test(e2e): drive the real Codex CLI through the gateway and add the command round trip row

Adds tests/e2e/codex: a harness that runs codex exec --json with a throwaway HOME and CODEX_HOME, a custom responses provider pointed at the gateway, the model list read from the gateway's /v1/models, the sandbox on and approvals off. Each test registers its setup's deployment via /model/new and calls it with a key scoped to it, both deleted on teardown. The first row checks that a value only a command can print comes back through the gateway on all five setups

tests/e2e/AGENTS.md

@@ -50,6 +50,7 @@ Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family
 - `gateway/` - proxy configuration only (`litellm-config.yml`). No tests
 - `claude_code/` - the Claude Code compatibility matrix: drives the real `claude` CLI (and HTTP probes) against a proxy for each feature x provider cell, reporting tagged-union outcomes via the `compat_result` fixture. Ships its own driver/builder/publisher, covered by `tests/e2e_harness/claude_code/`. The HTTP probes ride the shared transport (`ProxyClient.count_tokens` / `ProxyClient.messages`). The CLI-driving path stays bespoke. `pr_gate_version_resolver.py` only forwards to `cli_harness/npm_version_resolver.py`, because the Buildkite runner images still run that path and install @latest without it
 - `cli_harness/` - no tests, only the plumbing every suite that drives a real coding-agent CLI shares: `npm_version_resolver.py` picks the CLI release to install (the newest npm release at least three days old, run by CI as `python tests/e2e/cli_harness/npm_version_resolver.py <npm package>`, stdlib only), `lockdown.py` gives the CLI subprocess an allowlisted env and a fresh empty HOME so it never sees the runner's credentials or dotfiles, and `rate_limiter.py` is the cross-process token bucket per upstream that keeps parallel CLI runs under one request rate. Covered by `tests/e2e_harness/cli_harness/`. Code only one CLI suite needs stays in that suite
+- `codex/` - the Codex compatibility suite: drives the real `codex exec --json` CLI through the gateway, one class per row (the command round trip so far), each row parametrized over `SETUPS` in `setups.py`, one `pytest.param` per setup (GPT on Bedrock Mantle, Bedrock Runtime, Azure OpenAI and OpenAI, plus Claude on Bedrock). The `setup` fixture registers that setup's deployment under a fresh name via `/model/new` and the `codex` fixture makes a key scoped to it, and both are deleted on teardown. `harness.py` gives each test a throwaway HOME and CODEX_HOME whose `config.toml` points a custom `responses` provider at the gateway and has Codex read its model list from the gateway's `/v1/models` (`model_catalog_url` on the provider), with the sandbox on (`workspace-write`) and approvals off, and runs the CLI under `cli_harness`'s env lockdown and rate limiter. `run.py` reads the JSONL events into a `CodexRun`: the commands with their output, the patched files, the last agent message as `answer`, the usage and the errors, where an error's `status_code` is what lets the suite's 5xx rerun rule catch a turn Codex gave up on. A test passes on a random value the model can only learn through the gateway, never on the model's wording or the exact command it typed. It needs `codex` on PATH and fails without it. Covered by `tests/e2e_harness/codex/`
 - `ui/` - the Admin UI browser suite: Playwright in TypeScript, driving the dashboard served by a live proxy on port 4000 (seeded postgres + mock LLM upstream, see its `run_e2e.sh`). It is a self-contained npm package with its own lockfile and does not use the Python harness, pytest markers, or the shared transport. The Python rules in this file (typed models, `Result` unions, basedpyright zero-error gate) do not apply inside it. Its only Python file, `fixtures/mock_llm_server/server.py`, is excluded from the e2e basedpyright gate via the root `pyrightconfig.json`. Its specs follow the dashboard's rule against testing what shadcn and Base UI already guarantee, in `ui/litellm-dashboard/AGENTS.md`
 
 ## MCP suite: real Datadog only
@@ -325,5 +326,6 @@ other.<area>.<case>.<assertion>
 - spin up a local proxy by running the litellm proxy locally (`litellm --config <your-e2e-config>.yml --port 4000`, see CONTRIBUTING.md), make sure all tests pass. if a test fails due to an internally found issue, let users know to create a linear ticket for it. 
 
 - do not use xfail markers, tests should be written in a form that the end user expects it to pass
+- the `codex/` suite is the one exception: a single setup that a LiteLLM or provider bug breaks may carry `pytest.mark.xfail(strict=True, reason=...)` with a plain one-sentence reason on that setup's `pytest.param`, so it keeps running and goes red once the bug is fixed. Strict only, and never on a flaky setup (one that passes some runs and fails others)
 
 - a cell addresses the stack through its front door (`PROXY_BASE_URL`) the way a customer does, never a gateway pod by address, and proves a cross-replica property with N independent calls through that door, naming the miss odds (2^-N at two pods) in its docstring. `PROXY_REPLICA_URLS` is for read-backs that are per pod by nature (polling until a management write has converged on every replica, `/metrics`, RSS), never for steering the scenario a cell asserts on at a chosen pod

tests/e2e/codex/conftest.py

@@ -0,0 +1,30 @@
+from __future__ import annotations
+
+from collections.abc import Iterator
+from typing import Final
+
+import pytest
+from cli_harness.rate_limiter import get_default_limiter
+from codex.harness import CodexHarness, codex_harness
+from codex.setups import Deployment, ModelSetup
+from e2e_config import PROXY_BASE_URL, unique_marker
+from models import KeyGenerateBody
+from proxy_client import ProxyClient
+
+
+@pytest.fixture
+def setup(deployment: Deployment, proxy: ProxyClient) -> Iterator[ModelSetup]:
+    model: Final = f"e2e-codex-{unique_marker()}"
+    model_id: Final = proxy.create_model(model, deployment.litellm_params, provider_live=True)
+    yield ModelSetup(model=model, rate_bucket=deployment.rate_bucket)
+    proxy.delete_model(model_id)
+
+
+@pytest.fixture
+def codex(setup: ModelSetup, proxy: ProxyClient) -> Iterator[CodexHarness]:
+    key: Final = proxy.generate_key(KeyGenerateBody(models=[setup.model]))
+    with codex_harness(
+        gateway=PROXY_BASE_URL, api_key=key, rate_bucket=setup.rate_bucket, limiter=get_default_limiter()
+    ) as harness:
+        yield harness
+    proxy.delete_key(key)

tests/e2e/codex/harness.py

@@ -0,0 +1,84 @@
+from __future__ import annotations
+
+import subprocess
+import tempfile
+from collections.abc import Generator, Mapping
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Final
+
+from cli_harness.lockdown import isolated_home, locked_down_env
+from cli_harness.rate_limiter import RateBucket, RateLimiter
+from codex.run import CodexRun, codex_run
+from e2e_metadata import step
+
+CODEX_CLI: Final = "codex"
+EXEC_TIMEOUT_SECONDS: Final = 600.0
+_API_KEY_ENV: Final = "LITELLM_API_KEY"
+_CONFIG_TOML: Final = """\
+model_provider = "litellm"
+approval_policy = "never"
+sandbox_mode = "workspace-write"
+check_for_update_on_startup = false
+
+[analytics]
+enabled = false
+
+[model_providers.litellm]
+name = "LiteLLM"
+base_url = "{gateway}/v1"
+env_key = "{api_key_env}"
+wire_api = "responses"
+model_catalog_url = "{gateway}/v1/models"
+"""
+
+
+@dataclass(frozen=True, slots=True)
+class CodexHarness:
+    home: Path
+    api_key: str = field(repr=False)
+    rate_bucket: RateBucket
+    limiter: RateLimiter
+    cli: str = CODEX_CLI
+    timeout_seconds: float = EXEC_TIMEOUT_SECONDS
+
+    @property
+    def codex_home(self) -> Path:
+        return self.home / ".codex"
+
+    def new_workspace(self, files: Mapping[str, str]) -> Path:
+        workspace: Final = Path(tempfile.mkdtemp(prefix="workspace-", dir=self.home))
+        for name, content in files.items():
+            _ = (workspace / name).write_text(content)
+        return workspace
+
+    @step('Run Codex on {model} through the proxy with the prompt "{prompt}"')
+    def exec(self, prompt: str, *, model: str, cwd: Path) -> CodexRun:
+        _ = self.limiter.acquire(self.rate_bucket)
+        completed: Final = subprocess.run(
+            (self.cli, "exec", "--json", "--skip-git-repo-check", "--model", model, "--", prompt),
+            cwd=cwd,
+            env=locked_down_env(
+                home=self.home, overrides={"CODEX_HOME": str(self.codex_home), _API_KEY_ENV: self.api_key}
+            ),
+            stdin=subprocess.DEVNULL,
+            capture_output=True,
+            text=True,
+            timeout=self.timeout_seconds,
+            check=False,
+        )
+        return codex_run(exit_code=completed.returncode, stdout=completed.stdout, stderr=completed.stderr)
+
+
+@contextmanager
+def codex_harness(
+    *, gateway: str, api_key: str, rate_bucket: RateBucket, limiter: RateLimiter
+) -> Generator[CodexHarness]:
+    with isolated_home("codex-cli-home-") as home:
+        harness: Final = CodexHarness(home=home, api_key=api_key, rate_bucket=rate_bucket, limiter=limiter)
+        harness.codex_home.mkdir()
+        _ = (harness.codex_home / "config.toml").write_text(
+            _CONFIG_TOML.format(gateway=gateway, api_key_env=_API_KEY_ENV)
+        )
+        yield harness

tests/e2e/codex/run.py

@@ -0,0 +1,187 @@
+"""What one `codex exec --json` run did, read from the JSONL events it printed.
+
+The event shapes are Codex's own (`codex-rs/exec/src/exec_events.rs`). Codex is unpinned in CI, so an event or item
+type this module does not model is skipped, while one it does model is validated strictly and fails loudly when
+Codex changes it.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Callable, Iterator, Mapping
+from dataclasses import dataclass
+from itertools import chain
+from pathlib import Path
+from typing import Annotated, Final, Literal, cast
+
+from pydantic import BaseModel, ConfigDict, Discriminator, Tag, TypeAdapter
+
+_HTTP_STATUS: Final = re.compile(r"\bstatus:? (\d{3})\b")
+
+
+class _Frozen(BaseModel):
+    model_config = ConfigDict(frozen=True)
+
+
+class TokenUsage(_Frozen):
+    input_tokens: int
+    cached_input_tokens: int
+    cache_write_input_tokens: int = 0
+    output_tokens: int
+    reasoning_output_tokens: int
+
+
+class _AgentMessage(_Frozen):
+    type: Literal["agent_message"]
+    text: str
+
+
+class _CommandExecution(_Frozen):
+    type: Literal["command_execution"]
+    command: str
+    aggregated_output: str
+    exit_code: int | None
+
+
+class _FileUpdate(_Frozen):
+    path: str
+    kind: Literal["add", "delete", "update"]
+
+
+class _FileChange(_Frozen):
+    type: Literal["file_change"]
+    changes: tuple[_FileUpdate, ...]
+    status: Literal["in_progress", "completed", "failed"]
+
+
+class _ErrorItem(_Frozen):
+    type: Literal["error"]
+    message: str
+
+
+class _Unmodelled(_Frozen):
+    type: str
+
+
+class _Message(_Frozen):
+    message: str
+
+
+def _known_or_unmodelled(known: frozenset[str]) -> Callable[[object], str]:
+    def tag(raw: object) -> str:
+        kind: Final = cast("Mapping[str, object]", raw).get("type") if isinstance(raw, Mapping) else None
+        return kind if isinstance(kind, str) and kind in known else "unmodelled"
+
+    return tag
+
+
+_Item = Annotated[
+    Annotated[_AgentMessage, Tag("agent_message")]
+    | Annotated[_CommandExecution, Tag("command_execution")]
+    | Annotated[_FileChange, Tag("file_change")]
+    | Annotated[_ErrorItem, Tag("error")]
+    | Annotated[_Unmodelled, Tag("unmodelled")],
+    Discriminator(_known_or_unmodelled(frozenset({"agent_message", "command_execution", "file_change", "error"}))),
+]
+
+
+class _ThreadStarted(_Frozen):
+    type: Literal["thread.started"]
+    thread_id: str
+
+
+class _ItemCompleted(_Frozen):
+    type: Literal["item.completed"]
+    item: _Item
+
+
+class _TurnCompleted(_Frozen):
+    type: Literal["turn.completed"]
+    usage: TokenUsage
+
+
+class _TurnFailed(_Frozen):
+    type: Literal["turn.failed"]
+    error: _Message
+
+
+class _StreamError(_Frozen):
+    type: Literal["error"]
+    message: str
+
+
+_Event = Annotated[
+    Annotated[_ThreadStarted, Tag("thread.started")]
+    | Annotated[_ItemCompleted, Tag("item.completed")]
+    | Annotated[_TurnCompleted, Tag("turn.completed")]
+    | Annotated[_TurnFailed, Tag("turn.failed")]
+    | Annotated[_StreamError, Tag("error")]
+    | Annotated[_Unmodelled, Tag("unmodelled")],
+    Discriminator(
+        _known_or_unmodelled(frozenset({"thread.started", "item.completed", "turn.completed", "turn.failed", "error"}))
+    ),
+]
+_EVENT: Final = TypeAdapter[_Event](_Event)
+
+
+@dataclass(frozen=True, slots=True)
+class Command:
+    command: str
+    output: str
+    exit_code: int | None
+
+
+@dataclass(frozen=True, slots=True)
+class CodexError:
+    message: str
+    status_code: int | None
+
+
+@dataclass(frozen=True, slots=True)
+class CodexRun:
+    exit_code: int
+    thread_id: str | None
+    commands: tuple[Command, ...]
+    patched_files: tuple[Path, ...]
+    answer: str
+    usage: TokenUsage | None
+    errors: tuple[CodexError, ...]
+    stderr: str
+
+
+def codex_run(*, exit_code: int, stdout: str, stderr: str) -> CodexRun:
+    events: Final = tuple(_EVENT.validate_json(line) for line in stdout.splitlines() if line.strip())
+    items: Final = tuple(event.item for event in events if isinstance(event, _ItemCompleted))
+    patches: Final = tuple(item for item in items if isinstance(item, _FileChange) and item.status == "completed")
+    answers: Final = tuple(item.text for item in items if isinstance(item, _AgentMessage))
+    usages: Final = tuple(event.usage for event in events if isinstance(event, _TurnCompleted))
+    return CodexRun(
+        exit_code=exit_code,
+        thread_id=next((event.thread_id for event in events if isinstance(event, _ThreadStarted)), None),
+        commands=tuple(
+            Command(command=item.command, output=item.aggregated_output, exit_code=item.exit_code)
+            for item in items
+            if isinstance(item, _CommandExecution)
+        ),
+        patched_files=tuple(Path(change.path) for change in chain.from_iterable(patch.changes for patch in patches)),
+        answer=answers[-1] if answers else "",
+        usage=usages[-1] if usages else None,
+        errors=tuple(_errors(events)),
+        stderr=stderr,
+    )
+
+
+def _errors(events: tuple[_Event, ...]) -> Iterator[CodexError]:
+    for event in events:
+        match event:
+            case _TurnFailed(error=_Message(message=message)) | _StreamError(message=message):
+                yield _codex_error(message)
+            case _ItemCompleted(item=_ErrorItem(message=message)):
+                yield _codex_error(message)
+            case _:
+                pass
+
+
+def _codex_error(message: str) -> CodexError:
+    status: Final = _HTTP_STATUS.search(message)
+    return CodexError(message=message, status_code=int(status.group(1)) if status else None)

tests/e2e/codex/setups.py

@@ -0,0 +1,84 @@
+"""The credentials are `os.environ/` references the gateway resolves, the same names the Claude Code matrix's
+`test_config.yaml` uses, and Bedrock rides the gateway's default AWS identity."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Final
+
+import pytest
+from cli_harness.rate_limiter import RateBucket
+from e2e_metadata import Domain, Mode, Provider, Route, Subject, meta
+from models import LiteLLMParamsBody
+
+GPT_ON_BEDROCK_MANTLE: Final = "bedrock_mantle/openai.gpt-6.1-sol"
+GPT_ON_BEDROCK_RUNTIME: Final = "bedrock/us.openai.gpt-6.1-sol"
+GPT_ON_AZURE_OPENAI: Final = "azure/gpt-6.1-sol"
+GPT_ON_OPENAI: Final = "openai/gpt-6.1-sol"
+CLAUDE_ON_BEDROCK: Final = "bedrock/us.anthropic.claude-sonnet-5-5"
+_BEDROCK_REGION: Final = "us-east-1"
+
+
+@dataclass(frozen=True, slots=True)
+class Deployment:
+    litellm_params: LiteLLMParamsBody
+    rate_bucket: RateBucket
+
+
+@dataclass(frozen=True, slots=True)
+class ModelSetup:
+    model: str
+    rate_bucket: RateBucket
+
+
+def _drives(provider: Provider, model: str) -> pytest.MarkDecorator:
+    return meta(
+        Subject(
+            domain=Domain.LLM_TRANSLATION,
+            route=Route.RESPONSES,
+            providers=(provider,),
+            models=(model,),
+            mode=Mode.STREAM,
+        )
+    )
+
+
+SETUPS: Final = (
+    pytest.param(
+        Deployment(
+            LiteLLMParamsBody(model=GPT_ON_BEDROCK_MANTLE, aws_region_name=_BEDROCK_REGION), RateBucket.BEDROCK_MANTLE
+        ),
+        id="bedrock_mantle",
+        marks=_drives(Provider.BEDROCK_MANTLE, GPT_ON_BEDROCK_MANTLE),
+    ),
+    pytest.param(
+        Deployment(
+            LiteLLMParamsBody(model=GPT_ON_BEDROCK_RUNTIME, aws_region_name=_BEDROCK_REGION),
+            RateBucket.BEDROCK_CONVERSE,
+        ),
+        id="bedrock_runtime",
+        marks=_drives(Provider.BEDROCK, GPT_ON_BEDROCK_RUNTIME),
+    ),
+    pytest.param(
+        Deployment(
+            LiteLLMParamsBody(
+                model=GPT_ON_AZURE_OPENAI, api_base="os.environ/AZURE_API_BASE", api_key="os.environ/AZURE_API_KEY"
+            ),
+            RateBucket.AZURE_OPENAI,
+        ),
+        id="azure_openai",
+        marks=_drives(Provider.AZURE, GPT_ON_AZURE_OPENAI),
+    ),
+    pytest.param(
+        Deployment(LiteLLMParamsBody(model=GPT_ON_OPENAI, api_key="os.environ/OPENAI_API_KEY"), RateBucket.OPENAI),
+        id="openai",
+        marks=_drives(Provider.OPENAI, GPT_ON_OPENAI),
+    ),
+    pytest.param(
+        Deployment(
+            LiteLLMParamsBody(model=CLAUDE_ON_BEDROCK, aws_region_name=_BEDROCK_REGION), RateBucket.BEDROCK_CONVERSE
+        ),
+        id="claude_bedrock",
+        marks=_drives(Provider.BEDROCK, CLAUDE_ON_BEDROCK),
+    ),
+)

tests/e2e/codex/test_command_round_trip_e2e.py

@@ -0,0 +1,22 @@
+from __future__ import annotations
+
+from typing import Final
+
+import pytest
+from codex.harness import CodexHarness
+from codex.setups import SETUPS, ModelSetup
+from e2e_config import unique_marker
+
+pytestmark = pytest.mark.e2e
+
+
+@pytest.mark.parametrize("deployment", SETUPS)
+class TestCodexCommandRoundTrip:
+    def test_answers_with_what_a_command_printed(self, codex: CodexHarness, setup: ModelSetup) -> None:
+        marker: Final = unique_marker()
+        workspace: Final = codex.new_workspace(files={"secret.txt": marker})
+        run: Final = codex.exec(
+            "Run `cat secret.txt`, then reply with exactly what it printed.", model=setup.model, cwd=workspace
+        )
+        assert any(marker in command.output for command in run.commands), run
+        assert marker in run.answer, run

tests/e2e_harness/AGENTS.md

@@ -2,7 +2,7 @@
 
 Tests of the harness under `tests/e2e/` (the transport, the clients, the fixture bundle and replay edge, the stack lock, the IdP launcher, the coverage collector, the JUnit properties and the Claude Code driver), not of the product. They live outside `tests/e2e/` because the Buildkite e2e run copies that folder into the runner image and runs every test in it, so a harness test in there counts as a product test in the nightly numbers. Nothing here needs a proxy, provider keys or the network
 
-The layout mirrors `tests/e2e/`: `test_e2e_http.py` covers `tests/e2e/e2e_http.py`, `logging/test_datadog_reader.py` covers `tests/e2e/logging/datadog_reader.py`, `claude_code/` covers the driver, builder and probe, and `cli_harness/` covers the version resolver, the CLI env lockdown and the rate limiter. Put a new harness test under the folder that mirrors the suite folder whose module it covers
+The layout mirrors `tests/e2e/`: `test_e2e_http.py` covers `tests/e2e/e2e_http.py`, `logging/test_datadog_reader.py` covers `tests/e2e/logging/datadog_reader.py`, `claude_code/` covers the driver, builder and probe, `cli_harness/` covers the version resolver, the CLI env lockdown and the rate limiter, and `codex/` covers how the Codex driver reads the CLI's JSONL events. Put a new harness test under the folder that mirrors the suite folder whose module it covers
 
 Run them from the repo root. `e2e_config` reads `LITELLM_MASTER_KEY` at import and any value will do, the CI lane sets a dummy:
 

tests/e2e_harness/codex/test_codex_run.py

@@ -0,0 +1,91 @@
+from __future__ import annotations
+
+import configparser
+import json
+import re
+import shlex
+from pathlib import Path
+from typing import Final
+
+from codex.run import Command, codex_run
+
+E2E_PYTEST_INI: Final = Path(__file__).resolve().parents[2] / "e2e" / "pytest.ini"
+
+ROUND_TRIP: Final = "\n".join(
+    (
+        '{"type":"thread.started","thread_id":"01a18f9f-9d08-72e0-9be3-aa34cf7489d2"}',
+        '{"type":"turn.started"}',
+        '{"type":"item.completed","item":{"id":"item_0","type":"agent_message","text":"I will run it."}}',
+        '{"type":"item.started","item":{"id":"item_1","type":"command_execution","command":"/bin/zsh -lc \'cat'
+        ' secret.txt\'","aggregated_output":"","exit_code":null,"status":"in_progress"}}',
+        '{"type":"item.completed","item":{"id":"item_1","type":"command_execution","command":"/bin/zsh -lc \'cat'
+        ' secret.txt\'","aggregated_output":"marker-7f3a91\\n","exit_code":0,"status":"completed"}}',
+        '{"type":"item.completed","item":{"id":"item_2","type":"agent_message","text":"marker-7f3a91"}}',
+        '{"type":"turn.completed","usage":{"input_tokens":22547,"cached_input_tokens":11227,'
+        '"cache_write_input_tokens":11314,"output_tokens":67,"reasoning_output_tokens":0}}',
+    )
+)
+
+
+def _line(event: dict[str, object]) -> str:
+    return json.dumps(event)
+
+
+class TestCodexRun:
+    def test_the_answer_is_the_last_agent_message_and_each_finished_command_keeps_its_output(self) -> None:
+        run: Final = codex_run(exit_code=0, stdout=ROUND_TRIP, stderr="")
+
+        assert run.thread_id == "01a18f9f-9d08-72e0-9be3-aa34cf7489d2"
+        assert run.commands == (
+            Command(command="/bin/zsh -lc 'cat secret.txt'", output="marker-7f3a91\n", exit_code=0),
+        )
+        assert run.answer == "marker-7f3a91"
+        assert run.usage is not None and run.usage.input_tokens == 22547
+        assert run.errors == ()
+
+    def test_only_a_patch_that_applied_counts_as_patching_its_files(self) -> None:
+        def patch(status: str, path: str) -> str:
+            item: Final = {
+                "id": path,
+                "type": "file_change",
+                "status": status,
+                "changes": [{"path": path, "kind": "update"}],
+            }
+            return _line({"type": "item.completed", "item": item})
+
+        run: Final = codex_run(
+            exit_code=0, stdout="\n".join((patch("completed", "/w/app.py"), patch("failed", "/w/other.py"))), stderr=""
+        )
+
+        assert run.patched_files == (Path("/w/app.py"),)
+
+    def test_a_turn_that_gave_up_on_a_5xx_is_rerun_by_the_suites_network_rule(self) -> None:
+        failed: Final = _line(
+            {"type": "turn.failed", "error": {"message": "exceeded retry limit, last status: 502 Bad Gateway"}}
+        )
+
+        run: Final = codex_run(exit_code=1, stdout=failed, stderr="")
+
+        assert run.errors[0].status_code == 502
+        rerun_patterns: Final = _only_rerun_patterns(E2E_PYTEST_INI)
+        assert any(re.search(pattern, repr(run)) for pattern in rerun_patterns), rerun_patterns
+
+    def test_an_event_or_item_type_it_does_not_model_is_skipped(self) -> None:
+        newer_codex: Final = "\n".join(
+            (
+                _line({"type": "thread.goal_set", "goal": "ship it"}),
+                _line({"type": "item.completed", "item": {"id": "item_9", "type": "todo_list", "items": []}}),
+                ROUND_TRIP,
+            )
+        )
+
+        assert codex_run(exit_code=0, stdout=newer_codex, stderr="") == codex_run(
+            exit_code=0, stdout=ROUND_TRIP, stderr=""
+        )
+
+
+def _only_rerun_patterns(ini: Path) -> tuple[str, ...]:
+    parser: Final = configparser.ConfigParser()
+    _ = parser.read(ini)
+    args: Final = shlex.split(parser["pytest"]["addopts"])
+    return tuple(value for flag, value in zip(args, args[1:]) if flag == "--only-rerun")