test(e2e): declare both models a codex model switch drives in its test metadata

tests/e2e/AGENTS.md

@@ -50,7 +50,7 @@ Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family
 - `gateway/` - proxy configuration only (`litellm-config.yml`). No tests
 - `claude_code/` - the Claude Code compatibility matrix: drives the real `claude` CLI (and HTTP probes) against a proxy for each feature x provider cell, reporting tagged-union outcomes via the `compat_result` fixture. Ships its own driver/builder/publisher, covered by `tests/e2e_harness/claude_code/`. The HTTP probes ride the shared transport (`ProxyClient.count_tokens` / `ProxyClient.messages`). The CLI-driving path stays bespoke. `pr_gate_version_resolver.py` only forwards to `cli_harness/npm_version_resolver.py`, because the Buildkite runner images still run that path and install @latest without it
 - `cli_harness/` - no tests, only the plumbing every suite that drives a real coding-agent CLI shares: `npm_version_resolver.py` picks the CLI release to install (the newest npm release at least three days old, run by CI as `python tests/e2e/cli_harness/npm_version_resolver.py <npm package>`, stdlib only), `lockdown.py` gives the CLI subprocess an allowlisted env and a fresh empty HOME so it never sees the runner's credentials or dotfiles, and `rate_limiter.py` is the cross-process token bucket per upstream that keeps parallel CLI runs under one request rate. Covered by `tests/e2e_harness/cli_harness/`. Code only one CLI suite needs stays in that suite
-- `codex/` - the Codex compatibility suite: drives the real `codex exec --json` CLI through the gateway, one class per row (the command round trip, a file edit with the patch tool, a long session that follows a chain of files one command at a time, auto-compaction, where Codex reads a random value, compacts through the gateway and is asked for the value after the file is deleted, and a model switch, where Codex reasons on the setup's model, then resumes the thread on a second model and is asked for the value after the file is deleted), each row parametrized over `SETUPS` in `setups.py`, one `pytest.param` per setup (GPT on Bedrock Mantle, Bedrock Runtime, Azure OpenAI and OpenAI, plus Claude on Bedrock). A row with one setup broken by a known bug parametrizes over `with_expected_failure(<setup id>, reason=...)`, which is `SETUPS` with a strict xfail on that one setup. The `setup` fixture registers that setup's deployment under a fresh name via `/model/new` (through `register_model`, which a row uses for any extra model) and the `codex` fixture makes a key scoped to `key_models`, the setup's model unless a row overrides it, and both are deleted on teardown. `harness.py` gives each test a throwaway HOME and CODEX_HOME whose `config.toml` points a custom `responses` provider at the gateway and has Codex read its model list from the gateway's `/v1/models` (`model_catalog_url` on the provider), with the sandbox on (`workspace-write`) and approvals off, and runs the CLI under `cli_harness`'s env lockdown and rate limiter. `run.py` reads the JSONL events into a `CodexRun`: the commands with their output, the patched files, the last agent message as `answer`, the usage and the errors, where an error's `status_code` is what lets the suite's 5xx rerun rule catch a turn Codex gave up on. `codex exec --json` prints nothing when Codex compacts or about the model a turn ran on, so `rollout.py` reads the compactions and each turn's model from the thread's rollout file under CODEX_HOME/sessions, and a compaction is remote when its replacement history kept the provider's `compaction` item. `exec` and `resume` take `compact_after_tokens`, which forces Codex's remote compaction (`compaction_trigger` items over `/v1/responses`, the path Codex otherwise only takes for OpenAI and Azure) on the gateway provider and compacts once the context grows past that many tokens, and `reasoning_effort`, which sets Codex's `model_reasoning_effort` so the model reasons and the history carries its reasoning. A test passes on a random value the model can only learn through the gateway, never on the model's wording or the exact command it typed. It needs `codex` on PATH and fails without it. Covered by `tests/e2e_harness/codex/`
+- `codex/` - the Codex compatibility suite: drives the real `codex exec --json` CLI through the gateway, one class per row (the command round trip, a file edit with the patch tool, a long session that follows a chain of files one command at a time, auto-compaction, where Codex reads a random value, compacts through the gateway and is asked for the value after the file is deleted, and a model switch, where Codex reasons on the setup's model, then resumes the thread on a second model and is asked for the value after the file is deleted), each row parametrized over `SETUPS` in `setups.py`, one `pytest.param` per setup (GPT on Bedrock Mantle, Bedrock Runtime, Azure OpenAI and OpenAI, plus Claude on Bedrock). A row with one setup broken by a known bug parametrizes over `with_expected_failure(<setup id>, reason=...)`, which is `SETUPS` with a strict xfail on that one setup, and a row that drives a second model wraps its setups in `also_driving`, which adds that model and its provider to each setup's `Subject`. The `setup` fixture registers that setup's deployment under a fresh name via `/model/new` (through `register_model`, which a row uses for any extra model) and the `codex` fixture makes a key scoped to `key_models`, the setup's model unless a row overrides it, and both are deleted on teardown. `harness.py` gives each test a throwaway HOME and CODEX_HOME whose `config.toml` points a custom `responses` provider at the gateway and has Codex read its model list from the gateway's `/v1/models` (`model_catalog_url` on the provider), with the sandbox on (`workspace-write`) and approvals off, and runs the CLI under `cli_harness`'s env lockdown and rate limiter. `run.py` reads the JSONL events into a `CodexRun`: the commands with their output, the patched files, the last agent message as `answer`, the usage and the errors, where an error's `status_code` is what lets the suite's 5xx rerun rule catch a turn Codex gave up on. `codex exec --json` prints nothing when Codex compacts or about the model a turn ran on, so `rollout.py` reads the compactions and each turn's model from the thread's rollout file under CODEX_HOME/sessions, and a compaction is remote when its replacement history kept the provider's `compaction` item. `exec` and `resume` take `compact_after_tokens`, which forces Codex's remote compaction (`compaction_trigger` items over `/v1/responses`, the path Codex otherwise only takes for OpenAI and Azure) on the gateway provider and compacts once the context grows past that many tokens, and `reasoning_effort`, which sets Codex's `model_reasoning_effort` so the model reasons and the history carries its reasoning. A test passes on a random value the model can only learn through the gateway, never on the model's wording or the exact command it typed. It needs `codex` on PATH and fails without it. Covered by `tests/e2e_harness/codex/`
 - `ui/` - the Admin UI browser suite: Playwright in TypeScript, driving the dashboard served by a live proxy on port 4000 (seeded postgres + mock LLM upstream, see its `run_e2e.sh`). It is a self-contained npm package with its own lockfile and does not use the Python harness, pytest markers, or the shared transport. The Python rules in this file (typed models, `Result` unions, basedpyright zero-error gate) do not apply inside it. Its only Python file, `fixtures/mock_llm_server/server.py`, is excluded from the e2e basedpyright gate via the root `pyrightconfig.json`. Its specs follow the dashboard's rule against testing what shadcn and Base UI already guarantee, in `ui/litellm-dashboard/AGENTS.md`
 
 ## MCP suite: real Datadog only

tests/e2e/codex/setups.py

@@ -3,7 +3,8 @@
 
 from __future__ import annotations
 
-from dataclasses import dataclass
+from collections.abc import Mapping
+from dataclasses import dataclass, replace
 from typing import Final, Literal
 
 import pytest
@@ -94,3 +95,23 @@ def with_expected_failure(setup_id: SetupId, *, reason: str) -> tuple[ParameterS
         else setup
         for setup in SETUPS
     )
+
+
+def also_driving(setups: tuple[ParameterSet, ...], extra: Mapping[str, Subject]) -> tuple[ParameterSet, ...]:
+    return tuple(_also_driving(setup, extra) for setup in setups)
+
+
+def _also_driving(setup: ParameterSet, extra: Mapping[str, Subject]) -> ParameterSet:
+    deployment: Final = setup.values[0]
+    assert isinstance(deployment, Deployment)
+    added: Final = extra[deployment.litellm_params.model]
+    return pytest.param(*setup.values, id=setup.id, marks=tuple(_widened(mark, added) for mark in setup.marks))
+
+
+def _widened(mark: pytest.MarkDecorator | pytest.Mark, added: Subject) -> pytest.MarkDecorator | pytest.Mark:
+    subject: Final = mark.args[0] if mark.name == "meta" and mark.args else None
+    if not isinstance(subject, Subject):
+        return mark
+    return meta(
+        replace(subject, providers=(*subject.providers, *added.providers), models=(*subject.models, *added.models))
+    )

tests/e2e/codex/test_switch_model_e2e.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 from collections.abc import Callable
+from dataclasses import dataclass
 from types import MappingProxyType
 from typing import Final
 
@@ -17,24 +18,43 @@ from codex.setups import (
     GPT_ON_OPENAI,
     Deployment,
     ModelSetup,
+    also_driving,
     with_expected_failure,
 )
 from e2e_config import unique_marker
+from e2e_metadata import Provider, Subject
 from models import LiteLLMParamsBody
 
 pytestmark = pytest.mark.e2e
 
-OTHER_GPT_ON_OPENAI: Final = LiteLLMParamsBody(model="openai/gpt-6.1-terra", api_key="os.environ/OPENAI_API_KEY")
+OTHER_GPT_ON_BEDROCK_MANTLE: Final = "bedrock_mantle/openai.gpt-6-luna"
+OTHER_GPT_ON_BEDROCK_RUNTIME: Final = "bedrock/us.openai.gpt-6-luna"
+OTHER_GPT_ON_OPENAI: Final = "openai/gpt-6.1-terra"
+OTHER_CLAUDE_ON_BEDROCK: Final = "bedrock/us.anthropic.claude-haiku-5-5"
+
+
+@dataclass(frozen=True, slots=True)
+class SwitchTo:
+    provider: Provider
+    litellm_params: LiteLLMParamsBody
+
+
+TO_OTHER_GPT_ON_OPENAI: Final = SwitchTo(
+    Provider.OPENAI, LiteLLMParamsBody(model=OTHER_GPT_ON_OPENAI, api_key="os.environ/OPENAI_API_KEY")
+)
 SWITCH_TO: Final = MappingProxyType(
     {
-        GPT_ON_BEDROCK_MANTLE: LiteLLMParamsBody(
-            model="bedrock_mantle/openai.gpt-6-luna", aws_region_name=BEDROCK_REGION
+        GPT_ON_BEDROCK_MANTLE: SwitchTo(
+            Provider.BEDROCK_MANTLE,
+            LiteLLMParamsBody(model=OTHER_GPT_ON_BEDROCK_MANTLE, aws_region_name=BEDROCK_REGION),
         ),
-        GPT_ON_BEDROCK_RUNTIME: LiteLLMParamsBody(model="bedrock/us.openai.gpt-6-luna", aws_region_name=BEDROCK_REGION),
-        GPT_ON_AZURE_OPENAI: OTHER_GPT_ON_OPENAI,
-        GPT_ON_OPENAI: OTHER_GPT_ON_OPENAI,
-        CLAUDE_ON_BEDROCK: LiteLLMParamsBody(
-            model="bedrock/us.anthropic.claude-haiku-5-5", aws_region_name=BEDROCK_REGION
+        GPT_ON_BEDROCK_RUNTIME: SwitchTo(
+            Provider.BEDROCK, LiteLLMParamsBody(model=OTHER_GPT_ON_BEDROCK_RUNTIME, aws_region_name=BEDROCK_REGION)
+        ),
+        GPT_ON_AZURE_OPENAI: TO_OTHER_GPT_ON_OPENAI,
+        GPT_ON_OPENAI: TO_OTHER_GPT_ON_OPENAI,
+        CLAUDE_ON_BEDROCK: SwitchTo(
+            Provider.BEDROCK, LiteLLMParamsBody(model=OTHER_CLAUDE_ON_BEDROCK, aws_region_name=BEDROCK_REGION)
         ),
     }
 )
@@ -42,7 +62,7 @@ SWITCH_TO: Final = MappingProxyType(
 
 @pytest.fixture
 def second_model(deployment: Deployment, register_model: Callable[[LiteLLMParamsBody], str]) -> str:
-    return register_model(SWITCH_TO[deployment.litellm_params.model])
+    return register_model(SWITCH_TO[deployment.litellm_params.model].litellm_params)
 
 
 @pytest.fixture
@@ -52,10 +72,18 @@ def key_models(setup: ModelSetup, second_model: str) -> tuple[str, ...]:
 
 @pytest.mark.parametrize(
     "deployment",
-    with_expected_failure(
-        "azure_openai",
-        reason="LiteLLM forwards the reasoning Azure encrypted to OpenAI, which cannot decrypt another account's"
-        " reasoning and rejects the turn with invalid_encrypted_content (#41792)",
+    also_driving(
+        with_expected_failure(
+            "azure_openai",
+            reason="LiteLLM forwards the reasoning Azure encrypted to OpenAI, which cannot decrypt another account's"
+            " reasoning and rejects the turn with invalid_encrypted_content (#41792)",
+        ),
+        MappingProxyType(
+            {
+                model: Subject(providers=(to.provider,), models=(to.litellm_params.model,))
+                for model, to in SWITCH_TO.items()
+            }
+        ),
     ),
 )
 class TestCodexSwitchModel:

tests/e2e_harness/AGENTS.md

@@ -2,7 +2,7 @@
 
 Tests of the harness under `tests/e2e/` (the transport, the clients, the fixture bundle and replay edge, the stack lock, the IdP launcher, the coverage collector, the JUnit properties and the Claude Code driver), not of the product. They live outside `tests/e2e/` because the Buildkite e2e run copies that folder into the runner image and runs every test in it, so a harness test in there counts as a product test in the nightly numbers. Nothing here needs a proxy, provider keys or the network
 
-The layout mirrors `tests/e2e/`: `test_e2e_http.py` covers `tests/e2e/e2e_http.py`, `logging/test_datadog_reader.py` covers `tests/e2e/logging/datadog_reader.py`, `claude_code/` covers the driver, builder and probe, `cli_harness/` covers the version resolver, the CLI env lockdown and the rate limiter, and `codex/` covers how the Codex driver reads the CLI's JSONL events and the compactions in a thread's rollout file. Put a new harness test under the folder that mirrors the suite folder whose module it covers
+The layout mirrors `tests/e2e/`: `test_e2e_http.py` covers `tests/e2e/e2e_http.py`, `logging/test_datadog_reader.py` covers `tests/e2e/logging/datadog_reader.py`, `claude_code/` covers the driver, builder and probe, `cli_harness/` covers the version resolver, the CLI env lockdown and the rate limiter, and `codex/` covers how the Codex driver reads the CLI's JSONL events and a thread's rollout file, and how a row adds a second model to its setups' metadata. Put a new harness test under the folder that mirrors the suite folder whose module it covers
 
 Run them from the repo root. `e2e_config` reads `LITELLM_MASTER_KEY` at import and any value will do, the CI lane sets a dummy:
 

tests/e2e_harness/codex/test_codex_setups.py

@@ -0,0 +1,42 @@
+from __future__ import annotations
+
+from types import MappingProxyType
+from typing import Final
+
+from _pytest.mark.structures import ParameterSet
+from codex.setups import SETUPS, Deployment, also_driving, with_expected_failure
+from e2e_metadata import Provider, Subject
+
+SECOND_MODEL: Final = "openai/second-model"
+
+
+def _subject(setup: ParameterSet) -> Subject:
+    (meta,) = (mark for mark in setup.marks if mark.name == "meta")
+    return _only_subject(meta.args)
+
+
+def _only_subject(args: tuple[object, ...]) -> Subject:
+    (subject,) = args
+    assert isinstance(subject, Subject)
+    return subject
+
+
+def _model(setup: ParameterSet) -> str:
+    deployment: Final = setup.values[0]
+    assert isinstance(deployment, Deployment)
+    return deployment.litellm_params.model
+
+
+def test_also_driving_declares_the_extra_model_on_every_setup_and_keeps_the_xfail() -> None:
+    setups: Final = also_driving(
+        with_expected_failure("azure_openai", reason="known bug"),
+        MappingProxyType(
+            {_model(setup): Subject(providers=(Provider.OPENAI,), models=(SECOND_MODEL,)) for setup in SETUPS}
+        ),
+    )
+
+    assert tuple(setup.id for setup in setups) == tuple(setup.id for setup in SETUPS)
+    assert all(_subject(setup).models == tuple(sorted((_model(setup), SECOND_MODEL))) for setup in setups)
+    azure: Final = next(setup for setup in setups if setup.id == "azure_openai")
+    assert _subject(azure).providers == (Provider.AZURE, Provider.OPENAI)
+    assert any(mark.name == "xfail" for mark in azure.marks)