test(e2e): check codex really gets the real model's own catalog entry before the round trip

tests/e2e/AGENTS.md

@@ -50,7 +50,7 @@ Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family
 - `gateway/` - proxy configuration only (`litellm-config.yml`). No tests
 - `claude_code/` - the Claude Code compatibility matrix: drives the real `claude` CLI (and HTTP probes) against a proxy for each feature x provider cell, reporting tagged-union outcomes via the `compat_result` fixture. Ships its own driver/builder/publisher, covered by `tests/e2e_harness/claude_code/`. The HTTP probes ride the shared transport (`ProxyClient.count_tokens` / `ProxyClient.messages`). The CLI-driving path stays bespoke. `pr_gate_version_resolver.py` only forwards to `cli_harness/npm_version_resolver.py`, because the Buildkite runner images still run that path and install @latest without it
 - `cli_harness/` - no tests, only the plumbing every suite that drives a real coding-agent CLI shares: `npm_version_resolver.py` picks the CLI release to install (the newest npm release at least three days old, run by CI as `python tests/e2e/cli_harness/npm_version_resolver.py <npm package>`, stdlib only), `lockdown.py` gives the CLI subprocess an allowlisted env and a fresh empty HOME so it never sees the runner's credentials or dotfiles, and `rate_limiter.py` is the cross-process token bucket per upstream that keeps parallel CLI runs under one request rate. Covered by `tests/e2e_harness/cli_harness/`. Code only one CLI suite needs stays in that suite
-- `codex/` - the Codex compatibility suite: drives the real `codex exec --json` CLI through the gateway, one class per row (the command round trip, a file edit with the patch tool, a long session that follows a chain of files one command at a time, auto-compaction, where Codex reads a random value, compacts through the gateway and is asked for the value after the file is deleted, and the round trip plus auto-compaction again with the deployment named after the real model, so Codex's `/v1/models` catalog gives it the provider's full entry instead of the generic one), each row parametrized over `SETUPS` in `setups.py`, one `pytest.param` per setup (GPT on Bedrock Mantle, Bedrock Runtime, Azure OpenAI and OpenAI, plus Claude on Bedrock). A row with one setup broken by a known bug parametrizes over `with_expected_failure(<setup id>, reason=...)`, which is `SETUPS` with a strict xfail on that one setup. The `setup` fixture registers that setup's deployment under a fresh name via `/model/new` with a key scoped to it, and the `codex` fixture points Codex at it. `real_name_setup` and `real_name_codex` do the same under the setup's `real_model_name` (`gpt-6.1-sol`, `claude-sonnet-5-5`) as a team model on a throwaway team with a team key, since LiteLLM pools deployments that share a name and parallel tests would otherwise share one. Everything is deleted on teardown. `harness.py` gives each test a throwaway HOME and CODEX_HOME whose `config.toml` points a custom `responses` provider at the gateway and has Codex read its model list from the gateway's `/v1/models` (`model_catalog_url` on the provider), with the sandbox on (`workspace-write`) and approvals off, and runs the CLI under `cli_harness`'s env lockdown and rate limiter. `run.py` reads the JSONL events into a `CodexRun`: the commands with their output, the patched files, the last agent message as `answer`, the usage and the errors, where an error's `status_code` is what lets the suite's 5xx rerun rule catch a turn Codex gave up on. `codex exec --json` prints nothing when Codex compacts, so `rollout.py` reads the compactions from the thread's rollout file under CODEX_HOME/sessions, and a compaction is remote when its replacement history kept the provider's `compaction` item. `exec` and `resume` take `compact_after_tokens`, which forces Codex's remote compaction (`compaction_trigger` items over `/v1/responses`, the path Codex otherwise only takes for OpenAI and Azure) on the gateway provider and compacts once the context grows past that many tokens. A test passes on a random value the model can only learn through the gateway, never on the model's wording or the exact command it typed. It needs `codex` on PATH and fails without it. Covered by `tests/e2e_harness/codex/`
+- `codex/` - the Codex compatibility suite: drives the real `codex exec --json` CLI through the gateway, one class per row (the command round trip, a file edit with the patch tool, a long session that follows a chain of files one command at a time, auto-compaction, where Codex reads a random value, compacts through the gateway and is asked for the value after the file is deleted, and the round trip plus auto-compaction again with the deployment named after the real model, so Codex's `/v1/models` catalog gives it the provider's full entry instead of the generic one, which the round trip first checks against the tool mode Codex bundles for that name), each row parametrized over `SETUPS` in `setups.py`, one `pytest.param` per setup (GPT on Bedrock Mantle, Bedrock Runtime, Azure OpenAI and OpenAI, plus Claude on Bedrock). A row with one setup broken by a known bug parametrizes over `with_expected_failure(<setup id>, reason=...)`, which is `SETUPS` with a strict xfail on that one setup. The `setup` fixture registers that setup's deployment under a fresh name via `/model/new` with a key scoped to it, and the `codex` fixture points Codex at it. `real_name_setup` and `real_name_codex` do the same under the setup's `real_model_name` (`gpt-6.1-sol`, `claude-sonnet-5-5`) as a team model on a throwaway team with a team key, since LiteLLM pools deployments that share a name and parallel tests would otherwise share one. Everything is deleted on teardown. `harness.py` gives each test a throwaway HOME and CODEX_HOME whose `config.toml` points a custom `responses` provider at the gateway and has Codex read its model list from the gateway's `/v1/models` (`model_catalog_url` on the provider), with the sandbox on (`workspace-write`) and approvals off, and runs the CLI under `cli_harness`'s env lockdown and rate limiter. `run.py` reads the JSONL events into a `CodexRun`: the commands with their output, the patched files, the last agent message as `answer`, the usage and the errors, where an error's `status_code` is what lets the suite's 5xx rerun rule catch a turn Codex gave up on. `codex exec --json` prints nothing when Codex compacts, so `rollout.py` reads the compactions from the thread's rollout file under CODEX_HOME/sessions, and a compaction is remote when its replacement history kept the provider's `compaction` item. `exec` and `resume` take `compact_after_tokens`, which forces Codex's remote compaction (`compaction_trigger` items over `/v1/responses`, the path Codex otherwise only takes for OpenAI and Azure) on the gateway provider and compacts once the context grows past that many tokens. A test passes on a random value the model can only learn through the gateway, never on the model's wording or the exact command it typed. It needs `codex` on PATH and fails without it. Covered by `tests/e2e_harness/codex/`
 - `ui/` - the Admin UI browser suite: Playwright in TypeScript, driving the dashboard served by a live proxy on port 4000 (seeded postgres + mock LLM upstream, see its `run_e2e.sh`). It is a self-contained npm package with its own lockfile and does not use the Python harness, pytest markers, or the shared transport. The Python rules in this file (typed models, `Result` unions, basedpyright zero-error gate) do not apply inside it. Its only Python file, `fixtures/mock_llm_server/server.py`, is excluded from the e2e basedpyright gate via the root `pyrightconfig.json`. Its specs follow the dashboard's rule against testing what shadcn and Base UI already guarantee, in `ui/litellm-dashboard/AGENTS.md`
 
 ## MCP suite: real Datadog only

tests/e2e/codex/harness.py

@@ -7,6 +7,7 @@ from contextlib import contextmanager
 from dataclasses import dataclass, field
 from itertools import chain
 from pathlib import Path
+from types import MappingProxyType
 from typing import Final
 
 from cli_harness.lockdown import isolated_home, locked_down_env
@@ -14,6 +15,7 @@ from cli_harness.rate_limiter import RateBucket, RateLimiter
 from codex.rollout import Compaction, rollout_compactions
 from codex.run import CodexRun, codex_run
 from e2e_metadata import step
+from pydantic import BaseModel, ConfigDict
 
 CODEX_CLI: Final = "codex"
 EXEC_TIMEOUT_SECONDS: Final = 600.0
@@ -37,6 +39,19 @@ model_catalog_url = "{gateway}/v1/models"
 """
 
 
+class _CatalogModel(BaseModel):
+    model_config = ConfigDict(frozen=True)
+
+    slug: str
+    tool_mode: str | None = None
+
+
+class _Catalog(BaseModel):
+    model_config = ConfigDict(frozen=True)
+
+    models: tuple[_CatalogModel, ...]
+
+
 @dataclass(frozen=True, slots=True)
 class CodexHarness:
     home: Path
@@ -75,14 +90,32 @@ class CodexHarness:
         rollouts: Final = sorted(self.codex_home.glob(f"sessions/**/rollout-*-{thread_id}.jsonl"))
         return tuple(chain.from_iterable(rollout_compactions(rollout.read_text()) for rollout in rollouts))
 
+    @step("Read the tool mode Codex gives each model in its catalog, bundled={bundled}")
+    def tool_modes(self, *, bundled: bool = False) -> Mapping[str, str | None]:
+        completed: Final = subprocess.run(
+            (self.cli, "debug", "models", *(("--bundled",) if bundled else ())),
+            cwd=self.home,
+            env=self._env(),
+            stdin=subprocess.DEVNULL,
+            capture_output=True,
+            text=True,
+            timeout=self.timeout_seconds,
+            check=True,
+        )
+        catalog: Final = _Catalog.model_validate_json(completed.stdout)
+        return MappingProxyType({model.slug: model.tool_mode for model in catalog.models})
+
+    def _env(self) -> Mapping[str, str]:
+        return locked_down_env(
+            home=self.home, overrides={"CODEX_HOME": str(self.codex_home), _API_KEY_ENV: self.api_key}
+        )
+
     def _run(self, subcommand: tuple[str, ...], args: tuple[str, ...], cwd: Path) -> CodexRun:
         _ = self.limiter.acquire(self.rate_bucket)
         completed: Final = subprocess.run(
             (self.cli, *subcommand, "--json", "--skip-git-repo-check", *args),
             cwd=cwd,
-            env=locked_down_env(
-                home=self.home, overrides={"CODEX_HOME": str(self.codex_home), _API_KEY_ENV: self.api_key}
-            ),
+            env=self._env(),
             stdin=subprocess.DEVNULL,
             capture_output=True,
             text=True,

tests/e2e/codex/test_real_model_name_e2e.py

@@ -18,6 +18,9 @@ class TestCodexUnderTheRealModelName:
     def test_answers_with_what_a_command_printed(
         self, real_name_codex: CodexHarness, real_name_setup: ModelSetup
     ) -> None:
+        served_by_gateway: Final = real_name_codex.tool_modes()
+        bundled_with_codex: Final = real_name_codex.tool_modes(bundled=True)
+        assert served_by_gateway[real_name_setup.model] == bundled_with_codex.get(real_name_setup.model)
         marker: Final = unique_marker()
         workspace: Final = real_name_codex.new_workspace(files={"secret.txt": marker})
         run: Final = real_name_codex.exec(