Commit graph

2 commits

Author SHA1 Message Date
mateo-berri
fa992071f3 test(e2e): build a Codex compatibility table from tests/e2e/codex and add Vertex AI and five rows
Merge main and move the Codex compatibility work onto main's tests/e2e/codex
suite, replacing the parallel tests/e2e/client_apps suite and its harness tests.

A makereport hook in codex/compat_results.py appends one JSON line per decided
cell when CODEX_COMPAT_RESULTS_PATH is set, and codex/matrix_builder.py builds a
table in the Claude Code matrix shape and exits 3 when a cell that passed no
longer does. The row id comes from the test file name and the label from the
class name, so a new row needs no other edit.

The suite now pins @openai/codex 0.166.0 in its own package.json and lockfile,
adds Gemini on Vertex AI as a setup, and adds prompt caching, reasoning effort,
vision, structured output and web search rows. Web search carries strict xfails
where Codex or LiteLLM leaves the search tool out, and auto-compaction now also
expects Vertex AI to fail, for the same bridge gap Claude on Bedrock has
2026-11-03 03:10:05 -08:00
ryan-crabbe-berri
6254860560
test(e2e): drive the Codex CLI through LiteLLM, command round trip (#49374)
* test(e2e): drive the real Codex CLI through the gateway and add the command round trip row

Adds tests/e2e/codex: a harness that runs codex exec --json with a throwaway HOME and CODEX_HOME, a custom responses provider pointed at the gateway, the model list read from the gateway's /v1/models, the sandbox on and approvals off. Each test registers its setup's deployment via /model/new and calls it with a key scoped to it, both deleted on teardown. The first row checks that a value only a command can print comes back through the gateway on all five setups

* test(e2e): type the Codex self-test event helper as a Mapping

* test(e2e): clean up Codex test keys even when setup fails, and fail runs where Codex exits with an error

* test(e2e): record a step when the Codex harness sets up its home and workspace

The suite rules ask every public harness helper that does something to record a step, so a failed run shows whether it died while setting up Codex or while running it

* Update run.py

wip
2026-11-02 17:32:16 -08:00