Commit graph

3 commits

Author SHA1 Message Date
yuneng-jiang
e1c03c5423
fix(e2e): restore main qualification across providers and UI (#49763)
* test: check an unsupported budget sort field

* test: wait for dashboard navigation readiness

* test: support clustered provider caches with live fallback

* Revert "test: check an unsupported budget sort field"

This reverts commit 9694787bb5.

* test: check an unsupported budget sort field

* test(e2e): refresh retired provider fixtures

* test(e2e): read Langfuse observations through the supported API

* test(e2e): identify trace roots by kind and request path

* fix(e2e): repair UI readiness and regional AWS qualification

* fix(e2e): scope LangSmith batch readback to its workspace

* fix(e2e): use the configured global Bedrock GPT profile

* test(e2e): defer unprovisioned GovCloud coverage

* test(e2e): defer Bedrock Runtime Codex cases blocked by IAM

* test(e2e): defer intermittent Azure translation coverage
2026-11-03 20:38:47 +00:00
ryan-crabbe-berri
50af27eaa6
test(e2e): add Codex file edit, long session and auto-compaction rows through LiteLLM (#49386)
* test(e2e): add the Codex file edit and long session rows

The file edit row has Codex change a random value into config.txt with the patch tool and checks the file on disk. The long session row has it follow a chain of random file names one cat at a time. new_workspace now returns the resolved path, because Codex reports patched files by their real path (/private/var on macOS).

* test(e2e): make the Codex long session prove every earlier command output reached the model

* test(e2e): Codex auto-compaction through LiteLLM (#49389)

* test(e2e): add the Codex auto-compaction row

Codex reads a random value, compacts through the gateway with its remote compaction forced on, and is asked for the value after the file holding it is deleted. The row reads the compaction from the thread's rollout file and marks Claude on Bedrock as a strict expected failure, since the Responses-to-chat bridge ignores the compaction_trigger item

* refactor(e2e): validate Codex rollout lines with the models directly
2026-11-03 01:52:29 +00:00
ryan-crabbe-berri
6254860560
test(e2e): drive the Codex CLI through LiteLLM, command round trip (#49374)
* test(e2e): drive the real Codex CLI through the gateway and add the command round trip row

Adds tests/e2e/codex: a harness that runs codex exec --json with a throwaway HOME and CODEX_HOME, a custom responses provider pointed at the gateway, the model list read from the gateway's /v1/models, the sandbox on and approvals off. Each test registers its setup's deployment via /model/new and calls it with a key scoped to it, both deleted on teardown. The first row checks that a value only a command can print comes back through the gateway on all five setups

* test(e2e): type the Codex self-test event helper as a Mapping

* test(e2e): clean up Codex test keys even when setup fails, and fail runs where Codex exits with an error

* test(e2e): record a step when the Codex harness sets up its home and workspace

The suite rules ask every public harness helper that does something to record a step, so a failed run shows whether it died while setting up Codex or while running it

* Update run.py

wip
2026-11-02 17:32:16 -08:00