test(e2e): let a Codex known gap be non-strict for intermittent cells

A known_gaps entry can now set strict: false (a real bool, default true). That
entry marks its cell xfail(strict=False), so the e2e lane stays green whether
the cell passes or fails, while the results file still records what happened.

Vertex structured output gets a non-strict entry, since gemini-3.8-flash
sometimes loops on shell calls or stalls under an output schema and times out.
This commit is contained in:
mateo-berri 2026-10-30 10:17:22 -07:00
parent 014edcbb00
commit f38ef7389e
6 changed files with 48 additions and 5 deletions

View file

@ -49,7 +49,7 @@ Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family
- `secret_manager/` - the gateway's `key_management_system` against a real secret manager: deployment keys resolved from it (`os.environ/<name>` where the name exists only in the manager) and virtual keys written to and deleted from it. The tests are backend-agnostic and each backend is its own lane, because the setting is global to the proxy: `E2E_SECRET_MANAGER=<system>` opts in and picks the backend from `secret_backends.BACKENDS`, the proxy is booted from `gateway/secret_manager_<system>_ci_config.yml` against the live manager, and the tests reach that manager through the backend's `SecretStore` (`secret_store_<system>.py`). A test needing something not every backend does carries `requires_capability(...)` and is deselected on lanes that lack it. `secret_manager/backend.sh up <system>` runs a backend in Docker and writes the proxy's and the tests' env. Marked `secret_manager`, deselected unless `E2E_SECRET_MANAGER` is set, and kept out of the per-PR selector. Backends today: `hashicorp_vault` and `cyberark` (CyberArk Conjur, which cannot delete, so the delete test is Vault-only)
- `gateway/` - proxy configuration only (`litellm-config.yml`). No tests
- `claude_code/` - the Claude Code compatibility matrix: drives the real `claude` CLI (and HTTP probes) against a proxy for each feature x provider cell, reporting tagged-union outcomes via the `compat_result` fixture. Ships its own driver/builder/publisher, covered by `tests/e2e_harness/claude_code/`. The HTTP probes ride the shared transport (`ProxyClient.count_tokens` / `ProxyClient.messages`). The CLI-driving path stays bespoke
- `client_apps/` - compatibility matrices for coding-agent CLIs other than Claude Code, one subfolder per client (`codex/` today). A client folder holds `manifest.yaml` (its provider columns and feature rows), `proxy_models.yaml` (the deployments each column registers through `/model/new` on first use and deletes at session end) and one cell per `<feature>/test_<provider>.py`. A cell drives the real CLI against the proxy with a scoped key for every model in its column and reports one tagged-union outcome per model through the `compat_result` fixture, and the matrix marks the cell green only when every model passed. A proxy gap we already know about goes in the manifest's `known_gaps` as `{feature, provider, reason}`, the reason naming the proxy behavior in plain words, and `client_apps/conftest.py` marks that cell `xfail(strict=True)`, so the lane stays green while the gap lasts and goes red when the cell starts passing and the entry should go. The results file still records what the cell really did, and an entry naming a feature or provider the manifest lacks stops the run at collection. Cells of one column take turns across xdist workers through a file lock, since a provider's rate limit is shared by the whole column, while different columns run in parallel. The CLIs are pinned in `client_apps/package.json` and its lockfile, the runner image installs them with `npm ci`, and the driver runs `node_modules/.bin/<cli>` before falling back to the one on `PATH`. Nothing here imports `claude_code/`. To run it locally, run `npm ci` in `tests/e2e/client_apps`, then `uv run pytest tests/e2e/client_apps -n 8` with `LITELLM_PROXY_URL` and `LITELLM_MASTER_KEY` set. Point pytest at `tests/e2e/client_apps` or below, because `client_apps/conftest.py` is what writes the results to `<CLIENT>_COMPAT_RESULTS_PATH` (default `<client>-compat-results.json`, merged across xdist workers). `cd tests/e2e && python -m client_apps.matrix_builder build --manifest client_apps/codex/manifest.yaml --results <results> --out <matrix>` turns them into the matrix, and `regressions --old <matrix> --new <matrix>` exits 3 when a green cell turned red. The builder, collector, parser and driver are covered by `tests/e2e_harness/client_apps/`
- `client_apps/` - compatibility matrices for coding-agent CLIs other than Claude Code, one subfolder per client (`codex/` today). A client folder holds `manifest.yaml` (its provider columns and feature rows), `proxy_models.yaml` (the deployments each column registers through `/model/new` on first use and deletes at session end) and one cell per `<feature>/test_<provider>.py`. A cell drives the real CLI against the proxy with a scoped key for every model in its column and reports one tagged-union outcome per model through the `compat_result` fixture, and the matrix marks the cell green only when every model passed. A proxy gap we already know about goes in the manifest's `known_gaps` as `{feature, provider, reason}`, the reason naming the proxy behavior in plain words, and `client_apps/conftest.py` marks that cell `xfail(strict=True)`, so the lane stays green while the gap lasts and goes red when the cell starts passing and the entry should go, while `strict: false` on an entry keeps the lane green either way for a cell that fails only some of the time. The results file still records what the cell really did, and an entry naming a feature or provider the manifest lacks stops the run at collection. Cells of one column take turns across xdist workers through a file lock, since a provider's rate limit is shared by the whole column, while different columns run in parallel. The CLIs are pinned in `client_apps/package.json` and its lockfile, the runner image installs them with `npm ci`, and the driver runs `node_modules/.bin/<cli>` before falling back to the one on `PATH`. Nothing here imports `claude_code/`. To run it locally, run `npm ci` in `tests/e2e/client_apps`, then `uv run pytest tests/e2e/client_apps -n 8` with `LITELLM_PROXY_URL` and `LITELLM_MASTER_KEY` set. Point pytest at `tests/e2e/client_apps` or below, because `client_apps/conftest.py` is what writes the results to `<CLIENT>_COMPAT_RESULTS_PATH` (default `<client>-compat-results.json`, merged across xdist workers). `cd tests/e2e && python -m client_apps.matrix_builder build --manifest client_apps/codex/manifest.yaml --results <results> --out <matrix>` turns them into the matrix, and `regressions --old <matrix> --new <matrix>` exits 3 when a green cell turned red. The builder, collector, parser and driver are covered by `tests/e2e_harness/client_apps/`
- `ui/` - the Admin UI browser suite: Playwright in TypeScript, driving the dashboard served by a live proxy on port 4000 (seeded postgres + mock LLM upstream, see its `run_e2e.sh`). It is a self-contained npm package with its own lockfile and does not use the Python harness, pytest markers, or the shared transport. The Python rules in this file (typed models, `Result` unions, basedpyright zero-error gate) do not apply inside it. Its only Python file, `fixtures/mock_llm_server/server.py`, is excluded from the e2e basedpyright gate via the root `pyrightconfig.json`. Its specs follow the dashboard's rule against testing what shadcn and Base UI already guarantee, in `ui/litellm-dashboard/AGENTS.md`
## MCP suite: real Datadog only

View file

@ -29,6 +29,12 @@ features:
name: Web search
known_gaps:
- feature: structured_output
provider: vertex_ai
strict: false
reason: >-
gemini-3.8-flash sometimes loops on shell calls or stalls under an output schema and times out (1 of 3 runs
passed)
- feature: web_search
provider: openai
reason: The proxy has no POST /v1/alpha/search route, so Codex's web search calls get a 404 and never complete

View file

@ -98,7 +98,7 @@ def _known_gap_mark(test_file: Path, clients_dir: Path) -> pytest.MarkDecorator
if address is None:
return None
gap: Final = client_manifest(clients_dir / address.client).known_gap(address.feature_id, address.provider)
return None if gap is None else pytest.mark.xfail(strict=True, reason=gap.reason)
return None if gap is None else pytest.mark.xfail(strict=gap.strict, reason=gap.reason)
def _raised(call: pytest.CallInfo[None]) -> bool:

View file

@ -11,7 +11,7 @@ from types import MappingProxyType
from typing import Annotated, Final, Literal, Self
import yaml
from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, model_validator
from pydantic import BaseModel, ConfigDict, Field, StrictBool, TypeAdapter, model_validator
from client_apps.compat_results import (
Failed,
@ -40,6 +40,7 @@ class KnownGap(_Frozen):
feature: str
provider: str
reason: str = Field(min_length=1)
strict: StrictBool = True
class Manifest(_Frozen):

View file

@ -179,10 +179,13 @@ providers: [openai, vertex_ai]
features:
- {id: vision, name: Vision}
- {id: web_search, name: Web search}
- {id: structured_output, name: Structured output}
known_gaps:
- {feature: web_search, provider: openai, reason: the proxy has no search route}
- {feature: web_search, provider: vertex_ai, reason: the proxy drops search tools}
- {feature: vision, provider: vertex_ai, reason: the relay forwards an unknown field}
- {feature: structured_output, provider: openai, strict: false, reason: the model sometimes loops}
- {feature: structured_output, provider: vertex_ai, strict: false, reason: the model sometimes stalls}
"""
CELL_BODIES: Final = (
(
@ -194,6 +197,12 @@ CELL_BODIES: Final = (
("web_search", "vertex_ai", 'compat_result.passed("m")'),
("vision", "openai", 'compat_result.passed("m")'),
("vision", "vertex_ai", 'raise RuntimeError("proxy unreachable")'),
("structured_output", "openai", 'compat_result.passed("m")'),
(
"structured_output",
"vertex_ai",
'compat_result.failed("m", "codex exec timed out")\n pytest.fail("m: codex exec timed out", pytrace=False)',
),
)
@ -219,7 +228,7 @@ def _run_cells(
return pytester.runpytest_inprocess("--import-mode=importlib", "-p", "no:cacheprovider", str(client)), results
def test_known_gap_cells_xfail_strictly_while_the_matrix_records_what_happened(
def test_known_gap_cells_xfail_while_the_matrix_records_what_happened(
pytester: pytest.Pytester, monkeypatch: pytest.MonkeyPatch
) -> None:
run, results = _run_cells(pytester, monkeypatch, GAP_MANIFEST)
@ -235,13 +244,15 @@ def test_known_gap_cells_xfail_strictly_while_the_matrix_records_what_happened(
(feature.id, provider): outcome for feature in matrix.features for provider, outcome in feature.providers.items()
}
run.assert_outcomes(passed=1, failed=1, xfailed=2)
run.assert_outcomes(passed=1, failed=1, xfailed=3, xpassed=1)
assert "the proxy drops search tools" in run.stdout.str()
assert cells == {
("vision", "openai"): Passed(),
("vision", "vertex_ai"): Failed(error="RuntimeError: proxy unreachable"),
("web_search", "openai"): Failed(error="m: no completed web_search item"),
("web_search", "vertex_ai"): Passed(),
("structured_output", "openai"): Passed(),
("structured_output", "vertex_ai"): Failed(error="m: codex exec timed out"),
}

View file

@ -194,3 +194,28 @@ def test_a_known_gap_needs_a_reason(tmp_path: Path) -> None:
_ = load_manifest(path)
assert [error["loc"] for error in raised.value.errors()] == [("known_gaps", 0, "reason")]
def test_a_known_gap_is_strict_unless_it_says_otherwise(tmp_path: Path) -> None:
path: Final = tmp_path / "manifest.yaml"
_ = path.write_text(
_manifest_yaml(
"[{feature: vision, provider: openai, reason: r}, "
"{feature: vision, provider: vertex_ai, strict: false, reason: flaky}]"
)
)
manifest: Final = load_manifest(path)
assert [(gap.provider, gap.strict) for gap in manifest.known_gaps] == [("openai", True), ("vertex_ai", False)]
@pytest.mark.parametrize("strict", ("'false'", "0", "no-thanks"))
def test_a_known_gaps_strict_flag_must_be_a_bool(tmp_path: Path, strict: str) -> None:
path: Final = tmp_path / "manifest.yaml"
_ = path.write_text(_manifest_yaml(f"[{{feature: vision, provider: openai, strict: {strict}, reason: r}}]"))
with pytest.raises(ValidationError) as raised:
_ = load_manifest(path)
assert [error["loc"] for error in raised.value.errors()] == [("known_gaps", 0, "strict")]