When a cell aggregates per-model results across three tiers (Haiku/Sonnet/Opus),
a mix of (pass, not_applicable) used to short-circuit to not_applicable on the
first NA match, discarding the passing tiers from the published matrix.
Treat not_applicable like not_tested when mixed with pass: only return
not_applicable when every observed row is NA. Otherwise any pass surfaces as
pass, so the cell answers 'does this feature work on this provider?' truthfully
when at least one tier passes.
Add two regression tests pinning the new precedence:
- mixed pass + NA → pass
- all NA → not_applicable (with first reason)
Addresses three Bugbot concerns flagged on PR #28027 that are real
behavioral / coverage gaps:
1. matrix_builder._aggregate_cell now joins every failing tier's error
in the published cell instead of silently dropping all but the first.
When Haiku 429s and Opus times out on the same cell, both diagnostics
land in the matrix JSON so docs-page triage can name both outliers.
2. _aggregate_cell treats 'not_tested' rows as absent data: they're
dropped before computing the cell status. Previously a mixed
(pass, not_tested) cell silently fell through to 'not_tested',
discarding the passing tiers and hiding real coverage from the
published matrix. A cell still aggregates to 'not_tested' when
*every* row is 'not_tested' (or there are no rows at all).
3. test_v0_layout.py now structurally validates every feature declared
in manifest.yaml (directory exists, __init__.py exists, every
per-provider test_<provider>.py exists), not just the original six
v0 rows. The EXPECTED_FEATURE_IDS / EXPECTED_PROVIDERS anchor
constants still pin v0 positions; the new manifest-driven tests
extend the same structural guarantees to every post-v0 row so a
broken directory in 'count_tokens', 'tool_search', 'web_search',
etc. fails CI instead of silently becoming a 'not_tested' cell.
Three new builder tests pin the new aggregation behavior:
- mixed pass + not_tested surfaces as pass
- all-not_tested stays not_tested
- multiple fail errors are joined with '; '
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Slice 1 of the Claude Code Compatibility Matrix: the thinnest end-to-end
path through every layer for a single (feature, provider) cell, so a
future docs page can render a real green cell sourced from a real test.
What landed in this repo:
- tests/claude_code/manifest.yaml — feature manifest with one entry
(basic_messaging_non_streaming) plus the v0 provider column order.
- tests/claude_code/cli_driver.py — Claude Code CLI Driver. One entry
point (run_claude); handles subprocess assembly, env overlay, stream-JSON
parsing, and structured failure modes. `runner=` is a unit-test seam.
- tests/claude_code/conftest.py — `compat_result` fixture (tagged-union
recorder) + pytest_runtest_makereport hook that infers (feature, provider)
from the file path and writes a structured compat-results.json artifact.
- tests/claude_code/basic_messaging_non_streaming/test_anthropic.py — the
one cell, parametrized over Haiku/Sonnet/Opus per the PRD's per-cell
model rule.
- tests/claude_code/matrix_builder.py — pure-function builder from
(manifest, results, run-metadata) to the v1 JSON schema. Aggregates per-
model results into one cell (pass iff all pass). build_from_paths is the
thin I/O wrapper for the publisher.
- tests/claude_code/sample_compatibility-matrix.json — hand-authored sample
of the v1 JSON; copied to the docs repo by hand as part of this slice.
- Unit tests: 10 driver tests (mocked subprocess), 9 compat_result tests,
10 matrix-builder golden-file tests. 29/29 pass.
Key decisions:
- (feature, provider) is inferred from file path, not declared in metadata —
mirrors the PRD's "no drift" goal.
- Driver injects subprocess via a `runner` kwarg so unit tests don't need
the real `claude` CLI; production callers leave it default.
- Builder is a pure function on Mappings/Sequences; load/write live in a
thin `build_from_paths` wrapper. Golden-file tests pin the schema.
- `_driver_unit_tests/` and `_builder_unit_tests/` are prefixed with `_`
so the conftest's path-inference hook skips them and they don't
pollute the matrix artifact.
- `compat-results.json` added to .gitignore (CI-only output).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The MDX page `docs/tutorials/claude-code-compatibility` and the
`<CompatibilityMatrix />` React component. The hand-authored
compatibility-matrix.json (`sample_compatibility-matrix.json` in this
repo) is the artifact those docs files will consume; opening that doc
PR is the next step in this slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
