Addresses three Bugbot concerns flagged on PR #28027 that are real
behavioral / coverage gaps:
1. matrix_builder._aggregate_cell now joins every failing tier's error
in the published cell instead of silently dropping all but the first.
When Haiku 429s and Opus times out on the same cell, both diagnostics
land in the matrix JSON so docs-page triage can name both outliers.
2. _aggregate_cell treats 'not_tested' rows as absent data: they're
dropped before computing the cell status. Previously a mixed
(pass, not_tested) cell silently fell through to 'not_tested',
discarding the passing tiers and hiding real coverage from the
published matrix. A cell still aggregates to 'not_tested' when
*every* row is 'not_tested' (or there are no rows at all).
3. test_v0_layout.py now structurally validates every feature declared
in manifest.yaml (directory exists, __init__.py exists, every
per-provider test_<provider>.py exists), not just the original six
v0 rows. The EXPECTED_FEATURE_IDS / EXPECTED_PROVIDERS anchor
constants still pin v0 positions; the new manifest-driven tests
extend the same structural guarantees to every post-v0 row so a
broken directory in 'count_tokens', 'tool_search', 'web_search',
etc. fails CI instead of silently becoming a 'not_tested' cell.
Three new builder tests pin the new aggregation behavior:
- mixed pass + not_tested surfaces as pass
- all-not_tested stays not_tested
- multiple fail errors are joined with '; '
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Slice 1 of the Claude Code Compatibility Matrix: the thinnest end-to-end
path through every layer for a single (feature, provider) cell, so a
future docs page can render a real green cell sourced from a real test.
What landed in this repo:
- tests/claude_code/manifest.yaml — feature manifest with one entry
(basic_messaging_non_streaming) plus the v0 provider column order.
- tests/claude_code/cli_driver.py — Claude Code CLI Driver. One entry
point (run_claude); handles subprocess assembly, env overlay, stream-JSON
parsing, and structured failure modes. `runner=` is a unit-test seam.
- tests/claude_code/conftest.py — `compat_result` fixture (tagged-union
recorder) + pytest_runtest_makereport hook that infers (feature, provider)
from the file path and writes a structured compat-results.json artifact.
- tests/claude_code/basic_messaging_non_streaming/test_anthropic.py — the
one cell, parametrized over Haiku/Sonnet/Opus per the PRD's per-cell
model rule.
- tests/claude_code/matrix_builder.py — pure-function builder from
(manifest, results, run-metadata) to the v1 JSON schema. Aggregates per-
model results into one cell (pass iff all pass). build_from_paths is the
thin I/O wrapper for the publisher.
- tests/claude_code/sample_compatibility-matrix.json — hand-authored sample
of the v1 JSON; copied to the docs repo by hand as part of this slice.
- Unit tests: 10 driver tests (mocked subprocess), 9 compat_result tests,
10 matrix-builder golden-file tests. 29/29 pass.
Key decisions:
- (feature, provider) is inferred from file path, not declared in metadata —
mirrors the PRD's "no drift" goal.
- Driver injects subprocess via a `runner` kwarg so unit tests don't need
the real `claude` CLI; production callers leave it default.
- Builder is a pure function on Mappings/Sequences; load/write live in a
thin `build_from_paths` wrapper. Golden-file tests pin the schema.
- `_driver_unit_tests/` and `_builder_unit_tests/` are prefixed with `_`
so the conftest's path-inference hook skips them and they don't
pollute the matrix artifact.
- `compat-results.json` added to .gitignore (CI-only output).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The MDX page `docs/tutorials/claude-code-compatibility` and the
`<CompatibilityMatrix />` React component. The hand-authored
compatibility-matrix.json (`sample_compatibility-matrix.json` in this
repo) is the artifact those docs files will consume; opening that doc
PR is the next step in this slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
