2 commits

Author	SHA1	Message	Date
mateo-berri	415cf3d6b1	RALPH: compat matrix slice 2 - add 4 provider columns for basic_messaging_non_streaming (#26478, PRD #26476) ... Slice 2 of the Claude Code Compatibility Matrix: extend the tracer-bullet cell from slice 1 across all four remaining provider columns for basic_messaging_non_streaming. Proves the multi-provider, multi-model, all-must-pass aggregation logic against a 1x5 grid that exercises every status state. What landed: - tests/claude_code/basic_messaging_non_streaming/test_bedrock_invoke.py - tests/claude_code/basic_messaging_non_streaming/test_bedrock_converse.py - tests/claude_code/basic_messaging_non_streaming/test_vertex_ai.py Per-provider files modeled on test_anthropic.py: each parametrizes over Haiku 4.5 / Sonnet 4.6 / Opus 4.7 (the three Claude tiers required by the PRD), drives the real `claude` CLI through the driver, and reports pass/fail via `compat_result`. Per-cell error strings always include `[<model>]` so the docs tooltip can name the failing model when a cell goes red. - tests/claude_code/basic_messaging_non_streaming/test_azure.py All three (Azure, Claude) cells report `not_applicable` with a reason: Azure OpenAI Service does not host Anthropic models. The test still parametrizes over the same three model ids so the test count per cell is uniform across columns, and a future "Azure adds Anthropic" announcement only requires flipping the body, not the parametrization. - tests/claude_code/sample_compatibility-matrix.json Hand-authored 1x5 sample updated to reflect the slice 2 outcome: anthropic / bedrock_invoke / bedrock_converse / vertex_ai = pass, azure = not_applicable. - tests/claude_code/_builder_unit_tests/test_matrix_builder.py Two new golden-file tests: 1. 1x5 grid: feed the per-model results the four new test files produce on a real run; assert the builder output equals the hand-authored sample byte-for-byte. 2. fail-with-model-named: feed pass/fail/pass for one cell and assert the cell aggregates to fail with the failing model id surfaced in the error string (acceptance criterion: "the error string identifies which model broke"). Key decisions: - Duplication across the four per-provider files is accepted (per the PRD) rather than extracted into a helper. Each file is self-contained so a test author touching one provider doesn't accidentally regress the others. - Per-provider model alias names: `claude-<tier>-<provider-suffix>` (e.g. `claude-haiku-4-5-bedrock-invoke`). These are the alias names the proxy operator wires up in the routing config; the test only knows the alias, the proxy knows the upstream model id and region. - Azure is `not_applicable` rather than `not_tested` because the cell will never apply, not "we haven't gotten to it yet" - the two states are visually and semantically distinct in the rendered grid. - Sample shows the realistic best-case outcome (4 pass + 1 NA). The React renderer's coverage of the `fail` and `not_tested` states is exercised by other cells in v1+, not the v0 sample. Tests: 31 -> 34 passing (added 2 builder golden tests + 3 Azure not_applicable parametrizations that pass without env vars). Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs): - The companion update to compatibility-matrix.json in the docs repo. The hand-authored sample in this repo is the artifact the docs PR copies; opening that doc PR is the next step in this slice. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-06 23:27:05 +00:00
mateo-berri	6c573de426	RALPH: tracer-bullet for Claude Code compatibility matrix (#26477, PRD #26476) ... Slice 1 of the Claude Code Compatibility Matrix: the thinnest end-to-end path through every layer for a single (feature, provider) cell, so a future docs page can render a real green cell sourced from a real test. What landed in this repo: - tests/claude_code/manifest.yaml — feature manifest with one entry (basic_messaging_non_streaming) plus the v0 provider column order. - tests/claude_code/cli_driver.py — Claude Code CLI Driver. One entry point (run_claude); handles subprocess assembly, env overlay, stream-JSON parsing, and structured failure modes. `runner=` is a unit-test seam. - tests/claude_code/conftest.py — `compat_result` fixture (tagged-union recorder) + pytest_runtest_makereport hook that infers (feature, provider) from the file path and writes a structured compat-results.json artifact. - tests/claude_code/basic_messaging_non_streaming/test_anthropic.py — the one cell, parametrized over Haiku/Sonnet/Opus per the PRD's per-cell model rule. - tests/claude_code/matrix_builder.py — pure-function builder from (manifest, results, run-metadata) to the v1 JSON schema. Aggregates per- model results into one cell (pass iff all pass). build_from_paths is the thin I/O wrapper for the publisher. - tests/claude_code/sample_compatibility-matrix.json — hand-authored sample of the v1 JSON; copied to the docs repo by hand as part of this slice. - Unit tests: 10 driver tests (mocked subprocess), 9 compat_result tests, 10 matrix-builder golden-file tests. 29/29 pass. Key decisions: - (feature, provider) is inferred from file path, not declared in metadata — mirrors the PRD's "no drift" goal. - Driver injects subprocess via a `runner` kwarg so unit tests don't need the real `claude` CLI; production callers leave it default. - Builder is a pure function on Mappings/Sequences; load/write live in a thin `build_from_paths` wrapper. Golden-file tests pin the schema. - `_driver_unit_tests/` and `_builder_unit_tests/` are prefixed with `_` so the conftest's path-inference hook skips them and they don't pollute the matrix artifact. - `compat-results.json` added to .gitignore (CI-only output). Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs): - The MDX page `docs/tutorials/claude-code-compatibility` and the `<CompatibilityMatrix />` React component. The hand-authored compatibility-matrix.json (`sample_compatibility-matrix.json` in this repo) is the artifact those docs files will consume; opening that doc PR is the next step in this slice. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-06 23:27:05 +00:00