run_claude validated empty prompt strings but silently accepted
stdin_input="", letting an empty stdin reach the subprocess and surface
as a confusing CLI failure instead of a clear ValueError.
These three cells were failing for reasons unrelated to LiteLLM
translation:
- vision: tests passed `--image <path>`, a flag that no longer exists
in Claude Code 2.x (image attachment is now via the Files API or via
`--input-format stream-json` with inline content blocks). Rewrite
the cells to feed an Anthropic-shaped user message containing both
text and a base64 `image` content block through stdin in stream-json
mode. Hermetic — no temp file or Files API upload needed.
- extended_thinking: tests set `MAX_THINKING_TOKENS=4096` as an env
var, which Claude Code 2.x ignores. Switch to `--effort max` (the
current CLI knob) and use a non-trivial prompt (3-gallon / 5-gallon
jug puzzle). With trivial arithmetic the modern Sonnet/Opus tiers
optimize away the thinking step and arrive without a thinking block,
which made the test silently false-fail.
- web_search: assertion looked for `server_tool_use` /
`web_search_tool_result` blocks, but Claude Code's `WebSearch` is
a *client-side* tool: the CLI executes the search itself and feeds
the result back as a regular `tool_result` block. The Anthropic
server-side `web_search_20250305` tool only fires when injected
into the request directly (which the CLI does not do). Update the
assertion to look for a `tool_use` block whose name is
`WebSearch` — that's the right signal that the proxy preserved
both the request-side tool definition and the response-side tool_use
block end-to-end.
Driver change required to support stream-json input + variadic flags:
- cli_driver: insert `--` before the prompt positional. Variadic
flags like `--allowed-tools <tools...>` (commander.js) greedily
consume every following token, so the prompt was being eaten as a
tool name and the CLI would error out with "Input must be provided
either through stdin or as a prompt argument when using --print".
- cli_driver: thread a `stdin_input` parameter through `run_claude`
and `run_claude_models_parallel` so the vision rewrite can pipe
stream-json events to the CLI on stdin.
Validated end-to-end against a live LiteLLM proxy: all three Anthropic
cells now pass on Haiku 4.5, Sonnet 4.6, and Opus 4.7. Driver unit
tests (120) still green.
Cursor security review flagged that run_claude() forwarded the entire
parent environment to the externally installed claude CLI binary. In
the PR gate flow the binary is dynamically installed from npm, and
the surrounding job loads every upstream provider credential
(ANTHROPIC_API_KEY, AWS_*, AZURE_FOUNDRY_*, VERTEXAI_CREDENTIALS,
GITHUB_TOKEN, ...) into its env so the proxy can route requests. A
compromised CLI release would have read access to all of them — even
though the CLI itself only ever talks to the proxy via the explicit
ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKEN we set.
Build the subprocess env from a small allowlist of process-runtime
vars (PATH, HOME, NVM_DIR, locale) rather than inheriting all of
os.environ. Caller-supplied extra_env still rides on top, which is
the sanctioned way for tests to opt-in to passing additional vars
(e.g. extended_thinking sets MAX_THINKING_TOKENS).
Add unit tests pinning the contract: PATH/HOME flow through, secrets
do not, and extra_env can still override anything.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Switch all per-cell tests from @pytest.mark.parametrize("model", ...)
(3 sequential invocations) to a single test that fans out to all 3
Claude tiers via run_claude_models_parallel. Per-cell wall time is now
bounded by the slowest model rather than the sum.
- Add 5 new v0 feature dirs (5 providers each, 25 new test files):
web_search, pdf_input, prompt_caching_1h,
tool_use_streaming, thinking_with_tool_use
Manifest expanded to match.
- Add cross-process token-bucket rate limiter (rate_limiter.py + tests)
so xdist workers stay under per-provider req/s limits during full-grid
runs. New env knobs: LITELLM_COMPAT_RATE_{ANTHROPIC,AZURE,VERTEX_AI,
BEDROCK_CONVERSE,BEDROCK_INVOKE}.
- conftest.py: write per-worker shards under <artifact>.shards/, merge
in the controller; preserve the "don't write empty artifact" guard so
unit-test runs don't clobber a real compat-results.json.
- Vertex test_config.yaml: route project/location through env so the
cron VM can target a different GCP project than the upstream default.
- Add run_compat.sh wrapper for binary-searching ideal req/s per
provider against compat-rate-limit-summary.json output.
Bugbot flagged that `run_claude` placed the prompt as cmd[7] and then
appended extra_args after it. `claude --print` takes the prompt as the
final positional argument; flags appearing after it (e.g.
`--allowed-tools Bash`, `--image <path>`) are swallowed by the prompt
parser, which silently breaks the tool_use and vision cells.
Build the flag list first, then append the prompt last. Add a unit
test that pins the ordering.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Slice 1 of the Claude Code Compatibility Matrix: the thinnest end-to-end
path through every layer for a single (feature, provider) cell, so a
future docs page can render a real green cell sourced from a real test.
What landed in this repo:
- tests/claude_code/manifest.yaml — feature manifest with one entry
(basic_messaging_non_streaming) plus the v0 provider column order.
- tests/claude_code/cli_driver.py — Claude Code CLI Driver. One entry
point (run_claude); handles subprocess assembly, env overlay, stream-JSON
parsing, and structured failure modes. `runner=` is a unit-test seam.
- tests/claude_code/conftest.py — `compat_result` fixture (tagged-union
recorder) + pytest_runtest_makereport hook that infers (feature, provider)
from the file path and writes a structured compat-results.json artifact.
- tests/claude_code/basic_messaging_non_streaming/test_anthropic.py — the
one cell, parametrized over Haiku/Sonnet/Opus per the PRD's per-cell
model rule.
- tests/claude_code/matrix_builder.py — pure-function builder from
(manifest, results, run-metadata) to the v1 JSON schema. Aggregates per-
model results into one cell (pass iff all pass). build_from_paths is the
thin I/O wrapper for the publisher.
- tests/claude_code/sample_compatibility-matrix.json — hand-authored sample
of the v1 JSON; copied to the docs repo by hand as part of this slice.
- Unit tests: 10 driver tests (mocked subprocess), 9 compat_result tests,
10 matrix-builder golden-file tests. 29/29 pass.
Key decisions:
- (feature, provider) is inferred from file path, not declared in metadata —
mirrors the PRD's "no drift" goal.
- Driver injects subprocess via a `runner` kwarg so unit tests don't need
the real `claude` CLI; production callers leave it default.
- Builder is a pure function on Mappings/Sequences; load/write live in a
thin `build_from_paths` wrapper. Golden-file tests pin the schema.
- `_driver_unit_tests/` and `_builder_unit_tests/` are prefixed with `_`
so the conftest's path-inference hook skips them and they don't
pollute the matrix artifact.
- `compat-results.json` added to .gitignore (CI-only output).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The MDX page `docs/tutorials/claude-code-compatibility` and the
`<CompatibilityMatrix />` React component. The hand-authored
compatibility-matrix.json (`sample_compatibility-matrix.json` in this
repo) is the artifact those docs files will consume; opening that doc
PR is the next step in this slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
