probe_count_tokens and probe_tool_search were sending requests
directly via httpx with no call to the cross-process RateLimiter
that cli_driver.run_claude uses. During a full matrix run those
unthrottled probes would silently violate the limiter's aggregate
per-provider budget and could push adjacent CLI cells over the
429 threshold.
Acquire one token from the same process-wide limiter (keyed by
infer_provider(model)) before each probe, with an injectable
rate_limiter seam matching run_claude's API for unit tests.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Matrix grows from 11 to 15 feature rows. All new tests collected + 180
unit tests still pass; smoke runs hit real LiteLLM bug surfaces on
bedrock_invoke, bedrock_converse, and vertex_ai (cells correctly red
in PR #142).
Rename
------
`extended_thinking` -> `thinking` (directory, manifest id+name, 5
test fn names, 5 docstrings, builder unit-test fixtures, sample JSON,
run_compat.sh). Existing test logic already covers both manual
(`thinking.type=enabled`, Haiku 4.5) and adaptive
(`thinking.type=adaptive`, Opus 4.7) shapes because Claude Code picks
the shape per model from `--effort max`; the name change just stops
the column from looking like a Claude 3.7 reference.
New rows
--------
- structured_outputs (5 files, CLI `--json-schema`). Claude Code
synthesizes a single `StructuredOutput` tool from the schema and
surfaces the tool_use input as `structured_output` on the trailing
`result` event. Test ships its own `_validate_against_schema` so
we don't take a jsonschema dep just for matrix surface.
- count_tokens (5 files, HTTP probe). POSTs the proxy's
`/v1/messages/count_tokens` directly and asserts the response is
`{input_tokens: positive int}`. No CLI hook exists for this
endpoint; the test goes through the new http_probe helper instead.
- tool_search (5 files, HTTP probe). Sends
`tools: [{type: tool_search_tool_regex_20251119, name:
tool_search_tool_regex}]` and asserts the proxy doesn't 400. MCP
fan-out via `--mcp-config` would also exercise the tool-search
beta header path, but it's flaky w.r.t. Claude Code's internal
tool-deferral threshold; the HTTP probe hits the actual bug surface
(per-provider beta-header translation `advanced-tool-use-2025-11-20`
vs `tool-search-tool-2025-10-19`).
- long_context_1m (5 files, CLI `--betas context-1m-2025-08-07
--max-budget-usd 6`). A ~210k-token padded prompt over stdin
exercises the 1M-context beta. Sonnet 4.6 + Opus 4.7 only --
Haiku 4.5's window is 200k, so it's excluded from MODELS (not
marked not_applicable) to keep the per-cell aggregator semantics
intact. Prompt uses a document-style preamble + 8 cycling pangrams
rather than repeating identical chunks; without that, Opus 4.7
trips the safety filter mid-response with a Usage Policy refusal.
`--max-budget-usd 6` is a runaway-loop guard, ~2x worst-case Opus
per-cell spend.
New helper
----------
`tests/claude_code/http_probe.py`: shared `ProbeResult` dataclass
plus per-endpoint `probe_*` + `assert_*_shape` pairs for the
HTTP-probe rows. Uses httpx with `anthropic-version: 2023-06-01` and
a 30s timeout.
The cron host has no write access to BerriAI/litellm-docs by design. PRs
now open from a long-lived fork at agent-shin/litellm-docs:
- run_daily.sh validates AGENT_SHIN_GITHUB_TOKEN up front (failing 30 min
into a run because the env file is missing one line is wasted spend).
- The pre-commit shim adds a transient `fork` remote with the token
embedded in the URL, force-pushes the branch, then removes the remote
so the token never lives on disk.
- `gh pr create --head agent-shin:<branch>` opens the cross-repo PR
with GH_TOKEN scoped to AGENT_SHIN_GITHUB_TOKEN. A second
`gh pr edit --add-reviewer` runs under GITHUB_TOKEN (mateo-berri's
PAT) because agent-shin's PAT lacks RequestReviewsByLogin permission
on the upstream repo.
- PR_REVIEWERS env var (default `mateo-berri`) controls who gets
auto-tagged; empty disables.
Also bring litellm-compat-matrix.service to working state:
- Hardcode `/home/mateo` paths everywhere %h was used. systemd expands
%h against the *manager's* home (/root for PID 1) in *system* units,
not against the User= directive. The mismatch made ReadWritePaths
point at /root/.cache and the namespace setup failed with
status=226/NAMESPACE before run_daily.sh ever started.
- Explicit Environment=PATH so `uv` and `claude` under
~/.local/bin are visible to the up-front command-presence check;
systemd's default PATH excludes them.
- Expand ReadWritePaths to include ~/.claude (CLI per-session state)
and ~/.config/gh (gh host config fallback); both are written under
ProtectHome=read-only.
env.example refreshed: drop AWS_ACCESS_KEY_ID/SECRET +
GOOGLE_APPLICATION_CREDENTIALS in favor of AWS_BEARER_TOKEN_BEDROCK and
ADC via the VM's metadata server; document AGENT_SHIN_GITHUB_TOKEN,
FORK_OWNER/FORK_REPO overrides, and VERTEXAI_LOCATION=global.
- pytest_runtest_makereport: skip the defensive fail-row append when
the test has already recorded a fail via .add(), so the common pattern
of '.add(fail) per failing model, then pytest.fail() to surface them'
no longer produces duplicate rows in compat-results.json.
- _infer_feature_and_provider: validate the parent directory against
manifest.yaml instead of relying on a negative '_-prefix' filter, so
non-feature sibling dirs (e.g. cron_vm) can't leak rows into the
artifact or pollute the rate-limit summary counters.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Two related gaps in pytest_runtest_makereport let real failures show up as
green (or absent) cells in the published compatibility matrix:
1. Setup-phase failures (broken fixtures / imports) only produce a report
with when="setup"; the call phase never runs. The hook filtered on
when=="call" and returned, leaving no row for the cell. The matrix
builder then aggregated the empty cell to "not_tested" instead of
"fail".
2. A test that called compat_result.add({"status": "pass"}) for some
models and then raised before completing the rest produced a partial
list of pass entries. The "if not collected" guard was bypassed
because the list was non-empty, so no fail row was added. The cell
aggregator's all-pass check then returned pass for a cell that was
never fully exercised.
Now the hook also handles when=="setup" on failure, and always appends a
fail row when report.failed — preserving any partial pass entries from
add() for diagnostics while ensuring the cell aggregator surfaces the
crash.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The shim was previously gated on `tests/claude_code/test_config.yaml`
not existing in the worktree, so the moment a v*-stable tag landed
with its own copy of `tests/claude_code/` the cron would happily run
whatever frozen tests that release happened to ship — even though the
populator's whole purpose is to exercise *today's* tests against
*today's* stable proxy.
Drop the if-guard and make the shim unconditional: `rm -rf` the
worktree's `tests/claude_code/` after the tag checkout, then
`cp -r` the dev checkout's copy back in. This way fixes that land
on the dev checkout (like the stream-json vision rewrite, the
`--effort` thinking knob, the WebSearch tool_use assertion change)
take effect on the next cron run rather than waiting for a stable
release. The tag's own version of the tree, if any, is discarded each
run, so the worktree is byte-identical to
`${LITELLM_REPO}/tests/claude_code` every time.
Also drops `-e tests/claude_code` from the post-checkout `git
clean` invocation. We used to keep the prior run's shim across
`git clean -fdx` so subsequent runs didn't have to re-shim;
that's no longer necessary now that we re-shim unconditionally,
and keeping it would only delay garbage-collecting deleted files
across runs.
The previous early-return guard in pytest_sessionfinish exited before
the merge step for the xdist controller process: the controller never
runs tests itself (so _COLLECTOR.items is empty) and is not detected
as a worker (it has no workerinput), so the guard always tripped.
Worker shards were written but no process ever produced the canonical
compat-results.json. Also check for shards on disk so the controller
still proceeds to merge them under pytest -n auto.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- conftest: combine the two pytest_sessionstart definitions so stale
shard cleanup actually runs (the second def previously shadowed the
first, leaving compat-results.json.shards/ from prior sessions in
place and polluting the merged artifact).
- circleci: forward VERTEXAI_PROJECT and VERTEXAI_LOCATION into the
compat-proxy container so test_config.yaml's os.environ refs for the
Vertex AI routes resolve in the PR gate.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
These three cells were failing for reasons unrelated to LiteLLM
translation:
- vision: tests passed `--image <path>`, a flag that no longer exists
in Claude Code 2.x (image attachment is now via the Files API or via
`--input-format stream-json` with inline content blocks). Rewrite
the cells to feed an Anthropic-shaped user message containing both
text and a base64 `image` content block through stdin in stream-json
mode. Hermetic — no temp file or Files API upload needed.
- extended_thinking: tests set `MAX_THINKING_TOKENS=4096` as an env
var, which Claude Code 2.x ignores. Switch to `--effort max` (the
current CLI knob) and use a non-trivial prompt (3-gallon / 5-gallon
jug puzzle). With trivial arithmetic the modern Sonnet/Opus tiers
optimize away the thinking step and arrive without a thinking block,
which made the test silently false-fail.
- web_search: assertion looked for `server_tool_use` /
`web_search_tool_result` blocks, but Claude Code's `WebSearch` is
a *client-side* tool: the CLI executes the search itself and feeds
the result back as a regular `tool_result` block. The Anthropic
server-side `web_search_20250305` tool only fires when injected
into the request directly (which the CLI does not do). Update the
assertion to look for a `tool_use` block whose name is
`WebSearch` — that's the right signal that the proxy preserved
both the request-side tool definition and the response-side tool_use
block end-to-end.
Driver change required to support stream-json input + variadic flags:
- cli_driver: insert `--` before the prompt positional. Variadic
flags like `--allowed-tools <tools...>` (commander.js) greedily
consume every following token, so the prompt was being eaten as a
tool name and the CLI would error out with "Input must be provided
either through stdin or as a prompt argument when using --print".
- cli_driver: thread a `stdin_input` parameter through `run_claude`
and `run_claude_models_parallel` so the vision rewrite can pipe
stream-json events to the CLI on stdin.
Validated end-to-end against a live LiteLLM proxy: all three Anthropic
cells now pass on Haiku 4.5, Sonnet 4.6, and Opus 4.7. Driver unit
tests (120) still green.
Cursor security review flagged that run_claude() forwarded the entire
parent environment to the externally installed claude CLI binary. In
the PR gate flow the binary is dynamically installed from npm, and
the surrounding job loads every upstream provider credential
(ANTHROPIC_API_KEY, AWS_*, AZURE_FOUNDRY_*, VERTEXAI_CREDENTIALS,
GITHUB_TOKEN, ...) into its env so the proxy can route requests. A
compromised CLI release would have read access to all of them — even
though the CLI itself only ever talks to the proxy via the explicit
ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKEN we set.
Build the subprocess env from a small allowlist of process-runtime
vars (PATH, HOME, NVM_DIR, locale) rather than inheriting all of
os.environ. Caller-supplied extra_env still rides on top, which is
the sanctioned way for tests to opt-in to passing additional vars
(e.g. extended_thinking sets MAX_THINKING_TOKENS).
Add unit tests pinning the contract: PATH/HOME flow through, secrets
do not, and extra_env can still override anything.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Bugbot flagged that the awk extracting PINNED_UV_VERSION required the
value to start with a specifier (`==`, `>=`, etc.) and silently
fell through to the system uv if pyproject.toml later switched to a
bare `required-version = "0.10.9"` form. Strip any leading
specifier prefix from the quoted value rather than requiring one to
be present, so both prefixed and bare forms resolve to the same
download URL.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Cursor security review flagged that the cron VM proxy binds to
0.0.0.0 (litellm's default when --host is omitted) while running
under the predictable default `LITELLM_MASTER_KEY=sk-cron-matrix`.
On a VM where :PROXY_PORT is reachable, anything that can hit the
port can authenticate with the predictable fallback secret and burn
upstream provider credentials.
The populator proxy is only ever talked to by the same-host pytest
run (the health check and tests both use http://127.0.0.1:${PORT}),
so there's no reason for it to listen on external interfaces. Pass
`--host 127.0.0.1` explicitly.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Switch all per-cell tests from @pytest.mark.parametrize("model", ...)
(3 sequential invocations) to a single test that fans out to all 3
Claude tiers via run_claude_models_parallel. Per-cell wall time is now
bounded by the slowest model rather than the sum.
- Add 5 new v0 feature dirs (5 providers each, 25 new test files):
web_search, pdf_input, prompt_caching_1h,
tool_use_streaming, thinking_with_tool_use
Manifest expanded to match.
- Add cross-process token-bucket rate limiter (rate_limiter.py + tests)
so xdist workers stay under per-provider req/s limits during full-grid
runs. New env knobs: LITELLM_COMPAT_RATE_{ANTHROPIC,AZURE,VERTEX_AI,
BEDROCK_CONVERSE,BEDROCK_INVOKE}.
- conftest.py: write per-worker shards under <artifact>.shards/, merge
in the controller; preserve the "don't write empty artifact" guard so
unit-test runs don't clobber a real compat-results.json.
- Vertex test_config.yaml: route project/location through env so the
cron VM can target a different GCP project than the upstream default.
- Add run_compat.sh wrapper for binary-searching ideal req/s per
provider against compat-rate-limit-summary.json output.
Cursor security review flagged that the cron VM downloads the uv
release tarball and pipes it straight through `tar -xzO ... > file ;
chmod +x`, with no integrity check. A tampered release artifact would
execute in a credentialed cron context with access to the docs-repo
push token.
Download the tarball + the official .sha256 sidecar Astral publishes
alongside every uv release to a temp dir, run `sha256sum -c` against
the sidecar, and only extract+install on success. On mismatch we wipe
the tempdir and `die` with a clear refusal.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Bugbot flagged that the resolver fetches only the first page (default
30 entries) of the releases endpoint. LiteLLM ships multiple non-stable
releases per day, so 30+ non-stable releases between consecutive
v*-stable tags is routinely the case — when it happens, the jq filter
matches nothing, LITELLM_VERSION is empty, and the daily matrix update
silently dies.
Walk pages 1..5 (100 per page = 500 releases max) and short-circuit as
soon as a page contains at least one v*-stable tag. Same final jq
filter, same sort order.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Bugbot flagged that `run_claude` placed the prompt as cmd[7] and then
appended extra_args after it. `claude --print` takes the prompt as the
final positional argument; flags appearing after it (e.g.
`--allowed-tools Bash`, `--image <path>`) are swallowed by the prompt
parser, which silently breaks the tool_use and vision cells.
Build the flag list first, then append the prompt last. Add a unit
test that pins the ordering.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The pytest_sessionfinish hook in tests/claude_code/conftest.py is loaded
for every pytest run under tests/claude_code/, including sibling unit-
test trees (_driver_unit_tests/, _builder_unit_tests/, ...). Without a
guard, those runs wrote an empty results artifact and silently
overwrote any real artifact from a prior compat-test run.
- Add pytest_sessionstart hook in tests/claude_code/conftest.py that
clears the module-level _COLLECTOR so results don't leak across
pytest.main() invocations within the same process.
- Export LITELLM_MASTER_KEY=${PROXY_API_KEY} when launching the cron
VM proxy so the auth token the tests send actually matches what
the proxy expects.
The Claude Code Compatibility Matrix PR gate booted the proxy with only
Anthropic + Bedrock credentials, so every Vertex AI and Azure cell (12
out of 30) failed at startup because test_config.yaml resolves
VERTEXAI_CREDENTIALS, AZURE_FOUNDRY_API_KEY, and AZURE_FOUNDRY_API_BASE
from the environment. Forward those three env vars from the CircleCI
context so the proxy has working routes for all five providers.
Also drop the leftover debug prints in
basic_messaging_non_streaming/test_anthropic.py that wrote the proxy
api_key to stdout (and from there to JUnit XML / CI logs).
The resolver picks the latest `v*-stable` tag of BerriAI/litellm. Until
the compat matrix work itself lands in a stable release, that tag's
tree won't contain `tests/claude_code/` at all — the proxy config and
test files only exist on the work-in-progress stack. Without a shim,
the cron dies with 'proxy config not found at .../test_config.yaml'
on every run.
Fix: after `git checkout <tag>`, if
`<worktree>/tests/claude_code/test_config.yaml` is missing, copy the
directory from ${LITELLM_REPO} (the dev checkout, which has the
in-flight matrix work). The `git clean -e tests/claude_code` line
preserves the shim across runs.
Once the matrix work is in the resolved tag, the if-branch is a no-op
and the shim is never used. No code path needs to be removed later;
the bridge self-disables.
Tier 1 (single anthropic cell) and tier 2 (full
basic_messaging_non_streaming row across 5 providers) now run cleanly
from `run_daily.sh` on the GCP VM. Five issues showed up during
validation; each is fixed in this commit.
1. uv version pin
----------------
The litellm worktree pins an exact uv version in
`pyproject.toml`'s `[tool.uv] required-version` field. The cron
VM's system uv (currently 0.11.8) refused to sync against the
v1.83.10-stable lockfile (which pins ==0.10.9). Fix: parse the
pinned version out of the worktree's pyproject.toml, download the
matching standalone binary into `<worktree>/.uv-bin/uv-<version>`,
and use it for sync/run/proxy. Cached across runs.
2. Missing proxy + dev extras
---------------------------
`uv sync --frozen` only installed the base dependency set, so
`uv run litellm` died at startup with
`ModuleNotFoundError: No module named 'websockets'`. Per the
repo's AGENTS.md the canonical incantation is
`uv sync --frozen --group proxy-dev --extra proxy`.
3. Wrong env vars for the test driver
-----------------------------------
The script was setting `ANTHROPIC_BASE_URL` and
`ANTHROPIC_AUTH_TOKEN`, which is what Claude Code itself reads,
but the test files read `LITELLM_PROXY_BASE_URL` and
`LITELLM_PROXY_API_KEY` (search the test_config-driven test files
for `PROXY_BASE_URL_ENV = "LITELLM_PROXY_BASE_URL"`). Tests were
marking themselves `fail` with
"missing required env: set LITELLM_PROXY_BASE_URL...". Fix: rename
the two env vars in the pytest invocation. The driver still
propagates them onward as ANTHROPIC_BASE_URL/AUTH_TOKEN to Claude.
4. Cleanup couldn't find the proxy
--------------------------------
The previous setup did
`( ... && setsid uv run litellm ... ) &; PROXY_PID=$!`. `setsid`
detaches the inner uv into its own session, but `$!` records the
PID of the outer subshell, not the long-lived python proxy. So
`kill -TERM "-${PROXY_PID}"` in the EXIT trap targeted the wrong
pgid and the proxy survived as an orphan whenever the script was
killed externally. Fix: replace the subshell with
`setsid bash -c '...'` that writes $$ to a known pid file before
exec'ing the proxy. The cleanup trap reads that file and uses it
as the pgid. Belt-and-braces: the trap also `pgrep -f`s by port
number and SIGKILLs survivors. Trap now fires on `INT TERM` too,
not just `EXIT`.
5. .uv-bin cache survives git clean
---------------------------------
The original `git clean -fdx -e .venv` wiped `.uv-bin/` between
runs, forcing re-download of the pinned uv binary on every
invocation. Now excluded.
Things that worked first try
----------------------------
* Worktree clone + checkout to the resolved tag.
* gh auth on this VM (mateo-berri account, collaborator on
BerriAI/litellm-docs).
* The matrix builder produced a well-formed
`compatibility-matrix.json` with the right per-cell aggregation
even when 4 of 5 cells failed (tier 2 was: anthropic=pass,
bedrock_invoke=fail, bedrock_converse=fail, vertex_ai=fail with a
real 403 from GCP for insufficient scopes, azure=fail with
timeout).
The previous iteration of this PR ported the populator to a Python
module (`publisher.py`) with 8 unit-tested pure helpers for branch
naming and PR-body rendering. After the docker code came out, the
GitHub App auth came out, and the worktree-vs-tempdir decision was
made, what was left was: 'git fetch + git checkout + uv sync + start
a subprocess + run pytest + clone docs + git commit + gh pr create'.
That's a bash script.
This commit replaces 800 lines of Python (publisher.py + resolver.py +
their unit tests) with a 297-line run_daily.sh and a 48-line
build_matrix.py whose only job is to be importable Python that can
call into the matrix_builder we already have. Net deletion: -822 lines.
Removed
-------
* tests/claude_code/publisher.py — the full Python orchestrator.
Every code path it had is now in run_daily.sh.
* tests/claude_code/resolver.py — the GitHub Releases v*-stable
resolver. Replaced by ~10 lines of jq inside run_daily.sh.
* tests/claude_code/_publisher_unit_tests/ — both test files. The
pure helpers they covered (commit message, file allowlist, branch
name, PR title/body) only existed because publisher.py was Python.
The bash equivalents are short heredoc strings.
Added
-----
* tests/claude_code/cron_vm/run_daily.sh — the actual cron job,
structured as numbered phases (resolve / worktree / proxy /
pytest / build / publish) so journalctl output is readable.
* tests/claude_code/cron_vm/build_matrix.py — a 48-line CLI that
calls the existing matrix_builder.build_from_paths. Kept in
Python because the builder itself is Python and well-tested.
Modified
--------
* tests/claude_code/cron_vm/litellm-compat-matrix.service —
ExecStart now invokes run_daily.sh instead of
'python -m tests.claude_code.publisher'.
* tests/claude_code/cron_vm/README.md — updated layout table,
file roles, and operating commands to match.
Why this is the right shape
---------------------------
* The failure mode at 06:00 UTC is 'read journalctl, see the literal
failing command with its + prefix, copy-paste it into a shell to
reproduce'. Bash makes that immediate; Python's subprocess.run
output looks similar but the surrounding orchestration is harder
to step through interactively.
* Every operation the script does is already a shell command (git,
uv, gh, jq, curl, pytest). The Python wrapper was translating
between argv arrays and back.
* The two pieces that genuinely benefit from being in a typed
language are matrix_builder (already Python) and the resolver's
semver sort (now done in jq, with the version_key tuple sort
inline). 'Already Python' wins, 'tiny jq pipeline' wins.
What's preserved
----------------
* Idempotency: same (litellm, claude, UTC date) -> same branch ->
same PR. force-with-lease push, gh-pr-create no-op-on-exists.
* Byte-identical-JSON early return (git diff --cached --quiet).
* Per-feature status table in the PR body (jq pipeline mirroring
the Python pr_body_for_matrix logic).
* Persistent worktree approach so disk doesn't grow unboundedly.
* Proxy bound to :4100 to avoid colliding with a developer's :4000.
* SKIP_PUBLISH=1 and PYTEST_K=... operator escape hatches.
Operator escape hatch for first-time validation after a Claude Code CLI
upgrade or a proxy-config change. Setting `PYTEST_K=anthropic and
basic_messaging_non_streaming` (for example) narrows the matrix run to
one cell, which is enough to confirm the worktree+uv+proxy+pytest
plumbing without burning a full $2 in provider tokens.
Cells the narrowed run doesn't touch are filled with `not_tested` by
the matrix builder, so a PYTEST_K-narrowed run is structurally safe to
publish — though in practice operators pair it with `--skip-publish`
during validation.
The daily compat-matrix runs from the dedicated GCP VM
`litellm-compatibility-matrix-populator` rather than from a GitHub
Actions runner. The VM has no docker daemon, has `gh` already
authenticated against an account with `pull-requests: write` on
`BerriAI/litellm-docs`, and runs a long-lived litellm checkout we can
reuse across runs. That makes Docker, the GitHub App auth flow, and the
GHA workflow itself dead code.
Removed
-------
* `.github/workflows/claude_code_compat_matrix.yml` — no longer
triggers anything; the systemd timer in this PR owns the daily fire.
* `docker_image_for_tag` + `DOCKER_IMAGE_BASE` constants and the two
unit tests that covered them.
* `_start_proxy(image, port)` / `_stop_proxy(container_id)` /
`docker run` flow, replaced by direct `uv run litellm` subprocess
management with a sigterm-the-process-group teardown.
* `--skip-proxy` CLI flag (was only useful when the GHA workflow
split docker-bringup from publish into separate jobs).
* `docs_token` parameter and `DOCS_REPO_TOKEN` env var; `gh` on
the VM is already authenticated, so we don't pass an explicit
token through the publisher.
Added
-----
* Persistent worktree flow in `publisher.py`. First run clones
`BerriAI/litellm` into `~/litellm-cron-worktree/`; subsequent runs
do `git fetch --tags && git checkout --force <stable-tag> &&
uv sync --frozen`. Disk footprint is bounded because uv sync
removes packages no longer pinned and `git clean -fdx -e .venv`
wipes per-run cruft while keeping the venv around.
* `tests/claude_code/cron_vm/` containing systemd units and a
setup README:
- `litellm-compat-matrix.service` (`Type=oneshot`, runs as the
`mateo` user, sources `/etc/litellm-compat-matrix.env` for
provider creds, hardened with `NoNewPrivileges` /
`ProtectSystem=strict` / `PrivateTmp`);
- `litellm-compat-matrix.timer` (`OnCalendar=*-*-* 06:00:00 UTC`,
`Persistent=true` so a missed run fires when the VM is back up,
`RandomizedDelaySec=10min`);
- `.env.example` documenting the provider-credential surface;
- `README.md` covering one-time install, daily operation,
`journalctl` debugging, and the gotchas (proxy port `4100` to
avoid colliding with a developer's `:4000`, `gh` token
rotation, what to do after a Claude Code CLI upgrade).
Operator notes
--------------
* The proxy now binds `:4100` by default so a developer SSH'd into
the VM with their own `:4000` proxy isn't preempted by the cron.
* The Claude Code CLI is exercised as-is from the system install;
the populator does NOT `npm install` it. Operators upgrade the
CLI by running `npm install -g @anthropic-ai/claude-code@latest`
out of band, typically after watching a `--skip-publish` run to
verify the matrix doesn't suddenly turn red.
* 20 publisher unit tests pass (`pytest
tests/claude_code/_publisher_unit_tests/`).
* End-to-end validation on the VM happens after this PR lands as
follow-up commits on the same branch — the systemd unit is
`Type=oneshot` so a manual `systemctl start` reproduces the cron.
The daily Claude Code compatibility-matrix cron has been direct-pushing
`compatibility-matrix.json` to litellm-docs's main branch. Switch to
opening (or updating) a pull request so docs maintainers can review each
matrix update before it ships to readers.
Behavioural changes
-------------------
publisher.publish() now:
* checks out a deterministic head branch
(`compat-matrix/<litellm>-<claude>-<UTC-date>`) before staging the
JSON, instead of committing on top of the docs branch directly;
* `git push --force-with-lease` so a same-day rerun updates the
existing branch (and therefore the existing PR), without
overwriting any docs-maintainer fixup commit on the same branch;
* shells out to `gh pr create` against `docs_repo` with a
title/body that surfaces the resolved versions and a per-feature
status summary, so reviewers can triage from the inbox;
* treats 'a pull request for branch ... already exists' as success,
so two cron runs on the same day produce one PR, not two.
Idempotency contract
--------------------
* Same (litellm_version, claude_code_version, UTC date) -> same
branch -> same PR. Verified by the new
`test_pr_branch_name_is_deterministic_per_inputs` /
`...changes_when_any_component_changes` tests.
* Byte-identical JSON to the docs branch -> early-return before
push, same as the previous direct-push path.
* Empty version inputs are rejected up front so two distinct PRs
can never silently collapse onto one branch.
Tests
-----
* 8 new tests in `_publisher_unit_tests/test_publisher.py` cover
`pr_branch_name`, `pr_title_for_matrix`, and `pr_body_for_matrix`
(determinism, content, ordering, missing-provider rectangularity,
empty-input rejection).
* Existing 7 `commit_message_for_matrix` /
`docker_image_for_tag` / `select_files_to_commit` tests are
unchanged and still pass.
Workflow
--------
`.github/workflows/claude_code_compat_matrix.yml` updates only the
header doc comment to reflect that the GitHub App now needs
`pull-requests: write` in addition to `contents: write`. `gh` is
preinstalled on `ubuntu-latest` (also used by
`auto_update_price_and_context_window.yml`), so no install step is
needed.
Operator action required (one-time)
-----------------------------------
The compat-matrix GitHub App installation on `BerriAI/litellm-docs`
needs `pull-requests: write` added to its installation permissions
before the next cron run. Without it, the new `gh pr create` call
will fail with a 403; `compat-results.json` and
`compatibility-matrix.json` will still upload as workflow artifacts
for debugging.
Anthropic and Microsoft announced Claude Haiku 4.5, Sonnet 4.5/4.6, and
Opus 4.1/4.6/4.7 in Microsoft Foundry on 2025-11-18, so the matrix's
Azure column should exercise a real route through the LiteLLM proxy
rather than report not_applicable.
Foundry serves Claude on an Anthropic-shape /anthropic/v1/messages
endpoint (not the Azure OpenAI chat-completions route), and LiteLLM
already supports it via the azure_ai/claude-* provider prefix
(litellm/llms/azure_ai/anthropic/{handler,transformation,messages_transformation}.py).
- test_config.yaml: add 3 azure aliases pointing at azure_ai/claude-*
with AZURE_FOUNDRY_API_BASE / AZURE_FOUNDRY_API_KEY env
- 6x test_azure.py: replace not_applicable stubs with real run_claude
drivers, mirroring the existing test_vertex_ai.py shape exactly
- sample_compatibility-matrix.json: Azure cells flip to pass
- _builder_unit_tests: pin the new invariant (run_claude is used,
not_applicable is gone) and feed pass results across all 5 providers
in the 6x5 golden test
Slice 5 of the Claude Code Compatibility Matrix: extend the published
matrix from the 1x5 grid that landed in slice 2 to the full v0 6x5
grid described in the PRD's "Features in v0" section. After this
slice merges and the daily cron runs, the docs page reflects all six
v0 features against all five providers.
What landed:
- tests/claude_code/manifest.yaml
Five new entries appended in PRD row order:
basic_messaging_streaming, tool_use, prompt_caching_5m, vision,
extended_thinking. The manifest is the row-order source of truth
the matrix builder respects.
- tests/claude_code/<feature>/test_<provider>.py (25 new files)
For each of the five new features, five per-provider test files
modeled on slice 2's basic_messaging_non_streaming/. Each non-Azure
file parametrizes over Haiku 4.5 / Sonnet 4.6 / Opus 4.7 and drives
the real `claude` CLI through the driver with feature-specific
options:
* basic_messaging_streaming — count-1-to-5 prompt; asserts the
stream-json wire actually emitted events plus a non-empty reply.
* tool_use — `--allowed-tools Bash` plus an `echo pong` prompt;
asserts a `tool_use` content block was emitted.
* prompt_caching_5m — same baseline prompt as non-streaming, but
asserts the upstream usage block reports
cache_creation_input_tokens or cache_read_input_tokens > 0
(Claude Code stamps cache_control on its system prompt by default,
so a single live call surfaces it).
* vision — decodes a checked-in 1x1 PNG (base64 const) into
`tmp_path` and attaches it via `--image`; asserts a non-empty
reply.
* extended_thinking — sets `MAX_THINKING_TOKENS=4096`; asserts a
`thinking` content block was emitted.
All five Azure files report `not_applicable` with the standard
reason: Azure OpenAI Service does not host Anthropic models.
- tests/claude_code/sample_compatibility-matrix.json
Hand-authored 6x5 sample showing the realistic best-case outcome:
4 pass + 1 not_applicable (Azure) per row.
- tests/claude_code/_builder_unit_tests/test_v0_layout.py
New structural unit tests pinning the on-disk shape so future edits
can't silently flip the matrix shape:
* manifest lists all six v0 feature ids in PRD order
* manifest lists all five v0 provider columns in PRD order
* every (feature, provider) has a test file at the inferred path
* every test file references all three required Claude tiers
* every Azure test file is a `not_applicable` declaration
- tests/claude_code/_builder_unit_tests/test_matrix_builder.py
Renamed the slice-2 1x5 golden test to
test_build_matrix_6x5_grid_matches_published_sample and rebuilt
its inputs to feed all six features. The golden file is now the
6x5 sample.
Key decisions:
- Per-feature per-provider test bodies are deliberately duplicated
(per the PRD: "Duplication across per-provider files is accepted").
Each file is self-contained so a contributor touching one cell
doesn't accidentally regress neighbors.
- Only Azure cells are marked `not_applicable` in this slice. Other
combinations that turn out to genuinely not apply on the live cron
run (e.g. a provider that doesn't support `thinking` for a tier)
will be tightened to `not_applicable` reasons in a follow-up; for
now they fail honestly, which the matrix renderer paints red.
- prompt_caching_5m's assertion (cache tokens > 0 in the usage block)
exercises the path Claude Code customers care about: that the proxy
preserves `cache_control` annotations end-to-end. It does not try
to differentiate cache_creation vs cache_read across runs.
- The vision PNG fixture is generated at test time from a base64
const rather than checked into git as a binary — keeps the diff
text-only and avoids needing PIL or any image-generation library.
Tests: 31 -> 142 unit tests passing (no proxy / no `claude` CLI
required). Test counts:
* 12 builder tests (was 11; +1 for 6x5 golden, the slice-2 1x5
test was renamed in place)
* 100 v0_layout structural tests (new)
* 10 driver tests (unchanged)
* 9 compat_result tests (unchanged)
* 14 publisher unit tests (unchanged)
* 8 PR-gate version-resolver tests (unchanged)
* 6 CircleCI structural tests (unchanged)
The 90 per-cell tests under tests/claude_code/<feature>/ continue to
require a running proxy + `claude` CLI; they only run inside the
CircleCI PR gate or the daily-cron VM (both established in slices
3 and 4).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The companion update to compatibility-matrix.json in the docs repo.
After slice 4's daily-cron lands the App credentials, the cron run
will replace the docs-side hand-authored JSON automatically; until
then the slice-2 1x5 sample remains in the docs repo.
Notes for next iteration:
- The exact `claude` CLI flags for tool-allowlist (`--allowed-tools`),
vision (`--image`), and extended thinking (`MAX_THINKING_TOKENS`)
are best-guess from the current Claude Code surface; if the live
PR-gate run reveals different flag names, tighten in place.
- Several non-Azure cells will likely need `not_applicable`
declarations once the cron VM produces real outcomes (e.g.
Bedrock Invoke + extended_thinking is uncertain). That refinement
is an iteration-2 follow-up driven by data, not a blocker for this
slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Slice 4 of the Claude Code Compatibility Matrix: stand up the daily-cron
pipeline that publishes `compatibility-matrix.json` to the docs repo. After
this slice lands, the hand-authored matrix in the docs repo is replaced by
auto-generated output, and the docs page begins reflecting real test runs
against the latest stable LiteLLM release.
What landed:
- tests/claude_code/resolver.py
Latest Stable LiteLLM Resolver. Calls the GitHub Releases API and
returns the newest tag matching `v*-stable`. Sort is numeric on
(major, minor, patch) so v1.10.0-stable correctly outranks
v1.9.5-stable. Injectable `http_get` so tests run offline.
- tests/claude_code/publisher.py
Daily-cron orchestrator. Resolves the latest stable tag, pulls
`ghcr.io/berriai/litellm:<tag>`, starts it as the proxy, installs
`@anthropic-ai/claude-code@latest`, runs `pytest tests/claude_code/`,
invokes the Matrix JSON Builder, and direct-pushes
`compatibility-matrix.json` to the docs repo's main branch using a
GitHub App installation token (`DOCS_REPO_TOKEN`). Idempotent: a no-op
if the JSON is byte-identical to what's already on main.
- tests/claude_code/_publisher_unit_tests/test_resolver.py
test_publisher.py
14 unit tests covering the small pure helpers — version sort,
non-stable filtering, http-get injection, commit message determinism,
Docker image-name builder, and the file allowlist that enforces the
"only `compatibility-matrix.json` ever ships" guarantee. Per the PRD's
"Testing Decisions" section, the publisher's full subprocess
orchestration intentionally ships without a unit-test harness; the
daily-cron failure surface is itself the test.
- .github/workflows/claude_code_compat_matrix.yml
GitHub Actions workflow with three triggers (daily cron at 06:00 UTC,
`release: published` filtered to `*-stable` tags, and
`workflow_dispatch`). Mints a docs-repo installation token from a
GitHub App scoped to `BerriAI/litellm-docs` only with `contents:
write`, then runs the publisher.
- .gitignore
Add `compatibility-matrix.json` (cron VM output).
Key decisions:
- "Isolated VM" is realized as a GitHub-hosted ubuntu-latest runner —
every run gets a fresh ephemeral VM, and the always-latest Claude
Code CLI is only ever installed inside that ephemeral environment,
so a malicious or broken Claude Code release cannot affect the
trusted PR-gate CI in CircleCI.
- File-level restriction on the GitHub App's broad `contents: write`
scope is enforced by `select_files_to_commit` (script correctness),
per the PRD's explicit acknowledgement that GitHub does not support
file-path-scoped tokens.
- `release` runs are filtered to tags ending in `-stable` at the
workflow level, so a `v1.84.0-rc1` release does not republish the
matrix.
- Resolver and publisher live under `tests/claude_code/` alongside
`matrix_builder.py` and `cli_driver.py` — production code that
supports the test suite, kept colocated with it to match the slice
1+2 layout.
Out of scope / blockers for next iteration:
- Provisioning the GitHub App itself (creating it under BerriAI's
org, installing it on litellm-docs only, generating the private key
and registering `COMPAT_MATRIX_APP_ID` / `COMPAT_MATRIX_APP_PRIVATE_KEY`
as repo secrets) is an operator/infra step that cannot land via a
code change in this repo.
- The first successful cron run is what removes the hand-authored
`compatibility-matrix.json` from the docs repo and replaces it with
generated output — that happens after this PR merges and the App is
installed; not a code change here.
Tests: 34 -> 45 passing (added 7 resolver tests + 7 publisher helper
tests, all unit-only and offline). The 12 per-cell failures under
`tests/claude_code/basic_messaging_non_streaming/` remain by design —
they require a running proxy which the cron VM provides.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Slice 3 of the Claude Code Compatibility Matrix: wire the
`tests/claude_code/` suite into CircleCI as a pre-merge gate. A red
status on the new `claude_code_compat_pr_gate` job blocks merge into
the staging branch.
What landed:
- tests/claude_code/pr_gate_version_resolver.py
The Claude Code PR-Gate Version Resolver described in the PRD's
"Version resolvers" section. Queries the npm registry for
`@anthropic-ai/claude-code` and returns the newest version whose
publish timestamp is at least 3 days old. The 3-day window is a
security review buffer: a malicious or broken Claude Code release
has at least 72 hours to be detected before it can land in our PR
gate. Importable function (with `metadata=` / `fetcher=` / `as_of=`
injection seams for tests) and a `python -m ...` CLI for the CI step.
- tests/claude_code/test_config.yaml
Proxy routing config that maps the per-cell aliases the tests use
(`claude-haiku-4-5`, `claude-haiku-4-5-bedrock-invoke`, ...,
`claude-opus-4-7-vertex`) to real upstream model ids on Anthropic /
Bedrock (Invoke + Converse) / Vertex AI. Azure intentionally has no
entries here because every Azure × claude-code cell is
`not_applicable` (Azure OpenAI doesn't host Claude).
- .circleci/config.yml
New `claude_code_compat_pr_gate` job. Pattern modeled on
`proxy_e2e_anthropic_messages_tests` (load PR-built docker image,
start postgres, mount config.yaml). New step in the middle:
resolve the Claude Code version from the resolver, install Node 20
via the machine image's preinstalled nvm, and `npm install -g
@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}` (pinned, never
`latest`). Wired into `workflows.build_and_test` with a
`requires: [build_docker_database_image]` gate and the same
`*main_branches` filter the other proxy e2e job uses.
- tests/claude_code/_pr_gate_unit_tests/
16 new unit tests:
* 8 against the version resolver: boundary (>= 3d inclusive),
empty / all-too-new metadata, semver-vs-publish-time tiebreak,
custom min_age, fetcher injection, npm `time.created` /
`time.modified` skipping.
* 8 structural tests against `.circleci/config.yml`: job exists,
is in the workflow, requires the docker image, invokes the
resolver, install command is pinned (rejects unpinned `latest`),
runs `tests/claude_code/`, mounts `test_config.yaml`, exports
`LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY`. Plus one
regression test: the existing `proxy_e2e_anthropic_messages_tests`
job is unchanged in shape (acceptance criterion).
Key decisions:
- "Newest version" in the resolver is by **publish time**, not by
semver string ordering — if a patch lands on an older major after a
newer release, the patched line is the eligible one. (Tested.)
- The resolver's CLI prints the announcement to stderr and the bare
version to stdout, so the CI step can do
`CLAUDE_CODE_VERSION=$(uv run python -m ...)` cleanly while still
surfacing the selected version in the job log (acceptance criterion:
"the selected Claude Code version is logged").
- The structural CircleCI tests live under `_pr_gate_unit_tests/` so
the conftest path-inference hook skips them (the leading underscore
is the existing convention from `_driver_unit_tests/` /
`_builder_unit_tests/`); they don't pollute the matrix artifact.
- No `--no-verify` style supply-chain safety relaxation. Per the PRD,
Claude Code's pinning is the 3-day publish-age window, not a fixed
hash — by design, since the daily cron also pulls newer versions.
Tests: 47 -> 47 passing for the unit suite (16 new + 31 from slices
1 and 2). The end-to-end cells under `basic_messaging_non_streaming/`
require `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` and a
running proxy + `claude` CLI; they only run inside the new CircleCI
job.
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- No docs PR is needed for this slice — the gate produces a status
check, not a published artifact. The compat matrix JSON the docs
page consumes is published by the daily-cron job (a future slice),
not by the PR gate.
Notes for next iteration:
- The daily cron / matrix publisher is the next slice. Several
pieces this slice introduces (the `tests/claude_code/test_config.yaml`
proxy config, the structure of the compat-results.json artifact)
will be reused by it.
- The bedrock-converse / vertex_ai aliases in `test_config.yaml` use
best-guess upstream model ids (`us.anthropic.claude-{tier}` and
`vertex_ai/claude-{tier}`); the real ids may need to be tightened
once the gate runs against live AWS / GCP credentials and we see
what resolves.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Slice 2 of the Claude Code Compatibility Matrix: extend the tracer-bullet
cell from slice 1 across all four remaining provider columns for
basic_messaging_non_streaming. Proves the multi-provider, multi-model,
all-must-pass aggregation logic against a 1x5 grid that exercises every
status state.
What landed:
- tests/claude_code/basic_messaging_non_streaming/test_bedrock_invoke.py
- tests/claude_code/basic_messaging_non_streaming/test_bedrock_converse.py
- tests/claude_code/basic_messaging_non_streaming/test_vertex_ai.py
Per-provider files modeled on test_anthropic.py: each parametrizes
over Haiku 4.5 / Sonnet 4.6 / Opus 4.7 (the three Claude tiers
required by the PRD), drives the real `claude` CLI through the
driver, and reports pass/fail via `compat_result`. Per-cell error
strings always include `[<model>]` so the docs tooltip can name the
failing model when a cell goes red.
- tests/claude_code/basic_messaging_non_streaming/test_azure.py
All three (Azure, Claude) cells report `not_applicable` with a
reason: Azure OpenAI Service does not host Anthropic models. The
test still parametrizes over the same three model ids so the test
count per cell is uniform across columns, and a future "Azure adds
Anthropic" announcement only requires flipping the body, not the
parametrization.
- tests/claude_code/sample_compatibility-matrix.json
Hand-authored 1x5 sample updated to reflect the slice 2 outcome:
anthropic / bedrock_invoke / bedrock_converse / vertex_ai = pass,
azure = not_applicable.
- tests/claude_code/_builder_unit_tests/test_matrix_builder.py
Two new golden-file tests:
1. 1x5 grid: feed the per-model results the four new test files
produce on a real run; assert the builder output equals the
hand-authored sample byte-for-byte.
2. fail-with-model-named: feed pass/fail/pass for one cell and assert
the cell aggregates to fail with the failing model id surfaced
in the error string (acceptance criterion: "the error string
identifies which model broke").
Key decisions:
- Duplication across the four per-provider files is accepted (per the
PRD) rather than extracted into a helper. Each file is self-contained
so a test author touching one provider doesn't accidentally regress
the others.
- Per-provider model alias names: `claude-<tier>-<provider-suffix>`
(e.g. `claude-haiku-4-5-bedrock-invoke`). These are the alias names
the proxy operator wires up in the routing config; the test only
knows the alias, the proxy knows the upstream model id and region.
- Azure is `not_applicable` rather than `not_tested` because the
cell will never apply, not "we haven't gotten to it yet" - the two
states are visually and semantically distinct in the rendered grid.
- Sample shows the realistic best-case outcome (4 pass + 1 NA). The
React renderer's coverage of the `fail` and `not_tested` states is
exercised by other cells in v1+, not the v0 sample.
Tests: 31 -> 34 passing (added 2 builder golden tests + 3 Azure
not_applicable parametrizations that pass without env vars).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The companion update to compatibility-matrix.json in the docs repo.
The hand-authored sample in this repo is the artifact the docs PR
copies; opening that doc PR is the next step in this slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Slice 1 of the Claude Code Compatibility Matrix: the thinnest end-to-end
path through every layer for a single (feature, provider) cell, so a
future docs page can render a real green cell sourced from a real test.
What landed in this repo:
- tests/claude_code/manifest.yaml — feature manifest with one entry
(basic_messaging_non_streaming) plus the v0 provider column order.
- tests/claude_code/cli_driver.py — Claude Code CLI Driver. One entry
point (run_claude); handles subprocess assembly, env overlay, stream-JSON
parsing, and structured failure modes. `runner=` is a unit-test seam.
- tests/claude_code/conftest.py — `compat_result` fixture (tagged-union
recorder) + pytest_runtest_makereport hook that infers (feature, provider)
from the file path and writes a structured compat-results.json artifact.
- tests/claude_code/basic_messaging_non_streaming/test_anthropic.py — the
one cell, parametrized over Haiku/Sonnet/Opus per the PRD's per-cell
model rule.
- tests/claude_code/matrix_builder.py — pure-function builder from
(manifest, results, run-metadata) to the v1 JSON schema. Aggregates per-
model results into one cell (pass iff all pass). build_from_paths is the
thin I/O wrapper for the publisher.
- tests/claude_code/sample_compatibility-matrix.json — hand-authored sample
of the v1 JSON; copied to the docs repo by hand as part of this slice.
- Unit tests: 10 driver tests (mocked subprocess), 9 compat_result tests,
10 matrix-builder golden-file tests. 29/29 pass.
Key decisions:
- (feature, provider) is inferred from file path, not declared in metadata —
mirrors the PRD's "no drift" goal.
- Driver injects subprocess via a `runner` kwarg so unit tests don't need
the real `claude` CLI; production callers leave it default.
- Builder is a pure function on Mappings/Sequences; load/write live in a
thin `build_from_paths` wrapper. Golden-file tests pin the schema.
- `_driver_unit_tests/` and `_builder_unit_tests/` are prefixed with `_`
so the conftest's path-inference hook skips them and they don't
pollute the matrix artifact.
- `compat-results.json` added to .gitignore (CI-only output).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The MDX page `docs/tutorials/claude-code-compatibility` and the
`<CompatibilityMatrix />` React component. The hand-authored
compatibility-matrix.json (`sample_compatibility-matrix.json` in this
repo) is the artifact those docs files will consume; opening that doc
PR is the next step in this slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
