The cron systemd unit's `ProtectHome=read-only` blocks writes to
/home/mateo but still allows reads. With `HOME=/home/mateo` forwarded
to the `claude` subprocess, a compromised @anthropic-ai/claude-code
release (running during the `claude --version` probe) — or a
model-directed `Read` tool call during a PDF cell (which passes
`--allowed-tools Read`) — could read host credential files like
~/.config/gh/hosts.yml (gh-host token), ~/.ssh/, or ~/.bash_history
and exfiltrate them.
Two complementary mitigations, addressing veria's exact recommendation:
1. Per-invocation isolated HOME for every `claude` subprocess:
* cli_driver.py: drop HOME from _CLI_ENV_ALLOWLIST; create a
fresh empty tmpdir under tempfile.gettempdir() (`PrivateTmp=true`
keeps it on a service-private tmpfs) and pass it as HOME to
each `claude` invocation. Cleaned up in a `finally` so
timeouts and CLI-not-found don't leak tmpdirs.
* run_daily.sh: the up-front `claude --version` probe also runs
under $CLAUDE_PROBE_HOME (a per-run dir under ${WORKDIR}) so
the probe can never reach the runtime user's real home; the
existing `cleanup` trap removes ${WORKDIR}.
* Closes the `os.path.expanduser('~/.config/gh/hosts.yml')`-style
attack from a compromised CLI / model.
2. Filesystem-level hiding of credential dotdirs in the systemd unit:
* Add `InaccessiblePaths=-/home/mateo/.config/gh -/home/mateo/.ssh
-/home/mateo/.aws -/home/mateo/.docker -/home/mateo/.kube
-/home/mateo/.gnupg`. The kernel hides these paths from every
process in the unit's mount namespace, defeating the absolute-path
attack (`Read('/home/mateo/.config/gh/...')`) that the per-
invocation HOME override alone cannot block.
* Drop `/home/mateo/.config/gh` from `ReadWritePaths=` (it's
now hidden, and we pass GH_TOKEN inline to every `gh` call).
* Pass GH_TOKEN inline to `gh repo clone` in run_daily.sh
(was relying on host gh-cli config); the docs repo is public
so this is a no-op functionally, but it lets us drop the
~/.config/gh dependency entirely.
Tests:
* test_run_claude_uses_isolated_per_invocation_home: pin that the
CLI subprocess never sees the parent's $HOME, and that the
isolated HOME is a fresh tmpdir prefixed claude-cli-home-.
* test_run_claude_isolated_home_is_distinct_per_invocation: pin that
each call gets its own dir (no cross-call planting).
* test_run_claude_isolated_home_cleaned_up_after_run / on_subprocess
_failure: pin that the tmpdir is rm-rf'd on both the happy path
and the timeout/CLI-error path.
* test_version_probe_uses_isolated_home_not_runtime_user_home: pin
that run_daily.sh's probe forwards $CLAUDE_PROBE_HOME, not
${HOME}, into its `env -i` block.
* test_systemd_unit_credential_isolation.py (new): pin that
InaccessiblePaths covers all credential dotdirs, that
.config/gh is not under ReadWritePaths, and that ProtectHome
stays at least read-only.
All 349 existing claude_code unit tests still pass.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- run_daily.sh: extract semver via grep -oE instead of awk '{print $1}' so
the parsed version survives a 'claude --version' output that ever prepends
a label (e.g. 'Claude Code vX.Y.Z'). The previous awk pattern would silently
publish the wrong string in that case.
- _driver_unit_tests/test_basic_messaging.py: narrow pytest.raises(BaseException)
to pytest.raises(pytest.fail.Exception). pytest.fail() raises Failed, which
inherits from BaseException; the new bound matches what the helper raises
without also swallowing KeyboardInterrupt/SystemExit.
Address the Greptile concern that basic_messaging_streaming and
basic_messaging_non_streaming used the same implementation, so a proxy
that buffered the upstream stream would silently show green for the
streaming row.
The fix:
- _basic_messaging.run_basic_messaging_cell accepts verify_streaming=True,
which passes --include-partial-messages to the claude CLI. That flag
causes the CLI to emit one stream_event record per upstream SSE event
(message_start, content_block_delta, message_stop, ...). A buffering
proxy collapses the stream to a single non-streaming response, so
zero stream_event records are emitted.
- The cell rejects any model whose stream_event count is below
MIN_STREAM_DELTA_EVENTS (2) -- safely above the buffered case for any
non-trivial reply. Same all-must-pass shape as the existing
tool_use_streaming row.
- All five basic_messaging_streaming/test_*.py per-provider cells now
pass verify_streaming=True; the non-streaming variants are unchanged.
- New unit tests cover the helper, the partial-messages flag wiring,
the streamed/buffered branching, and the all-models-must-stream
contract.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
- run_claude_models_parallel: catch all exceptions in the per-model
worker and wrap unexpected ones into a ClaudeCLIError so the
documented 'errors as values' contract holds for OSError, ValueError,
etc., not just ClaudeCLIError. Without this, an unexpected raise in
any layer (rate limiter file I/O, infer_provider, etc.) abandons the
remaining models' results and crashes the calling test.
- test_run_claude_places_extra_args_before_prompt: drop the dead first
branch of the 'or' assertion — cmd[-3:] never matches that shape, so
the alternative was misleading dead code.
- basic_messaging_{non_streaming,streaming}/test_*.py: extract the
shared cell body into tests/claude_code/_basic_messaging.py.
Each per-provider file now declares its model list and calls
run_basic_messaging_cell(), eliminating ~700 lines of copy-paste
across 10 files. Updated _builder_unit_tests/test_v0_layout.py to
accept the helper-based pattern alongside direct run_claude() calls.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
run_claude validated empty prompt strings but silently accepted
stdin_input="", letting an empty stdin reach the subprocess and surface
as a confusing CLI failure instead of a clear ValueError.
These three cells were failing for reasons unrelated to LiteLLM
translation:
- vision: tests passed `--image <path>`, a flag that no longer exists
in Claude Code 2.x (image attachment is now via the Files API or via
`--input-format stream-json` with inline content blocks). Rewrite
the cells to feed an Anthropic-shaped user message containing both
text and a base64 `image` content block through stdin in stream-json
mode. Hermetic — no temp file or Files API upload needed.
- extended_thinking: tests set `MAX_THINKING_TOKENS=4096` as an env
var, which Claude Code 2.x ignores. Switch to `--effort max` (the
current CLI knob) and use a non-trivial prompt (3-gallon / 5-gallon
jug puzzle). With trivial arithmetic the modern Sonnet/Opus tiers
optimize away the thinking step and arrive without a thinking block,
which made the test silently false-fail.
- web_search: assertion looked for `server_tool_use` /
`web_search_tool_result` blocks, but Claude Code's `WebSearch` is
a *client-side* tool: the CLI executes the search itself and feeds
the result back as a regular `tool_result` block. The Anthropic
server-side `web_search_20250305` tool only fires when injected
into the request directly (which the CLI does not do). Update the
assertion to look for a `tool_use` block whose name is
`WebSearch` — that's the right signal that the proxy preserved
both the request-side tool definition and the response-side tool_use
block end-to-end.
Driver change required to support stream-json input + variadic flags:
- cli_driver: insert `--` before the prompt positional. Variadic
flags like `--allowed-tools <tools...>` (commander.js) greedily
consume every following token, so the prompt was being eaten as a
tool name and the CLI would error out with "Input must be provided
either through stdin or as a prompt argument when using --print".
- cli_driver: thread a `stdin_input` parameter through `run_claude`
and `run_claude_models_parallel` so the vision rewrite can pipe
stream-json events to the CLI on stdin.
Validated end-to-end against a live LiteLLM proxy: all three Anthropic
cells now pass on Haiku 4.5, Sonnet 4.6, and Opus 4.7. Driver unit
tests (120) still green.
Cursor security review flagged that run_claude() forwarded the entire
parent environment to the externally installed claude CLI binary. In
the PR gate flow the binary is dynamically installed from npm, and
the surrounding job loads every upstream provider credential
(ANTHROPIC_API_KEY, AWS_*, AZURE_FOUNDRY_*, VERTEXAI_CREDENTIALS,
GITHUB_TOKEN, ...) into its env so the proxy can route requests. A
compromised CLI release would have read access to all of them — even
though the CLI itself only ever talks to the proxy via the explicit
ANTHROPIC_BASE_URL/ANTHROPIC_AUTH_TOKEN we set.
Build the subprocess env from a small allowlist of process-runtime
vars (PATH, HOME, NVM_DIR, locale) rather than inheriting all of
os.environ. Caller-supplied extra_env still rides on top, which is
the sanctioned way for tests to opt-in to passing additional vars
(e.g. extended_thinking sets MAX_THINKING_TOKENS).
Add unit tests pinning the contract: PATH/HOME flow through, secrets
do not, and extra_env can still override anything.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Switch all per-cell tests from @pytest.mark.parametrize("model", ...)
(3 sequential invocations) to a single test that fans out to all 3
Claude tiers via run_claude_models_parallel. Per-cell wall time is now
bounded by the slowest model rather than the sum.
- Add 5 new v0 feature dirs (5 providers each, 25 new test files):
web_search, pdf_input, prompt_caching_1h,
tool_use_streaming, thinking_with_tool_use
Manifest expanded to match.
- Add cross-process token-bucket rate limiter (rate_limiter.py + tests)
so xdist workers stay under per-provider req/s limits during full-grid
runs. New env knobs: LITELLM_COMPAT_RATE_{ANTHROPIC,AZURE,VERTEX_AI,
BEDROCK_CONVERSE,BEDROCK_INVOKE}.
- conftest.py: write per-worker shards under <artifact>.shards/, merge
in the controller; preserve the "don't write empty artifact" guard so
unit-test runs don't clobber a real compat-results.json.
- Vertex test_config.yaml: route project/location through env so the
cron VM can target a different GCP project than the upstream default.
- Add run_compat.sh wrapper for binary-searching ideal req/s per
provider against compat-rate-limit-summary.json output.
Bugbot flagged that `run_claude` placed the prompt as cmd[7] and then
appended extra_args after it. `claude --print` takes the prompt as the
final positional argument; flags appearing after it (e.g.
`--allowed-tools Bash`, `--image <path>`) are swallowed by the prompt
parser, which silently breaks the tool_use and vision cells.
Build the flag list first, then append the prompt last. Add a unit
test that pins the ordering.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Slice 1 of the Claude Code Compatibility Matrix: the thinnest end-to-end
path through every layer for a single (feature, provider) cell, so a
future docs page can render a real green cell sourced from a real test.
What landed in this repo:
- tests/claude_code/manifest.yaml — feature manifest with one entry
(basic_messaging_non_streaming) plus the v0 provider column order.
- tests/claude_code/cli_driver.py — Claude Code CLI Driver. One entry
point (run_claude); handles subprocess assembly, env overlay, stream-JSON
parsing, and structured failure modes. `runner=` is a unit-test seam.
- tests/claude_code/conftest.py — `compat_result` fixture (tagged-union
recorder) + pytest_runtest_makereport hook that infers (feature, provider)
from the file path and writes a structured compat-results.json artifact.
- tests/claude_code/basic_messaging_non_streaming/test_anthropic.py — the
one cell, parametrized over Haiku/Sonnet/Opus per the PRD's per-cell
model rule.
- tests/claude_code/matrix_builder.py — pure-function builder from
(manifest, results, run-metadata) to the v1 JSON schema. Aggregates per-
model results into one cell (pass iff all pass). build_from_paths is the
thin I/O wrapper for the publisher.
- tests/claude_code/sample_compatibility-matrix.json — hand-authored sample
of the v1 JSON; copied to the docs repo by hand as part of this slice.
- Unit tests: 10 driver tests (mocked subprocess), 9 compat_result tests,
10 matrix-builder golden-file tests. 29/29 pass.
Key decisions:
- (feature, provider) is inferred from file path, not declared in metadata —
mirrors the PRD's "no drift" goal.
- Driver injects subprocess via a `runner` kwarg so unit tests don't need
the real `claude` CLI; production callers leave it default.
- Builder is a pure function on Mappings/Sequences; load/write live in a
thin `build_from_paths` wrapper. Golden-file tests pin the schema.
- `_driver_unit_tests/` and `_builder_unit_tests/` are prefixed with `_`
so the conftest's path-inference hook skips them and they don't
pollute the matrix artifact.
- `compat-results.json` added to .gitignore (CI-only output).
Out of scope per CLAUDE.md (docs live in BerriAI/litellm-docs):
- The MDX page `docs/tutorials/claude-code-compatibility` and the
`<CompatibilityMatrix />` React component. The hand-authored
compatibility-matrix.json (`sample_compatibility-matrix.json` in this
repo) is the artifact those docs files will consume; opening that doc
PR is the next step in this slice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
