Matrix grows from 11 to 15 feature rows. All new tests collected + 180
unit tests still pass; smoke runs hit real LiteLLM bug surfaces on
bedrock_invoke, bedrock_converse, and vertex_ai (cells correctly red
in PR #142).
Rename
------
`extended_thinking` -> `thinking` (directory, manifest id+name, 5
test fn names, 5 docstrings, builder unit-test fixtures, sample JSON,
run_compat.sh). Existing test logic already covers both manual
(`thinking.type=enabled`, Haiku 4.5) and adaptive
(`thinking.type=adaptive`, Opus 4.7) shapes because Claude Code picks
the shape per model from `--effort max`; the name change just stops
the column from looking like a Claude 3.7 reference.
New rows
--------
- structured_outputs (5 files, CLI `--json-schema`). Claude Code
synthesizes a single `StructuredOutput` tool from the schema and
surfaces the tool_use input as `structured_output` on the trailing
`result` event. Test ships its own `_validate_against_schema` so
we don't take a jsonschema dep just for matrix surface.
- count_tokens (5 files, HTTP probe). POSTs the proxy's
`/v1/messages/count_tokens` directly and asserts the response is
`{input_tokens: positive int}`. No CLI hook exists for this
endpoint; the test goes through the new http_probe helper instead.
- tool_search (5 files, HTTP probe). Sends
`tools: [{type: tool_search_tool_regex_20251119, name:
tool_search_tool_regex}]` and asserts the proxy doesn't 400. MCP
fan-out via `--mcp-config` would also exercise the tool-search
beta header path, but it's flaky w.r.t. Claude Code's internal
tool-deferral threshold; the HTTP probe hits the actual bug surface
(per-provider beta-header translation `advanced-tool-use-2025-11-20`
vs `tool-search-tool-2025-10-19`).
- long_context_1m (5 files, CLI `--betas context-1m-2025-08-07
--max-budget-usd 6`). A ~210k-token padded prompt over stdin
exercises the 1M-context beta. Sonnet 4.6 + Opus 4.7 only --
Haiku 4.5's window is 200k, so it's excluded from MODELS (not
marked not_applicable) to keep the per-cell aggregator semantics
intact. Prompt uses a document-style preamble + 8 cycling pangrams
rather than repeating identical chunks; without that, Opus 4.7
trips the safety filter mid-response with a Usage Policy refusal.
`--max-budget-usd 6` is a runaway-loop guard, ~2x worst-case Opus
per-cell spend.
New helper
----------
`tests/claude_code/http_probe.py`: shared `ProbeResult` dataclass
plus per-endpoint `probe_*` + `assert_*_shape` pairs for the
HTTP-probe rows. Uses httpx with `anthropic-version: 2023-06-01` and
a 30s timeout.
- Switch all per-cell tests from @pytest.mark.parametrize("model", ...)
(3 sequential invocations) to a single test that fans out to all 3
Claude tiers via run_claude_models_parallel. Per-cell wall time is now
bounded by the slowest model rather than the sum.
- Add 5 new v0 feature dirs (5 providers each, 25 new test files):
web_search, pdf_input, prompt_caching_1h,
tool_use_streaming, thinking_with_tool_use
Manifest expanded to match.
- Add cross-process token-bucket rate limiter (rate_limiter.py + tests)
so xdist workers stay under per-provider req/s limits during full-grid
runs. New env knobs: LITELLM_COMPAT_RATE_{ANTHROPIC,AZURE,VERTEX_AI,
BEDROCK_CONVERSE,BEDROCK_INVOKE}.
- conftest.py: write per-worker shards under <artifact>.shards/, merge
in the controller; preserve the "don't write empty artifact" guard so
unit-test runs don't clobber a real compat-results.json.
- Vertex test_config.yaml: route project/location through env so the
cron VM can target a different GCP project than the upstream default.
- Add run_compat.sh wrapper for binary-searching ideal req/s per
provider against compat-rate-limit-summary.json output.
