Ran the before/after proof live against Vertex Claude (global endpoint,
project vertex-check-481318): base transform 400s an unflagged model
(claude-opus-4-7) on a mid-conversation role:system reminder, the fix
hoists it to a 200, and a flagged model (claude-opus-4-8) keeps the
reminder in messages with cache_read held at 15615 across the reminder
turn. Flip both vertex.mid_conversation_system rows to fail_before_fix:
proven.
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted
Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400
Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag
Closes the Vertex/Azure gap from the customer RCA
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping
Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly.
* test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse
Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL,
TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path,
Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm
chat, and Nova Sonic realtime. Registry cells updated for the new markers.
* test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes
Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and
Langfuse driver models now use Anthropic haiku so local runs stay green when
Gemini daily quota is exhausted.
* test(e2e): drop Langfuse spend suite; feature is being deprecated
Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the
langfuse-only conftest driver/credentials fixtures.
* test(e2e): fold provider/batch feature tests into their endpoint suites
Keep the e2e layout endpoint- and suite-scoped instead of one file per
provider or feature
Move the virtual-key auth case into access_control/test_access_control_e2e.py
as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone
test_virtual_key_auth_e2e.py
Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock
assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py.
The hosted_vllm batch case is skipped for now since it needs a live vLLM server
(HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the
gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant
Merge the cohere, gemini and hosted_vllm chat cases into
llm_translation/test_chat_completions_regression_e2e.py so /chat/completions
coverage lives in one endpoint file, and repoint the coverage_registry source
fields to the new homes
Move the shared CacheControl / TextBlock / RichMessage request blocks into the
root models.py (re-exported from endpoints_client) so quota_management can use
them without a cross-suite import, which also clears the basedpyright errors in
test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in
test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed
json.loads path
* test(e2e): address review feedback and re-home virtual-key coverage
Replace the tautological Bedrock assume-role batch id assertion (`startswith(...)
or batch.id`, always true) with a managed-id shape check, since the unified
target_model_names path re-encodes the id rather than returning a raw ARN
Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no
longer consume the key's sole request unit before batch create runs; the batch
create then clears the generic per-request limiter and the batch limiter is what
returns the "Batch rate limit exceeded" body the assertions check
Set exercised_on to [] on the pass-through header test; it drives a pass-through
endpoint, not /chat/completions
Move the virtual-key valid_allows / invalid_denied cells from other.yaml to
mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management,
and point its covers marker at the new ids
Introduce the e2e coverage denominator: 282 behavior cells across the six
tracking modules (LLMs, MCPs, Management/UI, Reliability & Performance,
Logging & Guardrails, Other), one validated YAML row each, plus a collector
that diffs the registry against @pytest.mark.covers markers and reports
coverage per module.
The registry rows validate against a pydantic discriminated union so a row
cannot carry a field from another module. The collector is static: a
collect-only pass reads the markers, so it runs no test and needs no live
proxy. Register the covers marker suite-wide so that pass works under
--strict-markers.
This is a draft for review. Tiers are proposed rather than signed off, and a
few cells still need a support check or a prune.
