The cache edge keyed every recording on its own process's PYTEST_CURRENT_TEST.
Under xdist that names whatever test the serving worker is in, which is
unrelated to the caller: the proxy is a separate pod, and the Claude Code compat
matrix registered its shared aliases from every worker, each pointing at that
worker's edge, so the router spread one worker's calls across all eight edges.
Builds 234 and 235 of litellm-e2e, same commit, credited the same Bedrock
request to unrelated tests 92% of the time, and Bedrock never converged past a
~20% hit rate while OpenAI, whose deployments are per test, sat at 90%.
A deployment registered from inside a test now carries its test's slug in the
edge URL it is pointed at, `{edge}/{mount}/t/{slug}`, and the edge reads that
segment off every request before forwarding. A request without one is forwarded
live and never cached, and the edge no longer falls back to process state. The
compat aliases are registered with provider_live=True and stay on their real
provider path: no single test owns them, and the matrix exists to prove the real
CLI against real providers.
Build 227 mounted Gemini and turned TestGeminiFiles::test_gemini_file_upload
red. litellm's two Gemini endpoints disagree about what api_base means.
Chat composes {api_base}/models/{model}:{endpoint} and defaults api_base to
https://generativelanguage.googleapis.com/v1beta, so the version lives
inside it. File upload composes {api_base}/upload/v1beta/files and defaults
to the host root, so the version lives outside it. A single api_base cannot
satisfy both, and a registration carries no signal about which endpoint the
deployment will be used for, so the edge cannot route one and not the other.
Backing it out rather than working around it. The cache must never turn a
passing test red, which is the same rule the Bedrock model allowlist
follows, and Gemini was 7 of roughly 1030 edge calls in that build. Anyone
pointing litellm's Gemini provider at an AI gateway or a corporate proxy
hits this too, so the fix belongs in litellm; mounting Gemini is one line
once it lands.
This reverts commit 8a553ceb58.
Build 226's 62 Bedrock rejections are the question this is trying to
answer, and "incomplete" would have covered both candidate causes at once.
Replaying the completeness rules over eight streams captured from live
Bedrock, covering tool use, extended thinking and a max-tokens stop on
both streaming endpoints, accepts every one of them, so a rule that is too
strict is the less likely half. A provider that answered 429 or 5xx and
was retried out of sight is the other, and it now counts as
rejected_error_status rather than being folded in with a grammar failure.
Build 226 routed Bedrock streaming for the first time and rejected 62 of
220 misses on that mount, and the counters could not say why. A flat
rejected count covers three unrelated things with opposite fixes: the
consumer walking away mid-capture, a body that arrived whole and failed
its endpoint's rule, and a provider that could not be reached. Each now
also counts its own reason.
A consumer that walks away was counting nothing at all. Abandoning the
capture generator raises GeneratorExit at its yield, so neither branch of
the old accounting ran and the miss simply vanished from the report, which
is also why misses could exceed writes plus rejected with nothing to
explain the gap. The decision moves into settle() so the generator's
finally owns the accounting and an abandoned capture is counted like any
other rejection.
Gemini needs none of the machinery Bedrock needed. litellm composes
{api_base}/models/{model}:{endpoint} from a custom api_base, so a plain
path-prefixed mount reaches it, and the credential travels as a static
x-goog-api-key header that no host rewrite invalidates. Nothing is
re-signed and nothing leaves the cache key, so a recording still cannot
cross credentials.
A finished turn names a finishReason on every candidate and reports
usageMetadata. The reason is read as a string rather than compared to
STOP: MAX_TOKENS and the safety reasons end a turn just as finally, and
rejecting them would send every one of them upstream forever. Streaming
is the half worth care. Gemini repeats usageMetadata on every chunk and
names a finishReason only on the last, so the terminator is the final
event rather than any event, and a stream the connection cut short ends
on a chunk carrying usage and no reason.
The mount's upstream base carries the API version, so the path the rules
see is /v1beta/models/..., not the one the proxy sent. The first version
of this anchored the rule at the start of that path, which passed every
test against a stub with no version prefix and would have cached nothing
at all in a real run. Caught by replaying the rules over responses
captured from live gemini-2.5-flash, which is also why the tests now
mount their stub under the version prefix.
Vertex stays unmounted and is a separate provider here: litellm grafts
the default Vertex path onto an api_base only when that api_base has no
path of its own, so Vertex needs a root-mounted edge on its own port.
The Claude Code compat cells drive the real CLI, which always streams, so
converse-stream and invoke-with-response-stream were most of the suite's
Bedrock traffic and all of it bypassed the edge.
AWS frames those as binary vnd.amazon.eventstream rather than SSE, so
botocore's own parser reads the frames and validates both CRCs, and each
endpoint is then held to its terminal grammar. Two details drove the rule.
A ConverseStream ends with metadata, not with messageStop, and metadata is
what carries the token usage litellm prices the call from, so a stream cut
between the two names a stop reason but would replay as a free call. And a
dropped connection is invisible to the parser: it yields the frames it did
receive and silently discards a trailing partial one, so a stream cut one
byte short parses clean. The body is checked against the frame lengths it
declares to catch that.
The invoke stream carries the ordinary Anthropic event grammar inside its
chunk frames, so it shares the completeness rule with the SSE mounts.
Validated against three real Bedrock eventstream captures, and the tests
build their own frames rather than pasting a capture, with one test holding
that framing to botocore's parser.
The allowlist rejects an unlisted model before the region resolver runs, so the
two negative cases that used to cover the resolver were passing for the wrong
reason and two mutations of it survived. Answering an env-referenced region with
the default mount is only sound because every allowlisted model is a `us.`
profile that fans out across the US regions, so assert that on the list itself
and drop the per-call branch it made unreachable.
The edge re-signs with the run pod's identity, whose IAM policy is an explicit
per-model allowlist. Matching on the `anthropic.` infix instead routed every
Anthropic-on-Bedrock model, so a model outside the policy came back 403 from
Bedrock with no fallback, taking the whole claude_code Bedrock matrix red.
An unlisted model now keeps its direct path and loses only caching.
Every OpenAI Responses body carries `error: null` at the top level, and the
completeness check tested the key's presence rather than its value, so it
rejected every single one. The cost was silent: nothing failed, the endpoint
simply never cached, which is exactly the outcome the endpoint was added for.
Found by driving the edge against the real providers rather than the synthetic
fixtures, which carried no error key at all. Reading the value instead of the
key is also more accurate for chat completions and messages, where a real error
body carries a populated error object.
Almost every Bedrock deployment in the suite declares
aws_region_name="os.environ/AWS_REGION". The mount resolver treated that
string as a region name, produced a mount nothing serves, and left the whole
Anthropic-on-Bedrock surface on its direct path, which is the one thing
mounting Bedrock was for.
The run pod does not share the proxy's environment, so the harness genuinely
cannot resolve that reference. A `us.` inference profile fans out across the US
regions and is reachable from any of them, so those route to the default mount
whatever the proxy resolved. A model that is not cross-region and declares its
region that way keeps its direct path rather than being sent to a region it may
not exist in.
Chat completions and messages were the only cacheable paths. The suite also
drives /v1/embeddings and /v1/responses through the same OpenAI mount, so both
now cache, each with its own completeness rule: a chat response's `choices`
check would reject a perfectly good embedding, and a Responses run that never
reached `response.completed` must stay out of the cache the same way a
truncated stream does.
Vertex and Gemini stay off the edge. litellm's `_check_custom_proxy` rewrites a
path-prefixed vertex api_base into `{api_base}:{endpoint}`, dropping project,
location and model, so a mount under a path prefix cannot work without a
root-mounted edge on its own port or a change in litellm. Shipping an
unvalidated URL guess would have been worse than saying so in PROVIDER_CACHE.md.
Also finishes the MountPolicy move: a mount now carries its signer and its
unkeyed headers together instead of a bare signer map.
The exact-request cache reused 5% of routed traffic (build 218: 19 hits,
350 misses) because every test salts its prompt with a fresh unique_marker(),
so the same test could never match itself across builds. It also routed only
openai and anthropic, while the week's flakiness was Bedrock.
Key is now HMAC(test id + method + URL + headers + body, with every
unique_marker() token replaced by a placeholder, + FIFO slot index). The slot
index is what keeps two marker-only-different calls in one test on two
recordings and therefore two provider response ids, so spend rows still
reconcile one per invocation. A call outside any test is not cacheable.
Bedrock gets a region-qualified mount and SigV4 re-signing, since the edge
rewrites the Host the proxy signed. Signature headers are excluded from the
key for signing mounts only, because x-amz-date would otherwise make every
Bedrock request a permanent miss; every other mount still keys on its
credentials whole. Only Anthropic-on-Bedrock chat deployments route:
embeddings, image generation, rerank and realtime keep their direct path, and
so do deployments carrying their own aws_role_name or static keys, whose whole
point is to prove the product's assume-role chain rather than the runner's.
The two eventstream actions bypass the cache and go live, still signed.
Counters are now attributed per mount as well as in total, so a build can
report a per-provider hit rate instead of one number.
The first cache-enabled litellm-e2e build (211) showed three gaps in the shared provider cache:
Every OpenAI response carries Cloudflare bot-management Set-Cookie headers, and the capture rejected any response with Set-Cookie, so no OpenAI response was ever recorded (179 of 372 misses rejected). The edge already withholds Set-Cookie from the proxy, so drop it before validating and storing instead of rejecting.
The provider prompt-caching tests need fresh provider state: a replayed priming response reports cache creation rather than a cache read, and the TPM test then trips the key limit. Mark both modules provider_live.
TestApiBaseSeam::test_live_mode_returns_none ran inside the cache-enabled runner and saw the shared edge; isolate it from E2E_PROVIDER_CACHE.
