A deployment registered while a module- or class-scoped fixture is being set up
was bound to whichever test asked for the fixture first, so every later test in
the module shared that partition. A session-scoped fixture is set up by every
xdist worker, so its deployment could never have one owner at all.
The e2e conftest now wraps pytest_fixture_setup and records the node the fixture
is scoped to: registrations made during a module or class fixture's setup carry
that node's slug, and a session- or package-scoped one has no owner and stays
live. The registration seam test moves from tests/e2e to the cache harness tests
beside the rest of the attribution coverage.
The cache edge keyed every recording on its own process's PYTEST_CURRENT_TEST.
Under xdist that names whatever test the serving worker is in, which is
unrelated to the caller: the proxy is a separate pod, and the Claude Code compat
matrix registered its shared aliases from every worker, each pointing at that
worker's edge, so the router spread one worker's calls across all eight edges.
Builds 234 and 235 of litellm-e2e, same commit, credited the same Bedrock
request to unrelated tests 92% of the time, and Bedrock never converged past a
~20% hit rate while OpenAI, whose deployments are per test, sat at 90%.
A deployment registered from inside a test now carries its test's slug in the
edge URL it is pointed at, `{edge}/{mount}/t/{slug}`, and the edge reads that
segment off every request before forwarding. A request without one is forwarded
live and never cached, and the edge no longer falls back to process state. The
compat aliases are registered with provider_live=True and stay on their real
provider path: no single test owns them, and the matrix exists to prove the real
CLI against real providers.
Team admins are refused until a proxy admin enables a field, then limited to the enabled fields, and resending unchanged budget settings keeps the team's budget reset times
It drives the real CLI for several seconds. The edge stamps every
upstream call with PYTEST_CURRENT_TEST, a process-global that names
whichever test the worker is in when the call arrives rather than the one
that made it, so a test that holds a worker that long collects other
tests' in-flight calls. Build 234's key report credits this test with 20
Bedrock and 7 Anthropic misses, and it makes no provider call at all.
Those misattributed calls take the wrong test id into the cache key and
write recordings under it, so the test was polluting the shared corpus it
exists to protect.
Deselected unless E2E_CLI_DETERMINISM is set, the same opt-in shape the
managed-files, prompt-caching and redis-chaos markers already use. The
attribution bug itself is older than this branch and is reported, not
fixed here.
_PROXY_MaxBudgetLimiter re-checked spend:user:{id} against user_max_budget in
async_pre_call_hook without the zero-cost model exemption that
_user_max_budget_check applies in auth, so free models were rejected with
"Max budget limit reached." once a user was over budget. Auth already owns
this check, so the hook is deleted rather than taught the exemption again
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Builds 232 and 233 held the Bedrock hit rate at 9% with the Claude Code
driver already sending byte-identical requests and headers, so something
between the proxy's ingress and the upstream still moves per build and
the flat key cannot say what.
Emit a digest per key component next to the counters: the test id, the
method, the URL, each keyed header, the whole body, and one digest per
top-level JSON body field. Values are digested, so no payload or
credential reaches the artifact. Diffing two builds' artifacts names the
field that moved.
Diagnostic, to be removed once it has answered.
Build 232 took two compat cells red with a FileNotFoundError renaming
`.claude.json.197` onto `.claude.json`. `run_claude_models_parallel`
drives several models from one process, so a pid-suffixed staged name is
shared between threads: one thread renamed the file the other was still
writing, and the loser died on a path that no longer existed.
mkstemp in the same directory gives a name that is unique per thread as
well as per process, and the rename stays atomic.
Adds integration contracts for MCP lifecycle, protocol errors and OAuth
configuration, A2A wire versions, the OpenAI consumer path, persisted
toolsets, callback delivery, guardrail effects, configured prices, the
filtered spend ledger, and a CircleCI-owned browser flow for project
detachment, with the ASGI, browser-state, client and MCP helpers they use.
Consolidates the eleven commits previously stacked on
litellm_integration_providers onto its rebased tip
The compat cells drove the CLI with a fresh HOME per invocation and the
pytest process's own working directory, and both reach the request body.
The system prompt names a memory directory built from
$CLAUDE_CONFIG_DIR/projects/<cwd slug>, so a per-invocation config
directory rewrote every body, and the CLI adds a git block for its
working directory, so inheriting the checkout rewrote every body once
per candidate. The device id churned for the same reason: the CLI mints
it once and persists it in .claude.json, which we threw away each call.
Nothing here was load-bearing. All three ride in metadata.user_id, whose
job is abuse detection, not quota, caching or continuity. So pin the
config directory and the working directory at fixed paths, seed the
device id, and pin the session id.
HOME stays fresh and empty per invocation, so the isolation is no weaker
than before, and the CLI's own state no longer outlives the pod either.
The working directory is deliberately not the checkout, so a
model-directed Read now sees an empty directory rather than the
repository.
A pinned session id needs --no-session-persistence beside it: the CLI
refuses a session id another live process holds, and the matrix runs its
cells across xdist workers. Without the flag, six of eight concurrent
invocations die on "Session ID is already in use".
Build 227 mounted Gemini and turned TestGeminiFiles::test_gemini_file_upload
red. litellm's two Gemini endpoints disagree about what api_base means.
Chat composes {api_base}/models/{model}:{endpoint} and defaults api_base to
https://generativelanguage.googleapis.com/v1beta, so the version lives
inside it. File upload composes {api_base}/upload/v1beta/files and defaults
to the host root, so the version lives outside it. A single api_base cannot
satisfy both, and a registration carries no signal about which endpoint the
deployment will be used for, so the edge cannot route one and not the other.
Backing it out rather than working around it. The cache must never turn a
passing test red, which is the same rule the Bedrock model allowlist
follows, and Gemini was 7 of roughly 1030 edge calls in that build. Anyone
pointing litellm's Gemini provider at an AI gateway or a corporate proxy
hits this too, so the fix belongs in litellm; mounting Gemini is one line
once it lands.
This reverts commit 8a553ceb58.
Build 226's 62 Bedrock rejections are the question this is trying to
answer, and "incomplete" would have covered both candidate causes at once.
Replaying the completeness rules over eight streams captured from live
Bedrock, covering tool use, extended thinking and a max-tokens stop on
both streaming endpoints, accepts every one of them, so a rule that is too
strict is the less likely half. A provider that answered 429 or 5xx and
was retried out of sight is the other, and it now counts as
rejected_error_status rather than being folded in with a grammar failure.
Build 226 routed Bedrock streaming for the first time and rejected 62 of
220 misses on that mount, and the counters could not say why. A flat
rejected count covers three unrelated things with opposite fixes: the
consumer walking away mid-capture, a body that arrived whole and failed
its endpoint's rule, and a provider that could not be reached. Each now
also counts its own reason.
A consumer that walks away was counting nothing at all. Abandoning the
capture generator raises GeneratorExit at its yield, so neither branch of
the old accounting ran and the miss simply vanished from the report, which
is also why misses could exceed writes plus rejected with nothing to
explain the gap. The decision moves into settle() so the generator's
finally owns the accounting and an abandoned capture is counted like any
other rejection.
Gemini needs none of the machinery Bedrock needed. litellm composes
{api_base}/models/{model}:{endpoint} from a custom api_base, so a plain
path-prefixed mount reaches it, and the credential travels as a static
x-goog-api-key header that no host rewrite invalidates. Nothing is
re-signed and nothing leaves the cache key, so a recording still cannot
cross credentials.
A finished turn names a finishReason on every candidate and reports
usageMetadata. The reason is read as a string rather than compared to
STOP: MAX_TOKENS and the safety reasons end a turn just as finally, and
rejecting them would send every one of them upstream forever. Streaming
is the half worth care. Gemini repeats usageMetadata on every chunk and
names a finishReason only on the last, so the terminator is the final
event rather than any event, and a stream the connection cut short ends
on a chunk carrying usage and no reason.
The mount's upstream base carries the API version, so the path the rules
see is /v1beta/models/..., not the one the proxy sent. The first version
of this anchored the rule at the start of that path, which passed every
test against a stub with no version prefix and would have cached nothing
at all in a real run. Caught by replaying the rules over responses
captured from live gemini-2.5-flash, which is also why the tests now
mount their stub under the version prefix.
Vertex stays unmounted and is a separate provider here: litellm grafts
the default Vertex path onto an api_base only when that api_base has no
path of its own, so Vertex needs a root-mounted edge on its own port.
The Claude Code compat cells drive the real CLI, which always streams, so
converse-stream and invoke-with-response-stream were most of the suite's
Bedrock traffic and all of it bypassed the edge.
AWS frames those as binary vnd.amazon.eventstream rather than SSE, so
botocore's own parser reads the frames and validates both CRCs, and each
endpoint is then held to its terminal grammar. Two details drove the rule.
A ConverseStream ends with metadata, not with messageStop, and metadata is
what carries the token usage litellm prices the call from, so a stream cut
between the two names a stop reason but would replay as a free call. And a
dropped connection is invisible to the parser: it yields the frames it did
receive and silently discards a trailing partial one, so a stream cut one
byte short parses clean. The body is checked against the frame lengths it
declares to catch that.
The invoke stream carries the ordinary Anthropic event grammar inside its
chunk frames, so it shares the completeness rule with the SSE mounts.
Validated against three real Bedrock eventstream captures, and the tests
build their own frames rather than pasting a capture, with one test holding
that framing to botocore's parser.
The key normalizes a unique marker so two builds match, which is the whole
point, but it makes this test's identity collide with an earlier run's: it mints
a fresh receipt, sends it through a tool result, and asserts the model echoes it
back verbatim, so a stale recording matched and answered with the old receipt.
Build 223 is where that surfaced, once the corpus was full enough for the first
call to hit. A test that asserts a provider echoed this run's own unique value
belongs on the live path.
The allowlist rejects an unlisted model before the region resolver runs, so the
two negative cases that used to cover the resolver were passing for the wrong
reason and two mutations of it survived. Answering an env-referenced region with
the default mount is only sound because every allowlisted model is a `us.`
profile that fans out across the US regions, so assert that on the list itself
and drop the per-call branch it made unreachable.
The edge re-signs with the run pod's identity, whose IAM policy is an explicit
per-model allowlist. Matching on the `anthropic.` infix instead routed every
Anthropic-on-Bedrock model, so a model outside the policy came back 403 from
Bedrock with no fallback, taking the whole claude_code Bedrock matrix red.
An unlisted model now keeps its direct path and loses only caching.
Every OpenAI Responses body carries `error: null` at the top level, and the
completeness check tested the key's presence rather than its value, so it
rejected every single one. The cost was silent: nothing failed, the endpoint
simply never cached, which is exactly the outcome the endpoint was added for.
Found by driving the edge against the real providers rather than the synthetic
fixtures, which carried no error key at all. Reading the value instead of the
key is also more accurate for chat completions and messages, where a real error
body carries a populated error object.
Almost every Bedrock deployment in the suite declares
aws_region_name="os.environ/AWS_REGION". The mount resolver treated that
string as a region name, produced a mount nothing serves, and left the whole
Anthropic-on-Bedrock surface on its direct path, which is the one thing
mounting Bedrock was for.
The run pod does not share the proxy's environment, so the harness genuinely
cannot resolve that reference. A `us.` inference profile fans out across the US
regions and is reachable from any of them, so those route to the default mount
whatever the proxy resolved. A model that is not cross-region and declares its
region that way keeps its direct path rather than being sent to a region it may
not exist in.
Chat completions and messages were the only cacheable paths. The suite also
drives /v1/embeddings and /v1/responses through the same OpenAI mount, so both
now cache, each with its own completeness rule: a chat response's `choices`
check would reject a perfectly good embedding, and a Responses run that never
reached `response.completed` must stay out of the cache the same way a
truncated stream does.
Vertex and Gemini stay off the edge. litellm's `_check_custom_proxy` rewrites a
path-prefixed vertex api_base into `{api_base}:{endpoint}`, dropping project,
location and model, so a mount under a path prefix cannot work without a
root-mounted edge on its own port or a change in litellm. Shipping an
unvalidated URL guess would have been worse than saying so in PROVIDER_CACHE.md.
Also finishes the MountPolicy move: a mount now carries its signer and its
unkeyed headers together instead of a bare signer map.
The exact-request cache reused 5% of routed traffic (build 218: 19 hits,
350 misses) because every test salts its prompt with a fresh unique_marker(),
so the same test could never match itself across builds. It also routed only
openai and anthropic, while the week's flakiness was Bedrock.
Key is now HMAC(test id + method + URL + headers + body, with every
unique_marker() token replaced by a placeholder, + FIFO slot index). The slot
index is what keeps two marker-only-different calls in one test on two
recordings and therefore two provider response ids, so spend rows still
reconcile one per invocation. A call outside any test is not cacheable.
Bedrock gets a region-qualified mount and SigV4 re-signing, since the edge
rewrites the Host the proxy signed. Signature headers are excluded from the
key for signing mounts only, because x-amz-date would otherwise make every
Bedrock request a permanent miss; every other mount still keys on its
credentials whole. Only Anthropic-on-Bedrock chat deployments route:
embeddings, image generation, rerank and realtime keep their direct path, and
so do deployments carrying their own aws_role_name or static keys, whose whole
point is to prove the product's assume-role chain rather than the runner's.
The two eventstream actions bypass the cache and go live, still signed.
Counters are now attributed per mount as well as in total, so a build can
report a per-provider hit rate instead of one number.
The first cache-enabled litellm-e2e build (211) showed three gaps in the shared provider cache:
Every OpenAI response carries Cloudflare bot-management Set-Cookie headers, and the capture rejected any response with Set-Cookie, so no OpenAI response was ever recorded (179 of 372 misses rejected). The edge already withholds Set-Cookie from the proxy, so drop it before validating and storing instead of rejecting.
The provider prompt-caching tests need fresh provider state: a replayed priming response reports cache creation rather than a cache read, and the TPM test then trips the key limit. Mark both modules provider_live.
TestApiBaseSeam::test_live_mode_returns_none ran inside the cache-enabled runner and saw the shared edge; isolate it from E2E_PROVIDER_CACHE.
