A deployment registered while a module- or class-scoped fixture is being set up
was bound to whichever test asked for the fixture first, so every later test in
the module shared that partition. A session-scoped fixture is set up by every
xdist worker, so its deployment could never have one owner at all.
The e2e conftest now wraps pytest_fixture_setup and records the node the fixture
is scoped to: registrations made during a module or class fixture's setup carry
that node's slug, and a session- or package-scoped one has no owner and stays
live. The registration seam test moves from tests/e2e to the cache harness tests
beside the rest of the attribution coverage.
The cache edge keyed every recording on its own process's PYTEST_CURRENT_TEST.
Under xdist that names whatever test the serving worker is in, which is
unrelated to the caller: the proxy is a separate pod, and the Claude Code compat
matrix registered its shared aliases from every worker, each pointing at that
worker's edge, so the router spread one worker's calls across all eight edges.
Builds 234 and 235 of litellm-e2e, same commit, credited the same Bedrock
request to unrelated tests 92% of the time, and Bedrock never converged past a
~20% hit rate while OpenAI, whose deployments are per test, sat at 90%.
A deployment registered from inside a test now carries its test's slug in the
edge URL it is pointed at, `{edge}/{mount}/t/{slug}`, and the edge reads that
segment off every request before forwarding. A request without one is forwarded
live and never cached, and the edge no longer falls back to process state. The
compat aliases are registered with provider_live=True and stay on their real
provider path: no single test owns them, and the matrix exists to prove the real
CLI against real providers.
Postgres 10 has no fast default path, so ADD COLUMN ... DEFAULT on
LiteLLM_SpendLogs rewrites the heap and every index under an ACCESS
EXCLUSIVE lock inside the boot-time migrate deploy. The checker now
reports it on LiteLLM_SpendLogs and LiteLLM_ErrorLogs; the two shipped
migrations that already do it are grandfathered
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Build 227 mounted Gemini and turned TestGeminiFiles::test_gemini_file_upload
red. litellm's two Gemini endpoints disagree about what api_base means.
Chat composes {api_base}/models/{model}:{endpoint} and defaults api_base to
https://generativelanguage.googleapis.com/v1beta, so the version lives
inside it. File upload composes {api_base}/upload/v1beta/files and defaults
to the host root, so the version lives outside it. A single api_base cannot
satisfy both, and a registration carries no signal about which endpoint the
deployment will be used for, so the edge cannot route one and not the other.
Backing it out rather than working around it. The cache must never turn a
passing test red, which is the same rule the Bedrock model allowlist
follows, and Gemini was 7 of roughly 1030 edge calls in that build. Anyone
pointing litellm's Gemini provider at an AI gateway or a corporate proxy
hits this too, so the fix belongs in litellm; mounting Gemini is one line
once it lands.
This reverts commit 8a553ceb58.
Build 226's 62 Bedrock rejections are the question this is trying to
answer, and "incomplete" would have covered both candidate causes at once.
Replaying the completeness rules over eight streams captured from live
Bedrock, covering tool use, extended thinking and a max-tokens stop on
both streaming endpoints, accepts every one of them, so a rule that is too
strict is the less likely half. A provider that answered 429 or 5xx and
was retried out of sight is the other, and it now counts as
rejected_error_status rather than being folded in with a grammar failure.
Build 226 routed Bedrock streaming for the first time and rejected 62 of
220 misses on that mount, and the counters could not say why. A flat
rejected count covers three unrelated things with opposite fixes: the
consumer walking away mid-capture, a body that arrived whole and failed
its endpoint's rule, and a provider that could not be reached. Each now
also counts its own reason.
A consumer that walks away was counting nothing at all. Abandoning the
capture generator raises GeneratorExit at its yield, so neither branch of
the old accounting ran and the miss simply vanished from the report, which
is also why misses could exceed writes plus rejected with nothing to
explain the gap. The decision moves into settle() so the generator's
finally owns the accounting and an abandoned capture is counted like any
other rejection.
Gemini needs none of the machinery Bedrock needed. litellm composes
{api_base}/models/{model}:{endpoint} from a custom api_base, so a plain
path-prefixed mount reaches it, and the credential travels as a static
x-goog-api-key header that no host rewrite invalidates. Nothing is
re-signed and nothing leaves the cache key, so a recording still cannot
cross credentials.
A finished turn names a finishReason on every candidate and reports
usageMetadata. The reason is read as a string rather than compared to
STOP: MAX_TOKENS and the safety reasons end a turn just as finally, and
rejecting them would send every one of them upstream forever. Streaming
is the half worth care. Gemini repeats usageMetadata on every chunk and
names a finishReason only on the last, so the terminator is the final
event rather than any event, and a stream the connection cut short ends
on a chunk carrying usage and no reason.
The mount's upstream base carries the API version, so the path the rules
see is /v1beta/models/..., not the one the proxy sent. The first version
of this anchored the rule at the start of that path, which passed every
test against a stub with no version prefix and would have cached nothing
at all in a real run. Caught by replaying the rules over responses
captured from live gemini-2.5-flash, which is also why the tests now
mount their stub under the version prefix.
Vertex stays unmounted and is a separate provider here: litellm grafts
the default Vertex path onto an api_base only when that api_base has no
path of its own, so Vertex needs a root-mounted edge on its own port.
The Claude Code compat cells drive the real CLI, which always streams, so
converse-stream and invoke-with-response-stream were most of the suite's
Bedrock traffic and all of it bypassed the edge.
AWS frames those as binary vnd.amazon.eventstream rather than SSE, so
botocore's own parser reads the frames and validates both CRCs, and each
endpoint is then held to its terminal grammar. Two details drove the rule.
A ConverseStream ends with metadata, not with messageStop, and metadata is
what carries the token usage litellm prices the call from, so a stream cut
between the two names a stop reason but would replay as a free call. And a
dropped connection is invisible to the parser: it yields the frames it did
receive and silently discards a trailing partial one, so a stream cut one
byte short parses clean. The body is checked against the frame lengths it
declares to catch that.
The invoke stream carries the ordinary Anthropic event grammar inside its
chunk frames, so it shares the completeness rule with the SSE mounts.
Validated against three real Bedrock eventstream captures, and the tests
build their own frames rather than pasting a capture, with one test holding
that framing to botocore's parser.
The allowlist rejects an unlisted model before the region resolver runs, so the
two negative cases that used to cover the resolver were passing for the wrong
reason and two mutations of it survived. Answering an env-referenced region with
the default mount is only sound because every allowlisted model is a `us.`
profile that fans out across the US regions, so assert that on the list itself
and drop the per-call branch it made unreachable.
The edge re-signs with the run pod's identity, whose IAM policy is an explicit
per-model allowlist. Matching on the `anthropic.` infix instead routed every
Anthropic-on-Bedrock model, so a model outside the policy came back 403 from
Bedrock with no fallback, taking the whole claude_code Bedrock matrix red.
An unlisted model now keeps its direct path and loses only caching.
Every OpenAI Responses body carries `error: null` at the top level, and the
completeness check tested the key's presence rather than its value, so it
rejected every single one. The cost was silent: nothing failed, the endpoint
simply never cached, which is exactly the outcome the endpoint was added for.
Found by driving the edge against the real providers rather than the synthetic
fixtures, which carried no error key at all. Reading the value instead of the
key is also more accurate for chat completions and messages, where a real error
body carries a populated error object.
Almost every Bedrock deployment in the suite declares
aws_region_name="os.environ/AWS_REGION". The mount resolver treated that
string as a region name, produced a mount nothing serves, and left the whole
Anthropic-on-Bedrock surface on its direct path, which is the one thing
mounting Bedrock was for.
The run pod does not share the proxy's environment, so the harness genuinely
cannot resolve that reference. A `us.` inference profile fans out across the US
regions and is reachable from any of them, so those route to the default mount
whatever the proxy resolved. A model that is not cross-region and declares its
region that way keeps its direct path rather than being sent to a region it may
not exist in.
Chat completions and messages were the only cacheable paths. The suite also
drives /v1/embeddings and /v1/responses through the same OpenAI mount, so both
now cache, each with its own completeness rule: a chat response's `choices`
check would reject a perfectly good embedding, and a Responses run that never
reached `response.completed` must stay out of the cache the same way a
truncated stream does.
Vertex and Gemini stay off the edge. litellm's `_check_custom_proxy` rewrites a
path-prefixed vertex api_base into `{api_base}:{endpoint}`, dropping project,
location and model, so a mount under a path prefix cannot work without a
root-mounted edge on its own port or a change in litellm. Shipping an
unvalidated URL guess would have been worse than saying so in PROVIDER_CACHE.md.
Also finishes the MountPolicy move: a mount now carries its signer and its
unkeyed headers together instead of a bare signer map.
The exact-request cache reused 5% of routed traffic (build 218: 19 hits,
350 misses) because every test salts its prompt with a fresh unique_marker(),
so the same test could never match itself across builds. It also routed only
openai and anthropic, while the week's flakiness was Bedrock.
Key is now HMAC(test id + method + URL + headers + body, with every
unique_marker() token replaced by a placeholder, + FIFO slot index). The slot
index is what keeps two marker-only-different calls in one test on two
recordings and therefore two provider response ids, so spend rows still
reconcile one per invocation. A call outside any test is not cacheable.
Bedrock gets a region-qualified mount and SigV4 re-signing, since the edge
rewrites the Host the proxy signed. Signature headers are excluded from the
key for signing mounts only, because x-amz-date would otherwise make every
Bedrock request a permanent miss; every other mount still keys on its
credentials whole. Only Anthropic-on-Bedrock chat deployments route:
embeddings, image generation, rerank and realtime keep their direct path, and
so do deployments carrying their own aws_role_name or static keys, whose whole
point is to prove the product's assume-role chain rather than the runner's.
The two eventstream actions bypass the cache and go live, still signed.
Counters are now attributed per mount as well as in total, so a build can
report a per-provider hit rate instead of one number.
The first cache-enabled litellm-e2e build (211) showed three gaps in the shared provider cache:
Every OpenAI response carries Cloudflare bot-management Set-Cookie headers, and the capture rejected any response with Set-Cookie, so no OpenAI response was ever recorded (179 of 372 misses rejected). The edge already withholds Set-Cookie from the proxy, so drop it before validating and storing instead of rejecting.
The provider prompt-caching tests need fresh provider state: a replayed priming response reports cache creation rather than a cache read, and the TPM test then trips the key limit. Mark both modules provider_live.
TestApiBaseSeam::test_live_mode_returns_none ran inside the cache-enabled runner and saw the shared edge; isolate it from E2E_PROVIDER_CACHE.
The recursion is bounded to depth 1: split parts each carry a single
service_tier, so the recursive call's partition has one key and the split
helper returns None.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): support percentile-based TTFT routing
* fix(router): apply routing_strategy_args updates to the live selector
Runtime routing_strategy_args updates (config reload, update_settings)
only rebuilt the strategy selector when routing_strategy itself changed,
so a newly added ttft_percentile sat unused until the proxy restarted.
Also drops a comment that only restated the code it sat above.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* refactor(router): drop unreachable empty-samples guard in percentile latency
_percentile_latency is only called behind use_ttft, which already requires
a non-empty ttft sample list, so the early return was dead code and the one
line Codecov flagged as uncovered on this patch.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* test(router): cover the no-selector path of a routing_strategy_args update
simple-shuffle has no selector attribute to re-link, so the early return
guards a setattr with a None attribute name. Dropping the guard makes the
new test fail with "attribute name must be string, not 'NoneType'".
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* fix(test): assert ValidationError on out-of-range ttft_percentile
pytest.raises(ValueError) tripped PT011 for being too broad. Pydantic
raises ValidationError for the gt/le constraint, so naming it satisfies
the rule and pins the assertion to the constraint under test.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* fix(router): drop Final from a per-deployment loop variable
basedpyright rejects "A Final variable cannot be assigned within a loop",
which pushed reportGeneralTypeIssues one over its budget. selected_latency
is rebound each iteration, so it matches its unannotated neighbours in the
same loop.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* test(router): exempt _apply_updated_routing_strategy_args from the name scan
The scan only reads test files with "router" in the filename, so it cannot
see the update_settings tests in router_strategy/test_lowest_latency.py.
Calling the private helper directly would test structure rather than
behaviour, so it joins the existing entries ignored for the same reason.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
`include:` or `exclude:` written as `${{ ... }}` read back as a string, and
the sweep treated that as the directive being absent, so it expanded every
combination GitHub would have dropped. A job whose `name:` holds no matrix
value then looked like it repeated one name across combinations that never
run. An absent directive still means no rows; anything that is not a list
of rows now joins the names left out of the comparison
A `name:` whose only leftover expressions read a `github.` property other
than `github.job` is filled in identically for every job of the run that
publishes it, so two jobs of one workflow carrying it land on the same
check run. Those names now compare against the other jobs of their own
file instead of sitting in the blind-spot bucket. They stay out of the
comparison across files, where two workflows can run on different events
An expression at `jobs.<id>.strategy` is legal on GitHub, but the model
required a mapping there, so a workflow using one made the whole file
unreadable and turned code-quality red. That job's names are now a blind
spot like any other name the sweep cannot work out offline.
A matrix whose `name:` holds no matrix value publishes that one name once
per combination, which leaves a required context just as ambiguous as two
jobs sharing a name, so it now reports instead of deduping.
A file that does not parse as one YAML document is reported the way the
module already promised, rather than escaping as a traceback.
Three ways the sweep could fail a workflow GitHub would publish fine.
`github.workflow` and `github.job` were counted as fixed for the whole run, so
two jobs naming themselves after the workflow they sit in were reported as a
collision. `runner` and `vars` were wrong the same way. Drop the exception
entirely: a name still holding an expression is one GitHub resolves per job, so
it is nothing to compare, which is what the rest of the module already does.
`format()` was resolved with Python's semantics, so an attribute lookup crashed
the script and a width specifier padded a name GitHub never pads. Fill `{0}`
holes and escaped braces, and treat anything richer as unresolved.
A matrix `include` or `exclude` row holding a value that is not a scalar lost
that key and became an empty row, which excludes every combination. Report the
row instead of quietly reshaping the matrix around it.
A job name holding an expression the sweep could not resolve was compared as
if it were the published name. Two jobs whose names differ per matrix value or
per caller input were reported as a collision, and a matrix that was itself an
expression collapsed onto the bare job id and did the same.
Model what a job publishes as known names beside the reasons the rest stay
unknown. Anything the sweep cannot work out contributes no name and is
reported as a note instead of guessed at. An expression over contexts that are
fixed for the whole run still compares, so two jobs sharing one of those are
still caught.
The collision sweep read a job's name as its `name:` or bare job id, which is
wrong for a matrix job that sets no name: GitHub publishes `build (3.12)`, one
per combination. That missed real duplicates and invented ones that don't exist.
It also crossed every matrix value while ignoring `exclude`, so it checked
combinations no job ever runs.
Four smaller gaps went with it. Boolean matrix values reached a name as `True`
rather than `true`. A `format()` whose arguments cannot fill its placeholders
raised straight out of the script instead of leaving the name unresolved. A job
calling a reusable workflow only ever chained one level, and a call outside the
repo fell back to the caller's own name, which GitHub never posts. A job whose
`name:` was not a string failed validation and silently dropped every job in
that file, so the sweep now renders any scalar and reports a file it cannot read
instead of skipping it.
The checker raised a custom exception and caught it two lines down in the
same module, which is the throw-then-catch the repo's coding guide rules
out. `main` now prints the same message and returns the exit code, so the
collision list stays a value the whole way out
The guard read each matrix key's values independently and crossed them, so
a job name reading two keys off one include row published pairs no job ever
runs, which could fail a valid workflow on a required check
It now builds the combinations GitHub builds: the listed keys crossed, each
include row folded into the combinations it overwrites nothing in, and a row
that fits nowhere standing on its own
The guard only substituted a bare `${{ matrix.key }}`, so any name built from a
larger expression stayed in the string as its own template. `_test-unit-base.yml`
names its job with a ternary over `format()`, which meant every shard published
an opaque name and 23 of the 33 required contexts, all of them `<shard> / Run
tests`, were invisible to the very check meant to protect them.
Job names are now evaluated per matrix combination over the pieces a name can
hold: string literals, `matrix.<key>`, `format()`, `==` and `!=`, and the
`<cond> && <a> || <b>` idiom. All 33 required contexts now resolve, and nothing
in the repo leaves an expression unresolved. An expression the evaluator does not
understand still falls back to its verbatim template, so two jobs sharing one
stays a collision.
The test also drops its `sys.path.insert`, which the test-quality budget counts
under TQ003; pytest already puts the file's own directory on the path.
A ruleset's required status check names a check run and GitHub matches it by
that name alone, so two jobs publishing the same name leave the gate unable to
say which job proved it. The new code-quality check reads every workflow,
expands matrix values and local reusable-workflow calls the way Actions does,
and fails when one name has more than one job behind it.
The selector picked up two suites that can never pass in this stack, so
editing either one turned the check permanently red: the presidio masking
suite calls pytest.fail without an analyzer and anonymizer that up.sh
never starts, and the pipecat audio suite skips itself at import time
unless the NLTK punkt_tab data is present, which nothing installs.
tests/e2e/coverage_registry/test_collector.py had the same problem for a
different reason. Its nested pytest.main autoloads pytest-retry from the
ci group the workflow installs and dies with "INTERNALERROR: no option
named 'filtered_exceptions'", so the collect-only pass now disables that
plugin. The plugin's entry point is pytest-retry, not retry, so the same
one-word fix lands on mutmut's pytest_add_cli_args, where "-p no:retry"
was disabling nothing.
Two smaller holes in the harness: a canary argument the shell never
expanded used to select nothing and let the gate pass green, and a secret
that cannot be represented in both bash and dotenv was rejected without
naming the key.
A harness-only change (proxy_client.py, conftest.py, pytest.ini, the gateway
config, .github/e2e-stack, or the workflow) selected nothing, so the stack was
never exercised by the change that touched it. select_tests.py keeps the
changed-file rule and adds the access_control suite whenever a harness file
changes. The run step now reports the pytest exit code before the evidence
check, prints pytest's summary line per pass so the rerun count is visible,
and assert_tests_ran.py names each failed or errored test as classname::name
A one-character value in the provider secret bundle was masked too, which
turned every 1 in the run log into ***, including the pass numbers and the
gateway addresses, so the only public diagnostics were unreadable
