Adds a scheduled GitHub Actions lane on top of the merged record/replay
transport. A Saturday cron records the `replayable` e2e tests against the
real providers and publishes the fixture bundle as a private
`e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull
that artifact by its pinned digest, verify the checksum before extracting,
and replay it with provider credentials set to bogus values, so a run that
ever reached a real provider fails instead of passing.
An egress sentinel pins the provider hostnames to a local sink for the whole
replay job and counts every connection that reaches them; the job asserts
that count is zero, so hermeticity is proven by measurement. A red Saturday
publishes no bundle, so the next weekday finds nothing fresh and fails loudly
rather than replaying a week-old recording, and the transport's seven-day
freshness gate hard-fails any bundle that has drifted too far. The lane also
runs on demand from the Actions tab with a record/replay `mode` input.
Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic
test now counts to twenty so its recorded response banks several content
deltas, matching the assertion that the stream arrives incrementally.
The record/replay harness stored a streamed provider response as one
buffered body, so a replayed stream arrived coalesced and the
/v1/messages streaming test could not be edge-wired. Keep each SSE
transfer chunk in the bundle in the order the provider sent it (a new
streamed response shape at BUNDLE_FORMAT_VERSION 4) so replay reproduces
the provider's split points, the recorded usage chunk keeps its
position, and a mid-stream upstream error replays as the same
mid-stream error rather than a clean body.
Resolves LIT-5742
Adversarial review of the new multipart keying turned up collisions where
two different provider requests computed the same replay key, which is the
dangerous failure for a replay harness: the second request silently gets the
first one's response instead of missing loudly.
- a part counts as an upload when it has a filename or declares its own
content type, and the declared content type joins the identity, so two
uploads of the same bytes under the same field no longer collapse
- the uploaded parts contribute a JSON list of [field, filename, type]
triples instead of a "field:filename" string, so a separator inside a
filename can no longer impersonate a field boundary
- repeated field names get a "name[n]" suffix with a literal "[" doubled
first, so a repeated field and a literally indexed one stay distinct
- a field value that is not UTF-8 is stored as a base64 sha256 digest;
base64 rather than hex because the canonicalizer rewrites 64-character
hex runs to <sha256> and folded every binary value onto one key
- a field whose name reads as a credential is stored as <secret>. This
stays key-preserving because the key is recomputed from the stored
request rather than saved beside it, so the live request carrying the
real value still matches its redacted fixture
- the uploaded byte length leaves the key. The canonicalizer absorbs
timestamp and id drift inside a file, and that drift moves the count,
so keeping it there made re-records miss
Also stops a lookalike parameter such as "xboundary=" from being read as
the multipart boundary, and gives the OpenAI batch backend model a single
constant instead of three copies of the literal.
BUNDLE_FORMAT_VERSION goes to 3 because all of this moves recorded keys.
A bundle recorded under the old rules now fails naming both versions
instead of missing on every call.
Chat completions, embeddings, the non-streaming /v1/messages tests, and the
OpenAI batch deployment now register through the provider edge, so
E2E_FIXTURE_MODE=record captures their provider calls and replay serves them
back offline. None of them was wired before, so record was a silent no-op over
these suites and replay quietly went live instead of using the bundle
Multipart uploads now key on their parsed parts: every ordinary form field,
plus the field name, filename, content digest, and length of each file part.
The boundary is envelope rather than content, so it stays out of the digest
instead of changing the key on every run. A body that does not parse as its
declared envelope still has the boundary normalized away before hashing, so
the fallback is at least stable, and it records a name that says why
Binary uploads hash byte for byte. Canonicalizing them first meant decoding
with errors="replace", which collapsed every invalid byte to one U+FFFD and
gave two different PDFs of the same length the same key
Bundles stay out of the repo: they hold verbatim provider response bodies and
expire seven days after recording. Publishing them for CI is LIT-5748, and
streaming fidelity is LIT-5742
Replaces the test-side fixture transport with an in-process provider-edge
HTTP server the proxy's deployments point their api_base at. Record forwards
provider calls verbatim and writes them to the bundle; replay answers them
from the bundle with zero provider calls while key auth, routing, cost
calculation, and spend-log writes still execute against the live proxy and
database. Drift comes back as HTTP 599 naming the computed and closest
recorded keys. Request headers are never stored and responses are kept
byte-identical between modes from the proxy's side of the socket.
Replay previously matched interactions by transport verb and path in
recorded order, so a request whose body drifted from the recording
silently replayed the stale response, and reordering two independent
calls broke replay even though both were recorded. Match keys are now
canonical: fixture_canonical.py strips volatile headers and credential
fields, replaces unique markers, generated ids, uuids, and timestamps
with fixed placeholders, sorts object keys, and hashes what remains, so
a key is stable across runs and machines while any real content drift
is a hard ReplayMiss naming the computed key, the closest recorded key
with its file, and a content diff, with no fallthrough to a live call.
Matching is order-independent across distinct keys and FIFO within one
key. Recording now also redacts credential body and form fields (not
just auth headers) so provider keys never land in bundles.
Resolves LIT-5741
E2E_FIXTURE_MODE selects the transport every e2e client is built on: live
(default, unchanged behavior), record (pass through to the live proxy while
writing every interaction to a fixture bundle), or replay (serve every
interaction from the bundle with no proxy and no provider spend). Both new
transports fulfil the existing Transport protocol, so no test changes shape.
A bundle is a directory with a manifest (record timestamp, harness version,
format version) and one JSON file per interaction, grouped per test in call
order. Replay against a manifest older than seven days hard-fails at
collection time naming the bundle age. Record always wipes and never reads
the previous bundle, refusing to wipe a directory that is not a bundle.
Auth header values are redacted on write; uploads store a sha256 digest.
unique_marker() becomes deterministic per test in record/replay modes so a
replay run regenerates exactly the requests the record run sent.
Content-based match keys, streaming chunk fidelity, and provider-scoping are
follow-ups (LIT-5741, LIT-5742, LIT-5745).
The Locust throughput SLO test is a different testing category from
functional e2e (variance-driven, historically flaky, currently
skip-annotated against LIT-5119) and erodes trust in the suite as a
release gate; it comes out of the default collection along with its
exclusive plumbing (locustfile, load-mock registration fixtures,
run_chat_load). Re-implementation as its own pipeline is tracked in
LIT-5163. The weekly session-anomaly test never ran in the suite (opt-in
via E2E_WEEKLY_ANOMALY, driven by its own workflow) and stays, as do the
markerless aggregation unit tests.
The vllm passthrough test read-times-out (60s) against the shared
vllm-cpu backend in every run on the per-SHA e2e stack; it is removed
until LIT-5164 establishes whether that is backend capacity or a
passthrough defect. Its registry cells return to the gap list, which is
the honest state
The collector read @pytest.mark.covers off every collected item, and
collection does not evaluate skips, so a test carrying both a skip and a
covers marker reported its cell as covered while asserting nothing. 17
files under tests/e2e do exactly that, which inflated the headline from
290/434 to 311/434.
A cell now counts as covered only when at least one test pytest would
actually run declares it; a cell claimed by both a live and a skipped
test stays covered. Skip state comes from pytest's own evaluator, so
skip, skipif (bool and string conditions), and module-level pytestmark
resolve exactly as they do in the e2e run. Cells left uncovered this way
are listed under the headline and exported as skipped_markers (JSON) and
litellm_e2e_coverage_skipped_markers (Prometheus) so the gap surfaces
instead of disappearing; the Loki line contract is unchanged. A marker
on a skipped test that points outside the registry is still an orphan,
so --strict keeps its reach.
Because skipif resolves against the environment the collector runs in,
the number now depends on that environment; run it where the e2e suite
runs. A pytest.skip() call inside a test body remains invisible to a
static pass, which the module docstring and README both state.
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end
suites live under tests/e2e. The suite stays in TypeScript and becomes a
self-contained npm package with its own package.json, lockfile and tsconfig
instead of leaning on the dashboard's toolchain; the dashboard drops its
@playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs.
CI paths follow the move: both CircleCI jobs (main e2e and the
SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now
install and run Playwright from tests/e2e/ui, with the node cache keyed on
both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec
edits keep skipping backend jobs. The suite's mock LLM fixture is excluded
from the e2e basedpyright zero-error gate in pyrightconfig.json since it
belongs to the TS suite, not the typed Python harness.
* test(e2e): add other suite covering master-key auth and health lifecycle
Covers the other.* holding-pen cells that were uncovered: master-key
valid_allows/invalid_denied on the admin /user/list gate, and the
lifecycle probes liveness.ping, readiness.public_probe,
readiness.reports_db_status, and readiness_details.authenticated_diagnostics.
New tests/e2e/other/ suite on the shared ProxyClient; the health probes
send no auth header to prove the public routes need no credential, and the
details route is asserted to reject an anonymous caller while exposing
version/db diagnostics to the master key.
* test(e2e): cover block_code_execution and openai_moderation guardrails
Extends the guardrails suite with two built-in guardrails registered per
request (default_on=False, opted in via the chat body's guardrails selector)
so neither intercepts unrelated traffic on the shared proxy.
block_code_execution.pre_call.blocks: a python code block plus a run-this
request is intercepted with the canned content-blocked message and the model
never runs, while the same code block asked about with don't-run-it reaches
the model. Verified live.
openai_moderations.pre_call.blocks: a flagged prompt is rejected 400 naming
the moderation policy while a benign prompt passes. The guardrail calls
OpenAI's moderation API; verifying it needs an OpenAI key with moderation
quota (this account currently 429s the moderation endpoint).
Adds a shared create_backend_model helper and a generic register() plus
per-request guardrails/max_tokens on the client so more built-ins can reuse
the same path.
* test(e2e): cover presidio PII masking (pre_call + post_call)
Registers a presidio guardrail per request (default_on=False) with the
analyzer/anonymizer bases supplied in the registration params, so the test
controls its own dependency and needs no proxy restart.
presidio.pre_call.masks: a repeat-verbatim request comes back with the
<EMAIL_ADDRESS> placeholder and never the raw email, proving the prompt was
anonymized before the model saw it.
presidio.post_call.masks: with apply_to_output the model's own emitted email
is masked on the way out, so the caller never receives the raw value.
Both verified live against real presidio analyzer + anonymizer containers.
logging_only is intentionally not covered: /spend/logs exposes no prompt
messages to read back the masked log, and a logging_only run also masked the
response, contradicting its contract; noted in the module docstring for a
follow-up.
* test(e2e): cover presidio logging_only masking via OTEL read-back
Adds the third presidio cell, guardrail.presidio.logging_only.masks. The
logging_only contract (mask what is logged, do not block) is verified by
reading the request's gen-AI span back from the real OTEL destination: the
span's gen_ai.input.messages attribute carries the <EMAIL_ADDRESS> placeholder,
never the raw email, and the call itself is not blocked.
Reads the trace via the shared OtelReader, promoted from logging/ to the suite
root so both suites use it. The masked prompt is polled to a deadline because
logging_only masks the payload asynchronously and the span can briefly export
before the mask lands. Drops the throwaway chat_send in favor of the existing
transport.send for the call-id capture.
* fix(e2e): tolerate cross-pod guardrail sync delay in team-opt-out test
Stage runs multiple gateway pods behind the shared key. POST /guardrails
registers a new default-on guardrail in-process immediately only on the
pod that served the create call; every other pod picks it up on its next
periodic DB sync (proxy_server.py, every 30s), so the very next chat call
can race a pod that has not synced yet. Poll to a 40s deadline instead of
asserting on the first response, matching the existing pattern in
test_budget_reset_advances_e2e.py.
* test(e2e): cover a guardrail on the MCP tool-call path (content_filter pre_mcp_call)
Adds guardrail.litellm_content_filter.pre_mcp_call.blocks: against the real
Datadog MCP server, a content_filter guardrail configured mode=pre_mcp_call
blocks a banned keyword in an MCP tool call's arguments with HTTP 400 attributed
to the pre_mcp_call hook, and lets a clean argument reach the upstream server.
The guardrail attaches with default_on because per-key/request guardrail
selection is dropped from the synthetic MCP request the hook sees; the banned
keyword is unique per run so default_on only intercepts this test's own call.
mode must be pre_mcp_call - a pre_call config silently no-ops on tools/call
because the event type is rewritten for call_mcp_tool.
Drives the tool directly via /mcp-rest/tools/call for a deterministic check of
the same pre_mcp_call enforcement the OpenAI-SDK chat path hits when a model
invokes an MCP tool.
* fix(e2e): mid-conversation messages test uses client.proxy not client.gateway
EndpointsClient exposes .proxy after the Gateway->ProxyClient rename; the
mid-conversation system test still referenced .gateway, which fails the e2e
basedpyright gate. Aligns it with the rest of the harness.
* test(e2e): address review on the guardrail coverage
MCP tool-call guardrail: poll the banned call until the guardrail is enforced
instead of asserting on the first call, so the control-plane -> data-plane
guardrail sync cannot race the check into a false pass-through; add a repeat
banned call after enforcement to guard against a partial-propagation state.
OpenAI moderation: distinguish a moderation-endpoint 429 (rate limit / no
moderation quota) from a guardrail failure, so an account-capability gap reads
as such rather than as "did not block". Runs green with a moderation-capable key.
* test(e2e): close partial-propagation false-pass in MCP guardrail block test
The single post-block repeat call could be load-balanced back to the same
already-synced data-plane pod, so the test could pass while another pod still
lacked the guardrail and let the banned MCP call reach Datadog. Anchor a wait to
the guardrail create time (every pod is guaranteed to have DB-synced only after a
full ~30s sync interval), then require the banned call to stay blocked across
several attempts; a pass-through after that window is a real leak, not a race.
* test(e2e): drop xfail-style rate-limit branch from openai_moderation test
OpenAI's /v1/moderations is free and returns 200 with the env key (verified
directly), so the RateLimitedError branch mislabeled the failure: a 429 there is
insufficient_quota (no account billing), not throttling. The branch also only
printed a softer message before failing anyway, an xfail-in-disguise the e2e rules
forbid. A 429 now falls through and fails loudly with the full result.
Add a live spend-tracking e2e that drives a streaming anthropic-format
/v1/messages request through litellm's anthropic-messages -> OpenAI Responses
adapter and asserts the consumed stream writes exactly one SpendLogs row with
nonzero cost and token counts, attributed to the calling key under
custom_llm_provider openai and the /v1/messages call_type.
The deployment is a Responses-only OpenAI model (gpt-5.3-codex), so a served,
costed row proves the Responses path was taken; the chat-completions bridge
would have failed at OpenAI on an endpoint the model does not expose. Adds a
streaming /v1/messages method to the shared Gateway and the suite client, the
model to the inline compose config and driver-model registration, a coverage
registry row (quota_management.spend_tracking.messages_bridge.logs_cost), and
the matching variant vocab entry. The _summarize spend-row detail also gains
call_type and custom_llm_provider so a failed assertion prints the fields it
asserts on.
Resolves LIT-4546
* test(e2e): harden stage flakes for batches, UI, and MCP
Unique batch model names avoid load-balancing onto stale azure-batch
deployments that still pointed at the retired gpt-4.1-mini-batch, which
only the managed/unified path was hitting. Retry batch retrieve on 500
and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite
when the compose-only mcp-upstream is unreachable on stage k8s
* test(e2e): cover Datadog remote MCP via search_datadog_logs
Register the regional Datadog MCP endpoint with DD-API-KEY /
DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is
not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*,
assert the proxy shipped it, list tools, call search_datadog_logs for the
marker, and delete the server on teardown. Math-upstream key-access tests
only skip when that compose service is unreachable
* test(e2e): drop compose math MCP upstream; use Datadog only
Key-access denial and happy-path MCP e2e both register the real regional
Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers.
Remove the mcp-upstream compose service and FastMCP add/multiply fixture
* docs(e2e): require real Datadog MCP for all mcp suite tests
Document that tests/e2e/mcp must register via datadog_mcp helpers against
mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams
* chore: restore mcp_e2e_upstream_server.py
Keep the FastMCP fixture file; e2e no longer wires it in compose, but the
module itself is not part of the Datadog-only cleanup
* fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load
pytest on the host never inherited compose env_file keys, so DD_API_KEY
stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the
dynamically loaded datadog_reader module in sys.modules so dataclasses
do not crash under Python 3.12
* test(e2e/batches): harden azure/vertex unified lifecycle flakes
Put the provider deployment name in every JSONL body so Azure does not
depend on a perfect model rewrite. Retry create/retrieve/cancel on
transient statuses with backoff. Drop cancel assertions for azure and
vertex (registry only has a shared basic cell; create+retrieve prove
routing, cancel stays best-effort cleanup)
* test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED
Post-login client redirects abort the first /ui/api-keys/ goto on stage.
Wait off /ui/login after cookie set, then accept the page once Create New
Key is visible even if goto raised ERR_ABORTED
* test(e2e): drop flaky key models dropdown Playwright suite
API management e2e already covers key generate/update persistence. The
UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and
no unique product signal. Remove the suite and unused browser fixtures
* refactor(e2e): fold claude_code HTTP probes onto shared Gateway methods
Migrate tests/e2e/claude_code/http_probe.py off its own httpx client onto the
shared transport, and promote count_tokens and native anthropic messages to
first-class Gateway methods (Gateway.count_tokens / Gateway.messages) with typed
request/response models in the shared models.py so other suites reuse them.
The probes now take an injected Gateway and issue their request through the
shared count_tokens/messages methods, reusing the split control/data-plane
routing, timeout, and typed Result handling the rest of tests/e2e uses. The wire
shape is preserved: the pydantic bodies serialize byte-for-byte to what the old
httpx probes sent, and the anthropic-version header is carried by a small
AnthropicHeaders model. httpx is gone from the module.
* test(e2e): drop unit-level probe harness test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CodSpeed benchmarks the SDK with no IO, so it can't catch regressions that
only appear under real concurrent load through the full proxy stack (auth,
routing, logging, spend, Postgres, Redis). This adds a Locust load test under
tests/e2e/load that drives concurrent POST /chat/completions traffic against a
mock deployment (litellm_params.mock_response), so the measured throughput
reflects proxy overhead rather than a provider's latency, and asserts an
aggregate RPS SLO with a failure-ratio guard. The test is marked load and the
parent conftest sorts load-marked items last so it never perturbs
latency-sensitive suites. Covers reliability.perf.throughput.under_slo.
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
Add an e2e suite at tests/e2e/mcp/ that proves MCP authorization over the
api_key auth family. An admin registers an upstream MCP server through the
management API (POST /v1/mcp/server, persisted in the DB and picked up without
a restart) and queues its deletion. Two keys are created against that one
server: one granted access through object_permission.mcp_servers and one with
no MCP grant. The permitted key is a live control proving the upstream is
reachable and the tool is callable, so a denial on the ungranted key is an
authorization decision rather than a dead server. The denied key then sees
none of the server's tools on tools/list and is refused a tools/call with a
403 access_denied.
A deterministic self-hosted FastMCP upstream (add/multiply over
streamable-http) is added to the e2e compose stack so the suite runs offline
with a known tool set. KeyGenerateBody gains an optional typed
object_permission so the shared gateway can create a key with an MCP grant.
The e2e docs claimed `e2e`-marked tests skip when no proxy answers the
liveness probe, but the harness has always hard-failed: conftest.py's
pytest_runtest_setup calls pytest.fail, its module docstring states
"hard failures only ... never skip", and logging/conftest.py forbids
skipping outright. Align the docs to the code so the single most
important contract reads the same everywhere; a dead proxy turns a run
red instead of being silently skipped and mistaken for a pass. The
per-suite conftest docstrings that described the shared hook as a
"proxy liveness skip" are corrected to "liveness gate" for the same
reason.
Also scope the no-unit-tests hard rule to what it means: never
substitute a unit test for e2e feature coverage, while explicitly
allowing tests that cover the harness itself (e.g.
coverage_registry/test_collector.py), which carry no e2e marker and
run whether or not a proxy is up.
No product code and no harness logic changed.
Resolves LIT-4554
* test(e2e): assert bare-key budget refusal is 429 and /key/info spend reaches the cap
* test(e2e): keep the bare-key budget assertion to the 429 refusal shape
* test(e2e): assert a team's max_budget blocks every key on the team
* test(e2e): focus the team budget case on the 429 blocking behavior
* test(e2e): harness fixes for long_context, complexity router, UI, and unit coverage
Point long_context_1m at 1M-capable models, harden complexity-smart-router
registration and spend-log assertions, fix key models dropdown selectors, and
add gateway/lifecycle/transport and claude_code unit tests
* test(e2e): harden remaining stage failures in harness
Register complexity-smart-router via create_model + callable probe, fix
create-key UI navigation race, retry management writes and budget ALB
502s, mark Vertex count_tokens N/A when unsupported, and tighten
tool_search model lists for Azure/Bedrock capability gaps
* test(e2e): drop claude_code and harness unit tests from this PR
Keep management, router, budget, and shared conftest harness fixes only
* test(e2e): restore E2E_RESULT pytest_runtest_makereport hook
Accidentally dropped in an earlier harness commit; Grafana status history
depends on these structured log lines
* test(e2e): drop management control-plane write retries
Transient 500 retries do not fix the underlying control plane failures
* test(e2e): skip stage-red claude_code cells; fix multi-window budget latency
Mark the twelve failing claude_code matrix cells skip until product/config
lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of
Claude so the reset wait stays under ALB target idle timeout rather than
masking awselb 502s
* test(e2e): require exactly one LLM-tier spend row for complexity router
Keep alias membership for compose vs stage model names, but assert
len(served) == 1 so a leaked classifier sub-call cannot pass. Also pin
LIT-4521 skip and align LIT-4522/23/24 skip reasons
* test(e2e): harden router callable probe and multi-window budget exhaustion
_router_is_callable treated any non-success chat whose body lacked "Invalid
model name" as callable, so an unpropagated probe key (401), a generic 502, or
a connection reset let the session proceed and hit real "Invalid model name"
failures inside the tests. Require a Success outcome instead; the reload-race
400 and every infra/auth error now correctly read as not-callable.
The multi-window budget test capped the tight window at 3e-6, which gpt-5.5
exhausts on the first call but a cheaper CHEAP_OPENAI_MODEL might not within the
20-call loop, turning a reset test into a spurious "window never enforced"
failure. Drop the tight cap to 1e-9 so the first billed call exhausts it
regardless of model price; the roomy 1m window stays at 1.0 and never blocks.
* test(e2e): use a tradeoff-decision prompt for the complexity router classifier
"Is P equal to NP?" reads to the LLM classifier as a short yes/no question, so
gpt-5.5 classified it SIMPLE and the request routed to the openai backend, which
made the test fail even though the classifier was running. The tier definitions
key on what the request demands, not how hard the answer is, and a short direct
question maps to SIMPLE regardless of subject.
Swap in "Should I pay off my mortgage early or invest the extra money instead?".
It carries none of the heuristic scorer's reasoning/technical/code keywords and
stays short, so heuristic scoring still lands SIMPLE (openai), but the LLM reads
it as a decision that has to weigh tradeoffs and lands it above SIMPLE, which the
config routes to anthropic. Any non-SIMPLE tier serves anthropic, so the classifier
only has to avoid SIMPLE for the test to distinguish a real classifier run from the
heuristic fallback.
* feat(e2e): emit structured E2E_RESULT lines for package status history
Pytest progress logs only expose file basenames and collapse multi-test files
into one status-history row. Emit one logfmt E2E_RESULT per finished node with
package, file, outcome, duration_ms, node_id, and covers so Grafana can roll up
by package (stable cardinality) and drill down by node_id in Explore
* test(e2e): drop unit tests from the live e2e tree
tests/e2e is for live proxy suites only. Remove harness, coverage-registry,
and claude_code unit trees so the e2e run does not collect them
* fix(e2e): type E2E_RESULT hook for basedpyright zero-error gate
Protocol-typed covers extraction and pluggy Result typing on the
makereport hook so tests/e2e stays under the e2e basedpyright ceiling
* fix(e2e): drop unused xfailed/xpassed from E2E_RESULT Outcome
We never emit those states; xfail collapses to skipped/passed via pytest
report flags. Keep the literal honest so the dashboard only sees real outcomes
* fix(e2e): strip tests/e2e prefix when deriving E2E_RESULT package
Repo-root pytest nodeids are tests/e2e/<suite>/...; without stripping,
every line would package=tests and status history would be useless
* fix(e2e): use pytest wrapper=True instead of deprecated hookwrapper
pytest 8.1+ deprecates hookwrapper; yield returns the TestReport directly
so we return it for the outer chain and drop pluggy.Result
* fix(e2e): import e2e_result_reporter at module load
Surface a missing module as a collection-time ImportError instead of a
per-test hook failure mid-run
* test(e2e): restore coverage_registry/test_collector.py
Needed to validate registry coverage math and the checked-in cell
denominator; not a live proxy suite
* refactor(e2e/claude_code): align proxy env names with the rest of tests/e2e
Every claude_code compat cell used to read its own `LITELLM_PROXY_BASE_URL` and `LITELLM_PROXY_API_KEY` and duplicate the same 12-line "missing env, hard fail" block. The rest of `tests/e2e/` reads `LITELLM_PROXY_URL` and `LITELLM_MASTER_KEY` from `e2e_config.py`, so anyone standing up a live proxy for one suite had to export a second spelling for claude_code, and every cell repeated the same boilerplate.
Centralize the resolution in `claude_code/_env.py`. `resolve_proxy()` prefers the suite-wide `LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY` names and falls back to the legacy pair so existing CI wiring on stage keeps working during the roll-out. `require_proxy(compat_result)` is the one-liner cells call to bind `(base_url, api_key)` or hard-fail with a message that names both spellings.
55 cell files, `_basic_messaging.py`, and the driver's own unit-test fixture now go through the helper. `run_compat.sh` accepts either spelling and normalizes to the primary names before invoking pytest. `cron_vm/run_daily.sh` exports the primary names when launching pytest.
`_pr_gate_unit_tests/test_env_resolution.py` pins the resolution rules so a future edit cannot silently reintroduce the drift: primary names win on tie, legacy names still resolve when primary is unset, mixed URL-primary key-legacy still resolves, empty-string exports are treated as unset, `require_proxy` names both spellings in its error message.
Net diff: 71 files, +370/-1240.
* fix(e2e): anchor claude_code Bash pin at parents[1] so container run collects
`test_bash_tool_restrictions.py` derived `REPO_ROOT = Path(__file__).resolve().parents[4]` and then joined `tests/e2e/claude_code/<feature>`. That works locally, but the stage container mounts tests/e2e/ at /app/e2e/, so parents[4] resolves to filesystem root and the `_bash_cells()` assertion looks for `/tests/e2e/claude_code/tool_use` — a path that doesn't exist. Collection interrupts before any test runs, so the entire e2e suite appears broken.
Fix: `CLAUDE_CODE_DIR = Path(__file__).resolve().parents[1]` resolves to the sibling `claude_code/` dir in either layout, and the `relative_to(REPO_ROOT)` calls become `relative_to(CLAUDE_CODE_DIR)` so test IDs and error messages read the same.
Adds `test_claude_code_dir_anchor_is_layout_independent` as a regression pin: it checks the anchor lands on a directory named `claude_code` that contains this test file, which would fail under the old parents[4] anchor when run from /app/e2e/.
* feat(e2e/claude_code): register compat deployments via /model/new from a session fixture
Every compat cell hardcodes a virtual model name like `claude-sonnet-4-6` or `claude-sonnet-4-6-bedrock-invoke` and hits the proxy expecting it to be routable. On stage those live in the deployed model_list; locally the `docker-config.yaml` under tests/e2e/ only declares one of them, so anything past haiku 400s with `Invalid model name`.
`claude_code/test_config.yaml` is the ground-truth compat matrix config the deployment already uses. `_compat_models.py` loads it, normalizes the yaml keys pydantic would silently drop (vertex_ai_* → vertex_*), and selects the subset whose provider credentials are present in the environment. An autouse session fixture in `conftest.py` POSTs each selected deployment to `/model/new`, blocks until it is servable on the data plane, and tears them all down on session exit. Skips silently when the proxy env is unset so pure-unit runs stay hermetic.
`test_compat_models.py` pins the invariants that keep this safe. Every cell-referenced name must have a yaml entry (drift check catches a cell probing a name the fixture never registered); the yaml has no unused declarations; the fixture registers exactly 15 deployments (3 tiers × 5 provider surfaces); vertex_ai_* yaml keys populate the pydantic body's vertex_* fields (they got silently dropped historically); Azure needs both AZURE_FOUNDRY_* env vars; Bedrock lifts creds from the ambient AWS chain; Vertex needs both the yaml refs AND ambient GCP credentials.
* refactor(e2e/claude_code): inject env + runner instead of monkeypatching
`require_proxy` and `_basic_messaging.run_basic_messaging_cell` now take the env mapping (and the CLI runner) as constructor-style arguments with `os.environ` and `run_claude_models_parallel` as defaults. Tests exercise the branching by passing dicts and callables directly, so `monkeypatch.setenv` and `monkeypatch.setattr(_basic_messaging, "run_claude_models_parallel", ...)` are gone from every unit test in this refactor's blast radius.
`test_env_resolution.py` drops the `monkeypatch.setenv`/`delenv` fixtures and passes `env={...}` dicts to `require_proxy`. Added a new pinned check that a successful resolution leaves `compat_result` untouched, and split the "unset env" test into three explicit shapes (empty, primary-only, legacy-only) so a regression that swaps the precedence rule can no longer hide behind a single monkeypatched fixture.
`test_basic_messaging.py` (driver) replaces the `_install_fake_runner(monkeypatch, ...)` helper with `_make_fake_runner(...)` that returns a `(callable, captured_dict)` pair the test passes in via the helper's new `runner=` kwarg. Also drops the autouse `_proxy_env` fixture in favor of a module-level `_PROXY_ENV` dict each test wires through the helper's new `env=` kwarg. Added a regression pin that a missing-env call hard-fails without ever invoking the runner (so the guard order stays correct).
`test_run_daily_pytest_scrubs_env.py` updates its pin to assert the new suite-wide env spellings (`LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY`) instead of the legacy `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` that `run_daily.sh` used to export.
* handwrote rules
* test(claude_code): move the Claude Code compatibility matrix under tests/e2e
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(claude_code): drop the CircleCI compat PR gate; the matrix runs in the scheduled e2e suite instead
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: restore the upload-coverage job dropped by mistake with the compat gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(e2e/claude_code): print rate-limit summary on failed compat runs and fix stale run_daily.sh header comments
* test(claude_code): assert fine-grained tool streaming via input_json_delta instead of an event-count floor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): scope key models dropdown options to the key's team
A teamless key no longer offers the all-team-models option in the create and
edit forms; the backend expands that sentinel to the full proxy model list when
no team is attached, which is rarely what the user intended. A team key no
longer surfaces the all-proxy-models sentinel that leaks in verbatim when the
team's own model list carries it; the dropdown keeps All Team Models plus the
team's individual models.
Adds browser coverage to the management e2e suite: playwright (an optional
dependency behind importorskip) drives the proxy-served dashboard at /ui,
asserts the dropdown options a real user sees for teamless and team keys on
both create and edit, and walks the create modal end to end, reading the
persisted key back through /key/info.
* fix(ui): offer all-proxy-models on teamless keys in the models dropdown
A teamless key has no team allowlist to inherit, so the dropdown now offers All
Proxy Models in place of All Team Models on both the create and edit forms, with
the same exclusive-selection handling. Component and browser e2e tests updated to
pin the swapped option pair; the teamless create case now also walks the modal end
to end and reads the persisted key back through /key/info.
* test(ui): update no-team key creation spec to pick All Proxy Models
The create modal no longer offers All Team Models without a team; the teamless
path now offers All Proxy Models, which is what this spec exercises
* fix(ui): gate All Team Models on the team object being loaded
When a key has a team_id but the teams prop does not yet include the matching team, availableModels stays empty and the models dropdown rendered All Team Models on its own with nothing to compare against. Gate the option on the team object being present so it only appears once team models are known, and add a regression test for the loading state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): filter all-proxy-models from teamless model fetch in key edit form
The teamless fetch path stored modelAvailableCall results without excludeProxyWideSentinel, so an all-proxy-models entry in the response rendered a second option colliding with the hardcoded All Proxy Models sentinel. Apply the same filter used on the team path and add a regression test asserting the sentinel option is not duplicated
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): add management suite covering key/team/user/org lifecycle and route permissions
* test(e2e): decouple the enforcement-flip assertion from upstream health
Polling for a 200 on the newly-allowed model required it to be a routable,
healthy upstream, which is not the contract under test; poll until the
key_model_access_denied 403 lifts instead, excluding 401 so a revoked key
cannot read as success. Also document that the delete test's deferred teardown
firing on an already-deleted key is deliberate: cleanup must survive the test
failing before the in-body delete, and the repeat delete is a warn-free no-op
(the proxy answers 404 No keys found)
* test(e2e): inline the management suite's model and tpm literals
* test(e2e): drop the models_mgmt suite line from the folder list
* test(e2e): write the tpm limit as a plain integer literal
* test(e2e): migrate access-control and inference-endpoint regression tests
Move the access-control and non-chat inference-endpoint cases from litellm-regression-tests onto the shared e2e harness so a regression in either fails here first
access_control/ asserts the gateway's authorization and error-shape contract: a key limited to one model is denied 403 (key_model_access_denied) when it calls another, a key scoped to allowed_routes=["llm_api_routes"] is forbidden 403 from a management route, and an unknown model is rejected 400 before any provider is called. The source asserted 401 for the disallowed-model case against an older proxy; the live contract is now a 403, so the guard tracks current behavior
llm_translation/ gains one file per non-chat inference endpoint (/v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, /v1/images/generations). Each test registers the deployment it needs through /model/new, drives real provider traffic, asserts the parsed body carries real content instead of just a 200, then deletes the model on teardown, so nothing is hardcoded into the gateway config
* Update endpoints_client.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
tests/e2e/CLAUDE.md captures the harness code-style rules (suite-as-a-class, shared transport, typed pydantic models, Result/unwrap, markers, typing) and the coverage-registry naming grammar; CONTRIBUTING.md gets the Contributors Guide intro and a Setup section
