* test(e2e): add failing reproducers for two open gateway bugs
Both tests assert the behavior a customer expects and both are red today. They
are reproducers, not regressions: the product is wrong, not the tests.
Native passthrough returns almost none of the operational headers the managed
route does. A /gemini/ generateContent call comes back with three x-litellm-*
headers and no x-ratelimit-* at all, against sixteen and four on
/v1beta/models/{m}:generateContent for the same prompt, and critically it omits
x-litellm-response-cost. Customers front provider-native traffic through this
route and read those headers to reconcile spend and pace themselves, so native
traffic is currently invisible to the tooling that covers every other route.
/budget/update rejects any model_max_budget with a 500. The reported symptom was
model ids containing dots, and that reproduces (prisma raises "Unexpected
`-5.2[FloatValue]` Expected `:`" because the key is interpolated into a GraphQL
query unquoted, so glm-5.2 lexes as an identifier followed by a float), but the
plain name gpt4o fails too, on a separate "model_max_budget should be of any of
the following types: Json" type mismatch at budget_management_endpoints.py:173.
Omitting the field returns 200. The test drives both names so the failure says
whether per-model budgets are broken outright or only for punctuated ids; today
it stops on the plain name, which is the wider bug.
* test(e2e): add reproducer for unenforced end-user per-model rate limits
model_max_budget accepts an rpm_limit alongside the spend cap, and /budget/new
stores it: the create response echoes {"gemini-2.5-flash": {"rpm_limit": 1,
"max_budget": 100.0, "budget_duration": "1d"}}. Attach that budget to an end
user, drive three calls as that user, and all three return 200. The limit is
accepted, persisted, and then ignored.
The same shape already works when the budget hangs off a key, which is what
makes this quietly dangerous: the API gives every indication the cap is in
force. A customer using it to hold one end user to a slow rate on a shared key
gets no throttling at all.
Harness additions this needs: ModelBudgetEntry carries the rpm_limit/tpm_limit
the route already accepts, BudgetNewBody and create_budget carry
model_max_budget, and create_customer can attach an existing budget_id rather
than only an inline max_budget.
Red today, for the reason in the assertion message.
* test(e2e): tighten model_max_budget reproducers and drop in-loop closure
Trim the reproducer docstrings to the contract they assert, keeping the
failure messages that document each red-by-design bug. Replace the nested
per-model closure in the /budget/update test with a module-level predicate
and a per-model helper so nothing closes over a loop variable, and fix the
import order the merge left unsorted.
* test(e2e): skip the three reproducers while their gateway bugs stay open
The passthrough header contract, /budget/update model_max_budget, and
end-user per-model rpm enforcement reproducers all still fail against
staging by design. Skip each with the product gap named so the combined
suite can gate merges on green while the collector keeps reporting the
cells as uncovered.
* test(e2e): validate model budget response contracts
* refactor(e2e): unify model budget schema
* refactor(e2e): reuse shared model budget type
A single read of key_info.spend races the batched spend writer: deltas
earned before a reset flush to the DB up to ~60s later
(proxy_batch_write_at) and land on the row after the reset zeroed it.
The stage runs on Jul 30 and Aug 2 failed
test_key_budget_reset_at_advances_after_window exactly this way, with
spend back at the driven total while budget_reset_at had advanced and
calls flowed again.
Replace the single reads in rung 3 (spend zeroed after reset) and rung 4
(roomy window keeps spend) with _poll_key_spend, which re-reads to a 90s
deadline covering one full flush-plus-reset cycle. A reset that never
zeroes the row keeps spend pinned and still times out, so the regression
guard keeps its teeth.
#35271 restored the hierarchy where a team-scoped key is governed by the
team and team-member budgets only; the owner's personal max_budget applies
to their personal keys. Three places in the e2e suite still encoded the
old direction and would fail against a proxy built from staging.
test_user_budget_enforced_across_all_their_keys asserted that the owner's
team-member key is refused once their personal budget is exhausted. It now
asserts only the personal keys are refused, and keeps the team key as the
control that must keep serving, which pins the restored direction instead
of leaving it unasserted. Renamed to match what it now covers.
test_team_member_key_user_budget_resets_after_window drove a team key to a
block off the owner's personal budget, so nothing can block it any more and
_drive_to_block could never succeed. Its premise is gone rather than moved,
so it is removed; the sibling personal-key test still covers
quota_management.budget.internal_user.resets_after_window.
The registry rationale for that row dropped its "and team-member keys"
clause for the same reason.
* test(e2e): cover key max_budget blocks on personal, team, and team-member keys
* refactor(e2e): convert budget enforcement cases to the resources-fixture pattern
The E2ECase class pattern existed only in this file; every other suite uses
plain pytest tests with the resources fixture. Rewrites the nine cases as two
spec classes and removes the now-dead E2ECase protocol and run_case driver
from lifecycle.py
* test(e2e): assert the long budget window keeps blocking after the short window resets
The multi-window budget tests proved the tight window blocks and self-heals
but never asserted the other direction: a long (1d) window whose cap the
accumulated spend already crossed must keep refusing calls even inside a
fresh short window. Adds one test per file (key and team) that drives spend
to a block, waits for the short window's reset_at to strictly advance (the
reset job zeroes that window's counter in the same pass), then polls until
the refusal is attributed to the 1d window ("over 1d budget"), failing
immediately if any call succeeds or a non-budget error leaks. Harness gains
per-window reset_at readback: BudgetWindowState in models.py and
key_window_reset_at / team_window_reset_at on BudgetClient.
* refactor(e2e): hoist shared budget-suite helpers into budget_client
drive_to_block and as_datetime existed as five and four per-file copies in
the budgets suite; both move to budget_client with each file keeping a thin
delegating wrapper so call sites and per-file pacing stay unchanged. The
three /team/info readers in budget_client now share a private _team_info.
Also guard the long-window reset_at snapshots with explicit non-None asserts
so the midnight-roll diagnostic cannot misreport when the window is missing
from the info response (greptile P2s).
* docs(e2e): tighten the multi-window module docstrings
* refactor(e2e): type window reset_at as datetime and expose plain window readers
BudgetWindowState.reset_at becomes a pydantic-parsed datetime, so the
multi-window tests compare real datetimes instead of hand-parsing strings.
The duration-keyed accessors are replaced by two plain readers,
key_budget_windows and team_budget_windows, with the pure window_reset_at
lookup exported; the client no longer encodes one test's access pattern.
* test(e2e): name the tiny short-window cap and comment the wait loops
* test(e2e): surface the 429 budget-block assert in the multi-window tests
drive_to_block now returns the blocking response so a test body can assert
on its shape; the two long-window tests assert status 429 explicitly, which
also pins the multi-window enforcement path's HTTP mapping (the enforcement
suite only covers the single-budget path). Other callers ignore the return
and are unchanged.
* refactor(e2e): scope this PR to the multi-window test, drop the cross-suite hoist
The helper hoist rewrote four unrelated budget test files (reset, reset_advances,
team_member_reset, user_across_keys) to pull drive_to_block and as_datetime out
of budget_client, which is refactor churn beyond this PR's multi-window scope.
This restores those four to their pre-PR state and gives the two multi-window
tests their own inline drive-to-block loop again, so the PR touches only the
multi-window feature: its two tests plus the budget_client window readers and the
reset_at datetime typing they actually use. The suite-wide helper dedup can land
on its own PR
* docs(e2e): number the long-window key test steps inline
* docs(e2e): number the long-window team test steps inline
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
* test(e2e): budget reset diagonal for team, org, user, and #32005 team-member keys
Adds E2E-7/8/10/11 from the budget-level x key-kind coverage matrix: each budget
level serves traffic again after its budget_duration window elapses, walking the
same ladder as the enforcement diagonal. New registry rows and tests cover the
team, organization, and internal-user reset rungs, plus the #32005 interplay
where a team-member key frozen by its owner's user budget comes back when the
user's window renews; the bare-key and per-team-member rungs already had coverage
Each case isolates the cap to one entity, drives spend to a budget_exceeded
block, then polls past the window until a call succeeds, holding every refusal
as a budget block so a reset that no-ops (stays blocked forever) or crashes
(leaks a 5xx) fails the test. budget_duration becomes an optional param on the
budget_client create_team / create_user / create_org helpers
* test(e2e): fold the reset diagonal into test_budget_reset_e2e.py and address greptile nits
Move the team / org / user / #32005 reset cases out of the standalone
test_budget_reset_diagonal_e2e.py and into test_budget_reset_e2e.py, absorbing
the pre-existing bare-key reset into the same TestBudgetResetDiagonal spec class
so the whole reset ladder reads as one file (mirroring how the enforcement
diagonal lives in test_budget_enforcement_e2e.py) and the drive/poll helpers are
defined once instead of duplicated across reset files.
Greptile nits: bound the drive phase to under one window (12 attempts x 2s < 30s)
so a block is observed before the reset job can fire, and replace the bare assert
in the poll loop with a pytest.fail that prints the HTTP status, so a provider 429
or a crashed reset path is distinguishable from a budget block at a glance.
* test(e2e): trim reset diagonal docstrings back to the file's original style
* test(e2e): inline single-use drive-loop bounds
* test(e2e): cut the reset module docstring to one line
* test(e2e): make the org reset test wait for a scheduled window (bugbot)
/organization/new stores budget_duration without scheduling budget_reset_at, so
the reset job's NULL catch-up branch zeroes org spend on its first 5-10s tick;
the org reset test could pass off that catch-up instead of a real window roll
(tracked as LIT-4570). The test now reads the org's budget_id and polls
/budget/info until budget_reset_at is scheduled before driving spend, so the
recovery it observes can only come from a genuine window expiry. Verified live:
the org case now runs ~33s (a full window) instead of beating the rescheduler
OpenAI GPT-5.6 Claude Code cells burn minutes on CLI timeouts under the
full stage suite; gate them behind COMPAT_OPENAI_GPT_CELLS=1 like Mantle.
Multi-window budget e2e used max_tokens=1 which gpt-5.5 rejects mid-message
The e2e docs claimed `e2e`-marked tests skip when no proxy answers the
liveness probe, but the harness has always hard-failed: conftest.py's
pytest_runtest_setup calls pytest.fail, its module docstring states
"hard failures only ... never skip", and logging/conftest.py forbids
skipping outright. Align the docs to the code so the single most
important contract reads the same everywhere; a dead proxy turns a run
red instead of being silently skipped and mistaken for a pass. The
per-suite conftest docstrings that described the shared hook as a
"proxy liveness skip" are corrected to "liveness gate" for the same
reason.
Also scope the no-unit-tests hard rule to what it means: never
substitute a unit test for e2e feature coverage, while explicitly
allowing tests that cover the harness itself (e.g.
coverage_registry/test_collector.py), which carry no e2e marker and
run whether or not a proxy is up.
No product code and no harness logic changed.
Resolves LIT-4554
* test(e2e): assert bare-key budget refusal is 429 and /key/info spend reaches the cap
* test(e2e): keep the bare-key budget assertion to the 429 refusal shape
* test(e2e): assert a team's max_budget blocks every key on the team
* test(e2e): focus the team budget case on the 429 blocking behavior
* test(e2e): assert an org budget block is a 429 naming the organization
* test(e2e): assert bare-key budget refusal is 429 and /key/info spend reaches the cap
* test(e2e): keep the bare-key budget assertion to the 429 refusal shape
* test(e2e): assert a team's max_budget blocks every key on the team
* test(e2e): focus the team budget case on the 429 blocking behavior
* test(e2e): harness fixes for long_context, complexity router, UI, and unit coverage
Point long_context_1m at 1M-capable models, harden complexity-smart-router
registration and spend-log assertions, fix key models dropdown selectors, and
add gateway/lifecycle/transport and claude_code unit tests
* test(e2e): harden remaining stage failures in harness
Register complexity-smart-router via create_model + callable probe, fix
create-key UI navigation race, retry management writes and budget ALB
502s, mark Vertex count_tokens N/A when unsupported, and tighten
tool_search model lists for Azure/Bedrock capability gaps
* test(e2e): drop claude_code and harness unit tests from this PR
Keep management, router, budget, and shared conftest harness fixes only
* test(e2e): restore E2E_RESULT pytest_runtest_makereport hook
Accidentally dropped in an earlier harness commit; Grafana status history
depends on these structured log lines
* test(e2e): drop management control-plane write retries
Transient 500 retries do not fix the underlying control plane failures
* test(e2e): skip stage-red claude_code cells; fix multi-window budget latency
Mark the twelve failing claude_code matrix cells skip until product/config
lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of
Claude so the reset wait stays under ALB target idle timeout rather than
masking awselb 502s
* test(e2e): require exactly one LLM-tier spend row for complexity router
Keep alias membership for compose vs stage model names, but assert
len(served) == 1 so a leaked classifier sub-call cannot pass. Also pin
LIT-4521 skip and align LIT-4522/23/24 skip reasons
* test(e2e): harden router callable probe and multi-window budget exhaustion
_router_is_callable treated any non-success chat whose body lacked "Invalid
model name" as callable, so an unpropagated probe key (401), a generic 502, or
a connection reset let the session proceed and hit real "Invalid model name"
failures inside the tests. Require a Success outcome instead; the reload-race
400 and every infra/auth error now correctly read as not-callable.
The multi-window budget test capped the tight window at 3e-6, which gpt-5.5
exhausts on the first call but a cheaper CHEAP_OPENAI_MODEL might not within the
20-call loop, turning a reset test into a spurious "window never enforced"
failure. Drop the tight cap to 1e-9 so the first billed call exhausts it
regardless of model price; the roomy 1m window stays at 1.0 and never blocks.
* test(e2e): use a tradeoff-decision prompt for the complexity router classifier
"Is P equal to NP?" reads to the LLM classifier as a short yes/no question, so
gpt-5.5 classified it SIMPLE and the request routed to the openai backend, which
made the test fail even though the classifier was running. The tier definitions
key on what the request demands, not how hard the answer is, and a short direct
question maps to SIMPLE regardless of subject.
Swap in "Should I pay off my mortgage early or invest the extra money instead?".
It carries none of the heuristic scorer's reasoning/technical/code keywords and
stays short, so heuristic scoring still lands SIMPLE (openai), but the LLM reads
it as a decision that has to weigh tradeoffs and lands it above SIMPLE, which the
config routes to anthropic. Any non-SIMPLE tier serves anthropic, so the classifier
only has to avoid SIMPLE for the test to distinguish a real classifier run from the
heuristic fallback.
