* test(e2e): add failing reproducers for two open gateway bugs
Both tests assert the behavior a customer expects and both are red today. They
are reproducers, not regressions: the product is wrong, not the tests.
Native passthrough returns almost none of the operational headers the managed
route does. A /gemini/ generateContent call comes back with three x-litellm-*
headers and no x-ratelimit-* at all, against sixteen and four on
/v1beta/models/{m}:generateContent for the same prompt, and critically it omits
x-litellm-response-cost. Customers front provider-native traffic through this
route and read those headers to reconcile spend and pace themselves, so native
traffic is currently invisible to the tooling that covers every other route.
/budget/update rejects any model_max_budget with a 500. The reported symptom was
model ids containing dots, and that reproduces (prisma raises "Unexpected
`-5.2[FloatValue]` Expected `:`" because the key is interpolated into a GraphQL
query unquoted, so glm-5.2 lexes as an identifier followed by a float), but the
plain name gpt4o fails too, on a separate "model_max_budget should be of any of
the following types: Json" type mismatch at budget_management_endpoints.py:173.
Omitting the field returns 200. The test drives both names so the failure says
whether per-model budgets are broken outright or only for punctuated ids; today
it stops on the plain name, which is the wider bug.
* test(e2e): add reproducer for unenforced end-user per-model rate limits
model_max_budget accepts an rpm_limit alongside the spend cap, and /budget/new
stores it: the create response echoes {"gemini-2.5-flash": {"rpm_limit": 1,
"max_budget": 100.0, "budget_duration": "1d"}}. Attach that budget to an end
user, drive three calls as that user, and all three return 200. The limit is
accepted, persisted, and then ignored.
The same shape already works when the budget hangs off a key, which is what
makes this quietly dangerous: the API gives every indication the cap is in
force. A customer using it to hold one end user to a slow rate on a shared key
gets no throttling at all.
Harness additions this needs: ModelBudgetEntry carries the rpm_limit/tpm_limit
the route already accepts, BudgetNewBody and create_budget carry
model_max_budget, and create_customer can attach an existing budget_id rather
than only an inline max_budget.
Red today, for the reason in the assertion message.
* test(e2e): tighten model_max_budget reproducers and drop in-loop closure
Trim the reproducer docstrings to the contract they assert, keeping the
failure messages that document each red-by-design bug. Replace the nested
per-model closure in the /budget/update test with a module-level predicate
and a per-model helper so nothing closes over a loop variable, and fix the
import order the merge left unsorted.
* test(e2e): skip the three reproducers while their gateway bugs stay open
The passthrough header contract, /budget/update model_max_budget, and
end-user per-model rpm enforcement reproducers all still fail against
staging by design. Skip each with the product gap named so the combined
suite can gate merges on green while the collector keeps reporting the
cells as uncovered.
* test(e2e): validate model budget response contracts
* refactor(e2e): unify model budget schema
* refactor(e2e): reuse shared model budget type
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
This reverts commit dcb4e5033c.
The suites landed without the proof-of-fix and QA runbook the PR body
itself flagged as outstanding, so the coverage they claim is unverified
against a live proxy
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
* fix(e2e): address greptile and veria review on vendor strategy suite
Store search tool keys as os.environ refs and resolve them in SearchAPIRouter.
Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells
* fix(e2e): drop search_api_router os.environ expansion from vendor suite
Keep the PR test-only. Search tools register without an api_key so the
proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a.
* test(e2e): drop search e2e suite from vendor strategy PR
Remove the /v1/search coverage file and its registry rows so this PR
no longer carries search endpoint testing.
The strict-priority e2e (added with the zero-increment limiter fix) can
never pass on stage: the proxy there does not run the
dynamic_rate_limiter_v3 callbacks + priority_reservation settings the
module requires, confirmed by zero limiter log lines across every
gateway and backend pod during the 2026-08-02 run. Config lives in the
infra repo; LIT-5118 tracks adding it.
The throughput SLO test failed the same run with 65.9% of requests dying
at the ELB as 502/503 before reaching a pod. The per-replica SLO rework
fixed the RPS-floor assertion but cannot help when stage idles at one
warm gateway replica; LIT-5119 tracks pre-scaling the fleet for the load
phase.
Both skips name their ticket, and the coverage registry returns the two
cells to the gap list while they are in place.
A single read of key_info.spend races the batched spend writer: deltas
earned before a reset flush to the DB up to ~60s later
(proxy_batch_write_at) and land on the row after the reset zeroed it.
The stage runs on Jul 30 and Aug 2 failed
test_key_budget_reset_at_advances_after_window exactly this way, with
spend back at the driven total while budget_reset_at had advanced and
calls flowed again.
Replace the single reads in rung 3 (spend zeroed after reset) and rung 4
(roomy window keeps spend) with _poll_key_spend, which re-reads to a 90s
deadline covering one full flush-plus-reset cycle. A reset that never
zeroes the row keeps spend pinned and still times out, so the regression
guard keeps its teeth.
The atomic check-and-increment path skipped any counter whose increment
was <= 0. The dynamic rate limiter always passes a zero token increment
pre-call because usage lands on the counters post-response, so on a model
configured with only tpm the limiter evaluated no counters at all: no
model-wide TPM cap and no priority reservation, in either generous or
strict mode. Regressed in dd57ae6691 when the pre-call flow moved off the
read-only should_rate_limit check, which did evaluate token limits.
Keep zero-increment counters in the payload so they act as a pure check
(current + 0 > limit), matching the pre-regression semantics in both the
Lua and in-memory paths. Adds unit regressions at the primitive and hook
level plus a live e2e covering the priority_generous/priority_strict
registry rows.
#35271 restored the hierarchy where a team-scoped key is governed by the
team and team-member budgets only; the owner's personal max_budget applies
to their personal keys. Three places in the e2e suite still encoded the
old direction and would fail against a proxy built from staging.
test_user_budget_enforced_across_all_their_keys asserted that the owner's
team-member key is refused once their personal budget is exhausted. It now
asserts only the personal keys are refused, and keeps the team key as the
control that must keep serving, which pins the restored direction instead
of leaving it unasserted. Renamed to match what it now covers.
test_team_member_key_user_budget_resets_after_window drove a team key to a
block off the owner's personal budget, so nothing can block it any more and
_drive_to_block could never succeed. Its premise is gone rather than moved,
so it is removed; the sibling personal-key test still covers
quota_management.budget.internal_user.resets_after_window.
The registry rationale for that row dropped its "and team-member keys"
clause for the same reason.
The cache-hit and paid rows for the two driver calls flush from different
pods on independent update_spend timers, so waiting only for the cache-hit
row can return a half-arrived result set where the paid-row assertion then
fails on an empty list. Requiring both row kinds in the poll predicate lets
the existing deadline absorb the slower flush without weakening any assertion
require_env hard-failed a test (and, for the shared litellm-ops secret, drove
piling every provider credential into one blob) whenever an optional cred was
absent. Most call sites either read a value the test actually uses or just
gated on the runner's env for a key the gateway consumes.
Read os.environ directly where the test uses the value; drop the presence-only
gates so those cases run against the proxy instead of pre-failing on the
runner's environment. Removes the require_env helper from e2e_config.
* test(e2e): cover key max_budget blocks on personal, team, and team-member keys
* refactor(e2e): convert budget enforcement cases to the resources-fixture pattern
The E2ECase class pattern existed only in this file; every other suite uses
plain pytest tests with the resources fixture. Rewrites the nine cases as two
spec classes and removes the now-dead E2ECase protocol and run_case driver
from lifecycle.py
* fix(e2e): reference client.proxy in mid-conversation native providers test
EndpointsClient exposes the shared ProxyClient as .proxy and has never had a
.gateway attribute, so these two calls raised AttributeError at runtime and
failed the tests/e2e basedpyright zero-error gate for any PR touching e2e
files. Introduced in 23b5b7d199.
* test(e2e): cover 12 non-core LLM coverage registry cells
Raises Non-Core LLMs registry coverage from 24/50 to 36/50 (overall 51.9%
to 54.8%). Four cells were already asserted by existing tests and only
gain their covers marker (openai embeddings, openai image generation,
openai TTS, cohere rerank); one is dual-marked onto the existing
spend-tracking embeddings test rather than duplicated.
New tests: bedrock and vertex embeddings, streaming TTS (asserts chunked
transfer encoding so a buffered body cannot pass), audio transcriptions
via the realtime suite's wav fixture, moderations flag/pass pair, and
files list/retrieve in the batches suite.
Harness: e2e_http.upload generalized to any form model with a
file_content_type override (batches path unchanged), new stream_binary
primitive + BinaryStream for binary chunked responses, transcribe and
moderations client methods, file retrieve/list client methods.
* fix(e2e): close streamed TTS response on error paths and surface the error body
With stream=True a non-2xx response returned with the body unread, keeping
the socket checked out until garbage collection; the sibling
_streaming_outcome already consumes resp.text on error. The response now
closes on every path and BinaryStream carries a bounded error_body so a
failed stream call is triageable.
* test(e2e): assert streamed TTS response carries no content-length
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping
Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly.
* test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse
Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL,
TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path,
Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm
chat, and Nova Sonic realtime. Registry cells updated for the new markers.
* test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes
Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and
Langfuse driver models now use Anthropic haiku so local runs stay green when
Gemini daily quota is exhausted.
* test(e2e): drop Langfuse spend suite; feature is being deprecated
Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the
langfuse-only conftest driver/credentials fixtures.
* test(e2e): fold provider/batch feature tests into their endpoint suites
Keep the e2e layout endpoint- and suite-scoped instead of one file per
provider or feature
Move the virtual-key auth case into access_control/test_access_control_e2e.py
as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone
test_virtual_key_auth_e2e.py
Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock
assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py.
The hosted_vllm batch case is skipped for now since it needs a live vLLM server
(HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the
gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant
Merge the cohere, gemini and hosted_vllm chat cases into
llm_translation/test_chat_completions_regression_e2e.py so /chat/completions
coverage lives in one endpoint file, and repoint the coverage_registry source
fields to the new homes
Move the shared CacheControl / TextBlock / RichMessage request blocks into the
root models.py (re-exported from endpoints_client) so quota_management can use
them without a cross-suite import, which also clears the basedpyright errors in
test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in
test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed
json.loads path
* test(e2e): address review feedback and re-home virtual-key coverage
Replace the tautological Bedrock assume-role batch id assertion (`startswith(...)
or batch.id`, always true) with a managed-id shape check, since the unified
target_model_names path re-encodes the id rather than returning a raw ARN
Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no
longer consume the key's sole request unit before batch create runs; the batch
create then clears the generic per-request limiter and the batch limiter is what
returns the "Batch rate limit exceeded" body the assertions check
Set exercised_on to [] on the pass-through header test; it drives a pass-through
endpoint, not /chat/completions
Move the virtual-key valid_allows / invalid_denied cells from other.yaml to
mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management,
and point its covers marker at the new ids
tests/e2e/conftest.py's pytest_sessionfinish truncated LiteLLM_SpendLogs
against whatever DATABASE_URL resolved to, gated only by "an e2e test body
ran". Pointed at a shared or staging DB, a routine local run wiped real
spend data. It also reached the truncate helper through a sys.path.insert
into quota_management/spend_tracking/spend_e2e_client.py, a cross-suite
import-by-path hack it then unwound in a finally.
The cleanup now routes through a new run_spend_log_cleanup in a top-level
tests/e2e/e2e_db.py, which fires the destructive truncate only when the
operator set E2E_RESET_SPEND_LOGS=1 and an e2e test actually ran. Any other
value (unset, 0, true, empty) leaves the DB untouched, so presence of the
variable alone or a test run alone never arms the truncate. The decision
plus the injectable truncate callable live in that pure helper, and conftest
is a thin adapter that supplies os.environ.get(...), the session stash, and
reset_spend_logs.
reset_spend_logs itself moved from spend_e2e_client.py into e2e_db.py
(implementation unchanged), sitting next to e2e_config and lifecycle so both
conftest and any suite import it by name; the sys.path munging is gone.
Nothing else imported reset_spend_logs, so spend_e2e_client.py drops the
definition, its __all__ entry, and the now-unused os import.
* test(e2e): assert the long budget window keeps blocking after the short window resets
The multi-window budget tests proved the tight window blocks and self-heals
but never asserted the other direction: a long (1d) window whose cap the
accumulated spend already crossed must keep refusing calls even inside a
fresh short window. Adds one test per file (key and team) that drives spend
to a block, waits for the short window's reset_at to strictly advance (the
reset job zeroes that window's counter in the same pass), then polls until
the refusal is attributed to the 1d window ("over 1d budget"), failing
immediately if any call succeeds or a non-budget error leaks. Harness gains
per-window reset_at readback: BudgetWindowState in models.py and
key_window_reset_at / team_window_reset_at on BudgetClient.
* refactor(e2e): hoist shared budget-suite helpers into budget_client
drive_to_block and as_datetime existed as five and four per-file copies in
the budgets suite; both move to budget_client with each file keeping a thin
delegating wrapper so call sites and per-file pacing stay unchanged. The
three /team/info readers in budget_client now share a private _team_info.
Also guard the long-window reset_at snapshots with explicit non-None asserts
so the midnight-roll diagnostic cannot misreport when the window is missing
from the info response (greptile P2s).
* docs(e2e): tighten the multi-window module docstrings
* refactor(e2e): type window reset_at as datetime and expose plain window readers
BudgetWindowState.reset_at becomes a pydantic-parsed datetime, so the
multi-window tests compare real datetimes instead of hand-parsing strings.
The duration-keyed accessors are replaced by two plain readers,
key_budget_windows and team_budget_windows, with the pure window_reset_at
lookup exported; the client no longer encodes one test's access pattern.
* test(e2e): name the tiny short-window cap and comment the wait loops
* test(e2e): surface the 429 budget-block assert in the multi-window tests
drive_to_block now returns the blocking response so a test body can assert
on its shape; the two long-window tests assert status 429 explicitly, which
also pins the multi-window enforcement path's HTTP mapping (the enforcement
suite only covers the single-budget path). Other callers ignore the return
and are unchanged.
* refactor(e2e): scope this PR to the multi-window test, drop the cross-suite hoist
The helper hoist rewrote four unrelated budget test files (reset, reset_advances,
team_member_reset, user_across_keys) to pull drive_to_block and as_datetime out
of budget_client, which is refactor churn beyond this PR's multi-window scope.
This restores those four to their pre-PR state and gives the two multi-window
tests their own inline drive-to-block loop again, so the PR touches only the
multi-window feature: its two tests plus the budget_client window readers and the
reset_at datetime typing they actually use. The suite-wide helper dedup can land
on its own PR
* docs(e2e): number the long-window key test steps inline
* docs(e2e): number the long-window team test steps inline
Add a live spend-tracking e2e that drives a streaming anthropic-format
/v1/messages request through litellm's anthropic-messages -> OpenAI Responses
adapter and asserts the consumed stream writes exactly one SpendLogs row with
nonzero cost and token counts, attributed to the calling key under
custom_llm_provider openai and the /v1/messages call_type.
The deployment is a Responses-only OpenAI model (gpt-5.3-codex), so a served,
costed row proves the Responses path was taken; the chat-completions bridge
would have failed at OpenAI on an endpoint the model does not expose. Adds a
streaming /v1/messages method to the shared Gateway and the suite client, the
model to the inline compose config and driver-model registration, a coverage
registry row (quota_management.spend_tracking.messages_bridge.logs_cost), and
the matching variant vocab entry. The _summarize spend-row detail also gains
call_type and custom_llm_provider so a failed assertion prints the fields it
asserts on.
Resolves LIT-4546
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
* test(e2e): budget reset diagonal for team, org, user, and #32005 team-member keys
Adds E2E-7/8/10/11 from the budget-level x key-kind coverage matrix: each budget
level serves traffic again after its budget_duration window elapses, walking the
same ladder as the enforcement diagonal. New registry rows and tests cover the
team, organization, and internal-user reset rungs, plus the #32005 interplay
where a team-member key frozen by its owner's user budget comes back when the
user's window renews; the bare-key and per-team-member rungs already had coverage
Each case isolates the cap to one entity, drives spend to a budget_exceeded
block, then polls past the window until a call succeeds, holding every refusal
as a budget block so a reset that no-ops (stays blocked forever) or crashes
(leaks a 5xx) fails the test. budget_duration becomes an optional param on the
budget_client create_team / create_user / create_org helpers
* test(e2e): fold the reset diagonal into test_budget_reset_e2e.py and address greptile nits
Move the team / org / user / #32005 reset cases out of the standalone
test_budget_reset_diagonal_e2e.py and into test_budget_reset_e2e.py, absorbing
the pre-existing bare-key reset into the same TestBudgetResetDiagonal spec class
so the whole reset ladder reads as one file (mirroring how the enforcement
diagonal lives in test_budget_enforcement_e2e.py) and the drive/poll helpers are
defined once instead of duplicated across reset files.
Greptile nits: bound the drive phase to under one window (12 attempts x 2s < 30s)
so a block is observed before the reset job can fire, and replace the bare assert
in the poll loop with a pytest.fail that prints the HTTP status, so a provider 429
or a crashed reset path is distinguishable from a budget block at a glance.
* test(e2e): trim reset diagonal docstrings back to the file's original style
* test(e2e): inline single-use drive-loop bounds
* test(e2e): cut the reset module docstring to one line
* test(e2e): make the org reset test wait for a scheduled window (bugbot)
/organization/new stores budget_duration without scheduling budget_reset_at, so
the reset job's NULL catch-up branch zeroes org spend on its first 5-10s tick;
the org reset test could pass off that catch-up instead of a real window roll
(tracked as LIT-4570). The test now reads the org's budget_id and polls
/budget/info until budget_reset_at is scheduled before driving spend, so the
recovery it observes can only come from a genuine window expiry. Verified live:
the org case now runs ~33s (a full window) instead of beating the rescheduler
OpenAI GPT-5.6 Claude Code cells burn minutes on CLI timeouts under the
full stage suite; gate them behind COMPAT_OPENAI_GPT_CELLS=1 like Mantle.
Multi-window budget e2e used max_tokens=1 which gpt-5.5 rejects mid-message
The e2e docs claimed `e2e`-marked tests skip when no proxy answers the
liveness probe, but the harness has always hard-failed: conftest.py's
pytest_runtest_setup calls pytest.fail, its module docstring states
"hard failures only ... never skip", and logging/conftest.py forbids
skipping outright. Align the docs to the code so the single most
important contract reads the same everywhere; a dead proxy turns a run
red instead of being silently skipped and mistaken for a pass. The
per-suite conftest docstrings that described the shared hook as a
"proxy liveness skip" are corrected to "liveness gate" for the same
reason.
Also scope the no-unit-tests hard rule to what it means: never
substitute a unit test for e2e feature coverage, while explicitly
allowing tests that cover the harness itself (e.g.
coverage_registry/test_collector.py), which carry no e2e marker and
run whether or not a proxy is up.
No product code and no harness logic changed.
Resolves LIT-4554
* test(e2e): assert bare-key budget refusal is 429 and /key/info spend reaches the cap
* test(e2e): keep the bare-key budget assertion to the 429 refusal shape
* test(e2e): assert a team's max_budget blocks every key on the team
* test(e2e): focus the team budget case on the 429 blocking behavior
* test(e2e): assert an org budget block is a 429 naming the organization
* test(e2e): assert bare-key budget refusal is 429 and /key/info spend reaches the cap
* test(e2e): keep the bare-key budget assertion to the 429 refusal shape
* test(e2e): assert a team's max_budget blocks every key on the team
* test(e2e): focus the team budget case on the 429 blocking behavior
* test(e2e): harness fixes for long_context, complexity router, UI, and unit coverage
Point long_context_1m at 1M-capable models, harden complexity-smart-router
registration and spend-log assertions, fix key models dropdown selectors, and
add gateway/lifecycle/transport and claude_code unit tests
* test(e2e): harden remaining stage failures in harness
Register complexity-smart-router via create_model + callable probe, fix
create-key UI navigation race, retry management writes and budget ALB
502s, mark Vertex count_tokens N/A when unsupported, and tighten
tool_search model lists for Azure/Bedrock capability gaps
* test(e2e): drop claude_code and harness unit tests from this PR
Keep management, router, budget, and shared conftest harness fixes only
* test(e2e): restore E2E_RESULT pytest_runtest_makereport hook
Accidentally dropped in an earlier harness commit; Grafana status history
depends on these structured log lines
* test(e2e): drop management control-plane write retries
Transient 500 retries do not fix the underlying control plane failures
* test(e2e): skip stage-red claude_code cells; fix multi-window budget latency
Mark the twelve failing claude_code matrix cells skip until product/config
lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of
Claude so the reset wait stays under ALB target idle timeout rather than
masking awselb 502s
* test(e2e): require exactly one LLM-tier spend row for complexity router
Keep alias membership for compose vs stage model names, but assert
len(served) == 1 so a leaked classifier sub-call cannot pass. Also pin
LIT-4521 skip and align LIT-4522/23/24 skip reasons
* test(e2e): harden router callable probe and multi-window budget exhaustion
_router_is_callable treated any non-success chat whose body lacked "Invalid
model name" as callable, so an unpropagated probe key (401), a generic 502, or
a connection reset let the session proceed and hit real "Invalid model name"
failures inside the tests. Require a Success outcome instead; the reload-race
400 and every infra/auth error now correctly read as not-callable.
The multi-window budget test capped the tight window at 3e-6, which gpt-5.5
exhausts on the first call but a cheaper CHEAP_OPENAI_MODEL might not within the
20-call loop, turning a reset test into a spurious "window never enforced"
failure. Drop the tight cap to 1e-9 so the first billed call exhausts it
regardless of model price; the roomy 1m window stays at 1.0 and never blocks.
* test(e2e): use a tradeoff-decision prompt for the complexity router classifier
"Is P equal to NP?" reads to the LLM classifier as a short yes/no question, so
gpt-5.5 classified it SIMPLE and the request routed to the openai backend, which
made the test fail even though the classifier was running. The tier definitions
key on what the request demands, not how hard the answer is, and a short direct
question maps to SIMPLE regardless of subject.
Swap in "Should I pay off my mortgage early or invest the extra money instead?".
It carries none of the heuristic scorer's reasoning/technical/code keywords and
stays short, so heuristic scoring still lands SIMPLE (openai), but the LLM reads
it as a decision that has to weigh tradeoffs and lands it above SIMPLE, which the
config routes to anthropic. Any non-SIMPLE tier serves anthropic, so the classifier
only has to avoid SIMPLE for the test to distinguish a real classifier run from the
heuristic fallback.
