1 commit

Author	SHA1	Message	Date
Mateo Wang	1b81148f2a	test: add e2e tests for spend, budgets and llms (#30869) ... * tests: add e2e tests for spend, budgets and llms * style: make chained comparison of status_code clearer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * remove e2e_tests folder * test: add spend tracking tests * test: multi-window budgets coverage * fix: p0 issues, added types and shared functions for each test suite * chore: add config.yml * test: passthrough endpoints stream/non-stream e2e * style: carry clearer status_code comparison into renamed e2e dir * fix: rename cost breakdown function * fix: pydantic validation for budget info, dont allow explicit type cast * refactor: migrate to gateway client * test: add custom pricing tests * chore: change master key * test(e2e): address greptile review feedback Remove the duplicate cache/cache_params block in the gateway config so the two can't silently diverge under future edits. Reorder the soft-budget test to assert the call isn't a budget block before require_successful_call, since that helper hard-fails any non-2xx and left the budget-block check unreachable; the misleading "skip" comment is corrected. Add a deferred delete in test_budget_delete_removes_it so a failed delete doesn't leak a budget on the shared proxy. Scope the spend_tracking sys.path insertion in pytest_sessionfinish to just the cleanup import so a broader "pytest tests/" run isn't left with a mutated path. * test(e2e): drop misleading skip comment on require_successful_call require_successful_call fails hard, it does not skip; the trailing comment was factually wrong. The function name already states intent, so the comment is removed in both per-model and tag budget helpers. * test(e2e): assert budget-isolation invariant before success check On the should-still-succeed path of the per-model and tag isolation tests, check is_budget_block before require_successful_call. If the isolation bug fires the unaffected model/tag is blocked, so asserting the specific 'blocked by X' invariant first yields the diagnostic message instead of a generic upstream-failure. Matches the ordering in test_soft_budget_e2e.py. * fix(e2e): guard spend-log truncate on skip and stop returning unrelated priced rows * fix(e2e): run case init() inside try so partial-init failures tear down run_case called case.init() outside the try/finally that runs teardown(), so a case that registers cleanups progressively (create team, then user, then key) and then fails partway through init() would leak the already-created entities on the long-lived shared proxy. Move init() inside the try so teardown always runs. Add a regression test that registers a cleanup then raises mid-init and asserts the resource is still released. * test(e2e): mark known pricing-leak isolation test xfail(strict) test_custom_pricing_is_isolated_from_sibling_deployment documents a real proxy gap (a deployment's custom per-token pricing leaks into the shared cost map for sibling deployments of the same underlying model) and was left unconditionally failing, which pollutes the suite's pass/fail signal. Mark it xfail(strict=True) so the suite stays green while the leak persists and turns into a failure the moment isolation is fixed, prompting the marker's removal. * refactor(e2e): make suite pass its shipped strict basedpyright config The suite ships tests/pyrightconfig.json (strict, no Any), but basedpyright --project tests reported four errors in it: three reportAny on the parametrize ids=lambda c: c.__name__, and one reportUnusedFunction on the underscore-prefixed autouse fixture _require_live_proxy. Replace the untyped lambda with a typed _case_id(case_cls: Type[_BudgetCase]) -> str so the ids are no longer Any, and rename the fixture to require_live_proxy so basedpyright no longer treats it as an unused private function (it is referenced only by pytest's autouse machinery). basedpyright --project tests now reports zero errors. * fix(tests/e2e): gate spend-log truncate on e2e marker, not test directory * test(e2e): run harness unit tests without a live proxy The autouse session fixture skipped the whole tests/e2e session when no proxy answered, which also skipped test_lifecycle.py, a pure unit test of run_case that never touches the proxy. A regression test that silently skips gives no signal, so the skip now lives in pytest_runtest_setup gated on the same e2e marker the spend-log truncate guard already uses: live tests skip when no proxy is up while harness unit coverage always runs. The liveness probe is cached with lru_cache so it still runs once per session * test(e2e): clean up gateway config comment debris Fix the typo on the header comment and drop the orphaned namespace/ttl comment remnants left indented under cache_params; the active values are already set above. Flagged by greptile review. * fix: add new tests, split gateway * test(e2e): type the redis spend-counter probe for strict basedpyright The new cold-counter reseed test drove its redis client untyped, so the strict tests/pyrightconfig.json (reportUnknown*, reportAny) flagged ten errors once the file landed: scan_iter/get came back unknown and the pool.map lambda had an untyped parameter. Annotate the client as redis.Redis[str] via a TYPE_CHECKING import (the runtime import stays lazy so the suite still skips, not errors, when redis is absent), which resolves scan_iter to Iterator[str] and get to str | None, and replace the lambda with a typed inner function mirroring _burst. basedpyright --project tests is back to zero errors. * test(e2e): xfail the known team multi-window failure and isolate member teardown Greptile flagged two issues in the mirrored split-gateway commit. The team multi-window budget test documents a real /team/new write bug (budget_limits go straight to the Json? column and Prisma 500s, unlike the json.dumps'd key and /team/update paths) and was left as an unconditional hard failure, which would turn any live-proxy CI run red; mark it xfail(strict=True) like the custom-pricing isolation test so the suite stays green while the bug persists and flips to a failure the moment the write is fixed and the marker should go. The class-scoped member fixture in test_team_member_budget_e2e.py tore down its key, user, and team sequentially with no exception isolation, so a failed delete_key would strand the user and team on the long-lived shared proxy. Route cleanup through a ResourceManager: register each delete progressively and run them LIFO best-effort in a finally, so a partial-setup failure still releases what came before and one failed delete never blocks the rest. * test(e2e): set fast budget-reset cadence in gateway config so staging windows reset within e2e timeouts * test(e2e): surface real /spend/tags errors instead of masking them as missing tags The spend-tracking e2e client swallowed every non-200 from /spend/tags into an empty list, so a real server error or a response-shape mismatch showed up only as the generic "tag never appeared in /spend/tags" with no diagnostics. That masking is what made the original cluster failure undiagnosable. spend_by_tags now raises SpendTagsError carrying the actual HTTP status and body for any non-Success result, and poll_tag_spend fails fast on a hard server error rather than polling it into a timeout; eventual consistency only manifests as a 200 whose payload does not yet carry the tag, so only that case waits. The tag test now reports the last observed status and asserts the endpoint returned 200 at least once, with no weakened assertions. Hardening surfaced the real defect in the test itself: /spend/tags returns a top-level JSON array (List[LiteLLM_SpendLogs]), but the client validated against a SpendTagsResponse dict wrapper that never matched, so every call fell through to the empty-list mask. Wired spend_by_tags to the existing TagSpends RootModel and removed the dead SpendTagsResponse model. Verified against the real Postgres that request_tags is stored as proper JSONB arrays and /spend/tags aggregates them correctly, so there is no encoding bug to fix here. * test(e2e): drop flaky test_tag_spend_matches_sum_of_tagged_logs The test wrote tagged requests and polled /spend/tags expecting read-after-write consistency. /spend/tags itself is fine; verified live that request_tags is stored as a JSON array and the endpoint reflects a fresh tag within seconds, so the failures were a timing flake under full-suite load rather than a real defect. Coverage is retained by test_request_tags_round_trip (tags persist onto the row) and the /spend/tags route probe in test_spend_routes.py. Also remove the now-dead tag-spend scaffolding this test was the only user of: poll_tag_spend, spend_by_tags, TagSpendPoll, SpendTagsError, the TagSpend/TagSpends models, and their imports. * test(e2e): widen budget-reset wait windows to de-flake wall-clock-aligned resets The short-window reset tests asserted the reset landed within WINDOW_SECONDS + 45 (~75s), but the 30s budget window is wall-clock-aligned, so the reset can land up to a full window after start, then the rescheduler (~15-20s) zeroes the spend, plus poll and DB lag. A real run measured 84s, just over the 75s bound, and which of the short-window siblings tripped flipped run to run. Widen the wait loops to 150s and the elapsed assertions to WINDOW_SECONDS + 90 (120s for the key test). A genuinely stuck rescheduler is still caught by the wait-loop timeout, so this only removes the timing flake, not the regression signal. * test(e2e): let the spend-counter reseed test reach a cluster-mode TLS redis The test's _redis() built a standalone, non-TLS client on the docker-compose defaults (localhost:6380), so against the EKS serverless ElastiCache (cluster-mode + TLS) it could never connect and the test skipped. Honor E2E_REDIS_SSL and E2E_REDIS_CLUSTER so it builds a TLS RedisCluster client when the deploy provides them, and E2E_REDIS_NAMESPACE so the counter is read with a direct GET (cluster-safe) rather than a keyspace scan that can't span shards. The local standalone path and the graceful skip-on-unreachable behavior are unchanged. * test(e2e): take the direct-GET spend-counter path on E2E_REDIS_CLUSTER The gateway's cache sets no namespace, so the counter key is the bare spend🔑<hash>. Trigger the cluster-safe direct GET on E2E_REDIS_CLUSTER (not only on E2E_REDIS_NAMESPACE) so the cluster deploy need not set a namespace it does not use; the namespaced key is still tried first when a namespace is given. * test(e2e): use REDIS_HOST/REDIS_PORT and drop the unused redis knobs The runner is a standalone test pod, so the proxy's own REDIS_HOST/REDIS_PORT names are unambiguous - no E2E_ prefix needed. The only deployed redis it talks to is the serverless ElastiCache (always TLS + cluster), so that is inferred from REDIS_HOST being set rather than carried as ssl/cluster knobs. Stage sets no cache namespace (bare counter key, read directly on the cluster) and is passwordless, so the namespace and password env are gone; the local namespace is still handled by the standalone SCAN. * test(e2e): replace the vacuous failure-row test with per-model attribution test_failure_call_writes_failure_status_row had two skip hatches (the call did not fail, or no failure row landed) and never asserted anything on this proxy - gemini accepts an empty message (HTTP 200), and live failure-row logging is non-deterministic across providers. Replace it with a deterministic check: one key calling gemini-2.5-flash and claude-haiku-4-5 gets one spend row per call, each carrying its own model and a nonzero cost, under distinct request_ids that match the call's response id. Verified live on stage (gemini/gemini-2.5-flash $0.00053, anthropic/claude-haiku-4-5 $0.000038, distinct ids matching the responses). Failure-status row construction stays covered by the unit suite. * test(e2e): assert /spend/logs returns the key's spend without 5xx Regression for the intermittent 500s on /spend/logs (DB query / serialization errors under load). The existing spend_logs() helper swallows non-success responses into an empty list, so a 500 looks identical to 'rows not flushed yet'. This test queries the endpoint directly and asserts a Success response on every poll, failing loudly on any 5xx, then requires the call's nonzero spend to surface. --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-24 15:01:57 -07:00