Commit graph

15 commits

Author SHA1 Message Date
Yassin Kortam
c208bec37f
test(e2e): cover user deletion removing it from user inventory (#34007) 2026-07-20 22:38:32 +00:00
Yassin Kortam
5c8e7e6924
test(e2e): cover organization update persistence via /organization/info (#34010) 2026-07-20 22:36:39 +00:00
Yassin Kortam
61b906f9b6
test(e2e): cover model deletion removing it from the catalog (#34006) 2026-07-20 22:36:15 +00:00
Yassin Kortam
f21704c672
test(e2e): cover user update persistence via /user/info (#33998)
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
2026-07-20 22:18:36 +00:00
Yassin Kortam
eb27447a1d
test(e2e): cover team update persistence via /team/info (#33997)
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
2026-07-20 21:56:07 +00:00
Yassin Kortam
68be053e96
test(e2e): cover created user appearing in /user/list (#34016) 2026-07-20 14:29:13 -07:00
Yassin Kortam
4c77a5433a
test(e2e): cover created team appearing in /team/list (#34015) 2026-07-20 14:27:49 -07:00
Yassin Kortam
53f5a8c380
test(e2e): cover key block persisting to /key/info (#34014)
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
2026-07-20 14:26:18 -07:00
Yassin Kortam
214945a223
test(e2e): cover organization deletion removing it from /organization/info (#34009)
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
2026-07-20 19:36:59 +00:00
Yassin Kortam
72be5a9bc0
test(e2e): cover tag creation persisting for spend categorization (#34018) 2026-07-20 12:24:50 -07:00
Yassin Kortam
51df801159
test(e2e): cover key regeneration rotating to a working new key (#34000) 2026-07-20 12:23:22 -07:00
Yassin Kortam
08fa25042c
test(e2e): rename Gateway to ProxyClient and expose it as a session-scoped fixture (#33750)
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.

Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.

Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
2026-07-18 18:41:18 +00:00
mateo-berri
55c8ca41b5 ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
mubashir1osmani
54d404ef2c
fix(e2e): batch credentials wiring and compose harness for live proxy suite (#32744)
* fix(e2e): wire batch provider secrets for docker and k8s

Point batch deployments at the credential field names and os.environ refs
the gateway actually resolves from process env (compose .env or EKS secret
mounts). Missing secrets skip instead of failing red so a red run means a
product bug. Mirror S3 bucket env aliases in docker-compose for provider_fallback

* fix(e2e): drop batch provider_env unit tests

The batches suite is live e2e only; no monkeypatch or unit-level tests

* fix: batch credentials, provider list, and team db lookup

Keep object-storage fields through CredentialLiteLLMParams and resolve
os.environ/ refs when reading deployment credentials so Vertex/Bedrock
batch file uploads see bucket and AWS keys from K8s/docker env

Skip managed batch list when the request is provider-scoped so
/{provider}/v1/batches list works instead of 500

Force DB on check_db_only team lookups and stop masking non-404 errors
as "team doesn't exist"

Drop e2e runner-side skip helpers; hard-fail on missing gateway secrets

* fix: tag reseed, team window spend, and remaining e2e flakes

Reseed spend:tag counters from LiteLLM_TagTable so cold redis still
enforces after the spend writer flushes

When applying post-call cost to team multi-window counters, load the
team from the DB if it is missing from the management cache so window
spend is not dropped on cache misses

Harden cold-counter reseed e2e (namespace-aware keys, burst success,
poll). Give tag budget more headroom. Retry /key/update on redis DNS
blips. Ensure NLTK punkt_tab is present for pipecat realtime audio

* revert: drop product code changes; e2e-only scope

Reverts all litellm/ and unit-test product edits. This branch is limited
to tests/e2e per contributor instruction

* fix(e2e): harden batch list and team member setup races

provider_fallback list falls back when managed batches reject provider
filtering. Team create waits for /team/info and member_add retries on
transient team-not-found so split control-plane lag does not red the suite

* fix(e2e): remove .env.example

Leave local .env and docker-compose env wiring as the secret source

* fix(e2e): wire files_settings and faster budget rescheduler for compose

OpenAI/Azure batch file uploads need files_settings; budget reset e2e needs a
short rescheduler window. Drop unsupported bedrock-encoded create_batch cells,
tolerate bedrock file.bytes=0, and surface team-info wait failures instead of
hanging silently

* chore(e2e): strip verbose comments from batch capabilities

* fix(e2e): assert managed list fallback before provider_fallback skip

When provider-scoped list is rejected, still fetch the unfiltered list and
check the envelope. Only skip membership when the id is a raw
provider_fallback batch that managed list cannot index
2026-07-10 11:31:40 -07:00
mubashir1osmani
a1873d89cc
test(e2e): add management suite covering key/team/user/org lifecycle and route permissions (#32300)
* test(e2e): add management suite covering key/team/user/org lifecycle and route permissions

* test(e2e): decouple the enforcement-flip assertion from upstream health

Polling for a 200 on the newly-allowed model required it to be a routable,
healthy upstream, which is not the contract under test; poll until the
key_model_access_denied 403 lifts instead, excluding 401 so a revoked key
cannot read as success. Also document that the delete test's deferred teardown
firing on an already-deleted key is deliberate: cleanup must survive the test
failing before the in-body delete, and the repeat delete is a warn-free no-op
(the proxy answers 404 No keys found)

* test(e2e): inline the management suite's model and tpm literals

* test(e2e): drop the models_mgmt suite line from the folder list

* test(e2e): write the tpm limit as a plain integer literal
2026-07-06 19:11:27 -07:00