require_env hard-failed a test (and, for the shared litellm-ops secret, drove
piling every provider credential into one blob) whenever an optional cred was
absent. Most call sites either read a value the test actually uses or just
gated on the runner's env for a key the gateway consumes.
Read os.environ directly where the test uses the value; drop the presence-only
gates so those cases run against the proxy instead of pre-failing on the
runner's environment. Removes the require_env helper from e2e_config.
* fix(e2e): reference client.proxy in mid-conversation native providers test
EndpointsClient exposes the shared ProxyClient as .proxy and has never had a
.gateway attribute, so these two calls raised AttributeError at runtime and
failed the tests/e2e basedpyright zero-error gate for any PR touching e2e
files. Introduced in 23b5b7d199.
* test(e2e): cover 12 non-core LLM coverage registry cells
Raises Non-Core LLMs registry coverage from 24/50 to 36/50 (overall 51.9%
to 54.8%). Four cells were already asserted by existing tests and only
gain their covers marker (openai embeddings, openai image generation,
openai TTS, cohere rerank); one is dual-marked onto the existing
spend-tracking embeddings test rather than duplicated.
New tests: bedrock and vertex embeddings, streaming TTS (asserts chunked
transfer encoding so a buffered body cannot pass), audio transcriptions
via the realtime suite's wav fixture, moderations flag/pass pair, and
files list/retrieve in the batches suite.
Harness: e2e_http.upload generalized to any form model with a
file_content_type override (batches path unchanged), new stream_binary
primitive + BinaryStream for binary chunked responses, transcribe and
moderations client methods, file retrieve/list client methods.
* fix(e2e): close streamed TTS response on error paths and surface the error body
With stream=True a non-2xx response returned with the body unread, keeping
the socket checked out until garbage collection; the sibling
_streaming_outcome already consumes resp.text on error. The response now
closes on every path and BinaryStream carries a bounded error_body so a
failed stream call is triageable.
* test(e2e): assert streamed TTS response carries no content-length
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping
Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly.
* test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse
Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL,
TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path,
Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm
chat, and Nova Sonic realtime. Registry cells updated for the new markers.
* test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes
Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and
Langfuse driver models now use Anthropic haiku so local runs stay green when
Gemini daily quota is exhausted.
* test(e2e): drop Langfuse spend suite; feature is being deprecated
Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the
langfuse-only conftest driver/credentials fixtures.
* test(e2e): fold provider/batch feature tests into their endpoint suites
Keep the e2e layout endpoint- and suite-scoped instead of one file per
provider or feature
Move the virtual-key auth case into access_control/test_access_control_e2e.py
as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone
test_virtual_key_auth_e2e.py
Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock
assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py.
The hosted_vllm batch case is skipped for now since it needs a live vLLM server
(HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the
gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant
Merge the cohere, gemini and hosted_vllm chat cases into
llm_translation/test_chat_completions_regression_e2e.py so /chat/completions
coverage lives in one endpoint file, and repoint the coverage_registry source
fields to the new homes
Move the shared CacheControl / TextBlock / RichMessage request blocks into the
root models.py (re-exported from endpoints_client) so quota_management can use
them without a cross-suite import, which also clears the basedpyright errors in
test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in
test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed
json.loads path
* test(e2e): address review feedback and re-home virtual-key coverage
Replace the tautological Bedrock assume-role batch id assertion (`startswith(...)
or batch.id`, always true) with a managed-id shape check, since the unified
target_model_names path re-encodes the id rather than returning a raw ARN
Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no
longer consume the key's sole request unit before batch create runs; the batch
create then clears the generic per-request limiter and the batch limiter is what
returns the "Batch rate limit exceeded" body the assertions check
Set exercised_on to [] on the pass-through header test; it drives a pass-through
endpoint, not /chat/completions
Move the virtual-key valid_allows / invalid_denied cells from other.yaml to
mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management,
and point its covers marker at the new ids
* test(e2e): harden stage flakes for batches, UI, and MCP
Unique batch model names avoid load-balancing onto stale azure-batch
deployments that still pointed at the retired gpt-4.1-mini-batch, which
only the managed/unified path was hitting. Retry batch retrieve on 500
and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite
when the compose-only mcp-upstream is unreachable on stage k8s
* test(e2e): cover Datadog remote MCP via search_datadog_logs
Register the regional Datadog MCP endpoint with DD-API-KEY /
DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is
not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*,
assert the proxy shipped it, list tools, call search_datadog_logs for the
marker, and delete the server on teardown. Math-upstream key-access tests
only skip when that compose service is unreachable
* test(e2e): drop compose math MCP upstream; use Datadog only
Key-access denial and happy-path MCP e2e both register the real regional
Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers.
Remove the mcp-upstream compose service and FastMCP add/multiply fixture
* docs(e2e): require real Datadog MCP for all mcp suite tests
Document that tests/e2e/mcp must register via datadog_mcp helpers against
mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams
* chore: restore mcp_e2e_upstream_server.py
Keep the FastMCP fixture file; e2e no longer wires it in compose, but the
module itself is not part of the Datadog-only cleanup
* fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load
pytest on the host never inherited compose env_file keys, so DD_API_KEY
stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the
dynamically loaded datadog_reader module in sys.modules so dataclasses
do not crash under Python 3.12
* test(e2e/batches): harden azure/vertex unified lifecycle flakes
Put the provider deployment name in every JSONL body so Azure does not
depend on a perfect model rewrite. Retry create/retrieve/cancel on
transient statuses with backoff. Drop cancel assertions for azure and
vertex (registry only has a shared basic cell; create+retrieve prove
routing, cancel stays best-effort cleanup)
* test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED
Post-login client redirects abort the first /ui/api-keys/ goto on stage.
Wait off /ui/login after cookie set, then accept the page once Create New
Key is visible even if goto raised ERR_ABORTED
* test(e2e): drop flaky key models dropdown Playwright suite
API management e2e already covers key generate/update persistence. The
UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and
no unique product signal. Remove the suite and unused browser fixtures
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
The e2e docs claimed `e2e`-marked tests skip when no proxy answers the
liveness probe, but the harness has always hard-failed: conftest.py's
pytest_runtest_setup calls pytest.fail, its module docstring states
"hard failures only ... never skip", and logging/conftest.py forbids
skipping outright. Align the docs to the code so the single most
important contract reads the same everywhere; a dead proxy turns a run
red instead of being silently skipped and mistaken for a pass. The
per-suite conftest docstrings that described the shared hook as a
"proxy liveness skip" are corrected to "liveness gate" for the same
reason.
Also scope the no-unit-tests hard rule to what it means: never
substitute a unit test for e2e feature coverage, while explicitly
allowing tests that cover the harness itself (e.g.
coverage_registry/test_collector.py), which carry no e2e marker and
run whether or not a proxy is up.
No product code and no harness logic changed.
Resolves LIT-4554
* test(e2e): read datadog log delivery back from the real datadog api (#33604)
* test(e2e): read datadog log delivery back from the real datadog api
* test(e2e): compare datadog-read cost with math.isclose, not bit-equality
The response_cost now round-trips through DataDog's attribute indexing
pipeline, whose float serialization is not guaranteed to preserve the
exact bit pattern the proxy shipped. rel_tol=1e-9 (equal to 9 significant
digits) still fails on any real cost discrepancy while tolerating
representation drift. Addresses the Greptile P2 on this PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): widen the duplicate-settle window to 30s for real DataDog
Against the local sink one poll interval (5s) after the first hit was
enough to catch a same-call duplicate, because both events arrived in the
same flush batch. Against real DataDog, ingestion jitter can make one
call's two events searchable tens of seconds apart, so a 5s settle could
let the LIT-4447 duplicate slip past the exactly-one assertion. The reader
now keeps re-reading for DD_SETTLE_SECONDS (default 30s, env-overridable
via E2E_DD_SETTLE_SECONDS) after the first event appears, returning early
only when a duplicate is already visible - more waiting cannot clear it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(e2e): point UI tests at dashboard service; register complexity router
Stage gateway 404s /ui; the Next.js dashboard is litellm-ui:3000. Drive
playwright against E2E_UI_BASE_URL and wait on login placeholders after
client render. Register complexity-smart-router via /model/new when the
proxy does not already list it so stage matches compose config
* docs(e2e): clarify E2E_UI_BASE_URL should be ALB when ingress splits UI
* docs(e2e): prefer single path-routing host for control plane and UI
CONTROL_PLANE and UI already default to PROXY_BASE_URL; clarify that
stage should set one ALB host rather than three endpoints
* fix(e2e): always capture complexity router model_id for teardown
Split /model/new from the data-plane wait so a propagation timeout still
deletes the control-plane registration (greptile orphan-model concern)
* fix(e2e): click exact Login button so SSO control is not matched
Playwright strict mode matched both Login and Login with SSO
* fix(router): score complexity by difficulty not request length
The LLM classifier prompt treated short wording as SIMPLE, so probes like
"Is P equal to NP?" stayed on the SIMPLE backend even though the classifier
ran. Judge intellectual difficulty so short hard questions route higher
* fix(e2e): open key edit via Key ID and wait for team models
Key Alias text is not the row open control on the virtual keys table;
KeyInfoView opens from the Key ID button in that row. Also wait for a
real team model in the edit Models dropdown so we do not race the async
availableModels fetch that only has All Team Models on first paint
* fix(e2e): keep settled DD events on empty search; bump mcp for OSV
Do not let a transient empty DataDog search wipe events already seen in
the settle window (Greptile P1). Make the logs-search from window
env-overridable via E2E_DD_SEARCH_FROM (Greptile P2). Prefer the mono
Key ID button when opening key edit. Bump mcp 1.26.0 -> 1.28.1 so OSV
clears the three high GHSA findings on the staging PR
* revert: drop mcp lock bump from e2e staging PR
OSV mcp upgrade is unrelated to the e2e fixes; leave the dep pin alone
---------
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The rate-limited batch spend test snapshotted unattributed rows via the
unpaginated /spend/logs whole-table read, which grows with the environment
(58MB on stage) and OOMKilled the e2e runner at its 512Mi limit on every
scheduled run. Gateway.spend_logs_window pages /spend/logs/v2 over an
explicit date window instead, and SpendLogsParams now rejects a filterless
read so the whole-table call cannot come back
* fix(e2e): wire batch provider secrets for docker and k8s
Point batch deployments at the credential field names and os.environ refs
the gateway actually resolves from process env (compose .env or EKS secret
mounts). Missing secrets skip instead of failing red so a red run means a
product bug. Mirror S3 bucket env aliases in docker-compose for provider_fallback
* fix(e2e): drop batch provider_env unit tests
The batches suite is live e2e only; no monkeypatch or unit-level tests
* fix: batch credentials, provider list, and team db lookup
Keep object-storage fields through CredentialLiteLLMParams and resolve
os.environ/ refs when reading deployment credentials so Vertex/Bedrock
batch file uploads see bucket and AWS keys from K8s/docker env
Skip managed batch list when the request is provider-scoped so
/{provider}/v1/batches list works instead of 500
Force DB on check_db_only team lookups and stop masking non-404 errors
as "team doesn't exist"
Drop e2e runner-side skip helpers; hard-fail on missing gateway secrets
* fix: tag reseed, team window spend, and remaining e2e flakes
Reseed spend:tag counters from LiteLLM_TagTable so cold redis still
enforces after the spend writer flushes
When applying post-call cost to team multi-window counters, load the
team from the DB if it is missing from the management cache so window
spend is not dropped on cache misses
Harden cold-counter reseed e2e (namespace-aware keys, burst success,
poll). Give tag budget more headroom. Retry /key/update on redis DNS
blips. Ensure NLTK punkt_tab is present for pipecat realtime audio
* revert: drop product code changes; e2e-only scope
Reverts all litellm/ and unit-test product edits. This branch is limited
to tests/e2e per contributor instruction
* fix(e2e): harden batch list and team member setup races
provider_fallback list falls back when managed batches reject provider
filtering. Team create waits for /team/info and member_add retries on
transient team-not-found so split control-plane lag does not red the suite
* fix(e2e): remove .env.example
Leave local .env and docker-compose env wiring as the secret source
* fix(e2e): wire files_settings and faster budget rescheduler for compose
OpenAI/Azure batch file uploads need files_settings; budget reset e2e needs a
short rescheduler window. Drop unsupported bedrock-encoded create_batch cells,
tolerate bedrock file.bytes=0, and surface team-info wait failures instead of
hanging silently
* chore(e2e): strip verbose comments from batch capabilities
* fix(e2e): assert managed list fallback before provider_fallback skip
When provider-scoped list is rejected, still fetch the unfiltered list and
check the envelope. Only skip membership when the id is a raw
provider_fallback batch that managed list cannot index
azure batch used azure/gpt-4.1-mini-batch; gpt-4.1-mini is deprecating (2026-11-04)
and can no longer be deployed, so point it at gpt-5.4-mini (Global Batch) and bump
the api_version to 2025-04-01-preview. Requires an Azure Global Batch deployment
named gpt-5.4-mini-batch plus AZURE_API_BASE/AZURE_API_KEY on the proxy.
xai/grok-4-1-fast-non-reasoning is deprecated (2026-05-15); update the commented
xai realtime provider and the coverage-matrix doc to xai/grok-4-1-fast.
* fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects
spend_tracking/spend_e2e_client.py imported SpendTagsResponse and
TagSpend from models, but neither was ever defined, so importing the
client raised ImportError and pytest aborted collection for the whole
e2e session. The tag-spend tests had never run.
Model /spend/tags as it actually answers: a bare array of per-tag
aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the
existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field
that also wouldn't match the array shape; it now reads .root, matching
how spend_logs consumes its RootModel.
* test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction
Adds regression nets and gap-surfacing tests:
A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the
DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document
that reasoning_effort='none' and thinking type='disabled' are silently dropped
(LIT-3686 / GH #27453)
A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py):
parametrized regression net asserting real completion content, not just a 200,
across the configured providers for /chat/completions and /responses (GH #28991)
A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is
honored and prompt-cache read tokens grow on a repeated cacheable prefix
A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call
rate limiter runs, then asserts no unattributed spend row is left behind by the
internal input-file retrieval (LIT-3266)
A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct
key_alias and asserts each alias gets its own labeled series on /metrics
A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict)
regression proving eviction must not close an httpx client still held by an
in-flight caller (LIT-3221 / GH #13034)
Extends tests/e2e/models.py with the typed request and response fields these
tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage
fields, spend-log api_key)
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(e2e): drop unused litellm-regression-tests submodule
The e2e suite migrated the regression cases into this repo; nothing
imports the submodule at runtime (only a provenance comment references
it), so the .gitmodules entry and gitlink pointing at a personal repo
would just make upstream CI init a submodule it never uses. Remove both
to keep the change test-only.
* test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage
The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client
close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread
cleanup vs shared client teardown) and belongs in its own PR, not this e2e
coverage PR, so revert the file to its base state.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* fix: p0 issues, added types and shared functions for each test suite
* style: carry clearer status_code comparison into renamed e2e dir
* refactor: migrate to gateway client
* fix: add new tests, split gateway
* test(e2e): add live batches suite across providers and routing scenarios
* test(batches): cover real cost tracking on completed batch retrieve
* test(e2e): assert managed vs raw file and batch id shapes per routing scenario
* test(e2e): assert full response shape of each batches and files endpoint
* test(e2e): only accept transitional statuses for a freshly created batch
* test(prompt-factory): make test_convert_url deterministic with a data URL
picsum.photos is down (HTTP 522), so test_convert_url failed on every
run. Swap the live external image for an inline data: URL and assert the
round-trip through convert_url_to_base64 genuinely.
A data URL is already inline base64 image data, so convert_url_to_base64
now short-circuits it instead of attempting an impossible HTTP fetch;
add a regression for that branch in the mapped image_handling test
* fix: pass through async image data urls
* fix(image-handling): short-circuit data URLs in async path too
Bugbot flagged that convert_url_to_base64 returns data: base64 URLs
unchanged but async_convert_url_to_base64 still tried to fetch them,
so async OCR flows (Bedrock, Azure) would reject inline images the sync
path accepts. Add the same guard to the async function and a regression
test that asserts the async path returns the data URL without touching
the HTTP client
* Fix: openai batches lifecycle
* Fix: add e2e azure openai tests
* Fix e2e for vertex ai
* Add all models for testing
* test(managed-files): assert idempotent upsert in store_unified_file_id
store_unified_file_id switched from create to upsert to avoid
UniqueViolationError when re-storing the same unified_file_id (e.g.
batch output files stored before metadata is available). Update the
unit test to assert the upsert call and its create payload instead of
the removed create call.
* test(batches): reconcile vertex_ai native batch-id comment with fallback guard
* fix(test-config): keep rust-ocr models in model_list by moving files_settings after it
* fix(test-config): move batch models after OCR block to keep merge with internal_staging clean
* fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: ruff format transformation.py and endpoints.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition
* test(vertex-ai/batches): align completion_window assertion to 24h
* fix: update managed file metadata on upsert
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
