mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-26 01:12:21 +00:00
ee5efaed7b
A test's JUnit report says what happened to it, never what it was about.
`@pytest.mark.covers("cell.id")` is a registry key, not a description: it
cannot answer "which tests drive /v1/responses on Anthropic", and nothing
in the report says where a failing test actually died.
Two halves, deliberately separated, both riding out as JUnit <property>
entries the downstream emitter already knows how to read.
DECLARED - `@meta(Subject(...))` from the new tests/e2e/e2e_metadata.py.
One frozen dataclass, every field a closed enum (domain, route, provider,
model, capabilities, mode), so a typo is a basedpyright error at the call
site rather than a property that silently never appears. Serialization is
one pass over `dataclasses.asdict`, so a new scalar field needs no
serializer edit; `capabilities` is deduped and sorted at declaration so
committed run artifacts diff cleanly whatever order a test spelled it in.
Empty fields emit nothing - the suite does not pad every testcase with
five empty entries.
RECORDED - `@step("POST /chat/completions")` on harness helpers, never on
tests. Each call appends its label to the running test's user_properties
in call order, so the list IS the test's user story and cannot drift from
what the test did. The label is recorded BEFORE the wrapped call, so a
helper that raises still leaves its own label last: a failing test's last
step is where it died. Consecutive duplicates collapse and the log caps at
50, so a poll loop is one beat of the story rather than fifty.
Steps cannot be attached where the other properties are -
`pytest_collection_modifyitems` runs before any test body, so the recorder
is empty there. They attach from the existing `pytest_runtest_makereport`
wrapper on the call phase, which is what puts them on failures too, and an
autouse fixture empties the log at setup. The attach drops any prior step
entries first, because the suite runs `--reruns 1` and a retry would
otherwise stack a second copy of the story behind the first.
`covers` is untouched: the marker is separate because a dataclass passed
to `covers` would be dropped silently by `dedupe_covers` and would hard-
fail collection in tests/integration/conftest.py. The fixed
package/covers/source prefix stays byte-identical, and `@meta` goes BELOW
`@covers` so `Item.location` still anchors at the first decorator and
every `source` deep link keeps pointing where it pointed.
`Provider` mirrors litellm's `LlmProviders` values instead of importing
them, so nothing here - the module or its call sites - imports litellm.
tests/e2e is a black-box HTTP suite that is copied to the runner image on
its own, so a `from litellm...` at the top of a test module would make the
package a COLLECTION-time dependency: where it is absent, every test in
the suite errors out before running rather than importing slowly. The
mirror cannot drift silently - `TestProviderMirrorsLitellm` asserts every
value is a real `LlmProviders` value wherever litellm is importable, and
skips where it is not, which is the property it is guarding.
A declared `model` names the constant the test drives, never a copy of its
value: `CHEAP_ANTHROPIC_MODEL` and `CHEAP_OPENAI_MODEL` are env-overridable
(`E2E_CHEAP_ANTHROPIC_MODEL`, `E2E_CHEAP_OPENAI_MODEL`), so a hardcoded
default would have reported a model the run never touched. The same holds
for a file's own `BACKEND`/`MODEL` constant, where the copy was merely
waiting to drift.
Pilot: tests/e2e/quota_management, all 85 tests annotated and its three
clients plus cost_rows @step-decorated, to prove the API against real
tests rather than a toy. The rest of the suite is a later backfill.
Verified: 42 harness unit tests in test_junit_properties.py (19 new), 550
harness unit tests green, basedpyright over tests/e2e at the same 29
pre-existing errors as origin/main (all in the untouched mcp/
oauth_chat_client.py), ruff clean, the coverage-registry collector
byte-identical before and after (473/586), and the test-quality gate OK
against origin/main. Collection was also run with the litellm package
blocked at the import hook: 1339/1350 collected either way, the one error
being the pre-existing missing `httpx2` in mcp/. The live e2e tests need a
deployed proxy and real provider keys and were not run.
17 lines
1.9 KiB
INI
17 lines
1.9 KiB
INI
[pytest]
|
|
# Config when any e2e suite under tests/e2e/ is run directly, e.g.
|
|
# uv run pytest tests/e2e/quota_management/spend_tracking/ -v
|
|
# The e2e marker is also registered in conftest.py for runs rooted elsewhere.
|
|
addopts = --strict-markers --strict-config --reruns 1 --only-rerun "kind='network'" --only-rerun "status_code=5[0-9][0-9]"
|
|
markers =
|
|
e2e: live test that requires a running proxy and real provider keys
|
|
meta: typed e2e_metadata.Subject describing what this test drives (domain/route/provider/model/capabilities/mode); attach it with @meta(Subject(...)), never as a bare pytest.mark
|
|
replayable: edge-wired test whose provider traffic replays from a fixture bundle, so it makes zero provider calls in replay mode; the record/replay CI lane selects it with -m replayable
|
|
load: heavy throughput/load test; collected last so it never perturbs latency-sensitive suites
|
|
weekly: real-provider anomaly load test that spends real money; deselected unless E2E_WEEKLY_ANOMALY is set
|
|
managed_files: needs a proxy running with require_managed_files enabled; deselected unless E2E_MANAGED_FILES_STACK is set
|
|
prompt_caching_stack: needs a proxy running with router_settings.optional_pre_call_checks including prompt_caching; deselected unless E2E_PROMPT_CACHING_STACK is set
|
|
cli_determinism: drives the real claude CLI for several seconds; deselected unless E2E_CLI_DETERMINISM is set
|
|
redis_chaos: load test that pauses the proxy's Redis outright mid-run; needs a proxy booted from gateway/redis_chaos_ci_config.yml on the same host, and is deselected unless E2E_REDIS_CHAOS is set
|
|
mcp_oauth_live: real Linear OAuth consent via a captured browser session; deselected unless E2E_MCP_OAUTH_LIVE is set
|
|
provider_edge_host: routes provider traffic through the pytest host's edge in every fixture mode, so the gateway must reach the pytest host; deselected unless E2E_PROVIDER_EDGE_HOST_REACHABLE is set
|