mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-26 01:12:21 +00:00
11dbe70092
Some checks failed
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
`@pytest.mark.covers("cell.id")` is a registry key, not a description: it cannot answer "which tests drive /v1/responses on Anthropic". This adds the declared half of the per-test metadata, on top of the recorded step log
`@meta(Subject(...))` from tests/e2e/e2e_metadata.py takes one frozen dataclass whose fields are closed enums (domain, route, providers, capabilities, mode) plus free-string models, so a typo is a basedpyright error at the call site rather than a property that silently never appears. providers, models and capabilities are tuples because one test node often drives several (the claude_code matrix runs haiku, sonnet and opus in one body), with no positional pairing between them. Each is deduped and sorted at declaration so committed run artifacts diff cleanly, and anything but a tuple is refused at import, so models=("gpt-5.5") is a collection error naming the file instead of one model per character
Serialization is one pass over dataclasses.asdict: each scalar is one <property> under its field name, each plural value a repeated property under its singular name (provider, model, capability). Empty fields emit nothing, and the fixed package/covers/source prefix stays byte-identical, with the declared fields appended behind it. `covers` is untouched: the marker is separate because a dataclass passed to covers would be dropped silently by dedupe_covers and hard-fail collection in tests/integration/conftest.py, and @meta goes below @covers so every source deep link keeps its line
Provider mirrors litellm's LlmProviders values instead of importing them, because tests/e2e is shipped to the runner image on its own and a module-scope `from litellm...` would make the package a collection-time dependency. TestProviderMirrorsLitellm fails on drift wherever litellm is importable
tests/e2e/quota_management/ (29 files) is annotated as the pilot, and tests that drive more than one provider or model declare all of them. Every field is optional until the backfill of the rest of the suite lands
Its harness tests sit beside the step log's in tests/code_coverage_tests/test_e2e_metadata.py and test_e2e_junit_report.py, since tests/e2e holds only tests that drive a live proxy
17 lines
1.9 KiB
INI
17 lines
1.9 KiB
INI
[pytest]
|
|
# Config when any e2e suite under tests/e2e/ is run directly, e.g.
|
|
# uv run pytest tests/e2e/quota_management/spend_tracking/ -v
|
|
# The e2e marker is also registered in conftest.py for runs rooted elsewhere.
|
|
addopts = --strict-markers --strict-config --reruns 1 --only-rerun "kind='network'" --only-rerun "status_code=5[0-9][0-9]"
|
|
markers =
|
|
e2e: live test that requires a running proxy and real provider keys
|
|
meta: typed e2e_metadata.Subject describing what this test drives (domain/route/providers/models/capabilities/mode); attach it with @meta(Subject(...)), never as a bare pytest.mark
|
|
replayable: edge-wired test whose provider traffic replays from a fixture bundle, so it makes zero provider calls in replay mode; the record/replay CI lane selects it with -m replayable
|
|
load: heavy throughput/load test; collected last so it never perturbs latency-sensitive suites
|
|
weekly: real-provider anomaly load test that spends real money; deselected unless E2E_WEEKLY_ANOMALY is set
|
|
managed_files: needs a proxy running with require_managed_files enabled; deselected unless E2E_MANAGED_FILES_STACK is set
|
|
prompt_caching_stack: needs a proxy running with router_settings.optional_pre_call_checks including prompt_caching; deselected unless E2E_PROMPT_CACHING_STACK is set
|
|
cli_determinism: drives the real claude CLI for several seconds; deselected unless E2E_CLI_DETERMINISM is set
|
|
redis_chaos: load test that pauses the proxy's Redis outright mid-run; needs a proxy booted from gateway/redis_chaos_ci_config.yml on the same host, and is deselected unless E2E_REDIS_CHAOS is set
|
|
mcp_oauth_live: real Linear OAuth consent via a captured browser session; deselected unless E2E_MCP_OAUTH_LIVE is set
|
|
provider_edge_host: routes provider traffic through the pytest host's edge in every fixture mode, so the gateway must reach the pytest host; deselected unless E2E_PROVIDER_EDGE_HOST_REACHABLE is set
|