scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-26 01:12:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 99

Let a test declare several providers and models, and keep steps on setup errors
Some checks are pending
LiteLLM Rust / rust-lint (push) Waiting to run
Details
LiteLLM Rust / rust-test (push) Waiting to run
Details
LiteLLM Rust / rust-wheel (push) Waiting to run
Details
Terraform Provider / gofmt, vet, build, test (push) Waiting to run
Details
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Waiting to run
Details

Browse source 
Subject.provider and Subject.model become the tuples providers and models, handled exactly like capabilities: empty by default, deduped and sorted at declaration, and written to JUnit as a repeated property under the singular name (provider, model). One test node often drives several, such as the claude_code matrix running haiku, sonnet and opus in one body, and the two lists are independent sets with no positional pairing. Every plural field now refuses anything but a tuple, so models=("gpt-5.5") is a collection error naming the file instead of one model per character

All 27 pilot files move to the plural form. The fallback, per-model budget, shared-key and key-attribution tests now declare every provider and model they drive, the health-check test declares the Gemini model it probes, and the cost-breakdown test names the backend it actually registers

Steps are now attached after setup as well as after call, so a test that errors in a fixture keeps the steps recorded before the crash. The step log is emptied from the pytest_runtest_setup hook instead of an autouse fixture, because a fixture runs after wider-scoped ones and a setup error would otherwise inherit steps left by the previous module's finalizer. Teardown deliberately does not attach, so a failing test's last step stays the one it died on

test_junit_report.py runs real pytest with --junitxml against this conftest, in-process and under -n 2, and asserts on the parsed XML for passing, failing, setup-error and rerun cases plus the repeated provider, model and capability properties. Decorative comments in the files this branch adds are removed
This commit is contained in:
ryan-crabbe-berri 2026-09-21 10:54:13 -07:00
parent ee5efaed7b
commit f484cc2366
34 changed files with 718 additions and 254 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
tests/e2e/AGENTS.md
View file

@ -138,19 +138,19 @@ Separate from the coverage registry and additive to it: `@meta(Subject(...))` fr
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=CHEAP_ANTHROPIC_MODEL,
providers=(Provider.ANTHROPIC,),
models=(CHEAP_ANTHROPIC_MODEL,),
mode=Mode.NONSTREAM,
)
)
def test_bare_key_blocks_over_its_own_budget(...) -> None: ...
```
Every field is optional today (the backfill of the rest of the suite is a later PR) and every field is a closed enum, so a typo is a basedpyright error at the call site rather than a property that silently never appears. `capabilities` is a tuple even with one member, and it is deduped and sorted at declaration so the committed run artifacts diff cleanly. `Subject` is serialized with `dataclasses.asdict`, so a new scalar field needs no serializer edit; empty fields emit no `<property>` at all. A declared `model` names the constant the test drives (`CHEAP_ANTHROPIC_MODEL`, the file's own `BACKEND`), never a copy of its value, so the property cannot claim one model while an env override runs another. `e2e_metadata` is stdlib-only and so are its call sites: `Provider` mirrors litellm's `LlmProviders` values instead of importing them, because tests/e2e is shipped to the runner image on its own and a `from litellm...` at module scope would make the litellm package a hard dependency of COLLECTING the suite. `TestProviderMirrorsLitellm` in `test_junit_properties.py` fails on drift wherever litellm is importable and skips where it is not, so adding a provider is one line in `e2e_metadata`
Every field is optional today (the backfill of the rest of the suite is a later PR) and every field is a closed enum, so a typo is a basedpyright error at the call site rather than a property that silently never appears. `providers`, `models` and `capabilities` are tuples even with one member, because one test node routinely drives several: the claude_code matrix runs haiku, sonnet and opus in a single body, and a spend test calls two providers on one key. Declare every provider and every model the test drives, fallbacks included. The three are independent sets with no positional pairing between them (one provider x three models is the common case), and each is deduped and sorted at declaration so the committed run artifacts diff cleanly. `models=("gpt-5.5")` is a str and not a tuple, so anything but a tuple raises a `TypeError` where the decorator runs and shows up as a collection error naming the file. `Subject` is serialized with `dataclasses.asdict`, so a new scalar field needs no serializer edit; empty fields emit no `<property>` at all. A declared model names the constant the test drives (`CHEAP_ANTHROPIC_MODEL`, the file's own `BACKEND`), never a copy of its value, so the property cannot claim one model while an env override runs another. `e2e_metadata` is stdlib-only and so are its call sites: `Provider` mirrors litellm's `LlmProviders` values instead of importing them, because tests/e2e is shipped to the runner image on its own and a `from litellm...` at module scope would make the litellm package a hard dependency of COLLECTING the suite. `TestProviderMirrorsLitellm` in `test_junit_properties.py` fails on drift wherever litellm is importable and skips where it is not, so adding a provider is one line in `e2e_metadata`
The other half is recorded, not declared. `@step("POST /chat/completions")` goes on HARNESS helpers - client methods, `ResourceManager.key`, poll loops - never on a test, and appends its label to the running test's `user_properties` in call order. The list IS the test's user story, and because the label is recorded BEFORE the wrapped call, a failing test's LAST step is where it died. Consecutive duplicates collapse and the log caps at 50 entries, so a poll loop is one beat of the story rather than fifty. Nothing about steps is hand-written: the call sequence cannot drift from what the test actually did
The other half is recorded, not declared. `@step("POST /chat/completions")` goes on HARNESS helpers - client methods, `ResourceManager.key`, poll loops - never on a test, and appends its label to the running test's `user_properties` in call order. The list IS the test's user story, and because the label is recorded BEFORE the wrapped call, a failing test's LAST step is where it died. Consecutive duplicates collapse and the log caps at 50 entries, so a poll loop is one beat of the story rather than fifty. Nothing about steps is hand-written: the call sequence cannot drift from what the test actually did. The log is emptied first thing in every test's setup phase and attached after setup and again after call, so a test that errors in a fixture keeps the steps recorded before the crash. Teardown steps are left out on purpose: they are cleanup, and listing them would put a finalizer's step after the one a failing test died on
Both halves ride out as JUnit `<property>` entries (`junit_properties.py`), repeated rather than delimiter-joined, since a free-text label has no separator that can be reserved
Both halves ride out as JUnit `<property>` entries (`junit_properties.py`). Every plural value is a repeated property under its SINGULAR name (`provider`, `model`, `capability`, `step`) rather than one delimiter-joined value, since a free-text label has no separator that can be reserved. The results JSON downstream regroups them under the plural key, so `providers` and `models` are arrays there, `[]` when empty. `test_junit_report.py` runs real pytest with `--junitxml`, in-process and under `-n 2`, and pins what reaches the XML
## Coverage registry

43
tests/e2e/conftest.py
View file

@ -113,7 +113,7 @@ def pytest_configure(config: pytest.Config) -> None:
config.addinivalue_line(
"markers",
"meta(subject): typed e2e_metadata.Subject describing what this test drives"
" (domain/route/provider/model/capabilities/mode); attach it with @meta(Subject(...))",
" (domain/route/providers/models/capabilities/mode); attach it with @meta(Subject(...))",
)
config.addinivalue_line(
"markers",
@ -238,7 +238,14 @@ def pytest_runtest_setup(item: pytest.Item) -> None:
"""Hard-fail `e2e`-marked tests unless a proxy answers its liveness probe.
Unmarked tests (unit coverage of the harness) don't touch the proxy, so they
run even when none is up. Never skip for a missing proxy. Replay mode needs
the proxy too: only provider-bound traffic replays from the bundle."""
the proxy too: only provider-bound traffic replays from the bundle.
Also empties the step log, so the story a test tells is its own. It happens
here, first in the setup phase, rather than in a fixture: a fixture only runs
once every wider-scoped fixture ahead of it has been set up, so a step a
module-scoped finalizer recorded after the previous test would still be in
the log when this test's setup dies early, and would be reported as its own."""
STEPS.reset()
LIVE_PROVIDER_REQUIRED.set(item.get_closest_marker("provider_live") is not None)
if item.get_closest_marker("e2e") is None or item.get_closest_marker("migration_startup") is not None:
return
@ -269,9 +276,19 @@ def pytest_runtest_makereport(
The steps cannot ride along with the other properties in
`pytest_collection_modifyitems`: that hook runs before any test body has, so
the recorder is empty there. They are attached on every call-phase outcome,
failures included -- a failing test's last step is where it died, which is the
whole reason the field exists.
the recorder is empty there. They are attached after setup and again after
call, on every outcome -- a failing test's last step is where it died, which
is the whole reason the field exists. Setup has to attach too because a test
whose fixture raises never reaches the call phase, and setup is where an e2e
test most often dies (proxy not ready, key creation failing). The second
attach replaces the first, so nothing is doubled. JUnit writes properties
from the teardown report, which pytest builds from `item.user_properties`
after both of these have run.
Teardown deliberately does not attach. Steps recorded by fixture finalizers
are cleanup, and appending them would put "delete virtual key" after the step
a failing test died on, which breaks the one guarantee the field makes. A
finalizer that raises is still reported by JUnit with its own traceback.
"""
report = yield
if item.get_closest_marker("mcp_oauth_live") is not None and call.excinfo is not None:
@ -283,6 +300,7 @@ def pytest_runtest_makereport(
report.user_properties = list(item.user_properties)
if report.when == "call":
item.stash[_CALL_PASSED] = report.passed
if report.when in ("setup", "call"):
attach_step_properties(item)
return report
@ -320,21 +338,6 @@ def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None:
)
@pytest.fixture(autouse=True)
def _record_steps() -> Iterator[None]: # pyright: ignore[reportUnusedFunction] # autouse: nothing names it
"""Empty the step log before each test, so the story a test tells is its own.
Autouse and unconditional: the recorder is written by @step-decorated harness
helpers, which a test reaches through fixtures as readily as through its own
body, so setup-phase steps have to be kept too. The log is read after the call
phase by `pytest_runtest_makereport`; a test that dies partway keeps the
partial list, which is the point -- its last element is where it died.
"""
STEPS.reset()
yield
STEPS.reset()
@pytest.fixture(scope="session")
def proxy() -> ProxyClient:
"""The shared ProxyClient every suite's client is built from. Suite `client`

145
tests/e2e/e2e_metadata.py
View file

@ -3,8 +3,8 @@
Two halves, deliberately separated.
The DECLARED half is `Subject`: one frozen dataclass passed as the single
positional argument of `@meta(...)`. Every field is a closed enum (or a free
string for `model`), so a typo is a basedpyright error at the call site rather
positional argument of `@meta(...)`. Every field is a closed enum (or free
strings for `models`), so a typo is a basedpyright error at the call site rather
than a silently dropped property. `dataclasses.asdict()` turns the whole thing
into <property> pairs with no per-field plumbing -- adding a scalar field later
needs zero serializer changes.
@ -29,7 +29,7 @@ from __future__ import annotations
import threading
from collections.abc import Callable
from dataclasses import dataclass
from dataclasses import asdict, dataclass
from enum import Enum
from functools import wraps
from typing import Final, ParamSpec, TypeVar, cast
@ -79,10 +79,12 @@ class Route(str, Enum):
images_generations + images_edits -> IMAGES, audio_speech +
audio_transcriptions -> AUDIO, bedrock_native + google_native ->
PASSTHROUGH. Those splits are wire detail, not a customer-facing surface,
and `model` + `capabilities` already carry them.
and `models` + `capabilities` already carry them.
The last four are ops surfaces: logging/, load/, other/ and ui/ have no LLM
route of their own and would otherwise have to lie.
"""
# Core: each is the subject of many e2e tests.
CHAT_COMPLETIONS = "chat_completions"
MESSAGES = "messages"
RESPONSES = "responses"
@ -97,8 +99,6 @@ class Route(str, Enum):
TEAM_MANAGEMENT = "team_management"
SPEND_REPORTING = "spend_reporting"
MODEL_MANAGEMENT = "model_management"
# Long tail: real, but one or two test files each.
IMAGES = "images"
AUDIO = "audio"
MODERATIONS = "moderations"
@ -109,9 +109,6 @@ class Route(str, Enum):
A2A = "a2a"
USER_MANAGEMENT = "user_management"
BUDGET_MANAGEMENT = "budget_management"
# Ops surfaces. logging/, load/, other/ and ui/ have no LLM route of their
# own and would otherwise have to lie.
HEALTH = "health"
METRICS = "metrics"
PROXY_CONFIG = "proxy_config"
@ -212,6 +209,52 @@ class Mode(str, Enum):
WEBSOCKET = "websocket"
_M = TypeVar("_M")
def _scalar(value: object) -> str:
"""`str(member)` on a (str, Enum) gives 'Route.RESPONSES', not 'responses'
-- StrEnum would not, but it is 3.11+ and this repo floors at 3.10. So the
value is read explicitly, once, for every enum field."""
if isinstance(value, Enum):
return str(value.value) # pyright: ignore[reportAny] # Enum.value is Any for every enum
return str(value)
def _members(value: object) -> tuple[object, ...] | None:
"""The elements of a plural field, or None for anything that is not a tuple.
Both callers hold the value as a plain object: `_canonical` because a call
site can pass anything at runtime, the serializer because `asdict` hands the
tuple back inside an untyped dict. The elements are re-declared as plain
objects here and converted by `_scalar` like any other value.
"""
return cast("tuple[object, ...]", value) if isinstance(value, tuple) else None
def _canonical(name: str, value: object, member_type: type[_M]) -> tuple[_M, ...]:
"""A plural field's members: validated, deduped, and sorted by the value
they serialize to.
`models=("gpt-5.5")` is a str, not a tuple, and iterating it would declare
one model per character. Anything that is not a tuple is refused here, which
runs where the decorator does: at import, so pytest reports a collection
error naming the file instead of shipping garbage properties. An empty
string member is dropped rather than refused, because `models` is fed from
env-overridable constants and a blank override must not break collection.
"""
members = _members(value)
if members is None:
raise TypeError(
f"Subject.{name} must be a tuple, got {type(value).__name__}: {value!r}."
f" A one-member tuple needs its trailing comma: {name}=(x,), not {name}=(x)"
)
typed = tuple(member for member in members if isinstance(member, member_type))
if len(typed) != len(members):
raise TypeError(f"Subject.{name} takes {member_type.__name__} members, got {value!r}")
return tuple(sorted(frozenset(member for member in typed if _scalar(member)), key=_scalar))
@dataclass(frozen=True, slots=True)
class Subject:
"""What a test is about.
@ -220,24 +263,29 @@ class Subject:
backfill of the existing ~908 tests comes later. Named `Subject` rather than
`TestMeta` because pytest tries to collect any imported class named `Test*`
and would warn in every one of the ~570 modules that import it.
`providers`, `models` and `capabilities` are plural because one test node
routinely drives several: the claude_code matrix runs haiku, sonnet and opus
in a single body, and a spend test calls two providers on one key. Each is
an independent set. No positional pairing is implied between `providers` and
`models` (one provider x three models is the common case), and none could
survive anyway, since each tuple is deduped and sorted on its own.
"""
domain: Domain | None = None
route: Route | None = None
provider: Provider | None = None
model: str | None = None
providers: tuple[Provider, ...] = ()
models: tuple[str, ...] = ()
capabilities: tuple[Capability, ...] = ()
mode: Mode | None = None
def __post_init__(self) -> None:
"""Canonicalize capabilities at declaration: deduped and sorted by
value, so the committed run files diff cleanly however a test spelled
the tuple, and the serializer stays field-agnostic."""
object.__setattr__(
self,
"capabilities",
tuple(sorted(dict.fromkeys(self.capabilities), key=lambda c: c.value)),
)
"""Canonicalize every plural field at declaration, so the committed run
files diff cleanly however a test spelled the tuple, and the serializer
stays field-agnostic."""
object.__setattr__(self, "providers", _canonical("providers", self.providers, Provider))
object.__setattr__(self, "models", _canonical("models", self.models, str))
object.__setattr__(self, "capabilities", _canonical("capabilities", self.capabilities, Capability))
def meta(subject: Subject) -> pytest.MarkDecorator:
@ -255,13 +303,9 @@ def meta(subject: Subject) -> pytest.MarkDecorator:
return pytest.mark.meta(subject)
# --- The recorded half: steps -------------------------------------------------
_P = ParamSpec("_P")
_R = TypeVar("_R")
# A retrying helper (poll_cost_row) or a load test calling a decorated helper in
# a loop would otherwise emit thousands of <property> entries per testcase.
MAX_STEPS: Final = 50
MAX_STEP_CHARS: Final = 200
@ -280,16 +324,23 @@ class _StepRecorder:
self._steps: list[str] = []
def reset(self) -> None:
"""Called by an autouse fixture at setup, so each test starts empty."""
"""Called first thing in every test's setup phase, so each test starts
empty."""
with self._lock:
self._steps.clear()
def record(self, label: str) -> None:
"""Append `label`, unless it repeats the previous step or the log is full.
A retrying helper (poll_cost_row) or a load test calling a decorated
helper in a loop would otherwise emit thousands of <property> entries per
testcase: a consecutive repeat collapses, so a poll loop is one step in
the story rather than fifty, and the log stops growing at MAX_STEPS.
"""
cleaned = " ".join(label.split())[:MAX_STEP_CHARS]
if not cleaned:
return
with self._lock:
# A poll loop is one step in the story, not fifty.
if self._steps and self._steps[-1] == cleaned:
return
if len(self._steps) >= MAX_STEPS:
@ -324,33 +375,7 @@ def step(label: str) -> Callable[[Callable[_P, _R]], Callable[_P, _R]]:
return decorate
# --- Serialization: asdict() -> <property> pairs, no per-field plumbing -------
# The only field-specific knowledge in the module: which fields are plural, and
# what their repeated <property> is called. A new SCALAR field needs no edit.
_REPEATED: Final[dict[str, str]] = {"capabilities": "capability"}
def _scalar(value: object) -> str:
"""`str(member)` on a (str, Enum) gives 'Route.RESPONSES', not 'responses'
-- StrEnum would not, but it is 3.11+ and this repo floors at 3.10. So the
value is read explicitly, once, for every enum field."""
if isinstance(value, Enum):
# `Enum.value` is `Any` by construction, for every enum there has ever
# been; this is the one place in the module that reads it, and `str()`
# lands it back in the type system immediately.
return str(value.value) # pyright: ignore[reportAny] # Enum.value is Any for every enum
return str(value)
def _members(value: object) -> tuple[object, ...]:
"""The elements of a plural field, whatever `asdict` rebuilt it as.
`asdict` reconstructs a tuple field as a tuple of the same members, but hands
it back inside an untyped dict, so the elements are re-declared as plain
objects here and converted by `_scalar` like any other value.
"""
return cast("tuple[object, ...]", value) if isinstance(value, tuple) else ()
_REPEATED: Final[dict[str, str]] = {"providers": "provider", "models": "model", "capabilities": "capability"}
def _declared_subject(args: tuple[object, ...]) -> Subject | None:
@ -366,10 +391,13 @@ def _declared_subject(args: tuple[object, ...]) -> Subject | None:
def subject_properties(item: pytest.Item) -> tuple[tuple[str, str], ...]:
"""The declared half, in dataclass field order. Empty fields emit nothing;
the emitter is what guarantees every key exists in the JSON."""
from dataclasses import asdict # local: keeps module import trivially cheap
"""The declared half, in dataclass field order.
`_REPEATED` is the only field-specific knowledge here: which fields are
plural, and the SINGULAR name their repeated <property> goes out under. A new
scalar field needs no edit. Empty fields emit nothing; the emitter is what
guarantees every key exists in the JSON, with `providers` and `models` as
`[]` when nothing was declared."""
marker = item.get_closest_marker("meta")
if marker is None:
return ()
@ -381,12 +409,13 @@ def subject_properties(item: pytest.Item) -> tuple[tuple[str, str], ...]:
for name, value in fields.items():
repeated = _REPEATED.get(name)
if repeated is not None:
pairs.extend((repeated, _scalar(member)) for member in _members(value))
pairs.extend((repeated, _scalar(member)) for member in _members(value) or ())
elif value is not None and value != "":
pairs.append((name, _scalar(value)))
return tuple(pairs)
def step_properties() -> tuple[tuple[str, str], ...]:
"""The recorded half. Appended after the call phase, never at collection."""
"""The recorded half. Appended after the setup and call phases, never at
collection."""
return tuple(("step", label) for label in STEPS.taken())

13
tests/e2e/junit_properties.py
View file

@ -118,17 +118,18 @@ def attach_result_properties(item: pytest.Item) -> None:
def attach_step_properties(item: pytest.Item) -> None:
"""Attach the runtime-recorded steps after the call phase.
"""Attach the runtime-recorded steps; called after setup and after call.
Separate from `attach_result_properties` because it cannot share its home:
that one runs in `pytest_collection_modifyitems`, before any test body has
executed, so the recorder is necessarily empty there.
Any `step` entries already on the item are dropped first. The suite runs with
`--reruns 1`, so a flaky test's second attempt would otherwise append a second
copy of the story behind the first, and the report would read as one very long
test that did everything twice. Last attempt wins, which is the attempt whose
outcome JUnit records.
Any `step` entries already on the item are dropped first, which is what makes
the second call of a test safe: the story attached after setup is replaced by
the longer one attached after call. It also covers `--reruns 1`, where a flaky
test's second attempt would otherwise append a second copy of the story behind
the first, and the report would read as one very long test that did everything
twice. Last attempt wins, which is the attempt whose outcome JUnit records.
"""
item.user_properties[:] = [entry for entry in item.user_properties if entry[0] != "step"]
item.user_properties.extend(step_properties())

2
tests/e2e/pytest.ini
View file

@ -5,7 +5,7 @@
addopts = --strict-markers --strict-config --reruns 1 --only-rerun "kind='network'" --only-rerun "status_code=5[0-9][0-9]"
markers =
e2e: live test that requires a running proxy and real provider keys
meta: typed e2e_metadata.Subject describing what this test drives (domain/route/provider/model/capabilities/mode); attach it with @meta(Subject(...)), never as a bare pytest.mark
meta: typed e2e_metadata.Subject describing what this test drives (domain/route/providers/models/capabilities/mode); attach it with @meta(Subject(...)), never as a bare pytest.mark
replayable: edge-wired test whose provider traffic replays from a fixture bundle, so it makes zero provider calls in replay mode; the record/replay CI lane selects it with -m replayable
load: heavy throughput/load test; collected last so it never perturbs latency-sensitive suites
weekly: real-provider anomaly load test that spends real money; deselected unless E2E_WEEKLY_ANOMALY is set

36
tests/e2e/quota_management/budgets/test_budget_enforcement_e2e.py
View file

@ -61,8 +61,8 @@ class TestBudgetBlocksPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -77,8 +77,8 @@ class TestBudgetBlocksPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -102,8 +102,8 @@ class TestBudgetBlocksPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -145,8 +145,8 @@ class TestBudgetBlocksPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -166,8 +166,8 @@ class TestBudgetBlocksPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -189,8 +189,8 @@ class TestBudgetBlocksPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -225,8 +225,8 @@ class TestKeyBudgetBlocksAcrossKeyKinds:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -248,8 +248,8 @@ class TestKeyBudgetBlocksAcrossKeyKinds:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -269,8 +269,8 @@ class TestKeyBudgetBlocksAcrossKeyKinds:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/budgets/test_budget_fallback_e2e.py
View file

@ -25,8 +25,8 @@ FALLBACK_MODEL = "gpt-5.5"
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.MESSAGES,
provider=Provider.ANTHROPIC,
model=PRIMARY_MODEL,
providers=(Provider.ANTHROPIC, Provider.OPENAI),
models=(PRIMARY_MODEL, FALLBACK_MODEL),
mode=Mode.NONSTREAM,
)
)

20
tests/e2e/quota_management/budgets/test_budget_reset_advances_e2e.py
View file

@ -97,8 +97,8 @@ def test_key_with_budget_duration_schedules_reset_at_creation(client: BudgetClie
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -123,8 +123,8 @@ def test_key_spend_blocks_at_cap(client: BudgetClient, resources: ResourceManage
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -168,8 +168,8 @@ def test_key_budget_reset_at_advances_after_window(client: BudgetClient, resourc
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -221,8 +221,8 @@ def test_multi_window_key_resets_each_window_independently(client: BudgetClient,
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -263,8 +263,8 @@ def test_team_member_budget_reset_at_advances(client: BudgetClient, resources: R
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)

28
tests/e2e/quota_management/budgets/test_budget_reset_e2e.py
View file

@ -54,8 +54,8 @@ class TestBudgetResetPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -71,8 +71,8 @@ class TestBudgetResetPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -92,8 +92,8 @@ class TestBudgetResetPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -123,8 +123,8 @@ class TestBudgetResetPerLevel:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -150,8 +150,8 @@ class TestKeyBudgetResetAcrossKeyKinds:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -169,8 +169,8 @@ class TestKeyBudgetResetAcrossKeyKinds:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -188,8 +188,8 @@ class TestKeyBudgetResetAcrossKeyKinds:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)

16
tests/e2e/quota_management/budgets/test_model_access_group_budget_e2e.py
View file

@ -107,8 +107,8 @@ class TestModelAccessGroupBudget:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=BACKEND,
providers=(Provider.OPENAI,),
models=(BACKEND,),
mode=Mode.NONSTREAM,
)
)
@ -128,8 +128,8 @@ class TestModelAccessGroupBudget:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=BACKEND,
providers=(Provider.OPENAI,),
models=(BACKEND,),
mode=Mode.NONSTREAM,
)
)
@ -150,8 +150,8 @@ class TestModelAccessGroupBudget:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=BACKEND,
providers=(Provider.OPENAI,),
models=(BACKEND,),
mode=Mode.NONSTREAM,
)
)
@ -173,8 +173,8 @@ class TestModelAccessGroupBudget:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.BUDGET_MANAGEMENT,
provider=Provider.OPENAI,
model=BACKEND,
providers=(Provider.OPENAI,),
models=(BACKEND,),
)
)
def test_the_budget_read_reports_the_spend_drawn_against_the_pool(

8
tests/e2e/quota_management/budgets/test_model_max_budget_e2e.py
View file

@ -35,8 +35,8 @@ def _call(client: BudgetClient, key: str, model: str):
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=CAPPED_MODEL,
providers=(Provider.ANTHROPIC, Provider.GEMINI),
models=(CAPPED_MODEL, FREE_MODEL),
mode=Mode.NONSTREAM,
)
)
@ -75,8 +75,8 @@ def test_model_max_budget_isolates_per_model(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model=FREE_MODEL,
providers=(Provider.GEMINI,),
models=(FREE_MODEL,),
mode=Mode.NONSTREAM,
)
)

8
tests/e2e/quota_management/budgets/test_multi_window_budget_e2e.py
View file

@ -62,8 +62,8 @@ def _drive_to_block(client: BudgetClient, key: str) -> StreamingResponse:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=CHEAP_OPENAI_MODEL,
providers=(Provider.OPENAI,),
models=(CHEAP_OPENAI_MODEL,),
mode=Mode.NONSTREAM,
)
)
@ -104,8 +104,8 @@ def test_short_window_blocks_then_resets(client: BudgetClient, resources: Resour
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=CHEAP_OPENAI_MODEL,
providers=(Provider.OPENAI,),
models=(CHEAP_OPENAI_MODEL,),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/budgets/test_soft_budget_e2e.py
View file

@ -23,8 +23,8 @@ pytestmark = pytest.mark.e2e
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/budgets/test_spend_counter_reseed_e2e.py
View file

@ -149,8 +149,8 @@ def _accumulate(client: BudgetClient, key: str, count: int) -> None:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=MODEL,
providers=(Provider.ANTHROPIC,),
models=(MODEL,),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/budgets/test_tag_budget_e2e.py
View file

@ -39,8 +39,8 @@ def _tagged_call(client: BudgetClient, key: str, tag: str):
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)

8
tests/e2e/quota_management/budgets/test_team_member_budget_e2e.py
View file

@ -84,8 +84,8 @@ class TestTeamMemberBudget:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=MODEL,
providers=(Provider.ANTHROPIC,),
models=(MODEL,),
mode=Mode.NONSTREAM,
)
)
@ -112,8 +112,8 @@ class TestTeamMemberBudget:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=MODEL,
providers=(Provider.ANTHROPIC,),
models=(MODEL,),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/budgets/test_team_member_budget_isolation_e2e.py
View file

@ -93,8 +93,8 @@ class TestTeamMemberBudgetIsolation:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=MODEL,
providers=(Provider.OPENAI,),
models=(MODEL,),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/budgets/test_team_member_budget_reset_e2e.py
View file

@ -22,8 +22,8 @@ def _as_datetime(value: str) -> datetime:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)

8
tests/e2e/quota_management/budgets/test_team_multi_window_budget_e2e.py
View file

@ -57,8 +57,8 @@ def _drive_to_block(client: BudgetClient, key: str) -> StreamingResponse:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)
@ -99,8 +99,8 @@ def test_team_short_window_blocks_then_resets(client: BudgetClient, resources: R
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model="claude-haiku-4-5",
providers=(Provider.ANTHROPIC,),
models=("claude-haiku-4-5",),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/budgets/test_user_budget_across_keys_e2e.py
View file

@ -63,8 +63,8 @@ class TestUserBudgetAcrossKeys:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=MODEL,
providers=(Provider.OPENAI,),
models=(MODEL,),
mode=Mode.NONSTREAM,
)
)

8
tests/e2e/quota_management/ratelimit/test_dynamic_rate_limit_priority_e2e.py
View file

@ -162,8 +162,8 @@ class TestDynamicRateLimitPriority:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=BACKEND,
providers=(Provider.ANTHROPIC,),
models=(BACKEND,),
mode=Mode.NONSTREAM,
)
)
@ -213,8 +213,8 @@ class TestDynamicRateLimitPriority:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=BACKEND,
providers=(Provider.ANTHROPIC,),
models=(BACKEND,),
mode=Mode.NONSTREAM,
)
)

16
tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py
View file

@ -181,8 +181,8 @@ class TestKeyRateLimits:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=CHEAP_ANTHROPIC_MODEL,
providers=(Provider.ANTHROPIC,),
models=(CHEAP_ANTHROPIC_MODEL,),
mode=Mode.NONSTREAM,
)
)
@ -202,8 +202,8 @@ class TestKeyRateLimits:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=CHEAP_ANTHROPIC_MODEL,
providers=(Provider.ANTHROPIC,),
models=(CHEAP_ANTHROPIC_MODEL,),
mode=Mode.NONSTREAM,
)
)
@ -230,8 +230,8 @@ class TestKeyRateLimits:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=CHEAP_ANTHROPIC_MODEL,
providers=(Provider.ANTHROPIC,),
models=(CHEAP_ANTHROPIC_MODEL,),
mode=Mode.NONSTREAM,
)
)
@ -264,8 +264,8 @@ class TestKeyRateLimits:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=CHEAP_ANTHROPIC_MODEL,
providers=(Provider.ANTHROPIC,),
models=(CHEAP_ANTHROPIC_MODEL,),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/ratelimit/test_redis_backed_ratelimit_e2e.py
View file

@ -45,8 +45,8 @@ class TestRedisBackedRateLimit:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=BACKEND,
providers=(Provider.ANTHROPIC,),
models=(BACKEND,),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/ratelimit/test_redis_circuit_breaker_e2e.py
View file

@ -50,8 +50,8 @@ class TestRedisCircuitBreakerPath:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=BACKEND,
providers=(Provider.ANTHROPIC,),
models=(BACKEND,),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/ratelimit/test_tpm_excludes_cached_tokens_e2e.py
View file

@ -106,8 +106,8 @@ class TestTpmExcludesCachedTokens:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.ANTHROPIC,
model=ANTHROPIC_MODEL,
providers=(Provider.ANTHROPIC,),
models=(ANTHROPIC_MODEL,),
capabilities=(Capability.PROMPT_CACHING,),
mode=Mode.NONSTREAM,
)

16
tests/e2e/quota_management/spend_tracking/test_cache_cost_accounting_e2e.py
View file

@ -127,8 +127,8 @@ class TestCacheCostAccounting:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=CACHE_WRITE_BACKEND,
providers=(Provider.OPENAI,),
models=(CACHE_WRITE_BACKEND,),
capabilities=(Capability.PROMPT_CACHING,),
mode=Mode.NONSTREAM,
)
@ -167,8 +167,8 @@ class TestCacheCostAccounting:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=CACHE_WRITE_BACKEND,
providers=(Provider.OPENAI,),
models=(CACHE_READ_BACKEND,),
capabilities=(Capability.PROMPT_CACHING, Capability.REASONING),
mode=Mode.NONSTREAM,
)
@ -241,8 +241,8 @@ class TestCacheCostAccounting:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=CACHE_READ_BACKEND,
providers=(Provider.OPENAI,),
models=(CACHE_READ_BACKEND,),
capabilities=(Capability.PROMPT_CACHING,),
mode=Mode.STREAM,
)
@ -282,8 +282,8 @@ class TestCacheCostAccounting:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.MESSAGES,
provider=Provider.OPENAI,
model=BRIDGE_BACKEND,
providers=(Provider.OPENAI,),
models=(BRIDGE_BACKEND,),
capabilities=(Capability.PROMPT_CACHING,),
mode=Mode.NONSTREAM,
)

4
tests/e2e/quota_management/spend_tracking/test_cost_headers_e2e.py
View file

@ -65,8 +65,8 @@ class TestCostHeaders:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=BACKEND,
providers=(Provider.OPENAI,),
models=(BACKEND,),
mode=Mode.NONSTREAM,
)
)

13
tests/e2e/quota_management/spend_tracking/test_key_attribution_e2e.py
View file

@ -62,6 +62,7 @@ EMBED_MODEL: Final = "openai-text-embedding-3-small"
BATCH_MODEL: Final = "openai-gpt-4o-mini"
BATCH_BACKEND_MODEL: Final = "gpt-4o-mini"
BATCH_PROVIDER: Final = "openai"
DRIVEN_MODELS: Final = (CHAT_MODEL, MESSAGES_MODEL, RESPONSES_MODEL, EMBED_MODEL, BATCH_MODEL)
HEALTH_SERVICE_ACCOUNT: Final = "litellm-internal-health-check"
BATCH_TERMINAL_STATUSES: Final = frozenset({"completed", "failed", "cancelled", "expired"})
FAILED_BATCH_POLL_SECONDS: Final = 120.0
@ -286,6 +287,8 @@ class TestKeyAttribution:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.SPEND_REPORTING,
providers=(Provider.GEMINI, Provider.ANTHROPIC, Provider.OPENAI),
models=DRIVEN_MODELS,
)
)
def test_every_write_path_row_joins_the_key(self, client: SpendClient, driven: DrivenKey) -> None:
@ -328,6 +331,8 @@ class TestKeyAttribution:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.SPEND_REPORTING,
providers=(Provider.GEMINI, Provider.ANTHROPIC, Provider.OPENAI),
models=DRIVEN_MODELS,
)
)
def test_spend_logs_by_key_return_every_row_with_the_alias(self, client: SpendClient, driven: DrivenKey) -> None:
@ -362,6 +367,8 @@ class TestKeyAttribution:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.SPEND_REPORTING,
providers=(Provider.GEMINI, Provider.ANTHROPIC, Provider.OPENAI),
models=DRIVEN_MODELS,
)
)
def test_user_daily_activity_reports_alias_and_email(self, client: SpendClient, driven: DrivenKey) -> None:
@ -390,6 +397,8 @@ class TestKeyAttribution:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.HEALTH,
providers=(Provider.GEMINI,),
models=(CHAT_MODEL,),
)
)
def test_health_check_rows_keep_the_service_account_key(self, client: SpendClient) -> None:
@ -409,8 +418,8 @@ class TestKeyAttribution:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.BATCHES,
provider=Provider.OPENAI,
model=BATCH_MODEL,
providers=(Provider.OPENAI,),
models=(BATCH_MODEL,),
mode=Mode.BATCH,
)
)

4
tests/e2e/quota_management/spend_tracking/test_provider_edge_spend_e2e.py
View file

@ -27,8 +27,8 @@ pytestmark = [pytest.mark.e2e, pytest.mark.replayable]
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=f"openai/{CHEAP_OPENAI_MODEL}",
providers=(Provider.OPENAI,),
models=(f"openai/{CHEAP_OPENAI_MODEL}",),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/spend_tracking/test_service_tier_pricing_e2e.py
View file

@ -50,8 +50,8 @@ class TestServiceTierPricing:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model=BACKEND,
providers=(Provider.OPENAI,),
models=(BACKEND,),
capabilities=(Capability.REASONING,),
mode=Mode.NONSTREAM,
)

60
tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py
View file

@ -67,8 +67,8 @@ def _require_row(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
mode=Mode.NONSTREAM,
)
)
@ -111,8 +111,8 @@ def test_chat_completion_writes_nonzero_spend_row(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
mode=Mode.STREAM,
)
)
@ -148,8 +148,8 @@ def test_streaming_chat_completion_tracks_spend(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.MESSAGES,
provider=Provider.OPENAI,
model="openai-responses-codex",
providers=(Provider.OPENAI,),
models=("openai-responses-codex",),
mode=Mode.STREAM,
)
)
@ -227,8 +227,8 @@ def test_streaming_messages_via_responses_bridge_tracks_spend(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.EMBEDDINGS,
provider=Provider.OPENAI,
model="openai-text-embedding-3-small",
providers=(Provider.OPENAI,),
models=("openai-text-embedding-3-small",),
mode=Mode.NONSTREAM,
)
)
@ -259,8 +259,8 @@ def test_embedding_writes_nonzero_spend_row(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
mode=Mode.NONSTREAM,
)
)
@ -304,8 +304,8 @@ def test_cache_hit_is_zero_cost_and_suffixed(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
mode=Mode.NONSTREAM,
)
)
@ -341,8 +341,8 @@ def test_key_spend_equals_sum_of_logs(client: SpendClient, scoped_key: str) -> N
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model="openai/gpt-5.6-luna",
providers=(Provider.OPENAI,),
models=("openai/gpt-5.6-luna",),
mode=Mode.NONSTREAM,
)
)
@ -367,8 +367,8 @@ def test_burst_of_concurrent_calls_loses_no_spend(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
mode=Mode.NONSTREAM,
)
)
@ -429,8 +429,8 @@ def test_spend_logs_v2_pagination_caps_pages_and_keeps_total(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
mode=Mode.NONSTREAM,
)
)
@ -455,8 +455,8 @@ def test_request_tags_round_trip(client: SpendClient, scoped_key: str) -> None:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
mode=Mode.NONSTREAM,
)
)
@ -502,8 +502,8 @@ def test_tag_spend_matches_sum_of_tagged_logs(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
mode=Mode.NONSTREAM,
)
)
@ -529,8 +529,8 @@ def test_end_user_spend_attributed_on_row(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI, Provider.ANTHROPIC),
models=("gemini-2.5-flash", "claude-haiku-4-5"),
mode=Mode.NONSTREAM,
)
)
@ -587,8 +587,8 @@ def test_each_model_on_a_shared_key_gets_its_own_row(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.OPENAI,
model="openai/gpt-5.5",
providers=(Provider.OPENAI,),
models=("openai/gpt-5.5",),
mode=Mode.NONSTREAM,
)
)
@ -621,8 +621,8 @@ def test_failure_call_writes_failure_status_row(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.SPEND_REPORTING,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
)
)
def test_spend_calculate_returns_nonzero_cost(client: SpendClient) -> None:
@ -639,8 +639,8 @@ def test_spend_calculate_returns_nonzero_cost(client: SpendClient) -> None:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
provider=Provider.GEMINI,
model="gemini-2.5-flash",
providers=(Provider.GEMINI,),
models=("gemini-2.5-flash",),
mode=Mode.NONSTREAM,
)
)

4
tests/e2e/quota_management/spend_tracking/test_team_daily_activity_e2e.py
View file

@ -87,8 +87,8 @@ class TestTeamDailyActivity:
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.SPEND_REPORTING,
provider=Provider.OPENAI,
model="openai/gpt-5.6-luna",
providers=(Provider.OPENAI,),
models=("openai/gpt-5.6-luna",),
)
)
def test_valid_date_range_returns_results_and_metadata(

112
tests/e2e/test_junit_properties.py
View file

@ -10,7 +10,7 @@ rollups and, for ``source``, the status page's per-test links to GitHub.
from __future__ import annotations
from dataclasses import fields
from dataclasses import fields, replace
from pathlib import Path
import pytest
@ -169,15 +169,16 @@ class TestSubjectProperties:
def test_every_declared_field_becomes_a_property_in_field_order(self, request: pytest.FixtureRequest) -> None:
"""One pass over `dataclasses.asdict`: declaration order is emission order,
a (str, Enum) member is written as its `.value` and never `str(member)`,
the plural field emits a repeated SINGULAR `capability`, and an unset field
(`provider` here) emits nothing at all."""
and every plural field emits a repeated SINGULAR name (`provider`, `model`,
`capability`), one <property> per member and never a delimiter-joined value."""
test = type(self).test_every_declared_field_becomes_a_property_in_field_order
request.applymarker(
meta(
Subject(
domain=Domain.SPEND_BUDGETS,
route=Route.CHAT_COMPLETIONS,
model="gpt-5.5",
providers=(Provider.GEMINI, Provider.ANTHROPIC),
models=("gemini-2.5-flash", "claude-haiku-4-5"),
capabilities=(Capability.VISION, Capability.FUNCTION_CALLING, Capability.VISION),
mode=Mode.NONSTREAM,
)
@ -186,30 +187,108 @@ class TestSubjectProperties:
assert subject_properties(collected_item(request, test.__name__)) == (
("domain", "spend-budgets"),
("route", "chat_completions"),
("model", "gpt-5.5"),
("provider", "anthropic"),
("provider", "gemini"),
("model", "claude-haiku-4-5"),
("model", "gemini-2.5-flash"),
("capability", "function_calling"),
("capability", "vision"),
("mode", "nonstream"),
)
def test_one_provider_with_three_models_pairs_nothing(self, request: pytest.FixtureRequest) -> None:
"""The claude_code matrix shape: one test node drives haiku, sonnet and opus
through a single provider. The two lists are independent sets, so their
lengths need not agree and no model is tied to a provider by position."""
test = type(self).test_one_provider_with_three_models_pairs_nothing
request.applymarker(
meta(
Subject(
providers=(Provider.BEDROCK,),
models=("claude-sonnet-4-5", "claude-opus-4-7", "claude-haiku-4-5"),
)
)
)
assert subject_properties(collected_item(request, test.__name__)) == (
("provider", "bedrock"),
("model", "claude-haiku-4-5"),
("model", "claude-opus-4-7"),
("model", "claude-sonnet-4-5"),
)
def test_an_empty_plural_field_emits_nothing(self, request: pytest.FixtureRequest) -> None:
"""No `provider`, `model` or `capability` property at all, rather than one
with an empty value: the emitter is what turns absence into `[]`."""
test = type(self).test_an_empty_plural_field_emits_nothing
request.applymarker(meta(Subject(domain=Domain.MANAGEMENT)))
assert subject_properties(collected_item(request, test.__name__)) == (("domain", "management"),)
def test_scalar_property_names_are_the_dataclass_field_names(self, request: pytest.FixtureRequest) -> None:
"""The mapping is `asdict`, not a hand-written table: a scalar field added
to `Subject` later serializes under its own name with no edit to the
serializer. Proven by reading the field list back off the dataclass."""
test = type(self).test_scalar_property_names_are_the_dataclass_field_names
request.applymarker(meta(Subject(domain=Domain.UNKNOWN, route=Route.HEALTH, model="m", mode=Mode.STREAM)))
request.applymarker(meta(Subject(domain=Domain.UNKNOWN, route=Route.HEALTH, mode=Mode.STREAM)))
declared = tuple(field.name for field in fields(Subject))
emitted = tuple(name for name, _ in subject_properties(collected_item(request, test.__name__)))
assert emitted == tuple(name for name in declared if name in {"domain", "route", "model", "mode"})
assert emitted == tuple(name for name in declared if name in {"domain", "route", "mode"})
def test_capabilities_are_deduped_and_sorted_at_declaration(self) -> None:
def test_every_plural_field_is_deduped_and_sorted_at_declaration(self) -> None:
"""Canonicalized in `__post_init__`, so two tests that spelled the same set
in different orders produce byte-identical properties and the committed run
files diff cleanly."""
assert Subject(capabilities=(Capability.VISION, Capability.REASONING, Capability.VISION)).capabilities == (
Capability.REASONING,
Capability.VISION,
files diff cleanly. Sorted by the value that is serialized, which for an
enum is its `.value` and not its member name."""
subject = Subject(
providers=(Provider.OPENAI, Provider.ANTHROPIC, Provider.OPENAI),
models=("gpt-5.5", "claude-haiku-4-5", "gpt-5.5"),
capabilities=(Capability.VISION, Capability.REASONING, Capability.VISION),
)
assert subject.providers == (Provider.ANTHROPIC, Provider.OPENAI)
assert subject.models == ("claude-haiku-4-5", "gpt-5.5")
assert subject.capabilities == (Capability.REASONING, Capability.VISION)
@pytest.mark.parametrize(
("field", "value"),
[
("models", "gpt-5.5"),
("models", ["gpt-5.5"]),
("providers", Provider.OPENAI),
("providers", [Provider.OPENAI]),
("capabilities", Capability.VISION),
("capabilities", frozenset({Capability.VISION})),
],
)
def test_a_plural_field_refuses_anything_but_a_tuple(self, field: str, value: object) -> None:
"""`models=("gpt-5.5")` is a str, not a one-member tuple: the parentheses
do nothing without the trailing comma, and iterating the str would declare
one model per character. basedpyright flags it at the call site; this is
the runtime half, raised where the decorator runs, so it lands as a
collection error naming the file. `replace` is the untyped way in, since
the typed constructor would not let the test spell the mistake."""
with pytest.raises(TypeError, match=rf"Subject\.{field} must be a tuple"):
_ = replace(Subject(), **{field: value})
@pytest.mark.parametrize(
("field", "value", "member_type"),
[
("providers", ("openai",), "Provider"),
("capabilities", ("vision",), "Capability"),
("models", (5,), "str"),
],
)
def test_a_plural_field_refuses_a_member_of_the_wrong_type(
self, field: str, value: object, member_type: str
) -> None:
"""A bare "openai" where `Provider.OPENAI` belongs would serialize fine
today and stop joining the day the enum value is renamed."""
with pytest.raises(TypeError, match=rf"Subject\.{field} takes {member_type} members"):
_ = replace(Subject(), **{field: value})
def test_a_blank_model_is_dropped_rather_than_refused(self) -> None:
"""`models` is fed from env-overridable constants. A blank override is the
operator's mistake, and it must cost one missing property, not the
collection of the whole module."""
assert Subject(models=("", "gpt-5.5")).models == ("gpt-5.5",)
def test_the_typed_marker_only_ever_appends_to_the_fixed_prefix(self, request: pytest.FixtureRequest) -> None:
"""Loki, Grafana and the status page read `package`/`covers`/`source`; the
@ -267,9 +346,6 @@ class TestProviderMirrorsLitellm:
needs it, but a value that is not a provider at all would ship a property
no consumer can join on."""
try:
# Local, and the only litellm import under tests/e2e: at module scope
# it would be exactly the collection-time dependency `Provider` exists
# to avoid.
from litellm.types.utils import LlmProviders
except ImportError: # pragma: no cover - the runner image's shape
pytest.skip("litellm is not importable here, which is the property under test")
@ -282,9 +358,9 @@ class TestStepRecording:
"""The recorded half: `@step`-decorated harness helpers append to the running
test's story as they execute.
Each test here starts from an empty log because conftest's autouse
`_record_steps` fixture resets the recorder at setup -- the same reset the
live suite relies on for per-test isolation.
Each test here starts from an empty log because conftest's
`pytest_runtest_setup` hook resets the recorder first thing in every test's
setup -- the same reset the live suite relies on for per-test isolation.
"""
def test_steps_land_in_call_order(self) -> None:

346
tests/e2e/test_junit_report.py Normal file
View file

@ -0,0 +1,346 @@
"""The JUnit report itself, written by a real pytest run.
No proxy and no ``e2e`` marker. test_junit_properties.py pins the functions that
build the properties; this pins what reaches the XML once pytest, its junitxml
plugin, pytest-rerunfailures and xdist are all in the loop. Each case writes a
throwaway suite into a tmp dir and runs it in a child interpreter with THIS
directory's conftest.py loaded as a plugin, so the hooks under test are the ones
the live suite runs and the recorder is the real one, never a copy of either.
The timing that makes the recorded half work is pytest's, which is why it is
pinned here against the real thing: junitxml writes a testcase's properties from
its TEARDOWN report, and pytest builds that report from ``item.user_properties``
after the setup and call phases have both attached the steps. The suite runs
distributed, so every assertion is made in-process and again under ``-n 2``.
"""
from __future__ import annotations
import os
import shlex
import subprocess
import sys
from collections.abc import Mapping
from importlib.util import find_spec
from pathlib import Path
from types import MappingProxyType
from typing import Final
from xml.etree import ElementTree
import pytest
SUITE_DIR: Final = Path(__file__).resolve().parent
CHILD_TIMEOUT_SECONDS: Final = 180
STORY_SUITE: Final = """
from collections.abc import Iterator
from pathlib import Path
import pytest
from e2e_metadata import Capability, Domain, Mode, Provider, Route, Subject, meta, step
FIRST_ATTEMPT_MADE = Path(__file__).with_name("first-attempt-made")
@step("generate virtual key")
def generate_key() -> None:
return None
@step("create team")
def create_team() -> None:
raise RuntimeError("/team/new answered 500")
@step("POST /chat/completions")
def chat(*, ok: bool) -> None:
if not ok:
raise AssertionError("status_code=502 from upstream")
@step("poll /spend/logs")
def poll_spend_logs() -> None:
return None
@step("delete virtual key")
def delete_key() -> None:
return None
@pytest.fixture
def key() -> Iterator[None]:
generate_key()
yield
delete_key()
@pytest.fixture
def team(key: None) -> None:
create_team()
def test_passes(key: None) -> None:
chat(ok=True)
poll_spend_logs()
def test_fails(key: None) -> None:
chat(ok=False)
poll_spend_logs()
def test_errors_in_setup(team: None) -> None:
poll_spend_logs()
def test_passes_on_the_rerun(key: None) -> None:
first_attempt = not FIRST_ATTEMPT_MADE.exists()
FIRST_ATTEMPT_MADE.touch()
chat(ok=not first_attempt)
poll_spend_logs()
@meta(
Subject(
domain=Domain.LLM_TRANSLATION,
route=Route.MESSAGES,
providers=(Provider.BEDROCK, Provider.ANTHROPIC),
models=("claude-sonnet-4-5", "claude-opus-4-7", "claude-haiku-4-5"),
capabilities=(Capability.VISION, Capability.FUNCTION_CALLING),
mode=Mode.STREAM,
)
)
def test_declares_two_providers_and_three_models() -> None:
assert Provider.BEDROCK.value == "bedrock"
"""
WIDE_FINALIZER_SUITE: Final = """
from collections.abc import Iterator
import pytest
from e2e_metadata import step
@step("generate virtual key")
def generate_key() -> None:
return None
@step("delete shared team")
def delete_shared_team() -> None:
return None
@pytest.fixture(scope="module")
def shared_team() -> Iterator[None]:
yield
delete_shared_team()
def test_uses_the_shared_team(shared_team: None) -> None:
generate_key()
"""
WIDE_SETUP_ERROR_SUITE: Final = """
import pytest
from e2e_metadata import step
@step("log in to the identity provider")
def log_in() -> None:
raise RuntimeError("identity provider is down")
@pytest.fixture(scope="module")
def identity() -> None:
log_in()
def test_dies_in_a_module_scoped_fixture(identity: None) -> None:
assert identity is None
"""
BARE_STR_SUITE: Final = """
from e2e_metadata import Subject, meta
@meta(Subject(models=("gpt-5.5")))
def test_never_collected() -> None:
assert Subject is not None
"""
Properties = tuple[tuple[str, str], ...]
def write_suite(directory: Path, modules: Mapping[str, str]) -> None:
"""Lay a child suite out in ``directory``, with an ini file of its own.
The ini pins the child's rootdir to the tmp dir wherever that lives, and its
``pythonpath`` is what makes this directory's conftest.py, and the harness
modules the child suite imports, importable under ``-I``.
"""
_ = (directory / "pytest.ini").write_text(f"[pytest]\npythonpath = {shlex.quote(str(SUITE_DIR))}\n")
for name, source in modules.items():
_ = (directory / name).write_text(source)
def run_child_pytest(suite: Path, *args: str) -> subprocess.CompletedProcess[str]:
"""Run pytest over ``suite`` in a fresh interpreter, hooked up like the live suite.
``-p conftest`` registers this directory's conftest.py as a plugin, since a
tmp dir outside tests/e2e would never pick it up by location. The parent's
fixture-mode and addopts settings are dropped so a replay lane cannot leak
into the child.
"""
inherited: Final = {
name: value
for name, value in os.environ.items()
if name != "PYTEST_ADDOPTS" and not name.startswith("E2E_FIXTURE_")
}
return subprocess.run(
[sys.executable, "-I", "-m", "pytest", "-p", "conftest", "-p", "no:cacheprovider", *args, str(suite)],
cwd=suite,
env=inherited,
capture_output=True,
text=True,
timeout=CHILD_TIMEOUT_SECONDS,
check=False,
)
def properties_by_test(testsuite: ElementTree.Element) -> Mapping[str, Properties]:
"""Every testcase's <property> pairs, in document order, keyed by test name."""
return MappingProxyType(
{
testcase.get("name", ""): tuple(
(prop.get("name", ""), prop.get("value", "")) for prop in testcase.iter("property")
)
for testcase in testsuite.iter("testcase")
}
)
def values(properties: Properties, name: str) -> tuple[str, ...]:
return tuple(value for prop, value in properties if prop == name)
@pytest.fixture(
scope="module",
params=[
pytest.param((), id="in-process"),
pytest.param(
("-n", "2"),
id="xdist",
marks=pytest.mark.skipif(find_spec("xdist") is None, reason="pytest-xdist is not installed"),
),
],
)
def report(request: pytest.FixtureRequest, tmp_path_factory: pytest.TempPathFactory) -> Mapping[str, Properties]:
"""One child run per distribution mode, shared by every assertion below.
``--reruns 1`` and the ``--only-rerun`` pattern are the live suite's own
addopts. The two wide-scope modules sort ahead of the story, and next to each
other, so in-process the second one's setup runs right after the first one's
module-scoped finalizer.
"""
distribution: Final[tuple[str, ...]] = request.param # pyright: ignore[reportAny] # pytest types request.param as Any
suite: Final = tmp_path_factory.mktemp("suite")
write_suite(
suite,
{
"test_scope_a_finalizer.py": WIDE_FINALIZER_SUITE,
"test_scope_b_setup_error.py": WIDE_SETUP_ERROR_SUITE,
"test_story.py": STORY_SUITE,
},
)
xml: Final = suite / "report.xml"
child: Final = run_child_pytest(
suite, f"--junitxml={xml}", "--reruns", "1", "--only-rerun", "status_code=5[0-9][0-9]", *distribution
)
assert xml.exists(), f"the child run wrote no JUnit report:\n{child.stdout}\n{child.stderr}"
testsuite: Final = next(ElementTree.parse(xml).getroot().iter("testsuite"))
outcomes: Final = {name: testsuite.get(name) for name in ("tests", "failures", "errors", "skipped")}
assert outcomes == {"tests": "7", "failures": "1", "errors": "2", "skipped": "0"}, child.stdout
return properties_by_test(testsuite)
class TestStepsReachTheReport:
def test_a_passing_test_tells_its_story_in_call_order(self, report: Mapping[str, Properties]) -> None:
"""Fixture setup first, then the body. The finalizer's "delete virtual key"
is cleanup and is deliberately not part of the story."""
assert values(report["test_passes"], "step") == (
"generate virtual key",
"POST /chat/completions",
"poll /spend/logs",
)
def test_a_failing_test_s_last_step_is_where_it_died(self, report: Mapping[str, Properties]) -> None:
"""The reason the field exists. Nothing the test never reached is listed,
and no teardown step is appended behind the one it died on."""
assert values(report["test_fails"], "step") == ("generate virtual key", "POST /chat/completions")
def test_a_setup_error_keeps_the_steps_recorded_before_the_crash(self, report: Mapping[str, Properties]) -> None:
"""A fixture that raises never reaches the call phase, and setup is where
an e2e test most often dies (proxy not ready, key creation failing), so
the steps have to be attached after setup too."""
assert values(report["test_errors_in_setup"], "step") == ("generate virtual key", "create team")
def test_a_rerun_reports_only_the_attempt_junit_records(self, report: Mapping[str, Properties]) -> None:
"""The first attempt died on the chat call and the rerun got through. Steps
are attached twice per attempt, and none of that may show up as a doubled
or a stale story."""
assert values(report["test_passes_on_the_rerun"], "step") == (
"generate virtual key",
"POST /chat/completions",
"poll /spend/logs",
)
def test_a_setup_error_does_not_inherit_a_wider_finalizer_s_steps(self, report: Mapping[str, Properties]) -> None:
"""A module-scoped finalizer runs after the last test of its module, and
a module-scoped fixture is set up before any function-scoped one. The log
is emptied ahead of both, so the next test's setup error reports its own
steps and not "delete shared team"."""
assert values(report["test_uses_the_shared_team"], "step") == ("generate virtual key",)
assert values(report["test_dies_in_a_module_scoped_fixture"], "step") == ("log in to the identity provider",)
def test_steps_ride_behind_the_fixed_prefix(self, report: Mapping[str, Properties]) -> None:
"""`package`/`covers`/`source` are what Loki, Grafana and the status page
already read, on every outcome including a setup error."""
for name in ("test_passes", "test_fails", "test_errors_in_setup"):
assert tuple(prop for prop, _ in report[name])[:4] == ("package", "covers", "source", "step"), name
class TestDeclaredPropertiesReachTheReport:
def test_repeated_provider_model_and_capability_round_trip(self, report: Mapping[str, Properties]) -> None:
"""One <property> per member under the SINGULAR name, deduped and sorted,
with no pairing between the two providers and the three models."""
declared: Final = tuple(
(prop, value)
for prop, value in report["test_declares_two_providers_and_three_models"]
if prop not in {"package", "covers", "source"}
)
assert declared == (
("domain", "llm-translation"),
("route", "messages"),
("provider", "anthropic"),
("provider", "bedrock"),
("model", "claude-haiku-4-5"),
("model", "claude-opus-4-7"),
("model", "claude-sonnet-4-5"),
("capability", "function_calling"),
("capability", "vision"),
("mode", "stream"),
)
class TestBareStrIsACollectionError:
def test_a_str_where_a_tuple_belongs_fails_collection_and_names_the_fix(self, tmp_path: Path) -> None:
"""`models=("gpt-5.5")` raises where the decorator runs, which is import, so
pytest stops at collection and points at the file. Nothing is run and no
one-letter `model` properties are ever shipped."""
write_suite(tmp_path, {"test_bare_str.py": BARE_STR_SUITE})
child: Final = run_child_pytest(tmp_path)
assert child.returncode == pytest.ExitCode.INTERRUPTED, child.stdout
assert "Subject.models must be a tuple, got str: 'gpt-5.5'" in child.stdout
assert "models=(x,), not models=(x)" in child.stdout