`@pytest.mark.covers("cell.id")` is a registry key, not a description: it cannot answer "which tests drive /v1/responses on Anthropic". This adds the declared half of the per-test metadata, on top of the recorded step log
`@meta(Subject(...))` from tests/e2e/e2e_metadata.py takes one frozen dataclass whose fields are closed enums (domain, route, providers, capabilities, mode) plus free-string models, so a typo is a basedpyright error at the call site rather than a property that silently never appears. providers, models and capabilities are tuples because one test node often drives several (the claude_code matrix runs haiku, sonnet and opus in one body), with no positional pairing between them. Each is deduped and sorted at declaration so committed run artifacts diff cleanly, and anything but a tuple is refused at import, so models=("gpt-5.5") is a collection error naming the file instead of one model per character
Serialization is one pass over dataclasses.asdict: each scalar is one <property> under its field name, each plural value a repeated property under its singular name (provider, model, capability). Empty fields emit nothing, and the fixed package/covers/source prefix stays byte-identical, with the declared fields appended behind it. `covers` is untouched: the marker is separate because a dataclass passed to covers would be dropped silently by dedupe_covers and hard-fail collection in tests/integration/conftest.py, and @meta goes below @covers so every source deep link keeps its line
Provider mirrors litellm's LlmProviders values instead of importing them, because tests/e2e is shipped to the runner image on its own and a module-scope `from litellm...` would make the package a collection-time dependency. TestProviderMirrorsLitellm fails on drift wherever litellm is importable
tests/e2e/quota_management/ (29 files) is annotated as the pilot, and tests that drive more than one provider or model declare all of them. Every field is optional until the backfill of the rest of the suite lands
Its harness tests sit beside the step log's in tests/code_coverage_tests/test_e2e_metadata.py and test_e2e_junit_report.py, since tests/e2e holds only tests that drive a live proxy
The log stopped recording at MAX_STEPS, so a test that ran past 50 distinct steps and then failed reported step 50 as its last, not the helper it died in. The cap now drops from the front: the newest 50 steps are kept, led by a line counting the ones dropped, so the story still ends where the test died and says when it is partial
tests/e2e holds only tests that drive a live proxy, so the new recorder tests move to tests/code_coverage_tests/test_e2e_metadata.py and the report test to test_e2e_junit_report.py. tests/e2e/test_junit_properties.py is back to what main has. Without the e2e conftest, an autouse fixture does the per-test reset its setup hook did, and the attach tests lay the fixed prefix down themselves. CircleCI's provider_replay_harness job runs both files, and classify_changes.sh triggers it when they change, the same as the other harness tests there
A test's JUnit report says whether it passed, never what it did or where a failing test died. This records that from the harness, so nothing about it is hand-written and it cannot drift from what the test actually ran
`@step("create team with a budget")` from the new tests/e2e/e2e_metadata.py goes on harness helpers, never on tests, and appends its label to the running test's step log in call order. The label is recorded before the wrapped call, so a helper that raises still leaves its own label last: a failing test's last step is where it died. Every public harness method that performs an action now carries one, 355 across the client modules, lifecycle, idp, the logging readers, migrations and the claude_code driver
Only the outermost step records, tracked per thread. Harness layers call each other (ResourceManager.key goes through ProxyClient.generate_key, a domain client wraps the shared ProxyClient), so every layer carries a label and the story still reads at the level the test called in at, one beat per action. A step above @contextmanager holds the guard through __enter__ and __exit__, so a context's cleanup never lands behind the step a test died on, and a bare generator function is refused at import because its body interleaves with its caller's. Consecutive duplicates collapse and the log caps at 50, so a poll loop is one beat rather than fifty. The wrapper is a frame, so the eight cleanup and retry warnings raised directly inside decorated helpers use stacklevel=2 + STEP_FRAMES to keep reporting at their caller
The log is emptied first thing in pytest_runtest_setup and attached from the existing pytest_runtest_makereport wrapper after setup and again after call, so a test that errors in a fixture keeps the steps recorded before the crash. Teardown does not attach: finalizer steps are cleanup. Each attach drops the item's earlier step entries, so the second attach and a --reruns 1 retry replace the story rather than doubling it
Steps ride out as repeated <property name="step"> entries behind the fixed package/covers/source prefix, which stays byte-identical. The project-releaser emitter already regroups them into the results JSON's steps array. test_junit_report.py runs real pytest with --junitxml against this conftest, in-process and under -n 2, and pins the passing, failing, setup-error, rerun and wide-scope-fixture cases on the parsed XML
* fix(guardrails): scan video prompts for key-attached guardrails on /v1/videos
/v1/videos dispatches call_type avideo_generation, which CallTypes did not
know and no guardrail translation handler covered, so the unified guardrail
hook returned the request unscanned. Add the video call types and an OpenAI
video guardrail translation package that scans the prompt for create, remix,
edit and extension requests
Resolves LIT-6685
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate api types for video call types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: skip avideo_generation in azure sdk client exhaustive check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): retry a leaked video job until the guardrail sync deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): satisfy the type-discipline gate in the video handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): gate the video e2e on a chat probe so a miss starts at most one paid job
Addresses Greptile review: typed RewritingGuardrail override, dropped routine docstrings, and the e2e waits for the key guardrail to sync via /chat/completions before its single /v1/videos call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Resolve conflicts in the e2e Responses suite: main replaced the bespoke
EndpointsClient with provider SDK clients (76bf0cd579), so the four added
Vertex and Azure OpenAI /v1/responses tests are ported onto the OpenAI SDK
and tool_choice=required now comes from the SDK instead of a custom
request model.
Addresses Greptile review: typed RewritingGuardrail override, dropped routine docstrings, and the e2e waits for the key guardrail to sync via /chat/completions before its single /v1/videos call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
