Commit graph

3 commits

Author SHA1 Message Date
yucheng-berri
817582e697
test(e2e): otel trace completeness on streaming chat, messages, and responses (LIT-3787) (#33234) 2026-07-14 20:23:54 -07:00
yucheng-berri
6a213de9f4
test(e2e): otel trace completeness on /v1/messages (#33133)
* test(e2e): OTEL trace completeness on /v1/messages

Extends the LIT-3787 trace-completeness suite to the Anthropic-native route:
one successful non-streaming /v1/messages call must land at the destination as
ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT
span, no dangling parents). Adds the raw /v1/messages sender to the logging
suite client.

* test(e2e): reuse the shared AnthropicMessagesBody per review

Drops the duplicate /v1/messages request model in favor of the one models.py
already provides (budget_client uses the same one), passes max_tokens at the
call site to match the sibling chat test, notes in the docstring why the
gen-AI span is named chat on this surface, and adopts the hardened read-back
signature

* test(e2e): author the messages trace test docstring

* test(e2e): declare the messages surface on the covers marker

* test(e2e): otel trace completeness on /v1/responses (#33134)

* test(e2e): OTEL trace completeness on /v1/responses

Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API
route: one successful non-streaming /v1/responses call must land at the
destination as ONE connected trace. Adds the raw /v1/responses sender, a
CHEAP_OPENAI_MODEL config constant, and registers responses in the otel
registry cell's exercised_on.

* test(e2e): author the responses trace test docstring

* test(e2e): declare the responses and chat surfaces on the covers markers
2026-07-13 20:19:06 -07:00
yuneng-jiang
a43f128a74
test(e2e): add coverage registry and collector (#32304)
Introduce the e2e coverage denominator: 282 behavior cells across the six
tracking modules (LLMs, MCPs, Management/UI, Reliability & Performance,
Logging & Guardrails, Other), one validated YAML row each, plus a collector
that diffs the registry against @pytest.mark.covers markers and reports
coverage per module.

The registry rows validate against a pydantic discriminated union so a row
cannot carry a field from another module. The collector is static: a
collect-only pass reads the markers, so it runs no test and needs no live
proxy. Register the covers marker suite-wide so that pass works under
--strict-markers.

This is a draft for review. Tiers are proposed rather than signed off, and a
few cells still need a support check or a prune.
2026-07-07 15:51:20 -04:00