fix(e2e): keep the tool-continuation echo-back test on the live path

The key normalizes a unique marker so two builds match, which is the whole
point, but it makes this test's identity collide with an earlier run's: it mints
a fresh receipt, sends it through a tool result, and asserts the model echoes it
back verbatim, so a stale recording matched and answered with the old receipt.
Build 223 is where that surfaced, once the corpus was full enough for the first
call to hit. A test that asserts a provider echoed this run's own unique value
belongs on the live path.
This commit is contained in:
Yuneng Jiang 2026-09-16 05:09:04 -07:00
parent 7c2234be3a
commit bd1c2d6f07
3 changed files with 9 additions and 2 deletions

View file

@ -44,7 +44,9 @@ The trusted runner receives:
Do not give cache credentials to candidate deployments. Counter artifacts contain no recorded payloads or credentials. Hits count shared-cache responses; upstream attempts count actual forwards from the edge. Every counter is emitted twice, once as a flat total and once under `mount:{mount}:`, so a hit rate can be read per provider rather than only in aggregate. Existing application-cache observations still count requests arriving at the edge, including shared-cache hits
Tests that require real provider timing, limits or state use `@pytest.mark.provider_live`. The marker keeps newly registered models on live routes without weakening their assertions. The provider prompt-caching tests carry it because a replayed priming response reports cache creation rather than a cache read. Ordinary assertion failures still fail E2E. The shared cache does not modify provider response IDs or make the proxy aware of replay
Tests that require real provider timing, limits or state use `@pytest.mark.provider_live`. The marker keeps newly registered models on live routes without weakening their assertions. The provider prompt-caching tests carry it because a replayed priming response reports cache creation rather than a cache read.
One more class needs it, and it is the cost of normalizing the marker. A test that mints a fresh marker, sends it, and then asserts the provider's answer contains that exact value is asserting on the marker rather than using it as a salt. The key treats two such requests as the same identity, so a stale recording matches and answers with the marker from the run that recorded it. `TestOpenAIMessagesToolContinuation` is the one in the suite today: it sends a freshly minted receipt through a tool result and asserts the model echoes it back verbatim. If you add a test that asserts a provider echoed your own unique value, it belongs on the live path. Ordinary assertion failures still fail E2E. The shared cache does not modify provider response IDs or make the proxy aware of replay
## Recorded response semantics

View file

@ -85,7 +85,11 @@ def jwt_identity(idp: Keycloak, resources: ResourceManager, proxy: ProxyClient)
def pytest_configure(config: pytest.Config) -> None:
config.addinivalue_line("markers", "provider_live: requires actual provider timing, limits or state; bypass shared cache")
config.addinivalue_line(
"markers",
"provider_live: requires actual provider timing, limits, state, or a response that echoes this"
" run's own unique value; bypass shared cache",
)
config.addinivalue_line(
"markers",
"e2e: live test that requires a running proxy and real provider keys",

View file

@ -372,6 +372,7 @@ def _request_tool(
class TestOpenAIMessagesToolContinuation:
@pytest.mark.provider_live
@pytest.mark.parametrize("stream", [True, False], ids=["stream", "nonstream"])
def test_required_tool_arguments_and_correlated_result(
self, endpoints_client: EndpointsClient, resources: ResourceManager, stream: bool