mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-20 00:11:50 +00:00
fix(e2e): keep the tool-continuation echo-back test on the live path
The key normalizes a unique marker so two builds match, which is the whole point, but it makes this test's identity collide with an earlier run's: it mints a fresh receipt, sends it through a tool result, and asserts the model echoes it back verbatim, so a stale recording matched and answered with the old receipt. Build 223 is where that surfaced, once the corpus was full enough for the first call to hit. A test that asserts a provider echoed this run's own unique value belongs on the live path.
This commit is contained in:
parent
7c2234be3a
commit
bd1c2d6f07
3 changed files with 9 additions and 2 deletions
|
|
@ -44,7 +44,9 @@ The trusted runner receives:
|
|||
|
||||
Do not give cache credentials to candidate deployments. Counter artifacts contain no recorded payloads or credentials. Hits count shared-cache responses; upstream attempts count actual forwards from the edge. Every counter is emitted twice, once as a flat total and once under `mount:{mount}:`, so a hit rate can be read per provider rather than only in aggregate. Existing application-cache observations still count requests arriving at the edge, including shared-cache hits
|
||||
|
||||
Tests that require real provider timing, limits or state use `@pytest.mark.provider_live`. The marker keeps newly registered models on live routes without weakening their assertions. The provider prompt-caching tests carry it because a replayed priming response reports cache creation rather than a cache read. Ordinary assertion failures still fail E2E. The shared cache does not modify provider response IDs or make the proxy aware of replay
|
||||
Tests that require real provider timing, limits or state use `@pytest.mark.provider_live`. The marker keeps newly registered models on live routes without weakening their assertions. The provider prompt-caching tests carry it because a replayed priming response reports cache creation rather than a cache read.
|
||||
|
||||
One more class needs it, and it is the cost of normalizing the marker. A test that mints a fresh marker, sends it, and then asserts the provider's answer contains that exact value is asserting on the marker rather than using it as a salt. The key treats two such requests as the same identity, so a stale recording matches and answers with the marker from the run that recorded it. `TestOpenAIMessagesToolContinuation` is the one in the suite today: it sends a freshly minted receipt through a tool result and asserts the model echoes it back verbatim. If you add a test that asserts a provider echoed your own unique value, it belongs on the live path. Ordinary assertion failures still fail E2E. The shared cache does not modify provider response IDs or make the proxy aware of replay
|
||||
|
||||
## Recorded response semantics
|
||||
|
||||
|
|
|
|||
|
|
@ -85,7 +85,11 @@ def jwt_identity(idp: Keycloak, resources: ResourceManager, proxy: ProxyClient)
|
|||
|
||||
|
||||
def pytest_configure(config: pytest.Config) -> None:
|
||||
config.addinivalue_line("markers", "provider_live: requires actual provider timing, limits or state; bypass shared cache")
|
||||
config.addinivalue_line(
|
||||
"markers",
|
||||
"provider_live: requires actual provider timing, limits, state, or a response that echoes this"
|
||||
" run's own unique value; bypass shared cache",
|
||||
)
|
||||
config.addinivalue_line(
|
||||
"markers",
|
||||
"e2e: live test that requires a running proxy and real provider keys",
|
||||
|
|
|
|||
|
|
@ -372,6 +372,7 @@ def _request_tool(
|
|||
|
||||
|
||||
class TestOpenAIMessagesToolContinuation:
|
||||
@pytest.mark.provider_live
|
||||
@pytest.mark.parametrize("stream", [True, False], ids=["stream", "nonstream"])
|
||||
def test_required_tool_arguments_and_correlated_result(
|
||||
self, endpoints_client: EndpointsClient, resources: ResourceManager, stream: bool
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue