mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-16 09:22:22 +00:00
test: preserve Bedrock coverage and prevent retry masking
This commit is contained in:
parent
71d9fe65a0
commit
3b3c071d20
8 changed files with 176 additions and 28 deletions
|
|
@ -96,11 +96,11 @@ E2E_FIXTURE_MODE=replay E2E_FIXTURE_DIR=/tmp/e2e-fixtures E2E_RESET_SPEND_LOGS=1
|
|||
|
||||
Point the proxy at bogus provider credentials for the replay run and it still has to pass: that is the whole proof that nothing left the process. Bundles are never committed. `tests/e2e/.fixtures` is gitignored because a bundle holds verbatim provider response bodies and hard-fails after seven days. CI records and replays this lane on a schedule in `.github/workflows/e2e_record_replay.yml`, publishing the bundle as a private `e2e-fixtures-bundle` artifact instead of committing it, selecting the tests with the `@pytest.mark.replayable` marker, and proving the bogus-credentials replay hermetic by counting provider egress with `.github/scripts/e2e_egress_sentinel.py`
|
||||
|
||||
Bedrock tests use `bedrock_config.bedrock_params()` for Chat Completions, Responses, Converse and Invoke. Live mode retains the normal AWS credential path. Record and replay use a `/bedrock-us-east-1` mount by default (`E2E_BEDROCK_REGION` selects one of `us-east-1`, `us-east-2`, `us-west-1`, `us-west-2`; use the same region in both runs). The proxy signs each edge request with public test credentials. The edge verifies that signature over the host, path and body before accepting it. Recording re-signs the unchanged payload for the AWS endpoint using the runner's `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and optional `AWS_SESSION_TOKEN`, assuming `AWS_ROLE_NAME` with the SDK when configured. Start the proxy without ambient `AWS_ROLE_NAME`, `AWS_PROFILE_NAME`, web identity or Bedrock bearer credentials in these modes so its public test credentials cannot trigger STS or bypass signature verification. The workflow isolates that proxy environment while retaining real credentials and role settings in the recording runner. Replay requires no real AWS credentials and never forwards a miss. This checks request signing and protocol behavior in replay; live runs remain necessary to check real IAM access and model availability. Binary AWS EventStream responses preserve transfer chunks in the bundle, and native stream assertions decode events with botocore and require text, ordered completion and usage
|
||||
Bedrock tests use `bedrock_config.bedrock_params()` for Chat Completions, Responses, Converse and Invoke. Live mode retains the normal AWS credential path. Record and replay use a `/bedrock-us-east-1` mount by default (`E2E_BEDROCK_REGION` selects one of `us-east-1`, `us-east-2`, `us-west-1`, `us-west-2`; use the same region in both runs). The proxy signs each edge request with public test credentials. The edge verifies that signature over the host, path and body before accepting it. Recording re-signs the unchanged payload for the AWS endpoint using the runner's `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and optional `AWS_SESSION_TOKEN`, assuming `AWS_ROLE_NAME` with the SDK when configured. Start the proxy without ambient `AWS_ROLE_NAME`, `AWS_PROFILE_NAME`, web identity or Bedrock bearer credentials in these modes so its public test credentials cannot trigger STS or bypass signature verification. The workflow isolates that proxy environment while retaining real credentials and role settings in the recording runner. Replay requires no real AWS credentials and never forwards a miss. This checks request signing and protocol behavior in replay; live runs remain necessary to check real IAM access and model availability. Binary AWS EventStream responses preserve transfer chunks in the bundle, and native stream assertions decode events with botocore and require complete frames, text, ordered completion and usage. The original Responses tool test keeps default tool selection, and a separate test covers `tool_choice="required"`
|
||||
|
||||
The recording workflow uses the existing `e2e-changed` environment and OIDC secrets reader (`E2E_AWS_ROLE_TO_ASSUME`, secret `litellm-e2e-changed-provider-keys`), and fails before starting the proxy if any required provider credential is missing. Recording requires approval under that environment's existing protection policy, including scheduled runs. Replay supplies only test credentials and includes the four supported Bedrock endpoints in the egress sentinel
|
||||
|
||||
HTTP assertions classify a Bedrock 503 as `ProviderUnavailableError` only when AWS headers with Bedrock route context, or a recognized Bedrock error envelope support that attribution. A decoded native `serviceUnavailableException` receives the same classification even when the committed HTTP status is 200, with `stream_event` recorded as its evidence. Interrupted binary or SSE transfers become modeled network failures with the received status and request IDs preserved. The default live lane allows one test rerun after ten seconds for an evidenced availability error or a transport failure. Generic proxy 5xx, authentication errors and assertion failures do not qualify. Persistent provider failures remain failed, and JUnit properties retain AWS and proxy request IDs from the failed attempt even when the rerun passes. Record and replay continue to use `--reruns 0`
|
||||
HTTP assertions classify a Bedrock 503 as `ProviderUnavailableError` only when AWS headers with Bedrock route context, or a recognized Bedrock error envelope support that attribution. A decoded native `serviceUnavailableException` receives the same classification even when the committed HTTP status is 200, with `stream_event` recorded as its evidence. Interrupted binary or SSE transfers produce `NetworkFailureError` with the received status and request IDs preserved. Both diagnostic exceptions hard-fail without a test rerun, including when their messages contain a legacy retry marker. Incomplete binary frames hard-fail even after a valid completion event. The existing typed `NetworkError` retry remains, with its filter anchored to that assertion rather than matching a marker embedded in another response, header or assertion message. No newly classified failure becomes eligible for a rerun. The exercised proxy 5xx, authentication and behavior failures do not qualify. This retains the existing text-based retry convention: an arbitrary assertion that deliberately imitates the exact typed network-error prefix can still match. JUnit retains AWS and proxy request IDs from the failed attempt. Record and replay continue to use `--reruns 0`
|
||||
|
||||
Current limits: deployments baked into the proxy's config file cannot be edge-wired (only `/model/new` registrations can carry the edge api_base), and a file upload routed by `custom_llm_provider` through the proxy's `files_settings` block never passes a deployment at all, so the batches `model_param` and `provider_fallback` scenarios keep uploading live in every mode
|
||||
|
||||
|
|
|
|||
|
|
@ -101,7 +101,7 @@ A couple of logging destinations are configured on the proxy rather than by the
|
|||
|
||||
### The pull request check
|
||||
|
||||
Every same-repository PR that adds, modifies, or renames a `tests/e2e/**/test_*.py` file runs those changed files three times. A change to the harness itself, meaning a root-level `tests/e2e/*.py` file or `pytest.ini`, `tests/e2e/gateway/`, `.github/e2e-stack/`, or the workflow, also runs the `access_control` suite and both JWT suites as canaries, because those files have no test of their own that exercises the stack. `.github/e2e-stack/select_tests.py` applies both rules. The stack config at `tests/e2e/gateway/stage_mirror_ci_config.yml` must declare every model the selected suites use; a missing one shows up as a failed test id in the public log. The suite's single rerun for network failures and evidenced Bedrock availability errors (see `pytest.ini`) applies on every pass, including outages reported inside native streams. Generic proxy 5xx and failed behavior assertions do not qualify, and persistent failures still fail the check. The stage-mirror stack has a control-plane backend, two gateways behind nginx, Postgres, Keycloak, Jaeger, and TLS cluster-mode Valkey. Realm-only edits also trigger these canaries. The stack exports every gateway address in `LITELLM_PROXY_REPLICA_URLS`, so model registration waits until each gateway lists the new model rather than whichever one the load balancer answered from. Documentation, deleted-file, and application-only changes do not start the stack or request environment approval. The `ui/`, `claude_code/`, and `load/` directories, `batches/test_managed_files_enforcement_e2e.py`, `llm_translation/realtime/test_realtime_pipecat_audio_e2e.py`, and `guardrails/test_presidio_masking_e2e.py` remain outside this check because they use separate tooling or need a differently configured stack: the pipecat audio suite skips itself at import time unless the NLTK `punkt_tab` data is installed, and the presidio suite fails without the analyzer and anonymizer services this stack does not start. The Redis chaos test under `load/` needs a proxy it can pause the Redis of on the same host (`gateway/redis_chaos_ci_config.yml`), which `.github/workflows/test-e2e-redis-chaos.yml` boots, and which the Buildkite `e2e-redis-chaos` step in project-releaser runs co-located with Postgres and Valkey in one pod; it is deselected unless `E2E_REDIS_CHAOS` is set
|
||||
Every same-repository PR that adds, modifies, or renames a `tests/e2e/**/test_*.py` file runs those changed files three times. A change to the harness itself, meaning a root-level `tests/e2e/*.py` file or `pytest.ini`, `tests/e2e/gateway/`, `.github/e2e-stack/`, or the workflow, also runs the `access_control` suite and both JWT suites as canaries, because those files have no test of their own that exercises the stack. `.github/e2e-stack/select_tests.py` applies both rules. The stack config at `tests/e2e/gateway/stage_mirror_ci_config.yml` must declare every model the selected suites use; a missing one shows up as a failed test id in the public log. The suite's existing single rerun for typed network results applies on every pass (see `pytest.ini`). Bedrock availability diagnostics and interrupted HTTP streams hard-fail without a rerun. The exercised proxy 5xx and behavior failures do not qualify either. The legacy filter matches assertion text, so custom assertions must not imitate the exact typed network-error prefix. The stage-mirror stack has a control-plane backend, two gateways behind nginx, Postgres, Keycloak, Jaeger, and TLS cluster-mode Valkey. Realm-only edits also trigger these canaries. The stack exports every gateway address in `LITELLM_PROXY_REPLICA_URLS`, so model registration waits until each gateway lists the new model rather than whichever one the load balancer answered from. Documentation, deleted-file, and application-only changes do not start the stack or request environment approval. The `ui/`, `claude_code/`, and `load/` directories, `batches/test_managed_files_enforcement_e2e.py`, `llm_translation/realtime/test_realtime_pipecat_audio_e2e.py`, and `guardrails/test_presidio_masking_e2e.py` remain outside this check because they use separate tooling or need a differently configured stack: the pipecat audio suite skips itself at import time unless the NLTK `punkt_tab` data is installed, and the presidio suite fails without the analyzer and anonymizer services this stack does not start. The Redis chaos test under `load/` needs a proxy it can pause the Redis of on the same host (`gateway/redis_chaos_ci_config.yml`), which `.github/workflows/test-e2e-redis-chaos.yml` boots, and which the Buildkite `e2e-redis-chaos` step in project-releaser runs co-located with Postgres and Valkey in one pod; it is deselected unless `E2E_REDIS_CHAOS` is set
|
||||
|
||||
Every selected file must execute at least one passing test in each pass, and any test failure, collection error, or entirely skipped or deselected file fails the check. A file whose tests are all marked skip therefore cannot pass this check, so unskip at least one of them, or add the file to `UNSUPPORTED` in `select_tests.py` with the reason, before changing one. A failed pass stops the run. The public log prints pytest's one-line summary for each pass, including the rerun count, and names each failed or errored test as `classname::name`, so a retried network error or a failing test is visible without the raw output. The final `e2e-changed-tests` job succeeds only when no supported test files changed or the approved run completed all three passes. Fork PRs with selected tests fail this gate until a maintainer brings the reviewed change onto a same-repository branch
|
||||
|
||||
|
|
|
|||
|
|
@ -17,6 +17,10 @@ class _InvokeChunk(BaseModel):
|
|||
bytes: str
|
||||
|
||||
|
||||
class _PreludeLength(BaseModel):
|
||||
total_length: int
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class BedrockEvent:
|
||||
payload: str
|
||||
|
|
@ -41,9 +45,24 @@ def _decode_event(event: EventStreamMessage) -> BedrockEvent:
|
|||
return BedrockEvent(payload=_JSON.dump_json({headers.event_type: _JSON.validate_json(event.payload)}).decode())
|
||||
|
||||
|
||||
def decode_bedrock_stream(chunks: Iterable[bytes]) -> Iterator[BedrockEvent]:
|
||||
buffer: Final = EventStreamBuffer()
|
||||
for chunk in chunks:
|
||||
buffer.add_data(chunk)
|
||||
for event in buffer:
|
||||
class _CompleteEventStream:
|
||||
def __init__(self) -> None:
|
||||
self._buffer: Final = EventStreamBuffer()
|
||||
self._pending_bytes: int = 0
|
||||
|
||||
def feed(self, chunk: bytes) -> Iterator[BedrockEvent]:
|
||||
self._pending_bytes += len(chunk)
|
||||
self._buffer.add_data(chunk)
|
||||
for event in self._buffer:
|
||||
self._pending_bytes -= _PreludeLength.model_validate(event.prelude, from_attributes=True).total_length
|
||||
yield _decode_event(event)
|
||||
|
||||
def finish(self) -> None:
|
||||
assert self._pending_bytes == 0, f"incomplete Bedrock frame: {self._pending_bytes} trailing bytes"
|
||||
|
||||
|
||||
def decode_bedrock_stream(chunks: Iterable[bytes]) -> Iterator[BedrockEvent]:
|
||||
stream: Final = _CompleteEventStream()
|
||||
for chunk in chunks:
|
||||
yield from stream.feed(chunk)
|
||||
stream.finish()
|
||||
|
|
|
|||
|
|
@ -280,7 +280,6 @@ class TestResponses:
|
|||
model,
|
||||
"What is the weather in San Francisco? Use the get_weather tool.",
|
||||
[WEATHER_TOOL],
|
||||
tool_choice="required",
|
||||
)
|
||||
require_successful_call(result)
|
||||
parsed = ResponsesResult.model_validate_json(result.body)
|
||||
|
|
@ -291,6 +290,32 @@ class TestResponses:
|
|||
arguments = WeatherArguments.model_validate(raw_arguments)
|
||||
assert arguments.location, f"function call arguments missing location: {function_call.arguments}"
|
||||
|
||||
@pytest.mark.replayable
|
||||
@pytest.mark.covers("llm.responses.bedrock_converse.tool_use.nonstream.works")
|
||||
def test_responses_bedrock_required_tool_returns_function_call(
|
||||
self, endpoints_client: EndpointsClient, resources: ResourceManager
|
||||
) -> None:
|
||||
model = f"e2e-responses-required-{unique_marker()}"
|
||||
model_id = endpoints_client.create_model(model, bedrock_params())
|
||||
resources.defer(lambda: endpoints_client.delete_model(model_id))
|
||||
key = resources.key()
|
||||
|
||||
result = endpoints_client.responses_with_tools(
|
||||
key,
|
||||
model,
|
||||
"What is the weather in San Francisco? Use the get_weather tool.",
|
||||
[WEATHER_TOOL],
|
||||
tool_choice="required",
|
||||
)
|
||||
require_successful_call(result)
|
||||
parsed = ResponsesResult.model_validate_json(result.body)
|
||||
function_call = next((call for call in parsed.function_calls if call.name == "get_weather"), None)
|
||||
assert function_call is not None, f"no required get_weather function call over bedrock: {result.body[:500]}"
|
||||
assert function_call.arguments is not None
|
||||
raw_arguments = cast(object, json.loads(function_call.arguments))
|
||||
arguments = WeatherArguments.model_validate(raw_arguments)
|
||||
assert arguments.location, f"function call arguments missing location: {function_call.arguments}"
|
||||
|
||||
@pytest.mark.skip(reason="stage red: product gap, /v1/responses 500s (aresponses TypeError) on missing input instead of 400")
|
||||
@pytest.mark.covers("llm.responses.openai.input_validation.nonstream.works")
|
||||
def test_missing_input_returns_error(
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ class ProviderUnavailableError(AssertionError):
|
|||
class NetworkFailureError(AssertionError):
|
||||
def __init__(self, failure: ProviderFailure, message: str) -> None:
|
||||
self.failure: Final = failure
|
||||
super().__init__(f"kind='network'; {failure.model_dump_json(exclude_none=True)}; body={message[:1000]}")
|
||||
super().__init__(f"HTTP transfer failed; {failure.model_dump_json(exclude_none=True)}; body={message[:1000]}")
|
||||
|
||||
|
||||
class _ErrorMessage(BaseModel):
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
# Config when any e2e suite under tests/e2e/ is run directly, e.g.
|
||||
# uv run pytest tests/e2e/quota_management/spend_tracking/ -v
|
||||
# The e2e marker is also registered in conftest.py for runs rooted elsewhere.
|
||||
addopts = --strict-markers --strict-config --reruns 1 --reruns-delay 10 --only-rerun "kind='network'" --only-rerun "ProviderUnavailableError"
|
||||
addopts = --strict-markers --strict-config --reruns 1 --only-rerun "^AssertionError: kind='network' message=" --rerun-except "ProviderUnavailableError" --rerun-except "NetworkFailureError"
|
||||
markers =
|
||||
e2e: live test that requires a running proxy and real provider keys
|
||||
replayable: edge-wired test whose provider traffic replays from a fixture bundle, so it makes zero provider calls in replay mode; the record/replay CI lane selects it with -m replayable
|
||||
|
|
|
|||
|
|
@ -125,6 +125,27 @@ class TestBedrockEventStream:
|
|||
with pytest.raises(ChecksumMismatch):
|
||||
tuple(decode_bedrock_stream((wire[:-1] + bytes([wire[-1] ^ 1]),)))
|
||||
|
||||
@pytest.mark.parametrize("invoke", [False, True])
|
||||
@pytest.mark.parametrize("cut", [1, 7, 11, 12, 30, -1])
|
||||
def test_incomplete_frame_after_valid_completion_is_rejected(self, invoke: bool, cut: int) -> None:
|
||||
wire: Final = (
|
||||
b"".join(
|
||||
_frame(json.dumps({"bytes": base64.b64encode(event.encode()).decode()}), "chunk")
|
||||
for event in _INVOKE
|
||||
)
|
||||
if invoke
|
||||
else b"".join(
|
||||
_frame(json.dumps(value), key)
|
||||
for payload in _CONVERSE
|
||||
for key, value in _JSON_OBJECT.validate_json(payload).items()
|
||||
)
|
||||
)
|
||||
assertion: Final = assert_invoke_stream if invoke else assert_converse_stream
|
||||
assertion(streaming_outcome(_BinaryResponse((wire,)), True, sent_at=0.0))
|
||||
extra: Final = _frame('{"message":"unavailable"}', "serviceUnavailableException", "exception")[:cut]
|
||||
with pytest.raises(AssertionError, match="incomplete Bedrock frame"):
|
||||
assertion(streaming_outcome(_BinaryResponse((wire, extra)), True, sent_at=0.0))
|
||||
|
||||
|
||||
class TestNativeStreamAssertions:
|
||||
@pytest.mark.parametrize("missing", [0, 1, 3, 4])
|
||||
|
|
@ -180,7 +201,7 @@ def test_interrupted_http_stream_preserves_network_classification_and_ids(path:
|
|||
)
|
||||
assert result.status_code == 200 and not result.ok
|
||||
assert result.network_error is not None and result.network_error.kind == "network"
|
||||
with pytest.raises(NetworkFailureError, match="kind='network'") as caught:
|
||||
with pytest.raises(NetworkFailureError, match="HTTP transfer failed") as caught:
|
||||
require_successful_call(result)
|
||||
assert caught.value.failure.provider is None
|
||||
assert caught.value.failure.request_id == "interrupted-request"
|
||||
|
|
|
|||
|
|
@ -6,14 +6,19 @@ from pathlib import Path
|
|||
from typing import Final
|
||||
|
||||
|
||||
def test_retry_policy_preserves_failures_and_first_attempt_evidence(tmp_path: Path) -> None:
|
||||
def test_retry_policy_does_not_turn_new_diagnostics_into_recovered_passes(tmp_path: Path) -> None:
|
||||
suite: Final = Path(__file__).resolve().parent
|
||||
scenario: Final = tmp_path / "test_scenarios.py"
|
||||
scenario.write_text(
|
||||
"""
|
||||
from pathlib import Path
|
||||
import json
|
||||
from e2e_http import NetworkError, StreamingResponse, UnknownApiError, require_successful_call, unwrap
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from threading import Thread
|
||||
import pytest
|
||||
from e2e_http import URL, NoBody, NetworkError, StreamingResponse, UnknownApiError, require_successful_call, send, unwrap
|
||||
from test_bedrock_eventstream import _CONVERSE, _frame
|
||||
from llm_translation.bedrock_stream import assert_converse_stream
|
||||
|
||||
def attempt(name):
|
||||
path = Path(__file__).with_name(name + ".attempts")
|
||||
|
|
@ -71,11 +76,79 @@ def test_stream_persistent():
|
|||
streamed_unavailable()
|
||||
|
||||
def test_stream_transport():
|
||||
attempt("stream_transport")
|
||||
require_successful_call(StreamingResponse(
|
||||
status_code=200, body="connection interrupted", network_error=NetworkError(message="connection interrupted"),
|
||||
headers={"x-amzn-requestid": "transport-request", "x-litellm-call-id": "transport-call"},
|
||||
))
|
||||
if attempt("stream_transport") == 1:
|
||||
require_successful_call(StreamingResponse(
|
||||
status_code=200, body="connection interrupted", network_error=NetworkError(message="kind='network'"),
|
||||
headers={"x-amzn-requestid": "transport-request", "x-litellm-call-id": "transport-call"},
|
||||
))
|
||||
|
||||
def test_connection_recovers():
|
||||
if attempt("connection_recovers") == 1:
|
||||
require_successful_call(StreamingResponse(
|
||||
status_code=-1, body="connection refused", network_error=NetworkError(message="kind='network'"),
|
||||
))
|
||||
|
||||
def test_connection_direct_recovers():
|
||||
if attempt("connection_direct_recovers") == 1:
|
||||
result = StreamingResponse(status_code=-1, body="connection refused", network_error=NetworkError(message="connection refused"))
|
||||
assert result.ok, f"call failed: {result}"
|
||||
|
||||
def test_header_marker_recovers():
|
||||
if attempt("header_marker_recovers") == 1:
|
||||
require_successful_call(StreamingResponse(status_code=403, body="invalid credentials", headers={"x-request-id": "kind='network'"}))
|
||||
|
||||
def test_behavior_recovers():
|
||||
assert attempt("behavior_recovers") > 1, "completion returned no content"
|
||||
|
||||
def test_healthy_control():
|
||||
require_successful_call(StreamingResponse(status_code=200, body="{}"))
|
||||
|
||||
class InterruptedOnce(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
self.rfile.read(int(self.headers.get("content-length", "0")))
|
||||
binary = self.path.endswith("-binary")
|
||||
wire = (
|
||||
b"".join(_frame(json.dumps(value), key) for payload in _CONVERSE for key, value in json.loads(payload).items())
|
||||
if binary else b'data: {"text":"Hello"}\\n\\ndata: [DONE]\\n\\n'
|
||||
)
|
||||
interrupted = attempt("http_" + self.path.lstrip("/")) == 1 and not self.path.startswith("/healthy-")
|
||||
data = wire[:30] if interrupted else wire
|
||||
self.send_response(200)
|
||||
self.send_header("content-type", "application/vnd.amazon.eventstream" if binary else "text/event-stream")
|
||||
self.send_header("transfer-encoding", "chunked")
|
||||
self.send_header("x-litellm-call-id", "controlled-disconnect")
|
||||
self.end_headers()
|
||||
self.wfile.write(f"{len(data):x}\\r\\n".encode() + data + b"\\r\\n")
|
||||
if not interrupted:
|
||||
self.wfile.write(b"0\\r\\n\\r\\n")
|
||||
self.wfile.flush()
|
||||
self.close_connection = True
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def endpoint():
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), InterruptedOnce)
|
||||
worker = Thread(target=server.serve_forever, daemon=True)
|
||||
worker.start()
|
||||
yield f"http://127.0.0.1:{server.server_port}"
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
worker.join(timeout=5)
|
||||
|
||||
@pytest.mark.parametrize("path", ["binary", "sse"])
|
||||
@pytest.mark.parametrize("mode", ["helper", "direct", "healthy"])
|
||||
def test_http_stream_would_recover_on_a_second_request(endpoint, path, mode):
|
||||
result = send(URL(f"{endpoint}/{mode}-{path}"), headers=NoBody(), json=NoBody(), stream=True, timeout=5)
|
||||
if mode == "direct":
|
||||
assert result.ok, f"stream was not established: {result}"
|
||||
else:
|
||||
require_successful_call(result)
|
||||
if path == "binary":
|
||||
assert_converse_stream(result)
|
||||
else:
|
||||
assert result.stream_done
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
|
@ -107,26 +180,36 @@ def test_stream_transport():
|
|||
check=False,
|
||||
)
|
||||
assert result.returncode == 1, result.stdout + result.stderr
|
||||
assert "7 failed, 2 passed, 7 rerun" in result.stdout, result.stdout
|
||||
assert "17 failed, 3 passed, 1 rerun" in result.stdout, result.stdout
|
||||
assert {path.stem: int(path.read_text()) for path in tmp_path.glob("*.attempts")} == {
|
||||
"recovers": 2,
|
||||
"persistent": 2,
|
||||
"recovers": 1,
|
||||
"persistent": 1,
|
||||
"proxy_503": 1,
|
||||
"auth": 1,
|
||||
"network": 2,
|
||||
"native": 2,
|
||||
"stream_recovers": 2,
|
||||
"stream_persistent": 2,
|
||||
"stream_transport": 2,
|
||||
"native": 1,
|
||||
"stream_recovers": 1,
|
||||
"stream_persistent": 1,
|
||||
"stream_transport": 1,
|
||||
"connection_recovers": 1,
|
||||
"behavior_recovers": 1,
|
||||
"connection_direct_recovers": 1,
|
||||
"header_marker_recovers": 1,
|
||||
"http_helper-binary": 1,
|
||||
"http_helper-sse": 1,
|
||||
"http_direct-binary": 1,
|
||||
"http_direct-sse": 1,
|
||||
"http_healthy-binary": 1,
|
||||
"http_healthy-sse": 1,
|
||||
}
|
||||
cases: Final = ET.parse(tmp_path / "results.xml").getroot()
|
||||
recovered: Final = cases.find(".//testcase[@name='test_recovers']")
|
||||
assert recovered is not None
|
||||
assert recovered.find("failure") is None
|
||||
assert recovered.find("failure") is not None
|
||||
assert recovered.find("properties/property[@name='upstream_request_id'][@value='aws-first-attempt']") is not None
|
||||
assert recovered.find("properties/property[@name='upstream_call_id'][@value='proxy-first-attempt']") is not None
|
||||
streamed: Final = cases.find(".//testcase[@name='test_stream_recovers']")
|
||||
assert streamed is not None and streamed.find("failure") is None
|
||||
assert streamed is not None and streamed.find("failure") is not None
|
||||
assert streamed.find("properties/property[@name='upstream_status_code'][@value='200']") is not None
|
||||
assert streamed.find("properties/property[@name='upstream_evidence'][@value='stream_event']") is not None
|
||||
assert streamed.find("properties/property[@name='upstream_request_id'][@value='stream-request']") is not None
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue