Address the Greptile concern that basic_messaging_streaming and
basic_messaging_non_streaming used the same implementation, so a proxy
that buffered the upstream stream would silently show green for the
streaming row.
The fix:
- _basic_messaging.run_basic_messaging_cell accepts verify_streaming=True,
which passes --include-partial-messages to the claude CLI. That flag
causes the CLI to emit one stream_event record per upstream SSE event
(message_start, content_block_delta, message_stop, ...). A buffering
proxy collapses the stream to a single non-streaming response, so
zero stream_event records are emitted.
- The cell rejects any model whose stream_event count is below
MIN_STREAM_DELTA_EVENTS (2) -- safely above the buffered case for any
non-trivial reply. Same all-must-pass shape as the existing
tool_use_streaming row.
- All five basic_messaging_streaming/test_*.py per-provider cells now
pass verify_streaming=True; the non-streaming variants are unchanged.
- New unit tests cover the helper, the partial-messages flag wiring,
the streamed/buffered branching, and the all-models-must-stream
contract.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
- conftest: pytest_runtest_makereport now early-returns on report.skipped
so pytest.skip(...) inside a compat test body doesn't get recorded as
a phantom 'fail' row via the not-failed/empty-collected branch.
- _basic_messaging: drop require_stream_events. The check (not outcome.events)
cannot catch a buffering regression because cli_driver uses
subprocess.run(capture_output=True), which only exposes the post-exit
stdout blob — buffered-then-flushed and truly streamed responses are
indistinguishable. The check was also unreachable as an independent
failure path (empty events -> empty text -> the text check fires first).
Update all five streaming callers and docstrings accordingly.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- run_claude_models_parallel: catch all exceptions in the per-model
worker and wrap unexpected ones into a ClaudeCLIError so the
documented 'errors as values' contract holds for OSError, ValueError,
etc., not just ClaudeCLIError. Without this, an unexpected raise in
any layer (rate limiter file I/O, infer_provider, etc.) abandons the
remaining models' results and crashes the calling test.
- test_run_claude_places_extra_args_before_prompt: drop the dead first
branch of the 'or' assertion — cmd[-3:] never matches that shape, so
the alternative was misleading dead code.
- basic_messaging_{non_streaming,streaming}/test_*.py: extract the
shared cell body into tests/claude_code/_basic_messaging.py.
Each per-provider file now declares its model list and calls
run_basic_messaging_cell(), eliminating ~700 lines of copy-paste
across 10 files. Updated _builder_unit_tests/test_v0_layout.py to
accept the helper-based pattern alongside direct run_claude() calls.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
