mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-22 00:31:44 +00:00
a1f0ef2a99
Address the Greptile concern that basic_messaging_streaming and basic_messaging_non_streaming used the same implementation, so a proxy that buffered the upstream stream would silently show green for the streaming row. The fix: - _basic_messaging.run_basic_messaging_cell accepts verify_streaming=True, which passes --include-partial-messages to the claude CLI. That flag causes the CLI to emit one stream_event record per upstream SSE event (message_start, content_block_delta, message_stop, ...). A buffering proxy collapses the stream to a single non-streaming response, so zero stream_event records are emitted. - The cell rejects any model whose stream_event count is below MIN_STREAM_DELTA_EVENTS (2) -- safely above the buffered case for any non-trivial reply. Same all-must-pass shape as the existing tool_use_streaming row. - All five basic_messaging_streaming/test_*.py per-provider cells now pass verify_streaming=True; the non-streaming variants are unchanged. - New unit tests cover the helper, the partial-messages flag wiring, the streamed/buffered branching, and the all-models-must-stream contract. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
46 lines
1.7 KiB
Python
46 lines
1.7 KiB
Python
"""basic_messaging_streaming x Anthropic.
|
|
|
|
Drive the real `claude` CLI in headless `--output-format stream-json
|
|
--include-partial-messages` mode against a running LiteLLM proxy that
|
|
routes to Anthropic, and report the outcome via `compat_result`.
|
|
|
|
The cell goes green only when every Claude tier (a) returns a non-empty
|
|
reply and (b) the proxy actually streamed it — i.e. the CLI observed
|
|
multiple `stream_event` records carrying raw SSE deltas. A proxy that
|
|
buffers the upstream stream and returns a single non-streaming chunk
|
|
emits zero such records, which is the regression this row catches.
|
|
|
|
The (feature, provider) for this cell is inferred from the file path by
|
|
`tests/claude_code/conftest.py`:
|
|
|
|
tests/claude_code/basic_messaging_streaming/test_anthropic.py
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^
|
|
feature_id provider
|
|
|
|
The shared `run_basic_messaging_cell` helper fans the three Claude tiers
|
|
out in parallel inside this single test, with one
|
|
`compat_result.add(...)` entry per model so the matrix builder still
|
|
sees three rows for this (feature, provider).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from tests.claude_code._basic_messaging import run_basic_messaging_cell
|
|
|
|
ANTHROPIC_MODELS = [
|
|
"claude-haiku-4-5",
|
|
"claude-sonnet-4-6",
|
|
"claude-opus-4-7",
|
|
]
|
|
|
|
|
|
def test_basic_messaging_streaming_anthropic(compat_result):
|
|
"""Drive the `claude` CLI against the LiteLLM proxy and assert a
|
|
non-empty streamed reply (one row per Claude tier).
|
|
"""
|
|
run_basic_messaging_cell(
|
|
compat_result=compat_result,
|
|
models=ANTHROPIC_MODELS,
|
|
prompt="Count from 1 to 5, one number per line.",
|
|
verify_streaming=True,
|
|
)
|