scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-24 00:52:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 91

Merge pull request #34358 from BerriAI/claude/e2e-tests-custom-endpoints-qxoi1o

Browse source 
test(e2e): replace custom endpoints_client with provider SDK clients
This commit is contained in:
Mateo Wang 2026-09-21 14:24:38 -07:00 • committed by GitHub
commit 124e5d6d53
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
29 changed files with 1254 additions and 1755 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
pyproject.toml
View file

@ -236,6 +236,7 @@ e2e-dev = [
"playwright==1.61.0",
"websockets>=15.0.1,<16.0",
"locust==2.45.0",
"anthropic==0.84.0",
"psutil==7.2.2",
"mcp>=2.2.0,<3",
]

2
tests/e2e/AGENTS.md
View file

@ -85,6 +85,8 @@ That snippet only conveys intent. What you actually write uses the real harness:
Every HTTP call goes through the shared transport, never through `requests.*` in a test. `e2e_http.py` is the only module permitted to call `requests.*`, and that is enforced in CI by `tests/code_coverage_tests/check_e2e_no_raw_requests.py`. A test that imports requests will fail the check
One deliberate exception: LLM-endpoint calls in `llm_translation/` go through the real provider SDKs (OpenAI, Anthropic) via the suite's `sdk` fixture (`llm_translation/sdk_clients.py`), because that is what customers actually run against the proxy (LIT-4577). The SDKs raise their own typed exceptions on failure, which is exactly the customer-observable contract; management routes (model/key CRUD, spend read-back) and endpoints no official SDK covers (e.g. `/v1/rerank`, `/v1/ocr`, custom passthrough paths) stay on the shared transport. Raw HTTP client imports remain banned either way
The shape is layered so tests stay declarative
`transport.py` exposes a `Transport` Protocol with `post`, `get`, `delete`, `send`, `stream`, `probe`, plus `bearer(key)` and the `master` header. `HttpTransport` fulfils it, and `SplitTransport` routes each call by path to the data plane or the control plane so a split control-plane/data-plane deployment works without any change in the test

8
tests/e2e/CONTRIBUTING.md
View file

@ -75,10 +75,10 @@ The suites run against a live proxy, so bring one up first by running the litell
Buildkite runs this suite against a Keycloak deployed beside the ephemeral stack by project-releaser. It fetches the realm from the test-runner revision even when it reuses a gateway image from another commit. The GitHub Actions changed-test stack starts the same digest-pinned Keycloak through `.github/e2e-stack/start-idp.sh`, imports the checked-out realm, and exports the IdP URL and credentials in `stack.env`. Both runners configure issuer/audience validation and store the realm, keys and users in a separate schema in the stack's PostgreSQL, so replacing Keycloak preserves token validity. Both wait for realm discovery before running tests. Losing the whole ephemeral database invalidates the stack. Keycloak skips imports into an existing realm, so changes to the realm export require a fresh stack (or deliberately replacing the local data volume). A stack without it fails the JWT tests rather than skipping them
4. Run a suite against it; the harness reads `LITELLM_PROXY_URL` (default `http://localhost:4000`):
4. Run a suite against it; the harness reads `LITELLM_PROXY_URL` (default `http://localhost:4000`). The suites' client dependencies (the provider SDKs, websockets) live in the `e2e-dev` dependency group; `make bootstrap` installs it, and naming the group on the run keeps the command working from any environment state:
```bash
uv run pytest tests/e2e/llm_translation/ -v
uv run --group e2e-dev pytest tests/e2e/llm_translation/ -v
```
The browser tests in the `management/` suite drive the dashboard the proxy serves at `/ui` through playwright, an optional dependency behind `importorskip` (the suite's API tests run without it). It lives in the `e2e-dev` dependency group; install it along with its browser:
@ -206,6 +206,8 @@ That snippet only conveys intent. What you actually write uses the real harness:
Every HTTP call goes through the shared transport, never through `requests.*` in a test. `e2e_http.py` is the only module permitted to call `requests.*`, and that is enforced in CI by `tests/code_coverage_tests/check_e2e_no_raw_requests.py`. A test that imports requests will fail the check
One deliberate exception: LLM-endpoint calls in `llm_translation/` go through the real provider SDKs (OpenAI, Anthropic) via the suite's `sdk` fixture (`llm_translation/sdk_clients.py`), because that is what customers actually run against the proxy (LIT-4577). Management routes and endpoints no official SDK covers stay on the shared transport, and raw HTTP client imports remain banned either way
The shape is layered so tests stay declarative
`transport.py` exposes a `Transport` Protocol with `post`, `get`, `delete`, `send`, `stream`, `probe`, plus `bearer(key)` and the `master` header. `HttpTransport` fulfils it, and `SplitTransport` routes each call by path to the data plane or the control plane so a split control-plane/data-plane deployment works without any change in the test
@ -230,7 +232,7 @@ Before you push
```bash
litellm --config <your-e2e-config>.yml --port 4000
uv run pytest tests/e2e/<your_suite>/ -v
uv run --group e2e-dev pytest tests/e2e/<your_suite>/ -v
```
4. Capture screenshots of the test run and attach them to the PR as proof

10
tests/e2e/llm_translation/conftest.py
View file

@ -2,14 +2,16 @@
The shared lifecycle (resources/scoped_key), proxy liveness gate, and e2e marker
live in the parent tests/e2e/conftest.py. PassthroughClient holds the shared
ProxyClient, so the `resources` fixture cleans up keys this suite creates.
ProxyClient, so the `resources` fixture cleans up keys this suite creates. The
`sdk` fixture hands tests real provider SDK clients (OpenAI, Anthropic) pointed
at the proxy, the way customers actually call it.
"""
import pytest
from endpoints_client import EndpointsClient, build_endpoints_client
from passthrough_client import PassthroughClient, build_client
from proxy_client import ProxyClient
from sdk_clients import SdkClients, build_sdk_clients
def pytest_configure(config: pytest.Config) -> None:
@ -25,5 +27,5 @@ def client(proxy: ProxyClient) -> PassthroughClient:
@pytest.fixture(scope="session")
def endpoints_client(proxy: ProxyClient) -> EndpointsClient:
return build_endpoints_client(proxy)
def sdk() -> SdkClients:
return build_sdk_clients()

476
tests/e2e/llm_translation/endpoints_client.py
View file

@ -1,476 +0,0 @@
"""Client for the non-chat inference endpoints (responses, messages, rerank,
embeddings, audio speech, image generation).
Each test registers the deployment it needs through /model/new (deleted on
teardown), so nothing is hardcoded into the gateway config, then drives the
endpoint with `send` and parses the provider-native body with a suite-local model
so the assertion is on real content, not just a 200.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
from e2e_config import SLOW_PROVIDER_TIMEOUT_SECONDS
from e2e_http import BinaryStream, Result, StreamingResponse
from models import CacheControl, ChatMessage, LiteLLMParamsBody, RichMessage, TextBlock
from proxy_client import ProxyClient
from pydantic import BaseModel
__all__ = [
"CacheControl",
"ImageEditForm",
"ImagesResult",
"RichMessage",
"TextBlock",
"TranscriptionForm",
"TranscriptionResult",
]
class FunctionParameterProperty(BaseModel):
type: str
description: str | None = None
class FunctionParameters(BaseModel):
type: Literal["object"] = "object"
properties: dict[str, FunctionParameterProperty]
required: list[str] = []
class ResponsesFunctionTool(BaseModel):
type: Literal["function"] = "function"
name: str
description: str | None = None
parameters: FunctionParameters
class ResponsesInputTextPart(BaseModel):
type: Literal["input_text"] = "input_text"
text: str
class ResponsesInputImagePart(BaseModel):
type: Literal["input_image"] = "input_image"
image_url: str
ResponsesInputContentPart = ResponsesInputTextPart | ResponsesInputImagePart
class ResponsesInputMessage(BaseModel):
role: Literal["user", "assistant", "system"] = "user"
content: list[ResponsesInputContentPart]
ResponsesInput = str | list[ResponsesInputMessage]
class ResponsesRequest(BaseModel):
model: str
input: ResponsesInput
instructions: str | None = None
stream: bool = False
tools: list[ResponsesFunctionTool] | None = None
guardrails: list[str] | None = None
safety_identifier: str | None = None
cache: dict[str, bool] | None = {"no-cache": True}
class MessagesRequest(BaseModel):
model: str
max_tokens: int
messages: list[ChatMessage]
cache: dict[str, bool] | None = {"no-cache": True}
class RichMessagesRequest(BaseModel):
model: str
max_tokens: int = 64
system: list[TextBlock]
messages: list[RichMessage]
cache: dict[str, bool] | None = {"no-cache": True}
class CompletionsRequest(BaseModel):
model: str
prompt: str
max_tokens: int = 32
cache: dict[str, bool] | None = {"no-cache": True}
class EmbeddingsRequest(BaseModel):
model: str
input: str
cache: dict[str, bool] | None = {"no-cache": True}
class RerankRequest(BaseModel):
model: str
query: str
documents: list[str]
top_n: int
cache: dict[str, bool] | None = {"no-cache": True}
class SpeechRequest(BaseModel):
model: str
input: str
voice: str
class ImageRequest(BaseModel):
model: str
prompt: str
n: int = 1
size: str = "1024x1024"
class ImageEditForm(BaseModel):
model: str
prompt: str
n: int = 1
class TranscriptionForm(BaseModel):
model: str
response_format: str = "json"
class ModerationRequest(BaseModel):
model: str
input: str
class GenerateContentPart(BaseModel):
text: str
class GenerateContentContent(BaseModel):
role: Literal["user"] = "user"
parts: tuple[GenerateContentPart, ...]
class GenerateContentBody(BaseModel):
contents: tuple[GenerateContentContent, ...]
class ResponsesOutputContent(BaseModel):
type: str | None = None
text: str | None = None
class ResponsesOutputItem(BaseModel):
type: str | None = None
content: list[ResponsesOutputContent] = []
name: str | None = None
arguments: str | None = None
call_id: str | None = None
class ResponsesResult(BaseModel):
id: str | None = None
status: str | None = None
model: str | None = None
output: list[ResponsesOutputItem] = []
@property
def text(self) -> str:
return "".join(
content.text or "" for item in self.output for content in item.content
)
@property
def function_calls(self) -> tuple[ResponsesOutputItem, ...]:
return tuple(
item
for item in self.output
if item.type == "function_call"
and item.name is not None
and item.arguments is not None
)
class ResponsesStreamEvent(BaseModel):
event_id: str | None = None
class ResponsesStreamEventType(BaseModel):
type: str
class ResponsesOutputTextDeltaEvent(ResponsesStreamEvent):
type: Literal["response.output_text.delta"]
delta: str
class AnthropicContentBlock(BaseModel):
type: str | None = None
text: str | None = None
class MessagesUsage(BaseModel):
input_tokens: int = 0
output_tokens: int = 0
cache_creation_input_tokens: int = 0
cache_read_input_tokens: int = 0
class MessagesResult(BaseModel):
id: str | None = None
role: str | None = None
model: str | None = None
content: list[AnthropicContentBlock] = []
usage: MessagesUsage = MessagesUsage()
@property
def text(self) -> str:
return "".join(block.text or "" for block in self.content)
class CompletionChoice(BaseModel):
text: str | None = None
class CompletionsResult(BaseModel):
choices: list[CompletionChoice] = []
class EmbeddingItem(BaseModel):
embedding: list[float] = []
class EmbeddingsResult(BaseModel):
data: list[EmbeddingItem] = []
@property
def first_vector(self) -> tuple[float, ...]:
return tuple(self.data[0].embedding) if self.data else ()
class RerankItem(BaseModel):
index: int | None = None
relevance_score: float | None = None
class RerankResult(BaseModel):
results: list[RerankItem] = []
class ImageItem(BaseModel):
url: str | None = None
b64_json: str | None = None
class ImagesResult(BaseModel):
data: list[ImageItem] = []
class TranscriptionResult(BaseModel):
text: str = ""
class ModerationResultItem(BaseModel):
flagged: bool
categories: dict[str, bool] = {}
@property
def flagged_categories(self) -> tuple[str, ...]:
return tuple(name for name, hit in self.categories.items() if hit)
class ModerationResult(BaseModel):
results: list[ModerationResultItem] = []
@property
def first(self) -> ModerationResultItem | None:
return self.results[0] if self.results else None
@dataclass(frozen=True, slots=True)
class EndpointsClient:
proxy: ProxyClient
def create_model(self, model_name: str, litellm_params: LiteLLMParamsBody) -> str:
return self.proxy.create_model(model_name, litellm_params)
def delete_model(self, model_id: str) -> None:
self.proxy.delete_model(model_id)
def _send(
self, path: str, key: str, body: BaseModel, *, stream: bool = False
) -> StreamingResponse:
return self.proxy.transport.send(
path,
headers=self.proxy.transport.bearer(key),
json=body,
stream=stream,
)
def responses(
self,
key: str,
model: str,
text: str,
*,
stream: bool = False,
guardrails: list[str] | None = None,
safety_identifier: str | None = None,
) -> StreamingResponse:
return self._send(
"/v1/responses",
key,
ResponsesRequest(
model=model,
input=text,
instructions="You are a helpful assistant",
stream=stream,
guardrails=guardrails,
safety_identifier=safety_identifier,
),
stream=stream,
)
def responses_vision(
self, key: str, model: str, text: str, image_url: str
) -> StreamingResponse:
return self._send(
"/v1/responses",
key,
ResponsesRequest(
model=model,
input=[
ResponsesInputMessage(
content=[
ResponsesInputTextPart(text=text),
ResponsesInputImagePart(image_url=image_url),
]
)
],
instructions="You are a helpful assistant",
),
)
def responses_with_tools(
self, key: str, model: str, text: str, tools: list[ResponsesFunctionTool]
) -> StreamingResponse:
return self._send(
"/v1/responses",
key,
ResponsesRequest(
model=model,
input=text,
instructions="You are a helpful assistant",
tools=tools,
),
)
def messages(
self, key: str, model: str, text: str, *, max_tokens: int = 64
) -> StreamingResponse:
return self._send(
"/v1/messages",
key,
MessagesRequest(
model=model,
max_tokens=max_tokens,
messages=[ChatMessage(role="user", content=text)],
),
)
def text_completions(
self, key: str, model: str, prompt: str, *, max_tokens: int = 32
) -> StreamingResponse:
return self._send(
"/v1/completions",
key,
CompletionsRequest(model=model, prompt=prompt, max_tokens=max_tokens),
)
def embeddings(self, key: str, model: str, text: str) -> StreamingResponse:
return self._send("/embeddings", key, EmbeddingsRequest(model=model, input=text))
def rerank(
self, key: str, model: str, query: str, documents: list[str], top_n: int
) -> StreamingResponse:
return self._send(
"/v1/rerank",
key,
RerankRequest(model=model, query=query, documents=documents, top_n=top_n),
)
def audio_speech(
self, key: str, model: str, text: str, *, voice: str = "alloy"
) -> StreamingResponse:
return self._send(
"/v1/audio/speech", key, SpeechRequest(model=model, input=text, voice=voice)
)
def audio_speech_stream(
self, key: str, model: str, text: str, *, voice: str = "alloy"
) -> BinaryStream:
return self.proxy.transport.stream_binary(
"/v1/audio/speech",
headers=self.proxy.transport.bearer(key),
json=SpeechRequest(model=model, input=text, voice=voice),
)
def transcribe(
self, key: str, model: str, *, filename: str, content: bytes
) -> Result[TranscriptionResult]:
return self.proxy.transport.upload(
"/v1/audio/transcriptions",
headers=self.proxy.transport.bearer(key),
form=TranscriptionForm(model=model),
filename=filename,
content=content,
file_content_type="audio/wav",
response_type=TranscriptionResult,
)
def moderations(self, key: str, model: str, text: str) -> Result[ModerationResult]:
return self.proxy.transport.post(
"/v1/moderations",
headers=self.proxy.transport.bearer(key),
json=ModerationRequest(model=model, input=text),
response_type=ModerationResult,
)
def images(self, key: str, model: str, prompt: str) -> StreamingResponse:
return self._send(
"/v1/images/generations", key, ImageRequest(model=model, prompt=prompt)
)
def image_edit(
self, key: str, model: str, prompt: str, image: bytes, *, filename: str = "image.png"
) -> Result[ImagesResult]:
return self.proxy.transport.upload(
"/v1/images/edits",
headers=self.proxy.transport.bearer(key),
form=ImageEditForm(model=model, prompt=prompt),
filename=filename,
content=image,
file_content_type="image/png",
file_field="image",
response_type=ImagesResult,
timeout=SLOW_PROVIDER_TIMEOUT_SECONDS,
)
def generate_content(
self, key: str, model: str, text: str, *, stream: bool = False
) -> StreamingResponse:
operation = "streamGenerateContent" if stream else "generateContent"
return self._send(
f"/v1beta/models/{model}:{operation}",
key,
GenerateContentBody(
contents=(GenerateContentContent(parts=(GenerateContentPart(text=text),)),)
),
stream=stream,
)
def build_endpoints_client(proxy: ProxyClient) -> EndpointsClient:
return EndpointsClient(proxy=proxy)

62
tests/e2e/llm_translation/sdk_clients.py Normal file
View file

@ -0,0 +1,62 @@
"""Real provider SDK clients pointed at the proxy, connected the way customers
connect (LIT-4577).
The OpenAI SDK drives the OpenAI-compatible surface (/responses, /embeddings,
/images/generations, /moderations, /audio/*) and the Anthropic SDK drives
/v1/messages, each authenticated with a litellm virtual key. Errors surface as
the SDK's own exceptions, exactly what an end user sees. Retries are disabled
so a proxy fault fails the test instead of being papered over, and the timeout
matches the shared transport's request budget.
"""
from __future__ import annotations
from collections.abc import Mapping
from dataclasses import dataclass
from types import MappingProxyType
from typing import Final
from anthropic import Anthropic
from openai import OpenAI
from e2e_config import PROXY_BASE_URL, REQUEST_TIMEOUT
NO_PROXY_CACHE: Final = MappingProxyType({"cache": {"no-cache": True}})
"""``extra_body`` for every cacheable SDK call (messages, responses, completions,
embeddings): the gateway under test caches those call types, so an identical
re-send would otherwise be served from Redis instead of reaching the provider,
which hides provider-side behavior such as prompt-cache warm-up. The SDKs
themselves cannot bypass it (``Cache-Control`` only sets a TTL on the proxy)."""
def response_header(headers: Mapping[str, str], name: str) -> str | None:
"""Typed read of an SDK response header: httpx.Headers.get returns Any and
httpx itself is a banned import in suite code, so tests read headers through
the Mapping[str, str] interface Headers fulfils."""
return headers[name] if name in headers else None
@dataclass(frozen=True, slots=True)
class SdkClients:
base_url: str
request_timeout: float
def openai(self, key: str) -> OpenAI:
return OpenAI(
base_url=self.base_url,
api_key=key,
timeout=self.request_timeout,
max_retries=0,
)
def anthropic(self, key: str) -> Anthropic:
return Anthropic(
base_url=self.base_url,
api_key=key,
timeout=self.request_timeout,
max_retries=0,
)
def build_sdk_clients() -> SdkClients:
return SdkClients(base_url=PROXY_BASE_URL, request_timeout=REQUEST_TIMEOUT)

117
tests/e2e/llm_translation/test_audio_speech_e2e.py
View file

@ -1,20 +1,23 @@
"""Live e2e: POST /v1/audio/speech returns audio, non-streamed and streamed.
The non-streamed call asserts an audio (not JSON) body. The streamed call consumes
the response the way a player would and asserts customer-observable streaming:
chunked transfer encoding (a buffered body would carry a content-length) with
non-zero audio bytes.
Both positive calls go through the real OpenAI SDK (LIT-4577). The non-streamed
call asserts an audio (not JSON) body. The streamed call consumes the response
the way a player would and asserts customer-observable streaming: chunked
transfer encoding (a buffered body would carry a content-length) with non-zero
audio bytes. The malformed-body negatives stay on the shared transport because
the SDK refuses to send a request missing its required fields.
"""
from __future__ import annotations
import pytest
from e2e_config import unique_marker
from e2e_http import assert_client_error, require_successful_call
from endpoints_client import EndpointsClient
from e2e_http import assert_client_error
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from pydantic import BaseModel
from sdk_clients import SdkClients, response_header
pytestmark = pytest.mark.e2e
@ -25,67 +28,75 @@ class _OptionalSpeechBody(BaseModel):
voice: str | None = None
def _register_tts(
endpoints_client: EndpointsClient, resources: ResourceManager
) -> tuple[str, str]:
def _register_tts(proxy: ProxyClient, resources: ResourceManager) -> tuple[str, str]:
model = f"e2e-speech-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(model="openai/gpt-4o-mini-tts", api_key="os.environ/OPENAI_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
return model, resources.key()
class TestAudioSpeech:
@pytest.mark.covers("llm.audio_speech.openai.basic.nonstream.works")
def test_audio_speech_returns_audio(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model, key = _register_tts(endpoints_client, resources)
result = endpoints_client.audio_speech(key, model, "Hello!")
require_successful_call(result)
assert "audio" in (result.content_type or ""), (
f"/audio/speech content-type is not audio: {result.content_type!r}"
model, key = _register_tts(proxy, resources)
client = sdk.openai(key)
response = client.audio.speech.with_raw_response.create(
model=model, voice="alloy", input="Hello!"
)
assert result.body, "/audio/speech returned an empty body"
content_type = response_header(response.headers, "content-type")
assert "audio" in (content_type or ""), (
f"/audio/speech content-type is not audio: {content_type!r}"
)
assert response.content, "/audio/speech returned an empty body"
@pytest.mark.covers("llm.audio_speech.openai.basic.stream.works")
def test_audio_speech_streams_audio_chunks(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model, key = _register_tts(endpoints_client, resources)
result = endpoints_client.audio_speech_stream(
key,
model,
"Streaming speech should arrive in several audio chunks so a client can "
"begin playback well before the whole clip has finished generating.",
model, key = _register_tts(proxy, resources)
client = sdk.openai(key)
with client.audio.speech.with_streaming_response.create(
model=model,
voice="alloy",
input=(
"Streaming speech should arrive in several audio chunks so a client can "
"begin playback well before the whole clip has finished generating."
),
) as response:
content_type = response_header(response.headers, "content-type")
transfer_encoding = response_header(response.headers, "transfer-encoding")
content_length = response_header(response.headers, "content-length")
total_bytes = sum(len(chunk) for chunk in response.iter_bytes(chunk_size=8192))
assert "audio" in (content_type or ""), (
f"/audio/speech content-type is not audio: {content_type!r}"
)
assert result.ok, (
f"/audio/speech stream failed (status {result.status_code}); body={result.error_body}"
assert "chunked" in (transfer_encoding or ""), (
f"/audio/speech did not stream: transfer-encoding={transfer_encoding!r}, "
f"content-length={content_length!r} (a buffered body is not a stream)"
)
assert "audio" in (result.content_type or ""), (
f"/audio/speech content-type is not audio: {result.content_type!r}"
)
assert result.chunked, (
f"/audio/speech did not stream: transfer-encoding={result.transfer_encoding!r}, "
f"content-length={result.content_length!r} (a buffered body is not a stream)"
)
assert result.content_length is None, (
f"/audio/speech advertised content-length={result.content_length!r} on a "
assert content_length is None, (
f"/audio/speech advertised content-length={content_length!r} on a "
f"streamed response (a buffered body is not a stream)"
)
assert result.total_bytes > 0, "/audio/speech stream returned no audio bytes"
assert total_bytes > 0, "/audio/speech stream returned no audio bytes"
@pytest.mark.skip(reason="stage red: product gap, /v1/audio/speech 500s on missing input instead of 400")
@pytest.mark.covers("llm.audio_speech.openai.input_validation.nonstream.works")
def test_missing_input_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
model, key = _register_tts(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
model, key = _register_tts(proxy, resources)
result = proxy.transport.send(
"/v1/audio/speech",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalSpeechBody(model=model, voice="alloy"),
)
assert_client_error(result, "speech missing input")
@ -93,12 +104,12 @@ class TestAudioSpeech:
@pytest.mark.skip(reason="stage red: product gap, /v1/audio/speech 500s on missing model instead of 400")
@pytest.mark.covers("llm.audio_speech.openai.input_validation.nonstream.works")
def test_missing_model_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
_, key = _register_tts(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
_, key = _register_tts(proxy, resources)
result = proxy.transport.send(
"/v1/audio/speech",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalSpeechBody(input="hello", voice="alloy"),
)
assert_client_error(result, "speech missing model")
@ -106,12 +117,12 @@ class TestAudioSpeech:
@pytest.mark.skip(reason="stage red: product gap, /v1/audio/speech 500s on invalid voice instead of surfacing the provider 4xx")
@pytest.mark.covers("llm.audio_speech.openai.input_validation.nonstream.works")
def test_invalid_voice_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
model, key = _register_tts(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
model, key = _register_tts(proxy, resources)
result = proxy.transport.send(
"/v1/audio/speech",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalSpeechBody(model=model, input="hello", voice="invalid_voice_xyz"),
)
assert_client_error(result, "speech invalid voice")
@ -119,12 +130,12 @@ class TestAudioSpeech:
@pytest.mark.skip(reason="stage red: product gap, /v1/audio/speech 500s on empty input instead of surfacing the provider 4xx")
@pytest.mark.covers("llm.audio_speech.openai.input_validation.nonstream.works")
def test_empty_input_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
model, key = _register_tts(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
model, key = _register_tts(proxy, resources)
result = proxy.transport.send(
"/v1/audio/speech",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalSpeechBody(model=model, input="", voice="alloy"),
)
assert_client_error(result, "speech empty input")

66
tests/e2e/llm_translation/test_audio_transcriptions_e2e.py
View file

@ -1,12 +1,13 @@
"""Live e2e: POST /v1/audio/transcriptions turns speech into text (vendor §9.7 / LIT-4778).
Registers an OpenAI speech-to-text deployment at runtime and uploads a spoken
weather question (the realtime suite's 24kHz WAV fixture) as multipart, asserting
the returned transcript is non-empty and mentions the word it was asked about.
Also pins missing file/model negatives. A model-less request comes back as one of
two 400s depending on whether any wildcard deployment happens to be registered on
the shared proxy, so the assertion accepts either phrasing and holds both to naming
the model as the problem.
weather question (the realtime suite's 24kHz WAV fixture) through the real
OpenAI SDK (LIT-4577), asserting the returned transcript is non-empty and
mentions the word it was asked about. Also pins missing file/model negatives on
the shared multipart transport, since the SDK refuses to send them. A model-less
request comes back as one of two 400s depending on whether any wildcard
deployment happens to be registered on the shared proxy, so the assertion
accepts either phrasing and holds both to naming the model as the problem.
"""
from __future__ import annotations
@ -16,11 +17,12 @@ from typing import Final
import pytest
from e2e_config import unique_marker
from e2e_http import UnknownApiError, unwrap
from endpoints_client import EndpointsClient, TranscriptionForm, TranscriptionResult
from e2e_http import UnknownApiError
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from pydantic import BaseModel
from sdk_clients import SdkClients
pytestmark = pytest.mark.e2e
@ -36,32 +38,34 @@ class _OptionalTranscriptionForm(BaseModel):
response_format: str = "json"
def _register(
endpoints_client: EndpointsClient, resources: ResourceManager
) -> tuple[str, str]:
class _TranscriptionResult(BaseModel):
text: str = ""
def _register(proxy: ProxyClient, resources: ResourceManager) -> tuple[str, str]:
model = f"e2e-transcribe-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(
model="openai/gpt-4o-mini-transcribe", api_key="os.environ/OPENAI_API_KEY"
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
return model, resources.key()
class TestAudioTranscriptions:
@pytest.mark.covers("llm.audio_transcriptions.openai.basic.nonstream.works")
def test_audio_transcriptions_returns_text(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model, key = _register(endpoints_client, resources)
result = unwrap(
endpoints_client.transcribe(
key, model, filename=WEATHER_WAV.name, content=WEATHER_WAV.read_bytes()
)
model, key = _register(proxy, resources)
client = sdk.openai(key)
transcription = client.audio.transcriptions.create(
model=model, file=(WEATHER_WAV.name, WEATHER_WAV.read_bytes(), "audio/wav")
)
text = result.text.strip()
text = transcription.text.strip()
assert text, "/audio/transcriptions returned an empty transcript"
assert "weather" in text.lower(), (
f"transcript of a spoken weather question does not mention weather: {text!r}"
@ -69,17 +73,17 @@ class TestAudioTranscriptions:
@pytest.mark.covers("llm.audio_transcriptions.openai.input_validation.nonstream.works")
def test_missing_file_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
model, key = _register(endpoints_client, resources)
result = endpoints_client.proxy.transport.upload(
model, key = _register(proxy, resources)
result = proxy.transport.upload(
"/v1/audio/transcriptions",
headers=endpoints_client.proxy.transport.bearer(key),
form=TranscriptionForm(model=model),
headers=proxy.transport.bearer(key),
form=_OptionalTranscriptionForm(model=model),
filename="empty.wav",
content=b"",
file_content_type="audio/wav",
response_type=TranscriptionResult,
response_type=_TranscriptionResult,
)
match result:
case UnknownApiError(status_code=400, body=body):
@ -95,17 +99,17 @@ class TestAudioTranscriptions:
@pytest.mark.covers("llm.audio_transcriptions.openai.input_validation.nonstream.works")
def test_missing_model_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
_, key = _register(endpoints_client, resources)
result = endpoints_client.proxy.transport.upload(
_, key = _register(proxy, resources)
result = proxy.transport.upload(
"/v1/audio/transcriptions",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
form=_OptionalTranscriptionForm(),
filename=WEATHER_WAV.name,
content=WEATHER_WAV.read_bytes(),
file_content_type="audio/wav",
response_type=TranscriptionResult,
response_type=_TranscriptionResult,
)
match result:
case UnknownApiError(status_code=400, body=body):

49
tests/e2e/llm_translation/test_bedrock_web_search_server_tool_e2e.py
View file

@ -34,27 +34,22 @@ block alone does not activate it.
from __future__ import annotations
import pytest
from anthropic.types import WebSearchTool20250305Param
from e2e_config import unique_marker
from e2e_http import unwrap
from endpoints_client import EndpointsClient
from lifecycle import ResourceManager
from models import (
AnthropicMessagesBody,
AnthropicWebSearchTool,
ChatMessage,
LiteLLMParamsBody,
)
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from sdk_clients import NO_PROXY_CACHE, SdkClients
pytestmark = pytest.mark.e2e
BEDROCK_INVOKE_BACKEND = "bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0"
WEB_SEARCH_TOOL = AnthropicWebSearchTool(
type="web_search_20250305",
name="web_search",
max_uses=3,
)
WEB_SEARCH_TOOL: WebSearchTool20250305Param = {
"type": "web_search_20250305",
"name": "web_search",
"max_uses": 3,
}
SEARCH_PROMPT = "Use web search to tell me one recent news headline about Anthropic."
@ -68,34 +63,30 @@ class TestBedrockWebSearchServerTool:
)
@pytest.mark.covers("llm.messages.bedrock_invoke.web_search_server_tool.nonstream.works")
def test_web_search_server_tool_is_served(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
"""A bedrock deployment must answer a web_search server-tool request
instead of handing the tool to AWS and returning its 400."""
model = f"e2e-bedrock-websearch-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(
model=BEDROCK_INVOKE_BACKEND,
aws_region_name="us-east-1",
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
resources.defer(lambda: proxy.delete_model(model_id))
client = sdk.anthropic(resources.key())
response = unwrap(
endpoints_client.proxy.messages(
key,
AnthropicMessagesBody(
model=model,
max_tokens=512,
tools=[WEB_SEARCH_TOOL],
messages=[ChatMessage(role="user", content=SEARCH_PROMPT)],
),
)
response = client.messages.create(
model=model,
max_tokens=512,
tools=[WEB_SEARCH_TOOL],
messages=[{"role": "user", "content": SEARCH_PROMPT}],
extra_body=NO_PROXY_CACHE,
)
assert response.content, f"no content blocks in response: {response}"
assert response.content, f"no content blocks in response: {response!r}"
block_types = [block.type for block in response.content]
assert "web_search_tool_result" in block_types, (
"the answer carries no web_search_tool_result block, so the search "

5
tests/e2e/llm_translation/test_cache_control.py
View file

@ -24,7 +24,7 @@ service_tier lives in test_provider_features_e2e.py.
The provider-native cache_control request shape is not expressible with the
shared ``ChatBody`` (whose content is a plain string), so the cacheable body is
built from the typed content blocks shared in ``endpoints_client.py``.
built from the typed content blocks shared in ``models.py``.
"""
from __future__ import annotations
@ -38,9 +38,8 @@ from pydantic import BaseModel
from e2e_config import unique_marker
from e2e_http import Result, UnknownApiError, unwrap
from endpoints_client import CacheControl, RichMessage, TextBlock
from lifecycle import ResourceManager
from models import ChatBody, ChatMessage, ChatResponse, LiteLLMParamsBody, Usage
from models import CacheControl, ChatBody, ChatMessage, ChatResponse, LiteLLMParamsBody, RichMessage, TextBlock, Usage
from passthrough_client import PassthroughClient
import os

33
tests/e2e/llm_translation/test_completions_endpoint_e2e.py
View file

@ -3,19 +3,19 @@
The legacy text-completion endpoint (prompt-style, non-chat) is the second-busiest
route in production yet was previously uncovered; the rest of the "completions"
surface is chat only. Registers an OpenAI instruct deployment at runtime (deleted
on teardown), drives /v1/completions through the gateway, and asserts real
generated text came back so a regression that empties the completion fails here.
on teardown), drives /v1/completions through the gateway with the real OpenAI SDK
(LIT-4577), and asserts real generated text came back so a regression that empties
the completion fails here.
"""
from __future__ import annotations
import pytest
from e2e_config import unique_marker
from e2e_http import require_successful_call
from endpoints_client import CompletionsResult, EndpointsClient
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from sdk_clients import NO_PROXY_CACHE, SdkClients
pytestmark = pytest.mark.e2e
@ -23,24 +23,25 @@ pytestmark = pytest.mark.e2e
class TestCompletionsEndpoint:
@pytest.mark.covers("llm.completions.openai.basic.nonstream.works")
def test_text_completion_returns_text(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-completions-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(
model="text-completion-openai/gpt-3.5-turbo-instruct",
api_key="os.environ/OPENAI_API_KEY",
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
resources.defer(lambda: proxy.delete_model(model_id))
client = sdk.openai(resources.key())
result = endpoints_client.text_completions(
key, model, "Finish this sentence in a few words: the capital of France is"
completion = client.completions.create(
model=model,
prompt="Finish this sentence in a few words: the capital of France is",
max_tokens=32,
extra_body=NO_PROXY_CACHE,
)
require_successful_call(result)
parsed = CompletionsResult.model_validate_json(result.body)
assert parsed.choices, f"/v1/completions returned no choices: {result.body[:300]}"
completion = (parsed.choices[0].text or "").strip()
assert completion, f"/v1/completions returned an empty completion: {result.body[:300]}"
assert completion.choices, f"/v1/completions returned no choices: {completion!r}"
text = (completion.choices[0].text or "").strip()
assert text, f"/v1/completions returned an empty completion: {completion!r}"

30
tests/e2e/llm_translation/test_credential_messages_e2e.py
View file

@ -7,43 +7,47 @@ import os
import pytest
from e2e_config import unique_marker
from e2e_http import require_successful_call
from endpoints_client import EndpointsClient, MessagesResult
from lifecycle import ResourceManager
from models import CredentialCreateBody, LiteLLMParamsBody
from proxy_client import ProxyClient
from sdk_clients import NO_PROXY_CACHE, SdkClients
pytestmark = pytest.mark.e2e
class TestCredentialBackedMessages:
@pytest.mark.covers("mgmt.credential.new.serves_request")
def test_credential_backed_messages(self, endpoints_client: EndpointsClient, resources: ResourceManager) -> None:
def test_credential_backed_messages(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
marker = unique_marker()
credential_name = f"e2e-cred-{marker}"
model = f"e2e-cred-messages-{marker}"
anthropic_api_key = os.getenv("ANTHROPIC_API_KEY")
assert anthropic_api_key, "ANTHROPIC_API_KEY must be set for this live e2e test"
endpoints_client.proxy.create_credential(
proxy.create_credential(
CredentialCreateBody(
credential_name=credential_name,
credential_values={"api_key": anthropic_api_key},
)
)
resources.defer(lambda: endpoints_client.proxy.delete_credential(credential_name))
resources.defer(lambda: proxy.delete_credential(credential_name))
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(
model="anthropic/claude-haiku-4-5",
litellm_credential_name=credential_name,
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
key = resources.key()
result = endpoints_client.messages(key, model, "reply with one word")
require_successful_call(result)
parsed = MessagesResult.model_validate_json(result.body)
assert parsed.role == "assistant", f"unexpected role: {result.body[:300]}"
assert parsed.text.strip(), f"/v1/messages returned no text: {result.body[:300]}"
client = sdk.anthropic(resources.key())
message = client.messages.create(
model=model,
max_tokens=64,
messages=[{"role": "user", "content": "reply with one word"}],
extra_body=NO_PROXY_CACHE,
)
assert message.role == "assistant", f"unexpected role: {message.role!r}"
text = "".join(block.text for block in message.content if block.type == "text")
assert text.strip(), f"/v1/messages returned no text: {message.content!r}"

33
tests/e2e/llm_translation/test_custom_pricing_e2e.py
View file

@ -25,7 +25,6 @@ from pydantic import BaseModel, RootModel
from e2e_config import unique_marker
from proxy_client import ProxyClient
from e2e_http import Success, unwrap
from endpoints_client import EndpointsClient
from lifecycle import ResourceManager
from models import (
ChatBody,
@ -71,7 +70,7 @@ def _approx_equal(actual: float, expected: float) -> bool:
def _provision(
endpoints_client: EndpointsClient,
proxy: ProxyClient,
resources: ResourceManager,
prefix: str,
*,
@ -84,7 +83,7 @@ def _provision(
marker keeps the name unique so concurrent runs on the shared proxy never
collide."""
model_name = f"{prefix}-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model_name,
LiteLLMParamsBody(
model=BACKEND_MODEL,
@ -93,15 +92,15 @@ def _provision(
output_cost_per_token=output_cost_per_token,
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
return model_name
def _provision_custom_priced(
endpoints_client: EndpointsClient, resources: ResourceManager
proxy: ProxyClient, resources: ResourceManager
) -> str:
return _provision(
endpoints_client,
proxy,
resources,
"custom-priced-flash",
input_cost_per_token=CUSTOM_INPUT_RATE,
@ -151,14 +150,14 @@ def _poll_breakdown_row(proxy: ProxyClient, key: str, response_id: str | None) -
class TestCustomPricing:
def test_custom_pricing_is_billed_at_configured_rate(
self,
endpoints_client: EndpointsClient,
proxy: ProxyClient,
resources: ResourceManager,
scoped_key: str,
) -> None:
model = _provision_custom_priced(endpoints_client, resources)
model = _provision_custom_priced(proxy, resources)
chat = unwrap(
endpoints_client.proxy.chat(
proxy.chat(
scoped_key,
ChatBody(
model=model,
@ -172,7 +171,7 @@ class TestCustomPricing:
)
)
row = _poll_breakdown_row(endpoints_client.proxy, scoped_key, chat.id)
row = _poll_breakdown_row(proxy, scoped_key, chat.id)
assert row.metadata and row.metadata.cost_breakdown # guaranteed by the poll
breakdown = row.metadata.cost_breakdown
@ -195,10 +194,10 @@ class TestCustomPricing:
)
def test_model_info_reports_custom_pricing(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
model = _provision_custom_priced(endpoints_client, resources)
entry = _model_info_entry(endpoints_client.proxy.model_info(), model)
model = _provision_custom_priced(proxy, resources)
entry = _model_info_entry(proxy.model_info(), model)
assert entry.litellm_params.input_cost_per_token == CUSTOM_INPUT_RATE, (
f"/model/info litellm_params input rate "
@ -210,20 +209,20 @@ class TestCustomPricing:
)
def test_custom_pricing_is_isolated_from_sibling_deployment(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
# Register the override first so its rate is in the backend cost map before
# the sibling resolves; a leak (LIT-3897) would then poison the sibling.
custom = _provision_custom_priced(endpoints_client, resources)
custom = _provision_custom_priced(proxy, resources)
sibling = _provision(
endpoints_client,
proxy,
resources,
"base-flash",
input_cost_per_token=None,
output_cost_per_token=None,
)
entries = {entry.model_name: entry for entry in endpoints_client.proxy.model_info()}
entries = {entry.model_name: entry for entry in proxy.model_info()}
custom_entry = entries.get(custom)
sibling_entry = entries.get(sibling)
assert custom_entry is not None, f"{custom} absent from /model/info"

170
tests/e2e/llm_translation/test_embeddings_endpoint_e2e.py
View file

@ -1,23 +1,23 @@
"""Live e2e: POST /embeddings returns a real vector across OpenAI, Bedrock, Vertex, Cohere.
Each test registers the deployment it needs at runtime (deleted on teardown) and
asserts a non-empty, non-zero vector came back. The LIT-3167 guard in
tests/e2e/embeddings/ covers the Gemini embedding path; embeddings cost tracking is
covered by tests/e2e/quota_management/spend_tracking/.
Each test registers the deployment it needs at runtime (deleted on teardown),
drives the endpoint with the real OpenAI SDK (LIT-4577), and asserts a
non-empty, non-zero vector came back. The LIT-3167 guard in
tests/e2e/embeddings/ covers the Gemini embedding path; embeddings cost tracking
is covered by tests/e2e/quota_management/spend_tracking/. Malformed bodies the
SDK refuses to build stay on the shared transport.
"""
from __future__ import annotations
import pytest
from e2e_config import provider_edge_base, unique_marker
from e2e_http import (
assert_client_error,
require_successful_call,
)
from endpoints_client import EmbeddingsResult, EndpointsClient
from e2e_http import assert_client_error
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from pydantic import BaseModel
from sdk_clients import NO_PROXY_CACHE, SdkClients
pytestmark = pytest.mark.e2e
@ -39,35 +39,47 @@ def _openai_embeddings_params() -> LiteLLMParamsBody:
)
def _register(
proxy: ProxyClient, resources: ResourceManager, prefix: str, params: LiteLLMParamsBody
) -> tuple[str, str]:
model = f"{prefix}-{unique_marker()}"
model_id = proxy.create_model(model, params)
resources.defer(lambda: proxy.delete_model(model_id))
return model, resources.key()
def _assert_embedding_vector(
proxy: ProxyClient,
resources: ResourceManager,
sdk: SdkClients,
prefix: str,
params: LiteLLMParamsBody,
) -> None:
model, key = _register(proxy, resources, prefix, params)
client = sdk.openai(key)
embeddings = client.embeddings.create(model=model, input="Say this is a test!", extra_body=NO_PROXY_CACHE)
assert embeddings.data, f"/embeddings returned no data: {embeddings!r}"
vector = embeddings.data[0].embedding
assert vector, f"/embeddings returned no vector: {embeddings!r}"
assert any(component != 0.0 for component in vector), "embedding vector is all zeros"
class TestEmbeddingsEndpoint:
@pytest.mark.replayable
@pytest.mark.covers("llm.embeddings.openai.basic.nonstream.works")
def test_embeddings_returns_vector(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model = f"e2e-embeddings-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
_openai_embeddings_params(),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
result = endpoints_client.embeddings(key, model, "Say this is a test!")
require_successful_call(result)
parsed = EmbeddingsResult.model_validate_json(result.body)
assert parsed.first_vector, f"/embeddings returned no vector: {result.body[:300]}"
assert any(component != 0.0 for component in parsed.first_vector), (
f"embedding vector is all zeros: {result.body[:300]}"
)
def test_embeddings_returns_vector(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
_assert_embedding_vector(proxy, resources, sdk, "e2e-embeddings", _openai_embeddings_params())
@pytest.mark.covers("llm.embeddings.bedrock.basic.nonstream.works")
def test_bedrock_embeddings_returns_vector(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-embeddings-bedrock-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
_assert_embedding_vector(
proxy,
resources,
sdk,
"e2e-embeddings-bedrock",
LiteLLMParamsBody(
model="bedrock/amazon.titan-embed-text-v2:0",
aws_access_key_id="os.environ/AWS_ACCESS_KEY_ID",
@ -75,110 +87,62 @@ class TestEmbeddingsEndpoint:
aws_region_name="os.environ/AWS_REGION",
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
result = endpoints_client.embeddings(key, model, "Say this is a test!")
require_successful_call(result)
parsed = EmbeddingsResult.model_validate_json(result.body)
assert parsed.first_vector, f"/embeddings returned no vector: {result.body[:300]}"
assert any(component != 0.0 for component in parsed.first_vector), (
f"embedding vector is all zeros: {result.body[:300]}"
)
@pytest.mark.covers("llm.embeddings.cohere.basic.nonstream.works")
def test_cohere_embeddings_returns_vector(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-embeddings-cohere-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
_assert_embedding_vector(
proxy,
resources,
sdk,
"e2e-embeddings-cohere",
LiteLLMParamsBody(model="cohere/embed-v4.0", api_key="os.environ/COHERE_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
result = endpoints_client.embeddings(key, model, "Say this is a test!")
require_successful_call(result)
parsed = EmbeddingsResult.model_validate_json(result.body)
assert parsed.first_vector, f"/embeddings returned no vector: {result.body[:300]}"
assert any(component != 0.0 for component in parsed.first_vector), (
f"embedding vector is all zeros: {result.body[:300]}"
)
@pytest.mark.covers("llm.embeddings.vertex.basic.nonstream.works")
def test_vertex_embeddings_returns_vector(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-embeddings-vertex-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
_assert_embedding_vector(
proxy,
resources,
sdk,
"e2e-embeddings-vertex",
LiteLLMParamsBody(
model="vertex_ai/text-embedding-005",
vertex_project="os.environ/VERTEXAI_PROJECT",
vertex_location="us-central1",
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
result = endpoints_client.embeddings(key, model, "Say this is a test!")
require_successful_call(result)
parsed = EmbeddingsResult.model_validate_json(result.body)
assert parsed.first_vector, f"/embeddings returned no vector: {result.body[:300]}"
assert any(component != 0.0 for component in parsed.first_vector), (
f"embedding vector is all zeros: {result.body[:300]}"
)
@pytest.mark.replayable
@pytest.mark.covers("llm.embeddings.openai.basic.nonstream.works")
def test_array_input_returns_vectors(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model = f"e2e-embeddings-array-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
_openai_embeddings_params(),
def test_array_input_returns_vectors(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model, key = _register(proxy, resources, "e2e-embeddings-array", _openai_embeddings_params())
embeddings = sdk.openai(key).embeddings.create(
model=model, input=["Hello", "World", "Test"], extra_body=NO_PROXY_CACHE
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
result = endpoints_client.proxy.transport.send(
"/embeddings",
headers=endpoints_client.proxy.transport.bearer(key),
json=_OptionalEmbeddingsBody(model=model, input=["Hello", "World", "Test"]),
)
require_successful_call(result)
parsed = EmbeddingsResult.model_validate_json(result.body)
assert len(parsed.data) == 3, f"expected 3 vectors: {result.body[:300]}"
assert len(embeddings.data) == 3, f"expected 3 vectors: {embeddings!r}"
@pytest.mark.replayable
@pytest.mark.covers("llm.embeddings.openai.input_validation.nonstream.works")
def test_missing_model_returns_client_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
def test_missing_model_returns_client_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
key = resources.key()
result = endpoints_client.proxy.transport.send(
result = proxy.transport.send(
"/embeddings",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalEmbeddingsBody(input="hello"),
)
assert_client_error(result, "embeddings missing model")
@pytest.mark.replayable
@pytest.mark.covers("llm.embeddings.openai.input_validation.nonstream.works")
def test_missing_input_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model = f"e2e-embeddings-missin-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
_openai_embeddings_params(),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
result = endpoints_client.proxy.transport.send(
def test_missing_input_returns_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model, key = _register(proxy, resources, "e2e-embeddings-missin", _openai_embeddings_params())
result = proxy.transport.send(
"/embeddings",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalEmbeddingsBody(model=model),
)
assert_client_error(result, "embeddings missing input")

60
tests/e2e/llm_translation/test_google_native_e2e.py
View file

@ -1,19 +1,41 @@
"""Live e2e: the Gemini-native generateContent routes through the gateway.
Google's own SDKs read these routes, and the streaming test asserts the exact SSE
framing they expect (no doubled ``data:`` prefix, no bytes literal, no OpenAI
``[DONE]`` sentinel), which an SDK would hide, so this passthrough surface stays on
the shared transport.
"""
from __future__ import annotations
import pytest
from pydantic import BaseModel
from typing import Literal
import pytest
from e2e_config import unique_marker
from e2e_http import StreamingResponse, require_successful_call
from endpoints_client import EndpointsClient
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from pydantic import BaseModel
pytestmark = pytest.mark.e2e
UPSTREAM_MODEL = "gemini/gemini-2.5-flash"
class _GenerateContentPart(BaseModel):
text: str
class _GenerateContentContent(BaseModel):
role: Literal["user"] = "user"
parts: tuple[_GenerateContentPart, ...]
class _GenerateContentBody(BaseModel):
contents: tuple[_GenerateContentContent, ...]
class _StreamPart(BaseModel):
text: str | None = None
@ -30,16 +52,27 @@ class _StreamEvent(BaseModel):
candidates: tuple[_StreamCandidate, ...] = ()
def _managed_deployment(client: EndpointsClient, resources: ResourceManager) -> str:
def _managed_deployment(proxy: ProxyClient, resources: ResourceManager) -> str:
model = f"e2e-google-native-{unique_marker()}"
model_id = client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(model=UPSTREAM_MODEL, api_key="os.environ/GEMINI_API_KEY"),
)
resources.defer(lambda: client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
return model
def _generate_content(proxy: ProxyClient, key: str, model: str, text: str, *, stream: bool = False) -> StreamingResponse:
operation = "streamGenerateContent" if stream else "generateContent"
body = _GenerateContentBody(contents=(_GenerateContentContent(parts=(_GenerateContentPart(text=text),)),))
return proxy.transport.send(
f"/v1beta/models/{model}:{operation}",
headers=proxy.transport.bearer(key),
json=body,
stream=stream,
)
def _streamed_text(result: StreamingResponse) -> str:
return "".join(
part.text
@ -54,15 +87,13 @@ class TestGoogleNativeGenerateContent:
@pytest.mark.covers("llm.google_native.gemini.basic.nonstream.cost_logged")
def test_generate_content_returns_response_cost_header(
self,
endpoints_client: EndpointsClient,
proxy: ProxyClient,
resources: ResourceManager,
scoped_key: str,
) -> None:
model = _managed_deployment(endpoints_client, resources)
model = _managed_deployment(proxy, resources)
result = endpoints_client.generate_content(
scoped_key, model, f"Reply with the single word ok. {unique_marker()}"
)
result = _generate_content(proxy, scoped_key, model, f"Reply with the single word ok. {unique_marker()}")
require_successful_call(result)
assert result.call_id, "generateContent must stamp x-litellm-call-id"
@ -75,13 +106,14 @@ class TestGoogleNativeGenerateContent:
@pytest.mark.covers("llm.google_native.gemini.basic.stream.works")
def test_stream_generate_content_frames_sse_the_way_google_sdks_expect(
self,
endpoints_client: EndpointsClient,
proxy: ProxyClient,
resources: ResourceManager,
scoped_key: str,
) -> None:
model = _managed_deployment(endpoints_client, resources)
model = _managed_deployment(proxy, resources)
result = endpoints_client.generate_content(
result = _generate_content(
proxy,
scoped_key,
model,
f"Count from one to five, one number per line. {unique_marker()}",

92
tests/e2e/llm_translation/test_image_edits_e2e.py
View file

@ -1,23 +1,24 @@
"""Live e2e: POST /v1/images/edits returns an edited image.
Registers an OpenAI image model, then sends a small PNG plus an edit prompt as a
multipart request to /v1/images/edits and asserts the response carries an image
(url or base64). /images/edits is a distinct native route from
/images/generations: it is multipart file upload with the image sent as the
`image` part, not a JSON body. The fixture image is a small generated 64x64 PNG,
so no external asset is needed.
Registers an OpenAI image model, then sends a small PNG plus an edit prompt
through the real OpenAI SDK (LIT-4577) to /v1/images/edits and asserts the
response carries an image (url or base64). /images/edits is a distinct native
route from /images/generations: it is multipart file upload with the image sent
as the `image` part, not a JSON body. The fixture image is a small generated
64x64 PNG, so no external asset is needed.
"""
from __future__ import annotations
import base64
import openai
import pytest
from e2e_config import unique_marker
from e2e_http import Result, UnknownApiError, unwrap
from endpoints_client import EndpointsClient, ImageEditForm, ImagesResult
from e2e_config import SLOW_PROVIDER_TIMEOUT_SECONDS, unique_marker
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from sdk_clients import SdkClients
pytestmark = pytest.mark.e2e
@ -28,51 +29,54 @@ _TEST_PNG = base64.b64decode(
)
def _register_image_model(endpoints_client: EndpointsClient, resources: ResourceManager) -> tuple[str, str]:
def _register_image_model(proxy: ProxyClient, resources: ResourceManager) -> tuple[str, str]:
model = f"e2e-image-edit-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(model="openai/gpt-image-1", api_key="os.environ/OPENAI_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
return model, resources.key()
def _assert_client_error(result: Result[ImagesResult], context: str) -> None:
match result:
case UnknownApiError(status_code=status) if 400 <= status < 500:
return
case other:
pytest.fail(f"{context}: expected 4xx, got {other!r}")
def _image_part(content: bytes) -> tuple[str, bytes, str]:
return ("image.png", content, "image/png")
def _assert_client_error(error: openai.APIStatusError, context: str) -> None:
assert 400 <= error.status_code < 500, f"{context}: expected 4xx, got {error.status_code}: {error.message}"
class TestImageEdit:
@pytest.mark.covers("llm.images_edits.openai.basic.nonstream.works")
def test_image_edit_returns_image(self, endpoints_client: EndpointsClient, resources: ResourceManager) -> None:
model, key = _register_image_model(endpoints_client, resources)
def test_image_edit_returns_image(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model, key = _register_image_model(proxy, resources)
client = sdk.openai(key)
edited = unwrap(endpoints_client.image_edit(key, model, "Add a small red circle in the center", _TEST_PNG))
assert edited.data, f"/images/edits returned no data: {edited}"
first = edited.data[0]
assert first.b64_json or first.url, f"edited image has neither b64_json nor url: {first}"
@pytest.mark.covers("llm.images_edits.openai.input_validation.nonstream.works")
def test_empty_prompt_returns_error(self, endpoints_client: EndpointsClient, resources: ResourceManager) -> None:
model, key = _register_image_model(endpoints_client, resources)
result = endpoints_client.image_edit(key, model, "", _TEST_PNG)
_assert_client_error(result, "empty image-edit prompt")
@pytest.mark.covers("llm.images_edits.openai.input_validation.nonstream.works")
def test_empty_image_returns_error(self, endpoints_client: EndpointsClient, resources: ResourceManager) -> None:
model, key = _register_image_model(endpoints_client, resources)
result = endpoints_client.proxy.transport.upload(
"/v1/images/edits",
headers=endpoints_client.proxy.transport.bearer(key),
form=ImageEditForm(model=model, prompt="add a red circle"),
filename="image.png",
content=b"",
file_content_type="image/png",
file_field="image",
response_type=ImagesResult,
edited = client.images.edit(
model=model,
image=_image_part(_TEST_PNG),
prompt="Add a small red circle in the center",
timeout=SLOW_PROVIDER_TIMEOUT_SECONDS,
)
_assert_client_error(result, "empty image-edit file")
assert edited.data, f"/images/edits returned no data: {edited!r}"
first = edited.data[0]
assert first.b64_json or first.url, f"edited image has neither b64_json nor url: {first!r}"
@pytest.mark.covers("llm.images_edits.openai.input_validation.nonstream.works")
def test_empty_prompt_returns_error(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model, key = _register_image_model(proxy, resources)
client = sdk.openai(key)
with pytest.raises(openai.APIStatusError) as raised:
client.images.edit(model=model, image=_image_part(_TEST_PNG), prompt="")
_assert_client_error(raised.value, "empty image-edit prompt")
@pytest.mark.covers("llm.images_edits.openai.input_validation.nonstream.works")
def test_empty_image_returns_error(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model, key = _register_image_model(proxy, resources)
client = sdk.openai(key)
with pytest.raises(openai.APIStatusError) as raised:
client.images.edit(model=model, image=_image_part(b""), prompt="add a red circle")
_assert_client_error(raised.value, "empty image-edit file")

114
tests/e2e/llm_translation/test_image_generation_e2e.py
View file

@ -1,22 +1,22 @@
"""Live e2e: POST /v1/images/generations returns an image.
Registers an OpenAI image deployment at runtime and asserts the response carries a
generated image (url or base64). Migrated from
litellm-regression-tests/tests/test_inference_endpoints.py.
Registers an image deployment at runtime, drives it through the real OpenAI SDK
(LIT-4577), and asserts the response carries a generated image (url or base64).
Malformed bodies the SDK refuses to build stay on the shared transport. Migrated
from litellm-regression-tests/tests/test_inference_endpoints.py.
"""
from __future__ import annotations
import pytest
from e2e_config import unique_marker
from e2e_http import (
assert_client_error,
require_successful_call,
)
from endpoints_client import EndpointsClient, ImagesResult
from e2e_http import assert_client_error
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from openai.types import ImagesResponse
from proxy_client import ProxyClient
from pydantic import BaseModel
from sdk_clients import SdkClients
pytestmark = pytest.mark.e2e
@ -28,44 +28,46 @@ class _OptionalImageBody(BaseModel):
size: str | None = None
def _assert_image_returned(body: str) -> None:
parsed = ImagesResult.model_validate_json(body)
assert parsed.data, f"/images/generations returned no data: {body[:300]}"
first = parsed.data[0]
assert first.b64_json or first.url, (
f"generated image has neither b64_json nor url: {body[:300]}"
)
def _assert_image_returned(images: ImagesResponse) -> None:
data = images.data or []
assert data, f"/images/generations returned no data: {images!r}"
first = data[0]
assert first.b64_json or first.url, f"generated image has neither b64_json nor url: {first!r}"
def _register_openai_image(
endpoints_client: EndpointsClient, resources: ResourceManager
) -> tuple[str, str]:
model = f"e2e-image-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
def _register(proxy: ProxyClient, resources: ResourceManager, prefix: str, params: LiteLLMParamsBody) -> tuple[str, str]:
model = f"{prefix}-{unique_marker()}"
model_id = proxy.create_model(model, params)
resources.defer(lambda: proxy.delete_model(model_id))
return model, resources.key()
def _register_openai_image(proxy: ProxyClient, resources: ResourceManager) -> tuple[str, str]:
return _register(
proxy,
resources,
"e2e-image",
LiteLLMParamsBody(model="openai/gpt-image-1-mini", api_key="os.environ/OPENAI_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
return model, resources.key()
class TestImageGeneration:
@pytest.mark.covers("llm.images_generations.openai.basic.nonstream.works")
def test_image_generation_returns_image(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model, key = _register_openai_image(endpoints_client, resources)
result = endpoints_client.images(key, model, "Draw a cute cat")
require_successful_call(result)
_assert_image_returned(result.body)
model, key = _register_openai_image(proxy, resources)
images = sdk.openai(key).images.generate(model=model, prompt="Draw a cute cat", n=1, size="1024x1024")
_assert_image_returned(images)
@pytest.mark.covers("llm.images_generations.bedrock.basic.nonstream.works", exercised_on=["images_generations"])
def test_bedrock_image_generation_returns_image(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-bedrock-image-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
model, key = _register(
proxy,
resources,
"e2e-bedrock-image",
LiteLLMParamsBody(
model="bedrock/amazon.nova-canvas-v1:0",
aws_access_key_id="os.environ/AWS_ACCESS_KEY_ID",
@ -73,58 +75,46 @@ class TestImageGeneration:
aws_region_name="os.environ/AWS_REGION",
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
result = endpoints_client.images(key, model, "Draw a cute cat")
require_successful_call(result)
_assert_image_returned(result.body)
images = sdk.openai(key).images.generate(model=model, prompt="Draw a cute cat", n=1, size="1024x1024")
_assert_image_returned(images)
@pytest.mark.skip(reason="stage red: product gap, /v1/images/generations 500s (aimage_generation TypeError) on missing prompt instead of 400")
@pytest.mark.covers("llm.images_generations.openai.input_validation.nonstream.works")
def test_missing_prompt_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = _register_openai_image(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
def test_missing_prompt_returns_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model, key = _register_openai_image(proxy, resources)
result = proxy.transport.send(
"/v1/images/generations",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalImageBody(model=model),
)
assert_client_error(result, "images missing prompt")
@pytest.mark.covers("llm.images_generations.openai.input_validation.nonstream.works")
def test_empty_prompt_returns_client_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = _register_openai_image(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
def test_empty_prompt_returns_client_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model, key = _register_openai_image(proxy, resources)
result = proxy.transport.send(
"/v1/images/generations",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalImageBody(model=model, prompt=""),
)
assert_client_error(result, "images empty prompt")
@pytest.mark.covers("llm.images_generations.openai.input_validation.nonstream.works")
def test_invalid_size_returns_client_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = _register_openai_image(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
def test_invalid_size_returns_client_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model, key = _register_openai_image(proxy, resources)
result = proxy.transport.send(
"/v1/images/generations",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalImageBody(model=model, prompt="a blue square", size="999x999"),
)
assert_client_error(result, "images invalid size")
@pytest.mark.covers("llm.images_generations.openai.input_validation.nonstream.works")
def test_invalid_n_returns_client_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = _register_openai_image(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
def test_invalid_n_returns_client_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model, key = _register_openai_image(proxy, resources)
result = proxy.transport.send(
"/v1/images/generations",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalImageBody(model=model, prompt="a blue square", n=0),
)
assert_client_error(result, "images invalid n")

190
tests/e2e/llm_translation/test_messages_azure_foundry_e2e.py
View file

@ -1,9 +1,9 @@
"""Live e2e: POST /v1/messages routed to Azure AI Foundry Anthropic deployments.
Registers `azure_ai/<claude>` deployments at runtime and drives the Messages
endpoint through the gateway across the behaviors an Anthropic client relies on:
a basic completion, a streamed completion, and tool use (non-streaming and
streaming). Auth is the Azure API key (`x-api-key`); the deployment reads
endpoint through the gateway with the real Anthropic SDK (LIT-4577) across the
behaviors an Anthropic client relies on: a basic completion, a streamed
completion, and tool use (non-streaming and streaming). The deployment reads
`AZURE_AI_API_BASE` / `AZURE_AI_API_KEY` from the proxy env, so no secret is
sent in the request.
"""
@ -11,52 +11,39 @@ sent in the request.
from __future__ import annotations
import pytest
from anthropic.types import RawMessageStreamEvent, ToolParam
from e2e_config import unique_marker
from e2e_http import StreamingResponse, require_successful_call, unwrap
from endpoints_client import EndpointsClient
from lifecycle import ResourceManager
from models import (
AnthropicCustomTool,
AnthropicMessagesBody,
ChatMessage,
JsonSchemaProperty,
LiteLLMParamsBody,
ToolInputSchema,
)
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from sdk_clients import NO_PROXY_CACHE, SdkClients
pytestmark = pytest.mark.e2e
AZURE_FOUNDRY_MODEL = "azure_ai/claude-haiku-4-5"
WEATHER_TOOL = AnthropicCustomTool(
name="get_weather",
description="Get the current weather for a city.",
input_schema=ToolInputSchema(
properties={"city": JsonSchemaProperty(type="string")},
required=["city"],
),
)
WEATHER_TOOL: ToolParam = {
"name": "get_weather",
"description": "Get the current weather for a city.",
"input_schema": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
}
def _assert_streamed_ok(result: StreamingResponse) -> None:
require_successful_call(result)
assert result.is_streaming, f"response was not streamed: {result.headers}"
assert not result.stream_error, f"stream errored: {result.stream_error}"
assert result.stream_events, "stream produced no SSE events"
assert any("content_block_delta" in event for event in result.stream_events), (
"stream carried no content deltas"
)
assert any("message_stop" in event for event in result.stream_events), (
"stream never reached message_stop"
)
def _assert_streamed_ok(event_types: list[str]) -> None:
assert event_types, "stream produced no SSE events"
assert "content_block_delta" in event_types, "stream carried no content deltas"
assert "message_stop" in event_types, "stream never reached message_stop"
class TestAzureFoundryMessages:
def _register(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> tuple[str, str]:
def _register(self, proxy: ProxyClient, resources: ResourceManager) -> str:
model = f"e2e-azure-foundry-messages-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(
model=AZURE_FOUNDRY_MODEL,
@ -64,91 +51,72 @@ class TestAzureFoundryMessages:
api_key="os.environ/AZURE_AI_API_KEY",
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
return model, resources.key(models=[model])
resources.defer(lambda: proxy.delete_model(model_id))
return model
@pytest.mark.covers("llm.messages.azure_foundry.basic.nonstream.works")
def test_basic_nonstream(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = self._register(endpoints_client, resources)
response = unwrap(
endpoints_client.proxy.messages(
key,
AnthropicMessagesBody(
model=model,
max_tokens=64,
messages=[ChatMessage(role="user", content="Reply with one word.")],
),
)
def test_basic_nonstream(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model = self._register(proxy, resources)
client = sdk.anthropic(resources.key(models=[model]))
message = client.messages.create(
model=model,
max_tokens=64,
messages=[{"role": "user", "content": "Reply with one word."}],
extra_body=NO_PROXY_CACHE,
)
assert response.content, f"no content blocks in response: {response}"
text = "".join(block.text or "" for block in response.content if block.type == "text")
assert text.strip(), f"/v1/messages returned no text: {response}"
assert message.content, f"no content blocks in response: {message!r}"
text = "".join(block.text for block in message.content if block.type == "text")
assert text.strip(), f"/v1/messages returned no text: {message.content!r}"
@pytest.mark.covers("llm.messages.azure_foundry.basic.stream.works")
def test_basic_stream(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = self._register(endpoints_client, resources)
result = endpoints_client.proxy.messages_stream(
key,
AnthropicMessagesBody(
model=model,
max_tokens=64,
stream=True,
messages=[ChatMessage(role="user", content="Count from one to three.")],
),
def test_basic_stream(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model = self._register(proxy, resources)
client = sdk.anthropic(resources.key(models=[model]))
stream = client.messages.create(
model=model,
max_tokens=64,
stream=True,
messages=[{"role": "user", "content": "Count from one to three."}],
extra_body=NO_PROXY_CACHE,
)
_assert_streamed_ok(result)
_assert_streamed_ok([event.type for event in stream])
@pytest.mark.covers("llm.messages.azure_foundry.tool_use.nonstream.works")
def test_tool_use_nonstream(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = self._register(endpoints_client, resources)
response = unwrap(
endpoints_client.proxy.messages(
key,
AnthropicMessagesBody(
model=model,
max_tokens=256,
tools=[WEATHER_TOOL],
messages=[
ChatMessage(role="user", content="What is the weather in Paris? Use the tool.")
],
),
)
def test_tool_use_nonstream(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model = self._register(proxy, resources)
client = sdk.anthropic(resources.key(models=[model]))
message = client.messages.create(
model=model,
max_tokens=256,
tools=[WEATHER_TOOL],
messages=[{"role": "user", "content": "What is the weather in Paris? Use the tool."}],
extra_body=NO_PROXY_CACHE,
)
assert response.content, f"no content blocks in response: {response}"
assert any(block.type == "tool_use" for block in response.content), (
f"model did not call the tool: {response}"
assert message.content, f"no content blocks in response: {message!r}"
assert any(block.type == "tool_use" for block in message.content), (
f"model did not call the tool: {message.content!r}"
)
@pytest.mark.covers("llm.messages.azure_foundry.tool_use.stream.works")
def test_tool_use_stream(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = self._register(endpoints_client, resources)
result = endpoints_client.proxy.messages_stream(
key,
AnthropicMessagesBody(
model=model,
max_tokens=256,
stream=True,
tools=[WEATHER_TOOL],
messages=[
ChatMessage(role="user", content="What is the weather in Paris? Use the tool.")
],
),
)
require_successful_call(result)
assert result.is_streaming, f"response was not streamed: {result.headers}"
assert not result.stream_error, f"stream errored: {result.stream_error}"
assert result.stream_events, "stream produced no SSE events"
assert any("tool_use" in event for event in result.stream_events), (
"stream carried no tool_use block"
)
assert any("message_stop" in event for event in result.stream_events), (
"stream never reached message_stop"
def test_tool_use_stream(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model = self._register(proxy, resources)
client = sdk.anthropic(resources.key(models=[model]))
stream = client.messages.create(
model=model,
max_tokens=256,
stream=True,
tools=[WEATHER_TOOL],
messages=[{"role": "user", "content": "What is the weather in Paris? Use the tool."}],
extra_body=NO_PROXY_CACHE,
)
events: list[RawMessageStreamEvent] = list(stream)
event_types = [event.type for event in events]
assert event_types, "stream produced no SSE events"
assert any(
event.type == "content_block_start" and event.content_block.type == "tool_use" for event in events
), "stream carried no tool_use block"
assert "message_stop" in event_types, "stream never reached message_stop"

476
tests/e2e/llm_translation/test_messages_e2e.py
View file

@ -1,40 +1,42 @@
"""Live e2e: POST /v1/messages (Anthropic Messages API) returns a real completion.
Registers an Anthropic deployment at runtime, drives the Messages endpoint through
the gateway, and asserts an assistant message with text came back, both
non-streaming and streamed. Migrated from
the gateway with the real Anthropic SDK, the client customers actually use
(LIT-4577), and asserts an assistant message with text came back, both
non-streaming and streamed. Malformed bodies the SDK refuses to build stay on the
shared transport. Migrated from
litellm-regression-tests/tests/test_inference_endpoints.py.
"""
from __future__ import annotations
import time
from typing import Final
import pytest
from e2e_config import (
STREAM_MIN_LEAD_SECONDS,
provider_edge_base,
provider_paces_stream,
unique_marker,
from anthropic import Anthropic
from anthropic.types import (
InputJSONDelta,
Message,
MessageParam,
RawContentBlockDeltaEvent,
RawContentBlockStartEvent,
RawContentBlockStopEvent,
RawMessageDeltaEvent,
RawMessageStreamEvent,
TextBlock,
TextDelta,
ToolChoiceParam,
ToolParam,
ToolUseBlock,
)
from e2e_http import assert_client_error, require_successful_call, unwrap
from endpoints_client import EndpointsClient, MessagesResult
from e2e_config import STREAM_MIN_LEAD_SECONDS, provider_edge_base, provider_paces_stream, unique_marker
from e2e_http import assert_client_error
from lifecycle import ResourceManager
from models import (
AnthropicAssistantTurn,
AnthropicContentBlock,
AnthropicCustomTool,
AnthropicMessagesBody,
AnthropicToolChoice,
AnthropicToolResultBlock,
AnthropicToolResultTurn,
ChatMessage,
JsonSchemaProperty,
LiteLLMParamsBody,
SpendLogRow,
ToolInputSchema,
)
from models import ChatMessage, LiteLLMParamsBody, SpendLogRow
from proxy_client import ProxyClient
from pydantic import BaseModel, ConfigDict
from sdk_clients import NO_PROXY_CACHE, SdkClients, response_header
pytestmark = [pytest.mark.e2e, pytest.mark.replayable]
@ -45,35 +47,17 @@ class _OptionalMessagesBody(BaseModel):
max_tokens: int | None = None
class _MessagesEventDelta(BaseModel):
text: str = ""
class _MessagesEventUsage(BaseModel):
output_tokens: int | None = None
class _MessagesStreamEvent(BaseModel):
"""One Anthropic SSE event, keeping only what the stream's shape is asserted on.
``delta.text`` is populated on ``content_block_delta`` and absent on the
``message_delta`` that closes the turn, which is the event carrying ``usage``."""
type: str
delta: _MessagesEventDelta | None = None
usage: _MessagesEventUsage | None = None
ANTHROPIC_BACKEND = "anthropic/claude-haiku-4-5"
WEATHER_TOOL = AnthropicCustomTool(
name="get_weather",
description="Get the current weather for a city.",
input_schema=ToolInputSchema(
properties={"city": JsonSchemaProperty(type="string")},
required=["city"],
),
)
WEATHER_TOOL: ToolParam = {
"name": "get_weather",
"description": "Get the current weather for a city.",
"input_schema": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
}
def _approx_equal(actual: float, expected: float) -> bool:
@ -87,60 +71,67 @@ def _anthropic_params() -> LiteLLMParamsBody:
handler appends ``/v1/messages`` to ``api_base`` itself, where the OpenAI handler
appends only ``/chat/completions``."""
base = provider_edge_base("anthropic")
return LiteLLMParamsBody(
model=ANTHROPIC_BACKEND, api_key="os.environ/ANTHROPIC_API_KEY", api_base=base
)
return LiteLLMParamsBody(model=ANTHROPIC_BACKEND, api_key="os.environ/ANTHROPIC_API_KEY", api_base=base)
def _register(
proxy: ProxyClient,
resources: ResourceManager,
params: LiteLLMParamsBody | None = None,
prefix: str = "e2e-messages",
) -> tuple[str, str]:
model = f"{prefix}-{unique_marker()}"
model_id = proxy.create_model(model, _anthropic_params() if params is None else params)
resources.defer(lambda: proxy.delete_model(model_id))
return model, resources.key()
def _text(message: Message) -> str:
return "".join(block.text for block in message.content if isinstance(block, TextBlock))
def _user_turn(text: str) -> MessageParam:
return {"role": "user", "content": text}
class TestAnthropicMessages:
def _register(
self,
endpoints_client: EndpointsClient,
resources: ResourceManager,
params: LiteLLMParamsBody | None = None,
) -> tuple[str, str]:
model = f"e2e-messages-{unique_marker()}"
model_id = endpoints_client.create_model(
model, _anthropic_params() if params is None else params
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
return model, resources.key()
@pytest.mark.covers("llm.messages.anthropic.basic.nonstream.works")
def test_messages_returns_completion(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = self._register(endpoints_client, resources)
def test_messages_returns_completion(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model, key = _register(proxy, resources)
client = sdk.anthropic(key)
result = endpoints_client.messages(key, model, "reply with one word")
require_successful_call(result)
parsed = MessagesResult.model_validate_json(result.body)
assert parsed.role == "assistant", f"unexpected role: {result.body[:300]}"
assert parsed.text.strip(), f"/v1/messages returned no text: {result.body[:300]}"
message = client.messages.create(
model=model, max_tokens=64, messages=[_user_turn("reply with one word")], extra_body=NO_PROXY_CACHE
)
assert message.role == "assistant", f"unexpected role: {message.role!r}"
assert _text(message).strip(), f"/v1/messages returned no text: {message.content!r}"
@pytest.mark.covers("llm.messages.anthropic.basic.nonstream.cost_logged")
def test_messages_logs_cost_matching_the_response_header(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-messages-cost-{unique_marker()}"
model_id = endpoints_client.create_model(model, _anthropic_params())
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
model, key = _register(proxy, resources, prefix="e2e-messages-cost")
client = sdk.anthropic(key)
result = endpoints_client.messages(key, model, f"reply with one word {unique_marker()}")
require_successful_call(result)
parsed = MessagesResult.model_validate_json(result.body)
assert parsed.role == "assistant" and parsed.text.strip(), (
f"/v1/messages returned no assistant text: {result.body[:300]}"
raw = client.messages.with_raw_response.create(
model=model,
max_tokens=64,
messages=[_user_turn(f"reply with one word {unique_marker()}")],
extra_body=NO_PROXY_CACHE,
)
message = raw.parse()
assert message.role == "assistant" and _text(message).strip(), (
f"/v1/messages returned no assistant text: {message.content!r}"
)
# The customer reads per-request cost off the response header (LIT-4076), so
# it must be present and positive on /v1/messages, not only /chat/completions.
header_cost = result.response_cost
assert header_cost is not None and header_cost > 0, (
"x-litellm-response-cost header missing or non-positive on /v1/messages; "
f"headers={result.headers}"
raw_header_cost = response_header(raw.headers, "x-litellm-response-cost")
assert raw_header_cost is not None, (
f"x-litellm-response-cost header missing on /v1/messages; headers={dict(raw.headers)}"
)
header_cost = float(raw_header_cost)
assert header_cost > 0, f"x-litellm-response-cost header non-positive on /v1/messages: {header_cost}"
# Correlate the spend row by the unique scoped key, not the Anthropic response
# id: on /v1/messages the spend-log request_id is the proxy's own call id, which
@ -150,11 +141,9 @@ class TestAnthropicMessages:
def _priced(rows: list[SpendLogRow]) -> bool:
return any(r.spend is not None and r.spend > 0 for r in rows)
rows = endpoints_client.proxy.poll_logs_for_key(key, predicate=_priced)
rows = proxy.poll_logs_for_key(key, predicate=_priced)
priced = [r for r in rows if r.spend is not None and r.spend > 0]
assert priced, (
f"no priced /spend/logs row landed for key {key} within the poll window; got {rows}"
)
assert priced, f"no priced /spend/logs row landed for key {key} within the poll window; got {rows}"
row = priced[0]
assert (row.prompt_tokens or 0) > 0 and (row.completion_tokens or 0) > 0, (
f"messages spend row missing token counts, so the cost is not real usage: {row}"
@ -166,9 +155,7 @@ class TestAnthropicMessages:
@pytest.mark.covers("llm.messages.anthropic.basic.stream.works")
@pytest.mark.provider_live
def test_messages_streams_completion(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
def test_messages_streams_completion(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
"""Edge-wired like its non-streaming siblings, so record and replay both
carry the streamed response.
@ -178,51 +165,45 @@ class TestAnthropicMessages:
the first content delta must instead reach the client well before
``message_stop``, which a buffered response cannot do. Replay serves chunks back
to back, so only live and record runs judge the timing."""
model, key = self._register(endpoints_client, resources)
model, key = _register(proxy, resources)
client = sdk.anthropic(key)
result = endpoints_client.proxy.messages_stream(
key,
AnthropicMessagesBody(
model=model,
max_tokens=800,
stream=True,
messages=[ChatMessage(role="user", content="Count from 1 to 200, one number per line.")],
),
started: Final = time.monotonic()
stream = client.messages.create(
model=model,
max_tokens=800,
stream=True,
messages=[_user_turn("Count from 1 to 200, one number per line.")],
extra_body=NO_PROXY_CACHE,
)
require_successful_call(result)
assert result.is_streaming, f"response was not streamed: {result.headers}"
assert not result.stream_error, f"stream errored: {result.stream_error}"
assert result.stream_events, "stream produced no SSE events"
arrivals: Final = tuple((event, time.monotonic() - started) for event in stream)
assert arrivals, "stream produced no SSE events"
events = [
_MessagesStreamEvent.model_validate_json(event) for event in result.stream_events
]
types = [event.type for event in events]
delta_positions = [
events: Final = tuple(event for event, _ in arrivals)
types: Final = tuple(event.type for event in events)
delta_positions: Final = tuple(
index for index, event in enumerate(events) if event.type == "content_block_delta"
]
)
assert delta_positions, f"stream carried no content deltas: {types}"
text = "".join(
text: Final = "".join(
event.delta.text
for event in events
if event.type == "content_block_delta" and event.delta is not None
if isinstance(event, RawContentBlockDeltaEvent) and isinstance(event.delta, TextDelta)
)
assert text.strip(), f"content deltas assembled to no text: {result.stream_events[:5]}"
assert text.strip(), f"content deltas assembled to no text: {events[:5]}"
usage_positions = [
index
for index, event in enumerate(events)
if event.type == "message_delta" and event.usage is not None
]
usage_positions: Final = tuple(
index for index, event in enumerate(events) if isinstance(event, RawMessageDeltaEvent)
)
assert usage_positions, f"stream never reported usage: {types}"
assert "message_stop" in types, f"stream never reached message_stop: {types}"
stop_position = types.index("message_stop")
stop_position: Final = types.index("message_stop")
assert delta_positions[-1] < usage_positions[0] < stop_position, (
f"usage did not land between the last content delta and message_stop: {types}"
)
first_delta_at: Final = result.stream_event_arrivals[delta_positions[0]]
stop_at: Final = result.stream_event_arrivals[stop_position]
first_delta_at: Final = arrivals[delta_positions[0]][1]
stop_at: Final = arrivals[stop_position][1]
if provider_paces_stream():
assert stop_at - first_delta_at >= STREAM_MIN_LEAD_SECONDS, (
f"first content delta reached the client {first_delta_at:.2f}s after the request "
@ -231,142 +212,125 @@ class TestAnthropicMessages:
)
@pytest.mark.covers("llm.messages.anthropic.tool_use.nonstream.works")
def test_messages_tool_use(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = self._register(endpoints_client, resources)
def test_messages_tool_use(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model, key = _register(proxy, resources)
client = sdk.anthropic(key)
response = unwrap(
endpoints_client.proxy.messages(
key,
AnthropicMessagesBody(
model=model,
max_tokens=256,
tools=[WEATHER_TOOL],
messages=[
ChatMessage(role="user", content="What is the weather in Paris? Use the tool.")
],
),
)
message = client.messages.create(
model=model,
max_tokens=256,
tools=[WEATHER_TOOL],
messages=[_user_turn("What is the weather in Paris? Use the tool.")],
extra_body=NO_PROXY_CACHE,
)
assert response.content, f"no content blocks in response: {response}"
assert any(block.type == "tool_use" for block in response.content), (
f"model did not call the tool: {response}"
assert message.content, f"no content blocks in response: {message!r}"
assert any(isinstance(block, ToolUseBlock) for block in message.content), (
f"model did not call the tool: {message.content!r}"
)
@pytest.mark.skip(reason="stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing messages instead of 400")
@pytest.mark.skip(
reason="stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing messages instead of 400"
)
@pytest.mark.covers("llm.messages.anthropic.input_validation.nonstream.works")
def test_missing_messages_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = self._register(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
def test_missing_messages_returns_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model, key = _register(proxy, resources)
result = proxy.transport.send(
"/v1/messages",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalMessagesBody(model=model, max_tokens=50),
)
assert_client_error(result, "messages missing messages")
@pytest.mark.skip(reason="stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing max_tokens instead of 400")
@pytest.mark.skip(
reason="stage red: product gap, /v1/messages 500s (anthropic_messages TypeError) on missing max_tokens instead of 400"
)
@pytest.mark.covers("llm.messages.anthropic.input_validation.nonstream.works")
def test_missing_max_tokens_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model, key = self._register(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
def test_missing_max_tokens_returns_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model, key = _register(proxy, resources)
result = proxy.transport.send(
"/v1/messages",
headers=endpoints_client.proxy.transport.bearer(key),
json=_OptionalMessagesBody(
model=model, messages=[ChatMessage(role="user", content="hi")]
),
headers=proxy.transport.bearer(key),
json=_OptionalMessagesBody(model=model, messages=[ChatMessage(role="user", content="hi")]),
)
assert_client_error(result, "messages missing max_tokens")
@pytest.mark.covers("llm.messages.anthropic.input_validation.nonstream.works")
def test_missing_model_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
_, key = self._register(endpoints_client, resources)
result = endpoints_client.proxy.transport.send(
def test_missing_model_returns_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
_, key = _register(proxy, resources)
result = proxy.transport.send(
"/v1/messages",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalMessagesBody(messages=[ChatMessage(role="user", content="hi")], max_tokens=50),
)
assert_client_error(result, "messages missing model")
class _BridgeDelta(BaseModel):
type: str | None = None
partial_json: str | None = None
stop_reason: str | None = None
class _BridgeEvent(BaseModel):
type: str
index: int | None = None
content_block: AnthropicContentBlock | None = None
delta: _BridgeDelta | None = None
class _ParcelInput(BaseModel):
model_config = ConfigDict(extra="forbid", strict=True)
parcel: str
shelf: int
def _tool_from_stream(events: tuple[_BridgeEvent, ...]) -> AnthropicContentBlock:
def _tool_from_stream(events: tuple[RawMessageStreamEvent, ...]) -> ToolUseBlock:
starts: Final = tuple(
event
for event in events
if event.type == "content_block_start"
and event.content_block is not None
and event.content_block.type == "tool_use"
(index, event.index, event.content_block)
for index, event in enumerate(events)
if isinstance(event, RawContentBlockStartEvent) and isinstance(event.content_block, ToolUseBlock)
)
assert len(starts) == 1, "expected exactly one tool call"
start: Final = starts[0]
block: Final = start.content_block
assert block is not None and block.id and start.index is not None
start_position, block_index, block = starts[0]
assert block.id
fragments: Final = tuple(
event
for event in events
if event.type == "content_block_delta" and event.delta is not None and event.delta.type == "input_json_delta"
(index, event.index, event.delta.partial_json)
for index, event in enumerate(events)
if isinstance(event, RawContentBlockDeltaEvent) and isinstance(event.delta, InputJSONDelta)
)
assert fragments, "tool stream contained no argument fragments"
assert all(event.index == start.index for event in fragments), "tool fragments changed index"
positions: Final = tuple(i for i, event in enumerate(events) if event in fragments)
assert all(fragment_block == block_index for _, fragment_block, _ in fragments), "tool fragments changed index"
positions: Final = tuple(index for index, _, _ in fragments)
stops: Final = tuple(
i for i, event in enumerate(events) if event.type == "content_block_stop" and event.index == start.index
index
for index, event in enumerate(events)
if isinstance(event, RawContentBlockStopEvent) and event.index == block_index
)
assert len(stops) == 1 and events.index(start) < positions[0] <= positions[-1] < stops[0]
assert tuple(
event.delta.stop_reason for event in events if event.type == "message_delta" and event.delta is not None
) == ("tool_use",)
terminal_positions: Final = tuple(i for i, event in enumerate(events) if event.type == "message_delta")
assert len(stops) == 1 and start_position < positions[0] <= positions[-1] < stops[0]
terminal_positions: Final = tuple(
index for index, event in enumerate(events) if isinstance(event, RawMessageDeltaEvent)
)
stop_reasons: Final = tuple(event.delta.stop_reason for event in events if isinstance(event, RawMessageDeltaEvent))
assert stop_reasons == ("tool_use",)
assert len(terminal_positions) == 1 and stops[0] < terminal_positions[0] < len(events) - 1
assert tuple(i for i, event in enumerate(events) if event.type == "message_stop") == (len(events) - 1,), (
assert tuple(index for index, event in enumerate(events) if event.type == "message_stop") == (len(events) - 1,), (
"tool stream did not terminate exactly once"
)
arguments: Final = _ParcelInput.model_validate_json(
"".join(event.delta.partial_json or "" for event in fragments if event.delta is not None)
)
return AnthropicContentBlock(type="tool_use", id=block.id, name=block.name, input=arguments.model_dump())
arguments: Final = _ParcelInput.model_validate_json("".join(partial for _, _, partial in fragments))
return ToolUseBlock(type="tool_use", id=block.id, name=block.name, input=arguments.model_dump())
def _parcel_result(tool: AnthropicContentBlock, result: AnthropicToolResultBlock) -> AnthropicToolResultTurn:
assert tool.id and result.tool_use_id == tool.id, "tool result ID does not match the emitted call"
return AnthropicToolResultTurn(content=[result])
def _request_tool(
client: EndpointsClient, key: str, request: AnthropicMessagesBody, stream: bool
) -> AnthropicContentBlock:
def _request_tool(client: Anthropic, model: str, question: MessageParam, tool: ToolParam, stream: bool) -> ToolUseBlock:
tool_choice: Final[ToolChoiceParam] = {"type": "tool", "name": tool["name"]}
if stream:
response: Final = client.proxy.messages_stream(key, request)
require_successful_call(response)
assert response.is_streaming and not response.stream_error
return _tool_from_stream(tuple(_BridgeEvent.model_validate_json(event) for event in response.stream_events))
response_body: Final = unwrap(client.proxy.messages(key, request))
blocks: Final = tuple(block for block in response_body.content or () if block.type == "tool_use")
events: Final = tuple(
client.messages.create(
model=model,
max_tokens=2048,
messages=[question],
tools=[tool],
tool_choice=tool_choice,
stream=True,
extra_body=NO_PROXY_CACHE,
)
)
return _tool_from_stream(events)
message: Final = client.messages.create(
model=model,
max_tokens=2048,
messages=[question],
tools=[tool],
tool_choice=tool_choice,
extra_body=NO_PROXY_CACHE,
)
blocks: Final = tuple(block for block in message.content if isinstance(block, ToolUseBlock))
assert len(blocks) == 1
return blocks[0]
@ -375,55 +339,49 @@ class TestOpenAIMessagesToolContinuation:
@pytest.mark.provider_live
@pytest.mark.parametrize("stream", [True, False], ids=["stream", "nonstream"])
def test_required_tool_arguments_and_correlated_result(
self, endpoints_client: EndpointsClient, resources: ResourceManager, stream: bool
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients, stream: bool
) -> None:
model: Final = f"e2e-bridge-tool-{unique_marker()}"
base: Final = provider_edge_base("openai")
model_id: Final = endpoints_client.create_model(
model_id: Final = proxy.create_model(
model,
LiteLLMParamsBody(
model="openai/gpt-5.6", api_key="os.environ/OPENAI_API_KEY", api_base=f"{base}/v1" if base else None
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key: Final = resources.key(models=[model])
tool: Final = AnthropicCustomTool(
name="locate_parcel",
description="Look up the receipt for a parcel on a shelf. Return the receipt verbatim.",
input_schema=ToolInputSchema(
properties={"parcel": JsonSchemaProperty(type="string"), "shelf": JsonSchemaProperty(type="integer")},
required=["parcel", "shelf"],
),
resources.defer(lambda: proxy.delete_model(model_id))
client: Final = sdk.anthropic(resources.key(models=[model]))
tool: Final[ToolParam] = {
"name": "locate_parcel",
"description": "Look up the receipt for a parcel on a shelf. Return the receipt verbatim.",
"input_schema": {
"type": "object",
"properties": {"parcel": {"type": "string"}, "shelf": {"type": "integer"}},
"required": ["parcel", "shelf"],
},
}
question: Final = _user_turn(
"Call locate_parcel with parcel exactly amber-kite and shelf exactly 7. "
"After the tool result, reply with only the receipt returned by the tool."
)
question: Final = ChatMessage(
role="user",
content="Call locate_parcel with parcel exactly amber-kite and shelf exactly 7. After the tool result, reply with only the receipt returned by the tool.",
)
request: Final = AnthropicMessagesBody(
model=model,
max_tokens=2048,
messages=[question],
tools=[tool],
tool_choice=AnthropicToolChoice(type="tool", name=tool.name),
stream=stream,
)
emitted: Final = _request_tool(endpoints_client, key, request, stream)
emitted: Final = _request_tool(client, model, question, tool, stream)
assert emitted.id and emitted.name == "locate_parcel"
assert emitted.input == {"parcel": "amber-kite", "shelf": 7}, "required tool arguments were lost or changed"
receipt: Final = f"receipt-{unique_marker()}"
result_turn: Final = _parcel_result(emitted, AnthropicToolResultBlock(tool_use_id=emitted.id, content=receipt))
continuation: Final = unwrap(
endpoints_client.proxy.messages(
key,
AnthropicMessagesBody(
model=model,
max_tokens=2048,
tools=[tool],
tool_choice=AnthropicToolChoice(type="none"),
messages=[question, AnthropicAssistantTurn(content=[emitted]), result_turn],
),
)
continuation: Final = client.messages.create(
model=model,
max_tokens=2048,
tools=[tool],
tool_choice={"type": "none"},
messages=[
question,
{
"role": "assistant",
"content": [{"type": "tool_use", "id": emitted.id, "name": emitted.name, "input": emitted.input}],
},
{"role": "user", "content": [{"type": "tool_result", "tool_use_id": emitted.id, "content": receipt}]},
],
extra_body=NO_PROXY_CACHE,
)
answer: Final = "".join(block.text or "" for block in continuation.content or ())
assert answer.strip() == receipt, "continuation did not consume the correlated tool result"
assert all(block.type != "tool_use" for block in continuation.content or ())
assert _text(continuation).strip() == receipt, "continuation did not consume the correlated tool result"
assert all(not isinstance(block, ToolUseBlock) for block in continuation.content)

196
tests/e2e/llm_translation/test_messages_mid_conversation_system_e2e.py
View file

@ -17,27 +17,28 @@ entry whose prefix spans ``system`` plus message turns is invalidated when the
reminder is hoisted (the ``system`` field mutates and a turn disappears from
``messages``), while an entry ending at the system block itself would survive
the hoist and mask the regression.
Calls go through the real Anthropic SDK (LIT-4577). The SDK's ``MessageParam``
type only admits user/assistant roles, so the system reminder turn is cast to
it; the SDK serializes the dict verbatim, which is exactly the wire shape under
test.
"""
from __future__ import annotations
import time
from collections.abc import Sequence
from typing import cast
import pytest
from pydantic import BaseModel
from anthropic import Anthropic
from anthropic.types import Message, MessageParam, TextBlockParam
from e2e_config import unique_marker
from e2e_http import Result, unwrap
from endpoints_client import (
CacheControl,
EndpointsClient,
MessagesResult,
RichMessage,
RichMessagesRequest,
TextBlock,
)
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from pydantic import BaseModel
from sdk_clients import NO_PROXY_CACHE, SdkClients
pytestmark = pytest.mark.e2e
@ -49,54 +50,54 @@ CACHE_PRIMING_INTERVAL_SECONDS = 3.0
CACHE_WARM_CONSECUTIVE_READS = 3
def _cacheable_system_block(marker: str) -> TextBlock:
def _cacheable_system_block(marker: str) -> TextBlockParam:
"""A system prompt at roughly twice the 4096-token minimum cacheable size of
Haiku 4.5 (the smallest model here), unique per run so no other run's cache
entry can satisfy the read. The marker appears once instead of in every
paragraph: repeating it swung the block's size by ~1800 tokens with the
marker's own tokenization and left it under the minimum on ~15% of runs, so
the system breakpoint went uncached and the priming loop never saw a read."""
text = f"Run {marker}.\n" + " ".join(
f"Reference paragraph {index}." for index in range(1500)
text = f"Run {marker}.\n" + " ".join(f"Reference paragraph {index}." for index in range(1500))
return {"type": "text", "text": text, "cache_control": {"type": "ephemeral"}}
def _user_turn(text: str, *, cached: bool = False) -> MessageParam:
block: TextBlockParam = (
{"type": "text", "text": text, "cache_control": {"type": "ephemeral"}}
if cached
else {"type": "text", "text": text}
)
return TextBlock(text=text, cache_control=CacheControl())
return {"role": "user", "content": [block]}
def _user_turn(text: str, *, cached: bool = False) -> RichMessage:
block = TextBlock(text=text, cache_control=CacheControl() if cached else None)
return RichMessage(role="user", content=[block])
def _system_reminder_turn() -> RichMessage:
return RichMessage(
role="system",
content=[
TextBlock(
text="<system-reminder>Answer with exactly one word.</system-reminder>"
)
],
def _system_reminder_turn() -> MessageParam:
return cast(
"MessageParam",
{
"role": "system",
"content": [{"type": "text", "text": "<system-reminder>Answer with exactly one word.</system-reminder>"}],
},
)
def _post_messages(
client: EndpointsClient, key: str, body: RichMessagesRequest
) -> Result[MessagesResult]:
return client.proxy.transport.post(
"/v1/messages",
headers=client.proxy.transport.bearer(key),
json=body,
response_type=MessagesResult,
def _assistant_turn(text: str) -> MessageParam:
return {"role": "assistant", "content": [{"type": "text", "text": text}]}
def _text(message: Message) -> str:
return "".join(block.text for block in message.content if block.type == "text")
def _send(client: Anthropic, model: str, system_block: TextBlockParam, messages: Sequence[MessageParam]) -> Message:
return client.messages.create(
model=model, max_tokens=64, system=[system_block], messages=messages, extra_body=NO_PROXY_CACHE
)
def _register_invoke_deployment(
client: EndpointsClient, resources: ResourceManager, bedrock_model: str
) -> str:
def _register_invoke_deployment(proxy: ProxyClient, resources: ResourceManager, bedrock_model: str) -> str:
model = f"e2e-midsys-{unique_marker()}"
model_id = client.create_model(
model, LiteLLMParamsBody(model=bedrock_model, aws_region_name=AWS_REGION)
)
resources.defer(lambda: client.delete_model(model_id))
model_id = proxy.create_model(model, LiteLLMParamsBody(model=bedrock_model, aws_region_name=AWS_REGION))
resources.defer(lambda: proxy.delete_model(model_id))
return model
@ -118,9 +119,7 @@ class PrimedCache(BaseModel):
return self.prefix_read_tokens + self.first_turn_creation_tokens
def _prime_prompt_cache(
client: EndpointsClient, key: str, model: str, system_block: TextBlock
) -> PrimedCache:
def _prime_prompt_cache(client: Anthropic, model: str, system_block: TextBlockParam) -> PrimedCache:
"""Send first-turn calls (fresh cache-marked user turn each attempt,
identical system prefix) until one both reads the system prefix back from
cache and writes its own user-turn chunk, then re-send that exact turn until
@ -132,19 +131,17 @@ def _prime_prompt_cache(
deadline = time.monotonic() + CACHE_PRIMING_DEADLINE_SECONDS
while True:
user_text = _first_turn_user_text(unique_marker())
body = RichMessagesRequest(
model=model,
system=[system_block],
messages=[_user_turn(user_text, cached=True)],
)
usage = unwrap(_post_messages(client, key, body)).usage
if usage.cache_read_input_tokens > 0 and usage.cache_creation_input_tokens > 0:
first_turn = (_user_turn(user_text, cached=True),)
usage = _send(client, model, system_block, first_turn).usage
read_tokens = usage.cache_read_input_tokens or 0
creation_tokens = usage.cache_creation_input_tokens or 0
if read_tokens > 0 and creation_tokens > 0:
primed = PrimedCache(
first_user_text=user_text,
prefix_read_tokens=usage.cache_read_input_tokens,
first_turn_creation_tokens=usage.cache_creation_input_tokens,
prefix_read_tokens=read_tokens,
first_turn_creation_tokens=creation_tokens,
)
if _first_turn_reads_back(client, key, body, primed.full_prefix_tokens, deadline):
if _first_turn_reads_back(client, model, system_block, first_turn, primed.full_prefix_tokens, deadline):
return primed
if time.monotonic() >= deadline:
pytest.fail(
@ -155,15 +152,20 @@ def _prime_prompt_cache(
def _reads_full_prefix(
client: EndpointsClient, key: str, body: RichMessagesRequest, full_prefix_tokens: int
client: Anthropic,
model: str,
system_block: TextBlockParam,
messages: Sequence[MessageParam],
full_prefix_tokens: int,
) -> bool:
return unwrap(_post_messages(client, key, body)).usage.cache_read_input_tokens >= full_prefix_tokens
return (_send(client, model, system_block, messages).usage.cache_read_input_tokens or 0) >= full_prefix_tokens
def _first_turn_reads_back(
client: EndpointsClient,
key: str,
body: RichMessagesRequest,
client: Anthropic,
model: str,
system_block: TextBlockParam,
messages: Sequence[MessageParam],
full_prefix_tokens: int,
deadline: float,
) -> bool:
@ -172,12 +174,24 @@ def _first_turn_reads_back(
fresh entry can be missing from the region the next request lands on; each miss
re-creates the entry there, so the streak converges as the regions warm up."""
while time.monotonic() < deadline:
if all(_reads_full_prefix(client, key, body, full_prefix_tokens) for _ in range(CACHE_WARM_CONSECUTIVE_READS)):
if all(
_reads_full_prefix(client, model, system_block, messages, full_prefix_tokens)
for _ in range(CACHE_WARM_CONSECUTIVE_READS)
):
return True
time.sleep(CACHE_PRIMING_INTERVAL_SECONDS)
return False
def _reminder_turn_messages(primed: PrimedCache) -> tuple[MessageParam, ...]:
return (
_user_turn(primed.first_user_text, cached=True),
_system_reminder_turn(),
_assistant_turn("OK."),
_user_turn("Reply with one word again.", cached=True),
)
#: Kept in sync with the copy in test_messages_mid_conversation_system_native_providers_e2e.py;
#: the e2e suites stay self-contained rather than importing across test modules.
MID_CONVERSATION_CACHE_SKIP_REASON = (
@ -195,32 +209,18 @@ class TestBedrockInvokeMidConversationSystem:
exercised_on=[],
)
def test_flagged_model_keeps_prompt_cache_across_system_reminder(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = _register_invoke_deployment(
endpoints_client, resources, FLAGGED_INVOKE_MODEL
)
key = resources.key(models=[model])
model = _register_invoke_deployment(proxy, resources, FLAGGED_INVOKE_MODEL)
client = sdk.anthropic(resources.key(models=[model]))
system_block = _cacheable_system_block(unique_marker())
primed = _prime_prompt_cache(endpoints_client, key, model, system_block)
primed = _prime_prompt_cache(client, model, system_block)
reminder_turn_body = RichMessagesRequest(
model=model,
system=[system_block],
messages=[
_user_turn(primed.first_user_text, cached=True),
_system_reminder_turn(),
RichMessage(role="assistant", content=[TextBlock(text="OK.")]),
_user_turn("Reply with one word again.", cached=True),
],
)
second = unwrap(_post_messages(endpoints_client, key, reminder_turn_body))
second = _send(client, model, system_block, _reminder_turn_messages(primed))
assert second.text.strip(), (
f"{model}: reminder turn returned no completion text"
)
assert second.usage.cache_read_input_tokens >= primed.full_prefix_tokens, (
assert _text(second).strip(), f"{model}: reminder turn returned no completion text"
assert (second.usage.cache_read_input_tokens or 0) >= primed.full_prefix_tokens, (
f"{model}: turn with a mid-conversation system reminder read "
f"{second.usage.cache_read_input_tokens} cached tokens, expected at "
f"least the {primed.full_prefix_tokens} cached on turn one "
@ -235,37 +235,23 @@ class TestBedrockInvokeMidConversationSystem:
exercised_on=[],
)
def test_unflagged_model_converts_system_reminder_and_succeeds(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = _register_invoke_deployment(
endpoints_client, resources, UNFLAGGED_INVOKE_MODEL
)
key = resources.key(models=[model])
model = _register_invoke_deployment(proxy, resources, UNFLAGGED_INVOKE_MODEL)
client = sdk.anthropic(resources.key(models=[model]))
system_block = _cacheable_system_block(unique_marker())
primed = _prime_prompt_cache(endpoints_client, key, model, system_block)
primed = _prime_prompt_cache(client, model, system_block)
reminder_turn_body = RichMessagesRequest(
model=model,
system=[system_block],
messages=[
_user_turn(primed.first_user_text, cached=True),
_system_reminder_turn(),
RichMessage(role="assistant", content=[TextBlock(text="OK.")]),
_user_turn("Reply with one word again.", cached=True),
],
)
second = unwrap(_post_messages(endpoints_client, key, reminder_turn_body))
second = _send(client, model, system_block, _reminder_turn_messages(primed))
assert second.role == "assistant", (
f"{model}: unexpected role {second.role!r}"
)
assert second.text.strip(), (
assert second.role == "assistant", f"{model}: unexpected role {second.role!r}"
assert _text(second).strip(), (
f"{model}: conversation with a mid-conversation system reminder "
f"returned no text; the reminder was forwarded in place to a model "
f"that rejects role 'system' inside messages instead of being converted to a user turn"
)
assert second.usage.cache_read_input_tokens >= primed.full_prefix_tokens, (
assert (second.usage.cache_read_input_tokens or 0) >= primed.full_prefix_tokens, (
f"{model}: reminder turn read {second.usage.cache_read_input_tokens} "
f"cached tokens, expected at least the {primed.full_prefix_tokens} "
f"cached on turn one ({primed.prefix_read_tokens} system prefix + "

196
tests/e2e/llm_translation/test_messages_mid_conversation_system_native_providers_e2e.py
View file

@ -24,27 +24,28 @@ entry whose prefix spans ``system`` plus message turns is invalidated when the
reminder is hoisted (the ``system`` field mutates and a turn disappears from
``messages``), while an entry ending at the system block itself would survive
the hoist and mask the regression.
Calls go through the real Anthropic SDK (LIT-4577). The SDK's ``MessageParam``
type only admits user/assistant roles, so the system reminder turn is cast to
it; the SDK serializes the dict verbatim, which is exactly the wire shape under
test.
"""
from __future__ import annotations
import time
from collections.abc import Sequence
from typing import cast
import pytest
from pydantic import BaseModel
from anthropic import Anthropic
from anthropic.types import Message, MessageParam, TextBlockParam
from e2e_config import unique_marker
from e2e_http import Result, unwrap
from endpoints_client import (
CacheControl,
EndpointsClient,
MessagesResult,
RichMessage,
RichMessagesRequest,
TextBlock,
)
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from proxy_client import ProxyClient
from pydantic import BaseModel
from sdk_clients import NO_PROXY_CACHE, SdkClients
pytestmark = pytest.mark.e2e
@ -69,46 +70,54 @@ def _vertex_params(model: str, location: str) -> LiteLLMParamsBody:
)
def _cacheable_system_block(marker: str) -> TextBlock:
def _cacheable_system_block(marker: str) -> TextBlockParam:
"""A system prompt at roughly twice the 4096-token minimum cacheable size of
Haiku 4.5 (the smallest model here), unique per run so no other run's cache
entry can satisfy the read. The marker appears once instead of in every
paragraph: repeating it swung the block's size by ~1800 tokens with the
marker's own tokenization and left it under the minimum on ~15% of runs, so
the system breakpoint went uncached and the priming loop never saw a read."""
text = f"Run {marker}.\n" + " ".join(
f"Reference paragraph {index}." for index in range(1500)
text = f"Run {marker}.\n" + " ".join(f"Reference paragraph {index}." for index in range(1500))
return {"type": "text", "text": text, "cache_control": {"type": "ephemeral"}}
def _user_turn(text: str, *, cached: bool = False) -> MessageParam:
block: TextBlockParam = (
{"type": "text", "text": text, "cache_control": {"type": "ephemeral"}}
if cached
else {"type": "text", "text": text}
)
return TextBlock(text=text, cache_control=CacheControl())
return {"role": "user", "content": [block]}
def _user_turn(text: str, *, cached: bool = False) -> RichMessage:
block = TextBlock(text=text, cache_control=CacheControl() if cached else None)
return RichMessage(role="user", content=[block])
def _system_reminder_turn() -> RichMessage:
return RichMessage(
role="system",
content=[TextBlock(text="<system-reminder>Answer with exactly one word.</system-reminder>")],
def _system_reminder_turn() -> MessageParam:
return cast(
"MessageParam",
{
"role": "system",
"content": [{"type": "text", "text": "<system-reminder>Answer with exactly one word.</system-reminder>"}],
},
)
def _post_messages(client: EndpointsClient, key: str, body: RichMessagesRequest) -> Result[MessagesResult]:
return client.proxy.transport.post(
"/v1/messages",
headers=client.proxy.transport.bearer(key),
json=body,
response_type=MessagesResult,
def _assistant_turn(text: str) -> MessageParam:
return {"role": "assistant", "content": [{"type": "text", "text": text}]}
def _text(message: Message) -> str:
return "".join(block.text for block in message.content if block.type == "text")
def _send(client: Anthropic, model: str, system_block: TextBlockParam, messages: Sequence[MessageParam]) -> Message:
return client.messages.create(
model=model, max_tokens=64, system=[system_block], messages=messages, extra_body=NO_PROXY_CACHE
)
def _register_deployment(
client: EndpointsClient, resources: ResourceManager, params: LiteLLMParamsBody
) -> str:
def _register_deployment(proxy: ProxyClient, resources: ResourceManager, params: LiteLLMParamsBody) -> str:
model = f"e2e-midsys-{unique_marker()}"
model_id = client.create_model(model, params)
resources.defer(lambda: client.delete_model(model_id))
model_id = proxy.create_model(model, params)
resources.defer(lambda: proxy.delete_model(model_id))
return model
@ -130,9 +139,7 @@ class PrimedCache(BaseModel):
return self.prefix_read_tokens + self.first_turn_creation_tokens
def _prime_prompt_cache(
client: EndpointsClient, key: str, model: str, system_block: TextBlock
) -> PrimedCache:
def _prime_prompt_cache(client: Anthropic, model: str, system_block: TextBlockParam) -> PrimedCache:
"""Send first-turn calls (fresh cache-marked user turn each attempt,
identical system prefix) until one both reads the system prefix back from
cache and writes its own user-turn chunk, then re-send that exact turn until
@ -144,19 +151,17 @@ def _prime_prompt_cache(
deadline = time.monotonic() + CACHE_PRIMING_DEADLINE_SECONDS
while True:
user_text = _first_turn_user_text(unique_marker())
body = RichMessagesRequest(
model=model,
system=[system_block],
messages=[_user_turn(user_text, cached=True)],
)
usage = unwrap(_post_messages(client, key, body)).usage
if usage.cache_read_input_tokens > 0 and usage.cache_creation_input_tokens > 0:
first_turn = (_user_turn(user_text, cached=True),)
usage = _send(client, model, system_block, first_turn).usage
read_tokens = usage.cache_read_input_tokens or 0
creation_tokens = usage.cache_creation_input_tokens or 0
if read_tokens > 0 and creation_tokens > 0:
primed = PrimedCache(
first_user_text=user_text,
prefix_read_tokens=usage.cache_read_input_tokens,
first_turn_creation_tokens=usage.cache_creation_input_tokens,
prefix_read_tokens=read_tokens,
first_turn_creation_tokens=creation_tokens,
)
if _first_turn_reads_back(client, key, body, primed.full_prefix_tokens, deadline):
if _first_turn_reads_back(client, model, system_block, first_turn, primed.full_prefix_tokens, deadline):
return primed
if time.monotonic() >= deadline:
pytest.fail(
@ -167,15 +172,20 @@ def _prime_prompt_cache(
def _reads_full_prefix(
client: EndpointsClient, key: str, body: RichMessagesRequest, full_prefix_tokens: int
client: Anthropic,
model: str,
system_block: TextBlockParam,
messages: Sequence[MessageParam],
full_prefix_tokens: int,
) -> bool:
return unwrap(_post_messages(client, key, body)).usage.cache_read_input_tokens >= full_prefix_tokens
return (_send(client, model, system_block, messages).usage.cache_read_input_tokens or 0) >= full_prefix_tokens
def _first_turn_reads_back(
client: EndpointsClient,
key: str,
body: RichMessagesRequest,
client: Anthropic,
model: str,
system_block: TextBlockParam,
messages: Sequence[MessageParam],
full_prefix_tokens: int,
deadline: float,
) -> bool:
@ -184,12 +194,24 @@ def _first_turn_reads_back(
fresh entry can be missing from the region the next request lands on; each miss
re-creates the entry there, so the streak converges as the regions warm up."""
while time.monotonic() < deadline:
if all(_reads_full_prefix(client, key, body, full_prefix_tokens) for _ in range(CACHE_WARM_CONSECUTIVE_READS)):
if all(
_reads_full_prefix(client, model, system_block, messages, full_prefix_tokens)
for _ in range(CACHE_WARM_CONSECUTIVE_READS)
):
return True
time.sleep(CACHE_PRIMING_INTERVAL_SECONDS)
return False
def _reminder_turn_messages(primed: PrimedCache) -> tuple[MessageParam, ...]:
return (
_user_turn(primed.first_user_text, cached=True),
_system_reminder_turn(),
_assistant_turn("OK."),
_user_turn("Reply with one word again.", cached=True),
)
#: Why the flagged-model cache checks are skipped rather than failing. The
#: assertions below are correct and must be restored unchanged when the bug is
#: fixed; they are the regression guard for a real billing cost.
@ -209,28 +231,18 @@ MID_CONVERSATION_CACHE_SKIP_REASON = (
def _assert_flagged_model_keeps_cache(
client: EndpointsClient, resources: ResourceManager, params: LiteLLMParamsBody
proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients, params: LiteLLMParamsBody
) -> None:
model = _register_deployment(client, resources, params)
key = resources.key(models=[model])
model = _register_deployment(proxy, resources, params)
client = sdk.anthropic(resources.key(models=[model]))
system_block = _cacheable_system_block(unique_marker())
primed = _prime_prompt_cache(client, key, model, system_block)
primed = _prime_prompt_cache(client, model, system_block)
reminder_turn_body = RichMessagesRequest(
model=model,
system=[system_block],
messages=[
_user_turn(primed.first_user_text, cached=True),
_system_reminder_turn(),
RichMessage(role="assistant", content=[TextBlock(text="OK.")]),
_user_turn("Reply with one word again.", cached=True),
],
)
second = unwrap(_post_messages(client, key, reminder_turn_body))
second = _send(client, model, system_block, _reminder_turn_messages(primed))
assert second.text.strip(), f"{model}: reminder turn returned no completion text"
assert second.usage.cache_read_input_tokens >= primed.full_prefix_tokens, (
assert _text(second).strip(), f"{model}: reminder turn returned no completion text"
assert (second.usage.cache_read_input_tokens or 0) >= primed.full_prefix_tokens, (
f"{model}: turn with a mid-conversation system reminder read "
f"{second.usage.cache_read_input_tokens} cached tokens, expected at "
f"least the {primed.full_prefix_tokens} cached on turn one "
@ -242,33 +254,23 @@ def _assert_flagged_model_keeps_cache(
def _assert_unflagged_model_converts_and_succeeds(
client: EndpointsClient, resources: ResourceManager, params: LiteLLMParamsBody
proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients, params: LiteLLMParamsBody
) -> None:
model = _register_deployment(client, resources, params)
key = resources.key(models=[model])
model = _register_deployment(proxy, resources, params)
client = sdk.anthropic(resources.key(models=[model]))
system_block = _cacheable_system_block(unique_marker())
primed = _prime_prompt_cache(client, key, model, system_block)
primed = _prime_prompt_cache(client, model, system_block)
reminder_turn_body = RichMessagesRequest(
model=model,
system=[system_block],
messages=[
_user_turn(primed.first_user_text, cached=True),
_system_reminder_turn(),
RichMessage(role="assistant", content=[TextBlock(text="OK.")]),
_user_turn("Reply with one word again.", cached=True),
],
)
second = unwrap(_post_messages(client, key, reminder_turn_body))
second = _send(client, model, system_block, _reminder_turn_messages(primed))
assert second.role == "assistant", f"{model}: unexpected role {second.role!r}"
assert second.text.strip(), (
assert _text(second).strip(), (
f"{model}: conversation with a mid-conversation system reminder returned "
f"no text; the reminder was forwarded in place to a model that rejects "
f"role 'system' inside messages instead of being converted to a user turn"
)
assert second.usage.cache_read_input_tokens >= primed.full_prefix_tokens, (
assert (second.usage.cache_read_input_tokens or 0) >= primed.full_prefix_tokens, (
f"{model}: reminder turn read {second.usage.cache_read_input_tokens} cached "
f"tokens, expected at least the {primed.full_prefix_tokens} cached on turn "
f"one ({primed.prefix_read_tokens} system prefix + "
@ -289,20 +291,18 @@ class TestAzureFoundryMidConversationSystem:
exercised_on=[],
)
def test_flagged_model_keeps_prompt_cache_across_system_reminder(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
_assert_flagged_model_keeps_cache(endpoints_client, resources, _azure_params(self.FLAGGED_MODEL))
_assert_flagged_model_keeps_cache(proxy, resources, sdk, _azure_params(self.FLAGGED_MODEL))
@pytest.mark.covers(
"llm.messages.azure_foundry.mid_conversation_system.nonstream.works",
exercised_on=[],
)
def test_unflagged_model_converts_system_reminder_and_succeeds(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
_assert_unflagged_model_converts_and_succeeds(
endpoints_client, resources, _azure_params(self.UNFLAGGED_MODEL)
)
_assert_unflagged_model_converts_and_succeeds(proxy, resources, sdk, _azure_params(self.UNFLAGGED_MODEL))
class TestVertexMidConversationSystem:
@ -323,10 +323,10 @@ class TestVertexMidConversationSystem:
exercised_on=[],
)
def test_flagged_model_keeps_prompt_cache_across_system_reminder(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
_assert_flagged_model_keeps_cache(
endpoints_client, resources, _vertex_params(self.FLAGGED_MODEL, self.FLAGGED_LOCATION)
proxy, resources, sdk, _vertex_params(self.FLAGGED_MODEL, self.FLAGGED_LOCATION)
)
@pytest.mark.covers(
@ -334,8 +334,8 @@ class TestVertexMidConversationSystem:
exercised_on=[],
)
def test_unflagged_model_converts_system_reminder_and_succeeds(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
_assert_unflagged_model_converts_and_succeeds(
endpoints_client, resources, _vertex_params(self.UNFLAGGED_MODEL, self.UNFLAGGED_LOCATION)
proxy, resources, sdk, _vertex_params(self.UNFLAGGED_MODEL, self.UNFLAGGED_LOCATION)
)

72
tests/e2e/llm_translation/test_moderations_e2e.py
View file

@ -1,19 +1,23 @@
"""Live e2e: POST /v1/moderations classifies content against the provider policy.
Registers OpenAI's omni moderation model at runtime and asserts the product
promise on both sides of the decision: clearly violent text comes back flagged
with at least one policy category tripped, and benign text comes back not flagged.
Registers OpenAI's omni moderation model at runtime, drives it through the real
OpenAI SDK (LIT-4577), and asserts the product promise on both sides of the
decision: clearly violent text comes back flagged with at least one policy
category tripped, and benign text comes back not flagged. The malformed-body
negative stays on the shared transport, since the SDK refuses to send it.
"""
from __future__ import annotations
import pytest
from e2e_config import unique_marker
from e2e_http import assert_client_error, unwrap
from endpoints_client import EndpointsClient
from e2e_http import assert_client_error
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from pydantic import BaseModel
from openai.types import Moderation
from proxy_client import ProxyClient
from pydantic import BaseModel, TypeAdapter
from sdk_clients import SdkClients
pytestmark = pytest.mark.e2e
@ -26,59 +30,63 @@ class _OptionalModerationBody(BaseModel):
input: str | None = None
def _register_moderation_model(
endpoints_client: EndpointsClient, resources: ResourceManager
) -> str:
def _register_moderation_model(proxy: ProxyClient, resources: ResourceManager) -> str:
model = f"e2e-moderation-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(
model="openai/omni-moderation-latest", api_key="os.environ/OPENAI_API_KEY"
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
return model
_CATEGORY_FLAGS = TypeAdapter(dict[str, bool | None])
def _flagged_categories(item: Moderation) -> tuple[str, ...]:
flags = _CATEGORY_FLAGS.validate_python(item.categories.model_dump())
return tuple(name for name, hit in flags.items() if hit)
class TestModerations:
@pytest.mark.covers("llm.moderations.openai.basic.nonstream.works")
def test_moderations_flags_violent_content(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = _register_moderation_model(endpoints_client, resources)
key = resources.key()
model = _register_moderation_model(proxy, resources)
client = sdk.openai(resources.key())
result = unwrap(endpoints_client.moderations(key, model, VIOLENT_TEXT))
item = result.first
assert item is not None, f"/moderations returned no results: {result}"
assert item.flagged, f"violent text was not flagged: {item}"
assert item.flagged_categories, (
f"flagged result reported no true category: {item}"
)
moderation = client.moderations.create(model=model, input=VIOLENT_TEXT)
assert moderation.results, f"/moderations returned no results: {moderation!r}"
item = moderation.results[0]
assert item.flagged, f"violent text was not flagged: {item!r}"
assert _flagged_categories(item), f"flagged result reported no true category: {item!r}"
def test_moderations_passes_benign_content(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = _register_moderation_model(endpoints_client, resources)
key = resources.key()
model = _register_moderation_model(proxy, resources)
client = sdk.openai(resources.key())
result = unwrap(endpoints_client.moderations(key, model, BENIGN_TEXT))
item = result.first
assert item is not None, f"/moderations returned no results: {result}"
moderation = client.moderations.create(model=model, input=BENIGN_TEXT)
assert moderation.results, f"/moderations returned no results: {moderation!r}"
item = moderation.results[0]
assert not item.flagged, (
f"benign text was flagged as {item.flagged_categories}: {item}"
f"benign text was flagged as {_flagged_categories(item)}: {item!r}"
)
@pytest.mark.skip(reason="stage red: product gap, /v1/moderations 500s (KeyError 'input') on missing input instead of 400")
@pytest.mark.covers("llm.moderations.openai.input_validation.nonstream.works")
def test_missing_input_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
model = _register_moderation_model(endpoints_client, resources)
model = _register_moderation_model(proxy, resources)
key = resources.key()
result = endpoints_client.proxy.transport.send(
result = proxy.transport.send(
"/v1/moderations",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalModerationBody(model=model),
)
assert_client_error(result, "moderations missing input")

20
tests/e2e/llm_translation/test_ocr_rust_e2e.py
View file

@ -21,9 +21,9 @@ from typing import Protocol
import pytest
from e2e_config import unique_marker
from e2e_http import assert_client_error, unwrap
from endpoints_client import EndpointsClient
from lifecycle import ResourceManager
from models import LiteLLMParamsBody, OcrBody, OcrDocument, OcrResponse
from proxy_client import ProxyClient
from pydantic import BaseModel
pytestmark = pytest.mark.e2e
@ -149,28 +149,28 @@ def _assert_ocr_document(response: OcrResponse) -> None:
class TestRustOcrGateway:
@pytest.mark.parametrize("case", RUST_OCR_CASES, ids=_CASE_IDS)
def test_rust_ocr_response(
self, endpoints_client: EndpointsClient, resources: ResourceManager, case: _OcrCase
self, proxy: ProxyClient, resources: ResourceManager, case: _OcrCase
) -> None:
model = f"rust-ocr-{case.suffix}-{unique_marker()}"
model_id = endpoints_client.create_model(model, case.provider.litellm_params())
resources.defer(lambda: endpoints_client.delete_model(model_id))
model_id = proxy.create_model(model, case.provider.litellm_params())
resources.defer(lambda: proxy.delete_model(model_id))
key = resources.key()
response = unwrap(endpoints_client.proxy.ocr(key, OcrBody(model=model, document=case.document)))
response = unwrap(proxy.ocr(key, OcrBody(model=model, document=case.document)))
_assert_ocr_document(response)
@pytest.mark.skip(reason="stage red: product gap, /v1/ocr 500s (aocr TypeError) on missing document instead of 400")
@pytest.mark.covers("llm.ocr.openai.input_validation.nonstream.works")
def test_missing_document_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
model = f"rust-ocr-val-{unique_marker()}"
model_id = endpoints_client.create_model(model, MistralOcr().litellm_params())
resources.defer(lambda: endpoints_client.delete_model(model_id))
model_id = proxy.create_model(model, MistralOcr().litellm_params())
resources.defer(lambda: proxy.delete_model(model_id))
key = resources.key()
result = endpoints_client.proxy.transport.send(
result = proxy.transport.send(
"/v1/ocr",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalOcrBody(model=model),
)
assert_client_error(result, "ocr missing document")

8
tests/e2e/llm_translation/test_passthrough_headers_e2e.py
View file

@ -20,9 +20,8 @@ from pydantic import BaseModel, Field
from e2e_config import unique_marker
from e2e_http import AuthHeaders, NoBody, require_successful_call, unwrap
from endpoints_client import MessagesResult
from lifecycle import ResourceManager
from models import ChatMessage, KeyGenerateBody
from models import AnthropicMessagesResponse, ChatMessage, KeyGenerateBody
from passthrough_client import PassthroughClient
pytestmark = pytest.mark.e2e
@ -165,8 +164,9 @@ class TestPassthroughHeaders:
json=_messages_body(),
)
require_successful_call(result)
completion = MessagesResult.model_validate_json(result.body)
assert completion.text.strip(), (
completion = AnthropicMessagesResponse.model_validate_json(result.body)
text = "".join(block.text or "" for block in (completion.content or []))
assert text.strip(), (
f"static x-api-key must reach Anthropic for the call to succeed at all; got {result.body[:300]}"
)

52
tests/e2e/llm_translation/test_rerank_e2e.py
View file

@ -1,19 +1,20 @@
"""Live e2e: POST /v1/rerank ranks documents by relevance.
Registers a Cohere rerank deployment at runtime and asserts the endpoint returns
scored results within the requested top_n. Migrated from
Registers Cohere and Bedrock rerank deployments at runtime and asserts the
endpoint returns scored results within the requested top_n. No official
OpenAI/Anthropic SDK covers /v1/rerank, so the call rides the shared typed
transport via ProxyClient.rerank. Migrated from
litellm-regression-tests/tests/test_inference_endpoints.py.
"""
from __future__ import annotations
import pytest
from e2e_config import unique_marker
from e2e_http import require_successful_call
from endpoints_client import EndpointsClient, RerankResult
from e2e_http import unwrap
from lifecycle import ResourceManager
from models import LiteLLMParamsBody
from models import LiteLLMParamsBody, RerankBody, RerankResponse
from proxy_client import ProxyClient
pytestmark = pytest.mark.e2e
@ -26,38 +27,39 @@ DOCUMENTS = [
QUERY = "What is the capital of the United States?"
def _assert_top_n_scored(body: str) -> None:
parsed = RerankResult.model_validate_json(body)
assert parsed.results, f"/rerank returned no results: {body[:300]}"
assert len(parsed.results) <= 3, f"top_n=3 not honored: {body[:300]}"
assert parsed.results[0].relevance_score is not None, (
f"top rerank result has no relevance_score: {body[:300]}"
def _assert_top_n_scored(response: RerankResponse) -> None:
assert response.results, f"/rerank returned no results: {response!r}"
assert len(response.results) <= 3, f"top_n=3 not honored: {response!r}"
assert response.results[0].relevance_score is not None, (
f"top rerank result has no relevance_score: {response!r}"
)
def _rerank_top_3(proxy: ProxyClient, key: str, model: str) -> RerankResponse:
return unwrap(
proxy.rerank(key, RerankBody(model=model, query=QUERY, documents=DOCUMENTS, top_n=3))
)
class TestRerank:
@pytest.mark.covers("llm.rerank.cohere.basic.nonstream.works")
def test_rerank_scores_top_n(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
def test_rerank_scores_top_n(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model = f"e2e-rerank-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(model="cohere/rerank-v3.5", api_key="os.environ/COHERE_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
key = resources.key()
result = endpoints_client.rerank(key, model, QUERY, DOCUMENTS, top_n=3)
require_successful_call(result)
_assert_top_n_scored(result.body)
_assert_top_n_scored(_rerank_top_3(proxy, key, model))
@pytest.mark.covers("llm.rerank.bedrock.basic.nonstream.works", exercised_on=["rerank"])
def test_bedrock_rerank_scores_top_n(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager
) -> None:
model = f"e2e-bedrock-rerank-{unique_marker()}"
model_id = endpoints_client.create_model(
model_id = proxy.create_model(
model,
LiteLLMParamsBody(
model="bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0",
@ -66,9 +68,7 @@ class TestRerank:
aws_region_name="os.environ/AWS_REGION",
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
key = resources.key()
result = endpoints_client.rerank(key, model, QUERY, DOCUMENTS, top_n=3)
require_successful_call(result)
_assert_top_n_scored(result.body)
_assert_top_n_scored(_rerank_top_3(proxy, key, model))

437
tests/e2e/llm_translation/test_responses_e2e.py
View file

@ -1,12 +1,15 @@
"""Live e2e: POST /v1/responses returns a real completion.
Registers an OpenAI deployment at runtime, drives the Responses API through the
gateway, and asserts output text came back. Migrated from
Registers an OpenAI deployment at runtime and drives the Responses API through
the gateway with the real OpenAI SDK, the client customers actually use
(LIT-4577), asserting output text came back. Malformed bodies the SDK refuses
to build stay on the shared transport. Migrated from
litellm-regression-tests/tests/test_inference_endpoints.py.
"""
from __future__ import annotations
import contextlib
import json
import threading
from collections.abc import Mapping
@ -14,26 +17,23 @@ from dataclasses import dataclass, field
from types import MappingProxyType
from typing import Final, cast
import openai
import pytest
from e2e_config import PROVIDER_EDGE_ADVERTISE_HOST, PROVIDER_EDGE_BIND_HOST, unique_marker
from e2e_http import (
assert_client_error,
require_successful_call,
)
from endpoints_client import (
EndpointsClient,
FunctionParameterProperty,
FunctionParameters,
ResponsesFunctionTool,
ResponsesOutputTextDeltaEvent,
ResponsesResult,
ResponsesStreamEventType,
)
from e2e_http import assert_client_error
from lifecycle import ResourceManager
from models import ChatBody, ChatMessage, LiteLLMParamsBody
from openai.types.responses import (
FunctionToolParam,
Response,
ResponseFunctionToolCall,
ResponseInputParam,
)
from provider_edge import LiveEdge, start_provider_edge
from provider_edge_bedrock import bedrock_signer
from pydantic import BaseModel, ValidationError
from proxy_client import ProxyClient
from pydantic import BaseModel
from sdk_clients import NO_PROXY_CACHE, SdkClients
pytestmark = pytest.mark.e2e
@ -45,6 +45,8 @@ class _OptionalResponsesBody(BaseModel):
BEDROCK_CONVERSE_BACKEND = "bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0"
INSTRUCTIONS = "You are a helpful assistant"
CAT_IMAGE_URL = "https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg"
BEDROCK_EDGE_REGION: Final = "us-east-1"
BEDROCK_EDGE_MOUNT: Final = f"bedrock/{BEDROCK_EDGE_REGION}"
@ -73,14 +75,25 @@ class ConverseRequestCapture:
return tuple(self._bodies)
WEATHER_TOOL = ResponsesFunctionTool(
name="get_weather",
description="Get the weather for a location",
parameters=FunctionParameters(
properties={"location": FunctionParameterProperty(type="string")},
required=["location"],
),
)
WEATHER_TOOL: FunctionToolParam = {
"type": "function",
"name": "get_weather",
"description": "Get the weather for a location",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"],
},
"strict": False,
}
def _openai_params() -> LiteLLMParamsBody:
return LiteLLMParamsBody(model="openai/gpt-4o-mini", api_key="os.environ/OPENAI_API_KEY")
def _anthropic_params() -> LiteLLMParamsBody:
return LiteLLMParamsBody(model="anthropic/claude-haiku-4-5", api_key="os.environ/ANTHROPIC_API_KEY")
def _bedrock_params() -> LiteLLMParamsBody:
@ -92,6 +105,27 @@ def _bedrock_params() -> LiteLLMParamsBody:
)
def _register(
proxy: ProxyClient, resources: ResourceManager, params: LiteLLMParamsBody, prefix: str = "e2e-responses"
) -> str:
model = f"{prefix}-{unique_marker()}"
model_id = proxy.create_model(model, params)
resources.defer(lambda: proxy.delete_model(model_id))
return model
def _function_calls(response: Response) -> tuple[ResponseFunctionToolCall, ...]:
return tuple(item for item in response.output if isinstance(item, ResponseFunctionToolCall))
def _assert_weather_call(response: Response) -> None:
function_call = next((call for call in _function_calls(response) if call.name == "get_weather"), None)
assert function_call is not None, f"no get_weather function call: {response.output!r}"
raw_arguments = cast(object, json.loads(function_call.arguments))
arguments = WeatherArguments.model_validate(raw_arguments)
assert arguments.location, f"function call arguments missing location: {function_call.arguments}"
class WeatherArguments(BaseModel):
location: str
@ -99,250 +133,184 @@ class WeatherArguments(BaseModel):
class TestResponses:
@pytest.mark.covers("llm.responses.openai.basic.nonstream.works")
def test_responses_returns_completion(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-responses-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(model="openai/gpt-4o-mini", api_key="os.environ/OPENAI_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
model = _register(proxy, resources, _openai_params())
client = sdk.openai(resources.key())
result = endpoints_client.responses(key, model, "reply with one word")
require_successful_call(result)
parsed = ResponsesResult.model_validate_json(result.body)
assert parsed.text.strip(), f"/responses returned no output text: {result.body[:300]}"
response = client.responses.create(
model=model, input="reply with one word", instructions=INSTRUCTIONS, extra_body=NO_PROXY_CACHE
)
assert response.output_text.strip(), f"/responses returned no output text: {response.output!r}"
@pytest.mark.covers("llm.responses.openai.basic.stream.works")
def test_responses_streaming_returns_completion(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-responses-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(model="openai/gpt-4o-mini", api_key="os.environ/OPENAI_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
model = _register(proxy, resources, _openai_params())
client = sdk.openai(resources.key())
result = endpoints_client.responses(key, model, "reply with one word", stream=True)
require_successful_call(result)
delta_events = tuple(
parsed
for event in result.stream_events
if (parsed := _parse_stream_event(event)) is not None
stream = client.responses.create(
model=model,
input="reply with one word",
instructions=INSTRUCTIONS,
stream=True,
extra_body=NO_PROXY_CACHE,
)
events = tuple(stream)
assert events, "responses stream returned no events"
deltas = tuple(event.delta for event in events if event.type == "response.output_text.delta")
assert any(delta for delta in deltas), "responses stream returned no text deltas"
assert events[-1].type == "response.completed", (
f"responses stream did not terminate with response.completed: {events[-1].type}"
)
assert any(event.delta for event in delta_events), "responses stream returned no text deltas"
assert result.stream_events, "responses stream returned no events"
assert (
ResponsesStreamEventType.model_validate_json(result.stream_events[-1]).type
== "response.completed"
), "responses stream did not terminate with response.completed"
@pytest.mark.covers("llm.responses.openai.basic.nonstream.cost_logged")
def test_responses_logs_cost(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model = f"e2e-responses-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(model="openai/gpt-4o-mini", api_key="os.environ/OPENAI_API_KEY"),
def test_responses_logs_cost(self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients) -> None:
model = _register(proxy, resources, _openai_params())
client = sdk.openai(resources.key())
raw = client.responses.with_raw_response.create(
model=model,
input=f"reply with one word {unique_marker()}",
instructions=INSTRUCTIONS,
extra_body=NO_PROXY_CACHE,
)
response = raw.parse()
assert response.output_text.strip(), f"/responses returned no output text: {response.output!r}"
assert raw.headers.get("x-litellm-call-id") and response.id, (
f"missing response identifiers: id={response.id!r}, headers={dict(raw.headers)}"
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
result = endpoints_client.responses(key, model, f"reply with one word {unique_marker()}")
require_successful_call(result)
parsed = ResponsesResult.model_validate_json(result.body)
assert parsed.text.strip(), f"/responses returned no output text: {result.body[:300]}"
assert result.call_id and parsed.id, f"missing response identifiers: {result.body[:300]}"
rows = endpoints_client.proxy.poll_logs_for_request_id(
parsed.id,
rows = proxy.poll_logs_for_request_id(
response.id,
predicate=lambda logged_rows: any((row.spend or 0) > 0 for row in logged_rows),
)
row = next((logged_row for logged_row in rows if (logged_row.spend or 0) > 0), None)
assert row is not None, f"no costed spend row for response id {parsed.id}"
assert row is not None, f"no costed spend row for response id {response.id}"
assert "gpt-4o-mini" in (row.model or ""), f"unexpected spend row model: {row.model}"
@pytest.mark.covers("llm.responses.openai.tool_use.nonstream.works")
def test_responses_returns_function_call(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-responses-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(model="openai/gpt-4o-mini", api_key="os.environ/OPENAI_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
model = _register(proxy, resources, _openai_params())
client = sdk.openai(resources.key())
result = endpoints_client.responses_with_tools(
key,
model,
"What is the weather in San Francisco? Use the get_weather tool.",
[
ResponsesFunctionTool(
name="get_weather",
description="Get the weather for a location",
parameters=FunctionParameters(
properties={"location": FunctionParameterProperty(type="string")},
required=["location"],
),
)
],
response = client.responses.create(
model=model,
input="What is the weather in San Francisco? Use the get_weather tool.",
instructions=INSTRUCTIONS,
tools=[WEATHER_TOOL],
extra_body=NO_PROXY_CACHE,
)
require_successful_call(result)
parsed = ResponsesResult.model_validate_json(result.body)
function_call = next(
(call for call in parsed.function_calls if call.name == "get_weather"),
None,
)
assert function_call is not None, f"no get_weather function call: {result.body[:500]}"
assert function_call.arguments is not None
raw_arguments = cast(object, json.loads(function_call.arguments))
arguments = WeatherArguments.model_validate(raw_arguments)
assert arguments.location, f"function call arguments missing location: {function_call.arguments}"
_assert_weather_call(response)
@pytest.mark.covers("llm.responses.openai.vision.nonstream.works")
def test_responses_vision_describes_image(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-responses-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
model = _register(
proxy,
resources,
LiteLLMParamsBody(model="openai/gpt-4o", api_key="os.environ/OPENAI_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
client = sdk.openai(resources.key())
result = endpoints_client.responses_vision(
key,
model,
"What animal is shown in this image? Answer in one word",
"https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg",
vision_input: ResponseInputParam = [
{
"role": "user",
"content": [
{"type": "input_text", "text": "What animal is shown in this image? Answer in one word"},
{"type": "input_image", "image_url": CAT_IMAGE_URL, "detail": "auto"},
],
}
]
response = client.responses.create(
model=model, input=vision_input, instructions=INSTRUCTIONS, extra_body=NO_PROXY_CACHE
)
text = response.output_text.strip().lower()
assert text, f"/responses vision returned no output text: {response.output!r}"
assert any(keyword in text for keyword in ("cat", "feline")), (
f"vision response did not describe the image: {text[:300]}"
)
require_successful_call(result)
parsed = ResponsesResult.model_validate_json(result.body)
text = parsed.text.strip().lower()
assert text, f"/responses vision returned no output text: {result.body[:300]}"
assert any(
keyword in text
for keyword in ("cat", "feline")
), f"vision response did not describe the image: {parsed.text[:300]}"
@pytest.mark.covers("llm.responses.anthropic.basic.nonstream.works")
def test_responses_anthropic_returns_completion(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-responses-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(
model="anthropic/claude-haiku-4-5", api_key="os.environ/ANTHROPIC_API_KEY"
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
model = _register(proxy, resources, _anthropic_params())
client = sdk.openai(resources.key())
result = endpoints_client.responses(key, model, "reply with one word")
require_successful_call(result)
parsed = ResponsesResult.model_validate_json(result.body)
assert parsed.text.strip(), f"/responses returned no output text: {result.body[:300]}"
response = client.responses.create(
model=model, input="reply with one word", instructions=INSTRUCTIONS, extra_body=NO_PROXY_CACHE
)
assert response.output_text.strip(), f"/responses returned no output text: {response.output!r}"
@pytest.mark.covers("llm.responses.anthropic.tool_use.nonstream.works")
def test_responses_anthropic_returns_function_call(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-responses-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(
model="anthropic/claude-haiku-4-5", api_key="os.environ/ANTHROPIC_API_KEY"
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
model = _register(proxy, resources, _anthropic_params())
client = sdk.openai(resources.key())
result = endpoints_client.responses_with_tools(
key,
model,
"What is the weather in San Francisco? Use the get_weather tool.",
[
ResponsesFunctionTool(
name="get_weather",
description="Get the weather for a location",
parameters=FunctionParameters(
properties={"location": FunctionParameterProperty(type="string")},
required=["location"],
),
)
],
response = client.responses.create(
model=model,
input="What is the weather in San Francisco? Use the get_weather tool.",
instructions=INSTRUCTIONS,
tools=[WEATHER_TOOL],
extra_body=NO_PROXY_CACHE,
)
require_successful_call(result)
parsed = ResponsesResult.model_validate_json(result.body)
function_call = next(
(call for call in parsed.function_calls if call.name == "get_weather"),
None,
)
assert function_call is not None, f"no get_weather function call: {result.body[:500]}"
assert function_call.arguments is not None
raw_arguments = cast(object, json.loads(function_call.arguments))
arguments = WeatherArguments.model_validate(raw_arguments)
assert arguments.location, f"function call arguments missing location: {function_call.arguments}"
_assert_weather_call(response)
@pytest.mark.covers("llm.responses.bedrock_converse.basic.nonstream.works")
def test_responses_bedrock_returns_completion(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-responses-{unique_marker()}"
model_id = endpoints_client.create_model(model, _bedrock_params())
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
model = _register(proxy, resources, _bedrock_params())
client = sdk.openai(resources.key())
result = endpoints_client.responses(key, model, "reply with one word")
require_successful_call(result)
parsed = ResponsesResult.model_validate_json(result.body)
assert parsed.text.strip(), f"/responses over bedrock returned no output text: {result.body[:300]}"
response = client.responses.create(
model=model, input="reply with one word", instructions=INSTRUCTIONS, extra_body=NO_PROXY_CACHE
)
assert response.output_text.strip(), f"/responses over bedrock returned no output text: {response.output!r}"
@pytest.mark.covers("llm.responses.bedrock_converse.tool_use.nonstream.works")
def test_responses_bedrock_returns_function_call(
self, endpoints_client: EndpointsClient, resources: ResourceManager
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients
) -> None:
model = f"e2e-responses-{unique_marker()}"
model_id = endpoints_client.create_model(model, _bedrock_params())
resources.defer(lambda: endpoints_client.delete_model(model_id))
key = resources.key()
model = _register(proxy, resources, _bedrock_params())
client = sdk.openai(resources.key())
result = endpoints_client.responses_with_tools(
key, model, "What is the weather in San Francisco? Use the get_weather tool.", [WEATHER_TOOL]
response = client.responses.create(
model=model,
input="What is the weather in San Francisco? Use the get_weather tool.",
instructions=INSTRUCTIONS,
tools=[WEATHER_TOOL],
extra_body=NO_PROXY_CACHE,
)
require_successful_call(result)
parsed = ResponsesResult.model_validate_json(result.body)
function_call = next((call for call in parsed.function_calls if call.name == "get_weather"), None)
assert function_call is not None, f"no get_weather function call over bedrock: {result.body[:500]}"
assert function_call.arguments is not None
raw_arguments = cast(object, json.loads(function_call.arguments))
arguments = WeatherArguments.model_validate(raw_arguments)
assert arguments.location, f"function call arguments missing location: {function_call.arguments}"
_assert_weather_call(response)
@pytest.mark.provider_edge_host
@pytest.mark.parametrize("endpoint", ["/v1/responses", "/v1/chat/completions"])
def test_bedrock_forwards_allowed_safety_identifier_as_additional_model_request_field(
self, endpoints_client: EndpointsClient, resources: ResourceManager, endpoint: str
self, proxy: ProxyClient, resources: ResourceManager, sdk: SdkClients, endpoint: str
) -> None:
"""Judges the Converse bodies the edge captured, not the reply: Claude on
Bedrock rejects the forwarded field with a 400, which the chat leg's
``Result`` carries as a value and the OpenAI SDK raises."""
capture: Final = ConverseRequestCapture()
edge: Final = start_provider_edge(
LiveEdge(observe_request=capture.observe, sign=bedrock_signer(BEDROCK_EDGE_REGION)),
mounts=MappingProxyType({BEDROCK_EDGE_MOUNT: f"https://bedrock-runtime.{BEDROCK_EDGE_REGION}.amazonaws.com"}),
mounts=MappingProxyType(
{BEDROCK_EDGE_MOUNT: f"https://bedrock-runtime.{BEDROCK_EDGE_REGION}.amazonaws.com"}
),
bind_host=PROVIDER_EDGE_BIND_HOST,
advertise_host=PROVIDER_EDGE_ADVERTISE_HOST,
)
resources.defer(edge.shutdown)
model: Final = f"e2e-responses-{unique_marker()}"
model_id: Final = endpoints_client.create_model(
model_id: Final = proxy.create_model(
model,
LiteLLMParamsBody(
model=BEDROCK_CONVERSE_BACKEND,
@ -353,14 +321,21 @@ class TestResponses:
allowed_openai_params=["safety_identifier"],
),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
resources.defer(lambda: proxy.delete_model(model_id))
key: Final = resources.key()
safety_identifier: Final = f"end-user-{unique_marker()}"
if endpoint == "/v1/responses":
endpoints_client.responses(key, model, "reply with one word", safety_identifier=safety_identifier)
with contextlib.suppress(openai.BadRequestError):
sdk.openai(key).responses.create(
model=model,
input="reply with one word",
instructions=INSTRUCTIONS,
safety_identifier=safety_identifier,
extra_body=NO_PROXY_CACHE,
)
else:
endpoints_client.proxy.chat(
proxy.chat(
key,
ChatBody(
model=model,
@ -375,59 +350,37 @@ class TestResponses:
f"{endpoint} did not forward safety_identifier to Bedrock Converse on every attempt: {capture.bodies}"
)
@pytest.mark.skip(reason="stage red: product gap, /v1/responses 500s (aresponses TypeError) on missing input instead of 400")
@pytest.mark.skip(
reason="stage red: product gap, /v1/responses 500s (aresponses TypeError) on missing input instead of 400"
)
@pytest.mark.covers("llm.responses.openai.input_validation.nonstream.works")
def test_missing_input_returns_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model = f"e2e-responses-val-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(model="openai/gpt-4o-mini", api_key="os.environ/OPENAI_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
def test_missing_input_returns_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model = _register(proxy, resources, _openai_params(), prefix="e2e-responses-val")
key = resources.key()
result = endpoints_client.proxy.transport.send(
result = proxy.transport.send(
"/v1/responses",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalResponsesBody(model=model),
)
assert_client_error(result, "responses missing input")
@pytest.mark.covers("llm.responses.openai.input_validation.nonstream.works")
def test_missing_model_returns_client_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
def test_missing_model_returns_client_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
key = resources.key()
result = endpoints_client.proxy.transport.send(
result = proxy.transport.send(
"/v1/responses",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalResponsesBody(input="ping"),
)
assert_client_error(result, "responses missing model")
@pytest.mark.covers("llm.responses.openai.input_validation.nonstream.works")
def test_empty_input_returns_client_error(
self, endpoints_client: EndpointsClient, resources: ResourceManager
) -> None:
model = f"e2e-responses-val-{unique_marker()}"
model_id = endpoints_client.create_model(
model,
LiteLLMParamsBody(model="openai/gpt-4o-mini", api_key="os.environ/OPENAI_API_KEY"),
)
resources.defer(lambda: endpoints_client.delete_model(model_id))
def test_empty_input_returns_client_error(self, proxy: ProxyClient, resources: ResourceManager) -> None:
model = _register(proxy, resources, _openai_params(), prefix="e2e-responses-val")
key = resources.key()
result = endpoints_client.proxy.transport.send(
result = proxy.transport.send(
"/v1/responses",
headers=endpoints_client.proxy.transport.bearer(key),
headers=proxy.transport.bearer(key),
json=_OptionalResponsesBody(model=model, input=""),
)
assert_client_error(result, "responses empty input")
def _parse_stream_event(
event: str,
) -> ResponsesOutputTextDeltaEvent | None:
try:
return ResponsesOutputTextDeltaEvent.model_validate_json(event)
except ValidationError:
return None

20
tests/e2e/models.py
View file

@ -697,6 +697,26 @@ class EmbedResponse(BaseModel):
model: str | None = None
# ---------- rerank ----------
class RerankBody(BaseModel):
model: str
query: str
documents: list[str]
top_n: int
cache: dict[str, bool] | None = {"no-cache": True}
class RerankItem(BaseModel):
index: int | None = None
relevance_score: float | None = None
class RerankResponse(BaseModel):
results: list[RerankItem] = []
# ---------- ocr ----------

12
tests/e2e/proxy_client.py
View file

@ -79,6 +79,8 @@ from models import (
ModelUpdateBody,
OcrBody,
OcrResponse,
RerankBody,
RerankResponse,
RouterCurrentValues,
RouterSettingsResponse,
SpendLogRow,
@ -940,6 +942,16 @@ class ProxyClient:
timeout=SLOW_PROVIDER_TIMEOUT_SECONDS,
)
def rerank(self, key: str, body: RerankBody) -> Result[RerankResponse]:
"""POST /v1/rerank (Cohere-format). No official OpenAI/Anthropic SDK
covers this route, so it stays on the shared typed transport."""
return self.transport.post(
"/v1/rerank",
headers=self.transport.bearer(key),
json=body,
response_type=RerankResponse,
)
def count_tokens(self, key: str, body: CountTokensBody) -> Result[CountTokensResponse]:
"""POST /v1/messages/count_tokens (Anthropic-native). Sends the
anthropic-version header so the native path accepts it; harmless on the

2
uv.lock generated
View file

@ -4719,6 +4719,7 @@ dev = [
{ name = "vcrpy" },
]
e2e-dev = [
{ name = "anthropic" },
{ name = "locust" },
{ name = "mcp" },
{ name = "playwright" },
@ -4920,6 +4921,7 @@ dev = [
{ name = "vcrpy", specifier = "==8.2.1" },
]
e2e-dev = [
{ name = "anthropic", specifier = "==0.84.0" },
{ name = "locust", specifier = "==2.45.0" },
{ name = "mcp", specifier = ">=2.2.0,<3" },
{ name = "playwright", specifier = "==1.61.0" },