litellm/tests/translation_characterization

Translation characterization corpus

Pins v1 core-translation behavior at the transform seams so the v2 rewrite (02-mateo-scope.md, "How you prove it's safe") has a differential gate. Plain pytest, no network, no API keys, fully deterministic.

Layout

• cases/ - request corpus: OpenAI-format chat cases, one JSON file per stable case id, generated by _corpus.py (the cross-product of message shapes x features: tools incl. parallel and streamed args, thinking, cache_control, response_format json_schema/json_object, images, pdf, sampling params)
• fixtures/ - handwritten recorded provider payloads: responses/ (provider response JSON per API spec), streams/ (anthropic raw SSE lines, bedrock parsed event payloads), adapter/ (anthropic-format requests, OpenAI responses, native /v1/messages cases)
• snapshots/ - the pinned v1 output, canonical JSON (pretty-printed, sorted keys, trailing newline): requests/<provider>/<case_id>.json, responses/..., streams/..., adapter/...
• _seams.py - invokes the REAL v1 entry points the way completion() does: get_llm_provider -> get_optional_params -> get_litellm_params -> ProviderConfigManager.get_provider_chat_config -> validate_environment -> transform_request/transform_response, and CustomStreamWrapper over the real provider stream decoders

Providers covered: anthropic chat, bedrock_converse, bedrock_invoke (anthropic messages route), plus both /v1/messages surfaces (the anthropic<->openai adapter and the native AnthropicMessagesConfig request transform with its max_tokens/reasoning_effort/advisor-block quirks).

Running

pytest tests/translation_characterization

Regenerate snapshots (and the generated cases/ files):

pytest tests/translation_characterization --snapshot-update
# or, when the conftest is not the initial one (e.g. `pytest tests/`):
SNAPSHOT_UPDATE=1 pytest tests/translation_characterization

Determinism: uuid.uuid4, fastuuid.uuid4 (via litellm._uuid) and time.time are frozen per test in conftest.py because v1 mints chatcmpl-<uuid> ids and created timestamps inside the seams.

How v2 consumes this

The differential gate runs the SAME corpus (cases/ + fixtures/) through the v2 implementation and renders its output with the same canonical JSON helper (_helpers.canonical_json); parity means byte-identical files against snapshots/. The diff is the review artifact.

Rule: behavior changes never ride along with a port. An intentional behavior change ships as its own PR that edits only the snapshots it means to change, regenerated with --snapshot-update.

Seams not pinned bit-exactly, and why

• Bedrock binary event-stream framing: production bytes pass through botocore's EventStreamBuffer before reaching the decoders. The corpus replays the parsed event payloads into the real AWSEventStreamDecoder.converse_chunk_parser / AmazonAnthropicClaudeStreamDecoder._chunk_parser; the binary framing is AWS plumbing, not translation logic.
• URL-sourced media on bedrock (image_url case): v1 downloads the URL during transform, so those provider cells are skipped with a recorded reason in the case's skip map (anthropic passes URLs through and is pinned).
• Anthropic SSE error events: the v1 iterator raises mid-stream instead of yielding a chunk, so there is no JSON output to snapshot; the raise path is exception-shaped, not corpus-shaped.