mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
test(e2e): key multipart uploads by structured part identity
Adversarial review of the new multipart keying turned up collisions where two different provider requests computed the same replay key, which is the dangerous failure for a replay harness: the second request silently gets the first one's response instead of missing loudly. - a part counts as an upload when it has a filename or declares its own content type, and the declared content type joins the identity, so two uploads of the same bytes under the same field no longer collapse - the uploaded parts contribute a JSON list of [field, filename, type] triples instead of a "field:filename" string, so a separator inside a filename can no longer impersonate a field boundary - repeated field names get a "name[n]" suffix with a literal "[" doubled first, so a repeated field and a literally indexed one stay distinct - a field value that is not UTF-8 is stored as a base64 sha256 digest; base64 rather than hex because the canonicalizer rewrites 64-character hex runs to <sha256> and folded every binary value onto one key - a field whose name reads as a credential is stored as <secret>. This stays key-preserving because the key is recomputed from the stored request rather than saved beside it, so the live request carrying the real value still matches its redacted fixture - the uploaded byte length leaves the key. The canonicalizer absorbs timestamp and id drift inside a file, and that drift moves the count, so keeping it there made re-records miss Also stops a lookalike parameter such as "xboundary=" from being read as the multipart boundary, and gives the OpenAI batch backend model a single constant instead of three copies of the literal. BUNDLE_FORMAT_VERSION goes to 3 because all of this moves recorded keys. A bundle recorded under the old rules now fails naming both versions instead of missing on every call.
This commit is contained in:
mateo-berri
parent
d4162bd1ca
commit
f5df60f106
7 changed files with 302 additions and 44 deletions
|
|
@ -77,7 +77,9 @@ Mark live tests with `@pytest.mark.e2e` (on the class or the module). Pure cover
|
|||
|
||||
The seam is `provider_edge.py`: `start_provider_edge` boots an in-process HTTP server (one shared instance per pytest process, `e2e_config.provider_edge_base` is the accessor) that mounts each supported provider under a path prefix (`EDGE_MOUNTS`: `/openai` -> `https://api.openai.com`, `/anthropic` -> `https://api.anthropic.com`). A test participates by registering its deployment with `api_base=provider_edge_base("openai")` plus the provider's path suffix; `quota_management/spend_tracking/test_provider_edge_spend_e2e.py` is the reference. In live mode the accessor returns None and the deployment defaults to the real provider, so an edge-wired test runs in all three modes unchanged. Non-wired tests hit their providers live in every mode. The edge binds `E2E_PROVIDER_EDGE_BIND_HOST` (default 127.0.0.1) and advertises `E2E_PROVIDER_EDGE_ADVERTISE_HOST` in the api_base it hands out, for proxies running in containers
|
||||
|
||||
A bundle (default `tests/e2e/.fixtures`, override with `E2E_FIXTURE_DIR`) is a directory: `manifest.json` carries the record timestamp, harness git version, and format version, and each test gets a subdirectory holding one JSON file per provider call in call order (`0000-post-openai-v1-chat-completions.json`). Request headers are never stored (provider credentials never touch disk), non-JSON request bodies store a canonicalized sha256 digest instead of the bytes, `multipart/form-data` bodies store their ordinary fields plus a `field:filename` label, a digest of the file part's content, and that part's length, so the per-request random boundary and the envelope never reach the key, and responses store status, filtered headers, and the verbatim body base64-encoded, which is part of why bundles are gitignored. `fixture_bundle.py` owns the format. Record serves the proxy the same filtered stored response replay will serve later, so the two modes are byte-identical from the proxy's side of the socket
|
||||
A bundle (default `tests/e2e/.fixtures`, override with `E2E_FIXTURE_DIR`) is a directory: `manifest.json` carries the record timestamp, harness git version, and format version, and each test gets a subdirectory holding one JSON file per provider call in call order (`0000-post-openai-v1-chat-completions.json`). Request headers are never stored (provider credentials never touch disk), non-JSON request bodies store a canonicalized sha256 digest instead of the bytes, `multipart/form-data` bodies store their ordinary fields plus a JSON list of the uploaded parts' `[field, filename, content-type]` triples and a digest of their content, so the per-request random boundary and the envelope never reach the key, and responses store status, filtered headers, and the verbatim body base64-encoded, which is part of why bundles are gitignored. `fixture_bundle.py` owns the format. Record serves the proxy the same filtered stored response replay will serve later, so the two modes are byte-identical from the proxy's side of the socket
|
||||
|
||||
Multipart identity is the fiddly corner, and the rules exist because each one had a collision behind it. A part counts as an upload when it carries a filename or declares its own content type, and everything else is an ordinary field. Field names get a `name[n]` suffix on repeats, with a literal `[` doubled first, so a form that repeats `purpose` never keys the same as one that literally sends `purpose[1]`. A field whose name reads as a credential is stored as `<secret>`, which stays key-preserving because the key is recomputed from the stored request rather than saved alongside it, so the live request carrying the real value still matches its redacted fixture. A field value that is not UTF-8 is stored as a base64 sha256 digest, base64 and not hex because the canonicalizer rewrites any 64-character hex run to `<sha256>` and would fold every binary value onto one key. The uploaded parts contribute a JSON list rather than a `field:filename` string, so a separator inside a filename cannot impersonate a field boundary, and their byte length is stored for a reader's benefit but deliberately left out of the key, since the canonicalizer absorbs timestamp and id drift inside a file that changes its length
|
||||
|
||||
Replay matches calls per test by canonical key: `fixture_canonical.py` canonicalizes the recorded request (volatile headers and credential fields out, unique markers, generated ids, uuids, and timestamps replaced with fixed placeholders, object keys sorted) and the key is the method, edge path, and a content hash, so identity survives re-records and machine changes while any real content drift comes back as an HTTP 599 naming the computed key, the closest recorded key with its file, and a content diff, and never falls through to a live call. Matching is order-independent across distinct keys (concurrent calls may interleave) and FIFO within one key (a retry loop replays its responses in recorded order); a passed test must also consume its whole recording, or teardown fails it naming a leftover key. Either way the fix is always to re-record with `E2E_FIXTURE_MODE=record`. Every rewrite rule lives in `fixture_canonical.py`, so a new volatile header, credential field name, or generated-id shape is one edit there. Record starts fresh every time: it wipes the previous bundle (refusing to wipe a directory that is not a bundle) and never reads it. A replay bundle whose manifest is older than seven days hard-fails at collection time naming the bundle's age, so replay can never certify against fixtures that have drifted more than a week from the live providers
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ from __future__ import annotations
|
|||
import base64
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
from typing import Final, Literal
|
||||
|
||||
from e2e_config import provider_edge_base, unique_marker
|
||||
from models import LiteLLMParamsBody
|
||||
|
|
@ -17,13 +17,16 @@ def batch_model_name(base: str) -> str:
|
|||
return f"{base}-{_BATCH_RUN}"
|
||||
|
||||
|
||||
OPENAI_BATCH_BACKEND: Final = "gpt-4o-mini"
|
||||
|
||||
|
||||
def openai_batch_params() -> LiteLLMParamsBody:
|
||||
"""The OpenAI batch deployment, wired through the record/replay edge when a fixture
|
||||
mode is active and straight at OpenAI otherwise (LIT-5974). Azure, Vertex, and
|
||||
Bedrock stay live: none of them has an edge mount."""
|
||||
base = provider_edge_base("openai")
|
||||
return LiteLLMParamsBody(
|
||||
model="openai/gpt-4o-mini",
|
||||
model=f"openai/{OPENAI_BATCH_BACKEND}",
|
||||
api_key="os.environ/OPENAI_API_KEY",
|
||||
api_base=None if base is None else f"{base}/v1",
|
||||
)
|
||||
|
|
@ -116,7 +119,11 @@ class Capability:
|
|||
|
||||
PROVIDERS: tuple[Provider, ...] = (
|
||||
Provider(
|
||||
"openai", batch_model_name("openai-batch"), "gpt-4o-mini", can_cancel=True, can_list=True
|
||||
"openai",
|
||||
batch_model_name("openai-batch"),
|
||||
OPENAI_BATCH_BACKEND,
|
||||
can_cancel=True,
|
||||
can_list=True,
|
||||
),
|
||||
Provider(
|
||||
"azure",
|
||||
|
|
|
|||
|
|
@ -42,6 +42,7 @@ from capabilities import (
|
|||
BATCH_ID_SHAPE,
|
||||
CAPABILITIES,
|
||||
FILE_ID_SHAPE,
|
||||
OPENAI_BATCH_BACKEND,
|
||||
OPENAI_BATCH_MODEL,
|
||||
PROVIDERS,
|
||||
Capability,
|
||||
|
|
@ -480,8 +481,6 @@ def test_rate_limited_batch_create_leaves_no_unattributed_spend_row(
|
|||
)
|
||||
|
||||
|
||||
OPENAI_FILE_CONTENT_BACKEND = "gpt-4o-mini"
|
||||
|
||||
FILE_CONTENT_CELLS = {
|
||||
"azure": "llm.files.azure_openai.content.nonstream.works",
|
||||
"vertex_ai": "llm.files.vertex.content.nonstream.works",
|
||||
|
|
@ -511,7 +510,7 @@ class TestBatchFileContent:
|
|||
resources.defer(lambda: client.delete_model(model_id))
|
||||
key = resources.key()
|
||||
|
||||
payload = render_jsonl(OPENAI_FILE_CONTENT_BACKEND)
|
||||
payload = render_jsonl(OPENAI_BATCH_BACKEND)
|
||||
file = unwrap(
|
||||
client.upload_file(
|
||||
content=payload,
|
||||
|
|
|
|||
|
|
@ -5,7 +5,9 @@ version + format version) plus one subdirectory per test, holding one JSON file
|
|||
per provider-bound interaction in call order. Bundles older than
|
||||
``MAX_BUNDLE_AGE`` hard-fail replay at collection time (see conftest), so a
|
||||
green replay run can never certify against fixtures that have drifted more than
|
||||
a week from the live providers.
|
||||
a week from the live providers. Bump ``BUNDLE_FORMAT_VERSION`` whenever a change
|
||||
moves recorded keys: a bundle recorded under the old rules then fails naming
|
||||
both versions instead of quietly missing on every call.
|
||||
|
||||
This module owns the format only. The provider-edge server that produces and
|
||||
consumes it lives in provider_edge.py (LIT-5745) and the canonical match keys
|
||||
|
|
@ -28,7 +30,7 @@ from typing import Final
|
|||
|
||||
from pydantic import BaseModel, JsonValue
|
||||
|
||||
BUNDLE_FORMAT_VERSION: Final = 2
|
||||
BUNDLE_FORMAT_VERSION: Final = 3
|
||||
MAX_BUNDLE_AGE: Final = timedelta(days=7)
|
||||
MANIFEST_FILENAME: Final = "manifest.json"
|
||||
|
||||
|
|
@ -47,7 +49,14 @@ class RecordedRequest(BaseModel):
|
|||
over ``method``, ``path`` (the edge path including the provider mount,
|
||||
query string excluded), and the canonicalized headers, params, body, form,
|
||||
and file identity. Non-JSON bodies store a canonicalized content digest
|
||||
instead of the bytes."""
|
||||
instead of the bytes.
|
||||
|
||||
``file_name`` is a JSON list of the uploaded parts' ``[field, filename,
|
||||
content-type]`` triples rather than a flat label, so a separator inside a
|
||||
filename cannot impersonate a field boundary. ``file_bytes`` is recorded for
|
||||
a reader's benefit and stays out of the key: the canonicalizer absorbs
|
||||
timestamp and id drift inside an uploaded file, and that drift moves the
|
||||
byte count."""
|
||||
|
||||
method: str
|
||||
path: str
|
||||
|
|
|
|||
|
|
@ -129,7 +129,6 @@ def canonicalize(request: RecordedRequest) -> CanonicalRequest:
|
|||
else {
|
||||
"name": None if request.file_name is None else canonical_string(request.file_name),
|
||||
"sha256": request.file_sha256,
|
||||
"bytes": request.file_bytes,
|
||||
}
|
||||
)
|
||||
content: Final[dict[str, JsonValue]] = {
|
||||
|
|
|
|||
|
|
@ -59,7 +59,13 @@ from fixture_bundle import (
|
|||
prepare_bundle,
|
||||
slug_for_test,
|
||||
)
|
||||
from fixture_canonical import CanonicalRequest, canonical_string, canonicalize
|
||||
from fixture_canonical import (
|
||||
SECRET_PLACEHOLDER,
|
||||
CanonicalRequest,
|
||||
canonical_string,
|
||||
canonicalize,
|
||||
is_secret_field,
|
||||
)
|
||||
from fixture_mode import (
|
||||
FIXTURE_MODES,
|
||||
InvalidFixtureMode,
|
||||
|
|
@ -104,12 +110,15 @@ _JSON: Final[TypeAdapter[JsonValue]] = TypeAdapter(JsonValue)
|
|||
|
||||
|
||||
_BOUNDARY_PATTERN: Final = re.compile(
|
||||
r'boundary=(?:"([^"]*)"|([^;,\s]+))', re.IGNORECASE
|
||||
r'(?:^|;)\s*boundary\s*=\s*(?:"([^"]*)"|([^;,\s]+))', re.IGNORECASE
|
||||
)
|
||||
_DISPOSITION_NAME_PATTERN: Final = re.compile(r'(?:^|;)\s*name="([^"]*)"', re.IGNORECASE)
|
||||
_DISPOSITION_FILENAME_PATTERN: Final = re.compile(
|
||||
r'(?:^|;)\s*filename="([^"]*)"', re.IGNORECASE
|
||||
)
|
||||
_DISPOSITION_NAME_PATTERN: Final = re.compile(r'(?:^|;)\s*name="([^"]*)"')
|
||||
_DISPOSITION_FILENAME_PATTERN: Final = re.compile(r'(?:^|;)\s*filename="([^"]*)"')
|
||||
_UNPARSED_MULTIPART: Final = "<unparsed-multipart>"
|
||||
_BOUNDARY_PLACEHOLDER: Final = b"--<boundary>"
|
||||
_BINARY_FIELD_PREFIX: Final = "<binary:sha256:"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
@ -117,6 +126,7 @@ class _MultipartPart:
|
|||
field_name: str
|
||||
filename: str | None
|
||||
content: bytes
|
||||
content_type: str = ""
|
||||
|
||||
|
||||
def _header_value(headers: Mapping[str, str], name: str) -> str:
|
||||
|
|
@ -125,22 +135,34 @@ def _header_value(headers: Mapping[str, str], name: str) -> str:
|
|||
|
||||
|
||||
def _multipart_boundary(content_type: str) -> str | None:
|
||||
"""The declared boundary, or None when the envelope is not multipart or names no
|
||||
usable boundary. ``boundary`` is matched only as a parameter in its own right, so a
|
||||
longer name ending in it (``myboundary=``) is not mistaken for one, and an empty
|
||||
boundary is refused rather than splitting the body on a bare ``--``."""
|
||||
if "multipart/form-data" not in content_type.lower():
|
||||
return None
|
||||
match: Final = _BOUNDARY_PATTERN.search(content_type)
|
||||
return None if match is None else match.group(1) or match.group(2)
|
||||
if match is None:
|
||||
return None
|
||||
quoted, bare = match.group(1), match.group(2)
|
||||
return (quoted if quoted is not None else bare) or None
|
||||
|
||||
|
||||
def _part_headers(head: bytes) -> dict[str, str]:
|
||||
return {
|
||||
name.strip().lower(): value.strip()
|
||||
for line in head.decode("utf-8", errors="replace").split("\r\n")
|
||||
for name, separator, value in [line.partition(":")]
|
||||
if separator
|
||||
}
|
||||
|
||||
|
||||
def _parse_multipart_part(segment: bytes) -> _MultipartPart | None:
|
||||
head, separator, content = segment.partition(b"\r\n\r\n")
|
||||
if not separator:
|
||||
return None
|
||||
disposition: Final = "".join(
|
||||
value
|
||||
for line in head.decode("utf-8", errors="replace").split("\r\n")
|
||||
for name, _, value in [line.partition(":")]
|
||||
if name.strip().lower() == "content-disposition"
|
||||
)
|
||||
headers: Final = _part_headers(head)
|
||||
disposition: Final = headers.get("content-disposition", "")
|
||||
name_match: Final = _DISPOSITION_NAME_PATTERN.search(disposition)
|
||||
if name_match is None:
|
||||
return None
|
||||
|
|
@ -149,6 +171,7 @@ def _parse_multipart_part(segment: bytes) -> _MultipartPart | None:
|
|||
field_name=name_match.group(1),
|
||||
filename=None if filename_match is None else filename_match.group(1),
|
||||
content=content,
|
||||
content_type=headers.get("content-type", ""),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -178,39 +201,72 @@ def _content_digest(content: bytes) -> str:
|
|||
return hashlib.sha256(canonical_string(text).encode()).hexdigest()
|
||||
|
||||
|
||||
def _is_file_part(part: _MultipartPart) -> bool:
|
||||
"""Whether a part is an upload rather than an ordinary field. A filename says so
|
||||
outright, and so does a declared content type: clients attach one per part only for
|
||||
a file, and a client that omits the filename (httpx drops the parameter when it is
|
||||
empty) would otherwise have the file's bytes stored inline as a field value and key
|
||||
identically to a plain field of the same name."""
|
||||
return part.filename is not None or bool(part.content_type)
|
||||
|
||||
|
||||
def _field_value(part: _MultipartPart) -> str:
|
||||
"""What a field part contributes to the stored form. A secret-named field never has
|
||||
its value written out, since the bundle is a file on disk and the key redacts that
|
||||
field to the same placeholder either way, so replay still matches. A value that is
|
||||
not UTF-8 is carried as a digest rather than decoded lossily, because a replacing
|
||||
decode collapses every binary value of one length onto one string. That digest is
|
||||
base64 rather than hex, since the canonicalizer rewrites any long hex run to a
|
||||
``<sha256>`` placeholder and would collapse the values right back together."""
|
||||
if is_secret_field(part.field_name):
|
||||
return SECRET_PLACEHOLDER
|
||||
try:
|
||||
return part.content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
digest: Final = base64.b64encode(hashlib.sha256(part.content).digest()).decode()
|
||||
return f"{_BINARY_FIELD_PREFIX}{digest}>"
|
||||
|
||||
|
||||
def _form_fields(fields: tuple[_MultipartPart, ...]) -> dict[str, str]:
|
||||
"""The ordinary field parts, flattened into the mapping the bundle format stores. A
|
||||
name sent more than once takes an index instead of overwriting the earlier value, so
|
||||
nothing an upload said is dropped from its key."""
|
||||
name sent more than once takes an occurrence suffix instead of overwriting the
|
||||
earlier value, so nothing an upload said is dropped from its key. The suffix is
|
||||
escaped so a field literally named ``x[1]`` cannot collide with a second ``x``."""
|
||||
form: dict[str, str] = {}
|
||||
for part in fields:
|
||||
name = part.field_name
|
||||
name = part.field_name.replace("[", "[[")
|
||||
occurrence = 1
|
||||
while name in form:
|
||||
name = f"{part.field_name}[{occurrence}]"
|
||||
name = f"{part.field_name.replace('[', '[[')}[{occurrence}]"
|
||||
occurrence += 1
|
||||
form[name] = part.content.decode("utf-8", errors="replace")
|
||||
form[name] = _field_value(part)
|
||||
return form
|
||||
|
||||
|
||||
def _file_identity(files: tuple[_MultipartPart, ...]) -> tuple[str | None, str | None, int | None]:
|
||||
"""Name, content digest, and total length for the uploaded file parts. The name
|
||||
carries each part's field name as well as its filename, so two uploads sending the
|
||||
same bytes under different field names stay apart. A lone file keeps its own content
|
||||
digest; several fold into one digest over the per-part identities, which is ordered,
|
||||
so parts arriving in a different order key differently."""
|
||||
"""Name, content digest, and total length for the uploaded file parts.
|
||||
|
||||
The name is a structured list of every part's field name, filename, and declared
|
||||
content type rather than a joined string, so a filename containing the separator
|
||||
cannot be confused for a different split, and two parts that differ only in the type
|
||||
they declare stay apart. It goes through the canonicalizer as one string, which is
|
||||
why per-run markers inside a filename do not move the key in the multi-file case any
|
||||
more than they do in the single-file one.
|
||||
|
||||
The digest covers content only. A lone file keeps its own canonicalized digest;
|
||||
several fold into one ordered digest, so parts arriving in a different order key
|
||||
differently. Total length is recorded for a reader but deliberately kept out of the
|
||||
key: it is the raw byte count, and keying on it would undo exactly the drift the
|
||||
canonicalized digest exists to absorb."""
|
||||
if not files:
|
||||
return None, None, None
|
||||
names: Final = ", ".join(f"{part.field_name}:{part.filename}" for part in files)
|
||||
names: Final = _JSON.dump_json(
|
||||
[[part.field_name, part.filename, part.content_type] for part in files]
|
||||
).decode()
|
||||
total: Final = sum(len(part.content) for part in files)
|
||||
if len(files) == 1:
|
||||
return names, _content_digest(files[0].content), total
|
||||
folded: Final = _JSON.dump_json(
|
||||
[
|
||||
[part.field_name, part.filename, _content_digest(part.content), len(part.content)]
|
||||
for part in files
|
||||
]
|
||||
)
|
||||
folded: Final = _JSON.dump_json([_content_digest(part.content) for part in files])
|
||||
return names, hashlib.sha256(folded).hexdigest(), total
|
||||
|
||||
|
||||
|
|
@ -220,9 +276,9 @@ def _multipart_request(
|
|||
"""A multipart upload keyed by what it says rather than by its wire bytes: every
|
||||
ordinary field, plus the identity of the uploaded file. The random per-request
|
||||
boundary is envelope, never content, so it never reaches the digest."""
|
||||
form: Final = _form_fields(tuple(part for part in parts if part.filename is None))
|
||||
form: Final = _form_fields(tuple(part for part in parts if not _is_file_part(part)))
|
||||
file_name, file_sha256, file_bytes = _file_identity(
|
||||
tuple(part for part in parts if part.filename is not None)
|
||||
tuple(part for part in parts if _is_file_part(part))
|
||||
)
|
||||
return RecordedRequest(
|
||||
method=method,
|
||||
|
|
@ -258,7 +314,7 @@ def _opaque_request(
|
|||
)
|
||||
|
||||
|
||||
def _edge_request(
|
||||
def edge_request(
|
||||
method: str, path: str, query: str, body: bytes | None, content_type: str = ""
|
||||
) -> RecordedRequest:
|
||||
"""The identity replay matches on: the edge path (mount included), the query as
|
||||
|
|
@ -519,7 +575,7 @@ def handle_edge_request(
|
|||
return _text_reply(
|
||||
404, f"unknown provider mount {mount!r}; known mounts: {', '.join(sorted(mounts))}"
|
||||
)
|
||||
request: Final = _edge_request(
|
||||
request: Final = edge_request(
|
||||
method, split.path, split.query, body, _header_value(headers, "content-type")
|
||||
)
|
||||
match backend:
|
||||
|
|
|
|||
|
|
@ -23,11 +23,13 @@ from concurrent.futures import ThreadPoolExecutor
|
|||
from contextlib import contextmanager
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from typing import Final
|
||||
|
||||
import pytest
|
||||
from pydantic import TypeAdapter
|
||||
|
||||
from e2e_http import RawResponse, forward
|
||||
from fixture_canonical import canonicalize
|
||||
from fixture_bundle import (
|
||||
BundleRecorder,
|
||||
Interaction,
|
||||
|
|
@ -46,6 +48,7 @@ from provider_edge import (
|
|||
RecordEdge,
|
||||
ReplayEdge,
|
||||
ReplaySource,
|
||||
edge_request,
|
||||
handle_edge_request,
|
||||
provider_edge_api_base,
|
||||
replay_leftover_error,
|
||||
|
|
@ -412,7 +415,9 @@ class TestMultipartIdentity:
|
|||
raw = this_tests_files(root)[0].read_text(encoding="utf-8")
|
||||
interaction = Interaction.model_validate_json(raw)
|
||||
assert interaction.request.form == {"purpose": "batch"}
|
||||
assert interaction.request.file_name == "file:batch.jsonl"
|
||||
assert interaction.request.file_name == json.dumps(
|
||||
[["file", "batch.jsonl", "application/octet-stream"]], separators=(",", ":")
|
||||
)
|
||||
assert interaction.request.file_bytes == len(BATCH_JSONL)
|
||||
stored = interaction.request.model_dump_json()
|
||||
assert boundary not in stored
|
||||
|
|
@ -528,6 +533,187 @@ class TestMultipartIdentity:
|
|||
assert replay_upload(root, opaque, absent).status_code == 200
|
||||
|
||||
|
||||
def raw_multipart(boundary: str, *parts: tuple[str, bytes]) -> bytes:
|
||||
"""A body assembled from literal part headers, so a test can send the shapes a
|
||||
well-formed helper cannot: a file part with no filename, a declared per-part content
|
||||
type, a repeated or bracketed field name, or a non-UTF-8 value."""
|
||||
return (
|
||||
b"".join(
|
||||
f"--{boundary}\r\n{head}\r\n\r\n".encode() + content + b"\r\n"
|
||||
for head, content in parts
|
||||
)
|
||||
+ f"--{boundary}--\r\n".encode()
|
||||
)
|
||||
|
||||
|
||||
def upload_key(body: bytes, boundary: str) -> str:
|
||||
content_type: Final = f"multipart/form-data; boundary={boundary}"
|
||||
return canonicalize(edge_request("POST", UPLOAD_PATH, "", body, content_type)).key
|
||||
|
||||
|
||||
DISPOSITION = 'Content-Disposition: form-data; name="{name}"'
|
||||
FILE_DISPOSITION = DISPOSITION + '; filename="{filename}"'
|
||||
|
||||
|
||||
class TestMultipartIdentityEdges:
|
||||
"""The identity a multipart upload keys on, pinned against the ways two materially
|
||||
different uploads could otherwise collapse onto one key. A collision here is the
|
||||
dangerous failure: replay would answer one request with another's response."""
|
||||
|
||||
def test_a_declared_part_content_type_separates_otherwise_identical_uploads(self) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
as_json = raw_multipart(
|
||||
boundary,
|
||||
(FILE_DISPOSITION.format(name="file", filename="a") + "\r\nContent-Type: application/json", b"xy"),
|
||||
)
|
||||
as_csv = raw_multipart(
|
||||
boundary,
|
||||
(FILE_DISPOSITION.format(name="file", filename="a") + "\r\nContent-Type: text/csv", b"xy"),
|
||||
)
|
||||
|
||||
assert upload_key(as_json, boundary) != upload_key(as_csv, boundary)
|
||||
|
||||
def test_a_file_part_without_a_filename_is_not_mistaken_for_a_plain_field(self) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
upload = raw_multipart(
|
||||
boundary,
|
||||
(DISPOSITION.format(name="file") + "\r\nContent-Type: application/octet-stream", b"CONTENT"),
|
||||
)
|
||||
plain_field = raw_multipart(boundary, (DISPOSITION.format(name="file"), b"CONTENT"))
|
||||
|
||||
request = edge_request(
|
||||
"POST", UPLOAD_PATH, "", upload, f"multipart/form-data; boundary={boundary}"
|
||||
)
|
||||
|
||||
assert upload_key(upload, boundary) != upload_key(plain_field, boundary)
|
||||
assert request.form == {}
|
||||
assert b"CONTENT".decode() not in request.model_dump_json()
|
||||
|
||||
def test_a_filename_carrying_a_per_run_marker_keys_the_same_next_run(self) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
|
||||
def upload(marker: str) -> str:
|
||||
body = raw_multipart(
|
||||
boundary,
|
||||
(FILE_DISPOSITION.format(name="one", filename=f"{marker}.jsonl"), b"first"),
|
||||
(FILE_DISPOSITION.format(name="two", filename="steady.jsonl"), b"second"),
|
||||
)
|
||||
return upload_key(body, boundary)
|
||||
|
||||
assert upload("a1b2c3d4e5f6") == upload("0f9e8d7c6b5a")
|
||||
|
||||
def test_a_separator_inside_a_filename_cannot_forge_a_different_split(self) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
colon_in_filename = raw_multipart(
|
||||
boundary, (FILE_DISPOSITION.format(name="file", filename="a:b.jsonl"), b"same")
|
||||
)
|
||||
colon_in_field = raw_multipart(
|
||||
boundary, (FILE_DISPOSITION.format(name="file:a", filename="b.jsonl"), b"same")
|
||||
)
|
||||
|
||||
assert upload_key(colon_in_filename, boundary) != upload_key(colon_in_field, boundary)
|
||||
|
||||
def test_a_repeated_field_cannot_collide_with_a_literal_indexed_name(self) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
repeated = raw_multipart(
|
||||
boundary,
|
||||
(DISPOSITION.format(name="purpose"), b"x"),
|
||||
(DISPOSITION.format(name="purpose"), b"y"),
|
||||
)
|
||||
literal_index = raw_multipart(
|
||||
boundary,
|
||||
(DISPOSITION.format(name="purpose"), b"x"),
|
||||
(DISPOSITION.format(name="purpose[1]"), b"y"),
|
||||
)
|
||||
|
||||
assert upload_key(repeated, boundary) != upload_key(literal_index, boundary)
|
||||
|
||||
def test_two_binary_field_values_of_one_length_stay_apart(self) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
first = raw_multipart(boundary, (DISPOSITION.format(name="blob"), b"\xff\xfe\xfd"))
|
||||
second = raw_multipart(boundary, (DISPOSITION.format(name="blob"), b"\xf0\xf1\xf2"))
|
||||
|
||||
assert upload_key(first, boundary) != upload_key(second, boundary)
|
||||
|
||||
def test_a_secret_named_field_never_reaches_the_stored_request(self) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
body = raw_multipart(
|
||||
boundary,
|
||||
(DISPOSITION.format(name="openai_api_key"), b"sk-live-DEADBEEF-0123456789abcd"),
|
||||
(DISPOSITION.format(name="purpose"), b"batch"),
|
||||
)
|
||||
|
||||
request = edge_request(
|
||||
"POST", UPLOAD_PATH, "", body, f"multipart/form-data; boundary={boundary}"
|
||||
)
|
||||
|
||||
assert "sk-live-DEADBEEF-0123456789abcd" not in request.model_dump_json()
|
||||
assert request.form == {"openai_api_key": "<secret>", "purpose": "batch"}
|
||||
|
||||
def test_a_redacted_field_still_matches_the_live_request_that_carried_the_secret(
|
||||
self,
|
||||
) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
|
||||
def upload(secret: str) -> str:
|
||||
body = raw_multipart(
|
||||
boundary,
|
||||
(DISPOSITION.format(name="openai_api_key"), secret.encode()),
|
||||
(DISPOSITION.format(name="purpose"), b"batch"),
|
||||
)
|
||||
return upload_key(body, boundary)
|
||||
|
||||
assert upload("sk-live-DEADBEEF-0123456789abcd") == upload("<secret>")
|
||||
|
||||
def test_a_length_change_the_canonicalizer_absorbs_does_not_move_the_key(self) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
|
||||
def upload(created: str) -> str:
|
||||
body = raw_multipart(
|
||||
boundary,
|
||||
(
|
||||
FILE_DISPOSITION.format(name="file", filename="batch.jsonl"),
|
||||
b'{"created_at":"' + created.encode() + b'"}',
|
||||
),
|
||||
)
|
||||
return upload_key(body, boundary)
|
||||
|
||||
assert upload("2026-08-21T02:08:19Z") == upload("2026-08-21T02:08:19.123456Z")
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"content_type",
|
||||
[
|
||||
pytest.param("multipart/form-data; myboundary=zzz; boundary={boundary}", id="lookalike-parameter"),
|
||||
pytest.param("multipart/form-data; BOUNDARY={boundary}", id="uppercase-parameter"),
|
||||
],
|
||||
)
|
||||
def test_the_boundary_parameter_is_read_the_way_the_client_meant_it(
|
||||
self, content_type: str
|
||||
) -> None:
|
||||
boundary = "0123456789abcdef0123456789abcdef"
|
||||
body = raw_multipart(
|
||||
boundary, (FILE_DISPOSITION.format(name="file", filename="batch.jsonl"), BATCH_JSONL)
|
||||
)
|
||||
|
||||
request = edge_request(
|
||||
"POST", UPLOAD_PATH, "", body, content_type.format(boundary=boundary)
|
||||
)
|
||||
|
||||
assert request.form == {}
|
||||
assert request.file_name is not None
|
||||
assert "batch.jsonl" in request.file_name
|
||||
|
||||
def test_an_empty_declared_boundary_falls_back_instead_of_splitting_on_dashes(self) -> None:
|
||||
body = b'--\r\nContent-Disposition: form-data; name="a"\r\n\r\nvalue\r\n----\r\n'
|
||||
|
||||
request = edge_request(
|
||||
"POST", UPLOAD_PATH, "", body, 'multipart/form-data; boundary=""'
|
||||
)
|
||||
|
||||
assert request.form is None
|
||||
assert request.file_sha256 is not None
|
||||
|
||||
|
||||
class TestReplayLeftover:
|
||||
def test_partially_consumed_recording_names_the_leftover(self, tmp_path: Path) -> None:
|
||||
root = tmp_path / "bundle"
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue