Adds a scheduled GitHub Actions lane on top of the merged record/replay
transport. A Saturday cron records the `replayable` e2e tests against the
real providers and publishes the fixture bundle as a private
`e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull
that artifact by its pinned digest, verify the checksum before extracting,
and replay it with provider credentials set to bogus values, so a run that
ever reached a real provider fails instead of passing.
An egress sentinel pins the provider hostnames to a local sink for the whole
replay job and counts every connection that reaches them; the job asserts
that count is zero, so hermeticity is proven by measurement. A red Saturday
publishes no bundle, so the next weekday finds nothing fresh and fails loudly
rather than replaying a week-old recording, and the transport's seven-day
freshness gate hard-fails any bundle that has drifted too far. The lane also
runs on demand from the Actions tab with a record/replay `mode` input.
Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic
test now counts to twenty so its recorded response banks several content
deltas, matching the assertion that the stream arrives incrementally.
Chat completions, embeddings, the non-streaming /v1/messages tests, and the
OpenAI batch deployment now register through the provider edge, so
E2E_FIXTURE_MODE=record captures their provider calls and replay serves them
back offline. None of them was wired before, so record was a silent no-op over
these suites and replay quietly went live instead of using the bundle
Multipart uploads now key on their parsed parts: every ordinary form field,
plus the field name, filename, content digest, and length of each file part.
The boundary is envelope rather than content, so it stays out of the digest
instead of changing the key on every run. A body that does not parse as its
declared envelope still has the boundary normalized away before hashing, so
the fallback is at least stable, and it records a name that says why
Binary uploads hash byte for byte. Canonicalizing them first meant decoding
with errors="replace", which collapsed every invalid byte to one U+FFFD and
gave two different PDFs of the same length the same key
Bundles stay out of the repo: they hold verbatim provider response bodies and
expire seven days after recording. Publishing them for CI is LIT-5748, and
streaming fidelity is LIT-5742
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
This reverts commit dcb4e5033c.
The suites landed without the proof-of-fix and QA runbook the PR body
itself flagged as outstanding, so the coverage they claim is unverified
against a live proxy
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
* fix(e2e): address greptile and veria review on vendor strategy suite
Store search tool keys as os.environ refs and resolve them in SearchAPIRouter.
Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells
* fix(e2e): drop search_api_router os.environ expansion from vendor suite
Keep the PR test-only. Search tools register without an api_key so the
proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a.
* test(e2e): drop search e2e suite from vendor strategy PR
Remove the /v1/search coverage file and its registry rows so this PR
no longer carries search endpoint testing.
* test(e2e): point four suites at models the providers still serve
Four llm_translation tests failed against upstream because the model they name
no longer exists. Each replacement was verified against the live stage proxy.
deepseek/deepseek-reasoner is gone; the DeepSeek API now lists only
deepseek-v4-flash and deepseek-v4-pro. Use deepseek/deepseek-v4-pro, which
still returns message.reasoning_content by default and still drops it for both
reasoning_effort="none" and thinking={"type": "disabled"} (litellm maps the
former to the latter, so the provider rejecting a bare "none" does not matter).
amazon.titan-image-generator-v2:0 returns "This model version has reached the
end of its life"; amazon.nova-canvas-v1:0 is the text-to-image model Bedrock
still offers in us-east-1.
Bedrock's Rerank API requires a full model ARN and rejects a bare model id with
"The provided model ARN for reranking is invalid", regardless of model or
region. Pass the ARN for cohere.rerank-v3-5:0, which is available in the
stack's us-east-1.
vertex_ai/gemini-embedding-2 404s as an unknown publisher model on this
project; vertex_ai/text-embedding-005 returns a vector.
* test(e2e): skip the hosted_vllm chat test when its server is unset
test_hosted_vllm_chat_returns_content read os.environ["HOSTED_VLLM_API_BASE"]
directly, so a stack without that env var failed the test with a bare KeyError
instead of reporting an environment gap. The batches suite already skips on the
same variable, and the vertex passthrough tests use pytest.skip for the same
reason, so follow that idiom here.
Drop the HOSTED_VLLM_API_KEY plumbing: the stage vLLM stand-in serves
/v1/chat/completions unauthenticated, and api_key is optional on
LiteLLMParamsBody, so passing it added nothing.
Default the backend to the model that server actually serves,
Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M, rather than a Llama id it never had.
Verified against the live stage proxy: a deployment with just that model and
api_base returns "hello".
* fix(model_map): mark deepseek v4-pro and v4-flash as reasoning-capable
Review on #34567 flagged that deepseek/deepseek-v4-pro is not marked
reasoning-capable while the e2e control case requires reasoning_content back
from it. The behavior premise is inverted, but it surfaced a real data gap: the
model map never gained supports_reasoning for the v4 models when DeepSeek
retired deepseek-reasoner, which did carry the flag.
Both models do reason. Against the live API with no reasoning params, v4-pro
returns 106 chars of reasoning_content and v4-flash returns 54, and both drop
it for thinking={"type":"disabled"}.
The stale flag had a real consequence beyond metadata: DeepSeekChatConfig
._thinking_mode_active() gates on supports_reasoning(), so with the flag unset
it returned False even when a caller passed thinking={"type": "enabled"},
skipping the multi-turn check that reasoning_content be passed back on
assistant messages. Param support itself was never gated, which is why
reasoning_effort="none" still mapped to thinking disabled.
Verified with LITELLM_LOCAL_MODEL_COST_MAP=True: supports_reasoning now
reports True for deepseek/deepseek-v4-pro and deepseek/deepseek-v4-flash.
tencent/deepseek-v4-pro is left alone; that route was not exercised here.
* fix(e2e): reference client.proxy in mid-conversation native providers test
EndpointsClient exposes the shared ProxyClient as .proxy and has never had a
.gateway attribute, so these two calls raised AttributeError at runtime and
failed the tests/e2e basedpyright zero-error gate for any PR touching e2e
files. Introduced in 23b5b7d199.
* test(e2e): cover 12 non-core LLM coverage registry cells
Raises Non-Core LLMs registry coverage from 24/50 to 36/50 (overall 51.9%
to 54.8%). Four cells were already asserted by existing tests and only
gain their covers marker (openai embeddings, openai image generation,
openai TTS, cohere rerank); one is dual-marked onto the existing
spend-tracking embeddings test rather than duplicated.
New tests: bedrock and vertex embeddings, streaming TTS (asserts chunked
transfer encoding so a buffered body cannot pass), audio transcriptions
via the realtime suite's wav fixture, moderations flag/pass pair, and
files list/retrieve in the batches suite.
Harness: e2e_http.upload generalized to any form model with a
file_content_type override (batches path unchanged), new stream_binary
primitive + BinaryStream for binary chunked responses, transcribe and
moderations client methods, file retrieve/list client methods.
* fix(e2e): close streamed TTS response on error paths and surface the error body
With stream=True a non-2xx response returned with the body unread, keeping
the socket checked out until garbage collection; the sibling
_streaming_outcome already consumes resp.text on error. The response now
closes on every path and BinaryStream carries a bounded error_body so a
failed stream call is triageable.
* test(e2e): assert streamed TTS response carries no content-length
* test(e2e): migrate access-control and inference-endpoint regression tests
Move the access-control and non-chat inference-endpoint cases from litellm-regression-tests onto the shared e2e harness so a regression in either fails here first
access_control/ asserts the gateway's authorization and error-shape contract: a key limited to one model is denied 403 (key_model_access_denied) when it calls another, a key scoped to allowed_routes=["llm_api_routes"] is forbidden 403 from a management route, and an unknown model is rejected 400 before any provider is called. The source asserted 401 for the disallowed-model case against an older proxy; the live contract is now a 403, so the guard tracks current behavior
llm_translation/ gains one file per non-chat inference endpoint (/v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, /v1/images/generations). Each test registers the deployment it needs through /model/new, drives real provider traffic, asserts the parsed body carries real content instead of just a 200, then deletes the model on teardown, so nothing is hardcoded into the gateway config
* Update endpoints_client.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
