* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
This reverts commit dcb4e5033c.
The suites landed without the proof-of-fix and QA runbook the PR body
itself flagged as outstanding, so the coverage they claim is unverified
against a live proxy
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity
Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure
* test(e2e): expand vendor API strategy coverage across endpoints
Adds validation cases on existing endpoint suites, plus vector stores, search,
bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches
contract, and chat stream SSE. Registers coverage cells for LIT-4778
* test(e2e): finish vendor strategy open items
Audio transcription negatives, vector-store file attach/poll/search,
OpenAI moderation category matrix across chat/messages/responses, and
smoke model matrix for chat (LIT-4778)
* test(e2e): harden vendor strategy suite against live env edges
Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing,
vector store list/search models, responses validation, and provider-denied
Bedrock paths so the suite is stable against a live proxy
* test(e2e): rename suites, drop vendor_contract, fix greptile gaps
Move shared status helpers into e2e_http, rename chat auth headers and
chat security suites, remove vendor_contract and dev_config files_settings,
and tighten transcription validation plus vector-store search assertions
* test(e2e): route bedrock stream disconnects through e2e_http
Catch mid-stream RequestException in the shared harness so bedrock native
tests do not import requests directly
* fix(e2e): address greptile and veria review on vendor strategy suite
Store search tool keys as os.environ refs and resolve them in SearchAPIRouter.
Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells
* fix(e2e): drop search_api_router os.environ expansion from vendor suite
Keep the PR test-only. Search tools register without an api_key so the
proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a.
* test(e2e): drop search e2e suite from vendor strategy PR
Remove the /v1/search coverage file and its registry rows so this PR
no longer carries search endpoint testing.
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
The e2e docs claimed `e2e`-marked tests skip when no proxy answers the
liveness probe, but the harness has always hard-failed: conftest.py's
pytest_runtest_setup calls pytest.fail, its module docstring states
"hard failures only ... never skip", and logging/conftest.py forbids
skipping outright. Align the docs to the code so the single most
important contract reads the same everywhere; a dead proxy turns a run
red instead of being silently skipped and mistaken for a pass. The
per-suite conftest docstrings that described the shared hook as a
"proxy liveness skip" are corrected to "liveness gate" for the same
reason.
Also scope the no-unit-tests hard rule to what it means: never
substitute a unit test for e2e feature coverage, while explicitly
allowing tests that cover the harness itself (e.g.
coverage_registry/test_collector.py), which carry no e2e marker and
run whether or not a proxy is up.
No product code and no harness logic changed.
Resolves LIT-4554
* fix: rust ocr tests finally pass
* fix: move realtime dir
* fix(realtime): normalize azure realtime api_base to host for Foundry endpoints
The azure realtime handler appended the realtime path to api_base verbatim, so a
Foundry base carrying a project path (.../api/projects/<name>) produced an invalid
realtime URL and the websocket handshake hung. Normalize api_base to scheme and host
before building the realtime path so both Azure OpenAI and Foundry bases connect
Point the e2e realtime azure deployment at the GA gpt-realtime model and stop passing
the os.environ refs the realtime path never unwraps, resolving them from the gateway
env by name instead. Drop the local docker-compose scaffolding from the tree
* test(e2e): add Gateway.list_files and list_fine_tuning_jobs for the discovery suite
The discovery endpoints suite calls client.gateway.list_files and
list_fine_tuning_jobs, which did not exist on Gateway, so both tests errored with
AttributeError before reaching the proxy. Add the two GET wrappers using the
existing FileListResponse / FineTuningJobsResponse models
* revert(realtime): drop azure realtime api_base host-normalization
The azure realtime handshake failure was a config issue, not a litellm bug: the
realtime base was set to the Azure AI Foundry project endpoint (.../api/projects/<p>),
but the OpenAI-compatible realtime route lives at the resource root. litellm correctly
appends the realtime path to whatever base it is given, so pointing the realtime
deployment at the resource root is the fix and no core change is needed
* fix(ocr): route azure_ai doc-intelligence to its own endpoint at the source
get_llm_provider inherits AZURE_AI_API_BASE into api_base for every azure_ai/* OCR
model, but Azure Document Intelligence is a separate resource reached via
AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT, so doc-intelligence requests went to the wrong
host. Stop inheriting the azure_ai base for doc-intelligence models so api_base stays
unset and both the rust bridge and the python get_complete_url fall back to the
document-intelligence endpoint. This drops the earlier _rust_bridge_api_base reorder,
which only covered the rust path and let the env silently override an explicit api_base
* refactor(ocr): consolidate azure doc-intelligence detection; keep explicit api_base
Extract is_azure_document_intelligence_model as the single source of truth for the azure_ai doc-intelligence sub-route so the check is no longer duplicated across _prepare_ocr_request and _rust_bridge_api_base, and gate the dynamic_api_base suppression on the caller not supplying an api_base so an explicit endpoint is always honoured. Restore xai to the realtime PROVIDERS as a documented disabled entry instead of dropping it silently, and add a regression test pinning doc-intelligence api_base resolution.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The rust OCR smoke lived under tests/e2e/gateway and spoke raw httpx with
ad-hoc dataclasses, diverging from the rest of tests/e2e. Move it to
tests/e2e/llm_translation and rebuild it on the shared harness: typed pydantic
bodies in models.py (OcrDocument/OcrBody/OcrPage/OcrResponse), a Gateway.ocr()
route through the shared transport, Result/unwrap for outcomes, the e2e marker,
and the client/scoped_key fixtures. No test touches httpx or requests directly
now.
Behavior preserved: the config-presence check still reads gateway/litellm-config.yml
without a proxy, /model/info confirms the proxy loaded every rust-ocr deployment,
and each provider case asserts a well-formed OCR document over /v1/ocr.
Also add tests/e2e/CONTRIBUTING.md documenting the end-to-end testing flow so new
features land with coverage that walks the feature like production does.
