10 commits

Author	SHA1	Message	Date
Yuneng Jiang	09a98f5505	test(e2e): settle control-plane writes across every replica, not just one ... The suite already waits for a new model or agent to become servable before handing it back, but that wait returns on the first successful read. Every request opens a fresh connection (e2e_http calls requests.* with no Session), so a load-balanced Service routes each one independently: one successful read proves one replica converged, and the caller's next request re-rolls and can land on a replica that has not reloaded yet. At replicaCount: 2 this surfaced as 30 failures on a SHA that is green at 1 replica -- 400 "Invalid model name passed", 404 "Guardrail not found", "no healthy deployments for this model", and a /model/info listing that contained one of two models created moments apart. Add PROPAGATION_TIMEOUT (default 15s, override E2E_PROPAGATION_TIMEOUT) and settle_propagation(), sized off the proxy's proxy_config_reload_interval_seconds (30s by default, 7s on the e2e stack) plus margin, and settle after every control-plane create whose object the suite then uses: - ProxyClient.create_model and A2AClient.register_agent, after their existing polls -- the poll still fails loudly if the object never appears at all - GuardrailsClient.register, which had no barrier; create_content_filter_guardrail and create_bedrock_guardrail now route through it instead of POSTing directly - the guardrail creates in mcp_client and logging_client - the vertex passthrough model, whose body cannot go through create_model Left alone: the /model/new calls that assert a 403 or read back a status code, since they never use the model.	2026-08-07 19:36:30 -07:00
mubashir1osmani	a10365e84d	test(e2e): stop racing control-plane writes across the mcp, a2a, guardrail and passthrough suites (#34833) ... * test(e2e): wait for MCP tool discovery instead of racing it /v1/mcp/server returns as soon as the DB row is written, but the gateway runs the initialize + tools/list handshake against the upstream lazily, on the first request that needs it. Every MCP test read tools/list immediately after registering, so it raced that handshake. The gateway reports a server it has not discovered yet exactly like a dead one: it catches the per-server handshake exception and returns an empty tool list. The tests asserted on a single read, so the race surfaced as "granted key never saw search_datadog_logs; tools=frozenset()" while a sibling test against the same upstream in the same run passed. Add McpClient.await_tool, which polls tools/list to the suite's existing poll_timeout and returns the qualified tool name, and route the four discovery sites through it. An unreachable upstream or an unapplied grant still fails, and the failure now names the last tools/list result. Refs LIT-4821 * test(e2e): wait for a2a agents to reach the data plane after registration POST /v1/agents is a control-plane write; the /a2a/{agent_id} routes that serve the card and run message/send are data plane and only see the agent after the next DB reload. Every test registered an agent and immediately read its card or sent it a message, so the first data-plane touch could 404 on the agent it had just created. register_agent now waits for the card to become servable before returning, the same way ProxyClient.create_model waits for a new model, so callers do not each have to poll. Registration failures skip the wait, leaving the two rejection tests unchanged. A genuine propagation failure now fails naming the agent id and the last card read rather than as a bare 404 on whichever /a2a call ran first. Refs LIT-4821 * test(e2e): wait for presidio guardrails to sync before asserting masking Registering a guardrail is a control-plane write; the data-plane worker that serves /chat/completions only picks it up on its next periodic DB sync (~30s), so the first call after the create ran against a worker with no guardrail and passed the raw email straight through. The tests asserted on that first call, so they read in-flight propagation as a PII leak. Confirmed directly against a live proxy: the same call is unmasked at t=0s and masked at t=8s, and the presidio analyzer itself correctly returns EMAIL_ADDRESS with score 1.0 the whole time. The MCP guardrail suite already documents and waits out this exact sync delay; presidio never got the same treatment. Poll the call until the placeholder replaces the PII, so the assertions judge the synced state. A guardrail that never masks still fails, on the last unmasked content. pre_call and post_call now pass repeatably. Refs LIT-4821 * test(e2e): drop the presidio logging_only check pending LIT-4841 pre_call and post_call masking both pass once the guardrail-sync wait is in place, but logging_only left the raw email in the OTEL span's gen_ai.input.messages on every attempt across a full poll deadline. Keeping an assertion against known-failing behavior just turns every run red, so the cell is tracked in LIT-4841 instead. The registry row stays, so guardrail.presidio.logging_only.masks now reports as an uncovered gap rather than silently disappearing. Refs LIT-4821, LIT-4841 * test(e2e): wait for guardrail sync in bedrock, moderation and block-code checks All three asserted on the first call after registering a guardrail, so they were served by a data-plane worker that had not synced it yet (~30s DB poll) and read in-flight propagation as a guardrail that failed to block. Verified directly: the openai_moderation guardrail lets a flagged prompt through at t=0s and returns "Violated OpenAI moderation policy" at t=8s. The reasoning-only responses noted in triage (content=None with reasoning_tokens set) were a symptom of the same thing, not the cause; these are pre_call guardrails, so a synced guardrail rejects the request before the model runs. Add poll_until_blocked to guardrails_client for the two that surface a non-success status, and poll on the block marker in the block_code_execution check, which replaces the reply rather than erroring. All eight guardrail tests now pass. Refs LIT-4821 * test(e2e): drop the openai prompt-cache check pending LIT-4841 Prompt caching never engages through the proxy: cached_tokens is 0 on every repeat, while the identical payload sent straight to OpenAI reports 3615 cached tokens on the second call. Pinning prompt_cache_key on the proxy request restores caching (3328 tokens), so something varying per request is defeating OpenAI's automatic prefix cache. That is a product bug with a direct billing cost, tracked in LIT-4841. The registry row stays, so llm.chat_completions.openai.prompt_cache_5m.nonstream.works now reports as an uncovered gap instead of failing every run. Refs LIT-4821, LIT-4841 * test(e2e): drop the responses metadata redis-ttl check It failed on a Redis read timeout against the stage serverless cache (berrie-litellm-stage-ieib2i.serverless.use1.cache.amazonaws.com:6379), a reachability problem this suite has hit before rather than a proxy defect the assertion can pin down. The file held only this test. Its other cell, llm.responses.openai.basic.nonstream.works, is still covered by test_responses_e2e.py; other.config.responses.metadata_redis_ttl_bounded becomes an uncovered registry row, taking headline coverage 314/431 -> 312/431. Refs LIT-4821 * test(e2e): fix passthrough header propagation and openai body, drop the cost check Three separate problems behind the two passthrough failures. The header test 404'd because POST /config/pass_through_endpoint is a control-plane write and the worker serving the route only registers it on its next config reload; measured at ~18s on a live proxy. Wait for the route to stop 404ing before calling it. The readiness probe reuses the master key and omits anthropic-version so polling does not bill a completion per attempt. The openai passthrough body sent max_tokens, which the gpt-5 family rejects outright ("Unsupported parameter: 'max_tokens' is not supported with this model"). Confirmed against OpenAI directly: max_tokens 400s, max_completion_tokens 200s. Passthrough forwards the body untouched by design, so the body was simply wrong. test_openai_passthrough_nonstreaming_logs_cost still finds no SpendLogs row for its call_id after the fix, so it is removed rather than left red; the gemini and anthropic passthrough cost checks still cover that path. Passthrough suite is 8/8 green. Refs LIT-4821	2026-07-27 21:19:49 +00:00
mubashir1osmani	64fc19d61a	fix(e2e): stop tests from breaking the shared proxy for every suite after them (#34664) ... * fix(e2e): stop the cache-settings test from persisting a degraded Redis config TestCacheSettings.test_update_persists_cache_backend_to_get read the live cache settings and wrote them back, intending a no-op. Its capture modelled only type/host/port, so on a TLS cluster the write-back silently dropped `ssl` and `redis_startup_nodes`. That is not recoverable on its own. `/cache/settings` persists what it receives into LiteLLM_CacheConfig, that row outranks the YAML `cache_params`, and init_cache_settings_in_db re-applies it on a timer, so a restart does not clear it. The proxy ends up driving a TLS-only cluster endpoint as a plaintext standalone node and every Redis call blocks to socket timeout. On the affected deployment that took out rate limiting entirely (the v3 limiter is a Lua script on Redis with no DB fallback), Redis-only budget levels (tag, per-model, team-member, per-window), spend tracking, `ResetBudgetJob` (which self-starved at 54 skipped runs per 15 min), and `ProxyConfig.add_deployment`, whose last statement syncs guardrails and never ran. 60 of 72 failures in one run traced back here. The settings blob is now round-tripped verbatim via a RootModel over an exhaustive value union, so a subset cannot be written. Two guards make a regression fail loudly at this test instead of silently downstream: - refuse to write when GET reports redis_type=cluster but omits redis_startup_nodes, which is the exact precondition for persisting a downgrade. GET resolves the stored row overlaid with REDIS_* env and never reads YAML, so a cluster configured only in YAML cannot round-trip here - compare /cache/ping before and after, so a write that breaks connectivity fails this test rather than every suite that follows The underlying product defect is filed as LIT-4816: GET cannot express the effective config, and a partial POST is allowed to downgrade transport. This change only stops the suite from triggering it; the Admin UI can still do so. basedpyright clean (0 errors) under the e2e gate. * fix(e2e): scope the bedrock guardrail per request and send OpenAI's current token param Two failures that had nothing to do with the guardrail or route under test. create_bedrock_guardrail registered with default_on=True, which applies the guardrail to every request the proxy serves. The upstream ApplyGuardrail call was answering 403, and that came back to unrelated traffic as `403 Bedrock guardrail request failed`, failing three a2a tests and a passthrough headers test alongside the bedrock one. The harness already supports the per-request `guardrails` selector, so the guardrail is now registered opted out of default_on and selected by the test that wants it. A broken upstream guardrail fails its own test instead of whatever else is running. Note this only contains the blast radius; the 403 itself still needs the bedrock:ApplyGuardrail permission (or a valid guardrail identifier) on the deployment, so test_bedrock_pre_call_blocks_harmful_prompt can still fail on its own until that is sorted. The OpenAI passthrough body sent `max_tokens`, which newer models reject with "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead." Passthrough forwards the body untranslated, so drop_params does not apply and the body has to satisfy OpenAI's contract directly. vllm_chat keeps max_tokens, which vLLM accepts. basedpyright clean (0 errors) under the e2e gate. * fix(e2e): drop the pinned a2a api_key that broke every message/send #34512 pinned `api_key="os.environ/ANTHROPIC_API_KEY"` on the a2a bridge agent. The a2a bridge forwards the agent's litellm_params straight into litellm.acompletion() without expanding "os.environ/" indirection, so that literal string was sent upstream as x-api-key and every message/send failed with `AnthropicException - {"type":"authentication_error","message":"invalid x-api-key"}`. Omitting api_key restores the normal provider resolution: litellm reads ANTHROPIC_API_KEY from the proxy's own environment for this provider, which is what the agent-owner flow depends on and what the suite did before #34512. Verified against a live proxy, same agent shape each time: api_key omitted -> message/send 200 api_key "os.environ/ANTHROPIC_API_KEY" -> message/send 500 invalid x-api-key api_key <literal key> -> message/send 200 and the key itself is valid (direct call to api.anthropic.com returns 200), so this was indirection that never got expanded rather than a bad credential. This accounts for four failures (test_semver_protocol_version_registers_and_serves, test_message_send_runs_completion_bridge, test_pinned_v0_3_serves_flat_message_shape, test_pinned_v1_0_serves_nested_message_shape). They were previously reported as `403 Bedrock guardrail request failed`, because a default_on Bedrock guardrail short-circuited the request before it ever reached the bridge and hid this. The bridge silently ignoring "os.environ/" in agent params is a product defect in its own right, filed separately; anyone configuring an agent credential that way through the UI hits the same wall. basedpyright clean (0 errors) under the e2e gate. * test(e2e): make the load suite less aggressive against a shared proxy 750 users at spawn rate 50 saturated the request path hard enough to distort the latency-sensitive suites sharing the same proxy, and it spends real provider money at that rate. Drop to 200 users at spawn rate 20. The RPS floor moves with the user count rather than staying put, so the assertion keeps its meaning instead of becoming a formality: 355 RPS over 750 users is ~0.47 RPS/user, and 90 over 200 holds that same per-user expectation with a similar pass margin. A request-path regression still trips it. All four knobs stay env-overridable (E2E_LOAD_USERS, E2E_LOAD_SPAWN_RATE, E2E_LOAD_DURATION_SECONDS, E2E_LOAD_MIN_RPS) for a deliberate load run. Note the recorded failure for this test was "no requests completed in 60s", which was the gateway wedged on unreachable Redis rather than a throughput regression; this change is about not perturbing its neighbours, not about that failure. * fix(e2e): make the reasoning-tokens assertion exercise a request that reasons test_openai_chat_reasoning_reports_reasoning_tokens asked "A train travels 60 miles in 1.5 hours. What is its average speed in mph?" at reasoning_effort="low", then asserted reasoning_tokens > 0. The model answers that directly without reasoning, so 0 is correct behavior and the assertion was testing the model's discretion rather than litellm's reporting. Verified against a live proxy on a dedicated openai/gpt-5.6 deployment, matching how the test provisions its model: reasoning_effort=low, one-step arithmetic -> reasoning_tokens=0 reasoning_effort=high, the prompt used here -> reasoning_tokens=114 Raised to high effort with a prompt that requires a proof plus a search, so the field under test is actually populated and the assertion fails only if litellm stops surfacing it. While confirming this I also checked prompt caching, which needed no change: cached_tokens comes back 3615 of 3618 prompt tokens on a repeated large prefix against a dedicated deployment. An earlier reading of 0 was an artifact of probing a fan-out alias whose requests land on different deployments, not a caching defect. * test(e2e): skip the files-list test while LIT-4820 is open GET /v1/files does not include a just-uploaded file. The upload returns 200 and GET /v1/files/{id} resolves it, but the listing never contains it: the returned set stays fixed at 27 entries whose newest created_at is roughly ten hours older than the upload, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Polled for 40s, so not an eventual-consistency window. Filed as LIT-4820. Skipping keeps a known, ticketed product bug from holding the suite red and masking a new regression somewhere else in the same test. The assertion is left exactly as it was on purpose. It encodes the contract we actually want, that a file retrievable by id is also enumerable, and anything that lists files (a UI picker, cleanup tooling that lists then deletes and would therefore leak provider-side files) depends on it. Relaxing it to get green would delete the signal. The skip reason says so and links the ticket, and the ticket records that removing this marker is part of its definition of done. Matches the existing pattern in this file, where test_unified_file_and_batch_create skips with a reason citing LIT-3266. While skipped, the registry cell llm.files.openai.list.nonstream.works has no passing covering test, so files-list coverage reports as uncovered rather than passing, which is the honest state. * fix(e2e): parse Sentinel node lists in the cache-settings model The value union covered scalar lists and lists of mappings, but not lists of lists. `redis_startup_nodes` holds host/port mappings while `sentinel_nodes` holds positional pairs (CACHE_SETTINGS_FIELDS documents `[['localhost', 26379]]`), so on a Sentinel deployment pydantic rejected the response: sentinel_nodes.list[dict[str,...]].1 Input should be a valid dictionary [input_value=['localhost', 26380]] The round-trip test reads GET /cache/settings before it writes anything, so that rejection failed the test at the read, before any assertion ran. A Sentinel deployment would have looked like a broken cache-settings route rather than a model too narrow to parse a documented shape. A list element may now be a scalar, a list or a mapping, which covers both node shapes without special-casing either and tolerates a heterogeneous list instead of rejecting the whole response. Adds TestCacheSettingsModel, harness-level with no `e2e` marker so it runs without a proxy, covering all four backend shapes (cluster mappings, sentinel pairs, plain node, url mode with a null discrete field) plus transport() key selection. Confirmed it fails on the previous union and passes on this one: old union -> 1 failed, 4 passed (the sentinel case) new union -> 5 passed * test(e2e): remove the cache-settings round-trip test The test could not fail for the thing it claimed to test, and could break the deployment it ran against. Both halves of that are worth stating. It read the live settings, wrote back identical values, and asserted the read-back matched. If POST /cache/settings were a complete no-op that returned 200 and touched nothing, GET would still return the values read a moment earlier and the test would pass. It verified that GET is stable, not that the route persists anything. Against that, /cache/settings persists what it receives into LiteLLM_CacheConfig, that row outranks YAML cache_params, and init_cache_settings_in_db re-applies it on a timer. A write that omits ssl or redis_startup_nodes converts a TLS cluster into a plaintext standalone client and every later Redis call blocks to socket timeout. On 2026-07-25 that failed 60 of 72 tests in one run: rate limiting stopped enforcing, Redis-only budgets admitted billable over-budget spend, ResetBudgetJob self-starved, and guardrail sync never ran. Guarding the previous shape was not sufficient. Writing the blob verbatim plus a cluster precondition and a /cache/ping check narrowed the hazard but did not remove it, because GET cannot express the effective config: it resolves the stored row overlaid with REDIS_* env and never reads YAML. On a fresh deploy it cannot see YAML's ssl to echo back, so a TLS non-cluster deployment could still have a row written that drops it. No round-trip through this route is safe on a shared proxy. Removed with the models and helpers it owned, and TestCacheSettingsModel with them since it existed only to protect that parsing. The registry row mgmt.cache_settings.update.happy_path stays, now carrying the rationale for why it is deliberately uncovered and what a safe test would require (an isolated proxy, or LIT-4816 fixed so a partial write cannot downgrade transport). Coverage therefore reports this cell as a gap, which is the honest state. Collector passes --strict; the module still collects 11 tests.	2026-07-25 23:12:55 +00:00
devin-ai-integration[bot]	6a180e9e5d	test(a2a): assert the property agent returns listings, not just non-empty text (#34515)	2026-07-24 11:03:28 -07:00
mubashir1osmani	0d483a7df9	test(e2e): pin the a2a bridge agent's Anthropic key so message/send works (#34512) ... The a2a completion-bridge tests registered the agent with only custom_llm_provider and model, so the bridge's litellm.acompletion had no api_key and relied on the gateway resolving ANTHROPIC_API_KEY from its ambient env. When that env var is absent, POST /a2a/{id} returns 500 with "Missing Anthropic API Key" and every message/send test (completion bridge, pinned v0.3/v1.0 message shapes, semver serves) fails while the register/discovery/rejection tests still pass. Give A2ABridgeParams an optional api_key and register the bridge agent with api_key="os.environ/ANTHROPIC_API_KEY", matching how the rest of the suite wires anthropic-backed models (e.g. the ratelimit redis tests). The agent now carries its provider key explicitly instead of depending on ambient gateway env.	2026-07-24 10:50:26 -07:00
mateo-berri	0bfdb37266	test(e2e): route external agent card fetch through the typed transport ... Adds get_external to e2e_http.py for absolute third-party GETs (no proxy base url or auth, same Result classification) and rewires fetch_agent_card through it, dropping the urllib.request escape hatch. Creates tests/code_coverage_tests/check_e2e_no_raw_requests.py, the checker tests/e2e/CLAUDE.md already referenced, and wires it into the code-quality workflow so raw HTTP client imports outside the transport fail CI; pre-existing uses (root conftest liveness probe, claude_code version resolver) are grandfathered and exception-type-only imports stay allowed.	2026-07-21 16:35:02 -07:00
yassin	51f0f40c2f	test(e2e): invoke a real published a2a agent and assert it replies ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-21 22:33:07 +00:00
mateo-berri	43402b04bd	test(e2e): register the verbatim a2a-sdk 0.3.x card and guard malformed protocolVersion	2026-07-21 14:43:09 -07:00
yassin	1315ebd1f9	test(e2e): guard 0.3.0-style semver protocolVersion registration ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-21 20:14:21 +00:00
yassin	5e68a00347	test(e2e): add live A2A agent e2e suite ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-21 00:10:49 +00:00