mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-11 22:51:28 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
ryan-crabbe-berri
|
88ab22fefc
|
test(e2e): skip the three Datadog MCP tool-call tests pending LIT-5052 (#35380)
All three send a `telemetry` object in the arguments to Datadog's search_datadog_logs tool. Datadog tightened that tool's input schema to reject unknown properties, so every call now fails validation with 'unexpected additional properties ["telemetry"]' before the behavior each test exists to prove is reached. `telemetry` was never a documented Datadog parameter; the tests relied on the server ignoring extra properties. The proxy transmitted exactly what the tests supplied and surfaced the upstream error faithfully, so this is test-side. The covers markers and registry rows stay put: the collector counts a cell as covered only when a test pytest would actually run declares it, so skipping hands all four cells back to the gap list where they belong. |
||
|
mubashir1osmani
|
c274cf321c
|
test(e2e): poll MCP tools across multi-worker lag (#35047)
* fix(mcp): resolve call_tool by registry without requiring tool map
Multi-worker reloads put MCP servers in the registry from the DB but do
not re-run tools/list on every process. Gating call_tool on
tool_name_to_mcp_server_name_mapping made cold workers 500 with Tool not
found after another worker had already listed the tool. Treat a registry
match on server id/name/alias as enough; upstream rejects unknown tools
* test(e2e): poll MCP register, tools/list, and tools/call across multi-worker lag
Stage multi-worker gateways only load MCP servers and tool maps on the
process that handled the request. Poll until the server is listed, the
tool appears on tools/list, and tools/call is not a cold-worker 500 so
key-access and Datadog MCP e2e stop racing the LB
* Revert "fix(mcp): resolve call_tool by registry without requiring tool map"
This reverts commit
|
||
|
mubashir1osmani
|
a10365e84d
|
test(e2e): stop racing control-plane writes across the mcp, a2a, guardrail and passthrough suites (#34833)
* test(e2e): wait for MCP tool discovery instead of racing it
/v1/mcp/server returns as soon as the DB row is written, but the gateway runs
the initialize + tools/list handshake against the upstream lazily, on the first
request that needs it. Every MCP test read tools/list immediately after
registering, so it raced that handshake.
The gateway reports a server it has not discovered yet exactly like a dead one:
it catches the per-server handshake exception and returns an empty tool list.
The tests asserted on a single read, so the race surfaced as "granted key never
saw search_datadog_logs; tools=frozenset()" while a sibling test against the
same upstream in the same run passed.
Add McpClient.await_tool, which polls tools/list to the suite's existing
poll_timeout and returns the qualified tool name, and route the four discovery
sites through it. An unreachable upstream or an unapplied grant still fails, and
the failure now names the last tools/list result.
Refs LIT-4821
* test(e2e): wait for a2a agents to reach the data plane after registration
POST /v1/agents is a control-plane write; the /a2a/{agent_id} routes that serve
the card and run message/send are data plane and only see the agent after the
next DB reload. Every test registered an agent and immediately read its card or
sent it a message, so the first data-plane touch could 404 on the agent it had
just created.
register_agent now waits for the card to become servable before returning, the
same way ProxyClient.create_model waits for a new model, so callers do not each
have to poll. Registration failures skip the wait, leaving the two rejection
tests unchanged. A genuine propagation failure now fails naming the agent id and
the last card read rather than as a bare 404 on whichever /a2a call ran first.
Refs LIT-4821
* test(e2e): wait for presidio guardrails to sync before asserting masking
Registering a guardrail is a control-plane write; the data-plane worker that
serves /chat/completions only picks it up on its next periodic DB sync (~30s), so
the first call after the create ran against a worker with no guardrail and passed
the raw email straight through. The tests asserted on that first call, so they
read in-flight propagation as a PII leak.
Confirmed directly against a live proxy: the same call is unmasked at t=0s and
masked at t=8s, and the presidio analyzer itself correctly returns EMAIL_ADDRESS
with score 1.0 the whole time. The MCP guardrail suite already documents and
waits out this exact sync delay; presidio never got the same treatment.
Poll the call until the placeholder replaces the PII, so the assertions judge the
synced state. A guardrail that never masks still fails, on the last unmasked
content. pre_call and post_call now pass repeatably.
Refs LIT-4821
* test(e2e): drop the presidio logging_only check pending LIT-4841
pre_call and post_call masking both pass once the guardrail-sync wait is in place,
but logging_only left the raw email in the OTEL span's gen_ai.input.messages on
every attempt across a full poll deadline. Keeping an assertion against
known-failing behavior just turns every run red, so the cell is tracked in
LIT-4841 instead.
The registry row stays, so guardrail.presidio.logging_only.masks now reports as an
uncovered gap rather than silently disappearing.
Refs LIT-4821, LIT-4841
* test(e2e): wait for guardrail sync in bedrock, moderation and block-code checks
All three asserted on the first call after registering a guardrail, so they were
served by a data-plane worker that had not synced it yet (~30s DB poll) and read
in-flight propagation as a guardrail that failed to block. Verified directly: the
openai_moderation guardrail lets a flagged prompt through at t=0s and returns
"Violated OpenAI moderation policy" at t=8s.
The reasoning-only responses noted in triage (content=None with reasoning_tokens
set) were a symptom of the same thing, not the cause; these are pre_call
guardrails, so a synced guardrail rejects the request before the model runs.
Add poll_until_blocked to guardrails_client for the two that surface a non-success
status, and poll on the block marker in the block_code_execution check, which
replaces the reply rather than erroring. All eight guardrail tests now pass.
Refs LIT-4821
* test(e2e): drop the openai prompt-cache check pending LIT-4841
Prompt caching never engages through the proxy: cached_tokens is 0 on every
repeat, while the identical payload sent straight to OpenAI reports 3615 cached
tokens on the second call. Pinning prompt_cache_key on the proxy request restores
caching (3328 tokens), so something varying per request is defeating OpenAI's
automatic prefix cache.
That is a product bug with a direct billing cost, tracked in LIT-4841. The
registry row stays, so llm.chat_completions.openai.prompt_cache_5m.nonstream.works
now reports as an uncovered gap instead of failing every run.
Refs LIT-4821, LIT-4841
* test(e2e): drop the responses metadata redis-ttl check
It failed on a Redis read timeout against the stage serverless cache
(berrie-litellm-stage-ieib2i.serverless.use1.cache.amazonaws.com:6379), a
reachability problem this suite has hit before rather than a proxy defect the
assertion can pin down.
The file held only this test. Its other cell,
llm.responses.openai.basic.nonstream.works, is still covered by
test_responses_e2e.py; other.config.responses.metadata_redis_ttl_bounded becomes
an uncovered registry row, taking headline coverage 314/431 -> 312/431.
Refs LIT-4821
* test(e2e): fix passthrough header propagation and openai body, drop the cost check
Three separate problems behind the two passthrough failures.
The header test 404'd because POST /config/pass_through_endpoint is a
control-plane write and the worker serving the route only registers it on its next
config reload; measured at ~18s on a live proxy. Wait for the route to stop 404ing
before calling it. The readiness probe reuses the master key and omits
anthropic-version so polling does not bill a completion per attempt.
The openai passthrough body sent max_tokens, which the gpt-5 family rejects
outright ("Unsupported parameter: 'max_tokens' is not supported with this model").
Confirmed against OpenAI directly: max_tokens 400s, max_completion_tokens 200s.
Passthrough forwards the body untouched by design, so the body was simply wrong.
test_openai_passthrough_nonstreaming_logs_cost still finds no SpendLogs row for
its call_id after the fix, so it is removed rather than left red; the gemini and
anthropic passthrough cost checks still cover that path.
Passthrough suite is 8/8 green.
Refs LIT-4821
|
||
|
mubashir1osmani
|
fdf380d0e3
|
test(e2e): harden stage flakes for batches, UI, and MCP (#33831)
* test(e2e): harden stage flakes for batches, UI, and MCP Unique batch model names avoid load-balancing onto stale azure-batch deployments that still pointed at the retired gpt-4.1-mini-batch, which only the managed/unified path was hitting. Retry batch retrieve on 500 and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite when the compose-only mcp-upstream is unreachable on stage k8s * test(e2e): cover Datadog remote MCP via search_datadog_logs Register the regional Datadog MCP endpoint with DD-API-KEY / DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*, assert the proxy shipped it, list tools, call search_datadog_logs for the marker, and delete the server on teardown. Math-upstream key-access tests only skip when that compose service is unreachable * test(e2e): drop compose math MCP upstream; use Datadog only Key-access denial and happy-path MCP e2e both register the real regional Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers. Remove the mcp-upstream compose service and FastMCP add/multiply fixture * docs(e2e): require real Datadog MCP for all mcp suite tests Document that tests/e2e/mcp must register via datadog_mcp helpers against mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams * chore: restore mcp_e2e_upstream_server.py Keep the FastMCP fixture file; e2e no longer wires it in compose, but the module itself is not part of the Datadog-only cleanup * fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load pytest on the host never inherited compose env_file keys, so DD_API_KEY stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the dynamically loaded datadog_reader module in sys.modules so dataclasses do not crash under Python 3.12 * test(e2e/batches): harden azure/vertex unified lifecycle flakes Put the provider deployment name in every JSONL body so Azure does not depend on a perfect model rewrite. Retry create/retrieve/cancel on transient statuses with backoff. Drop cancel assertions for azure and vertex (registry only has a shared basic cell; create+retrieve prove routing, cancel stays best-effort cleanup) * test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED Post-login client redirects abort the first /ui/api-keys/ goto on stage. Wait off /ui/login after cookie set, then accept the page once Create New Key is visible even if goto raised ERR_ABORTED * test(e2e): drop flaky key models dropdown Playwright suite API management e2e already covers key generate/update persistence. The UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and no unique product signal. Remove the suite and unused browser fixtures |