mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mubashir1osmani
|
ac5b51253a
|
test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149)
* test(e2e): add other suite covering master-key auth and health lifecycle Covers the other.* holding-pen cells that were uncovered: master-key valid_allows/invalid_denied on the admin /user/list gate, and the lifecycle probes liveness.ping, readiness.public_probe, readiness.reports_db_status, and readiness_details.authenticated_diagnostics. New tests/e2e/other/ suite on the shared ProxyClient; the health probes send no auth header to prove the public routes need no credential, and the details route is asserted to reject an anonymous caller while exposing version/db diagnostics to the master key. * test(e2e): cover block_code_execution and openai_moderation guardrails Extends the guardrails suite with two built-in guardrails registered per request (default_on=False, opted in via the chat body's guardrails selector) so neither intercepts unrelated traffic on the shared proxy. block_code_execution.pre_call.blocks: a python code block plus a run-this request is intercepted with the canned content-blocked message and the model never runs, while the same code block asked about with don't-run-it reaches the model. Verified live. openai_moderations.pre_call.blocks: a flagged prompt is rejected 400 naming the moderation policy while a benign prompt passes. The guardrail calls OpenAI's moderation API; verifying it needs an OpenAI key with moderation quota (this account currently 429s the moderation endpoint). Adds a shared create_backend_model helper and a generic register() plus per-request guardrails/max_tokens on the client so more built-ins can reuse the same path. * test(e2e): cover presidio PII masking (pre_call + post_call) Registers a presidio guardrail per request (default_on=False) with the analyzer/anonymizer bases supplied in the registration params, so the test controls its own dependency and needs no proxy restart. presidio.pre_call.masks: a repeat-verbatim request comes back with the <EMAIL_ADDRESS> placeholder and never the raw email, proving the prompt was anonymized before the model saw it. presidio.post_call.masks: with apply_to_output the model's own emitted email is masked on the way out, so the caller never receives the raw value. Both verified live against real presidio analyzer + anonymizer containers. logging_only is intentionally not covered: /spend/logs exposes no prompt messages to read back the masked log, and a logging_only run also masked the response, contradicting its contract; noted in the module docstring for a follow-up. * test(e2e): cover presidio logging_only masking via OTEL read-back Adds the third presidio cell, guardrail.presidio.logging_only.masks. The logging_only contract (mask what is logged, do not block) is verified by reading the request's gen-AI span back from the real OTEL destination: the span's gen_ai.input.messages attribute carries the <EMAIL_ADDRESS> placeholder, never the raw email, and the call itself is not blocked. Reads the trace via the shared OtelReader, promoted from logging/ to the suite root so both suites use it. The masked prompt is polled to a deadline because logging_only masks the payload asynchronously and the span can briefly export before the mask lands. Drops the throwaway chat_send in favor of the existing transport.send for the call-id capture. * fix(e2e): tolerate cross-pod guardrail sync delay in team-opt-out test Stage runs multiple gateway pods behind the shared key. POST /guardrails registers a new default-on guardrail in-process immediately only on the pod that served the create call; every other pod picks it up on its next periodic DB sync (proxy_server.py, every 30s), so the very next chat call can race a pod that has not synced yet. Poll to a 40s deadline instead of asserting on the first response, matching the existing pattern in test_budget_reset_advances_e2e.py. * test(e2e): cover a guardrail on the MCP tool-call path (content_filter pre_mcp_call) Adds guardrail.litellm_content_filter.pre_mcp_call.blocks: against the real Datadog MCP server, a content_filter guardrail configured mode=pre_mcp_call blocks a banned keyword in an MCP tool call's arguments with HTTP 400 attributed to the pre_mcp_call hook, and lets a clean argument reach the upstream server. The guardrail attaches with default_on because per-key/request guardrail selection is dropped from the synthetic MCP request the hook sees; the banned keyword is unique per run so default_on only intercepts this test's own call. mode must be pre_mcp_call - a pre_call config silently no-ops on tools/call because the event type is rewritten for call_mcp_tool. Drives the tool directly via /mcp-rest/tools/call for a deterministic check of the same pre_mcp_call enforcement the OpenAI-SDK chat path hits when a model invokes an MCP tool. * fix(e2e): mid-conversation messages test uses client.proxy not client.gateway EndpointsClient exposes .proxy after the Gateway->ProxyClient rename; the mid-conversation system test still referenced .gateway, which fails the e2e basedpyright gate. Aligns it with the rest of the harness. * test(e2e): address review on the guardrail coverage MCP tool-call guardrail: poll the banned call until the guardrail is enforced instead of asserting on the first call, so the control-plane -> data-plane guardrail sync cannot race the check into a false pass-through; add a repeat banned call after enforcement to guard against a partial-propagation state. OpenAI moderation: distinguish a moderation-endpoint 429 (rate limit / no moderation quota) from a guardrail failure, so an account-capability gap reads as such rather than as "did not block". Runs green with a moderation-capable key. * test(e2e): close partial-propagation false-pass in MCP guardrail block test The single post-block repeat call could be load-balanced back to the same already-synced data-plane pod, so the test could pass while another pod still lacked the guardrail and let the banned MCP call reach Datadog. Anchor a wait to the guardrail create time (every pod is guaranteed to have DB-synced only after a full ~30s sync interval), then require the banned call to stay blocked across several attempts; a pass-through after that window is a real leak, not a race. * test(e2e): drop xfail-style rate-limit branch from openai_moderation test OpenAI's /v1/moderations is free and returns 200 with the env key (verified directly), so the RateLimitedError branch mislabeled the failure: a 429 there is insufficient_quota (no account billing), not throttling. The branch also only printed a softer message before failing anyway, an xfail-in-disguise the e2e rules forbid. A 429 now falls through and fails loudly with the full result. |
||
|
mubashir1osmani
|
fdf380d0e3
|
test(e2e): harden stage flakes for batches, UI, and MCP (#33831)
* test(e2e): harden stage flakes for batches, UI, and MCP Unique batch model names avoid load-balancing onto stale azure-batch deployments that still pointed at the retired gpt-4.1-mini-batch, which only the managed/unified path was hitting. Retry batch retrieve on 500 and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite when the compose-only mcp-upstream is unreachable on stage k8s * test(e2e): cover Datadog remote MCP via search_datadog_logs Register the regional Datadog MCP endpoint with DD-API-KEY / DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*, assert the proxy shipped it, list tools, call search_datadog_logs for the marker, and delete the server on teardown. Math-upstream key-access tests only skip when that compose service is unreachable * test(e2e): drop compose math MCP upstream; use Datadog only Key-access denial and happy-path MCP e2e both register the real regional Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers. Remove the mcp-upstream compose service and FastMCP add/multiply fixture * docs(e2e): require real Datadog MCP for all mcp suite tests Document that tests/e2e/mcp must register via datadog_mcp helpers against mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams * chore: restore mcp_e2e_upstream_server.py Keep the FastMCP fixture file; e2e no longer wires it in compose, but the module itself is not part of the Datadog-only cleanup * fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load pytest on the host never inherited compose env_file keys, so DD_API_KEY stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the dynamically loaded datadog_reader module in sys.modules so dataclasses do not crash under Python 3.12 * test(e2e/batches): harden azure/vertex unified lifecycle flakes Put the provider deployment name in every JSONL body so Azure does not depend on a perfect model rewrite. Retry create/retrieve/cancel on transient statuses with backoff. Drop cancel assertions for azure and vertex (registry only has a shared basic cell; create+retrieve prove routing, cancel stays best-effort cleanup) * test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED Post-login client redirects abort the first /ui/api-keys/ goto on stage. Wait off /ui/login after cookie set, then accept the page once Create New Key is visible even if goto raised ERR_ABORTED * test(e2e): drop flaky key models dropdown Playwright suite API management e2e already covers key generate/update persistence. The UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and no unique product signal. Remove the suite and unused browser fixtures |
||
|
Yassin Kortam
|
08fa25042c
|
test(e2e): rename Gateway to ProxyClient and expose it as a session-scoped fixture (#33750)
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is not a gateway server, it is the client every suite uses to talk to the proxy (keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the module to proxy_client.py and the class to ProxyClient, with build_gateway becoming build_proxy_client and the GatewayProvider protocol becoming ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only identifiers changed; prose and string literals that use the word gateway for the proxy-server concept were left alone. Each suite previously built its own instance through a per-suite build_client() that called build_gateway() inside, duplicating the proxy wiring across suites. There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every suite's client fixture depends on it and injects it, so the wiring lives in one place. claude_code keeps building its own client directly since it has its own harness and does not use the shared fixtures. Behavior is unchanged: shared transport, data-plane/control-plane split routing, poll budget, typed request/response models, and resource cleanup all go through the same object. |
||
|
Yassin Kortam
|
89c87ae59a
|
test(e2e): mcp suite for key-without-access denial (#33752)
Add an e2e suite at tests/e2e/mcp/ that proves MCP authorization over the api_key auth family. An admin registers an upstream MCP server through the management API (POST /v1/mcp/server, persisted in the DB and picked up without a restart) and queues its deletion. Two keys are created against that one server: one granted access through object_permission.mcp_servers and one with no MCP grant. The permitted key is a live control proving the upstream is reachable and the tool is callable, so a denial on the ungranted key is an authorization decision rather than a dead server. The denied key then sees none of the server's tools on tools/list and is refused a tools/call with a 403 access_denied. A deterministic self-hosted FastMCP upstream (add/multiply over streamable-http) is added to the e2e compose stack so the suite runs offline with a known tool set. KeyGenerateBody gains an optional typed object_permission so the shared gateway can create a key with an MCP grant. |