Enterprise MCP users mint virtual keys against tool access groups rather than
explicit server ids. Nothing exercised that end to end.
Registers the upstream MCP server tagged with a server-side access group
(mcp_access_groups), then mints one key granted that group and one granted a
different group. Asserts the granted key sees the tagged server's tools on
tools/list and the other key sees none, so access-group scoping can't leak the
server across the boundary.
Adds mcp_access_groups support to the e2e MCP client (server registration, key
generation, ObjectPermission) and the registry cell
mcp.list_tools.api_key.access_group_scoped.
* test(e2e): add other suite covering master-key auth and health lifecycle
Covers the other.* holding-pen cells that were uncovered: master-key
valid_allows/invalid_denied on the admin /user/list gate, and the
lifecycle probes liveness.ping, readiness.public_probe,
readiness.reports_db_status, and readiness_details.authenticated_diagnostics.
New tests/e2e/other/ suite on the shared ProxyClient; the health probes
send no auth header to prove the public routes need no credential, and the
details route is asserted to reject an anonymous caller while exposing
version/db diagnostics to the master key.
* test(e2e): cover block_code_execution and openai_moderation guardrails
Extends the guardrails suite with two built-in guardrails registered per
request (default_on=False, opted in via the chat body's guardrails selector)
so neither intercepts unrelated traffic on the shared proxy.
block_code_execution.pre_call.blocks: a python code block plus a run-this
request is intercepted with the canned content-blocked message and the model
never runs, while the same code block asked about with don't-run-it reaches
the model. Verified live.
openai_moderations.pre_call.blocks: a flagged prompt is rejected 400 naming
the moderation policy while a benign prompt passes. The guardrail calls
OpenAI's moderation API; verifying it needs an OpenAI key with moderation
quota (this account currently 429s the moderation endpoint).
Adds a shared create_backend_model helper and a generic register() plus
per-request guardrails/max_tokens on the client so more built-ins can reuse
the same path.
* test(e2e): cover presidio PII masking (pre_call + post_call)
Registers a presidio guardrail per request (default_on=False) with the
analyzer/anonymizer bases supplied in the registration params, so the test
controls its own dependency and needs no proxy restart.
presidio.pre_call.masks: a repeat-verbatim request comes back with the
<EMAIL_ADDRESS> placeholder and never the raw email, proving the prompt was
anonymized before the model saw it.
presidio.post_call.masks: with apply_to_output the model's own emitted email
is masked on the way out, so the caller never receives the raw value.
Both verified live against real presidio analyzer + anonymizer containers.
logging_only is intentionally not covered: /spend/logs exposes no prompt
messages to read back the masked log, and a logging_only run also masked the
response, contradicting its contract; noted in the module docstring for a
follow-up.
* test(e2e): cover presidio logging_only masking via OTEL read-back
Adds the third presidio cell, guardrail.presidio.logging_only.masks. The
logging_only contract (mask what is logged, do not block) is verified by
reading the request's gen-AI span back from the real OTEL destination: the
span's gen_ai.input.messages attribute carries the <EMAIL_ADDRESS> placeholder,
never the raw email, and the call itself is not blocked.
Reads the trace via the shared OtelReader, promoted from logging/ to the suite
root so both suites use it. The masked prompt is polled to a deadline because
logging_only masks the payload asynchronously and the span can briefly export
before the mask lands. Drops the throwaway chat_send in favor of the existing
transport.send for the call-id capture.
* fix(e2e): tolerate cross-pod guardrail sync delay in team-opt-out test
Stage runs multiple gateway pods behind the shared key. POST /guardrails
registers a new default-on guardrail in-process immediately only on the
pod that served the create call; every other pod picks it up on its next
periodic DB sync (proxy_server.py, every 30s), so the very next chat call
can race a pod that has not synced yet. Poll to a 40s deadline instead of
asserting on the first response, matching the existing pattern in
test_budget_reset_advances_e2e.py.
* test(e2e): cover a guardrail on the MCP tool-call path (content_filter pre_mcp_call)
Adds guardrail.litellm_content_filter.pre_mcp_call.blocks: against the real
Datadog MCP server, a content_filter guardrail configured mode=pre_mcp_call
blocks a banned keyword in an MCP tool call's arguments with HTTP 400 attributed
to the pre_mcp_call hook, and lets a clean argument reach the upstream server.
The guardrail attaches with default_on because per-key/request guardrail
selection is dropped from the synthetic MCP request the hook sees; the banned
keyword is unique per run so default_on only intercepts this test's own call.
mode must be pre_mcp_call - a pre_call config silently no-ops on tools/call
because the event type is rewritten for call_mcp_tool.
Drives the tool directly via /mcp-rest/tools/call for a deterministic check of
the same pre_mcp_call enforcement the OpenAI-SDK chat path hits when a model
invokes an MCP tool.
* fix(e2e): mid-conversation messages test uses client.proxy not client.gateway
EndpointsClient exposes .proxy after the Gateway->ProxyClient rename; the
mid-conversation system test still referenced .gateway, which fails the e2e
basedpyright gate. Aligns it with the rest of the harness.
* test(e2e): address review on the guardrail coverage
MCP tool-call guardrail: poll the banned call until the guardrail is enforced
instead of asserting on the first call, so the control-plane -> data-plane
guardrail sync cannot race the check into a false pass-through; add a repeat
banned call after enforcement to guard against a partial-propagation state.
OpenAI moderation: distinguish a moderation-endpoint 429 (rate limit / no
moderation quota) from a guardrail failure, so an account-capability gap reads
as such rather than as "did not block". Runs green with a moderation-capable key.
* test(e2e): close partial-propagation false-pass in MCP guardrail block test
The single post-block repeat call could be load-balanced back to the same
already-synced data-plane pod, so the test could pass while another pod still
lacked the guardrail and let the banned MCP call reach Datadog. Anchor a wait to
the guardrail create time (every pod is guaranteed to have DB-synced only after a
full ~30s sync interval), then require the banned call to stay blocked across
several attempts; a pass-through after that window is a real leak, not a race.
* test(e2e): drop xfail-style rate-limit branch from openai_moderation test
OpenAI's /v1/moderations is free and returns 200 with the env key (verified
directly), so the RateLimitedError branch mislabeled the failure: a 429 there is
insufficient_quota (no account billing), not throttling. The branch also only
printed a softer message before failing anyway, an xfail-in-disguise the e2e rules
forbid. A 429 now falls through and fails loudly with the full result.
* test(e2e): harden stage flakes for batches, UI, and MCP
Unique batch model names avoid load-balancing onto stale azure-batch
deployments that still pointed at the retired gpt-4.1-mini-batch, which
only the managed/unified path was hitting. Retry batch retrieve on 500
and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite
when the compose-only mcp-upstream is unreachable on stage k8s
* test(e2e): cover Datadog remote MCP via search_datadog_logs
Register the regional Datadog MCP endpoint with DD-API-KEY /
DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is
not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*,
assert the proxy shipped it, list tools, call search_datadog_logs for the
marker, and delete the server on teardown. Math-upstream key-access tests
only skip when that compose service is unreachable
* test(e2e): drop compose math MCP upstream; use Datadog only
Key-access denial and happy-path MCP e2e both register the real regional
Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers.
Remove the mcp-upstream compose service and FastMCP add/multiply fixture
* docs(e2e): require real Datadog MCP for all mcp suite tests
Document that tests/e2e/mcp must register via datadog_mcp helpers against
mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams
* chore: restore mcp_e2e_upstream_server.py
Keep the FastMCP fixture file; e2e no longer wires it in compose, but the
module itself is not part of the Datadog-only cleanup
* fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load
pytest on the host never inherited compose env_file keys, so DD_API_KEY
stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the
dynamically loaded datadog_reader module in sys.modules so dataclasses
do not crash under Python 3.12
* test(e2e/batches): harden azure/vertex unified lifecycle flakes
Put the provider deployment name in every JSONL body so Azure does not
depend on a perfect model rewrite. Retry create/retrieve/cancel on
transient statuses with backoff. Drop cancel assertions for azure and
vertex (registry only has a shared basic cell; create+retrieve prove
routing, cancel stays best-effort cleanup)
* test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED
Post-login client redirects abort the first /ui/api-keys/ goto on stage.
Wait off /ui/login after cookie set, then accept the page once Create New
Key is visible even if goto raised ERR_ABORTED
* test(e2e): drop flaky key models dropdown Playwright suite
API management e2e already covers key generate/update persistence. The
UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and
no unique product signal. Remove the suite and unused browser fixtures
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
Add an e2e suite at tests/e2e/mcp/ that proves MCP authorization over the
api_key auth family. An admin registers an upstream MCP server through the
management API (POST /v1/mcp/server, persisted in the DB and picked up without
a restart) and queues its deletion. Two keys are created against that one
server: one granted access through object_permission.mcp_servers and one with
no MCP grant. The permitted key is a live control proving the upstream is
reachable and the tool is callable, so a denial on the ungranted key is an
authorization decision rather than a dead server. The denied key then sees
none of the server's tools on tools/list and is refused a tools/call with a
403 access_denied.
A deterministic self-hosted FastMCP upstream (add/multiply over
streamable-http) is added to the e2e compose stack so the suite runs offline
with a known tool set. KeyGenerateBody gains an optional typed
object_permission so the shared gateway can create a key with an MCP grant.
