mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
fdf380d0e3
* test(e2e): harden stage flakes for batches, UI, and MCP Unique batch model names avoid load-balancing onto stale azure-batch deployments that still pointed at the retired gpt-4.1-mini-batch, which only the managed/unified path was hitting. Retry batch retrieve on 500 and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite when the compose-only mcp-upstream is unreachable on stage k8s * test(e2e): cover Datadog remote MCP via search_datadog_logs Register the regional Datadog MCP endpoint with DD-API-KEY / DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*, assert the proxy shipped it, list tools, call search_datadog_logs for the marker, and delete the server on teardown. Math-upstream key-access tests only skip when that compose service is unreachable * test(e2e): drop compose math MCP upstream; use Datadog only Key-access denial and happy-path MCP e2e both register the real regional Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers. Remove the mcp-upstream compose service and FastMCP add/multiply fixture * docs(e2e): require real Datadog MCP for all mcp suite tests Document that tests/e2e/mcp must register via datadog_mcp helpers against mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams * chore: restore mcp_e2e_upstream_server.py Keep the FastMCP fixture file; e2e no longer wires it in compose, but the module itself is not part of the Datadog-only cleanup * fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load pytest on the host never inherited compose env_file keys, so DD_API_KEY stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the dynamically loaded datadog_reader module in sys.modules so dataclasses do not crash under Python 3.12 * test(e2e/batches): harden azure/vertex unified lifecycle flakes Put the provider deployment name in every JSONL body so Azure does not depend on a perfect model rewrite. Retry create/retrieve/cancel on transient statuses with backoff. Drop cancel assertions for azure and vertex (registry only has a shared basic cell; create+retrieve prove routing, cancel stays best-effort cleanup) * test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED Post-login client redirects abort the first /ui/api-keys/ goto on stage. Wait off /ui/login after cookie set, then accept the page once Create New Key is visible even if goto raised ERR_ABORTED * test(e2e): drop flaky key models dropdown Playwright suite API management e2e already covers key generate/update persistence. The UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and no unique product signal. Remove the suite and unused browser fixtures
53 lines
1.8 KiB
Python
53 lines
1.8 KiB
Python
"""MCP suite's `client` fixture.
|
|
|
|
The shared lifecycle (resources/scoped_key), proxy liveness handling, and the
|
|
`e2e`/`covers` markers live in the parent tests/e2e/conftest.py. McpClient holds
|
|
the shared ProxyClient, so the `resources` fixture tears down whatever this suite
|
|
creates (keys via the ProxyClient, MCP servers via the deferred cleanups).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib.util
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import Protocol, cast
|
|
|
|
import pytest
|
|
|
|
from mcp_client import McpClient, build_client
|
|
from proxy_client import ProxyClient
|
|
|
|
|
|
class DdLogsReader(Protocol):
|
|
def poll_events_for_marker(self, marker: str) -> list[object]: ...
|
|
|
|
|
|
class _DdLogsReaderBuilder(Protocol):
|
|
def __call__(self) -> DdLogsReader: ...
|
|
|
|
|
|
def _build_dd_logs_reader() -> DdLogsReader:
|
|
# Load logging/datadog_reader.py by path so basedpyright does not require a
|
|
# package layout. Register the module in sys.modules before exec so
|
|
# dataclasses inside it can resolve cls.__module__ (otherwise Python 3.12
|
|
# raises AttributeError: 'NoneType' object has no attribute '__dict__').
|
|
path = Path(__file__).resolve().parent.parent / "logging" / "datadog_reader.py"
|
|
name = "e2e_logging_datadog_reader"
|
|
spec = importlib.util.spec_from_file_location(name, path)
|
|
assert spec is not None and spec.loader is not None
|
|
module = importlib.util.module_from_spec(spec)
|
|
sys.modules[name] = module
|
|
spec.loader.exec_module(module)
|
|
builder = cast(_DdLogsReaderBuilder, getattr(module, "build_dd_logs_reader"))
|
|
return builder()
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def client(proxy: ProxyClient) -> McpClient:
|
|
return build_client(proxy)
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def dd_logs() -> DdLogsReader:
|
|
return _build_dd_logs_reader()
|