mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
224fe67f10
* test(e2e): read datadog log delivery back from the real datadog api (#33604) * test(e2e): read datadog log delivery back from the real datadog api * test(e2e): compare datadog-read cost with math.isclose, not bit-equality The response_cost now round-trips through DataDog's attribute indexing pipeline, whose float serialization is not guaranteed to preserve the exact bit pattern the proxy shipped. rel_tol=1e-9 (equal to 9 significant digits) still fails on any real cost discrepancy while tolerating representation drift. Addresses the Greptile P2 on this PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): widen the duplicate-settle window to 30s for real DataDog Against the local sink one poll interval (5s) after the first hit was enough to catch a same-call duplicate, because both events arrived in the same flush batch. Against real DataDog, ingestion jitter can make one call's two events searchable tens of seconds apart, so a 5s settle could let the LIT-4447 duplicate slip past the exactly-one assertion. The reader now keeps re-reading for DD_SETTLE_SECONDS (default 30s, env-overridable via E2E_DD_SETTLE_SECONDS) after the first event appears, returning early only when a duplicate is already visible - more waiting cannot clear it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): point UI tests at dashboard service; register complexity router Stage gateway 404s /ui; the Next.js dashboard is litellm-ui:3000. Drive playwright against E2E_UI_BASE_URL and wait on login placeholders after client render. Register complexity-smart-router via /model/new when the proxy does not already list it so stage matches compose config * docs(e2e): clarify E2E_UI_BASE_URL should be ALB when ingress splits UI * docs(e2e): prefer single path-routing host for control plane and UI CONTROL_PLANE and UI already default to PROXY_BASE_URL; clarify that stage should set one ALB host rather than three endpoints * fix(e2e): always capture complexity router model_id for teardown Split /model/new from the data-plane wait so a propagation timeout still deletes the control-plane registration (greptile orphan-model concern) * fix(e2e): click exact Login button so SSO control is not matched Playwright strict mode matched both Login and Login with SSO * fix(router): score complexity by difficulty not request length The LLM classifier prompt treated short wording as SIMPLE, so probes like "Is P equal to NP?" stayed on the SIMPLE backend even though the classifier ran. Judge intellectual difficulty so short hard questions route higher * fix(e2e): open key edit via Key ID and wait for team models Key Alias text is not the row open control on the virtual keys table; KeyInfoView opens from the Key ID button in that row. Also wait for a real team model in the edit Models dropdown so we do not race the async availableModels fetch that only has All Team Models on first paint * fix(e2e): keep settled DD events on empty search; bump mcp for OSV Do not let a transient empty DataDog search wipe events already seen in the settle window (Greptile P1). Make the logs-search from window env-overridable via E2E_DD_SEARCH_FROM (Greptile P2). Prefer the mono Key ID button when opening key edit. Bump mcp 1.26.0 -> 1.28.1 so OSV clears the three high GHSA findings on the staging PR * revert: drop mcp lock bump from e2e staging PR OSV mcp upgrade is unrelated to the e2e fixes; leave the dep pin alone --------- Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
63 lines
2.2 KiB
Python
63 lines
2.2 KiB
Python
"""Management suite fixtures: the client plus a logged-in dashboard page.
|
|
|
|
Lifecycle/skip/marker live in the parent conftest. The browser fixtures drive
|
|
the dashboard the proxy serves at /ui, so browser tests exercise exactly what an
|
|
end user sees. playwright is an optional dependency loaded behind importorskip
|
|
inside the fixture, so the API tests in this suite collect and run without it:
|
|
|
|
uv pip install playwright && uv run playwright install chromium
|
|
"""
|
|
|
|
from typing import TYPE_CHECKING, Iterator
|
|
|
|
import pytest
|
|
|
|
from e2e_config import UI_BASE_URL, UI_PASSWORD, UI_USERNAME
|
|
from management_client import ManagementClient, build_client
|
|
|
|
if TYPE_CHECKING:
|
|
from playwright.sync_api import Browser, Page
|
|
|
|
|
|
def pytest_configure(config: pytest.Config) -> None:
|
|
config.addinivalue_line(
|
|
"markers",
|
|
"covers: registry cell a test covers, e.g. mgmt.key.generate.persists",
|
|
)
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def client() -> ManagementClient:
|
|
return build_client()
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def browser() -> "Iterator[Browser]":
|
|
pytest.importorskip("playwright.sync_api", reason="playwright not installed")
|
|
from playwright.sync_api import sync_playwright
|
|
|
|
with sync_playwright() as playwright:
|
|
launched = playwright.chromium.launch()
|
|
yield launched
|
|
launched.close()
|
|
|
|
|
|
@pytest.fixture
|
|
def ui_page(browser: "Browser") -> "Iterator[Page]":
|
|
context = browser.new_context()
|
|
try:
|
|
page = context.new_page()
|
|
# Split deploys serve the Next.js dashboard on the UI service, not the
|
|
# data-plane gateway (which 404s /ui). Login is a client-rendered form
|
|
# that appears after LoadingScreen; wait on the placeholder, not #id
|
|
# (Ant Design Input does not always set id="username").
|
|
page.goto(f"{UI_BASE_URL}/ui/login")
|
|
username = page.get_by_placeholder("Enter your username")
|
|
username.wait_for(state="visible", timeout=30_000)
|
|
username.fill(UI_USERNAME)
|
|
page.get_by_placeholder("Enter your password").fill(UI_PASSWORD)
|
|
page.get_by_role("button", name="Login", exact=True).click()
|
|
page.wait_for_function("() => document.cookie.includes('token=')")
|
|
yield page
|
|
finally:
|
|
context.close()
|