* test(e2e): read datadog log delivery back from the real datadog api (#33604)
* test(e2e): read datadog log delivery back from the real datadog api
* test(e2e): compare datadog-read cost with math.isclose, not bit-equality
The response_cost now round-trips through DataDog's attribute indexing
pipeline, whose float serialization is not guaranteed to preserve the
exact bit pattern the proxy shipped. rel_tol=1e-9 (equal to 9 significant
digits) still fails on any real cost discrepancy while tolerating
representation drift. Addresses the Greptile P2 on this PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): widen the duplicate-settle window to 30s for real DataDog
Against the local sink one poll interval (5s) after the first hit was
enough to catch a same-call duplicate, because both events arrived in the
same flush batch. Against real DataDog, ingestion jitter can make one
call's two events searchable tens of seconds apart, so a 5s settle could
let the LIT-4447 duplicate slip past the exactly-one assertion. The reader
now keeps re-reading for DD_SETTLE_SECONDS (default 30s, env-overridable
via E2E_DD_SETTLE_SECONDS) after the first event appears, returning early
only when a duplicate is already visible - more waiting cannot clear it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(e2e): point UI tests at dashboard service; register complexity router
Stage gateway 404s /ui; the Next.js dashboard is litellm-ui:3000. Drive
playwright against E2E_UI_BASE_URL and wait on login placeholders after
client render. Register complexity-smart-router via /model/new when the
proxy does not already list it so stage matches compose config
* docs(e2e): clarify E2E_UI_BASE_URL should be ALB when ingress splits UI
* docs(e2e): prefer single path-routing host for control plane and UI
CONTROL_PLANE and UI already default to PROXY_BASE_URL; clarify that
stage should set one ALB host rather than three endpoints
* fix(e2e): always capture complexity router model_id for teardown
Split /model/new from the data-plane wait so a propagation timeout still
deletes the control-plane registration (greptile orphan-model concern)
* fix(e2e): click exact Login button so SSO control is not matched
Playwright strict mode matched both Login and Login with SSO
* fix(router): score complexity by difficulty not request length
The LLM classifier prompt treated short wording as SIMPLE, so probes like
"Is P equal to NP?" stayed on the SIMPLE backend even though the classifier
ran. Judge intellectual difficulty so short hard questions route higher
* fix(e2e): open key edit via Key ID and wait for team models
Key Alias text is not the row open control on the virtual keys table;
KeyInfoView opens from the Key ID button in that row. Also wait for a
real team model in the edit Models dropdown so we do not race the async
availableModels fetch that only has All Team Models on first paint
* fix(e2e): keep settled DD events on empty search; bump mcp for OSV
Do not let a transient empty DataDog search wipe events already seen in
the settle window (Greptile P1). Make the logs-search from window
env-overridable via E2E_DD_SEARCH_FROM (Greptile P2). Prefer the mono
Key ID button when opening key edit. Bump mcp 1.26.0 -> 1.28.1 so OSV
clears the three high GHSA findings on the staging PR
* revert: drop mcp lock bump from e2e staging PR
OSV mcp upgrade is unrelated to the e2e fixes; leave the dep pin alone
---------
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(complexity_router): return empty dict from _classifier_call_metadata when metadata is absent
The LLM classifier reads request_kwargs.get("litellm_metadata"), but the proxy stores request metadata under "metadata", so this returned None. _classifier_call_metadata then passed None straight through to the classifier acompletion call, which assumes a dict and blows up with 'NoneType' object has no attribute 'update'; the router swallowed it and silently fell back to heuristic scoring, so the configured LLM classifier never ran. Returning an empty dict keeps the classifier call well-formed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover complexity-router LLM classifier routes over the proxy
Add a live e2e regression for the complexity auto-router: a lexically simple but hard prompt ("Is P equal to NP?") is routed by the LLM classifier to the higher-tier anthropic backend, read back from the spend log's model. Before the metadata fix the classifier silently crashed and the router fell back to heuristic SIMPLE scoring on the openai backend, so this test fails pre-fix and passes post-fix.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
