mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yassin Kortam
|
442fdc181e
|
docs(tests/e2e): align docs with the hard-fail-on-dead-proxy contract and scope the no-unit-tests rule (#33755)
The e2e docs claimed `e2e`-marked tests skip when no proxy answers the liveness probe, but the harness has always hard-failed: conftest.py's pytest_runtest_setup calls pytest.fail, its module docstring states "hard failures only ... never skip", and logging/conftest.py forbids skipping outright. Align the docs to the code so the single most important contract reads the same everywhere; a dead proxy turns a run red instead of being silently skipped and mistaken for a pass. The per-suite conftest docstrings that described the shared hook as a "proxy liveness skip" are corrected to "liveness gate" for the same reason. Also scope the no-unit-tests hard rule to what it means: never substitute a unit test for e2e feature coverage, while explicitly allowing tests that cover the harness itself (e.g. coverage_registry/test_collector.py), which carry no e2e marker and run whether or not a proxy is up. No product code and no harness logic changed. Resolves LIT-4554 |
||
|
mubashir1osmani
|
10462eddaf
|
test(e2e): harness fixes for stage job green (skips + router/UI/budget) (#33634)
* test(e2e): harness fixes for long_context, complexity router, UI, and unit coverage Point long_context_1m at 1M-capable models, harden complexity-smart-router registration and spend-log assertions, fix key models dropdown selectors, and add gateway/lifecycle/transport and claude_code unit tests * test(e2e): harden remaining stage failures in harness Register complexity-smart-router via create_model + callable probe, fix create-key UI navigation race, retry management writes and budget ALB 502s, mark Vertex count_tokens N/A when unsupported, and tighten tool_search model lists for Azure/Bedrock capability gaps * test(e2e): drop claude_code and harness unit tests from this PR Keep management, router, budget, and shared conftest harness fixes only * test(e2e): restore E2E_RESULT pytest_runtest_makereport hook Accidentally dropped in an earlier harness commit; Grafana status history depends on these structured log lines * test(e2e): drop management control-plane write retries Transient 500 retries do not fix the underlying control plane failures * test(e2e): skip stage-red claude_code cells; fix multi-window budget latency Mark the twelve failing claude_code matrix cells skip until product/config lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of Claude so the reset wait stays under ALB target idle timeout rather than masking awselb 502s * test(e2e): require exactly one LLM-tier spend row for complexity router Keep alias membership for compose vs stage model names, but assert len(served) == 1 so a leaked classifier sub-call cannot pass. Also pin LIT-4521 skip and align LIT-4522/23/24 skip reasons * test(e2e): harden router callable probe and multi-window budget exhaustion _router_is_callable treated any non-success chat whose body lacked "Invalid model name" as callable, so an unpropagated probe key (401), a generic 502, or a connection reset let the session proceed and hit real "Invalid model name" failures inside the tests. Require a Success outcome instead; the reload-race 400 and every infra/auth error now correctly read as not-callable. The multi-window budget test capped the tight window at 3e-6, which gpt-5.5 exhausts on the first call but a cheaper CHEAP_OPENAI_MODEL might not within the 20-call loop, turning a reset test into a spurious "window never enforced" failure. Drop the tight cap to 1e-9 so the first billed call exhausts it regardless of model price; the roomy 1m window stays at 1.0 and never blocks. * test(e2e): use a tradeoff-decision prompt for the complexity router classifier "Is P equal to NP?" reads to the LLM classifier as a short yes/no question, so gpt-5.5 classified it SIMPLE and the request routed to the openai backend, which made the test fail even though the classifier was running. The tier definitions key on what the request demands, not how hard the answer is, and a short direct question maps to SIMPLE regardless of subject. Swap in "Should I pay off my mortgage early or invest the extra money instead?". It carries none of the heuristic scorer's reasoning/technical/code keywords and stays short, so heuristic scoring still lands SIMPLE (openai), but the LLM reads it as a decision that has to weigh tradeoffs and lands it above SIMPLE, which the config routes to anthropic. Any non-SIMPLE tier serves anthropic, so the classifier only has to avoid SIMPLE for the test to distinguish a real classifier run from the heuristic fallback. |
||
|
mubashir1osmani
|
224fe67f10
|
test: e2e staging leftovers (#33613)
* test(e2e): read datadog log delivery back from the real datadog api (#33604) * test(e2e): read datadog log delivery back from the real datadog api * test(e2e): compare datadog-read cost with math.isclose, not bit-equality The response_cost now round-trips through DataDog's attribute indexing pipeline, whose float serialization is not guaranteed to preserve the exact bit pattern the proxy shipped. rel_tol=1e-9 (equal to 9 significant digits) still fails on any real cost discrepancy while tolerating representation drift. Addresses the Greptile P2 on this PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): widen the duplicate-settle window to 30s for real DataDog Against the local sink one poll interval (5s) after the first hit was enough to catch a same-call duplicate, because both events arrived in the same flush batch. Against real DataDog, ingestion jitter can make one call's two events searchable tens of seconds apart, so a 5s settle could let the LIT-4447 duplicate slip past the exactly-one assertion. The reader now keeps re-reading for DD_SETTLE_SECONDS (default 30s, env-overridable via E2E_DD_SETTLE_SECONDS) after the first event appears, returning early only when a duplicate is already visible - more waiting cannot clear it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): point UI tests at dashboard service; register complexity router Stage gateway 404s /ui; the Next.js dashboard is litellm-ui:3000. Drive playwright against E2E_UI_BASE_URL and wait on login placeholders after client render. Register complexity-smart-router via /model/new when the proxy does not already list it so stage matches compose config * docs(e2e): clarify E2E_UI_BASE_URL should be ALB when ingress splits UI * docs(e2e): prefer single path-routing host for control plane and UI CONTROL_PLANE and UI already default to PROXY_BASE_URL; clarify that stage should set one ALB host rather than three endpoints * fix(e2e): always capture complexity router model_id for teardown Split /model/new from the data-plane wait so a propagation timeout still deletes the control-plane registration (greptile orphan-model concern) * fix(e2e): click exact Login button so SSO control is not matched Playwright strict mode matched both Login and Login with SSO * fix(router): score complexity by difficulty not request length The LLM classifier prompt treated short wording as SIMPLE, so probes like "Is P equal to NP?" stayed on the SIMPLE backend even though the classifier ran. Judge intellectual difficulty so short hard questions route higher * fix(e2e): open key edit via Key ID and wait for team models Key Alias text is not the row open control on the virtual keys table; KeyInfoView opens from the Key ID button in that row. Also wait for a real team model in the edit Models dropdown so we do not race the async availableModels fetch that only has All Team Models on first paint * fix(e2e): keep settled DD events on empty search; bump mcp for OSV Do not let a transient empty DataDog search wipe events already seen in the settle window (Greptile P1). Make the logs-search from window env-overridable via E2E_DD_SEARCH_FROM (Greptile P2). Prefer the mono Key ID button when opening key edit. Bump mcp 1.26.0 -> 1.28.1 so OSV clears the three high GHSA findings on the staging PR * revert: drop mcp lock bump from e2e staging PR OSV mcp upgrade is unrelated to the e2e fixes; leave the dep pin alone --------- Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
fac43df9b9
|
fix(complexity_router): return empty dict from _classifier_call_metadata when metadata is absent (#33452)
* fix(complexity_router): return empty dict from _classifier_call_metadata when metadata is absent
The LLM classifier reads request_kwargs.get("litellm_metadata"), but the proxy stores request metadata under "metadata", so this returned None. _classifier_call_metadata then passed None straight through to the classifier acompletion call, which assumes a dict and blows up with 'NoneType' object has no attribute 'update'; the router swallowed it and silently fell back to heuristic scoring, so the configured LLM classifier never ran. Returning an empty dict keeps the classifier call well-formed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover complexity-router LLM classifier routes over the proxy
Add a live e2e regression for the complexity auto-router: a lexically simple but hard prompt ("Is P equal to NP?") is routed by the LLM classifier to the higher-tier anthropic backend, read back from the spend log's model. Before the metadata fix the classifier silently crashed and the router fell back to heuristic SIMPLE scoring on the openai backend, so this test fails pre-fix and passes post-fix.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|