* test(e2e): cover google-native generateContent framing and prometheus queue time
Adds live coverage for three shipped regressions that had none, all reached
through surfaces a customer drives from Google SDKs and operator dashboards.
The managed google-native route (`/v1beta/models/{model}:generateContent`) had
no harness support at all, so EndpointsClient gains generate_content and
stream_generate_content plus the request body models, and a new suite asserts
the two contracts that broke there: the response carries
x-litellm-response-cost so SDK traffic reconciles against spend (LIT-4076), and
the stream relays single-prefixed SSE frames with no OpenAI [DONE] terminator.
A doubled `data:` prefix, a leaked bytes literal, or the [DONE] sentinel each
fail the stream test; [DONE] absence is only asserted once real content has
arrived, because a first-chunk upstream error legitimately falls back to the
OpenAI error shape and does emit it.
The prometheus test pins litellm_request_queue_time_seconds to an actual
observation on our own key's series rather than to the family merely existing,
which is the distinction the original regression turned on: the histogram stayed
registered while nothing was ever written to it (LIT-2034).
Each assertion was mutation-checked against the live proxy; inverting the
[DONE] expectation, the cost-header expectation, or the metric name fails the
corresponding test.
* refactor(e2e): simplify google native coverage
* fix(e2e): make the datadog read-back find what DataDog actually indexes
Live verification of the merged #33604 against real DataDog (us5) exposed
three read-back defects that the local-sink tests could never see; all
three fixes are verified against the real API:
- Marker search: DataDog consumes the shipped JSON message into the
event's attributes and leaves the indexed message EMPTY, so the
full-text '"marker"' query matched nothing and every test failed with
zero events. The query is now '*:*marker*', which scans all attributes
(the marker sits in messages.content); verified to return exactly the
event for the call.
- Rate limit: the Logs Search API budget is 2 requests per 10s org-wide
(x-ratelimit-name logs_public_search_api). Polling at POLL_INTERVAL=5s
sat exactly at the limit and the reader hard-failed on the first 429.
Searches now pace at DD_SEARCH_INTERVAL (10s default) and a 429 backs
off and retries up to 5 times; only non-429 failures stay hard fails.
- Envelope status: DataDog re-derives the indexed event status from the
parsed payload's status attribute ('success') and normalizes it to its
OK severity, so the assertion expects 'ok', not the shipped 'info'.
Live run: chat_completions and responses pass every assertion including
the exact response-cost cross-check; messages red-pins the LIT-4447
duplicate for real (one call -> two sync-sweep copies + one async batch
copy, same request id, confirmed in proxy debug logs). The duplicate is
race-dependent, so the pin flickers until #33589 lands.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): datadog log delivery for streamed chat, messages, and responses
Rewritten from the dd-sink version (original #33566) to judge delivery on
what real DataDog ingested, matching the merged #33604 conversion: the
dd_logs reader searches events back through the Logs Search API and the
assertions validate the indexed envelope (source:litellm tag, ok status)
and the StandardLoggingPayload fields under the event's attributes.
Each streamed test drives one STREAMED call per route, asserts the stream
actually streamed (event-stream content type, >0 chunks, no upstream error
event), then pins exactly one DataDog event whose payload records
stream=true, the aggregated token count, and a response_cost equal to the
/spend/logs row for the call - a stream's headers ship before its cost
exists, so the spend row is the cross-check anchor, and the spend row and
DataDog event must also agree on total_tokens.
Coverage registry: adds logging.datadog.stream.exports_metric exercised on
chat_completions, messages, and responses.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update test_datadog_log_e2e.py
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(e2e): establish litellm_e2e_staging integration line
Long-lived berri branch for e2e suite recovery work (LIT-4479 through LIT-4486) before merge to litellm_internal_staging
* test(e2e): remove langfuse_otel logging e2e suite (#33558)
* test(e2e): remove langfuse_otel logging e2e suite
Removes the LIT-4483 dynamic per-team/key/org langfuse_otel logging e2e tests (tests/e2e/logging/test_langfuse_e2e.py, added in #32857). The shared logging_client harness and the langfuse coverage-registry cells are left in place; only the test module is removed. The otel and prometheus logging e2e suites are unaffected.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop orphaned langfuse coverage-registry cells
The three logging.langfuse.*.logs_spend P0 cells were only exercised by the deleted langfuse_otel e2e suite. Remove them so the coverage registry has no orphaned rows.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(e2e): log into the react admin ui in the management browser fixture (#33562)
The management ui_page fixture drove the old server-rendered login form: it clicked input[type="submit"] and treated wait_for_url("**/ui/**") as the done signal. /ui/ now serves the react (antd) dashboard whose submit is a <button type="submit">, so the click waited out the full 30s timeout and errored every browser test in the suite. wait_for_url also matched instantly because the login page already lives at /ui/, so on the fast path the fixture navigated before the auth cookie landed and got bounced back to login.
Click the antd submit button and wait for the token cookie loginCall sets on document.cookie, the real post-login signal.
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(e2e): make ui login readiness robust to httpOnly token cookies (#33564)
The login readiness check waited only on document.cookie including token=, which is empty when the token cookie is httpOnly. If the server ever sets it via a Set-Cookie header, the wait would spin to the 30s timeout and silently reproduce the original hang. Also accept the login form detaching (#username gone after the post-login redirect) so readiness holds regardless of how the cookie is delivered.
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
* test(e2e): datadog log delivery for successful chat, messages, and responses
Covers logging.datadog.success.exports_metric on all three routes: one
successful non-streaming call must reach the DataDog logs intake as exactly
one log event whose StandardLoggingPayload message carries the model group,
real token counts, and a response cost equal to the x-litellm-response-cost
header of the same response. Delivery is judged at the intake: the compose
stack gains a dd-sink service recording every batch the datadog callback
ships via the DD_BASE_URL testing override, and a typed reader replays it.
Writing these caught a live product bug: /v1/messages double-logs every
success (two byte-identical events per call), filed as LIT-4447; the messages
test tolerates byte-identical duplicates of the one event until it lands,
while a second differing event still fails
* test(e2e): address review findings on the datadog delivery suite
Consolidates the fresh-key first_ok helper into logging_client now that the
otel PR it mirrored has merged (both test files use the shared copy), moves
intake batch parsing into a helper so no path can leave the batch unbound,
and gives the sink's /health endpoint a truthful text/plain content type
* test(e2e): tolerate same-logical-event duplicates by call id, not byte identity
A clean LIT-4447 repro showed the duplicated payload is built twice and can
mint a fresh synthetic completion id per emission, arriving as two separate
intake POSTs with the same litellm_call_id and identical substantive fields.
Byte-identity was therefore a flaky criterion; duplicates now qualify only
when they share the call id, call type, model group, tokens, and cost, and a
second differing event still fails
* test(e2e): assert the scenario strictly; the messages test is the LIT-4447 regression pin
Per review direction the tests now assert exactly what the scenario promises:
exactly one DataDog log event per successful call, on every route. The
/v1/messages test therefore fails on current code against the known
double-log (LIT-4447) and is its regression pin; it goes green when the fix
lands. The duplicate-tolerance machinery is removed
* Simplify docstrings for DataDog log tests
Removed redundant phrasing about cost cross-checking in docstrings.
* Update test_datadog_log_e2e.py
* test(e2e): OTEL trace completeness on /v1/messages
Extends the LIT-3787 trace-completeness suite to the Anthropic-native route:
one successful non-streaming /v1/messages call must land at the destination as
ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT
span, no dangling parents). Adds the raw /v1/messages sender to the logging
suite client.
* test(e2e): reuse the shared AnthropicMessagesBody per review
Drops the duplicate /v1/messages request model in favor of the one models.py
already provides (budget_client uses the same one), passes max_tokens at the
call site to match the sibling chat test, notes in the docstring why the
gen-AI span is named chat on this surface, and adopts the hardened read-back
signature
* test(e2e): author the messages trace test docstring
* test(e2e): declare the messages surface on the covers marker
* test(e2e): otel trace completeness on /v1/responses (#33134)
* test(e2e): OTEL trace completeness on /v1/responses
Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API
route: one successful non-streaming /v1/responses call must land at the
destination as ONE connected trace. Adds the raw /v1/responses sender, a
CHEAP_OPENAI_MODEL config constant, and registers responses in the otel
registry cell's exercised_on.
* test(e2e): author the responses trace test docstring
* test(e2e): declare the responses and chat surfaces on the covers markers
Introduce the e2e coverage denominator: 282 behavior cells across the six
tracking modules (LLMs, MCPs, Management/UI, Reliability & Performance,
Logging & Guardrails, Other), one validated YAML row each, plus a collector
that diffs the registry against @pytest.mark.covers markers and reports
coverage per module.
The registry rows validate against a pydantic discriminated union so a row
cannot carry a field from another module. The collector is static: a
collect-only pass reads the markers, so it runs no test and needs no live
proxy. Register the covers marker suite-wide so that pass works under
--strict-markers.
This is a draft for review. Tiers are proposed rather than signed off, and a
few cells still need a support check or a prune.
