* test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata
* test(e2e): tag guardrails and logging tests with Subject metadata and record client steps
* test(e2e): leave the guardrails and logging harness unit tests untagged
* test(e2e): let the inner create_model step name the guardrail backend deployment
* docs(e2e): name every markerless harness test file that carries no Subject
* test(e2e): keep the step discovery comprehensions to one for clause
* test(e2e): declare the default guardrail backend model on the tests that drive it
* update logic that marks a logging callback as complete
* test(logging): cover streaming failure dedupe in mark_logging_complete
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): cover streaming failure dedupe in S3 and DataDog
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): keep has_run_logging as a deprecated alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): audit streaming failure dedupe across surfaces, fallbacks and sink outage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): assert anthropic upstream path in streaming failure audit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): count only provider posts in streaming audit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): assert the sink outage rejects uploads in burst audit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(logging): configure datadog retries with router override
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: add logging e2e coverage (s3_v2, gcs_bucket, team langfuse callback, datadog failure)
Five new live e2e scenarios raising Logging & Guardrails registry coverage:
s3_v2 success and failure objects read back from the real S3 bucket,
gcs_bucket success record read back through the GCS JSON API (with
nextPageToken pagination and per-request bearer minting), team-scoped
Langfuse callback delivery with non-team isolation, and DataDog failure
event delivery queried by indexed model_group. datadog_reader gains
query-based variants of the marker search; the langfuse cell is a new
registry row. Bucket readers settle past a full flush interval so a
late duplicate cannot hide from the exactly-one assertions
* test: cover clock-skew day prefix in gcs read-back and retry team callback propagation
* test: key the s3 failure read-back on the provider error, not payload absence
* chore: rerun ci
* chore: rerun ci after config sync
* chore: rerun ci with pr lane env
* chore: rerun ci
* chore: rerun ci
* chore: rerun ci
* chore: rerun ci
* chore: rerun ci
* test: add guardrail e2e coverage (presidio masking, bedrock post and during call, moderation on messages) (#38553)
* test: add guardrail e2e coverage (presidio masking, bedrock post/during, moderation on messages)
* test: require the phone placeholder positively in the presidio masking predicate
* test: count only the 400 verdict body as a bedrock post_call block
* test(e2e): exempt the guardrail config echo from the post_call leak assertion
* test(e2e): pin the fail-closed contract for an unknown guardrail name (skipped, product gap)
* test(e2e): tolerate the readiness 503 from a transient db blip in the callback-config probes
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is
not a gateway server, it is the client every suite uses to talk to the proxy
(keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the
module to proxy_client.py and the class to ProxyClient, with build_gateway
becoming build_proxy_client and the GatewayProvider protocol becoming
ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only
identifiers changed; prose and string literals that use the word gateway for the
proxy-server concept were left alone.
Each suite previously built its own instance through a per-suite build_client()
that called build_gateway() inside, duplicating the proxy wiring across suites.
There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every
suite's client fixture depends on it and injects it, so the wiring lives in one
place. claude_code keeps building its own client directly since it has its own
harness and does not use the shared fixtures.
Behavior is unchanged: shared transport, data-plane/control-plane split routing,
poll budget, typed request/response models, and resource cleanup all go through
the same object.
* fix(e2e): make the datadog read-back find what DataDog actually indexes
Live verification of the merged #33604 against real DataDog (us5) exposed
three read-back defects that the local-sink tests could never see; all
three fixes are verified against the real API:
- Marker search: DataDog consumes the shipped JSON message into the
event's attributes and leaves the indexed message EMPTY, so the
full-text '"marker"' query matched nothing and every test failed with
zero events. The query is now '*:*marker*', which scans all attributes
(the marker sits in messages.content); verified to return exactly the
event for the call.
- Rate limit: the Logs Search API budget is 2 requests per 10s org-wide
(x-ratelimit-name logs_public_search_api). Polling at POLL_INTERVAL=5s
sat exactly at the limit and the reader hard-failed on the first 429.
Searches now pace at DD_SEARCH_INTERVAL (10s default) and a 429 backs
off and retries up to 5 times; only non-429 failures stay hard fails.
- Envelope status: DataDog re-derives the indexed event status from the
parsed payload's status attribute ('success') and normalizes it to its
OK severity, so the assertion expects 'ok', not the shipped 'info'.
Live run: chat_completions and responses pass every assertion including
the exact response-cost cross-check; messages red-pins the LIT-4447
duplicate for real (one call -> two sync-sweep copies + one async batch
copy, same request id, confirmed in proxy debug logs). The duplicate is
race-dependent, so the pin flickers until #33589 lands.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): datadog log delivery for streamed chat, messages, and responses
Rewritten from the dd-sink version (original #33566) to judge delivery on
what real DataDog ingested, matching the merged #33604 conversion: the
dd_logs reader searches events back through the Logs Search API and the
assertions validate the indexed envelope (source:litellm tag, ok status)
and the StandardLoggingPayload fields under the event's attributes.
Each streamed test drives one STREAMED call per route, asserts the stream
actually streamed (event-stream content type, >0 chunks, no upstream error
event), then pins exactly one DataDog event whose payload records
stream=true, the aggregated token count, and a response_cost equal to the
/spend/logs row for the call - a stream's headers ship before its cost
exists, so the spend row is the cross-check anchor, and the spend row and
DataDog event must also agree on total_tokens.
Coverage registry: adds logging.datadog.stream.exports_metric exercised on
chat_completions, messages, and responses.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update test_datadog_log_e2e.py
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): read datadog log delivery back from the real datadog api (#33604)
* test(e2e): read datadog log delivery back from the real datadog api
* test(e2e): compare datadog-read cost with math.isclose, not bit-equality
The response_cost now round-trips through DataDog's attribute indexing
pipeline, whose float serialization is not guaranteed to preserve the
exact bit pattern the proxy shipped. rel_tol=1e-9 (equal to 9 significant
digits) still fails on any real cost discrepancy while tolerating
representation drift. Addresses the Greptile P2 on this PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): widen the duplicate-settle window to 30s for real DataDog
Against the local sink one poll interval (5s) after the first hit was
enough to catch a same-call duplicate, because both events arrived in the
same flush batch. Against real DataDog, ingestion jitter can make one
call's two events searchable tens of seconds apart, so a 5s settle could
let the LIT-4447 duplicate slip past the exactly-one assertion. The reader
now keeps re-reading for DD_SETTLE_SECONDS (default 30s, env-overridable
via E2E_DD_SETTLE_SECONDS) after the first event appears, returning early
only when a duplicate is already visible - more waiting cannot clear it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(e2e): point UI tests at dashboard service; register complexity router
Stage gateway 404s /ui; the Next.js dashboard is litellm-ui:3000. Drive
playwright against E2E_UI_BASE_URL and wait on login placeholders after
client render. Register complexity-smart-router via /model/new when the
proxy does not already list it so stage matches compose config
* docs(e2e): clarify E2E_UI_BASE_URL should be ALB when ingress splits UI
* docs(e2e): prefer single path-routing host for control plane and UI
CONTROL_PLANE and UI already default to PROXY_BASE_URL; clarify that
stage should set one ALB host rather than three endpoints
* fix(e2e): always capture complexity router model_id for teardown
Split /model/new from the data-plane wait so a propagation timeout still
deletes the control-plane registration (greptile orphan-model concern)
* fix(e2e): click exact Login button so SSO control is not matched
Playwright strict mode matched both Login and Login with SSO
* fix(router): score complexity by difficulty not request length
The LLM classifier prompt treated short wording as SIMPLE, so probes like
"Is P equal to NP?" stayed on the SIMPLE backend even though the classifier
ran. Judge intellectual difficulty so short hard questions route higher
* fix(e2e): open key edit via Key ID and wait for team models
Key Alias text is not the row open control on the virtual keys table;
KeyInfoView opens from the Key ID button in that row. Also wait for a
real team model in the edit Models dropdown so we do not race the async
availableModels fetch that only has All Team Models on first paint
* fix(e2e): keep settled DD events on empty search; bump mcp for OSV
Do not let a transient empty DataDog search wipe events already seen in
the settle window (Greptile P1). Make the logs-search from window
env-overridable via E2E_DD_SEARCH_FROM (Greptile P2). Prefer the mono
Key ID button when opening key edit. Bump mcp 1.26.0 -> 1.28.1 so OSV
clears the three high GHSA findings on the staging PR
* revert: drop mcp lock bump from e2e staging PR
OSV mcp upgrade is unrelated to the e2e fixes; leave the dep pin alone
---------
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(e2e): datadog log delivery for successful chat, messages, and responses
Covers logging.datadog.success.exports_metric on all three routes: one
successful non-streaming call must reach the DataDog logs intake as exactly
one log event whose StandardLoggingPayload message carries the model group,
real token counts, and a response cost equal to the x-litellm-response-cost
header of the same response. Delivery is judged at the intake: the compose
stack gains a dd-sink service recording every batch the datadog callback
ships via the DD_BASE_URL testing override, and a typed reader replays it.
Writing these caught a live product bug: /v1/messages double-logs every
success (two byte-identical events per call), filed as LIT-4447; the messages
test tolerates byte-identical duplicates of the one event until it lands,
while a second differing event still fails
* test(e2e): address review findings on the datadog delivery suite
Consolidates the fresh-key first_ok helper into logging_client now that the
otel PR it mirrored has merged (both test files use the shared copy), moves
intake batch parsing into a helper so no path can leave the batch unbound,
and gives the sink's /health endpoint a truthful text/plain content type
* test(e2e): tolerate same-logical-event duplicates by call id, not byte identity
A clean LIT-4447 repro showed the duplicated payload is built twice and can
mint a fresh synthetic completion id per emission, arriving as two separate
intake POSTs with the same litellm_call_id and identical substantive fields.
Byte-identity was therefore a flaky criterion; duplicates now qualify only
when they share the call id, call type, model group, tokens, and cost, and a
second differing event still fails
* test(e2e): assert the scenario strictly; the messages test is the LIT-4447 regression pin
Per review direction the tests now assert exactly what the scenario promises:
exactly one DataDog log event per successful call, on every route. The
/v1/messages test therefore fails on current code against the known
double-log (LIT-4447) and is its regression pin; it goes green when the fix
lands. The duplicate-tolerance machinery is removed
* Simplify docstrings for DataDog log tests
Removed redundant phrasing about cost cross-checking in docstrings.
* Update test_datadog_log_e2e.py
