mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
21 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
666648d58c | fix(otel): map /v1/messages provider errors before failure logging | ||
|
Yuneng Jiang
|
bf93483b95
|
test(e2e): assert provider error shape instead of pinned prose
Both providers reworded the error strings these two cells pinned, so the suite went red without any behavior changing. Anthropic's auth error is now "API key is invalid." rather than "invalid x-api-key", and OpenAI rejects an empty upload with "This model does not support the format you provided.", which names neither "file" nor "audio". Assert the durable shape instead. The otel cell pins the machine-readable authentication_error type plus a non-empty message, and the embedded JSON still has to parse, which is what proves the attribute survived untruncated. The transcription cell pins that the 400 relays the provider's own rejection and is typed as a client input error, so a regression that swallows the provider reason or returns a 500 still fails. |
||
|
mubashir1osmani
|
4725cb4661
|
test(e2e): cover google-native generateContent framing and prometheus queue time (#34650)
* test(e2e): cover google-native generateContent framing and prometheus queue time
Adds live coverage for three shipped regressions that had none, all reached
through surfaces a customer drives from Google SDKs and operator dashboards.
The managed google-native route (`/v1beta/models/{model}:generateContent`) had
no harness support at all, so EndpointsClient gains generate_content and
stream_generate_content plus the request body models, and a new suite asserts
the two contracts that broke there: the response carries
x-litellm-response-cost so SDK traffic reconciles against spend (LIT-4076), and
the stream relays single-prefixed SSE frames with no OpenAI [DONE] terminator.
A doubled `data:` prefix, a leaked bytes literal, or the [DONE] sentinel each
fail the stream test; [DONE] absence is only asserted once real content has
arrived, because a first-chunk upstream error legitimately falls back to the
OpenAI error shape and does emit it.
The prometheus test pins litellm_request_queue_time_seconds to an actual
observation on our own key's series rather than to the family merely existing,
which is the distinction the original regression turned on: the histogram stayed
registered while nothing was ever written to it (LIT-2034).
Each assertion was mutation-checked against the live proxy; inverting the
[DONE] expectation, the cost-header expectation, or the metric name fails the
corresponding test.
* refactor(e2e): simplify google native coverage
|
||
|
Yassin Kortam
|
a0d499e131
|
fix(e2e): assert on the gen-AI span that served the stream, not the span count (#36582)
The otel trace tests asserted that a streamed call produces exactly one gen-AI span. The proxy opens one gen-AI span per upstream attempt, so a call the router retried carries an error span for every failed attempt beside the one that answered, and the assertion fails on a request that succeeded. Select the served attempt instead: drop spans whose otel.status_code is ERROR, require exactly one survivor, and run the TTFT and streaming-flag assertions against it. That keeps what these assertions exist for, a split trace or a stream logged as two served spans, while tolerating a retry. Only the failed attempt lacks TTFT, so the old code also had a second failure mode: when the first span happened to be the error one, the test reported the attribute as missing rather than as belonging to a different attempt. test_span_selection.py covers the selection itself against Jaeger-shaped payloads and carries no e2e marker, since reproducing a first-attempt failure live is not something a test can arrange. |
||
|
Yuneng Jiang
|
09a98f5505
|
test(e2e): settle control-plane writes across every replica, not just one
The suite already waits for a new model or agent to become servable before handing it back, but that wait returns on the first successful read. Every request opens a fresh connection (e2e_http calls requests.* with no Session), so a load-balanced Service routes each one independently: one successful read proves one replica converged, and the caller's next request re-rolls and can land on a replica that has not reloaded yet. At replicaCount: 2 this surfaced as 30 failures on a SHA that is green at 1 replica -- 400 "Invalid model name passed", 404 "Guardrail not found", "no healthy deployments for this model", and a /model/info listing that contained one of two models created moments apart. Add PROPAGATION_TIMEOUT (default 15s, override E2E_PROPAGATION_TIMEOUT) and settle_propagation(), sized off the proxy's proxy_config_reload_interval_seconds (30s by default, 7s on the e2e stack) plus margin, and settle after every control-plane create whose object the suite then uses: - ProxyClient.create_model and A2AClient.register_agent, after their existing polls -- the poll still fails loudly if the object never appears at all - GuardrailsClient.register, which had no barrier; create_content_filter_guardrail and create_bedrock_guardrail now route through it instead of POSTing directly - the guardrail creates in mcp_client and logging_client - the vertex passthrough model, whose body cannot go through create_model Left alone: the /model/new calls that assert a 403 or read back a status code, since they never use the model. |
||
|
ryan-crabbe-berri
|
daf22ec871
|
test(e2e): make MCP and prometheus e2e tests robust to data-plane sync lag (#34854)
* test(e2e): harden harness and tests against data-plane pod churn A stage autoscaler scale-down produced a 2s window of ALB 502s that killed six budget tests on their first management call, and a freshly scaled-up pod that had not run its 30s DB object sync yet failed two MCP tests and one prometheus cardinality test. Retry transient gateway errors (502/503/504, connection errors) once at the shared e2e_http dispatch seam, poll MCP server registration to the poll deadline instead of asserting a single-shot listing, anchor the MCP guardrail full-sync wait to the later of the guardrail and server writes, and turn the prometheus alias poll into a drive-and-scrape convergence loop that re-sends traffic for missing aliases and unions results across scrapes * test(e2e): drain request body in retry stub handler so keep-alive reuse cannot misparse leftovers as requests * revert(e2e): drop the transient-502 retry seam A raw 502 during a pod scale-down is what a real client sees, so the suite retrying past it hides an availability gap instead of flagging it. The gateway-side fix is graceful drain on the deployment; until then the failures are signal * test(e2e): cap per-alias driver re-drives in the prometheus cardinality poll Bounds worst-case provider spend to 4 completions per alias while scrapes keep polling to the deadline; counters persist on whichever pod served them, so the cap costs no convergence unless that pod dies * test(e2e): drop driver re-drives from the prometheus cardinality poll The per-key cardinality contract is process-local and counters persist on whichever pod served the driver call, so unioning aliases across free scrape polls converges without re-sending billable traffic. The residual gap, a pod dying inside the poll window, is deferred to direct per-pod scraping |
||
|
mubashir1osmani
|
ac5b51253a
|
test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149)
* test(e2e): add other suite covering master-key auth and health lifecycle Covers the other.* holding-pen cells that were uncovered: master-key valid_allows/invalid_denied on the admin /user/list gate, and the lifecycle probes liveness.ping, readiness.public_probe, readiness.reports_db_status, and readiness_details.authenticated_diagnostics. New tests/e2e/other/ suite on the shared ProxyClient; the health probes send no auth header to prove the public routes need no credential, and the details route is asserted to reject an anonymous caller while exposing version/db diagnostics to the master key. * test(e2e): cover block_code_execution and openai_moderation guardrails Extends the guardrails suite with two built-in guardrails registered per request (default_on=False, opted in via the chat body's guardrails selector) so neither intercepts unrelated traffic on the shared proxy. block_code_execution.pre_call.blocks: a python code block plus a run-this request is intercepted with the canned content-blocked message and the model never runs, while the same code block asked about with don't-run-it reaches the model. Verified live. openai_moderations.pre_call.blocks: a flagged prompt is rejected 400 naming the moderation policy while a benign prompt passes. The guardrail calls OpenAI's moderation API; verifying it needs an OpenAI key with moderation quota (this account currently 429s the moderation endpoint). Adds a shared create_backend_model helper and a generic register() plus per-request guardrails/max_tokens on the client so more built-ins can reuse the same path. * test(e2e): cover presidio PII masking (pre_call + post_call) Registers a presidio guardrail per request (default_on=False) with the analyzer/anonymizer bases supplied in the registration params, so the test controls its own dependency and needs no proxy restart. presidio.pre_call.masks: a repeat-verbatim request comes back with the <EMAIL_ADDRESS> placeholder and never the raw email, proving the prompt was anonymized before the model saw it. presidio.post_call.masks: with apply_to_output the model's own emitted email is masked on the way out, so the caller never receives the raw value. Both verified live against real presidio analyzer + anonymizer containers. logging_only is intentionally not covered: /spend/logs exposes no prompt messages to read back the masked log, and a logging_only run also masked the response, contradicting its contract; noted in the module docstring for a follow-up. * test(e2e): cover presidio logging_only masking via OTEL read-back Adds the third presidio cell, guardrail.presidio.logging_only.masks. The logging_only contract (mask what is logged, do not block) is verified by reading the request's gen-AI span back from the real OTEL destination: the span's gen_ai.input.messages attribute carries the <EMAIL_ADDRESS> placeholder, never the raw email, and the call itself is not blocked. Reads the trace via the shared OtelReader, promoted from logging/ to the suite root so both suites use it. The masked prompt is polled to a deadline because logging_only masks the payload asynchronously and the span can briefly export before the mask lands. Drops the throwaway chat_send in favor of the existing transport.send for the call-id capture. * fix(e2e): tolerate cross-pod guardrail sync delay in team-opt-out test Stage runs multiple gateway pods behind the shared key. POST /guardrails registers a new default-on guardrail in-process immediately only on the pod that served the create call; every other pod picks it up on its next periodic DB sync (proxy_server.py, every 30s), so the very next chat call can race a pod that has not synced yet. Poll to a 40s deadline instead of asserting on the first response, matching the existing pattern in test_budget_reset_advances_e2e.py. * test(e2e): cover a guardrail on the MCP tool-call path (content_filter pre_mcp_call) Adds guardrail.litellm_content_filter.pre_mcp_call.blocks: against the real Datadog MCP server, a content_filter guardrail configured mode=pre_mcp_call blocks a banned keyword in an MCP tool call's arguments with HTTP 400 attributed to the pre_mcp_call hook, and lets a clean argument reach the upstream server. The guardrail attaches with default_on because per-key/request guardrail selection is dropped from the synthetic MCP request the hook sees; the banned keyword is unique per run so default_on only intercepts this test's own call. mode must be pre_mcp_call - a pre_call config silently no-ops on tools/call because the event type is rewritten for call_mcp_tool. Drives the tool directly via /mcp-rest/tools/call for a deterministic check of the same pre_mcp_call enforcement the OpenAI-SDK chat path hits when a model invokes an MCP tool. * fix(e2e): mid-conversation messages test uses client.proxy not client.gateway EndpointsClient exposes .proxy after the Gateway->ProxyClient rename; the mid-conversation system test still referenced .gateway, which fails the e2e basedpyright gate. Aligns it with the rest of the harness. * test(e2e): address review on the guardrail coverage MCP tool-call guardrail: poll the banned call until the guardrail is enforced instead of asserting on the first call, so the control-plane -> data-plane guardrail sync cannot race the check into a false pass-through; add a repeat banned call after enforcement to guard against a partial-propagation state. OpenAI moderation: distinguish a moderation-endpoint 429 (rate limit / no moderation quota) from a guardrail failure, so an account-capability gap reads as such rather than as "did not block". Runs green with a moderation-capable key. * test(e2e): close partial-propagation false-pass in MCP guardrail block test The single post-block repeat call could be load-balanced back to the same already-synced data-plane pod, so the test could pass while another pod still lacked the guardrail and let the banned MCP call reach Datadog. Anchor a wait to the guardrail create time (every pod is guaranteed to have DB-synced only after a full ~30s sync interval), then require the banned call to stay blocked across several attempts; a pass-through after that window is a real leak, not a race. * test(e2e): drop xfail-style rate-limit branch from openai_moderation test OpenAI's /v1/moderations is free and returns 200 with the env key (verified directly), so the RateLimitedError branch mislabeled the failure: a 429 there is insufficient_quota (no account billing), not throttling. The branch also only printed a softer message before failing anyway, an xfail-in-disguise the e2e rules forbid. A 429 now falls through and fails loudly with the full result. |
||
|
mubashir1osmani
|
28f012bb52
|
test(true_rabbit): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping (#33843)
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly. * test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL, TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path, Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm chat, and Nova Sonic realtime. Registry cells updated for the new markers. * test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and Langfuse driver models now use Anthropic haiku so local runs stay green when Gemini daily quota is exhausted. * test(e2e): drop Langfuse spend suite; feature is being deprecated Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the langfuse-only conftest driver/credentials fixtures. * test(e2e): fold provider/batch feature tests into their endpoint suites Keep the e2e layout endpoint- and suite-scoped instead of one file per provider or feature Move the virtual-key auth case into access_control/test_access_control_e2e.py as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone test_virtual_key_auth_e2e.py Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py. The hosted_vllm batch case is skipped for now since it needs a live vLLM server (HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant Merge the cohere, gemini and hosted_vllm chat cases into llm_translation/test_chat_completions_regression_e2e.py so /chat/completions coverage lives in one endpoint file, and repoint the coverage_registry source fields to the new homes Move the shared CacheControl / TextBlock / RichMessage request blocks into the root models.py (re-exported from endpoints_client) so quota_management can use them without a cross-suite import, which also clears the basedpyright errors in test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed json.loads path * test(e2e): address review feedback and re-home virtual-key coverage Replace the tautological Bedrock assume-role batch id assertion (`startswith(...) or batch.id`, always true) with a managed-id shape check, since the unified target_model_names path re-encodes the id rather than returning a raw ARN Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no longer consume the key's sole request unit before batch create runs; the batch create then clears the generic per-request limiter and the batch limiter is what returns the "Batch rate limit exceeded" body the assertions check Set exercised_on to [] on the pass-through header test; it drives a pass-through endpoint, not /chat/completions Move the virtual-key valid_allows / invalid_denied cells from other.yaml to mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management, and point its covers marker at the new ids |
||
|
Yassin Kortam
|
08fa25042c
|
test(e2e): rename Gateway to ProxyClient and expose it as a session-scoped fixture (#33750)
The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is not a gateway server, it is the client every suite uses to talk to the proxy (keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the module to proxy_client.py and the class to ProxyClient, with build_gateway becoming build_proxy_client and the GatewayProvider protocol becoming ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only identifiers changed; prose and string literals that use the word gateway for the proxy-server concept were left alone. Each suite previously built its own instance through a per-suite build_client() that called build_gateway() inside, duplicating the proxy wiring across suites. There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every suite's client fixture depends on it and injects it, so the wiring lives in one place. claude_code keeps building its own client directly since it has its own harness and does not use the shared fixtures. Behavior is unchanged: shared transport, data-plane/control-plane split routing, poll budget, typed request/response models, and resource cleanup all go through the same object. |
||
|
yucheng-berri
|
0223383d94
|
test(e2e): datadog log delivery for streamed routes, read back from the real datadog api (#33566)
* fix(e2e): make the datadog read-back find what DataDog actually indexes Live verification of the merged #33604 against real DataDog (us5) exposed three read-back defects that the local-sink tests could never see; all three fixes are verified against the real API: - Marker search: DataDog consumes the shipped JSON message into the event's attributes and leaves the indexed message EMPTY, so the full-text '"marker"' query matched nothing and every test failed with zero events. The query is now '*:*marker*', which scans all attributes (the marker sits in messages.content); verified to return exactly the event for the call. - Rate limit: the Logs Search API budget is 2 requests per 10s org-wide (x-ratelimit-name logs_public_search_api). Polling at POLL_INTERVAL=5s sat exactly at the limit and the reader hard-failed on the first 429. Searches now pace at DD_SEARCH_INTERVAL (10s default) and a 429 backs off and retries up to 5 times; only non-429 failures stay hard fails. - Envelope status: DataDog re-derives the indexed event status from the parsed payload's status attribute ('success') and normalizes it to its OK severity, so the assertion expects 'ok', not the shipped 'info'. Live run: chat_completions and responses pass every assertion including the exact response-cost cross-check; messages red-pins the LIT-4447 duplicate for real (one call -> two sync-sweep copies + one async batch copy, same request id, confirmed in proxy debug logs). The duplicate is race-dependent, so the pin flickers until #33589 lands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): datadog log delivery for streamed chat, messages, and responses Rewritten from the dd-sink version (original #33566) to judge delivery on what real DataDog ingested, matching the merged #33604 conversion: the dd_logs reader searches events back through the Logs Search API and the assertions validate the indexed envelope (source:litellm tag, ok status) and the StandardLoggingPayload fields under the event's attributes. Each streamed test drives one STREAMED call per route, asserts the stream actually streamed (event-stream content type, >0 chunks, no upstream error event), then pins exactly one DataDog event whose payload records stream=true, the aggregated token count, and a response_cost equal to the /spend/logs row for the call - a stream's headers ship before its cost exists, so the spend row is the cross-check anchor, and the spend row and DataDog event must also agree on total_tokens. Coverage registry: adds logging.datadog.stream.exports_metric exercised on chat_completions, messages, and responses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update test_datadog_log_e2e.py --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
yucheng-berri
|
ac29a6a283
|
test(e2e): otel streaming spans record a real ttft below span duration (#33588) | ||
|
mubashir1osmani
|
224fe67f10
|
test: e2e staging leftovers (#33613)
* test(e2e): read datadog log delivery back from the real datadog api (#33604) * test(e2e): read datadog log delivery back from the real datadog api * test(e2e): compare datadog-read cost with math.isclose, not bit-equality The response_cost now round-trips through DataDog's attribute indexing pipeline, whose float serialization is not guaranteed to preserve the exact bit pattern the proxy shipped. rel_tol=1e-9 (equal to 9 significant digits) still fails on any real cost discrepancy while tolerating representation drift. Addresses the Greptile P2 on this PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): widen the duplicate-settle window to 30s for real DataDog Against the local sink one poll interval (5s) after the first hit was enough to catch a same-call duplicate, because both events arrived in the same flush batch. Against real DataDog, ingestion jitter can make one call's two events searchable tens of seconds apart, so a 5s settle could let the LIT-4447 duplicate slip past the exactly-one assertion. The reader now keeps re-reading for DD_SETTLE_SECONDS (default 30s, env-overridable via E2E_DD_SETTLE_SECONDS) after the first event appears, returning early only when a duplicate is already visible - more waiting cannot clear it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): point UI tests at dashboard service; register complexity router Stage gateway 404s /ui; the Next.js dashboard is litellm-ui:3000. Drive playwright against E2E_UI_BASE_URL and wait on login placeholders after client render. Register complexity-smart-router via /model/new when the proxy does not already list it so stage matches compose config * docs(e2e): clarify E2E_UI_BASE_URL should be ALB when ingress splits UI * docs(e2e): prefer single path-routing host for control plane and UI CONTROL_PLANE and UI already default to PROXY_BASE_URL; clarify that stage should set one ALB host rather than three endpoints * fix(e2e): always capture complexity router model_id for teardown Split /model/new from the data-plane wait so a propagation timeout still deletes the control-plane registration (greptile orphan-model concern) * fix(e2e): click exact Login button so SSO control is not matched Playwright strict mode matched both Login and Login with SSO * fix(router): score complexity by difficulty not request length The LLM classifier prompt treated short wording as SIMPLE, so probes like "Is P equal to NP?" stayed on the SIMPLE backend even though the classifier ran. Judge intellectual difficulty so short hard questions route higher * fix(e2e): open key edit via Key ID and wait for team models Key Alias text is not the row open control on the virtual keys table; KeyInfoView opens from the Key ID button in that row. Also wait for a real team model in the edit Models dropdown so we do not race the async availableModels fetch that only has All Team Models on first paint * fix(e2e): keep settled DD events on empty search; bump mcp for OSV Do not let a transient empty DataDog search wipe events already seen in the settle window (Greptile P1). Make the logs-search from window env-overridable via E2E_DD_SEARCH_FROM (Greptile P2). Prefer the mono Key ID button when opening key edit. Bump mcp 1.26.0 -> 1.28.1 so OSV clears the three high GHSA findings on the staging PR * revert: drop mcp lock bump from e2e staging PR OSV mcp upgrade is unrelated to the e2e fixes; leave the dep pin alone --------- Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
mubashir1osmani
|
ebdf0bbfd7
|
chore(e2e): establish litellm_e2e_staging integration line (#33502)
* chore(e2e): establish litellm_e2e_staging integration line Long-lived berri branch for e2e suite recovery work (LIT-4479 through LIT-4486) before merge to litellm_internal_staging * test(e2e): remove langfuse_otel logging e2e suite (#33558) * test(e2e): remove langfuse_otel logging e2e suite Removes the LIT-4483 dynamic per-team/key/org langfuse_otel logging e2e tests (tests/e2e/logging/test_langfuse_e2e.py, added in #32857). The shared logging_client harness and the langfuse coverage-registry cells are left in place; only the test module is removed. The otel and prometheus logging e2e suites are unaffected. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): drop orphaned langfuse coverage-registry cells The three logging.langfuse.*.logs_spend P0 cells were only exercised by the deleted langfuse_otel e2e suite. Remove them so the coverage registry has no orphaned rows. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(e2e): log into the react admin ui in the management browser fixture (#33562) The management ui_page fixture drove the old server-rendered login form: it clicked input[type="submit"] and treated wait_for_url("**/ui/**") as the done signal. /ui/ now serves the react (antd) dashboard whose submit is a <button type="submit">, so the click waited out the full 30s timeout and errored every browser test in the suite. wait_for_url also matched instantly because the login page already lives at /ui/, so on the fast path the fixture navigated before the auth cookie landed and got bounced back to login. Click the antd submit button and wait for the token cookie loginCall sets on document.cookie, the real post-login signal. Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(e2e): make ui login readiness robust to httpOnly token cookies (#33564) The login readiness check waited only on document.cookie including token=, which is empty when the token cookie is httpOnly. If the server ever sets it via a Set-Cookie header, the wait would spin to the 30s timeout and silently reproduce the original hang. Also accept the login form detaching (#username gone after the post-login redirect) so readiness holds regardless of how the cookie is delivered. Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Mubashir Osmani <mubashir@berri.ai> |
||
|
yucheng-berri
|
edc30ea515
|
test(e2e): datadog log delivery for successful chat, messages, and responses (LIT-4447) (#33415)
* test(e2e): datadog log delivery for successful chat, messages, and responses Covers logging.datadog.success.exports_metric on all three routes: one successful non-streaming call must reach the DataDog logs intake as exactly one log event whose StandardLoggingPayload message carries the model group, real token counts, and a response cost equal to the x-litellm-response-cost header of the same response. Delivery is judged at the intake: the compose stack gains a dd-sink service recording every batch the datadog callback ships via the DD_BASE_URL testing override, and a typed reader replays it. Writing these caught a live product bug: /v1/messages double-logs every success (two byte-identical events per call), filed as LIT-4447; the messages test tolerates byte-identical duplicates of the one event until it lands, while a second differing event still fails * test(e2e): address review findings on the datadog delivery suite Consolidates the fresh-key first_ok helper into logging_client now that the otel PR it mirrored has merged (both test files use the shared copy), moves intake batch parsing into a helper so no path can leave the batch unbound, and gives the sink's /health endpoint a truthful text/plain content type * test(e2e): tolerate same-logical-event duplicates by call id, not byte identity A clean LIT-4447 repro showed the duplicated payload is built twice and can mint a fresh synthetic completion id per emission, arriving as two separate intake POSTs with the same litellm_call_id and identical substantive fields. Byte-identity was therefore a flaky criterion; duplicates now qualify only when they share the call id, call type, model group, tokens, and cost, and a second differing event still fails * test(e2e): assert the scenario strictly; the messages test is the LIT-4447 regression pin Per review direction the tests now assert exactly what the scenario promises: exactly one DataDog log event per successful call, on every route. The /v1/messages test therefore fails on current code against the known double-log (LIT-4447) and is its regression pin; it goes green when the fix lands. The duplicate-tolerance machinery is removed * Simplify docstrings for DataDog log tests Removed redundant phrasing about cost cross-checking in docstrings. * Update test_datadog_log_e2e.py |
||
|
yucheng-berri
|
d6f498ff5c
|
test(e2e): failed request error span carries the full untruncated message and status (LIT-4179) (#33304)
* test(e2e): failed request error span carries the full untruncated message and status Covers logging.otel.failure.exports_metric on chat_completions: a request that fails at the provider (invalid upstream key deployment) must export one complete trace whose gen-AI span carries the LIT-4179 error contract, declared as one reviewable payload (EXPECTED_ERROR_SPAN_ATTRIBUTES) plus an untruncated error.message proven by parsing the embedded provider error JSON back out of the attribute. The root SERVER span must record the 401 the client received. Adds STORE_MODEL_IN_DB to the compose stack so /model/new works locally, which the suite's model-registering tests already assume * test(e2e): clean failure diagnostics on the error-span contract per review A truncated error.message with missing braces now fails with a readable assertion instead of an unhandled ValueError, an unparseable embedded JSON fails via pytest.fail with the truncation context, and the retry loop now asserts the upstream provider failure was actually observed so a fresh-key propagation deadline cannot masquerade as a trace-export failure * test(e2e): pin the full error attribute set including the litellm.provider.error keys The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*, which the initial contract missed. The payload now pins error, error.type, otel.status_code, litellm.provider.error.code=401, and litellm.provider.error.llm_provider=anthropic exactly, plus non-empty litellm.provider.error.stack_trace and the untruncated error.message * test(e2e): author the error-span test docstring |
||
|
yucheng-berri
|
817582e697
|
test(e2e): otel trace completeness on streaming chat, messages, and responses (LIT-3787) (#33234) | ||
|
yucheng-berri
|
6a213de9f4
|
test(e2e): otel trace completeness on /v1/messages (#33133)
* test(e2e): OTEL trace completeness on /v1/messages Extends the LIT-3787 trace-completeness suite to the Anthropic-native route: one successful non-streaming /v1/messages call must land at the destination as ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT span, no dangling parents). Adds the raw /v1/messages sender to the logging suite client. * test(e2e): reuse the shared AnthropicMessagesBody per review Drops the duplicate /v1/messages request model in favor of the one models.py already provides (budget_client uses the same one), passes max_tokens at the call site to match the sibling chat test, notes in the docstring why the gen-AI span is named chat on this surface, and adopts the hardened read-back signature * test(e2e): author the messages trace test docstring * test(e2e): declare the messages surface on the covers marker * test(e2e): otel trace completeness on /v1/responses (#33134) * test(e2e): OTEL trace completeness on /v1/responses Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API route: one successful non-streaming /v1/responses call must land at the destination as ONE connected trace. Adds the raw /v1/responses sender, a CHEAP_OPENAI_MODEL config constant, and registers responses in the otel registry cell's exercised_on. * test(e2e): author the responses trace test docstring * test(e2e): declare the responses and chat surfaces on the covers markers |
||
|
yucheng-berri
|
948a43cd64
|
test(e2e): otel trace completeness on /chat/completions (#33132)
* test(e2e): OTEL trace completeness on /chat/completions against a local Jaeger destination Adds the logging-suite infrastructure for LIT-3787 trace-completeness coverage: a jaeger service in the compose stack as the OTEL v2 destination (arize_phoenix preset pointed at it via PHOENIX_COLLECTOR_HTTP_ENDPOINT, so gen-AI spans export through a preset-owned provider - the code path where trace splits happen), a typed Jaeger query read-back client, and the first test: one successful non-streaming /chat/completions call exports ONE complete trace (root SERVER span + auth/db/cost children + gen-AI CLIENT span, no dangling parents). * test(e2e): harden the otel trace read-back per review Jaeger reads now query server-side by the litellm.call_id span tag instead of paging recent traces and filtering client-side; the compose stack's background jobs alone can push a request trace past the page. A failed query hard-fails instead of reading as an empty result, the settle predicate now also waits for the prefix-matched db span the assertion demands, parent-chain walking follows CHILD_OF references only, the zero-trace and split-trace failures get distinct messages, jaeger gets a healthcheck so the depends_on condition is accurate, and the chat docstring names the route the code actually asserts * test(e2e): author the chat trace test docstring * Update logging section in CLAUDE.md Removed mention of OTEL trace-tree completeness from logging integration section. |
||
|
yucheng-berri
|
69c5839cc0
|
fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712)
* fix(guardrails): filter Add-Guardrail mode dropdown per provider The GET /guardrails/ui/add_guardrail_settings endpoint returned every GuardrailEventHooks value in one flat supported_modes list, so the Admin UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving Content Filter or Tool Permission with pre_mcp_call then failed with a 400 because those guardrails' server-side supported_event_hooks list excludes it. Expose each guardrail's supported hooks as a get_supported_event_hooks classmethod on CustomGuardrail (mirrors the existing get_config_model pattern) and have the endpoint iterate guardrail_class_registry to build a supported_modes_by_provider map. The UI Mode dropdown filters by that map when the selected provider is known and falls back to the global list otherwise. __init__ now sources its own supported_event_hooks list from the classmethod so the two sides can't drift. Also register BedrockGuardrail, ToolPermissionGuardrail, lakera, lakera_v2, and presidio in guardrail_class_registry so they participate in the map (they were previously only in guardrail_initializer_registry and had no class-registry entry). Behavior change: guardrails that previously had no supported_event_hooks declared (aim, javelin, azure/text_moderation, cato_networks, crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx, prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai, lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now validate the configured mode at instantiation. Existing configs where the mode was silently a no-op will fail at proxy startup with a clear validation error rather than running as a broken guardrail. Resolves LIT-4226 * fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form Address Greptile P1 (startup break) and P2 (edit form UX): LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported event_hook, unchanged behavior for the guardrails validated pre-PR). Setting it to false logs a warning and continues, giving deployments an opt-out while they fix configs that now surface as errors instead of silently no-op'ing. Regression test covers both modes. Edit form now surfaces the currently-saved mode even when it is not in the filtered per-provider list, so a legacy row (e.g. content_filter saved with pre_mcp_call before this fix) no longer disappears from the dropdown; the option renders with a 'not supported by <provider>' note so the user knows to pick another. * fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint Audited every get_supported_event_hooks classmethod against the hooks each guardrail's own tests exercise and its handler methods. Five were too narrow and their tests caught it in CI: rubrik gains pre_call, presidio gains during_call and pre_mcp_call, prompt_security, onyx and qualifire gain during_call. The remaining classes match either their original __init__ declarations or their exercised modes exactly. Cursor review fixes: the Add form now drops selected modes the new provider does not support when the user switches providers, so a pre_mcp_call selection cannot ride along into a provider that rejects it at save; the edit form handles list-shaped stored modes instead of treating mode as always a string. Extracted shared toModeArray and getSupportedModesForProvider helpers into guardrail_info_helpers so both forms use one implementation, typed the remaining any usages in both forms, removed nested ternaries, and committed the ratcheted-down eslint metrics and pruned suppressions |
||
|
mubashir1osmani
|
1bf98c0687
|
test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857)
* test(e2e): cover Langfuse logging.yaml P0 logs_spend cells Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat traffic and assert calculatedTotalCost matches StandardLogging response_cost and proxy spend. Also assert tool calls and applied guardrails land on the trace. Missing env or proxy is a hard failure, never a skip * test(e2e): use langfuse_otel callback for Langfuse spend coverage Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to Langfuse) instead of the classic langfuse SDK. Match generations named litellm_request by prompt marker and user_api_key_alias * test(e2e): require Langfuse spend assert; drop AGENTS.md Guardrail path no longer soft-gates logs_spend. Non-stream responses must return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md * test(e2e): fail when Langfuse spend is missing on guardrail path Always run logs_spend assertions for tool_permission; require positive x-litellm-response-cost on non-stream and positive /spend/logs spend * test(e2e): do not fall back to unmatched spend log rows poll_proxy_spend_for_key returns None when response_id or positive-spend filters match nothing, instead of silently using rows[0] |
||
|
mubashir1osmani
|
31c1ffc5a4
|
test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction (#32165)
* fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects spend_tracking/spend_e2e_client.py imported SpendTagsResponse and TagSpend from models, but neither was ever defined, so importing the client raised ImportError and pytest aborted collection for the whole e2e session. The tag-spend tests had never run. Model /spend/tags as it actually answers: a bare array of per-tag aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field that also wouldn't match the array shape; it now reads .root, matching how spend_logs consumes its RootModel. * test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction Adds regression nets and gap-surfacing tests: A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document that reasoning_effort='none' and thinking type='disabled' are silently dropped (LIT-3686 / GH #27453) A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py): parametrized regression net asserting real completion content, not just a 200, across the configured providers for /chat/completions and /responses (GH #28991) A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is honored and prompt-cache read tokens grow on a repeated cacheable prefix A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call rate limiter runs, then asserts no unattributed spend row is left behind by the internal input-file retrieval (LIT-3266) A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct key_alias and asserts each alias gets its own labeled series on /metrics A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict) regression proving eviction must not close an httpx client still held by an in-flight caller (LIT-3221 / GH #13034) Extends tests/e2e/models.py with the typed request and response fields these tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage fields, spend-log api_key) Co-authored-by: Cursor <cursoragent@cursor.com> * test(e2e): drop unused litellm-regression-tests submodule The e2e suite migrated the regression cases into this repo; nothing imports the submodule at runtime (only a provenance comment references it), so the .gitmodules entry and gitlink pointing at a personal repo would just make upstream CI init a submodule it never uses. Remove both to keep the change test-only. * test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread cleanup vs shared client teardown) and belongs in its own PR, not this e2e coverage PR, so revert the file to its base state. --------- Co-authored-by: Cursor <cursoragent@cursor.com> |