mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
25 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
b9e5eec28c | test(e2e): pin require_managed_files enforcement behind a marker-gated stack phase | ||
|
yuneng-jiang
|
7aef79b774
|
test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957)
* test(e2e): send no-cache on every cacheable request body, opt in only where a hit is the assertion
The e2e proxy runs with the response cache on, so any test that re-sends an
identical chat, messages, responses, completions, embeddings or rerank body
reads back a redis copy of an earlier call instead of reaching the provider.
Five tests in the last week failed that way. Default cache: {"no-cache": true}
on those request models and pass cache=None only in the two tests whose
assertion is the cache hit itself.
* test(e2e): give image edits and OCR a 180s client timeout
Both routes wait on providers that can legitimately take longer than the
60s transport-wide request timeout (gpt-image edits, Azure Document
Intelligence), and a client-side read timeout there fails a green request.
post/upload now accept a per-call timeout like get already does; only those
two call sites use it.
* test(e2e): rerun once on network errors and upstream 5xx only
Assertion failures still fail on the first attempt; only an outcome whose
error string carries the e2e_http network kind or a 5xx status gets one
more try. Test Engine records every attempt, so the flake rate stays
visible while a single provider blip no longer reds the rc run.
* test(e2e): let the reseed burst survive one upstream failure and print why
The burst is the precondition, not the property: one 5xx among six
concurrent calls still leaves five workers racing the cold counter, which
is what the reseed assertion measures. Two or more failures still abort,
and the failing bodies are now in the message instead of only the status
codes.
* test(e2e): keep polling Jaeger through a transient query failure
poll_traces_for_call already waits up to POLL_TIMEOUT for spans to land,
but a single refused connection to the query API failed the test on the
spot. Jaeger restarted twice during today's gate runs (19:05 and 19:41
UTC, each under a minute) and took ten and three otel tests with it while
the same tests passed on the rc build minutes later. A network failure
now counts as not-yet inside the same deadline; if Jaeger is still
unreachable when the deadline passes the test fails with that error, and
any non-network failure still fails immediately.
|
||
|
mateo-berri
|
601d6ff2c8 |
[e2e] Pin the OpenAI websocket passthrough prefixes
The websocket routes under /openai_passthrough and /openai had no e2e coverage, so nothing catches the regression from issue #36088, where both prefixes carried HTTP routes only and refused every upgrade with a 403 before a socket ever existed. Two tests cover it. The realtime one opens /openai_passthrough/v1/realtime and asserts OpenAI's own session.created frame comes back, which proves the route is registered and relayed upstream. The responses one asserts /openai/v1/responses accepts the upgrade, since a responses.connect socket waits for the client to speak first and has no opening frame to check. A refused upgrade is an HTTP response rather than a close frame, so both assert on the handshake. ws_base_url moves into e2e_config now that a second suite needs it |
||
|
mateo-berri
|
367dd537b9 |
feat(e2e): move record/replay to the provider edge (LIT-5745)
Replaces the test-side fixture transport with an in-process provider-edge HTTP server the proxy's deployments point their api_base at. Record forwards provider calls verbatim and writes them to the bundle; replay answers them from the bundle with zero provider calls while key auth, routing, cost calculation, and spend-log writes still execute against the live proxy and database. Drift comes back as HTTP 599 naming the computed and closest recorded keys. Request headers are never stored and responses are kept byte-identical between modes from the proxy's side of the socket. |
||
|
mateo-berri
|
2adf8aa581 |
feat(e2e): add record/replay transport seam and fixture bundle format
E2E_FIXTURE_MODE selects the transport every e2e client is built on: live (default, unchanged behavior), record (pass through to the live proxy while writing every interaction to a fixture bundle), or replay (serve every interaction from the bundle with no proxy and no provider spend). Both new transports fulfil the existing Transport protocol, so no test changes shape. A bundle is a directory with a manifest (record timestamp, harness version, format version) and one JSON file per interaction, grouped per test in call order. Replay against a manifest older than seven days hard-fails at collection time naming the bundle age. Record always wipes and never reads the previous bundle, refusing to wipe a directory that is not a bundle. Auth header values are redacted on write; uploads store a sha256 digest. unique_marker() becomes deterministic per test in record/replay modes so a replay run regenerates exactly the requests the record run sent. Content-based match keys, streaming chunk fidelity, and provider-scoping are follow-ups (LIT-5741, LIT-5742, LIT-5745). |
||
|
Yuneng Jiang
|
09a98f5505
|
test(e2e): settle control-plane writes across every replica, not just one
The suite already waits for a new model or agent to become servable before handing it back, but that wait returns on the first successful read. Every request opens a fresh connection (e2e_http calls requests.* with no Session), so a load-balanced Service routes each one independently: one successful read proves one replica converged, and the caller's next request re-rolls and can land on a replica that has not reloaded yet. At replicaCount: 2 this surfaced as 30 failures on a SHA that is green at 1 replica -- 400 "Invalid model name passed", 404 "Guardrail not found", "no healthy deployments for this model", and a /model/info listing that contained one of two models created moments apart. Add PROPAGATION_TIMEOUT (default 15s, override E2E_PROPAGATION_TIMEOUT) and settle_propagation(), sized off the proxy's proxy_config_reload_interval_seconds (30s by default, 7s on the e2e stack) plus margin, and settle after every control-plane create whose object the suite then uses: - ProxyClient.create_model and A2AClient.register_agent, after their existing polls -- the poll still fails loudly if the object never appears at all - GuardrailsClient.register, which had no barrier; create_content_filter_guardrail and create_bedrock_guardrail now route through it instead of POSTing directly - the guardrail creates in mcp_client and logging_client - the vertex passthrough model, whose body cannot go through create_model Left alone: the /model/new calls that assert a 403 or read back a status code, since they never use the model. |
||
|
Yassin Kortam
|
3d35eee560
|
test(e2e): derive the throughput SLO per replica and surface locust's error breakdown (#35494)
The floor was an absolute fleet number, so it asserted replicas x per-replica rate and went red on how many gateway pods happened to be warm rather than on the request path. The test now measures one replica first, with a short serial pass that only ever occupies a single pod, and requires the concurrent phase to reach at least that rate. A serial latency budget carries the request-path assertion the floor used to imply, and both hold at one replica or seven. Zero-error runs that "sustained 16.7 RPS" were queueing, not slow requests: the load model is a mock_response deployment with no upstream, a single-worker replica serves it in about 57ms, and 100 closed-loop users against 1/0.057 RPS of capacity sit at 6s each by Little's law. The runner also kept locust's --json summary and threw away everything else, so a run where 93% of requests failed said nothing about what they got. It now passes --csv, reads the failure breakdown back, and reports locust's own generator-saturation warnings, both folded into the assertion messages. Resolves LIT-5054 |
||
|
mubashir1osmani
|
64fc19d61a
|
fix(e2e): stop tests from breaking the shared proxy for every suite after them (#34664)
* fix(e2e): stop the cache-settings test from persisting a degraded Redis config TestCacheSettings.test_update_persists_cache_backend_to_get read the live cache settings and wrote them back, intending a no-op. Its capture modelled only type/host/port, so on a TLS cluster the write-back silently dropped `ssl` and `redis_startup_nodes`. That is not recoverable on its own. `/cache/settings` persists what it receives into LiteLLM_CacheConfig, that row outranks the YAML `cache_params`, and init_cache_settings_in_db re-applies it on a timer, so a restart does not clear it. The proxy ends up driving a TLS-only cluster endpoint as a plaintext standalone node and every Redis call blocks to socket timeout. On the affected deployment that took out rate limiting entirely (the v3 limiter is a Lua script on Redis with no DB fallback), Redis-only budget levels (tag, per-model, team-member, per-window), spend tracking, `ResetBudgetJob` (which self-starved at 54 skipped runs per 15 min), and `ProxyConfig.add_deployment`, whose last statement syncs guardrails and never ran. 60 of 72 failures in one run traced back here. The settings blob is now round-tripped verbatim via a RootModel over an exhaustive value union, so a subset cannot be written. Two guards make a regression fail loudly at this test instead of silently downstream: - refuse to write when GET reports redis_type=cluster but omits redis_startup_nodes, which is the exact precondition for persisting a downgrade. GET resolves the stored row overlaid with REDIS_* env and never reads YAML, so a cluster configured only in YAML cannot round-trip here - compare /cache/ping before and after, so a write that breaks connectivity fails this test rather than every suite that follows The underlying product defect is filed as LIT-4816: GET cannot express the effective config, and a partial POST is allowed to downgrade transport. This change only stops the suite from triggering it; the Admin UI can still do so. basedpyright clean (0 errors) under the e2e gate. * fix(e2e): scope the bedrock guardrail per request and send OpenAI's current token param Two failures that had nothing to do with the guardrail or route under test. create_bedrock_guardrail registered with default_on=True, which applies the guardrail to every request the proxy serves. The upstream ApplyGuardrail call was answering 403, and that came back to unrelated traffic as `403 Bedrock guardrail request failed`, failing three a2a tests and a passthrough headers test alongside the bedrock one. The harness already supports the per-request `guardrails` selector, so the guardrail is now registered opted out of default_on and selected by the test that wants it. A broken upstream guardrail fails its own test instead of whatever else is running. Note this only contains the blast radius; the 403 itself still needs the bedrock:ApplyGuardrail permission (or a valid guardrail identifier) on the deployment, so test_bedrock_pre_call_blocks_harmful_prompt can still fail on its own until that is sorted. The OpenAI passthrough body sent `max_tokens`, which newer models reject with "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead." Passthrough forwards the body untranslated, so drop_params does not apply and the body has to satisfy OpenAI's contract directly. vllm_chat keeps max_tokens, which vLLM accepts. basedpyright clean (0 errors) under the e2e gate. * fix(e2e): drop the pinned a2a api_key that broke every message/send #34512 pinned `api_key="os.environ/ANTHROPIC_API_KEY"` on the a2a bridge agent. The a2a bridge forwards the agent's litellm_params straight into litellm.acompletion() without expanding "os.environ/" indirection, so that literal string was sent upstream as x-api-key and every message/send failed with `AnthropicException - {"type":"authentication_error","message":"invalid x-api-key"}`. Omitting api_key restores the normal provider resolution: litellm reads ANTHROPIC_API_KEY from the proxy's own environment for this provider, which is what the agent-owner flow depends on and what the suite did before #34512. Verified against a live proxy, same agent shape each time: api_key omitted -> message/send 200 api_key "os.environ/ANTHROPIC_API_KEY" -> message/send 500 invalid x-api-key api_key <literal key> -> message/send 200 and the key itself is valid (direct call to api.anthropic.com returns 200), so this was indirection that never got expanded rather than a bad credential. This accounts for four failures (test_semver_protocol_version_registers_and_serves, test_message_send_runs_completion_bridge, test_pinned_v0_3_serves_flat_message_shape, test_pinned_v1_0_serves_nested_message_shape). They were previously reported as `403 Bedrock guardrail request failed`, because a default_on Bedrock guardrail short-circuited the request before it ever reached the bridge and hid this. The bridge silently ignoring "os.environ/" in agent params is a product defect in its own right, filed separately; anyone configuring an agent credential that way through the UI hits the same wall. basedpyright clean (0 errors) under the e2e gate. * test(e2e): make the load suite less aggressive against a shared proxy 750 users at spawn rate 50 saturated the request path hard enough to distort the latency-sensitive suites sharing the same proxy, and it spends real provider money at that rate. Drop to 200 users at spawn rate 20. The RPS floor moves with the user count rather than staying put, so the assertion keeps its meaning instead of becoming a formality: 355 RPS over 750 users is ~0.47 RPS/user, and 90 over 200 holds that same per-user expectation with a similar pass margin. A request-path regression still trips it. All four knobs stay env-overridable (E2E_LOAD_USERS, E2E_LOAD_SPAWN_RATE, E2E_LOAD_DURATION_SECONDS, E2E_LOAD_MIN_RPS) for a deliberate load run. Note the recorded failure for this test was "no requests completed in 60s", which was the gateway wedged on unreachable Redis rather than a throughput regression; this change is about not perturbing its neighbours, not about that failure. * fix(e2e): make the reasoning-tokens assertion exercise a request that reasons test_openai_chat_reasoning_reports_reasoning_tokens asked "A train travels 60 miles in 1.5 hours. What is its average speed in mph?" at reasoning_effort="low", then asserted reasoning_tokens > 0. The model answers that directly without reasoning, so 0 is correct behavior and the assertion was testing the model's discretion rather than litellm's reporting. Verified against a live proxy on a dedicated openai/gpt-5.6 deployment, matching how the test provisions its model: reasoning_effort=low, one-step arithmetic -> reasoning_tokens=0 reasoning_effort=high, the prompt used here -> reasoning_tokens=114 Raised to high effort with a prompt that requires a proof plus a search, so the field under test is actually populated and the assertion fails only if litellm stops surfacing it. While confirming this I also checked prompt caching, which needed no change: cached_tokens comes back 3615 of 3618 prompt tokens on a repeated large prefix against a dedicated deployment. An earlier reading of 0 was an artifact of probing a fan-out alias whose requests land on different deployments, not a caching defect. * test(e2e): skip the files-list test while LIT-4820 is open GET /v1/files does not include a just-uploaded file. The upload returns 200 and GET /v1/files/{id} resolves it, but the listing never contains it: the returned set stays fixed at 27 entries whose newest created_at is roughly ten hours older than the upload, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Polled for 40s, so not an eventual-consistency window. Filed as LIT-4820. Skipping keeps a known, ticketed product bug from holding the suite red and masking a new regression somewhere else in the same test. The assertion is left exactly as it was on purpose. It encodes the contract we actually want, that a file retrievable by id is also enumerable, and anything that lists files (a UI picker, cleanup tooling that lists then deletes and would therefore leak provider-side files) depends on it. Relaxing it to get green would delete the signal. The skip reason says so and links the ticket, and the ticket records that removing this marker is part of its definition of done. Matches the existing pattern in this file, where test_unified_file_and_batch_create skips with a reason citing LIT-3266. While skipped, the registry cell llm.files.openai.list.nonstream.works has no passing covering test, so files-list coverage reports as uncovered rather than passing, which is the honest state. * fix(e2e): parse Sentinel node lists in the cache-settings model The value union covered scalar lists and lists of mappings, but not lists of lists. `redis_startup_nodes` holds host/port mappings while `sentinel_nodes` holds positional pairs (CACHE_SETTINGS_FIELDS documents `[['localhost', 26379]]`), so on a Sentinel deployment pydantic rejected the response: sentinel_nodes.list[dict[str,...]].1 Input should be a valid dictionary [input_value=['localhost', 26380]] The round-trip test reads GET /cache/settings before it writes anything, so that rejection failed the test at the read, before any assertion ran. A Sentinel deployment would have looked like a broken cache-settings route rather than a model too narrow to parse a documented shape. A list element may now be a scalar, a list or a mapping, which covers both node shapes without special-casing either and tolerates a heterogeneous list instead of rejecting the whole response. Adds TestCacheSettingsModel, harness-level with no `e2e` marker so it runs without a proxy, covering all four backend shapes (cluster mappings, sentinel pairs, plain node, url mode with a null discrete field) plus transport() key selection. Confirmed it fails on the previous union and passes on this one: old union -> 1 failed, 4 passed (the sentinel case) new union -> 5 passed * test(e2e): remove the cache-settings round-trip test The test could not fail for the thing it claimed to test, and could break the deployment it ran against. Both halves of that are worth stating. It read the live settings, wrote back identical values, and asserted the read-back matched. If POST /cache/settings were a complete no-op that returned 200 and touched nothing, GET would still return the values read a moment earlier and the test would pass. It verified that GET is stable, not that the route persists anything. Against that, /cache/settings persists what it receives into LiteLLM_CacheConfig, that row outranks YAML cache_params, and init_cache_settings_in_db re-applies it on a timer. A write that omits ssl or redis_startup_nodes converts a TLS cluster into a plaintext standalone client and every later Redis call blocks to socket timeout. On 2026-07-25 that failed 60 of 72 tests in one run: rate limiting stopped enforcing, Redis-only budgets admitted billable over-budget spend, ResetBudgetJob self-starved, and guardrail sync never ran. Guarding the previous shape was not sufficient. Writing the blob verbatim plus a cluster precondition and a /cache/ping check narrowed the hazard but did not remove it, because GET cannot express the effective config: it resolves the stored row overlaid with REDIS_* env and never reads YAML. On a fresh deploy it cannot see YAML's ssl to echo back, so a TLS non-cluster deployment could still have a row written that drops it. No round-trip through this route is safe on a shared proxy. Removed with the models and helpers it owned, and TestCacheSettingsModel with them since it existed only to protect that parsing. The registry row mgmt.cache_settings.update.happy_path stays, now carrying the rationale for why it is deliberately uncovered and what a safe test would require (an isolated proxy, or LIT-4816 fixed so a partial write cannot downgrade transport). Coverage therefore reports this cell as a gap, which is the honest state. Collector passes --strict; the module still collects 11 tests. |
||
|
mubashir1osmani
|
b5bc3631e1
|
refactor(e2e): drop require_env, read os.environ where a cred is used (#34413)
require_env hard-failed a test (and, for the shared litellm-ops secret, drove piling every provider credential into one blob) whenever an optional cred was absent. Most call sites either read a value the test actually uses or just gated on the runner's env for a key the gateway consumes. Read os.environ directly where the test uses the value; drop the presence-only gates so those cases run against the proxy instead of pre-failing on the runner's environment. Removes the require_env helper from e2e_config. |
||
|
Tin Chi Lo
|
3cdd6ab9a1 | test(e2e): drive a real Linear OAuth MCP through chat completions under both ingress headers | ||
|
Yassin Kortam
|
3f98f62748
|
test(e2e): fail the run when a Rust gateway silently serves /messages through Python (#34208)
A gateway whose native extension is unavailable falls back to the Python implementation without raising, so it answers /v1/messages normally and the only difference on the wire is the absent x-litellm-rust header. Nothing in the suite read that header, so a Rust deployment that had stopped running Rust produced a fully green e2e run. Assert the marker on the streamed Messages assertions when E2E_EXPECT_RUST is set. It stays opt-in because the same suite image also runs against the standard gateway, which has no Rust path and must keep passing; the two deployments are already separate Applications, so this is one value on the Rust instance rather than branching inside the tests. |
||
|
mateo-berri
|
1255094de3 | fix(e2e): retry transient turn failures and move the weekly anomaly run to Saturday before the stable release cut | ||
|
mateo-berri
|
48295df0e9 | test(e2e): raise warm cache read floor to 0.65 from measured healthy and regression baselines | ||
|
mateo-berri
|
1692170264 | fix(e2e): count aborted-session turns as failures and require a spend stability window | ||
|
mateo-berri
|
b572cb80d1 | test(e2e): add weekly session-anomaly load test against real providers | ||
|
mubashir1osmani
|
28f012bb52
|
test(true_rabbit): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping (#33843)
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly. * test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL, TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path, Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm chat, and Nova Sonic realtime. Registry cells updated for the new markers. * test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and Langfuse driver models now use Anthropic haiku so local runs stay green when Gemini daily quota is exhausted. * test(e2e): drop Langfuse spend suite; feature is being deprecated Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the langfuse-only conftest driver/credentials fixtures. * test(e2e): fold provider/batch feature tests into their endpoint suites Keep the e2e layout endpoint- and suite-scoped instead of one file per provider or feature Move the virtual-key auth case into access_control/test_access_control_e2e.py as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone test_virtual_key_auth_e2e.py Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py. The hosted_vllm batch case is skipped for now since it needs a live vLLM server (HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant Merge the cohere, gemini and hosted_vllm chat cases into llm_translation/test_chat_completions_regression_e2e.py so /chat/completions coverage lives in one endpoint file, and repoint the coverage_registry source fields to the new homes Move the shared CacheControl / TextBlock / RichMessage request blocks into the root models.py (re-exported from endpoints_client) so quota_management can use them without a cross-suite import, which also clears the basedpyright errors in test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed json.loads path * test(e2e): address review feedback and re-home virtual-key coverage Replace the tautological Bedrock assume-role batch id assertion (`startswith(...) or batch.id`, always true) with a managed-id shape check, since the unified target_model_names path re-encodes the id rather than returning a raw ARN Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no longer consume the key's sole request unit before batch create runs; the batch create then clears the generic per-request limiter and the batch limiter is what returns the "Batch rate limit exceeded" body the assertions check Set exercised_on to [] on the pass-through header test; it drives a pass-through endpoint, not /chat/completions Move the virtual-key valid_allows / invalid_denied cells from other.yaml to mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management, and point its covers marker at the new ids |
||
|
mubashir1osmani
|
fdf380d0e3
|
test(e2e): harden stage flakes for batches, UI, and MCP (#33831)
* test(e2e): harden stage flakes for batches, UI, and MCP Unique batch model names avoid load-balancing onto stale azure-batch deployments that still pointed at the retired gpt-4.1-mini-batch, which only the managed/unified path was hitting. Retry batch retrieve on 500 and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite when the compose-only mcp-upstream is unreachable on stage k8s * test(e2e): cover Datadog remote MCP via search_datadog_logs Register the regional Datadog MCP endpoint with DD-API-KEY / DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*, assert the proxy shipped it, list tools, call search_datadog_logs for the marker, and delete the server on teardown. Math-upstream key-access tests only skip when that compose service is unreachable * test(e2e): drop compose math MCP upstream; use Datadog only Key-access denial and happy-path MCP e2e both register the real regional Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers. Remove the mcp-upstream compose service and FastMCP add/multiply fixture * docs(e2e): require real Datadog MCP for all mcp suite tests Document that tests/e2e/mcp must register via datadog_mcp helpers against mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams * chore: restore mcp_e2e_upstream_server.py Keep the FastMCP fixture file; e2e no longer wires it in compose, but the module itself is not part of the Datadog-only cleanup * fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load pytest on the host never inherited compose env_file keys, so DD_API_KEY stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the dynamically loaded datadog_reader module in sys.modules so dataclasses do not crash under Python 3.12 * test(e2e/batches): harden azure/vertex unified lifecycle flakes Put the provider deployment name in every JSONL body so Azure does not depend on a perfect model rewrite. Retry create/retrieve/cancel on transient statuses with backoff. Drop cancel assertions for azure and vertex (registry only has a shared basic cell; create+retrieve prove routing, cancel stays best-effort cleanup) * test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED Post-login client redirects abort the first /ui/api-keys/ goto on stage. Wait off /ui/login after cookie set, then accept the page once Create New Key is visible even if goto raised ERR_ABORTED * test(e2e): drop flaky key models dropdown Playwright suite API management e2e already covers key generate/update persistence. The UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and no unique product signal. Remove the suite and unused browser fixtures |
||
|
Yassin Kortam
|
a2614b1239
|
test(e2e): add Locust throughput load test that runs last (#33748)
CodSpeed benchmarks the SDK with no IO, so it can't catch regressions that only appear under real concurrent load through the full proxy stack (auth, routing, logging, spend, Postgres, Redis). This adds a Locust load test under tests/e2e/load that drives concurrent POST /chat/completions traffic against a mock deployment (litellm_params.mock_response), so the measured throughput reflects proxy overhead rather than a provider's latency, and asserts an aggregate RPS SLO with a failure-ratio guard. The test is marked load and the parent conftest sorts load-marked items last so it never perturbs latency-sensitive suites. Covers reliability.perf.throughput.under_slo. |
||
|
yucheng-berri
|
0223383d94
|
test(e2e): datadog log delivery for streamed routes, read back from the real datadog api (#33566)
* fix(e2e): make the datadog read-back find what DataDog actually indexes Live verification of the merged #33604 against real DataDog (us5) exposed three read-back defects that the local-sink tests could never see; all three fixes are verified against the real API: - Marker search: DataDog consumes the shipped JSON message into the event's attributes and leaves the indexed message EMPTY, so the full-text '"marker"' query matched nothing and every test failed with zero events. The query is now '*:*marker*', which scans all attributes (the marker sits in messages.content); verified to return exactly the event for the call. - Rate limit: the Logs Search API budget is 2 requests per 10s org-wide (x-ratelimit-name logs_public_search_api). Polling at POLL_INTERVAL=5s sat exactly at the limit and the reader hard-failed on the first 429. Searches now pace at DD_SEARCH_INTERVAL (10s default) and a 429 backs off and retries up to 5 times; only non-429 failures stay hard fails. - Envelope status: DataDog re-derives the indexed event status from the parsed payload's status attribute ('success') and normalizes it to its OK severity, so the assertion expects 'ok', not the shipped 'info'. Live run: chat_completions and responses pass every assertion including the exact response-cost cross-check; messages red-pins the LIT-4447 duplicate for real (one call -> two sync-sweep copies + one async batch copy, same request id, confirmed in proxy debug logs). The duplicate is race-dependent, so the pin flickers until #33589 lands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): datadog log delivery for streamed chat, messages, and responses Rewritten from the dd-sink version (original #33566) to judge delivery on what real DataDog ingested, matching the merged #33604 conversion: the dd_logs reader searches events back through the Logs Search API and the assertions validate the indexed envelope (source:litellm tag, ok status) and the StandardLoggingPayload fields under the event's attributes. Each streamed test drives one STREAMED call per route, asserts the stream actually streamed (event-stream content type, >0 chunks, no upstream error event), then pins exactly one DataDog event whose payload records stream=true, the aggregated token count, and a response_cost equal to the /spend/logs row for the call - a stream's headers ship before its cost exists, so the spend row is the cross-check anchor, and the spend row and DataDog event must also agree on total_tokens. Coverage registry: adds logging.datadog.stream.exports_metric exercised on chat_completions, messages, and responses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update test_datadog_log_e2e.py --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
mubashir1osmani
|
224fe67f10
|
test: e2e staging leftovers (#33613)
* test(e2e): read datadog log delivery back from the real datadog api (#33604) * test(e2e): read datadog log delivery back from the real datadog api * test(e2e): compare datadog-read cost with math.isclose, not bit-equality The response_cost now round-trips through DataDog's attribute indexing pipeline, whose float serialization is not guaranteed to preserve the exact bit pattern the proxy shipped. rel_tol=1e-9 (equal to 9 significant digits) still fails on any real cost discrepancy while tolerating representation drift. Addresses the Greptile P2 on this PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): widen the duplicate-settle window to 30s for real DataDog Against the local sink one poll interval (5s) after the first hit was enough to catch a same-call duplicate, because both events arrived in the same flush batch. Against real DataDog, ingestion jitter can make one call's two events searchable tens of seconds apart, so a 5s settle could let the LIT-4447 duplicate slip past the exactly-one assertion. The reader now keeps re-reading for DD_SETTLE_SECONDS (default 30s, env-overridable via E2E_DD_SETTLE_SECONDS) after the first event appears, returning early only when a duplicate is already visible - more waiting cannot clear it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): point UI tests at dashboard service; register complexity router Stage gateway 404s /ui; the Next.js dashboard is litellm-ui:3000. Drive playwright against E2E_UI_BASE_URL and wait on login placeholders after client render. Register complexity-smart-router via /model/new when the proxy does not already list it so stage matches compose config * docs(e2e): clarify E2E_UI_BASE_URL should be ALB when ingress splits UI * docs(e2e): prefer single path-routing host for control plane and UI CONTROL_PLANE and UI already default to PROXY_BASE_URL; clarify that stage should set one ALB host rather than three endpoints * fix(e2e): always capture complexity router model_id for teardown Split /model/new from the data-plane wait so a propagation timeout still deletes the control-plane registration (greptile orphan-model concern) * fix(e2e): click exact Login button so SSO control is not matched Playwright strict mode matched both Login and Login with SSO * fix(router): score complexity by difficulty not request length The LLM classifier prompt treated short wording as SIMPLE, so probes like "Is P equal to NP?" stayed on the SIMPLE backend even though the classifier ran. Judge intellectual difficulty so short hard questions route higher * fix(e2e): open key edit via Key ID and wait for team models Key Alias text is not the row open control on the virtual keys table; KeyInfoView opens from the Key ID button in that row. Also wait for a real team model in the edit Models dropdown so we do not race the async availableModels fetch that only has All Team Models on first paint * fix(e2e): keep settled DD events on empty search; bump mcp for OSV Do not let a transient empty DataDog search wipe events already seen in the settle window (Greptile P1). Make the logs-search from window env-overridable via E2E_DD_SEARCH_FROM (Greptile P2). Prefer the mono Key ID button when opening key edit. Bump mcp 1.26.0 -> 1.28.1 so OSV clears the three high GHSA findings on the staging PR * revert: drop mcp lock bump from e2e staging PR OSV mcp upgrade is unrelated to the e2e fixes; leave the dep pin alone --------- Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
yucheng-berri
|
edc30ea515
|
test(e2e): datadog log delivery for successful chat, messages, and responses (LIT-4447) (#33415)
* test(e2e): datadog log delivery for successful chat, messages, and responses Covers logging.datadog.success.exports_metric on all three routes: one successful non-streaming call must reach the DataDog logs intake as exactly one log event whose StandardLoggingPayload message carries the model group, real token counts, and a response cost equal to the x-litellm-response-cost header of the same response. Delivery is judged at the intake: the compose stack gains a dd-sink service recording every batch the datadog callback ships via the DD_BASE_URL testing override, and a typed reader replays it. Writing these caught a live product bug: /v1/messages double-logs every success (two byte-identical events per call), filed as LIT-4447; the messages test tolerates byte-identical duplicates of the one event until it lands, while a second differing event still fails * test(e2e): address review findings on the datadog delivery suite Consolidates the fresh-key first_ok helper into logging_client now that the otel PR it mirrored has merged (both test files use the shared copy), moves intake batch parsing into a helper so no path can leave the batch unbound, and gives the sink's /health endpoint a truthful text/plain content type * test(e2e): tolerate same-logical-event duplicates by call id, not byte identity A clean LIT-4447 repro showed the duplicated payload is built twice and can mint a fresh synthetic completion id per emission, arriving as two separate intake POSTs with the same litellm_call_id and identical substantive fields. Byte-identity was therefore a flaky criterion; duplicates now qualify only when they share the call id, call type, model group, tokens, and cost, and a second differing event still fails * test(e2e): assert the scenario strictly; the messages test is the LIT-4447 regression pin Per review direction the tests now assert exactly what the scenario promises: exactly one DataDog log event per successful call, on every route. The /v1/messages test therefore fails on current code against the known double-log (LIT-4447) and is its regression pin; it goes green when the fix lands. The duplicate-tolerance machinery is removed * Simplify docstrings for DataDog log tests Removed redundant phrasing about cost cross-checking in docstrings. * Update test_datadog_log_e2e.py |
||
|
yucheng-berri
|
6a213de9f4
|
test(e2e): otel trace completeness on /v1/messages (#33133)
* test(e2e): OTEL trace completeness on /v1/messages Extends the LIT-3787 trace-completeness suite to the Anthropic-native route: one successful non-streaming /v1/messages call must land at the destination as ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT span, no dangling parents). Adds the raw /v1/messages sender to the logging suite client. * test(e2e): reuse the shared AnthropicMessagesBody per review Drops the duplicate /v1/messages request model in favor of the one models.py already provides (budget_client uses the same one), passes max_tokens at the call site to match the sibling chat test, notes in the docstring why the gen-AI span is named chat on this surface, and adopts the hardened read-back signature * test(e2e): author the messages trace test docstring * test(e2e): declare the messages surface on the covers marker * test(e2e): otel trace completeness on /v1/responses (#33134) * test(e2e): OTEL trace completeness on /v1/responses Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API route: one successful non-streaming /v1/responses call must land at the destination as ONE connected trace. Adds the raw /v1/responses sender, a CHEAP_OPENAI_MODEL config constant, and registers responses in the otel registry cell's exercised_on. * test(e2e): author the responses trace test docstring * test(e2e): declare the responses and chat surfaces on the covers markers |
||
|
yucheng-berri
|
948a43cd64
|
test(e2e): otel trace completeness on /chat/completions (#33132)
* test(e2e): OTEL trace completeness on /chat/completions against a local Jaeger destination Adds the logging-suite infrastructure for LIT-3787 trace-completeness coverage: a jaeger service in the compose stack as the OTEL v2 destination (arize_phoenix preset pointed at it via PHOENIX_COLLECTOR_HTTP_ENDPOINT, so gen-AI spans export through a preset-owned provider - the code path where trace splits happen), a typed Jaeger query read-back client, and the first test: one successful non-streaming /chat/completions call exports ONE complete trace (root SERVER span + auth/db/cost children + gen-AI CLIENT span, no dangling parents). * test(e2e): harden the otel trace read-back per review Jaeger reads now query server-side by the litellm.call_id span tag instead of paging recent traces and filtering client-side; the compose stack's background jobs alone can push a request trace past the page. A failed query hard-fails instead of reading as an empty result, the settle predicate now also waits for the prefix-matched db span the assertion demands, parent-chain walking follows CHILD_OF references only, the zero-trace and split-trace failures get distinct messages, jaeger gets a healthcheck so the depends_on condition is accurate, and the chat docstring names the route the code actually asserts * test(e2e): author the chat trace test docstring * Update logging section in CLAUDE.md Removed mention of OTEL trace-tree completeness from logging integration section. |
||
|
mubashir1osmani
|
a05a1eef94
|
fix(ui): scope key models dropdown options to the key's team (#32382)
* fix(ui): scope key models dropdown options to the key's team A teamless key no longer offers the all-team-models option in the create and edit forms; the backend expands that sentinel to the full proxy model list when no team is attached, which is rarely what the user intended. A team key no longer surfaces the all-proxy-models sentinel that leaks in verbatim when the team's own model list carries it; the dropdown keeps All Team Models plus the team's individual models. Adds browser coverage to the management e2e suite: playwright (an optional dependency behind importorskip) drives the proxy-served dashboard at /ui, asserts the dropdown options a real user sees for teamless and team keys on both create and edit, and walks the create modal end to end, reading the persisted key back through /key/info. * fix(ui): offer all-proxy-models on teamless keys in the models dropdown A teamless key has no team allowlist to inherit, so the dropdown now offers All Proxy Models in place of All Team Models on both the create and edit forms, with the same exclusive-selection handling. Component and browser e2e tests updated to pin the swapped option pair; the teamless create case now also walks the modal end to end and reads the persisted key back through /key/info. * test(ui): update no-team key creation spec to pick All Proxy Models The create modal no longer offers All Team Models without a team; the teamless path now offers All Proxy Models, which is what this spec exercises * fix(ui): gate All Team Models on the team object being loaded When a key has a team_id but the teams prop does not yet include the matching team, availableModels stays empty and the models dropdown rendered All Team Models on its own with nothing to compare against. Gate the option on the team object being present so it only appears once team models are known, and add a regression test for the loading state Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): filter all-proxy-models from teamless model fetch in key edit form The teamless fetch path stored modelAvailableCall results without excludeProxyWideSentinel, so an all-proxy-models entry in the response rendered a second option colliding with the hardcoded All Proxy Models sentinel. Apply the same filter used on the team path and add a regression test asserting the sentinel option is not duplicated Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
1b81148f2a
|
test: add e2e tests for spend, budgets and llms (#30869)
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* test: multi-window budgets coverage
* fix: p0 issues, added types and shared functions for each test suite
* chore: add config.yml
* test: passthrough endpoints stream/non-stream e2e
* style: carry clearer status_code comparison into renamed e2e dir
* fix: rename cost breakdown function
* fix: pydantic validation for budget info, dont allow explicit type cast
* refactor: migrate to gateway client
* test: add custom pricing tests
* chore: change master key
* test(e2e): address greptile review feedback
Remove the duplicate cache/cache_params block in the gateway config so the two
can't silently diverge under future edits. Reorder the soft-budget test to assert
the call isn't a budget block before require_successful_call, since that helper
hard-fails any non-2xx and left the budget-block check unreachable; the misleading
"skip" comment is corrected. Add a deferred delete in test_budget_delete_removes_it
so a failed delete doesn't leak a budget on the shared proxy. Scope the
spend_tracking sys.path insertion in pytest_sessionfinish to just the cleanup
import so a broader "pytest tests/" run isn't left with a mutated path.
* test(e2e): drop misleading skip comment on require_successful_call
require_successful_call fails hard, it does not skip; the trailing
comment was factually wrong. The function name already states intent,
so the comment is removed in both per-model and tag budget helpers.
* test(e2e): assert budget-isolation invariant before success check
On the should-still-succeed path of the per-model and tag isolation
tests, check is_budget_block before require_successful_call. If the
isolation bug fires the unaffected model/tag is blocked, so asserting
the specific 'blocked by X' invariant first yields the diagnostic
message instead of a generic upstream-failure. Matches the ordering in
test_soft_budget_e2e.py.
* fix(e2e): guard spend-log truncate on skip and stop returning unrelated priced rows
* fix(e2e): run case init() inside try so partial-init failures tear down
run_case called case.init() outside the try/finally that runs teardown(), so a
case that registers cleanups progressively (create team, then user, then key)
and then fails partway through init() would leak the already-created entities on
the long-lived shared proxy. Move init() inside the try so teardown always runs.
Add a regression test that registers a cleanup then raises mid-init and asserts
the resource is still released.
* test(e2e): mark known pricing-leak isolation test xfail(strict)
test_custom_pricing_is_isolated_from_sibling_deployment documents a real proxy
gap (a deployment's custom per-token pricing leaks into the shared cost map for
sibling deployments of the same underlying model) and was left unconditionally
failing, which pollutes the suite's pass/fail signal. Mark it xfail(strict=True)
so the suite stays green while the leak persists and turns into a failure the
moment isolation is fixed, prompting the marker's removal.
* refactor(e2e): make suite pass its shipped strict basedpyright config
The suite ships tests/pyrightconfig.json (strict, no Any), but basedpyright
--project tests reported four errors in it: three reportAny on the parametrize
ids=lambda c: c.__name__, and one reportUnusedFunction on the underscore-prefixed
autouse fixture _require_live_proxy. Replace the untyped lambda with a typed
_case_id(case_cls: Type[_BudgetCase]) -> str so the ids are no longer Any, and
rename the fixture to require_live_proxy so basedpyright no longer treats it as an
unused private function (it is referenced only by pytest's autouse machinery).
basedpyright --project tests now reports zero errors.
* fix(tests/e2e): gate spend-log truncate on e2e marker, not test directory
* test(e2e): run harness unit tests without a live proxy
The autouse session fixture skipped the whole tests/e2e session when no proxy
answered, which also skipped test_lifecycle.py, a pure unit test of run_case that
never touches the proxy. A regression test that silently skips gives no signal,
so the skip now lives in pytest_runtest_setup gated on the same e2e marker the
spend-log truncate guard already uses: live tests skip when no proxy is up while
harness unit coverage always runs. The liveness probe is cached with lru_cache so
it still runs once per session
* test(e2e): clean up gateway config comment debris
Fix the typo on the header comment and drop the orphaned namespace/ttl
comment remnants left indented under cache_params; the active values are
already set above. Flagged by greptile review.
* fix: add new tests, split gateway
* test(e2e): type the redis spend-counter probe for strict basedpyright
The new cold-counter reseed test drove its redis client untyped, so the strict
tests/pyrightconfig.json (reportUnknown*, reportAny) flagged ten errors once the
file landed: scan_iter/get came back unknown and the pool.map lambda had an
untyped parameter. Annotate the client as redis.Redis[str] via a TYPE_CHECKING
import (the runtime import stays lazy so the suite still skips, not errors, when
redis is absent), which resolves scan_iter to Iterator[str] and get to str | None,
and replace the lambda with a typed inner function mirroring _burst. basedpyright
--project tests is back to zero errors.
* test(e2e): xfail the known team multi-window failure and isolate member teardown
Greptile flagged two issues in the mirrored split-gateway commit. The team
multi-window budget test documents a real /team/new write bug (budget_limits go
straight to the Json? column and Prisma 500s, unlike the json.dumps'd key and
/team/update paths) and was left as an unconditional hard failure, which would
turn any live-proxy CI run red; mark it xfail(strict=True) like the custom-pricing
isolation test so the suite stays green while the bug persists and flips to a
failure the moment the write is fixed and the marker should go.
The class-scoped member fixture in test_team_member_budget_e2e.py tore down its
key, user, and team sequentially with no exception isolation, so a failed
delete_key would strand the user and team on the long-lived shared proxy. Route
cleanup through a ResourceManager: register each delete progressively and run them
LIFO best-effort in a finally, so a partial-setup failure still releases what came
before and one failed delete never blocks the rest.
* test(e2e): set fast budget-reset cadence in gateway config so staging windows reset within e2e timeouts
* test(e2e): surface real /spend/tags errors instead of masking them as missing tags
The spend-tracking e2e client swallowed every non-200 from /spend/tags into an
empty list, so a real server error or a response-shape mismatch showed up only as
the generic "tag never appeared in /spend/tags" with no diagnostics. That masking
is what made the original cluster failure undiagnosable.
spend_by_tags now raises SpendTagsError carrying the actual HTTP status and body
for any non-Success result, and poll_tag_spend fails fast on a hard server error
rather than polling it into a timeout; eventual consistency only manifests as a
200 whose payload does not yet carry the tag, so only that case waits. The tag
test now reports the last observed status and asserts the endpoint returned 200
at least once, with no weakened assertions.
Hardening surfaced the real defect in the test itself: /spend/tags returns a
top-level JSON array (List[LiteLLM_SpendLogs]), but the client validated against a
SpendTagsResponse dict wrapper that never matched, so every call fell through to
the empty-list mask. Wired spend_by_tags to the existing TagSpends RootModel and
removed the dead SpendTagsResponse model. Verified against the real Postgres that
request_tags is stored as proper JSONB arrays and /spend/tags aggregates them
correctly, so there is no encoding bug to fix here.
* test(e2e): drop flaky test_tag_spend_matches_sum_of_tagged_logs
The test wrote tagged requests and polled /spend/tags expecting read-after-write
consistency. /spend/tags itself is fine; verified live that request_tags is stored
as a JSON array and the endpoint reflects a fresh tag within seconds, so the
failures were a timing flake under full-suite load rather than a real defect.
Coverage is retained by test_request_tags_round_trip (tags persist onto the row)
and the /spend/tags route probe in test_spend_routes.py.
Also remove the now-dead tag-spend scaffolding this test was the only user of:
poll_tag_spend, spend_by_tags, TagSpendPoll, SpendTagsError, the TagSpend/TagSpends
models, and their imports.
* test(e2e): widen budget-reset wait windows to de-flake wall-clock-aligned resets
The short-window reset tests asserted the reset landed within WINDOW_SECONDS + 45
(~75s), but the 30s budget window is wall-clock-aligned, so the reset can land up
to a full window after start, then the rescheduler (~15-20s) zeroes the spend, plus
poll and DB lag. A real run measured 84s, just over the 75s bound, and which of the
short-window siblings tripped flipped run to run. Widen the wait loops to 150s and
the elapsed assertions to WINDOW_SECONDS + 90 (120s for the key test). A genuinely
stuck rescheduler is still caught by the wait-loop timeout, so this only removes the
timing flake, not the regression signal.
* test(e2e): let the spend-counter reseed test reach a cluster-mode TLS redis
The test's _redis() built a standalone, non-TLS client on the docker-compose
defaults (localhost:6380), so against the EKS serverless ElastiCache (cluster-mode
+ TLS) it could never connect and the test skipped. Honor E2E_REDIS_SSL and
E2E_REDIS_CLUSTER so it builds a TLS RedisCluster client when the deploy provides
them, and E2E_REDIS_NAMESPACE so the counter is read with a direct GET (cluster-safe)
rather than a keyspace scan that can't span shards. The local standalone path and the
graceful skip-on-unreachable behavior are unchanged.
* test(e2e): take the direct-GET spend-counter path on E2E_REDIS_CLUSTER
The gateway's cache sets no namespace, so the counter key is the bare
spend🔑<hash>. Trigger the cluster-safe direct GET on E2E_REDIS_CLUSTER (not
only on E2E_REDIS_NAMESPACE) so the cluster deploy need not set a namespace it
does not use; the namespaced key is still tried first when a namespace is given.
* test(e2e): use REDIS_HOST/REDIS_PORT and drop the unused redis knobs
The runner is a standalone test pod, so the proxy's own REDIS_HOST/REDIS_PORT
names are unambiguous - no E2E_ prefix needed. The only deployed redis it talks
to is the serverless ElastiCache (always TLS + cluster), so that is inferred from
REDIS_HOST being set rather than carried as ssl/cluster knobs. Stage sets no cache
namespace (bare counter key, read directly on the cluster) and is passwordless, so
the namespace and password env are gone; the local namespace is still handled by
the standalone SCAN.
* test(e2e): replace the vacuous failure-row test with per-model attribution
test_failure_call_writes_failure_status_row had two skip hatches (the call did
not fail, or no failure row landed) and never asserted anything on this proxy -
gemini accepts an empty message (HTTP 200), and live failure-row logging is
non-deterministic across providers. Replace it with a deterministic check: one
key calling gemini-2.5-flash and claude-haiku-4-5 gets one spend row per call,
each carrying its own model and a nonzero cost, under distinct request_ids that
match the call's response id. Verified live on stage (gemini/gemini-2.5-flash
$0.00053, anthropic/claude-haiku-4-5 $0.000038, distinct ids matching the
responses). Failure-status row construction stays covered by the unit suite.
* test(e2e): assert /spend/logs returns the key's spend without 5xx
Regression for the intermittent 500s on /spend/logs (DB query / serialization
errors under load). The existing spend_logs() helper swallows non-success
responses into an empty list, so a 500 looks identical to 'rows not flushed yet'.
This test queries the endpoint directly and asserts a Success response on every
poll, failing loudly on any 5xx, then requires the call's nonzero spend to surface.
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
|