* test: migrate local_testing SDK and provider tests out of build_and_test
* test: cover sagemaker region, hf template and nova paths end to end
Region rows now go through litellm.completion on the sagemaker/ handler that
the legacy tests exercised, the prompt template test mocks the Hugging Face
tokenizer config instead of switching to custom_prompt_dict, streaming tests
assert role, finish reason and body, and Nova single turn and streaming run
through litellm.completion with a recorded request body
* test: tighten migrated integration tests for local OpenAI routes, timeouts and air-gapped boot
Split the catch-all providers migration file by behavior. The fake endpoint
route tests now hit the owned upstream directly, as the legacy harness tests
did. LM Studio asserts the outbound key, path and stripped model. The embedding
timeout test asserts the proxy maps a deployment timeout to a 408 and forwards
the read timeout to the provider client, instead of timing out the test client
The Azure streaming tool-call test also asserts the second identical stream is
served from the Redis cache with the same reassembled tool call
The Docker network test always skipped in CI because the security job has no
litellm/litellm image and no egress to pull one, and it tested a published image
rather than the checkout. It now boots the checkout proxy with every outbound
HTTP connection refused and asserts readiness, the bundled cost map fallback and
that the cost map host is the only egress attempted
* test: tighten the migrated e2e rows for local_testing SDK and provider tests
Drop the duplicated Anthropic covers marker and the Anthropic cell wrongly claimed by
the Bedrock and Vertex cache tests. Keep the file-wide Anthropic model as it is on main.
Send the 2023-06-01 version header to Anthropic through the deployment's extra_headers
so the header actually reaches the provider, and mark the creation-token case 1h then
5m like the legacy test did
Reuse the Bedrock file's own registration for the pydantic response test, swap the
hand-built PNGs for the shared cat.jpg fixture, assert the SageMaker Nova image answer
names the cat and that top_logprobs=3 is honored, and register the SageMaker Nova opt-in
markers the same way as the other opt-in markers
* test: restore lost assertions in migrated test_main tests
Restore the role alternation check in test_completion_anthropic_hanging, the
embedding and streaming response-header checks, the logit_bias drop check and
the gemini system and safety settings checks. Add an azure 429 retry-after
test for test_exception_with_headers, trigger the router fallback with
mock_timeout like the legacy test, and stop the mock timeout tests sleeping 3s
* test: restore lost legacy assertions in migrated vertex ai unit tests
* test: tighten migrated cost, streaming and embedding tests and scope the hosted mock guard
* test: strengthen migrated anthropic, azure, bedrock and cohere provider tests
Assert full request bodies and usage instead of presence checks, restore the
dummy-tool and custom_llm_provider assertions the legacy tests made, add the
async and per-model variants the legacy parametrization covered, split
streamed tool arguments across deltas, and move the Azure chat streaming
tests to the azure.py mirror file
* test: tighten migrated llms provider tests and cover legacy intents lost in the move
Strengthen the migrated fireworks, gemini, mistral, together, openai text completion, openrouter, perplexity and openai error tests so they assert what the legacy tests proved (cost, provider tag, usage-only-on-last-chunk, function_call rebuild, reasoning param, sync/async/stream error fields). Replace the huggingface and jina tests that doubled litellm code with respx end-to-end tests. Add the tests the pre-existing mapped tests never covered: cohere 400 mapping, token-array text completion, hosted_vllm/llamafile/lm_studio env api base embeddings, watsonx embedding auth, ollama ssl_verify client and ollama vision prompt
* test: cover residual legacy paths for anthropic tool streams, watsonx async stream, ollama http images and caller async embedding clients
* test: restore single request assertions in the deep_infra and azure_ai embedding tests
* test: drop a no-op streaming cost flag from the openai streaming cost test
* test: cover residual legacy-test lines in caching, main, utils, replicate, vertex and bedrock invoke
* test: cover streamed audio assembly, choice membership, untranslated responses events and the stream log guard
* test: pin cost rates in unit tests, use sdk for cohere e2e, drop vertex llama e2e
* test: send the vertex maps e2e request through the openai sdk
* test: drop the local_testing jobs that select no tests after the migrations
* test: wait for the streamed cache entry and keep custom Mistral rates test-local
* test: keep untouched Mistral test lines as they were
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test(e2e): jwt auto_register map-existing-key repro
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(jwt): auto_register_map_existing_key maps JWT to the user's existing virtual key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): exclude blocked keys from auto_register_map_existing_key reuse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(jwt): route existing-key lookup through VerificationTokenRepository
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): stop requiring LITELLM_SALT_KEY for the owned JWT gateway
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): gate the owned JWT gateway tests behind E2E_OWNED_GATEWAY
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): only reuse keys that can call LLM routes in auto_register_map_existing_key
Skip Admin UI session keys and keys whose allowed_routes restrict them to
anything other than llm_api_routes (management, read_only, password-reset
sessions). Mapping a JWT to one of those left the user with 401s or 403s on
every LLM call, since the mapping persists.
* fix(jwt): scope auto_register_map_existing_key reuse to the JWT-resolved team
Only reuse a key whose team_id matches the team auth_builder resolved for
the JWT (no team matches no team), so a personal key can no longer bypass
the resolved team's model and budget limits.
With the flag on, the first JWT request now falls through to the same
virtual-key checks later mapped requests get, instead of returning early,
so a reused key's own limits apply from request one rather than 200 then
403. Flag off keeps the early return unchanged.
* fix(jwt): keep the early return when no master key is set
Without a master key the generic virtual-key path returns a bare
INTERNAL_USER object, so falling through on the first auto-registered
request dropped the key's team, models and budgets. Only fall through when
a master key is configured.
Tests now assert the reused key per team rather than the query shape, and
cover the flag-off early return and the no-master-key case.
* test(jwt): assert on race-loser's returned key, not only mocks (TQ002)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(jwt): close the auto_register_map_existing_key race, shared-claim and expiry holes
A key auto_register just minted is never adopted by a concurrent request, so the race loser's cleanup can no longer delete a key another request mapped and cascade its mapping away (503, user left with no key)
Reuse only happens when the claim value is the JWT-resolved user_id. A shared claim such as azp or client_id falls back to minting, so one user can no longer land on another user's personal key and budget
Only keys that never expire are reused, so an expiring key can no longer pin the claim to a permanent 401
Integration tests on a real proxy and Postgres cover all three. The race test holds the first mapping insert in a Postgres relay, so the interleaving is forced rather than timed. The where-clause shape unit tests are replaced by these, since only a real database proves the filter
* test(e2e): create the reused key in the team the JWT resolves to
The flag only reuses a key in the JWT-resolved team, and this identity's groups claim resolves to its team, so a teamless key was never eligible and the test could not pass
* test(integration): match the held statement across TCP reads
The relay looked for the trigger inside one read, so an insert split across two reads was never held and the race test would fail waiting for it. It now matches one exact trigger over a window that keeps the end of the previous read
* fix(jwt): gate key reuse on the claim field, not on the claim value
Requiring the claim value to equal the resolved user_id skipped reuse for users matched through the sso_user_id or case-insensitive email fallback, whose stored user_id differs from the JWT sub. That is the lookup LIT-5378 asks for. Reuse is now allowed when the virtual key claim is the user_id or user_email JWT field, globally or for the token's issuer, which still keeps shared claims such as azp or client_id on the mint path
* fix(jwt): let an issuer's own user field replace the global one when gating key reuse
An issuer that identifies users by uid no longer treats the global sub field as a user identity claim, so a shared sub under that issuer mints instead of reusing a personal key
* test(jwt): make the flag-off test fail when the flag no longer gates key reuse
The flag-off test used a config where sub was not a user identity claim, so deleting the flag check still passed. Configure user_id_jwt_field=sub so only the flag keeps the lookup off, and drop test docstrings
* chore(lint): drop mutable-ok suppressions that LIT013 flags as no-ops
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Mrinal Chanshetty <mchanshetty@Mrinals-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(e2e): give e2e tests typed metadata for what they drive
@meta(Subject(domain, route, providers, models, capabilities, mode)) declares
what a test is about with closed enums, and each field lands in the JUnit report
as a property. The quota_management suites are the first to declare it.
* docs(e2e): say e2e_metadata avoids litellm, not that it is stdlib-only
It already imports pydantic and pytest, both of which the suite needs to collect. The rule that matters is no litellm import
* test(e2e): declare models through the constant each test drives
43 @meta declarations in quota_management typed the model name out again, so changing the call would leave the coverage report naming the old model. Each file now has one constant used by both, and a guard fails on any model written as a string literal in @meta
* refactor(e2e): set route only when the endpoint is what the test checks
A budget or rate-limit test whose chat call only triggers the block now leaves route unset, since its steps already name the call. Tests of an endpoint keep it: budget CRUD, key creation, spend reporting reads, and the per-endpoint spend tests for chat, messages, embeddings, batches and health. The two /spend/logs tests tagged chat_completions are now spend_reporting
* refactor(e2e): build the declared properties without mutating a list
subject_properties seeded a list and grew it with append and extend. It now flattens one tuple per field, and the plural-name table is a read-only mapping
* fix(e2e): tag each spend-route probe with the endpoint it checks
The breadth test gave all 33 probes spend_reporting, so /key/list, /user/list, /team/list, /organization/list and /customer/list counted as spend reporting. Each case now carries its own route, with organization and customer management added to Route
* test(e2e): add a HashiCorp Vault secret manager lane
key_management_system had no end-to-end coverage: the Rust crates and the
Python unit tests all run against mocked managers. This adds a secret_manager
suite that drives a proxy configured with hashicorp_vault against a real Vault.
The tests seed a fresh secret name per test with the runner's OPENAI_API_KEY and
register a deployment pointing at os.environ/<name>. The proxy's env never holds
that name, so get_secret's os.environ fallback cannot mask a broken manager, and
a bogus value in Vault must come back as the provider's 401. Virtual keys are
checked written to and removed from Vault under prefix_for_stored_virtual_keys.
The setting is global to the proxy, so the lane has its own config and the
secret_manager_vault opt-in marker, and stays out of the per-PR selector.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): make the secret manager suite backend-agnostic
One marker and opt-in (secret_manager / E2E_SECRET_MANAGER=<system>) pick the
backend from secret_backends.BACKENDS. The tests reach the manager through a
SecretStore protocol, and each backend contributes a secret_store_<system>.py
module, a registry entry, and gateway/secret_manager_<system>_ci_config.yml.
requires_capability deselects tests a backend cannot support (CyberArk does
not delete), and test_secret_backends.py checks every lane config against its
backend without a live stack.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): add a CyberArk Conjur secret manager lane
Adds cyberark as the second secret_manager backend: a Conjur store over its
REST API (policy-declared variables, raw-text values, policy-patch teardown),
its lane config, and a registry entry without deletes_stored_keys, since the
proxy's CyberArk delete answers not_supported and Conjur keeps the key.
secret_manager/backend.sh up|down <system> boots any backend in Docker and
writes proxy.env and tests.env, so every lane runs the same way; the registry
test checks the script boots exactly the registered backends. e2e_http gains
send_text_external for APIs that speak raw text rather than JSON.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(e2e): give the secret manager suite a client with .proxy and address review
The shared resources fixture reads client.proxy, so a bare ProxyClient errored every
live test at setup. backend.sh now writes its env under a per-user directory with
umask 077, the markerless unit tests are gone per tests/e2e/AGENTS.md, and routine
comments are trimmed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* test(e2e): run the memory cell alone on the shared stack
* test(e2e): hold the stack lock for every collected test, marker or not
* test(e2e): prove the stack lock's reader sharing and writer preference across processes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(otel): honor SSL_CERT_FILE and ssl_verify in OTLP HTTP exporters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): assert OTLP HTTP TLS behavior against a real TLS sink
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): assert rejected exports by outcome, not by exception type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): honor SSL_CERT_FILE and ssl_verify in the v2 OTLP HTTP exporters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): hoist otlp_tls imports and type the TLS sink fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): gate the OTLP TLS export test behind an otel_tls opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(e2e): drop CONTRIBUTING.md edit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(otel v2): map completions, images, speech, transcription and moderation output onto the generation output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(redaction): redact text completion choices in the standard logging payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): compare decoded generation output text and follow the live moderation verdict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel v2): compare logged byte counts with the received media and move e2e schemas into models.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(e2e): keep the otel_v2 Langfuse output e2e file out of the stage-mirror gate it cannot run in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Buildkite ephemeral stack runs the gateway in another pod, so it cannot reach the pytest host's provider edge. The GitHub changed-e2e lane runs gateways on the runner and sets E2E_PROVIDER_EDGE_HOST_REACHABLE
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The cache edge keyed every recording on its own process's PYTEST_CURRENT_TEST.
Under xdist that names whatever test the serving worker is in, which is
unrelated to the caller: the proxy is a separate pod, and the Claude Code compat
matrix registered its shared aliases from every worker, each pointing at that
worker's edge, so the router spread one worker's calls across all eight edges.
Builds 234 and 235 of litellm-e2e, same commit, credited the same Bedrock
request to unrelated tests 92% of the time, and Bedrock never converged past a
~20% hit rate while OpenAI, whose deployments are per test, sat at 90%.
A deployment registered from inside a test now carries its test's slug in the
edge URL it is pointed at, `{edge}/{mount}/t/{slug}`, and the edge reads that
segment off every request before forwarding. A request without one is forwarded
live and never cached, and the edge no longer falls back to process state. The
compat aliases are registered with provider_live=True and stay on their real
provider path: no single test owns them, and the matrix exists to prove the real
CLI against real providers.
It drives the real CLI for several seconds. The edge stamps every
upstream call with PYTEST_CURRENT_TEST, a process-global that names
whichever test the worker is in when the call arrives rather than the one
that made it, so a test that holds a worker that long collects other
tests' in-flight calls. Build 234's key report credits this test with 20
Bedrock and 7 Anthropic misses, and it makes no provider call at all.
Those misattributed calls take the wrong test id into the cache key and
write recordings under it, so the test was polluting the shared corpus it
exists to protect.
Deselected unless E2E_CLI_DETERMINISM is set, the same opt-in shape the
managed-files, prompt-caching and redis-chaos markers already use. The
attribution bug itself is older than this branch and is reported, not
fixed here.
create-release.yml is back to calling the chaos test through no mechanism at
all; it never called it. Also drops the E2E_REDIS_CHAOS opt-in gate itself:
the redis_chaos marker still exists for -m selection and is still excluded
from the per-PR selector by path (tests/e2e/(ui|claude_code|load)/), but the
test no longer needs an env var to run once its file is targeted.
Co-Authored-By: Claude Code <noreply@anthropic.com>
CLIENT PAUSE ALL for the length of the chaos phase instead of CLIENT PAUSE WRITE, so every Redis touchpoint on the request path times out rather than just the writes. The pause is sized to the phase because it freezes the control connection too; teardown's CLIENT UNPAUSE is a safety net for a phase that overran
Latency, RSS and CPU are now budgeted as chaos-over-baseline ratios (p50/p90/p99 for latency and RSS, CPU seconds per request once) through a small phase_budget module, replacing the machine-shaped absolutes. The Redis timeout rate is reported but no longer asserted
The final /metrics scrape waits for litellm_deployment_failure_responses_total to stop moving, since that counter is bumped from the async logging queue and lagged the load generator by thousands of increments. The model group carries a unique marker so a deployment left behind by an aborted run cannot absorb this run's retries
Co-Authored-By: Claude Code <noreply@anthropic.com>
The sequential version sent one request at a time, so a Redis outage never
reached the concurrency where the failed-tracking alert body actually grows.
This drives the proxy with locust against one model group of three mock
deployments, two failing at order 1 and one serving at order 2, so every
request spends its retries on the failing pair and lands on the serving
deployment through the order-based fallback. Two phases, a healthy baseline
and a CLIENT PAUSE WRITE window, and every request must succeed in both.
Latency, RSS and CPU are reported as p50/p90/p99 per phase rather than
asserted on: RSS and CPU come from psutil on the proxy's process tree, since
a multi-worker proxy serves /metrics from the prometheus multiprocess
collector and that drops the process collector's series. Thresholds stay open
until weekly runs give real baselines.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Add tests/e2e/router/test_redis_timeout_e2e.py against a proxy booted from
tests/e2e/gateway/redis_timeout_ci_config.yml: a real Redis with socket_timeout 0.001, so every
command times out and the circuit breaker opens, plus a primary deployment that always fails and
falls back to a healthy one, so every request carries retry breadcrumbs into cost tracking. The
test drives twenty chat requests through the proxy and asserts each answers within ten seconds,
the last third is no slower than the first, /health/liveliness stays fast, and every request
still reaches the spend log.
Gate it behind the redis_timeout marker and E2E_REDIS_TIMEOUT, exclude it from the per-PR
e2e-changed selector, register the reliability.circuit_breaker.redis_timeout.stays_responsive
cell, and run it as its own job in the weekly load anomaly workflow with a Postgres and Valkey
service. Against a v1.100.0 proxy the run wedges the worker: requests time out and liveliness
stops answering (LIT-6780).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
The prompt-cache affinity test needs the prompt_caching pre-call check on the
proxy, which the CI stack does not carry until project-releaser #223 lands, so
it now sits behind a prompt_caching_stack marker that is deselected unless
E2E_PROMPT_CACHING_STACK is set, the same shape as managed_files. The
per-directory deselection hooks for weekly and managed_files move into the
parent conftest as one OPT_IN_MARKERS table, so the collector counts a gated
cell only where its env var is set (31/36 today, 32/36 with #223).
The simple-shuffle cell asked for three picks, which a shuffle ignoring the
weights passes one time in eight; it now asks for ten.
Adds a scheduled GitHub Actions lane on top of the merged record/replay
transport. A Saturday cron records the `replayable` e2e tests against the
real providers and publishes the fixture bundle as a private
`e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull
that artifact by its pinned digest, verify the checksum before extracting,
and replay it with provider credentials set to bogus values, so a run that
ever reached a real provider fails instead of passing.
An egress sentinel pins the provider hostnames to a local sink for the whole
replay job and counts every connection that reaches them; the job asserts
that count is zero, so hermeticity is proven by measurement. A red Saturday
publishes no bundle, so the next weekday finds nothing fresh and fails loudly
rather than replaying a week-old recording, and the transport's seven-day
freshness gate hard-fails any bundle that has drifted too far. The lane also
runs on demand from the Actions tab with a record/replay `mode` input.
Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic
test now counts to twenty so its recorded response banks several content
deltas, matching the assertion that the stream arrives incrementally.
* test(e2e): send no-cache on every cacheable request body, opt in only where a hit is the assertion
The e2e proxy runs with the response cache on, so any test that re-sends an
identical chat, messages, responses, completions, embeddings or rerank body
reads back a redis copy of an earlier call instead of reaching the provider.
Five tests in the last week failed that way. Default cache: {"no-cache": true}
on those request models and pass cache=None only in the two tests whose
assertion is the cache hit itself.
* test(e2e): give image edits and OCR a 180s client timeout
Both routes wait on providers that can legitimately take longer than the
60s transport-wide request timeout (gpt-image edits, Azure Document
Intelligence), and a client-side read timeout there fails a green request.
post/upload now accept a per-call timeout like get already does; only those
two call sites use it.
* test(e2e): rerun once on network errors and upstream 5xx only
Assertion failures still fail on the first attempt; only an outcome whose
error string carries the e2e_http network kind or a 5xx status gets one
more try. Test Engine records every attempt, so the flake rate stays
visible while a single provider blip no longer reds the rc run.
* test(e2e): let the reseed burst survive one upstream failure and print why
The burst is the precondition, not the property: one 5xx among six
concurrent calls still leaves five workers racing the cold counter, which
is what the reseed assertion measures. Two or more failures still abort,
and the failing bodies are now in the message instead of only the status
codes.
* test(e2e): keep polling Jaeger through a transient query failure
poll_traces_for_call already waits up to POLL_TIMEOUT for spans to land,
but a single refused connection to the query API failed the test on the
spot. Jaeger restarted twice during today's gate runs (19:05 and 19:41
UTC, each under a minute) and took ten and three otel tests with it while
the same tests passed on the rc build minutes later. A network failure
now counts as not-yet inside the same deadline; if Jaeger is still
unreachable when the deadline passes the test fails with that error, and
any non-network failure still fails immediately.
CodSpeed benchmarks the SDK with no IO, so it can't catch regressions that
only appear under real concurrent load through the full proxy stack (auth,
routing, logging, spend, Postgres, Redis). This adds a Locust load test under
tests/e2e/load that drives concurrent POST /chat/completions traffic against a
mock deployment (litellm_params.mock_response), so the measured throughput
reflects proxy overhead rather than a provider's latency, and asserts an
aggregate RPS SLO with a failure-ratio guard. The test is marked load and the
parent conftest sorts load-marked items last so it never perturbs
latency-sensitive suites. Covers reliability.perf.throughput.under_slo.
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* test: multi-window budgets coverage
* fix: p0 issues, added types and shared functions for each test suite
* chore: add config.yml
* test: passthrough endpoints stream/non-stream e2e
* style: carry clearer status_code comparison into renamed e2e dir
* fix: rename cost breakdown function
* fix: pydantic validation for budget info, dont allow explicit type cast
* refactor: migrate to gateway client
* test: add custom pricing tests
* chore: change master key
* test(e2e): address greptile review feedback
Remove the duplicate cache/cache_params block in the gateway config so the two
can't silently diverge under future edits. Reorder the soft-budget test to assert
the call isn't a budget block before require_successful_call, since that helper
hard-fails any non-2xx and left the budget-block check unreachable; the misleading
"skip" comment is corrected. Add a deferred delete in test_budget_delete_removes_it
so a failed delete doesn't leak a budget on the shared proxy. Scope the
spend_tracking sys.path insertion in pytest_sessionfinish to just the cleanup
import so a broader "pytest tests/" run isn't left with a mutated path.
* test(e2e): drop misleading skip comment on require_successful_call
require_successful_call fails hard, it does not skip; the trailing
comment was factually wrong. The function name already states intent,
so the comment is removed in both per-model and tag budget helpers.
* test(e2e): assert budget-isolation invariant before success check
On the should-still-succeed path of the per-model and tag isolation
tests, check is_budget_block before require_successful_call. If the
isolation bug fires the unaffected model/tag is blocked, so asserting
the specific 'blocked by X' invariant first yields the diagnostic
message instead of a generic upstream-failure. Matches the ordering in
test_soft_budget_e2e.py.
* fix(e2e): guard spend-log truncate on skip and stop returning unrelated priced rows
* fix(e2e): run case init() inside try so partial-init failures tear down
run_case called case.init() outside the try/finally that runs teardown(), so a
case that registers cleanups progressively (create team, then user, then key)
and then fails partway through init() would leak the already-created entities on
the long-lived shared proxy. Move init() inside the try so teardown always runs.
Add a regression test that registers a cleanup then raises mid-init and asserts
the resource is still released.
* test(e2e): mark known pricing-leak isolation test xfail(strict)
test_custom_pricing_is_isolated_from_sibling_deployment documents a real proxy
gap (a deployment's custom per-token pricing leaks into the shared cost map for
sibling deployments of the same underlying model) and was left unconditionally
failing, which pollutes the suite's pass/fail signal. Mark it xfail(strict=True)
so the suite stays green while the leak persists and turns into a failure the
moment isolation is fixed, prompting the marker's removal.
* refactor(e2e): make suite pass its shipped strict basedpyright config
The suite ships tests/pyrightconfig.json (strict, no Any), but basedpyright
--project tests reported four errors in it: three reportAny on the parametrize
ids=lambda c: c.__name__, and one reportUnusedFunction on the underscore-prefixed
autouse fixture _require_live_proxy. Replace the untyped lambda with a typed
_case_id(case_cls: Type[_BudgetCase]) -> str so the ids are no longer Any, and
rename the fixture to require_live_proxy so basedpyright no longer treats it as an
unused private function (it is referenced only by pytest's autouse machinery).
basedpyright --project tests now reports zero errors.
* fix(tests/e2e): gate spend-log truncate on e2e marker, not test directory
* test(e2e): run harness unit tests without a live proxy
The autouse session fixture skipped the whole tests/e2e session when no proxy
answered, which also skipped test_lifecycle.py, a pure unit test of run_case that
never touches the proxy. A regression test that silently skips gives no signal,
so the skip now lives in pytest_runtest_setup gated on the same e2e marker the
spend-log truncate guard already uses: live tests skip when no proxy is up while
harness unit coverage always runs. The liveness probe is cached with lru_cache so
it still runs once per session
* test(e2e): clean up gateway config comment debris
Fix the typo on the header comment and drop the orphaned namespace/ttl
comment remnants left indented under cache_params; the active values are
already set above. Flagged by greptile review.
* fix: add new tests, split gateway
* test(e2e): type the redis spend-counter probe for strict basedpyright
The new cold-counter reseed test drove its redis client untyped, so the strict
tests/pyrightconfig.json (reportUnknown*, reportAny) flagged ten errors once the
file landed: scan_iter/get came back unknown and the pool.map lambda had an
untyped parameter. Annotate the client as redis.Redis[str] via a TYPE_CHECKING
import (the runtime import stays lazy so the suite still skips, not errors, when
redis is absent), which resolves scan_iter to Iterator[str] and get to str | None,
and replace the lambda with a typed inner function mirroring _burst. basedpyright
--project tests is back to zero errors.
* test(e2e): xfail the known team multi-window failure and isolate member teardown
Greptile flagged two issues in the mirrored split-gateway commit. The team
multi-window budget test documents a real /team/new write bug (budget_limits go
straight to the Json? column and Prisma 500s, unlike the json.dumps'd key and
/team/update paths) and was left as an unconditional hard failure, which would
turn any live-proxy CI run red; mark it xfail(strict=True) like the custom-pricing
isolation test so the suite stays green while the bug persists and flips to a
failure the moment the write is fixed and the marker should go.
The class-scoped member fixture in test_team_member_budget_e2e.py tore down its
key, user, and team sequentially with no exception isolation, so a failed
delete_key would strand the user and team on the long-lived shared proxy. Route
cleanup through a ResourceManager: register each delete progressively and run them
LIFO best-effort in a finally, so a partial-setup failure still releases what came
before and one failed delete never blocks the rest.
* test(e2e): set fast budget-reset cadence in gateway config so staging windows reset within e2e timeouts
* test(e2e): surface real /spend/tags errors instead of masking them as missing tags
The spend-tracking e2e client swallowed every non-200 from /spend/tags into an
empty list, so a real server error or a response-shape mismatch showed up only as
the generic "tag never appeared in /spend/tags" with no diagnostics. That masking
is what made the original cluster failure undiagnosable.
spend_by_tags now raises SpendTagsError carrying the actual HTTP status and body
for any non-Success result, and poll_tag_spend fails fast on a hard server error
rather than polling it into a timeout; eventual consistency only manifests as a
200 whose payload does not yet carry the tag, so only that case waits. The tag
test now reports the last observed status and asserts the endpoint returned 200
at least once, with no weakened assertions.
Hardening surfaced the real defect in the test itself: /spend/tags returns a
top-level JSON array (List[LiteLLM_SpendLogs]), but the client validated against a
SpendTagsResponse dict wrapper that never matched, so every call fell through to
the empty-list mask. Wired spend_by_tags to the existing TagSpends RootModel and
removed the dead SpendTagsResponse model. Verified against the real Postgres that
request_tags is stored as proper JSONB arrays and /spend/tags aggregates them
correctly, so there is no encoding bug to fix here.
* test(e2e): drop flaky test_tag_spend_matches_sum_of_tagged_logs
The test wrote tagged requests and polled /spend/tags expecting read-after-write
consistency. /spend/tags itself is fine; verified live that request_tags is stored
as a JSON array and the endpoint reflects a fresh tag within seconds, so the
failures were a timing flake under full-suite load rather than a real defect.
Coverage is retained by test_request_tags_round_trip (tags persist onto the row)
and the /spend/tags route probe in test_spend_routes.py.
Also remove the now-dead tag-spend scaffolding this test was the only user of:
poll_tag_spend, spend_by_tags, TagSpendPoll, SpendTagsError, the TagSpend/TagSpends
models, and their imports.
* test(e2e): widen budget-reset wait windows to de-flake wall-clock-aligned resets
The short-window reset tests asserted the reset landed within WINDOW_SECONDS + 45
(~75s), but the 30s budget window is wall-clock-aligned, so the reset can land up
to a full window after start, then the rescheduler (~15-20s) zeroes the spend, plus
poll and DB lag. A real run measured 84s, just over the 75s bound, and which of the
short-window siblings tripped flipped run to run. Widen the wait loops to 150s and
the elapsed assertions to WINDOW_SECONDS + 90 (120s for the key test). A genuinely
stuck rescheduler is still caught by the wait-loop timeout, so this only removes the
timing flake, not the regression signal.
* test(e2e): let the spend-counter reseed test reach a cluster-mode TLS redis
The test's _redis() built a standalone, non-TLS client on the docker-compose
defaults (localhost:6380), so against the EKS serverless ElastiCache (cluster-mode
+ TLS) it could never connect and the test skipped. Honor E2E_REDIS_SSL and
E2E_REDIS_CLUSTER so it builds a TLS RedisCluster client when the deploy provides
them, and E2E_REDIS_NAMESPACE so the counter is read with a direct GET (cluster-safe)
rather than a keyspace scan that can't span shards. The local standalone path and the
graceful skip-on-unreachable behavior are unchanged.
* test(e2e): take the direct-GET spend-counter path on E2E_REDIS_CLUSTER
The gateway's cache sets no namespace, so the counter key is the bare
spend🔑<hash>. Trigger the cluster-safe direct GET on E2E_REDIS_CLUSTER (not
only on E2E_REDIS_NAMESPACE) so the cluster deploy need not set a namespace it
does not use; the namespaced key is still tried first when a namespace is given.
* test(e2e): use REDIS_HOST/REDIS_PORT and drop the unused redis knobs
The runner is a standalone test pod, so the proxy's own REDIS_HOST/REDIS_PORT
names are unambiguous - no E2E_ prefix needed. The only deployed redis it talks
to is the serverless ElastiCache (always TLS + cluster), so that is inferred from
REDIS_HOST being set rather than carried as ssl/cluster knobs. Stage sets no cache
namespace (bare counter key, read directly on the cluster) and is passwordless, so
the namespace and password env are gone; the local namespace is still handled by
the standalone SCAN.
* test(e2e): replace the vacuous failure-row test with per-model attribution
test_failure_call_writes_failure_status_row had two skip hatches (the call did
not fail, or no failure row landed) and never asserted anything on this proxy -
gemini accepts an empty message (HTTP 200), and live failure-row logging is
non-deterministic across providers. Replace it with a deterministic check: one
key calling gemini-2.5-flash and claude-haiku-4-5 gets one spend row per call,
each carrying its own model and a nonzero cost, under distinct request_ids that
match the call's response id. Verified live on stage (gemini/gemini-2.5-flash
$0.00053, anthropic/claude-haiku-4-5 $0.000038, distinct ids matching the
responses). Failure-status row construction stays covered by the unit suite.
* test(e2e): assert /spend/logs returns the key's spend without 5xx
Regression for the intermittent 500s on /spend/logs (DB query / serialization
errors under load). The existing spend_logs() helper swallows non-success
responses into an empty list, so a 500 looks identical to 'rows not flushed yet'.
This test queries the endpoint directly and asserts a Success response on every
poll, failing loudly on any 5xx, then requires the call's nonzero spend to surface.
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
