The cassette has been sharing the litellm proxy's Redis (REDIS_SSL_URL
/ REDIS_URL / REDIS_HOST) since day one. That Redis is also used by
litellm proxy instances elsewhere (staging, dev, other branches' CI),
and *something* on it calls FLUSHALL roughly 125x/day:
cmdstat_flushall: { calls: 4150 } # over 33 days uptime
Each FLUSHALL nukes the entire keyspace — including ~700-800 cassette
entries — and the next test in the same CI run has to fall through to
billable upstream calls. Net effect: even after fixing the replay path
in 916e9851, the proxy was still effectively record-only for any test
unlucky enough to run after a flush. We never found the FLUSHALL
caller in our own code (no /cache/flushall in tests, no .flushall() on
the cassette client) so the conclusion is that some other workload
shares this instance.
Switch to a dedicated cassette Redis via a new `CASSETTE_REDIS_URL`
project env var (set via the CircleCI API to an Upstash 10 GB
instance). Falls back to the old REDIS_* vars if the new one isn't
set so this works on forks without the secret. Also threads the
cassette host into NO_PROXY since RESP-over-TLS must bypass mitmproxy
(mitmproxy only speaks HTTP/HTTPS).
mitmproxy serves its current-instance CA certificate at the magic
host ``mitm.it/cert/pem``. The CI pipeline downloads that CA in the
``Fetch CA from cassette-proxy`` step and trusts it (certifi append +
system trust store) so subsequent in-process tests can validate
mitmproxy's MITM leaf certs.
Once the previous commit fixed the replay path so cache hits actually
serve, the very first ``mitm.it/cert/pem`` lookup hit a stale entry
from a prior CI run — the proxy handed back a CA generated by a *previous*
mitmdump instance with a different private key. The CI then trusted that
stale CA, mitmproxy in the *current* run signed leaf certs with its fresh
CA, and every TLS handshake failed:
SSL: CERTIFICATE_VERIFY_FAILED:
certificate verify failed: authority and subject key identifier mismatch
This took down langfuse_logging_unit_tests (at ``test_embedding.py``
collection), search_testing, and image_gen_testing.
Adding ``mitm.it`` to the passthrough host list short-circuits in
``_should_skip`` before any Redis lookup, so the CA is always served
fresh from the running mitmdump instance. The poisoned key in the
shared dev Redis was deleted manually before pushing this commit.
Adds a regression test that exercises both hooks: ``request()`` must
return without setting ``flow.response``, and ``response()`` must not
persist anything to Redis.
Every cache hit was silently falling through to the upstream because
the addon's replay path called http.Response.make(status, body, list[(str,str)]),
and mitmproxy 11's Headers constructor demands bytes — it raised
``TypeError: Header fields must be bytes.`` inside the addon, mitmdump
logged ``Addon error: Header fields must be bytes.`` to its background
log, and the request continued out to the real provider as if it had
been a miss. (Stats counted it as a hit because the log line was
emitted before the raise.) Net effect: the proxy was record-only.
Three fixes:
1. _build_replay_response constructs http.Response directly,
encoding the cached headers back to bytes (latin-1, RFC 7230) and
handing mitmproxy the raw on-the-wire body. Going through
Response.make/set_content would also have re-encoded the
body (e.g. double-gzip), so we bypass that codepath entirely.
2. The recording side now calls flow.response.headers.items(multi=True)
so repeated headers (notably multiple Set-Cookie) are preserved
as distinct entries instead of being silently merged.
3. Adds a contract test file that runs against *real* mitmproxy
(skipped automatically when only the test stub is loaded). This is
what would have caught the original bug — the existing fake stubs
don't model Headers's bytes-strictness, which is precisely why
the issue hid for the whole record-only run on the previous commit.
The existing addon unit tests now prefer real mitmproxy when it's
installed, so the contract tests run in the same process when both
are available.
The previous `enable_cassette_proxy_for_pytest` exported HTTP_PROXY /
HTTPS_PROXY / SSL_CERT_FILE etc. into $BASH_ENV, which routed every
subsequent shell command through mitmproxy — including the CircleCI
agent's own `circleci tests run` plugin auto-installer. The Go binary
ignores SSL_CERT_FILE, so the plugin download failed with
`tls: failed to verify certificate: x509: certificate signed by
unknown authority` and broke local_testing_part1/part2.
Confine the proxy env to a sourced file plus a `/usr/local/bin/cassette-pytest`
wrapper, and prefix the 18 pytest invocations in proxied jobs with that
wrapper. The CircleCI agent's own egress is no longer affected.
Two new reusable CircleCI commands plus a refactor of start_cassette_proxy
to support in-process pytest jobs (which don't have docker daemon access).
New commands
- start_cassette_proxy: now launches mitmdump as a background subprocess
via 'uv tool install mitmproxy', so the same command works on both
docker: and machine: executors. Exports
CASSETTE_PROXY_URL (host.docker.internal:8080 — for SUT containers)
CASSETTE_PROXY_HOST_URL (localhost:8080 — for the runner shell)
CASSETTE_PROXY_CA (/tmp/cassette-proxy-ca.crt)
into $BASH_ENV.
- export_cassette_proxy_docker_args: composes a single
$CASSETTE_PROXY_DOCKER_ARGS string of '-e ...' / '-v ...' flags
ready to splice into the SUT's 'docker run', so opt-in for an
in-Docker job is exactly two lines + one variable.
- enable_cassette_proxy_for_pytest: routes the runner shell's egress
through the sidecar for in-process pytest jobs. Patches certifi's
bundled cacert.pem in every venv on the runner (so openai-python /
httpx / langfuse / google-auth trust the proxy CA), exports
HTTPS_PROXY / NO_PROXY / SSL_CERT_FILE / etc. for every subsequent
step, and crucially flips AIOHTTP_TRUST_ENV=true — without that flag
litellm's aiohttp transport silently ignores HTTPS_PROXY (see
litellm/llms/custom_httpx/http_handler.py:951-952).
NO_PROXY now includes $REDIS_HOST automatically when set. mitmproxy
only handles HTTP/HTTPS; the redis client's TCP+TLS connection to the
project's managed Redis would be broken if it were sent through the
proxy. Same logic in both opt-in commands.
Jobs wired (Pattern A — SUT runs in a Docker container)
- e2e_openai_endpoints (already wired in the previous commit, now uses
the new $CASSETTE_PROXY_DOCKER_ARGS shorthand)
- build_and_test
- proxy_logging_guardrails_model_info_tests
- proxy_spend_accuracy_tests
- proxy_store_model_in_db_tests
- proxy_build_from_pip_tests
- proxy_pass_through_endpoint_tests
- proxy_e2e_anthropic_messages_tests
Jobs wired (Pattern B — pytest runs in-process)
- llm_translation_testing
- realtime_translation_testing
- agent_testing
- guardrails_testing
- google_generate_content_endpoint_testing
- llm_responses_api_testing
- ocr_testing
- search_testing
- litellm_mapped_enterprise_tests
- batches_testing
- litellm_utils_testing
- pass_through_unit_testing
- image_gen_testing
- logging_testing
- audio_testing
- local_testing_part1
- local_testing_part2
- langfuse_logging_unit_tests
Total: 25 jobs now route their LLM-provider HTTP egress through the
cassette proxy. The remaining CI jobs either don't make real provider
calls (proxy_multi_instance_tests, e2e_ui_testing,
auth_ui_unit_tests, redis_caching_unit_tests, ui_*, helm_*, install_*,
etc.) or are pure infrastructure (db_migration_disable_update_check,
test_bad_database_url, build_docker_database_image).
README updated with both opt-in patterns side by side.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Introduces a mitmproxy-based recording HTTP/HTTPS sidecar that any CI
job can opt into to cache LLM-provider responses across runs. Unlike
the in-process VCR persister at tests/_vcr_redis_persister.py — which
can only intercept HTTP traffic from the same Python process where it
was loaded — this sidecar operates at the network layer, so it works
for any e2e job whose system-under-test runs in a Docker container
(every job under e2e_*, proxy_*, etc.).
Components
- tests/e2e_cassette_proxy/cache_key.py: pure-function cache-key
derivation. Hashes (method, scheme, host, path, sorted query,
allowlisted headers, canonical-JSON body); strips auth, tracing,
and SDK-metadata headers so equivalent requests collide regardless
of run-to-run noise.
- tests/e2e_cassette_proxy/redis_store.py: thin Redis wrapper that
stores one (request, response) pair per key as MessagePack
(JSON+base64 fallback). Caps per-key payload size, drops oversize
responses with a log line, and never blocks the request path on
Redis errors.
- tests/e2e_cassette_proxy/addon.py: mitmproxy addon that ties the
two together. Hosts on the passthrough list (localhost, the proxy
itself) are never cached; non-2xx upstream responses are not
persisted.
- tests/e2e_cassette_proxy/Dockerfile: pinned python:3.12-slim base +
pinned mitmproxy 11.0.2 + pinned redis-py + pinned msgpack.
- tests/e2e_cassette_proxy/trust_ca.sh: helper for SUT containers to
trust the proxy CA in every Python / curl / boto3 / node trust
store at once.
- tests/e2e_cassette_proxy/README.md: usage guide + opt-in checklist
for other e2e jobs.
CI integration
- New reusable command 'start_cassette_proxy' in .circleci/config.yml.
Builds the image, runs the sidecar wired to the project Redis, fetches
the proxy CA, and exports CASSETTE_PROXY_URL / CASSETTE_PROXY_CA into
$BASH_ENV for downstream steps.
- e2e_openai_endpoints is wired up as the canonical demo: two-line opt-in
pattern documented in the README.
- Job logs include a 'Cassette-proxy stats' step that dumps the
hit/miss/store summary via 'docker logs cassette-proxy | grep
[E2ECASS]'.
Tests
- 31 hermetic unit tests under tests/test_litellm/e2e_cassette_proxy:
- test_cache_key.py: 14 tests pinning equivalence-class behavior of
the key derivation (auth header, tracing header, JSON key order,
query order, host case all collapse; method/path/body/allowlisted
headers / query-param values do not).
- test_redis_store.py: 9 tests covering set/get round-trip, default
TTL, binary body round-trip, oversize-payload rejection, corrupt-
blob eviction, and graceful behavior when the Redis client raises.
- test_addon.py: 8 tests using a fake-mitmproxy flow to exercise
the addon end-to-end (passthrough miss, persist on 2xx, hit on
canonicalized-equivalent re-request, no-persist on 5xx, host
passthrough, replay-only 599-on-miss, record-only never serves
cache).
- 31/31 pass.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The async/sync delete_response_api_handler always passed json=data into
httpx.delete, where data is {} from the transformer. httpx serializes that
to a 2-byte body. The Azure Responses DELETE endpoint now rejects any
request body with code: unexpected_body, breaking
test_basic_openai_responses_delete_endpoint on the llm_responses_api_testing
job. Build the kwargs dict and only set json= when data is truthy.
Add unit tests that patch httpx.delete and assert json/data are not in the
captured kwargs for the Azure DELETE path (sync and async).
The proxy's ingress hardening (commit 842eea0131) now strips client-supplied
`mock_response` from the request body unless the calling key or team has the
`allow_client_mock_response: true` admin-metadata flag set. The e2e model
access tests rely on `mock_response` to short-circuit the LLM call, so without
the flag they hit real backends — the bedrock wildcard route fakes out to a
shared example endpoint that now 404s on unsupported paths, causing
`test_model_access_patterns[key_models2-bedrock/anthropic.claude-3-True]`
(and the bedrock/anthropic.* row that pytest -x never reaches) to fail.
Set `allow_client_mock_response: true` on every key and team this test file
provisions so `mock_response` is preserved end-to-end.
Cover the full litellm.rerank()/arerank() path with HTTP mocked, asserting
metadata.requester_metadata reaches the Discovery Engine :rank body as
userLabels (and stays absent when no metadata is set). Catches plumbing
regressions that unit tests on transform_rerank_request alone would miss.
