The previous `enable_cassette_proxy_for_pytest` exported HTTP_PROXY /
HTTPS_PROXY / SSL_CERT_FILE etc. into $BASH_ENV, which routed every
subsequent shell command through mitmproxy — including the CircleCI
agent's own `circleci tests run` plugin auto-installer. The Go binary
ignores SSL_CERT_FILE, so the plugin download failed with
`tls: failed to verify certificate: x509: certificate signed by
unknown authority` and broke local_testing_part1/part2.
Confine the proxy env to a sourced file plus a `/usr/local/bin/cassette-pytest`
wrapper, and prefix the 18 pytest invocations in proxied jobs with that
wrapper. The CircleCI agent's own egress is no longer affected.
Two new reusable CircleCI commands plus a refactor of start_cassette_proxy
to support in-process pytest jobs (which don't have docker daemon access).
New commands
- start_cassette_proxy: now launches mitmdump as a background subprocess
via 'uv tool install mitmproxy', so the same command works on both
docker: and machine: executors. Exports
CASSETTE_PROXY_URL (host.docker.internal:8080 — for SUT containers)
CASSETTE_PROXY_HOST_URL (localhost:8080 — for the runner shell)
CASSETTE_PROXY_CA (/tmp/cassette-proxy-ca.crt)
into $BASH_ENV.
- export_cassette_proxy_docker_args: composes a single
$CASSETTE_PROXY_DOCKER_ARGS string of '-e ...' / '-v ...' flags
ready to splice into the SUT's 'docker run', so opt-in for an
in-Docker job is exactly two lines + one variable.
- enable_cassette_proxy_for_pytest: routes the runner shell's egress
through the sidecar for in-process pytest jobs. Patches certifi's
bundled cacert.pem in every venv on the runner (so openai-python /
httpx / langfuse / google-auth trust the proxy CA), exports
HTTPS_PROXY / NO_PROXY / SSL_CERT_FILE / etc. for every subsequent
step, and crucially flips AIOHTTP_TRUST_ENV=true — without that flag
litellm's aiohttp transport silently ignores HTTPS_PROXY (see
litellm/llms/custom_httpx/http_handler.py:951-952).
NO_PROXY now includes $REDIS_HOST automatically when set. mitmproxy
only handles HTTP/HTTPS; the redis client's TCP+TLS connection to the
project's managed Redis would be broken if it were sent through the
proxy. Same logic in both opt-in commands.
Jobs wired (Pattern A — SUT runs in a Docker container)
- e2e_openai_endpoints (already wired in the previous commit, now uses
the new $CASSETTE_PROXY_DOCKER_ARGS shorthand)
- build_and_test
- proxy_logging_guardrails_model_info_tests
- proxy_spend_accuracy_tests
- proxy_store_model_in_db_tests
- proxy_build_from_pip_tests
- proxy_pass_through_endpoint_tests
- proxy_e2e_anthropic_messages_tests
Jobs wired (Pattern B — pytest runs in-process)
- llm_translation_testing
- realtime_translation_testing
- agent_testing
- guardrails_testing
- google_generate_content_endpoint_testing
- llm_responses_api_testing
- ocr_testing
- search_testing
- litellm_mapped_enterprise_tests
- batches_testing
- litellm_utils_testing
- pass_through_unit_testing
- image_gen_testing
- logging_testing
- audio_testing
- local_testing_part1
- local_testing_part2
- langfuse_logging_unit_tests
Total: 25 jobs now route their LLM-provider HTTP egress through the
cassette proxy. The remaining CI jobs either don't make real provider
calls (proxy_multi_instance_tests, e2e_ui_testing,
auth_ui_unit_tests, redis_caching_unit_tests, ui_*, helm_*, install_*,
etc.) or are pure infrastructure (db_migration_disable_update_check,
test_bad_database_url, build_docker_database_image).
README updated with both opt-in patterns side by side.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Introduces a mitmproxy-based recording HTTP/HTTPS sidecar that any CI
job can opt into to cache LLM-provider responses across runs. Unlike
the in-process VCR persister at tests/_vcr_redis_persister.py — which
can only intercept HTTP traffic from the same Python process where it
was loaded — this sidecar operates at the network layer, so it works
for any e2e job whose system-under-test runs in a Docker container
(every job under e2e_*, proxy_*, etc.).
Components
- tests/e2e_cassette_proxy/cache_key.py: pure-function cache-key
derivation. Hashes (method, scheme, host, path, sorted query,
allowlisted headers, canonical-JSON body); strips auth, tracing,
and SDK-metadata headers so equivalent requests collide regardless
of run-to-run noise.
- tests/e2e_cassette_proxy/redis_store.py: thin Redis wrapper that
stores one (request, response) pair per key as MessagePack
(JSON+base64 fallback). Caps per-key payload size, drops oversize
responses with a log line, and never blocks the request path on
Redis errors.
- tests/e2e_cassette_proxy/addon.py: mitmproxy addon that ties the
two together. Hosts on the passthrough list (localhost, the proxy
itself) are never cached; non-2xx upstream responses are not
persisted.
- tests/e2e_cassette_proxy/Dockerfile: pinned python:3.12-slim base +
pinned mitmproxy 11.0.2 + pinned redis-py + pinned msgpack.
- tests/e2e_cassette_proxy/trust_ca.sh: helper for SUT containers to
trust the proxy CA in every Python / curl / boto3 / node trust
store at once.
- tests/e2e_cassette_proxy/README.md: usage guide + opt-in checklist
for other e2e jobs.
CI integration
- New reusable command 'start_cassette_proxy' in .circleci/config.yml.
Builds the image, runs the sidecar wired to the project Redis, fetches
the proxy CA, and exports CASSETTE_PROXY_URL / CASSETTE_PROXY_CA into
$BASH_ENV for downstream steps.
- e2e_openai_endpoints is wired up as the canonical demo: two-line opt-in
pattern documented in the README.
- Job logs include a 'Cassette-proxy stats' step that dumps the
hit/miss/store summary via 'docker logs cassette-proxy | grep
[E2ECASS]'.
Tests
- 31 hermetic unit tests under tests/test_litellm/e2e_cassette_proxy:
- test_cache_key.py: 14 tests pinning equivalence-class behavior of
the key derivation (auth header, tracing header, JSON key order,
query order, host case all collapse; method/path/body/allowlisted
headers / query-param values do not).
- test_redis_store.py: 9 tests covering set/get round-trip, default
TTL, binary body round-trip, oversize-payload rejection, corrupt-
blob eviction, and graceful behavior when the Redis client raises.
- test_addon.py: 8 tests using a fake-mitmproxy flow to exercise
the addon end-to-end (passthrough miss, persist on 2xx, hit on
canonicalized-equivalent re-request, no-persist on 5xx, host
passthrough, replay-only 599-on-miss, record-only never serves
cache).
- 31/31 pass.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The async/sync delete_response_api_handler always passed json=data into
httpx.delete, where data is {} from the transformer. httpx serializes that
to a 2-byte body. The Azure Responses DELETE endpoint now rejects any
request body with code: unexpected_body, breaking
test_basic_openai_responses_delete_endpoint on the llm_responses_api_testing
job. Build the kwargs dict and only set json= when data is truthy.
Add unit tests that patch httpx.delete and assert json/data are not in the
captured kwargs for the Azure DELETE path (sync and async).
The proxy's ingress hardening (commit 842eea0131) now strips client-supplied
`mock_response` from the request body unless the calling key or team has the
`allow_client_mock_response: true` admin-metadata flag set. The e2e model
access tests rely on `mock_response` to short-circuit the LLM call, so without
the flag they hit real backends — the bedrock wildcard route fakes out to a
shared example endpoint that now 404s on unsupported paths, causing
`test_model_access_patterns[key_models2-bedrock/anthropic.claude-3-True]`
(and the bedrock/anthropic.* row that pytest -x never reaches) to fail.
Set `allow_client_mock_response: true` on every key and team this test file
provisions so `mock_response` is preserved end-to-end.
Cover the full litellm.rerank()/arerank() path with HTTP mocked, asserting
metadata.requester_metadata reaches the Discovery Engine :rank body as
userLabels (and stays absent when no metadata is set). Catches plumbing
regressions that unit tests on transform_rerank_request alone would miss.
The wrapper had no production callers after transform_parsed_response
was refactored to call _resolve_json_mode_non_streaming directly.
Updated the parametrized test to call the underlying method.
