Introduces a mitmproxy-based recording HTTP/HTTPS sidecar that any CI
job can opt into to cache LLM-provider responses across runs. Unlike
the in-process VCR persister at tests/_vcr_redis_persister.py — which
can only intercept HTTP traffic from the same Python process where it
was loaded — this sidecar operates at the network layer, so it works
for any e2e job whose system-under-test runs in a Docker container
(every job under e2e_*, proxy_*, etc.).
Components
- tests/e2e_cassette_proxy/cache_key.py: pure-function cache-key
derivation. Hashes (method, scheme, host, path, sorted query,
allowlisted headers, canonical-JSON body); strips auth, tracing,
and SDK-metadata headers so equivalent requests collide regardless
of run-to-run noise.
- tests/e2e_cassette_proxy/redis_store.py: thin Redis wrapper that
stores one (request, response) pair per key as MessagePack
(JSON+base64 fallback). Caps per-key payload size, drops oversize
responses with a log line, and never blocks the request path on
Redis errors.
- tests/e2e_cassette_proxy/addon.py: mitmproxy addon that ties the
two together. Hosts on the passthrough list (localhost, the proxy
itself) are never cached; non-2xx upstream responses are not
persisted.
- tests/e2e_cassette_proxy/Dockerfile: pinned python:3.12-slim base +
pinned mitmproxy 11.0.2 + pinned redis-py + pinned msgpack.
- tests/e2e_cassette_proxy/trust_ca.sh: helper for SUT containers to
trust the proxy CA in every Python / curl / boto3 / node trust
store at once.
- tests/e2e_cassette_proxy/README.md: usage guide + opt-in checklist
for other e2e jobs.
CI integration
- New reusable command 'start_cassette_proxy' in .circleci/config.yml.
Builds the image, runs the sidecar wired to the project Redis, fetches
the proxy CA, and exports CASSETTE_PROXY_URL / CASSETTE_PROXY_CA into
$BASH_ENV for downstream steps.
- e2e_openai_endpoints is wired up as the canonical demo: two-line opt-in
pattern documented in the README.
- Job logs include a 'Cassette-proxy stats' step that dumps the
hit/miss/store summary via 'docker logs cassette-proxy | grep
[E2ECASS]'.
Tests
- 31 hermetic unit tests under tests/test_litellm/e2e_cassette_proxy:
- test_cache_key.py: 14 tests pinning equivalence-class behavior of
the key derivation (auth header, tracing header, JSON key order,
query order, host case all collapse; method/path/body/allowlisted
headers / query-param values do not).
- test_redis_store.py: 9 tests covering set/get round-trip, default
TTL, binary body round-trip, oversize-payload rejection, corrupt-
blob eviction, and graceful behavior when the Redis client raises.
- test_addon.py: 8 tests using a fake-mitmproxy flow to exercise
the addon end-to-end (passthrough miss, persist on 2xx, hit on
canonicalized-equivalent re-request, no-persist on 5xx, host
passthrough, replay-only 599-on-miss, record-only never serves
cache).
- 31/31 pass.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The async/sync delete_response_api_handler always passed json=data into
httpx.delete, where data is {} from the transformer. httpx serializes that
to a 2-byte body. The Azure Responses DELETE endpoint now rejects any
request body with code: unexpected_body, breaking
test_basic_openai_responses_delete_endpoint on the llm_responses_api_testing
job. Build the kwargs dict and only set json= when data is truthy.
Add unit tests that patch httpx.delete and assert json/data are not in the
captured kwargs for the Azure DELETE path (sync and async).
The proxy's ingress hardening (commit 842eea0131) now strips client-supplied
`mock_response` from the request body unless the calling key or team has the
`allow_client_mock_response: true` admin-metadata flag set. The e2e model
access tests rely on `mock_response` to short-circuit the LLM call, so without
the flag they hit real backends — the bedrock wildcard route fakes out to a
shared example endpoint that now 404s on unsupported paths, causing
`test_model_access_patterns[key_models2-bedrock/anthropic.claude-3-True]`
(and the bedrock/anthropic.* row that pytest -x never reaches) to fail.
Set `allow_client_mock_response: true` on every key and team this test file
provisions so `mock_response` is preserved end-to-end.
Cover the full litellm.rerank()/arerank() path with HTTP mocked, asserting
metadata.requester_metadata reaches the Discovery Engine :rank body as
userLabels (and stays absent when no metadata is set). Catches plumbing
regressions that unit tests on transform_rerank_request alone would miss.
The wrapper had no production callers after transform_parsed_response
was refactored to call _resolve_json_mode_non_streaming directly.
Updated the parametrized test to call the underlying method.
Two Greptile P2s addressed:
1. (security) The audit-log row for an ``add_team_callbacks`` call would
serialize the entire ``callback_vars`` block — including
``langfuse_secret_key``, ``langsmith_api_key``, and the GCS service
account path — verbatim into ``LiteLLM_AuditLogs``. Anyone with read
access to the audit table could harvest team callback credentials.
Same risk for ``disable_team_logging`` when the team's existing row
has populated ``callback_settings.callback_vars``.
Add ``_redact_callback_secrets``: deep-copies the metadata snapshot
and replaces every ``callback_vars`` value with ``***REDACTED***``.
The keys are kept so an auditor can still see *which* fields
changed. Applied to both before and after snapshots.
2. ``asyncio.create_task`` is fire-and-forget; if the audit-log write
raises (transient DB error etc.) the exception is silently
discarded by the event loop and the audit row is just missing —
the exact gap this PR is closing. Attach a ``done_callback`` that
logs the exception at warning level via ``verbose_proxy_logger`` so
the operator sees there's a gap.
Tests assert that callback values are not present in the serialized
audit payload (both for ``add_team_callbacks`` and for
``disable_team_logging`` when the team's existing row has
populated secrets).
The two mutating endpoints in team_callback_endpoints.py
(``/team/{id}/callback`` POST and ``/team/{id}/disable_logging``) wrote
team metadata without emitting an audit-log row. The disable variant
is the worst case: a logging-control action that itself isn't logged,
so an admin (or compromised admin) could zero out a team's
observability with no forensic trail.
Add ``_emit_team_callback_audit_log`` mirroring the
``store_audit_logs``-gated pattern already used in team_endpoints.py
for /team/new and /team/update. When ``litellm.store_audit_logs`` is
True, both endpoints now emit an ``LiteLLM_AuditLogs`` row capturing
the calling user, the API key, and the before/after team metadata.
When the flag is False the helper is a no-op, so non-Enterprise
deployments are unaffected.
The ``litellm_changed_by`` header is now also accepted on
``/team/{id}/disable_logging`` to match the existing
``add_team_callbacks`` shape; the header is optional so existing
callers are unaffected.
Variant scope: the file has three endpoints — both mutating variants
are now logged. The read-only ``GET /team/{id}/callback`` is unchanged.
Other unlogged callback / logging-control admin endpoints elsewhere in
the proxy (e.g. ``/cache/settings``, ``/config_overrides/hashicorp_vault``)
are out of scope here and would be addressed in a separate PR.
Tests cover both endpoints in both ``store_audit_logs`` states and
verify that the captured before/after metadata reflects the actual
mutation, plus that the ``litellm-changed-by`` header overrides the
auth user_id when supplied.
