* fix(tests): drop deprecated OpenAI-Beta realtime header
OpenAI deprecated the 'OpenAI-Beta: realtime=v1' header; the live
service now returns code 4000 invalid_beta with
"Unknown beta requested: 'realtime'.". Two integration tests in
tests/llm_translation/realtime/test_realtime_guardrails_openai.py
hardcoded the header and started failing across all PRs.
Library code is unaffected: the OpenAI realtime handler only
forwards 'OpenAI-Beta: realtime=v1' upstream when the proxy *client*
sends it (litellm/llms/openai/realtime/handler.py). Default proxy
behavior uses the GA protocol.
Connect to OpenAI without the deprecated header, and accept the GA
event name 'response.output_audio_transcript.delta' alongside the
beta-protocol name 'response.audio_transcript.delta' for the
transcript-delta assertion.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(logging): post_call tolerates non-JSON-serializable values
post_call() did json.dumps(original_response) without default=str, so
any provider passing a dict containing datetime/Decimal/etc. would
raise TypeError. Bedrock batch retrieval hits this with
get_model_invocation_job() responses that include datetime fields
(submitTime, lastModifiedTime, endTime), failing
tests/batches_tests/test_bedrock_files_and_batches.py::test_async_file_and_batch
across all PRs.
Pass default=str so non-serializable values fall back to str().
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(tests): mock boto3 in bedrock retrieve batch test
The test patched AsyncHTTPHandler.get, but the bedrock retrieve
handler uses boto3.client('bedrock').get_model_invocation_job
directly, so the real AWS call was being made on every run, failing
with AccessDeniedException because the hardcoded test ARN belongs to
a different AWS account.
- Mock boto3.client and BedrockBatchesConfig.get_credentials so the
test never touches AWS.
- Use status=Completed in the mock response so output_file_id is
populated (the handler intentionally leaves it None for
non-completed jobs).
- Assert the predicted per-job output object URI (matches what the
handler actually returns) instead of the bare output prefix.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* docs(tests): include GA event name in guardrail-block test docstring
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
OpenAI deprecated the gpt-4o-realtime-preview-2024-10-01 snapshot,
which caused these E2E tests to fail consistently in CI. Bump to the
unversioned gpt-4o-realtime-preview alias to match the sibling
test_openai_realtime_simple.py and stay current as OpenAI rolls the
alias forward.
- Add `_strip_image_b64_payloads` filter: rewrites `data[*].b64_json` in
image-gen responses to a 4-byte placeholder before the cassette is saved.
Image-edit and image-gen cassettes (193 MB / 184 MB / 104 MB / ...) will
shrink to <100 KB on next record. Tests assert response shape only, so
coverage is preserved.
- Add `_normalize_multipart_boundary` filter: replaces httpx's per-request
random multipart boundary with a fixed string in both Content-Type header
and body bytes. Audio-transcription / Whisper tests have been effectively
unmocked — every CI run hit live providers and was silently capped at
MAX_EPISODES_PER_CASSETTE=50. Both record and replay now see identical
bytes; the safe_body matcher works.
- Fix test_evals_api.py body poisoning: replace `int(time.time())` in eval
names with `hashlib.sha1(test_node_name)[:12]`, add a function-scoped
`managed_eval` fixture that creates and deletes the eval, and switch
`get_eval` / `update_eval` from `list_evals().data[0].id` (which made
the URL vary by run) to `managed_eval.id`. Net coverage gain: delete is
now actually exercised.
- Swap arxiv PDF URL in BaseOCRTest for the in-repo `dummy.pdf` (589 B)
served via sha-pinned jsdelivr.
- Swap etsystatic image URL in BaseLLMChatTest.test_image_url for the
in-repo LiteLLM logo (9.2 KB) served via the same jsdelivr pin.
- Add `tests/llm_translation/test_vcr_filters.py` with 14 unit tests
covering both new filters: replacement, idempotency, nesting, content-
length update, two-distinct-boundaries-converge-after-normalize, etc.
Cassettes recorded with the prior patterns will mismatch on the first CI
run after merge; recommend flushing the cassette Redis once (post-merge)
so re-records save under the new format from the start.
* test: add 24hr Redis-backed VCR cache to additional test suites
Extracts the existing llm_translation VCR plumbing into a reusable helper
(tests/_vcr_conftest_common.py) and wires it into the conftest.py files
of the test directories listed in LIT-2787:
audio_tests, batches_tests, guardrails_tests, image_gen_tests,
litellm_utils_tests, local_testing, logging_callback_tests,
pass_through_unit_tests, router_unit_tests, unified_google_tests
The same helper is also adopted by the pre-existing llm_translation and
llm_responses_api_testing conftests to remove the copy-pasted VCR setup.
Each consuming conftest:
- registers the Redis persister via pytest_recording_configure
- auto-marks collected tests with pytest.mark.vcr (skipping respx-using
files where applicable, since respx and vcrpy both patch httpx)
- gates cassette writes on test success via _vcr_outcome_gate
The cache is opt-in via CASSETTE_REDIS_URL; when unset, VCR is disabled
and tests hit live providers as before. LITELLM_VCR_DISABLE=1 still
forces a bypass for ad-hoc local runs.
Test directories that run LiteLLM proxy in Docker (build_and_test,
proxy_logging_guardrails_model_info_tests, proxy_store_model_in_db_tests)
are intentionally not included: VCR.py patches the in-process httpx
transport and cannot intercept calls made from inside a Docker container.
The installing_litellm_on_python* jobs make no LLM calls and don't
benefit from caching.
https://linear.app/litellm-ai/issue/LIT-2787/add-24hr-caching-to-additional-test-suites
* test(vcr): add safe-body matcher to handle JSONL and binary request bodies
vcrpy's stock body matcher inspects Content-Type and unconditionally
runs json.loads on application/json bodies. JSON Lines payloads (used
by the Bedrock batch S3 PUT and other upload paths) crash that with
json.JSONDecodeError: Extra data, before the matcher can return
'not a match'.
This was the root cause of the batches_testing CI job failing on
test_async_create_file once VCR auto-marking was applied to the
batches_tests directory.
Add a conservative byte-equality body matcher and use it in place of
'body' in the shared match_on tuple. The matcher is strictly more
conservative than vcrpy's default — the only thing it gives up is
'different JSON key order is treated as the same body', which doesn't
apply to deterministic litellm-built request payloads. It can never
produce a false positive that the default would have rejected, so
there is no cross-contamination risk.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): exclude tests that VCR replay actively breaks
A few tests are incompatible with cassette replay and were failing on
the latest CI run after VCR auto-marking was extended to local_testing
and logging_callback_tests:
- test_amazing_s3_logs.py (logging_callback_tests): the test asserts on
a per-run response_id that should round-trip through a real S3
PUT/LIST. vcrpy's boto3 stub intercepts the PUT and the LIST replays
stale keys, so the freshly-generated id is never found.
- test_async_embedding_azure (logging_callback_tests) and
test_amazing_sync_embedding (local_testing): the failure branches
deliberately pass api_key='my-bad-key' to assert that the failure
callback fires. We scrub auth headers from cassettes (so the bad-key
request matches the prior good-key request), and vcrpy replays the
recorded 200 — the failure callback never fires.
- test_assistants.py (local_testing): the OpenAI Assistants polling
APIs mint fresh thread/run IDs every recording session and then poll
until status=='completed'. Replays of those polled GETs can never
match a freshly-generated run id, so every CI run effectively
re-records and the suite blows past the 15m no_output_timeout.
Skip these from VCR auto-marking so they continue to hit live providers
as they did before this change. The remaining tests in each directory
still get cached.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): expand skip lists for second batch of incompatible tests
Followup to the previous commit. After re-running CI on the rebuilt
branch, three more tests surfaced as VCR-replay-incompatible:
- litellm_utils_testing :: test_get_valid_models_from_dynamic_api_key
Calls GET /v1/models with api_key='123' to assert the result is empty.
We scrub auth headers, so the bad-key request matches the prior
good-key cassette and replays the recorded model list.
- litellm_utils_testing :: test_litellm_overhead.py
Measures litellm_overhead_time_ms as a percentage of total wall-clock
time. With cached responses the upstream 'network' time collapses to
microseconds, blowing past the 40%% threshold the test asserts on.
Skip the whole file (every parametrization is at risk).
- local_testing_part1 :: test_async_custom_handler_completion and
test_async_custom_handler_embedding
Same bad-key failure-callback pattern as the already-skipped
test_amazing_sync_embedding.
- litellm_router_testing :: test_router_caching.py
Asserts on litellm's own router-level response cache by comparing
response1.id to response2.id across repeat upstream calls (test
bypasses litellm cache via ttl=0 and expects upstream to return a
*new* id). With VCR replay both upstream calls return the same
cassette body, so the ids are identical. Skip the whole file.
- logging_callback_tests :: test_async_chat_azure (preemptive)
Same shape as already-skipped test_async_embedding_azure; was masked
by upstream OpenAI rate-limit failures on baseline.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): use item.path and tighten matcher docstring
- Replace pytest's deprecated item.fspath with item.path in
apply_vcr_auto_marker_to_items so we don't emit deprecation
warnings under pytest 8.
- Clarify _safe_body_matcher docstring to reflect actual behavior
(direct == first, then UTF-8 bytes comparison, no repr fallback).
Addresses Greptile review feedback on PR #27159.
* test(vcr): swallow all RedisError on cassette save/load
Cassette persistence is strictly best-effort: any Redis-side failure
(connection blip, timeout, OutOfMemoryError when the maxmemory cap is
hit, READONLY replicas, etc.) should degrade to 'test passed but
cassette not cached' rather than fail the test on teardown.
Previously the persister only caught ConnectionError and TimeoutError,
so OutOfMemoryError — which Redis Cloud raises when the cassette cache
hits its memory cap and there are no evictable keys — propagated out of
vcrpy's autouse fixture and ERRORed otherwise-passing tests on
teardown. This caused the litellm_utils_testing CircleCI job to fail on
the latest commit's run, even though the underlying test was a unit
test that used mock_response and produced no real upstream traffic
(the cassette was dirtied by a background langfuse callback). The
rerun only succeeded because Redis evictions happened to free enough
room before the SET — i.e. it was timing-dependent flakiness.
Catch redis.exceptions.RedisError (the common base of all server- and
client-side Redis exceptions) on both save and load, and parametrize
the regression tests across ConnectionError, TimeoutError, and
OutOfMemoryError to pin the new behavior.
* test(vcr): surface cassette-cache failures with warnings + session banner
When the persister silently swallows a Redis OOM (or any RedisError) on
save/load there is otherwise no visible signal that the cache is
degraded — tests pass, the cassette just isn't persisted, and the next
session still hits the same Redis at the same near-cap memory.
Add three layers of observability so that failure mode is loud:
1. Per-process health counters ("save_failures", "load_failures", and
the last error string for each), exposed via cassette_cache_health()
and reset via reset_cassette_cache_health(). The persister
increments these in addition to logging.
2. VCRCassetteCacheWarning (UserWarning subclass) emitted via
warnings.warn() inside the persister's except block. Pytest's
built-in warnings summary at session end automatically lists every
such warning, so the failure is visible in CI logs without any
conftest-level wiring.
3. Session-end banner via emit_cassette_cache_session_banner() and a
stderr-fallback atexit handler registered from
register_persister_if_enabled(). Two states:
- red "VCR CASSETTE CACHE DEGRADED" when save_failures or
load_failures > 0
- yellow "VCR CASSETTE CACHE NEAR CAPACITY" (no failures, but
used_memory >= 85% of maxmemory) so the next session knows
the Redis is approaching OOM before any SET actually fails
Capacity comes from a best-effort INFO memory probe
(cassette_cache_capacity_snapshot) that returns None on any failure or
when maxmemory is uncapped. The atexit handler skips xdist workers so
only the controller emits.
Tests: parametrize the existing save/load swallow-error tests across
ConnectionError/TimeoutError/OutOfMemoryError, add direct tests for
the health counters and warning emission, and a new
test_vcr_conftest_common_banner.py covering banner output for every
state (silent/red/yellow/disabled/xdist-worker).
* test(vcr): bucket cassettes by API key fingerprint, drop bad-key skips
Tests that deliberately call an LLM API with a bad key (e.g. to assert
that the failure callback fires, or that check_valid_key returns False)
were being silently served the prior good-key cassette: we scrub the
real Authorization / x-api-key header from the cassette before storing
it, so a follow-up bad-key call is byte-identical to the good-key call
under the existing match_on tuple.
Add a 'key_fingerprint' custom matcher that distinguishes requests by
the SHA-256 of their API-key headers. The fingerprint is stamped into
a synthetic 'x-litellm-key-fp' header by a new before_record_request
hook, which then strips the real auth headers (we have to do the
scrubbing here instead of via vcrpy's filter_headers knob, because
filter_headers runs *first* and would erase the value we want to hash).
Bad-key requests now get a different cassette bucket than good-key
requests, so vcrpy will not replay a recorded 200 in place of the
expected 401. The fingerprint is a one-way hash of the secret, so
cassettes never contain the key.
This permanently removes the 'bad-key' category of skips:
- tests/local_testing: dropped ::test_amazing_sync_embedding,
::test_async_custom_handler_completion,
::test_async_custom_handler_embedding
- tests/logging_callback_tests: dropped ::test_async_chat_azure,
::test_async_embedding_azure
- tests/litellm_utils_tests: dropped
::test_get_valid_models_from_dynamic_api_key
Coverage: 7 new unit tests in tests/test_litellm/test_vcr_safe_body_matcher.py
covering header stripping, fingerprint determinism, no-auth bucketing,
good-vs-bad key discrimination, x-api-key (Anthropic/Azure) discrimination,
and idempotence under replay.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): drop redundant comments and docstrings
Trim narration of code that is already self-evident from function and
variable names. Keep the two genuinely non-obvious bits:
- ordering constraint between filter_headers and before_record_request,
which would invite a maintainer to re-introduce the bug if removed
- the per-directory _VCR_INCOMPATIBLE_FILES rationale, since 'why
exactly is this skipped' is not knowable from the test name alone
Also drop the 40-line commented-out drop-in conftest snippet at the
bottom of _vcr_conftest_common.py — the consuming conftests are the
canonical reference.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): make _before_record_request idempotent
vcrpy invokes before_record_request more than once per request:
can_play_response_for calls it, then __contains__ /
_responses (reached via play_response) call it again on the
result. The second invocation sees a request whose auth headers we
already stripped, so a naive recompute yields "no-key" and
overwrites the real fingerprint stored in the header.
This makes can_play_response_for and play_response disagree on
matchability — the former says "yes, we have a stored response for
this" (matching no-key to no-key) and the latter throws
UnhandledHTTPRequestError because it computes a fresh real
fingerprint that doesn't match the stored no-key.
In CI this manifested as ~30 failing tests across guardrails_testing,
audio_testing, batches_testing, image_gen_testing, llm_responses_api,
litellm_router_unit_testing, etc. Skip the recompute when the header
is already set, so re-applying the hook is a no-op.
Adds a regression test that fires the hook twice on the same dict and
asserts the fingerprint stays put.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): drop more redundant docstrings and headers
* test(vcr): enable 24hr cache for ocr_tests and search_tests
These two directories were the only non-dockerized test suites in the
build_and_test workflow that make live LLM/provider API calls but were
not VCR-enabled by this PR. Together they account for 96 tests:
- tests/ocr_tests/ (31): Mistral OCR, Azure AI OCR, Azure Document
Intelligence, Vertex AI OCR. Pure-unit tests inside the same files
(e.g. TestAzureDocumentIntelligencePagesParam) make no HTTP calls
and become benign VCR NOOPs.
- tests/search_tests/ (65): Brave, DataForSEO, DuckDuckGo, Exa,
Firecrawl, Google PSE, Linkup, Parallel.ai, Perplexity, SearchAPI,
Searxng, Serper, Tavily.
Both directories use the canonical minimal conftest pattern from
tests/audio_tests/conftest.py with no skip lists. None of the test
files use respx, none assert on per-call upstream non-determinism
(no response1.id != response2.id, no overhead-as-fraction-of-total,
no live polling), so the default match_on tuple should cache cleanly.
If a flake surfaces during the first cassette-recording CI run, we
can add a targeted skip the same way we did for the other dirs.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The Azure o-series tests were excluded from the conftest's VCR auto-marker
because of a respx/vcrpy transport-patching conflict, but the only respx
reference in the file was an unused `MockRouter` import. Drop the dead
import and remove the file from the conflict set so cassettes record on
first run and replay thereafter, eliminating the 60-95s live Azure latency
that was crashing xdist workers under --timeout=120 thread-mode timeouts.
Newer Cloudflare Workers AI models (e.g. Nemotron) emit 'response_text'
instead of 'response' on streamed chunks. The non-streaming path was
already updated to fall back to 'response_text' (#26385), but the
streaming chunk parser still only read 'response', which caused
streaming requests against those models to silently produce empty
content.
Mirror the non-streaming fallback in CloudflareChatResponseIterator.chunk_parser
and add a streaming test for the response_text shape.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
`os.path.relpath` with no `start` arg uses the current working
directory, so running pytest from a subdirectory produced a
different Redis key than running from the repo root. CI-recorded
cassettes and locally-replayed runs would silently miss each
other's cache.
Anchor the path to the repo root (derived from `__file__`) so the
key is stable regardless of CWD.
https://claude.ai/code/session_018uCx7pcrkdUJZrCVMaTdPx
Remove explanatory comments that restated what the code already says.
Kept only those that document non-obvious external contracts (the aiohttp
record-path patch's reason for re-feeding the body, and the warning
messages inside save_cassette that reach the user).
Previous attempt wrote to sys.__stderr__ from the test fixture. Under
xdist, fixtures run inside worker subprocesses whose stderr is captured
by the controller and only released to the live log on test failure —
so passing tests' verdicts were silently swallowed.
Round-trip via report.user_properties: the worker-side fixture stashes
the verdict on user_properties, xdist serializes it onto the report,
and a controller-side pytest_runtest_logreport hook writes it via the
TerminalReporter (the same plugin that emits PASSED/FAILED markers).
TerminalReporter is resolved lazily on first hook call because it's
not yet registered when conftest's pytest_configure runs.
Verified locally in both serial and xdist modes.
Raw github serves application/octet-stream which OpenAI/Gemini reject
when LiteLLM fetches the URL client-side. jsDelivr serves the same
file with content-type: application/pdf. Pin to a commit SHA so the
asset is immutable and jsDelivr can cache it for a year.
Previously, the per-test [VCR HIT/MISS/...] line was written via
TerminalReporter.write_line from inside fixture teardown. Pytest
captures that stream by default and only surfaces it on FAILED tests
(under 'Captured stdout teardown'), so passing tests' verdicts were
invisible in CI logs and the user couldn't tell whether the cache
was working.
Write directly to sys.__stderr__ so the line bypasses pytest's
capture entirely. Under xdist each worker has its own __stderr__
which CircleCI aggregates into the live job log alongside the
PASSED/FAILED markers.
GitHub serves PDFs from raw.githubusercontent.com, github.com/.../raw/..., LFS, and Releases as application/octet-stream by deliberate anti-hotlinking policy. Anyone who passes a GitHub-hosted PDF URL as an OpenAI / Gemini / Bedrock file_id hits "unsupported MIME type 'application/octet-stream'" because _process_image_response inlines the URL with whatever Content-Type the server sent.
When the server-provided Content-Type is application/octet-stream or binary/octet-stream and the URL extension maps to a known MIME type (.pdf, .png, .jpg, etc.), trust the extension instead. Specific Content-Types (image/png, application/pdf) still win over the extension; the override only applies to generic binary types.
Also restores the Greptile SHA-pinned raw.githubusercontent.com URL on the file_id integration test so we test against the same hosting real users hit, no third-party CDN.
Anthropic's URL fetcher intermittently returns 400 'Unable to download
the file' for the Wikipedia URL the test was using. Point it at the
repo's existing tests/llm_translation/fixtures/dummy.pdf via raw
GitHub instead — small, deterministic, reliably fetchable.
With a stable URL the test no longer needs to be opted out of VCR;
remove it from the incompatible list so it can replay from cassette.
Resolve URL conflict: keep Greptile's commit-SHA pin for immutability, but route through jsDelivr (cdn.jsdelivr.net/gh/BerriAI/litellm@<sha>/...) so the response Content-Type is application/pdf instead of application/octet-stream. Without this, OpenAI / Gemini / Router PDF tests reject the inlined file_data with "unsupported MIME type 'application/octet-stream'".
A test that produces non-deterministic request bodies (e.g. uuid in
the prompt) under record_mode=new_episodes never replays — every CI
run appends fresh unmatched episodes. The cassette grows unbounded
over time and silently inflates Redis (we observed one cassette at
22 episodes / ~860KB after ~5 CI runs).
Refuse the save when episode count exceeds MAX_EPISODES_PER_CASSETTE
so the pathology surfaces with a loud warning that points to the
opt-out fix instead of festering invisibly.
The previous URL switch to raw.githubusercontent.com fixed Anthropic's "Unable to download" failure but caused OpenAI / Gemini / Router PDF tests to fail with "unsupported MIME type 'application/octet-stream'": those providers download the URL and inline it as data:<Content-Type>;base64,..., and raw.githubusercontent.com serves PDFs as application/octet-stream.
jsDelivr proxies the same in-repo fixture (cdn.jsdelivr.net/gh/BerriAI/litellm@main/...) and returns the correct Content-Type: application/pdf, so all providers (Anthropic forwards the URL natively; OpenAI/Gemini/Bedrock fetch and inline) get the right MIME type without changing transformer code.
Some tests can't benefit from cassette replay because they assert on
state that only exists in the live provider between two calls (e.g.
prompt-cache propagation, intermittent provider quirks). Marking them
with @pytest.mark.vcr just wastes cycles trying to record cassettes
they will never replay against successfully.
Opt-out by nodeid suffix so subclassed/parametrized variants are
covered:
- ::test_prompt_caching — Anthropic/Bedrock prompt-cache propagation
isn't deterministic in the 0–1s window the test gives it.
- ::test_async_pdf_handling_with_file_id — flaky upstream Wikipedia
fetch through the Anthropic Files API.
- TestBedrockInvokeNovaJson::test_json_response_pydantic_obj —
Bedrock Nova returns tool_call vs JSON nondeterministically (other
providers' subclasses are healthy).
- ::test_bedrock_converse__streaming_passthrough — Bedrock streaming
response_cost calc returns None intermittently.
These tests keep their existing @pytest.mark.flaky retry behavior.
A test that fails (incl. all the failing retries before a passing one)
can otherwise overwrite a known-good cassette with a 'bad luck'
recording. Tests like test_prompt_caching, which assert on provider
state across two calls, can produce a 200 response that semantically
fails the assertion — the 2xx filter doesn't catch this because the
HTTP layer is fine.
- pytest_runtest_makereport hook attaches each phase report to the
pytest item.
- _vcr_outcome_gate fixture (combining the verbose-mode reporter)
reads the call-phase outcome at teardown and informs the persister
via mark_test_outcome_for_cassette before vcrpy's Cassette.__exit__
triggers save_cassette.
- save_cassette consults the per-key 'did the test pass?' flag and
short-circuits when False, leaving any prior good recording intact.
- Defaults to passed=True when no marker is present so non-test
usage of the persister still works.
Set LITELLM_VCR_VERBOSE=1 to print a one-line cassette verdict per
test (HIT / MISS / PARTIAL / NOOP) showing replay vs new-recording
counts. Useful for local QA to confirm which tests actually exercised
the cache and which fell through to the live provider.
Managed Redis (e.g. Upstash) drops idle TLS connections, which surfaced
in CI as a teardown ERROR on test_gemini_image_size_limit_exceeded:
redis.exceptions.ConnectionError: EOF occurred in violation of
protocol (_ssl.c:2427)
Cassette persistence is a cache, not test correctness, so:
- Configure the redis client with Retry(ExponentialBackoff, retries=2)
on ConnectionError/TimeoutError to absorb single-socket drops.
- Wrap save_cassette so a final failure logs a warning instead of
failing teardown — the next run re-records.
- Wrap load_cassette so an outage on read becomes a cache miss
(CassetteNotFoundError) instead of erroring in setup.
Stop falling back to REDIS_URL/REDIS_SSL_URL/REDIS_HOST for the VCR
persister. Sharing a Redis with the application cache risks cassettes
being wiped by tests that flush the app Redis.
The test was passing the Wikipedia URL https://upload.wikimedia.org/wikipedia/commons/2/20/Re_example.pdf as the file_id, which Anthropic's URL fetcher can no longer download (returns "Unable to download the file"). The URL is healthy for generic clients but Anthropic's fetcher fails on it deterministically, so the test has been red across PRs on litellm_internal_staging.
Switch to the in-repo fixture at tests/llm_translation/fixtures/dummy.pdf served via raw.githubusercontent.com - same fixture used elsewhere in the repo, no external CDN dependency that can block by user-agent.
Stop forcing Gemini 3 thinkingLevel for Anthropic-style thinking params by default, and gate legacy low/minimal mapping behind an explicit feature flag to avoid provider-default confusion.
Made-with: Cursor
Trailing slashes on custom API base examples cause double-slash in
get_complete_url. Also fixes inconsistent list indentation in
test_crusoe_models_configuration.
- Remove trailing slash from docs Base URL to match providers.json
- Wrap model_cost mutations in try/finally to prevent test state leakage
- Add missing __init__.py to crusoe test package
Replace hand-written CrusoeChatConfig class and manual registrations
across constants.py, __init__.py, get_llm_provider_logic.py, and
_lazy_imports_registry.py with a single entry in
litellm/llms/openai_like/providers.json, consistent with the
recommended pattern for OpenAI-compatible providers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
vcrpy's aiohttp stub captures response bodies via 'await response.read()',
which drains aiohttp's StreamReader. Downstream consumers of the same
ClientResponse (litellm's AiohttpResponseStream, which iterates
response.content.iter_chunked) then see an empty body and surface as
JSON 'Expecting value: line 1 column 1 (char 0)' errors on every
record-path call.
The previous workaround set litellm.disable_aiohttp_transport=True for
the whole VCR-active session, which made the tests exercise pure httpx
instead of the production aiohttp transport. That hid the production
transport from coverage and surfaced its own bugs (e.g. the Azure
DELETE-with-empty-body case fixed in upstream staging).
Replace the workaround with a targeted monkey-patch that re-feeds the
captured body into the StreamReader via unread_data after vcrpy records
it. Tests now run through the same transport customers do, both on
first record and on replay, for both unary and streaming endpoints.
Verified locally against api.anthropic.com with the production
LiteLLMAiohttpTransport: record path passes (real network, 4.2s),
replay path passes (Redis cache, 1.8s).
The Anthropic replay tests hardcoded specific token counts and content
strings ('Hello! How can I help you today?', prompt_tokens == 12). On a
fresh CI Redis those values must match a pre-recorded cassette that
doesn't exist, so the first run hits the live API and gets different
real bytes back.
Assert on shape instead: non-empty content, positive token counts,
finish_reason in the known set, and (for streaming) more than one chunk.
The tests still exercise the full transformation pipeline end-to-end and
catch shape regressions; drift in the exact text/token counts is
expected and now tolerated.
litellm's default LiteLLMAiohttpTransport routes requests through aiohttp,
which sits below httpx and is invisible to vcrpy's httpx-stub interception.
Under vcrpy + aiohttp, requests reach the real network but responses come
back through the stubbed httpx transport as empty 200s, surfacing as
'Unable to get json response - Expecting value: line 1 column 1 (char 0)'
in providers like Anthropic, Gemini, and any other path that exercises the
aiohttp transport.
Disabling the aiohttp transport when the VCR persister is registered
forces all calls through pure httpx, which vcrpy can record and replay
correctly.
record_mode='once' refused to add new requests once any cassette
existed in Redis. Combined with filter_non_2xx_response (which drops
non-2xx responses from the saved cassette) and a 24h shared-Redis TTL,
a single transient API failure mid-test left the cassette stuck with
only the leading non-API requests (e.g. the model_prices fetch from
raw.githubusercontent.com), and every subsequent run for the next 24h
errored with 'Can't overwrite existing cassette'.
new_episodes records anything not already present, so partially
populated cassettes recover on the next run instead of poisoning the
suite for a full TTL window.
Provider SDKs already retry transient 5xx/429 with exponential backoff
(default max_retries=2), and pytest.mark.flaky covers test-level
retries on top of that. Setting litellm.num_retries=3 here just
multiplied the existing layers — worst case 6 (flaky) x 3 (this) x
2 (CI rerunfailures) = 36 attempts on a single test.
Removing it keeps SDK-level network-blip protection intact and
shortens worst-case latency on cache-miss runs.
Removes commentary that restated the code, including:
- module-level banners explaining what the conftest does (covered by
Readme.md and the function bodies)
- docstrings on _scrub_response, _before_record_response, vcr_config,
_vcr_disabled, pytest_recording_configure (function names + bodies
are self-evident)
- inline notes about header filtering, match_on, etc.
- per-test docstrings restating the test name
Keeps the two non-obvious notes that aren't recoverable from the code:
the vcrpy/respx httpx-transport collision rationale on
_RESPX_CONFLICTING_FILES, the vcrpy "return None to skip persisting"
contract on filter_non_2xx_response, and the fixture-ordering
dependency on _vcr_record_retries.
Removes the YAML cassette feature entirely and replaces it with a
Redis-only flow. Every test in tests/llm_translation/ and
tests/llm_responses_api_testing/ is auto-marked @pytest.mark.vcr via
conftest.pytest_collection_modifyitems, so any provider call lands in
the Redis cache (litellm:vcr:cassette:<rel_path>, 24h TTL). First run
records, runs within the day replay, day rollover re-records and
surfaces upstream API drift within 24h.
VCR is on by default. Set LITELLM_VCR_DISABLE=1, or simply leave
REDIS_HOST unset, to opt out — both bypass the auto-marker entirely so
nothing about cassettes runs. record_mode is "once" so cache-miss
records and cache-hit replays.
The 8 existing respx-using files in tests/llm_translation are excluded
from the auto-marker (vcrpy and respx both patch the httpx transport;
applying both makes one silently win). The persister's own unit-test
file is also excluded so it doesn't recursively run inside a cassette.
The persister moved from tests/llm_translation/_vcr_redis_persister.py
to tests/_vcr_redis_persister.py so both conftests share it. The two
demo tests in test_anthropic_completion_vcr.py were ported into
test_anthropic_completion.py and the demo file was deleted.
Adds tests/_flush_vcr_cache.py + a Make target
(test-llm-translation-flush-vcr-cache) that scans
litellm:vcr:cassette:* and pipelines DELETEs, for the
"I want the next CI run to re-record now" workflow. Drops the now-dead
test-llm-translation-record target.
Provider keys are still required on cache-miss (which happens on first
run and once a day after that). Replay-mode runs need only Redis.
Stores VCR cassettes in Redis under litellm:vcr:cassette:<rel_path> with
a 24h expiry instead of YAML on disk. The TTL means each daily CI run
starts with an aged-out cache, naturally re-records against live providers,
and surfaces upstream API drift within a day without a manual `make`
re-record sweep. Opt-in via LITELLM_VCR_REDIS=1; default behaviour is
unchanged so local dev keeps the on-disk cassettes.
before_record_response now drops non-2xx responses so a transient 5xx or
429 from a provider can't poison the cache for the rest of the TTL window.
Vcr-marked tests bump litellm.num_retries to 3 during recording so
provider-SDK exponential backoff kicks in on the cache-miss path.
Tests cover the three surfaces we depend on in CI: serialize/deserialize
roundtrip via the real vcrpy serializer, TTL is actually applied to saved
keys, cache miss raises CassetteNotFoundError so vcrpy falls through to
record mode, and 2xx-only filtering across the status-code matrix
(2xx kept, 3xx/4xx/5xx dropped, with 429 and 503 explicitly pinned).
CI's license check fails on the new dev dep because liccheck cannot read
the PEP 639 'License-Expression' field that pytest-recording uses. Add
the package to the manually-verified allowlist (MIT, confirmed via PyPI
classifier).
Also addresses greptile P2 review comments:
- Add 'anthropic-version' to the request-header filter list so live and
mock recordings produce structurally identical cassettes.
- Replace the indentation-sensitive regex in
'_strip_nondeterministic_headers' with a YAML parse-and-rewrite so the
helper keeps working if vcrpy ever changes its serialization style.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
