* fix(tests): use gpt-realtime in realtime guardrails test
OpenAI shut down gpt-4o-realtime-preview-2024-12-17 on 2026-05-07, so
the live OpenAI realtime guardrails integration test now fails with
model_not_found (session.created never arrives, _wait_for_event times
out). Point OPENAI_REALTIME_URL at the current GA model, gpt-realtime.
Scope limited to this test: the pricing-catalog JSON keeps the retired
entries intentionally (historical cost calc + separate Azure timeline),
and the Azure realtime cost-calc test is unaffected.
* fix(tests): mock nvidia_nim rerank instead of hitting EOL'd endpoint
NVIDIA reached end-of-life for the hosted nvidia/llama-3.2-nv-rerankqa-1b-v2
rerank API on 2026-05-18 with no published replacement, so the live
BaseLLMRerankTest.test_basic_rerank for nvidia_nim now returns HTTP 410
("Gone"). NVIDIA's hosted catalog rotates on a schedule, so swapping in
another live model would only defer the failure.
Override test_basic_rerank in TestNvidiaNim to mock the sync/async HTTP
transport (same pattern as test_nvidia_nim_rerank_ranking_endpoint in this
file) and inject a fake NVIDIA_NIM_API_KEY via monkeypatch. The
request/response transformation and cost calculation stay covered offline.
Scope limited to nvidia_nim; other BaseLLMRerankTest providers untouched.
* fix(tests): migrate remaining realtime tests off shut-down gpt-4o-realtime-preview
OpenAI's 2026-05-07 shutdown removed the entire gpt-4o-realtime-preview
family, including the undated 'gpt-4o-realtime-preview' alias (not just the
dated snapshot fixed earlier). Three live tests still connected with the
dead alias and failed with messages_received=1 (an error event instead of
session.created):
- test_openai_realtime_simple.py: get_model() -> gpt-realtime (drives
TestOpenAIRealtime.test_realtime_connection / test_realtime_with_query_params)
- test_openai_realtime.py: test_openai_realtime_direct_call_no_intent and
test_openai_realtime_direct_call_with_intent -> openai/gpt-realtime
(the with_intent test shares the same dead alias even though it was not
in the failing set this run)
Mocked unit tests (test_realtime_query_params_construction,
test_realtime_query_params_use_normalized_model_name) are left as-is: they
never hit the network and assert string plumbing only.
Also fixes test_text_message_blocked_by_guardrail_no_ai_response, which now
connects (the earlier URL swap worked) but tripped a model-wording-brittle
assertion. The guardrail flow asks the model to voice the block message
verbatim; gpt-4o-realtime-preview complied (output contained 'blocked'),
gpt-realtime refuses verbatim-repeat instructions ('I'm sorry, but I can't
repeat that message.'). Since the original user message is blocked before
it reaches OpenAI, the refusal is still a safe outcome. Assertion #3 now
accepts both voicing and refusal, and adds a hard check that the blocked
phrase never leaks into AI output.
* fix(tests): stabilize image-edit VCR cassettes to stop live gpt-image-1 spend
The image-edit cassettes for ``gpt-image-1`` were accumulating >50
episodes and being refused by the persister
(``tests/_vcr_redis_persister.py``), so every CI run was hitting the
real OpenAI endpoint. The async parametrize was the clearest tell:
``test_openai_image_edit_litellm_sdk[True]`` cached to 1 entry, but the
``[False]`` (async) sibling grew to 51 entries and never replayed.
Two non-deterministic sources were fueling the growth, both fixed
here. After this patch, the cassettes settle at one episode per
unique call and replay for the 24-hour TTL like every other suite.
1. Pin httpx's multipart boundary at the source. The existing
``_normalize_multipart_boundary`` rewrites the boundary in the
``Content-Type`` header reliably, but on the async transport path
the body is not always a contiguous ``bytes`` object when
``before_record_request`` runs, so the body-side replacement
silently no-ops and the recorded cassette retains the random
``boundary=<hex>`` string. The next CI run gets a fresh random
boundary, the ``safe_body`` matcher misses, and
``record_mode="new_episodes"`` appends another episode. Wrapping
``httpx._multipart.MultipartStream.__init__`` so it always uses
``vcr-static-boundary`` when no boundary is supplied eliminates
the variance for both sync and async paths and leaves the normalizer
in place as a backstop. Exposed as
``pin_httpx_multipart_boundary`` so other multipart-heavy suites
(audio, ocr, batches) can adopt the same fixture later.
2. Pass raw ``bytes`` (not ``BytesIO`` streams) through the
image-edit fixtures. A ``BytesIO`` whose file pointer is at EOF
after the first multipart upload silently encodes an empty image on
the next SDK / Router retry — yet another divergent body that VCR
records as a new episode. ``bytes`` are immutable and position-less,
so retries re-encode an identical payload every time. This is also
a small production-correctness improvement: a customer passing
``BytesIO`` today would hit the same empty-body retry bug. The
BytesIO-specific smoke test
(``test_openai_image_edit_with_bytesio``) is preserved by giving
``get_test_images_as_bytesio`` its own factory instead of aliasing
the bytes one.
3. Add ``scripts/flush_image_edit_vcr_cassettes.py`` — a one-shot
Redis SCAN/DEL helper that clears the bloated pre-fix cassettes
under ``litellm:vcr:cassette:tests/image_gen_tests/test_image_edits/*``.
Without this, the next CI run still loads the existing 51-entry
cassette, the new fixed-boundary body still doesn't match any of
the stale entries, the persister still refuses to save, and the
bleed continues. Run once with the production
``CASSETTE_REDIS_URL`` after merge (dry-run by default).
* DIAGNOSTIC: log VCR body mismatches + per-episode body hashes
Temporary observability boost so we can root-cause why
``test_image_edits.py`` async parametrizes still record fresh
episodes on every CI run even though the multipart boundary is now
pinned (sync parametrizes cache cleanly as VCR HIT). The matcher
currently raises ``AssertionError("request bodies differ")`` with
zero context, so we cannot tell whether the live body genuinely
varies, the matcher is comparing a bytes object to a stream object,
or the normalizer is silently skipping the body because it is not
bytes/str.
Three logs added; the first two are worth keeping permanently, the
third is intended to be reverted after the diagnosis lands:
1. ``_safe_body_matcher`` now emits a structured stderr block on
mismatch (type of each side, length, SHA-256, first divergent
byte offset, ±100-byte window). Always-on -- mismatches are
signal, not noise, and the existing per-test verdict already
logs once per test. PERMANENT.
2. ``_normalize_multipart_boundary`` now logs to stderr when the
body type is not bytes/bytearray/str -- the silent ``else:
return`` branch was masking exactly the case we suspect is
firing on async (httpx ``MultipartStream`` handed to vcrpy
before the body is read). PERMANENT.
3. ``_RedisPersister.save_cassette`` now logs every episode's body
SHA-256, length, and 120-byte preview at save time. This lets
two consecutive CI runs be diffed: if the same test records a
different hash run-to-run, the live body genuinely varies; if
both runs record the same hash but the matcher still misses, the
bug is in the matcher itself. TEMPORARY -- revert once the
async variance is identified and fixed.
Once a single ``image_gen_testing`` CI run produces these logs,
revert this commit (or just the persister hash block) with a force
push so the cassette save path is not noisy in steady-state.
* DIAGNOSTIC: route VCR diagnostics through per-PID files (bypass xdist capture)
Re-push of the diagnostic logging from the previous commit, this
time wired so the output actually survives to the CI log. xdist
captures stdout/stderr from every passing test in the worker
process; the body-matcher and normalizer-skip diagnostics fire from
inside vcrpy machinery during the test, so for any test that
ultimately passes (which is all of them once the cassettes are
recorded), the diagnostic lines are silently swallowed.
Fix: write each diagnostic line to a per-PID file under
``test-results/vcr-diagnostics/<pid>.log`` instead of writing to
stderr. The controller's ``pytest_terminal_summary`` aggregates
those files and writes them through ``terminalreporter.write_line``,
which is not subject to per-test capture. As a bonus,
``test-results/`` is already collected by the ``store_test_results``
step in CircleCI, so the raw per-worker logs survive as build
artifacts even after the test session ends.
Three call sites updated:
1. ``_emit_body_mismatch_diagnostic`` (matcher) -- writes the
structured type/length/sha/window block via ``vcr_diag_write_line``.
2. ``_normalize_multipart_boundary`` -- logs the silent-skip path
(body not bytes/bytearray/str) the same way.
3. ``_maybe_log_episode_body_hashes`` (persister) -- replaces the
``_log.warning`` calls (which the root-logger config also
swallows in CI) with ``vcr_diag_write_line``.
Image-gen conftest is the only suite wired to dump the aggregated
log at session end. Other suites can opt in by adding
``emit_vcr_diagnostic_log(terminalreporter)`` to their own
``pytest_terminal_summary``. The diagnostic dir is cleared at the
start of each session (controller-only) so a local rerun does not
mix output from prior runs.
Same revert plan as the previous diagnostic commit: keep the
matcher + normalizer skip diagnostics permanently (they only fire
on signal events), revert the persister body-hash dump once the
async variance is identified.
* fix(tests): coalesce iterable request bodies before matching/recording
Root cause of the residual async image-edit cassette leak. The
diagnostic run for ``ba3915d9`` printed:
[vcr-safe-body-matcher] request body mismatch
body[a]: type='list_iterator' length=unknown sha256=N/A
body[b]: type='list_iterator' length=unknown sha256=N/A
httpx's async transport hands vcrpy a ``request.body`` that is a
``list_iterator`` over multipart chunks rather than a contiguous
``bytes`` blob. Two consequences:
1. ``_safe_body_matcher`` compares the two iterator objects with
``==``, which is identity comparison for arbitrary iterators -
semantically identical multipart bodies never compare equal, and
``record_mode="new_episodes"`` appends a new episode on every CI
run until the cassette crosses ``MAX_EPISODES_PER_CASSETTE`` and
the persister refuses to save (this is exactly what the OVERFLOW
warning has been catching).
2. ``_normalize_multipart_boundary`` short-circuits its
``else: return`` branch because the body is neither bytes nor
str, so any residual random boundary characters in the body bytes
are never rewritten.
Sync requests do not hit this code path: httpx's sync transport
hands vcrpy a single ``bytes`` body, so ``==`` works and the
boundary normalizer runs as intended. That is why
``test_openai_image_edit_litellm_sdk[True]`` records to ``entries=1``
and replays cleanly while ``[False]`` (async) kept growing by one
episode per run.
Fix: add ``_materialize_iterable_body`` which coalesces an iterable
``request.body`` into ``bytes`` in-place. Call it from two places:
* The top of ``_before_record_request``, so the boundary normalizer
and the cassette serializer both see bytes from then on.
* The top of ``_safe_body_matcher``, as defense in depth in case a
future vcrpy code path invokes the matcher without first going
through ``_before_record_request``.
The vcrpy ``Request`` is a wrapper used for matching and recording;
the underlying httpx transport sends its own request body
separately, so replacing the iterator on the vcrpy wrapper does
not starve the live HTTP send.
After this lands the async parametrizes should flip from
``[VCR MISS:RECORDED] entries=N+1`` to ``[VCR HIT] entries=N`` on
the next CI run, matching the sync side and dropping the residual
~$3/day to $0.
* fix(tests): handle bytes_iterator + never leave an exhausted body
Follow-up to 8e08272b. The previous attempt at coalescing iterable
request bodies bailed out (``return`` without writing
``request.body``) whenever it could not classify the chunk type.
That was the wrong failure mode for one critical case: vcrpy
sometimes presents the body as ``iter(some_bytes)``, whose Python
type is ``bytes_iterator`` and which yields ``int`` byte values
(0-255), not byte chunks. The old code saw an ``int`` chunk, hit
the ``else: return`` branch, and left ``request.body`` pointing at
the now-exhausted iterator.
The post-fix diagnostic run made this loud:
[vcr-safe-body-matcher] request body mismatch
body[a]: type='bytes_iterator' length=unknown sha256=N/A
body[b]: type='bytes_iterator' length=unknown sha256=N/A
Every async image-edit test then ballooned from entries=2 to
entries=10 in that single CI run -- the exhausted iterator meant
the live multipart upload went out as an empty body, OpenAI
returned 400, the SDK + flaky retries fired, each retry got a
fresh iterator that my hook exhausted again, and ``new_episodes``
recorded each failed attempt as a new cassette episode.
This patch:
* Recognizes ``bytes_iterator`` (chunks are ``int``) and
reconstructs the buffer via ``bytes(chunks)``.
* Keeps the existing ``list_iterator``-over-bytes-chunks handling
via ``b"".join(...)``.
* **Always writes a bytes value back to ``request.body`` after
consuming the iterator.** If the chunk shape is unrecognized,
``request.body`` is set to ``b""`` rather than left as an
exhausted iterator. That is wrong in the sense of "we lost the
body" but right in the sense of "the failure mode is now visible
(live API call sends empty body and fails fast) instead of
invisible (corrupt cassette grows silently)". Combined with the
matcher diagnostic, any future regression in this code path will
surface in the CI log immediately.
Local verification covers ``bytes_iterator``, ``list_iterator``
over bytes chunks, generator over bytes chunks, empty iterator,
already-bytes (idempotent), identical-content iterator equality
in the matcher (now matches), and differing-content iterator
inequality (still raises).
* fix(tests): clear vcrpy's sticky _was_iter flag so materialized bodies stay bytes
Actual root cause of the async image-edit cassette leak. The
previous diagnostic run produced this dead giveaway:
[vcr-episode-body-hash] ... episode[0]: body type='bytes_iterator'
is not bytes/bytearray/str -- cannot hash
[vcr-safe-body-matcher] request body mismatch
body[a]: type='bytes_iterator' length=unknown sha256=N/A
body[b]: type='bytes_iterator' length=unknown sha256=N/A
Both sides of the matcher were ``bytes_iterator`` **after** the
materializer had supposedly converted them to bytes. That made no
sense until I read vcrpy's ``Request`` class.
vcrpy's ``Request`` keeps two private flags that are set in
``__init__`` from the original body's type and **never cleared by
the setter**:
def __init__(self, method, uri, body, headers):
self._was_file = hasattr(body, "read")
self._was_iter = _is_nonsequence_iterator(body)
...
@property
def body(self):
if self._was_file: return BytesIO(self._body)
if self._was_iter: return iter(self._body)
return self._body
@body.setter
def body(self, value):
if isinstance(value, str): value = value.encode("utf-8")
self._body = value # <-- does NOT touch _was_iter / _was_file
So when httpx's async transport hands vcrpy an iterator body,
``_was_iter`` becomes ``True`` and stays there forever. Even after
``_materialize_iterable_body`` writes plain bytes via
``request.body = out``, the next read of ``.body`` re-wraps the
stored bytes in ``iter()`` -- producing a fresh ``bytes_iterator``
that compares unequal to any other ``bytes_iterator`` via object
identity. The matcher missed every time, the cassette grew by one
episode per run, and the persister saw the same iterator type when
trying to hash the body for the diagnostic log.
Fix: after writing the materialized bytes, also force
``_was_iter`` and ``_was_file`` to ``False``. vcrpy exposes no
public API for this, so we touch the private flags directly --
acknowledged as a pragmatic test-only hack with a clear unit
boundary (the only call site is ``_materialize_iterable_body``).
Local repro reproduces the exact production setup:
``Request('POST', url, iter(b'multipart-content'), {})`` on two
sides, runs the matcher, asserts HIT. Verified the matcher hits on
identical content and still raises on differing content.
Should be the last fix needed. Existing cassettes that contain
oddly-shaped bodies (lists of int chunks, etc. from the previous
``_was_iter=True`` save path) still match because the materializer
canonicalises both sides to bytes before comparison -- no fourth
re-flush required.
* revert(tests): drop the temp per-episode body-hash diagnostic
Removed now that 1c51ad13 has confirmed the root cause (vcrpy's
sticky ``_was_iter`` flag making the body getter re-wrap stored
bytes in ``iter()`` on every access). The hash dump did its job --
the post-1c51ad13 image_gen_testing run shows all five async
image-edit tests as ``[VCR HIT]`` with stable entry counts and
zero billing errors -- and is too noisy to keep on by default
(over 100 lines per session at steady state).
Kept permanently:
* ``_safe_body_matcher`` mismatch diagnostic in
``_vcr_conftest_common.py``. Only fires on a body mismatch,
which is signal worth surfacing whenever it happens.
* ``_normalize_multipart_boundary`` "skipped" log line. Same
rationale -- only fires when the body shape is something the
normalizer cannot rewrite in place.
* The ``test-results/vcr-diagnostics/<pid>.log`` per-PID file
plumbing (``vcr_diag_write_line`` /
``emit_vcr_diagnostic_log``). Useful for any future diagnostic
that needs to bypass xdist stdout/stderr capture; cheap to keep.
* chore(tests): delete unused flush script + wire VCR diagnostic dump everywhere
* Remove ``scripts/flush_image_edit_vcr_cassettes.py``. It was a
one-shot helper for the initial cassette flush; the iterator and
``_was_iter`` fixes mean no future flush should be required, and
the script was never run anywhere (the actual flushes happened
inside the CI conftest via the temp hacks that have since been
reverted).
* The matcher mismatch + normalizer skip diagnostics already write
per-PID files for every suite that imports the shared VCR
plumbing, but ``emit_vcr_diagnostic_log`` -- the controller-side
dump that surfaces those files into the CI log at session end --
was only wired into ``image_gen_tests``. Add the one-line call to
the 12 sibling conftests that already use VCR so the diagnostics
surface in any suite's terminal output if a body matcher ever
misses. No new output in steady state -- the dump is a no-op when
no diagnostics were recorded that session.
* chore(tests): trim non-essential comments per project comment policy
Strips docstrings, inline comments, and block comments that this PR
introduced where the code itself was already self-evident. Keeps the
few lines that document non-obvious behaviour (raw-bytes-not-BytesIO
rationale on the image fixtures, the per-PID-files-bypass-xdist note
on the diagnostic directory). Touches only comments this PR added --
no pre-existing comment is removed.
Net: -161 lines of comment/docstring across 3 files, no code
behaviour change.
* chore(tests): forward **kwargs in pin_httpx_multipart_boundary wrapper
Defensive against future httpx MultipartStream.__init__ adding new
optional kwargs. Without the forward, the wrapper would silently drop
them. No behaviour change today.
* chore(tests): canonicalize VCR matchers and surface shouldn't-happen branches
Bundles the "follow-up cleanup PR" into this one so it does not get
lost. Four small changes:
1. Introduce ``_canonical_body(req) -> (bytes, pre_type)`` and route
``_safe_body_matcher`` through it. The matcher now operates on
bytes by construction; the "compare two iterator objects via
``==`` and silently get object-identity semantics" failure mode
(which cost us this entire PR to diagnose) is structurally
impossible to reintroduce. ``pre_type`` is the body type *before*
canonicalization, surfaced by the mismatch diagnostic so a future
regression involving a new body shape is still visible.
2. Add a structured diagnostic to ``_key_fingerprint_matcher``. It
was previously raising a bare ``AssertionError("API key
fingerprints differ")`` with zero context -- exactly the
anti-pattern the body matcher had before this PR.
3. Surface "shouldn't-happen" branches via ``vcr_diag_write_line``:
* ``_strip_image_b64_payloads`` -- logs when ``response``,
``response['body']``, or ``response['body']['string']`` arrives
in an unexpected shape (vcrpy contract violation).
* ``_compute_key_fingerprint`` -- logs the ``"no-key"`` fallback
with the request method/URL so a stripped-auth-header bug is
visible instead of masked.
* ``_canonical_body`` -- logs its own empty-bytes fallback when a
body has a shape ``_materialize_iterable_body`` did not handle.
4. Re-introduce per-episode body-hash logging in
``_RedisPersister.save_cassette`` (was reverted in 927c5548 as
"noisy"). Quantified cost: ~25 KB of CI log per session at peak,
~ms-scale CPU, zero output in steady state (no save = no log).
Trade-off favours keeping it: lets two consecutive CI runs be
diffed by body hash, which is how we will spot the next regression
in the same class.
All call sites still work: local repro confirms iter==iter HIT,
iter!=iter raises, plain-bytes HIT, body-hash log emits via the same
per-PID file plumbing as the matcher diagnostics.
* chore(tests): symmetrize diag-log cleanup across every VCR-using conftest
``image_gen_tests/conftest.py`` was the only suite that cleared
``test-results/vcr-diagnostics/*.log`` at session start. The other 12
VCR-using conftests inherited any stale per-PID logs from a previous
local run and would dump them in the terminal summary -- harmless in
CI (fresh container) but confusing locally when running multiple
suites in sequence.
Extracts the cleanup into a ``reset_vcr_diag_dir`` helper in
``tests/_vcr_conftest_common.py`` and calls it from every VCR-using
conftest's ``pytest_configure``. Same single source of truth, no
inline duplication.
* fix(tests): gate body materialization on __next__ and strip PR comments
aiohttp/vcrpy stores the json kwarg as a dict; _materialize_iterable_body
was iterating it via __iter__ and joining the keys, replacing the request
body with concatenated key names ("textlanguageentities"). Gate on
__next__ so containers (dict/list/tuple) are left alone — only single-use
iterators like httpx's bytes_iterator / list_iterator are materialized.
Log diagnostic line when chunk type is unrecognized.
* fix(tests): JSON-encode dict bodies in canonical_body for stable matching
aiohttp stubs store the json kwarg as a dict; the fallback that compared
all dicts as b"" caused concurrent presidio analyze calls to be served
the wrong cassette episode. JSON-encode with sort_keys for stable bytes.
* fix(tests): guard emit_vcr_diagnostic_log against multi-conftest re-emission
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(tests): globalize multipart-boundary pin + stabilize whisper fixtures
Diagnostic shows audio_testing was silently re-recording 50+ live Whisper
episodes per CI run (over MAX_EPISODES_PER_CASSETTE, so the persister
refused to save). Two changes:
* Move the session-autouse _pin_multipart_boundary fixture into the
shared _vcr_conftest_common module so every VCR-using suite picks it
up via a single import. image_gen had it inline; the other 12 suites
silently lacked it.
* Replace the module-level open("rb") audio file handles in test_whisper
with cached bytes + a per-call (filename, bytes, mimetype) tuple,
mirroring the image_edits raw-bytes pattern. Stops the file-pointer-
at-EOF bug where the second test got an empty multipart body.
* chore(tests): drop per-episode body-hash dump and redundant emit guard
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Resolve conflicts in the five unrelated CI-flake fixes I previously landed
on this branch -- staging shipped stronger versions (mocked HTTP for the
Fireworks tests, mocked image-fetch for the Gemini size-limit test, switched
the openapi-compliance test to the Interaction response schema instead of
dropping the assertion). Take staging's version of all five files and drop
my now-unreachable 429-skip lines from the Gemini test that the auto-merge
left behind.
Two P2 nits flagged by Greptile on PR 28036:
1. _build_completion_kwargs() defaulted vertex_project to "vertex-check-481318"
when VERTEX_PROJECT was unset. That value is a specific GCP project that
doesn't belong to this repo, so if the env-var skip guard were ever
bypassed (misconfig, direct helper call), the test would silently issue
calls to a foreign project rather than failing loudly. Drop the fallback
and read os.environ["VERTEX_PROJECT"] directly, mirroring how
AZURE_FOUNDRY_* are handled.
2. _build_messages_kwargs() was a one-liner that returned the result of
_build_completion_kwargs() unchanged -- a dead abstraction with one
caller. Inline at the _call_messages call site and delete the helper.
litellm.ImageFetchError is a subclass of BadRequestError, so when
Wikimedia returns 429 the pytest.raises(ImageFetchError) block matches
and swallows the exception -- the outer try/except never fires. Drop the
try/except and check the captured error message for "Status code: 429"
after the raises block, calling pytest.skip in that case. Same intent,
right control flow.
Four pre-existing flakes on main that gate this branch's workflow even
though they're unrelated to the reasoning_effort_grid suite:
1. tests/local_testing/test_completion.py::test_completion_fireworks_ai
2. tests/local_testing/test_completion_cost.py::test_completion_cost_fireworks_ai[fireworks_ai/llama-v3p3-70b-instruct]
3. tests/llm_translation/test_fireworks_ai_translation.py::test_document_inlining_example[False]
The Fireworks-hosted `llama-v3p3-70b-instruct` deployment is currently
returning 404 "Model not found, inaccessible, and/or not deployed".
These tests pass when the model is deployed; the issue is upstream
capacity, not our code path. Wrap the live call in a try/except that
pytest.skip's on litellm.NotFoundError so a Fireworks deployment hiccup
no longer fails CI for unrelated PRs.
4. tests/llm_translation/test_gemini.py::test_gemini_image_size_limit_exceeded
The test fetches the 32MB "Blue Marble 2002" image from Wikimedia to
exercise the 50MB image-size cap. CI runners share an IP pool with
noisy traffic, so Wikimedia routinely returns HTTP 429. The size-limit
check never gets a chance to fire. Catch the 429 BadRequestError and
pytest.skip in that case.
None of these belong on this PR conceptually, but they're included per
request to unblock the workflow before morning.
The anthropic_messages route wraps client-side BadRequestError as
AnthropicError (a BaseLLMException subclass) with status_code=400, so
"except BadRequestError" missed those cells and they fell through to the
generic Exception arm, returning 500 instead of the expected 400.
Replace the isinstance-on-BadRequestError check with a tiny classifier
that prefers BadRequestError membership, then falls back to the exception's
status_code attribute (set by every BaseLLMException subclass), then 500.
Apply to both _call_chat and _call_messages for consistency.
Fixes the 13 CircleCI llm_translation_testing failures on
bedrock_invoke_messages cells where the effort was disabled / invalid /
empty / xhigh-on-unsupported / max-on-unsupported.
Two CI failures, both pre-existing in different ways:
1. reasoning_effort_grid: all 33 bedrock_invoke_messages cells failed with
AttributeError("module 'litellm' has no attribute 'messages'"). litellm
exposes the async Anthropic Messages entrypoint as litellm.anthropic_messages
(via "from .llms.anthropic.experimental_pass_through.messages.handler
import *" in litellm/__init__.py), not litellm.messages.acreate. Swap
the call.
2. tests/test_litellm/interactions/test_openapi_compliance.py::TestResponseCompliance::test_interaction_response_fields
asserts the live Google spec contains "steps". Google's spec has churned
through "outputs" -> "steps" -> neither, and presently carries neither.
The test broke on main as soon as upstream dropped "steps"; pulling the
key off the assert list realigns the test with the live schema. Re-add
the per-turn output field once upstream stabilizes on a name.
The openapi-compliance fix doesn't belong to this PR conceptually but is
included here per request to unblock CI before the morning.
The Content-Length header check in _process_image_response rejects the
image before the body is streamed, so the mock body never needs to be
materialized. Use an empty body instead of b"x" * 100MB (addresses
greptile/cursor review feedback).
Mock the image fetch instead of downloading a 50MB+ image from
upload.wikimedia.org. The runner was intermittently rate-limited
(HTTP 429), so the code raised "Unable to fetch image ... Status
code: 429" and the size-limit assertions failed even though
pytest.raises(litellm.ImageFetchError) still matched.
Mirror the established LargeImageClient pattern in
tests/test_litellm/litellm_core_utils/test_image_handling.py: stub
litellm.module_level_client with a response whose Content-Length
exceeds the 50MB limit and bypass SSRF validation, so the
size-limit rejection path is exercised deterministically with no
external network dependency.
The live deepseek-v3p1 call kept hitting Fireworks NOT_FOUND because
Fireworks rotates its serverless catalog and no externally-verifiable
list exists. The [False] branch also never sent an image, so it only
proved the model responded.
Mock the HTTP post (mirrors test_global_disable_flag_with_transform_
messages_helper) and assert the real behavior: #transform=inline is
appended to the PDF URL unless disabled. No network, no model
dependency, and stronger coverage than the old live test.
Fireworks removed llama-v3p3-70b-instruct from serverless, so every
live test using it now fails with NotFoundError ("Model not found,
inaccessible, and/or not deployed").
Swap the 6 references (3 files) to the currently-served
accounts/fireworks/models/deepseek-v3p1 — the canonical model in
Fireworks' current docs examples and present in LiteLLM's cost map.
test_get_model_params_fireworks_ai is a pure pricing-heuristic test
(no network) asserting the >16b branch, so it uses llama-v3p1-70b-
instruct instead to keep the "fireworks-ai-above-16b" assertion and
branch coverage intact.
- Drop the "v4" suffix throughout: it referred to the QA sweep iteration,
not this test suite. There's only one regression suite, so just call it
reasoning_effort_grid.
- Move tests/test_litellm/reasoning_effort_grid_v4/ -> tests/llm_translation/
reasoning_effort_grid/. Two reasons:
1. The parent tests/test_litellm/conftest.py installs an autouse fixture
(isolate_host_aws_config) that clears every AWS_* env var before each
test, which would silently skip every Bedrock cell.
2. tests/llm_translation/conftest.py already wires up the Redis-backed
VCR persister and auto-applies @pytest.mark.vcr to every collected
item via apply_vcr_auto_marker_to_items. Living under that conftest
means the suite gets cassette replay for free -- first CI run with
provider creds records 231 cassettes, every subsequent run replays
them with no live spend.
- Trim the suite's own conftest down to just the wire_capture fixture; the
inherited llm_translation conftest covers the VCR plumbing.
- Drop the dedicated reasoning_effort_grid_v4_e2e CircleCI job. The existing
llm_translation_testing job globs tests/llm_translation/**/test_*.py, so
the suite is gated by an existing job with no new wiring.
* test: modernize models used in CircleCI e2e test suites
Replaces obsolete models (gpt-4o, gpt-4o-mini, gpt-3.5-turbo,
claude-3-5-sonnet-20240620, claude-sonnet-4-20250514) with current
equivalents across the e2e_openai_endpoints and
proxy_e2e_anthropic_messages_tests CircleCI jobs.
- gpt-4o -> gpt-5.5 (responses API e2e tests)
- gpt-4o-mini -> gpt-5-mini (websocket responses, oai_misc_config)
- gpt-4o-mini-2024-07-18 -> gpt-4.1-mini-2025-04-14 (fine-tuning,
still actively fine-tunable)
- gpt-4 / gpt-3.5-turbo target_model_names example -> gpt-5.5 /
gpt-5-mini
- bedrock claude-3-5-sonnet-20240620 batch entry -> haiku-4-5-20251001
(also aligning oai_misc_config model_name with what
test_bedrock_batches_api.py actually requests)
- bedrock claude-sonnet-4-20250514 (deprecated, retires 2026-06-15)
-> claude-sonnet-4-5-20250929
* test: point bedrock-claude-sonnet-4 alias at Sonnet 4.6, not 4.5
Greptile/Cursor flagged that after the previous commit, the
bedrock-claude-sonnet-4 alias collided with bedrock-claude-sonnet-4.5
(both pointed to claude-sonnet-4-5-20250929). Rename to
bedrock-claude-sonnet-4.6 and point it at the Sonnet 4.6 Bedrock ID
(us.anthropic.claude-sonnet-4-6, already in the litellm model
registry) so the alias name matches the underlying model version.
* test: modernize models across remaining CI-mounted configs & tests
Expands the modernization sweep to all CircleCI-mounted proxy configs
and to test directories where the model literal is a fixture/route key
(not the test's subject).
Config changes:
- proxy_server_config.yaml: bump gpt-3.5-turbo / gpt-3.5-turbo-1106 /
gpt-4o / gemini-1.5-flash / dall-e-3 underlying models; rename
gpt-3.5-turbo-end-user-test alias to gpt-5-mini-end-user-test; bump
text-embedding-ada-002 underlying to text-embedding-3-small. User-
facing aliases (gpt-3.5-turbo, gpt-4, text-embedding-ada-002, etc.)
preserved for backward compatibility with tests.
- simple_config.yaml, otel_test_config.yaml, spend_tracking_config.yaml:
bump gpt-3.5-turbo underlying to gpt-5-mini.
- pass_through_config.yaml: claude-3-5-sonnet / claude-3-7-sonnet /
claude-3-haiku entries replaced with claude-sonnet-4-5 / claude-
haiku-4-5 / claude-opus-4-7.
- oai_misc_config.yaml: align alias name with the gpt-5-mini rename.
Test changes (proactive: claude-sonnet-4-20250514 / claude-opus-4-
20250514 retire 2026-06-15):
- tests/llm_translation/test_anthropic_completion.py: bump 3 references
+ paired Vertex AI ID to claude-sonnet-4-5.
- tests/llm_translation/test_optional_params.py: bump 2 references.
- tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py
and test_bedrock_anthropic_messages_test.py: bump router fixtures
using the deprecated model IDs.
- tests/pass_through_unit_tests/base_anthropic_messages_tool_search_test.py:
modernize docstring examples.
- tests/test_end_users.py: update references to renamed alias.
* test: modernize placeholder model literals in router_unit_tests
Mass replace_all on fixture/placeholder model literals across the
router_unit_tests/ suite (model name is a routing key / label, not the
test subject). Sub-agent sweep so far — additional commits will follow
for logging_callback_tests/, enterprise/, top-level tests/test_*.py,
and other CI-mounted dirs.
Mappings applied:
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 / claude-3-opus-20240229 /
claude-3-haiku-20240307 / claude-3-5-sonnet-20240620 ->
claude-sonnet-4-5-20250929 / claude-opus-4-7 /
claude-haiku-4-5-20251001 as appropriate
Explicitly preserved:
- gpt-4o-mini-* variants (transcribe, tts, etc.) where they're current
- gpt-4-turbo / gpt-4-vision-preview / gpt-4-0613 (subject literals)
- JSONL batch body literals
- Mock LLM response model fields (must match upstream)
- Fake/mock identifiers
* test: modernize placeholder model literals across remaining CI suites
Sub-agent sweep across logging_callback_tests/, guardrails_tests/,
enterprise/, pass_through_unit_tests/, otel_tests/,
llm_responses_api_testing/, batches_tests/, spend_tracking_tests/,
litellm_utils_tests/, unified_google_tests/, and a few top-level
tests/test_*.py files where the model literal is a fixture or
placeholder (router model_list, mock standard logging payload, mock
callback data) rather than the test's subject.
Mappings applied (see scope notes below):
- gpt-3.5-turbo -> gpt-5-mini
- gpt-4 (bare) -> gpt-5.5
- gpt-4o (bare) -> gpt-5.5 (corrected from initial gpt-5 — bare gpt-5
is not a valid OpenAI alias; only gpt-5.5 / gpt-5.4 / gpt-5.2-codex
/ gpt-5-mini exist)
- gpt-4o-mini (bare) -> gpt-5-mini
- text-embedding-ada-002 -> text-embedding-3-small
- claude-3-sonnet-20240229 -> claude-sonnet-4-5-20250929
- claude-3-opus-20240229 -> claude-opus-4-7
- claude-3-haiku-20240307 -> claude-haiku-4-5-20251001
- claude-3-5-sonnet-20240620/20241022 -> claude-sonnet-4-5-20250929
- claude-3-7-sonnet-20250219 -> claude-sonnet-4-6
- gemini-1.5-flash -> gemini-2.5-flash
- gemini-1.5-pro -> gemini-2.5-pro
Explicitly preserved (not modernized):
- llm_translation/ tests where model is the SUBJECT (provider-specific
translation/transformation logic). Only the deprecated 20250514
references were already bumped in a prior commit.
- Cost-calc / tokenizer subject tests in test_utils.py (skip-ranges
documented by the sub-agent).
- Bedrock model IDs in test_health_check.py path-stripping tests.
- JSONL batch request bodies and mock LLM response bodies (must match
upstream literal).
- Langfuse expected-request-body JSON fixtures (cost values are exact-
match-asserted; changing the model would shift response_cost).
- gpt-3.5-turbo-instruct (text-completion endpoint; no modern OpenAI
equivalent).
- Top-level tests calling the proxy through user-facing aliases
(gpt-3.5-turbo, gpt-4, text-embedding-ada-002, dall-e-3) — aliases
in proxy_server_config.yaml stay; only the underlying model was
bumped.
- tests/test_gpt5_azure_temperature_support.py (the test's whole point
is model-name handling).
- Fake / mock / openai/fake identifiers.
Notable side fixes:
- test_spend_accuracy_tests.py: UPSTREAM_MODEL now matches what
spend_tracking_config.yaml's proxy actually routes to (gpt-5-mini),
resolving a latent inconsistency.
- proxy_server_config.yaml: bare `gpt-5` alias renamed to `gpt-5.5`
(bare gpt-5 is not a valid OpenAI alias).
- test_batches_logging_unit_tests.py: explicit_models list entries
kept distinct (gpt-5-mini + gpt-5.5) after bulk rename.
* test: fix CI failures from model modernization sweep
CI surfaced 4 categories of regression from the bulk modernization:
1. Azure deployment names are customer-specific. Reverted:
- tests/litellm_utils_tests/test_health_check.py: azure/text-
embedding-3-small -> azure/text-embedding-ada-002 (the CI Azure
account does not have a text-embedding-3-small deployment).
- tests/logging_callback_tests/test_custom_callback_router.py:
same revert for two router fixtures driving aembedding.
2. gpt-5 family does not accept temperature != 1. Tests that pass a
custom temperature swapped from gpt-5-mini to gpt-4.1-mini (modern
non-reasoning OpenAI mini that still accepts temperature/logprobs):
- tests/logging_callback_tests/test_datadog.py
- tests/logging_callback_tests/test_langsmith_unit_test.py
- tests/logging_callback_tests/test_otel_logging.py
3. proxy_server_config.yaml's gpt-3.5-turbo-large alias was routing to
gpt-5.5 (a reasoning model that rejects logprobs). The proxy test
tests/test_openai_endpoints.py::test_chat_completion_streaming
exercises logprobs/top_logprobs through that alias. Bumped the
underlying model to gpt-4.1 (non-reasoning, still modern).
4. tests/logging_callback_tests/test_gcs_pub_sub.py asserts against a
pinned JSON fixture (gcs_pub_sub_body/spend_logs_payload.json) with
hardcoded model="gpt-4o" and a model-specific spend value. Reverted
the litellm.acompletion calls in the test to model="gpt-4o" so the
fixture's exact-match assertions still hold.
5. tests/pass_through_unit_tests/test_anthropic_messages_passthrough.py:
anthropic.messages.create routing to openai/gpt-5-mini returned an
empty content[0] with max_tokens=100 (reasoning-token consumption).
Swapped to openai/gpt-4.1-mini.
* test: fix Assistants API model + 2 cursor[bot] review nits
1. pass_through_unit_tests/test_custom_logger_passthrough.py: gpt-5.5
isn't accepted by the /v1/assistants endpoint
("unsupported_model"). Switch to gpt-4.1-mini (modern, Assistants-
API-supported, non-reasoning).
2. example_config_yaml/pass_through_config.yaml: the previous sweep
bumped the claude-3-7-sonnet alias to claude-opus-4-7, which is a
tier change (Sonnet -> Opus). Map to claude-sonnet-4-6 to keep the
Sonnet tier intact. (Cursor bugbot review.)
3. example_config_yaml/simple_config.yaml: model_name was left as
gpt-3.5-turbo while the underlying was bumped to gpt-5-mini, which
muddles the "simple" example. Make both sides gpt-5-mini so the
most basic example is a straight 1:1 mapping again. (Cursor bugbot
review.)
* fix: revert gpt-4/gpt-3.5-turbo alias underlying to non-reasoning models
tests/test_openai_endpoints.py::test_completion calls the proxy alias
"gpt-4" with temperature=0, and other tests call gpt-3.5-turbo with
custom temperature / logprobs / the legacy /v1/completions endpoint.
The earlier modernization mapped both aliases to gpt-5.5 / gpt-5-mini,
which are reasoning models that reject temperature != 1 and don't
expose /v1/completions. Map the aliases to gpt-4.1 / gpt-4.1-mini
(modern non-reasoning OpenAI models) instead — keeps user-facing
aliases preserved while picking a current underlying that still
supports the parameters/endpoints the tests exercise.
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint (#27943)
* docs: add one-line docstring to _disable_debugging (#27894)
Squash-merged by litellm-agent from oss-agent-shin's PR.
* Add jp. Bedrock cross-region inference profile for claude-sonnet-4-6 (#27831)
Squash-merged by litellm-agent from Cyberfilo's PR.
* Sanitize empty text content blocks on /v1/messages (#27832)
Squash-merged by litellm-agent from Cyberfilo's PR.
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint
The bedrock-mantle gateway (Claude Mythos Preview) serves the Anthropic
Messages API at /anthropic/v1/messages; /v1/messages returns 404 Not
Found. Both AmazonMantleConfig (chat/completions caller route) and
AmazonMantleMessagesConfig (anthropic-messages caller route) hardcoded
the wrong path, so every Mantle request 404'd before reaching the model.
Per the Anthropic docs: "[Claude in Amazon Bedrock] uses the Messages
API at /anthropic/v1/messages with SSE streaming."
https://platform.claude.com/docs/en/api/claude-on-amazon-bedrock
Confirmed independently against the live endpoint:
/v1/chat/completions -> 200 OK
/v1/messages -> 404 Not Found (what litellm used)
/anthropic/v1/messages -> 200 OK (Claude only)
Adds a regression test asserting both Mantle configs build the
/anthropic/v1/messages path, and updates the existing assertions that
encoded the wrong path.
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
* fix: sanitize empty text blocks in sync anthropic_messages_handler path
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com>
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
`_session_stats` is a module-level dict mutated inside `_vcr_outcome_gate`
— which runs in each xdist worker process. The controller's
`pytest_terminal_summary` then reads its own empty `_session_stats` and
bails on `if not counts: return`, so the OVERFLOW / LIVE_CALL sections
the rest of this PR adds never make it into CI logs in the dist mode CI
actually uses.
Ship a structured `vcr_outcome` payload via `user_properties` (which
xdist round-trips) and add `aggregate_report_outcome` on the controller
to fold worker outcomes into `_session_stats`. The recording process
tags `vcr_recorded_by` with `PYTEST_XDIST_WORKER` so the controller can
tell "single-process — already counted locally" apart from "produced by
a worker — needs aggregation here", and not double-count when there's
no xdist.
Covered by 9 new unit tests in test_vcr_classification.py including the
end-to-end summary render path.
Address two greptile P2 review concerns on PR #27795:
1. MISS:OVERFLOW was firing whenever total > MAX_EPISODES_PER_CASSETTE
regardless of cassette state. A cassette that grew past the cap
historically but this run only *replayed* (dirty=False) is
healthy — the persister never tries to save, so the cache state is
stable and the next run will replay too. Only flag OVERFLOW when
dirty=True (new episodes were recorded that the persister would
refuse to save). Add a regression test covering the
dirty=False + large-total case.
2. _module_uses_respx did substring matching on the module source,
which false-positives on comments / docstrings / string literals.
A comment like # Previously tried respx.mock but switched to
vcrpy would keep a file pinned on the opt-out list, defeating the
dead-import pruning goal of this PR. Replace the substring scan
with an ast.NodeVisitor (_RespxUsageVisitor) that only
counts:
- @pytest.mark.respx / @respx.mock decorators
- with respx.mock(): ... (sync + async) context managers
- respx.mock(...) calls outside a with/decorator
- function parameters / fixture names equal to respx_mock
Add tests for the comment / docstring / string-literal cases plus
each real-usage pattern.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The suffix '.bedrock-runtime.amazonaws.com' never matched real Bedrock
endpoints, which use the format 'bedrock-runtime[-fips].{region}.amazonaws.com'
(region between 'bedrock-runtime' and 'amazonaws.com'). Add an explicit
host check for that pattern so Bedrock live calls are visible to the
probe, and update the unit test accordingly. Also drop the unused
'_LIVE_CALL_PROBE_INSTALLED' module variable.
These seven test files were on _RESPX_CONFLICTING_FILES, which made the
auto-marker skip them entirely. Inspecting the source shows the only
respx artifact is a top-level 'from respx import MockRouter' that no
test ever uses - no @pytest.mark.respx, no respx_mock fixture, no
respx.mock context manager. The import is dead code left over from a
previous mocking pattern.
Now that apply_vcr_auto_marker_to_items detects respx per-item via the
marker / fixture chain (b637d9f64a), the file-level skip is no longer
needed for these files - they were the reason the OpenAI tests
(test_o3_reasoning_effort, test_streaming_response[o1/o3-mini],
TestOpenAIO1::test_streaming, TestOpenAIChatCompletion::test_web_search,
TestOpenAIO3::test_web_search, etc.) ran live every CI build despite
the cassette cache being healthy.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Convert the per-test VCR verdict line from a single 'NOOP / HIT / MISS /
PARTIAL' tag into a classified outcome that distinguishes the cases that
silently bill the live API on every CI run from the ones that don't:
HIT pure replay
PARTIAL mixed replay + new recordings
MISS:RECORDED new cassette saved to Redis (cached next run)
MISS:OVERFLOW cassette > MAX_EPISODES_PER_CASSETTE; persister
refused to save; re-bills every run
MISS:NOT_PERSISTED test failed; save_cassette skipped; re-bills
NOOP VCR-marked but no HTTP traffic (mocked elsewhere)
UNMARKED:LIVE_CALL test bypassed VCR AND opened a TCP connection
to a known LLM provider host -> wasted spend
UNMARKED:NO_TRAFFIC test bypassed VCR but didn't call out
The UNMARKED:LIVE_CALL signal is what converts 'this test probably hits
live' into 'this test connected to api.openai.com'. We install a
socket.connect / socket.create_connection wrapper for the duration of
each non-VCR-marked test and record any outbound TCP to a known LLM
provider hostname. The probe sits below the httpx layer so vcrpy and
respx (which both patch above the socket) are unaffected.
Replace the file-level _RESPX_CONFLICTING_FILES blacklists in the
llm_translation and local_testing conftests with per-item respx
detection in apply_vcr_auto_marker_to_items. A test now skips VCR when
it actually carries @pytest.mark.respx or has respx_mock in its fixture
chain - not just because some other test in the same file imports
MockRouter. Items skipped by skip_files are split into respx_conflict
(real conflict, the module wires up respx) vs file_opt_out (dead skip-
list entry whose module never touches respx) so the session summary
makes pruning obvious.
Stabilize the AWS SigV4 fingerprint: the Authorization header on
Bedrock requests rotates its Credential date and Signature on every
call, which previously pushed every Bedrock test past the 50-episode
overflow threshold. Extract the access-key id only
('aws-sigv4:AKIA...') so two requests with the same identity match.
Always emit verdict logging when VCR is active (set
LITELLM_VCR_VERBOSE=0 to opt back into the legacy quiet mode). Add a
session-end classification summary that lists overflow tests, unmarked
live-call tests, and the skip-reason breakdown.
Wire the live-call probe + summary hook into every test directory that
already uses the Redis-backed VCR cache (audio_tests, guardrails_tests,
image_gen_tests, litellm_utils_tests, llm_responses_api_testing,
llm_translation, local_testing, logging_callback_tests, ocr_tests,
pass_through_unit_tests, router_unit_tests, search_tests,
unified_google_tests).
Add tests/llm_translation/test_vcr_classification.py covering the
verdict classifier, skip-reason tagging, AWS SigV4 fingerprint stability,
live-host classification, and session summary rendering.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(tests): drop deprecated OpenAI-Beta realtime header
OpenAI deprecated the 'OpenAI-Beta: realtime=v1' header; the live
service now returns code 4000 invalid_beta with
"Unknown beta requested: 'realtime'.". Two integration tests in
tests/llm_translation/realtime/test_realtime_guardrails_openai.py
hardcoded the header and started failing across all PRs.
Library code is unaffected: the OpenAI realtime handler only
forwards 'OpenAI-Beta: realtime=v1' upstream when the proxy *client*
sends it (litellm/llms/openai/realtime/handler.py). Default proxy
behavior uses the GA protocol.
Connect to OpenAI without the deprecated header, and accept the GA
event name 'response.output_audio_transcript.delta' alongside the
beta-protocol name 'response.audio_transcript.delta' for the
transcript-delta assertion.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(logging): post_call tolerates non-JSON-serializable values
post_call() did json.dumps(original_response) without default=str, so
any provider passing a dict containing datetime/Decimal/etc. would
raise TypeError. Bedrock batch retrieval hits this with
get_model_invocation_job() responses that include datetime fields
(submitTime, lastModifiedTime, endTime), failing
tests/batches_tests/test_bedrock_files_and_batches.py::test_async_file_and_batch
across all PRs.
Pass default=str so non-serializable values fall back to str().
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(tests): mock boto3 in bedrock retrieve batch test
The test patched AsyncHTTPHandler.get, but the bedrock retrieve
handler uses boto3.client('bedrock').get_model_invocation_job
directly, so the real AWS call was being made on every run, failing
with AccessDeniedException because the hardcoded test ARN belongs to
a different AWS account.
- Mock boto3.client and BedrockBatchesConfig.get_credentials so the
test never touches AWS.
- Use status=Completed in the mock response so output_file_id is
populated (the handler intentionally leaves it None for
non-completed jobs).
- Assert the predicted per-job output object URI (matches what the
handler actually returns) instead of the bare output prefix.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* docs(tests): include GA event name in guardrail-block test docstring
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
OpenAI deprecated the gpt-4o-realtime-preview-2024-10-01 snapshot,
which caused these E2E tests to fail consistently in CI. Bump to the
unversioned gpt-4o-realtime-preview alias to match the sibling
test_openai_realtime_simple.py and stay current as OpenAI rolls the
alias forward.
- Add `_strip_image_b64_payloads` filter: rewrites `data[*].b64_json` in
image-gen responses to a 4-byte placeholder before the cassette is saved.
Image-edit and image-gen cassettes (193 MB / 184 MB / 104 MB / ...) will
shrink to <100 KB on next record. Tests assert response shape only, so
coverage is preserved.
- Add `_normalize_multipart_boundary` filter: replaces httpx's per-request
random multipart boundary with a fixed string in both Content-Type header
and body bytes. Audio-transcription / Whisper tests have been effectively
unmocked — every CI run hit live providers and was silently capped at
MAX_EPISODES_PER_CASSETTE=50. Both record and replay now see identical
bytes; the safe_body matcher works.
- Fix test_evals_api.py body poisoning: replace `int(time.time())` in eval
names with `hashlib.sha1(test_node_name)[:12]`, add a function-scoped
`managed_eval` fixture that creates and deletes the eval, and switch
`get_eval` / `update_eval` from `list_evals().data[0].id` (which made
the URL vary by run) to `managed_eval.id`. Net coverage gain: delete is
now actually exercised.
- Swap arxiv PDF URL in BaseOCRTest for the in-repo `dummy.pdf` (589 B)
served via sha-pinned jsdelivr.
- Swap etsystatic image URL in BaseLLMChatTest.test_image_url for the
in-repo LiteLLM logo (9.2 KB) served via the same jsdelivr pin.
- Add `tests/llm_translation/test_vcr_filters.py` with 14 unit tests
covering both new filters: replacement, idempotency, nesting, content-
length update, two-distinct-boundaries-converge-after-normalize, etc.
Cassettes recorded with the prior patterns will mismatch on the first CI
run after merge; recommend flushing the cassette Redis once (post-merge)
so re-records save under the new format from the start.
* test: add 24hr Redis-backed VCR cache to additional test suites
Extracts the existing llm_translation VCR plumbing into a reusable helper
(tests/_vcr_conftest_common.py) and wires it into the conftest.py files
of the test directories listed in LIT-2787:
audio_tests, batches_tests, guardrails_tests, image_gen_tests,
litellm_utils_tests, local_testing, logging_callback_tests,
pass_through_unit_tests, router_unit_tests, unified_google_tests
The same helper is also adopted by the pre-existing llm_translation and
llm_responses_api_testing conftests to remove the copy-pasted VCR setup.
Each consuming conftest:
- registers the Redis persister via pytest_recording_configure
- auto-marks collected tests with pytest.mark.vcr (skipping respx-using
files where applicable, since respx and vcrpy both patch httpx)
- gates cassette writes on test success via _vcr_outcome_gate
The cache is opt-in via CASSETTE_REDIS_URL; when unset, VCR is disabled
and tests hit live providers as before. LITELLM_VCR_DISABLE=1 still
forces a bypass for ad-hoc local runs.
Test directories that run LiteLLM proxy in Docker (build_and_test,
proxy_logging_guardrails_model_info_tests, proxy_store_model_in_db_tests)
are intentionally not included: VCR.py patches the in-process httpx
transport and cannot intercept calls made from inside a Docker container.
The installing_litellm_on_python* jobs make no LLM calls and don't
benefit from caching.
https://linear.app/litellm-ai/issue/LIT-2787/add-24hr-caching-to-additional-test-suites
* test(vcr): add safe-body matcher to handle JSONL and binary request bodies
vcrpy's stock body matcher inspects Content-Type and unconditionally
runs json.loads on application/json bodies. JSON Lines payloads (used
by the Bedrock batch S3 PUT and other upload paths) crash that with
json.JSONDecodeError: Extra data, before the matcher can return
'not a match'.
This was the root cause of the batches_testing CI job failing on
test_async_create_file once VCR auto-marking was applied to the
batches_tests directory.
Add a conservative byte-equality body matcher and use it in place of
'body' in the shared match_on tuple. The matcher is strictly more
conservative than vcrpy's default — the only thing it gives up is
'different JSON key order is treated as the same body', which doesn't
apply to deterministic litellm-built request payloads. It can never
produce a false positive that the default would have rejected, so
there is no cross-contamination risk.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): exclude tests that VCR replay actively breaks
A few tests are incompatible with cassette replay and were failing on
the latest CI run after VCR auto-marking was extended to local_testing
and logging_callback_tests:
- test_amazing_s3_logs.py (logging_callback_tests): the test asserts on
a per-run response_id that should round-trip through a real S3
PUT/LIST. vcrpy's boto3 stub intercepts the PUT and the LIST replays
stale keys, so the freshly-generated id is never found.
- test_async_embedding_azure (logging_callback_tests) and
test_amazing_sync_embedding (local_testing): the failure branches
deliberately pass api_key='my-bad-key' to assert that the failure
callback fires. We scrub auth headers from cassettes (so the bad-key
request matches the prior good-key request), and vcrpy replays the
recorded 200 — the failure callback never fires.
- test_assistants.py (local_testing): the OpenAI Assistants polling
APIs mint fresh thread/run IDs every recording session and then poll
until status=='completed'. Replays of those polled GETs can never
match a freshly-generated run id, so every CI run effectively
re-records and the suite blows past the 15m no_output_timeout.
Skip these from VCR auto-marking so they continue to hit live providers
as they did before this change. The remaining tests in each directory
still get cached.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): expand skip lists for second batch of incompatible tests
Followup to the previous commit. After re-running CI on the rebuilt
branch, three more tests surfaced as VCR-replay-incompatible:
- litellm_utils_testing :: test_get_valid_models_from_dynamic_api_key
Calls GET /v1/models with api_key='123' to assert the result is empty.
We scrub auth headers, so the bad-key request matches the prior
good-key cassette and replays the recorded model list.
- litellm_utils_testing :: test_litellm_overhead.py
Measures litellm_overhead_time_ms as a percentage of total wall-clock
time. With cached responses the upstream 'network' time collapses to
microseconds, blowing past the 40%% threshold the test asserts on.
Skip the whole file (every parametrization is at risk).
- local_testing_part1 :: test_async_custom_handler_completion and
test_async_custom_handler_embedding
Same bad-key failure-callback pattern as the already-skipped
test_amazing_sync_embedding.
- litellm_router_testing :: test_router_caching.py
Asserts on litellm's own router-level response cache by comparing
response1.id to response2.id across repeat upstream calls (test
bypasses litellm cache via ttl=0 and expects upstream to return a
*new* id). With VCR replay both upstream calls return the same
cassette body, so the ids are identical. Skip the whole file.
- logging_callback_tests :: test_async_chat_azure (preemptive)
Same shape as already-skipped test_async_embedding_azure; was masked
by upstream OpenAI rate-limit failures on baseline.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): use item.path and tighten matcher docstring
- Replace pytest's deprecated item.fspath with item.path in
apply_vcr_auto_marker_to_items so we don't emit deprecation
warnings under pytest 8.
- Clarify _safe_body_matcher docstring to reflect actual behavior
(direct == first, then UTF-8 bytes comparison, no repr fallback).
Addresses Greptile review feedback on PR #27159.
* test(vcr): swallow all RedisError on cassette save/load
Cassette persistence is strictly best-effort: any Redis-side failure
(connection blip, timeout, OutOfMemoryError when the maxmemory cap is
hit, READONLY replicas, etc.) should degrade to 'test passed but
cassette not cached' rather than fail the test on teardown.
Previously the persister only caught ConnectionError and TimeoutError,
so OutOfMemoryError — which Redis Cloud raises when the cassette cache
hits its memory cap and there are no evictable keys — propagated out of
vcrpy's autouse fixture and ERRORed otherwise-passing tests on
teardown. This caused the litellm_utils_testing CircleCI job to fail on
the latest commit's run, even though the underlying test was a unit
test that used mock_response and produced no real upstream traffic
(the cassette was dirtied by a background langfuse callback). The
rerun only succeeded because Redis evictions happened to free enough
room before the SET — i.e. it was timing-dependent flakiness.
Catch redis.exceptions.RedisError (the common base of all server- and
client-side Redis exceptions) on both save and load, and parametrize
the regression tests across ConnectionError, TimeoutError, and
OutOfMemoryError to pin the new behavior.
* test(vcr): surface cassette-cache failures with warnings + session banner
When the persister silently swallows a Redis OOM (or any RedisError) on
save/load there is otherwise no visible signal that the cache is
degraded — tests pass, the cassette just isn't persisted, and the next
session still hits the same Redis at the same near-cap memory.
Add three layers of observability so that failure mode is loud:
1. Per-process health counters ("save_failures", "load_failures", and
the last error string for each), exposed via cassette_cache_health()
and reset via reset_cassette_cache_health(). The persister
increments these in addition to logging.
2. VCRCassetteCacheWarning (UserWarning subclass) emitted via
warnings.warn() inside the persister's except block. Pytest's
built-in warnings summary at session end automatically lists every
such warning, so the failure is visible in CI logs without any
conftest-level wiring.
3. Session-end banner via emit_cassette_cache_session_banner() and a
stderr-fallback atexit handler registered from
register_persister_if_enabled(). Two states:
- red "VCR CASSETTE CACHE DEGRADED" when save_failures or
load_failures > 0
- yellow "VCR CASSETTE CACHE NEAR CAPACITY" (no failures, but
used_memory >= 85% of maxmemory) so the next session knows
the Redis is approaching OOM before any SET actually fails
Capacity comes from a best-effort INFO memory probe
(cassette_cache_capacity_snapshot) that returns None on any failure or
when maxmemory is uncapped. The atexit handler skips xdist workers so
only the controller emits.
Tests: parametrize the existing save/load swallow-error tests across
ConnectionError/TimeoutError/OutOfMemoryError, add direct tests for
the health counters and warning emission, and a new
test_vcr_conftest_common_banner.py covering banner output for every
state (silent/red/yellow/disabled/xdist-worker).
* test(vcr): bucket cassettes by API key fingerprint, drop bad-key skips
Tests that deliberately call an LLM API with a bad key (e.g. to assert
that the failure callback fires, or that check_valid_key returns False)
were being silently served the prior good-key cassette: we scrub the
real Authorization / x-api-key header from the cassette before storing
it, so a follow-up bad-key call is byte-identical to the good-key call
under the existing match_on tuple.
Add a 'key_fingerprint' custom matcher that distinguishes requests by
the SHA-256 of their API-key headers. The fingerprint is stamped into
a synthetic 'x-litellm-key-fp' header by a new before_record_request
hook, which then strips the real auth headers (we have to do the
scrubbing here instead of via vcrpy's filter_headers knob, because
filter_headers runs *first* and would erase the value we want to hash).
Bad-key requests now get a different cassette bucket than good-key
requests, so vcrpy will not replay a recorded 200 in place of the
expected 401. The fingerprint is a one-way hash of the secret, so
cassettes never contain the key.
This permanently removes the 'bad-key' category of skips:
- tests/local_testing: dropped ::test_amazing_sync_embedding,
::test_async_custom_handler_completion,
::test_async_custom_handler_embedding
- tests/logging_callback_tests: dropped ::test_async_chat_azure,
::test_async_embedding_azure
- tests/litellm_utils_tests: dropped
::test_get_valid_models_from_dynamic_api_key
Coverage: 7 new unit tests in tests/test_litellm/test_vcr_safe_body_matcher.py
covering header stripping, fingerprint determinism, no-auth bucketing,
good-vs-bad key discrimination, x-api-key (Anthropic/Azure) discrimination,
and idempotence under replay.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): drop redundant comments and docstrings
Trim narration of code that is already self-evident from function and
variable names. Keep the two genuinely non-obvious bits:
- ordering constraint between filter_headers and before_record_request,
which would invite a maintainer to re-introduce the bug if removed
- the per-directory _VCR_INCOMPATIBLE_FILES rationale, since 'why
exactly is this skipped' is not knowable from the test name alone
Also drop the 40-line commented-out drop-in conftest snippet at the
bottom of _vcr_conftest_common.py — the consuming conftests are the
canonical reference.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): make _before_record_request idempotent
vcrpy invokes before_record_request more than once per request:
can_play_response_for calls it, then __contains__ /
_responses (reached via play_response) call it again on the
result. The second invocation sees a request whose auth headers we
already stripped, so a naive recompute yields "no-key" and
overwrites the real fingerprint stored in the header.
This makes can_play_response_for and play_response disagree on
matchability — the former says "yes, we have a stored response for
this" (matching no-key to no-key) and the latter throws
UnhandledHTTPRequestError because it computes a fresh real
fingerprint that doesn't match the stored no-key.
In CI this manifested as ~30 failing tests across guardrails_testing,
audio_testing, batches_testing, image_gen_testing, llm_responses_api,
litellm_router_unit_testing, etc. Skip the recompute when the header
is already set, so re-applying the hook is a no-op.
Adds a regression test that fires the hook twice on the same dict and
asserts the fingerprint stays put.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(vcr): drop more redundant docstrings and headers
* test(vcr): enable 24hr cache for ocr_tests and search_tests
These two directories were the only non-dockerized test suites in the
build_and_test workflow that make live LLM/provider API calls but were
not VCR-enabled by this PR. Together they account for 96 tests:
- tests/ocr_tests/ (31): Mistral OCR, Azure AI OCR, Azure Document
Intelligence, Vertex AI OCR. Pure-unit tests inside the same files
(e.g. TestAzureDocumentIntelligencePagesParam) make no HTTP calls
and become benign VCR NOOPs.
- tests/search_tests/ (65): Brave, DataForSEO, DuckDuckGo, Exa,
Firecrawl, Google PSE, Linkup, Parallel.ai, Perplexity, SearchAPI,
Searxng, Serper, Tavily.
Both directories use the canonical minimal conftest pattern from
tests/audio_tests/conftest.py with no skip lists. None of the test
files use respx, none assert on per-call upstream non-determinism
(no response1.id != response2.id, no overhead-as-fraction-of-total,
no live polling), so the default match_on tuple should cache cleanly.
If a flake surfaces during the first cassette-recording CI run, we
can add a targeted skip the same way we did for the other dirs.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The Azure o-series tests were excluded from the conftest's VCR auto-marker
because of a respx/vcrpy transport-patching conflict, but the only respx
reference in the file was an unused `MockRouter` import. Drop the dead
import and remove the file from the conflict set so cassettes record on
first run and replay thereafter, eliminating the 60-95s live Azure latency
that was crashing xdist workers under --timeout=120 thread-mode timeouts.
Newer Cloudflare Workers AI models (e.g. Nemotron) emit 'response_text'
instead of 'response' on streamed chunks. The non-streaming path was
already updated to fall back to 'response_text' (#26385), but the
streaming chunk parser still only read 'response', which caused
streaming requests against those models to silently produce empty
content.
Mirror the non-streaming fallback in CloudflareChatResponseIterator.chunk_parser
and add a streaming test for the response_text shape.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
`os.path.relpath` with no `start` arg uses the current working
directory, so running pytest from a subdirectory produced a
different Redis key than running from the repo root. CI-recorded
cassettes and locally-replayed runs would silently miss each
other's cache.
Anchor the path to the repo root (derived from `__file__`) so the
key is stable regardless of CWD.
https://claude.ai/code/session_018uCx7pcrkdUJZrCVMaTdPx
Remove explanatory comments that restated what the code already says.
Kept only those that document non-obvious external contracts (the aiohttp
record-path patch's reason for re-feeding the body, and the warning
messages inside save_cassette that reach the user).
Previous attempt wrote to sys.__stderr__ from the test fixture. Under
xdist, fixtures run inside worker subprocesses whose stderr is captured
by the controller and only released to the live log on test failure —
so passing tests' verdicts were silently swallowed.
Round-trip via report.user_properties: the worker-side fixture stashes
the verdict on user_properties, xdist serializes it onto the report,
and a controller-side pytest_runtest_logreport hook writes it via the
TerminalReporter (the same plugin that emits PASSED/FAILED markers).
TerminalReporter is resolved lazily on first hook call because it's
not yet registered when conftest's pytest_configure runs.
Verified locally in both serial and xdist modes.
Raw github serves application/octet-stream which OpenAI/Gemini reject
when LiteLLM fetches the URL client-side. jsDelivr serves the same
file with content-type: application/pdf. Pin to a commit SHA so the
asset is immutable and jsDelivr can cache it for a year.
Previously, the per-test [VCR HIT/MISS/...] line was written via
TerminalReporter.write_line from inside fixture teardown. Pytest
captures that stream by default and only surfaces it on FAILED tests
(under 'Captured stdout teardown'), so passing tests' verdicts were
invisible in CI logs and the user couldn't tell whether the cache
was working.
Write directly to sys.__stderr__ so the line bypasses pytest's
capture entirely. Under xdist each worker has its own __stderr__
which CircleCI aggregates into the live job log alongside the
PASSED/FAILED markers.
GitHub serves PDFs from raw.githubusercontent.com, github.com/.../raw/..., LFS, and Releases as application/octet-stream by deliberate anti-hotlinking policy. Anyone who passes a GitHub-hosted PDF URL as an OpenAI / Gemini / Bedrock file_id hits "unsupported MIME type 'application/octet-stream'" because _process_image_response inlines the URL with whatever Content-Type the server sent.
When the server-provided Content-Type is application/octet-stream or binary/octet-stream and the URL extension maps to a known MIME type (.pdf, .png, .jpg, etc.), trust the extension instead. Specific Content-Types (image/png, application/pdf) still win over the extension; the override only applies to generic binary types.
Also restores the Greptile SHA-pinned raw.githubusercontent.com URL on the file_id integration test so we test against the same hosting real users hit, no third-party CDN.
Anthropic's URL fetcher intermittently returns 400 'Unable to download
the file' for the Wikipedia URL the test was using. Point it at the
repo's existing tests/llm_translation/fixtures/dummy.pdf via raw
GitHub instead — small, deterministic, reliably fetchable.
With a stable URL the test no longer needs to be opted out of VCR;
remove it from the incompatible list so it can replay from cassette.
Resolve URL conflict: keep Greptile's commit-SHA pin for immutability, but route through jsDelivr (cdn.jsdelivr.net/gh/BerriAI/litellm@<sha>/...) so the response Content-Type is application/pdf instead of application/octet-stream. Without this, OpenAI / Gemini / Router PDF tests reject the inlined file_data with "unsupported MIME type 'application/octet-stream'".
A test that produces non-deterministic request bodies (e.g. uuid in
the prompt) under record_mode=new_episodes never replays — every CI
run appends fresh unmatched episodes. The cassette grows unbounded
over time and silently inflates Redis (we observed one cassette at
22 episodes / ~860KB after ~5 CI runs).
Refuse the save when episode count exceeds MAX_EPISODES_PER_CASSETTE
so the pathology surfaces with a loud warning that points to the
opt-out fix instead of festering invisibly.
The previous URL switch to raw.githubusercontent.com fixed Anthropic's "Unable to download" failure but caused OpenAI / Gemini / Router PDF tests to fail with "unsupported MIME type 'application/octet-stream'": those providers download the URL and inline it as data:<Content-Type>;base64,..., and raw.githubusercontent.com serves PDFs as application/octet-stream.
jsDelivr proxies the same in-repo fixture (cdn.jsdelivr.net/gh/BerriAI/litellm@main/...) and returns the correct Content-Type: application/pdf, so all providers (Anthropic forwards the URL natively; OpenAI/Gemini/Bedrock fetch and inline) get the right MIME type without changing transformer code.
Some tests can't benefit from cassette replay because they assert on
state that only exists in the live provider between two calls (e.g.
prompt-cache propagation, intermittent provider quirks). Marking them
with @pytest.mark.vcr just wastes cycles trying to record cassettes
they will never replay against successfully.
Opt-out by nodeid suffix so subclassed/parametrized variants are
covered:
- ::test_prompt_caching — Anthropic/Bedrock prompt-cache propagation
isn't deterministic in the 0–1s window the test gives it.
- ::test_async_pdf_handling_with_file_id — flaky upstream Wikipedia
fetch through the Anthropic Files API.
- TestBedrockInvokeNovaJson::test_json_response_pydantic_obj —
Bedrock Nova returns tool_call vs JSON nondeterministically (other
providers' subclasses are healthy).
- ::test_bedrock_converse__streaming_passthrough — Bedrock streaming
response_cost calc returns None intermittently.
These tests keep their existing @pytest.mark.flaky retry behavior.
A test that fails (incl. all the failing retries before a passing one)
can otherwise overwrite a known-good cassette with a 'bad luck'
recording. Tests like test_prompt_caching, which assert on provider
state across two calls, can produce a 200 response that semantically
fails the assertion — the 2xx filter doesn't catch this because the
HTTP layer is fine.
- pytest_runtest_makereport hook attaches each phase report to the
pytest item.
- _vcr_outcome_gate fixture (combining the verbose-mode reporter)
reads the call-phase outcome at teardown and informs the persister
via mark_test_outcome_for_cassette before vcrpy's Cassette.__exit__
triggers save_cassette.
- save_cassette consults the per-key 'did the test pass?' flag and
short-circuits when False, leaving any prior good recording intact.
- Defaults to passed=True when no marker is present so non-test
usage of the persister still works.
Set LITELLM_VCR_VERBOSE=1 to print a one-line cassette verdict per
test (HIT / MISS / PARTIAL / NOOP) showing replay vs new-recording
counts. Useful for local QA to confirm which tests actually exercised
the cache and which fell through to the live provider.
Managed Redis (e.g. Upstash) drops idle TLS connections, which surfaced
in CI as a teardown ERROR on test_gemini_image_size_limit_exceeded:
redis.exceptions.ConnectionError: EOF occurred in violation of
protocol (_ssl.c:2427)
Cassette persistence is a cache, not test correctness, so:
- Configure the redis client with Retry(ExponentialBackoff, retries=2)
on ConnectionError/TimeoutError to absorb single-socket drops.
- Wrap save_cassette so a final failure logs a warning instead of
failing teardown — the next run re-records.
- Wrap load_cassette so an outage on read becomes a cache miss
(CassetteNotFoundError) instead of erroring in setup.
