scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 7
43876 commits 3745 branches 1395 tags 3.9 GiB
Commit graph

7 commits

Author SHA1 Message Date
Mateo Wang
7e13256fee
test: add 24hr Redis-backed VCR cache to additional test suites (#27159) 
* test: add 24hr Redis-backed VCR cache to additional test suites

Extracts the existing llm_translation VCR plumbing into a reusable helper
(tests/_vcr_conftest_common.py) and wires it into the conftest.py files
of the test directories listed in LIT-2787:

  audio_tests, batches_tests, guardrails_tests, image_gen_tests,
  litellm_utils_tests, local_testing, logging_callback_tests,
  pass_through_unit_tests, router_unit_tests, unified_google_tests

The same helper is also adopted by the pre-existing llm_translation and
llm_responses_api_testing conftests to remove the copy-pasted VCR setup.

Each consuming conftest:
- registers the Redis persister via pytest_recording_configure
- auto-marks collected tests with pytest.mark.vcr (skipping respx-using
  files where applicable, since respx and vcrpy both patch httpx)
- gates cassette writes on test success via _vcr_outcome_gate

The cache is opt-in via CASSETTE_REDIS_URL; when unset, VCR is disabled
and tests hit live providers as before. LITELLM_VCR_DISABLE=1 still
forces a bypass for ad-hoc local runs.

Test directories that run LiteLLM proxy in Docker (build_and_test,
proxy_logging_guardrails_model_info_tests, proxy_store_model_in_db_tests)
are intentionally not included: VCR.py patches the in-process httpx
transport and cannot intercept calls made from inside a Docker container.
The installing_litellm_on_python* jobs make no LLM calls and don't
benefit from caching.

https://linear.app/litellm-ai/issue/LIT-2787/add-24hr-caching-to-additional-test-suites

* test(vcr): add safe-body matcher to handle JSONL and binary request bodies

vcrpy's stock body matcher inspects Content-Type and unconditionally
runs json.loads on application/json bodies. JSON Lines payloads (used
by the Bedrock batch S3 PUT and other upload paths) crash that with
json.JSONDecodeError: Extra data, before the matcher can return
'not a match'.

This was the root cause of the batches_testing CI job failing on
test_async_create_file once VCR auto-marking was applied to the
batches_tests directory.

Add a conservative byte-equality body matcher and use it in place of
'body' in the shared match_on tuple. The matcher is strictly more
conservative than vcrpy's default — the only thing it gives up is
'different JSON key order is treated as the same body', which doesn't
apply to deterministic litellm-built request payloads. It can never
produce a false positive that the default would have rejected, so
there is no cross-contamination risk.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(vcr): exclude tests that VCR replay actively breaks

A few tests are incompatible with cassette replay and were failing on
the latest CI run after VCR auto-marking was extended to local_testing
and logging_callback_tests:

- test_amazing_s3_logs.py (logging_callback_tests): the test asserts on
  a per-run response_id that should round-trip through a real S3
  PUT/LIST. vcrpy's boto3 stub intercepts the PUT and the LIST replays
  stale keys, so the freshly-generated id is never found.
- test_async_embedding_azure (logging_callback_tests) and
  test_amazing_sync_embedding (local_testing): the failure branches
  deliberately pass api_key='my-bad-key' to assert that the failure
  callback fires. We scrub auth headers from cassettes (so the bad-key
  request matches the prior good-key request), and vcrpy replays the
  recorded 200 — the failure callback never fires.
- test_assistants.py (local_testing): the OpenAI Assistants polling
  APIs mint fresh thread/run IDs every recording session and then poll
  until status=='completed'. Replays of those polled GETs can never
  match a freshly-generated run id, so every CI run effectively
  re-records and the suite blows past the 15m no_output_timeout.

Skip these from VCR auto-marking so they continue to hit live providers
as they did before this change. The remaining tests in each directory
still get cached.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(vcr): expand skip lists for second batch of incompatible tests

Followup to the previous commit. After re-running CI on the rebuilt
branch, three more tests surfaced as VCR-replay-incompatible:

- litellm_utils_testing :: test_get_valid_models_from_dynamic_api_key
  Calls GET /v1/models with api_key='123' to assert the result is empty.
  We scrub auth headers, so the bad-key request matches the prior
  good-key cassette and replays the recorded model list.
- litellm_utils_testing :: test_litellm_overhead.py
  Measures litellm_overhead_time_ms as a percentage of total wall-clock
  time. With cached responses the upstream 'network' time collapses to
  microseconds, blowing past the 40%% threshold the test asserts on.
  Skip the whole file (every parametrization is at risk).
- local_testing_part1 :: test_async_custom_handler_completion and
  test_async_custom_handler_embedding
  Same bad-key failure-callback pattern as the already-skipped
  test_amazing_sync_embedding.
- litellm_router_testing :: test_router_caching.py
  Asserts on litellm's own router-level response cache by comparing
  response1.id to response2.id across repeat upstream calls (test
  bypasses litellm cache via ttl=0 and expects upstream to return a
  *new* id). With VCR replay both upstream calls return the same
  cassette body, so the ids are identical. Skip the whole file.
- logging_callback_tests :: test_async_chat_azure (preemptive)
  Same shape as already-skipped test_async_embedding_azure; was masked
  by upstream OpenAI rate-limit failures on baseline.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(vcr): use item.path and tighten matcher docstring

- Replace pytest's deprecated item.fspath with item.path in
  apply_vcr_auto_marker_to_items so we don't emit deprecation
  warnings under pytest 8.
- Clarify _safe_body_matcher docstring to reflect actual behavior
  (direct == first, then UTF-8 bytes comparison, no repr fallback).

Addresses Greptile review feedback on PR #27159.

* test(vcr): swallow all RedisError on cassette save/load

Cassette persistence is strictly best-effort: any Redis-side failure
(connection blip, timeout, OutOfMemoryError when the maxmemory cap is
hit, READONLY replicas, etc.) should degrade to 'test passed but
cassette not cached' rather than fail the test on teardown.

Previously the persister only caught ConnectionError and TimeoutError,
so OutOfMemoryError — which Redis Cloud raises when the cassette cache
hits its memory cap and there are no evictable keys — propagated out of
vcrpy's autouse fixture and ERRORed otherwise-passing tests on
teardown. This caused the litellm_utils_testing CircleCI job to fail on
the latest commit's run, even though the underlying test was a unit
test that used mock_response and produced no real upstream traffic
(the cassette was dirtied by a background langfuse callback). The
rerun only succeeded because Redis evictions happened to free enough
room before the SET — i.e. it was timing-dependent flakiness.

Catch redis.exceptions.RedisError (the common base of all server- and
client-side Redis exceptions) on both save and load, and parametrize
the regression tests across ConnectionError, TimeoutError, and
OutOfMemoryError to pin the new behavior.

* test(vcr): surface cassette-cache failures with warnings + session banner

When the persister silently swallows a Redis OOM (or any RedisError) on
save/load there is otherwise no visible signal that the cache is
degraded — tests pass, the cassette just isn't persisted, and the next
session still hits the same Redis at the same near-cap memory.

Add three layers of observability so that failure mode is loud:

1. Per-process health counters ("save_failures", "load_failures", and
   the last error string for each), exposed via cassette_cache_health()
   and reset via reset_cassette_cache_health(). The persister
   increments these in addition to logging.

2. VCRCassetteCacheWarning (UserWarning subclass) emitted via
   warnings.warn() inside the persister's except block. Pytest's
   built-in warnings summary at session end automatically lists every
   such warning, so the failure is visible in CI logs without any
   conftest-level wiring.

3. Session-end banner via emit_cassette_cache_session_banner() and a
   stderr-fallback atexit handler registered from
   register_persister_if_enabled(). Two states:
     - red "VCR CASSETTE CACHE DEGRADED" when save_failures or
       load_failures > 0
     - yellow "VCR CASSETTE CACHE NEAR CAPACITY" (no failures, but
       used_memory >= 85% of maxmemory) so the next session knows
       the Redis is approaching OOM before any SET actually fails

Capacity comes from a best-effort INFO memory probe
(cassette_cache_capacity_snapshot) that returns None on any failure or
when maxmemory is uncapped. The atexit handler skips xdist workers so
only the controller emits.

Tests: parametrize the existing save/load swallow-error tests across
ConnectionError/TimeoutError/OutOfMemoryError, add direct tests for
the health counters and warning emission, and a new
test_vcr_conftest_common_banner.py covering banner output for every
state (silent/red/yellow/disabled/xdist-worker).

* test(vcr): bucket cassettes by API key fingerprint, drop bad-key skips

Tests that deliberately call an LLM API with a bad key (e.g. to assert
that the failure callback fires, or that check_valid_key returns False)
were being silently served the prior good-key cassette: we scrub the
real Authorization / x-api-key header from the cassette before storing
it, so a follow-up bad-key call is byte-identical to the good-key call
under the existing match_on tuple.

Add a 'key_fingerprint' custom matcher that distinguishes requests by
the SHA-256 of their API-key headers. The fingerprint is stamped into
a synthetic 'x-litellm-key-fp' header by a new before_record_request
hook, which then strips the real auth headers (we have to do the
scrubbing here instead of via vcrpy's filter_headers knob, because
filter_headers runs *first* and would erase the value we want to hash).

Bad-key requests now get a different cassette bucket than good-key
requests, so vcrpy will not replay a recorded 200 in place of the
expected 401. The fingerprint is a one-way hash of the secret, so
cassettes never contain the key.

This permanently removes the 'bad-key' category of skips:

- tests/local_testing: dropped ::test_amazing_sync_embedding,
  ::test_async_custom_handler_completion,
  ::test_async_custom_handler_embedding
- tests/logging_callback_tests: dropped ::test_async_chat_azure,
  ::test_async_embedding_azure
- tests/litellm_utils_tests: dropped
  ::test_get_valid_models_from_dynamic_api_key

Coverage: 7 new unit tests in tests/test_litellm/test_vcr_safe_body_matcher.py
covering header stripping, fingerprint determinism, no-auth bucketing,
good-vs-bad key discrimination, x-api-key (Anthropic/Azure) discrimination,
and idempotence under replay.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(vcr): drop redundant comments and docstrings

Trim narration of code that is already self-evident from function and
variable names. Keep the two genuinely non-obvious bits:

- ordering constraint between filter_headers and before_record_request,
  which would invite a maintainer to re-introduce the bug if removed
- the per-directory _VCR_INCOMPATIBLE_FILES rationale, since 'why
  exactly is this skipped' is not knowable from the test name alone

Also drop the 40-line commented-out drop-in conftest snippet at the
bottom of _vcr_conftest_common.py — the consuming conftests are the
canonical reference.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(vcr): make _before_record_request idempotent

vcrpy invokes before_record_request more than once per request:
can_play_response_for calls it, then __contains__ /
_responses (reached via play_response) call it again on the
result. The second invocation sees a request whose auth headers we
already stripped, so a naive recompute yields "no-key" and
overwrites the real fingerprint stored in the header.

This makes can_play_response_for and play_response disagree on
matchability — the former says "yes, we have a stored response for
this" (matching no-key to no-key) and the latter throws
UnhandledHTTPRequestError because it computes a fresh real
fingerprint that doesn't match the stored no-key.

In CI this manifested as ~30 failing tests across guardrails_testing,
audio_testing, batches_testing, image_gen_testing, llm_responses_api,
litellm_router_unit_testing, etc. Skip the recompute when the header
is already set, so re-applying the hook is a no-op.

Adds a regression test that fires the hook twice on the same dict and
asserts the fingerprint stays put.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(vcr): drop more redundant docstrings and headers

* test(vcr): enable 24hr cache for ocr_tests and search_tests

These two directories were the only non-dockerized test suites in the
build_and_test workflow that make live LLM/provider API calls but were
not VCR-enabled by this PR. Together they account for 96 tests:

- tests/ocr_tests/ (31): Mistral OCR, Azure AI OCR, Azure Document
  Intelligence, Vertex AI OCR. Pure-unit tests inside the same files
  (e.g. TestAzureDocumentIntelligencePagesParam) make no HTTP calls
  and become benign VCR NOOPs.
- tests/search_tests/ (65): Brave, DataForSEO, DuckDuckGo, Exa,
  Firecrawl, Google PSE, Linkup, Parallel.ai, Perplexity, SearchAPI,
  Searxng, Serper, Tavily.

Both directories use the canonical minimal conftest pattern from
tests/audio_tests/conftest.py with no skip lists. None of the test
files use respx, none assert on per-call upstream non-determinism
(no response1.id != response2.id, no overhead-as-fraction-of-total,
no live polling), so the default match_on tuple should cache cleanly.
If a flake surfaces during the first cassette-recording CI run, we
can add a targeted skip the same way we did for the other dirs.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
 2026-05-05 15:13:31 -07:00
mateo-berri
468b849072
tests(vcr): drop YAML/cassettes-directory metaphor from Redis keys 2026-04-30 23:37:17 +00:00
mateo-berri
c7d647b567
tests: drop YAML cassettes, make Redis-backed VCR the default 
Removes the YAML cassette feature entirely and replaces it with a
Redis-only flow. Every test in tests/llm_translation/ and
tests/llm_responses_api_testing/ is auto-marked @pytest.mark.vcr via
conftest.pytest_collection_modifyitems, so any provider call lands in
the Redis cache (litellm:vcr:cassette:<rel_path>, 24h TTL). First run
records, runs within the day replay, day rollover re-records and
surfaces upstream API drift within 24h.

VCR is on by default. Set LITELLM_VCR_DISABLE=1, or simply leave
REDIS_HOST unset, to opt out — both bypass the auto-marker entirely so
nothing about cassettes runs. record_mode is "once" so cache-miss
records and cache-hit replays.

The 8 existing respx-using files in tests/llm_translation are excluded
from the auto-marker (vcrpy and respx both patch the httpx transport;
applying both makes one silently win). The persister's own unit-test
file is also excluded so it doesn't recursively run inside a cassette.

The persister moved from tests/llm_translation/_vcr_redis_persister.py
to tests/_vcr_redis_persister.py so both conftests share it. The two
demo tests in test_anthropic_completion_vcr.py were ported into
test_anthropic_completion.py and the demo file was deleted.

Adds tests/_flush_vcr_cache.py + a Make target
(test-llm-translation-flush-vcr-cache) that scans
litellm:vcr:cassette:* and pipelines DELETEs, for the
"I want the next CI run to re-record now" workflow. Drops the now-dead
test-llm-translation-record target.

Provider keys are still required on cache-miss (which happens on first
run and once a day after that). Replay-mode runs need only Redis.
 2026-04-30 21:40:58 +00:00
Cursor Agent
05333e42ba
tests(llm_translation): switch to pytest-recording for marker-based bulk capture 
Per Yuneng's feedback, use a single @pytest.mark.vcr marker so one record
sweep populates cassettes for every marked test across all providers,
instead of forcing each test to bind to a hard-coded cassette path.

Changes vs. the initial scaffolding:

- Add 'pytest-recording==0.13.4' on top of vcrpy. Adopt its layout:
  cassettes live at 'cassettes/<test_module>/<test_name>.yaml', resolved
  automatically. New tests just decorate with '@pytest.mark.vcr' — no
  imports or path bookkeeping.
- Move the shared filter/match config into a 'vcr_config' fixture in
  'tests/llm_translation/conftest.py' (consumed by pytest-recording for
  every marked test in the dir). Drop the standalone 'vcr_config.py'.
- Bulk record / replay via the standard '--record-mode' CLI flag:
  'make test-llm-translation-record' now sweeps every '@pytest.mark.vcr'
  test under tests/llm_translation in one shot. Optional 'TARGET=' var
  scopes to a single file.
- Move existing cassettes to the per-test paths and update the local
  in-process Anthropic regenerator to write to the same paths.
- Refresh README + Makefile target docs to match the sweep workflow.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
 2026-04-30 18:08:57 +00:00
Cursor Agent
94b319c577
tests(llm_translation): add VCR cassette infrastructure for offline replay 
Live LLM e2e tests have been draining provider billing accounts and going
flaky on outages (LIT-2683). This change introduces vcrpy-backed cassette
replay so CI can exercise the same end-to-end LiteLLM transformation paths
without hitting the live provider:

- Add 'vcrpy==8.1.1' to the dev dependency group.
- New 'tests/llm_translation/vcr_config.py' centralises the VCR config:
  filters auth/secret headers and per-request response headers, matches on
  method+URI+body, and exposes 'LITELLM_VCR_RECORD_MODE' for re-recording.
- New 'tests/llm_translation/test_anthropic_completion_vcr.py' demonstrates
  the pattern with one non-streaming and one streaming Anthropic test that
  replay from cassettes shipped under 'cassettes/'.
- New 'tests/llm_translation/cassettes/_record_anthropic_fixtures.py' lets
  contributors regenerate the canned Anthropic cassettes against a local
  in-process mock (no API key required), and 'cassettes/README.md' documents
  the full record/replay/refresh workflow.
- New 'make test-llm-translation-record FILE=...' Makefile target to refresh
  cassettes against the live API.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
 2026-04-30 00:45:50 +00:00
Krish Dholakia
859b47f08b
LiteLLM Minor Fixes & Improvements (11/29/2024) (#6965) 
* fix(factory.py): ensure tool call converts image url

Fixes https://github.com/BerriAI/litellm/issues/6953

* fix(transformation.py): support mp4 + pdf url's for vertex ai

Fixes https://github.com/BerriAI/litellm/issues/6936

* fix(http_handler.py): mask gemini api key in error logs

Fixes https://github.com/BerriAI/litellm/issues/6963

* docs(prometheus.md): update prometheus FAQs

* feat(auth_checks.py): ensure specific model access > wildcard model access

if wildcard model is in access group, but specific model is not - deny access

* fix(auth_checks.py): handle auth checks for team based model access groups

handles scenario where model access group used for wildcard models

* fix(internal_user_endpoints.py): support adding guardrails on `/user/update`

Fixes https://github.com/BerriAI/litellm/issues/6942

* fix(key_management_endpoints.py): fix prepare_metadata_fields helper

* fix: fix tests

* build(requirements.txt): bump openai dep version

fixes proxies argument

* test: fix tests

* fix(http_handler.py): fix error message masking

* fix(bedrock_guardrails.py): pass in prepped data

* test: fix test

* test: fix nvidia nim test

* fix(http_handler.py): return original response headers

* fix: revert maskedhttpstatuserror

* test: update tests

* test: cleanup test

* fix(key_management_endpoints.py): fix metadata field update logic

* fix(key_management_endpoints.py): maintain initial order of guardrails in key update

* fix(key_management_endpoints.py): handle prepare metadata

* fix: fix linting errors

* fix: fix linting errors

* fix: fix linting errors

* fix: fix key management errors

* fix(key_management_endpoints.py): update metadata

* test: update test

* refactor: add more debug statements

* test: skip flaky test

* test: fix test

* fix: fix test

* fix: fix update metadata logic

* fix: fix test

* ci(config.yml): change db url for e2e ui testing
 2024-12-01 05:24:11 -08:00
Krish Dholakia
234185ec13
LiteLLM Minor Fixes & Improvements (09/16/2024) (#5723) (#5731) 
* LiteLLM Minor Fixes & Improvements (09/16/2024)  (#5723)

* coverage (#5713)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Move (#5714)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(litellm_logging.py): fix logging client re-init (#5710)

Fixes https://github.com/BerriAI/litellm/issues/5695

* fix(presidio.py): Fix logging_hook response and add support for additional presidio variables in guardrails config

Fixes https://github.com/BerriAI/litellm/issues/5682

* feat(o1_handler.py): fake streaming for openai o1 models

Fixes https://github.com/BerriAI/litellm/issues/5694

* docs: deprecated traceloop integration in favor of native otel (#5249)

* fix: fix linting errors

* fix: fix linting errors

* fix(main.py): fix o1 import

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view (#5730)

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view

Supports having `MonthlyGlobalSpend` view be a material view, and exposes an endpoint to refresh it

* fix(custom_logger.py): reset calltype

* fix: fix linting errors

* fix: fix linting error

* fix: fix import

* test(test_databricks.py): fix databricks tests

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>
 2024-09-17 08:05:52 -07:00