mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
b637d9f64a
Convert the per-test VCR verdict line from a single 'NOOP / HIT / MISS /
PARTIAL' tag into a classified outcome that distinguishes the cases that
silently bill the live API on every CI run from the ones that don't:
HIT pure replay
PARTIAL mixed replay + new recordings
MISS:RECORDED new cassette saved to Redis (cached next run)
MISS:OVERFLOW cassette > MAX_EPISODES_PER_CASSETTE; persister
refused to save; re-bills every run
MISS:NOT_PERSISTED test failed; save_cassette skipped; re-bills
NOOP VCR-marked but no HTTP traffic (mocked elsewhere)
UNMARKED:LIVE_CALL test bypassed VCR AND opened a TCP connection
to a known LLM provider host -> wasted spend
UNMARKED:NO_TRAFFIC test bypassed VCR but didn't call out
The UNMARKED:LIVE_CALL signal is what converts 'this test probably hits
live' into 'this test connected to api.openai.com'. We install a
socket.connect / socket.create_connection wrapper for the duration of
each non-VCR-marked test and record any outbound TCP to a known LLM
provider hostname. The probe sits below the httpx layer so vcrpy and
respx (which both patch above the socket) are unaffected.
Replace the file-level _RESPX_CONFLICTING_FILES blacklists in the
llm_translation and local_testing conftests with per-item respx
detection in apply_vcr_auto_marker_to_items. A test now skips VCR when
it actually carries @pytest.mark.respx or has respx_mock in its fixture
chain - not just because some other test in the same file imports
MockRouter. Items skipped by skip_files are split into respx_conflict
(real conflict, the module wires up respx) vs file_opt_out (dead skip-
list entry whose module never touches respx) so the session summary
makes pruning obvious.
Stabilize the AWS SigV4 fingerprint: the Authorization header on
Bedrock requests rotates its Credential date and Signature on every
call, which previously pushed every Bedrock test past the 50-episode
overflow threshold. Extract the access-key id only
('aws-sigv4:AKIA...') so two requests with the same identity match.
Always emit verdict logging when VCR is active (set
LITELLM_VCR_VERBOSE=0 to opt back into the legacy quiet mode). Add a
session-end classification summary that lists overflow tests, unmarked
live-call tests, and the skip-reason breakdown.
Wire the live-call probe + summary hook into every test directory that
already uses the Redis-backed VCR cache (audio_tests, guardrails_tests,
image_gen_tests, litellm_utils_tests, llm_responses_api_testing,
llm_translation, local_testing, logging_callback_tests, ocr_tests,
pass_through_unit_tests, router_unit_tests, search_tests,
unified_google_tests).
Add tests/llm_translation/test_vcr_classification.py covering the
verdict classifier, skip-reason tagging, AWS SigV4 fingerprint stability,
live-host classification, and session summary rendering.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| gcs_pub_sub_body | ||
| langfuse_expected_request_body | ||
| base_test.py | ||
| conftest.py | ||
| create_mock_standard_logging_payload.py | ||
| gettysburg.wav | ||
| log.txt | ||
| test_alerting.py | ||
| test_amazing_s3_logs.py | ||
| test_assemble_streaming_responses.py | ||
| test_bedrock_knowledgebase_hook.py | ||
| test_built_in_tools_cost_tracking.py | ||
| test_custom_callback_router.py | ||
| test_datadog.py | ||
| test_datadog_llm_obs.py | ||
| test_dynamic_otel_keys.py | ||
| test_gcs_pub_sub.py | ||
| test_generic_api_callback.py | ||
| test_humanloop_unit_tests.py | ||
| test_langfuse_dynamic_credentials.py | ||
| test_langfuse_e2e_test.py | ||
| test_langfuse_unit_tests.py | ||
| test_langsmith_dynamic_credentials.py | ||
| test_langsmith_unit_test.py | ||
| test_log_db_redis_services.py | ||
| test_logging_redaction_e2e_test.py | ||
| test_moderations_api_logging.py | ||
| test_opentelemetry_unit_tests.py | ||
| test_otel_logging.py | ||
| test_pagerduty_alerting.py | ||
| test_posthog.py | ||
| test_spend_logs.py | ||
| test_sqs_logger.py | ||
| test_standard_logging_payload.py | ||
| test_standard_logging_payload_excluded_fields.py | ||
| test_token_counting.py | ||
| test_unit_test_litellm_logging.py | ||
| test_unit_tests_init_callbacks.py | ||
| test_view_request_resp_logs.py | ||