litellm's default LiteLLMAiohttpTransport routes requests through aiohttp,
which sits below httpx and is invisible to vcrpy's httpx-stub interception.
Under vcrpy + aiohttp, requests reach the real network but responses come
back through the stubbed httpx transport as empty 200s, surfacing as
'Unable to get json response - Expecting value: line 1 column 1 (char 0)'
in providers like Anthropic, Gemini, and any other path that exercises the
aiohttp transport.
Disabling the aiohttp transport when the VCR persister is registered
forces all calls through pure httpx, which vcrpy can record and replay
correctly.
The async/sync delete_response_api_handler always passed json=data into
httpx.delete, where data is {} from the transformer. httpx serializes that
to a 2-byte body. The Azure Responses DELETE endpoint now rejects any
request body with code: unexpected_body, breaking
test_basic_openai_responses_delete_endpoint on the llm_responses_api_testing
job. Build the kwargs dict and only set json= when data is truthy.
Add unit tests that patch httpx.delete and assert json/data are not in the
captured kwargs for the Azure DELETE path (sync and async).
Two Greptile review findings addressed:
1. (P1, security) The ``litellm_oauth_state`` cookie is the sole
guard against Login-CSRF in the PKCE flow but was set without the
``Secure`` attribute, so a network observer on plain HTTP could
read and replay it — bypassing the protection this PR adds.
Thread the originating ``Request`` down through
``get_sso_login_redirect`` and ``get_generic_sso_redirect_response``
and set ``Secure`` based on ``request.url.scheme == "https"``.
When no request is supplied (programmatic callers / tests) default
to ``Secure=True`` — production-safe. Local HTTP dev still works
because the request scheme is observed at runtime.
2. (P2) The cookie was set unconditionally, but the callback only
validates it inside the PKCE branch. Two concurrent SSO sessions
(one PKCE, one plain) could overwrite each other's state cookie
and produce spurious 400s for the plain-flow user.
Move the ``set_cookie`` call inside the existing
``if code_verifier and "state" in redirect_params`` block so the
cookie is only written when PKCE is active and the validation
will actually fire.
Tests cover both paths: PKCE-on (cookie set with Secure default),
PKCE-off (cookie not set), and HTTP dev request (Secure dropped so
the browser will actually attach the cookie on the callback hop).
``variant`` is user-controlled (passed through from
``litellm.video_content(variant=...)``) and was interpolated raw into
the URL query string. A value like ``thumbnail&extra=1`` would inject
additional query parameters into the upstream request — the same
class of issue this PR's path-segment encoding addresses. Wrap the
value in ``quote(value, safe="")`` so ``&`` / ``=`` / ``#`` cannot
terminate the ``variant`` value or open a new parameter.
Adds a regression test asserting that a malicious ``thumbnail&extra=1``
ends up percent-encoded in the URL, and that the legitimate
``thumbnail`` value still round-trips cleanly.
record_mode='once' refused to add new requests once any cassette
existed in Redis. Combined with filter_non_2xx_response (which drops
non-2xx responses from the saved cassette) and a 24h shared-Redis TTL,
a single transient API failure mid-test left the cassette stuck with
only the leading non-API requests (e.g. the model_prices fetch from
raw.githubusercontent.com), and every subsequent run for the next 24h
errored with 'Can't overwrite existing cassette'.
new_episodes records anything not already present, so partially
populated cassettes recover on the next run instead of poisoning the
suite for a full TTL window.
The Generic SSO PKCE flow used the URL ``state`` parameter as the
cache key for the PKCE ``code_verifier`` without binding the state
to the caller's browser. An attacker who pre-minted a state and
cached a verifier under it could hand the resulting login link to a
victim; the victim's auth code would then be exchanged with the
attacker's verifier on the callback, producing an access token
under the attacker's control (Login CSRF / token theft).
The non-PKCE branch is unaffected because it delegates to
fastapi-sso's ``verify_and_process``, which performs its own
session-cookie check. The PKCE branch bypasses that helper, which
is exactly the gap this commit closes.
Two-part fix in ``ui_sso.py``:
- ``get_generic_sso_redirect_response`` now sets a
``litellm_oauth_state`` cookie (HttpOnly, SameSite=Lax, 10-min TTL)
carrying the state value used in the redirect URL. The cookie is
set on the redirect response just like the existing
``litellm_cp_return_to`` cookie a few lines earlier in the file.
- ``get_generic_sso_response`` validates ``request.cookies.get(
"litellm_oauth_state")`` against ``request.query_params.get(
"state")`` via ``secrets.compare_digest`` before invoking the
PKCE token exchange. Mismatch (or either being missing) raises a
``ProxyException`` with HTTP 400.
The pre-existing TODO above the redirect logic ("state should be a
random string and added to the user session with cookie") is now
addressed and removed.
Tests cover the redirect-side cookie set, the missing-cookie reject
shape, the URL/cookie-mismatch reject shape, and the matching-cookie
happy path.
The proxy's ingress hardening (commit 842eea0131) now strips client-supplied
`mock_response` from the request body unless the calling key or team has the
`allow_client_mock_response: true` admin-metadata flag set. The e2e model
access tests rely on `mock_response` to short-circuit the LLM call, so without
the flag they hit real backends — the bedrock wildcard route fakes out to a
shared example endpoint that now 404s on unsupported paths, causing
`test_model_access_patterns[key_models2-bedrock/anthropic.claude-3-True]`
(and the bedrock/anthropic.* row that pytest -x never reaches) to fail.
Set `allow_client_mock_response: true` on every key and team this test file
provisions so `mock_response` is preserved end-to-end.
Provider SDKs already retry transient 5xx/429 with exponential backoff
(default max_retries=2), and pytest.mark.flaky covers test-level
retries on top of that. Setting litellm.num_retries=3 here just
multiplied the existing layers — worst case 6 (flaky) x 3 (this) x
2 (CI rerunfailures) = 36 attempts on a single test.
Removing it keeps SDK-level network-blip protection intact and
shortens worst-case latency on cache-miss runs.
This commit updates the codebase to replace instances of DualCache with UserApiKeyCache in various files, including utils, expired_ui_session_key_cleanup_manager, and team_member_permission_checks. Additionally, it enhances the UserApiKeyCache class with new methods for cache management, improving type safety and consistency across the application.
This commit deletes the AuthMetrics class and its associated methods, which were responsible for tracking combined_view SQL query metrics. The PrometheusLogger integration has been updated to remove references to these metrics, streamlining the codebase. Additionally, minor whitespace adjustments were made in the cache coordinator for consistency.
Removes commentary that restated the code, including:
- module-level banners explaining what the conftest does (covered by
Readme.md and the function bodies)
- docstrings on _scrub_response, _before_record_response, vcr_config,
_vcr_disabled, pytest_recording_configure (function names + bodies
are self-evident)
- inline notes about header filtering, match_on, etc.
- per-test docstrings restating the test name
Keeps the two non-obvious notes that aren't recoverable from the code:
the vcrpy/respx httpx-transport collision rationale on
_RESPX_CONFLICTING_FILES, the vcrpy "return None to skip persisting"
contract on filter_non_2xx_response, and the fixture-ordering
dependency on _vcr_record_retries.
Removes the YAML cassette feature entirely and replaces it with a
Redis-only flow. Every test in tests/llm_translation/ and
tests/llm_responses_api_testing/ is auto-marked @pytest.mark.vcr via
conftest.pytest_collection_modifyitems, so any provider call lands in
the Redis cache (litellm:vcr:cassette:<rel_path>, 24h TTL). First run
records, runs within the day replay, day rollover re-records and
surfaces upstream API drift within 24h.
VCR is on by default. Set LITELLM_VCR_DISABLE=1, or simply leave
REDIS_HOST unset, to opt out — both bypass the auto-marker entirely so
nothing about cassettes runs. record_mode is "once" so cache-miss
records and cache-hit replays.
The 8 existing respx-using files in tests/llm_translation are excluded
from the auto-marker (vcrpy and respx both patch the httpx transport;
applying both makes one silently win). The persister's own unit-test
file is also excluded so it doesn't recursively run inside a cassette.
The persister moved from tests/llm_translation/_vcr_redis_persister.py
to tests/_vcr_redis_persister.py so both conftests share it. The two
demo tests in test_anthropic_completion_vcr.py were ported into
test_anthropic_completion.py and the demo file was deleted.
Adds tests/_flush_vcr_cache.py + a Make target
(test-llm-translation-flush-vcr-cache) that scans
litellm:vcr:cassette:* and pipelines DELETEs, for the
"I want the next CI run to re-record now" workflow. Drops the now-dead
test-llm-translation-record target.
Provider keys are still required on cache-miss (which happens on first
run and once a day after that). Replay-mode runs need only Redis.
