test_extract_langfuse_metadata_with_header_enrichment replaced
sys.modules["litellm.integrations.langfuse.langfuse"] with a stub
module but never restored it. This caused subsequent tests using
patch("litellm.integrations.langfuse.langfuse._add_prompt_to_generation_params")
to patch the stub instead of the real module, while _log_langfuse_v2
executed from the real module's globals (unpatched), triggering
ModuleNotFoundError and assertion failures.
Fix: use monkeypatch.setitem() so pytest automatically restores the
original module after the test completes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The `if hasattr(...)` guards in test_acompletion_with_mcp_adds_metadata_to_streaming
and test_acompletion_with_mcp_streaming_metadata_in_correct_chunks could silently skip
the provider_specific_fields assertions if chunks lacked choices/delta. Replace with
unconditional `assert hasattr(...)` so failures surface immediately.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use importlib.import_module + reload uniformly in both code paths
to ensure fresh module state regardless of whether litellm was
previously in sys.modules. This fixes the inconsistency where the
"not in sys.modules" branch didn't reload the module.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- test_pillar_guardrails.py: Fix fixture to properly update module-level
litellm reference using global keyword and assignment from reload
- test_anthropic_experimental_pass_through_messages_handler.py: Add missing
assert keywords to kwargs comparison statements (lines 36, 60-62)
- test_proxy_server.py: Replace silent pytest.skip with explicit assertion
to catch router initialization regressions
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes test failures that occur during parallel test execution (pytest -n 4)
due to module reloading issues with conftest.py reloading litellm.
Changes:
- Add module reload fixtures to ensure fresh references after conftest reloads
- Use patch.object and string-based patches instead of direct attribute assignment
- Use class name comparison instead of isinstance for reloaded modules
- Handle case where litellm is missing from sys.modules during parallel runs
- Move stream consumption inside patch contexts to avoid real API calls
- Mock litellm.acompletion instead of low-level HTTP handlers
- Add skipif decorator for enterprise-only test classes
Affected test files:
- test_container_integration.py
- test_responses_background_cost.py
- test_huggingface_embedding_handler.py
- test_vertex_ai_rerank_integration.py
- test_volcengine_responses_transformation.py
- test_pillar_guardrails.py
- test_litellm_pre_call_utils.py
- test_proxy_server.py
- test_converse_transformation.py
- test_chat_completions_handler.py
- test_aresponses_api_with_mcp.py
- test_anthropic_experimental_pass_through_messages_handler.py
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Many tests across the llms group (sap, compactifai, vercel_ai_gateway, mistral,
zai, heroku) set litellm.disable_aiohttp_transport = True without restoring it.
When these tests run before test_ssl_context_transport or test_session_reuse_chain
in the same xdist worker, _create_async_transport() returns None (because aiohttp
is disabled AND force_ipv4 is False), causing both tests to fail with
'assert None is not None'.
Fix: extend isolate_litellm_state in conftest.py to also save and restore
disable_aiohttp_transport and force_ipv4, following the same pattern already
used for callbacks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add _reset_litellm_http_client_cache autouse fixture (matching
test_vertex_gemma_transformation.py) to flush in_memory_llm_clients_cache
before each test. Without this, a cached real AsyncHTTPHandler from an
earlier test could bypass the class-level mock and cause real HTTP calls.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace instance-level patch.object(client, "post", side_effect=...) with
class-level patch of AsyncHTTPHandler and AsyncMock to reliably intercept
HTTP calls in CI where real Google credentials are available.
The old approach patched a specific instance's post method and passed
client=client to acompletion(). In CI, the mock wasn't intercepting actual
HTTP calls, causing 401 ACCESS_TOKEN_TYPE_UNSUPPORTED errors. The new
approach patches AsyncHTTPHandler at the class level so any instance
created internally by get_async_httpx_client() is also mocked.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two test files were reloading modules in setup_method/fixtures, which
caused class-reference staleness for subsequent tests in the same worker:
1. test_huggingface_embedding_handler.py reloaded
litellm.llms.custom_httpx.http_handler, creating a new HTTPHandler
class. Subsequent tests (e.g. hosted_vllm embedding) created
client = HTTPHandler() from the new class, but llm_http_handler.py
still held the old class reference. isinstance(client, HTTPHandler)
returned False, so a new unpatched client was used and
client.post was never called.
2. test_vertex_ai_rerank_integration.py reloaded
litellm.llms.vertex_ai.rerank.transformation in setup_method,
creating a new VertexAIRerankConfig class. The transformation test
file's module-level import still referenced the old class, so
@patch('...VertexAIRerankConfig._ensure_access_token') patched the
new class while self.config was an instance of the old class,
leaving the mock unapplied and hitting real Google credentials.
Fix: remove the reload calls. The module-level class references are
stable across tests within a worker; the reloads were solving a problem
that doesn't exist and actively created cross-test contamination.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two independent fixes for test_token_counter.py failures in CI:
1. test_disable_hf_tokenizer_download leaked litellm.disable_hf_tokenizer_download=True
because pytest.MonkeyPatch() was never undone. The setting persisted into the
alphabetically-subsequent test_llama2/3_tokenizer_api_failure tests, causing
_select_tokenizer_helper to short-circuit before calling from_pretrained.
Fix: wrap the test body in try/finally and call monkeypatch.undo().
2. encode() returns a HuggingFace Encoding object when the HF tokenizer loads, but
falls back to returning a plain List[int] (tiktoken) when the model hub is
unreachable. test_encoding_and_decoding called .ids on the result, which raises
AttributeError when the list-based fallback is active.
Fix: normalize encode() to always return List[int] by extracting .ids when present,
and remove the now-unnecessary .ids access in the test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- encode() now always returns List[int] by extracting .ids from HuggingFace
Encoding objects, making the return type consistent regardless of tokenizer backend
- test_encoding_and_decoding: remove .ids access since encode() now returns a list
- test_tokenizers: skip llama2 differentiation assertion when HuggingFace tokenizer
is unavailable (CI without network access falls back to tiktoken)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This test has failed repeatedly in CI with:
'Expected _add_prompt_to_generation_params to have been called once. Called 0 times.'
Root cause: _add_prompt_to_generation_params is only called when _supports_prompt()
returns True. Under cross-test state contamination in CI (parallel workers),
langfuse_sdk_version can be in an unexpected state, causing _supports_prompt() to
return False and silently skip the call (exception swallowed by the outer try/except).
Fixes:
- Use reset_mock(side_effect=True) so setUp's trace side_effect is cleared and the
explicit return_value assignment actually takes effect
- Patch _supports_prompt on the logger instance to always return True, making the
_add_prompt_to_generation_params assertion independent of SDK version state
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix isinstance() checks failing due to module reload in conftest.py.
The conftest.py fixture reloads the litellm module between test modules,
which causes class references imported at module-level to become stale.
When AsyncHTTPHandler is imported at the top of the file and then litellm
is reloaded by the fixture, the isinstance() check fails because the
returned instance is of the NEW AsyncHTTPHandler class while the test
is checking against the OLD class reference.
Solution: Import AsyncHTTPHandler locally within each test function that
uses isinstance() checks. This ensures we get the fresh class reference
after the module reload.
Fixed tests:
- test_session_reuse_integration
- test_get_async_httpx_client_with_shared_session
- test_get_async_httpx_client_without_shared_session
This resolves intermittent CI failures where parallel test execution
triggers the module reload behavior.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Critical fix for Greptile feedback: The setup_and_teardown fixture was
missing the autouse=True parameter, causing the module reload logic to
never execute. This would result in test pollution as callbacks would
chain across modules.
Changes:
- Add autouse=True to setup_and_teardown fixture in conftest.py
- Add autouse=True to setup_and_teardown fixture in conftest_improved.py
Note: conftest_improved.py is intentionally kept as a reference
implementation showing the recommended improvements. It demonstrates
better patterns for test isolation that can be adopted later.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Replace asyncio.get_event_loop_policy() with asyncio.new_event_loop()
- Use asyncio.set_event_loop() to set the event loop
- Fixes deprecation warning in Python 3.16
- Updated both conftest.py and conftest_improved.py
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Default user_id to caller's own ID for non-admins instead of 403 when
omitted, preserving backward compatibility for API consumers
- Apply same fix to aggregated endpoint
- Update test to verify defaulting behavior instead of expecting 403
- Add useEffect to sync selectedUserId when auth state settles in
UsagePageView to handle async auth initialization
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix HTTPException swallowed by broad except block in get_user_daily_activity
and get_user_daily_activity_aggregated: re-raise HTTPException before the
generic handler so 403 status codes propagate correctly
- Add status_code assertions in non-admin access tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add 6 new EU PII patterns for GDPR compliance
- fr_nir: French Social Security Number (NIR/INSEE) with validation
- eu_iban_enhanced: Enhanced IBAN detection with specific format
- fr_phone: French phone numbers (+33, 0033, 0 formats)
- eu_vat: EU VAT identification numbers (all 27 member states)
- eu_passport_generic: Generic EU passport format
- fr_postal_code: French postal codes with contextual keywords
* Add GDPR Art. 32 EU PII Protection policy template
- Comprehensive GDPR Article 32 compliance policy
- 4 guardrail groups: National IDs, Financial, Contact Info, Business IDs
- Masks French NIR/INSEE, EU IBANs, French phones, EU VAT numbers
- Includes EU passport numbers and email addresses
- Medium complexity template with indigo icon
* Add comprehensive tests for EU PII patterns
- Test French NIR validation (sex digit, month range)
- Test enhanced IBAN detection (French, German)
- Test French phone number formats
- Test EU VAT numbers
- Test generic EU passport format
- Test French postal code pattern
* Add EU pattern loading and category validation tests
- Verify all 6 EU PII patterns are loaded correctly
- Verify patterns are categorized as 'EU PII Patterns'
- Ensure pattern loading consistency
* Add end-to-end tests for GDPR policy template
- 4 tests for PII that should be masked (NIR, IBAN, phone, VAT)
- 4 tests for text that should pass through (invalid patterns, no PII)
- 1 bonus test for multiple PII types in same message
- All tests verify correct masking behavior
* Add region field to policy templates
- Added region field to all 6 templates (EU, AU, Global)
- Updated both main and backup JSON files
- Enables region-based filtering in UI
* Add region filter to policy templates UI
- Added Radio.Group filter for regions (All, AU, EU, Global)
- Efficient filtering with useMemo hooks
- Clean button-based UI matching existing design
- Defaults missing regions to Global
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Address Greptile review: add contextual guards and negative tests
- Added keyword_pattern to eu_vat (VAT, tax number, fiscal code, etc.)
- Added keyword_pattern to eu_passport_generic (passport, travel document, etc.)
- Added 3 negative unit tests for false positive prevention
- Added 2 E2E tests verifying no masking without keyword context
- All patterns now require contextual keywords to prevent false positives
* address greptile review feedback (greploop iteration 1)
- Remove unused HTTPException import from test file
- Add keyword_pattern to eu_vat for contextual VAT matching
- Add allow_word_numbers: false to eu_passport_generic
- Add negative test cases for EU VAT false positives
- All 5 Greptile comments addressed
* Address Greptile feedback: fix patterns and sync backup
- Fix fr_phone pattern: use negative lookbehind (?<!\d) to prevent false matches in longer digit strings
- Add keyword_pattern to eu_passport_generic to reduce false positives on version strings/SKUs
- Sync policy_templates_backup.json with main file (add GDPR template)
- Add keyword_pattern to eu_vat (was auto-added by formatter)
All pattern tests passing
* address greptile review feedback (greploop iteration 2)
- Update test to document that eu_vat raw pattern is intentionally broad
- Test verifies pattern DOES match common words (by design)
- Documents that keyword_pattern guard prevents false positives in production
- Addresses Greptile's false positive risk concern
* address greptile review feedback (greploop iteration 3)
- Fix test_eu_vat_masked: change gap from 2 words to 1 word (VAT number: FR...)
- This ensures keyword matching works within MAX_KEYWORD_VALUE_GAP_WORDS=1 limit
- fr_phone pattern already works correctly (verified with tests)
- test_pattern_requires_keyword_context already updated in iteration 2
Addresses final issues from Greptile 2/5 review
* fix: remove country-specific passport patterns from GDPR template
- Remove passport_france, passport_germany, passport_netherlands from template
- These patterns lack keyword guards and cause false positives
- Only eu_passport_generic remains (has keyword_pattern guard)
- Sync policy_templates_backup.json with main file
- Update test setup to match template
All 11 E2E tests now passing ✅
* Update tests/test_litellm/proxy/guardrails/guardrail_hooks/content_filter/test_gdpr_policy_e2e.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/content_filter.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
PR #19809 changed stateless=True to stateless=False to enable progress
notifications for MCP tool calls. This caused the mcp library to enforce
mcp-session-id headers on all non-initialize requests, breaking MCP
Inspector, curl, and any client without automatic session management.
Revert to stateless=True to restore compatibility with all MCP clients.
The progress notification code already handles missing sessions gracefully
(defensive checks + try/except), so no other changes are needed.
Fixes#20242
