Read summary from the original thinking dict instead of hardcoding "detailed"
in _route_openai_thinking_to_responses_api_if_needed(). This preserves the
user's chosen summary value (e.g. "concise", "auto") for non-Claude models
routed through the Anthropic Messages adapter to OpenAI's Responses API.
Fixes#20998
Three test failures caused by the real langfuse SDK import being triggered
at test time:
1. test_langfuse_prompt_management.py: Both tests create LangfusePromptManagement()
which calls `import langfuse`. Since earlier TestLangfuseUsageDetails tests
remove sys.modules["langfuse"] via patch.dict teardown, the real langfuse
import runs and fails on Python 3.14 (pydantic v1 incompatibility).
Fix: add setup_method/teardown_method to mock sys.modules["langfuse"].
2. test_langfuse.py::test_max_langfuse_clients_limit: Same root cause — creates
LangFuseLogger() without mocking sys.modules["langfuse"].
Fix: wrap test body with patch.dict("sys.modules", {"langfuse": mock}).
3. test_langfuse_otel.py::test_extract_langfuse_metadata_with_header_enrichment:
Replaces sys.modules["litellm.integrations.langfuse.langfuse"] with a stub
without restoring it, causing patch() in later tests to target the stub
instead of the real module.
Fix: use monkeypatch.setitem() which auto-restores after the test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
test_extract_langfuse_metadata_with_header_enrichment replaced
sys.modules["litellm.integrations.langfuse.langfuse"] with a stub
module but never restored it. This caused subsequent tests using
patch("litellm.integrations.langfuse.langfuse._add_prompt_to_generation_params")
to patch the stub instead of the real module, while _log_langfuse_v2
executed from the real module's globals (unpatched), triggering
ModuleNotFoundError and assertion failures.
Fix: use monkeypatch.setitem() so pytest automatically restores the
original module after the test completes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The `if hasattr(...)` guards in test_acompletion_with_mcp_adds_metadata_to_streaming
and test_acompletion_with_mcp_streaming_metadata_in_correct_chunks could silently skip
the provider_specific_fields assertions if chunks lacked choices/delta. Replace with
unconditional `assert hasattr(...)` so failures surface immediately.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use importlib.import_module + reload uniformly in both code paths
to ensure fresh module state regardless of whether litellm was
previously in sys.modules. This fixes the inconsistency where the
"not in sys.modules" branch didn't reload the module.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- test_pillar_guardrails.py: Fix fixture to properly update module-level
litellm reference using global keyword and assignment from reload
- test_anthropic_experimental_pass_through_messages_handler.py: Add missing
assert keywords to kwargs comparison statements (lines 36, 60-62)
- test_proxy_server.py: Replace silent pytest.skip with explicit assertion
to catch router initialization regressions
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fixes test failures that occur during parallel test execution (pytest -n 4)
due to module reloading issues with conftest.py reloading litellm.
Changes:
- Add module reload fixtures to ensure fresh references after conftest reloads
- Use patch.object and string-based patches instead of direct attribute assignment
- Use class name comparison instead of isinstance for reloaded modules
- Handle case where litellm is missing from sys.modules during parallel runs
- Move stream consumption inside patch contexts to avoid real API calls
- Mock litellm.acompletion instead of low-level HTTP handlers
- Add skipif decorator for enterprise-only test classes
Affected test files:
- test_container_integration.py
- test_responses_background_cost.py
- test_huggingface_embedding_handler.py
- test_vertex_ai_rerank_integration.py
- test_volcengine_responses_transformation.py
- test_pillar_guardrails.py
- test_litellm_pre_call_utils.py
- test_proxy_server.py
- test_converse_transformation.py
- test_chat_completions_handler.py
- test_aresponses_api_with_mcp.py
- test_anthropic_experimental_pass_through_messages_handler.py
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Many tests across the llms group (sap, compactifai, vercel_ai_gateway, mistral,
zai, heroku) set litellm.disable_aiohttp_transport = True without restoring it.
When these tests run before test_ssl_context_transport or test_session_reuse_chain
in the same xdist worker, _create_async_transport() returns None (because aiohttp
is disabled AND force_ipv4 is False), causing both tests to fail with
'assert None is not None'.
Fix: extend isolate_litellm_state in conftest.py to also save and restore
disable_aiohttp_transport and force_ipv4, following the same pattern already
used for callbacks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add _reset_litellm_http_client_cache autouse fixture (matching
test_vertex_gemma_transformation.py) to flush in_memory_llm_clients_cache
before each test. Without this, a cached real AsyncHTTPHandler from an
earlier test could bypass the class-level mock and cause real HTTP calls.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The _safe_get_request_headers caching uses request.state._cached_headers,
which returns a truthy MagicMock on bare MagicMock() objects instead of
None, breaking content-type detection for form-data tests.
Replace instance-level patch.object(client, "post", side_effect=...) with
class-level patch of AsyncHTTPHandler and AsyncMock to reliably intercept
HTTP calls in CI where real Google credentials are available.
The old approach patched a specific instance's post method and passed
client=client to acompletion(). In CI, the mock wasn't intercepting actual
HTTP calls, causing 401 ACCESS_TOKEN_TYPE_UNSUPPORTED errors. The new
approach patches AsyncHTTPHandler at the class level so any instance
created internally by get_async_httpx_client() is also mocked.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two test files were reloading modules in setup_method/fixtures, which
caused class-reference staleness for subsequent tests in the same worker:
1. test_huggingface_embedding_handler.py reloaded
litellm.llms.custom_httpx.http_handler, creating a new HTTPHandler
class. Subsequent tests (e.g. hosted_vllm embedding) created
client = HTTPHandler() from the new class, but llm_http_handler.py
still held the old class reference. isinstance(client, HTTPHandler)
returned False, so a new unpatched client was used and
client.post was never called.
2. test_vertex_ai_rerank_integration.py reloaded
litellm.llms.vertex_ai.rerank.transformation in setup_method,
creating a new VertexAIRerankConfig class. The transformation test
file's module-level import still referenced the old class, so
@patch('...VertexAIRerankConfig._ensure_access_token') patched the
new class while self.config was an instance of the old class,
leaving the mock unapplied and hitting real Google credentials.
Fix: remove the reload calls. The module-level class references are
stable across tests within a worker; the reloads were solving a problem
that doesn't exist and actively created cross-test contamination.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two independent fixes for test_token_counter.py failures in CI:
1. test_disable_hf_tokenizer_download leaked litellm.disable_hf_tokenizer_download=True
because pytest.MonkeyPatch() was never undone. The setting persisted into the
alphabetically-subsequent test_llama2/3_tokenizer_api_failure tests, causing
_select_tokenizer_helper to short-circuit before calling from_pretrained.
Fix: wrap the test body in try/finally and call monkeypatch.undo().
2. encode() returns a HuggingFace Encoding object when the HF tokenizer loads, but
falls back to returning a plain List[int] (tiktoken) when the model hub is
unreachable. test_encoding_and_decoding called .ids on the result, which raises
AttributeError when the list-based fallback is active.
Fix: normalize encode() to always return List[int] by extracting .ids when present,
and remove the now-unnecessary .ids access in the test.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- encode() now always returns List[int] by extracting .ids from HuggingFace
Encoding objects, making the return type consistent regardless of tokenizer backend
- test_encoding_and_decoding: remove .ids access since encode() now returns a list
- test_tokenizers: skip llama2 differentiation assertion when HuggingFace tokenizer
is unavailable (CI without network access falls back to tiktoken)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This test has failed repeatedly in CI with:
'Expected _add_prompt_to_generation_params to have been called once. Called 0 times.'
Root cause: _add_prompt_to_generation_params is only called when _supports_prompt()
returns True. Under cross-test state contamination in CI (parallel workers),
langfuse_sdk_version can be in an unexpected state, causing _supports_prompt() to
return False and silently skip the call (exception swallowed by the outer try/except).
Fixes:
- Use reset_mock(side_effect=True) so setUp's trace side_effect is cleared and the
explicit return_value assignment actually takes effect
- Patch _supports_prompt on the logger instance to always return True, making the
_add_prompt_to_generation_params assertion independent of SDK version state
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix isinstance() checks failing due to module reload in conftest.py.
The conftest.py fixture reloads the litellm module between test modules,
which causes class references imported at module-level to become stale.
When AsyncHTTPHandler is imported at the top of the file and then litellm
is reloaded by the fixture, the isinstance() check fails because the
returned instance is of the NEW AsyncHTTPHandler class while the test
is checking against the OLD class reference.
Solution: Import AsyncHTTPHandler locally within each test function that
uses isinstance() checks. This ensures we get the fresh class reference
after the module reload.
Fixed tests:
- test_session_reuse_integration
- test_get_async_httpx_client_with_shared_session
- test_get_async_httpx_client_without_shared_session
This resolves intermittent CI failures where parallel test execution
triggers the module reload behavior.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Critical fix for Greptile feedback: The setup_and_teardown fixture was
missing the autouse=True parameter, causing the module reload logic to
never execute. This would result in test pollution as callbacks would
chain across modules.
Changes:
- Add autouse=True to setup_and_teardown fixture in conftest.py
- Add autouse=True to setup_and_teardown fixture in conftest_improved.py
Note: conftest_improved.py is intentionally kept as a reference
implementation showing the recommended improvements. It demonstrates
better patterns for test isolation that can be adopted later.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Replace asyncio.get_event_loop_policy() with asyncio.new_event_loop()
- Use asyncio.set_event_loop() to set the event loop
- Fixes deprecation warning in Python 3.16
- Updated both conftest.py and conftest_improved.py
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add infrastructure for JSON-declared providers to support /v1/responses
via `supported_endpoints` field in providers.json. Simplify Perplexity
responses config from 410 to 40 lines by moving cost dict→float parsing
to generic validators in ResponseAPIUsage and Usage.
- Add `supported_endpoints` field to SimpleProviderConfig (default: [])
- Add `supports_responses_api()` to JSONProviderRegistry
- Create OpenAILikeResponsesConfig base class for responses API
- Add `create_responses_config_class()` with class caching
- ProviderConfigManager: Python classes take priority over JSON fallback
- Fix ResponseAPIUsage.cost field_validator to handle dict cost objects
- Fix Usage.__init__ to handle dict cost from chat completions
- Simplify PerplexityResponsesConfig with get_supported_openai_params guard
- Add 20 unit tests including Python-over-JSON priority test
- Rename _is_nova_lite_2_model → _is_nova_2_model to match all nova-2-* variants
- Add bedrock/converse/ routing prefix stripping in model detection
- Fix pre-existing test_get_supported_openai_params_bedrock_converse failure
- Remove thinking_blocks tests from Nova 2 test file (not Nova 2 behavior)
- Add end-to-end request, response, multi-turn, and model detection tests
- Parametrize key tests across both nova-2-lite and nova-2-pro model IDs
* allow filtering by user in global usage
* add server root path test to github actions
* Update .github/workflows/test_server_root_path.yml
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* address greptile review feedback (greploop iteration 1)
- Fix HTTPException swallowed by broad except block in get_user_daily_activity
and get_user_daily_activity_aggregated: re-raise HTTPException before the
generic handler so 403 status codes propagate correctly
- Add status_code assertions in non-admin access tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* address greptile review feedback (greploop iteration 2)
- Default user_id to caller's own ID for non-admins instead of 403 when
omitted, preserving backward compatibility for API consumers
- Apply same fix to aggregated endpoint
- Update test to verify defaulting behavior instead of expecting 403
- Add useEffect to sync selectedUserId when auth state settles in
UsagePageView to handle async auth initialization
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fixing syntax
* remove artifacts
* feat: guardrail tracing UI - policy, detection method, match details (#21349)
* feat: add GuardrailTracingDetail TypedDict and tracing fields to StandardLoggingGuardrailInformation
* feat: add policy_template field to Guardrail config TypedDict
* feat: accept GuardrailTracingDetail in base guardrail logging method
* feat: populate tracing fields in content filter guardrail
* test: add tracing fields tests for custom guardrail base class
* test: add tracing fields e2e tests for content filter guardrail
* feat: add guardrail tracing UI - policy badges, match details, timeline
* feat: redesign GuardrailViewer to Guardrails & Policy Compliance layout
Two-column layout with request lifecycle timeline on the left
and compact evaluation detail cards on the right. Header shows
guardrail count, pass/fail status, total overhead, policy info,
and an export button.
* feat: add clickable guardrail link in metrics + show policy names
* feat: add risk_score field to StandardLoggingGuardrailInformation
* feat: compute risk_score in content filter guardrail
* feat: display backend risk_score badge on evaluation cards
* fix: fallback to frontend risk score when backend doesn't provide one
* passing in masster key for api calls
* Fix: Add blog as incident report
* Fix: Add blog as incident report
* remove timeline
* feat(models): add github_copilot/gpt-5.3-codex and github_copilot/claude-opus-4.6-fast (#21316)
Add missing GitHub Copilot model entries for gpt-5.3-codex (GA) and
claude-opus-4.6-fast (Public Preview) to both the root and backup
model pricing JSON files.
* only tests for /ui
* bump: version 1.81.12 → 1.81.13
* Fixing mapped tests
* fixing no_config test
* fixing container tests
* fixing test_basic_openai_responses_api
* Adding bedrock thinking budget tokens to docs
* fixing regen key tests
* fix: add missing OpenAI chat completion params to OPENAI_CHAT_COMPLETION_PARAMS
Add store, prompt_cache_key, prompt_cache_retention, safety_identifier, and verbosity
to OPENAI_CHAT_COMPLETION_PARAMS list.
These params were already in DEFAULT_CHAT_COMPLETION_PARAM_VALUES but missing from
the OPENAI_CHAT_COMPLETION_PARAMS list, causing them to be dropped when passed to
OpenAI-compatible providers.
---------
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
