litellm/tests/test_litellm/llms
Ishaan Jaff 28c33f53a3
CircleCI test stability (#23055)
* fix: resolve ruff lint errors and mypy type error

- Remove unused import get_user_credential (F401)
- Add noqa: PLR0915 for 3 large functions exceeding 50 statements
- Cast result_data['q'] to str for _append_domain_filters (mypy arg-type)

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: add /vertex_ai/live to supported endpoints and azure gpt-5.1 reasoning flags

- Add /vertex_ai/live to JSON schema validation enum in test_utils.py
- Add supports_none_reasoning_effort=true to 10 azure/gpt-5.1 model entries
  (matching the OpenAI gpt-5.1 behavior)

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: handle non-string team_alias/key_alias in PolicyMatchContext

Prevent Pydantic validation errors when team_alias or key_alias are not
proper strings (e.g. MagicMock in tests). Only pass values that are
actually strings; default to None otherwise.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: initialize jwt_handler.litellm_jwtauth in JWT test

The test_jwt_non_admin_team_route_access test was failing because
user_api_key_auth now accesses jwt_handler.litellm_jwtauth.virtual_key_claim_field
before reaching the mocked JWTAuthManager.auth_builder. Initialize the
jwt_handler with a default LiteLLM_JWTAuth object.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: add missing mock attributes to MCP server test

The test_add_update_server_fallback_to_server_id test was failing because
MagicMock auto-creates attributes when accessed. build_mcp_server_from_table
accesses many fields via getattr(), which on a MagicMock returns another
MagicMock instead of None, causing Pydantic validation errors in MCPServer.

Explicitly set all required mock attributes.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: update UI tests for leftnav, navbar, and KeyLifecycleSettings

- leftnav: Add mock for useTeams hook, add isUserTeamAdminForAnyTeam to
  roles mock, update topLevelLabels to match current component menu items
- navbar: Add mocks for useDisableBouncingIcon, BlogDropdown, UserDropdown,
  and serverRootPath. Update test to work with the new component structure.
- KeyLifecycleSettings: Fix placeholder and tooltip assertions to match
  actual component behavior

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: update health check test assertion from 'connected' to 'healthy'

The /health/readiness endpoint now returns {"status": "healthy"} with the
DB status in a separate field, instead of the previous {"status": "connected"}.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: clear litellm.api_key in OpenRouter validate_environment test

The test_validate_environment_raises_without_key test was failing because
litellm.api_key may be set globally in the test environment. Clear it
along with OPENROUTER_API_KEY and OR_API_KEY env vars using monkeypatch.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: patch HTTPHandler class-level in VLLM embedding test

The test_encoding_format_not_sent_in_actual_request test was patching
client.post on an instance, but the handler uses the class method.
Patch HTTPHandler.post at class level, add caching=False to prevent
cache hits, and remove broad try/except that hid errors.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: make test_redaction_responses_api_stream resilient to async callback timing

Replace fixed 1s sleep with polling wait for async_log_success_event.
Streaming success handler runs via asyncio.create_task; 1s was insufficient
in CI. Add 0.5s initial sleep for event loop to schedule the task, then
poll up to 10s for the callback to fire.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: update dompurify and svgo to fix security CVEs

- CVE-2026-0540: dompurify XSS vulnerability - fix by upgrading to 3.3.2+
- CVE-2026-29074: svgo DoS via entity expansion - fix by upgrading to 3.3.3+

Added npm overrides in docs/my-website/package.json and regenerated
package-lock.json.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: remove unused json import in config_override_endpoints.py

Ruff F401: json is imported but unused (safe_json_loads/safe_dumps
are used instead)

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: add missing MCP mock attributes and provider documentation entries

- Add missing mock attributes to test_add_update_server_with_alias and
  test_add_update_server_without_alias (same fix as fallback test)
- Add bedrock_mantle and searchapi to provider_endpoints_support.json
- Remove unused json import from config_override_endpoints.py

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: override _supports_reasoning_effort_level for Azure gpt5_series prefix

The Azure GPT-5 config uses 'gpt5_series/' as a routing prefix, but
_supports_factory(model='gpt5_series/gpt-5.1') fails to resolve because
'gpt5_series' is not a recognized provider. Override the method to strip
the prefix and prepend 'azure/' for correct model info lookup.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: accept both 'healthy' and 'connected' in health check test

The test_health_and_chat_completion test runs against both source builds
(which return 'healthy') and pip-installed versions (which may return
'connected'). Accept both values.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: mock extract_mcp_auth_context in streamable HTTP MCP handler test

The handle_streamable_http_mcp function now calls extract_mcp_auth_context
before session_manager.handle_request, but the test didn't mock it. The
auth extraction fails with the minimal mock scope, preventing
handle_request from being called. Also relax assertion to not check
exact args since the send wrapper may be modified by debug injection.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: add test for _combine_fallback_usage to satisfy router code coverage

The router_code_coverage.py check requires all functions in router.py
to be called in test files. Add a basic test for _combine_fallback_usage.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: add @log_guardrail_information decorator to CrowdStrike AIDR guardrail

The check_guardrail_apply_decorator.py CI check requires all guardrail
apply_guardrail methods to have the @log_guardrail_information decorator.
The CrowdStrike AIDR handler was missing it.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: document PRISMA_RECONNECT_ESCALATION_THRESHOLD and REDIS_CLUSTER_NODES env keys

Add missing environment variable documentation to config_settings.md
to satisfy the test_env_keys.py CI check.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: document enforced_file_expires_after and enforced_batch_output_expires_after in new_team docstring

The test_api_docs.py CI check validates that all Pydantic model fields
are documented in the function docstring. Add missing parameter docs
for enforced_file_expires_after and enforced_batch_output_expires_after.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: regenerate poetry.lock to match pyproject.toml

The poetry.lock file was out of sync with pyproject.toml, causing
proxy_e2e_azure_batches_tests to fail during dependency installation.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: set master_key=None in test_create_file_with_deep_nested_litellm_metadata

The test was missing the master_key monkeypatch that other tests in the
same file set. In CI with parallel execution (-n 4), another test may
set master_key to a non-None value, causing auth failures (500) when
the test sends 'Bearer test-key'.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: document enforced_*_expires_after in update_team docstring too

Same missing params as new_team - also needed in update_team docstring
for the test_api_docs.py CI check to pass.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: use get_async_httpx_client in a2a_protocol and add master_key monkeypatch to files tests

- Replace httpx.AsyncClient() with get_async_httpx_client() in a2a_protocol/main.py
  to satisfy the ensure_async_clients_test CI check
- Add httpxSpecialProvider.A2AProvider enum value
- Add master_key=None monkeypatch to test_managed_files_with_loadbalancing

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: remove unused httpx import from a2a_protocol/main.py

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: use cache-key-only param for A2A extra_headers to avoid AsyncHTTPHandler init error

The 'extra_headers' key in params was being passed to AsyncHTTPHandler.__init__()
which doesn't accept it. Use 'disable_aiohttp_transport' as the cache-key-only
param since it's explicitly filtered out before reaching the constructor.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: add additionalProperties:false and resolve $defs/$ref in Anthropic output_format schemas

Anthropic API now requires additionalProperties=false for all object-type
schemas in output_format. Also resolve $defs/$ref references by inlining
them using unpack_defs before sending to Anthropic, since Anthropic
doesn't support external schema references.

Fixes: llm_translation_testing Anthropic JSON schema failures

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: allowlist CVE-2026-2297 and GHSA-qffp-2rhf-9h96 in security scans

- CVE-2026-2297: Python 3.13 SourcelessFileLoader audit hook bypass,
  no fix available in base image
- GHSA-qffp-2rhf-9h96: tar hardlink path traversal, from nodejs_wheel
  bundled npm, not used in application runtime code

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: isolate files endpoint tests from shared proxy state in CI parallel execution

Override user_api_key_auth dependency to return a fixed UserAPIKeyAuth
with PROXY_ADMIN role, avoiding auth lookups via prisma_client,
user_api_key_cache, or master_key. Set prisma_client=None to prevent
DB state contamination. Use try/finally to clean up dependency overrides.

Fixes persistent test_create_file_with_deep_nested_litellm_metadata and
test_managed_files_with_loadbalancing 500 errors in CI with -n 4.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

* fix: apply same auth override to test_managed_files_with_loadbalancing

Same CI parallel execution fix as test_create_file_with_deep_nested -
override user_api_key_auth dependency and set prisma_client=None.

Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
2026-03-07 15:19:39 -08:00
..
amazon_nova/chat [New Model] Add Amazon Nova as first party provider for chat completions (#17351) 2025-12-06 14:43:55 -08:00
anthropic fix(proxy): do not forward Authorization header to LLM provider when used for LiteLLM proxy auth 2026-03-06 18:20:48 +05:30
azure Exclude aresponses_websocket from Azure SDK client init test 2026-03-04 11:13:57 -03:00
azure_ai feat(azure_ai): add router flat cost when response contains actual model 2026-03-06 18:18:06 +05:30
baseten/chat add testing and remove redundant function 2025-08-21 11:25:26 -07:00
bedrock fix(bedrock): filter out custom field from tools to prevent 400 errors (#22861) 2026-03-05 13:54:23 -08:00
bedrock_mantle feat(provider): add Amazon Bedrock Mantle as a first-class provider 2026-03-05 00:03:40 -05:00
bytez/chat Integration: Bytez as a model provider (#12121) 2025-07-12 10:50:39 -07:00
chat test(bedrock): add unit tests for region extraction from bedrock/{region}/{model} path 2026-03-02 11:06:27 +05:30
chatgpt test(chatgpt): restore gpt-5.2 codex transformation coverage 2026-03-05 17:00:49 -05:00
cohere fix(unified_guardrail.py): support during_call event type for unified guardrails (#17514) 2025-12-04 22:06:13 -08:00
cometapi/chat feat: add CometAPI provider support with chat completions and streaming (#13458) 2025-08-11 18:06:37 -07:00
compactifai Fix CompactifAI provider tests and implementation 2025-09-15 22:03:42 +02:00
custom_httpx [Release Fix] (#22411) 2026-02-28 09:46:35 -08:00
dashscope fix: remove list-to-str transformation from dashscope 2026-02-19 07:30:13 +00:00
databricks Address Greptile review: fix SDK auth fallback and remove unused imports 2026-02-18 15:57:50 +09:00
datarobot Updated URL handling for DataRobot provider base 2025-08-21 19:42:46 -06:00
deepgram fix: return the diarized transcript when it's required in the request (#16133) 2025-11-02 09:45:18 -08:00
deepinfra Fix: tool content should be str 2026-01-07 11:01:31 +05:30
docker_model_runner fix docker model runner tests 2025-11-22 11:07:56 -08:00
featherless_ai/chat fix(tests): use monkeypatch for env var isolation in featherless tests 2026-03-01 23:46:15 +01:00
fireworks_ai Fix: Extra inputs are not permitted, field: 'messages[2].provider_specific_fields' 2026-02-03 16:23:18 +05:30
gemini Add get files API support and tests 2026-02-03 18:57:39 +05:30
github_copilot fix(github_copilot): preserve encrypted_content in reasoning items for multi-turn conversations (#17130) 2025-12-05 12:42:25 -08:00
heroku adds tool calling test 2025-08-06 14:23:55 -06:00
hosted_vllm CircleCI test stability (#23055) 2026-03-07 15:19:39 -08:00
huggingface fix: improve test isolation for parallel execution 2026-02-17 21:28:23 -03:00
jina_ai/embedding feat(JinaAI): support multimodal embedding models (#13181) 2025-08-05 19:21:56 -07:00
lemonade Adding unit tests and documentation 2025-09-30 12:12:24 -06:00
litellm_proxy/chat LiteLLM SDK <-> Proxy: support user param + Prisma - remove use_prisma_migrate flag - redundant as this is now default (#13555) 2025-08-12 22:03:39 -07:00
llamafile/chat test llamafile 2025-07-03 22:35:22 -07:00
lm_studio fix(lm_studio): resolve illegal Bearer header value issue 2025-09-12 22:41:30 +02:00
manus [Feat] New provider - Manus API on /responses, GET /responses (#18804) 2026-01-08 18:37:42 +05:30
meta_llama fix: add missing OpenAI chat completion params to OPENAI_CHAT_COMPLETION_PARAMS (#21360) 2026-02-16 20:31:21 -08:00
minimax Add tests for chat completion minmax 2025-12-23 17:10:46 +05:30
mistral Added thinking streaming support for mistral (#16434) 2025-11-10 18:41:45 -08:00
moonshot fix(moonshot): preserve image_url blocks in multimodal messages 2026-02-19 16:44:38 -03:00
nebius test: fixes 2025-05-31 12:42:56 -07:00
novita/chat
nscale/chat
oci/chat Fixes #20957 2026-02-11 11:20:18 +00:00
ocr Add OCR guardrail_translation handler and support (#22145) 2026-02-28 17:39:36 -08:00
ollama fix(ollama): thread api_base to get_model_info + graceful fallback (#21970) 2026-02-23 21:00:37 -08:00
openai feat(openai): drop reasoning_effort for gpt-5.4 when tools present 2026-03-06 22:20:47 +05:30
openai_like fix(proxy): fix master key rotation Prisma validation errors (#21330) 2026-02-16 15:13:05 -08:00
openrouter CircleCI test stability (#23055) 2026-03-07 15:19:39 -08:00
ovhcloud Add support of audio transcription for OVHcloud (#17305) 2025-12-01 18:26:39 -08:00
perplexity feat(perplexity): add embedding support for pplx-embed-v1 models (#22610) 2026-03-02 17:37:50 -08:00
pg_vector/vector_stores [Bug Fix] QA - Use PG Vector Vector Store with LiteLLM (#12716) 2025-07-18 08:41:18 -07:00
publicai fix: add missing OpenAI chat completion params to OPENAI_CHAT_COMPLETION_PARAMS (#21360) 2026-02-16 20:31:21 -08:00
ragflow/chat fix failing vertex tests 2025-12-06 00:04:04 +05:30
recraft [Feat] Add Recraft API - Image Edits Support (#12874) 2025-07-22 15:03:08 -07:00
runwayml [Feat] RunwayML - Add support for /audio/speech eleven_multilingual_v2 endpoint (#16604) 2025-11-13 14:32:09 -08:00
s3_vectors [Feat] RAG API - Add s3_vectors as provider on /vector_store/search API + UI for creating + PDF support for /rag/ingest (#19895) 2026-01-27 16:30:59 -08:00
sagemaker fix(sagemaker): Support TEI raw array response format for embeddings (#20487) 2026-02-12 19:39:05 +05:30
sambanova Feat/sambanova embeddings (#13308) 2025-08-12 17:15:26 -07:00
sap fix(sap provider layer): enable response-format for anthropic models and improve compatibility for GPT models via LangChain (#22804) 2026-03-04 16:03:59 -08:00
snowflake Snowflake provider support: added embeddings, PAT, account_id (#15727) 2025-11-17 20:27:46 -08:00
stability feat(stability): add Stability AI image generation support (#17894) 2025-12-14 08:29:45 +05:30
vercel_ai_gateway feat(vercel_ai_gateway): add embeddings support 2026-01-23 15:11:37 -03:00
vertex_ai feat(vertex_ai): support explicit AWS credentials for WIF auth (#21472) 2026-03-06 09:27:20 -08:00
volcengine test: Fix additional broken tests 2026-02-15 13:08:41 -03:00
voyage/rerank [Fix] CI/CD – Clean Up Performance PR Changes & others (#17838) 2025-12-11 12:50:03 -08:00
wandb (feat): Add W&B Inference to LiteLLM 2025-09-11 00:07:30 +05:30
watsonx [Release Fix] (#22411) 2026-02-28 09:46:35 -08:00
xai Add xai websearch params support 2026-01-28 09:54:43 +05:30
zai feat(zai): Add GLM-4.7 model with reasoning support (#18476) 2026-01-04 00:44:19 +05:30
test_cache_control_and_reasoning.py Add Prompt caching and reasoning support for MiniMax, GLM, Xiaomi 2026-01-28 17:25:26 +05:30
test_lifecycle_fix.py fix: revert httpx client caching that caused closed client errors (#20025) 2026-02-03 16:15:04 -08:00
test_oom_fixes.py fix: HTTP client memory leaks in Presidio, OpenAI, and Gemini (#19190) 2026-01-19 19:02:55 -08:00