scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-17 04:34:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 172
litellm/tests/integration/providers
History
Yuneng Jiang f0bc6c31f8
test(integration): hand every harness server a port from one per-worker block 
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.

tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.

The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
2026-10-15 14:07:13 -07:00
..
_api_base_trailing_slash_support.py fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
_assistants_route_errors_support.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
_cache_control_marks_support.py fix(responses): propagate cache_control through responses->chat completion bridge (#32511) 2026-10-14 15:44:34 -07:00
_coralbricks.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
_count_tokens_system_lift.py fix(anthropic): count a leading system run through count_tokens' system parameter (#45463) 2026-10-08 22:34:25 -07:00
_dashscope_video_support.py feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos (#43706) 2026-10-14 10:10:07 -07:00
_gemini_tts_voice_support.py perf(ci): isolate and shard integration suites (#44146) 2026-10-10 11:15:19 -07:00
_idle_upstream_support.py fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355) 2026-10-14 10:50:11 -07:00
_mantle_gpt_prompt_cache_support.py fix(bedrock_mantle): send OpenAI explicit prompt cache breakpoints for GPT-5.6 and newer (#38729) 2026-10-08 02:07:22 -07:00
_responses_bridge_prompt_cache_breakpoint.py fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge (#44119) 2026-10-07 16:25:36 -07:00
_sagemaker_chat_container.py fix(sagemaker): keep AWS credentials out of the sagemaker_chat request body (#45724) 2026-10-10 15:30:54 -07:00
_tool_search_support.py fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440) 2026-10-14 18:52:32 -07:00
_vertex_batch_rows.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
fine_tuning_proxy_config.yaml test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
openai_fine_tuning.jsonl test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_a2a_sdk_upstream_wire.py test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940) 2026-10-11 06:24:57 -07:00
test_anthropic_count_tokens_system_lift_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_anthropic_passthrough_wire.py test(integration): cover pass_through endpoint contracts (#44690) 2026-10-14 07:38:17 +00:00
test_anthropic_refusal_stop_details_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_anthropic_refusal_stop_details_wire.py fix(anthropic): keep refusal stop_reason and stop_details on the logged response (#46104) 2026-10-14 14:42:49 -07:00
test_anthropic_skills_create_wire.py test(integration): cover anthropic_messages endpoint contracts (#44704) 2026-10-14 06:57:54 +00:00
test_anthropic_system_cache_control_wire.py test(integration): assert /v1/responses usage reports Anthropic system cache write then read (#42855) 2026-09-23 18:52:06 -07:00
test_anthropic_thinking_signature_logging_wire.py fix(anthropic): stop repeating streamed thinking text in the signature chunk (#44127) 2026-10-02 19:38:41 -07:00
test_anthropic_thinking_signature_stream_wire.py test: expect the list-position index on streamed thinking blocks in the integration suite (#46618) 2026-10-15 05:52:56 +00:00
test_anthropic_tool_search_result_chaos.py fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440) 2026-10-14 18:52:32 -07:00
test_anthropic_tool_search_result_matrix_wire.py fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440) 2026-10-14 18:52:32 -07:00
test_anthropic_tool_search_result_modify_params_wire.py fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440) 2026-10-14 18:52:32 -07:00
test_anthropic_tool_search_result_passthrough_wire.py fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440) 2026-10-14 18:52:32 -07:00
test_anthropic_tool_search_result_wire.py fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440) 2026-10-14 18:52:32 -07:00
test_api_base_trailing_slash_chaos.py fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
test_api_base_trailing_slash_owned_proxy.py fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
test_api_base_trailing_slash_wire.py fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
test_assemblyai_passthrough.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_assistants_evals_wire.py test(integration): pin assistants and evals endpoint contracts (#44734) 2026-10-14 07:08:35 +00:00
test_assistants_route_errors_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_assistants_route_errors_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_audit_any_sweep_harness_endpoint.py refactor(types): replace Any with proven types in 9 files (#44389) 2026-10-03 21:03:16 +00:00
test_audit_any_sweep_vector_store.py refactor(types): replace Any with proven types in 9 files (#44389) 2026-10-03 21:03:16 +00:00
test_azure_ai_chat_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_azure_ai_claude_messages_wire.py test(integration): cover anthropic_messages endpoint contracts (#44704) 2026-10-14 06:57:54 +00:00
test_azure_ai_flux2_image_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_azure_ai_rerank_auth_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_azure_embedding_wire.py test(integration): cover embeddings endpoint contracts (#44660) 2026-10-14 03:38:12 +00:00
test_bedrock_anthropic_beta_header_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_bedrock_auth_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_batch_blank_s3_env_wire.py test(e2e): assert only litellm-owned batch behavior and move the blank S3 env pin to an integration test (#43321) 2026-09-26 11:17:03 -07:00
test_bedrock_batch_files_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_batch_retrieve_status_wire.py fix(bedrock): map Scheduled batch jobs to in_progress on retrieve (#45157) 2026-10-07 18:08:51 -07:00
test_bedrock_claude_thinking_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_converse_client_metadata_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_converse_config_blocks_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_converse_lookaround_regex_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_bedrock_converse_lookaround_regex_wire.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_bedrock_converse_missing_content_wire.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_bedrock_converse_redacted_thinking_wire.py test: expect the list-position index on streamed thinking blocks in the integration suite (#46618) 2026-10-15 05:52:56 +00:00
test_bedrock_converse_stream_event_frames_wire.py test(integration): update stale CircleCI integration expectations (#46570) 2026-10-14 18:21:59 -07:00
test_bedrock_deepseek_reasoning_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_embedding_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_gpt5_reasoning_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_gpt_responses_native_wire.py feat(bedrock): serve gpt-5.6+ chat completions natively by default, with chat_completions/ opt-in for gpt-oss and grok (#44307) 2026-10-02 19:55:54 -07:00
test_bedrock_invoke_cache_usage_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_kimi_k3_cache_point_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_bedrock_knowledge_base_user_context_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_mantle_claude_chat_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_bedrock_mantle_claude_chat_wire.py test(integration): assert the Messages API health probe on Mantle Claude (#44612) 2026-10-05 15:28:02 -07:00
test_bedrock_mantle_codex_input_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_mantle_context_overflow_fallbacks_wire.py test(integration): update stale CircleCI integration expectations (#46570) 2026-10-14 18:21:59 -07:00
test_bedrock_mantle_context_overflow_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_bedrock_mantle_count_tokens_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_bedrock_mantle_gpt_prompt_cache_breakpoint_wire.py fix(bedrock_mantle): send OpenAI explicit prompt cache breakpoints for GPT-5.6 and newer (#38729) 2026-10-08 02:07:22 -07:00
test_bedrock_mantle_output_limit_wire.py test(integration): update stale CircleCI integration expectations (#46570) 2026-10-14 18:21:59 -07:00
test_bedrock_mantle_responses_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_mantle_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_marengo_embed_3_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_model_discovery_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_bedrock_model_discovery_wire.py fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on (#43981) 2026-10-09 14:14:17 -07:00
test_bedrock_passthrough_stream_wire.py test(integration): cover pass_through endpoint contracts (#44690) 2026-10-14 07:38:17 +00:00
test_bedrock_rerank_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_role_configuration.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_runtime_chat_completions_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_bedrock_runtime_chat_completions_sad_wire.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_bedrock_runtime_chat_completions_wire.py fix(bedrock): drop stop sequences for GPT 5.6 and newer instead of forwarding them to a 400 (#45664) 2026-10-09 15:00:24 -07:00
test_bedrock_stream_timeout_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_bedrock_stream_timeout_wire.py fix(responses): stop SDK retries nesting under router retries, and unbreak CircleCI integration tests (#44791) 2026-10-06 15:57:33 +00:00
test_bedrock_thinking_tokens_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_cache_control_tool_call_marks_owned_proxy.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_cache_control_tool_call_marks_wire.py test(integration): follow #46101 in the breakpoint cache mark census expectation (#46509) 2026-10-14 20:55:43 -07:00
test_chat_completions_tool_search_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_chat_completions_tool_search_history_wire.py fix(anthropic)!: translate the OpenAI tool_search tool on /v1/chat/completions for Claude models (#46389) 2026-10-15 04:47:05 +00:00
test_chat_completions_tool_search_wire.py fix(anthropic)!: translate the OpenAI tool_search tool on /v1/chat/completions for Claude models (#46389) 2026-10-15 04:47:05 +00:00
test_chatgpt_responses_caller_stream_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_chatgpt_responses_caller_stream_wire.py fix(responses): honor caller stream flag when provider forces SSE (internal copy of #34095) (#41235) 2026-10-06 03:30:47 +00:00
test_chatgpt_responses_top_level_error_wire.py fix(responses): keep the provider code and message on top-level stream error events (#46438) 2026-10-14 16:16:46 -07:00
test_cloudflare_clef_images_chaos.py feat(decisions)!: send images to Cloudflare Clef through its images field (#45734) 2026-10-14 12:14:48 +00:00
test_cloudflare_clef_images_wire.py feat(decisions)!: send images to Cloudflare Clef through its images field (#45734) 2026-10-14 12:14:48 +00:00
test_cloudflare_clef_omni_media_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_cloudflare_clef_omni_media_wire.py feat(decisions): support Clef-omni audio and video (#45811) 2026-10-12 20:29:49 +00:00
test_container_file_routes_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_coralbricks_provider_chaos.py feat(providers): add CoralBricks as a native provider (#44700) 2026-10-12 21:56:59 +00:00
test_coralbricks_provider_env_wire.py feat(providers): add CoralBricks as a native provider (#44700) 2026-10-12 21:56:59 +00:00
test_coralbricks_provider_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_cursor_responses_wire.py fix(cache): keep cache hits free of unset fields, and follow intentional behavior changes in CircleCI suites (#46359) 2026-10-14 08:56:05 -07:00
test_dashscope_chat_wire.py fix(proxy): keep provider refusal, logprobs and system_fingerprint in chat completions (#44788) 2026-10-13 13:17:40 -07:00
test_dashscope_video_chaos.py feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos (#43706) 2026-10-14 10:10:07 -07:00
test_dashscope_video_wire.py feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos (#43706) 2026-10-14 10:10:07 -07:00
test_databricks_agent_wire.py feat(agents): support Databricks Agent Platform agents in the agent gateway (#45681) 2026-10-15 12:42:06 -07:00
test_databricks_chat_wire.py test: cover response contracts across translation endpoints (#46632) 2026-10-15 00:05:09 -07:00
test_databricks_decisions_wire.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_databricks_oauth_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_decision_model_guardrail.py fix(guardrails): resolve decision models in caller team scope (#46085) 2026-10-15 11:43:36 -07:00
test_decisions_box_deployment_flag_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_decisions_box_deployment_flag_wire.py fix(decisions): read a deployment's own supports_box_questions flag before the cost map (#46696) 2026-10-15 12:34:36 -07:00
test_decisions_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_decisions_openai_format_wire.py fix(decisions): keep box answers from OpenAI-format providers instead of turning them into refusals (#46633) 2026-10-15 14:01:59 +00:00
test_decisions_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_deepseek_vision_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_device_code_login_guard_boot.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_device_code_login_guard_wire.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_embedding_cache_partial_hit_wire.py test(integration): cover embeddings endpoint contracts (#44660) 2026-10-14 03:38:12 +00:00
test_embedding_token_array_wire.py test(integration): cover embeddings endpoint contracts (#44660) 2026-10-14 03:38:12 +00:00
test_fake_openai_upstream_routes.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_fal_ai_chat_wire.py fix(fal_ai): honour global api_base for image generation and reject non-string reasoning_effort with 400 (#42512) 2026-09-22 13:13:54 -07:00
test_fal_ai_image_wire.py test(integration): cover images endpoint contracts (#44653) 2026-10-14 03:40:29 +00:00
test_fal_ai_passthrough_wire.py fix(fal_ai): align /fal_ai queue gate with the pricer and normalise resolution type (#42505) 2026-09-22 13:12:07 -07:00
test_fal_ai_video_wire.py fix(fal_ai): reuse the status client and headers on the video result probe (#42511) 2026-09-22 13:12:51 -07:00
test_files_batches_wire.py test(integration): cover files_batches endpoint contracts (#44703) 2026-10-14 06:57:41 +00:00
test_fireworks_ai_router_slug_wire.py feat(fireworks_ai): route and list the auto, auto-instant and firerouter routers (#43641) 2026-09-28 19:08:16 -07:00
test_fireworks_ai_session_affinity_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_gemini_agents_wire.py test(integration): cover google_genai endpoint contracts (#44680) 2026-10-14 04:01:35 +00:00
test_gemini_chat_video_metadata_wire.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_gemini_embedding_file_block_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_gemini_embedding_file_block_wire.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_gemini_image_config_and_usage_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_gemini_interactions_wire.py test(integration): cover google_genai endpoint contracts (#44680) 2026-10-14 04:01:35 +00:00
test_gemini_omitted_temperature.py test(translation): share rich fixtures across Claude, GPT and Gemini providers (#46585) 2026-10-14 19:53:09 -07:00
test_gemini_thinking_replay_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_gemini_thinking_replay_wire.py test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs (#45298) 2026-10-08 02:37:55 -07:00
test_gemini_tool_result_part_reference_chaos.py fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results (#38284) 2026-10-15 06:26:47 -07:00
test_gemini_tool_result_part_reference_wire.py fix(vertex_ai): prevent Gemini from misinterpreting $ref in tool results (#38284) 2026-10-15 06:26:47 -07:00
test_gemini_tts_voice_mapping_owned_proxy.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_gemini_tts_voice_mapping_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_google_genai_adapter_wire.py fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836) 2026-10-15 08:06:49 -07:00
test_google_genai_native_wire.py test(integration): cover google_genai endpoint contracts (#44680) 2026-10-14 04:01:35 +00:00
test_google_genai_proxy_sdk_wire.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_google_genai_vertex_routes_wire.py fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836) 2026-10-15 08:06:49 -07:00
test_hosted_vllm_reasoning_content_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_hosted_vllm_reasoning_content_wire.py test: cover response contracts across translation endpoints (#46632) 2026-10-15 00:05:09 -07:00
test_idle_upstream_keepalive_edges_wire.py test(integration): update stale CircleCI integration expectations (#46570) 2026-10-14 18:21:59 -07:00
test_idle_upstream_keepalive_override_wire.py fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355) 2026-10-14 10:50:11 -07:00
test_idle_upstream_reconnect_chaos_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_idle_upstream_reconnect_routes_wire.py fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355) 2026-10-14 10:50:11 -07:00
test_idle_upstream_reconnect_rust_messages_wire.py fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355) 2026-10-14 10:50:11 -07:00
test_idle_upstream_reconnect_wire.py fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355) 2026-10-14 10:50:11 -07:00
test_image_gen_drop_params_wire.py test(integration): cover images endpoint contracts (#44653) 2026-10-14 03:40:29 +00:00
test_inferenceapis_context_window_fallbacks_wire.py feat(providers): add Inference APIs as OpenAI-compatible provider (#41669) 2026-10-15 08:17:36 -07:00
test_inferenceapis_wire.py feat(providers): add Inference APIs as OpenAI-compatible provider (#41669) 2026-10-15 08:17:36 -07:00
test_interactions_openai_bridge_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_internal_params_wire.py fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
test_lm_studio_chat_wire.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_moderations.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_moderations_wire.py test(integration): cover small_inference endpoint contracts (#44654) 2026-10-14 04:06:15 +00:00
test_nvidia_nim_ranking_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_ocr_multipart_wire.py test(integration): cover small_inference endpoint contracts (#44654) 2026-10-14 04:06:15 +00:00
test_ocr_router_wire.py test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
test_ollama_prompt_tools_chaos.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_ollama_prompt_tools_wire.py fix(ollama): one stream id, 400 on non-text tool content, separated tool results, no empty message item (#45177) 2026-10-07 19:08:26 -07:00
test_openai_chat_prompt_cache_breakpoint_wire.py fix(openai): carry cache_control to prompt_cache_breakpoint on chat completions to gpt-5.6 (#46101) 2026-10-13 15:45:49 -07:00
test_openai_chat_wire.py test(integration): cover chat_completions endpoint contracts (#44688) 2026-10-14 03:45:38 +00:00
test_openai_completion_wire.py test(integration): cover chat_completions endpoint contracts (#44688) 2026-10-14 03:45:38 +00:00
test_openai_count_tokens_tools_wire.py fix(count_tokens): count system and tools on Vertex Gemini, fall back to Mantle on Bedrock 404 (#46430) 2026-10-15 10:36:10 -04:00
test_openai_dialect_prompt_cache_breakpoint_owned_proxy.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_openai_embedding_timeout_wire.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_openai_embedding_wire.py test(integration): cover embeddings endpoint contracts (#44660) 2026-10-14 03:38:12 +00:00
test_openai_error_handling.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_openai_fine_tuning.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_openai_image_edit_wire.py test(integration): cover images endpoint contracts (#44653) 2026-10-14 03:40:29 +00:00
test_openai_image_variation_wire.py test(integration): cover images endpoint contracts (#44653) 2026-10-14 03:40:29 +00:00
test_openai_passthrough_files_wire.py test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
test_openai_passthrough_multipart_wire.py test(integration): cover pass_through endpoint contracts (#44690) 2026-10-14 07:38:17 +00:00
test_openai_realtime_transcription_url.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_openai_responses_prompt_cache_breakpoint_wire.py fix(openai): carry cache_control to prompt_cache_breakpoint on /v1/responses to gpt-5.6 (#46105) 2026-10-13 15:23:07 -07:00
test_openai_responses_tool_search_wire.py fix(responses): map Anthropic tool search to OpenAI server tool search (#46441) 2026-10-15 05:13:56 +00:00
test_openai_responses_websocket_wire.py fix(responses): keep the provider code and message on top-level stream error events (#46438) 2026-10-14 16:16:46 -07:00
test_openai_stream_text_usage_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00
test_provider_lookup_status.py test(integration): pin assistants and evals endpoint contracts (#44734) 2026-10-14 07:08:35 +00:00
test_rag_ingest_wire.py test(integration): cover vector_stores endpoint contracts (#44739) 2026-10-14 03:59:08 +00:00
test_realtime_transcription_guardrail_session_update.py test(integration): isolate shared proxy state behind six order-dependent flakes (#46172) 2026-10-13 09:55:11 -07:00
test_realtime_unguarded_voice_turns.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_realtime_webrtc_http.py test(integration): cover realtime endpoint contracts (#44689) 2026-10-14 03:49:34 +00:00
test_request_boundary.py fix(tests): resolve the integration support package without run.py's PYTHONPATH 2026-09-15 21:10:08 -07:00
test_rerank_latency_headers_wire.py test(integration): cover small_inference endpoint contracts (#44654) 2026-10-14 04:06:15 +00:00
test_responses_bridge_incomplete.py fix(responses): return the first message item as content for structured output on the chat bridge (#46508) 2026-10-14 16:58:00 -07:00
test_responses_bridge_input_audio_wire.py fix(responses): map input_audio blocks in the chat-to-Responses bridge (#45224) 2026-10-07 22:25:35 -07:00
test_responses_bridge_item_marks_guardrail_wire.py fix(responses): propagate cache_control through responses->chat completion bridge (#32511) 2026-10-14 15:44:34 -07:00
test_responses_bridge_message_cache_control_wire.py fix(responses): propagate cache_control through responses->chat completion bridge (#32511) 2026-10-14 15:44:34 -07:00
test_responses_bridge_message_cache_control_wire_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_responses_bridge_namespace_tools.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_responses_bridge_prompt_cache_breakpoint.py fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge (#44119) 2026-10-07 16:25:36 -07:00
test_responses_bridge_prompt_cache_breakpoint_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_responses_bridge_prompt_cache_breakpoint_wire.py fix(anthropic): cache tool result screenshots on the /v1/messages to Responses bridge (#46324) 2026-10-14 08:12:11 +00:00
test_responses_bridge_prompt_cache_breakpoint_wire_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_responses_bridge_stream_output_items_wire.py fix(ollama): one stream id, 400 on non-text tool content, separated tool results, no empty message item (#45177) 2026-10-07 19:08:26 -07:00
test_responses_bridge_tool_search_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_responses_bridge_tool_search_replay.py fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444) 2026-10-14 21:40:14 -07:00
test_responses_bridge_tool_search_replay_wire.py fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models (#45882) 2026-10-14 20:42:22 -07:00
test_responses_bridge_tool_search_undeferred_wire.py fix(responses): translate OpenAI tool_search to Anthropic tool search for Claude models (#45882) 2026-10-14 20:42:22 -07:00
test_responses_bridge_tool_search_wire.py fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444) 2026-10-14 21:40:14 -07:00
test_responses_client_header_forwarding_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_responses_compact_wire.py fix(proxy): check previous_response_id ownership on /v1/responses/compact (#46408) 2026-10-14 17:57:34 +00:00
test_responses_core_wire.py test(integration): cover responses endpoint contracts (#44733) 2026-10-14 06:59:15 +00:00
test_responses_error_status_wire.py test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
test_responses_lifecycle_wire.py test(integration): cover responses endpoint contracts (#44733) 2026-10-14 06:59:15 +00:00
test_responses_minted_reasoning_replay_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_responses_minted_reasoning_replay_wire.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_responses_reasoning_item_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_responses_websocket_session_limit.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_s3_wire.py test: fix stale and flaky tests across CircleCI, GHA and Buildkite (#45530) 2026-10-09 11:40:14 -07:00
test_sagemaker_chat_credential_sources.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_sagemaker_chat_wire.py fix(sagemaker): keep AWS credentials out of the sagemaker_chat request body (#45724) 2026-10-10 15:30:54 -07:00
test_sagemaker_embedding_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_sagemaker_embedding_wire.py feat(sagemaker)!: inference components and OpenAI-compatible embeddings (#44985) 2026-10-12 22:25:52 +00:00
test_search_wire.py fix(search): bill an unpriced fallback search tool at its own price, not the failed primary's (#46734) 2026-10-15 12:20:33 -07:00
test_strands_decider_agentcore_wire.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_tencent_chat_wire.py fix(proxy): keep provider refusal, logprobs and system_fingerprint in chat completions (#44788) 2026-10-13 13:17:40 -07:00
test_validated_reply_boundaries_under_faults.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_vector_store_create_wire.py test(integration): cover vector_stores endpoint contracts (#44739) 2026-10-14 03:59:08 +00:00
test_vector_store_search_wire.py test(integration): cover vector_stores endpoint contracts (#44739) 2026-10-14 03:59:08 +00:00
test_vertex_anthropic_inline_tools_beta_wire.py test(integration): remove six setup causes of CircleCI integration reds (#46490) 2026-10-14 12:00:35 -07:00
test_vertex_anthropic_structured_output_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_vertex_batch_output_info_wire.py fix(vertex_ai): keep batch output_file_id null until Vertex reports outputInfo (#43030) 2026-09-24 14:05:31 -07:00
test_vertex_batch_row_line_errors_chaos.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_vertex_batch_row_line_errors_owned_proxy.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_vertex_batch_row_line_errors_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_vertex_credentials_source_wire.py fix(vertex): report an unreadable credentials file as a read failure (#40173) 2026-10-15 02:03:56 -07:00
test_vertex_embedding_batch_file_block_wire.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_vertex_embedding_file_block_wire.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_vertex_embedding_wire.py test(integration): cover embeddings endpoint contracts (#44660) 2026-10-14 03:38:12 +00:00
test_vertex_gemini_count_tokens_wire.py fix(count_tokens): count system and tools on Vertex Gemini, fall back to Mantle on Bedrock 404 (#46430) 2026-10-15 10:36:10 -04:00
test_vertex_gemini_fragmented_stream_wire.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_vertex_gemini_function_call_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00
test_vertex_partner_count_tokens_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_vertex_partner_messages_wire.py test(integration): cover anthropic_messages endpoint contracts (#44704) 2026-10-14 06:57:54 +00:00
test_vertex_passthrough_wire.py fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440) 2026-10-14 18:52:32 -07:00
test_vertex_realtime_multi_region_host.py fix(vertex_ai): dial the multi-region Live API host for realtime sessions (#45166) 2026-10-07 17:39:38 -07:00
test_vertex_timestamps_utc_wire.py fix(vertex_ai): parse Vertex timestamps as UTC so created_at is not shifted by the host timezone (#45550) 2026-10-10 15:18:15 -07:00
test_voyage_mongodb_host_wire.py test(integration): hand every harness server a port from one per-worker block 2026-10-15 14:07:13 -07:00
test_xai_batch_output_file_wire.py fix(xai): name a completed xAI batch's results file and stop looking it up on xAI's files API (#45559) 2026-10-14 22:05:43 -07:00
test_xai_server_tool_usage_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_xai_web_search_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_xiaomi_mimo_wire.py fix(proxy): keep provider refusal, logprobs and system_fingerprint in chat completions (#44788) 2026-10-13 13:17:40 -07:00