litellm/tests/integration/sdk
Mateo Wang c4360bbb59
fix(harness): set PWD to the working directory of local sandbox commands (#47530)
* fix(harness): set PWD to the working directory of local sandbox commands

LocalSandbox started CLIs in the sandbox workdir but passed the parent's PWD through. OpenCode reads PWD as its project directory, so a litellm.agent() turn worked in the Python process cwd instead of the sandbox.

* fix(harness): re-home resumed OpenCode sessions into the sandbox folder

OpenCode files each session under the folder it ran in and only streams a resumed session's events from that folder, so a session saved before PWD pointed at the sandbox, or resumed in another sandbox, hung. Resume now runs opencode export and import in the sandbox first. Also types the CLI handler's config so its member accesses are known.

* fix(harness): fail an OpenCode resume that cannot move its session, with a timeout

The export and import step now has a 30 s bound, and a failure or timeout raises HarnessError instead of leaving the resumed turn waiting forever. A restart of a session the handler already ran skips the step.

* fix(harness): resume an OpenCode session elsewhere as a copy and fail the turn when it cannot

Importing the exported session moved it, so the original stopped working
where it was saved. Resume now imports a copy under fresh ids, filed under
the sandbox folder, and continues the copy. A failed or timed-out export or
import ends the next turn with runtime_error, as base does for a missing
session, instead of raising from agent_resume.

* fix(harness): retry a failed OpenCode resume on the next turn and keep resumed copies stable

Resume now only records the saved id, and the first turn runs the OpenCode export and import under the caller's timeout. A failed export fails that turn and the next turn tries again. The local sandbox kills a command when its run is cancelled, and the copy names the sandbox folder so OpenCode 1.15.5 and older keep it there.

* test(harness): keep subprocess tests out of tests/unit

The PWD regression test moves to tests/integration/sdk with an isolated child interpreter, and the cancel test checks collect_output with an in-process process instead of a real one. collect_output takes the sandbox Process protocol.

* fix(harness): rewrite only id fields when copying an OpenCode session for resume
2026-10-20 20:06:11 +00:00
..
conftest.py fix(tests): drain SDK logging before restoring callbacks (#46924) 2026-10-16 12:39:26 -07:00
test_aiohttp_session_rebuild_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_anthropic_messages_bridge_mid_stream_failure_sdk.py test: extend runtime regression contracts for Messages and SDK (#46641) 2026-10-15 17:51:15 +00:00
test_azure_prompt_shield_tuple_messages.py fix(guardrails): restore Azure guardrail get_user_prompt dispatch and allow logging (#44067) 2026-10-01 23:30:32 -07:00
test_bedrock_converse_redacted_thinking_router_sdk.py fix(bedrock)!: replay redacted_thinking on Converse and keep its data as Claude's own string (#46532) 2026-10-14 19:08:14 -07:00
test_bedrock_converse_stream_sync_decoder_wire.py fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it (#47100) 2026-10-18 00:06:07 +00:00
test_bedrock_model_discovery_sdk.py fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on (#43981) 2026-10-09 14:14:17 -07:00
test_bedrock_native_output_fallback_sdk.py fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it (#47100) 2026-10-18 00:06:07 +00:00
test_bedrock_native_output_register_model_sdk.py fix(bedrock): use native structured output on Haiku 5.5 and Sonnet 5.5, and fix Fable 5 and Sonnet 4.6 schema 400s (#46870) 2026-10-16 20:57:23 -07:00
test_cache_control_census_pydantic_messages_sdk.py fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it (#46591) 2026-10-15 14:46:43 -07:00
test_caching_redis_sdk.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_callback_logging_cache.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_databricks_json_schema_refs_sdk.py fix(databricks): keep #/$defs refs in json_schema response_format (#45659) 2026-10-09 18:51:36 -07:00
test_device_code_login_guard_sdk.py fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it (#46591) 2026-10-15 14:46:43 -07:00
test_dual_cache_redis.py ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) 2026-10-01 13:00:54 -07:00
test_exa_search_envelope_sdk.py fix(exa): return highlights, highlight scores, score and the response envelope from Exa search (#47423) 2026-10-19 15:45:56 -07:00
test_gemini_long_decode_sync_sdk.py feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240) 2026-10-15 22:54:23 +00:00
test_gemini_tts_speech_logging_sdk.py fix(speech)!: correctly propagate spend logging and usage for Gemini TTS bridge (#44551) 2026-10-19 23:02:11 +00:00
test_generic_api_callback.py perf(ci): isolate and shard integration suites (#44146) 2026-10-10 11:15:19 -07:00
test_google_genai_adapter_stream_usage_sdk.py fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836) 2026-10-15 08:06:49 -07:00
test_http2_wire.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
test_huggingface_tokenizer_downloads_sdk.py fix(token_counter): count DeepSeek V4 text with DeepSeek's own tokenizer (#47353) 2026-10-20 03:43:35 +00:00
test_idle_upstream_reconnect_sdk.py fix(http): restore the 15s aiohttp keep-alive default in the Python and Rust clients (#46355) 2026-10-14 10:50:11 -07:00
test_interception_converted_stream_options_sdk.py fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream (#45433) 2026-10-09 22:16:02 -07:00
test_langfuse_redaction_sdk.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_local_sandbox_sdk.py fix(harness): set PWD to the working directory of local sandbox commands (#47530) 2026-10-20 20:06:11 +00:00
test_ocr_matrix_sdk.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_openai_sdk_client_sessions_wire.py fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927) 2026-10-09 19:45:50 -07:00
test_owned_redis_ports.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
test_provider_budget_redis.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_provider_error_and_extra_body_families_wire.py feat(providers): add Inference APIs as OpenAI-compatible provider (#41669) 2026-10-15 08:17:36 -07:00
test_redis_breaker_metrics_multiprocess_sdk.py fix(caching): report Redis circuit-breaker metrics through the Prometheus services logger (#46484) 2026-10-14 14:29:54 -07:00
test_redis_cluster_iam_auth.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
test_redis_service_metrics.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_response_cache_sync_sdk.py fix(anthropic): keep refusal stop_reason and stop_details on the logged response (#46104) 2026-10-14 14:42:49 -07:00
test_responses_bridge_stream_errors_sdk.py test(integration): update stale CircleCI integration expectations (#46570) 2026-10-14 18:21:59 -07:00
test_responses_bridge_sync_reasoning_sdk.py fix(bedrock)!: keep reasoning, thinking blocks and the json_mode finish_reason on fake streams (#46863) 2026-10-16 17:27:57 +00:00
test_responses_bridge_tool_search_sdk.py fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444) 2026-10-14 21:40:14 -07:00
test_router_behavior_wire.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_router_budget_limiter_wire.py fix(router): stop queueing budget increments for Redis when no Redis is configured (#46831) 2026-10-16 15:52:42 +00:00
test_router_cooldown_fallback_sdk.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_router_include_fallback_errors_wire.py fix(router): keep include_fallback_errors off the provider call on the sync Router path (#45218) 2026-10-07 22:25:12 -07:00
test_router_lowest_latency_wire.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_router_redis_tls_url.py test: move routing, caching and callback tests out of local_testing (#45761) 2026-10-10 09:33:09 -07:00
test_router_sync_stream_fallback_wire.py fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493) 2026-10-14 14:19:25 -07:00
test_router_tool_call_stream_fallback_budget_wire.py fix(router): read the first stream chunk before returning so a failure before any token runs through the retry and fallback budget (#46493) 2026-10-14 14:19:25 -07:00
test_router_unsupported_params_fallback_wire.py fix(router): add fallback_on_unsupported_params so param rejections can raise instead of falling back (#46350) 2026-10-14 14:43:01 -07:00
test_sagemaker_embedding_sdk.py feat(sagemaker)!: inference components and OpenAI-compatible embeddings (#44985) 2026-10-12 22:25:52 +00:00
test_slack_daily_report_redis.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_stream_error_api_base_sdk.py fix(streaming): report the deployment's api_base when a stream fails mid-way (#47045) 2026-10-17 15:41:24 +00:00
test_stream_open_error_provider_headers_sdk.py fix(streaming): keep provider response headers on SDK streaming errors (#41528) 2026-10-15 08:11:54 -07:00
test_sync_stream_connection_release_wire.py fix(streaming): add CustomStreamWrapper.close() and release the connection when a sync stream ends (#46354) 2026-10-14 19:14:04 +00:00
test_text_completion_stream_rebuild_sdk.py fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions (#44349) 2026-10-20 05:31:39 +00:00
test_tool_loop_server_tool_replay_sdk.py fix(anthropic): keep tool_search_tool_result so the search call replays with its result (#46440) 2026-10-14 18:52:32 -07:00
test_topaz_image_variation_trailing_slash_sdk.py fix(llms): sanitize trailing slashes in provider get_complete_url (#42887) 2026-10-15 02:59:46 -07:00
test_usage_based_routing_sdk_redis_reads.py refactor: clean up fresh tech debt from 2026-09-29 (#43830) 2026-09-30 02:40:37 -07:00
test_usage_routing_counter_ttl.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_vertex_batch_row_line_errors_sdk.py fix(vertex_ai): return 400 with file line number for malformed batch JSONL row (#40097) 2026-10-15 03:21:48 -07:00
test_vertex_gemini_image_batch_cost_sdk.py refactor: expose public hidden_params accessors (#44668) 2026-10-07 00:01:33 -07:00
test_vertex_gemini_stream_timeout_wire.py fix(vertex_ai): pass the per-request timeout to Gemini streaming calls (#46349) 2026-10-14 10:51:18 -07:00
test_vertex_partner_count_tokens_sdk.py fix(responses)!: count input tokens the way inference sends the request (#47363) 2026-10-20 15:13:59 +00:00