scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-10 03:28:53 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 84
litellm/tests/integration/providers
History
Exact
devin-ai-integration[bot] d34ad35281
fix(anthropic): count a leading system run through count_tokens' system parameter (#45463) 
* fix(anthropic): count a leading system run through count_tokens' system parameter

Anthropic's count_tokens rejects role "system" at the head of messages, so
/v1/responses/input_tokens with instructions, and /v1/messages/count_tokens
with a system-role message, fell back to the local tokenizer. The shared
Anthropic count_tokens transformation now lifts the leading run of system
messages into the top-level system parameter, the way the chat path sends
it, after any system the caller set. Anthropic direct, Azure AI Anthropic,
and Bedrock Mantle share that transformation.

* test(integration): cover the count_tokens leading-system lift across Anthropic, Azure AI and Bedrock Mantle

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-08 22:34:25 -07:00
..
_cache_control_marks_support.py fix(caching): count tool_call cache_control marks in the injection census (#43556) 2026-10-03 17:33:48 -07:00
_count_tokens_system_lift.py fix(anthropic): count a leading system run through count_tokens' system parameter (#45463) 2026-10-08 22:34:25 -07:00
_mantle_gpt_prompt_cache_support.py fix(bedrock_mantle): send OpenAI explicit prompt cache breakpoints for GPT-5.6 and newer (#38729) 2026-10-08 02:07:22 -07:00
_responses_bridge_prompt_cache_breakpoint.py fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge (#44119) 2026-10-07 16:25:36 -07:00
test_anthropic_count_tokens_system_lift_wire.py fix(anthropic): count a leading system run through count_tokens' system parameter (#45463) 2026-10-08 22:34:25 -07:00
test_anthropic_system_cache_control_wire.py test(integration): assert /v1/responses usage reports Anthropic system cache write then read (#42855) 2026-09-23 18:52:06 -07:00
test_anthropic_thinking_signature_logging_wire.py fix(anthropic): stop repeating streamed thinking text in the signature chunk (#44127) 2026-10-02 19:38:41 -07:00
test_anthropic_thinking_signature_stream_wire.py fix(anthropic): stop repeating streamed thinking text in the signature chunk (#44127) 2026-10-02 19:38:41 -07:00
test_audit_any_sweep_harness_endpoint.py refactor(types): replace Any with proven types in 9 files (#44389) 2026-10-03 21:03:16 +00:00
test_audit_any_sweep_vector_store.py refactor(types): replace Any with proven types in 9 files (#44389) 2026-10-03 21:03:16 +00:00
test_azure_ai_chat_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_azure_ai_flux2_image_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_azure_ai_rerank_auth_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_anthropic_beta_header_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_bedrock_auth_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_batch_blank_s3_env_wire.py test(e2e): assert only litellm-owned batch behavior and move the blank S3 env pin to an integration test (#43321) 2026-09-26 11:17:03 -07:00
test_bedrock_batch_files_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_batch_retrieve_status_wire.py fix(bedrock): map Scheduled batch jobs to in_progress on retrieve (#45157) 2026-10-07 18:08:51 -07:00
test_bedrock_claude_thinking_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_converse_client_metadata_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_converse_config_blocks_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_converse_lookaround_regex_chaos.py feat(bedrock): drop lookaround regex patterns from tool schemas for Converse models that reject them (#44138) 2026-10-02 19:47:25 -07:00
test_bedrock_converse_lookaround_regex_wire.py feat(bedrock): drop lookaround regex patterns from tool schemas for Converse models that reject them (#44138) 2026-10-02 19:47:25 -07:00
test_bedrock_converse_missing_content_wire.py fix(bedrock): accept Converse messages with no content key (#43936) 2026-10-01 15:14:34 -07:00
test_bedrock_converse_stream_event_frames_wire.py fix(bedrock): surface unrecognized converse-stream event frames instead of an empty turn (#44112) 2026-10-02 16:06:26 -07:00
test_bedrock_deepseek_reasoning_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_embedding_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_gpt5_reasoning_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_gpt_responses_native_wire.py feat(bedrock): serve gpt-5.6+ chat completions natively by default, with chat_completions/ opt-in for gpt-oss and grok (#44307) 2026-10-02 19:55:54 -07:00
test_bedrock_invoke_cache_usage_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_kimi_k3_cache_point_wire.py fix(bedrock): stop emitting Converse cachePoint blocks for Kimi K3 (#44292) 2026-10-03 18:11:46 +00:00
test_bedrock_knowledge_base_user_context_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_mantle_claude_chat_chaos.py fix(bedrock_mantle): route Claude chat completions to the native Messages endpoint (#43646) 2026-10-03 15:24:41 -07:00
test_bedrock_mantle_claude_chat_wire.py test(integration): assert the Messages API health probe on Mantle Claude (#44612) 2026-10-05 15:28:02 -07:00
test_bedrock_mantle_codex_input_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_mantle_count_tokens_wire.py fix(anthropic): count a leading system run through count_tokens' system parameter (#45463) 2026-10-08 22:34:25 -07:00
test_bedrock_mantle_gpt_prompt_cache_breakpoint_wire.py fix(bedrock_mantle): send OpenAI explicit prompt cache breakpoints for GPT-5.6 and newer (#38729) 2026-10-08 02:07:22 -07:00
test_bedrock_mantle_responses_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_mantle_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_marengo_embed_3_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_passthrough_stream_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_rerank_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_bedrock_role_configuration.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_bedrock_runtime_chat_completions_chaos.py test(ci): fix six CircleCI test regressions on main (#44429) 2026-10-03 20:55:01 -07:00
test_bedrock_runtime_chat_completions_sad_wire.py feat(bedrock): serve gpt-5.6+ chat completions natively by default, with chat_completions/ opt-in for gpt-oss and grok (#44307) 2026-10-02 19:55:54 -07:00
test_bedrock_runtime_chat_completions_wire.py feat(bedrock): serve gpt-5.6+ chat completions natively by default, with chat_completions/ opt-in for gpt-oss and grok (#44307) 2026-10-02 19:55:54 -07:00
test_bedrock_stream_timeout_chaos.py fix(bedrock): honor the per-request timeout on Converse and Invoke streaming (internal copy of #38210) (#44134) 2026-10-03 19:29:15 +00:00
test_bedrock_stream_timeout_wire.py fix(responses): stop SDK retries nesting under router retries, and unbreak CircleCI integration tests (#44791) 2026-10-06 15:57:33 +00:00
test_bedrock_thinking_tokens_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_cache_control_tool_call_marks_owned_proxy.py fix(caching): count tool_call cache_control marks in the injection census (#43556) 2026-10-03 17:33:48 -07:00
test_cache_control_tool_call_marks_wire.py test(integration): run the response cache tool-call tests on a proxy without the message cap (#45350) 2026-10-08 04:24:13 -07:00
test_chatgpt_responses_caller_stream_chaos.py fix(responses): honor caller stream flag when provider forces SSE (internal copy of #34095) (#41235) 2026-10-06 03:30:47 +00:00
test_chatgpt_responses_caller_stream_wire.py fix(responses): honor caller stream flag when provider forces SSE (internal copy of #34095) (#41235) 2026-10-06 03:30:47 +00:00
test_container_file_routes_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_dashscope_chat_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_databricks_chat_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_databricks_oauth_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_decisions_chaos.py feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions (#45184) 2026-10-07 21:41:03 -07:00
test_decisions_wire.py feat(ui): add the evaluation mode to the Add Model form (#45481) 2026-10-08 19:53:27 -07:00
test_deepseek_vision_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_device_code_login_guard_boot.py fix(chatgpt,github_copilot): refuse device-code login inside an event loop or worker thread (#39585) 2026-10-06 03:20:07 +00:00
test_device_code_login_guard_wire.py fix(chatgpt,github_copilot): refuse device-code login inside an event loop or worker thread (#39585) 2026-10-06 03:20:07 +00:00
test_fal_ai_chat_wire.py fix(fal_ai): honour global api_base for image generation and reject non-string reasoning_effort with 400 (#42512) 2026-09-22 13:13:54 -07:00
test_fal_ai_image_wire.py fix(fal_ai): honour global api_base for image generation and reject non-string reasoning_effort with 400 (#42512) 2026-09-22 13:13:54 -07:00
test_fal_ai_passthrough_wire.py fix(fal_ai): align /fal_ai queue gate with the pricer and normalise resolution type (#42505) 2026-09-22 13:12:07 -07:00
test_fal_ai_video_wire.py fix(fal_ai): reuse the status client and headers on the video result probe (#42511) 2026-09-22 13:12:51 -07:00
test_fireworks_ai_router_slug_wire.py feat(fireworks_ai): route and list the auto, auto-instant and firerouter routers (#43641) 2026-09-28 19:08:16 -07:00
test_fireworks_ai_session_affinity_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_gemini_chat_video_metadata_wire.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_gemini_embedding_file_block_chaos.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_gemini_embedding_file_block_wire.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_gemini_image_config_and_usage_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_gemini_thinking_replay_chaos.py fix(gemini): stop replaying thinking block signatures to Gemini (#44661) 2026-10-06 02:31:53 +00:00
test_gemini_thinking_replay_wire.py test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs (#45298) 2026-10-08 02:37:55 -07:00
test_hosted_vllm_reasoning_content_chaos.py fix(hosted_vllm): keep reasoning_content on replayed assistant messages (#43599) 2026-09-30 13:12:28 -07:00
test_hosted_vllm_reasoning_content_wire.py test(ci): repair stale tests and flaky CI infrastructure (#43983) 2026-10-01 17:46:43 +00:00
test_image_gen_drop_params_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00
test_interactions_openai_bridge_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_internal_params_wire.py fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
test_nvidia_nim_ranking_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_ocr_router_wire.py test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
test_ollama_prompt_tools_chaos.py test(e2e): cover ollama and ollama_chat on chat completions, responses and messages (#45079) 2026-10-07 20:17:35 +00:00
test_ollama_prompt_tools_wire.py fix(ollama): one stream id, 400 on non-text tool content, separated tool results, no empty message item (#45177) 2026-10-07 19:08:26 -07:00
test_openai_chat_wire.py fix(responses): honor request cache controls on chat completions bridged to the Responses API (#44676) 2026-10-05 17:52:44 -07:00
test_openai_dialect_prompt_cache_breakpoint_owned_proxy.py fix(bedrock_mantle): send OpenAI explicit prompt cache breakpoints for GPT-5.6 and newer (#38729) 2026-10-08 02:07:22 -07:00
test_openai_image_edit_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_openai_passthrough_files_wire.py test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
test_openai_realtime_transcription_url.py fix(openai/realtime): drop model from upstream URL for intent=transcription (#43854) 2026-10-05 19:14:56 +00:00
test_openai_responses_websocket_wire.py test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
test_openai_stream_text_usage_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00
test_provider_lookup_status.py fix(proxy): return 4xx instead of 500 for missing required params, invalid pagination and unknown ids (#43787) 2026-10-02 22:48:53 -07:00
test_realtime_transcription_guardrail_session_update.py fix(realtime): run transcript guardrails on raw-path transcription sessions with a transcription-safe block (#44844) 2026-10-07 11:59:34 -07:00
test_request_boundary.py fix(tests): resolve the integration support package without run.py's PYTHONPATH 2026-09-15 21:10:08 -07:00
test_rerank_latency_headers_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_responses_bridge_incomplete.py fix(responses): merge bridged tool calls into the same choice as the text (#44346) 2026-10-03 07:00:52 +00:00
test_responses_bridge_input_audio_wire.py fix(responses): map input_audio blocks in the chat-to-Responses bridge (#45224) 2026-10-07 22:25:35 -07:00
test_responses_bridge_namespace_tools.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_responses_bridge_prompt_cache_breakpoint.py fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge (#44119) 2026-10-07 16:25:36 -07:00
test_responses_bridge_prompt_cache_breakpoint_chaos.py fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge (#44119) 2026-10-07 16:25:36 -07:00
test_responses_bridge_prompt_cache_breakpoint_wire.py fix(responses): map input_audio blocks in the chat-to-Responses bridge (#45224) 2026-10-07 22:25:35 -07:00
test_responses_bridge_prompt_cache_breakpoint_wire_chaos.py fix(responses): map input_audio blocks in the chat-to-Responses bridge (#45224) 2026-10-07 22:25:35 -07:00
test_responses_bridge_stream_output_items_wire.py fix(ollama): one stream id, 400 on non-text tool content, separated tool results, no empty message item (#45177) 2026-10-07 19:08:26 -07:00
test_responses_client_header_forwarding_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_responses_error_status_wire.py test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
test_responses_minted_reasoning_replay_chaos.py fix(responses): drop bridge-minted reasoning items from OpenAI replays (#44132) 2026-10-02 19:39:48 -07:00
test_responses_minted_reasoning_replay_wire.py fix(responses): drop bridge-minted reasoning items from OpenAI replays (#44132) 2026-10-02 19:39:48 -07:00
test_responses_reasoning_item_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_responses_websocket_session_limit.py fix(proxy): keep idle responses websockets open until a configurable session limit (#44433) 2026-10-07 19:26:30 -07:00
test_s3_wire.py test: cover provider wire contracts, streaming and recovery 2026-09-16 12:48:30 -07:00
test_sagemaker_chat_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_tencent_chat_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_validated_reply_boundaries_under_faults.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_vertex_anthropic_inline_tools_beta_wire.py fix(vertex_ai): forward the inline-tools-2026-09-15 beta to Vertex and Anthropic (#45297) 2026-10-08 12:27:52 -07:00
test_vertex_batch_output_info_wire.py fix(vertex_ai): keep batch output_file_id null until Vertex reports outputInfo (#43030) 2026-09-24 14:05:31 -07:00
test_vertex_embedding_batch_file_block_wire.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_vertex_embedding_file_block_wire.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_vertex_gemini_fragmented_stream_wire.py test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) 2026-09-23 09:51:06 -07:00
test_vertex_gemini_function_call_wire.py refactor(types): replace Any with proven types in 7 files (#43704) 2026-09-29 06:12:58 -07:00
test_vertex_partner_count_tokens_wire.py fix(vertex_ai): forward system and tools to partner model count_tokens (#43900) 2026-10-03 16:36:14 -07:00
test_vertex_realtime_multi_region_host.py fix(vertex_ai): dial the multi-region Live API host for realtime sessions (#45166) 2026-10-07 17:39:38 -07:00
test_voyage_mongodb_host_wire.py feat(voyage): rebrand to VoyageAI by MongoDB and route MongoDB keys to ai.mongodb.com (#41812) 2026-10-08 13:12:47 -07:00
test_xai_server_tool_usage_wire.py refactor(types): replace Any with proven types in 157 files (#44798) 2026-10-06 15:31:54 +00:00
test_xai_web_search_wire.py test(integration): regression tests for August provider translation and streaming bugs (#42621) 2026-09-22 20:52:00 -07:00
test_xiaomi_mimo_wire.py test(integration): move Xiaomi MiMo coverage from live e2e to the providers wire shard (#42395) 2026-09-21 19:51:26 -07:00