scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-15 04:19:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 121
litellm/tests/unit/llms
History
devin-ai-integration[bot] e0398455d0
test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 
* test: migrate selected llm translation contracts

* test: migrate anthropic completion translation tests

* test: migrate bedrock and cohere translation tests

* test: tighten bedrock translation migration

* test: migrate azure and openai model translation tests

* test: migrate openai and small-provider translation tests

* test: migrate gemini and vertex translation tests

* test: migrate optional params and rerank translation tests

* test: migrate huggingface, json and openrouter translation tests

* test: fix openrouter integration expectations

* test: migrate databricks, xai, cometapi and fireworks translation tests

* test: migrate ovhcloud, prompt and small provider translation tests

* test: split reasoning effort grid into unit and e2e

* test: migrate router and realtime guardrail translation tests

* test: migrate google interactions translation tests

* test: migrate a2a, evals, skills and hosted vllm translation tests

* test: remove remaining llm translation suite

* test: tighten migrated llm translation unit tests and land missing rows

Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover

* test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP

Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths

* test: close lost-intent gaps found in review of the llm translation migration

Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised

* test: assert provider acceptance in the reasoning effort e2e

The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid

* test: cover async and stream-end paths the legacy translation suite exercised

Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker

* test: type the new llm translation e2e calls and mark moved fixture dirs as packages

* test: name the reasoning effort e2e models as module constants

* test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name

* test: block the prompt the shared guardrail denies and drop the reasoning presence check

* test: drop the duplicate litellm import left by the main sync

---------

Co-authored-by: yuneng <yuneng@berri.ai>
2026-10-11 07:58:53 +00:00
..
a2a refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
aiml test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
aiohttp_openai refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
anthropic test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
apiserpent test(unit): add package markers to migrated unit test directories 2026-09-20 11:53:14 +00:00
aws_polly refactor(sdk): separate core AWS and tokenizer dependencies (#44447) 2026-10-08 19:07:57 -07:00
azure test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
azure_ai test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
base_llm test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
baseten test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
bedrock test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
bedrock_mantle refactor(sdk): separate core AWS and tokenizer dependencies (#44447) 2026-10-08 19:07:57 -07:00
black_forest_labs refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
bytez test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
cerebras test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
chat refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
chatgpt refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
clarifai refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
claude_code refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
cloudflare test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554) 2026-10-09 09:48:10 -07:00
codex refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
cohere test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
cometapi test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
compactifai test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
crusoe refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
custom_httpx test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
dashscope refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
databricks test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
dataforseo test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
datarobot test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
deepagents feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
deepgram test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
deepinfra refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
deepseek test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
docker_model_runner test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
duckduckgo test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
e2b refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
edenai test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
elevenlabs test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
exa_ai test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
fal_ai test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
fastcrw test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
featherless_ai refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
firecrawl test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
fireworks_ai test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
gdc test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
gemini test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
gigachat test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
github_copilot chore(github_copilot): sync model catalog with Copilot's served models (#45386) 2026-10-09 15:19:08 -07:00
gradient_ai refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
groq test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
heroku test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
hosted_vllm test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
huggingface test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
hyperbolic refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
inception refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
infinity test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
jina_ai test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
lambda_ai test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
langflow fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
langgraph test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
laya feat: add Bespoke Nimble gateway and OSS classifier support (#44246) 2026-10-02 19:37:08 -07:00
lemonade refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
linkup test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
litellm_proxy test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
llamafile refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
lm_studio refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
manus test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
meta test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
meta_llama refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
microsoft_365_copilot feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange (#45158) 2026-10-09 15:19:07 -07:00
minimax test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
mistral test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
modelscope refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
mongodb test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
moonshot chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
nadir test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
nebius test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
neosantara refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
nimble test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
novita test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
nscale test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
nvidia_nim test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554) 2026-10-09 09:48:10 -07:00
nvidia_riva test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
oci refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
ocr test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
ollama test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
oobabooga test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
openai test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
openai_like test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
opencode feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
openrouter test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
ovhcloud test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
parallel_ai test: add __init__.py to phase 12 unit test directories 2026-09-20 12:36:15 +00:00
parasail refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
pass_through feat(guardrails): per-mode stream_scope with bedrock stream and pass-through fixes (#43801) 2026-10-08 13:46:53 -07:00
perplexity test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
pg_vector fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
publicai test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
ragflow refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
recraft refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
reducto test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
replicate test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
runwayml test: add __init__.py to phase 12 unit test directories 2026-09-20 12:36:15 +00:00
s3_vectors test: add __init__.py to intermediate phase 12 unit test directories 2026-09-20 12:37:21 +00:00
sagemaker test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
sail test(e2e): align completion, SAIL, and spend-log fixtures with supported contracts (#43902) 2026-09-30 14:21:22 -07:00
sambanova refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
sap fix(sap): correctly handle cache_control (#39122) 2026-10-06 08:54:06 -07:00
scaledown feat(scaledown): add native text provider support (#44168) 2026-10-10 14:01:17 -07:00
scaleway test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
searchapi test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
searxng test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
serper test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
snowflake refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
soniox refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
stability refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
strands_decider feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN (#45678) 2026-10-09 22:51:19 -07:00
tavily test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
tencent refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
tinyfish test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
together_ai test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
tool_loop feat(harness): add Harness.TOOL_LOOP, a minimal in-process tool-calling loop (#44391) 2026-10-03 17:33:01 +00:00
triton test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554) 2026-10-09 09:48:10 -07:00
v0 test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
valkey test: migrate legacy provider tests to tests/unit (wave 2, phase 13) 2026-09-20 14:18:49 +00:00
vercel_ai_gateway refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
vertex_ai test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
volcengine test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
voyage test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
wandb test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
watsonx test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
xai test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
xinference test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
you_com test: migrate phase 14 wave 2 provider tests to tests/unit 2026-09-20 11:01:40 +00:00
zai test: add __init__.py to every tests/unit directory this migration touches 2026-09-20 11:52:09 +00:00
__init__.py test(unit): add package markers to migrated unit test directories 2026-09-20 11:53:14 +00:00
test_cache_control_and_reasoning.py test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
test_custom_llm.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_file_content_block.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_file_search_responses.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_lifecycle_fix.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_oss_decision.py feat: add Bespoke Nimble gateway and OSS classifier support (#44246) 2026-10-02 19:37:08 -07:00
test_polling_url_origin_match.py test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
test_predibase_transformation.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00