scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-13 03:59:12 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 66
litellm/tests/unit/llms
History
devin-ai-integration[bot] 1dbde0845e
test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 
* test: move guardrails, image, audio, batches, search legacy tests to unit

* test: restore the custom guardrail and remaining chunk 1 unit rows

* test: move the ocr matrix to e2e

* test: drop comments from moved tests and fix vertex ocr credentials

* test: move litellm utils and unified google legacy tests to their tiers

* test: move logging callback and pass-through unit legacy tests to their tiers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: move store_model_in_db legacy tests and start the admin ui migration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: finish the proxy job rows and drop their CircleCI jobs

Moves the remaining multi-instance, openai endpoint, otel, pass-through and
top-level proxy tests into tests/integration with scripted upstreams, rewrites
the weak replacements to assert the outbound wire and exact responses, and
removes the build_and_test job group this PR owns from CircleCI

* test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names

* test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist

* test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in

* test: type the e2e beta header, prompt cache and secret manager values

* test: assert the missing model error message on the OpenAI routes

* test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks

* test: make the langfuse fixture directory a package for the layout check

* test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane

* test: keep the ocr e2e pdf fixture next to the test

* test: put meta below covers on the streaming tool search e2e

* test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures

* test: count only OCR calls in the OCR matrix recorder

* test: pin OCR parse bodies and billed cost, group new e2e cases into classes

* ci: restore the python image anchor dropped in the main sync

---------

Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-11 00:17:21 -07:00
..
a2a refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
aiml test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
aiohttp_openai refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
anthropic test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
apiserpent test(unit): add package markers to migrated unit test directories 2026-09-20 11:53:14 +00:00
aws_polly refactor(sdk): separate core AWS and tokenizer dependencies (#44447) 2026-10-08 19:07:57 -07:00
azure test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
azure_ai test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
base_llm test: make 38 legacy live base-class translation tests offline (#45351) 2026-10-08 23:18:25 +00:00
baseten test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
bedrock test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
bedrock_mantle refactor(sdk): separate core AWS and tokenizer dependencies (#44447) 2026-10-08 19:07:57 -07:00
black_forest_labs refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
bytez test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
cerebras test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
chat refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
chatgpt refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
clarifai refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
claude_code refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
cloudflare test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554) 2026-10-09 09:48:10 -07:00
codex refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
cohere test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
cometapi refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
compactifai test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
crusoe refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
custom_httpx test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
dashscope refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
databricks fix(databricks): call the ai_decide AI Function route for decisions and the auto-router (#45722) 2026-10-10 23:58:16 -07:00
dataforseo test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
datarobot test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
deepagents feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
deepgram test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
deepinfra refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
deepseek test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
docker_model_runner test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
duckduckgo test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
e2b refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
edenai test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
elevenlabs test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
exa_ai test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
fal_ai test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
fastcrw test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
featherless_ai refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
firecrawl test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
fireworks_ai test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
gdc test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
gemini test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
gigachat refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
github_copilot chore(github_copilot): sync model catalog with Copilot's served models (#45386) 2026-10-09 15:19:08 -07:00
gradient_ai refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
groq test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
heroku test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
hosted_vllm test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
huggingface test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
hyperbolic refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
inception refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
infinity test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
jina_ai test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
lambda_ai test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
langflow fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
langgraph test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
laya feat: add Bespoke Nimble gateway and OSS classifier support (#44246) 2026-10-02 19:37:08 -07:00
lemonade refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
linkup test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
litellm_proxy test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
llamafile refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
lm_studio refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
manus test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
meta test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
meta_llama refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
microsoft_365_copilot feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange (#45158) 2026-10-09 15:19:07 -07:00
minimax test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
mistral test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
modelscope refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
mongodb test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
moonshot chore(cost-map): remove models past their deprecation date (#42435) 2026-09-22 21:19:26 +00:00
nadir test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
nebius test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
neosantara refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
nimble test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
novita test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
nscale test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
nvidia_nim test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554) 2026-10-09 09:48:10 -07:00
nvidia_riva test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
oci refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
ocr test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
ollama test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
oobabooga test(unit): make every tests/unit directory a package so pytest collection is unique 2026-09-20 11:50:59 +00:00
openai test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
openai_like test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
opencode feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
openrouter test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
ovhcloud refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
parallel_ai test: add __init__.py to phase 12 unit test directories 2026-09-20 12:36:15 +00:00
parasail refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
pass_through feat(guardrails): per-mode stream_scope with bedrock stream and pass-through fixes (#43801) 2026-10-08 13:46:53 -07:00
perplexity test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
pg_vector fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
publicai test: add __init__.py to phase 12 unit test directories 2026-09-20 12:36:15 +00:00
ragflow refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
recraft refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
reducto test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
replicate test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
runwayml test: add __init__.py to phase 12 unit test directories 2026-09-20 12:36:15 +00:00
s3_vectors test: add __init__.py to intermediate phase 12 unit test directories 2026-09-20 12:37:21 +00:00
sagemaker test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
sail test(e2e): align completion, SAIL, and spend-log fixtures with supported contracts (#43902) 2026-09-30 14:21:22 -07:00
sambanova refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
sap fix(sap): correctly handle cache_control (#39122) 2026-10-06 08:54:06 -07:00
scaledown feat(scaledown): add native text provider support (#44168) 2026-10-10 14:01:17 -07:00
scaleway feat(scaleway): add rerank support (#44160) 2026-10-02 21:20:05 -07:00
searchapi test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
searxng test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
serper test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
snowflake refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
soniox refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
stability refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
strands_decider feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN (#45678) 2026-10-09 22:51:19 -07:00
tavily test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
tencent refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
tinyfish test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
together_ai test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
tool_loop feat(harness): add Harness.TOOL_LOOP, a minimal in-process tool-calling loop (#44391) 2026-10-03 17:33:01 +00:00
triton test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554) 2026-10-09 09:48:10 -07:00
v0 test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
valkey test: migrate legacy provider tests to tests/unit (wave 2, phase 13) 2026-09-20 14:18:49 +00:00
vercel_ai_gateway refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
vertex_ai test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
volcengine test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00
voyage fix(ci): drop the publicly known master key prefix from the voyage routing test (#45431) 2026-10-08 13:41:03 -07:00
wandb test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
watsonx test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
xai test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
xinference test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
you_com test: migrate phase 14 wave 2 provider tests to tests/unit 2026-09-20 11:01:40 +00:00
zai test: add __init__.py to every tests/unit directory this migration touches 2026-09-20 11:52:09 +00:00
__init__.py test(unit): add package markers to migrated unit test directories 2026-09-20 11:53:14 +00:00
test_cache_control_and_reasoning.py test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
test_custom_llm.py test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
test_file_content_block.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_file_search_responses.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_lifecycle_fix.py refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_oss_decision.py feat: add Bespoke Nimble gateway and OSS classifier support (#44246) 2026-10-02 19:37:08 -07:00
test_polling_url_origin_match.py test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
test_predibase_transformation.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00