scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-09 22:31:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 133
litellm/tests/test_litellm/llms
History
Exact
Mateo Wang 75613bf22f
test: add regression coverage for twelve closed issues (#37974) 
* test: add regression coverage for twelve closed issues

Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:

- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges

Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.

* fix: stop the new regression tests polluting and tripping over shared global state

Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.

test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.

The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
2026-08-22 22:24:05 +00:00
..
a2a/chat fix(a2a): populate response usage in a2a chat transformation (#31980) 2026-07-03 09:28:36 +05:30
aiml/image_generation test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
amazon_nova/chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
anthropic test: add regression coverage for twelve closed issues (#37974) 2026-08-22 22:24:05 +00:00
apiserpent test: unwind environment writes in tests/test_litellm with monkeypatch (#37806) 2026-08-21 20:28:37 -07:00
azure test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
azure_ai test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
base_llm test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
baseten/chat style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
bedrock test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
bedrock_mantle test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
black_forest_labs test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
bytez/chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
chatgpt test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
cloudflare fix(cloudflare): route native Workers AI provider through OpenAI-compatible endpoint (#31053) 2026-06-23 12:09:11 -07:00
cohere test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
cometapi/chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
compactifai style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
crusoe test: unwind environment writes in tests/test_litellm with monkeypatch (#37806) 2026-08-21 20:28:37 -07:00
custom_httpx test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
dashscope test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
databricks test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
datarobot test: unwind environment writes in tests/test_litellm with monkeypatch (#37806) 2026-08-21 20:28:37 -07:00
deepgram test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
deepinfra test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
deepseek test: merge three stranded twins into the files that shadow them (#37600) 2026-08-20 14:25:23 -07:00
docker_model_runner test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
elevenlabs test: avoid provider test module name collision 2026-04-29 22:36:57 -07:00
fal_ai test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
fastcrw/search chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
featherless_ai/chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
fireworks_ai test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
gdc/chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
gemini test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
github_copilot test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
gradient_ai test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
groq fix: omit thinking.type=disabled for always-on thinking Claude models (#37510) 2026-08-21 10:27:26 -07:00
heroku style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
hosted_vllm test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
huggingface test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
inception test: unwind environment writes in tests/test_litellm with monkeypatch (#37806) 2026-08-21 20:28:37 -07:00
jina_ai/embedding test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
langflow test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
lemonade test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
litellm_proxy test: remove tests that never execute 2026-08-12 10:45:38 -07:00
llamafile/chat style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
lm_studio style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
manus test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
meta_llama test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
minimax test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
mistral test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
modelscope test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
moonshot test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
nebius test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
neosantara Litellm oss staging 050626 (#29774) 2026-06-05 13:51:51 -07:00
nimble/search feat(search): add Nimble as a search provider (#36347) 2026-08-14 17:09:58 -07:00
novita/chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
nscale/chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
nvidia_nim/rerank test: drop rerank package marker colliding with voyage test package 2026-08-11 23:37:27 -07:00
nvidia_riva test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
oci test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
ocr test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
ollama test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
oobabooga/chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
openai fix(otel): emit LLM Call spans for speech, image, moderation, ocr and transcription (#37752) 2026-08-22 11:11:21 -07:00
openai_like Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_scx_ai_provider 2026-08-21 17:27:49 +10:00
openrouter test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
ovhcloud test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
parallel_ai test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
parasail Litellm oss staging 080626 (#29932) 2026-06-08 13:49:52 -07:00
pass_through test: remove four mirror test files that exercise none of their module 2026-07-25 10:57:19 -07:00
perplexity test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
pg_vector/vector_stores test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
publicai test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
ragflow/chat test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
recraft test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
reducto Litellm oss staging 04 21 2026 2 (#26569) 2026-05-20 21:25:19 -07:00
runwayml test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
s3_vectors Fix extra body error 2026-04-29 08:34:31 +05:30
sagemaker test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
sambanova
sap style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
scaleway Feature/add audio support for scaleway (#26110) 2026-04-20 14:49:41 -07:00
snowflake chore(oss): litellm oss staging 120626 (#30292) 2026-06-12 09:49:25 -07:00
soniox chore: litellm oss staging (#30745) 2026-06-18 13:55:35 -07:00
stability test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
tencent fix: omit thinking.type=disabled for always-on thinking Claude models (#37510) 2026-08-21 10:27:26 -07:00
tinyfish test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
valkey/vector_stores feat(vector_stores): add Valkey as a managed vector store provider (#37002) 2026-08-18 21:45:22 +00:00
vercel_ai_gateway test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
vertex_ai test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
volcengine test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
voyage test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769) 2026-08-20 20:24:49 -07:00
wandb test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
watsonx test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
xai test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
you_com test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
zai test: stop the zai tests from leaking env and litellm globals into the session (#37834) 2026-08-21 22:10:34 -07:00
test_cache_control_and_reasoning.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_file_content_block.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_file_search_responses.py fix(tests): use canonical litellm_enterprise import path (#27699) 2026-05-12 12:32:57 -07:00
test_lifecycle_fix.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_polling_url_origin_match.py chore(security): close two unaddressed SSRF cases 2026-05-01 18:43:47 +00:00
test_predibase_transformation.py refactor(predibase): migrate transform_request and transform_response… (#25249) 2026-04-27 08:58:22 +05:30