litellm/tests
Mateo Wang 79ca820933
fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert (#46772)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does

* test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends

* fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types

* fix(count_tokens): count Anthropic-format requests for Gemini the way /v1/messages inference converts them

* fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert

* test(count_tokens): compare tools by value in the public api test

* fix(count_tokens): reject unconvertible blocks nested inside a tool_result

* fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI

Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field.

* test(count_tokens): expect the image detail the adapter now sends after merging main

* fix(count_tokens): fetch Gemini remote media without blocking the event loop

* fix(count_tokens): type normalize_native_messages without the unimported Any

On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError

* chore(count_tokens): give the Gemini count helpers' list returns a reason

* fix(count_tokens): normalize Anthropic messages for Gemini counts the way /v1/messages does

* fix(count_tokens): count Gemini bodies after inference's history rewrites and answer converter errors with 400

* test(count_tokens): cover the Gemini count error paths and compaction with a text-block system

---------

Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
2026-10-16 03:19:55 +00:00
..
_support fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
base_sdk_tests test: extend runtime regression contracts for Messages and SDK (#46641) 2026-10-15 17:51:15 +00:00
benchmarks refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
code_coverage_tests feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043) 2026-10-15 18:44:30 +00:00
documentation_tests fix(proxy): let configured pass-through endpoints outrank built-in provider prefix routes (#44133) 2026-10-10 12:26:17 -07:00
e2e test(e2e): perf datastore limits count the Postgres statements and Redis commands a request waits on, background work reported on its own (#46798) 2026-10-15 20:10:17 -07:00
e2e_harness test(e2e): attach perf case metrics and limits as JUnit properties (#46774) 2026-10-15 19:12:07 -07:00
fixtures test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
harness_e2e feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
integration fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert (#46772) 2026-10-16 03:19:55 +00:00
rust-python-harness docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
test_litellm_rust test(rust-bridge): select native routes per test without LITELLM_RUST (#46620) 2026-10-15 09:12:08 -07:00
typing fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927) 2026-10-09 19:45:50 -07:00
unit fix(count_tokens): return a 400 naming Anthropic blocks the OpenAI and Gemini count can't convert (#46772) 2026-10-16 03:19:55 +00:00
windows_tests test: move packaging and install checks off CircleCI build_and_test (#45757) 2026-10-10 09:33:51 -07:00
__init__.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_logging.conf
test_rust_python_harness.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
VCR_TESTING.md test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.