scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-29 06:33:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 200
litellm/tests
History
Chloe Lu cbe5c288b8
fix(anthropic): keep signed empty thinking blocks on /v1/messages requests (#41362) 
The request-side sanitizer on /v1/messages dropped every thinking block whose
text is empty, even with a valid signature. Claude Opus 4.8 / Fable 5 / Fable 5.1
return exactly that shape by default (thinking.display omitted): the signature
is the preserved reasoning. Replaying the assistant turn as returned therefore
lost the model's reasoning on every multi-turn request, silently (the provider
answers 200 without it), and left preserved-thinking enforcement
(thinking.block_binding) nothing to bind.

Anthropic accepts an empty thinking block that carries a signature and binds it
to its prefix; only an empty block with no signature (the LIT-6357 case, a turn
a non-Anthropic model served through the bridge) is rejected with "each
thinking block must contain thinking". Use the signature-aware predicate
is_empty_unsigned_thinking_block, which #38809 already applied on the emit
side, for the request side too. The unsigned case is still dropped.

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 897bfea38d)
2026-10-19 22:29:53 -07:00
..
_support fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
agent_tests/local_only_agent_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
audio_tests test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
base_sdk_tests refactor(sdk): separate core AWS and tokenizer dependencies (#44447) 2026-10-08 19:07:57 -07:00
basic_proxy_startup_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
batches_tests test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
benchmarks refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
code_coverage_tests docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI (#45838) 2026-10-10 16:30:24 -07:00
documentation_tests fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
e2e test(e2e): backport three stale e2e test fixes from main to rc/1.106.0 (#47229) 2026-10-17 20:22:41 -07:00
e2e_harness ci: render lint, unit and smoke checks as <tier> / <job> with one collector per tier (#45480) 2026-10-09 04:28:41 +00:00
guardrails_tests test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
harness_e2e feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
image_gen_tests test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
integration fix(responses): fold bridged Responses output into one chat choice (#46099) 2026-10-19 22:27:55 -07:00
litellm_utils_tests test: delete 34 legacy tests already covered by e2e (#45340) 2026-10-08 09:43:56 -07:00
llm_translation test: make 38 legacy live base-class translation tests offline (#45351) 2026-10-08 23:18:25 +00:00
load_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
local_testing test: delete 34 legacy tests already covered by e2e (#45340) 2026-10-08 09:43:56 -07:00
logging_callback_tests test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
mcp_tests feat(mcp): add portable catalog pagination (#44446) 2026-10-06 18:54:50 -07:00
multi_instance_e2e_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
ocr_tests test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
openai_endpoints_tests test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
otel_tests test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362) 2026-10-08 15:55:57 -07:00
pass_through_tests test: delete 34 legacy tests already covered by e2e (#45340) 2026-10-08 09:43:56 -07:00
pass_through_unit_tests test: delete 34 legacy tests already covered by e2e (#45340) 2026-10-08 09:43:56 -07:00
proxy_admin_ui_tests test: move the unit half of 126 mixed legacy files into tests/unit (#45090) 2026-10-07 14:07:43 -07:00
proxy_behavior test: realign stale decisions and lens tests with merged behaviour (#45346) 2026-10-08 09:44:18 -07:00
proxy_e2e_anthropic_messages_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_migration_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_security_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_unit_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
router_unit_tests test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs (#45298) 2026-10-08 02:37:55 -07:00
rust-python-harness test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
search_tests test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288) 2026-10-08 02:38:12 -07:00
store_model_in_db_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_litellm refactor(llms): expose public names for private provider helpers (#45037) 2026-10-07 08:49:25 -07:00
test_litellm_rust fix(ci): import seed_tracing_fixtures from the pytest scripts path in rust trace tests (#45485) 2026-10-08 19:41:22 -07:00
unified_google_tests test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) 2026-10-07 05:02:28 +00:00
unit fix(anthropic): keep signed empty thinking blocks on /v1/messages requests (#41362) 2026-10-19 22:29:53 -07:00
vector_store_tests refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
windows_tests fix(packaging): keep wheel paths under Windows MAX_PATH for Store Python (#43903) 2026-09-30 22:20:36 +00:00
__init__.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
_fake_openai_endpoint_server.py test(timeout): time out against the local fake endpoint instead of api.openai.com 2026-09-03 09:53:30 -07:00
_flush_vcr_cache.py
_live_test_helpers.py refactor(test): validate cost-map entries into a typed model 2026-09-16 14:32:08 -07:00
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py Extend the record/replay proxy to chat, embeddings, moderations, rerank, and Anthropic (#29847) 2026-06-06 14:33:42 -07:00
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py fix(tests): stop VCR recording and replaying a test's own localhost upstream (#43346) 2026-09-26 15:49:39 -07:00
_vcr_redis_persister.py
_wait_helpers.py test: replace blind sleeps with deadline waits in callback and caching tests (#37660) 2026-08-20 18:48:43 +00:00
_ws_vcr.py test(realtime): record and replay websocket traffic in redis vcr cassettes (#32390) 2026-07-08 00:19:06 -07:00
AGENTS.md test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py Prompt Compression - add it to the proxy (#25729) 2026-04-20 15:08:00 -07:00
fake_openai_endpoint.py test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900) 2026-06-20 16:20:35 -07:00
gettysburg.wav feat(main.py): support openai transcription endpoints 2024-03-08 10:25:19 -08:00
large_text.py fix(router.py): check for context window error when handling 400 status code errors 2024-03-26 08:08:15 -07:00
openai_batch_completions.jsonl feat(router.py): Support Loadbalancing batch azure api endpoints (#5469) 2024-09-02 21:32:55 -07:00
pyrightconfig.json test: add e2e tests for spend, budgets and llms (#30869) 2026-06-24 15:01:57 -07:00
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_debug_warning.py fix(utils.py): fix togetherai streaming cost calculation 2024-08-01 15:03:08 -07:00
test_keys.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_litellm_proxy_responses_config.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_logging.conf
test_models.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_new_vector_store_endpoints.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_openai_endpoints.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_proxy_server_non_root.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_rust_python_harness.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_service_logger_otel.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_spend_logs.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
test_team_logging.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_users.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.