litellm/tests
devin-ai-integration[bot] f61b3c3f38
refactor(types): declare litellm-owned kwargs as typed objects and derive the lists from their fields (#42843)
* refactor(types): declare litellm-owned params in one registry

* refactor(types): re-export registry constants without redundant aliases

* refactor(types): satisfy type-discipline rules in registry projections and tests

* refactor(types): classify every registry entry and check groups against typed config models

* test(types): pin load-bearing names and exact projections in registry tests

* style(types): keep agentic projection comment within ruff format

* refactor(types): declare litellm-owned params as typed objects and derive the lists from their fields

* refactor(types): fields of the typed objects become the registry; tests use a hand-written inventory

* refactor(types): split traversal into wire_names and owned_wire_names, move rust to kwarg artifacts

rust is a module-level switch (litellm.rust) that nothing reads from a call's kwargs, so it
joins self, use_client and model_config as a registered artifact instead of a DispatchOptions
field. The field constants now import from litellm.types.litellm_params directly instead of
through a re-export in litellm.types.utils. metadata and litellm_metadata are MutableMapping
because their readers mutate them in place, and client accepts raw httpx clients

* refactor(types): own max_agentic_loops as an option and walk only nested leaves

Move max_agentic_loops from AgenticLoopState to a new AgenticLoopOptions leaf under
LiteLLMOptions, since the interception handlers read it as a deployment ceiling rather
than stamping it. Drop the owned_wire_names fallback that treated an unresolved annotation
as a direct field, which under postponed annotations silently shrank the registry. Re-export
TRUSTED_CALLBACK_VARS_FIELD and ADDRESSED_RESPONSE_ID_FIELD from types.utils so that import
path keeps working. Tests use hand-written inventories for the callback and pricing names

* refactor(types): move data_residency to call state and drop aliased re-exports

data_residency is stamped by get_litellm_params and responses.main during the
call, so it lives on CallState, not CostOptions. mock_response also accepts a
float sequence, which main.py reads for mock embeddings. The types/utils.py
re-exports become one plain import with an exact F401 suppression instead of
two X as X aliases that pushed PLC0414 over its strict-gate ceiling. Redundant
leaf docstrings and the structural artifact test are gone; the re-exported
FIELD constants are checked by identity instead

* refactor(types): project owned kwarg names once and keep pass-through extraction in request order

* refactor(types): type caching_groups from its cache reader and hoist the pass-through ownership set

caching_groups is a sequence of flat model-group sequences, which is what
Cache._get_caching_group iterates. A regression test drives the public
cache key path so two groups in one caching group share a key and a third
does not. The pass-through endpoint builds its frozenset of owned names
once at import instead of per request, reads the two metadata carriers
from the extracted mapping instead of popping them, and its extraction
mappings are read-only. Concatenation tests assert the whole derived list
and tuple, docstrings drop reader claims that nothing in the module backs

* refactor(types): read owned names live in pass-through and pin tests to literal inventories

The pass-through endpoint checks body keys against the public all_litellm_params
list at request time again, as the base does, instead of a frozenset taken at
import, so a name registered after import is still extracted. A test drives
that path with a name added after import, and another sends both metadata
carriers interleaved with provider keys and asserts the whole merged result.

retry_policy accepts the mapping form its router reader builds a RetryPolicy
from. The pricing inventory in the typed tests is a literal tuple checked
against the model's fields, the agentic compatibility test asserts type, length
and set instead of declaration order, and the typed-model overlap tests assert
the exact intersection.

* refactor(types): move model_alias_map to CallState and read the owned registry in registry order in pass-through

* refactor(types): drop restating docstrings, keep FIELD importers on types.utils, pin pass-through registry order

* fix(types): satisfy strict lint for public FIELD re-exports

* fix(types): restore clean parameter re-exports

* fix(tests): compare pass-through extraction order to registry body keys

* refactor(types): type owned request parameter leaves

* refactor(types): share routing strategy literal and tighten leaf tests

* fix(proxy): drop client-supplied proxy-stamped names from pass-through litellm_params

* refactor(proxy): name pass-through litellm key split for what it holds

* refactor(types): drop TODO markers on the kept readerless fields

* fix(types): keep deployment tag_regex and max_file_size_mb out of provider requests

* fix(types): include every routing strategy the router accepts

---------

Co-authored-by: shrey kharbanda <shreshth@berri.ai>
2026-09-24 20:18:41 -07:00
..
_support fix(params): stop stream_chunk_size reaching provider request bodies (#42664) 2026-09-23 10:27:47 -07:00
agent_tests
audio_tests
base_sdk_tests fix(mcp): explain missing public client dependencies 2026-09-19 19:52:13 -07:00
basic_proxy_startup_tests
batches_tests
benchmarks feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
code_coverage_tests fix(router): match provider-prefixed fallback keys for bare model groups served by wildcard deployments (#43062) 2026-09-24 18:08:56 -07:00
documentation_tests fix(docs-test): restore line-anchored table regex in router settings check 2026-09-18 20:10:20 +00:00
e2e test(e2e): pin end-user and tag attribution from Codex-style headers on /v1/responses (#43093) 2026-09-24 18:33:49 -07:00
guardrails_tests Revert "Merge pull request #41220 from BerriAI/litellm_post_call_guardrail_context" 2026-09-19 17:45:15 +00:00
image_gen_tests fix(logging): pass provider response headers to callbacks on every endpoint (#42824) 2026-09-24 13:01:12 -07:00
integration fix(proxy): authorize key model aliases the same way as team aliases (#43049) 2026-09-24 19:51:02 -07:00
litellm_utils_tests test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
llm_responses_api_testing
llm_translation fix(logging): pass provider response headers to callbacks on every endpoint (#42824) 2026-09-24 13:01:12 -07:00
load_tests
local_testing ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests (#42902) 2026-09-24 15:49:59 -07:00
logging_callback_tests test: point CircleCI-only suites at models still in the cost map (#42617) 2026-09-22 17:28:34 -07:00
mcp_tests ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests (#42904) 2026-09-24 23:07:48 +00:00
multi_instance_e2e_tests
ocr_tests refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
openai_endpoints_tests
otel_tests test: fix stale budget-status and bad-database-url assertions 2026-09-21 15:00:54 -07:00
pass_through_tests fix(mcp): preserve legacy behavior on SDK2 and streamline verification 2026-09-18 22:28:31 -07:00
pass_through_unit_tests feat(proxy): add TinyFish Agent API passthrough with per-step billing (#41099) 2026-09-21 21:21:43 -07:00
proxy_admin_ui_tests
proxy_behavior feat(proxy): server-side Team Usage export beyond the top-N key cap (#42996) 2026-09-24 15:40:36 -05:00
proxy_e2e_anthropic_messages_tests fix(test): run the all-beta-headers bedrock cases on Claude Fable 5.1 2026-09-19 23:33:29 +00:00
proxy_migration_tests test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
proxy_security_tests refactor(proxy): rename the local development override to dangerously_permit_weak_or_unset_master_key so the name says exactly what it permits 2026-09-19 18:53:14 -07:00
proxy_unit_tests ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests (#42903) 2026-09-24 22:59:11 +00:00
router_unit_tests fix(router): match provider-prefixed fallback keys for bare model groups served by wildcard deployments (#43062) 2026-09-24 18:08:56 -07:00
rust-python-harness refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
search_tests
spend_tracking_tests
store_model_in_db_tests test(mcp): patch create_mcp_server_if_identifier_free in the store-model-in-db MCP tests (#42916) 2026-09-24 11:59:52 -07:00
test_litellm refactor(types): declare litellm-owned kwargs as typed objects and derive the lists from their fields (#42843) 2026-09-24 20:18:41 -07:00
test_litellm_rust refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
unified_google_tests test(google_genai): move unified_google_tests to gemini-3.5-flash-lite (#42520) 2026-09-22 13:07:37 -07:00
unit refactor(types): declare litellm-owned kwargs as typed objects and derive the lists from their fields (#42843) 2026-09-24 20:18:41 -07:00
vector_store_tests
windows_tests
__init__.py
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py
test_fallbacks.py
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py fix(proxy): authorize key model aliases the same way as team aliases (#43049) 2026-09-24 19:51:02 -07:00
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py
test_new_vector_store_endpoints.py
test_openai_endpoints.py
test_organizations.py
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py
test_resource_cleanup.py
test_rust_python_harness.py
test_service_logger_otel.py
test_spend_logs.py
test_team.py
test_team_logging.py
test_team_members.py
test_users.py

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.