scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-11 03:38:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 82
litellm/tests
History
Ninad Phalak b55622e8ba
fix(guardrails): keep restored llm_shield_proxy values out of telemetry, refuse SDK streams 
- The post-call restore hook no longer goes through log_guardrail_information,
  which recorded its whole return value, the restored reply, as guardrail_response.
  That field is exported to traces even with message logging turned off.
- A model-level stream outside the proxy is refused once redacted. Nothing restores
  an SDK stream, and its cache writer reads the request from before the deployment
  hook, so it also got cached despite the no-store bypass.
- Drop a narrating comment, and keep example_config.yaml to config only; the
  how-to lives in the docs PR.
2026-10-05 09:58:34 +00:00
..
_support fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
agent_tests
audio_tests
base_sdk_tests fix(mcp): explain missing public client dependencies 2026-09-19 19:52:13 -07:00
basic_proxy_startup_tests
batches_tests
benchmarks feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
code_coverage_tests feat(proxy): bounded daily activity routes for all usage entities (#43408) 2026-10-01 16:57:23 -07:00
documentation_tests fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
e2e test(e2e): keep 1ms-timeout deployments off the provider cache (#44082) 2026-10-01 15:54:02 -07:00
guardrails_tests fix(packaging): keep wheel paths under Windows MAX_PATH for Store Python (#43903) 2026-09-30 22:20:36 +00:00
harness_e2e feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
image_gen_tests fix(logging): pass provider response headers to callbacks on every endpoint (#42824) 2026-09-24 13:01:12 -07:00
integration feat(lens): simplify setup and investigation workflow (#44089) 2026-10-01 17:21:41 -07:00
litellm_utils_tests fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
llm_responses_api_testing test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
llm_translation test(bedrock): restore the AWS env after a failed live call in the auth tests (#43921) 2026-09-30 22:36:57 +00:00
load_tests
local_testing ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) 2026-10-01 13:00:54 -07:00
logging_callback_tests test(ci): repair stale tests and flaky CI infrastructure (#43983) 2026-10-01 17:46:43 +00:00
mcp_tests ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests (#42904) 2026-09-24 23:07:48 +00:00
multi_instance_e2e_tests
ocr_tests refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
openai_endpoints_tests
otel_tests test: fix stale budget-status and bad-database-url assertions 2026-09-21 15:00:54 -07:00
pass_through_tests fix(mcp): preserve legacy behavior on SDK2 and streamline verification 2026-09-18 22:28:31 -07:00
pass_through_unit_tests refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
proxy_admin_ui_tests
proxy_behavior refactor(proxy): inject tracing receiver and access context (#44035) 2026-10-01 13:45:32 -07:00
proxy_e2e_anthropic_messages_tests fix(test): run the all-beta-headers bedrock cases on Claude Fable 5.1 2026-09-19 23:33:29 +00:00
proxy_migration_tests fix(proxy-extras): bound the lock waits of the partitioned SpendLogs index build (#44109) 2026-10-01 18:28:50 -07:00
proxy_security_tests refactor(proxy): rename the local development override to dangerously_permit_weak_or_unset_master_key so the name says exactly what it permits 2026-09-19 18:53:14 -07:00
proxy_unit_tests ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests (#42903) 2026-09-24 22:59:11 +00:00
router_unit_tests fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
rust-python-harness refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
search_tests fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows (#42951) 2026-09-25 19:12:36 -07:00
spend_tracking_tests
store_model_in_db_tests test(ci): repair MCP Responses and budget fixtures (#43788) 2026-09-29 23:29:52 -07:00
test_litellm refactor(tracing): normalize agent spans in Rust (#44071) 2026-10-01 18:01:20 -07:00
test_litellm_rust refactor(tracing): normalize agent spans in Rust (#44071) 2026-10-01 18:01:20 -07:00
unified_google_tests test(google_genai): move unified_google_tests to gemini-3.5-flash-lite (#42520) 2026-09-22 13:07:37 -07:00
unit fix(guardrails): keep restored llm_shield_proxy values out of telemetry, refuse SDK streams 2026-10-05 09:58:34 +00:00
vector_store_tests
windows_tests fix(packaging): keep wheel paths under Windows MAX_PATH for Store Python (#43903) 2026-09-30 22:20:36 +00:00
__init__.py
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py fix(tests): stop VCR recording and replaying a test's own localhost upstream (#43346) 2026-09-26 15:49:39 -07:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py
test_fallbacks.py test(ci): refresh qualified retired OpenAI fixtures (#43938) 2026-09-30 16:10:18 -07:00
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py fix(proxy): authorize key model aliases the same way as team aliases (#43049) 2026-09-24 19:51:02 -07:00
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py test(proxy): move management_endpoints, management_helpers and guardrails tests into tests/unit/proxy (#44003) 2026-10-01 10:52:03 -07:00
test_new_vector_store_endpoints.py
test_openai_endpoints.py test(ci): refresh qualified retired OpenAI fixtures (#43938) 2026-09-30 16:10:18 -07:00
test_organizations.py
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py test(proxy): delete the legacy proxy test tree and shard tests/unit/proxy by glob (#44018) 2026-10-01 11:40:45 -07:00
test_resource_cleanup.py
test_rust_python_harness.py test: fix stale and state-leaking tests red on scheduled CircleCI (#43266) 2026-09-25 19:10:20 -07:00
test_service_logger_otel.py
test_spend_logs.py test(proxy): move middleware, spend_tracking, pass_through, common_utils and root proxy tests into tests/unit/proxy (#44015) 2026-10-01 18:23:31 +00:00
test_team.py
test_team_logging.py
test_team_members.py test(proxy): move management_endpoints, management_helpers and guardrails tests into tests/unit/proxy (#44003) 2026-10-01 10:52:03 -07:00
test_users.py
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.