litellm/tests
devin-ai-integration[bot] c008d5e2bd
fix(scim): propagate team roster write failures on group and user writes (#37700)
SCIM roster writes were swallowed, so a group or user push returned 200 while the
team roster never received the membership. Surfacing the failure fixes that, but
aborting on the first failed write leaves the rest of the batch unattempted on top
of unrolled-back, which is worse than what it replaces.

Every roster write in a reconciliation is now attempted, and the ones that did not
land are reported together, naming each failed add and remove. Rollback would be the
other option and it is not safe here: the compensating write can fail too, and it can
strip a membership that pre-dated the push. SCIM reconciliation is idempotent, so a
named partial failure is what the IdP's next push needs to close the gap.

The reported status still follows the failures, so a unanimous 404 stays a 404 and
only a batch whose failures disagree falls back to 500.

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-20 17:23:39 -07:00
..
agent_tests test: repair stale CircleCI contracts 2026-08-08 12:19:29 -07:00
audio_tests test: remove tests that never execute 2026-08-12 10:45:38 -07:00
base_sdk_tests test(cli): cover the keyless token record and keep keyring to the cli extra 2026-08-20 03:45:00 -07:00
basic_proxy_startup_tests
batches_tests test: build redaction and batch limiter fixtures the way production does (#37416) 2026-08-18 19:37:50 -07:00
benchmarks
code_coverage_tests test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
documentation_tests ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
e2e Merge pull request #37679 from mubashir1osmani/litellm_lit5890_replay_key_params_form 2026-08-20 16:16:33 -07:00
enterprise fix(proxy): split agent inference and management routes so admin nodes can create agents (#37730) 2026-08-20 16:03:15 -07:00
guardrails_tests fix(guardrails): honor configured timeout in Zscaler AI Guard (#36110) 2026-08-07 00:25:52 +00:00
image_gen_tests test: point the live web search, groq and vertex image suites at models that still exist (#37733) 2026-08-20 17:03:35 -07:00
integration
litellm/proxy/_experimental/mcp_server test: merge three stranded twins into the files that shadow them (#37600) 2026-08-20 14:25:23 -07:00
litellm-proxy-extras test: remove tests that never execute 2026-08-12 10:45:38 -07:00
litellm_utils_tests test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
llm_responses_api_testing test: refresh three suites that drifted from the code they cover 2026-08-15 15:37:09 -07:00
llm_translation test: point the live web search, groq and vertex image suites at models that still exist (#37733) 2026-08-20 17:03:35 -07:00
load_tests
local_testing test(ci): serve /moderations from the canned OpenAI mock (#37739) 2026-08-20 16:58:38 -07:00
logging_callback_tests test: point the live web search, groq and vertex image suites at models that still exist (#37733) 2026-08-20 17:03:35 -07:00
mcp_tests test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
multi_instance_e2e_tests
ocr_tests test(ocr): update Azure DI supported-params assertion for req_format 2026-08-18 19:21:46 -07:00
openai_endpoints_tests fix(batches): register managed output files on batch cancel 2026-08-05 18:28:52 -07:00
otel_tests
pass_through_tests
pass_through_unit_tests test: allow protocol-constrained pass-through routes to declare fewer methods (#37415) 2026-08-18 19:38:05 -07:00
proxy_admin_ui_tests fix(access groups): sync assigned_team_ids from the team write paths (#36825) 2026-08-14 04:45:36 +00:00
proxy_behavior feat(proxy): add /team/daily/activity/aggregated and switch the Usage team tab to it (#36562) 2026-08-18 11:29:57 -07:00
proxy_e2e_anthropic_messages_tests ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
proxy_migration_tests test(docker): gate the componentized gateway and backend images on an arbitrary-uid offline boot (#36136) 2026-08-07 09:57:37 -07:00
proxy_security_tests
proxy_unit_tests fix(proxy): claim batch cost rows atomically so multi-pod polling can't double-bill (#37685) 2026-08-20 16:21:04 -07:00
router_unit_tests feat(complexity_router): custom classifier plugins via classifier_type 'custom' (#37249) 2026-08-18 14:09:19 -07:00
search_tests Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_pr32448_tinyfish_headers 2026-08-18 13:03:33 -07:00
spend_tracking_tests
store_model_in_db_tests test: refresh three suites that drifted from the code they cover 2026-08-15 15:37:09 -07:00
test_litellm fix(scim): propagate team roster write failures on group and user writes (#37700) 2026-08-20 17:23:39 -07:00
unified_google_tests
vector_store_tests
windows_tests
__init__.py
_fake_openai_endpoint_server.py test(ci): serve /moderations from the canned OpenAI mock (#37739) 2026-08-20 16:58:38 -07:00
_flush_vcr_cache.py
_live_test_helpers.py
_openai_record_replay_proxy.py
_vcr_conftest_common.py
_vcr_redis_persister.py
_wait_helpers.py test: replace blind sleeps with deadline waits in callback and caching tests (#37660) 2026-08-20 18:48:43 +00:00
_ws_vcr.py
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_anthropic_compaction_usage.py
test_budget_management.py
test_callbacks_on_proxy.py
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py ci: lint the test tree for undefined names and fix all 30 (#37671) 2026-08-20 13:30:34 -07:00
test_fallbacks.py
test_gpt5_azure_temperature_support.py
test_health.py
test_keys.py
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py
test_new_vector_store_endpoints.py
test_openai_endpoints.py test: point the live web search, groq and vertex image suites at models that still exist (#37733) 2026-08-20 17:03:35 -07:00
test_organizations.py
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py
test_resource_cleanup.py
test_service_logger_otel.py
test_spend_logs.py
test_team.py test: add six ruff rules that catch tests which cannot fail (#37709) 2026-08-20 14:21:26 -07:00
test_team_logging.py
test_team_members.py
test_users.py

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.