litellm/tests
devin-ai-integration[bot] abc543e701
feat(proxy): add opt-in vector_store_deny_by_default for least-privilege vector store access (#44244)
* feat(proxy): add opt-in vector_store_deny_by_default for standalone virtual keys

Adds general_settings.vector_store_deny_by_default (typed bool, default false). When enabled, a virtual key
with no team must list the requested vector store in its object_permission.vector_stores; no permission
record, null, or an empty list is denied with key_vector_store_access_denied. Omitted or false keeps the
existing behavior, including nonempty allowlist enforcement. The master key is unchanged in both modes.
Team keys and keyless callers are deferred to later increments of LIT-6035

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(proxy): require key and team vector store grants for team keys under vector_store_deny_by_default

With the flag enabled, a virtual key on a team needs both its own grant and its team's grant for every requested vector store. A missing permission record, an empty list or an unresolved team grants nothing. Dashboard session keys and the master key keep their existing behavior, and flag-off behavior is unchanged

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(proxy): require user or team vector store grants for keyless requests under vector_store_deny_by_default

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): cover vector_store_deny_by_default through a real proxy with key, team and JWT identities

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(ui): regenerate schema.d.ts for vector_store_deny_by_default

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): cover path and file_search vector store ids under vector_store_deny_by_default

Strict mode now reads vector_store_ids and tools[].vector_store_ids from the request body without needing a vector store registry, so /v1/vector_stores/{id}/search and Responses file_search are checked. User grants load through the object permission cache, and the proxy admin user rebuild keeps object_permission_id

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): broadcast user entitlement cache eviction to every worker

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(auth): reuse VectorStoreRegistry id extraction for vector_store_deny_by_default

Strict mode now uses get_vector_store_ids_to_run on an empty registry when none is loaded, instead of a parallel set of request-shape helpers, and vector_store_access_check documents the policy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add strict vector store audit cells for routes, SDKs, workers and concurrency

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): consolidate vector_store_deny_by_default coverage to core cases

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): reject invalid vector_store_deny_by_default at config load and return 400 for malformed vector_store_ids

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): read vector store key and team grants through the object permission cache

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(auth): return typed permission rows in request flow vector store tests

Co-authored-by: mrinal <mrinal@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(auth): validate vector store ids and tools as immutable sequences

Co-authored-by: mrinal <mrinal@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-06 12:50:25 -07:00
..
_support fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
agent_tests
audio_tests test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
base_sdk_tests fix(mcp): explain missing public client dependencies 2026-09-19 19:52:13 -07:00
basic_proxy_startup_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
batches_tests test(e2e): move live-provider legacy tests into tests/e2e (#44120) 2026-10-02 00:02:18 -07:00
benchmarks feat(tokenizer): preserve Python defaults with opt-in Rust dispatch (#42174) 2026-09-22 04:41:11 +00:00
code_coverage_tests fix(ci): skip the generated dashboard bundle in the master key guard (#44890) 2026-10-06 18:15:09 +00:00
documentation_tests fix(s3_v2): upload fresh events first, drop terminal failures and hour-old retries by default, opt-in adaptive concurrency (#43022) 2026-09-26 14:58:28 -07:00
e2e fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
guardrails_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
harness_e2e feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885) 2026-10-01 22:27:49 +00:00
image_gen_tests test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
integration feat(proxy): add opt-in vector_store_deny_by_default for least-privilege vector store access (#44244) 2026-10-06 12:50:25 -07:00
litellm_utils_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
llm_responses_api_testing test: remove 130 legacy tests owned by stronger unit proofs (#44157) 2026-10-02 10:18:50 -07:00
llm_translation fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
load_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
local_testing fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
logging_callback_tests test(ci): pin the ROI estimator flag and the prompt-cache counter in two drifted tests (#44499) 2026-10-04 08:40:23 -07:00
mcp_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
multi_instance_e2e_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
ocr_tests refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
openai_endpoints_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
otel_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
pass_through_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
pass_through_unit_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_admin_ui_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_behavior fix(lens): reuse trace reviews and consolidate findings across runs (#44778) 2026-10-06 12:09:56 -07:00
proxy_e2e_anthropic_messages_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_migration_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_security_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
proxy_unit_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
router_unit_tests fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
rust-python-harness refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
search_tests fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows (#42951) 2026-09-25 19:12:36 -07:00
spend_tracking_tests test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) 2026-10-01 23:00:54 -07:00
store_model_in_db_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_litellm fix(lens): reconstruct native coding agent conversations (#44711) 2026-10-05 17:51:05 -07:00
test_litellm_rust refactor(cache): remove dead Cache._native_cache runtime path (#44785) 2026-10-05 20:58:19 -07:00
unified_google_tests fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
unit feat(proxy): add opt-in vector_store_deny_by_default for least-privilege vector store access (#44244) 2026-10-06 12:50:25 -07:00
vector_store_tests
windows_tests fix(packaging): keep wheel paths under Windows MAX_PATH for Store Python (#43903) 2026-09-30 22:20:36 +00:00
__init__.py
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py fix(tests): stop VCR recording and replaying a test's own localhost upstream (#43346) 2026-09-26 15:49:39 -07:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_anthropic_compaction_usage.py
test_budget_management.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_callbacks_on_proxy.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_debug_warning.py
test_default_encoding_non_root.py
test_end_users.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_fallbacks.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_gpt5_azure_temperature_support.py
test_health.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_keys.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_litellm_proxy_responses_config.py
test_logging.conf
test_models.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_new_vector_store_endpoints.py
test_openai_endpoints.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_otel_thread_leak.py
test_presidio_latency.py
test_proxy_server_non_root.py
test_ratelimit.py test(proxy): delete the legacy proxy test tree and shard tests/unit/proxy by glob (#44018) 2026-10-01 11:40:45 -07:00
test_resource_cleanup.py
test_rust_python_harness.py test: fix stale and state-leaking tests red on scheduled CircleCI (#43266) 2026-09-25 19:10:20 -07:00
test_service_logger_otel.py
test_spend_logs.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_team.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_team_logging.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_team_members.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
test_users.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.