scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-27 01:22:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 82
litellm/tests/test_litellm
History
devin-ai-integration[bot] c6c3881d7f
fix(proxy): share model rate-limit buckets between a model_group_alias and its target (#42516) 
* fix(proxy): share model rate-limit buckets between a model_group_alias and its target

A request sent under a model_group_alias counted in its own per-key, per-team,
per-org, and per-project model bucket, so a key could double a deployment's
default_api_key_rpm_limit / tpm_limit by alternating the alias and the model
group name, and a metadata model_rpm_limit / model_tpm_limit keyed by the
model group never applied to alias requests. The limiter now resolves the
requested name to its model group before keying any model bucket, looks the
limit up by the requested name first and the model group second, and charges
post-call tokens to the same bucket.

* fix(proxy): charge the model group resolved at admission when reconciling reserved tokens

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-22 13:31:58 -07:00
..
a2a_protocol fix(a2a): send message/stream for Bedrock AgentCore streaming requests 2026-09-21 13:39:32 +00:00
batches test: migrate wave 1 phase 1 legacy tests to tests/unit 2026-09-20 09:17:57 +00:00
caching fix(cache-qdrant-semantic): wait for Qdrant upserts to be indexed 2026-09-21 21:47:56 +00:00
chat_completions test: migrate wave 1 phase 1 legacy tests to tests/unit 2026-09-20 09:17:57 +00:00
completion_extras fix(edenai): restore unrelated files the squashed PR commit had reverted 2026-09-21 19:42:37 +00:00
containers test: delete assertions that pin vendor cost map facts 2026-09-18 00:28:49 +00:00
endpoints test: migrate wave 1 phase 1 legacy tests to tests/unit 2026-09-20 09:17:57 +00:00
enterprise test: migrate wave 1 phase 1 legacy tests to tests/unit 2026-09-20 09:17:57 +00:00
expected_fine_tuning_api
expected_responses_api_request
experimental_mcp_client fix(mcp): restore legacy SSE and bounded cancellation cleanup (#42382) 2026-09-22 11:27:50 -07:00
fixtures/together_ai_sync feat(models): add daily Together AI model registry sync script and workflow 2026-08-25 13:12:06 -07:00
google_genai fix(google_genai): drop non-object tool parameters instead of forwarding them 2026-09-19 19:19:33 -07:00
images test(images): pin scalar-array edit params survive as repeated multipart fields 2026-08-24 12:49:54 -07:00
integrations fix(websearch_interception): keep intercepted searches under the parent request's session and trace (#41711) 2026-09-22 12:35:27 -07:00
interactions refactor(interactions): remove expired use_legacy_interactions_schema shim 2026-09-17 20:14:07 +00:00
litellm_core_utils feat: add configurable provider affinity header mapping (#41033) 2026-09-22 13:31:33 -07:00
llms feat: add configurable provider affinity header mapping (#41033) 2026-09-22 13:31:33 -07:00
messages test: migrate phase 15 legacy tests to tests/unit 2026-09-20 11:55:43 +00:00
ocr test: migrate phase 15 legacy tests to tests/unit 2026-09-20 11:55:43 +00:00
passthrough test: migrate phase 15 legacy tests to tests/unit 2026-09-20 11:55:43 +00:00
proxy fix(proxy): share model rate-limit buckets between a model_group_alias and its target (#42516) 2026-09-22 13:31:58 -07:00
rag test: migrate phase 15 legacy tests to tests/unit 2026-09-20 11:55:43 +00:00
rerank_api ci(unit): fail a hung test in 120s with a traceback instead of idling the shard to its step timeout 2026-09-19 02:53:36 -07:00
responses feat: add configurable provider affinity header mapping (#41033) 2026-09-22 13:31:33 -07:00
router_strategy feat(router): add group-scoped priority routing strategy (#42378) 2026-09-22 13:09:38 -07:00
router_utils feat(openai): add GPT-6 Sol and GPT-6 Luna (#42515) 2026-09-22 11:34:01 -07:00
rust_bridge test(rust_bridge): drop route dispatch assertions, test the bridge directly (#42536) 2026-09-22 19:21:32 +00:00
secret_managers fix(aws_secret_manager_v2): restore secret scheduled for deletion instead of failing CreateSecret (#42454) 2026-09-22 14:12:20 -05:00
types feat: add configurable provider affinity header mapping (#41033) 2026-09-22 13:31:33 -07:00
vector_stores fix(bedrock): forward userContext in Knowledge Base Retrieve requests 2026-09-16 11:50:36 -07:00
videos test: remove phase 16 legacy test files from tests/test_litellm 2026-09-20 10:59:22 +00:00
__init__.py
conftest.py fix(bedrock): gate Invoke tool search on the model map's supports_tool_search flag 2026-09-16 18:23:14 -07:00
log.txt
readme.md
test_a2a_registry_lookup.py fix(a2a): Entra credentials own the chat route bearer over a stored api_key or authorization header 2026-09-17 16:09:36 -07:00
test_acompletion_session_reuse_e2e.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_add_deployment_no_master_key.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_aembedding_session_reuse_e2e.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_anthropic_beta_headers_filtering.py feat(router): native compact-to-fit across conversation APIs (#42074) 2026-09-21 22:52:29 -07:00
test_anthropic_skills_transformation.py
test_assert_ci_coverage.py test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
test_assert_workflow_dir_hygiene.py feat(ci): assert .github/workflows holds only workflows, correctly named (#37616) 2026-08-20 21:36:26 +00:00
test_audio_transcription_rust_bridge.py refactor(rust_bridge): group route modules into packages and split ocr into main and rust 2026-09-16 20:34:51 +00:00
test_auto_update_price_and_context_window_file.py feat(cost_map): derive source_revision from the loaded bytes instead of a _metadata stamp 2026-09-07 17:47:51 -07:00
test_azure_ad_token_credential_resolution.py test(router): cover s3_output_bucket_name surviving the trusted credential snapshot 2026-08-17 14:51:10 -07:00
test_azure_ai_grok_4_3_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_azure_ai_grok_4_6_model_metadata.py test: keep behavior tests that read the cost map for a later fixture rewrite 2026-09-18 04:52:06 +00:00
test_baseten_glm_5_3_model_metadata.py test: drop tests that pin provider-owned cost map values 2026-09-18 03:55:51 +00:00
test_batch_completion_models_all_responses.py
test_bedrock_marengo_embed_3_model_metadata.py test: drop tests that pin provider-owned cost map values 2026-09-18 03:55:51 +00:00
test_budget_ratchet_check.py test(ci): annotate new test locals as Final 2026-09-20 09:07:51 +00:00
test_chat_ui_responses_session.py
test_check_licenses.py fix(ci): retry transient PyPI license lookups 2026-08-23 09:23:35 +00:00
test_check_mcp_operation_boundary.py refactor(mcp): extract explicit operation context and dispatch 2026-09-21 12:24:15 -07:00
test_check_migrations_no_data_rewrites.py fix(migrations-check): read the table name past comments, ignore referential SET DEFAULT 2026-09-16 18:14:36 +00:00
test_check_py310_typing_imports.py fix: keep litellm importable on Python 3.10 and guard 3.11-only typing imports in CI (#39448) 2026-09-02 18:27:19 -07:00
test_check_test_quality.py test(ci): annotate new test locals as Final 2026-09-20 09:07:51 +00:00
test_check_type_discipline.py perf(ci): fan the budget checkers out across cores (#37784) 2026-08-22 22:44:58 -07:00
test_circleci_path_filter.py fix(mcp): preserve session expiry signals and scope dependency CI 2026-09-18 22:52:10 -07:00
test_circleci_rust_toolchain.py fix(ci): pin workflow toolchain dependencies 2026-09-02 12:16:25 -07:00
test_claude_fable_5_config.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_claude_opus_4_6_config.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_claude_opus_4_8_config.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_claude_opus_5_config.py feat(anthropic): add Claude Opus 5.5 (#42489) 2026-09-22 09:52:35 -07:00
test_claude_sonnet_5_config.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_cloudflare_workers_ai_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_completion_timeout_resolution.py
test_component_entrypoint.py feat(proxy): share database connections across workers with an in-container pgbouncer (#39683) 2026-09-10 22:14:37 +00:00
test_compression.py
test_conftest.py test: trim the PROXY_BASE_URL fixture and regression docstrings 2026-08-19 00:56:37 -07:00
test_conftest_isolation.py test: roll back live router replay membership between tests (#36278) 2026-08-08 10:45:43 -07:00
test_constants.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_container_router.py
test_cost_calculation_log_level.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_cost_calculator.py fix(cost): honor deployment pricing for image generation (#39311) 2026-09-22 11:40:39 -07:00
test_cost_map_guard.py ci: skip cost map file checks on PRs that leave the cost map untouched (#42406) 2026-09-21 21:40:48 -07:00
test_count_tokens_public_api.py fix(proxy): run the remaining inline token counts off the event loop 2026-09-08 18:42:38 +00:00
test_dashscope_image_generation.py test: keep the pinning-test removal free of unrelated reformatting 2026-09-18 04:27:28 +00:00
test_daybreak_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_deepseek_model_metadata.py test: drop tests that pin provider-owned cost map values 2026-09-18 03:55:51 +00:00
test_default_branch.py ci: avoid duplicate default branch fetches 2026-09-07 15:28:01 -07:00
test_detect_changes.py perf(ci): gate the lint, MCP and dashboard jobs on the pull request's file list (#37559) 2026-08-19 18:32:21 -07:00
test_dockerfile_apk_repository.py fix(docker): add public Wolfi apk repo to runtime image (#39033) 2026-09-01 15:11:15 -07:00
test_dockerfile_bedrock_realtime_extra.py fix(bedrock): keep realtime SDK error range inside websocket close reason 2026-09-17 01:57:33 +00:00
test_dockerfile_non_root.py
test_drop_params_env_var.py fix(init): keep non-flag LITELLM_DROP_PARAMS values on with a warning 2026-09-07 22:13:23 -07:00
test_e2e_egress_sentinel.py ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
test_eager_tiktoken_load.py
test_env_key_doc_gate.py fix(ci): make the env-key doc gate see bare get_secret and get_secret_str reads (#35996) 2026-08-05 14:49:55 -07:00
test_exception_exports.py
test_exception_header_preservation.py fix(bedrock): keep x-amzn-RequestId on chat error responses (#40089) 2026-09-07 17:16:47 -07:00
test_exception_mapping_request_attribute.py
test_filter_out_litellm_params.py
test_fireworks_serverless_model_costs.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_gate_slot_lock.py ci: avoid duplicate default branch fetches 2026-09-07 15:28:01 -07:00
test_gemini_3_1_flash_lite_image_pricing.py test: drop gemini-3.1-flash-lite-image capability pins 2026-09-16 19:21:50 +00:00
test_gemini_tts_native_audio_pricing.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_get_blog_posts.py test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
test_git_hooks.py
test_gpt_5_4_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_gpt_5_5_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_gpt_image_cost_calculator.py fix(cost): honor deployment pricing for image generation (#39311) 2026-09-22 11:40:39 -07:00
test_gpt_realtime_mode.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_groq_streaming_encoding.py
test_guardrail_exception_status_codes.py
test_lazy_imports.py perf: defer fastapi and tiktoken BPE imports out of import litellm 2026-09-17 08:59:48 +00:00
test_lint_workflow_diff_gates.py ci(lint): gate top-level tests/e2e and litellm files in the diff-scoped lint steps 2026-09-01 22:30:46 -07:00
test_litellm_params_reserved_keys.py
test_logging.py fix(logging): stamp scrubbed records with a private sentinel a caller cannot supply 2026-09-16 17:30:49 -07:00
test_lowest_latency_zero_tokens.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_main.py fix(edenai): restore unrelated files the squashed PR commit had reverted 2026-09-21 19:42:37 +00:00
test_main_module_header.py
test_mistral_medium_3_5_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_mistral_small_4_0_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_mistral_zai_glm_5_2_model_metadata.py test: keep behavior tests that read the cost map for a later fixture rewrite 2026-09-18 04:52:06 +00:00
test_model_block_unblock.py fix(proxy): serialize model reconciles so concurrent model writes stop evicting each other (#36687) 2026-08-12 13:42:26 -07:00
test_model_cost_aliases.py
test_model_param_helper.py
test_model_prices_schema.py fix(cost): bill DeepSeek V4.1 Flash and V4 Pro at their off-peak rates outside peak hours 2026-09-19 04:31:55 -07:00
test_model_response_normalization.py
test_muse_spark_1_1_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_muse_spark_1_2_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_muse_spark_1_3_model_metadata.py test: delete assertions that pin vendor cost map facts 2026-09-18 00:28:49 +00:00
test_mutation_report.py fix(ci): stop the mutation report publishing a score it never measured (#37825) 2026-08-21 20:15:52 -07:00
test_nested_drop_params.py
test_non_chat_routes_open_llm_spans.py fix(otel): emit LLM Call spans for speech, image, moderation, ocr and transcription (#37752) 2026-08-22 11:11:21 -07:00
test_openai_embedding_encoding_format_default.py test(embeddings): move legacy intercepts to the wire for the omitted-format path 2026-08-29 12:04:44 -07:00
test_openai_service_tier_long_context_pricing.py feat(openai): add GPT-6 Sol and GPT-6 Luna (#42515) 2026-09-22 11:34:01 -07:00
test_pre_commit_lint.py ci: follow the default branch in development tooling 2026-09-07 14:34:45 -07:00
test_prisma_generate_if_needed.py fix(lint): generate the prisma client into the gate-owned venv 2026-08-06 01:54:26 -07:00
test_project_alias_tracking.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_project_tags_pydantic.py test(lint): ban blind pytest.raises(Exception) with ruff B017 (#37731) 2026-08-20 18:09:42 -07:00
test_proxy_auth.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_rag_openai_ingestion.py
test_rate_limit_error_unification.py feat(proxy): add budget_exceeded_status_code setting to restore 429 for budget refusals 2026-09-20 07:00:10 +00:00
test_redact_string_in_error_paths.py fix(proxy): stop leaking internal exception details to clients (#39380) 2026-09-02 17:32:00 -07:00
test_redis.py test(redis): pin ElastiCache IAM signing and TLS coercion invariants 2026-09-10 10:13:03 -04:00
test_redis_credential_provider.py test(redis): pin ElastiCache IAM signing and TLS coercion invariants 2026-09-10 10:13:03 -04:00
test_register_model_custom_pricing.py fix(cost): bill off-peak rates for deployments that set only off_peak_pricing 2026-09-01 12:14:46 -07:00
test_register_model_zero_cost_persistence.py
test_replicate_model_key_format.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_responses_api_bridge_non_stream.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_responses_id_security.py fix(proxy): authorize every Responses API id, not only the ones the proxy issued (#39548) 2026-09-11 11:47:05 -07:00
test_responses_streaming_container_ownership.py
test_retrieve_batch_bedrock_dispatch.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router.py feat(router): time-windowed team reservation of deployments via model_info.access_windows (#42398) 2026-09-22 13:22:15 -05:00
test_router_block_helpers.py
test_router_exception_redaction.py fix(router): explain fallback outcome in plain words in the raised error (#42509) 2026-09-22 18:37:42 +00:00
test_router_google_genai.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_router_model_cost_isolation.py fix(router): preserve discovered limits and model info fallbacks 2026-09-17 00:23:50 +00:00
test_router_order_fallback.py fix(router): skip the refusing deployment when retrying a non-transient error 2026-09-05 22:25:13 -07:00
test_router_per_deployment_num_retries.py refactor(router): resolve retry policy by exception MRO and add DefaultRetries 2026-09-04 16:09:01 -07:00
test_router_redis_init.py
test_router_retry_backoff_headers.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_router_retry_non_retryable_errors.py fix(timing): union provider timing windows and anchor detailed pre-processing at receive time 2026-09-19 00:48:17 +00:00
test_router_retry_policy_update.py fix(proxy): persist only the router settings keys the request set 2026-09-18 14:01:05 -07:00
test_router_silent_experiment.py fix(router): snapshot shadow kwargs per target so concurrent shadows never share metadata 2026-09-16 04:31:43 +00:00
test_router_streaming_fallback_metadata.py
test_router_weighted_failover.py fix(router): skip the refusing deployment when retrying a non-transient error 2026-09-05 22:25:13 -07:00
test_ruff_strict_gate.py fix: address cross-version CI failures 2026-09-02 14:17:19 -07:00
test_sambanova_model_metadata.py test: keep behavior tests that read the cost map for a later fixture rewrite 2026-09-18 04:52:06 +00:00
test_secret_redaction.py fix(bedrock): keep batch S3 credentials out of chat requests and debug logs 2026-09-21 20:42:04 +00:00
test_select_ui_test_scope.py test(ui): align workflow expectations with Vitest 4 2026-09-08 16:13:58 -07:00
test_service_logger.py
test_setup_wizard.py
test_shared_session_integration.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_ssl_verify_unit.py refactor(bedrock): remove the dead BedrockLLM invoke code path 2026-07-29 20:25:36 -07:00
test_stream_chunk_builder_annotations.py
test_stream_chunk_builder_citations.py fix(streaming): join block-list citation deltas without extra nesting 2026-08-28 13:56:07 -07:00
test_stream_chunk_builder_images.py test: run the 30 test files stranded in the second mirror (#37595) 2026-08-20 10:59:43 -07:00
test_streaming_connection_cleanup.py test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 2026-08-22 09:25:58 -07:00
test_sync_together_ai_models.py feat(cost_map): derive source_revision from the loaded bytes instead of a _metadata stamp 2026-09-07 17:47:51 -07:00
test_system_message_format_bug.py
test_test_quality_gate.py ci(tests): wire tests/unit into CircleCI and drain legacy unit shards green 2026-09-20 07:05:42 +00:00
test_thinking_enabled.py test: drop restating comment and wrap long call in thinking tests 2026-08-18 19:55:22 -07:00
test_together_ai_model_metadata.py test: delete assertions that pin vendor cost map facts 2026-09-18 00:28:49 +00:00
test_type_check_gate.py fix(lint): retire the single-slot base-counts cache 2026-08-06 02:23:52 -07:00
test_type_discipline_gate.py fix(lint): pick the merge-aware base so in-progress merges are not blamed for base drift 2026-08-04 17:58:41 -07:00
test_typesafe_model_metadata.py feat(proxy): add TypeSafe Jev passthrough spend tracking 2026-09-17 15:53:19 +00:00
test_unit_shard_per_test_timeout.py test(ci): drop the structure-only assertion on the shard script; the parametrized hang test covers both invocations 2026-09-19 03:20:48 -07:00
test_utils.py fix(utils): stop a nested additional_drop_params entry from crashing openai-compatible calls (#42492) 2026-09-22 10:50:49 -07:00
test_utils_module_docstring.py
test_uuid_helper.py
test_vcr_safe_body_matcher.py
test_vertex_ai_xai_grok_prompt_caching_metadata.py test(pricing): assert cache-priced vertex grok rows advertise supports_prompt_caching (#41526) 2026-09-21 20:58:11 -07:00
test_video_generation.py test: delete assertions that pin vendor cost map facts 2026-09-18 00:28:49 +00:00
test_with_dashboard_node.py fix(bootstrap): fail fast when nvm cannot activate the pinned node 2026-08-04 21:18:58 -07:00
test_xai_grok_4_3_model_metadata.py test: keep tests that survive correct cost-map updates 2026-09-15 22:16:08 +00:00
test_xai_responses_auto_routing.py fix(xai): keep 'instructions' on the xAI Responses API so system messages survive web_search bridging 2026-09-16 01:35:13 +00:00

readme.md

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py