scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-06 08:16:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 78
litellm/tests/test_litellm
History
yucheng-berri 7680cedf42
test(logging): regression coverage for streaming /v1/messages OpenAI Responses spend logs (#31388) 
* test(logging): cover streaming /v1/messages OpenAI Responses spend logs

The #28595 fix added unit tests that call _handle_anthropic_messages_response_logging
directly, but nothing exercises the streaming wiring that actually regressed:
a streaming /v1/messages call cross-routed to the OpenAI Responses backend whose
success handler took the no-op async_log_stream_event path and dropped the SpendLogs
row. Add an end-to-end test that drives litellm.anthropic_messages(stream=True) with a
mocked upstream Responses SSE and asserts async_log_success_event fires with non-zero
cost and call_type anthropic_messages, plus a key-gated live counterpart.

* test(logging): exercise stream deltas and assert single success log

Address review on the streaming bridge regression test: emit output_item.added
plus text deltas before response.completed so it covers mid-stream delta handling
rather than only end-of-stream success logging, assert at least one
content_block_delta surfaces, restore litellm.callbacks via monkeypatch instead of
leaking global state, and assert async_log_success_event fires exactly once.

* test(logging): drop live network test from mock-only suite

Greptile flagged that tests/test_litellm only permits mock tests; network calls
belong in tests/e2e. Remove the key-gated live counterpart and keep the
deterministic mocked test as the regression guard. The live verification stays
in the PR description as the proof of fix.
2026-06-25 20:30:04 -07:00
..
a2a_protocol fix(a2a): forward agent_extra_headers through completion bridge (#28277) 2026-06-11 21:56:18 -07:00
anthropic_interface/exceptions style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
caching fix(cache): apply Redis namespace to all key operations (#31288) 2026-06-25 15:39:07 -07:00
completion_extras chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
containers fix(containers): record ownership for service-account keys + fix Prisma Json serialization (#28990) 2026-05-27 21:00:07 -07:00
enterprise chore: litellm oss staging (#30745) 2026-06-18 13:55:35 -07:00
expected_fine_tuning_api refactor: refactor testing 2026-03-28 18:39:32 -07:00
expected_responses_api_request [Feat] Adds support for server-side compaction on the OpenAI Responses API context_management (#21058) 2026-02-12 10:00:30 -08:00
experimental_mcp_client feat(mcp): graft v2 resolver onto _create_mcp_client (none + api_key static family) (#31058) 2026-06-24 14:53:33 -07:00
google_genai fix(google_genai): preserve complete SSE events in Vertex/Gemini image streaming (#30270) 2026-06-12 07:49:30 -07:00
images fix(image_edit): read vertex_project/location from litellm_params in Imagen get_complete_url 2026-04-21 15:03:40 +05:30
integrations fix(otel): hashable scope for _emit_once when guardrail_mode is list (#31262) 2026-06-25 11:59:35 -07:00
interactions chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
litellm_core_utils test(logging): regression coverage for streaming /v1/messages OpenAI Responses spend logs (#31388) 2026-06-25 20:30:04 -07:00
llms fix(vertex): preserve Gemini Embedding 2 usageMetadata for cost tracking (#31354) 2026-06-25 19:53:16 -07:00
models feat(litellm): add models and repository layers (#29686) 2026-06-06 20:59:33 -07:00
ocr feat(ocr): thin Rust OCR Python bridge (#31368) 2026-06-25 18:42:59 -07:00
passthrough refactor: trim explanatory comments from streaming-flush fix 2026-04-30 02:39:28 +00:00
proxy feat(mcp): opt-in least-privilege default for team key MCP access (#31380) 2026-06-25 18:49:15 -07:00
repositories feat(litellm): add models and repository layers (#29686) 2026-06-06 20:59:33 -07:00
responses chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
router_strategy fix(router): enforce deployment budgets for dynamically added models (#29273) 2026-05-29 19:43:14 -07:00
router_utils chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
sandbox feat: add opensandbox sandbox provider (#31024) 2026-06-23 09:05:13 -07:00
secret_managers chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_router fix: use atomic increment-first pattern for model RPM rate limiting 2026-02-24 09:55:07 -03:00
types refactor(completion): extract provider dispatch into typed helpers so basedpyright can analyze it (#30813) 2026-06-23 07:29:31 -07:00
vector_stores style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
__init__.py
conftest.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
log.txt
readme.md
test__types.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_a2a_registry_lookup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_acompletion_session_reuse_e2e.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_add_deployment_no_master_key.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_aembedding_session_reuse_e2e.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_anthropic_beta_headers_filtering.py chore(oss): litellm oss staging 120626 (#30292) 2026-06-12 09:49:25 -07:00
test_anthropic_skills_transformation.py fix: encode upstream URL path identifiers 2026-04-29 22:02:39 -07:00
test_anthropic_sonnet_1hr_cache_pricing.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_azure_ad_token_credential_resolution.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_azure_video_router.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_batch_completion_models_all_responses.py test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) 2026-06-16 10:20:59 -07:00
test_bedrock_anthropic_1hr_cache_pricing.py Litellm oss staging 030626 (#29578) 2026-06-03 11:01:51 -07:00
test_bedrock_usgov_haiku_1hr_cache.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_bedrock_usgov_pricing.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_budget_ratchet_check.py chore(lint): widen ANN slack to 10% of baseline and drop PLR0913 from the strict gate (#31335) 2026-06-25 14:43:45 -07:00
test_chat_ui_responses_session.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_check_licenses.py fix(check_licenses): read PEP 639 license-expression metadata (#28529) 2026-05-22 11:22:38 -07:00
test_check_type_discipline.py ci(lint): ratcheted type-discipline gate (mutable collections, casts, guards, kwargs, suppressions) (#30500) 2026-06-16 16:59:21 -07:00
test_claude_fable_5_config.py Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064) 2026-06-10 08:50:15 +05:30
test_claude_haiku_4_5_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_opus_4_6_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_opus_4_8_config.py fix(anthropic): route Claude Opus 4.8 through adaptive thinking (#29702) 2026-06-05 16:19:01 +05:30
test_claude_sonnet_4_6_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_cloudflare_workers_ai_model_metadata.py feat(cloudflare): add current Workers AI text-generation models to the cost map (#31051) 2026-06-23 10:44:37 -07:00
test_command_r7b_pricing.py chore: litellm oss 170626 (#30637) 2026-06-17 21:11:12 -07:00
test_completion_timeout_resolution.py fix(router): honor litellm_settings.request_timeout as an independent per-attempt timeout (#31119) 2026-06-23 14:22:54 -07:00
test_compression.py Prompt Compression - add it to the proxy (#25729) 2026-04-20 15:08:00 -07:00
test_constants.py Fix test 2026-03-27 21:21:43 +05:30
test_container_router.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_cost_calculation_log_level.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_cost_calculator.py fix(realtime): stop revalidating realtime events at the logging boundary (#31054) 2026-06-22 20:43:03 -07:00
test_count_tokens_public_api.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_dashscope_image_generation.py fix(dashscope): fail fast on image generation API errors 2026-04-23 18:41:01 +05:30
test_deepseek_model_metadata.py Revert "Merge pull request #16590 from Chesars/refactor/remove-backup-file-dry-principle" 2026-04-25 17:10:41 -03:00
test_dockerfile_non_root.py fix(docker.non_root): use numeric UID 65534 for K8s runAsNonRoot (#26268) 2026-04-22 18:00:04 -07:00
test_eager_tiktoken_load.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_exception_exports.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_exception_header_preservation.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_exception_mapping_request_attribute.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_filter_out_litellm_params.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_get_blog_posts.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_git_hooks.py chore(hooks): enforce Conventional Commits and Conventional Branches (#30174) 2026-06-11 10:00:23 -07:00
test_github_close_low_quality_prs.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_review_gate.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_triage_with_llm.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_triage_workflows.py fix(test): drop references to removed Agent Shin workflows (#30791) 2026-06-18 15:47:51 -07:00
test_gpt_5_5_model_metadata.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_gpt_image_cost_calculator.py Add gpt-image-2 support (#26644) (#26705) 2026-04-28 20:10:42 -07:00
test_groq_streaming_encoding.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_guardrail_exception_status_codes.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_lazy_imports.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_litellm_params_reserved_keys.py fix(snowflake): transform tool_choice string to object format (#23268) 2026-03-11 01:41:24 +05:30
test_logging.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_lowest_latency_zero_tokens.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_main.py Litellm oss staging 040626 (#29671) 2026-06-04 11:07:20 -07:00
test_main_module_header.py Add main module header comment 2026-05-06 00:26:17 +00:00
test_mistral_medium_3_5_model_metadata.py fix(cost-map): retarget mistral-medium-latest to Medium 3.5 and add date-pinned aliases (#31373) 2026-06-25 18:27:18 -07:00
test_model_block_unblock.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
test_model_cost_aliases.py [Feat] - Ishaan main merge branch (#23596) 2026-03-14 09:40:00 -07:00
test_model_param_helper.py style: black format test_model_param_helper.py 2026-04-15 18:18:52 -07:00
test_model_response_normalization.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_nested_drop_params.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_openai_embedding_encoding_format_default.py fix(embeddings): allow omitting encoding_format via env sentinel none 2026-05-01 23:03:13 +05:30
test_project_alias_tracking.py feat(proxy): add project_alias tracking through callback metadata pipeline 2026-03-23 10:44:17 -07:00
test_project_tags_pydantic.py fix: req changes 2026-02-27 13:33:34 +05:30
test_rag_openai_ingestion.py chore: litellm oss staging (#30745) 2026-06-18 13:55:35 -07:00
test_rate_limit_error_unification.py feat: standardize rate limit errors with category, rate_limit_type, model, and llm_provider fields (#27687) 2026-06-06 17:50:29 -07:00
test_redact_string_in_error_paths.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_redis.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_register_model_custom_pricing.py fix(register_model): preserve built-in cache pricing when registering custom overrides under unmapped keys (#30044) 2026-06-10 12:11:03 -07:00
test_register_model_zero_cost_persistence.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_responses_api_bridge_non_stream.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_responses_id_security.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_responses_streaming_container_ownership.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_retrieve_batch_bedrock_dispatch.py feat(batch-job): bedrock batch model invocation job retrieval (#26834) 2026-05-11 13:22:26 -07:00
test_router.py fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads (#31036) 2026-06-24 13:19:57 -07:00
test_router_block_helpers.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
test_router_exception_redaction.py chore: litellm oss 170626 (#30637) 2026-06-17 21:11:12 -07:00
test_router_google_genai.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_model_cost_isolation.py fix(router): isolate all per-deployment pricing overrides from sibling deployments (#31021) 2026-06-23 00:11:22 -07:00
test_router_order_fallback.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_router_per_deployment_num_retries.py fix(router): guard num_retries=None in async_function_with_retries (#30036) 2026-06-23 10:16:15 -07:00
test_router_redis_init.py fix: handle deprecated 'redis_db' arg to prevent crash (#19808) 2026-02-02 18:18:05 +05:30
test_router_retry_non_retryable_errors.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_silent_experiment.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_streaming_fallback_metadata.py chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
test_router_weighted_failover.py feat: add weighted-routing failover (#27980) 2026-05-15 17:28:54 +00:00
test_ruff_strict_gate.py feat(auth): resolve caller identity once into a Principal at the auth seam (#30887) 2026-06-20 18:49:41 -07:00
test_secret_redaction.py Litellm oss staging (#29492) 2026-06-02 08:48:10 -07:00
test_service_logger.py fix: missing span for guardrail passthrough (#29552) 2026-06-03 01:25:15 +00:00
test_setup_wizard.py test: test 2026-03-28 19:17:38 -07:00
test_shared_session_integration.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_ssl_verify_unit.py Litellm OSS Staging (#29161) 2026-06-01 21:22:35 -07:00
test_stream_chunk_builder_annotations.py fix: merge annotations from all streaming chunks in stream_chunk_builder 2026-03-15 14:20:45 +05:30
test_streaming_connection_cleanup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_system_message_format_bug.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_thinking_enabled.py Litellm oss staging 250526 (#28770) 2026-05-26 11:57:39 -07:00
test_triage_rollout_heads_up.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_type_check_gate.py ci: make the basedpyright budget gate delta-vs-base (#31106) 2026-06-23 11:10:05 -07:00
test_type_discipline_gate.py ci(lint): ratcheted type-discipline gate (mutable collections, casts, guards, kwargs, suppressions) (#30500) 2026-06-16 16:59:21 -07:00
test_utils.py fix(anthropic): drop unsupported speed param with drop_params (#31152) 2026-06-23 22:22:49 -07:00
test_utils_module_docstring.py Add utils module docstring 2026-05-06 00:42:49 +00:00
test_uuid_helper.py
test_vcr_safe_body_matcher.py test: stabilize batch VCR coverage and stop live upload/network leaks (#29477) 2026-06-02 16:11:52 -07:00
test_video_generation.py fix(vertex-ai): use DB credentials in video handlers + implement Veo video edit (#29098) 2026-05-28 11:45:41 -07:00
test_xai_grok_4_3_model_metadata.py feat(xai): add grok-4.3 and grok-4.3-latest to model_prices_and_conte… (#27154) 2026-05-07 09:06:56 -07:00
test_xai_responses_auto_routing.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00

readme.md

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py