scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-07 07:58:09 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 150
litellm/tests/test_litellm
History
yucheng-berri 0e9da56f89
fix(batches): strip NUL bytes from passthrough batch tags before the managed object write (#36688) 
PostgreSQL rejects NUL in jsonb with 22P05, and the tags go into the managed
object's CREATE payload, so one poisoned tag aborts the whole row insert rather
than just that column. With no LiteLLM_ManagedObjectTable row, CheckBatchCost
never discovers the batch, so a batch that really ran and billed at the provider
produces no spend at all. The create-time write is fire and forget, so nothing
retries it.

This regressed in #36468, which started passing request_tags and
persist_attribution from the Anthropic passthrough; before that no
caller-supplied string reached the column.

Sanitize in the shared helper that builds the value, matching how
spend_tracking_utils already handles LiteLLM_SpendLogs.request_tags. Both the
Anthropic and the Vertex passthrough build tags through that one helper, so this
covers both. Rename it to _sanitized_str_tuple since it no longer merely
coerces.
2026-08-12 13:31:51 -07:00
..
a2a_protocol fix(http): stop pooled clients persisting cookies on the aiohttp jar too (#36149) 2026-08-07 11:05:59 -07:00
anthropic_interface fix(rust): route agentic-completion-hook /messages requests to Python for all stream modes (#34126) 2026-07-22 00:36:47 +00:00
batches test(batches): run Responses coverage in CI 2026-07-31 10:24:10 -04:00
caching test(caching): align closer tests with self-healing handlers 2026-08-04 18:23:06 -07:00
completion_extras fix(litellm): make the responses bridge and cursor routing total over the surfaces they now serve 2026-08-01 11:26:09 -07:00
compression fix(guardrails/headroom): stop compressing the turn the model must act on (#35294) 2026-07-30 18:53:31 -07:00
containers fix(containers): record ownership for service-account keys + fix Prisma Json serialization (#28990) 2026-05-27 21:00:07 -07:00
enterprise fix(batches): attribute Anthropic passthrough batch cost to the creating key, team and tags (#36468) 2026-08-11 20:52:43 -07:00
expected_fine_tuning_api
expected_responses_api_request test(responses): replace perma-skip azure shell e2e with offline coverage (#32444) 2026-07-08 10:01:41 -07:00
experimental_mcp_client fix(mcp): bound MCP client requests with a session read timeout (#36675) 2026-08-12 12:36:24 -07:00
google_genai chore: litellm oss staging 250626 (#31305) 2026-06-25 21:00:28 -07:00
images
integrations Merge pull request #36685 from BerriAI/litellm_restore_shadowed_tests 2026-08-12 12:06:37 -07:00
interactions fix(tests): assert Content variants are identified by type, not by the discriminator keyword (#35161) 2026-07-29 17:49:30 -07:00
litellm_core_utils Merge pull request #36685 from BerriAI/litellm_restore_shadowed_tests 2026-08-12 12:06:37 -07:00
llms Merge pull request #36685 from BerriAI/litellm_restore_shadowed_tests 2026-08-12 12:06:37 -07:00
models feat(litellm): add models and repository layers (#29686) 2026-06-06 20:59:33 -07:00
ocr test: litellm fix failing tests (#32577) 2026-07-09 13:54:45 -07:00
passthrough fix(passthrough): stream non-sse passthrough responses instead of buffering in memory (#32386) 2026-07-07 20:51:15 -07:00
proxy fix(batches): strip NUL bytes from passthrough batch tags before the managed object write (#36688) 2026-08-12 13:31:51 -07:00
rag fix(rag): track LLM completion usage and spend for /v1/rag/query (#32438) 2026-07-17 17:45:27 +00:00
realtime_api fix(realtime): preserve nested transcription model and session-first model priority 2026-07-03 14:21:22 -07:00
repositories fix(reset_budget_job): atomic budget cascade with chunked reset scans (#36287) 2026-08-10 14:42:36 -07:00
rerank_api fix(rerank): log optional_rerank_params at debug to stop leaking request content (#32533) 2026-07-08 18:48:03 -07:00
responses Merge branch 'litellm_internal_staging' into litellm_/remove-no-guard-mirror-tests 2026-08-12 10:38:24 -07:00
router_strategy fix(router): spend only the router-selecting tags, keep the caller's other tags constraining the routed tier 2026-08-12 01:10:55 -07:00
router_utils feat(router): make routing groups callable as virtual models and list them in /v1/models (#36519) 2026-08-11 18:41:19 -07:00
sandbox feat: add opensandbox sandbox provider (#31024) 2026-06-23 09:05:13 -07:00
secret_managers fix(bedrock): stop replaying expired Google OIDC tokens to STS on guardrail auth (#34637) 2026-07-25 16:27:54 -07:00
test_router fix(logging): stop pinning large request payloads past request end (#33455) 2026-07-15 15:28:37 -07:00
types refactor(types): declare mirrored pricing fields on ModelInfo 2026-08-08 03:25:09 +00:00
vector_stores
videos test(videos): add 1:1 test file scaffold for videos component paths (#30631) 2026-06-29 09:12:51 +05:30
__init__.py
conftest.py test: roll back live router replay membership between tests (#36278) 2026-08-08 10:45:43 -07:00
log.txt
readme.md
test__types.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_a2a_registry_lookup.py
test_acompletion_session_reuse_e2e.py
test_add_deployment_no_master_key.py
test_aembedding_session_reuse_e2e.py
test_anthropic_beta_headers_filtering.py chore(oss): litellm oss staging 120626 (#30292) 2026-06-12 09:49:25 -07:00
test_anthropic_skills_transformation.py fix: encode upstream URL path identifiers 2026-04-29 22:02:39 -07:00
test_anthropic_sonnet_1hr_cache_pricing.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_audio_transcription_rust_bridge.py perf(bedrock): audio transcription via rust core (py->rust bridge) (#33990) 2026-07-20 14:09:41 -07:00
test_azure_ad_token_credential_resolution.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_batch_completion_models_all_responses.py test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) 2026-06-16 10:20:59 -07:00
test_bedrock_anthropic_1hr_cache_pricing.py Litellm oss staging 030626 (#29578) 2026-06-03 11:01:51 -07:00
test_bedrock_usgov_haiku_1hr_cache.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_bedrock_usgov_pricing.py Litellm OSS Staging 010626 (#29422) 2026-06-01 21:42:51 -07:00
test_budget_ratchet_check.py fix(lint): let the ratchet guard recognise a graduated rule 2026-08-07 23:11:23 -07:00
test_chat_ui_responses_session.py
test_check_licenses.py fix(check_licenses): read PEP 639 license-expression metadata (#28529) 2026-05-22 11:22:38 -07:00
test_check_type_discipline.py feat(lint): gate writable TypedDict fields with LIT012 2026-08-11 17:57:34 -07:00
test_circleci_path_filter.py ci: gate CircleCI jobs on changed paths (#32080) 2026-07-04 19:15:08 -07:00
test_circleci_rust_toolchain.py ci(circleci): install a pinned Rust toolchain on the Linux jobs (#35519) 2026-08-03 18:39:35 -07:00
test_claude_fable_5_config.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_claude_haiku_4_5_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_opus_4_6_config.py fix(azure_ai): advertise 1M context window for Claude Opus 4.6+ on Foundry 2026-07-24 21:53:12 +00:00
test_claude_opus_4_8_config.py fix(azure_ai): advertise 1M context window for Claude Opus 4.6+ on Foundry 2026-07-24 21:53:12 +00:00
test_claude_opus_5_config.py feat(anthropic): add Claude Opus 5 2026-07-24 10:43:49 -07:00
test_claude_sonnet_4_6_config.py feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags (#29238) 2026-05-28 18:50:33 -07:00
test_claude_sonnet_5_config.py fix(model_prices): apply claude-sonnet-5 introductory pricing through 2026-08-31 (#31917) 2026-07-01 17:45:57 -07:00
test_cloudflare_workers_ai_model_metadata.py feat(cloudflare): add current Workers AI text-generation models to the cost map (#31051) 2026-06-23 10:44:37 -07:00
test_command_r7b_pricing.py chore: litellm oss 170626 (#30637) 2026-06-17 21:11:12 -07:00
test_completion_timeout_resolution.py fix(router): honor litellm_settings.request_timeout as an independent per-attempt timeout (#31119) 2026-06-23 14:22:54 -07:00
test_component_entrypoint.py fix(docker): honor USE_DDTRACE in the componentized gateway and backend images (#35490) 2026-08-01 14:12:59 -07:00
test_compression.py
test_conftest_isolation.py test: roll back live router replay membership between tests (#36278) 2026-08-08 10:45:43 -07:00
test_constants.py
test_container_router.py
test_cost_calculation_log_level.py
test_cost_calculator.py feat(spend): rebuild the auto-router benchmarks backend as a per-session rollup (#35910) 2026-08-05 20:06:32 +00:00
test_count_tokens_public_api.py
test_dashscope_image_generation.py
test_deepseek_model_metadata.py
test_dockerfile_non_root.py
test_eager_tiktoken_load.py
test_env_key_doc_gate.py fix(ci): make the env-key doc gate see bare get_secret and get_secret_str reads (#35996) 2026-08-05 14:49:55 -07:00
test_exception_exports.py
test_exception_header_preservation.py
test_exception_mapping_request_attribute.py
test_filter_out_litellm_params.py
test_get_blog_posts.py
test_git_hooks.py chore(hooks): enforce Conventional Commits and Conventional Branches (#30174) 2026-06-11 10:00:23 -07:00
test_github_close_low_quality_prs.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_review_gate.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_github_triage_with_llm.py fix(triage): track and credit feature dead-end evidence in the verdict 2026-08-11 10:57:46 -07:00
test_github_triage_workflows.py fix(test): drop references to removed Agent Shin workflows (#30791) 2026-06-18 15:47:51 -07:00
test_gpt_5_4_model_metadata.py fix(pricing): correct gpt-5.4-mini and gpt-5.4-nano token limits 2026-07-30 02:43:54 +00:00
test_gpt_5_5_model_metadata.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_gpt_image_cost_calculator.py chore: litellm oss staging (#31185) 2026-06-26 09:17:44 -07:00
test_gpt_realtime_mode.py test(pricing): pin the realtime mode assertion to the bundled cost map (#33806) 2026-07-18 02:52:55 +00:00
test_groq_streaming_encoding.py
test_guardrail_exception_status_codes.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_lazy_imports.py
test_litellm_params_reserved_keys.py
test_logging.py feat(logging): add opt-in session_id and trace_id correlation to JSON log records via contextvars (#34418) 2026-08-10 10:40:13 -07:00
test_lowest_latency_zero_tokens.py
test_main.py fix(bridge): resolve the effective OpenAI base once, shared by gate and chat handler 2026-08-01 11:26:09 -07:00
test_main_module_header.py Add main module header comment 2026-05-06 00:26:17 +00:00
test_mistral_medium_3_5_model_metadata.py fix(cost-map): retarget mistral-medium-latest to Medium 3.5 and add date-pinned aliases (#31373) 2026-06-25 18:27:18 -07:00
test_model_block_unblock.py fix(proxy): stop model writes 500ing on another pod's delete (#35400) 2026-07-31 18:10:48 -07:00
test_model_cost_aliases.py
test_model_param_helper.py
test_model_prices_schema.py fix(pricing): sync flex/priority tier keys to dated OpenAI snapshot variants 2026-08-04 23:56:52 -07:00
test_model_response_normalization.py
test_muse_spark_1_1_model_metadata.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_nested_drop_params.py
test_openai_embedding_encoding_format_default.py fix(embeddings): allow omitting encoding_format via env sentinel none 2026-05-01 23:03:13 +05:30
test_pre_commit_lint.py build(lint): count deleted files toward check triggers 2026-08-08 10:41:06 -07:00
test_prisma_generate_if_needed.py fix(lint): generate the prisma client into the gate-owned venv 2026-08-06 01:54:26 -07:00
test_project_alias_tracking.py
test_project_tags_pydantic.py
test_rag_openai_ingestion.py chore: litellm oss staging (#30745) 2026-06-18 13:55:35 -07:00
test_rate_limit_error_unification.py test(rate-limits): drop the removed data kwarg from the v3 dynamic limiter raise-branch test 2026-07-30 15:20:44 -07:00
test_redact_string_in_error_paths.py fix(router): redact fallback tracebacks at the call site and cover the sync deferred stream (#35843) 2026-08-05 10:06:52 -07:00
test_redis.py perf: install hiredis so redis-py parses replies with its C parser (#35709) 2026-08-03 20:47:43 -07:00
test_register_model_custom_pricing.py fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) (#32389) 2026-07-07 20:49:03 -07:00
test_register_model_zero_cost_persistence.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_replicate_model_key_format.py fix(model-prices): correct replicate model key typo (#34800) 2026-08-05 16:37:44 -07:00
test_responses_api_bridge_non_stream.py Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
test_responses_id_security.py fix(responses_id_security): decrypt response ids for input_items follow-ups (#32269) 2026-07-06 14:03:15 -07:00
test_responses_streaming_container_ownership.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_retrieve_batch_bedrock_dispatch.py feat(batch-job): bedrock batch model invocation job retrieval (#26834) 2026-05-11 13:22:26 -07:00
test_router.py fix(router): spend only the router-selecting tags, keep the caller's other tags constraining the routed tier 2026-08-12 01:10:55 -07:00
test_router_block_helpers.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
test_router_exception_redaction.py fix(router): mask provider credentials embedded in fallback error messages (#32083) 2026-07-03 18:48:06 -07:00
test_router_google_genai.py
test_router_model_cost_isolation.py fix(router): keep custom model_info across a price data reload (#35491) 2026-08-05 19:56:50 +00:00
test_router_order_fallback.py chore: litellm oss staging160626 (#30527) 2026-06-16 18:23:13 -07:00
test_router_per_deployment_num_retries.py fix(router): honor request-level num_retries over a deployment's litellm_params value (#35483) 2026-08-01 13:51:29 -07:00
test_router_redis_init.py
test_router_retry_non_retryable_errors.py
test_router_retry_policy_update.py fix(proxy): tighten role gating on /get/config/callbacks response (#31745) 2026-07-01 17:58:31 -07:00
test_router_silent_experiment.py
test_router_streaming_fallback_metadata.py chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
test_router_weighted_failover.py fix(router): cool down failed fallback deployments and correct cooldown TTL after Redis backfill (#35104) 2026-08-10 16:51:55 -07:00
test_ruff_strict_gate.py refactor(lint): graduate the 35 zero-violation strict rules into ruff.toml 2026-08-07 23:10:33 -07:00
test_secret_redaction.py fix(logging): extend secret redaction to records litellm does not emit directly (#35977) 2026-08-05 14:50:42 -07:00
test_service_logger.py fix: missing span for guardrail passthrough (#29552) 2026-06-03 01:25:15 +00:00
test_setup_wizard.py
test_shared_session_integration.py
test_ssl_verify_unit.py refactor(bedrock): remove the dead BedrockLLM invoke code path 2026-07-29 20:25:36 -07:00
test_stream_chunk_builder_annotations.py
test_streaming_connection_cleanup.py
test_system_message_format_bug.py
test_thinking_enabled.py Litellm oss staging 250526 (#28770) 2026-05-26 11:57:39 -07:00
test_triage_rollout_heads_up.py feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433) 2026-06-17 20:42:27 -07:00
test_type_check_gate.py fix(lint): retire the single-slot base-counts cache 2026-08-06 02:23:52 -07:00
test_type_discipline_gate.py fix(lint): pick the merge-aware base so in-progress merges are not blamed for base drift 2026-08-04 17:58:41 -07:00
test_utils.py test: remove tests that never execute 2026-08-12 10:45:38 -07:00
test_utils_module_docstring.py Add utils module docstring 2026-05-06 00:42:49 +00:00
test_uuid_helper.py
test_vcr_safe_body_matcher.py test: stabilize batch VCR coverage and stop live upload/network leaks (#29477) 2026-06-02 16:11:52 -07:00
test_video_generation.py fix(vertex-ai): use DB credentials in video handlers + implement Veo video edit (#29098) 2026-05-28 11:45:41 -07:00
test_with_dashboard_node.py fix(bootstrap): fail fast when nvm cannot activate the pinned node 2026-08-04 21:18:58 -07:00
test_xai_grok_4_3_model_metadata.py feat(xai): add grok-4.3 and grok-4.3-latest to model_prices_and_conte… (#27154) 2026-05-07 09:06:56 -07:00
test_xai_responses_auto_routing.py

readme.md

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py