scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-11 03:38:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 82
litellm/tests/unit
History
devin-ai-integration[bot] 6f5ca84f69
feat(prometheus): cap series per metric for every labeled metric (#44420) 
* feat(prometheus): cap series per metric for every labeled metric

Add prometheus_metrics_max_series_per_metric: per metric and per worker process, the first N label
sets keep a series of their own. Counters and histograms record every later label set on one series
whose labels are all "other", so totals stay exact, and gauges skip it. The cap holds with multiple
workers because it never needs to remove a series.

Add prometheus_metrics_ttl_seconds: a series idle for that long is removed and its slot is freed.
The prometheus client cannot remove a series in multi-process mode, so the TTL is ignored there with
a startup warning.

Both settings are off by default. The end_user caps are unchanged.

* fix(prometheus): share the series cap across workers of one proxy instance

Workers writing to one PROMETHEUS_MULTIPROC_DIR now agree on which label
sets get a series through an append-only admissions file per metric, so a
merged scrape stays at the cap plus `other` instead of growing with every
worker and every worker restart. The two fallback counters now pass their
label names as a keyword so the cap and prometheus_exclude_labels apply to
them, admission and child creation happen under one lock, the test fixture
restores the shared registry, and the `other` label value lives in
constants.py.

* test(prometheus): check emitted labels instead of wrapper types, close the admission match

The exclude-labels test now emits through the spend and provider budget
metrics and checks the scrape keeps all their labels. The admission match
arms end in assert_never so the match is exhaustive.

* fix(prometheus): return the exhaustive-match fallback so every admission arm returns

* fix(prometheus): pick the series tracker with isinstance so every path of _admits returns

* fix(prometheus): skip an admissions line a worker could only write part of

* fix(prometheus): frame each admissions record with newlines so a cut-off record cannot swallow the next

A record a worker could only write part of used to merge with the next worker's record, and both were skipped for one request. Each record is now written between two newlines, so the fragment is a line of its own. The clock fixture in the series tests starts from a constant instead of reading the real clock

* fix(prometheus): ignore a non-positive series cap or TTL with a warning instead of failing the logger

A cap or TTL of 0 or less raised at logger init. The proxy logs that as a non-blocking error and keeps serving, so the result was a running proxy with no Prometheus metrics at all. The setting is now ignored with a startup warning naming it, the same rule the end_user cap already follows for a non-positive value

* fix(prometheus): start the series cap over on a one-worker restart and audit it live

A proxy with one worker and an operator-set PROMETHEUS_MULTIPROC_DIR now drops litellm's admission files at boot, so a restart frees every slot there the way it already does with several workers. A cap or TTL that is not a number greater than 0 (a bool, a non-numeric string, an empty value) is ignored with the startup warning instead of breaking the logger

The integration cells drive the cap on every endpoint through the OpenAI and Anthropic SDKs and raw httpx, streaming and not, plus gauges, cache hits, failures, both workers of one instance, the TTL on one worker and its warning on two, ignored settings, excluded labels on the fallback counters, a null cap, /config/update, a concurrent burst scraped mid-flight, a provider outage, a killed worker, and restarts with one and two workers

* fix(prometheus): wipe an operator-set multiprocess directory on a one-worker boot too

* fix(prometheus): leave the multiprocess directory alone on a setup-only run

A run with --skip_server_startup starts no worker, so it no longer creates or
wipes PROMETHEUS_MULTIPROC_DIR. Wiping there deleted the samples of a proxy
already running against the same directory

* fix(prometheus): free the capped series slots when a gateway or backend container restarts

The component image entrypoint starts uvicorn without the proxy CLI and wiped only the .db sample files at container start, so the admitted-series files of the previous container survived an in-place restart. Every label set seen after the restart was then counted on `other` once the previous container had filled the cap

* test(prometheus): cover a setup-only run and a gateway image restart under the cap

Two integration cells from the audit: a `--skip_server_startup` run pointed at a live
two-worker proxy's operator directory leaves its samples alone, and the gateway image
(`docker/component_entrypoint.sh` running `python -m gateway.launch`) restarted on a kept
PROMETHEUS_MULTIPROC_DIR starts the cap over. The burst cells now wait for every counter
they assert on, since the request and failure counters of one call increment at different
points of the logging callback

* test(prometheus): prove the cap reaches the fallback counters in the X1 cell

* fix(prometheus): ignore a cleanup interval that is not a number of at least 0

A string or negative prometheus_metrics_cleanup_interval_seconds reached the
series tracker unvalidated, so the first labeled emit with a TTL on raised
TypeError inside the callback and recorded no series. The interval is now
validated the way the cap and the TTL are: an invalid value is ignored with a
warning and the default 60 seconds applies. The I2 integration cell drives a
string interval through a live proxy and reads the warning from its log

* test(prometheus): give the restart cells the boot budget of their siblings

C4 and C5 boot two proxies each and hit the file's 240 s budget on a loaded
box; C3 and D1 already carry 420 s

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-06 01:28:44 +00:00
..
a2a_protocol refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
anthropic_interface fix(proxy): send a real error event when a /v1/messages stream fails (#41826) 2026-09-25 15:36:35 -07:00
batches feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
caching fix(caching): key response cache by router model group in litellm_metadata (#44542) 2026-10-05 17:06:21 +00:00
chat_completions feat(rust): connect Python inference bindings to shared routes (#43465) 2026-09-27 19:05:39 -07:00
completion_extras fix(responses): honor request cache controls on chat completions bridged to the Responses API (#44676) 2026-10-05 17:52:44 -07:00
compression
containers test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
decisions feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers (#44236) 2026-10-03 17:38:38 +00:00
embeddings test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
endpoints
enterprise feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
expected_fine_tuning_api test: move tests/test_litellm/llms into tests/unit/llms (#43191) 2026-09-25 12:43:23 -07:00
expected_responses_api_request test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) 2026-09-25 17:10:13 -07:00
experimental_mcp_client fix(mcp): preserve upstream tool schemas and parameter headers (#44425) 2026-10-03 15:00:43 -07:00
files refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
fixtures test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
gateway test(ci): fix six CircleCI test regressions on main (#44429) 2026-10-03 20:55:01 -07:00
google_genai fix(google_genai): preserve proxy_server_request in completion adapter (#43536) 2026-09-28 21:29:55 -07:00
harness feat(harness): add Harness.TOOL_LOOP, a minimal in-process tool-calling loop (#44391) 2026-10-03 17:33:01 +00:00
images fix(params): keep _litellm_* kwargs out of provider request bodies by construction (#43221) 2026-09-26 15:20:09 -07:00
integration_support fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses (#43560) 2026-10-05 22:36:25 +00:00
integrations feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
interactions feat(interactions): durable cross-pod settlement for background interaction billing (#41955) 2026-10-02 21:08:38 -07:00
litellm_core_utils fix(logging): deduplicate streaming failure callbacks (#44442) 2026-10-04 23:13:43 -07:00
litellm_proxy_extras fix(proxy-extras): log v1 migration failures at ERROR so LITELLM_LOG=ERROR shows them (#44202) 2026-10-03 22:45:16 -07:00
llms feat: add Reka as an OpenAI-compatible provider (#44278) 2026-10-05 17:47:23 -07:00
messages refactor(anthropic): rename experimental_pass_through to pass_through (#43329) 2026-09-26 13:00:50 -07:00
models feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
ocr refactor(ocr): remove the Python OCR execution path and require the Rust route (#43081) 2026-09-24 18:18:50 -07:00
passthrough test: repair stale and polluting tests red on scheduled main CI (#44229) 2026-10-02 21:32:17 +00:00
proxy feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
rag refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
realtime_api fix(openai/realtime): drop model from upstream URL for intent=transcription (#43854) 2026-10-05 19:14:56 +00:00
repositories refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
rerank_api test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
responses fix(responses): report truncated bridged output as incomplete and echo request params (#44460) 2026-10-05 11:27:12 -07:00
router_strategy refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
router_utils fix(router): retry a /v1/messages stream the provider drops before the first content chunk (#44276) 2026-10-05 22:53:39 +00:00
rust_bridge test(rust_bridge): expect PartRow start_time and end_time as required LENS_CONTENT fields (#44719) 2026-10-06 00:14:41 +00:00
sandbox
secret_managers refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
skills ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests (#42903) 2026-09-24 22:59:11 +00:00
test_router fix(router): retry a /v1/messages stream the provider drops before the first content chunk (#44276) 2026-10-05 22:53:39 +00:00
tracing fix(tracing): unify ClickHouse storage configuration (#43941) 2026-10-02 16:31:26 +00:00
types feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
vector_stores refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
videos
__init__.py
AGENTS.md test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
conftest.py test(ci): repair stale tests and flaky CI infrastructure (#43983) 2026-10-01 17:46:43 +00:00
test_a2a_registry_lookup.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_acompletion_session_reuse_e2e.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_add_deployment_no_master_key.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_aembedding_session_reuse_e2e.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_anthropic_beta_headers_filtering.py fix(anthropic): forward the dangerous-tool-use beta to Azure AI Foundry (#43934) 2026-09-30 16:40:11 -07:00
test_anthropic_skills_transformation.py feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
test_assert_ci_coverage.py ci: run unit selections from GHA test-path and drop the CircleCI unit jobs (#44461) 2026-10-05 10:49:16 -07:00
test_assert_workflow_dir_hygiene.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_audio_transcription_rust_bridge.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_auto_update_price_and_context_window_file.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_azure_ad_token_credential_resolution.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_azure_ai_grok_4_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_azure_ai_grok_4_6_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_baseten_glm_5_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_batch_completion_models_all_responses.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_bedrock_marengo_embed_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_budget_ratchet_check.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_chat_ui_responses_session.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_check_licenses.py feat(proxy): embed enterprise LiteAdmin MCP in LiteLLM images (#44610) 2026-10-05 15:24:34 -07:00
test_check_mcp_operation_boundary.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_check_migrations_no_data_rewrites.py fix(proxy-extras): build the SpendLogs indexes in the migration job instead of in migrations (#43948) 2026-10-01 14:12:22 -07:00
test_check_py310_typing_imports.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_check_test_quality.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_check_type_discipline.py chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
test_circleci_path_filter.py ci: move Postgres, MCP and Redis suites to CircleCI integration (#44453) 2026-10-05 09:33:14 -07:00
test_circleci_rust_toolchain.py ci: cut CircleCI wall time without loosening test isolation (#43347) 2026-09-26 15:34:53 -07:00
test_claude_fable_5_config.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_claude_opus_4_6_config.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_claude_opus_4_8_config.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_claude_opus_5_config.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_claude_sonnet_5_config.py fix(model_prices): correct Claude Sonnet 5.5 capabilities and provider keys (#43587) 2026-09-28 19:12:31 +00:00
test_cloudflare_workers_ai_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_completion_timeout_resolution.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_component_entrypoint.py feat(prometheus): cap series per metric for every labeled metric (#44420) 2026-10-06 01:28:44 +00:00
test_compression.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_conftest_isolation.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_constants.py test(ci): fix six CircleCI test regressions on main (#44429) 2026-10-03 20:55:01 -07:00
test_container_router.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_cost_calculation_log_level.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_cost_calculator.py fix(cost): bill per-second transcription models outside chat modes (#44458) 2026-10-05 13:36:29 -07:00
test_cost_map_guard.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_count_tokens_public_api.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_dashscope_image_generation.py refactor(types): replace Any with proven types in 137 files (#44478) 2026-10-05 11:01:45 -07:00
test_daybreak_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_deepseek_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_default_branch.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_detect_changes.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_dockerfile_apk_repository.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_dockerfile_bedrock_realtime_extra.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_dockerfile_non_root.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_drop_params_env_var.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_e2e_egress_sentinel.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_eager_tiktoken_load.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_env_key_doc_gate.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_exception_exports.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_exception_header_preservation.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_exception_mapping_request_attribute.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_filter_out_litellm_params.py fix(params): validate stream_chunk_size once, before any provider call (#43222) 2026-09-26 23:01:20 +00:00
test_fireworks_serverless_model_costs.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gate_slot_lock.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gemini_3_1_flash_lite_image_pricing.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gemini_tts_native_audio_pricing.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_get_blog_posts.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_git_hooks.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_5_4_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_5_5_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_image_cost_calculator.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_gpt_realtime_mode.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_groq_streaming_encoding.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_guardrail_exception_status_codes.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_integration_run.py test(integration): let run.py select cells by pytest node id (#44319) 2026-10-02 21:01:57 -07:00
test_internal_context.py fix(otel): nest cache spans under their operation and name service spans by purpose (#44150) 2026-10-03 09:20:27 -07:00
test_lazy_imports.py feat(anthropic): workload identity federation and pluggable identity sources (#44448) 2026-10-03 17:08:30 -07:00
test_lens_dev.py refactor(lens): storage-independent trace reads, shared keyset pager, typed read failures (#44422) 2026-10-04 12:39:47 -07:00
test_lint_workflow_diff_gates.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_litellm_params_reserved_keys.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_logging.py fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294) 2026-09-26 09:25:13 -07:00
test_lowest_latency_zero_tokens.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_main.py revert(responses): revert "fix(responses): keep gpt-5.4/5.5 tool calls on chat and merge bridged tool calls into one choice" (#44295) (#44344) 2026-10-03 06:47:11 +00:00
test_main_module_header.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mistral_medium_3_5_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mistral_small_4_0_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mistral_zai_glm_5_2_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_model_block_unblock.py fix(proxy): return 4xx instead of 500 for missing required params, invalid pagination and unknown ids (#43787) 2026-10-02 22:48:53 -07:00
test_model_cost_aliases.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_model_param_helper.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_model_prices_schema.py fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows (#42951) 2026-09-25 19:12:36 -07:00
test_model_response_normalization.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_muse_spark_1_1_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_muse_spark_1_2_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_muse_spark_1_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_mutation_report.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_nested_drop_params.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_non_chat_routes_open_llm_spans.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_openai_embedding_encoding_format_default.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_openai_service_tier_long_context_pricing.py fix(cost_calculator): bill ultrafast prompts above 272k at the ultrafast long-context rates (#43764) 2026-09-30 07:32:16 -07:00
test_openrouter_gpt_image_model_metadata.py fix(model_prices): registry audit 2026-10-03, add cohere embed v5, grok-imagine-video-1.5-lite and openrouter gpt-image rows (#44376) 2026-10-03 12:53:07 -07:00
test_package_layout.py
test_pre_commit_lint.py ci: cut CircleCI wall time without loosening test isolation (#43347) 2026-09-26 15:34:53 -07:00
test_prisma_generate_if_needed.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_process_helpers.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_project_alias_tracking.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_project_tags_pydantic.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_proxy_auth.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_rag_openai_ingestion.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_rate_limit_error_unification.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_read_rc_version.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_redact_string_in_error_paths.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_redis.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_redis_credential_provider.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_register_model_custom_pricing.py fix(cost_calculator): bill chat per-second pricing once with a new cost_per_second field (#43614) 2026-09-29 11:27:14 -07:00
test_register_model_zero_cost_persistence.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_replicate_model_key_format.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_responses_api_bridge_non_stream.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_responses_id_security.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_responses_streaming_container_ownership.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_retrieve_batch_bedrock_dispatch.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_block_helpers.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_exception_redaction.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_get_settings.py ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) 2026-10-01 13:00:54 -07:00
test_router_google_genai.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_model_cost_isolation.py feat(bedrock): drop lookaround regex patterns from tool schemas for Converse models that reject them (#44138) 2026-10-02 19:47:25 -07:00
test_router_order_fallback.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_per_deployment_num_retries.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_redis_init.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_retry_backoff_headers.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_retry_non_retryable_errors.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_retry_policy_update.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_silent_experiment.py fix(router): keep silent_model out of embedding provider requests (#44064) 2026-10-01 13:06:29 -07:00
test_router_streaming_fallback_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_router_weighted_failover.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_ruff_strict_gate.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_sambanova_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_secret_redaction.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_seed_tracing_fixtures.py feat(tracing)!: return only data from SQL queries (#44609) 2026-10-05 12:11:59 -07:00
test_select_ui_test_scope.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_service_logger.py fix(otel): name postgres service spans by operation and table (#44240) 2026-10-03 09:20:28 -07:00
test_setup_wizard.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_shared_session_integration.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_socket_policy.py
test_ssl_verify_unit.py test(unit): stop test modules from putting their own directory on sys.path (#43421) 2026-09-28 12:20:20 -07:00
test_stream_chunk_builder_annotations.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_stream_chunk_builder_citations.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_stream_chunk_builder_images.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_streaming_connection_cleanup.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_sync_together_ai_models.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_system_message_format_bug.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_test_quality_gate.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_thinking_enabled.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_together_ai_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_tool_loop.py feat(sdk): add run_tool_loop and arun_tool_loop helpers (#44381) 2026-10-03 20:10:30 -07:00
test_type_check_gate.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_type_discipline_gate.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_typesafe_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_unit_shard_missing_paths.py ci: run unit selections from GHA test-path and drop the CircleCI unit jobs (#44461) 2026-10-05 10:49:16 -07:00
test_unit_shard_per_test_timeout.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_utils.py perf(proxy): stop prompt-cache eligibility from tokenizing the whole conversation (#44221) 2026-10-03 09:20:28 -07:00
test_utils_module_docstring.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_uuid_helper.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_vcr_safe_body_matcher.py fix(tests): stop VCR recording and replaying a test's own localhost upstream (#43346) 2026-09-26 15:49:39 -07:00
test_vertex_ai_xai_grok_prompt_caching_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_video_generation.py fix(proxy): return 4xx instead of 500 for missing required params, invalid pagination and unknown ids (#43787) 2026-10-02 22:48:53 -07:00
test_with_dashboard_node.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_xai_grok_4_3_model_metadata.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00
test_xai_responses_auto_routing.py test: move tests/test_litellm root and small trees into tests/unit (#43186) 2026-09-25 11:30:43 -07:00