scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9
litellm/tests
History
ryan-crabbe-berri fe5cc1eb0c
fix(proxy): global max_budget ignores budget_duration; enforce against the resettable proxy budget row (#33732) 
* fix(proxy): enforce global max_budget against the resettable proxy budget row

The global proxy budget check compared litellm.max_budget against
SUM(spend) from the MonthlyGlobalSpend view, whose window is hardcoded
to a trailing 30 days. litellm.budget_duration was stored and reset on
a user row that enforcement never read, and startup budgeted the admin
user's own row (default_user_id) instead of the litellm-proxy-budget
aggregate row the spend writer increments per request. Net effect: 1d,
7d and 30d all behaved as a trailing 30 day cap that never reset on the
configured duration.

Startup now upserts the budget onto the litellm-proxy-budget row (and
zeroes lifetime accrual when first putting a row on a reset schedule),
enforcement loads global spend from that row, and ResetBudgetJob drops
the cached global spend accumulator when it resets that row so the cap
unblocks immediately after each window.

Fixes https://github.com/BerriAI/litellm/issues/31292

* refactor(proxy): address review nits on global proxy budget fix

Drop the redundant litellm_proxy_budget_name parameter from
_upsert_proxy_budget_with_reset_at_backfill; its only caller always passed
LITELLM_PROXY_BUDGET_NAME, and any other value would write the budget to a
row enforcement never reads.

Introduce GLOBAL_PROXY_SPEND_CACHE_KEY in constants.py and use it at every
site that previously built the key from litellm_proxy_admin_name (auth
loads, spend-writer increments, startup warm, reset-job invalidation), so
the reader and invalidator can no longer drift apart. The literal key value
is unchanged. Also drop the now-pointless litellm_proxy_admin_name
parameter from _warm_global_spend_cache and the proxy_server import from
the reset-job helper.
2026-07-25 11:29:39 -07:00
..
agent_tests feat(a2a): support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents (#30950) 2026-06-29 09:32:39 +05:30
audio_tests fix(tests): stabilize image-edit VCR cassettes to stop live gpt-image-1 spend (#28110) 2026-05-18 09:15:39 -07:00
basic_proxy_startup_tests build: migrate packaging, CI, and Docker from Poetry to uv (#25007) 2026-04-09 11:46:23 -07:00
batches_tests Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
benchmarks test(benchmarks): run shared logging executor inline to make CodSpeed measurements deterministic (#32435) 2026-07-09 11:14:22 +03:00
code_coverage_tests test(e2e): drive a real Linear OAuth MCP through chat completions under both ingress headers 2026-07-22 23:29:10 -07:00
documentation_tests feat(messages): route native Anthropic /messages through Rust behind LITELLM_RUST env var (#33848) 2026-07-18 18:27:20 -07:00
e2e test(e2e): cover /openai chat passthrough cost logging (#34470) 2026-07-25 17:54:28 +00:00
enterprise fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException (#32289) 2026-07-09 13:18:51 -07:00
guardrails_tests feat: add deepkeep as custom guardrail (#33844) 2026-07-20 19:27:40 -07:00
image_gen_tests test(bedrock): switch image gen live test off EOL Titan to Nova Canvas (#31937) 2026-07-01 22:57:29 -07:00
integration feat(sandbox): e2b code execution primitive (#30898) 2026-06-20 16:30:01 -07:00
litellm fix(anthropic): strip all remaining output_format schema keywords rejected by Anthropic 2026-07-22 16:16:04 -07:00
litellm-proxy-extras style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
litellm_core_utils Merge branch 'litellm_internal_staging' into litellm_staging_03_22_2026 2026-04-20 19:56:00 +05:30
litellm_utils_tests feat(budgets): add configurable budget_reset_time of day (#31007) 2026-07-21 13:35:01 -07:00
llm_responses_api_testing Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) 2026-07-14 19:32:25 -07:00
llm_translation Merge pull request #34520 from BerriAI/litellm_/replace-gpt5-codex-test-27520b 2026-07-24 12:03:24 -07:00
load_tests style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
local_testing fix(anthropic): self-heal on missing thinking-signature errors from Bedrock/Vertex (#33719) 2026-07-17 18:18:38 +00:00
logging_callback_tests test(logging): pin compression_savings in the gcs pubsub spend log fixture (#34204) 2026-07-22 00:56:20 +00:00
mcp_tests Merge pull request #33631 from BerriAI/litellm_lit4517_messages_mcp_gateway 2026-07-20 16:22:03 -07:00
multi_instance_e2e_tests
ocr_tests test(ocr): use mistral-document-ai-2512 in azure_ai OCR tests 2026-07-15 18:13:22 -07:00
old_proxy_tests/tests fix: cleanup tests 2026-03-30 16:24:35 -07:00
openai_endpoints_tests chore(ci): modernize model references in tests and configs (#27856) 2026-05-15 15:44:28 -07:00
otel_tests fix(cli): mint per-session agent credential on lite login (#31072) 2026-06-26 09:05:15 -07:00
pass_through_tests test(pass-through): de-flake vertex spend-log test by routing through the proxy (#31689) 2026-06-30 15:27:48 -07:00
pass_through_unit_tests fix(streaming): stamp completion_start_time on first chunk for /v1/messages and /v1/responses (#32284) 2026-07-06 19:30:31 -07:00
proxy_admin_ui_tests Revert "chore: remove _experimental/out (#31546)" 2026-07-01 13:25:47 -07:00
proxy_behavior fix(proxy): resolve team org from team_id so org admins can update team budgets 2026-07-08 19:10:35 -07:00
proxy_e2e_anthropic_messages_tests Extend the record/replay proxy to chat, embeddings, moderations, rerank, and Anthropic (#29847) 2026-06-06 14:33:42 -07:00
proxy_migration_tests fix(docker): bake non_root prisma engines at /opt/prisma so migrations run offline for any uid (#34325) 2026-07-22 23:03:39 -07:00
proxy_security_tests test(proxy): stop running real-DB tests in GitHub Actions unit jobs (#29700) 2026-06-04 14:56:02 -07:00
proxy_unit_tests feat(guardrails): add run_in_parallel opt-in for concurrent pre_call and post_call guardrails (#33770) 2026-07-24 13:25:58 -07:00
router_unit_tests fix(router): tag-aware pre-routing strategy selection for shared model_name (#33691) 2026-07-17 09:26:07 -07:00
scim_tests
search_tests feat(tinyfish): make search provider permissive, attribute errors (#31997) 2026-07-03 10:17:11 -07:00
spend_tracking_tests chore(ci): modernize model references in tests and configs (#27856) 2026-05-15 15:44:28 -07:00
store_model_in_db_tests test: de-flake langfuse callbacks-in-db e2e test 2026-07-04 19:14:53 -07:00
test_litellm fix(proxy): global max_budget ignores budget_duration; enforce against the resettable proxy budget row (#33732) 2026-07-25 11:29:39 -07:00
unified_google_tests fix(google_genai): preserve complete SSE events in Vertex/Gemini image streaming (#30270) 2026-06-12 07:49:30 -07:00
vector_store_tests Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326) 2026-05-30 11:26:24 -07:00
windows_tests ci: reproduce default-Windows wheel install to guard MAX_PATH (#29597) 2026-06-03 11:28:08 -07:00
__init__.py
_fake_openai_endpoint_server.py test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900) 2026-06-20 16:20:35 -07:00
_flush_vcr_cache.py tests(vcr): isolate cassette redis to CASSETTE_REDIS_URL 2026-05-01 12:32:59 -07:00
_live_test_helpers.py test(vcr): close out the remaining VCR live-call leaks (#29603) 2026-06-03 13:46:43 -07:00
_openai_record_replay_proxy.py Extend the record/replay proxy to chat, embeddings, moderations, rerank, and Anthropic (#29847) 2026-06-06 14:33:42 -07:00
_vcr_conftest_common.py test(vcr): close out the remaining VCR live-call leaks (#29603) 2026-06-03 13:46:43 -07:00
_vcr_redis_persister.py test(vcr): stop refreshing cassette TTL on read so cassettes lapse after 24h (#29784) 2026-06-05 10:22:41 -07:00
_ws_vcr.py test(realtime): record and replay websocket traffic in redis vcr cassettes (#32390) 2026-07-08 00:19:06 -07:00
eval_swe_bench.py Prompt Compression - add it to the proxy (#25729) 2026-04-20 15:08:00 -07:00
fake_openai_endpoint.py test: point router/completion/triton tests at the local fake OpenAI endpoint (#30900) 2026-06-20 16:20:35 -07:00
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json test: add e2e tests for spend, budgets and llms (#30869) 2026-06-24 15:01:57 -07:00
README.MD
test_anthropic_compaction_usage.py chore(oss): litellm oss staging 150626 (#30463) 2026-06-16 12:06:41 -07:00
test_budget_management.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_callbacks_on_proxy.py test(callbacks): harden flaky proxy callback-leak detector (#28195) 2026-05-18 16:39:02 -07:00
test_config.py
test_debug_warning.py
test_default_encoding_non_root.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_end_users.py chore(ci): modernize model references in tests and configs (#27856) 2026-05-15 15:44:28 -07:00
test_entrypoint.py
test_fallbacks.py
test_gpt5_azure_temperature_support.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_health.py fix(tests): swap dall-e to gpt-image-1 after openai deprecation 2026-05-12 16:55:18 -07:00
test_keys.py test: harden remaining pass-through CI flakes (image-gen spend poll, ruby assistants timeout) (#30685) 2026-06-17 14:35:47 -07:00
test_litellm_proxy_responses_config.py chore(ci): modernize model references in tests and configs (#27856) 2026-05-15 15:44:28 -07:00
test_logging.conf
test_models.py test: replace test_add_and_delete_models integration test with mock 2026-03-30 21:30:57 -07:00
test_new_vector_store_endpoints.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_openai_endpoints.py Extend the record/replay proxy to chat, embeddings, moderations, rerank, and Anthropic (#29847) 2026-06-06 14:33:42 -07:00
test_organizations.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_otel_thread_leak.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_passthrough_endpoints.py
test_presidio_latency.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_proxy_server_non_root.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_ratelimit.py chore(ci): modernize model references in tests and configs (#27856) 2026-05-15 15:44:28 -07:00
test_resource_cleanup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_service_logger_otel.py fix(langfuse): send v4 ingestion header for otel callback (#33907) 2026-07-18 20:36:51 -07:00
test_spend_logs.py Litellm oss staging 04 21 2026 2 (#26569) 2026-05-20 21:25:19 -07:00
test_team.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_team_logging.py test: cleanup dead tests 2026-03-28 20:49:02 -07:00
test_team_members.py Litellm oss staging 04 21 2026 2 (#26569) 2026-05-20 21:25:19 -07:00
test_users.py Fix: tag budget reset must drop stale management-cache entry (#27568) 2026-05-10 00:18:55 +00:00

README.MD

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/test_litellm

This folder can only run mock tests.