mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-17 04:34:25 +00:00
7da34e8aed
Per-model budgets were three separate things pretending to be one. The enforcement check, the post-call increment and the info endpoints each derived their own cache key, so a budget could refuse traffic at 429 while /key/info reported zero usage, and a Bedrock model id never matched a budget keyed on the bare family name. /user/new echoed a model_max_budget back and stored an empty dict, and nothing enforced a user-scoped per-model budget at all. One owner now builds the counter key from the configured budget model, and enforcement, the increment and the info endpoints all read it. Bedrock ids resolve through the model-cost map. Auth carries the user's budget onto the token on every branch that reaches the spend hook, including JWT and auto-registration. Native passthrough attaches the three budget metadata keys its StandardLoggingUserAPIKeyMetadata does not carry, so /anthropic/... and /bedrock/... traffic is counted and capped like /v1/chat/completions. The dashboard gains the per-model budget editor it never had, on the key create, key edit and internal-user edit forms. It is read-only without an enterprise license, matching the write gate the proxy already enforces, and an untouched budget is left out of an update so an unrelated edit cannot trip that gate. The editor hydrates from either BudgetConfig spelling, since model_max_budget is a plain dict that the proxy stores exactly as the client sent it, and it carries through the fields it does not model. Without both, editing one model would drop another model row entirely and silently discard its tpm_limit and rpm_limit. /user/info refreshes its local copy of the user field by field after a save, so model_max_budget joins that list. Left out, a saved cap read back as the old one when the form was reopened, and clearing the row to recover would then wipe the value that had actually persisted. A zero-dollar cap is the strictest limit expressible, not the absence of one, so it is enforced rather than skipped on falsiness, spend exactly at the cap is refused the way every sibling budget check already refuses it, and a counter that was never written reads as zero spend rather than as unknown. The usage endpoints read every counter in one batched lookup, so a large model_max_budget cannot fan out into one concurrent cache call per configured model. Every auth path honours the same zero-cost skip flag, so none of them can refuse a free request that another serves. The custom-auth helper gains the flag it never had, which also changes its pre-existing key and end-user checks. The compaction summary gate checks the user scope alongside the key and end-user ones. This file propagates all three budgets into the summary subrequest, so enforcing only two let compaction increment a counter it could not be refused by. Custom auth attaches the user's budget to the token unconditionally, since the post-call spend hook reads it there: gating the attach on the same condition as enforcement left the counter uncharged whenever the request was not itself enforceable. An entry that will not validate is skipped rather than raised on, so one malformed scope cannot abort every other scope's increment or turn a config typo into a 500. The edit forms re-seed the budget editor when a different key or user is loaded. Its rows are seeded once and cannot re-read their own value prop, so without this a save wrote the previously loaded record's budgets onto the current one. Only the built-in provider pass-through routes carry the budget metadata. get_model_from_request deliberately resolves no model for a user-defined pass-through, since its body is forwarded verbatim and names an upstream model, so attaching there would charge a counter nothing on that route can refuse. |
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| base_sdk_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| integration | ||
| litellm/proxy/_experimental/mcp_server | ||
| litellm-proxy-extras | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_behavior | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_migration_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _openai_record_replay_proxy.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_anthropic_compaction_usage.py | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.