scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-06 02:48:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103
litellm/litellm
History
ryan-crabbe-berri abe3289398
fix(proxy): retry model cost map fetch with Retry-After-aware backoff and keep current map on reload failure (#35739) 
* fix(proxy): retry model cost map fetch with Retry-After-aware backoff and stop downgrading to the packaged backup on reload failure

A 429 or transient network error during a manual or scheduled model cost map
reload used to silently replace litellm.model_cost with the stale backup JSON
bundled in the installed wheel, stamp the reload as successful, and clear the
force_reload flag, so a fleet could serve months-old pricing until the next
interval. Runtime reloads now go through refetch_model_cost_map, which retries
429/5xx/transport errors up to 3 times honoring Retry-After (capped at 30s,
exponential backoff with jitter otherwise) and returns a failure value instead
of the backup when the fetch or integrity validation fails. On failure the pod
keeps its currently loaded map, the periodic job leaves last_run and
force_reload untouched so it retries on the next config poll, and the manual
endpoint returns 502 with the reason instead of reporting a fake success.
Startup behavior is unchanged: boot still falls back to the packaged backup
since there is no previously loaded map to keep.

* fix(proxy): use shared async httpx client for cost map reload and make retry tests CI-env-proof

The reload fetch now goes through get_async_httpx_client with a dedicated
httpxSpecialProvider.ModelCostMap pool instead of constructing a raw
httpx.AsyncClient, so it inherits deployment-level TLS and transport settings
and passes the ensure_async_clients gate. Tests inject a MockTransport-backed
client through the same seam. An autouse fixture clears
LITELLM_LOCAL_MODEL_COST_MAP, which CI exports and which short-circuited the
retry tests; the two TestPriceDataReloadAPI tests and the config sync pubsub
reload test that still patched get_model_cost_map now patch
refetch_model_cost_map instead.
2026-08-03 22:08:58 -07:00
..
a2a_protocol perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
anthropic_interface refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
assistants refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
batch_completion refactor(logging): drop redundant !s conversion flags from f-strings 2026-08-01 16:59:05 -07:00
batches perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
caching perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
completion_extras perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
compression refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
containers refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
endpoints/speech/speech_to_completion_bridge refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
evals refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
experimental_mcp_client perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
files refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
fine_tuning refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
google_genai perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
images refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
integrations perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
interactions perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
litellm_core_utils fix(proxy): retry model cost map fetch with Retry-After-aware backoff and keep current map on reload failure (#35739) 2026-08-03 22:08:58 -07:00
llms perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
models refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
ocr perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
passthrough refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
proxy fix(proxy): retry model cost map fetch with Retry-After-aware backoff and keep current map on reload failure (#35739) 2026-08-03 22:08:58 -07:00
proxy_auth refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
rag perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
realtime_api refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
repositories perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
rerank_api perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
responses perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
router_strategy perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
router_utils perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
rust_bridge refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
sandbox perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
search perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
secret_managers perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
skills refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
types fix(proxy): retry model cost map fetch with Retry-After-aware backoff and keep current map on reload failure (#35739) 2026-08-03 22:08:58 -07:00
vector_store_files refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
vector_stores perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
videos refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
__init__.py feat(spend): add net auto-router savings to the cost-optimization dashboard (#35521) 2026-08-04 03:10:37 +00:00
_internal_context.py fix(proxy): harden request parameter handling 2026-04-16 01:38:12 +00:00
_lazy_imports.py fix(lint): restore group-header comments the import sort displaced in _lazy_imports.py 2026-08-01 16:49:25 -07:00
_lazy_imports_registry.py feat(tencent): add Tencent TokenHub as a provider (#31903) 2026-07-02 18:31:59 -07:00
_logging.py refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
_redis.py perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
_redis_credential_provider.py refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
_service_logger.py refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
_uuid.py build(deps-dev): bump black to 26.3.1 and apply formatting (#28525) 2026-05-21 17:24:18 -07:00
_version.py
anthropic_beta_headers_config.json fix(bedrock-invoke): retain clear_tool_uses_20250919 context_management edits and emit context-management-2025-06-27 beta (LIT-3393) (#32658) 2026-07-09 14:31:27 -07:00
anthropic_beta_headers_manager.py perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
blog_posts.json docs(blog): add WebRTC blog post link 2026-03-13 17:54:33 +05:30
budget_manager.py refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
constants.py fix(caching): close evicted LLM clients so their connections are reclaimed (#35492) 2026-08-03 13:28:38 -07:00
cost.json
cost_calculator.py perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
exceptions.py refactor(logging): drop redundant !s conversion flags from f-strings 2026-08-01 16:59:05 -07:00
main.py perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
model_prices_and_context_window_backup.json fix(bedrock): stop forwarding no-op toolSpec.strict to Converse (#35688) 2026-08-03 18:58:40 -07:00
policy_templates_backup.json feat(add-new-block_code_execution-guardrail): prevent agent from executing code (#22154) 2026-02-25 22:02:14 -08:00
provider_endpoints_support_backup.json chore: litellm oss staging (#30968) 2026-06-23 07:31:44 -07:00
py.typed
router.py perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
scheduler.py refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
setup_wizard.py refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets 2026-08-01 15:43:29 -07:00
timeout.py style: unify ruff format width on 120 (#31518) 2026-06-27 12:39:29 -07:00
utils.py perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00