scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-17 23:51:30 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 66
litellm/litellm
History
Krish Dholakia 711601e22a
Add key-level multi-instance tpm/rpm/max parallel request limiting (#10458) 
* fix: initial commit of v2 parallel request limiter hook

enables multi-instance rate limiting to work

* fix: subsequent commit with additional refactors

* fix(parallel_request_limiter_v2.py): cleanup initial call hook

simplify it

* fix(parallel_request_limiter_v2.py): working v2 parallel request limiter

* fix: more updates - still not passing testing

* fix(test_parallel_request_limiter_v2.py): update test + add conftest

* fix: fix ruff checks

* fix(parallel_request_limiter_v2.py): use pull via pattern method to load in keys instance wouldn't have seen yet

Fixes issue where redis syncing was not pulling key until instance had seen it

* test: update testing to cover tpm and rpm

* fix(parallel_request_limiter_v2.py): fix ruff errors

* fix(proxy/hooks/__init__.py): feature flag export

* fix(proxy/hooks/__init_.py): fix linting error

* ci(config.yml): add tests/enterprise to ci/cd

* fix: fix ruff check

* test: update testing
2025-04-30 21:32:31 -07:00
..
anthropic_interface docs anthropic messages interface 2025-03-31 16:53:39 -07:00
assistants Contributor PR - Support OPENAI_BASE_URL in addition to OPENAI_API_BASE (#9995) (#10423) 2025-04-29 21:27:37 -07:00
batch_completion (code quality) run ruff rule to ban unused imports (#7313) 2024-12-19 12:33:42 -08:00
batches Contributor PR - Support OPENAI_BASE_URL in addition to OPENAI_API_BASE (#9995) (#10423) 2025-04-29 21:27:37 -07:00
caching Add key-level multi-instance tpm/rpm/max parallel request limiting (#10458) 2025-04-30 21:32:31 -07:00
experimental_mcp_client fix types on tools.py 2025-03-29 16:48:15 -07:00
files Contributor PR - Support OPENAI_BASE_URL in addition to OPENAI_API_BASE (#9995) (#10423) 2025-04-29 21:27:37 -07:00
fine_tuning Contributor PR - Support OPENAI_BASE_URL in addition to OPENAI_API_BASE (#9995) (#10423) 2025-04-29 21:27:37 -07:00
integrations fix(langsmith.py): respect langsmith batch size param (#10411) 2025-04-29 18:55:22 -07:00
litellm_core_utils Schedule budget resets at expectable times (#10331) (#10333) 2025-04-29 20:59:44 -07:00
llms Contributor PR - Support OPENAI_BASE_URL in addition to OPENAI_API_BASE (#9995) (#10423) 2025-04-29 21:27:37 -07:00
proxy Add key-level multi-instance tpm/rpm/max parallel request limiting (#10458) 2025-04-30 21:32:31 -07:00
realtime_api Realtime API: Support 'base_model' cost tracking + show response in spend logs (if enabled) (#9897) 2025-04-10 21:24:45 -07:00
rerank_api build(pyproject.toml): add new dev dependencies - for type checking (#9631) 2025-03-29 11:02:13 -07:00
responses [Bug Fix] Responses API - fix for handling multiturn responses API sessions (#10415) 2025-04-29 17:22:47 -07:00
router_strategy Add key-level multi-instance tpm/rpm/max parallel request limiting (#10458) 2025-04-30 21:32:31 -07:00
router_utils [Feat] Add Support for DELETE /v1/responses/{response_id} on OpenAI, Azure OpenAI (#10205) 2025-04-22 18:27:03 -07:00
secret_managers Squashed commit of the following: (#9709) 2025-04-02 21:24:54 -07:00
types [Feat] Add support for using Bedrock Knowledge Bases with LiteLLM /chat/completions requests (#10413) 2025-04-29 17:29:02 -07:00
__init__.py [Feat] Add support for using Bedrock Knowledge Bases with LiteLLM /chat/completions requests (#10413) 2025-04-29 17:29:02 -07:00
_logging.py fix(logging): add json formatting for uncaught exceptions (#9615) (#9619) 2025-03-28 15:16:15 -07:00
_redis.py Squashed commit of the following: (#9709) 2025-04-02 21:24:54 -07:00
_service_logger.py clean up service metrics 2025-04-02 17:50:41 -07:00
_version.py Virtual key based policies in Aim Guardrails (#9499) 2025-04-01 21:57:23 -07:00
budget_manager.py Squashed commit of the following: (#9709) 2025-04-02 21:24:54 -07:00
constants.py Update fireworks ai pricing (#10425) 2025-04-29 20:58:05 -07:00
cost.json store llm costs in budget manager 2023-09-09 19:11:35 -07:00
cost_calculator.py [Feat] Add logging callback support for /moderations API (#10390) 2025-04-28 20:24:17 -07:00
exceptions.py feat(openai.py): bubble all error information back to client 2025-03-10 15:27:43 -07:00
main.py Contributor PR - Support OPENAI_BASE_URL in addition to OPENAI_API_BASE (#9995) (#10423) 2025-04-29 21:27:37 -07:00
model_prices_and_context_window_backup.json Update fireworks ai pricing (#10425) 2025-04-29 20:58:05 -07:00
py.typed feature - Types for mypy - #360 2024-05-30 14:14:41 -04:00
router.py [Feat] Add GET, DELETE Responses endpoints on LiteLLM Proxy (#10297) 2025-04-24 17:34:26 -07:00
scheduler.py Squashed commit of the following: (#9709) 2025-04-02 21:24:54 -07:00
timeout.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
utils.py Embedding caching fixes - handle str -> list cache, set usage tokens for cache hits, combine usage tokens on partial cache hits (#10424) 2025-04-29 21:21:28 -07:00