5 commits

Author	SHA1	Message	Date
Krish Dholakia	9bfd3e4819	fix(router.py): write file to all deployments (#10708) ... * fix(router.py): write file to all deployments allows unified file id to work across multiple deployments * fix(view_logs/index.tsx): show call type in request logs * fix(router.py): pass a deep copy of kwargs to avoid conflict across multiple runs * fix(batch_utils.py): broaden check * fix(router_utils.py): handle null type for function name * fix(proxy_track_cost_callback.py): fix ruff check error * fix(router.py): handle healthy_deployments as a dict * feat(managed_files.py): support encoding / decoding unified batch id … (#10711) * feat(managed_files.py): support encoding / decoding unified batch id when using managed files allows routing retrieve batch to the right model id * fix: fix linting error * test: add unit tests * fix: fix ruff check	2025-05-10 00:08:30 -07:00
Krish Dholakia	b8b78f1fde	Support unified file id (managed files) for batches (#10650) ... * refactor(managed_files.py): move enterprise feature into enterprise folder prevent unexpected surprises * refactor: safely handle enterprise hooks * fix: fix ruff check errors * fix(files_endpoints.py): cleanup enterprise code from OSS * refactor: complete cleanup * fix(managed_files.py): complete cleanup * fix(managed_files.py): instrument to be able to update deployment values post-router selection and just before making llm call * fix(managed_files.py): instrument to be able to update deployment values post-router selection and just before making llm call * fix: fix linting error * fix: fix linting error	2025-05-07 23:39:40 -07:00
Krish Dholakia	552b7e4013	Add customer + model per key level multi-instance tpm/rpm limiting (#10518) ... * fix(redis_cache.py): handle multiple event loops * fix(parallel_request_limiter_v2.py): add customer tpm limiting * fix(parallel_request_limiter.py): add customer rpm limiting * fix(parallel_request_limiter_v2.py): add model per key + customer tpm/rpm limiting * fix(parallel_request_limiter_v2.py): make error more informative * fix: fix ruff error * fix: generate new poetry lock	2025-05-03 10:28:55 -07:00
Krish Dholakia	132bdb1380	Add user + team based multi-instance rate limiting (#10497) ... * fix(parallel_request_limiter_v2.py): add user multi-instance rate limiting * fix(parallel_request_limiter_v2.py): add user multi-instance rpm limiting * fix(parallel_request_limiter_v2.py): add team based multi-instance rate limiting	2025-05-01 22:09:26 -07:00
Krish Dholakia	711601e22a	Add key-level multi-instance tpm/rpm/max parallel request limiting (#10458) ... * fix: initial commit of v2 parallel request limiter hook enables multi-instance rate limiting to work * fix: subsequent commit with additional refactors * fix(parallel_request_limiter_v2.py): cleanup initial call hook simplify it * fix(parallel_request_limiter_v2.py): working v2 parallel request limiter * fix: more updates - still not passing testing * fix(test_parallel_request_limiter_v2.py): update test + add conftest * fix: fix ruff checks * fix(parallel_request_limiter_v2.py): use pull via pattern method to load in keys instance wouldn't have seen yet Fixes issue where redis syncing was not pulling key until instance had seen it * test: update testing to cover tpm and rpm * fix(parallel_request_limiter_v2.py): fix ruff errors * fix(proxy/hooks/__init__.py): feature flag export * fix(proxy/hooks/__init_.py): fix linting error * ci(config.yml): add tests/enterprise to ci/cd * fix: fix ruff check * test: update testing	2025-04-30 21:32:31 -07:00