Commit graph

148 commits

Author SHA1 Message Date
Jugal D. Bhatt
a754a25828
Feat add retention config (#10815)
* add function to check config flag

* added unit tests

* convert to seconds support

* added in settings.md

* Updated config_settings.md

* remove extra point

* change config var

* resolve conflict
2025-05-14 18:16:25 -07:00
Ishaan Jaff
3130c4f8f9
[Refactor] Move LLM Guard, Secret Detection to Enterprise Pip packagea (#10782)
* refactor: move guardrails to pip

* refactor: move guardrails to pip

* testing fix: move guardrails to pip

* git commit setup_litellm_enterprise_pip
2025-05-13 09:42:22 -07:00
Krish Dholakia
a1964eab18
Realtime API - Set 'headers' in scope for websocket auth requests + reliability fix infinite loop when model_name not found for realtime models (#10679)
* fix(user_api_key_auth.py): add 'headers' to constructed request for websocket

Fix issue on some datastructure versions which require a headers field in scope

* test(test_user_api_key_auth.py): add unit testing for headers in scope change

* fix(router.py): migrate `_arealtime` to generic router endpoint

Fix infinite loop on model name missing for realtime api calls

* test(test_router_helper_utils.py): cleanup test post refactor
2025-05-08 22:50:09 -07:00
Ishaan Jaff
c67ca115ae
Return 404 when delete_verification_tokens (POST /key/delete) fails to find keys (#10604) (#10605)
* Return 404 when delete_verification_tokens fails to find keys

* Add test_delete_nonexistent_key_returns_404

Co-authored-by: Marc Abramowitz <abramowi@adobe.com>
2025-05-06 20:45:15 -07:00
Ishaan Jaff
299f8f18a0 text fix redis cache 2025-05-03 16:28:43 -07:00
Prathamesh Saraf
ac7b1efe5c
Refactor budget assertions in tests to improve clarity and accuracy. Updated remaining hours check to ensure positive values and adjusted budget reset time validation for better range checks. (#10500) 2025-05-02 09:02:02 -07:00
Krish Dholakia
711601e22a
Add key-level multi-instance tpm/rpm/max parallel request limiting (#10458)
* fix: initial commit of v2 parallel request limiter hook

enables multi-instance rate limiting to work

* fix: subsequent commit with additional refactors

* fix(parallel_request_limiter_v2.py): cleanup initial call hook

simplify it

* fix(parallel_request_limiter_v2.py): working v2 parallel request limiter

* fix: more updates - still not passing testing

* fix(test_parallel_request_limiter_v2.py): update test + add conftest

* fix: fix ruff checks

* fix(parallel_request_limiter_v2.py): use pull via pattern method to load in keys instance wouldn't have seen yet

Fixes issue where redis syncing was not pulling key until instance had seen it

* test: update testing to cover tpm and rpm

* fix(parallel_request_limiter_v2.py): fix ruff errors

* fix(proxy/hooks/__init__.py): feature flag export

* fix(proxy/hooks/__init_.py): fix linting error

* ci(config.yml): add tests/enterprise to ci/cd

* fix: fix ruff check

* test: update testing
2025-04-30 21:32:31 -07:00
Krish Dholakia
290e2528cd
Schedule budget resets at expectable times (#10331) (#10333)
* Schedule budget resets at expectable times (#10331)

* Enhance budget reset functionality with timezone support and standardized reset times

- Added `get_next_standardized_reset_time` function to calculate budget reset times based on specified durations and timezones.
- Introduced `timezone_utils.py` to manage timezone retrieval and budget reset time calculations.
- Updated budget reset logic in `reset_budget_job.py`, `internal_user_endpoints.py`, `key_management_endpoints.py`, and `team_endpoints.py` to utilize the new timezone-aware reset time calculations.
- Added unit tests for the new reset time functionality in `test_duration_parser.py`.
- Updated `.gitignore` to include `test.py` and made minor formatting adjustments in `docker-compose.yml` for consistency.

* Fixed linting

* Fix for mypy

* Fixed testcase for reset

* fix(duration_parser.py): move off zoneinfo - doesn't work with python 3.8

* test: update test

* refactor: improve budget reset time calculation and update related tests for accuracy

* clean up imports in team_endpoints.py

* test: update budget remaining hours assertions to reflect new reset time logic

* build(model_prices_and_context_window.json): update model

---------

Co-authored-by: Prathamesh Saraf <pratamesh1867@gmail.com>
2025-04-29 20:59:44 -07:00
Krish Dholakia
839878f4f5
Support x-litellm-api-key header param + allow key at max budget to call non-llm api endpoints (#10392)
* fix(user_api_key_auth.py): fix passing `x-litellm-api-key` to user api key auth

Support using this when given, or bearer token when given

 Fixes issue with auth on vertex passthrough

* test(test_user_api_key_auth.py): use new fastapi.security check

* fix(user_api_key_auth.py): allow key at budget, to still call non-llm api endpoints

Fixes issue where key at budget, couldn't call `/key/info`
2025-04-29 18:57:57 -07:00
Ishaan Jaff
f984089b01
[Reliability fix] Redis transaction buffer - ensure all redis queues are periodically flushed (#10393)
* test_e2e_size_of_redis_buffer

* fix store_in_memory_spend_updates_in_redis

* _commit_spend_updates_to_db_with_redis

* daily_tag_spend_update_transactions

* pip install fakeredis==2.28.1
2025-04-28 21:36:54 -07:00
Ishaan Jaff
8ae2653280 fix calculated cache key for tests 2025-04-19 09:25:11 -07:00
Krish Dholakia
1ea046cc61
test: update tests to new deployment model (#10142)
* test: update tests to new deployment model

* test: update model name

* test: skip cohere rbac issue test

* test: update test - replace gpt-4o model
2025-04-18 14:22:12 -07:00
Ishaan Jaff
94a553dbb2
[Feat] Emit Key, Team Budget metrics on a cron job schedule (#9528)
* _initialize_remaining_budget_metrics

* initialize_budget_metrics_cron_job

* initialize_budget_metrics_cron_job

* initialize_budget_metrics_cron_job

* test_initialize_budget_metrics_cron_job

* LITELLM_PROXY_ADMIN_NAME

* fix code qa checks

* test_initialize_budget_metrics_cron_job

* test_initialize_budget_metrics_cron_job

* pod lock manager allow dynamic cron job ID

* fix pod lock manager

* require cronjobid for PodLockManager

* fix DB_SPEND_UPDATE_JOB_NAME acquire / release lock

* add comment on prometheus logger

* add debug statements for emitting key, team budget metrics

* test_pod_lock_manager.py

* test_initialize_budget_metrics_cron_job

* initialize_budget_metrics_cron_job

* initialize_remaining_budget_metrics

* remove outdated test
2025-04-10 16:59:14 -07:00
Krish Dholakia
0d503ad8ad
Move daily user transaction logging outside of 'disable_spend_logs' flag - different tables (#9772)
* refactor(db_spend_update_writer.py): aggregate table is entirely different

* test(test_db_spend_update_writer.py): add unit test to ensure if disable_spend_logs is true daily user transactions is still logged

* test: fix test
2025-04-05 09:58:16 -07:00
Ishaan Jaff
a64631edfb test pod lock manager 2025-04-02 14:39:40 -07:00
Ishaan Jaff
8dc792139e refactor file structure 2025-04-01 18:30:48 -07:00
Ishaan Jaff
290e837515 test_update_logs_with_spend_logs_url 2025-04-01 18:15:01 -07:00
Ishaan Jaff
4ddca7a79c
Merge branch 'main' into litellm_fix_service_account_behavior 2025-04-01 12:04:28 -07:00
Ishaan Jaff
c2c5dbf24f test_get_enforced_params 2025-04-01 08:41:53 -07:00
Ishaan Jaff
13aa7f75f6 test_enforced_params_check 2025-04-01 07:40:31 -07:00
Ishaan Jaff
55763ae276 test_end_user_transactions_reset 2025-04-01 07:13:25 -07:00
Ishaan Jaff
923ac2303b test_end_user_transactions_reset 2025-03-31 20:55:13 -07:00
Ishaan Jaff
aa8261af89 test fixes 2025-03-31 19:33:10 -07:00
Ishaan Jaff
a753fc9d9f test_long_term_spend_accuracy_with_bursts 2025-03-31 19:17:13 -07:00
Ishaan Jaff
ce5f55d04e test fix update spend 2025-03-31 14:20:47 -07:00
Krish Dholakia
5c107c64dd
Add gemini audio input support + handle special tokens in sagemaker response (#9640)
* fix(internal_user_endpoints.py): cleanup unused variables on beta endpoint

no team/org split on daily user endpoint

* build(model_prices_and_context_window.json): gemini-2.0-flash supports audio input

* feat(gemini/transformation.py): support passing audio input to gemini

* test: fix test

* fix(gemini/transformation.py): support audio input as a url

enables passing google cloud bucket urls

* fix(gemini/transformation.py): support explicitly passing format of file

* fix(gemini/transformation.py): expand support for inferred file types from url

* fix(sagemaker/completion/transformation.py): fix special token error when counting sagemaker tokens

* test: fix import
2025-03-29 19:23:09 -07:00
Krish Dholakia
1604f87663
install prisma migration files - connects litellm proxy to litellm's prisma migration files (#9637)
* build(README.md): initial commit adding a separate folder for additional proxy files. Meant to reduce size of core package

* build(litellm-proxy-extras/): new pip package for storing migration files

allows litellm proxy to use migration files, without adding them to core repo

* build(litellm-proxy-extras/): cleanup pyproject.toml

* build: move prisma migration files inside new proxy extras package

* build(run_migration.py): update script to write to correct folder

* build(proxy_cli.py): load in migration files from litellm-proxy-extras

Closes https://github.com/BerriAI/litellm/issues/9558

* build: add MIT license to litellm-proxy-extras

* test: update test

* fix: fix schema

* bump: version 0.1.0 → 0.1.1

* build(publish-proxy-extras.sh): add script for publishing new proxy-extras version

* build(liccheck.ini): add litellm-proxy-extras to authorized packages

* fix(litellm-proxy-extras/utils.py): move prisma migrate logic inside extra proxy pkg

easier since migrations folder already there

* build(pre-commit-config.yaml): add litellm_proxy_extras to ci tests

* docs(config_settings.md): document new env var

* build(pyproject.toml): bump relevant files when litellm-proxy-extras version changed

* build(pre-commit-config.yaml): run poetry check on litellm-proxy-extras as well
2025-03-29 15:27:09 -07:00
Krrish Dholakia
217e8d7d44 test: make script to run clearer 2025-03-29 08:23:18 -07:00
Ishaan Jaff
7e8a02099c Merge branch 'main' into litellm_use_redis_for_updates 2025-03-28 20:12:29 -07:00
Ishaan Jaff
1eaf847f8a test pod lock manager 2025-03-28 13:31:45 -07:00
Ishaan Jaff
758182fc7f fix typo on codebase 2025-03-27 22:36:00 -07:00
Krrish Dholakia
79175ddb53 test: fix test 2025-03-27 22:04:59 -07:00
Krrish Dholakia
b3c7785240 test: skip flaky test - failing due to db timeouts - unrelated to test 2025-03-27 20:34:26 -07:00
Krrish Dholakia
e2d4597588 test: mark flaky test 2025-03-27 20:10:57 -07:00
Ishaan Jaff
21e3b764f5 use DBSpendUpdateWriter class for managing DB spend updates 2025-03-27 16:31:23 -07:00
Krish Dholakia
11838e1c3b
Litellm fix db testing (#9593)
* ci: fix test

* test: safely change db url

* fix: print db url

* test: remove delenv
2025-03-27 14:50:41 -07:00
Ishaan Jaff
0155b0eba2
Merge pull request #9533 from BerriAI/litellm_stability_fixes
[Reliability Fixes] - Gracefully handle exceptions when DB is having an outage
2025-03-26 18:57:38 -07:00
Krrish Dholakia
04490c99d7 test: fix test 2025-03-26 17:12:09 -07:00
Krrish Dholakia
d4adc9764b test(test_db_schema_migration.py): ci/cd test to enforce schema migrations are documented in .sql files 2025-03-26 16:59:50 -07:00
Krish Dholakia
132d3f7baa
feat(prisma-migrations): add baseline db migration file (#9565)
adds initial baseline db migration file

enables future schema changes to be documented via .sql files
2025-03-26 16:22:56 -07:00
Ishaan Jaff
6e5d2b1ac7 handle failed db connections 2025-03-25 23:14:44 -07:00
Krrish Dholakia
364ea3b7dc test: fix test 2025-03-21 22:02:39 -07:00
Krrish Dholakia
95ef5f1009 refactor(user_api_key_auth.py): move is_route_allowed to inside common_checks
ensures consistent behaviour inside api key + jwt routes
2025-03-21 17:21:07 -07:00
Krrish Dholakia
91cf3fc40d test: initial e2e testing to ensure non admin jwt token cannot create new teams 2025-03-21 16:40:18 -07:00
Ishaan Jaff
63a32ff02c test_reset_budget_job 2025-03-17 19:37:09 -07:00
Krrish Dholakia
6ed995952f fix: fix test 2025-03-14 20:28:50 -07:00
Krish Dholakia
c93a5e2301
Merge pull request #9047 from BerriAI/litellm_dev_03_06_2025_p4
feat(handle_jwt.py): support multiple jwt url's
2025-03-10 22:37:35 -07:00
Ishaan Jaff
9dcc25d63b Merge branch 'main' into litellm_fix_team_model_access_checks 2025-03-10 19:05:11 -07:00
Krish Dholakia
c58941d49c
Merge branch 'main' into litellm_dev_03_06_2025_p4 2025-03-10 18:41:10 -07:00
Krrish Dholakia
c08705517b test: fix test 2025-03-09 19:40:03 -07:00