Commit graph

894 commits

Author SHA1 Message Date
Ishaan Jaff
82a03fd163 fixes build from pip 2025-06-14 09:03:50 -07:00
Ishaan Jaff
690513994b security - add back grype scans 2025-06-14 08:42:58 -07:00
Ishaan Jaff
7947139913
[Feat] MCP expose streamable https endpoint for LiteLLM Proxy (#11645)
* feat - add https mcp support

* fixes for MCP http integration

* fix code QA

* bump mcp dep

* test_mcp_server_manager_https_server

* test mcp server https

* fix linting error

* bump mcp in poetry

* fix import streamablehttp_client

* fix streamablehttp_client

* fix streamablehttp_client

* add streamablehttp_client

* add simple https server

* working mounted app

* working HTTPS mcp streamable

* fix code QA check

* feat: add MCP Server

* fix - init just as fastapi app

* add LITELLM_MCP_SERVER_DESCRIPTION

* fix importing / init litellm app

* Update litellm/proxy/_experimental/mcp_server/server.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update litellm/proxy/_experimental/mcp_server/server.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update server.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fixes based on review + code check

* fix linting

* test_streamable_http_mcp_handler_mock

* fix python 3.13 install

* fix deps test

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-06-12 16:32:04 -07:00
Ishaan Jaff
02b02c739b
[Feat] MCP - Add support for streamablehttp_client MCP Servers (#11628)
* feat - add https mcp support

* fixes for MCP http integration

* fix code QA

* bump mcp dep

* test_mcp_server_manager_https_server

* test mcp server https

* fix linting error

* bump mcp in poetry

* fix import streamablehttp_client

* fix streamablehttp_client

* fix streamablehttp_client

* add streamablehttp_client
2025-06-11 17:09:46 -07:00
Krrish Dholakia
196f05016e build(config.yml): weaken grype check for nightly - python3.13 known cve - blocked on chainguard to release a fix
Will call this out on release note, but this an upstream python3.13 issue  and shouldn't block nightly releases
2025-06-06 12:06:23 -07:00
Ishaan Jaff
3db272b6d2
[Perf] - Add Async + Batched S3 Logging (#11340)
* fix: add s3 v2 async

* fix: add s3 v2 async

* fix: add s3 v2 async

* test: s3 v2 logging

* fixes: s3 logging

* fixes: s3 logging use max upload batch size

* fixes: s3 logging tests

* fixes: s3 logging tests

* fixes: s3 logging tests
2025-06-02 21:52:34 -07:00
Ishaan Jaff
62a083de02 fix: linting pin mypy 2025-05-29 19:58:31 -07:00
Ishaan Jaff
226c3b74ba build: fix ui build 2025-05-28 21:26:18 -07:00
Ishaan Jaff
6b527a77aa ci/cd fix build ui 2025-05-28 21:00:39 -07:00
Krish Dholakia
4c82dd9b27
Ollama Chat - parse tool calls on streaming (#11171)
* fix(user_api_key_auth.py): fix else block

Fixes https://github.com/BerriAI/litellm/issues/11170

* refactor(ollama/chat): refactor to base config pattern

easier to maintain fixes

* fix(ollama/chat): support tool call parsing on streaming

Closes https://github.com/BerriAI/litellm/issues/11104

* test: update import location

* fix: cleanup unused import

* fix: fix ruff check error

* test: update import

* test: update test on ci

* ci: cleanup

* fix: fix chekc

* fix: fix api key check order

* test: fix import

* ci: fix script

* test: fix imports

* fix: fix tests
2025-05-27 16:14:49 -07:00
Krish Dholakia
a44bc9f613
Remove + Check for unsafe enterprise/ folder imports (#11107)
* refactor(litellm/): wrap enterprise package usage in safe try-excepts

prevents missing package error for OSS users

Fixes https://github.com/BerriAI/litellm/issues/10795

* ci(config.yml): add script to check for unsafe imports to ci/cd

Closes https://github.com/BerriAI/litellm/issues/10795

* fix: add missing docs

* test: test_openai_image_edit_litellm_sdk

* use n 4 for mapped tests (#11109)

* Fix/background health check (#10887)

* fix: improve health check logic by deep copying model list on each iteration

* test: add async test for background health check reflecting model list changes

* fix: validate health check interval before executing background health check

* fix: specify type for health check results dictionary

* fix(user_api_key_auth.py): handle user custom auth set with no custom settings

* bump: version 0.1.21 → 0.2.0

* ci(config.yml): run enterprise and litellm tests separately

* fix: fix linting error

* docs: add missing docs

* [Feat] Add content policy violation error mapping for image editd (#11113)

* feat: add image edit mapping for content policy violations

* test fix

* Expose `/list` and `/info` endpoints for Audit Log events (#11102)

* feat(audit_logging_endpoints.py): expose list endpoint to show all audit logs

make it easier for user to retrieve individual endpoints

* feat(enterprise/): add audit logging endpoint

* feat(audit_logging_endpoints.py): expose new GET `/audit/{id}` endpoint

make it easier to retrieve view individual audit logs

* feat(key_management_event_hooks.py): correctly show the key of the user who initiated the change

* fix(key_management_event_hooks.py): add key rotations as an audit log event

'

* test(test_audit_logging_endpoints.py): add simple unit testing for audit log endpoint

* fix: testing fixes

* fix: fix ruff check

* [Feat] Use aiohttp transport by default - 97% lower median latency  (#11097)

* fix: add flag for disabling use_aiohttp_transport

* feat: add _create_async_transport

* feat: fixes for transport

* add httpx-aiohttp

* feat: fixes for transport

* refactor: fixes for transport

* build: fix deps

* fixes: test fixes

* fix: ensure aiohttp does not auto set content type

* test: test fixes

* feat: add LiteLLMAiohttpTransport

* fix: fixes for responses API handling

* test: fixes for responses API handling

* test: fixes for responses API handling

* feat: fixes for transport

* fix: base embedding handler

* test: test_async_http_handler_force_ipv4

* test: fix failing deepeval test

* fix: add YARL for bedrock urls

* fix: issues with transport

* fix: comment out linting issues

* test fix

* test: XAI is unstable

* test: fixes for using respx

* test: XAI fixes

* test: XAI fixes

* test: infinity testing fixes

* docs(config_settings.md): document param

* test: test_openai_image_edit_litellm_sdk

* test: remove deprecated test

* bump respx==0.22.0

* test: test_xai_message_name_filtering

* test: fix anthropic test after bumping httpx

* use n 4 for mapped tests (#11109)

* fix: use 1 session per event loop

* test: test_client_session_helper

* fix: linting error

* fix: resolving GET requests on httpx 0.28.1

* test fixes proxy unit tests

* fix: add ssl verify settings

* fix: proxy unit tests

* fix: refactor

* tests: basic unit tests for aiohttp transports

* tests: fixes xai

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>

* test: cleanup redundant test

* fix: fix import

* fix: fix linting errors

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: JuHyun Bae <jhyun0408@nate.com>
2025-05-24 08:59:02 -07:00
Ishaan Jaff
86cdb8382b
[Feat] Use aiohttp transport by default - 97% lower median latency (#11097)
* fix: add flag for disabling use_aiohttp_transport

* feat: add _create_async_transport

* feat: fixes for transport

* add httpx-aiohttp

* feat: fixes for transport

* refactor: fixes for transport

* build: fix deps

* fixes: test fixes

* fix: ensure aiohttp does not auto set content type

* test: test fixes

* feat: add LiteLLMAiohttpTransport

* fix: fixes for responses API handling

* test: fixes for responses API handling

* test: fixes for responses API handling

* feat: fixes for transport

* fix: base embedding handler

* test: test_async_http_handler_force_ipv4

* test: fix failing deepeval test

* fix: add YARL for bedrock urls

* fix: issues with transport

* fix: comment out linting issues

* test fix

* test: XAI is unstable

* test: fixes for using respx

* test: XAI fixes

* test: XAI fixes

* test: infinity testing fixes

* docs(config_settings.md): document param

* test: test_openai_image_edit_litellm_sdk

* test: remove deprecated test

* bump respx==0.22.0

* test: test_xai_message_name_filtering

* test: fix anthropic test after bumping httpx

* use n 4 for mapped tests (#11109)

* fix: use 1 session per event loop

* test: test_client_session_helper

* fix: linting error

* fix: resolving GET requests on httpx 0.28.1

* test fixes proxy unit tests

* fix: add ssl verify settings

* fix: proxy unit tests

* fix: refactor

* tests: basic unit tests for aiohttp transports

* tests: fixes xai

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
2025-05-23 22:55:35 -07:00
Krrish Dholakia
247c364451 ci(config.yml): run enterprise and litellm tests separately 2025-05-23 21:37:06 -07:00
Ishaan Jaff
716d7c21f0
use n 4 for mapped tests (#11109) 2025-05-23 18:49:58 -07:00
Ishaan Jaff
dd4a65b83a
Feat: add MCP to Responses API and bump openai python sdk (#11029)
* feat: add MCP to responses API

* feat: bump openai version to 1.75.0

* docs MCP + responses API

* fixes: type checking

* fixes: type checking

* build: use latest openai 1.81.0

* fix: linting error

* fix: linting error

* fix: test

* fix: linting errors

* fix: test

* fix: test

* fix: linting

* Revert "fix: linting"

This reverts commit ebb19ff8cb.

* fix: linting
2025-05-22 07:24:10 -07:00
Ishaan Jaff
5ab9fc0b7a
fix linting cryptography (#10921) 2025-05-17 17:21:22 -07:00
Ishaan Jaff
e87b536d03 fix config.yml 2025-05-17 16:59:03 -07:00
Ishaan Jaff
01d0983c29 fix config.yml 2025-05-17 16:57:17 -07:00
Ishaan Jaff
93ec0c48bc
[UI QA Guardrails] - Minor UI Fixes (#10920)
* fix: presidio params on ui

* move mode and always on to the top

* explain each mode

* show switch for default on / off for guards

* allow multiple modes for guardrails

* fix: presidio output parsing with stream

* Revert "git commit fix linting error on cryptography bindings"

This reverts commit 2615f04afd.
2025-05-17 16:27:57 -07:00
Ishaan Jaff
9846abb5c0 allow using output parsing for guardrails 2025-05-17 14:52:39 -07:00
Ishaan Jaff
ddf1b20f4f Revert "git commit fix linting error on cryptography bindings"
This reverts commit 2615f04afd.
2025-05-17 14:49:54 -07:00
Ishaan Jaff
2615f04afd git commit fix linting error on cryptography bindings 2025-05-17 12:16:37 -07:00
Ishaan Jaff
9e06c42f72 test: fix litellm_mapped_tests 2025-05-14 18:29:01 -07:00
Ishaan Jaff
d558c223b3 bump resource_class: xlarge 2025-05-14 08:31:53 -07:00
Ishaan Jaff
98bed9b698 update resource_class for litellm_mapped_tests 2025-05-14 07:43:56 -07:00
Ishaan Jaff
741ea6e149 ci/cd: remove n -4 change 2025-05-13 21:21:37 -07:00
Ishaan Jaff
6b6794a926 fix mapped tests 2025-05-13 21:17:41 -07:00
Ishaan Jaff
19ca5f7307 test - increase duration of litellm mapped test 2025-05-13 20:56:48 -07:00
Ishaan Jaff
8142c20c98
[Feat] Allow specifying PII Entities Config when using Presidio Guardrails (#10810)
* refactor: use analyze_text, anonymize_text

* feat: allow defining pii_entities_config for presidio

* feat: use entities config for presidio analyze request

* feat: add test_presidio_pii.py

* testing: add guardrails testing job

* feat: allow blocking specific entities pii

* test: use 1 file for presidio guard tests

* fix: presidio pii tests

* test: presidio blocked entity

* clean up docs

* docs presidio pii parsing

* fix: raise_exception_if_blocked_entities_detected

* fix: linting errors
2025-05-13 19:48:56 -07:00
Ishaan Jaff
3130c4f8f9
[Refactor] Move LLM Guard, Secret Detection to Enterprise Pip packagea (#10782)
* refactor: move guardrails to pip

* refactor: move guardrails to pip

* testing fix: move guardrails to pip

* git commit setup_litellm_enterprise_pip
2025-05-13 09:42:22 -07:00
Ishaan Jaff
3731ee436a
[Refactor] Use pip package for enterprise/ folder (#10709)
* init enterprise pip

* init enterprise pip

* init enterprise pip

* test: enterprise pip

* add litellm-enterprise to pip

* litellm ent check

* litellm ent check

* fix import email router

* fix setup_litellm_enterprise_pip

* fix local testing with enterprise pip
2025-05-09 17:18:48 -07:00
Ishaan Jaff
7c4fae477a
[Bug Fix] UnicodeDecodeError: 'charmap' on Windows during litellm import (#10542)
* test using litellm on windows

* fix using litellm on windows

* test using litellm on windows

* fix change on ci/cd

* fix test on ci/cd
2025-05-03 21:31:05 -07:00
Krrish Dholakia
4ab0ee0b65 test: more testing fixes 2025-05-01 15:36:13 -07:00
Krrish Dholakia
7502a21279 ci(config.yml): bump config 2025-05-01 15:27:16 -07:00
Krish Dholakia
c0a464e4eb
fix(all_keys_table.tsx): fix filtering on key alias across multiple p… (#10455)
* fix(all_keys_table.tsx): fix filtering on key alias across multiple pages

* fix(key_management_endpoints.py): enable global sorting on `/key/list`

* feat(all_keys_table.tsx): support sorting on keys table

* feat(all_keys_table.tsx): enable global sorting

* ci(config.yml): add requests mock
2025-04-30 22:21:41 -07:00
Krish Dholakia
711601e22a
Add key-level multi-instance tpm/rpm/max parallel request limiting (#10458)
* fix: initial commit of v2 parallel request limiter hook

enables multi-instance rate limiting to work

* fix: subsequent commit with additional refactors

* fix(parallel_request_limiter_v2.py): cleanup initial call hook

simplify it

* fix(parallel_request_limiter_v2.py): working v2 parallel request limiter

* fix: more updates - still not passing testing

* fix(test_parallel_request_limiter_v2.py): update test + add conftest

* fix: fix ruff checks

* fix(parallel_request_limiter_v2.py): use pull via pattern method to load in keys instance wouldn't have seen yet

Fixes issue where redis syncing was not pulling key until instance had seen it

* test: update testing to cover tpm and rpm

* fix(parallel_request_limiter_v2.py): fix ruff errors

* fix(proxy/hooks/__init__.py): feature flag export

* fix(proxy/hooks/__init_.py): fix linting error

* ci(config.yml): add tests/enterprise to ci/cd

* fix: fix ruff check

* test: update testing
2025-04-30 21:32:31 -07:00
Ishaan Jaff
616c1ad666 fix orjson ci/cd test 2025-04-30 19:37:12 -07:00
Ishaan Jaff
f984089b01
[Reliability fix] Redis transaction buffer - ensure all redis queues are periodically flushed (#10393)
* test_e2e_size_of_redis_buffer

* fix store_in_memory_spend_updates_in_redis

* _commit_spend_updates_to_db_with_redis

* daily_tag_spend_update_transactions

* pip install fakeredis==2.28.1
2025-04-28 21:36:54 -07:00
Krrish Dholakia
d2a462fc93 ci: handle whl 2025-04-12 11:03:21 -07:00
Krrish Dholakia
4ea9887ff6 ci: see what's in tmp dir 2025-04-12 10:59:53 -07:00
Krrish Dholakia
7b465c24a9 fix(config.yml): only run publish_proxy_extras check on main 2025-04-12 10:30:32 -07:00
Krish Dholakia
d004fb542f
fix(litellm_proxy_extras): add baselining db script (#9942)
* fix(litellm_proxy_extras): add baselining db script

Fixes https://github.com/BerriAI/litellm/issues/9885

* fix(prisma_client.py): fix ruff errors

* ci(config.yml): add publish_proxy_extras step

* fix(config.yml): compare contents between versions to check for changes

* fix(config.yml): fix check

* fix: install toml

* fix: update check

* fix: ensure versions in sync

* fix: fix version compare

* fix: correct the cost for 'gemini/gemini-2.5-pro-preview-03-25' (#9896)

* fix: Typo in the cost 'gemini/gemini-2.5-pro-preview-03-25', closes #9854

* chore: update in backup file as well

* Litellm add managed files db (#9930)

* fix(openai.py): ensure openai file object shows up on logs

* fix(managed_files.py): return unified file id as b64 str

allows retrieve file id to work as expected

* fix(managed_files.py): apply decoded file id transformation

* fix: add unit test for file id + decode logic

* fix: initial commit for litellm_proxy support with CRUD Endpoints

* fix(managed_files.py): support retrieve file operation

* fix(managed_files.py): support for DELETE endpoint for files

* fix(managed_files.py): retrieve file content support

supports retrieve file content api from openai

* fix: fix linting error

* test: update tests

* fix: fix linting error

* feat(managed_files.py): support reading / writing files in DB

* feat(managed_files.py): support deleting file from DB on delete

* test: update testing

* fix(spend_tracking_utils.py): ensure each file create request is logged correctly

* fix(managed_files.py): fix storing / returning managed file object from cache

* fix(files/main.py): pass litellm params to azure route

* test: fix test

* build: add new prisma migration

* build: bump requirements

* test: add more testing

* refactor: cleanup post merge w/ main

* fix: fix code qa errors

* [DB / Infra] Add new column team_member_permissions  (#9941)

* add team_member_permissions to team table

* add migration.sql file

* fix poetry lock

* fix prisma migrations

* fix poetry lock

* fix migration

* ui new build

* fix(factory.py): correct indentation for message index increment in ollama,  This fixes bug #9822 (#9943)

* fix(factory.py): correct indentation for message index increment in ollama_pt function

* test: add unit tests for ollama_pt function handling various message types

* ci: update test

* fix: fix check

* ci: see what dir looks like

* ci: more checks

* ci: fix filepath

* ci: cleanup

* ci: fix ci

---------

Co-authored-by: Nilanjan De <nilanjan.de@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Dan Shaw <dan@danieljshaw.com>
2025-04-12 10:29:34 -07:00
Krish Dholakia
ac4f32fb1e
Cost tracking for gemini-2.5-pro (#9837)
* build(model_prices_and_context_window.json): add google/gemini-2.0-flash-lite-001 versioned pricing

Closes https://github.com/BerriAI/litellm/issues/9829

* build(model_prices_and_context_window.json): add initial support for 'supported_output_modalities' param

* build(model_prices_and_context_window.json): add initial support for 'supported_output_modalities' param

* build(model_prices_and_context_window.json): add supported endpoints to gemini-2.5-pro

* build(model_prices_and_context_window.json): add gemini 200k+ pricing

* feat(utils.py): support cost calculation for gemini-2.5-pro above 200k tokens

Fixes https://github.com/BerriAI/litellm/issues/9807

* build: test dockerfile change

* build: revert apk change

* ci(config.yml): pip install wheel

* ci: test problematic package first

* ci(config.yml): pip install only binary

* ci: try more things

* ci: test different ml_dtypes version

* ci(config.yml): check ml_dtypes==0.4.0

* ci: test

* ci: cleanup config.yml

* ci: specify ml dtypes in requirements.txt

* ci: remove redisvl depedency (temporary)

* fix: fix linting errors

* test: update test

* test: fix test
2025-04-09 18:48:43 -07:00
Ishaan Jaff
c403dfb615 pip install --upgrade pip wheel setuptools 2025-04-08 16:38:44 -07:00
Ishaan Jaff
8a596dbe8c pip install wheel 2025-04-08 16:27:09 -07:00
Ishaan Jaff
12b3923579 update circle ci requirements 2025-04-04 14:45:14 -07:00
Ishaan Jaff
4ed0ab5b1c Revert "remove google dns for img tests"
This reverts commit d3fc8b563c.
2025-04-02 20:42:29 -07:00
Ishaan Jaff
d3fc8b563c remove google dns for img tests 2025-04-02 20:34:47 -07:00
Krrish Dholakia
6e8d9860a5 ci(config.yml): remove -s from large tests
easier to track test progress
2025-04-02 08:50:47 -07:00
Ishaan Jaff
6d4a6a8402 add spend tracking config.yaml 2025-03-31 19:42:00 -07:00