Commit graph

855 commits

Author SHA1 Message Date
Ishaan Jaff
5ab3d74371 DEFAULT_NUM_WORKERS_LITELLM_PROXY 2025-09-01 20:34:23 -07:00
Ishaan Jaff
c839b2c751 installing_litellm_on_python 2025-08-26 19:51:02 -07:00
Ishaan Jaff
b9132968b2
[Perf] Improvements for Async Success Handler (Logging Callbacks) - Approx +130 RPS (#13905)
* [Performance] Reduce Significant CPU overhead from litellm_logging.py (#13895)

* fix: litellm.configured_cold_storage_logger

* fix Session Management - Non-OpenAI Models docs

* ruff fix

* test fix

* create LoggingWorker

* add GLOBAL_LOGGING_WORKER for async task handling

* fix logging tests

* add conftest

* fix conftest

* test fix location of encode bedrock runtime modelid arn

* fix conftest.py

* tuning LoggingWorker

* conftest.py

* fix conftest batches/

* test_async_chat_azure

* event_loop

* test_bedrock_streaming_passthrough_test2

* fix GLOBAL_LOGGING_WORKER

* logging worker

* add flush for global logging worker

* Revert "fix GLOBAL_LOGGING_WORKER"

This reverts commit d254f508f4.

* fix conftest clear_queue

* fix conftest clear_queue

* setup_and_teardown for llm translation

* docs AWS_REGION

* test_async_chat_azure

* change test DIR

* run ci/cd again

* use 1 job for litellm_router_unit_testing

* fix space

* fix litellm_router_unit_testing

* test_aaarouter_dynamic_cooldown_message_retry_time

* litellm_router_unit_testing

* conftest.py clearing qu

* fixes litellm_router_unit_testing

* fixes clear_queue

* fix router_unit_tests

* remove conftest

* add back conftest for router

* fix event loop test

* test fix

* fixes for LoggingWorker

* ruff fix
2025-08-23 13:13:23 -07:00
Ishaan Jaff
76f1064229
[Bug Fix] litellm incompatible with newest release of openAI v1.100.0 (#13728)
* fix imports OpenAI SDK

* ResponseText fixes

* fixes ResponseText

* fix imports

* catch AttributeError

* fix import

* use openai==1.100.1

* fix build from PIP

* fix lint test

* Print OpenAI version

* fix Install dependencies
2025-08-18 18:26:17 -07:00
0x-fang
9f17bedb48
bump up boto3 and aioboto3 versions (#13665) 2025-08-15 17:06:08 -07:00
Ishaan Jaff
f5a1c14929 Revert "update boto3 version to 1.37.38 (#13656)"
This reverts commit 6f2264a13d.
2025-08-15 11:24:59 -07:00
Ishaan Jaff
c331473b75 Revert "fix aioboto3"
This reverts commit 82d1cb48b7.
2025-08-15 11:24:50 -07:00
Ishaan Jaff
82d1cb48b7 fix aioboto3 2025-08-15 11:19:14 -07:00
0x-fang
6f2264a13d
update boto3 version to 1.37.38 (#13656) 2025-08-15 11:06:54 -07:00
Ishaan Jaff
eb4bd26f24
[Bug Fix] - Get Routes (#13466)
* fixes get_routes_for_mounted_app

* fix - use _safe_get_endpoint_name

* fix code QA check

* test_get_routes_for_mounted_app_with_static_files

* test fixes
2025-08-09 12:52:23 -07:00
Ishaan Jaff
3905cee579 test fixes 2025-08-08 18:50:09 -07:00
Krrish Dholakia
fedd805b6a test: loosen check 2025-08-01 09:21:13 -07:00
Krrish Dholakia
4a66617d71 build(config.yml): use ci/cd postgres in test 2025-08-01 08:48:07 -07:00
Krish Dholakia
c7e4435bdc
Fix - using managed files w/ OTEL + UI - add model group alias on UI (#13171)
* fix(router.py): safe deep copy kwargs

OTEL adds a parent_otel_span which cannot be deepcopied

* fix: use safe deep copy in other places as well

* test: add script to check and ban copy.deepcopy of kwargs

enforce safe_deep_copy usage

* build(ui/): new component for adding model group alias on UI

* fix(proxy_server.py): support updating model_group_alias via /config/update

allows ui component to work

* fix(router.py): update model_group_alias in router settings based on db value

* fix: fix code qa error
2025-07-31 21:22:04 -07:00
Ishaan Jaff
ee70d593c1
[Feat] Allow redacting message / response content for specific logging integrations - DD LLM Observability (#13158)
* fix redact_standard_logging_payload

* add StandardCustomLoggerInitParams

* allow defining DatadogLLMObsInitParams

* fix init DataDogLLMObsLogger

* fix import

* update redact_standard_logging_payload_from_model_call_details

* test_dd_llms_obs_redaction

* docs DD logging

* docs DD

* docs DD

* Redacting Messages, Response docs DD LLM Obs

* fix redaction logic

* fix create_llm_obs_payload

* fix logging response

* fixes

* ruff fix

* fix test

* test_dd_llms_obs_redaction

* test_create_llm_obs_payload

* redact_standard_logging_payload_from_model_call_details

* img - dd_llm_obs

* docs DD

* fix linting

* fix linting

* fix mypy

* test_create_llm_obs_payload

* test_create_llm_obs_payload

* fix mock_env_vars

* fix _handle_anthropic_messages_response_logging
2025-07-31 16:44:16 -07:00
Ishaan Jaff
cbb922b1bb
[Bug Fix] Gemini-CLI Integration - ensure tool calling works as expected on generateContent (#13189)
* transform_generate_content_request

* add tools in GenerateContentRequestDict

* add generate_content_handler tool calling

* google_generate_content_endpoint_testing

* test_mock_stream_generate_content_with_tools

* test_validate_post_request_parameters

* fixes for generate_content_handler

* fix VertexAIGoogleGenAIConfig

* fixes veretx ai

* google_generate_content_endpoint_testing

* test_async_streaming_with_logging

* load_vertex_ai_credentials

* test_vertex_anthropic.py
2025-07-31 16:42:57 -07:00
Krish Dholakia
c6a8733234
build(config.yml): migrate build_and_test to ci/cd pg db (#13166) 2025-07-30 18:19:36 -07:00
Krrish Dholakia
e3ba350345 ci: migrate to postgres in ci/cd 2025-07-30 17:00:06 -07:00
Krrish Dholakia
2266938184 ci(config.yml): remove check 2025-07-30 16:33:42 -07:00
Krrish Dholakia
88aed06bbb ci: migrate to db in pipeline 2025-07-30 15:48:15 -07:00
Ishaan Jaff
5aa4a70bd3 fix grype scan 2025-07-30 15:35:16 -07:00
Krrish Dholakia
3bf7411491 ci(config.yml): run prisma generate before running enterprise tests 2025-07-30 15:28:45 -07:00
Krrish Dholakia
6da36049fe ci: remove bad script 2025-07-30 15:25:32 -07:00
Krish Dholakia
3ca99f1def
Litellm explore postgres db ci cd (#13156)
* ci(config.yml): testing with ci/cd db

* build: spin up pg db in ci/cd test
2025-07-30 15:17:12 -07:00
Ishaan Jaff
8f86355361 fix ci/cd testing 2025-07-25 09:05:27 -07:00
Ishaan Jaff
29cff4f4e6 pip install semantic_router --no-deps 2025-07-24 20:09:37 -07:00
Ishaan Jaff
b8e404dd95
[Feat] Backend Router - Add Auto-Router powered by semantic-router (#12955)
* add router.json

* test_router_auto_router

* async_pre_routing_hook

* fixes for auto router

* add async_pre_routing_hook

* add LiteLLMRouterEncoder

* update test auto_router_embedding_model

* add auto_router_embedding_model

* add AutoRouter

* fix async_pre_routing_hook

* update async_pre_routing_hook

* fix auto router

* fix router.json

* working router init

* working embedding encoder

* working auto router

* test_router_auto_router

* test auto router

* add semantic-router as optional for litellm

* add extras

* semantic_router==0.1.10

* ruff fix

* use aiohttp==3.10.11

* python-dotenv==1.0.1

* test auto router

* test_router_auto_router

* semantic_router

* test_is_auto_router_deployment

* fix check

* fix docker build step

* add semantic_router

* Revert "add semantic_router"

This reverts commit 537b672887.
2025-07-24 18:32:56 -07:00
Ishaan Jaff
d93e85282e test_bad_database_url 2025-07-22 18:37:51 -07:00
Krrish Dholakia
aeb270b759 build: move build_and_test to use prisma migrate 2025-07-17 12:16:06 -07:00
Ishaan Jaff
57b0b4edf3
[Bug fix] [Bug]: Verbose log is enabled by default (#12596)
* test find_set_verbose_assignments

* fix set verbose

* test set verbose

* fix unused import
2025-07-14 20:06:16 -07:00
Ishaan Jaff
57d9ca5f3a ci/cd remove flaky load testing scripts. We will do load testing on post releases 2025-07-12 11:17:54 -07:00
Ishaan Jaff
a0a57687ca
fix security - mcp (#12362) 2025-07-07 15:53:54 -07:00
Ishaan Jaff
08f3b06d82
[Feat] Bump langfuse python SDK version and LANGFUSE_TRACING_ENVIRONMENT (#12376)
* bump langfuse to 2.59.7

* _set_langfuse_specific_attributes

* fix files

* test_set_langfuse_environment_attribute
2025-07-07 15:53:43 -07:00
Ishaan Jaff
f285765862 add code_qa_check_tests 2025-07-04 10:06:40 -07:00
Ishaan Jaff
39955129f5
fix mapped tests (#12320)
* fix - use flush llm client cache

* faster mapped tests

* test_async_multiple_response_ids_routing

* fix tests

* test_ateam_member_update_admin_requires_premium

* regular mapped tests

* Revert "Fix: Initialize JSON logging for all loggers when JSON_LOGS=True (#12206)"

This reverts commit 2c60c316ec.

* reset num workers
2025-07-04 10:04:43 -07:00
Ishaan Jaff
0c19414b36
[️ Python SDK import] - reduce python sdk import time by .3s (#12140)
* use 1 file for KeyManagementSystem

* move key management settings

* fix import locs

* test_proxy_types_not_imported

* test the import loc

* fix import item

* fix imports

* fix import loc

* fix imports
2025-06-28 14:57:10 -07:00
Ishaan Jaff
7f11b5b76c
[Security] - Add Trivy Security Scan for UI + Docs folder - remove all vulnerabilities (#11778)
* docs - sec scan

* ui - fix 0 security issues

* fix - 0 medium vuln on docs

* add MEDIUM scan for Docs

* Run Trivy scan on LiteLLM UI
2025-06-16 17:13:19 -07:00
Ishaan Jaff
327868fff6 speed up llm_translation_testing 2025-06-14 16:56:02 -07:00
Ishaan Jaff
d477c1915d fix - speed up litellm_proxy_unit_testing 2025-06-14 16:30:28 -07:00
Ishaan Jaff
82a03fd163 fixes build from pip 2025-06-14 09:03:50 -07:00
Ishaan Jaff
690513994b security - add back grype scans 2025-06-14 08:42:58 -07:00
Ishaan Jaff
7947139913
[Feat] MCP expose streamable https endpoint for LiteLLM Proxy (#11645)
* feat - add https mcp support

* fixes for MCP http integration

* fix code QA

* bump mcp dep

* test_mcp_server_manager_https_server

* test mcp server https

* fix linting error

* bump mcp in poetry

* fix import streamablehttp_client

* fix streamablehttp_client

* fix streamablehttp_client

* add streamablehttp_client

* add simple https server

* working mounted app

* working HTTPS mcp streamable

* fix code QA check

* feat: add MCP Server

* fix - init just as fastapi app

* add LITELLM_MCP_SERVER_DESCRIPTION

* fix importing / init litellm app

* Update litellm/proxy/_experimental/mcp_server/server.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update litellm/proxy/_experimental/mcp_server/server.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update server.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fixes based on review + code check

* fix linting

* test_streamable_http_mcp_handler_mock

* fix python 3.13 install

* fix deps test

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-06-12 16:32:04 -07:00
Ishaan Jaff
02b02c739b
[Feat] MCP - Add support for streamablehttp_client MCP Servers (#11628)
* feat - add https mcp support

* fixes for MCP http integration

* fix code QA

* bump mcp dep

* test_mcp_server_manager_https_server

* test mcp server https

* fix linting error

* bump mcp in poetry

* fix import streamablehttp_client

* fix streamablehttp_client

* fix streamablehttp_client

* add streamablehttp_client
2025-06-11 17:09:46 -07:00
Krrish Dholakia
196f05016e build(config.yml): weaken grype check for nightly - python3.13 known cve - blocked on chainguard to release a fix
Will call this out on release note, but this an upstream python3.13 issue  and shouldn't block nightly releases
2025-06-06 12:06:23 -07:00
Ishaan Jaff
3db272b6d2
[Perf] - Add Async + Batched S3 Logging (#11340)
* fix: add s3 v2 async

* fix: add s3 v2 async

* fix: add s3 v2 async

* test: s3 v2 logging

* fixes: s3 logging

* fixes: s3 logging use max upload batch size

* fixes: s3 logging tests

* fixes: s3 logging tests

* fixes: s3 logging tests
2025-06-02 21:52:34 -07:00
Ishaan Jaff
62a083de02 fix: linting pin mypy 2025-05-29 19:58:31 -07:00
Ishaan Jaff
226c3b74ba build: fix ui build 2025-05-28 21:26:18 -07:00
Ishaan Jaff
6b527a77aa ci/cd fix build ui 2025-05-28 21:00:39 -07:00
Krish Dholakia
4c82dd9b27
Ollama Chat - parse tool calls on streaming (#11171)
* fix(user_api_key_auth.py): fix else block

Fixes https://github.com/BerriAI/litellm/issues/11170

* refactor(ollama/chat): refactor to base config pattern

easier to maintain fixes

* fix(ollama/chat): support tool call parsing on streaming

Closes https://github.com/BerriAI/litellm/issues/11104

* test: update import location

* fix: cleanup unused import

* fix: fix ruff check error

* test: update import

* test: update test on ci

* ci: cleanup

* fix: fix chekc

* fix: fix api key check order

* test: fix import

* ci: fix script

* test: fix imports

* fix: fix tests
2025-05-27 16:14:49 -07:00
Krish Dholakia
a44bc9f613
Remove + Check for unsafe enterprise/ folder imports (#11107)
* refactor(litellm/): wrap enterprise package usage in safe try-excepts

prevents missing package error for OSS users

Fixes https://github.com/BerriAI/litellm/issues/10795

* ci(config.yml): add script to check for unsafe imports to ci/cd

Closes https://github.com/BerriAI/litellm/issues/10795

* fix: add missing docs

* test: test_openai_image_edit_litellm_sdk

* use n 4 for mapped tests (#11109)

* Fix/background health check (#10887)

* fix: improve health check logic by deep copying model list on each iteration

* test: add async test for background health check reflecting model list changes

* fix: validate health check interval before executing background health check

* fix: specify type for health check results dictionary

* fix(user_api_key_auth.py): handle user custom auth set with no custom settings

* bump: version 0.1.21 → 0.2.0

* ci(config.yml): run enterprise and litellm tests separately

* fix: fix linting error

* docs: add missing docs

* [Feat] Add content policy violation error mapping for image editd (#11113)

* feat: add image edit mapping for content policy violations

* test fix

* Expose `/list` and `/info` endpoints for Audit Log events (#11102)

* feat(audit_logging_endpoints.py): expose list endpoint to show all audit logs

make it easier for user to retrieve individual endpoints

* feat(enterprise/): add audit logging endpoint

* feat(audit_logging_endpoints.py): expose new GET `/audit/{id}` endpoint

make it easier to retrieve view individual audit logs

* feat(key_management_event_hooks.py): correctly show the key of the user who initiated the change

* fix(key_management_event_hooks.py): add key rotations as an audit log event

'

* test(test_audit_logging_endpoints.py): add simple unit testing for audit log endpoint

* fix: testing fixes

* fix: fix ruff check

* [Feat] Use aiohttp transport by default - 97% lower median latency  (#11097)

* fix: add flag for disabling use_aiohttp_transport

* feat: add _create_async_transport

* feat: fixes for transport

* add httpx-aiohttp

* feat: fixes for transport

* refactor: fixes for transport

* build: fix deps

* fixes: test fixes

* fix: ensure aiohttp does not auto set content type

* test: test fixes

* feat: add LiteLLMAiohttpTransport

* fix: fixes for responses API handling

* test: fixes for responses API handling

* test: fixes for responses API handling

* feat: fixes for transport

* fix: base embedding handler

* test: test_async_http_handler_force_ipv4

* test: fix failing deepeval test

* fix: add YARL for bedrock urls

* fix: issues with transport

* fix: comment out linting issues

* test fix

* test: XAI is unstable

* test: fixes for using respx

* test: XAI fixes

* test: XAI fixes

* test: infinity testing fixes

* docs(config_settings.md): document param

* test: test_openai_image_edit_litellm_sdk

* test: remove deprecated test

* bump respx==0.22.0

* test: test_xai_message_name_filtering

* test: fix anthropic test after bumping httpx

* use n 4 for mapped tests (#11109)

* fix: use 1 session per event loop

* test: test_client_session_helper

* fix: linting error

* fix: resolving GET requests on httpx 0.28.1

* test fixes proxy unit tests

* fix: add ssl verify settings

* fix: proxy unit tests

* fix: refactor

* tests: basic unit tests for aiohttp transports

* tests: fixes xai

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>

* test: cleanup redundant test

* fix: fix import

* fix: fix linting errors

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: JuHyun Bae <jhyun0408@nate.com>
2025-05-24 08:59:02 -07:00