Commit graph

26945 commits

Author SHA1 Message Date
Ishaan Jaffer
2fe2704a21 fixes code QA check 2025-10-31 17:55:49 -07:00
Ishaan Jaffer
73f745ee09 fix claude-sonnet-4-5-20250929 2025-10-31 17:48:49 -07:00
Ishaan Jaffer
09e18ea0ac fix get_guardrail_info 2025-10-31 17:45:38 -07:00
Ishaan Jaff
c59a0c9681
[Feat] UI - Allow setting cache settings on UI (#16143)
* add LiteLLM_CacheConfig

* add CacheSettingsField

* add UI cache saver

* feat add cache_settings_router

* fix schema

* fix ssl_check_hostname

* refactor into utils

* add groups for field names

* add test_connection in base cache

* add test_connection inredis and redis cluster

* feat _decrypt_db_variables

* add cache settings endpoints

* test_test_cache_connection_calls_cache_test_connection_with_params

* fix: add switch_on_llm_response_caching

* feat use CacheSettingsManager

* feat use CacheSettingsManager

* TestCacheSettingsManager

* fix update_config

* Cache Field test
2025-10-31 17:43:59 -07:00
yuneng-jiang
725671395a
[Feature] UI - Config Guardrails should not be editable and guardrail info fix (#16142)
* UI Config Guardrails should not be editable and guardrails info definition location fix

* Remove unused import

* Added literals for guardrail definition location
2025-10-31 16:29:24 -07:00
yuneng-jiang
cf0d84c789
[Feature] UI - Add Tags to update model (#16140)
* Add Tags to update model

* Test Key Page Tag Selector
2025-10-31 16:28:46 -07:00
yuneng-jiang
dd053e1d12
[Feature] UI - Add tags in model creation (#16138)
* Add tags in model creation

* Removing unused import in test
2025-10-31 16:26:30 -07:00
Ishaan Jaff
b6312e68f0
[Feat] UI + Management - Allow setting all routing strategies, tag filtering on UI (#16139)
* add ROUTER_SETTINGS_FIELDS

* feat: add router_settings_router on proxy

* add RouterSettingsField

* update RouterSettingsResponse

* add ROUTER_SETTINGS_FIELDS

* add getRouterSettingsCall

* add UI friendly name

* fix router settings

* fix get descriptions

* fix toggles /refator

* fix refactor

* fix refactor

* fix ROUTING_STRATEGY_DESCRIPTIONS

* fix fields
2025-10-31 11:29:37 -07:00
Raghav Jhavar
1f10e8e49c
make debug log debug (#16137) 2025-10-31 11:17:24 -07:00
1stprinciple
2824388c30
Don't add "accounts/fireworks/models" prefix for Fireworks Provider (#15938)
* init

* if not model.startswith("accounts/") and "#" not in model:
2025-10-30 20:36:42 -07:00
Sameer Kankute
689b210c18
Add Xai websearch cost (#16001)
* Add xai websearch cost

* Add test for websearch cost in xai

* remove not required changes
2025-10-30 20:35:34 -07:00
Boxuan Li
297c2a0bf4
prompt_caching.md: Fix wrong prompt_tokens definition (#16044) 2025-10-30 20:13:13 -07:00
Nicholas Couture
f747a4a38a
fix: Handle multiple rate limit types per descriptor and prevent IndexError (#16039)
* improve descriptor_key handling for multiple and missing rate limit descriptors in parallel request limiter v3

* Add tests for parallel request limiter v3 in proxy hooks
2025-10-30 20:12:54 -07:00
Sameer Kankute
c1369a07ba
Add Add per model group header forwarding for Bedrock Invoke API (#16042) 2025-10-30 20:10:17 -07:00
tanvithakur
bb79a75a89
Changes to fix frequency_penalty and presence_penalty issue for gemini-2.5-pro model (#16041)
* Update vertex_and_google_ai_studio_gemini.py

* Fix logic

* Fix mypy error

* Fix lgic

---------

Co-authored-by: tanvi thakur <tanvi_thakur@intuit.com>
2025-10-30 20:02:58 -07:00
John Lathouwers
a6f740f28b
Add OCI Signer Authentication. Closes #16048, Closes #15654 (#16064)
* Add OCI Signer Authentication. Closes #16048,  Closes #15654

* Fix linting error

* Remove Recommended, Catch None, Trim Whitespace

* Make method clear re Proxy vs SDK
2025-10-30 19:59:01 -07:00
pale-aura
0d84c11bbe
Fix vercel_ai_gateway entry for glm-4.6 (#16084)
* Fix vercel_ai_gateway entry for glm-4.6

* Fix: add unit test for vercel_ai_gateway/zai/glm-4.6

* Fix:

Add fix for unit test on vercel_ai_gateway/zai/glm-4.6

* Remove print statements in test_vercel_ai_gateway_glm46_cost_math
2025-10-30 19:40:35 -07:00
Geoffray Viossat
b29ba9e3be
feat: handle Deepgram detected language when available (#16093) 2025-10-30 19:19:34 -07:00
Daniel Dimenshtein
cde9ba5f93
Update model_prices_and_context_window.json, qwen-vl models (#16106)
Add supports vision, to qwen vl models
2025-10-30 19:18:35 -07:00
Ishaan Jaffer
4d88f21393 docs fix 2025-10-30 18:25:51 -07:00
Seongbin Lim
18e7eaaddb
Add gemini/gemini-embedding-001 pricing entry for Google GenAI API (#16078)
* Add gemini/gemini-embedding-001 pricing entry for Google GenAI API

Add pricing and configuration entry for "gemini/gemini-embedding-001"
model when called via Google GenAI API (google-genai provider). This
mirrors the existing Vertex AI entry ("gemini-embedding-001") but uses
the 'gemini' provider instead.

Co-Authored-By: Claude <noreply@anthropic.com>

* Update tpm and rpm for gemini/gemini-embedding-001

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-10-30 17:39:19 -07:00
Ishaan Jaff
0428229032
[Docs] readme fixes add supported providers (#16109)
* add provider test

* docs readme.md

* docs providers

* order providers

* test_providers_alphabetically_ordered

* docs endpoint

* fix config

* add ENDPOINT_COLUMNS

* add provider endpoints

* docs fix
2025-10-30 17:23:20 -07:00
Krrish Dholakia
30f343c3cb docs(deploy.md): document new terraform module 2025-10-30 17:11:55 -07:00
Ishaan Jaff
fd073ee682
Litellm docs readme fixes (#16107)
* add provider test

* docs readme.md

* docs providers

* order providers

* test_providers_alphabetically_ordered

* docs endpoint

* fix config
2025-10-30 17:05:32 -07:00
Ishaan Jaffer
40c663439a fix readme.md 2025-10-30 15:54:50 -07:00
yuneng-jiang
720ba865fb
[Infra] Litellm Backend SSO Changes (#16029)
* SSO Backend changes

* Encrypt and Decrypt, load into os env

* Linting and addressing comments
2025-10-30 14:32:08 -07:00
Sameer Kankute
eed3ad0bdb
Fix: Moderations endpoint now respects api_base configuration parameter (#16087)
* Update moderation to use api base

* Update moderation to use api base

* Fix mypy error
2025-10-30 11:01:26 -07:00
yuneng-jiang
b9850accaa
Circle CI e2e test fix (#16100) 2025-10-30 09:54:46 -07:00
Thomas Mildner
6fc33add9a
Opik user auth key metadata Documentation (#16004)
* fix(opik): enhance requester metadata retrieval from API key auth

* docs(opik): add automatic metadata association from API keys in documentation
2025-10-29 19:13:19 -07:00
Sameer Kankute
192935120d
Add OpenAI-compatible annotations support for Cohere v2 citations 2025-10-29 19:12:17 -07:00
Talal
5e10ea4136
Improve(mcp): respect X-Forwarded- headers in OAuth endpoints (#16036)
* fix(mcp): respect X-Forwarded-Proto header in OAuth endpoints

When LiteLLM proxy is deployed behind a reverse proxy (like nginx or a load balancer) that terminates SSL/TLS, the proxy receives HTTP requests internally but should expose HTTPS URLs externally. This change detects the X-Forwarded-Proto header and uses it to construct correct redirect URIs and endpoint URLs.

Changes:
- Added X-Forwarded-Proto detection to authorize, token, oauth_protected_resource_mcp, oauth_authorization_server_mcp, and register_client endpoints
- Added comprehensive tests for X-Forwarded-Proto header support across all affected endpoints
- Fixed existing tests to properly mock request.headers

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix formatting

* feat(mcp): support X-Forwarded-Host for proxy base URL reconstruction

Extended X-Forwarded-Proto support to also handle X-Forwarded-Host and X-Forwarded-Port headers. This allows LiteLLM to correctly construct redirect URIs and endpoint URLs when deployed behind a reverse proxy that changes the host/port.

Example scenario:
- Internal URL: http://localhost:8888/github/mcp
- External URL: https://proxy.abc.com/github/mcp
- Proxy sets: X-Forwarded-Proto: https, X-Forwarded-Host: proxy.abc.com

Changes:
- Added get_request_base_url() helper function to centralize X-Forwarded-* header handling
- Replaced all inline X-Forwarded-Proto checks with calls to the helper function
- Helper handles X-Forwarded-Proto, X-Forwarded-Host, and X-Forwarded-Port
- Added tests for X-Forwarded-Host scenarios in authorize and token endpoints

Fixes issue where protected resource URL mismatch occurred:
  Error: Protected resource http://proxy.abc.com:8888/github/mcp
  does not match expected https://proxy.abc.com/github/mcp

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: replace Yelp-specific hostnames with generic examples

Changed all references from chatproxy.yelpcorp.com to proxy.example.com in:
- test_proxy_forwarding.py (default host parameter)
- TEST_PROXY_FORWARDING.md (documentation examples)
- discoverable_endpoints.py (docstring example)
- test_discoverable_endpoints.py (test mock data)

This makes the code more generic and suitable for open source.
All 13 tests still passing.

* remove accidentally added files

* fix formatting

* add new test for get_base_url

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-10-29 19:11:32 -07:00
Timothée Lecomte
eb0e4f34dc
docs: use custom-llm-provider header in examples (#16055) 2025-10-29 19:06:48 -07:00
Jason Roberts
6672250769
feat(guardrails): Add per-request profile overrides to PANW Prisma AIRS (#16069)
- Per-request metadata: profile_name, profile_id, user_ip, app_name
- Full litellm_call_id for traceability
- Optional profile_name (supports API keys with linked profiles)
- Backward compatible
2025-10-29 19:05:59 -07:00
yuneng-jiang
1b234108c0
[Feature] UI - Add Key Type Select in Key Settings (#16034)
* Add Key Type Select in Key Settings

* Frontend changes to fully support editing key type in key settings
2025-10-29 18:41:44 -07:00
yuneng-jiang
cd6d6cfdb5
Test Key UI Embeddings (#16065) 2025-10-29 18:40:19 -07:00
yuneng-jiang
5c71455d22
Validation for Proxy Base URL in SSO Settings (#16082) 2025-10-29 18:39:59 -07:00
Ishaan Jaffer
044e26081d test_get_request_body_nova_canvas_inference_profile_arn 2025-10-29 18:30:59 -07:00
Ishaan Jaff
aea78b8d1a
[Feat] Add support for Batch API Rate limiting - PR1 adds support for input based rate limits (#16075)
* add count_input_file_usage

* add count_input_file_usage

* fix count_input_file_usage

* _get_batch_job_input_file_usage

* fixes imports

* use _get_batch_job_input_file_usage

* test_batch_rate_limits

* add _check_and_increment_batch_counters

* add get_rate_limiter_for_call_type

* test_batch_rate_limit_multiple_requests

* fixes for batch limits

* fix linting

* fix MYPY linting
2025-10-29 18:28:52 -07:00
Ishaan Jaffer
8a7f39daa4 tes numeric constants 2025-10-29 18:23:38 -07:00
Ishaan Jaffer
f538caaa01 fix proxy_build_from_pip_tests 2025-10-29 18:22:55 -07:00
Ishaan Jaffer
a3e70b8885 fix model by provider test 2025-10-29 18:21:06 -07:00
OrionCodeDev
5f52533294
Fix spend tracking for OCR/aOCR requests (log pages_processed + recognize OCRResponse) (#16070)
* fix-ocr-cost

* fix1
2025-10-29 18:11:21 -07:00
Ishaan Jaffer
a10b0b829e docs fix rbac improvements 2025-10-29 17:42:24 -07:00
Ishaan Jaffer
06449df9d3 fix codestral-embed 2025-10-29 13:15:32 -07:00
Ishaan Jaff
abbb1476ee
feat: add codestral-embed-2505 (#16071) 2025-10-29 13:13:55 -07:00
Ishaan Jaff
99feefd614
[Feat] Add FAL AI Image Generations on LiteLLM (#16067)
* add fal-ai provider

* fix image_generation_handler

* init FalAIImageGenerationConfig

* init cost_calculator

* init FAL AI

* TestFAL_AI_ImageGeneration

* fix load_custom_provider_entrypoints

* TestFAL_AI_ImageGeneration

* add imagen4 transform FAL AI

* add FAL AI imagen 4 transform

* BaseImageGenTest

* test_fal_ai_image_generation_basic

* add BRIA + Recraft img gen

* add recraft + BRIA

* test_fal_ai_image_generation_basic

* tests for flux PRO v11

* Add FAL AI SD

* test FAL AI SD

* docs FAL AI

* docs fal ai

* Using Model-Specific Parameters

* add fal ai model prices

* add fall_ai JPG logo

* ui fixes FAL AI

* fix linting

* fix linting

* fix bedrock test_get_request_body_stability3

* test_custom_llm_provider_entrypoint
2025-10-29 13:10:51 -07:00
Alexsander Hamir
4939793ade
fix: prevent httpx DeprecationWarning memory leak in AsyncHTTPHandler (#16024)
* fix: prevent httpx DeprecationWarning memory leak in AsyncHTTPHandler

Route bytes/str to content= parameter instead of data= to avoid deprecation warning that causes memory leak

* refactor: extract data/content preparation into helper function

Create _prepare_request_data_and_content() helper to DRY up the logic
for routing data/content parameters correctly in httpx requests.

This helper prevents httpx DeprecationWarnings (which cause memory leaks)
by moving bytes/str from data= to content= parameter while keeping
dict/Mapping in data= parameter.

Applied the helper consistently across all HTTP methods in both
AsyncHTTPHandler and HTTPHandler classes:
- post(), put(), patch(), delete()
- single_connection_post_request()

Related to: b850ed1188

* fix: Python 3.8 compatibility - use Tuple instead of tuple in type hints

Replace lowercase tuple[...] with typing.Tuple[...] in http_handler.py
to fix 'TypeError: type object is not subscriptable' on Python 3.8
2025-10-29 12:57:41 -07:00
langping
5bba1e8405
Added fallback logic for detecting file content-type when S3 returns generic (#15635)
* enhance image processing fallback logic

* Extract to comment utils
2025-10-29 08:24:20 -07:00
Dmitrii Komarov
1dfdcb0762
Allow using ARNs when generation images via Bedrock (#15789)
* Use model_id in Bedrock's image_handler

* Fix MyPy for converse_handler and invoke_hanlder
2025-10-28 19:41:35 -07:00
Albert DeFusco
559ae96e38
Python entry-point for CustomLLM subclasses (#15881)
* load entrypoints

* mock loading entry-point in pyproject.toml

* simpler group name

* create CustomLLM subclass instance after load
2025-10-28 19:39:14 -07:00