* docs(deploy.md): move docker recommendation to `main-stable`
* feat(enterprise/internal_user_endpoints.py): expose endpoint for checking available premium users
* feat(usage_indictor.tsx): add new element to help track remaining premium users
* feat(usage_indicator.tsx): show premium user remaining usage
allows users with user caps to know how much is left
* fix(vertex_and_google_ai_studio_gemini.py): bubble up stream is not finished, even if stop reason is given
prevents early completion of stream
Closes https://github.com/BerriAI/litellm/issues/11549
* fix(streaming_handler.py): respect is_finished = False in hidden params
internal logic for preventing ending stream early
* fix(litellm_license.py): add function to check if user is over limit
* fix(internal_user_endpoints.py): add function to check if user is over limit
* refactor: move test
* docs(customer_endpoints.py): document new param
* fix: fixes for transfer encoding error on aiohttp transport
* Update tests/test_litellm/llms/custom_httpx/test_aiohttp_transport.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix(streaming_handler.py): maintain same 'created' across all chunks
Fixes https://github.com/BerriAI/litellm/issues/11437
* test: add unit test to ensure created is always the same across all chunks
* fix(types/utils.py): set a tool call id, if missing in delta tool call
Ensures stream chunk builder can reconstruct tool calls correctly
Fixes https://github.com/BerriAI/litellm/issues/11262
* fix(responses/transformation.py): support passing mcp server tool call to anthropic
allows switching between openai and anthropic for mcp tool calling
* fix(ollama/chat/transformation.py): set tool call id's when missing
* fix(onboarding_link.tsx): fix adding ui/invitation id
* fix(onboarding_link.tsx): update invitation link function to handle w/ and w/out custom server path cases
* fix(model_checks.py): ensure team only models returned when all proxy models set for team
* feat(anthropic/): initial commit adding working mcp tool call support
pass in mcp tool via `tools` and litellm will handle translating it to the right anthropic param
* feat(anthropic/): map openai mcp tool to anthropic mcp tool
allows usage within responses api
* fix(databricks/transformation.py): fix databricks linting error
* test(test_anthropic_chat_transformation.py): fix test
* test: update test
* fix(anthropic/chat/transformation.py): add dummy tool call
* refactor: comment out circuit breaker
causes incorrect rate limiting in high traffic
* fix(base_routing_strategy.py): don't reset value if redis val is lower than current in-memory value
Fixes issue where redis might be trailing in-memory value
* fix(parallel_request_limiter_v2.py): if in-memory higher than redis, don't reset value; add previous slot keys to redis increment to correctly 'get' them
* fix(parallel_request_limiter_v3.py): v3 implementation of parallel request limiter
does not use background redis syncing - increments redis in call
simplify rate limiting logic, to improve accuracy
* fix: fix ruff errors
* fix(parallel_request_limiter_v3.py): don't decrement limit on post call success - causes double decrements
* fix(parallel_request_limiter_v3.py): working accurate multi-instance logic
ensured just 100 requests allowed on 100 users, 10 ramp up, 100 rpm limit key, 2 instances
* fix(parallel_request_limiter_v3.py): working accurate rate limiting with time window resets
allows rate limiting to work across multiple windows
* test: add unit tests for v3 rate limiter
* fix(parallel_request_limiter_v3.py): return window value into in-memory cache
allows in-memory cache checks to be used correctly
* refactor(parallel_request_limiter_v3.py): refactor rate limiting to work for multiple window/counter key pairs
enables using for user/team/model rate limiting
* feat(parallel_request_limiter_v3.py): working rate limiting, across key/user/team/end-user
* fix(parallel_request_limiter_v3.py): add model specific rate limiting
* fix(parallel_request_limiter_v3.py): ignore if no rate limits set
skip unecessary rate limit checks - if no limits set
* fix(parallel_request_limiter_v3.py): initial commit bringing token rate limits back
* fix(parallel_request_limiter_v3.py): increment by value in list + update assertions to handle tokens + max parallel requests
* test(parallel_request_limiter_v3.py): more testing
* fix(parallel_request_limiter.py): working in-memory cache limiter
* fix(redis_cache.py): ignore linting error - use safe hasattr
* fix(parallel_request_limiter_v3.py): fix linting error
* refactor: remove redundant parallel_Request_limiter_v2.py
old / inaccurate implementation
* test: update tests
* style: cleanup
* test: update test
* docs(config_settings.md): document new env var
* test(test_base_routing_strategy.py): update test
* Enhance proxy CLI with Rich formatting and improved user experience
- Integrated Rich library for better console output in `proxy_cli.py`, including version display, health check results, and test completion responses.
- Updated health check and test completion methods to provide progress indicators and formatted tables.
- Refactored feedback display in `proxy_server.py` to use Rich for a more visually appealing user interface.
- Adjusted tests in `test_proxy_cli.py` to mock console output instead of using print statements, ensuring compatibility with Rich formatting.
* fix linting error
* refactor(proxy_cli.py): simplify DB setup logging
- Removed progress indicators for IAM token generation and environment variable decryption to simplify the code.
- Consolidated the logic for generating the database URL and setting environment variables.
- Enhanced error handling for configuration loading and database setup, ensuring clearer feedback
* Update test-linting workflow to include proxy-dev dependencies in Poetry installation
* Enhance proxy server initialization with Rich console for improved model display. Added support for loading model parameters from environment variables and refined provider identification logic. Fallback to original print formatting if Rich is not available.
* Refactor feedback handling: Moved feedback message generation and custom warning display to utils.py. Enhanced feedback box with rich formatting and fallback to ASCII for environments without rich. Cleaned up proxy_server.py by removing obsolete code.
* fix linting error
* Refactor model initialization display: Moved model initialization logic to a new utility function `display_model_initialization` for improved readability and maintainability. Enhanced model provider extraction with a dedicated function. Fallback to basic logging if Rich console is unavailable.
* Refactor model provider extraction: Replace the `_extract_provider_from_model` function with a more robust approach using `get_llm_provider`. Implement fallback logic for provider identification and improve error handling. Ensure compatibility with Rich console for model initialization display.
* Refactor get_end_user_id_from_request_body to support user ID retrieval from custom headers and multiple request body formats. Enhance tests to cover various scenarios including header precedence and fallback mechanisms.
* Refactor get_end_user_id_from_request_body function to accept request_body as the first parameter, improving clarity and flexibility. Update tests for compatibility and add new cases to ensure correct functionality across various request body formats.
* Update _user_api_key_auth_builder and user_api_key_auth to pass request object to get_end_user_id_from_request_body, enhancing user ID retrieval from request data.
* refactor(auth_utils.py): update get_end_user_id_from_request_body to accept request_headers instead of request, and adjust related function calls in user_api_key_auth and tests
* refactor(tests): update mock request handling in LLM pass-through endpoint tests
- Replaced the Request object with a Mock for better flexibility in testing.
- Enhanced mock setup to include user API key handling and virtual key retrieval.
- Updated test calls to reflect changes in mock request structure and added necessary patches for new dependencies.
* refactor(vertex_and_google_ai_studio_gemini.py): remove redundant variable declaration for url_context_metadata, linting error
* Handle file content type transformation in responses api (#11310)
* Handle file content type transformation in responses api
* change to use input_file
* -
* TestLiteLLMCompletionResponsesConfig
* test: TestLiteLLMCompletionResponsesConfig
* fix: fix linting
---------
Co-authored-by: Jayme Gordon <jayme_gordon@icloud.com>
* fix(vertex_and_google_ai_studio_gemini.py): add web search request tracking
Enables cost calculation for google web search
* fix(vertex_and_gemini): use common processing logic across stream / non-stream calls
* fix(vertex_And_google_ai_studio_Gemini.py): fix initial choice
* fix: fix linting error
* fix: add initial support for google search cost tracking
* fix(tool_call_cost_tracking.py): working tool cost tracking for gemini
* fix(vertex_ai/gemini/cost_calculator.py): add google web search tool cost tracking for vertex ai
Closes LIT-210
* fix: fix check
* build(model_prices_and_context_window.json): fix amazon nova max output tokens
Closes https://github.com/BerriAI/litellm/issues/11441
* fix: fix ruff check
* fix: using litellm with claude code bedrock
* fix: usage for bedrock with /messages
* fix: bedrock_sse_wrapper
* tests: test for test_chunk_parser_usage_transformation
* test fix
* fix(huggingface): use get() instead of pop() for input_type parameter
Fixes embedding generation for HuggingFace models where input_type override
is required (e.g. BAAI/bge-m3). The pop() method was mutating optional_params
and removing input_type before downstream functions could access it.
* Add unit tests to catch regression
* Move tests around
- Renamed SSOSettingsResponse to inherit from a new base class SettingsResponse for better structure.
- Introduced InternalUserSettingsResponse and DefaultTeamSettingsResponse models for internal user and default team settings.
- Updated endpoint responses to use field_schema instead of schema for consistency.
- Enhanced test cases to validate the new response structure and ensure proper functionality of SSO settings.
* fix(anthropic/): fix regression when passing file url's to the 'file_id' parameter
add test and ensure anthropic file url's are correctly sent as 'document' blocks
* fix(vertex_and_google_ai_studio.py): Use same usage calculation function as non-streaming
Closes https://github.com/BerriAI/litellm/issues/10667
* test(test_vertex_and_google_ai_studio_gemini.py): update test
* fix(ui_sso.py): update user as proxy admin in db table, when checking for proxy_admin_id
Fixes issue where existing internal user, unable to make calls when set as proxy admin id
* fix(utils.py): fix custom base path
* Fixes issue with team_endpoints on member budget update
* refactored location of budget membership fix
* added test for _upsert_budget_membership func
* fix(proxy_server.py): working swagger on custom base
removes the swagger monkey patch - this seems to render the swagger on custom base paths
* fix(ui/): working custom auth uptil login success event
* fix(ui/): working custom server root path for login
* fix(proxy_server.py): create typed dict for ui returned token
allows better documentation of expected params
* refactor(proxy_server.py): refactor all ui login endpoints to use same returned ui token object
* feat(ui_sso.py): add server root path to ui token
* feat(ui_sso.py): allows ui to call correct endpoint
* fix(networking.tsx): update proxy base url with custom root path
* fix(networking.tsx): handle updating proxy base url for non-local instances
* refactor: remove uneccessary references to proxybaseurl in ui code - reduce potential for errors
* fix: fix linting error
* fix(onboarding_link.tsx): fix onboarding link when custom server path is set
* feat(ui_discovery_endpoints.py): add new public .well-known/ route for litellm ui config
returns the server root path and proxy base url for constructing api calls
* feat(_types.py): add litellm well known config as public route
allows ui to query it
* fix(/_types.py): add .well-known config to as public route
* fix(page.tsx): create pattern for loading in ui config before making network requests
ensures requests are formatted correctly
* fix(page.tsx): call credential endpoint once ui config is loaded
* fix(page.tsx): route correctly to litellm dashboard from new user login
* fix(page.tsx): remove hardcoded `/litellm` for /sso/key/generate request
* fix(proxy_server.py): re-add moderations endpoint
* fix(proxy_server.py): mount __next__ at / and /litellm
allows it to work when proxy is mounted on root
* docs(contributing.md): remove /ui on ui doc - it will now run on root
* docs(custom_root_ui.md): add docs on custom root path
* Add support for DataRobot as a router in LiteLLM
* Updates to logic
* Changes to make things work better
* Capitalize bearer
* Revert change
* Undo and simplify things
* Add basic testing
* Add some extra handling
* More tests
* Lowercase
* Fix
* Comment
* Add local test_completion for datarobot
* Sync repo to main
* Update get_complete_url to accept deployments
* Migrate to OpenAILike
* Mock datarobot in test
* Migrate everything to OpenAI LIke
* Apply fixes and changes from review
* Update docs
* Update env vars
* Move tests
* fix(base_routing_strategy.py): compress increments to redis - reduces write ops
* fix(base_routing_strategy.py): make get and reset in memory keys atomic
* fix(base_routing_strategy.py): don't reset keys - causes discrepency on subsequent requests to instance
* fix(parallel_request_limiter.py): retrieve values of previous slots from cache
more accurate rate limiting with sliding window
* fix: fix test
* fix: fix linting error
* feat: Add Gemini TTS audio parameter support
- Add is_model_gemini_audio_model() method to detect TTS models
- Include 'audio' parameter in supported params for TTS models
- Map OpenAI audio parameter to Gemini speechConfig format
- Add _extract_audio_response_from_parts() method to transform audio
output to openai format
* updated unit-test to use pcm16
* - created typedict for speechconfig
- simplified gemini tts model detection
- moved gemini_tts test to test_litellm
* simplified is_model_gemini_audio_model more
* feat(parallel_request_limiter_v2.py): add sliding window logic
allows rate limiting to work across minutes
* fix(parallel_request_limiter_v2.py): decrement usage on rate limit error
* fix(base_routing_strategy.py): fix merge from redis - preserve values in in-memory cache during gap b/w push to redis and read from redis
* fix(base_routing_strategy.py): catch the delta change during redis sync
ensures values are kept in sync
* fix(parallel_request_limiter_v2.py): update tpm tracking to use slot key logic
* fix: fix linting error
* test: update testing
* test: update tests
* test: skip on rate limit or internal server errors
* test: use pytest fixture instead
* test: bump mistral model
