* feat(internal_user_endpoints.py): allow passing team-specific details when creating new user with team
allows assigning users to teams on creation
* feat(internal_user_endpoints.py): support for setting team values in default internal user params
Ensures admin can assign all new users to a default team
* test(test_internal_user_endpoints.py): add unit tests
* feat(ui/): allow setting default team on admin UI
easier onboarding for users
* fix(internal_user_endpoints.py): fix exception check
* fix(spend_tracking_utils.py): add user agent tags from standard logging payload, in spend logs payload
* feat(litellm_logging.py): identify user agent tags as `User-Agent: ..` and allow admin to disable storing user agent as tag
* fix(azure_ai/): pass content type header in azure ai request
Fixes https://github.com/BerriAI/litellm/issues/11227
* test: add unit test
* fix(router.py): fix passing dynamic credentials to retrieve batch
Fixes batch retrieval when using router
* test: add more unit tests
* fix SCIM memberships Patch
* fixes for SCIM updates
* fixes for SCIM
* working provisioning for teams on SCIM
* working user patch / PUT ops SCIM
* fixes SCIM
* test_scim_v2_endpoints.py
* handle_existing_user_by_email
* fixes for provisioning SCIMUser
* fixes SCIM provisioning
* test scim v2
* fixes for linting
* fix _apply_patch_ops
* fixes code QA check for team membership checks
* use cost_per_request
* fix cost_per_request
* fixes cost_per_request
* fixes for cost per request for pass through
* ui fix param name
* fixes for _set_cost_per_request
* test cost per request pass through endpoints
* Update tests/test_litellm/proxy/pass_through_endpoints/test_pass_through_endpoints.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* tests pass through endpoints
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: add summarize parameter to /spend/logs endpoint for individual transaction logs
- Introduced a new `summarize` parameter to control data format when querying spend logs.
- `summarize=true` (default) returns aggregated data, while `summarize=false` provides individual transaction logs.
- Updated documentation and added tests to validate the new functionality.
* fix: wrap boto3.Session() with tracer for observability - Add tracer.trace wrapper around boto3.Session() call in _get_aws_region_name method - Ensures all boto3 initializations in base_aws_llm.py are properly instrumented - Fixes test_boto3_init_tracer_wrapping test failure - Maintains consistency with other boto3 calls in the same file
* fix(litellm_pre_call_utils.py): add user agent tags to spend logs in standard logging payload logic
avoid clash when tag based routing is enabled
* test: remove redundant test
* test: rename oidc test to run earlier
quicker debuging
* fix(azure.py): return more detailed error message
* fix(azure/common_utils.py): use default scope, if scope is none
fixes oidc test
* fix: always default to cognitiveservices.azure.com
* test: update test
* feat: Add health check functionality and endpoints
- Introduced methods for saving health check results to the database, including validation and cleaning of data.
- Added new health check endpoints to retrieve health check history and latest health statuses for models.
- Updated model prices and context window configuration for new Azure transcription models.
* test: Add unit tests for health check functionality
- Introduced tests for PrismaClient health check methods, including saving results and retrieving health check history.
- Added tests for the _save_health_check_to_db function to ensure proper handling of healthy and unhealthy endpoints.
- Implemented mock objects to simulate database interactions and validate method behaviors.
* Refactor health endpoint model ID handling and improve logging
- Updated health endpoint to use `get_deployment` for retrieving model names based on model IDs, enhancing error handling for missing models.
- Changed health check result saving to the database to be non-blocking by using `asyncio.create_task`.
- Cleaned up code for better readability and maintainability.
* Refactor utility functions in proxy module for improved readability and error handling
- Removed unused imports and simplified exception handling in `_get_redoc_url` and `_get_docs_url` functions to manage circular imports.
- Cleaned up logging statements for consistency and clarity.
- Streamlined error message formatting in `handle_exception_on_proxy` function.
* Enhance type hinting and default values in ProxyUpdateSpend class for improved clarity and robustness
- Added type hints for `_end_user_list_transactions` to specify it as a dictionary mapping end user IDs to spend amounts.
- Updated default values for optional fields in `SpendLogsPayload` to ensure they are initialized properly, enhancing error handling.
- Refactored `_premium_user_check` function to improve model validation logic and error handling.
* Fix disable_spend_updates method to handle None return value gracefully
- Updated the disable_spend_updates method to return False if the environment variable DISABLE_SPEND_UPDATES is not set or is None, improving robustness in configuration handling.
* Refactor join_paths function in utils.py for improved path handling
- Enhanced the join_paths function to better manage leading and trailing slashes, ensuring correct path concatenation.
- Added logic to handle cases where either base_path or route is empty, improving robustness and usability.
* Enhance health check functionality and improve error handling
- Introduced a new method `_save_health_check_to_db` for saving health check results to the database, utilizing safe JSON functions for data integrity.
- Refactored existing health check methods to streamline the process and improve error logging.
- Updated email sending logic to ensure secure connections and better error handling.
- Improved spend update logic with batch processing and retry mechanisms for database operations.
- Added utility functions for projected spend calculations and enhanced validation for team configurations.
* Add health check methods for database interaction
- Introduced `save_health_check_result` method to save health check results with detailed logging and validation.
- Added `get_health_check_history` method for retrieving health check records with optional filtering.
- Implemented `get_all_latest_health_checks` method to fetch the latest health checks for each model.
- Enhanced error handling and logging for all new methods to improve reliability and traceability.
* Refactor health check result saving to use typed arguments
- Updated the `_save_health_check_to_db` function to call `save_health_check_result` with explicitly typed arguments instead of a dictionary spread, enhancing code clarity and type safety.
- Removed unused method bindings in the mock Prisma client tests to streamline the test setup.
* Remove unused `_save_health_check_to_db` function from utils.py to streamline code and improve maintainability.
* Implement response time validation and details cleaning in health check result saving
- Added `_validate_response_time` method to ensure response time values are valid and handle exceptions gracefully.
- Introduced `_clean_details` method to validate and clean details JSON, improving data integrity.
- Refactored `save_health_check_result` to utilize these new methods for optional fields, enhancing code clarity and maintainability.
- Updated tests to bind new methods to the mock Prisma client for comprehensive testing.
* Add health check utility functions and refactor existing endpoints
- Introduced `_convert_health_check_to_dict` to standardize health check record conversion to dictionary format for JSON responses.
- Added `_check_prisma_client` helper function to streamline database availability checks and improve error handling.
- Refactored health check endpoints to utilize the new utility functions, enhancing code clarity and maintainability.
* Refactor health check tests for improved clarity and coverage
- Simplified the mock PrismaClient setup by consolidating method bindings.
- Updated health check result saving tests to use parameterized scenarios for better coverage.
- Added tests for health check history retrieval and graceful handling when no database client is provided.
- Removed redundant mock functions to streamline the test suite.
* Implement helper function for health check and database saving
- Added `_perform_health_check_and_save` to encapsulate health check execution and optional database saving.
- Refactored health endpoint logic to utilize the new helper function, improving code clarity and reducing redundancy.
- Enhanced error handling and streamlined the process of saving health check results to the database.
* refactor: rename target_model parameter to model in health check function
* fix(handle_jwt.py): check user object, if jwt user is proxy admin
correctly return user role - if jwt user has role updated in UI
* test(test_handle_jwt.py): add unit test for passing correct user role
* feat(model_info_view.tsx): separate UI component for updating edit model component
* feat(model_info_view.tsx): allow updating model access group on UI
show all available access groups in ui component
* docs: minor fixes
* feat(ui_sso.py): allow admin to specify additional headers for sso provider
some sso providers require special headers to return a json response
* test(test_ui_sso.py): add unit tests to ensure custom headers are respect3ed
* docs(config_settings.md): document new header param
* fix(litellm_pre_call_utils.py): add spend tag tracking by user agent
allows checking spend for cli tools like claude code
* feat(litellm_pre_call_utils.py): track spend by user agent part if user agent contains "/"
allows tracking spend across user agent versions
Better cost tracking for claude cod
* test(test_litellm_pre_call_utils.py): add testing for pre call utils, user agent parsing
* fix: fix linting check
* feat: update the feature of ollama_embeddings to work on a sync api
Signed-off-by: Abinand P <abinand0911@gmail.com>
* lint:fixing of the lint file
Signed-off-by: Abinand P <abinand0911@gmail.com>
* fix:test
Signed-off-by: Abinand P <abinand0911@gmail.com>
* chore: added test for ollama embedding and refactored handler
Signed-off-by: Abinand P <abinand0911@gmail.com>
* fix:lint error
Signed-off-by: Abinand P <abinand0911@gmail.com>
---------
Signed-off-by: Abinand P <abinand0911@gmail.com>
* fix(prometheus.py): remove request increment from inside the log success event
it's only done on post-call success/failure
* fix(litellm_logging.py): add additional validation step for checking if 'stream' is true
prevent double counting on non-stream requests
* test: add unit testing to ensure stream is not incorrectly set to true
* feat(litellm_logging.py): emit request route in standard logging payload
used by prometheus streaming metrics for route
* fix: fix otel test
* fix: fix linting errors
* test: update test
* fix: fix linting error
* feat: Add health check functionality and endpoints
- Introduced methods for saving health check results to the database, including validation and cleaning of data.
- Added new health check endpoints to retrieve health check history and latest health statuses for models.
- Updated model prices and context window configuration for new Azure transcription models.
* test: Add unit tests for health check functionality
- Introduced tests for PrismaClient health check methods, including saving results and retrieving health check history.
- Added tests for the _save_health_check_to_db function to ensure proper handling of healthy and unhealthy endpoints.
- Implemented mock objects to simulate database interactions and validate method behaviors.
* Refactor health endpoint model ID handling and improve logging
- Updated health endpoint to use `get_deployment` for retrieving model names based on model IDs, enhancing error handling for missing models.
- Changed health check result saving to the database to be non-blocking by using `asyncio.create_task`.
- Cleaned up code for better readability and maintainability.
* Refactor utility functions in proxy module for improved readability and error handling
- Removed unused imports and simplified exception handling in `_get_redoc_url` and `_get_docs_url` functions to manage circular imports.
- Cleaned up logging statements for consistency and clarity.
- Streamlined error message formatting in `handle_exception_on_proxy` function.
* Enhance type hinting and default values in ProxyUpdateSpend class for improved clarity and robustness
- Added type hints for `_end_user_list_transactions` to specify it as a dictionary mapping end user IDs to spend amounts.
- Updated default values for optional fields in `SpendLogsPayload` to ensure they are initialized properly, enhancing error handling.
- Refactored `_premium_user_check` function to improve model validation logic and error handling.
* Fix disable_spend_updates method to handle None return value gracefully
- Updated the disable_spend_updates method to return False if the environment variable DISABLE_SPEND_UPDATES is not set or is None, improving robustness in configuration handling.
* Refactor join_paths function in utils.py for improved path handling
- Enhanced the join_paths function to better manage leading and trailing slashes, ensuring correct path concatenation.
- Added logic to handle cases where either base_path or route is empty, improving robustness and usability.
* Enhance health check functionality and improve error handling
- Introduced a new method `_save_health_check_to_db` for saving health check results to the database, utilizing safe JSON functions for data integrity.
- Refactored existing health check methods to streamline the process and improve error logging.
- Updated email sending logic to ensure secure connections and better error handling.
- Improved spend update logic with batch processing and retry mechanisms for database operations.
- Added utility functions for projected spend calculations and enhanced validation for team configurations.
* Add health check methods for database interaction
- Introduced `save_health_check_result` method to save health check results with detailed logging and validation.
- Added `get_health_check_history` method for retrieving health check records with optional filtering.
- Implemented `get_all_latest_health_checks` method to fetch the latest health checks for each model.
- Enhanced error handling and logging for all new methods to improve reliability and traceability.
* Refactor health check result saving to use typed arguments
- Updated the `_save_health_check_to_db` function to call `save_health_check_result` with explicitly typed arguments instead of a dictionary spread, enhancing code clarity and type safety.
- Removed unused method bindings in the mock Prisma client tests to streamline the test setup.
* Remove unused `_save_health_check_to_db` function from utils.py to streamline code and improve maintainability.
* Implement response time validation and details cleaning in health check result saving
- Added `_validate_response_time` method to ensure response time values are valid and handle exceptions gracefully.
- Introduced `_clean_details` method to validate and clean details JSON, improving data integrity.
- Refactored `save_health_check_result` to utilize these new methods for optional fields, enhancing code clarity and maintainability.
- Updated tests to bind new methods to the mock Prisma client for comprehensive testing.
* Add health check utility functions and refactor existing endpoints
- Introduced `_convert_health_check_to_dict` to standardize health check record conversion to dictionary format for JSON responses.
- Added `_check_prisma_client` helper function to streamline database availability checks and improve error handling.
- Refactored health check endpoints to utilize the new utility functions, enhancing code clarity and maintainability.
* Refactor health check tests for improved clarity and coverage
- Simplified the mock PrismaClient setup by consolidating method bindings.
- Updated health check result saving tests to use parameterized scenarios for better coverage.
- Added tests for health check history retrieval and graceful handling when no database client is provided.
- Removed redundant mock functions to streamline the test suite.
* Implement helper function for health check and database saving
- Added `_perform_health_check_and_save` to encapsulate health check execution and optional database saving.
- Refactored health endpoint logic to utilize the new helper function, improving code clarity and reducing redundancy.
- Enhanced error handling and streamlined the process of saving health check results to the database.
* docs(index.md): add stable pip package
* fix(anthropic/chat/transformation.py): add 'none' tool choice mapping
Allows disabling anthropic tool calling
Maintain parity
* fix(transformation.py): if tool_choice="none" ignore 'disable_parallel_Tool_use'
unsupported param from anthropic - makes sense as the 'none' implies no tool calls are being made
* fix(anthropic/chat/transformation.py): append prefix to start of assistant response, if set
ensures assistant response contains complete response
* fix(anthropic/chat/transformation.py): add flag to allow user to opt out of enabling prefix in prompt
* fix(anthropic/chat/transformation.py): working e2e support for prefix prompt in assistant response
* feat(networking.tsx): always include model access groups on UI
show admin created access groups when giving key/user/team model permissions
* feat(add_model_tab.tsx): initial ui component for adding to an existing model access group
allows user to add model to an access group (simplify giving users/keys/teams model access)
* feat(proxy_server.py): add 'only_model_access_groups' flag support to `/v1/models`
simplifies listing available access groups on UI
* test: add e2e test for new only_model_access_groups param
* feat(add_model_tab.tsx): allow adding+viewing model access groups on models tab
make feature functional on UI
* feat(view_users.tsx): route edit user to user info page
more detailed user edit
* feat(columns.tsx): route edit user to user info page
more detailed user edit
* fix(columns.tsx): fix linting error
* build(ui/): fix linting errors
* feat(anthropic/passthrough): pass dynamic api key/api base params to litellm.completion
allows calls to work with config.yaml
* fix(responses_api/transformation): fix passing dynamic params to responses api from .completion()
Allows responses api to work with config.yaml
* fix(langfuse.py): fix responses api usage logging to langfuse
* refactor(litellm_logging.py): add more generic solution for responses api usage logging
ensures it works across all logging integrations
* fix(litellm_logging.py): patch for anthropic messages not returning a pydantic object
it should ideally return a pydantic object, which would simplify checks and reduce errors
* fix(handler.py): correctly bubble up empty choices errors to litellm.completion
causes downstream errors as it is expected there is at least one choice set
* feat(litellm_logging.py): prevent double logging litellm responses
ensures accurate spend tracking for calls when bridges are used
* fix(litellm_logging.py): ensure logging is consistently enforced across all call types
* fix: patch - set calltype before entering bridge api
ensures logging object is applying the correct logic on the event hooks
* fix(types/router.py): loosen type hint for mock response
* change space_key header to space_id for Arize (#11595)
* feat(schema): add additional indexes to LiteLLM_SpendLogs for improved query performance (#11675)
* Revert "feat(schema): add additional indexes to LiteLLM_SpendLogs for improve…" (#11683)
This reverts commit 2a7f113fde.
* [Feat] Use dedicated Rest endpoints for list, calling MCP tools (#11684)
* fix: (fix) use specific rest endpoints for MCP
* ui - use rest mcp endpoints
* fix imports
* docs DISABLE_AIOHTTP_TRUST_ENV
* docs(caching.md): remove batch redis get recommendation - old code path, no longer necessary
* fix(vertex_and_google_ai_studio_gemini.py): handle gemini not passing audio token usage data
* Chat Completions <-> Responses API Bridge Improvements (#11685)
* feat(anthropic/passthrough): pass dynamic api key/api base params to litellm.completion
allows calls to work with config.yaml
* fix(responses_api/transformation): fix passing dynamic params to responses api from .completion()
Allows responses api to work with config.yaml
* fix(langfuse.py): fix responses api usage logging to langfuse
* refactor(litellm_logging.py): add more generic solution for responses api usage logging
ensures it works across all logging integrations
* fix(litellm_logging.py): patch for anthropic messages not returning a pydantic object
it should ideally return a pydantic object, which would simplify checks and reduce errors
* fix(handler.py): correctly bubble up empty choices errors to litellm.completion
causes downstream errors as it is expected there is at least one choice set
* fix(response_metadata.py): allow model_info to be none
* fix(litellm_logging.py): copy object before mutating
* fix: fix lint check
* fix: fix linting error
* fix: fix linting error
---------
Co-authored-by: vanities <mischkeaa@gmail.com>
Co-authored-by: Cole McIntosh <82463175+colesmcintosh@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* feat(anthropic/passthrough): pass dynamic api key/api base params to litellm.completion
allows calls to work with config.yaml
* fix(responses_api/transformation): fix passing dynamic params to responses api from .completion()
Allows responses api to work with config.yaml
* fix(langfuse.py): fix responses api usage logging to langfuse
* refactor(litellm_logging.py): add more generic solution for responses api usage logging
ensures it works across all logging integrations
* fix(litellm_logging.py): patch for anthropic messages not returning a pydantic object
it should ideally return a pydantic object, which would simplify checks and reduce errors
* fix(handler.py): correctly bubble up empty choices errors to litellm.completion
causes downstream errors as it is expected there is at least one choice set
* fix(response_metadata.py): allow model_info to be none
* fix(litellm_logging.py): copy object before mutating
* fix: fix lint check
- Updated the `_add_reasoning_system_prompt_if_needed` method to maintain the original format of list content when prepending the reasoning prompt.
- Adjusted tests to verify that both string and list content types are correctly handled, ensuring the reasoning prompt is added without altering the content structure.
- Updated the `_add_reasoning_system_prompt_if_needed` method to convert list content to strings before prepending the reasoning prompt.
- Adjusted tests to verify that system messages with list content are correctly transformed into strings, ensuring original content is preserved.
- Revised the reasoning support indicators in the Mistral model documentation for clarity.
- Improved the `_add_reasoning_system_prompt_if_needed` method to handle both string and list content types for system messages, ensuring the reasoning prompt is correctly prepended.
- Added a new test case to verify the functionality of adding the reasoning system prompt when the existing content is a list.
* fix(utils.py): convert stringified numbers to numbers
Closes https://github.com/BerriAI/litellm/issues/11266
* fix(convert_dict_to_model_response_object/): bubble up azure content_filter_results
* fix: fix linting error
* fix: fix linting errors
* fix(types/utils.py): ensure choices is correctly set
* fix: delete field if not set
* fix: expand scope of choicelogprobs value
* feat(parallel_request_limiter_v3.py): allows admin to enforce token rate limit based on just output tokens
Useful when trying to rate limit for primarily self hosted model use-cases
* test(test_parallel_request_limiter_v3.py): add unit test for token rate limit type
* feat(parallel_request_limiter_v3.py): return remaining token limits in header
* feat: return rate limit headers in response
* feat(parallel_request_limiter_v3.py): working rate limit response headers
* feat(parallel_request_limiter_v3.py): fix rate limit tracking for tpm when rpm also set
* feat(parallel_request_limiter_v3.py): show headers for key/user/team
* feat(parallel_request_limiter_v3.py): decrement max parallel request limiter on failure event
* feat(parallel_request_limiter_v3.py): add in-memory cache implementation of parallel request rate limiter
allows rate limiter to work even without redis cache setup
Work for GA of parallel request limiter v3
* refactor(proxy/hooks/__init__.py): replace with new parallel request handler
* test: update testing
* fix: fix ruff check
* fix: revert ga of multi instance rate limiting - needs more work to pass testing
* Added support for reasoning parameters in magistral models, including "reasoning_effort" and "thinking".
* Updated the MistralConfig class to handle reasoning system prompts.
* Implemented tests to verify reasoning functionality and ensure correct parameter mapping for magistral models.
* Enhanced the model prices JSON to reflect new reasoning capabilities.
* Checkpoint before follow-up message
* Add comprehensive tests for Deepgram transcription functionality
* clean up transform
* just use 1 test
* test cleanup
* test fix get_complete_url
* test rename file
* refactor deepgram URL construction
* add logging_obj.pre_call
* fix unused imports
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(internal_user_endpoints.py): support user with `+` in email on user info
ensures user is correctly parsed from input
* fix(factory.py): support vertex function call args as None
handles empty string in args for vertex gemini calls
* docs(langfuse_integration.md): pin langfuse sdk version on docs
* fix(vertex_ai/): return empty dict, instead of none when empty string given
* refactor: reduce function size
* fix: fix linting errors
* fix: revert check
* fix(internal_user_endpoints.py): fix check
* test: update tests
* test: update tests
