* fix: Handle circular references in spend tracking metadata JSON serialization
- Fixes issue #12634 where circular references in metadata caused
ValueError: Circular reference detected when logging spend data
- Adds _safe_json_dumps() function that detects and handles circular
references by replacing them with placeholder strings
- Maintains full functionality for normal objects while preventing
crashes from circular references
- Adds comprehensive tests for circular reference handling
- Critical fix for v1.74.3 stable release
* fix: Replace bare except clauses with specific Exception handling
- Fixes E722 linting errors in _safe_json_dumps function
- Maintains same error handling behavior while following best practices
- All tests continue to pass
* refactor: Use existing safe_dumps utility instead of custom implementation
- Replace custom _safe_json_dumps() with existing safe_dumps() from litellm_core_utils
- Remove duplicate code and leverage existing circular reference handling
- Update tests to use safe_dumps function
- Maintains same functionality while reducing code duplication
- All tests continue to pass
* fix(proxy_server.py): fixes for handling team only models via `/v2/model/info`
ensures team only models show up on the correct team on `Models + Endpoints`
* test: update tests
* fix(router.py): use more descriptive error message
* fix(proxy/_types.py): note `/team/member_update` is a self-managed route
route has it's own logic for rbac - enables team admins to update member permissions
Fixes issue where team admins on UI could not update member permissions
* fix(token_counter.py): move log line to being '.debug' instead of '.error'
Fixes https://github.com/BerriAI/litellm/issues/12269
* (#11794) use upsert for managed object table rather than create to avoid UniqueViolationError
* (#11794) use upsert for managed object table rather than create to avoid UniqueViolationError
* fix(bedrock): auto-generate session name when only aws_role_name is provided
Fixes#12583 - AWS role assumption not working correctly when aws_role_name
is provided without aws_session_name.
Previously, if only aws_role_name was provided in the config without
aws_session_name, the code would fall back to using environment credentials
instead of assuming the specified role. This was problematic in EKS/IRSA
environments where users want to assume a different role.
The fix:
- When aws_role_name is provided without aws_session_name, we now
auto-generate a session name with format 'litellm-session-{timestamp}'
- This ensures role assumption happens as expected
- Added comprehensive test coverage for this scenario
* style: format test file with black
---------
Co-authored-by: openhands <openhands@all-hands.dev>
* fix(anthropic): fix streaming + response_format + tools bug
- Fix _handle_json_mode_chunk to only convert response_format tools to content
- Regular user tools now remain as proper tool_calls in streaming mode
- Add comprehensive test for the fix
- Resolves issue where all tools were incorrectly converted to content chunks
Before: All tools converted to content with different indices
After: Only response_format tool converted, regular tools remain as tool_calls
* fix(anthropic): improve streaming + response_format + tools handling
* fix: lint error (too many statements)
* fix(anthropic): correct finish_reason for streaming response_format tools
- Add ttl parameter to cache_control for Gemini models
- Support Google's TTL format (e.g., '3600s', '7200s')
- Implement robust TTL extraction and validation
- Extract TTL before system message transformation to handle all cases
- Add comprehensive test suite with 17 test cases in tests/test_litellm/
- Update documentation with TTL usage examples
- Maintain backward compatibility with existing cache_control usage
Fixes#9810
* feat(route_checks.py): allow admin to disable proxy management endpoints on instance
useful for preventing multiple instances from doing admin actions
* docs(scaling_multiple_instances.md): add architecture doc on scaling multiple litellm instances
provide guidance on scaling proxy
* docs(scaling_multiple_instances.md): add doc on scaling across multiple regions for litellm
* fix(route_checks.py): allow disabling llm api endpoints on an instance
allows pure admin instance to exist
* refactor(enterprise/route_checks.py): refactor env var checks
* refactor: finish refactoring
* docs(control_plane_and_data_plane.md): refactor docs
* test: update tests
Added cache_read_input_token_cost (25% of regular input cost) to all 39 Gemini 2.x models
to properly support implicit context caching cost calculations. Previously, cached tokens
were being charged at full price instead of the discounted rate.
Fixes#11156
* [Feat] New LLM API Integration - Add Moonshot API (Kimi) (#12551)
* Add Moonshot AI provider support to LiteLLM
Co-authored-by: ishaan <ishaan@berri.ai>
* Refactor Moonshot provider params handling and transformation logic
Co-authored-by: ishaan <ishaan@berri.ai>
* fix constants
* add Moonshot AI
* fix get_supported_openai_params
* handle kimi temp
* add tool choice handling
* test moonshot unit tests
* fix kimi
* fix linting
* Add pricing information for Moonshot AI's kimi-k2 model (#12566)
* Add pricing information for Moonshot AI's kimi-k2 model
* Update model name to kimi-k2-0711-preview
- Changed model name from moonshot/kimi-k2 to moonshot/kimi-k2-0711-preview
- This reflects the specific model version as requested
* Update moonshot_models list to match model_context JSON
---------
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* update docs
* docs moonshot
* fixes model cost map
* fix map_openai_params
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* just use 1 param for mcp groups
* fix just use 1 param for access groups
* test_get_tools_from_mcp_servers
* docs access groups
* group MCPs
* test fix
* fix screenshots on docs
* TestMCPAccessGroupsE2E
* update img
* fix MCP connect
* Get the basics of the integration working.
* Cleanup bytez integration.
* Update user agent for Bytez integration.
* Use the config class directly. Create the start of the docs.
* Finish up bytez documentation. Include a provider integration guide.
* Fix typing bug in custom_logger_utils. Add tests for bytez integration.
* Add token tracking for model usage for Bytez integration.
* Create a units test for the Bytez config.
* Make changes to Bytez transformation code per PR feedback.
* Cleanup coment in Bytez transformation test.
* Remove LRU usage for bytez integration.
* Consolidate Bytez tests into a single file. Conform to project structure for tests.
* Fix linting error with Bytez impl.
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* mcp acces group init
* add ability to change server display on ui through access groups
* Mcp access group names UI (#12486)
* Added ui changes to reflect mcp_access_groups
* fix edit mcp page
* change to string array (#12491)
* change to string array
* Remove print
* add ability to change server display on ui through access groups
* Litellm mcp access groups accesses (#12498)
* added mcp access groups for keys and teams
* added access groups above servers
* fixed ruff
* fixed mypy
* revert couple changes
* fix test
* fixed double asterisks
* Litellm mcp groups UI (#12522)
* add ui for teams
* fix object permissions
* fix mcp servers test object permission
* remove print
* add helper method
* add tests + remove logs
* add mcp access group servers to test key
* add mcp access group support for headers
* lint fix
* add tests and helper function
* fixed test
* change list -> List
* tests
* fix(proxy_setting_endpoints.py): require store model in db is enabled for setting user default settings
* test(test_proxy_server.py): update test
* fix(reset_budget_job.py): initial commit adding reset budget logic for team members
* test: update unit testing
* test(test_proxy_budget_reset.py): validate team member budget was reset
* test(test_reset_budget_job.py): update unit tests
* test: update tests
* fix(internal_user_endpoints.py): initial commit removing logic to create new budget for new user if default max budget in team set
* feat(proxy_setting_endpoints.py): update team member budget when set via default internal user endpoint
removes need to create a unique budget per user
* feat(proxy_server.py): set team max member budget on startup, if set on config.yaml
* fix(prometheus.py): support custom tags for tracking on prometheus
Allows tracking user agent values on prometheus metrics
* test(test_internal_user_endpoints.py): fix test
* update types
* add STDIO to client
* allow using STDIO with MCP manager
* add Stdio-specific fields to schema.prisma
* fixes for MCP mgmt
* fix for adding stdio MCP to DB
* ui - allow adding stdio MCPs
* fix MCP server manager
* docs stdio MCP
* add_stdio_mcp.png
* new stdio tests
* allow adding MCPs through config.yaml
* fix tool test panel
* use TestMCPClient
* ui fixes for testing circle ci mcp
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* mcp acces group init
* add ability to change server display on ui through access groups
* Mcp access group names UI (#12486)
* Added ui changes to reflect mcp_access_groups
* fix edit mcp page
* change to string array (#12491)
* change to string array
* Remove print
* add ability to change server display on ui through access groups
* Litellm mcp access groups accesses (#12498)
* added mcp access groups for keys and teams
* added access groups above servers
* fixed ruff
* fixed mypy
* revert couple changes
* fix test
* fixed double asterisks
* Litellm mcp groups UI (#12522)
* add ui for teams
* fix object permissions
* fix mcp servers test object permission
* remove print
* add helper method
* add tests + remove logs
* added mcp tools on internal user and divide it by teams
* add support for server api call
* Added frontend for test key
* added tools used output
* fix ui for servers
* All servers to personal
* change columns format
* revert ui logic
* Added vertical align
* fix mapped tests
* fix lint
* fix lint
* remove extra file
* fix ui test
* comments fixes
* change query type
* change query type
* mcp acces group init
* add ability to change server display on ui through access groups
* Mcp access group names UI (#12486)
* Added ui changes to reflect mcp_access_groups
* fix edit mcp page
* change to string array (#12491)
* change to string array
* Remove print
* add ability to change server display on ui through access groups
* Litellm mcp access groups accesses (#12498)
* added mcp access groups for keys and teams
* added access groups above servers
* fixed ruff
* fixed mypy
* revert couple changes
* fix test
* fixed double asterisks
* feat(custom_guardrail.py): allow admin to run guardrails on specific modes based on request tags
Enables pre-call checks for openwebui but logging for claude code guardrails
* refactor(custom_guardrail.py): move tag based mode to enterprise
* test(test_custom_guardrail.py): update test
* feat(quick_start.md): add doc on tag-based guardrail mode
* docs(quick_start.md): cleanup doc
* docs: cleanup
* docs(openweb_ui.md): add tutorial to doc
* fix(custom_guardrail.py): ensure user is premium user
* test: update test
* fix(guardrails_ai.py): initial commit adding pre-call hook support for guardrails ai
enables running user input through guardrails ai - if set
* feat(guardrails_ai.py): working pre call guardrail
enables pii detection to work via guardrails ai
* feat(guardrails_ai.py): support logging hook
enables masking input via guardrails ai on logging integrations
* test(test_guardrails_ai.py): add unit test for new input processing function
