* fix(bedrock): auto-generate session name when only aws_role_name is provided
Fixes#12583 - AWS role assumption not working correctly when aws_role_name
is provided without aws_session_name.
Previously, if only aws_role_name was provided in the config without
aws_session_name, the code would fall back to using environment credentials
instead of assuming the specified role. This was problematic in EKS/IRSA
environments where users want to assume a different role.
The fix:
- When aws_role_name is provided without aws_session_name, we now
auto-generate a session name with format 'litellm-session-{timestamp}'
- This ensures role assumption happens as expected
- Added comprehensive test coverage for this scenario
* style: format test file with black
---------
Co-authored-by: openhands <openhands@all-hands.dev>
* fix(anthropic): fix streaming + response_format + tools bug
- Fix _handle_json_mode_chunk to only convert response_format tools to content
- Regular user tools now remain as proper tool_calls in streaming mode
- Add comprehensive test for the fix
- Resolves issue where all tools were incorrectly converted to content chunks
Before: All tools converted to content with different indices
After: Only response_format tool converted, regular tools remain as tool_calls
* fix(anthropic): improve streaming + response_format + tools handling
* fix: lint error (too many statements)
* fix(anthropic): correct finish_reason for streaming response_format tools
- Add ttl parameter to cache_control for Gemini models
- Support Google's TTL format (e.g., '3600s', '7200s')
- Implement robust TTL extraction and validation
- Extract TTL before system message transformation to handle all cases
- Add comprehensive test suite with 17 test cases in tests/test_litellm/
- Update documentation with TTL usage examples
- Maintain backward compatibility with existing cache_control usage
Fixes#9810
* [Feat] New LLM API Integration - Add Moonshot API (Kimi) (#12551)
* Add Moonshot AI provider support to LiteLLM
Co-authored-by: ishaan <ishaan@berri.ai>
* Refactor Moonshot provider params handling and transformation logic
Co-authored-by: ishaan <ishaan@berri.ai>
* fix constants
* add Moonshot AI
* fix get_supported_openai_params
* handle kimi temp
* add tool choice handling
* test moonshot unit tests
* fix kimi
* fix linting
* Add pricing information for Moonshot AI's kimi-k2 model (#12566)
* Add pricing information for Moonshot AI's kimi-k2 model
* Update model name to kimi-k2-0711-preview
- Changed model name from moonshot/kimi-k2 to moonshot/kimi-k2-0711-preview
- This reflects the specific model version as requested
* Update moonshot_models list to match model_context JSON
---------
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* update docs
* docs moonshot
* fixes model cost map
* fix map_openai_params
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* Get the basics of the integration working.
* Cleanup bytez integration.
* Update user agent for Bytez integration.
* Use the config class directly. Create the start of the docs.
* Finish up bytez documentation. Include a provider integration guide.
* Fix typing bug in custom_logger_utils. Add tests for bytez integration.
* Add token tracking for model usage for Bytez integration.
* Create a units test for the Bytez config.
* Make changes to Bytez transformation code per PR feedback.
* Cleanup coment in Bytez transformation test.
* Remove LRU usage for bytez integration.
* Consolidate Bytez tests into a single file. Conform to project structure for tests.
* Fix linting error with Bytez impl.
* Add unit tests
* Fix tool call handling in Anthropic pass-through adapter
- Move tool message placement before user messages to maintain proper order
- Add safety check for empty function arguments in tool calls
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Added dashscope as a provider
* Fix some leftover references on nebius
* Porting the dashscope api endpoit international version
* explicit tool_choice = True in config
* add support of bearer token for bedrock integration
* fix linting issue
* fix type checking issue
* reoder arguments to address type checking issue
* switch to use get_secret_str to fetch env variable
Co-authored-by: 0x-fang <fanggong@amazon.com>
* Litellm dev 03 05 2025 contributor prs (#9079)
* feat: add support for copilot provider
* test: add tests for github copilot
* chore: clean up github copilot authenticator
* test: add test for github copilot authenticator
* test: add test for github copilot for sonnet 3.7 thought model
* Fix#7629 - Add tzdata package to Dockerfile (#8915)
* Add tzdata package to Dockerfile
* Move tzdata to python requirement.txt
* feat: add support for copilot provider (#8577)
* feat: add support for copilot provider
* test: add tests for github copilot
* chore: clean up github copilot authenticator
* test: add test for github copilot authenticator
* test: add test for github copilot for sonnet 3.7 thought model
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* feat: add model information for copilot models
* fix: fix linting errors
* test: remove integration test for github_copilot + fix misisng mock
* fix: use print to make sure the logger message shown
* test: remove debug print
* fix lint (#11112)
* Add init files to make test directories Python packages and update import paths in test_token_counter.py (#11119)
* Update litellm/model_prices_and_context_window_backup.json
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
---------
Co-authored-by: Son H. Nguyen <nhs.000.dev@gmail.com>
Co-authored-by: subnet.dev <50828879+subnet-dev@users.noreply.github.com>
Co-authored-by: Son H. Nguyen <33925625+nhs000@users.noreply.github.com>
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* refactor github copilot
* test_github_copilot_transformation.py
* test_github_copilot_authenticator.py
* add GitHub Copilot
* fix order
* doc fix
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Son H. Nguyen <nhs.000.dev@gmail.com>
Co-authored-by: subnet.dev <50828879+subnet-dev@users.noreply.github.com>
Co-authored-by: Son H. Nguyen <33925625+nhs000@users.noreply.github.com>
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
* Enhance Mistral API: Add support for parallel tool calls and refine name handling in tool messages. Plus, introduce a new test for parallel tool calls in the Mistral model.
* tests
* make mypy happy
* Refine name handling in Mistral chat transformation: clarify conditions for removing the 'name' field based on message role and content.
* refactor: streamline Mistral integration by removing deprecated references and adding a new handler
- Removed "mistral" from the list of compatible providers in constants.
- Updated the completion function in main.py to utilize the new Mistral handler.
- Deleted outdated Mistral chat and embedding files.
- Introduced a new handler for Mistral chat completions, implementing the llm_http_handler pattern.
- Added integration tests for the Mistral handler to ensure proper API base and key handling.
* lint
* fix: remove unneeded handler object
* add tests
* Addres PR comments
* Enhance Mistral API: Add support for parallel tool calls and refine name handling in tool messages. Plus, introduce a new test for parallel tool calls in the Mistral model.
* tests
* make mypy happy
* Refine name handling in Mistral chat transformation: clarify conditions for removing the 'name' field based on message role and content.
* handle mistral returning '' instead of None
* Fix: Preserve full path structure for Gemini custom api_base (Fixes#11959)
This fix addresses an issue where custom api_base URLs (like Cloudflare AI Gateway)
were not working correctly with Google AI Studio (Gemini) models.
The problem was that the _check_custom_proxy method was simply appending the endpoint
to the custom base URL, resulting in malformed URLs like:
https://gateway.ai.cloudflare.com/v1/my-id/my-gateway/google-ai-studio:generateContent
Instead of the correct format:
https://gateway.ai.cloudflare.com/v1/my-id/my-gateway/google-ai-studio/v1beta/models/gemini-2.5-flash:generateContent
Changes:
- Modified _check_custom_proxy to preserve the full path structure from the original URL
- Extracts the path from the original Google AI URL and appends it to the custom base
- Maintains backward compatibility for Vertex AI models (unchanged behavior)
- Added comprehensive tests to verify the fix works correctly
Fixes#11959
* Fix: Update test to match actual Gemini URL format and fix double colon issue
- Fixed test expectation to include the full model path with 'gemini/' prefix
- Fixed double colon issue in Vertex AI URL construction when using custom api_base
- All tests now pass successfully
* fix(proxy_server.py): handle empty config yaml
Fixes https://github.com/BerriAI/litellm/issues/12163
* fix(gemini/common_utils.py): replace models/ as expected, instead of using 'strip'
Fixes https://github.com/BerriAI/litellm/issues/12160
* fix(anthropic/experimental_pass_through/messages/transformation.py): check for env var when selecting api key
* fix(anthropic/transformation.py): return tool_use content block start on anthropic bridge
Closes https://github.com/BerriAI/litellm/issues/12158
* fix(anthropic/streaming_iterator.py): fix setting index in block
ensure index is set just once and increments correctly when a new block is created
* fix(anthropic/adapters/handler.py): update logging obj with stream options value if set
* feat(anthropic/streaming_iterator.py): return usage from chat completion to messages bridge
enables usage tracking for non-anthropic models
Closes https://github.com/BerriAI/litellm/issues/12132
* fix(streaming_iterator.py): safely access usage chunk
* fix: suppress linting error
* test: update tests
* fix: fix streaming errors
* fix(proxy_server.py): handle empty config yaml
Fixes https://github.com/BerriAI/litellm/issues/12163
* fix(gemini/common_utils.py): replace models/ as expected, instead of using 'strip'
Fixes https://github.com/BerriAI/litellm/issues/12160
* fix(anthropic/experimental_pass_through/messages/transformation.py): check for env var when selecting api key
* docs(config_settings.md): add api key to docs
* fix(anthropic/experimental_pass_through): use given model name when returning streaming chunks
don't harcode model name on streaming
confusing for user
* fix(anthropic/streaming_iterator.py): remove scope of import
* feat(litellm_logging.py): allow admin to specify additional headers for using as spend tags
Closes https://github.com/BerriAI/litellm/issues/12129
* test(test_litellm_logging.py): add unit tests
* feat(openweb_ui.md): add custom tag tutorial to docs
* docs(cost_tracking.md): add tag based usage UI screenshot
* test: update test
* fix: fix import
* refactor(passthrough_endpoints-success-handler): refactor llm passthrough logging logic
isolate the llm translation work to enable cost tracking on sdk
* feat: initial implementation of passthrough SDK cost calculation
enables bedrock passthrough cost tracking to work
* feat(cost_calculator.py): working cost calculation for bedrock passthrough
* feat(litellm_logging.py): consider allm_passthrough in cost tracking
allows async calls (e.g. via proxy) to work
* feat(bedrock/passthrough): working event stream decoding for bedrock passthrough calls + logging instrumentation for passthrough sdk calls (log on stream completion)
Enables bedrock streaming cost calculation
* feat(litellm_logging.py): support streaming passthrough cost tracking
* feat(passthrough/main.py): working async streaming cost calculation
Closes https://github.com/BerriAI/litellm/issues/11359
* feat(proxy_server.py): fix passthrough routing when llm router enabled
* feat: further fixes
* feat(bedrock/): working bedrock passthrough cost tracking (non-streaming)
* feat(litellm_logging.py): working usage tracking for bedrock passthrough calls
ensures tokens are logged
* feat(bedrock/passthrough): add converse passthrough cost tracking support
* feat(base_llm/passthrough): remove redundant function
* refactor(litellm_logging.py): refactor function to be below 50 LOC
* test: update test
* test: remove redundant test
* [Bug Fix] Anthropic - Token Usage Null Handling in calculate_usage (BerriAI/litellm#11920)
* [Fix] Missed a null check and used a cast instead by error
* fix(handler.py): support routing custom llm's to chat completion handler
Adds custom llm support for anthropic
* test(test_anthropic_experimental_pass_through_messages_handler.py): add unit test confirming custom llm respected
* docs(custom_llm_server.md): document anthropic custom llm translation
* test(volcengine.py): map thinking in extra body
Fixes https://github.com/BerriAI/litellm/issues/11879
* feat(main.py): support `azure/responses/<deployment-name>` model string
this allows us to route the model correctly
Closes https://github.com/BerriAI/litellm/issues/11879
* docs(azure_responses.md): document calling azure responses api models via chat completions bridge
Closes https://github.com/BerriAI/litellm/issues/11917
* fix: fix custom provider check
* test: update tests
* fix(litellm_logging.py): fix using router model id for logging calls
Fixes https://github.com/BerriAI/litellm/issues/11975#issuecomment-2995882238
* test(test_litellm_logging.py): add unit test for custom price tracking
* fix(vertex_ai/): don't send invalid format parameter to vertex
causes calls to fail
* fix(vertex_ai_context_caching.py): if cached content present and tools in message, cache tools as well
gemini throws errors if tools passed in alongside cached content
* test: add unit tests
* fix: fix linting errors
* test: test_vertex_ai_common_utils.py
update test
* fix(streaming_handler.py): unset response cost when creating model response
* feat: add citation_cost_per_token and search_queries_cost_per_1000 fields to ModelInfoBase
- Add citation_cost_per_token field to ModelInfoBase for Perplexity citation token costs
- Add search_queries_cost_per_1000 field to ModelInfoBase for Perplexity search query costs
- Update _get_model_info_helper to include these fields in model info responses
- Enables proper cost calculation for Perplexity-specific usage metrics
* feat: update Perplexity sonar-deep-research model pricing configuration
- Update input/output token costs to / per million tokens respectively
- Add reasoning token cost at per million tokens
- Add citation_cost_per_token at per million tokens (same as input)
- Add search_queries_cost_per_1000 at /bin/zsh.005 per 1000 search queries
- Remove deprecated search_context_cost_per_query structure
- Aligns with Perplexity's updated pricing model for deep research capabilities
* feat: implement Perplexity-specific cost calculator
- Create cost_per_token function for Perplexity provider
- Calculate standard input/output token costs
- Add citation token cost calculation using citation_cost_per_token rate
- Add reasoning token cost calculation with fallback to completion_tokens_details
- Add search query cost calculation using search_queries_cost_per_1000 rate
- Return separate prompt_cost and completion_cost for accurate billing
- Handles all Perplexity-specific usage metrics: citation_tokens, num_search_queries, reasoning_tokens
* feat: integrate Perplexity cost calculator with main cost calculation system
- Import perplexity_cost_per_token function in main cost calculator
- Add perplexity provider case to cost_per_token function
- Enables automatic routing of Perplexity cost calculations to provider-specific logic
- Maintains compatibility with existing cost calculation patterns
- Supports all Perplexity-specific cost metrics through unified interface
* feat: enhance Perplexity response transformation to extract cost-related fields
- Override transform_response method to extract Perplexity-specific usage fields
- Add _enhance_usage_with_perplexity_fields method to process API responses
- Extract citation_tokens from citations array using character-based estimation (~4 chars/token)
- Extract num_search_queries from both usage field and root level with priority handling
- Create usage object when none exists to ensure cost fields are always captured
- Handle empty citations and missing fields gracefully
- Enables automatic extraction of cost metrics from Perplexity API responses
* test: add comprehensive test suite for Perplexity cost calculation features
Add 82 comprehensive tests across 3 test files:
- test_perplexity_cost_calculator.py (59 tests):
* Cost calculation with citation tokens, search queries, reasoning tokens
* Various combinations and edge cases
* Integration with main cost calculator
* Model info access and validation
* Zero values and missing fields handling
- test_perplexity_chat_transformation.py (12 tests):
* Citation token extraction from API responses
* Search query extraction from usage and root fields
* Priority handling and field aggregation
* Empty citations and missing fields handling
* Token estimation accuracy validation
- test_perplexity_integration.py (11 tests):
* End-to-end cost calculation workflows
* High-volume and edge case scenarios
* Model info integration validation
* Case-insensitive provider matching
* Transformation preservation of existing fields
Ensures reliability and correctness of all Perplexity cost features with comprehensive coverage of happy path, edge cases, and error conditions.
* fix: remove unused Union import from Perplexity transformation
- Remove unused typing.Union import from litellm/llms/perplexity/chat/transformation.py
- Fixes F401 linting error: 'typing.Union imported but unused'
- Maintains only necessary imports: Any, List, Optional, Tuple
* Fix JSON schema validation and use web_search_requests field
- Add citation_cost_per_token and search_queries_cost_per_1000 to JSON schema
- Update Perplexity transformation to use web_search_requests in PromptTokensDetailsWrapper
- Update Perplexity cost calculator to read from web_search_requests field
- Maintain backward compatibility while using standard LiteLLM fields
* Fix type errors in Perplexity cost calculator
- Add null checks for token counts and cost values to prevent None multiplication errors
- Use .get() with fallback values instead of direct dictionary access
- Ensure all arithmetic operations handle None values safely
This fixes the failing job 44517525148 type errors.
* Refactor Perplexity cost calculation tests to improve accuracy and consistency
- Replace absolute difference assertions with math.isclose for better precision in cost comparisons
- Update tests to utilize PromptTokensDetailsWrapper for handling web search requests
- Ensure all test cases correctly reflect the new structure of usage fields, enhancing clarity and maintainability
* fix: address type hinting issues in PerplexityChatConfig usage handling
- Add type ignore comments to model_response.usage assignments to resolve type checking errors
- Ensures compatibility with type definitions while maintaining existing functionality
* Update model pricing configuration in JSON backup
- Add citation_cost_per_token and search_queries_cost_per_1000 fields to enhance cost tracking
- Remove deprecated search_context_cost_per_query structure to streamline pricing model
- Aligns with recent updates in Perplexity's pricing strategy
* Update search queries cost structure in model_prices_and_context_window.json to use search_context_cost_per_query
* Refactor search queries cost structure in model_prices_and_context_window_backup.json and update related code to use search_queries_cost_per_query. Remove deprecated search_queries_cost_per_1000 references across model info and tests.
* Enhance cost calculation in cost_calculator.py by introducing a safe float casting function to handle potential None and invalid values. Update cost calculations for input, citation, output, reasoning, and search query tokens to use this new function, ensuring more robust handling of model pricing data.
* Refactor cost calculation in cost_calculator.py to support both legacy and current search cost keys. Enhance handling of search cost values by accommodating both dictionary and float formats, ensuring robust cost computation for search queries.
* Update test cases to reflect changes in cost structure, renaming search_queries_cost_per_query to search_context_cost_per_query for consistency with recent refactor. Ensure assertions in tests align with updated cost keys.
* Update test_perplexity_integration.py to rename search_queries_cost_per_query to search_context_cost_per_query, ensuring consistency with recent cost structure changes. Adjust assertions to align with updated cost keys.
* Fixing watsonx error: 'model_id' or 'model' cannot be specified in the request body for models in a deployment space
* Revert "Fixing watsonx error: 'model_id' or 'model' cannot be specified in the request body for models in a deployment space"
This reverts commit 9d16a3000b.
* Implementing feedback from code review
* add test_anthropic_messages_litellm_router_streaming_with_logging to base tests
* move test
* fixes for base ant tests
* working bedrock ant logging
* use BaseAnthropicMessagesStreamingIterator
* use common iterator for messages streaming
* TestAnthropicDirectAPI
* test_anthropic_claude3_transformation.py
* fix code QA checks
* fix logging for anthropic messages in SLP
* fix TestAnthropicOpenAIAPI
* remove hard coded usage for adapter
* test_anthropic_messages_litellm_router_streaming_with_logging
* fix(factory.py): handle qs:.. in mime type
Fixes https://github.com/BerriAI/litellm/issues/11839
* feat(litellm_proxy/): don't transform messages client-side
leave litellm proxy messages untouched - allow proxy to handle transformation
prevents double transformation
* feat(tag_management_endpoints.py): support adding models to tag by adding model_name
Closes https://github.com/BerriAI/litellm/issues/11884
* test(test_tag_management_endpoints.py): add unit tests for adding new model by public model name
* test: update test
