* fix(handler.py): support routing custom llm's to chat completion handler
Adds custom llm support for anthropic
* test(test_anthropic_experimental_pass_through_messages_handler.py): add unit test confirming custom llm respected
* docs(custom_llm_server.md): document anthropic custom llm translation
* test(volcengine.py): map thinking in extra body
Fixes https://github.com/BerriAI/litellm/issues/11879
* feat(main.py): support `azure/responses/<deployment-name>` model string
this allows us to route the model correctly
Closes https://github.com/BerriAI/litellm/issues/11879
* docs(azure_responses.md): document calling azure responses api models via chat completions bridge
Closes https://github.com/BerriAI/litellm/issues/11917
* fix: fix custom provider check
* test: update tests
* fix(litellm_logging.py): fix using router model id for logging calls
Fixes https://github.com/BerriAI/litellm/issues/11975#issuecomment-2995882238
* test(test_litellm_logging.py): add unit test for custom price tracking
* fix(vertex_ai/): don't send invalid format parameter to vertex
causes calls to fail
* fix(vertex_ai_context_caching.py): if cached content present and tools in message, cache tools as well
gemini throws errors if tools passed in alongside cached content
* test: add unit tests
* fix: fix linting errors
* test: test_vertex_ai_common_utils.py
update test
* fix(streaming_handler.py): unset response cost when creating model response
* feat: add citation_cost_per_token and search_queries_cost_per_1000 fields to ModelInfoBase
- Add citation_cost_per_token field to ModelInfoBase for Perplexity citation token costs
- Add search_queries_cost_per_1000 field to ModelInfoBase for Perplexity search query costs
- Update _get_model_info_helper to include these fields in model info responses
- Enables proper cost calculation for Perplexity-specific usage metrics
* feat: update Perplexity sonar-deep-research model pricing configuration
- Update input/output token costs to / per million tokens respectively
- Add reasoning token cost at per million tokens
- Add citation_cost_per_token at per million tokens (same as input)
- Add search_queries_cost_per_1000 at /bin/zsh.005 per 1000 search queries
- Remove deprecated search_context_cost_per_query structure
- Aligns with Perplexity's updated pricing model for deep research capabilities
* feat: implement Perplexity-specific cost calculator
- Create cost_per_token function for Perplexity provider
- Calculate standard input/output token costs
- Add citation token cost calculation using citation_cost_per_token rate
- Add reasoning token cost calculation with fallback to completion_tokens_details
- Add search query cost calculation using search_queries_cost_per_1000 rate
- Return separate prompt_cost and completion_cost for accurate billing
- Handles all Perplexity-specific usage metrics: citation_tokens, num_search_queries, reasoning_tokens
* feat: integrate Perplexity cost calculator with main cost calculation system
- Import perplexity_cost_per_token function in main cost calculator
- Add perplexity provider case to cost_per_token function
- Enables automatic routing of Perplexity cost calculations to provider-specific logic
- Maintains compatibility with existing cost calculation patterns
- Supports all Perplexity-specific cost metrics through unified interface
* feat: enhance Perplexity response transformation to extract cost-related fields
- Override transform_response method to extract Perplexity-specific usage fields
- Add _enhance_usage_with_perplexity_fields method to process API responses
- Extract citation_tokens from citations array using character-based estimation (~4 chars/token)
- Extract num_search_queries from both usage field and root level with priority handling
- Create usage object when none exists to ensure cost fields are always captured
- Handle empty citations and missing fields gracefully
- Enables automatic extraction of cost metrics from Perplexity API responses
* test: add comprehensive test suite for Perplexity cost calculation features
Add 82 comprehensive tests across 3 test files:
- test_perplexity_cost_calculator.py (59 tests):
* Cost calculation with citation tokens, search queries, reasoning tokens
* Various combinations and edge cases
* Integration with main cost calculator
* Model info access and validation
* Zero values and missing fields handling
- test_perplexity_chat_transformation.py (12 tests):
* Citation token extraction from API responses
* Search query extraction from usage and root fields
* Priority handling and field aggregation
* Empty citations and missing fields handling
* Token estimation accuracy validation
- test_perplexity_integration.py (11 tests):
* End-to-end cost calculation workflows
* High-volume and edge case scenarios
* Model info integration validation
* Case-insensitive provider matching
* Transformation preservation of existing fields
Ensures reliability and correctness of all Perplexity cost features with comprehensive coverage of happy path, edge cases, and error conditions.
* fix: remove unused Union import from Perplexity transformation
- Remove unused typing.Union import from litellm/llms/perplexity/chat/transformation.py
- Fixes F401 linting error: 'typing.Union imported but unused'
- Maintains only necessary imports: Any, List, Optional, Tuple
* Fix JSON schema validation and use web_search_requests field
- Add citation_cost_per_token and search_queries_cost_per_1000 to JSON schema
- Update Perplexity transformation to use web_search_requests in PromptTokensDetailsWrapper
- Update Perplexity cost calculator to read from web_search_requests field
- Maintain backward compatibility while using standard LiteLLM fields
* Fix type errors in Perplexity cost calculator
- Add null checks for token counts and cost values to prevent None multiplication errors
- Use .get() with fallback values instead of direct dictionary access
- Ensure all arithmetic operations handle None values safely
This fixes the failing job 44517525148 type errors.
* Refactor Perplexity cost calculation tests to improve accuracy and consistency
- Replace absolute difference assertions with math.isclose for better precision in cost comparisons
- Update tests to utilize PromptTokensDetailsWrapper for handling web search requests
- Ensure all test cases correctly reflect the new structure of usage fields, enhancing clarity and maintainability
* fix: address type hinting issues in PerplexityChatConfig usage handling
- Add type ignore comments to model_response.usage assignments to resolve type checking errors
- Ensures compatibility with type definitions while maintaining existing functionality
* Update model pricing configuration in JSON backup
- Add citation_cost_per_token and search_queries_cost_per_1000 fields to enhance cost tracking
- Remove deprecated search_context_cost_per_query structure to streamline pricing model
- Aligns with recent updates in Perplexity's pricing strategy
* Update search queries cost structure in model_prices_and_context_window.json to use search_context_cost_per_query
* Refactor search queries cost structure in model_prices_and_context_window_backup.json and update related code to use search_queries_cost_per_query. Remove deprecated search_queries_cost_per_1000 references across model info and tests.
* Enhance cost calculation in cost_calculator.py by introducing a safe float casting function to handle potential None and invalid values. Update cost calculations for input, citation, output, reasoning, and search query tokens to use this new function, ensuring more robust handling of model pricing data.
* Refactor cost calculation in cost_calculator.py to support both legacy and current search cost keys. Enhance handling of search cost values by accommodating both dictionary and float formats, ensuring robust cost computation for search queries.
* Update test cases to reflect changes in cost structure, renaming search_queries_cost_per_query to search_context_cost_per_query for consistency with recent refactor. Ensure assertions in tests align with updated cost keys.
* Update test_perplexity_integration.py to rename search_queries_cost_per_query to search_context_cost_per_query, ensuring consistency with recent cost structure changes. Adjust assertions to align with updated cost keys.
* Fixing watsonx error: 'model_id' or 'model' cannot be specified in the request body for models in a deployment space
* Revert "Fixing watsonx error: 'model_id' or 'model' cannot be specified in the request body for models in a deployment space"
This reverts commit 9d16a3000b.
* Implementing feedback from code review
* add test_anthropic_messages_litellm_router_streaming_with_logging to base tests
* move test
* fixes for base ant tests
* working bedrock ant logging
* use BaseAnthropicMessagesStreamingIterator
* use common iterator for messages streaming
* TestAnthropicDirectAPI
* test_anthropic_claude3_transformation.py
* fix code QA checks
* fix logging for anthropic messages in SLP
* fix TestAnthropicOpenAIAPI
* remove hard coded usage for adapter
* test_anthropic_messages_litellm_router_streaming_with_logging
* fix(factory.py): handle qs:.. in mime type
Fixes https://github.com/BerriAI/litellm/issues/11839
* feat(litellm_proxy/): don't transform messages client-side
leave litellm proxy messages untouched - allow proxy to handle transformation
prevents double transformation
* feat(tag_management_endpoints.py): support adding models to tag by adding model_name
Closes https://github.com/BerriAI/litellm/issues/11884
* test(test_tag_management_endpoints.py): add unit tests for adding new model by public model name
* test: update test
* build(model_prices_and_context_window.json): mark all gemini-2.5 models as supporting pdf input
Closes https://github.com/BerriAI/litellm/issues/11881
* fix(anthropic_transformation.py): set custom llm provider custom property
Fixes https://github.com/BerriAI/litellm/issues/11861
* test: add unit test for checking supports_reasoning
* test: add test for vertex ai flow
* feat(bedrock/anthropic): ensure thinking param correctly passed for bedrock/invoke
* Enhance Mistral API: Add support for parallel tool calls and refine name handling in tool messages. Plus, introduce a new test for parallel tool calls in the Mistral model.
* tests
* make mypy happy
* Refine name handling in Mistral chat transformation: clarify conditions for removing the 'name' field based on message role and content.
* fix(vertex_ai): Handle missing tokenCount in promptTokensDetails (#11581)
This PR is a Solution to the Error converting to a valid response block='tokenCount'. File an issue if litellm error - https://github.com/BerriAI/litellm/issues
It's happening because vertex_ai is not sometimes sending the token count for the audio modality.
* test_vertex_ai_usage_metadata_missing_token_count
---------
Co-authored-by: Nishith Jain <167524748+KingNish24@users.noreply.github.com>
* Enable Tool Calling for `meta_llama` (#11825)
* feat: enable tools and function_call features
* fix: ignore pydantic warnings for StreamingChoices from llama-api
* docs: add tool calling examples
* docs: change default models to Maverick
* docs: fix output of tool use
* test_map_openai_params
---------
Co-authored-by: Young Han <110819238+seyeong-han@users.noreply.github.com>
* fix(spend_tracking_utils.py): add user agent tags from standard logging payload, in spend logs payload
* feat(litellm_logging.py): identify user agent tags as `User-Agent: ..` and allow admin to disable storing user agent as tag
* fix(azure_ai/): pass content type header in azure ai request
Fixes https://github.com/BerriAI/litellm/issues/11227
* test: add unit test
* fix(router.py): fix passing dynamic credentials to retrieve batch
Fixes batch retrieval when using router
* test: add more unit tests
* fix(litellm_pre_call_utils.py): add user agent tags to spend logs in standard logging payload logic
avoid clash when tag based routing is enabled
* test: remove redundant test
* test: rename oidc test to run earlier
quicker debuging
* fix(azure.py): return more detailed error message
* fix(azure/common_utils.py): use default scope, if scope is none
fixes oidc test
* fix: always default to cognitiveservices.azure.com
* test: update test
* feat: update the feature of ollama_embeddings to work on a sync api
Signed-off-by: Abinand P <abinand0911@gmail.com>
* lint:fixing of the lint file
Signed-off-by: Abinand P <abinand0911@gmail.com>
* fix:test
Signed-off-by: Abinand P <abinand0911@gmail.com>
* chore: added test for ollama embedding and refactored handler
Signed-off-by: Abinand P <abinand0911@gmail.com>
* fix:lint error
Signed-off-by: Abinand P <abinand0911@gmail.com>
---------
Signed-off-by: Abinand P <abinand0911@gmail.com>
* docs(index.md): add stable pip package
* fix(anthropic/chat/transformation.py): add 'none' tool choice mapping
Allows disabling anthropic tool calling
Maintain parity
* fix(transformation.py): if tool_choice="none" ignore 'disable_parallel_Tool_use'
unsupported param from anthropic - makes sense as the 'none' implies no tool calls are being made
* fix(anthropic/chat/transformation.py): append prefix to start of assistant response, if set
ensures assistant response contains complete response
* fix(anthropic/chat/transformation.py): add flag to allow user to opt out of enabling prefix in prompt
* fix(anthropic/chat/transformation.py): working e2e support for prefix prompt in assistant response
* feat(networking.tsx): always include model access groups on UI
show admin created access groups when giving key/user/team model permissions
* feat(add_model_tab.tsx): initial ui component for adding to an existing model access group
allows user to add model to an access group (simplify giving users/keys/teams model access)
* feat(proxy_server.py): add 'only_model_access_groups' flag support to `/v1/models`
simplifies listing available access groups on UI
* test: add e2e test for new only_model_access_groups param
* feat(add_model_tab.tsx): allow adding+viewing model access groups on models tab
make feature functional on UI
* feat(view_users.tsx): route edit user to user info page
more detailed user edit
* feat(columns.tsx): route edit user to user info page
more detailed user edit
* fix(columns.tsx): fix linting error
* build(ui/): fix linting errors
* feat(anthropic/passthrough): pass dynamic api key/api base params to litellm.completion
allows calls to work with config.yaml
* fix(responses_api/transformation): fix passing dynamic params to responses api from .completion()
Allows responses api to work with config.yaml
* fix(langfuse.py): fix responses api usage logging to langfuse
* refactor(litellm_logging.py): add more generic solution for responses api usage logging
ensures it works across all logging integrations
* fix(litellm_logging.py): patch for anthropic messages not returning a pydantic object
it should ideally return a pydantic object, which would simplify checks and reduce errors
* fix(handler.py): correctly bubble up empty choices errors to litellm.completion
causes downstream errors as it is expected there is at least one choice set
* fix(response_metadata.py): allow model_info to be none
* fix(litellm_logging.py): copy object before mutating
* fix: fix lint check
- Updated the `_add_reasoning_system_prompt_if_needed` method to maintain the original format of list content when prepending the reasoning prompt.
- Adjusted tests to verify that both string and list content types are correctly handled, ensuring the reasoning prompt is added without altering the content structure.
- Updated the `_add_reasoning_system_prompt_if_needed` method to convert list content to strings before prepending the reasoning prompt.
- Adjusted tests to verify that system messages with list content are correctly transformed into strings, ensuring original content is preserved.
- Revised the reasoning support indicators in the Mistral model documentation for clarity.
- Improved the `_add_reasoning_system_prompt_if_needed` method to handle both string and list content types for system messages, ensuring the reasoning prompt is correctly prepended.
- Added a new test case to verify the functionality of adding the reasoning system prompt when the existing content is a list.
* Added support for reasoning parameters in magistral models, including "reasoning_effort" and "thinking".
* Updated the MistralConfig class to handle reasoning system prompts.
* Implemented tests to verify reasoning functionality and ensure correct parameter mapping for magistral models.
* Enhanced the model prices JSON to reflect new reasoning capabilities.
* Checkpoint before follow-up message
* Add comprehensive tests for Deepgram transcription functionality
* clean up transform
* just use 1 test
* test cleanup
* test fix get_complete_url
* test rename file
* refactor deepgram URL construction
* add logging_obj.pre_call
* fix unused imports
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(internal_user_endpoints.py): support user with `+` in email on user info
ensures user is correctly parsed from input
* fix(factory.py): support vertex function call args as None
handles empty string in args for vertex gemini calls
* docs(langfuse_integration.md): pin langfuse sdk version on docs
* fix(vertex_ai/): return empty dict, instead of none when empty string given
* refactor: reduce function size
* fix: fix linting errors
* fix: revert check
* fix(internal_user_endpoints.py): fix check
* test: update tests
* test: update tests
