Added cache_read_input_token_cost (25% of regular input cost) to all 39 Gemini 2.x models
to properly support implicit context caching cost calculations. Previously, cached tokens
were being charged at full price instead of the discounted rate.
Fixes#11156
* [Feat] New LLM API Integration - Add Moonshot API (Kimi) (#12551)
* Add Moonshot AI provider support to LiteLLM
Co-authored-by: ishaan <ishaan@berri.ai>
* Refactor Moonshot provider params handling and transformation logic
Co-authored-by: ishaan <ishaan@berri.ai>
* fix constants
* add Moonshot AI
* fix get_supported_openai_params
* handle kimi temp
* add tool choice handling
* test moonshot unit tests
* fix kimi
* fix linting
* Add pricing information for Moonshot AI's kimi-k2 model (#12566)
* Add pricing information for Moonshot AI's kimi-k2 model
* Update model name to kimi-k2-0711-preview
- Changed model name from moonshot/kimi-k2 to moonshot/kimi-k2-0711-preview
- This reflects the specific model version as requested
* Update moonshot_models list to match model_context JSON
---------
Co-authored-by: openhands <openhands@all-hands.dev>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* update docs
* docs moonshot
* fixes model cost map
* fix map_openai_params
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
Co-authored-by: Xingyao Wang <xingyao@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
* Added dashscope as a provider
* Fix some leftover references on nebius
* Porting the dashscope api endpoit international version
* explicit tool_choice = True in config
- Add azure/o3-deep-research model to pricing configuration
- Input cost: 0 per million tokens (1e-05 per token)
- Output cost: 0 per million tokens (4e-05 per token)
- Supports reasoning, web search, function calling, vision, and other advanced features
- Based on Azure AI Foundry Deep Research pricing documentation
Resolves#12318
Co-authored-by: openhands <openhands@all-hands.dev>
* build(model_prices_and_context_window.json): remove 'supports_tool_choice' for specific mistral models
Closes https://github.com/BerriAI/litellm/issues/11750
* feat: initial commit adding cleaner ui for azure text moderation guardrails
* feat(guardrail_endpoints.py): add discoverable guardrail configs and improve converting base model to dict with types
* fix(guardrail_provider_fields.tsx): render from api endpoint correctly
* fix(guardrail_provider_fields.tsx): cleanup
* refactor(guardrail_endpoints.py): refactor to handle dictionaries with literal - allows multiselect
* feat(ui/): render dictionary with known keys correctly
* feat(ui/): render optional params on separate page
* style(ui/): style improvements to rendering optional params on the UI
* feat(azure/prompt_shield.py): add azure prompt shield back on UI
* fix(add_guardrail_form.tsx): fix form to handle updated api
* fix(guardrail_optional_params.tsx): ensure values are nested correctly for writing to api
* fix: fix linting error
* test: update tests
* fix mapped tests (#12320)
* fix - use flush llm client cache
* faster mapped tests
* test_async_multiple_response_ids_routing
* fix tests
* test_ateam_member_update_admin_requires_premium
* regular mapped tests
* Revert "Fix: Initialize JSON logging for all loggers when JSON_LOGS=True (#12206)"
This reverts commit 2c60c316ec.
* reset num workers
* check_for_litellm_module_deletion
* add test
* add code_qa_check_tests
* ci/cd new release
* docs(opentelemetry_integration.md): add otel debug flag to docs
help people debug their traces easily
* [Feat] Add failure logging support for s3 logger (#12299)
* add async_log_failure_event
* test_basic_s3_v2_logging_failure
* Comma separated spend and budget (#12317)
* all_keys_table, key_info_view, organizations
* teams
* the whole usage page
* teams page
* users table
* user info
* more commas
* some more commas
* Segregate MCP tools on connections using headers (#12296)
* Add get tools segregation
* add ui changes (#12302)
* resolve comments
* add mapped tests
* remove advanced settings (#12323)
* [Bug Fix] /generateContent API - Only pass supported params when using OpenAI models (#12297)
* fix - only pass GenericLiteLLMParams
* test_google_generate_content_with_openai
* Fix: Fix custom ca bundle support in aiohttp transport (#12281)
* Unify usage of get_ssl_configuration
* Fix doc
* [Feat] Add github co-pilot as a new LLM API provider (#12325)
* Litellm dev 03 05 2025 contributor prs (#9079)
* feat: add support for copilot provider
* test: add tests for github copilot
* chore: clean up github copilot authenticator
* test: add test for github copilot authenticator
* test: add test for github copilot for sonnet 3.7 thought model
* Fix#7629 - Add tzdata package to Dockerfile (#8915)
* Add tzdata package to Dockerfile
* Move tzdata to python requirement.txt
* feat: add support for copilot provider (#8577)
* feat: add support for copilot provider
* test: add tests for github copilot
* chore: clean up github copilot authenticator
* test: add test for github copilot authenticator
* test: add test for github copilot for sonnet 3.7 thought model
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* feat: add model information for copilot models
* fix: fix linting errors
* test: remove integration test for github_copilot + fix misisng mock
* fix: use print to make sure the logger message shown
* test: remove debug print
* fix lint (#11112)
* Add init files to make test directories Python packages and update import paths in test_token_counter.py (#11119)
* Update litellm/model_prices_and_context_window_backup.json
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
---------
Co-authored-by: Son H. Nguyen <nhs.000.dev@gmail.com>
Co-authored-by: subnet.dev <50828879+subnet-dev@users.noreply.github.com>
Co-authored-by: Son H. Nguyen <33925625+nhs000@users.noreply.github.com>
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* refactor github copilot
* test_github_copilot_transformation.py
* test_github_copilot_authenticator.py
* add GitHub Copilot
* fix order
* doc fix
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Son H. Nguyen <nhs.000.dev@gmail.com>
Co-authored-by: subnet.dev <50828879+subnet-dev@users.noreply.github.com>
Co-authored-by: Son H. Nguyen <33925625+nhs000@users.noreply.github.com>
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
* add new env vars to docs
* ssl_cert_error (#12327)
* feat - add new banner on startup (#12328)
* [Security Bug Fix] Ensure only LLM API route fails get logged on Langfuse (and other loggers) (#12308)
* _is_proxy_only_llm_api_error
* test_proxy_only_error_true_for_llm_route
* add not on change
* Update tests/test_litellm/proxy/test_proxy_utils.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* add test_post_call_failure_hook_auth_error_key_info_route
* test fix _is_proxy_only_llm_api_error
* test_chat_completion_request_with_redaction
* test_post_call_failure_hook_auth_error_llm_api_route
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Ensure message redaction works for responses API logging (#12291)
* add fixes to choice implementation redaction
* add isInstance check on responses API
* change datadog to revert back
* change datadog to revert back
* fix type errors
* Redaction test changes
* Redaction test changes
* Redaction test changes - remove changes
* [Feat] UI - Allow Viewing/Editing Team Based Callbacks (#12329)
* add logging settings view on UI
* fix change ordering
* fix type error
* test fix - import loc
* [Feat] QA - Arize Team based logging (#12331)
* add _get_tracer_with_dynamic_headers
* fix construct_dynamic_arize_headers
* [Feat] UI - Allow Viewing/Editing Team Based Callbacks (#12329)
* add logging settings view on UI
* fix change ordering
* add construct_dynamic_otel_headers for arize
* refactor common code
* test_construct_dynamic_arize_headers
* otel unit tests
* test_arize_dynamic_params
* test_arize_dynamic_headers_in_grpc_requests
* [Feat] UI - Allow setting Logging Callback Setting per Key (#12333)
* allow adding logging settings on a key
* add ability to edit key logging settings
* dont leak api keys on ui
* fix order
* add langsmith settings
* ui new build
* test_default_api_base
* fix new utils tests
* bump: version 1.73.7 → 1.74.0
* Add mcp server segregation comma separated support (#12326)
* add mcp comma separated support
* dont support legacy json array
* lint
* add live tail state to session storage (#12335)
* add aiml
* fix: fix unused imports
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: tanjiro <56165694+NANDINI-star@users.noreply.github.com>
Co-authored-by: Jugal D. Bhatt <55304795+jugaldb@users.noreply.github.com>
Co-authored-by: Joost van Doorn <joost.van.doorn@gmail.com>
Co-authored-by: Son H. Nguyen <nhs.000.dev@gmail.com>
Co-authored-by: subnet.dev <50828879+subnet-dev@users.noreply.github.com>
Co-authored-by: Son H. Nguyen <33925625+nhs000@users.noreply.github.com>
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* add watsonx/mistralai/mistral-large to model prices and context window json
- Set max_tokens and max_input_tokens to 131072 and max_output_tokens to 16384 to match IBM's official context window limit for Mistral Large models hosted on watsonx.
- Correct input_cost_per_token and output_cost_per_token to reflect IBM's pricing of $0.003 and $0.01 per 1000 tokens, respectively, converted to per-token rates (0.000003 and 0.00001).
- Confirmed function-calling support based on IBM documentation and Mistral's own API schema.
- Aligns with Mistral's own model entries that specify token windows of ~128K and matches LiteLLM's expected cost format.
**Sources:**
- IBM watsonx Foundation Models docs: https://www.ibm.com/docs/en/watsonx/w-and-w/2.1.x?topic=models-foundation-model-details#mistral-large
- IBM Pricing information for watsonx.ai models: https://www.ibm.com/products/watsonx-ai/pricing
- LiteLLM repo docs on token cost formatting and limits https://docs.litellm.ai/docs/proxy/custom_pricing.
- Imitating entries from litellm/model_prices_and_context_window.json for watsonx/ibm/granite-3-8b-instruct as well as Mistral's own mistral/mistral-large-latest and similar.
* Update model_prices_and_context_window.json
typo
* Update model_prices_and_context_window.json
* Update model_prices_and_context_window.json
* Add Azure OpenAI assistant features cost tracking
Implements cost tracking for Azure's new assistant features:
- File Search: $0.1 USD per 1 GB/Day (storage-based pricing)
- Code Interpreter: $0.03 USD per session
- Computer Use: $0.003 input + $0.012 output per 1K tokens
Features:
- Provider-specific pricing (Azure vs OpenAI)
- Model-specific pricing overrides via JSON config
- Environment variable configuration
- Backwards compatible with existing OpenAI pricing
* Add comprehensive tests for Azure assistant features cost tracking
- Unit tests for file search, code interpreter, computer use, vector store
- Integration tests for combined cost calculation
- Provider-specific pricing tests (Azure vs OpenAI)
- Model-specific pricing override tests
- Edge case handling (None inputs, zero values)
- All 17 tests passing
* Fix test and ensure all Azure assistant cost tracking tests pass
- Fixed integration test approach
- All 17 tests now passing
- Comprehensive coverage of Azure assistant features cost tracking
* Enhance cost tracking for Azure assistant features
- Safely convert and extract parameters for file search, computer use, and code interpreter sessions.
- Ensure model_info is consistently converted to a dictionary format.
- Improve error handling for input values to prevent type-related issues.
- Maintain compatibility with existing cost calculation methods.
* Refactor cost tracking for Azure assistant features
- Introduced separate methods for handling costs related to web search, file search, vector store, computer use, and code interpreter.
- Enhanced parameter extraction and conversion for file search and computer use.
- Improved error handling and type safety throughout the cost calculation process.
- Maintained compatibility with existing cost calculation methods while streamlining the overall structure.
* feat(model_prices_and_context_window.json): add mistral-small-3.2-24b-instruct model with token costs and chat mode support
* fix(model_prices_and_context_window.json): update model paths to include 'openrouter' prefix for mistral-small-3.1 and 3.2
* feat: add citation_cost_per_token and search_queries_cost_per_1000 fields to ModelInfoBase
- Add citation_cost_per_token field to ModelInfoBase for Perplexity citation token costs
- Add search_queries_cost_per_1000 field to ModelInfoBase for Perplexity search query costs
- Update _get_model_info_helper to include these fields in model info responses
- Enables proper cost calculation for Perplexity-specific usage metrics
* feat: update Perplexity sonar-deep-research model pricing configuration
- Update input/output token costs to / per million tokens respectively
- Add reasoning token cost at per million tokens
- Add citation_cost_per_token at per million tokens (same as input)
- Add search_queries_cost_per_1000 at /bin/zsh.005 per 1000 search queries
- Remove deprecated search_context_cost_per_query structure
- Aligns with Perplexity's updated pricing model for deep research capabilities
* feat: implement Perplexity-specific cost calculator
- Create cost_per_token function for Perplexity provider
- Calculate standard input/output token costs
- Add citation token cost calculation using citation_cost_per_token rate
- Add reasoning token cost calculation with fallback to completion_tokens_details
- Add search query cost calculation using search_queries_cost_per_1000 rate
- Return separate prompt_cost and completion_cost for accurate billing
- Handles all Perplexity-specific usage metrics: citation_tokens, num_search_queries, reasoning_tokens
* feat: integrate Perplexity cost calculator with main cost calculation system
- Import perplexity_cost_per_token function in main cost calculator
- Add perplexity provider case to cost_per_token function
- Enables automatic routing of Perplexity cost calculations to provider-specific logic
- Maintains compatibility with existing cost calculation patterns
- Supports all Perplexity-specific cost metrics through unified interface
* feat: enhance Perplexity response transformation to extract cost-related fields
- Override transform_response method to extract Perplexity-specific usage fields
- Add _enhance_usage_with_perplexity_fields method to process API responses
- Extract citation_tokens from citations array using character-based estimation (~4 chars/token)
- Extract num_search_queries from both usage field and root level with priority handling
- Create usage object when none exists to ensure cost fields are always captured
- Handle empty citations and missing fields gracefully
- Enables automatic extraction of cost metrics from Perplexity API responses
* test: add comprehensive test suite for Perplexity cost calculation features
Add 82 comprehensive tests across 3 test files:
- test_perplexity_cost_calculator.py (59 tests):
* Cost calculation with citation tokens, search queries, reasoning tokens
* Various combinations and edge cases
* Integration with main cost calculator
* Model info access and validation
* Zero values and missing fields handling
- test_perplexity_chat_transformation.py (12 tests):
* Citation token extraction from API responses
* Search query extraction from usage and root fields
* Priority handling and field aggregation
* Empty citations and missing fields handling
* Token estimation accuracy validation
- test_perplexity_integration.py (11 tests):
* End-to-end cost calculation workflows
* High-volume and edge case scenarios
* Model info integration validation
* Case-insensitive provider matching
* Transformation preservation of existing fields
Ensures reliability and correctness of all Perplexity cost features with comprehensive coverage of happy path, edge cases, and error conditions.
* fix: remove unused Union import from Perplexity transformation
- Remove unused typing.Union import from litellm/llms/perplexity/chat/transformation.py
- Fixes F401 linting error: 'typing.Union imported but unused'
- Maintains only necessary imports: Any, List, Optional, Tuple
* Fix JSON schema validation and use web_search_requests field
- Add citation_cost_per_token and search_queries_cost_per_1000 to JSON schema
- Update Perplexity transformation to use web_search_requests in PromptTokensDetailsWrapper
- Update Perplexity cost calculator to read from web_search_requests field
- Maintain backward compatibility while using standard LiteLLM fields
* Fix type errors in Perplexity cost calculator
- Add null checks for token counts and cost values to prevent None multiplication errors
- Use .get() with fallback values instead of direct dictionary access
- Ensure all arithmetic operations handle None values safely
This fixes the failing job 44517525148 type errors.
* Refactor Perplexity cost calculation tests to improve accuracy and consistency
- Replace absolute difference assertions with math.isclose for better precision in cost comparisons
- Update tests to utilize PromptTokensDetailsWrapper for handling web search requests
- Ensure all test cases correctly reflect the new structure of usage fields, enhancing clarity and maintainability
* fix: address type hinting issues in PerplexityChatConfig usage handling
- Add type ignore comments to model_response.usage assignments to resolve type checking errors
- Ensures compatibility with type definitions while maintaining existing functionality
* Update model pricing configuration in JSON backup
- Add citation_cost_per_token and search_queries_cost_per_1000 fields to enhance cost tracking
- Remove deprecated search_context_cost_per_query structure to streamline pricing model
- Aligns with recent updates in Perplexity's pricing strategy
* Update search queries cost structure in model_prices_and_context_window.json to use search_context_cost_per_query
* Refactor search queries cost structure in model_prices_and_context_window_backup.json and update related code to use search_queries_cost_per_query. Remove deprecated search_queries_cost_per_1000 references across model info and tests.
* Enhance cost calculation in cost_calculator.py by introducing a safe float casting function to handle potential None and invalid values. Update cost calculations for input, citation, output, reasoning, and search query tokens to use this new function, ensuring more robust handling of model pricing data.
* Refactor cost calculation in cost_calculator.py to support both legacy and current search cost keys. Enhance handling of search cost values by accommodating both dictionary and float formats, ensuring robust cost computation for search queries.
* Update test cases to reflect changes in cost structure, renaming search_queries_cost_per_query to search_context_cost_per_query for consistency with recent refactor. Ensure assertions in tests align with updated cost keys.
* Update test_perplexity_integration.py to rename search_queries_cost_per_query to search_context_cost_per_query, ensuring consistency with recent cost structure changes. Adjust assertions to align with updated cost keys.
* build(model_prices_and_context_window.json): mark all gemini-2.5 models as supporting pdf input
Closes https://github.com/BerriAI/litellm/issues/11881
* fix(anthropic_transformation.py): set custom llm provider custom property
Fixes https://github.com/BerriAI/litellm/issues/11861
* test: add unit test for checking supports_reasoning
* test: add test for vertex ai flow
* feat(bedrock/anthropic): ensure thinking param correctly passed for bedrock/invoke
* Added support for reasoning parameters in magistral models, including "reasoning_effort" and "thinking".
* Updated the MistralConfig class to handle reasoning system prompts.
* Implemented tests to verify reasoning functionality and ensure correct parameter mapping for magistral models.
* Enhanced the model prices JSON to reflect new reasoning capabilities.
* Add Deepgram provider to supported providers list and mappings
* add logo
* Add deepgram to model cost map
* ui - require api key for deepgram
* fix logo path
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* build(model_prices_and_context_window.json): add o3-pro pricing
* build(model_prices_and_context_window.json): add updated o3 model pricing
* build(model_prices_and_context_window.json): add new o3-pro model version
* build(model_prices_and_context_window.json): add 'claude-opus-4' on vertexai (no @)
* build(model_prices_and_context_window.json): add claude sonnet 4 without 2
*@
* feat(internal_user_endpoints.py): assign user to orgs on user creation
allows user to be a member of orgs on creation - work to enable default orgs on UI
* fix(internal_user_endpoints.py): fix http_request
