* Add gemini api key in the custom api url
* Update tests
* Use api key n the header
* Use api key n the header
* fix mypy error
* fix mypy error
* fix test gemini auth
* add helper functions
* update generic_cost_per_token function
* add test
* formatting
* add examples in docstring for _calculate_tiered_cost
* Restore files to upstream/main version
* dashscope specific calculation
* improve for different costs
* remove _calculate_flat_cost function
* fix(anthropic-adapter): properly translate Anthropic image format to OpenAI
Fixed bug where images were stripped during Anthropic Messages API to Azure
OpenAI translation. Image source data was being stringified instead of having
fields properly extracted.
- Added _translate_anthropic_image_to_openai() helper method
- Support both base64 and URL image formats per Anthropic API spec
- Refactored user message and tool result image handling
* test(anthropic-adapter): add comprehensive image translation tests
Add 5 unit tests covering image translation from Anthropic to OpenAI format:
- User messages with base64 images
- User messages with URL images
- Tool results with base64 images
- Tool results with URL images
- Mixed content with multiple images
* Add v1 cut of container api
* fix lint errors
* Add proxy support to container apis & logging support (#16049)
* Add proxy support to container apis
* Add logging support
* Add cost tracking support for containers and documentation
* Add new constant documentation
* Add container cost in model map
* fix failing azure tests
* Update tests based on model map changes
* fix model map tests
* fix model map tests
* Container modeshould be container
* Container tests fix
* Merge branch 'main' into litellm_sameer_oct_staging_2
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* fix model error for apis which don't need model
* fix print statments:
* fix mypy lint errors
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* Fix vercel_ai_gateway entry for glm-4.6
* Fix: add unit test for vercel_ai_gateway/zai/glm-4.6
* Fix:
Add fix for unit test on vercel_ai_gateway/zai/glm-4.6
* Remove print statements in test_vercel_ai_gateway_glm46_cost_math
* perf: Skip sleep delays in base_mail.py during tests to improve test speed
* perf: Mock datetime.now in parallel_request_limiter_v3.py to improve test speed
* pref: Mock urllib system calls in test_aiohttp_transport.py to improve test speed
* chore: add --durations=50 to visualize slowest tests
* pref: reduce setup phase overhead by widening fixture scope in conftest.py
* test: stabilize flaky tests
* fix: minor issue
* fix(presidio.py): handle content as a list of texts
covers openai + anthropic messages api
* fix(presidio.py): safe get messages
* test: add unit testing for presidio guardrails
* fix(unified_guardrail.py): initial commit
* fix(enkryptai.py): implement apply_guardrail to enkrypt guardrail
* fix(unified_guardrail.py): support unified guardrail on input
* feat(unified_guardrail.py): add post call success hook implementation
allows us to just have 1 place to handle llm translation to guardrail api spec
* refactor: refactor initial unified guardrail component
* refactor: more refactoring
* feat(responses/): add guardrails to responses api
allows existing guardrails to work for new llm endpoints
* docs(adding_guardrail_support.md): document new guardrail endpoint support
* test: add unit tests
* feat(image_generation/): add guardrail support for image generation endpoint
* feat(openai/text_completion): support guardrails on `/v1/completions` API
* docs: document guardrails support on new endpoints
* docs: clarify when guardrails run
* feat(openai/speech): add guardrail support for input
* docs(rerank/): add guardrail support on input query
* fix: fix ruff check
* Addd v2/chat support for cohere
* fix streaming
* Use v2_transformation for logging passthrough:
* Use v2_transformation for logging passthrough:
* Add test for checking if document and citation_options is getting passed
* Update the cohere model
* Add cost tracking for vertex ai passthrough batch jobs
* Add full passthrough support
* refactor code according to the comments
* Add passthrough handler
* remove invalid params
* Updated documentation
* Updated documentation
* Updated documentation
* Correct the import
* Add openai videos generation and retrieval support
* add retrieval endpoint
* Add docs
* Add imports
* remove orjson
* remove double import
* fix openai videos format
* remove mock code
* remove not required comments
* Add tests
* Add tests
* Add other video endpoints
* Fix cost calculation and transformation
* Fixed mypy tests
* remove not used imports
* fix documentation for get batch req (#15742)
* Add grounding info to responses API (#15737)
* Add grounding info to responses API
* fix lint errors
* Use typed objects for annotations
* Use typed objects for annotations
* fix mypy error
* Litellm fix json serialize alreting 2 (#15741)
* fix json serializable error for alerts
* Add test
* fix mypt errors
* fix mypt errors
* Add Qwen3 imported model support for AWS Bedrock (#15783)
* Add qwen imported model support
* fix mypy errors
* fix empty user message error (#15784)
* fix typed dict for list
* Add azure supported videos endpoint
* fix mapped tests
* add azure sora models to model map
* Add OpenAI video generation and content retrieval support (#15745)
* Add openai videos generation and retrieval support
* add retrieval endpoint
* Add docs
* Add imports
* remove orjson
* remove double import
* fix openai videos format
* remove mock code
* remove not required comments
* Add tests
* Add tests
* Add other video endpoints
* Fix cost calculation and transformation
* Fixed mypy tests
* remove not used imports
* fix typed dict for list
* fix mypy errors
* move directory
* make v2 chat default
* Fix mypy tests
* Fix mypy tests
* Fix mypy tests
* Fix mypy tests
* Revert "Add Azure Video Generation Support with Sora Integration"
* refactor videos repo
* add test
* Add azure openai videos support
* Add azure openai videos support
* Add router endpoint support for videos
* fix mypy error
* add azure models
* fix mapped test
* fix mypy error
* Add proxy router test
* Add proxy router test
* remove deprecated model name from tests
* fix import error
* fix import error
* Add gaurdrail integration in videos endpoint
* Add logging support for videos endpoint
* Add final documentation supporting videos integration
* fix model name and document input
* Update literals to avoid mypy errors
* Remove unused imports and print statements
* revert guardrail support for video generation and video remix
* revert guardrail support for video generation and video remix
* Fix failing mapped and llm translation tests
* feat(cohere): Enhance embedding transformation to support Bedrock's embeddings by type
* test(cohere): Add unit tests for embedding transformation responses
* Implement fix for thinking_blocks and converse API calls
This fixes Claude's models via the Converse API, which should also fix
Claude Code.
* Add thinking literal
* Fix mypy issues
* Type fix for redacted thinking
* Add voyage model integration in sagemaker
* Add config file logic
* Use already exiting voyage transformation
* refactor code as per comments
* fix merge error
* refactor code as per comments
* refactor code as per comments
* UI new build
* [Fix] router - regression when adding/removing models (#15451)
* fix(router): update model_name_to_deployment_indices on deployment removal
When a deployment is deleted, the model_name_to_deployment_indices map
was not being updated, causing stale index references. This could lead
to incorrect routing behavior when deployments with the same model_name
were dynamically removed.
Changes:
- Update _update_deployment_indices_after_removal to maintain
model_name_to_deployment_indices mapping
- Remove deleted indices and decrement indices greater than removed index
- Clean up empty entries when no deployments remain for a model name
- Update test to verify proper index shifting and cleanup behavior
* fix(router): remove redundant index building during initialization
Remove duplicate index building operations that were causing unnecessary
work during router initialization:
1. Removed redundant `_build_model_id_to_deployment_index_map` call in
__init__ - `set_model_list` already builds all indices from scratch
2. Removed redundant `_build_model_name_index` call at end of
`set_model_list` - the index is already built incrementally via
`_create_deployment` -> `_add_model_to_list_and_index_map`
Both indices (model_id_to_deployment_index_map and
model_name_to_deployment_indices) are properly maintained as lookup
indexes through existing helper methods. This change eliminates O(N)
duplicate work during initialization without any behavioral changes.
The indices continue to be correctly synchronized with model_list on
all operations (add/remove/upsert).
* fix(prometheus): Fix Prometheus metric collection in a multi-workers environment (#14929)
Co-authored-by: sotazhang <sotazhang@tencent.com>
* Add tiered pricing and cost calculation for xai
* Use generic cost calculator
* Resolve conflicts in generated HTML files
* Remove penalty params as supported params for gemini preview model (#15503)
* fix conversion of thinking block
* add application level encryption in SQS (#15512)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* add application level encryption in SQS
* add application level encryption in SQS
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
* [Feat] Bedrock Knowledgebase - return search_response when using /chat/completions API with LiteLLM (#15509)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* add AnthropicCitation
* fix async_post_call_success_deployment_hook
* fix add vector_store_custom_logger to global callbacks
* test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call
* async_post_call_success_deployment_hook
* add async_post_call_streaming_deployment_hook
* async def test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call_streaming(setup_vector_store_registry):
* fix _call_post_streaming_deployment_hook
* fix async_post_call_streaming_deployment_hook
* test update
* docs: Accessing Search Results
* docs KB
* fix chatUI
* fix searchResults
* fix onSearchResults
* fix kb
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* [Feat] Add dynamic rate limits on LiteLLM Gateway (#15518)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* fix: KeyRequestBase
* fix rpm_limit_type
* fix dynamic rate limits
* fix use dynamic limits here
* fix _should_enforce_rate_limit
* fix _should_enforce_rate_limit
* fix counter
* test_dynamic_rate_limiting_v3
* use _create_rate_limit_descriptors
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* Add google rerank endpoint
* Add docs
* fix mypy error
* fix mypy and lint errors
* Add haiku 4.5 integration
* Add haiku 4.5 integration for other regions as well
* Handle citation field correctly
* Fix filtering headers for signature calcs
* Add haiku 4.5 integration (#15650)
---------
Co-authored-by: Leslie Cheng <leslie.cheng5@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com>
Co-authored-by: Lucas <10226902+LoadingZhang@users.noreply.github.com>
Co-authored-by: sotazhang <sotazhang@tencent.com>
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
Fixes RuntimeError "Session is closed" by:
- Checking session.closed before use and recreating if needed
- Catching RuntimeError during requests and retrying with new session
- Validating newly created sessions aren't already closed
Adds tests for both proactive detection and reactive retry scenarios.
Fixes#15263
This PR fixes the cost calculation for Bedrock Anthropic models with prompt caching.
**Root Cause:**
PR #9838 incorrectly removed adding `cacheWriteInputTokens` to `prompt_tokens`
for Bedrock, based on the assumption that it would cause double counting (similar
to an Anthropic API issue). However, Bedrock's token structure is different:
- **Bedrock API**: `inputTokens`, `cacheReadInputTokens`, and `cacheWriteInputTokens`
are ALL separate values that should be summed for total input tokens
- **Anthropic API**: Same structure - all three token types are separate
The fix in #9838 was later reverted for Anthropic (correctly re-adding
`cache_creation_input_tokens` to `prompt_tokens`), but Bedrock was never fixed.
**Changes:**
1. Re-add `cacheWriteInputTokens` to `input_tokens` in Bedrock transformation
2. Update test assertions to reflect correct behavior
3. Add regression test for prompt caching cost calculation
4. Fix typo in Anthropic transformation where `cache_creation_tokens` was
incorrectly set to `cache_read_input_tokens`
**Testing:**
- All existing Bedrock transformation tests pass
- New test validates correct cost calculation with prompt caching
- Verified costs are non-negative and accurate
