Commit graph

606 commits

Author SHA1 Message Date
Krrish Dholakia
e1f54ef02c docs: refactor placement of adding guardrails to endpoints doc 2025-10-27 10:26:00 -07:00
Sameer Kankute
70650a044c
Add all sora models (#15937) 2025-10-26 10:10:57 -07:00
Krish Dholakia
346e036399
fix(opentelemetry.py): fix issue where headers were not being split correctly + feat(bedrock): add titan image generations w/ cost tracking (#15916)
* fix(opentelemetry.py): fix issue where headers were not being split correctly

* feat(bedrock/image): Support bedrock titan image generation

Closes https://github.com/BerriAI/litellm/issues/361

* build(model_prices_and_context_window.json): track titan image gen pricing

enables cost tracking per request

* feat(amazon_titan_transformation.py): support titan image generation cost tracking

* docs: document new model

* docs: update docs to indicate cost tracking + refactor rerank into separate doc

* fix: fix mypy linting error

* fix: fix type ignore
2025-10-25 13:45:13 -07:00
Krish Dholakia
2bd41dc034
Guardrails - Responses API, Image Gen, Text completions, Audio transcriptions, Audio Speech, Rerank, Anthropic Messages API support via the unified apply_guardrails function (#15706)
* fix(presidio.py): handle content as a list of texts

covers openai + anthropic messages api

* fix(presidio.py): safe get messages

* test: add unit testing for presidio guardrails

* fix(unified_guardrail.py): initial commit

* fix(enkryptai.py): implement apply_guardrail to enkrypt guardrail

* fix(unified_guardrail.py): support unified guardrail on input

* feat(unified_guardrail.py): add post call success hook implementation

allows us to just have 1 place to handle llm translation to guardrail api spec

* refactor: refactor initial unified guardrail component

* refactor: more refactoring

* feat(responses/): add guardrails to responses api

allows existing guardrails to work for new llm endpoints

* docs(adding_guardrail_support.md): document new guardrail endpoint support

* test: add unit tests

* feat(image_generation/): add guardrail support for image generation endpoint

* feat(openai/text_completion): support guardrails on `/v1/completions` API

* docs: document guardrails support on new endpoints

* docs: clarify when guardrails run

* feat(openai/speech): add guardrail support for input

* docs(rerank/): add guardrail support on input query

* fix: fix ruff check
2025-10-25 13:38:57 -07:00
Krish Dholakia
86524fcaf5
VertexAI Search Vector Store - Passthrough endpoint support + Vector store search Cost tracking support (#15824)
* feat(vector_stores/): initial commit adding Vertex AI Search API support for litellm

new vector store provider

* feat(vector_store/): use vector store id for vertex ai search api

* fix: transformation.py

cleanup

* fix: implement abstract function

* fix: fix linting error

* fix: main.py

fix check

* feat: initial commit with working passthrough support for vertex ai search api through litellm

* feat(llm_passthrough_endpoints.py): fix passing correct project on datastore passthrough

* feat(vertex_ai/): support passthrough call for vertex ai search vector store

* docs(vertex_ai_search_datastore.md): document new vertex ai passthrough endpoint

* docs(sidebars.js): document new endpoint

* feat: initial commit adding logging for vertex ai passthrough api

 allows vertex ai vector search api to work with cost calculation

* feat(vertex_ai/): search vector store cost tracking

* fix(vertex_passthrough_logging_handler.py): log the cost

* fix: improve logged response

* fix(vertex_passthrough_logging_handler.py): logging

* feat(litellm_logging): main.py

add cost tracking for vertex ai search api via unified api

* refactor: fix ruff checks

* fix(llm_passthrough_endpoints.py): fix linting
2025-10-25 13:17:15 -07:00
Krish Dholakia
f8d6a6edb9
fix(managed_files.py): don't raise error if managed object is not found + (Feat) Azure AI - Search Vector Stores + (Fix) Batches - “User default_user_id does not have access to the object” when object not in db + (fix) Vector Stores - show config.yaml vector stores on UI (#15873)
* fix(managed_files.py): don't raise error if managed object is not found

* feat(vector_stores): add azure ai search vector store support

Enables direct querying a vector store on azure

* fix(azure/vector_stores): working azure ai search api vector stores

allows azure direct querying on vector stores

* test: update env vars

* docs(docs/): document new azure ai vector store search

* docs(azure_ai_vector_stores.md): add table

* docs: clarify support for 'create' vector stores

* fix(vector_stores/endpoints.py): Fixes https://github.com/BerriAI/litellm/issues/14606

* fix: fix linting errors
2025-10-25 12:06:24 -07:00
Krish Dholakia
ddacaf6c32
(feat) Organizations: allow org admins to create teams on UI + (feat) IBM Guardrails (#15924)
* fix(oldteams.tsx): allow org admin to create team on ui

* fix(oldteams.tsx): show org admin a dropdown of allowed orgs for team creation

* docs(access_control.md): cleanup doc

* feat(ibm_guardrails/): initial commit adding support for ibm guardrails on litellm

allows user to use self-hosted ibm guardrails

* feat(ibm_detector.py): working detector

* docs(ibm_guardrails.md): document new ibm guardrails

* fix: fix linting errors
2025-10-25 11:13:39 -07:00
Ishaan Jaff
e4d5f00990
[Feat] New Guardrail - Dynamo AI Guardrail (#15920)
* add dynamo types

* fix Dynamo guard

* add dynamo guardrail

* add dynamo ai docs guard

* docs fix

* test dynamo

* test LASSO
2025-10-24 17:11:04 -07:00
Ishaan Jaff
3e4b5ef3a5
[Feat] Add cost tracking for Search API requests - Google PSE, Tavily, Parallel AI, Exa AI (#15821)
* add search cost tracking

* add cost tracking for tavily tiers

* add search to call types

* add search_provider_cost_per_query

* add cost tracking for search APIs

* add cost tracking search APIs

* docs cost tracking search

* docs search

* fix linting
2025-10-22 17:29:09 -07:00
Ishaan Jaff
f5a80110c1
[Feat] Add /search endpoint on LiteLLM Gateway (#15780)
* add SearchProvider

* add SearchToolTypedDict

* add search

* add SearchAPIRouter

* working router level search

* add search to allowed llm / ocr routes

* feat: add search_router

* add routing + proxy for search APIs

* /v1/search/{search_tool_name}

* fix search routing

* feat: parse_search_tools

* clean up sidebar

* docs fix

* router tests for search tools

* docs fix
2025-10-21 19:05:20 -07:00
Krrish Dholakia
2a1dbb5b9e docs(creating_adapters.md): document how to write an adapter 2025-10-21 16:20:31 -07:00
Krrish Dholakia
1e0368521e refactor: cleanup 2025-10-21 13:46:19 -07:00
Ishaan Jaff
8ad9bbbd02
[Docs] Add Azure AI - OCR to docs (#15768)
* add Azure OCR to docs

* docs fix

* docs fix

* docs fix

* docs OCR
2025-10-21 13:10:45 -07:00
Ishaan Jaffer
185182bebc Revert "add Azure OCR to docs"
This reverts commit a3699e28a4.
2025-10-21 13:02:31 -07:00
Ishaan Jaffer
a3699e28a4 add Azure OCR to docs 2025-10-21 13:02:21 -07:00
Ishaan Jaffer
6605aba307 docs grayswan 2025-10-21 11:16:25 -07:00
Ishaan Jaff
73a23a6c78
[Feat] Add Azure AVA TTS integration (#15749)
* add AzureBaseIssueTokenHandler

* add BaseTextToSpeechConfig

* async_text_to_speech_handler

* add AzureAVATextToSpeechConfig

* add get_provider_text_to_speech_config

* add AzureAVATextToSpeechConfig

* fixes for base_llm_http_handler

* fix transform_text_to_speech_request

* test_azure_ava_tts_async

* test_azure_ava_tts_async

* fix TextToSpeechRequestData

* fix transform_text_to_speech_request

* add text_to_speech_handler in LLMHttpHandler

* remove old file

* fix transform_text_to_speech_request

* fix dispatch_text_to_speech

* fix azure TTS

* fix AVA TTS

* fix transform

* fix linting

* ci/cd - use one job for audio testing

* fix tests

* fix llm http handler debugging

* unit tests azure tts

* docs Azure speech

* docs fix

* docs azure AVA

* docs azure AVA

* fix handlers

* test_async_realtime_uses_max_size_parameter
2025-10-20 16:52:23 -07:00
Krrish Dholakia
bbb7b75672 docs: sidebars.js
add vertex ai live api to docs
2025-10-17 23:55:44 -07:00
Ishaan Jaffer
ea69f4547d Merge branch 'main' into litellm_oct_staging2 2025-10-16 17:06:29 -07:00
Ishaan Jaffer
79516563a9 docs add /invoke and /converse routes 2025-10-16 16:25:27 -07:00
TensorNull
55a6dd3a8b feat(cometapi): Add CometAPI provider support (embeddings, image generation, docs)
- Add CometAPI embedding and image generation transformations and configs
- Add image cost calculator and export/init files
- Register provider in constants, utils, main (embedding path) and sidebars
- Add CometAPI docs page and cookbook notebook (Colab) for usage examples
2025-10-16 13:08:14 +08:00
Ishaan Jaff
bc26845ec4
[Feat] Native /ocr endpoint support (#15573)
* [Feat] Add native litellm.ocr() functions (#15567)

* fix get_supported_ocr_params

* add get_provider_ocr_config

* init OCR

* init ocr functions

* add OCRResponse Base Model

* add ocr to llm http handlers

* add main.py for OCR

* fix linting for OCR

* TestMistralOCR

* update to use DocumentType for Mistral

* fix _prepare_ocr_request

* fix transform

* add main.py for OCR

* add spec to init

* fix OCR

* TestMistralOCR

* ruff fix

* Potential fix for code scanning alert no. 3521: Clear-text logging of sensitive information

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* [Feat] Add /ocr route on LiteLLM AI Gateway - Adds support for native mistral ocr calling (#15571)

* fix get_supported_ocr_params

* add get_provider_ocr_config

* init OCR

* init ocr functions

* add OCRResponse Base Model

* add ocr to llm http handlers

* add main.py for OCR

* fix linting for OCR

* TestMistralOCR

* update to use DocumentType for Mistral

* fix _prepare_ocr_request

* fix transform

* add main.py for OCR

* add spec to init

* fix OCR

* TestMistralOCR

* ruff fix

* add router.ocr() methods

* add OCR routes

* feat add ocr routes

* add OCR routes

* feat: add OCR routes in proxy server

* working /ocr routes

* test_router_aocr_with_mistral

* docs Mistral OCR

* docs OCR

* [Feat] Add Azure AI Mistral OCR Integration  (#15572)

* fix get_supported_ocr_params

* add get_provider_ocr_config

* init OCR

* init ocr functions

* add OCRResponse Base Model

* add ocr to llm http handlers

* add main.py for OCR

* fix linting for OCR

* TestMistralOCR

* update to use DocumentType for Mistral

* fix _prepare_ocr_request

* fix transform

* add main.py for OCR

* add spec to init

* fix OCR

* TestMistralOCR

* ruff fix

* add router.ocr() methods

* add OCR routes

* feat add ocr routes

* add OCR routes

* feat: add OCR routes in proxy server

* working /ocr routes

* test_router_aocr_with_mistral

* docs Mistral OCR

* docs OCR

* add azure ai to get_provider_ocr_config

* add AzureAIOCRConfig

* TestAzureAIOCR

* TestAzureAIOCR

* test fixes for azure ai ocr

* fix async OCR transform for Azure

* fix transform_ocr_request

---------

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-10-15 17:20:01 -07:00
Ishaan Jaff
a6c57cb5bd
[Feat] Cost Tracking - specify a global vendor discount for costs. (#15546)
* fix cost_discount_config

* add CostBreakdown

* fix: set_cost_breakdown

* test_cost_discount_vertex_ai

* docs fix

* docs fix discounts

* docs fix

* docs custom pricing

* docs fix

* fixes for getting cost breakdown in response headers

* test - response headers wth discount
2025-10-14 20:07:04 -07:00
Ishaan Jaffer
7288f4355f docs fix 2025-10-11 16:24:21 -07:00
Ishaan Jaffer
54b8bde60c docs fix 2025-10-11 16:23:46 -07:00
Ishaan Jaff
527c8f59fa
[Feat] Tag Management - Add support for setting tag based budgets (#15433)
* feat: add LiteLLM_TagTable

* fix: use new table for tag management

* fix - allow setting budgets for tags

* working tag creation

* fix schema.prisma

* add tag info

* ui fixes

* ui fix tag info

* TAG_CACHE_IN_MEMORY_TTL_SECONDS

* add Litellm_EntityType

* fix get_aggregated_db_spend_update_transactions

* fix: _update_entity_spend_in_db

* fix _tag_max_budget_check

* add tag budget check

* add tag_list_transactions

* test_get_tag_objects_batch

* test_update_tag_db_without_prisma_client

* fix get_tags_from_request_body

* get_tags_from_request_body

* fix get_tags_from_request_body

* fix spend tracking utils

* get_tags_from_request_body

* test_get_tags_from_request_body_with_metadata_tags

* feat: add _update_tag_cache spend tracking

* fix _PROXY_track_cost_callback

* test_tag_cache_update_multiple_tags

* fix tag info

* docs fix

* docs tag budgets

* doc fix

* docs fix

* fix tag budget

* docs tag budgets

* docs fix

* ruff fix
2025-10-10 19:24:50 -07:00
Ishaan Jaff
bf209415da
[Feat] VertexAI Gemma model family streaming support + Added MedGemma (#15427)
* test_acompletion_filters_stream_and_stream_options

* fix: stream_options

* docs medgemma

* lint fix

* docs
2025-10-10 14:22:27 -07:00
Krish Dholakia
d944717d4b
Merge pull request #15278 from xprilion/wandb-inference-docs
Add W&B Inference documentation
2025-10-09 22:35:16 -07:00
Ishaan Jaff
6cf57d85c3
[Feat] Add EnkryptAI Guardrails on LiteLLM (#15390)
* feat: add EnkryptAIGuardrail

* fix types

* fix EnkryptAIGuardrails

* __all__ = ["EnkryptAIGuardrail"]
fix

* fix EnkryptAIGuardrail

* TestEnkryptAIExtraction

* fix mypy linting

* docs enkrypt ai

* add EnkryptAIGuardrailConfigModel

* fix EnkryptAIGuardrailConfigModel

* fix import struct

* fix EnkryptAIGuardrails
2025-10-09 16:27:04 -07:00
Anubhav Singh
f59f1971cc
Merge branch 'BerriAI:main' into wandb-inference-docs 2025-10-09 22:26:37 +05:30
Krrish Dholakia
f88e3ebdfa docs: refactor to have function call / computer use / etc. on top level 2025-10-09 07:25:25 -07:00
Ishaan Jaff
36c971a6fd
[MCP Gateway] QA/Fixes - Ensure Team/Key level enforcement works for MCPs (#15305)
* fix: _set_object_permission

* fix: _set_object_permission on teams

* fix: _set_object_permission

* fixes for team/key permissions

* statsh: object permission view

* fix: MCPServerPermissions

* fix: _get_team_object_permission

* test mcp checks for permissions

* fix server checks with prefix names

* test_list_tools_strips_prefix_when_matching_permissions

* ruff fix

* docs - refactor MCP

* docs update MCP docs

* docs allowed tools
2025-10-07 17:34:48 -07:00
xprilion
888391127f Add W&B Inference documentation 2025-10-07 19:18:45 +05:30
Krrish Dholakia
fbf7ca17ba docs(security_encryption_faq.md): answers common questions about data at rest vs. in transit 2025-10-06 17:34:10 -07:00
Ishaan Jaff
efa782d6d2
[Feat] Add Nvidia NIM Rerank Support (#15152)
* feat: add NvidiaNimRerankConfig

* fix: NvidiaNimRerankConfig

* fix: NvidiaNimRerankConfig

* fix routing to nvidia nim

* docs nvidia nim rerank

* TestNvidiaNim

* nvidia nim rerank fixes

* fix rerank

* transform_rerank_response

* Usage with LiteLLM Proxy

* fixes linting

* NvidiaNimRerankConfig.DEFAULT_NIM_RERANK_API_BASE

* fix Custom API Base URL

* fix rerank base

* fix main.py

* fix transform

* fix linting

* map_cohere_rerank_params

* ruff fix

* linting fixes

* ruff fix
2025-10-02 18:58:52 -07:00
Amir Refaee
8991657d67
added railtracks to projects using litellm (#15144) 2025-10-02 14:21:38 -07:00
Eddie Richter
929510ef5d Adding unit tests and documentation 2025-09-30 12:12:24 -06:00
abhijitjavelin
adea1bfbc4
Merge branch 'BerriAI:main' into main 2025-09-28 01:08:16 +05:30
Abhijit L
7bba372ac0 feat: add javelin guardrails 2025-09-28 01:07:08 +05:30
Krish Dholakia
3e3e3837a1
Merge branch 'main' into litellm_dev_09_24_2025_p1 2025-09-27 11:21:07 -07:00
Krrish Dholakia
5c6a863e40 docs: cleanup 2025-09-27 11:10:44 -07:00
Teddy Amkie
fdfff39222
Add Azure passthrough documentation and sidebar entry (#14958)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-09-27 10:13:16 -07:00
Ishaan Jaff
d04c6d4eea
[Feat] Add new anthropic web fetch tool support (#14951)
* add web_fetch tool for ANTHROPIC_HOSTED_TOOLS

* add ANTHROPIC_BETA_HEADER_VALUES, ANTHROPIC_HOSTED_TOOLS

* feat: add web fetch tool anthropic

* test_anthropic_tool_use

* docs web fetch

* docs fix

* docs fix
2025-09-26 11:04:11 -07:00
Teddy Amkie
dcbccd1fea
Corrected docs updates sept 2025 (#14916)
* docs: Corrected documentation updates from Sept 2025

This PR contains the actual intended documentation changes, properly synced with main:

✅ Real changes applied:
- Added AWS authentication link to bedrock guardrails documentation
- Updated Vertex AI with Gemini API alternative configuration
- Added async_post_call_success_hook code snippet to custom callback docs
- Added SSO free for up to 5 users information to enterprise and custom_sso docs
- Added SSO free information block to security.md
- Added cancel response API usage and curl example to response_api.md
- Added image for modifying default user budget via admin UI
- Re-ordered sidebars in documentation

❌ Sync issues resolved:
- Kept all upstream changes that were added to main after branch diverged
- Preserved Provider-Specific Metadata Parameters section that was added upstream
- Maintained proper curl parameter formatting (-d instead of -D)

This corrects the sync issues from the original PR #14769.

* docs: Restore missing files from original PR

Added back ~16 missing documentation files that were part of the original PR:

✅ Restored files:
- docs/my-website/docs/completion/usage.md
- docs/my-website/docs/fine_tuning.md
- docs/my-website/docs/getting_started.md
- docs/my-website/docs/image_edits.md
- docs/my-website/docs/image_generation.md
- docs/my-website/docs/index.md
- docs/my-website/docs/moderation.md
- docs/my-website/docs/observability/callbacks.md
- docs/my-website/docs/providers/bedrock.md
- docs/my-website/docs/proxy/caching.md
- docs/my-website/docs/proxy/config_settings.md
- docs/my-website/docs/proxy/db_deadlocks.md
- docs/my-website/docs/proxy/load_balancing.md
- docs/my-website/docs/proxy_api.md
- docs/my-website/docs/rerank.md

✅ Fixed context-caching issue:
- Restored provider_specific_params.md to main version (preserving Provider-Specific Metadata Parameters section)
- Your original PR didn't intend to modify this file - it was just a sync issue

Now includes all ~26 documentation files from the original PR #14769.

* docs: Remove files that were deleted in original PR

- Removed docs/my-website/docs/providers/azure_ai_img_edit.md (was deleted in original PR)
- sdk/headers.md was already not present

Now matches the complete intended changes from original PR #14769.

* docs: Restore azure_ai_img_edit.md from main

- Restored docs/my-website/docs/providers/azure_ai_img_edit.md from main branch
- This file should not have been deleted as it was a newer commit
- SDK headers file doesn't exist in main (was reverted) and wasn't part of your original changes

Fixes the file restoration issues.

* docs: Fix vertex.md - preserve context caching from newer commit

- Restored vertex.md to main version to preserve context caching content (lines 817-887)
- Added back only your intended change: alternative gemini config example
- Context caching content from newer commit is now preserved

Fixes the vertex.md sync issue where newer content was incorrectly deleted.

* docs: Fix providers/bedrock.md - restore deleted content from newer commit

- Restored providers/bedrock.md to main version
- Preserves 'Usage - Request Metadata' section that was added in newer commit
- Your actual intended change was to proxy/guardrails/bedrock.md (authentication tip) which is preserved
- Now only has additions, no subtractions as intended

Fixes the bedrock.md sync issue.

* docs: Restore missing IAM policy section in bedrock.md

Added back your intended IAM policy documentation that was lost when restoring main version:

✅ Added IAM AssumeRole Policy section:
- Explains requirement for sts:AssumeRole permission
- Shows error message example when permission missing
- Provides complete IAM policy JSON example
- Links to AWS AssumeRole documentation
- Clarifies trust policy requirements

Now bedrock.md has both:
- All newer content preserved (Request Metadata section)
- Your intended IAM policy addition restored

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-09-25 15:49:19 -07:00
Mubashir Osmani
fc1c39645a
docs: Letta Guide (#14798)
* fix: flaky passthrough tests

* Revert "fix: flaky passthrough tests"

This reverts commit ffe692e017.

* fix: serialize prisma objects

* docs: added letta

* arranged endpoints in alphabetical order

* added user_api_key fields to custom auth docs

* user onboarding

* cleaning up

This reverts commit 40e2aade73.

* cleaning up
This reverts commit ffe692e017.

* Revert "fix: serialize prisma objects"

This reverts commit 1e7bd13c26.
2025-09-23 16:18:57 -07:00
Ishaan Jaffer
45f835227b Revert "Merge pull request #14769 from TeddyAmkie/doc-updates-sept-2025"
This reverts commit 666bcac2af, reversing
changes made to b7803bcbb0.
2025-09-23 14:14:58 -07:00
Krish Dholakia
af151e6ba8
Merge branch 'main' into doc-updates-sept-2025 2025-09-22 22:41:02 -07:00
Krish Dholakia
66f7e4a832
Merge pull request #14790 from eycjur/support_flux_image_edit
[Feat] Support flux image edit
2025-09-22 22:38:45 -07:00
eycjur
f24d65b807 add docs 2025-09-23 02:51:19 +00:00
Krrish Dholakia
4c66e1f6bf docs(provider_specific_params.md): fix docs 2025-09-22 17:18:57 -07:00