scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-10 22:41:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 81

Merge branch 'BerriAI:main' into main

Browse source
This commit is contained in:
AnilAren 2025-10-31 09:03:22 +05:30 • committed by GitHub
commit 15fcf4477c
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
356 changed files with 15053 additions and 1508 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
.circleci/config.yml
View file

@ -2609,6 +2609,7 @@ jobs:
pip install "pytest-mock==3.12.0"
pip install "pytest-asyncio==0.21.1"
pip install "mypy==1.18.2"
pip install apscheduler
- run:
name: Build Docker image
command: |

3
.github/workflows/test-litellm.yml vendored
View file

@ -33,6 +33,7 @@ jobs:
poetry run pip install "google-genai==1.22.0"
poetry run pip install "google-cloud-aiplatform>=1.38"
poetry run pip install "fastapi-offline==1.7.3"
poetry run pip install "python-multipart==0.0.18"
- name: Setup litellm-enterprise as local package
run: |
cd enterprise
@ -40,4 +41,4 @@ jobs:
cd ..
- name: Run tests
run: |
poetry run pytest tests/test_litellm --tb=short -vv --maxfail=10 -n 4
poetry run pytest tests/test_litellm --tb=short -vv --maxfail=10 -n 4 --durations=50

147
README.md
View file

@ -301,53 +301,108 @@ curl 'http://0.0.0.0:4000/key/generate' \
}
```
## Supported Providers ([Docs](https://docs.litellm.ai/docs/providers))
## Supported Providers ([Website Supported Models](https://models.litellm.ai/) | [Docs](https://docs.litellm.ai/docs/providers))
| Provider | [Completion](https://docs.litellm.ai/docs/#basic-usage) | [Streaming](https://docs.litellm.ai/docs/completion/stream#streaming-responses) | [Async Completion](https://docs.litellm.ai/docs/completion/stream#async-completion) | [Async Streaming](https://docs.litellm.ai/docs/completion/stream#async-streaming) | [Async Embedding](https://docs.litellm.ai/docs/embedding/supported_embedding) | [Async Image Generation](https://docs.litellm.ai/docs/image_generation) |
|-------------------------------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------|-------------------------------------------------------------------------|
| [openai](https://docs.litellm.ai/docs/providers/openai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Meta - Llama API](https://docs.litellm.ai/docs/providers/meta_llama) | ✅ | ✅ | ✅ | ✅ | | |
| [azure](https://docs.litellm.ai/docs/providers/azure) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [AI/ML API](https://docs.litellm.ai/docs/providers/aiml) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [aws - sagemaker](https://docs.litellm.ai/docs/providers/aws_sagemaker) | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [aws - bedrock](https://docs.litellm.ai/docs/providers/bedrock) | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [google - vertex_ai](https://docs.litellm.ai/docs/providers/vertex) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [google - palm](https://docs.litellm.ai/docs/providers/palm) | ✅ | ✅ | ✅ | ✅ | | |
| [google AI Studio - gemini](https://docs.litellm.ai/docs/providers/gemini) | ✅ | ✅ | ✅ | ✅ | | |
| [mistral ai api](https://docs.litellm.ai/docs/providers/mistral) | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [cloudflare AI Workers](https://docs.litellm.ai/docs/providers/cloudflare_workers) | ✅ | ✅ | ✅ | ✅ | | |
| [CompactifAI](https://docs.litellm.ai/docs/providers/compactifai) | ✅ | ✅ | ✅ | ✅ | | |
| [cohere](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [anthropic](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | ✅ | | |
| [empower](https://docs.litellm.ai/docs/providers/empower) | ✅ | ✅ | ✅ | ✅ |
| [huggingface](https://docs.litellm.ai/docs/providers/huggingface) | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [replicate](https://docs.litellm.ai/docs/providers/replicate) | ✅ | ✅ | ✅ | ✅ | | |
| [together_ai](https://docs.litellm.ai/docs/providers/togetherai) | ✅ | ✅ | ✅ | ✅ | | |
| [openrouter](https://docs.litellm.ai/docs/providers/openrouter) | ✅ | ✅ | ✅ | ✅ | | |
| [ai21](https://docs.litellm.ai/docs/providers/ai21) | ✅ | ✅ | ✅ | ✅ | | |
| [baseten](https://docs.litellm.ai/docs/providers/baseten) | ✅ | ✅ | ✅ | ✅ | | |
| [vllm](https://docs.litellm.ai/docs/providers/vllm) | ✅ | ✅ | ✅ | ✅ | | |
| [nlp_cloud](https://docs.litellm.ai/docs/providers/nlp_cloud) | ✅ | ✅ | ✅ | ✅ | | |
| [aleph alpha](https://docs.litellm.ai/docs/providers/aleph_alpha) | ✅ | ✅ | ✅ | ✅ | | |
| [petals](https://docs.litellm.ai/docs/providers/petals) | ✅ | ✅ | ✅ | ✅ | | |
| [ollama](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [deepinfra](https://docs.litellm.ai/docs/providers/deepinfra) | ✅ | ✅ | ✅ | ✅ | | |
| [perplexity-ai](https://docs.litellm.ai/docs/providers/perplexity) | ✅ | ✅ | ✅ | ✅ | | |
| [Groq AI](https://docs.litellm.ai/docs/providers/groq) | ✅ | ✅ | ✅ | ✅ | | |
| [Deepseek](https://docs.litellm.ai/docs/providers/deepseek) | ✅ | ✅ | ✅ | ✅ | | |
| [anyscale](https://docs.litellm.ai/docs/providers/anyscale) | ✅ | ✅ | ✅ | ✅ | | |
| [IBM - watsonx.ai](https://docs.litellm.ai/docs/providers/watsonx) | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [voyage ai](https://docs.litellm.ai/docs/providers/voyage) | | | | | ✅ | |
| [xinference [Xorbits Inference]](https://docs.litellm.ai/docs/providers/xinference) | | | | | ✅ | |
| [FriendliAI](https://docs.litellm.ai/docs/providers/friendliai) | ✅ | ✅ | ✅ | ✅ | | |
| [Galadriel](https://docs.litellm.ai/docs/providers/galadriel) | ✅ | ✅ | ✅ | ✅ | | |
| [GradientAI](https://docs.litellm.ai/docs/providers/gradient_ai) | ✅ | ✅ | | | | |
| [Novita AI](https://novita.ai/models/llm?utm_source=github_litellm&utm_medium=github_readme&utm_campaign=github_link) | ✅ | ✅ | ✅ | ✅ | | |
| [Featherless AI](https://docs.litellm.ai/docs/providers/featherless_ai) | ✅ | ✅ | ✅ | ✅ | | |
| [Nebius AI Studio](https://docs.litellm.ai/docs/providers/nebius) | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [Heroku](https://docs.litellm.ai/docs/providers/heroku) | ✅ | ✅ | | | | |
| [OVHCloud AI Endpoints](https://docs.litellm.ai/docs/providers/ovhcloud) | ✅ | ✅ | | | | |
| [CometAPI](https://docs.litellm.ai/docs/providers/cometapi) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Provider | `/chat/completions` | `/messages` | `/responses` | `/embeddings` | `/image/generations` | `/audio/transcriptions` | `/audio/speech` | `/moderations` | `/batches` | `/rerank` |
|-------------------------------------------------------------------------------------|---------------------|-------------|--------------|---------------|----------------------|-------------------------|-----------------|----------------|-----------|-----------|
| [AI/ML API (`aiml`)](https://docs.litellm.ai/docs/providers/aiml) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
| [AI21 (`ai21`)](https://docs.litellm.ai/docs/providers/ai21) | ✅ | ✅ | ✅ | | | | | | | |
| [AI21 Chat (`ai21_chat`)](https://docs.litellm.ai/docs/providers/ai21) | ✅ | ✅ | ✅ | | | | | | | |
| [Aleph Alpha](https://docs.litellm.ai/docs/providers/aleph_alpha) | ✅ | ✅ | ✅ | | | | | | | |
| [Anthropic (`anthropic`)](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | |
| [Anthropic Text (`anthropic_text`)](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | |
| [Anyscale](https://docs.litellm.ai/docs/providers/anyscale) | ✅ | ✅ | ✅ | | | | | | | |
| [AssemblyAI (`assemblyai`)](https://docs.litellm.ai/docs/pass_through/assembly_ai) | ✅ | ✅ | ✅ | | | ✅ | | | | |
| [Auto Router (`auto_router`)](https://docs.litellm.ai/docs/proxy/auto_routing) | ✅ | ✅ | ✅ | | | | | | | |
| [AWS - Bedrock (`bedrock`)](https://docs.litellm.ai/docs/providers/bedrock) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ |
| [AWS - Sagemaker (`sagemaker`)](https://docs.litellm.ai/docs/providers/aws_sagemaker) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| [Azure (`azure`)](https://docs.litellm.ai/docs/providers/azure) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [Azure AI (`azure_ai`)](https://docs.litellm.ai/docs/providers/azure_ai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [Azure Text (`azure_text`)](https://docs.litellm.ai/docs/providers/azure) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |
| [Baseten (`baseten`)](https://docs.litellm.ai/docs/providers/baseten) | ✅ | ✅ | ✅ | | | | | | | |
| [Bytez (`bytez`)](https://docs.litellm.ai/docs/providers/bytez) | ✅ | ✅ | ✅ | | | | | | | |
| [Cerebras (`cerebras`)](https://docs.litellm.ai/docs/providers/cerebras) | ✅ | ✅ | ✅ | | | | | | | |
| [Clarifai (`clarifai`)](https://docs.litellm.ai/docs/providers/clarifai) | ✅ | ✅ | ✅ | | | | | | | |
| [Cloudflare AI Workers (`cloudflare`)](https://docs.litellm.ai/docs/providers/cloudflare_workers) | ✅ | ✅ | ✅ | | | | | | | |
| [Codestral (`codestral`)](https://docs.litellm.ai/docs/providers/codestral) | ✅ | ✅ | ✅ | | | | | | | |
| [Cohere (`cohere`)](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ |
| [Cohere Chat (`cohere_chat`)](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | ✅ | | | | | | | |
| [CometAPI (`cometapi`)](https://docs.litellm.ai/docs/providers/cometapi) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| [CompactifAI (`compactifai`)](https://docs.litellm.ai/docs/providers/compactifai) | ✅ | ✅ | ✅ | | | | | | | |
| [Custom (`custom`)](https://docs.litellm.ai/docs/providers/custom_llm_server) | ✅ | ✅ | ✅ | | | | | | | |
| [Custom OpenAI (`custom_openai`)](https://docs.litellm.ai/docs/providers/openai_compatible) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |
| [Dashscope (`dashscope`)](https://docs.litellm.ai/docs/providers/dashscope) | ✅ | ✅ | ✅ | | | | | | | |
| [Databricks (`databricks`)](https://docs.litellm.ai/docs/providers/databricks) | ✅ | ✅ | ✅ | | | | | | | |
| [DataRobot (`datarobot`)](https://docs.litellm.ai/docs/providers/datarobot) | ✅ | ✅ | ✅ | | | | | | | |
| [Deepgram (`deepgram`)](https://docs.litellm.ai/docs/providers/deepgram) | ✅ | ✅ | ✅ | | | ✅ | | | | |
| [DeepInfra (`deepinfra`)](https://docs.litellm.ai/docs/providers/deepinfra) | ✅ | ✅ | ✅ | | | | | | | |
| [Deepseek (`deepseek`)](https://docs.litellm.ai/docs/providers/deepseek) | ✅ | ✅ | ✅ | | | | | | | |
| [ElevenLabs (`elevenlabs`)](https://docs.litellm.ai/docs/providers/elevenlabs) | ✅ | ✅ | ✅ | | | | ✅ | | | |
| [Empower (`empower`)](https://docs.litellm.ai/docs/providers/empower) | ✅ | ✅ | ✅ | | | | | | | |
| [Fal AI (`fal_ai`)](https://docs.litellm.ai/docs/providers/fal_ai) | ✅ | ✅ | ✅ | | ✅ | | | | | |
| [Featherless AI (`featherless_ai`)](https://docs.litellm.ai/docs/providers/featherless_ai) | ✅ | ✅ | ✅ | | | | | | | |
| [Fireworks AI (`fireworks_ai`)](https://docs.litellm.ai/docs/providers/fireworks_ai) | ✅ | ✅ | ✅ | | | | | | | |
| [FriendliAI (`friendliai`)](https://docs.litellm.ai/docs/providers/friendliai) | ✅ | ✅ | ✅ | | | | | | | |
| [Galadriel (`galadriel`)](https://docs.litellm.ai/docs/providers/galadriel) | ✅ | ✅ | ✅ | | | | | | | |
| [GitHub Copilot (`github_copilot`)](https://docs.litellm.ai/docs/providers/github_copilot) | ✅ | ✅ | ✅ | | | | | | | |
| [GitHub Models (`github`)](https://docs.litellm.ai/docs/providers/github) | ✅ | ✅ | ✅ | | | | | | | |
| [Google - PaLM](https://docs.litellm.ai/docs/providers/palm) | ✅ | ✅ | ✅ | | | | | | | |
| [Google - Vertex AI (`vertex_ai`)](https://docs.litellm.ai/docs/providers/vertex) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
| [Google AI Studio - Gemini (`gemini`)](https://docs.litellm.ai/docs/providers/gemini) | ✅ | ✅ | ✅ | | | | | | | |
| [GradientAI (`gradient_ai`)](https://docs.litellm.ai/docs/providers/gradient_ai) | ✅ | ✅ | ✅ | | | | | | | |
| [Groq AI (`groq`)](https://docs.litellm.ai/docs/providers/groq) | ✅ | ✅ | ✅ | | | | | | | |
| [Heroku (`heroku`)](https://docs.litellm.ai/docs/providers/heroku) | ✅ | ✅ | ✅ | | | | | | | |
| [Hosted VLLM (`hosted_vllm`)](https://docs.litellm.ai/docs/providers/vllm) | ✅ | ✅ | ✅ | | | | | | | |
| [Huggingface (`huggingface`)](https://docs.litellm.ai/docs/providers/huggingface) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ |
| [Hyperbolic (`hyperbolic`)](https://docs.litellm.ai/docs/providers/hyperbolic) | ✅ | ✅ | ✅ | | | | | | | |
| [IBM - Watsonx.ai (`watsonx`)](https://docs.litellm.ai/docs/providers/watsonx) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| [Infinity (`infinity`)](https://docs.litellm.ai/docs/providers/infinity) | | | | ✅ | | | | | | |
| [Jina AI (`jina_ai`)](https://docs.litellm.ai/docs/providers/jina_ai) | | | | ✅ | | | | | | |
| [Lambda AI (`lambda_ai`)](https://docs.litellm.ai/docs/providers/lambda_ai) | ✅ | ✅ | ✅ | | | | | | | |
| [Lemonade (`lemonade`)](https://docs.litellm.ai/docs/providers/lemonade) | ✅ | ✅ | ✅ | | | | | | | |
| [LiteLLM Proxy (`litellm_proxy`)](https://docs.litellm.ai/docs/providers/litellm_proxy) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
| [Llamafile (`llamafile`)](https://docs.litellm.ai/docs/providers/llamafile) | ✅ | ✅ | ✅ | | | | | | | |
| [LM Studio (`lm_studio`)](https://docs.litellm.ai/docs/providers/lm_studio) | ✅ | ✅ | ✅ | | | | | | | |
| [Maritalk (`maritalk`)](https://docs.litellm.ai/docs/providers/maritalk) | ✅ | ✅ | ✅ | | | | | | | |
| [Meta - Llama API (`meta_llama`)](https://docs.litellm.ai/docs/providers/meta_llama) | ✅ | ✅ | ✅ | | | | | | | |
| [Mistral AI API (`mistral`)](https://docs.litellm.ai/docs/providers/mistral) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| [Moonshot (`moonshot`)](https://docs.litellm.ai/docs/providers/moonshot) | ✅ | ✅ | ✅ | | | | | | | |
| [Morph (`morph`)](https://docs.litellm.ai/docs/providers/morph) | ✅ | ✅ | ✅ | | | | | | | |
| [Nebius AI Studio (`nebius`)](https://docs.litellm.ai/docs/providers/nebius) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| [NLP Cloud (`nlp_cloud`)](https://docs.litellm.ai/docs/providers/nlp_cloud) | ✅ | ✅ | ✅ | | | | | | | |
| [Novita AI (`novita`)](https://novita.ai/models/llm?utm_source=github_litellm&utm_medium=github_readme&utm_campaign=github_link) | ✅ | ✅ | ✅ | | | | | | | |
| [Nscale (`nscale`)](https://docs.litellm.ai/docs/providers/nscale) | ✅ | ✅ | ✅ | | | | | | | |
| [Nvidia NIM (`nvidia_nim`)](https://docs.litellm.ai/docs/providers/nvidia_nim) | ✅ | ✅ | ✅ | | | | | | | |
| [OCI (`oci`)](https://docs.litellm.ai/docs/providers/oci) | ✅ | ✅ | ✅ | | | | | | | |
| [Ollama (`ollama`)](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| [Ollama Chat (`ollama_chat`)](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | ✅ | | | | | | | |
| [Oobabooga (`oobabooga`)](https://docs.litellm.ai/docs/providers/openai_compatible) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |
| [OpenAI (`openai`)](https://docs.litellm.ai/docs/providers/openai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| [OpenAI-like (`openai_like`)](https://docs.litellm.ai/docs/providers/openai_compatible) | | | | ✅ | | | | | | |
| [OpenRouter (`openrouter`)](https://docs.litellm.ai/docs/providers/openrouter) | ✅ | ✅ | ✅ | | | | | | | |
| [OVHCloud AI Endpoints (`ovhcloud`)](https://docs.litellm.ai/docs/providers/ovhcloud) | ✅ | ✅ | ✅ | | | | | | | |
| [Perplexity AI (`perplexity`)](https://docs.litellm.ai/docs/providers/perplexity) | ✅ | ✅ | ✅ | | | | | | | |
| [Petals (`petals`)](https://docs.litellm.ai/docs/providers/petals) | ✅ | ✅ | ✅ | | | | | | | |
| [Predibase (`predibase`)](https://docs.litellm.ai/docs/providers/predibase) | ✅ | ✅ | ✅ | | | | | | | |
| [Recraft (`recraft`)](https://docs.litellm.ai/docs/providers/recraft) | | | | | ✅ | | | | | |
| [Replicate (`replicate`)](https://docs.litellm.ai/docs/providers/replicate) | ✅ | ✅ | ✅ | | | | | | | |
| [Sagemaker Chat (`sagemaker_chat`)](https://docs.litellm.ai/docs/providers/aws_sagemaker) | ✅ | ✅ | ✅ | | | | | | | |
| [Sambanova (`sambanova`)](https://docs.litellm.ai/docs/providers/sambanova) | ✅ | ✅ | ✅ | | | | | | | |
| [Snowflake (`snowflake`)](https://docs.litellm.ai/docs/providers/snowflake) | ✅ | ✅ | ✅ | | | | | | | |
| [Text Completion Codestral (`text-completion-codestral`)](https://docs.litellm.ai/docs/providers/codestral) | ✅ | ✅ | ✅ | | | | | | | |
| [Text Completion OpenAI (`text-completion-openai`)](https://docs.litellm.ai/docs/providers/text_completion_openai) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |
| [Together AI (`together_ai`)](https://docs.litellm.ai/docs/providers/togetherai) | ✅ | ✅ | ✅ | | | | | | | |
| [Topaz (`topaz`)](https://docs.litellm.ai/docs/providers/topaz) | ✅ | ✅ | ✅ | | | | | | | |
| [Triton (`triton`)](https://docs.litellm.ai/docs/providers/triton-inference-server) | ✅ | ✅ | ✅ | | | | | | | |
| [V0 (`v0`)](https://docs.litellm.ai/docs/providers/v0) | ✅ | ✅ | ✅ | | | | | | | |
| [Vercel AI Gateway (`vercel_ai_gateway`)](https://docs.litellm.ai/docs/providers/vercel_ai_gateway) | ✅ | ✅ | ✅ | | | | | | | |
| [VLLM (`vllm`)](https://docs.litellm.ai/docs/providers/vllm) | ✅ | ✅ | ✅ | | | | | | | |
| [Volcengine (`volcengine`)](https://docs.litellm.ai/docs/providers/volcano) | ✅ | ✅ | ✅ | | | | | | | |
| [Voyage AI (`voyage`)](https://docs.litellm.ai/docs/providers/voyage) | | | | ✅ | | | | | | |
| [WandB Inference (`wandb`)](https://docs.litellm.ai/docs/providers/wandb_inference) | ✅ | ✅ | ✅ | | | | | | | |
| [Watsonx Text (`watsonx_text`)](https://docs.litellm.ai/docs/providers/watsonx) | ✅ | ✅ | ✅ | | | | | | | |
| [xAI (`xai`)](https://docs.litellm.ai/docs/providers/xai) | ✅ | ✅ | ✅ | | | | | | | |
| [Xinference (`xinference`)](https://docs.litellm.ai/docs/providers/xinference) | | | | ✅ | | | | | | |
[**Read the Docs**](https://docs.litellm.ai/docs/)

4
batch_small.jsonl Normal file
View file

@ -0,0 +1,4 @@
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello, how are you?"}]}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "What is the weather today?"}]}}
{"custom_id": "request-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Tell me a short joke"}]}}

2
docs/my-website/docs/completion/prompt_caching.md
View file

@ -27,7 +27,7 @@ For the supported providers, LiteLLM follows the OpenAI prompt caching usage obj
}
```
- `prompt_tokens`: These are the non-cached prompt tokens (same as Anthropic, equivalent to Deepseek `prompt_cache_miss_tokens`).
- `prompt_tokens`: These are all prompt tokens including cache-miss and cache-hit input tokens.
- `completion_tokens`: These are the output tokens generated by the model.
- `total_tokens`: Sum of prompt_tokens + completion_tokens.
- `prompt_tokens_details`: Object containing cached_tokens.

10
docs/my-website/docs/files_endpoints.md
View file

@ -57,7 +57,7 @@ client = OpenAI(
client.files.create(
file=wav_data,
purpose="user_data",
extra_body={"custom_llm_provider": "openai"}
extra_headers={"custom-llm-provider": "openai"}
)
```
@ -71,7 +71,7 @@ client = OpenAI(
base_url="http://0.0.0.0:4000/v1"
)
files = client.files.list(extra_body={"custom_llm_provider": "openai"})
files = client.files.list(extra_headers={"custom-llm-provider": "openai"})
print("files=", files)
```
@ -85,7 +85,7 @@ client = OpenAI(
base_url="http://0.0.0.0:4000/v1"
)
file = client.files.retrieve(file_id="file-abc123", extra_body={"custom_llm_provider": "openai"})
file = client.files.retrieve(file_id="file-abc123", extra_headers={"custom-llm-provider": "openai"})
print("file=", file)
```
@ -99,7 +99,7 @@ client = OpenAI(
base_url="http://0.0.0.0:4000/v1"
)
response = client.files.delete(file_id="file-abc123", extra_body={"custom_llm_provider": "openai"})
response = client.files.delete(file_id="file-abc123", extra_headers={"custom-llm-provider": "openai"})
print("delete response=", response)
```
@ -113,7 +113,7 @@ client = OpenAI(
base_url="http://0.0.0.0:4000/v1"
)
content = client.files.content(file_id="file-abc123", extra_body={"custom_llm_provider": "openai"})
content = client.files.content(file_id="file-abc123", extra_headers={"custom-llm-provider": "openai"})
print("content=", content)
```

19
docs/my-website/docs/fine_tuning.md
View file

@ -62,7 +62,7 @@ client = AsyncOpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000") # base_u
file_name = "openai_batch_completions.jsonl"
response = await client.files.create(
extra_body={"custom_llm_provider": "azure"}, # tell litellm proxy which provider to use
extra_headers={"custom-llm-provider": "azure"}, # tell litellm proxy which provider to use
file=open(file_name, "rb"),
purpose="fine-tune",
)
@ -73,8 +73,8 @@ response = await client.files.create(
```shell
curl http://localhost:4000/v1/files \
-H "Authorization: Bearer sk-1234" \
-H "custom-llm-provider: azure" \
-F purpose="batch" \
-F custom_llm_provider="azure"\
-F file="@mydata.jsonl"
```
</TabItem>
@ -92,7 +92,7 @@ curl http://localhost:4000/v1/files \
ft_job = await client.fine_tuning.jobs.create(
model="gpt-35-turbo-1106", # Azure OpenAI model you want to fine-tune
training_file="file-abc123", # file_id from create file response
extra_body={"custom_llm_provider": "azure"}, # tell litellm proxy which provider to use
extra_headers={"custom-llm-provider": "azure"}, # tell litellm proxy which provider to use
)
```
</TabItem>
@ -103,8 +103,8 @@ ft_job = await client.fine_tuning.jobs.create(
curl http://localhost:4000/v1/fine_tuning/jobs \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-H "custom-llm-provider: azure" \
-d '{
"custom_llm_provider": "azure",
"model": "gpt-35-turbo-1106",
"training_file": "file-abc123"
}'
@ -215,7 +215,7 @@ curl http://localhost:4000/v1/fine_tuning/jobs \
# cancel specific fine tuning job
cancel_ft_job = await client.fine_tuning.jobs.cancel(
fine_tuning_job_id="123", # fine tuning job id
extra_body={"custom_llm_provider": "azure"}, # tell litellm proxy which provider to use
extra_headers={"custom-llm-provider": "azure"}, # tell litellm proxy which provider to use
)
print("response from cancel ft job={}".format(cancel_ft_job))
@ -228,7 +228,7 @@ print("response from cancel ft job={}".format(cancel_ft_job))
curl -X POST http://localhost:4000/v1/fine_tuning/jobs/ftjob-abc123/cancel \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{"custom_llm_provider": "azure"}'
-H "custom-llm-provider: azure"
```
</TabItem>
@ -242,7 +242,7 @@ curl -X POST http://localhost:4000/v1/fine_tuning/jobs/ftjob-abc123/cancel \
```python
list_ft_jobs = await client.fine_tuning.jobs.list(
extra_query={"custom_llm_provider": "azure"} # tell litellm proxy which provider to use
extra_headers={"custom-llm-provider": "azure"} # tell litellm proxy which provider to use
)
print("list of ft jobs={}".format(list_ft_jobs))
@ -252,9 +252,10 @@ print("list of ft jobs={}".format(list_ft_jobs))
<TabItem value="curl" label="curl">
```shell
curl -X GET 'http://localhost:4000/v1/fine_tuning/jobs?custom_llm_provider=azure' \
curl -X GET 'http://localhost:4000/v1/fine_tuning/jobs' \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234"
-H "Authorization: Bearer sk-1234" \
-H "custom-llm-provider: azure"
```
</TabItem>

34
docs/my-website/docs/observability/opik_integration.md
View file

@ -220,7 +220,41 @@ curl --location --request POST 'http://0.0.0.0:4000/chat/completions' \
}'
```
## Automatic Metadata from API Keys
In some cases, the requester may be unable or unaware of how to add Opik metadata to their requests. To ensure all Opik-related actions are properly tracked, LiteLLM Proxy can automatically associate metadata from a user-specific API key when none is provided in the request.
### How It Works
When you create an API key in LiteLLM Proxy, you can attach Opik-specific metadata to the key itself. This metadata will be automatically applied to all requests made with that key, unless the request explicitly provides its own Opik metadata (which takes precedence).
### Usage
**Step 1: Save Opik Metadata to the corresponding Api Key**
Go to 'Virtual Keys', click on your choosen api key and edit 'Settings'.
Now save the opik metadata as user api key metdata.
<Image img={require('../../img/opik_key_metadata.png')} />
**Step 2: Use the key - Opik metadata is automatically applied**
```bash
curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-key-from-step-1' \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "user",
"content": "What's the weather like in Boston today?"
}
]
}'
```
All requests made with this key will automatically be tracked in the "TestProject" Opik project with the specified tags, without requiring the user to pass metadata in each request.
## Support & Talk to Founders

10
docs/my-website/docs/providers/azure/azure.md
View file

@ -834,7 +834,7 @@ client = OpenAI(
batch_input_file = client.files.create(
file=open("mydata.jsonl", "rb"),
purpose="batch",
extra_body={"custom_llm_provider": "azure"}
extra_headers={"custom-llm-provider": "azure"}
)
file_id = batch_input_file.id
```
@ -870,7 +870,7 @@ batch = client.batches.create( # re use client from above
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={"description": "My batch job"},
extra_body={"custom_llm_provider": "azure"}
extra_headers={"custom-llm-provider": "azure"}
)
```
@ -898,7 +898,7 @@ curl http://localhost:4000/v1/batches \
```python
retrieved_batch = client.batches.retrieve(
batch.id,
extra_query={"custom_llm_provider": "azure"}
extra_headers={"custom-llm-provider": "azure"}
)
```
@ -922,7 +922,7 @@ curl http://localhost:4000/v1/batches/batch_abc123 \
```python
cancelled_batch = client.batches.cancel(
batch.id,
extra_body={"custom_llm_provider": "azure"}
extra_headers={"custom-llm-provider": "azure"}
)
```
@ -945,7 +945,7 @@ curl http://localhost:4000/v1/batches/batch_abc123/cancel \
<TabItem value="sdk" label="OpenAI Python SDK">
```python
client.batches.list(extra_query={"custom_llm_provider": "azure"})
client.batches.list(extra_headers={"custom-llm-provider": "azure"})
```
</TabItem>

12
docs/my-website/docs/providers/azure/videos.md
View file

@ -31,7 +31,7 @@ os.environ["AZURE_OPENAI_API_VERSION"] = "2024-02-15-preview"
### Basic Usage
```python
from litellm import video_generation, video_status, video_retrieval
from litellm import video_generation, video_status, video_content
import os
import time
@ -68,7 +68,7 @@ while True:
time.sleep(10) # Wait 10 seconds before checking again
# Download video content when ready
video_bytes = video_retrieval(
video_bytes = video_content(
video_id=response.id,
model="azure/sora-2"
)
@ -146,10 +146,10 @@ client = openai.OpenAI(
)
# request sent to model set on litellm proxy, `litellm --model`
response = client.videos.generations.create(
response = client.videos.create(
model="azure-sora-2",
prompt="A cat playing with a ball of yarn in a sunny garden",
seconds="8",
seconds=8,
size="720x1280"
)
@ -210,7 +210,7 @@ general_settings:
```python
# Download video content
video_bytes = video_retrieval(
video_bytes = video_content(
video_id="video_1234567890",
model="azure/sora-2"
)
@ -242,7 +242,7 @@ def generate_and_download_video(prompt):
time.sleep(30)
# Step 3: Download video
video_bytes = litellm.video_retrieval(
video_bytes = litellm.video_content(
video_id=video_id,
model="azure/sora-2"
)

310
docs/my-website/docs/providers/fal_ai.md Normal file
View file

@ -0,0 +1,310 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Fal AI
Fal AI provides fast, scalable access to state-of-the-art image generation models including FLUX, Stable Diffusion, Imagen, and more.
## Overview
| Property | Details |
|----------|---------|
| Description | Fal AI offers optimized infrastructure for running image generation models at scale with low latency. |
| Provider Route on LiteLLM | `fal_ai/` |
| Provider Doc | [Fal AI Documentation ↗](https://fal.ai/models) |
| Supported Operations | [`/images/generations`](#image-generation) |
## Setup
### API Key
```python showLineNumbers
import os
# Set your Fal AI API key
os.environ["FAL_AI_API_KEY"] = "your-fal-api-key"
```
Get your API key from [fal.ai](https://fal.ai/).
## Supported Models
| Model Name | Description | Documentation |
|------------|-------------|---------------|
| `fal_ai/fal-ai/flux-pro/v1.1-ultra` | FLUX Pro v1.1 Ultra - High-quality image generation | [Docs ↗](https://fal.ai/models/fal-ai/flux-pro/v1.1-ultra) |
| `fal_ai/fal-ai/imagen4/preview` | Google's Imagen 4 - Highest quality model | [Docs ↗](https://fal.ai/models/fal-ai/imagen4/preview) |
| `fal_ai/fal-ai/recraft/v3/text-to-image` | Recraft v3 - Multiple style options | [Docs ↗](https://fal.ai/models/fal-ai/recraft/v3/text-to-image) |
| `fal_ai/fal-ai/stable-diffusion-v35-medium` | Stable Diffusion v3.5 Medium | [Docs ↗](https://fal.ai/models/fal-ai/stable-diffusion-v35-medium) |
| `fal_ai/bria/text-to-image/3.2` | Bria 3.2 - Commercial-grade generation | [Docs ↗](https://fal.ai/models/bria/text-to-image/3.2) |
## Image Generation
### Usage - LiteLLM Python SDK
<Tabs>
<TabItem value="basic" label="Basic Usage">
```python showLineNumbers title="Basic Image Generation"
import litellm
import os
# Set your API key
os.environ["FAL_AI_API_KEY"] = "your-fal-api-key"
# Generate an image
response = litellm.image_generation(
model="fal_ai/fal-ai/flux-pro/v1.1-ultra",
prompt="A serene mountain landscape at sunset with vibrant colors"
)
print(response.data[0].url)
```
</TabItem>
<TabItem value="imagen4" label="Imagen 4">
```python showLineNumbers title="Google Imagen 4 Generation"
import litellm
import os
os.environ["FAL_AI_API_KEY"] = "your-fal-api-key"
# Generate with Imagen 4
response = litellm.image_generation(
model="fal_ai/fal-ai/imagen4/preview",
prompt="A vintage 1960s kitchen with flour package on countertop",
aspect_ratio="16:9",
num_images=1
)
print(response.data[0].url)
```
</TabItem>
<TabItem value="recraft" label="Recraft v3">
```python showLineNumbers title="Recraft v3 with Style"
import litellm
import os
os.environ["FAL_AI_API_KEY"] = "your-fal-api-key"
# Generate with specific style
response = litellm.image_generation(
model="fal_ai/fal-ai/recraft/v3/text-to-image",
prompt="A red panda eating bamboo",
style="realistic_image",
image_size="landscape_4_3"
)
print(response.data[0].url)
```
</TabItem>
<TabItem value="async" label="Async Usage">
```python showLineNumbers title="Async Image Generation"
import litellm
import asyncio
import os
async def generate_image():
os.environ["FAL_AI_API_KEY"] = "your-fal-api-key"
response = await litellm.aimage_generation(
model="fal_ai/fal-ai/stable-diffusion-v35-medium",
prompt="A cyberpunk cityscape with neon lights",
guidance_scale=7.5,
num_inference_steps=50
)
print(response.data[0].url)
return response
asyncio.run(generate_image())
```
</TabItem>
<TabItem value="advanced" label="Advanced Parameters">
```python showLineNumbers title="Advanced FLUX Pro Generation"
import litellm
import os
os.environ["FAL_AI_API_KEY"] = "your-fal-api-key"
# Generate with advanced parameters
response = litellm.image_generation(
model="fal_ai/fal-ai/flux-pro/v1.1-ultra",
prompt="A majestic dragon soaring over mountains",
n=2,
size="1792x1024", # Maps to aspect_ratio="16:9"
seed=42,
safety_tolerance="2",
enhance_prompt=True
)
for image in response.data:
print(f"Generated image: {image.url}")
```
</TabItem>
</Tabs>
### Usage - LiteLLM Proxy Server
#### 1. Configure your config.yaml
```yaml showLineNumbers title="Fal AI Image Generation Configuration"
model_list:
- model_name: flux-ultra
litellm_params:
model: fal_ai/fal-ai/flux-pro/v1.1-ultra
api_key: os.environ/FAL_AI_API_KEY
model_info:
mode: image_generation
- model_name: imagen4
litellm_params:
model: fal_ai/fal-ai/imagen4/preview
api_key: os.environ/FAL_AI_API_KEY
model_info:
mode: image_generation
- model_name: stable-diffusion
litellm_params:
model: fal_ai/fal-ai/stable-diffusion-v35-medium
api_key: os.environ/FAL_AI_API_KEY
model_info:
mode: image_generation
general_settings:
master_key: sk-1234
```
#### 2. Start LiteLLM Proxy Server
```bash showLineNumbers title="Start Proxy Server"
litellm --config /path/to/config.yaml
# RUNNING on http://0.0.0.0:4000
```
#### 3. Make requests
<Tabs>
<TabItem value="openai-sdk" label="OpenAI SDK">
```python showLineNumbers title="Generate via Proxy - OpenAI SDK"
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000",
api_key="sk-1234"
)
response = client.images.generate(
model="flux-ultra",
prompt="A beautiful sunset over the ocean",
n=1,
size="1024x1024"
)
print(response.data[0].url)
```
</TabItem>
<TabItem value="litellm-sdk" label="LiteLLM SDK">
```python showLineNumbers title="Generate via Proxy - LiteLLM SDK"
import litellm
response = litellm.image_generation(
model="litellm_proxy/imagen4",
prompt="A cozy coffee shop interior",
api_base="http://localhost:4000",
api_key="sk-1234"
)
print(response.data[0].url)
```
</TabItem>
<TabItem value="curl" label="cURL">
```bash showLineNumbers title="Generate via Proxy - cURL"
curl --location 'http://localhost:4000/v1/images/generations' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
"model": "stable-diffusion",
"prompt": "A serene Japanese garden with cherry blossoms",
"n": 1,
"size": "1024x1024"
}'
```
</TabItem>
</Tabs>
## Using Model-Specific Parameters
LiteLLM forwards any additional parameters directly to the Fal AI API. You can pass model-specific parameters in your request and they will be sent to Fal AI.
```python showLineNumbers title="Pass Model-Specific Parameters"
import litellm
# Any parameters beyond the standard ones are forwarded to Fal AI
response = litellm.image_generation(
model="fal_ai/fal-ai/flux-pro/v1.1-ultra",
prompt="A beautiful sunset",
# Model-specific Fal AI parameters
aspect_ratio="16:9",
safety_tolerance="2",
enhance_prompt=True,
seed=42
)
```
For the complete list of parameters supported by each model, see:
- [FLUX Pro v1.1-ultra Parameters ↗](https://fal.ai/models/fal-ai/flux-pro/v1.1-ultra/api)
- [Imagen 4 Parameters ↗](https://fal.ai/models/fal-ai/imagen4/preview/api)
- [Recraft v3 Parameters ↗](https://fal.ai/models/fal-ai/recraft/v3/text-to-image/api)
- [Stable Diffusion v3.5 Parameters ↗](https://fal.ai/models/fal-ai/stable-diffusion-v35-medium/api)
- [Bria 3.2 Parameters ↗](https://fal.ai/models/bria/text-to-image/3.2/api)
## Supported Parameters
Standard OpenAI-compatible parameters that work across all models:
| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `prompt` | string | Text description of desired image | Required |
| `model` | string | Fal AI model to use | Required |
| `n` | integer | Number of images to generate (1-4) | `1` |
| `size` | string | Image dimensions (maps to model-specific format) | Model default |
| `api_key` | string | Your Fal AI API key | Environment variable |
## Getting Started
1. Sign up at [fal.ai](https://fal.ai/)
2. Get your API key from your account settings
3. Set `FAL_AI_API_KEY` environment variable
4. Choose a model from the [Fal AI model gallery](https://fal.ai/models)
5. Start generating images with LiteLLM
## Additional Resources
- [Fal AI Documentation](https://fal.ai/docs)
- [Model Gallery](https://fal.ai/models)
- [API Reference](https://fal.ai/docs/api-reference)
- [Pricing](https://fal.ai/pricing)

2
docs/my-website/docs/providers/github.md
View file

@ -1,7 +1,7 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# 🆕 Github
# Github
https://github.com/marketplace/models
:::tip

2
docs/my-website/docs/providers/google_ai_studio/files.md
View file

@ -39,7 +39,7 @@ encoded_string = base64.b64encode(wav_data).decode('utf-8')
file = create_file(
file=wav_data,
purpose="user_data",
extra_body={"custom_llm_provider": "gemini"},
extra_headers={"custom-llm-provider": "gemini"},
api_key=os.getenv("GEMINI_API_KEY"),
)

199
docs/my-website/docs/providers/oci.md
View file

@ -29,17 +29,38 @@ Check the [OCI Models List](https://docs.oracle.com/en-us/iaas/Content/generativ
## Authentication
LiteLLM uses OCI signing key authentication. Follow the [official Oracle tutorial](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm) to create a signing key and obtain the following parameters:
LiteLLM supports two authentication methods for OCI:
### Method 1: Manual Credentials
Provide individual OCI credentials directly to LiteLLM. Follow the [official Oracle tutorial](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm) to create a signing key and obtain the following parameters:
- `user`
- `fingerprint`
- `tenancy`
- `region`
- `key_file`
- `key_file` or `key`
- `compartment_id`
This is the default method for LiteLLM AI Gateway (LLM Proxy) access to OCI GenAI models.
### Method 2: OCI SDK Signer
Use an OCI SDK `Signer` object for authentication. This method:
- Leverages the official [OCI SDK for signing](https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html)
- Supports additional authentication methods (instance principals, workload identity, etc.)
To use this method, install the OCI SDK:
```bash
pip install oci
```
This method is an alternative when using the LiteLLM SDK on Oracle Cloud Infrastructure (instances or Oracle Kubernetes Engine).
## Usage
Input the parameters obtained from the OCI signing key creation process into the `completion` function.
<Tabs>
<TabItem value="manual" label="Manual Credentials">
Input the parameters obtained from the OCI signing key creation process into the `completion` function:
```python
import os
@ -64,10 +85,119 @@ response = completion(
print(response)
```
</TabItem>
<TabItem value="oci-sdk" label="OCI SDK Signer" default>
Use the OCI SDK `Signer` for authentication:
```python
from litellm import completion
from oci.signer import Signer
# Create an OCI Signer
signer = Signer(
tenancy="ocid1.tenancy.oc1..",
user="ocid1.user.oc1..",
fingerprint="xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx",
private_key_file_location="~/.oci/key.pem",
# Or use private_key_content="<your_private_key_content>"
)
messages = [{"role": "user", "content": "Hey! how's it going?"}]
response = completion(
model="oci/xai.grok-4",
messages=messages,
oci_signer=signer,
oci_region="us-chicago-1", # Optional, defaults to us-ashburn-1
oci_serving_mode="ON_DEMAND", # Optional, default is "ON_DEMAND". Other option is "DEDICATED"
oci_compartment_id="<oci_compartment_id>",
)
print(response)
```
**Alternative: Use OCI Config File**
The OCI SDK can automatically load credentials from `~/.oci/config`:
```python
from litellm import completion
from oci.config import from_file
from oci.signer import Signer
# Load config from file
config = from_file("~/.oci/config", "DEFAULT") # "DEFAULT" is the profile name
signer = Signer(
tenancy=config["tenancy"],
user=config["user"],
fingerprint=config["fingerprint"],
private_key_file_location=config["key_file"],
pass_phrase=config.get("pass_phrase") # Optional if key is encrypted
)
messages = [{"role": "user", "content": "Hey! how's it going?"}]
response = completion(
model="oci/xai.grok-4",
messages=messages,
oci_signer=signer,
oci_region=config["region"],
oci_compartment_id="<oci_compartment_id>",
)
print(response)
```
**Instance Principal Authentication**
For applications running on OCI compute instances:
```python
from litellm import completion
from oci.auth.signers import InstancePrincipalsSecurityTokenSigner
oci.auth.signers.get_oke_workload_identity_resource_principal_signer()
# Use instance principal authentication
signer = InstancePrincipalsSecurityTokenSigner()
messages = [{"role": "user", "content": "Hey! how's it going?"}]
response = completion(
model="oci/xai.grok-4",
messages=messages,
oci_signer=signer,
oci_region="us-chicago-1",
oci_compartment_id="<oci_compartment_id>",
)
print(response)
```
**Use workload identity authentication**
For applications running in Oracle Kubernetes Engine (OKE):
```python
from litellm import completion
from oci.auth.signers import get_oke_workload_identity_resource_principal_signer
# Use instance principal authentication
signer = get_oke_workload_identity_resource_principal_signer()
messages = [{"role": "user", "content": "Hey! how's it going?"}]
response = completion(
model="oci/xai.grok-4",
messages=messages,
oci_signer=signer,
oci_region="us-chicago-1",
oci_compartment_id="<oci_compartment_id>",
)
print(response)
```
</TabItem>
</Tabs>
## Usage - Streaming
Just set `stream=True` when calling completion.
<Tabs>
<TabItem value="manual-stream" label="Manual Credentials">
```python
import os
from litellm import completion
@ -93,10 +223,68 @@ for chunk in response:
print(chunk["choices"][0]["delta"]["content"]) # same as openai format
```
</TabItem>
<TabItem value="oci-sdk-stream" label="OCI SDK Signer" default>
```python
from litellm import completion
from oci.signer import Signer
signer = Signer(
tenancy="ocid1.tenancy.oc1..",
user="ocid1.user.oc1..",
fingerprint="xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx",
private_key_file_location="~/.oci/key.pem",
)
messages = [{"role": "user", "content": "Hey! how's it going?"}]
response = completion(
model="oci/xai.grok-4",
messages=messages,
stream=True,
oci_signer=signer,
oci_region="us-chicago-1",
oci_compartment_id="<oci_compartment_id>",
)
for chunk in response:
print(chunk["choices"][0]["delta"]["content"]) # same as openai format
```
</TabItem>
</Tabs>
## Usage Examples by Model Type
### Using Cohere Models
<Tabs>
<TabItem value="cohere-sdk" label="OCI SDK Signer" default>
```python
from litellm import completion
from oci.signer import Signer
signer = Signer(
tenancy="ocid1.tenancy.oc1..",
user="ocid1.user.oc1..",
fingerprint="xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx",
private_key_file_location="~/.oci/key.pem",
)
messages = [{"role": "user", "content": "Explain quantum computing"}]
response = completion(
model="oci/cohere.command-latest",
messages=messages,
oci_signer=signer,
oci_region="us-chicago-1",
oci_compartment_id="<oci_compartment_id>",
)
print(response)
```
</TabItem>
<TabItem value="cohere-manual" label="Manual Credentials">
```python
from litellm import completion
@ -112,4 +300,7 @@ response = completion(
oci_compartment_id=<oci_compartment_id>,
)
print(response)
```
```
</TabItem>
</Tabs>

10
docs/my-website/docs/providers/openai/videos.md
View file

@ -17,7 +17,7 @@ os.environ["OPENAI_API_KEY"] = "your-api-key"
### Basic Usage
```python
from litellm import video_generation, video_retrieval
from litellm import video_generation, video_content
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"
@ -34,7 +34,7 @@ print(f"Video ID: {response.id}")
print(f"Status: {response.status}")
# Download video content when ready
video_bytes = video_retrieval(
video_bytes = video_content(
video_id=response.id,
model="sora-2"
)
@ -63,7 +63,7 @@ with open("generated_video.mp4", "wb") as f:
```python
# Download video content
video_bytes = video_retrieval(
video_bytes = video_content(
video_id="video_1234567890",
model="sora-2"
)
@ -95,7 +95,7 @@ def generate_and_download_video(prompt):
time.sleep(30)
# Step 3: Download video
video_bytes = litellm.video_retrieval(
video_bytes = litellm.video_content(
video_id=video_id,
model="sora-2"
)
@ -118,7 +118,7 @@ video_file = generate_and_download_video(
# Video editing with reference image
response = litellm.video_generation(
prompt="Make the cat jump higher",
input_reference="path/to/image.jpg", # Reference image
input_reference=open("path/to/image.jpg", "rb"), # Reference image
model="sora-2",
seconds="8"
)

33
docs/my-website/docs/providers/openrouter.md
View file

@ -9,10 +9,9 @@ LiteLLM supports all the text / chat / vision models from [OpenRouter](https://o
```python
import os
from litellm import completion
os.environ["OPENROUTER_API_KEY"] = ""
os.environ["OPENROUTER_API_BASE"] = "" # [OPTIONAL] defaults to https://openrouter.ai/api/v1
os.environ["OR_SITE_URL"] = "" # [OPTIONAL]
os.environ["OR_APP_NAME"] = "" # [OPTIONAL]
@ -22,8 +21,32 @@ response = completion(
)
```
## OpenRouter Completion Models
## Configuration with Environment Variables
For production environments, you can dynamically configure the base_url using environment variables:
```python
import os
from litellm import completion
# Configure with environment variables
OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY")
OPENROUTER_BASE_URL = os.getenv("OPENROUTER_API_BASE", "https://openrouter.ai/api/v1")
# Set environment for LiteLLM
os.environ["OPENROUTER_API_KEY"] = OPENROUTER_API_KEY
os.environ["OPENROUTER_API_BASE"] = OPENROUTER_BASE_URL
response = completion(
model="openrouter/google/palm-2-chat-bison",
messages=messages,
base_url=OPENROUTER_BASE_URL # Explicitly pass base_url for clarity
)
```
This approach provides better flexibility for managing configurations across different environments (dev, staging, production) and makes it easier to switch between self-hosted and cloud endpoints.
## OpenRouter Completion Models
🚨 LiteLLM supports ALL OpenRouter models, send `model=openrouter/<your-openrouter-model>` to send it to open router. See all openrouter models [here](https://openrouter.ai/models)
| Model Name | Function Call |
@ -40,12 +63,12 @@ response = completion(
| openrouter/meta-llama/llama-2-70b-chat | `completion('openrouter/meta-llama/llama-2-70b-chat', messages)` | `os.environ['OR_SITE_URL']`,`os.environ['OR_APP_NAME']`,`os.environ['OPENROUTER_API_KEY']` |
## Passing OpenRouter Params - transforms, models, route
Pass `transforms`, `models`, `route`as arguments to `litellm.completion()`
```python
import os
from litellm import completion
os.environ["OPENROUTER_API_KEY"] = ""
response = completion(
@ -54,4 +77,4 @@ response = completion(
transforms = [""],
route= ""
)
```
```

10
docs/my-website/docs/providers/vertex.md
View file

@ -2935,7 +2935,7 @@ finetune_settings:
ft_job = await client.fine_tuning.jobs.create(
model="gemini-1.0-pro-002", # Vertex model you want to fine-tune
training_file="gs://cloud-samples-data/ai-platform/generative_ai/sft_train_data.jsonl", # file_id from create file response
extra_body={"custom_llm_provider": "vertex_ai"}, # tell litellm proxy which provider to use
extra_headers={"custom-llm-provider": "vertex_ai"}, # tell litellm proxy which provider to use
)
```
</TabItem>
@ -2946,8 +2946,8 @@ ft_job = await client.fine_tuning.jobs.create(
curl http://localhost:4000/v1/fine_tuning/jobs \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-H "custom-llm-provider: vertex_ai" \
-d '{
"custom_llm_provider": "vertex_ai",
"model": "gemini-1.0-pro-002",
"training_file": "gs://cloud-samples-data/ai-platform/generative_ai/sft_train_data.jsonl"
}'
@ -2975,9 +2975,7 @@ ft_job = client.fine_tuning.jobs.create(
"learning_rate_multiplier": 0.1, # learning_rate_multiplier on Vertex
"adapter_size": "ADAPTER_SIZE_ONE" # type: ignore, vertex specific hyperparameter
},
extra_body={
"custom_llm_provider": "vertex_ai",
},
extra_headers={"custom-llm-provider": "vertex_ai"},
)
```
</TabItem>
@ -2988,8 +2986,8 @@ ft_job = client.fine_tuning.jobs.create(
curl http://localhost:4000/v1/fine_tuning/jobs \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-H "custom-llm-provider: vertex_ai" \
-d '{
"custom_llm_provider": "vertex_ai",
"model": "gemini-1.0-pro-002",
"training_file": "gs://cloud-samples-data/ai-platform/generative_ai/sft_train_data.jsonl",
"hyperparameters": {

16
docs/my-website/docs/providers/vertex_batch.md
View file

@ -50,7 +50,7 @@ oai_client = OpenAI(
file_obj = oai_client.files.create(
file=open("batch_requests.jsonl", "rb"),
purpose="batch",
extra_body={"custom_llm_provider": "vertex_ai"}
extra_headers={"custom-llm-provider": "vertex_ai"}
)
print(f"File uploaded with ID: {file_obj.id}")
@ -63,9 +63,9 @@ print(f"File uploaded with ID: {file_obj.id}")
curl --request POST \
--url http://localhost:4000/v1/files \
--header 'Content-Type: multipart/form-data' \
--header 'custom-llm-provider: vertex_ai' \
--form purpose=batch \
--form file=@batch_requests.jsonl \
--form custom_llm_provider=vertex_ai
--form file=@batch_requests.jsonl
```
</TabItem>
@ -100,7 +100,7 @@ create_batch_response = oai_client.batches.create(
completion_window="24h",
endpoint="/v1/chat/completions",
input_file_id=batch_input_file_id, # e.g. "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-2.5-flash-lite/abc123-def4-5678-9012-34567890abcd"
extra_body={"custom_llm_provider": "vertex_ai"}
extra_headers={"custom-llm-provider": "vertex_ai"}
)
print(f"Batch created with ID: {create_batch_response.id}")
@ -113,11 +113,11 @@ print(f"Batch created with ID: {create_batch_response.id}")
curl --request POST \
--url http://localhost:4000/v1/batches \
--header 'Content-Type: application/json' \
--header 'custom-llm-provider: vertex_ai' \
--data '{
"input_file_id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-2.5-flash-lite/abc123-def4-5678-9012-34567890abcd",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"custom_llm_provider": "vertex_ai"
"completion_window": "24h"
}'
```
@ -162,7 +162,7 @@ Check the status of your batch job. The batch will progress through states: `val
```python showLineNumbers title="retrieve_batch.py"
retrieved_batch = oai_client.batches.retrieve(
batch_id=create_batch_response.id, # Created batch id, e.g. 7814463557919047680
extra_query={"custom_llm_provider": "vertex_ai"}
extra_headers={"custom-llm-provider": "vertex_ai"}
)
print(f"Batch status: {retrieved_batch.status}")
@ -230,7 +230,7 @@ encoded_file_id = urllib.parse.quote_plus(output_file_id)
# Get file content
file_content = oai_client.files.content(
file_id=encoded_file_id,
extra_body={"custom_llm_provider": "vertex_ai"}
extra_headers={"custom-llm-provider": "vertex_ai"}
)
# Process the results

19
docs/my-website/docs/proxy/access_control.md
View file

@ -10,8 +10,25 @@ Role-based access control (RBAC) is based on Organizations, Teams and Internal U
- `Organizations` are the top-level entities that contain Teams.
- `Team` - A Team is a collection of multiple `Internal Users`
- `Internal Users` - users that can create keys, make LLM API calls, view usage on LiteLLM. Users can be on multiple teams.
- `Virtual Keys` - Keys are used for authentication to the LiteLLM API. Keys are tied to a `Internal User` and `Team`
- `Virtual Keys` - Keys are used for authentication to the LiteLLM API. Each key can optionally be associated with a `user_id`, a `team_id`, or both:
- **User-only key**: Has a `user_id` but no `team_id`. Tracked individually, deleted when the user is deleted.
- **Team key (Service Account)**: Has a `team_id` but no `user_id`. Shared by the team, not deleted when users are removed. [Learn more about service account keys](https://docs.litellm.ai/docs/proxy/virtual_keys#service-account-keys).
- **User + Team key**: Has both `user_id` and `team_id`. Belongs to a specific user within a team context.
### When to Use Each Key Type
| Key Type | Use Case | Spend Tracking | Lifecycle |
|----------|----------|----------------|-----------|
| **User-only** | Personal API keys for individual developers | Tracked to the user | Deleted when user is deleted |
| **Team (Service Account)** | Production apps, CI/CD pipelines, shared services | Tracked to the team only | Persists even when team members leave |
| **User + Team** | User working within a team context | Tracked to both user and team | Deleted when user is deleted |
**Example scenarios:**
- Use **user-only keys** for developers testing locally
- Use **team service account keys** for your production application that shouldn't break when employees leave
- Use **user + team keys** when you want individual accountability within a team budget
---
## User Roles

8
docs/my-website/docs/proxy/config_settings.md
View file

@ -232,7 +232,7 @@ router_settings:
| max_response_size_mb | int | The maximum size for responses in MB. LLM Responses above this size will not be sent. |
| proxy_budget_rescheduler_min_time | int | The minimum time (in seconds) to wait before checking db for budget resets. **Default is 597 seconds** |
| proxy_budget_rescheduler_max_time | int | The maximum time (in seconds) to wait before checking db for budget resets. **Default is 605 seconds** |
| proxy_batch_write_at | int | Time (in seconds) to wait before batch writing spend logs to the db. **Default is 10 seconds** |
| proxy_batch_write_at | int | Time (in seconds) to wait before batch writing spend logs to the db. **Default is 30 seconds** |
| proxy_batch_polling_interval | int | Time (in seconds) to wait before polling a batch, to check if it's completed. **Default is 6000 seconds (1 hour)** |
| alerting_args | dict | Args for Slack Alerting [Doc on Slack Alerting](./alerting.md) |
| custom_key_generate | str | Custom function for key generation [Doc on custom key generation](./virtual_keys.md#custom--key-generate) |
@ -361,6 +361,10 @@ router_settings:
| AIOHTTP_TRUST_ENV | Flag to enable aiohttp trust environment. When this is set to True, aiohttp will respect HTTP(S)_PROXY env vars. **Default is False**
| AIOHTTP_TTL_DNS_CACHE | DNS cache time-to-live for aiohttp in seconds. **Default is 300**
| ALLOWED_EMAIL_DOMAINS | List of email domains allowed for access
| APSCHEDULER_COALESCE | Whether to combine multiple pending executions of a job into one. **Default is False**
| APSCHEDULER_MAX_INSTANCES | Maximum number of concurrent instances of each job. **Default is 1**
| APSCHEDULER_MISFIRE_GRACE_TIME | Grace time in seconds for misfired jobs. **Default is 1**
| APSCHEDULER_REPLACE_EXISTING | Whether to replace existing jobs with the same ID. **Default is False**
| ARIZE_API_KEY | API key for Arize platform integration
| ARIZE_SPACE_KEY | Space key for Arize platform
| ARGILLA_BATCH_SIZE | Batch size for Argilla logging
@ -726,7 +730,7 @@ router_settings:
| PROMPTLAYER_API_KEY | API key for PromptLayer integration
| PROXY_ADMIN_ID | Admin identifier for proxy server
| PROXY_BASE_URL | Base URL for proxy service
| PROXY_BATCH_WRITE_AT | Time in seconds to wait before batch writing spend logs to the database. Default is 10
| PROXY_BATCH_WRITE_AT | Time in seconds to wait before batch writing spend logs to the database. Default is 30
| PROXY_BATCH_POLLING_INTERVAL | Time in seconds to wait before polling a batch, to check if it's completed. Default is 6000s (1 hour)
| PROXY_BUDGET_RESCHEDULER_MAX_TIME | Maximum time in seconds to wait before checking database for budget resets. Default is 605
| PROXY_BUDGET_RESCHEDULER_MIN_TIME | Minimum time in seconds to wait before checking database for budget resets. Default is 597

4
docs/my-website/docs/proxy/deploy.md
View file

@ -2,7 +2,7 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# Docker, Deployment
# Docker, Helm, Terraform
You can find the Dockerfile to build litellm proxy [here](https://github.com/BerriAI/litellm/blob/main/Dockerfile)
@ -195,7 +195,7 @@ docker run \
s/o [Nicholas Cecere](https://www.linkedin.com/in/nicholas-cecere-24243549/) for his LiteLLM User Management Terraform
👉 [Go here for Terraform](https://github.com/ncecere/terraform-litellm-user-mgmt)
👉 [Go here for Terraform](https://github.com/BerriAI/terraform-provider-litellm)
### Kubernetes

2
docs/my-website/docs/proxy/guardrails/grayswan.md
View file

@ -18,6 +18,7 @@ Cygnal returns a `violation` score between `0` and `1` (higher means more likely
```bash
export GRAYSWAN_API_KEY="your-grayswan-key"
export GRAYSWAN_API_BASE="https://api.grayswan.ai"
```
### 2. Configure `config.yaml`
@ -37,6 +38,7 @@ guardrails:
guardrail: grayswan
mode: [pre_call, post_call] # monitor both input and output
api_key: os.environ/GRAYSWAN_API_KEY
api_base: os.environ/GRAYSWAN_API_BASE # optional
optional_params:
on_flagged_action: monitor # or "block"
violation_threshold: 0.5 # score >= threshold is flagged

17
docs/my-website/docs/proxy/guardrails/ibm_guardrails.md
View file

@ -3,11 +3,18 @@ import TabItem from '@theme/TabItem';
# IBM Guardrails
LiteLLM works with IBM's FMS Guardrails for content safety. You can use it to detect jailbreaks, PII, hate speech, and more.
LiteLLM works with [IBM's FMS Guardrails](https://github.com/foundation-model-stack/fms-guardrails-orchestrator) for content safety. You can use it to detect jailbreaks, PII, hate speech, and more.
## What it does
IBM Guardrails analyzes text and tells you if it contains things you want to avoid. It gives each detection a score. Higher scores mean it's more confident.
IBM's FMS Guardrails is a framework for invoking detectors on LLM inputs and outputs. To configure these detectors, you can use e.g. [TrustyAI detectors](https://github.com/trustyai-explainability/guardrails-detectors), an open-source project maintained by the Red Hat's [TrustyAI team](https://github.com/trustyai-explainability) that allows the user to configure detectors that are:
- regex patterns
- file type validators
- custom Python functions
- Hugging Face [AutoModelForSequenceClassification](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForSequenceClassification), i.e. sequence classification models
Each detector outputs an API response based on the following [openapi schema](https://foundation-model-stack.github.io/fms-guardrails-orchestrator/docs/api/openapi_detector_api.yaml).
You can run these checks:
- Before sending to the LLM (on user input)
@ -73,7 +80,7 @@ curl -i http://localhost:4000/v1/chat/completions \
- `guardrail` - str - Set to `ibm_guardrails`
- `auth_token` - str - Your IBM Guardrails auth token. Can use `os.environ/IBM_GUARDRAILS_AUTH_TOKEN`
- `base_url` - str - URL of your IBM Guardrails server
- `base_url` - str - URL of your IBM Detector or Guardrails server
- `detector_id` - str - Which detector to use (e.g., "jailbreak-detector", "pii-detector")
### Optional params
@ -97,7 +104,7 @@ IBM Guardrails has two APIs you can use:
### Detector Server (recommended)
The simpler one. Sends all messages at once.
[This Detectors API](https://foundation-model-stack.github.io/fms-guardrails-orchestrator/?urls.primaryName=Detector+API#/Text) uses `api/v1/text/contents` endpoint to run a single detector; it can accept multiple text inputs within a request.
```yaml
guardrails:
@ -113,7 +120,7 @@ guardrails:
### Orchestrator
If you're using the IBM FMS Guardrails Orchestrator, you can use this.
If you're using the IBM FMS Guardrails Orchestrator, you can use [FMS Orchestrator API](https://foundation-model-stack.github.io/fms-guardrails-orchestrator/?urls.primaryName=Orchestrator+API), specifically by leveraging the `api/v2/text/detection/content` to potentially run multiple detectors in a single request; however, this endpoint can only accept one text input per request.
```yaml
guardrails:

73
docs/my-website/docs/proxy/guardrails/panw_prisma_airs.md
View file

@ -199,10 +199,44 @@ Expected successful response:
| Parameter | Required | Description | Default |
|-----------|----------|-------------|---------|
| `api_key` | Yes | Your PANW Prisma AIRS API key from Strata Cloud Manager | - |
| `profile_name` | Yes | Security profile name configured in Strata Cloud Manager | - |
| `profile_name` | No | Security profile name configured in Strata Cloud Manager. Optional if API key has linked profile | - |
| `app_name` | No | Application identifier for tracking in Prisma AIRS analytics (will be prefixed with "LiteLLM-") | `LiteLLM` |
| `api_base` | No | Custom API base URL (without /v1/scan/sync/request path) | `https://service.api.aisecurity.paloaltonetworks.com` |
| `mode` | No | When to run the guardrail | `pre_call` |
## Per-Request Metadata Overrides
You can override guardrail settings on a per-request basis using the `metadata` field:
```json
{
"model": "gpt-4",
"messages": [...],
"metadata": {
"profile_name": "dev-allow-all", // Override profile name
"profile_id": "uuid-here", // Override profile ID (takes precedence)
"user_ip": "192.168.1.100", // Track user IP
"app_name": "MyApp" // Custom app name (becomes "LiteLLM-MyApp")
}
}
```
**Supported Metadata Fields:**
| Field | Description | Priority |
|-------|-------------|----------|
| `profile_name` | PANW AI security profile name | Per-request > config |
| `profile_id` | PANW AI security profile ID (takes precedence over profile_name) | Per-request only |
| `user_ip` | User IP address for tracking in Prisma AIRS | Per-request only |
| `app_name` | Application identifier (prefixed with "LiteLLM-") | Per-request > config > "LiteLLM" |
:::info Profile Resolution
- If both `profile_id` and `profile_name` are provided, PANW API uses `profile_id` (it takes precedence)
- If no profile is specified in metadata, uses the config `profile_name`
- If no profile is specified at all, PANW API will use the profile linked to your API key in Strata Cloud Manager
- **Note:** If your API key is not linked to a profile, you must provide `profile_name` or `profile_id`
:::
## Environment Variables
```bash
@ -235,6 +269,43 @@ guardrails:
profile_name: "permissive-policy" # Lower security profile
```
### Multiple API Keys (Multi-Tenant)
For multi-tenant deployments where different customers need different PANW API keys, create separate guardrail instances:
```yaml
guardrails:
- guardrail_name: "panw-customer-a"
litellm_params:
guardrail: panw_prisma_airs
mode: "pre_call"
api_key: os.environ/PANW_CUSTOMER_A_KEY # Linked to Customer A profile in SCM
- guardrail_name: "panw-customer-b"
litellm_params:
guardrail: panw_prisma_airs
mode: "pre_call"
api_key: os.environ/PANW_CUSTOMER_B_KEY # Linked to Customer B profile in SCM
```
Then route requests to the appropriate guardrail:
```bash
curl -X POST http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "Hello"}],
"guardrails": ["panw-customer-a"]
}'
```
**Use Cases:**
- **Multi-tenant deployments**: Different customers with different security policies
- **Environment-specific policies**: Dev/staging/prod with different API keys and profiles
- **A/B testing**: Compare different security profiles side-by-side
### Content Masking
PANW Prisma AIRS can automatically mask sensitive content (PII, credit cards, SSNs, etc.) instead of blocking requests. This allows your application to continue functioning while protecting sensitive data.

66
docs/my-website/docs/proxy/guardrails/pillar_security.md
View file

@ -29,7 +29,7 @@ Use Pillar Security for comprehensive LLM security including:
Add Pillar Security to your `config.yaml`:
**🌟 Recommended Configuration (Dual Mode):**
**🌟 Recommended Configuration:**
```yaml
model_list:
- model_name: gpt-4.1-mini
@ -45,6 +45,8 @@ guardrails:
api_key: os.environ/PILLAR_API_KEY # Your Pillar API key
api_base: os.environ/PILLAR_API_BASE # Pillar API endpoint
on_flagged_action: "monitor" # Log threats but allow requests
fallback_on_error: "allow" # Gracefully degrade if Pillar is down (default)
timeout: 5.0 # Timeout for Pillar API calls in seconds (default)
persist_session: true # Keep conversations visible in Pillar dashboard
async_mode: false # Request synchronous verdicts
include_scanners: true # Return scanner category breakdown
@ -207,6 +209,8 @@ You can configure Pillar Security using environment variables:
export PILLAR_API_KEY="your_api_key_here"
export PILLAR_API_BASE="https://api.pillar.security"
export PILLAR_ON_FLAGGED_ACTION="monitor"
export PILLAR_FALLBACK_ON_ERROR="allow"
export PILLAR_TIMEOUT="30.0"
```
### Session Tracking
@ -245,6 +249,66 @@ Logs the violation but allows the request to proceed:
on_flagged_action: "monitor"
```
### Resilience and Error Handling
#### Graceful Degradation (`fallback_on_error`)
Control what happens when the Pillar API is unavailable (network errors, timeouts, service outages):
```yaml
fallback_on_error: "allow" # Default - recommended for production resilience
```
**Available Options:**
- **`allow` (Default - Recommended)**: Proceed without scanning when Pillar is unavailable
- **No service interruption** if Pillar is down
- **Best for production** where availability is critical
- Security scans are skipped during outages (logged as warnings)
```yaml
guardrails:
- guardrail_name: "pillar-resilient"
litellm_params:
guardrail: pillar
fallback_on_error: "allow" # Graceful degradation
```
- **`block`**: Reject all requests when Pillar is unavailable
- **Fail-secure approach** - no request proceeds without scanning
- **Service interruption** during Pillar outages
- Returns 503 Service Unavailable error
```yaml
guardrails:
- guardrail_name: "pillar-fail-secure"
litellm_params:
guardrail: pillar
fallback_on_error: "block" # Fail secure
```
#### Timeout Configuration
Configure how long to wait for Pillar API responses:
**Example Configurations:**
```yaml
# Production: Default - Fast with graceful degradation
guardrails:
- guardrail_name: "pillar-production"
litellm_params:
guardrail: pillar
timeout: 5.0 # Default - fast failure detection
fallback_on_error: "allow" # Graceful degradation (required)
```
**Environment Variables:**
```bash
export PILLAR_FALLBACK_ON_ERROR="allow"
export PILLAR_TIMEOUT="5.0"
```
## Advanced Configuration
**Quick takeaways**

22
docs/my-website/docs/proxy/logging.md
View file

@ -1415,18 +1415,26 @@ AWS_REGION_NAME = ""
```yaml
model_list:
- model_name: gpt-4o
- model_name: gpt-4o
litellm_params:
model: gpt-4o
litellm_settings:
callbacks: ["aws_sqs"]
aws_sqs_callback_params:
sqs_queue_url: https://sqs.us-west-2.amazonaws.com/123456789012/my-queue # AWS SQS Queue URL
sqs_region_name: us-west-2 # AWS Region Name for SQS
sqs_aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID # use os.environ/<variable name> to pass environment variables. This is AWS Access Key ID for SQS
sqs_aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY # AWS Secret Access Key for SQS
sqs_batch_size: 10 # [OPTIONAL] Number of messages to batch before sending (default: 10)
sqs_flush_interval: 30 # [OPTIONAL] Time in seconds to wait before flushing batch (default: 30)
# --- 🧱 Required Parameters ---
sqs_queue_url: https://sqs.us-west-2.amazonaws.com/123456789012/my-queue
# The AWS SQS Queue URL to which LiteLLM will send log events.
sqs_region_name: us-west-2
# AWS Region for your SQS queue (e.g., us-east-1, eu-central-1, etc.)
# --- Logging Controls ---
sqs_strip_base64_files: true
# If true, LiteLLM will remove or redact base64-encoded binary data (e.g., PDFs, images, audio)
# from logged messages to avoid large payloads. SQS has a 1 MB payload size limit.
```
**Step 3**: Start the proxy, make a test request

266
docs/my-website/docs/videos.md
View file

@ -21,7 +21,7 @@ LiteLLM follows the [OpenAI Video Generation API specification](https://platform
### Quick Start
```python
from litellm import video_generation, video_status, video_retrieval
from litellm import video_generation, video_status, video_content
import os
import time
@ -56,7 +56,7 @@ while True:
time.sleep(10) # Wait 10 seconds before checking again
# Download video content when ready
video_bytes = video_retrieval(
video_bytes = video_content(
video_id=response.id,
model="openai/sora-2"
)
@ -69,7 +69,7 @@ with open("generated_video.mp4", "wb") as f:
### Async Usage
```python
from litellm import avideo_generation, avideo_status, avideo_retrieval
from litellm import avideo_generation, avideo_status, avideo_content
import os, asyncio
os.environ["OPENAI_API_KEY"] = "sk-.."
@ -103,7 +103,7 @@ async def test_async_video():
await asyncio.sleep(10) # Wait 10 seconds before checking again
# Download video content when ready
video_bytes = await avideo_retrieval(
video_bytes = await avideo_content(
video_id=response.id,
model="openai/sora-2"
)
@ -241,57 +241,45 @@ litellm --config /path/to/config.yaml
Test video generation request
```bash
curl http://0.0.0.0:4000/videos/generations \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
curl --location 'http://localhost:4000/v1/videos' \
--header 'Content-Type: application/json' \
--header 'x-litellm-api-key: sk-1234' \
--data '{
"model": "sora-2",
"prompt": "A cat playing with a ball of yarn in a sunny garden",
"seconds": "8",
"size": "720x1280"
}'
"prompt": "A beautiful sunset over the ocean"
}'
```
Test video status request
```bash
curl http://0.0.0.0:4000/videos/status \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"video_id": "video_1234567890",
"model": "sora-2"
}'
curl --location 'http://localhost:4000/v1/videos/video_id' \
--header 'Accept: application/json' \
--header 'x-litellm-api-key: sk-1234'
```
Test video retrieval request
```bash
curl http://0.0.0.0:4000/videos/retrieval \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"video_id": "video_1234567890",
"model": "sora-2"
}'
curl --location 'http://localhost:4000/v1/videos/video_id/content' \
--header 'Accept: application/json' \
--header 'x-litellm-api-key: sk-1234'
```
Test video remix request
```bash
curl http://0.0.0.0:4000/videos/remix \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: multipart/form-data" \
-F 'model=sora-2' \
-F 'prompt=Make the cat jump higher' \
-F 'input_reference=@path/to/image.jpg' \
-F 'seconds=8'
curl --location --request POST 'http://localhost:4000/v1/videos/string/remix' \
--header 'Accept: application/json' \
--header 'x-litellm-api-key: sk-1234'
```
Test Azure video generation request
```bash
curl http://0.0.0.0:4000/videos/generations \
curl http://localhost:4000/v1/videos \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
@ -302,6 +290,188 @@ curl http://0.0.0.0:4000/videos/generations \
}'
```
## **Using OpenAI Client with LiteLLM Proxy**
You can use the standard OpenAI Python client to interact with LiteLLM's video endpoints. This provides a familiar interface while leveraging LiteLLM's provider abstraction and proxy features.
### Setup
First, configure your OpenAI client to point to your LiteLLM proxy:
```python
from openai import OpenAI
# Point the OpenAI client to your LiteLLM proxy
client = OpenAI(
api_key="sk-1234", # Your LiteLLM proxy API key
base_url="http://localhost:4000/v1" # Your LiteLLM proxy URL
)
```
### Video Generation
Generate a new video using the OpenAI client interface:
```python
# Basic video generation
response = client.videos.create(
model="sora-2",
prompt="A cat playing with a ball of yarn in a sunny garden",
seconds=8,
size="720x1280"
)
print(f"Video ID: {response.id}")
print(f"Status: {response.status}")
```
### Video Generation with Reference Image
Create a video using a reference image:
```python
# Video generation with reference image
response = client.videos.create(
model="sora-2",
prompt="Add clouds to the video",
seconds=4,
input_reference=open("/path/to/your/image.jpg", "rb")
)
print(f"Video ID: {response.id}")
print(f"Status: {response.status}")
```
### Video Status Checking
Check the status of a video generation:
```python
# Check video status
status_response = client.videos.retrieve(
video_id="video_6900378779308191a7359266e59b53fc01cd6bbd27a70763"
)
print(f"Status: {status_response.status}")
print(f"Progress: {status_response.progress}%")
# Poll until completion
import time
while status_response.status not in ["completed", "failed"]:
time.sleep(10) # Wait 10 seconds
status_response = client.videos.retrieve(
video_id="video_6900378779308191a7359266e59b53fc01cd6bbd27a70763"
)
print(f"Current status: {status_response.status}")
```
### List Videos
Get a list of your videos:
```python
# List all videos
videos = client.videos.list()
for video in videos.data:
print(f"Video ID: {video.id}, Status: {video.status}")
```
### Download Video Content
Download the completed video:
```python
# Download video content
response = client.videos.download_content(
video_id="video_68fa2938848c8190bb718f977503aba6092ab18d68938fed"
)
# Save the video to file
with open("generated_video.mp4", "wb") as f:
f.write(response.content)
print("Video downloaded successfully!")
```
### Video Remix (Editing)
Edit an existing video with new instructions:
```python
# Remix/edit an existing video
response = client.videos.remix(
video_id="video_68fa2574bdd88190873a8af06a370ff407094ddbc4bbb91b",
prompt="Slow the cloud movement",
seconds=8
)
print(f"Remix Video ID: {response.id}")
print(f"Status: {response.status}")
```
### Complete Workflow Example
Here's a complete example showing the full video generation workflow:
```python
from openai import OpenAI
import time
# Initialize client
client = OpenAI(
api_key="sk-1234",
base_url="http://localhost:4000/v1"
)
# 1. Generate video
print("Generating video...")
response = client.videos.create(
model="sora-2",
prompt="A serene lake with mountains in the background",
seconds=8,
size="1280x720"
)
video_id = response.id
print(f"Video generation started. ID: {video_id}")
# 2. Poll for completion
print("Waiting for video to complete...")
while True:
status = client.videos.retrieve(video_id=video_id)
print(f"Status: {status.status}")
if status.status == "completed":
print("Video generation completed!")
break
elif status.status == "failed":
print("Video generation failed!")
break
time.sleep(10)
# 3. Download video
if status.status == "completed":
print("Downloading video...")
video_content = client.videos.download_content(video_id=video_id)
with open(f"video_{video_id}.mp4", "wb") as f:
f.write(video_content.content)
print("Video saved successfully!")
# 4. Optional: Remix the video
print("Creating a remix...")
remix_response = client.videos.remix(
video_id=video_id,
prompt="Add gentle ripples to the lake surface"
)
print(f"Remix started. ID: {remix_response.id}")
```
## **Request/Response Format**
:::info
@ -381,19 +551,21 @@ The response follows OpenAI's video generation format with the following structu
```json
{
"id": "video_1234567890",
"object": "video",
"status": "queued",
"created_at": 1712697600,
"model": "sora-2",
"size": "720x1280",
"seconds": "8",
"usage": {
"prompt_tokens": 0,
"completion_tokens": 0,
"total_tokens": 0,
"duration_seconds": 8.0
}
"id": "video_6900378779308191a7359266e59b53fc01cd6bbd27a70763",
"object": "video",
"status": "queued",
"created_at": 1761621895,
"completed_at": null,
"expires_at": null,
"error": null,
"progress": 0,
"remixed_from_video_id": null,
"seconds": "4",
"size": "720x1280",
"model": "sora-2",
"usage": {
"duration_seconds": 4.0
}
}
```

BIN
docs/my-website/img/opik_key_metadata.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 58 KiB

6
docs/my-website/sidebars.js
View file

@ -32,6 +32,7 @@ const sidebars = {
items: [
"proxy/guardrails/quick_start",
...[
"adding_provider/adding_guardrail_support",
"proxy/guardrails/aim_security",
"proxy/guardrails/aporia_api",
"proxy/guardrails/azure_content_guardrail",
@ -431,6 +432,7 @@ const sidebars = {
"providers/openai",
"providers/openai/responses_api",
"providers/openai/text_to_speech",
"providers/openai/videos",
]
},
"providers/text_completion_openai",
@ -443,6 +445,7 @@ const sidebars = {
"providers/azure/azure_responses",
"providers/azure/azure_embedding",
"providers/azure/azure_speech",
"providers/azure/videos",
]
},
{
@ -534,6 +537,7 @@ const sidebars = {
"providers/groq",
"providers/deepseek",
"providers/elevenlabs",
"providers/fal_ai",
"providers/fireworks_ai",
"providers/clarifai",
"providers/compactifai",
@ -713,7 +717,7 @@ const sidebars = {
items: [
"adding_provider/directory_structure",
"adding_provider/new_rerank_provider",
"adding_provider/adding_guardrail_support"],
]
},
"extras/contributing",
"contributing",

3
enterprise/litellm_enterprise/integrations/prometheus.py
View file

@ -2189,6 +2189,9 @@ class PrometheusLogger(CustomLogger):
prometheus_logger.initialize_remaining_budget_metrics,
"interval",
minutes=PROMETHEUS_BUDGET_METRICS_REFRESH_INTERVAL_MINUTES,
# REMOVED jitter parameter - major cause of memory leak
id="prometheus_budget_metrics_job",
replace_existing=True,
)
@staticmethod

43
enterprise/litellm_enterprise/proxy/guardrails/endpoints.py
View file

@ -1,43 +0,0 @@
"""
Enterprise Guardrail Routes on LiteLLM Proxy
To see all free guardrails see litellm/proxy/guardrails/*
Exposed Routes:
- /mask_pii
"""
from typing import Optional
from fastapi import APIRouter, Depends
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.guardrails.guardrail_endpoints import GUARDRAIL_REGISTRY
from litellm.types.guardrails import ApplyGuardrailRequest, ApplyGuardrailResponse
router = APIRouter(tags=["guardrails"], prefix="/guardrails")
@router.post("/apply_guardrail", response_model=ApplyGuardrailResponse)
async def apply_guardrail(
request: ApplyGuardrailRequest,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
Mask PII from a given text, requires a guardrail to be added to litellm.
"""
active_guardrail: Optional[
CustomGuardrail
] = GUARDRAIL_REGISTRY.get_initialized_guardrail_callback(
guardrail_name=request.guardrail_name
)
if active_guardrail is None:
raise Exception(f"Guardrail {request.guardrail_name} not found")
response_text = await active_guardrail.apply_guardrail(
text=request.text, language=request.language, entities=request.entities
)
return ApplyGuardrailResponse(response_text=response_text)

8
litellm-js/spend-logs/package-lock.json generated
View file

@ -6,7 +6,7 @@
"": {
"dependencies": {
"@hono/node-server": "^1.10.1",
"hono": "^4.9.7"
"hono": "^4.10.3"
},
"devDependencies": {
"@types/node": "^20.11.17",
@ -463,9 +463,9 @@
}
},
"node_modules/hono": {
"version": "4.9.7",
"resolved": "https://registry.npmjs.org/hono/-/hono-4.9.7.tgz",
"integrity": "sha512-t4Te6ERzIaC48W3x4hJmBwgNlLhmiEdEE5ViYb02ffw4ignHNHa5IBtPjmbKstmtKa8X6C35iWwK4HaqvrzG9w==",
"version": "4.10.3",
"resolved": "https://registry.npmjs.org/hono/-/hono-4.10.3.tgz",
"integrity": "sha512-2LOYWUbnhdxdL8MNbNg9XZig6k+cZXm5IjHn2Aviv7honhBMOHb+jxrKIeJRZJRmn+htUCKhaicxwXuUDlchRA==",
"license": "MIT",
"engines": {
"node": ">=16.9.0"

2
litellm-js/spend-logs/package.json
View file

@ -4,7 +4,7 @@
},
"dependencies": {
"@hono/node-server": "^1.10.1",
"hono": "^4.9.7"
"hono": "^4.10.3"
},
"devDependencies": {
"@types/node": "^20.11.17",

8
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -580,4 +580,12 @@ model LiteLLM_SearchToolsTable {
search_tool_info Json?
created_at DateTime @default(now())
updated_at DateTime @updatedAt
}
// SSO configuration table
model LiteLLM_SSOConfig {
id String @id @default("sso_config")
sso_settings Json
created_at DateTime @default(now())
updated_at DateTime @updatedAt
}

5
litellm/__init__.py
View file

@ -475,6 +475,7 @@ nlp_cloud_models: Set = set()
aleph_alpha_models: Set = set()
bedrock_models: Set = set()
bedrock_converse_models: Set = set(BEDROCK_CONVERSE_MODELS)
fal_ai_models: Set = set()
fireworks_ai_models: Set = set()
fireworks_ai_embedding_models: Set = set()
deepinfra_models: Set = set()
@ -663,6 +664,8 @@ def add_known_models():
text_completion_codestral_models.add(key)
elif value.get("litellm_provider") == "xai":
xai_models.add(key)
elif value.get("litellm_provider") == "fal_ai":
fal_ai_models.add(key)
elif value.get("litellm_provider") == "deepseek":
deepseek_models.add(key)
elif value.get("litellm_provider") == "meta_llama":
@ -813,6 +816,7 @@ model_list = list(
| gemini_models
| text_completion_codestral_models
| xai_models
| fal_ai_models
| deepseek_models
| azure_ai_models
| voyage_models
@ -895,6 +899,7 @@ models_by_provider: dict = {
"aleph_alpha": aleph_alpha_models,
"text-completion-codestral": text_completion_codestral_models,
"xai": xai_models,
"fal_ai": fal_ai_models,
"deepseek": deepseek_models,
"mistral": mistral_chat_models,
"azure_ai": azure_ai_models,

54
litellm/batches/batch_utils.py
View file

@ -1,10 +1,15 @@
import json
from typing import Any, List, Literal, Tuple, Optional
import time
from typing import Any, List, Literal, Optional, Tuple
import httpx
import litellm
from litellm._logging import verbose_logger
from litellm._uuid import uuid
from litellm.types.llms.openai import Batch
from litellm.types.utils import CallTypes, Usage
from litellm.types.utils import CallTypes, ModelResponse, Usage
from litellm.utils import token_counter
async def calculate_batch_cost_and_usage(
@ -107,6 +112,10 @@ def calculate_vertex_ai_batch_cost_and_usage(
"""
Calculate both cost and usage from Vertex AI batch responses
"""
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
VertexGeminiConfig,
)
total_cost = 0.0
total_tokens = 0
prompt_tokens = 0
@ -115,14 +124,7 @@ def calculate_vertex_ai_batch_cost_and_usage(
for response in vertex_ai_batch_responses:
if response.get("status") == "JOB_STATE_SUCCEEDED": # Check if response was successful
# Transform Vertex AI response to OpenAI format if needed
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexGeminiConfig
from litellm import ModelResponse
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.types.utils import CallTypes
from litellm._uuid import uuid
import httpx
import time
# Create required arguments for the transformation method
model_response = ModelResponse()
@ -163,8 +165,9 @@ def calculate_vertex_ai_batch_cost_and_usage(
total_cost += cost
# Extract usage from the transformed response
if hasattr(openai_format_response, 'usage') and openai_format_response.usage:
usage = openai_format_response.usage
usage_obj = getattr(openai_format_response, 'usage', None)
if usage_obj:
usage = usage_obj
else:
# Fallback: create usage from response dict
response_dict = openai_format_response.dict() if hasattr(openai_format_response, 'dict') else {}
@ -278,6 +281,33 @@ def _get_batch_job_total_usage_from_file_content(
completion_tokens=completion_tokens,
)
def _get_batch_job_input_file_usage(
file_content_dictionary: List[dict],
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
model_name: Optional[str] = None,
) -> Usage:
"""
Count the number of tokens in the input file
Used for batch rate limiting to count the number of tokens in the input file
"""
prompt_tokens: int = 0
completion_tokens: int = 0
for _item in file_content_dictionary:
body = _item.get("body", {})
model = body.get("model", model_name or "")
messages = body.get("messages", [])
if messages:
item_tokens = token_counter(model=model, messages=messages)
prompt_tokens += item_tokens
return Usage(
total_tokens=prompt_tokens + completion_tokens,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
)
def _get_batch_job_usage_from_response_body(response_body: dict) -> Usage:
"""

12
litellm/constants.py
View file

@ -1050,7 +1050,17 @@ PROXY_BATCH_POLLING_INTERVAL = int(os.getenv("PROXY_BATCH_POLLING_INTERVAL", 360
PROXY_BUDGET_RESCHEDULER_MAX_TIME = int(
os.getenv("PROXY_BUDGET_RESCHEDULER_MAX_TIME", 605)
)
PROXY_BATCH_WRITE_AT = int(os.getenv("PROXY_BATCH_WRITE_AT", 10)) # in seconds
# MEMORY LEAK FIX: Increased from 10s to 30s minimum to prevent memory issues with APScheduler
# Very frequent intervals (<30s) can cause memory leaks in APScheduler's internal functions
PROXY_BATCH_WRITE_AT = int(os.getenv("PROXY_BATCH_WRITE_AT", 30)) # in seconds, increased from 10
# APScheduler Configuration - MEMORY LEAK FIX
# These settings prevent memory leaks in APScheduler's normalize() and _apply_jitter() functions
APSCHEDULER_COALESCE = os.getenv("APSCHEDULER_COALESCE", "True").lower() in ["true", "1"] # collapse many missed runs into one
APSCHEDULER_MISFIRE_GRACE_TIME = int(os.getenv("APSCHEDULER_MISFIRE_GRACE_TIME", 3600)) # ignore runs older than 1 hour (was 120)
APSCHEDULER_MAX_INSTANCES = int(os.getenv("APSCHEDULER_MAX_INSTANCES", 1)) # prevent concurrent job instances
APSCHEDULER_REPLACE_EXISTING = os.getenv("APSCHEDULER_REPLACE_EXISTING", "True").lower() in ["true", "1"] # always replace existing jobs
DEFAULT_HEALTH_CHECK_INTERVAL = int(
os.getenv("DEFAULT_HEALTH_CHECK_INTERVAL", 300)
) # 5 minutes

1
litellm/images/main.py
View file

@ -342,6 +342,7 @@ def image_generation( # noqa: PLR0915
litellm.LlmProviders.RECRAFT,
litellm.LlmProviders.AIML,
litellm.LlmProviders.GEMINI,
litellm.LlmProviders.FAL_AI,
):
if image_generation_config is None:
raise ValueError(

39
litellm/integrations/_types/open_inference.py
View file

@ -387,3 +387,42 @@ class OpenInferenceLLMProviderValues(Enum):
GOOGLE = "google"
AZURE = "azure"
AWS = "aws"
class ErrorAttributes:
"""
Attributes for error information in spans.
These attributes follow OpenTelemetry semantic conventions for exceptions
and are used to record error information from StandardLoggingPayloadErrorInformation.
"""
ERROR_TYPE = "error.type"
"""
The type/class of the error (e.g., 'ValueError', 'OpenAIError', 'RateLimitError').
Corresponds to StandardLoggingPayloadErrorInformation.error_class
"""
ERROR_MESSAGE = "error.message"
"""
The error message describing what went wrong.
Corresponds to StandardLoggingPayloadErrorInformation.error_message
"""
ERROR_CODE = "error.code"
"""
The error code (e.g., HTTP status code like '500', '429', or provider-specific codes).
Corresponds to StandardLoggingPayloadErrorInformation.error_code
"""
ERROR_STACK_TRACE = "error.stack_trace"
"""
The full stack trace of the error.
Corresponds to StandardLoggingPayloadErrorInformation.traceback
"""
ERROR_LLM_PROVIDER = "error.llm_provider"
"""
The LLM provider where the error occurred (e.g., 'openai', 'anthropic', 'azure').
Corresponds to StandardLoggingPayloadErrorInformation.llm_provider
"""

97
litellm/integrations/custom_logger.py
View file

@ -1,5 +1,6 @@
#### What this does ####
# On success, logs events to Promptlayer
import re
import traceback
from typing import (
TYPE_CHECKING,
@ -15,7 +16,9 @@ from typing import (
from pydantic import BaseModel
from litellm._logging import verbose_logger
from litellm.caching.caching import DualCache
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER
from litellm.types.integrations.argilla import ArgillaItem
from litellm.types.llms.openai import AllMessageValues, ChatCompletionRequest
from litellm.types.utils import (
@ -53,6 +56,12 @@ else:
PreRoutingHookResponse = Any
_BASE64_INLINE_PATTERN = re.compile(
r"data:(?:application|image|audio|video)/[a-zA-Z0-9.+-]+;base64,[A-Za-z0-9+/=\s]+",
re.MULTILINE,
)
class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callback#callback-class
# Class variables or attributes
def __init__(
@ -567,3 +576,91 @@ class CustomLogger: # https://docs.litellm.ai/docs/observability/custom_callbac
Get the proxy server request from cold storage using the object key directly.
"""
pass
async def _strip_base64_from_messages(
self, payload: "StandardLoggingPayload", max_depth: int = DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER
) -> "StandardLoggingPayload":
"""
Removes or redacts base64-encoded file data (e.g., PDFs, images, audio)
from messages and responses before sending to SQS.
Behavior:
• Drop entries with a 'file' key.
• Drop entries with type == 'file' or any non-text type.
• Keep untyped or text content.
• Recursively redact inline base64 blobs in *any* string field, at any depth.
"""
raw_messages: Any = payload.get("messages", [])
messages: list[Any] = raw_messages if isinstance(raw_messages, list) else []
verbose_logger.debug(f"[CustomLogger] Stripping base64 from {len(messages)} messages")
if messages:
payload["messages"] = self._process_messages(messages=messages, max_depth=max_depth)
total_items = 0
for m in payload.get("messages", []) or []:
if isinstance(m, dict):
content = m.get("content", [])
if isinstance(content, list):
total_items += len(content)
verbose_logger.debug(
f"[CustomLogger] Completed base64 strip; retained {total_items} content items"
)
return payload
def _redact_base64(self, value: Any, depth: int = 0, max_depth: int = DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER) -> Any:
"""Recursively redact inline base64 from any nested structure with a max recursion depth limit."""
if depth > max_depth:
verbose_logger.warning(
f"[CustomLogger] Max recursion depth {max_depth} reached while redacting base64"
)
return "[MAX_DEPTH_REACHED]"
if isinstance(value, str):
if _BASE64_INLINE_PATTERN.search(value):
verbose_logger.debug(
f"[CustomLogger] Redacted inline base64 string: {value[:40]}..."
)
return _BASE64_INLINE_PATTERN.sub("[BASE64_REDACTED]", value)
return value
if isinstance(value, list):
return [self._redact_base64(value=v, depth=depth + 1, max_depth=max_depth) for v in value]
if isinstance(value, dict):
return {k: self._redact_base64(value=v, depth=depth + 1, max_depth=max_depth) for k, v in value.items()}
return value
def _should_keep_content(self, content: Any) -> bool:
"""Return True if this content item should be retained."""
if not isinstance(content, dict):
return True
if "file" in content:
return False
ctype = content.get("type")
return not (isinstance(ctype, str) and ctype != "text")
def _process_messages(self, messages: list[Any], max_depth: int = DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER) -> List[Dict[str, Any]]:
filtered_messages: List[Dict[str, Any]] = []
for msg in messages:
if not isinstance(msg, dict):
continue
contents: Any = msg.get("content")
if isinstance(contents, list):
cleaned: list[Any] = []
for c in contents:
if self._should_keep_content(content=c):
cleaned.append(self._redact_base64(value=c, max_depth=max_depth))
msg["content"] = cleaned
else:
msg["content"] = self._redact_base64(value=contents, max_depth=max_depth)
for key, val in list(msg.items()):
if key != "content":
msg[key] = self._redact_base64(value=val, max_depth=max_depth)
filtered_messages.append(msg)
return filtered_messages

19
litellm/integrations/gitlab/__init__.py
View file

@ -8,7 +8,7 @@ if TYPE_CHECKING:
from litellm.types.prompts.init_prompts import SupportedPromptIntegrations
from litellm.integrations.custom_prompt_management import CustomPromptManagement
from litellm.types.prompts.init_prompts import PromptSpec, PromptLiteLLMParams
from .gitlab_prompt_manager import GitLabPromptManager
from .gitlab_prompt_manager import GitLabPromptManager, GitLabPromptCache
# Global instances
global_gitlab_config: Optional[dict] = None
@ -16,13 +16,13 @@ global_gitlab_config: Optional[dict] = None
def set_global_gitlab_config(config: dict) -> None:
"""
Set the global BitBucket configuration for prompt management.
Set the global gitlab configuration for prompt management.
Args:
config: Dictionary containing BitBucket configuration
- workspace: BitBucket workspace name
config: Dictionary containing gitlab configuration
- workspace: gitlab workspace name
- repository: Repository name
- access_token: BitBucket access token
- access_token: gitlab access token
- branch: Branch to fetch prompts from (default: main)
"""
import litellm
@ -34,7 +34,7 @@ def prompt_initializer(
litellm_params: "PromptLiteLLMParams", prompt_spec: "PromptSpec"
) -> "CustomPromptManagement":
"""
Initialize a prompt from a BitBucket repository.
Initialize a prompt from a Gitlab repository.
"""
gitlab_config = getattr(litellm_params, "gitlab_config", None)
prompt_id = getattr(litellm_params, "prompt_id", None)
@ -42,16 +42,16 @@ def prompt_initializer(
if not gitlab_config:
raise ValueError(
"bitbucket_config is required for BitBucket prompt integration"
"gitlab_config is required for gitlab prompt integration"
)
try:
bitbucket_prompt_manager = GitLabPromptManager(
gitlab_prompt_manager = GitLabPromptManager(
gitlab_config=gitlab_config,
prompt_id=prompt_id,
)
return bitbucket_prompt_manager
return gitlab_prompt_manager
except Exception as e:
raise e
@ -90,6 +90,7 @@ prompt_initializer_registry = {
# Export public API
__all__ = [
"GitLabPromptManager",
"GitLabPromptCache",
"set_global_gitlab_config",
"global_gitlab_config",
]

180
litellm/integrations/gitlab/gitlab_prompt_manager.py
View file

@ -12,10 +12,24 @@ from litellm.integrations.prompt_management_base import (
)
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import StandardCallbackDynamicParams
from litellm.integrations.gitlab.gitlab_client import GitLabClient
GITLAB_PREFIX = "gitlab::"
def encode_prompt_id(raw_id: str) -> str:
"""Convert GitLab path IDs like 'invoice/extract' → 'gitlab::invoice::extract'"""
if raw_id.startswith(GITLAB_PREFIX):
return raw_id # already encoded
return f"{GITLAB_PREFIX}{raw_id.replace('/', '::')}"
def decode_prompt_id(encoded_id: str) -> str:
"""Convert 'gitlab::invoice::extract' → 'invoice/extract'"""
if not encoded_id.startswith(GITLAB_PREFIX):
return encoded_id
return encoded_id[len(GITLAB_PREFIX):].replace("::", "/")
class GitLabPromptTemplate:
def __init__(
self,
@ -87,6 +101,7 @@ class GitLabTemplateManager:
def _id_to_repo_path(self, prompt_id: str) -> str:
"""Map a prompt_id to a repo path (respects prompts_path and adds .prompt)."""
prompt_id = decode_prompt_id(prompt_id)
if self.prompts_path:
return f"{self.prompts_path}/{prompt_id}.prompt"
return f"{prompt_id}.prompt"
@ -101,26 +116,27 @@ class GitLabTemplateManager:
path = path[len(self.prompts_path.strip("/")) + 1 :]
if path.endswith(".prompt"):
path = path[: -len(".prompt")]
return path
return encode_prompt_id(path)
# ---------- loading ----------
def _load_prompt_from_gitlab(self, prompt_id: str, *, ref: Optional[str] = None) -> None:
"""Load a specific .prompt file from GitLab (scoped under prompts_path if set)."""
try:
# prompt_id = decode_prompt_id(prompt_id)
file_path = self._id_to_repo_path(prompt_id)
prompt_content = self.gitlab_client.get_file_content(file_path, ref=ref)
if prompt_content:
template = self._parse_prompt_file(prompt_content, prompt_id)
self.prompts[prompt_id] = template
except Exception as e:
raise Exception(f"Failed to load prompt '{prompt_id}' from GitLab: {e}")
raise Exception(f"Failed to load prompt '{encode_prompt_id(prompt_id)}' from GitLab: {e}")
def load_all_prompts(self, *, recursive: bool = True) -> List[str]:
"""
Eagerly load all .prompt files from prompts_path. Returns loaded IDs.
"""
files = self.list_templates(recursive=recursive) # reuse logic
files = self.list_templates(recursive=recursive)
loaded: List[str] = []
for pid in files:
if pid not in self.prompts:
@ -195,9 +211,6 @@ class GitLabTemplateManager:
return self.prompts.get(template_id)
def list_templates(self, *, recursive: bool = True) -> List[str]:
"""
List available prompt IDs discovered under prompts_path (no extension, relative to prompts_path).
"""
"""
List available prompt IDs under prompts_path (no extension).
Compatible with both list_files signatures:
@ -248,7 +261,7 @@ class GitLabPromptManager(CustomPromptManagement):
"access_token": "glpat_***",
"tag": "v1.2.3", # optional; takes precedence
"branch": "main", # default fallback
"prompts_path": "prompts/chat" # <--- NEW
"prompts_path": "prompts/chat"
}
"""
@ -438,9 +451,11 @@ class GitLabPromptManager(CustomPromptManagement):
prompt_version: Optional[int] = None,
) -> PromptManagementClient:
try:
if prompt_id not in self.prompt_manager.prompts:
decoded_id = decode_prompt_id(prompt_id)
if decoded_id not in self.prompt_manager.prompts:
git_ref = getattr(dynamic_callback_params, "extra", {}).get("git_ref") if hasattr(dynamic_callback_params, "extra") else None
self.prompt_manager._load_prompt_from_gitlab(prompt_id, ref=git_ref)
self.prompt_manager._load_prompt_from_gitlab(decoded_id, ref=git_ref)
rendered_prompt, prompt_metadata = self.get_prompt_template(
prompt_id, prompt_variables
@ -486,3 +501,148 @@ class GitLabPromptManager(CustomPromptManagement):
prompt_label,
prompt_version,
)
class GitLabPromptCache:
"""
Cache all .prompt files from a GitLab repo into memory.
- Keys are the *repo file paths* (e.g. "prompts/chat/greet/hi.prompt")
mapped to JSON-like dicts containing content + metadata.
- Also exposes a by-ID view (ID == path relative to prompts_path without ".prompt",
e.g. "greet/hi").
Usage:
cfg = {
"project": "group/subgroup/repo",
"access_token": "glpat_***",
"prompts_path": "prompts/chat", # optional, can be empty for repo root
# "branch": "main", # default is "main"
# "tag": "v1.2.3", # takes precedence over branch
# "base_url": "https://gitlab.com/api/v4" # default
}
cache = GitLabPromptCache(cfg)
cache.load_all() # fetch + parse all .prompt files
print(cache.list_files()) # repo file paths
print(cache.list_ids()) # template IDs relative to prompts_path
prompt_json = cache.get_by_file("prompts/chat/greet/hi.prompt")
prompt_json2 = cache.get_by_id("greet/hi")
# If GitLab content changes and you want to refresh:
cache.reload() # re-scan and refresh all
"""
def __init__(
self,
gitlab_config: Dict[str, Any],
*,
ref: Optional[str] = None,
gitlab_client: Optional[GitLabClient] = None,
) -> None:
# Build a PromptManager (which internally builds TemplateManager + Client)
self.prompt_manager = GitLabPromptManager(
gitlab_config=gitlab_config,
prompt_id=None,
ref=ref,
gitlab_client=gitlab_client,
)
self.template_manager: GitLabTemplateManager = self.prompt_manager.prompt_manager
# In-memory stores
self._by_file: Dict[str, Dict[str, Any]] = {}
self._by_id: Dict[str, Dict[str, Any]] = {}
# -------------------------
# Public API
# -------------------------
def load_all(self, *, recursive: bool = True) -> Dict[str, Dict[str, Any]]:
"""
Scan GitLab for all .prompt files under prompts_path, load and parse each,
and return the mapping of repo file path -> JSON-like dict.
"""
ids = self.template_manager.list_templates(recursive=recursive) # IDs relative to prompts_path
for pid in ids:
# Ensure template is loaded into TemplateManager
if pid not in self.template_manager.prompts:
self.template_manager._load_prompt_from_gitlab(pid)
tmpl = self.template_manager.get_template(pid)
if tmpl is None:
# If something raced/failed, try once more
self.template_manager._load_prompt_from_gitlab(pid)
tmpl = self.template_manager.get_template(pid)
if tmpl is None:
continue
file_path = self.template_manager._id_to_repo_path(pid) # "prompts/chat/..../file.prompt"
entry = self._template_to_json(pid, tmpl)
self._by_file[file_path] = entry
# prefixed_id = pid if pid.startswith("gitlab::") else f"gitlab::{pid}"
encoded_id = encode_prompt_id(pid)
self._by_id[encoded_id] = entry
# self._by_id[pid] = entry
return self._by_id
def reload(self, *, recursive: bool = True) -> Dict[str, Dict[str, Any]]:
"""Clear the cache and re-load from GitLab."""
self._by_file.clear()
self._by_id.clear()
return self.load_all(recursive=recursive)
def list_files(self) -> List[str]:
"""Return the repo file paths currently cached."""
return list(self._by_file.keys())
def list_ids(self) -> List[str]:
"""Return the template IDs (relative to prompts_path, without extension) currently cached."""
return list(self._by_id.keys())
def get_by_file(self, file_path: str) -> Optional[Dict[str, Any]]:
"""Get a cached prompt JSON by repo file path."""
return self._by_file.get(file_path)
def get_by_id(self, prompt_id: str) -> Optional[Dict[str, Any]]:
"""Get a cached prompt JSON by prompt ID (relative to prompts_path)."""
if prompt_id in self._by_id:
return self._by_id[prompt_id]
# Try normalized forms
decoded = decode_prompt_id(prompt_id)
encoded = encode_prompt_id(decoded)
return self._by_id.get(encoded) or self._by_id.get(decoded)
# -------------------------
# Internals
# -------------------------
def _template_to_json(self, prompt_id: str, tmpl: GitLabPromptTemplate) -> Dict[str, Any]:
"""
Normalize a GitLabPromptTemplate into a JSON-like dict that is easy to serialize.
"""
# Safer copy of metadata (avoid accidental mutation)
md = dict(tmpl.metadata or {})
# Pull standard fields (also present in metadata sometimes)
model = tmpl.model
temperature = tmpl.temperature
max_tokens = tmpl.max_tokens
optional_params = dict(tmpl.optional_params or {})
return {
"id": prompt_id, # e.g. "greet/hi"
"path": self.template_manager._id_to_repo_path(prompt_id), # e.g. "prompts/chat/greet/hi.prompt"
"content": tmpl.content, # rendered content (without frontmatter)
"metadata": md, # parsed frontmatter
"model": model,
"temperature": temperature,
"max_tokens": max_tokens,
"optional_params": optional_params,
}

77
litellm/integrations/langfuse/langfuse_otel.py
View file

@ -48,11 +48,12 @@ class LangfuseOtelLogger(OpenTelemetry):
_utils.set_attributes(span, kwargs, response_obj)
#########################################################
# Set Langfuse specific attributes eg Langfuse Environment
# Set Langfuse specific attributes
#########################################################
LangfuseOtelLogger._set_langfuse_specific_attributes(
span=span,
kwargs=kwargs
kwargs=kwargs,
response_obj=response_obj
)
return
@ -86,7 +87,7 @@ class LangfuseOtelLogger(OpenTelemetry):
return metadata
@staticmethod
def _set_langfuse_specific_attributes(span: Span, kwargs):
def _set_langfuse_specific_attributes(span: Span, kwargs, response_obj):
"""
Sets Langfuse specific metadata attributes onto the OTEL span.
@ -96,6 +97,7 @@ class LangfuseOtelLogger(OpenTelemetry):
compatibility.
"""
from litellm.integrations.arize._utils import safe_set_attribute
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
# 1) Environment variable override
langfuse_environment = os.environ.get("LANGFUSE_TRACING_ENVIRONMENT")
@ -141,6 +143,75 @@ class LangfuseOtelLogger(OpenTelemetry):
value = str(value)
safe_set_attribute(span, enum_attr.value, value)
# 3) Set observation input/output for better UI display
#
# These Langfuse-specific attributes provide better UI display,
# especially for tool calls and function calling.
# Set observation input (messages)
messages = kwargs.get("messages")
if messages:
safe_set_attribute(
span,
LangfuseSpanAttributes.OBSERVATION_INPUT.value,
safe_dumps(messages),
)
# Set observation output (response with tool_calls if present)
if response_obj and hasattr(response_obj, "get"):
choices = response_obj.get("choices", [])
if choices:
# Extract the first choice's message
first_choice = choices[0]
message = first_choice.get("message", {})
# Check if there are tool_calls
tool_calls = message.get("tool_calls")
if tool_calls:
# Transform tool_calls to Langfuse-expected format
transformed_tool_calls = []
for tool_call in tool_calls:
function = tool_call.get("function", {})
arguments_str = function.get("arguments", "{}")
# Parse arguments from JSON string to object
try:
arguments_obj = json.loads(arguments_str) if isinstance(arguments_str, str) else arguments_str
except json.JSONDecodeError:
arguments_obj = {}
# Create Langfuse-compatible tool call object
langfuse_tool_call = {
"id": response_obj.get("id", ""),
"name": function.get("name", ""),
"call_id": tool_call.get("id", ""),
"type": "function_call",
"arguments": arguments_obj,
}
transformed_tool_calls.append(langfuse_tool_call)
# Set the observation output with transformed tool_calls
safe_set_attribute(
span,
LangfuseSpanAttributes.OBSERVATION_OUTPUT.value,
safe_dumps(transformed_tool_calls),
)
else:
# No tool_calls, use regular content-based output
output_data = {}
if message.get("role"):
output_data["role"] = message.get("role")
if message.get("content") is not None:
output_data["content"] = message.get("content")
if output_data:
safe_set_attribute(
span,
LangfuseSpanAttributes.OBSERVATION_OUTPUT.value,
safe_dumps(output_data),
)
@staticmethod
def _get_langfuse_otel_host() -> Optional[str]:
"""

85
litellm/integrations/opentelemetry.py
View file

@ -841,6 +841,10 @@ class OpenTelemetry(CustomLogger):
)
span.set_status(Status(StatusCode.ERROR))
self.set_attributes(span, kwargs, response_obj)
# Record exception information using OTEL standard method
self._record_exception_on_span(span=span, kwargs=kwargs)
span.end(end_time=self._to_ns(end_time))
# Create span for guardrail information
@ -849,6 +853,87 @@ class OpenTelemetry(CustomLogger):
if parent_otel_span is not None:
parent_otel_span.end(end_time=self._to_ns(datetime.now()))
def _record_exception_on_span(self, span: Span, kwargs: dict):
"""
Record exception information on the span using OTEL standard methods.
This extracts error information from StandardLoggingPayload and:
1. Uses span.record_exception() for the actual exception object (OTEL standard)
2. Sets structured error attributes from StandardLoggingPayloadErrorInformation
"""
try:
from litellm.integrations._types.open_inference import ErrorAttributes
# Get the exception object if available
exception = kwargs.get("exception")
# Record the exception using OTEL's standard method
if exception is not None:
span.record_exception(exception)
# Get StandardLoggingPayload for structured error information
standard_logging_payload: Optional[StandardLoggingPayload] = kwargs.get(
"standard_logging_object"
)
if standard_logging_payload is None:
return
# Extract error_information from StandardLoggingPayload
error_information = standard_logging_payload.get("error_information")
if error_information is None:
# Fallback to error_str if error_information is not available
error_str = standard_logging_payload.get("error_str")
if error_str:
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_MESSAGE,
value=error_str,
)
return
# Set structured error attributes from StandardLoggingPayloadErrorInformation
if error_information.get("error_code"):
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_CODE,
value=error_information["error_code"],
)
if error_information.get("error_class"):
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_TYPE,
value=error_information["error_class"],
)
if error_information.get("error_message"):
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_MESSAGE,
value=error_information["error_message"],
)
if error_information.get("llm_provider"):
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_LLM_PROVIDER,
value=error_information["llm_provider"],
)
if error_information.get("traceback"):
self.safe_set_attribute(
span=span,
key=ErrorAttributes.ERROR_STACK_TRACE,
value=error_information["traceback"],
)
except Exception as e:
verbose_logger.exception(
"OpenTelemetry: Error recording exception on span: %s", str(e)
)
def set_tools_attributes(self, span: Span, tools):
import json

7
litellm/integrations/opik/opik.py
View file

@ -204,6 +204,13 @@ class OpikLogger(CustomBatchLogger):
# Update litellm_opik_metadata with opik metadata from requester
standard_logging_metadata = standard_logging_object.get("metadata", {}) or {}
requester_metadata = standard_logging_metadata.get("requester_metadata", {}) or {}
# If requester_metadata is empty, try to get it from user_api_key_auth_metadata saved in api key
if not requester_metadata:
requester_metadata = standard_logging_metadata.get(
"user_api_key_auth_metadata", {}
) or {}
requester_opik_metadata = requester_metadata.get("opik", {}) or {}
litellm_opik_metadata.update(requester_opik_metadata)

90
litellm/integrations/sqs.py
View file

@ -9,6 +9,7 @@ from __future__ import annotations
import asyncio
import base64
import json
import re
import traceback
from typing import List, Optional
@ -30,6 +31,11 @@ from litellm.types.utils import StandardLoggingPayload
from .custom_batch_logger import CustomBatchLogger
_BASE64_INLINE_PATTERN = re.compile(
r"data:(?:application|image|audio|video)/[a-zA-Z0-9.+-]+;base64,[A-Za-z0-9+/=\s]+",
re.MULTILINE,
)
class SQSLogger(CustomBatchLogger, BaseAWSLLM):
"""Batching logger that writes logs to an AWS SQS queue, optionally encrypting the payload."""
@ -54,6 +60,7 @@ class SQSLogger(CustomBatchLogger, BaseAWSLLM):
sqs_flush_interval: Optional[int] = DEFAULT_SQS_FLUSH_INTERVAL_SECONDS,
sqs_batch_size: Optional[int] = DEFAULT_SQS_BATCH_SIZE,
sqs_config=None,
sqs_strip_base64_files: bool = False,
# --- 🔐 Application-level encryption params ---
sqs_aws_use_application_level_encryption: bool = False,
sqs_app_encryption_key_b64: Optional[str] = None,
@ -84,6 +91,7 @@ class SQSLogger(CustomBatchLogger, BaseAWSLLM):
sqs_aws_role_name=sqs_aws_role_name,
sqs_aws_web_identity_token=sqs_aws_web_identity_token,
sqs_aws_sts_endpoint=sqs_aws_sts_endpoint,
sqs_strip_base64_files=sqs_strip_base64_files,
sqs_aws_use_application_level_encryption=sqs_aws_use_application_level_encryption,
sqs_app_encryption_key_b64=sqs_app_encryption_key_b64,
sqs_app_encryption_aad=sqs_app_encryption_aad,
@ -113,25 +121,26 @@ class SQSLogger(CustomBatchLogger, BaseAWSLLM):
raise e
def _init_sqs_params(
self,
sqs_queue_url: Optional[str] = None,
sqs_region_name: Optional[str] = None,
sqs_api_version: Optional[str] = None,
sqs_use_ssl: bool = True,
sqs_verify: Optional[bool] = None,
sqs_endpoint_url: Optional[str] = None,
sqs_aws_access_key_id: Optional[str] = None,
sqs_aws_secret_access_key: Optional[str] = None,
sqs_aws_session_token: Optional[str] = None,
sqs_aws_session_name: Optional[str] = None,
sqs_aws_profile_name: Optional[str] = None,
sqs_aws_role_name: Optional[str] = None,
sqs_aws_web_identity_token: Optional[str] = None,
sqs_aws_sts_endpoint: Optional[str] = None,
sqs_aws_use_application_level_encryption: bool = False,
sqs_app_encryption_key_b64: Optional[str] = None,
sqs_app_encryption_aad: Optional[str] = None,
sqs_config=None,
self,
sqs_queue_url: Optional[str] = None,
sqs_region_name: Optional[str] = None,
sqs_api_version: Optional[str] = None,
sqs_use_ssl: bool = True,
sqs_verify: Optional[bool] = None,
sqs_endpoint_url: Optional[str] = None,
sqs_aws_access_key_id: Optional[str] = None,
sqs_aws_secret_access_key: Optional[str] = None,
sqs_aws_session_token: Optional[str] = None,
sqs_aws_session_name: Optional[str] = None,
sqs_aws_profile_name: Optional[str] = None,
sqs_aws_role_name: Optional[str] = None,
sqs_aws_web_identity_token: Optional[str] = None,
sqs_aws_sts_endpoint: Optional[str] = None,
sqs_strip_base64_files: bool = False,
sqs_aws_use_application_level_encryption: bool = False,
sqs_app_encryption_key_b64: Optional[str] = None,
sqs_app_encryption_aad: Optional[str] = None,
sqs_config=None,
) -> None:
litellm.aws_sqs_callback_params = litellm.aws_sqs_callback_params or {}
@ -141,55 +150,59 @@ class SQSLogger(CustomBatchLogger, BaseAWSLLM):
litellm.aws_sqs_callback_params[key] = litellm.get_secret(value)
self.sqs_queue_url = (
litellm.aws_sqs_callback_params.get("sqs_queue_url") or sqs_queue_url
litellm.aws_sqs_callback_params.get("sqs_queue_url") or sqs_queue_url
)
self.sqs_region_name = (
litellm.aws_sqs_callback_params.get("sqs_region_name") or sqs_region_name
litellm.aws_sqs_callback_params.get("sqs_region_name") or sqs_region_name
)
self.sqs_api_version = (
litellm.aws_sqs_callback_params.get("sqs_api_version") or sqs_api_version
litellm.aws_sqs_callback_params.get("sqs_api_version") or sqs_api_version
)
self.sqs_use_ssl = (
litellm.aws_sqs_callback_params.get("sqs_use_ssl", True) or sqs_use_ssl
litellm.aws_sqs_callback_params.get("sqs_use_ssl", True) or sqs_use_ssl
)
self.sqs_verify = litellm.aws_sqs_callback_params.get("sqs_verify") or sqs_verify
self.sqs_endpoint_url = (
litellm.aws_sqs_callback_params.get("sqs_endpoint_url") or sqs_endpoint_url
litellm.aws_sqs_callback_params.get("sqs_endpoint_url") or sqs_endpoint_url
)
self.sqs_aws_access_key_id = (
litellm.aws_sqs_callback_params.get("sqs_aws_access_key_id")
or sqs_aws_access_key_id
litellm.aws_sqs_callback_params.get("sqs_aws_access_key_id")
or sqs_aws_access_key_id
)
self.sqs_aws_secret_access_key = (
litellm.aws_sqs_callback_params.get("sqs_aws_secret_access_key")
or sqs_aws_secret_access_key
litellm.aws_sqs_callback_params.get("sqs_aws_secret_access_key")
or sqs_aws_secret_access_key
)
self.sqs_aws_session_token = (
litellm.aws_sqs_callback_params.get("sqs_aws_session_token")
or sqs_aws_session_token
litellm.aws_sqs_callback_params.get("sqs_aws_session_token")
or sqs_aws_session_token
)
self.sqs_aws_session_name = (
litellm.aws_sqs_callback_params.get("sqs_aws_session_name") or sqs_aws_session_name
litellm.aws_sqs_callback_params.get("sqs_aws_session_name") or sqs_aws_session_name
)
self.sqs_aws_profile_name = (
litellm.aws_sqs_callback_params.get("sqs_aws_profile_name") or sqs_aws_profile_name
litellm.aws_sqs_callback_params.get("sqs_aws_profile_name") or sqs_aws_profile_name
)
self.sqs_aws_role_name = (
litellm.aws_sqs_callback_params.get("sqs_aws_role_name") or sqs_aws_role_name
litellm.aws_sqs_callback_params.get("sqs_aws_role_name") or sqs_aws_role_name
)
self.sqs_aws_web_identity_token = (
litellm.aws_sqs_callback_params.get("sqs_aws_web_identity_token")
or sqs_aws_web_identity_token
litellm.aws_sqs_callback_params.get("sqs_aws_web_identity_token")
or sqs_aws_web_identity_token
)
self.sqs_aws_sts_endpoint = (
litellm.aws_sqs_callback_params.get("sqs_aws_sts_endpoint") or sqs_aws_sts_endpoint
litellm.aws_sqs_callback_params.get("sqs_aws_sts_endpoint") or sqs_aws_sts_endpoint
)
self.sqs_strip_base64_files = (
litellm.aws_sqs_callback_params.get("sqs_strip_base64_files", False)
or sqs_strip_base64_files
)
self.sqs_aws_use_application_level_encryption = (
@ -217,13 +230,15 @@ class SQSLogger(CustomBatchLogger, BaseAWSLLM):
self.sqs_config = litellm.aws_sqs_callback_params.get("sqs_config") or sqs_config
async def async_log_success_event(
self, kwargs, response_obj, start_time, end_time
self, kwargs, response_obj, start_time, end_time
) -> None:
try:
verbose_logger.debug(
"SQS Logging - Enters logging function for model %s", kwargs
)
standard_logging_payload = kwargs.get("standard_logging_object")
if self.sqs_strip_base64_files:
standard_logging_payload = await self._strip_base64_from_messages(standard_logging_payload)
if standard_logging_payload is None:
raise ValueError("standard_logging_payload is None")
@ -337,4 +352,3 @@ class SQSLogger(CustomBatchLogger, BaseAWSLLM):
response.raise_for_status()
except Exception as e:
verbose_logger.exception(f"Error sending to SQS: {str(e)}")

6
litellm/litellm_core_utils/app_crypto.py
View file

@ -1,16 +1,18 @@
import base64
import json
import os
from typing import Optional
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
class AppCrypto:
def __init__(self, master_key: bytes):
if len(master_key) != 32:
raise ValueError("Master key must be 32 bytes for AES-256-GCM")
self.key = master_key
def encrypt_json(self, data: dict, aad: bytes | None = None) -> dict:
def encrypt_json(self, data: dict, aad: Optional[bytes] = None) -> dict:
aes = AESGCM(self.key)
nonce = os.urandom(12)
plaintext = json.dumps(data).encode("utf-8")
@ -22,7 +24,7 @@ class AppCrypto:
"tag": base64.b64encode(tag).decode(),
}
def decrypt_json(self, enc: dict, aad: bytes | None = None) -> dict:
def decrypt_json(self, enc: dict, aad: Optional[bytes] = None) -> dict:
aes = AESGCM(self.key)
nonce = base64.b64decode(enc["nonce"])
ct = base64.b64decode(enc["ciphertext"])

18
litellm/litellm_core_utils/core_helpers.py
View file

@ -1,6 +1,6 @@
# What is this?
## Helper utilities
from typing import TYPE_CHECKING, Any, Iterable, List, Optional, Union
from typing import TYPE_CHECKING, Any, Iterable, List, Literal, Optional, Union
import httpx
@ -138,6 +138,22 @@ def add_missing_spend_metadata_to_litellm_metadata(
return litellm_metadata
def get_metadata_variable_name_from_kwargs(
kwargs: dict,
) -> Literal["metadata", "litellm_metadata"]:
"""
Helper to return what the "metadata" field should be called in the request data
- New endpoints return `litellm_metadata`
- Old endpoints return `metadata`
Context:
- LiteLLM used `metadata` as an internal field for storing metadata
- OpenAI then started using this field for their metadata
- LiteLLM is now moving to using `litellm_metadata` for our metadata
"""
return "litellm_metadata" if "litellm_metadata" in kwargs else "metadata"
def get_litellm_metadata_from_kwargs(kwargs: dict):
"""
Helper to get litellm metadata from all litellm request kwargs

1
litellm/litellm_core_utils/exception_mapping_utils.py
View file

@ -67,6 +67,7 @@ class ExceptionCheckers:
"string too long. expected a string with maximum length",
"model's maximum context limit",
"is longer than the model's context length",
"input tokens exceed the configured limit",
]
for substring in known_exception_substrings:
if substring in _error_str_lowercase:

2
litellm/litellm_core_utils/litellm_logging.py
View file

@ -117,6 +117,7 @@ from litellm.types.utils import (
)
from litellm.types.videos.main import VideoObject
from litellm.utils import _get_base_model_from_metadata, executor, print_verbose
from litellm.llms.base_llm.ocr.transformation import OCRResponse
from ..integrations.argilla import ArgillaLogger
from ..integrations.arize.arize_phoenix import ArizePhoenixLogger
@ -1619,6 +1620,7 @@ class Logging(LiteLLMLoggingBaseClass):
or isinstance(logging_result, OpenAIFileObject)
or isinstance(logging_result, LiteLLMRealtimeStreamLoggingObject)
or isinstance(logging_result, OpenAIModerationResponse)
or isinstance(logging_result, OCRResponse) # OCR
or isinstance(logging_result, dict)
and logging_result.get("object") == "vector_store.search_results.page"
or isinstance(logging_result, VideoObject)

96
litellm/litellm_core_utils/prompt_templates/common_utils.py
View file

@ -654,6 +654,102 @@ def _get_image_mime_type_from_url(url: str) -> Optional[str]:
return None
def infer_content_type_from_url_and_content(
url: str,
content: bytes,
current_content_type: Optional[str] = None,
) -> str:
"""
Infer content type from URL extension and binary content when content-type header is missing or generic.
This helper implements a fallback strategy for determining MIME types when HTTP headers
are missing or provide generic values (like binary/octet-stream). It's commonly used
when processing images and documents from various sources (S3, URLs, etc.).
Fallback Strategy:
1. If current_content_type is valid (not None and not generic octet-stream), return it
2. Try to infer from URL extension (handles query parameters)
3. Try to detect from binary content signature (magic bytes)
4. Raise ValueError if all methods fail
Args:
url: The URL of the content (used to extract file extension)
content: The binary content (first ~100 bytes are sufficient for detection)
current_content_type: The current content-type from headers (may be None or generic)
Returns:
str: The inferred MIME type (e.g., "image/png", "application/pdf")
Raises:
ValueError: If content type cannot be determined by any method
Example:
>>> content_type = infer_content_type_from_url_and_content(
... url="https://s3.amazonaws.com/bucket/image.png?AWSAccessKeyId=123",
... content=png_binary_data,
... current_content_type="binary/octet-stream"
... )
>>> print(content_type)
"image/png"
"""
from litellm.litellm_core_utils.token_counter import get_image_type
# If we have a valid content type that's not generic, use it
if current_content_type and current_content_type not in [
"binary/octet-stream",
"application/octet-stream",
]:
return current_content_type
# Extension to MIME type mapping
# Supports images, documents, and other common file types
extension_to_mime = {
# Image formats
"jpg": "image/jpeg",
"jpeg": "image/jpeg",
"png": "image/png",
"gif": "image/gif",
"webp": "image/webp",
# Document formats
"pdf": "application/pdf",
"csv": "text/csv",
"doc": "application/msword",
"docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"xls": "application/vnd.ms-excel",
"xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"html": "text/html",
"txt": "text/plain",
"md": "text/markdown",
}
# Try to infer from URL extension
if url:
extension = url.split(".")[-1].lower().split("?")[0] # Remove query params
inferred_type = extension_to_mime.get(extension)
if inferred_type:
return inferred_type
# Try to detect from binary content signature (magic bytes)
if content:
detected_type = get_image_type(content[:100])
if detected_type:
type_to_mime = {
"png": "image/png",
"jpeg": "image/jpeg",
"gif": "image/gif",
"webp": "image/webp",
"heic": "image/heic",
}
if detected_type in type_to_mime:
return type_to_mime[detected_type]
# If all fallbacks failed, raise error
raise ValueError(
f"Unable to determine content type from URL: {url}. "
f"Response content-type: {current_content_type}"
)
def get_tool_call_names(tools: List[ChatCompletionToolParam]) -> List[str]:
"""
Get tool call names from tools

24
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -38,7 +38,11 @@ from litellm.types.llms.vertex_ai import FunctionResponse as VertexFunctionRespo
from litellm.types.llms.vertex_ai import PartType as VertexPartType
from litellm.types.utils import GenericImageParsingChunk
from .common_utils import convert_content_list_to_str, is_non_content_values_set
from .common_utils import (
convert_content_list_to_str,
infer_content_type_from_url_and_content,
is_non_content_values_set,
)
from .image_handling import convert_url_to_base64
@ -2536,13 +2540,17 @@ class BedrockImageProcessor:
"""Handles both sync and async image processing for Bedrock conversations."""
@staticmethod
def _post_call_image_processing(response: httpx.Response) -> Tuple[str, str]:
def _post_call_image_processing(response: httpx.Response, image_url: str = "") -> Tuple[str, str]:
# Check the response's content type to ensure it is an image
content_type = response.headers.get("content-type")
if not content_type:
raise ValueError(
f"URL does not contain content-type (content-type: {content_type})"
)
# Use helper function to infer content type with fallback logic
content_type = infer_content_type_from_url_and_content(
url=image_url,
content=response.content,
current_content_type=content_type,
)
content_type = _parse_content_type(content_type)
# Convert the image content to base64 bytes
@ -2561,7 +2569,7 @@ class BedrockImageProcessor:
response = await client.get(image_url, follow_redirects=True)
response.raise_for_status() # Raise an exception for HTTP errors
return BedrockImageProcessor._post_call_image_processing(response)
return BedrockImageProcessor._post_call_image_processing(response, image_url)
except Exception as e:
raise e
@ -2574,7 +2582,7 @@ class BedrockImageProcessor:
response = client.get(image_url, follow_redirects=True)
response.raise_for_status() # Raise an exception for HTTP errors
return BedrockImageProcessor._post_call_image_processing(response)
return BedrockImageProcessor._post_call_image_processing(response, image_url)
except Exception as e:
raise e

78
litellm/litellm_core_utils/redact_messages.py
View file

@ -14,6 +14,9 @@ import litellm
from litellm.integrations.custom_logger import CustomLogger
from litellm.secret_managers.main import str_to_bool
from litellm.types.utils import StandardCallbackDynamicParams
from litellm.litellm_core_utils.core_helpers import (
get_metadata_variable_name_from_kwargs,
)
import asyncio
if TYPE_CHECKING:
@ -37,6 +40,38 @@ def redact_message_input_output_from_custom_logger(
return result
def _redact_choice_content(choice):
"""Helper to redact content in a choice (message or delta)."""
if isinstance(choice, litellm.Choices):
choice.message.content = "redacted-by-litellm"
if hasattr(choice.message, "reasoning_content"):
choice.message.reasoning_content = "redacted-by-litellm"
if hasattr(choice.message, "thinking_blocks"):
choice.message.thinking_blocks = None
elif isinstance(choice, litellm.utils.StreamingChoices):
choice.delta.content = "redacted-by-litellm"
if hasattr(choice.delta, "reasoning_content"):
choice.delta.reasoning_content = "redacted-by-litellm"
if hasattr(choice.delta, "thinking_blocks"):
choice.delta.thinking_blocks = None
def _redact_responses_api_output(output_items):
"""Helper to redact ResponsesAPIResponse output items."""
for output_item in output_items:
if hasattr(output_item, "content") and isinstance(output_item.content, list):
for content_part in output_item.content:
if hasattr(content_part, "text"):
content_part.text = "redacted-by-litellm"
# Redact reasoning items in output array
if hasattr(output_item, "type") and output_item.type == "reasoning":
if hasattr(output_item, "summary") and isinstance(output_item.summary, list):
for summary_item in output_item.summary:
if hasattr(summary_item, "text"):
summary_item.text = "redacted-by-litellm"
def perform_redaction(model_call_details: dict, result):
"""
Performs the actual redaction on the logging object and result.
@ -56,19 +91,12 @@ def perform_redaction(model_call_details: dict, result):
_streaming_response = model_call_details["complete_streaming_response"]
if hasattr(_streaming_response, "choices"):
for choice in _streaming_response.choices:
if isinstance(choice, litellm.Choices):
choice.message.content = "redacted-by-litellm"
elif isinstance(choice, litellm.utils.StreamingChoices):
choice.delta.content = "redacted-by-litellm"
_redact_choice_content(choice)
elif hasattr(_streaming_response, "output"):
# Handle ResponsesAPIResponse format
for output_item in _streaming_response.output:
if hasattr(output_item, "content") and isinstance(
output_item.content, list
):
for content_part in output_item.content:
if hasattr(content_part, "text"):
content_part.text = "redacted-by-litellm"
_redact_responses_api_output(_streaming_response.output)
# Redact reasoning field in ResponsesAPIResponse
if hasattr(_streaming_response, "reasoning") and _streaming_response.reasoning is not None:
_streaming_response.reasoning = None
# Redact result
if result is not None:
@ -84,17 +112,13 @@ def perform_redaction(model_call_details: dict, result):
if isinstance(_result, litellm.ModelResponse):
if hasattr(_result, "choices") and _result.choices is not None:
for choice in _result.choices:
if isinstance(choice, litellm.Choices):
choice.message.content = "redacted-by-litellm"
elif isinstance(choice, litellm.utils.StreamingChoices):
choice.delta.content = "redacted-by-litellm"
_redact_choice_content(choice)
elif isinstance(_result, litellm.ResponsesAPIResponse):
if hasattr(_result, "output"):
for output_item in _result.output:
if hasattr(output_item, "content") and isinstance(output_item.content, list):
for content_part in output_item.content:
if hasattr(content_part, "text"):
content_part.text = "redacted-by-litellm"
_redact_responses_api_output(_result.output)
# Redact reasoning field in ResponsesAPIResponse
if hasattr(_result, "reasoning") and _result.reasoning is not None:
_result.reasoning = None
elif isinstance(_result, litellm.EmbeddingResponse):
if hasattr(_result, "data") and _result.data is not None:
_result.data = []
@ -107,11 +131,13 @@ def should_redact_message_logging(model_call_details: dict) -> bool:
"""
Determine if message logging should be redacted.
"""
_request_headers = (
model_call_details.get("litellm_params", {}).get("metadata", {}) or {}
)
request_headers = _request_headers.get("headers", {})
litellm_params = model_call_details.get("litellm_params", {})
metadata_field = get_metadata_variable_name_from_kwargs(litellm_params)
metadata = litellm_params.get(metadata_field, {})
# Get headers from the metadata
request_headers = metadata.get("headers", {}) if isinstance(metadata, dict) else {}
possible_request_headers = [
"litellm-enable-message-redaction", # old header. maintain backwards compatibility

2
litellm/llms/azure/common_utils.py
View file

@ -759,4 +759,4 @@ class BaseAzureLLM(BaseOpenAILLM):
def _is_azure_v1_api_version(api_version: Optional[str]) -> bool:
if api_version is None:
return False
return api_version == "preview" or api_version == "latest"
return api_version in {"preview", "latest", "v1"}

98
litellm/llms/bedrock/base_aws_llm.py
View file

@ -1,6 +1,7 @@
import hashlib
import json
import os
import urllib.parse
from datetime import datetime
from typing import (
TYPE_CHECKING,
@ -331,16 +332,61 @@ class BaseAWSLLM:
return provider
return None
@staticmethod
def get_bedrock_model_id(
optional_params: dict,
provider: Optional[BEDROCK_INVOKE_PROVIDERS_LITERAL],
model: str,
) -> str:
model_id = optional_params.pop("model_id", None)
if model_id is not None:
model_id = BaseAWSLLM.encode_model_id(model_id=model_id)
else:
model_id = model
model_id = model_id.replace("invoke/", "", 1)
if provider == "llama" and "llama/" in model_id:
model_id = BaseAWSLLM._get_model_id_from_model_with_spec(
model_id, spec="llama"
)
elif provider == "deepseek_r1" and "deepseek_r1/" in model_id:
model_id = BaseAWSLLM._get_model_id_from_model_with_spec(
model_id, spec="deepseek_r1"
)
return model_id
@staticmethod
def _get_model_id_from_model_with_spec(
model: str,
spec: str,
) -> str:
"""
Remove `llama` from modelID since `llama` is simply a spec to follow for custom bedrock models
"""
model_id = model.replace(spec + "/", "")
return BaseAWSLLM.encode_model_id(model_id=model_id)
@staticmethod
def encode_model_id(model_id: str) -> str:
"""
Double encode the model ID to ensure it matches the expected double-encoded format.
Args:
model_id (str): The model ID to encode.
Returns:
str: The double-encoded model ID.
"""
return urllib.parse.quote(model_id, safe="")
@staticmethod
def get_bedrock_embedding_provider(
model: str,
) -> Optional[BEDROCK_EMBEDDING_PROVIDERS_LITERAL]:
"""
Helper function to get the bedrock embedding provider from the model
Handles scenarios like:
1. model=cohere.embed-english-v3:0 -> Returns `cohere`
2. model=amazon.titan-embed-text-v1 -> Returns `amazon`
2. model=amazon.titan-embed-text-v1 -> Returns `amazon`
3. model=us.twelvelabs.marengo-embed-2-7-v1:0 -> Returns `twelvelabs`
4. model=twelvelabs.marengo-embed-2-7-v1:0 -> Returns `twelvelabs`
"""
@ -349,20 +395,24 @@ class BaseAWSLLM:
parts = model.split(".")
# Check if the second part (after potential region) is a known provider
if len(parts) >= 2:
potential_provider = parts[1] # e.g., "twelvelabs" from "us.twelvelabs.marengo-embed-2-7-v1:0"
potential_provider = parts[
1
] # e.g., "twelvelabs" from "us.twelvelabs.marengo-embed-2-7-v1:0"
if potential_provider in get_args(BEDROCK_EMBEDDING_PROVIDERS_LITERAL):
return cast(BEDROCK_EMBEDDING_PROVIDERS_LITERAL, potential_provider)
# Check if the first part is a known provider (standard format)
potential_provider = parts[0] # e.g., "cohere" from "cohere.embed-english-v3:0"
potential_provider = parts[
0
] # e.g., "cohere" from "cohere.embed-english-v3:0"
if potential_provider in get_args(BEDROCK_EMBEDDING_PROVIDERS_LITERAL):
return cast(BEDROCK_EMBEDDING_PROVIDERS_LITERAL, potential_provider)
# Fallback: check if any provider name appears in the model string
for provider in get_args(BEDROCK_EMBEDDING_PROVIDERS_LITERAL):
if provider in model:
return cast(BEDROCK_EMBEDDING_PROVIDERS_LITERAL, provider)
return None
def _get_aws_region_name(
@ -984,20 +1034,23 @@ class BaseAWSLLM:
raise ImportError(
"Missing boto3 to call bedrock. Run 'pip install boto3'."
)
# Filter headers for AWS signature calculation
# AWS SigV4 only includes specific headers in signature calculation
aws_signature_headers = self._filter_headers_for_aws_signature(headers)
sigv4 = SigV4Auth(credentials, "bedrock", aws_region_name)
request = AWSRequest(
method="POST", url=endpoint_url, data=data, headers=aws_signature_headers
method="POST",
url=endpoint_url,
data=data,
headers=aws_signature_headers,
)
sigv4.add_auth(request)
# Add back all original headers (including forwarded ones) after signature calculation
for header_name, header_value in headers.items():
request.headers[header_name] = header_value
if (
extra_headers is not None and "Authorization" in extra_headers
): # prevent sigv4 from overwriting the auth header
@ -1013,16 +1066,27 @@ class BaseAWSLLM:
"""
aws_signature_headers = {}
aws_headers = {
'host', 'content-type', 'date', 'x-amz-date', 'x-amz-security-token',
'x-amz-content-sha256', 'x-amz-algorithm', 'x-amz-credential',
'x-amz-signedheaders', 'x-amz-signature'
"host",
"content-type",
"date",
"x-amz-date",
"x-amz-security-token",
"x-amz-content-sha256",
"x-amz-algorithm",
"x-amz-credential",
"x-amz-signedheaders",
"x-amz-signature",
}
for header_name, header_value in headers.items():
header_lower = header_name.lower()
if header_lower in aws_headers or header_lower.startswith('x-amz-') or header_lower.startswith('x-amzn-'):
if (
header_lower in aws_headers
or header_lower.startswith("x-amz-")
or header_lower.startswith("x-amzn-")
):
aws_signature_headers[header_name] = header_value
return aws_signature_headers
def _sign_request(

11
litellm/llms/bedrock/chat/converse_handler.py
View file

@ -1,5 +1,4 @@
import json
import urllib
from typing import Any, Optional, Union
import httpx
@ -84,16 +83,6 @@ class BedrockConverseLLM(BaseAWSLLM):
def __init__(self) -> None:
super().__init__()
def encode_model_id(self, model_id: str) -> str:
"""
Double encode the model ID to ensure it matches the expected double-encoded format.
Args:
model_id (str): The model ID to encode.
Returns:
str: The double-encoded model ID.
"""
return urllib.parse.quote(model_id, safe="") # type: ignore
async def async_streaming(
self,
model: str,

42
litellm/llms/bedrock/chat/invoke_handler.py
View file

@ -3,20 +3,15 @@ TODO: DELETE FILE. Bedrock LLM is no longer used. Goto `litellm/llms/bedrock/cha
"""
import copy
import json
import time
import types
import urllib.parse
from functools import partial
from typing import (
Any,
AsyncIterator,
Callable,
Iterator,
List,
Optional,
Tuple,
Union,
cast,
get_args,
)
@ -672,16 +667,6 @@ class BedrockLLM(BaseAWSLLM):
return model_response
def encode_model_id(self, model_id: str) -> str:
"""
Double encode the model ID to ensure it matches the expected double-encoded format.
Args:
model_id (str): The model ID to encode.
Returns:
str: The double-encoded model ID.
"""
return urllib.parse.quote(model_id, safe="")
def completion( # noqa: PLR0915
self,
model: str,
@ -1176,33 +1161,6 @@ class BedrockLLM(BaseAWSLLM):
return cast(litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL, provider)
return None
def get_bedrock_model_id(
self,
optional_params: dict,
provider: Optional[litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL],
model: str,
) -> str:
modelId = optional_params.pop("model_id", None)
if modelId is not None:
modelId = self.encode_model_id(model_id=modelId)
else:
modelId = model
if provider == "llama" and "llama/" in modelId:
modelId = self._get_model_id_for_llama_like_model(modelId)
return modelId
def _get_model_id_for_llama_like_model(
self,
model: str,
) -> str:
"""
Remove `llama` from modelID since `llama` is simply a spec to follow for custom bedrock models
"""
model_id = model.replace("llama/", "")
return self.encode_model_id(model_id=model_id)
def get_response_stream_shape():
global _response_stream_shape_cache

63
litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py
View file

@ -1,7 +1,6 @@
import copy
import json
import time
import urllib.parse
from functools import partial
from typing import TYPE_CHECKING, Any, List, Optional, Tuple, Union, cast, get_args
@ -190,14 +189,16 @@ class AmazonInvokeConfig(BaseConfig, BaseAWSLLM):
] = True # cohere requires stream = True in inference params
request_data = {"prompt": prompt, **inference_params}
elif provider == "anthropic":
transformed_request = litellm.AmazonAnthropicClaudeConfig().transform_request(
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
headers=headers,
transformed_request = (
litellm.AmazonAnthropicClaudeConfig().transform_request(
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
headers=headers,
)
)
return transformed_request
elif provider == "nova":
return litellm.AmazonInvokeNovaConfig().transform_request(
@ -327,7 +328,9 @@ class AmazonInvokeConfig(BaseConfig, BaseAWSLLM):
elif provider == "meta" or provider == "llama" or provider == "deepseek_r1":
outputText = completion_response["generation"]
elif provider == "mistral":
outputText = litellm.AmazonMistralConfig.get_outputText(completion_response, model_response)
outputText = litellm.AmazonMistralConfig.get_outputText(
completion_response, model_response
)
else: # amazon titan
outputText = completion_response.get("results")[0].get("outputText")
except Exception as e:
@ -549,48 +552,6 @@ class AmazonInvokeConfig(BaseConfig, BaseAWSLLM):
return cast(litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL, provider)
return None
def get_bedrock_model_id(
self,
optional_params: dict,
provider: Optional[litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL],
model: str,
) -> str:
modelId = optional_params.pop("model_id", None)
if modelId is not None:
modelId = self.encode_model_id(model_id=modelId)
else:
modelId = model
modelId = modelId.replace("invoke/", "", 1)
if provider == "llama" and "llama/" in modelId:
modelId = self._get_model_id_from_model_with_spec(modelId, spec="llama")
elif provider == "deepseek_r1" and "deepseek_r1/" in modelId:
modelId = self._get_model_id_from_model_with_spec(
modelId, spec="deepseek_r1"
)
return modelId
def _get_model_id_from_model_with_spec(
self,
model: str,
spec: str,
) -> str:
"""
Remove `llama` from modelID since `llama` is simply a spec to follow for custom bedrock models
"""
model_id = model.replace(spec + "/", "")
return self.encode_model_id(model_id=model_id)
def encode_model_id(self, model_id: str) -> str:
"""
Double encode the model ID to ensure it matches the expected double-encoded format.
Args:
model_id (str): The model ID to encode.
Returns:
str: The double-encoded model ID.
"""
return urllib.parse.quote(model_id, safe="")
def convert_messages_to_prompt(
self, model, messages, provider, custom_prompt_dict
) -> Tuple[str, Optional[list]]:

18
litellm/llms/bedrock/image/image_handler.py
View file

@ -7,6 +7,7 @@ import httpx
from pydantic import BaseModel
import litellm
from litellm import BEDROCK_INVOKE_PROVIDERS_LITERAL
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.litellm_logging import Logging as LitellmLogging
from litellm.llms.bedrock.image.amazon_nova_canvas_transformation import (
@ -183,8 +184,14 @@ class BedrockImageGeneration(BaseAWSLLM):
optional_params, model
)
# Use the existing ARN-aware provider detection method
bedrock_provider = self.get_bedrock_invoke_provider(model)
### SET RUNTIME ENDPOINT ###
modelId = model
modelId = self.get_bedrock_model_id(
model=model,
provider=bedrock_provider,
optional_params=optional_params,
)
_, proxy_endpoint_url = self.get_runtime_endpoint(
api_base=api_base,
aws_bedrock_runtime_endpoint=boto3_credentials_info.aws_bedrock_runtime_endpoint,
@ -192,7 +199,10 @@ class BedrockImageGeneration(BaseAWSLLM):
)
proxy_endpoint_url = f"{proxy_endpoint_url}/model/{modelId}/invoke"
data = self._get_request_body(
model=model, prompt=prompt, optional_params=optional_params
model=model,
prompt=prompt,
optional_params=optional_params,
bedrock_provider=bedrock_provider,
)
# Make POST Request
@ -231,6 +241,7 @@ class BedrockImageGeneration(BaseAWSLLM):
def _get_request_body(
self,
model: str,
bedrock_provider: Optional[BEDROCK_INVOKE_PROVIDERS_LITERAL],
prompt: str,
optional_params: dict,
) -> dict:
@ -242,9 +253,6 @@ class BedrockImageGeneration(BaseAWSLLM):
Returns:
dict: The request body to use for the Bedrock Image Generation API
"""
# Use the existing ARN-aware provider detection method
bedrock_provider = self.get_bedrock_invoke_provider(model)
if bedrock_provider == "amazon" or bedrock_provider == "nova":
# Handle Amazon Nova Canvas models
provider = "amazon"

87
litellm/llms/cohere/chat/v2_transformation.py
View file

@ -6,7 +6,12 @@ import httpx
import litellm
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.types.llms.cohere import CohereV2ChatResponse
from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolCallChunk
from litellm.types.llms.openai import (
AllMessageValues,
ChatCompletionToolCallChunk,
ChatCompletionAnnotation,
ChatCompletionAnnotationURLCitation,
)
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm.types.utils import ModelResponse, Usage
@ -207,9 +212,15 @@ class CohereV2ChatConfig(OpenAIGPTConfig):
]
)
## ADD CITATIONS
if "citations" in cohere_v2_chat_response:
setattr(model_response, "citations", cohere_v2_chat_response["citations"])
## ADD CITATIONS AS ANNOTATIONS
annotations: Optional[List[ChatCompletionAnnotation]] = None
citations = None
if "message" in cohere_v2_chat_response and "citations" in cohere_v2_chat_response["message"]:
citations = cohere_v2_chat_response["message"]["citations"]
if citations:
annotations = self._translate_citations_to_openai_annotations(citations)
## Tool calling response
cohere_tools_response = cohere_v2_chat_response["message"].get("tool_calls", [])
@ -225,8 +236,13 @@ class CohereV2ChatConfig(OpenAIGPTConfig):
_message = litellm.Message(
tool_calls=tool_calls,
content=None,
annotations=annotations,
)
model_response.choices[0].message = _message # type: ignore
else:
if annotations:
current_message = model_response.choices[0].message # type: ignore
current_message.annotations = annotations
## CALCULATING USAGE - use cohere `billed_units` for returning usage
token_usage = cohere_v2_chat_response["usage"].get("tokens", {})
@ -276,3 +292,66 @@ class CohereV2ChatConfig(OpenAIGPTConfig):
self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers]
) -> BaseLLMException:
return CohereError(status_code=status_code, message=error_message)
def _translate_citations_to_openai_annotations(self, citations: List[dict]) -> List[ChatCompletionAnnotation]:
"""
Transform Cohere citations to OpenAI annotations format.
Creates separate annotations for each source in a citation, allowing multiple
annotations with the same start/end index if they reference different sources.
Args:
citations: List of Cohere citation objects with format:
{
"start": int,
"end": int,
"text": str,
"sources": [
{
"type": "document",
"document": {
"title": str,
"snippet": str,
...
},
"id": str
}
]
}
Returns:
List of OpenAI ChatCompletionAnnotation objects (one per source)
"""
annotations: List[ChatCompletionAnnotation] = []
for citation in citations:
start_index = citation.get("start", 0)
end_index = citation.get("end", 0)
# Extract source information - loop through all sources
sources = citation.get("sources", [])
if not sources:
continue
# Create an annotation for each source
for source in sources:
if source.get("type") == "document" and "document" in source:
document = source["document"]
title = document.get("title", "")
url = source.get("url") or f"source:{source.get('id', 'unknown')}"
url_citation: ChatCompletionAnnotationURLCitation = {
"start_index": start_index,
"end_index": end_index,
"title": title,
"url": url,
}
annotation: ChatCompletionAnnotation = {
"type": "url_citation",
"url_citation": url_citation,
}
annotations.append(annotation)
return annotations

120
litellm/llms/custom_httpx/http_handler.py
View file

@ -3,7 +3,7 @@ import os
import ssl
import sys
import time
from typing import TYPE_CHECKING, Any, Callable, Dict, List, Mapping, Optional, Union
from typing import TYPE_CHECKING, Any, Callable, Dict, List, Mapping, Optional, Tuple, Union
import certifi
import httpx
@ -47,6 +47,46 @@ headers = {
_DEFAULT_TIMEOUT = httpx.Timeout(timeout=5.0, connect=5.0)
def _prepare_request_data_and_content(
data: Optional[Union[dict, str, bytes]] = None,
content: Any = None,
) -> Tuple[Optional[Union[dict, Mapping]], Any]:
"""
Helper function to route data/content parameters correctly for httpx requests
This prevents httpx DeprecationWarnings that cause memory leaks.
Background:
- httpx shows a DeprecationWarning when you pass bytes/str to `data=`
- It wants you to use `content=` instead for bytes/str
- The warning itself leaks memory when triggered repeatedly
Solution:
- Move bytes/str from `data=` to `content=` before calling build_request
- Keep dicts in `data=` (that's still the correct parameter for dicts)
Args:
data: Request data (can be dict, str, or bytes)
content: Request content (raw bytes/str)
Returns:
Tuple of (request_data, request_content) properly routed for httpx
"""
request_data = None
request_content = content
if data is not None:
if isinstance(data, (bytes, str)):
# Bytes/strings belong in content= (only if not already provided)
if content is None:
request_content = data
else:
# dict/Mapping stays in data= parameter
request_data = data
return request_data, request_content
def get_ssl_configuration(
ssl_verify: Optional[VerifyTypes] = None,
) -> Union[bool, str, ssl.SSLContext]:
@ -301,17 +341,20 @@ class AsyncHTTPHandler:
if timeout is None:
timeout = self.timeout
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
request_data, request_content = _prepare_request_data_and_content(data, content)
req = self.client.build_request(
"POST",
url,
data=data, # type: ignore
data=request_data,
json=json,
params=params,
headers=headers,
timeout=timeout,
files=files,
content=content,
)
content=request_content,
)
response = await self.client.send(req, stream=stream)
response.raise_for_status()
return response
@ -364,19 +407,23 @@ class AsyncHTTPHandler:
async def put(
self,
url: str,
data: Optional[Union[dict, str]] = None, # type: ignore
data: Optional[Union[dict, str, bytes]] = None, # type: ignore
json: Optional[dict] = None,
params: Optional[dict] = None,
headers: Optional[dict] = None,
timeout: Optional[Union[float, httpx.Timeout]] = None,
stream: bool = False,
content: Any = None,
):
try:
if timeout is None:
timeout = self.timeout
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
request_data, request_content = _prepare_request_data_and_content(data, content)
req = self.client.build_request(
"PUT", url, data=data, json=json, params=params, headers=headers, timeout=timeout # type: ignore
"PUT", url, data=request_data, json=json, params=params, headers=headers, timeout=timeout, content=request_content # type: ignore
)
response = await self.client.send(req)
response.raise_for_status()
@ -424,19 +471,23 @@ class AsyncHTTPHandler:
async def patch(
self,
url: str,
data: Optional[Union[dict, str]] = None, # type: ignore
data: Optional[Union[dict, str, bytes]] = None, # type: ignore
json: Optional[dict] = None,
params: Optional[dict] = None,
headers: Optional[dict] = None,
timeout: Optional[Union[float, httpx.Timeout]] = None,
stream: bool = False,
content: Any = None,
):
try:
if timeout is None:
timeout = self.timeout
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
request_data, request_content = _prepare_request_data_and_content(data, content)
req = self.client.build_request(
"PATCH", url, data=data, json=json, params=params, headers=headers, timeout=timeout # type: ignore
"PATCH", url, data=request_data, json=json, params=params, headers=headers, timeout=timeout, content=request_content # type: ignore
)
response = await self.client.send(req)
response.raise_for_status()
@ -484,18 +535,23 @@ class AsyncHTTPHandler:
async def delete(
self,
url: str,
data: Optional[Union[dict, str]] = None, # type: ignore
data: Optional[Union[dict, str, bytes]] = None, # type: ignore
json: Optional[dict] = None,
params: Optional[dict] = None,
headers: Optional[dict] = None,
timeout: Optional[Union[float, httpx.Timeout]] = None,
stream: bool = False,
content: Any = None,
):
try:
if timeout is None:
timeout = self.timeout
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
request_data, request_content = _prepare_request_data_and_content(data, content)
req = self.client.build_request(
"DELETE", url, data=data, json=json, params=params, headers=headers, timeout=timeout # type: ignore
"DELETE", url, data=request_data, json=json, params=params, headers=headers, timeout=timeout, content=request_content # type: ignore
)
response = await self.client.send(req, stream=stream)
response.raise_for_status()
@ -543,8 +599,11 @@ class AsyncHTTPHandler:
Used for retrying connection client errors.
"""
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
request_data, request_content = _prepare_request_data_and_content(data, content)
req = client.build_request(
"POST", url, data=data, json=json, params=params, headers=headers, content=content # type: ignore
"POST", url, data=request_data, json=json, params=params, headers=headers, content=request_content # type: ignore
)
response = await client.send(req, stream=stream)
response.raise_for_status()
@ -798,21 +857,24 @@ class HTTPHandler:
logging_obj: Optional[LiteLLMLoggingObject] = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
request_data, request_content = _prepare_request_data_and_content(data, content)
if timeout is not None:
req = self.client.build_request(
"POST",
url,
data=data, # type: ignore
data=request_data, # type: ignore
json=json,
params=params,
headers=headers,
timeout=timeout,
files=files,
content=content, # type: ignore
content=request_content, # type: ignore
)
else:
req = self.client.build_request(
"POST", url, data=data, json=json, params=params, headers=headers, files=files, content=content # type: ignore
"POST", url, data=request_data, json=json, params=params, headers=headers, files=files, content=request_content # type: ignore
)
response = self.client.send(req, stream=stream)
response.raise_for_status()
@ -840,21 +902,25 @@ class HTTPHandler:
def patch(
self,
url: str,
data: Optional[Union[dict, str]] = None,
data: Optional[Union[dict, str, bytes]] = None,
json: Optional[Union[dict, str]] = None,
params: Optional[dict] = None,
headers: Optional[dict] = None,
stream: bool = False,
timeout: Optional[Union[float, httpx.Timeout]] = None,
content: Any = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
request_data, request_content = _prepare_request_data_and_content(data, content)
if timeout is not None:
req = self.client.build_request(
"PATCH", url, data=data, json=json, params=params, headers=headers, timeout=timeout # type: ignore
"PATCH", url, data=request_data, json=json, params=params, headers=headers, timeout=timeout, content=request_content # type: ignore
)
else:
req = self.client.build_request(
"PATCH", url, data=data, json=json, params=params, headers=headers # type: ignore
"PATCH", url, data=request_data, json=json, params=params, headers=headers, content=request_content # type: ignore
)
response = self.client.send(req, stream=stream)
response.raise_for_status()
@ -883,21 +949,25 @@ class HTTPHandler:
def put(
self,
url: str,
data: Optional[Union[dict, str]] = None,
data: Optional[Union[dict, str, bytes]] = None,
json: Optional[Union[dict, str]] = None,
params: Optional[dict] = None,
headers: Optional[dict] = None,
stream: bool = False,
timeout: Optional[Union[float, httpx.Timeout]] = None,
content: Any = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
request_data, request_content = _prepare_request_data_and_content(data, content)
if timeout is not None:
req = self.client.build_request(
"PUT", url, data=data, json=json, params=params, headers=headers, timeout=timeout # type: ignore
"PUT", url, data=request_data, json=json, params=params, headers=headers, timeout=timeout, content=request_content # type: ignore
)
else:
req = self.client.build_request(
"PUT", url, data=data, json=json, params=params, headers=headers # type: ignore
"PUT", url, data=request_data, json=json, params=params, headers=headers, content=request_content # type: ignore
)
response = self.client.send(req, stream=stream)
return response
@ -913,21 +983,25 @@ class HTTPHandler:
def delete(
self,
url: str,
data: Optional[Union[dict, str]] = None, # type: ignore
data: Optional[Union[dict, str, bytes]] = None, # type: ignore
json: Optional[dict] = None,
params: Optional[dict] = None,
headers: Optional[dict] = None,
timeout: Optional[Union[float, httpx.Timeout]] = None,
stream: bool = False,
content: Any = None,
):
try:
# Prepare data/content parameters to prevent httpx DeprecationWarning (memory leak fix)
request_data, request_content = _prepare_request_data_and_content(data, content)
if timeout is not None:
req = self.client.build_request(
"DELETE", url, data=data, json=json, params=params, headers=headers, timeout=timeout # type: ignore
"DELETE", url, data=request_data, json=json, params=params, headers=headers, timeout=timeout, content=request_content # type: ignore
)
else:
req = self.client.build_request(
"DELETE", url, data=data, json=json, params=params, headers=headers # type: ignore
"DELETE", url, data=request_data, json=json, params=params, headers=headers, content=request_content # type: ignore
)
response = self.client.send(req, stream=stream)
response.raise_for_status()

7
litellm/llms/custom_httpx/llm_http_handler.py
View file

@ -1774,11 +1774,16 @@ class BaseLLMHTTPHandler:
provider_specific_header=provider_specific_header,
custom_llm_provider=custom_llm_provider,
)
forwarded_headers = kwargs.get("headers", None)
if forwarded_headers and extra_headers:
merged_headers = {**forwarded_headers, **extra_headers}
else:
merged_headers = forwarded_headers or extra_headers
(
headers,
api_base,
) = anthropic_messages_provider_config.validate_anthropic_messages_environment(
headers=extra_headers or {},
headers=merged_headers or {},
model=model,
messages=messages,
optional_params=anthropic_messages_optional_request_params,

18
litellm/llms/deepgram/audio_transcription/transformation.py
View file

@ -58,7 +58,7 @@ class DeepgramAudioTranscriptionConfig(BaseAudioTranscriptionConfig):
) -> AudioTranscriptionRequestData:
"""
Processes the audio file input based on its type and returns AudioTranscriptionRequestData.
For Deepgram, the binary audio data is sent directly as the request body.
Args:
@ -69,12 +69,11 @@ class DeepgramAudioTranscriptionConfig(BaseAudioTranscriptionConfig):
"""
# Use common utility to process the audio file
processed_audio = process_audio_file(audio_file)
# Return structured data with binary content and no files
# For Deepgram, we send binary data directly as request body
return AudioTranscriptionRequestData(
data=processed_audio.file_content,
files=None
data=processed_audio.file_content, files=None
)
def transform_audio_transcription_response(
@ -99,9 +98,11 @@ class DeepgramAudioTranscriptionConfig(BaseAudioTranscriptionConfig):
# Add additional metadata matching OpenAI format
response["task"] = "transcribe"
response["language"] = (
"english" # Deepgram auto-detects but doesn't return language
)
# Use detected_language if available, otherwise default to "en"
detected_language = first_channel.get("detected_language")
response["language"] = detected_language if detected_language else "en"
response["duration"] = response_json["metadata"]["duration"]
# Transform words to match OpenAI format
@ -150,7 +151,6 @@ class DeepgramAudioTranscriptionConfig(BaseAudioTranscriptionConfig):
return url
def _format_param_value(self, value) -> str:
"""
Formats a parameter value for use in query string.
@ -180,7 +180,7 @@ class DeepgramAudioTranscriptionConfig(BaseAudioTranscriptionConfig):
provider_specific_params = self.get_provider_specific_params(
optional_params=optional_params,
model=model,
openai_params=self.get_supported_openai_params(model)
openai_params=self.get_supported_openai_params(model),
)
for key, value in provider_specific_params.items():

24
litellm/llms/fal_ai/__init__.py Normal file
View file

@ -0,0 +1,24 @@
from .cost_calculator import cost_calculator
from .image_generation import (
FalAIBaseConfig,
FalAIBriaConfig,
FalAIFluxProV11UltraConfig,
FalAIImageGenerationConfig,
FalAIImagen4Config,
FalAIRecraftV3Config,
FalAIStableDiffusionConfig,
get_fal_ai_image_generation_config,
)
__all__ = [
"cost_calculator",
"FalAIBaseConfig",
"FalAIImageGenerationConfig",
"FalAIImagen4Config",
"FalAIRecraftV3Config",
"FalAIBriaConfig",
"FalAIFluxProV11UltraConfig",
"FalAIStableDiffusionConfig",
"get_fal_ai_image_generation_config",
]

26
litellm/llms/fal_ai/cost_calculator.py Normal file
View file

@ -0,0 +1,26 @@
from typing import Any
import litellm
from litellm.types.utils import ImageResponse
def cost_calculator(
model: str,
image_response: Any,
) -> float:
"""
fal.ai image generation cost calculator
"""
_model_info = litellm.get_model_info(
model=model,
custom_llm_provider=litellm.LlmProviders.FAL_AI.value,
)
output_cost_per_image: float = _model_info.get("output_cost_per_image") or 0.0
num_images: int = 0
if isinstance(image_response, ImageResponse):
if image_response.data:
num_images = len(image_response.data)
return output_cost_per_image * num_images
else:
raise ValueError(f"image_response must be of type ImageResponse got type={type(image_response)}")

49
litellm/llms/fal_ai/image_generation/__init__.py Normal file
View file

@ -0,0 +1,49 @@
from litellm.llms.base_llm.image_generation.transformation import (
BaseImageGenerationConfig,
)
from .bria_transformation import FalAIBriaConfig
from .flux_pro_v11_ultra_transformation import FalAIFluxProV11UltraConfig
from .imagen4_transformation import FalAIImagen4Config
from .recraft_v3_transformation import FalAIRecraftV3Config
from .stable_diffusion_transformation import FalAIStableDiffusionConfig
from .transformation import FalAIBaseConfig, FalAIImageGenerationConfig
__all__ = [
"FalAIBaseConfig",
"FalAIImageGenerationConfig",
"FalAIImagen4Config",
"FalAIRecraftV3Config",
"FalAIBriaConfig",
"FalAIFluxProV11UltraConfig",
"FalAIStableDiffusionConfig",
]
def get_fal_ai_image_generation_config(model: str) -> BaseImageGenerationConfig:
"""
Get the appropriate Fal AI image generation configuration based on the model.
Args:
model: The Fal AI model name (e.g., "fal-ai/imagen4/preview", "fal-ai/recraft/v3/text-to-image")
Returns:
The appropriate configuration class for the specified model
"""
model_lower = model.lower()
# Map model names to their corresponding configuration classes
if "imagen4" in model_lower or "imagen-4" in model_lower:
return FalAIImagen4Config()
elif "recraft" in model_lower:
return FalAIRecraftV3Config()
elif "bria" in model_lower:
return FalAIBriaConfig()
elif "flux-pro" in model_lower and "ultra" in model_lower:
return FalAIFluxProV11UltraConfig()
elif "stable-diffusion" in model_lower:
return FalAIStableDiffusionConfig()
# Default to generic Fal AI configuration
return FalAIImageGenerationConfig()

231
litellm/llms/fal_ai/image_generation/bria_transformation.py Normal file
View file

@ -0,0 +1,231 @@
from typing import TYPE_CHECKING, Any, List, Optional
import httpx
from litellm.types.llms.openai import OpenAIImageGenerationOptionalParams
from litellm.types.utils import ImageObject, ImageResponse
from .transformation import FalAIBaseConfig
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class FalAIBriaConfig(FalAIBaseConfig):
"""
Configuration for Bria Text-to-Image 3.2 model.
Bria 3.2 is a commercial-grade text-to-image model with prompt enhancement
and multiple aspect ratio options.
Model endpoint: bria/text-to-image/3.2
Documentation: https://fal.ai/models/bria/text-to-image/3.2
"""
IMAGE_GENERATION_ENDPOINT: str = "bria/text-to-image/3.2"
def get_supported_openai_params(
self, model: str
) -> List[OpenAIImageGenerationOptionalParams]:
"""
Get supported OpenAI parameters for Bria 3.2.
"""
return [
"n",
"response_format",
"size",
]
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool,
) -> dict:
"""
Map OpenAI parameters to Bria 3.2 parameters.
Mappings:
- size -> aspect_ratio (1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9)
- response_format -> ignored (Bria returns URLs)
- n -> ignored (Bria doesn't support multiple images in one call)
"""
supported_params = self.get_supported_openai_params(model)
# Map OpenAI params to Bria params
param_mapping = {
"size": "aspect_ratio",
}
for k in non_default_params.keys():
if k not in optional_params.keys():
if k in supported_params:
# Use mapped parameter name if exists
mapped_key = param_mapping.get(k, k)
mapped_value = non_default_params[k]
# Transform specific parameters
if k == "response_format":
# Bria always returns URLs, so we can ignore this
continue
elif k == "n":
# Bria doesn't support multiple images, ignore
continue
elif k == "size":
# Map OpenAI size format to Bria aspect ratio
mapped_value = self._map_aspect_ratio(mapped_value)
optional_params[mapped_key] = mapped_value
elif drop_params:
pass
else:
raise ValueError(
f"Parameter {k} is not supported for model {model}. Supported parameters are {supported_params}. Set drop_params=True to drop unsupported parameters."
)
return optional_params
def _map_aspect_ratio(self, size: str) -> str:
"""
Map OpenAI size format to Bria aspect ratio format.
OpenAI format: "1024x1024", "1792x1024", etc.
Bria format: "1:1", "2:3", "3:2", "3:4", "4:3", "4:5", "5:4", "9:16", "16:9"
"""
# Map common OpenAI sizes to Bria aspect ratios
size_to_aspect_ratio = {
"1024x1024": "1:1",
"512x512": "1:1",
"1792x1024": "16:9",
"1024x1792": "9:16",
"1024x768": "4:3",
"768x1024": "3:4",
"1280x960": "4:3",
"960x1280": "3:4",
}
if size in size_to_aspect_ratio:
return size_to_aspect_ratio[size]
# Parse custom size format "WIDTHxHEIGHT" and calculate aspect ratio
if "x" in size:
try:
width_str, height_str = size.split("x")
width = int(width_str)
height = int(height_str)
# Calculate aspect ratio and find closest match
ratio = width / height
# Map to closest supported aspect ratio
if 0.95 <= ratio <= 1.05: # Close to 1:1
return "1:1"
elif ratio >= 1.7: # Close to 16:9
return "16:9"
elif ratio <= 0.6: # Close to 9:16
return "9:16"
elif 1.3 <= ratio <= 1.4: # Close to 4:3
return "4:3"
elif 0.7 <= ratio <= 0.8: # Close to 3:4
return "3:4"
elif 1.45 <= ratio <= 1.55: # Close to 3:2
return "3:2"
elif 0.65 <= ratio <= 0.7: # Close to 2:3
return "2:3"
elif 1.2 <= ratio <= 1.3: # Close to 5:4
return "5:4"
elif 0.75 <= ratio <= 0.85: # Close to 4:5
return "4:5"
except (ValueError, AttributeError, ZeroDivisionError):
pass
# Default to 1:1
return "1:1"
def transform_image_generation_request(
self,
model: str,
prompt: str,
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transform the image generation request to Bria 3.2 request body.
Required parameters:
- prompt: Prompt for image generation
Optional parameters:
- aspect_ratio: "1:1", "2:3", "3:2", "3:4", "4:3", "4:5", "5:4", "9:16", "16:9" (default: "1:1")
- prompt_enhancer: Improve the prompt (default: true)
- sync_mode: Return image directly in response (default: false)
- truncate_prompt: Truncate the prompt (default: true)
- guidance_scale: Guidance scale 1-10 (default: 5)
- num_inference_steps: Inference steps 20-50 (default: 30)
- seed: Random seed for reproducibility (default: 5555)
- negative_prompt: Negative prompt string
"""
bria_request_body = {
"prompt": prompt,
**optional_params,
}
return bria_request_body
def transform_image_generation_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ImageResponse,
logging_obj: LiteLLMLoggingObj,
request_data: dict,
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
"""
Transform the Bria 3.2 response to litellm ImageResponse format.
Expected response format:
{
"image": {
"url": "https://...",
"content_type": "image/png",
"file_name": "...",
"file_size": 123456,
"width": 1024,
"height": 1024
}
}
"""
try:
response_data = raw_response.json()
except Exception as e:
raise self.get_error_class(
error_message=f"Error transforming image generation response: {e}",
status_code=raw_response.status_code,
headers=raw_response.headers,
)
if not model_response.data:
model_response.data = []
# Handle Bria response format - uses "image" (singular) not "images"
image_data = response_data.get("image")
if image_data and isinstance(image_data, dict):
model_response.data.append(
ImageObject(
url=image_data.get("url", None),
b64_json=None, # Bria returns URLs only
)
)
return model_response

263
litellm/llms/fal_ai/image_generation/flux_pro_v11_ultra_transformation.py Normal file
View file

@ -0,0 +1,263 @@
from typing import TYPE_CHECKING, Any, List, Optional
import httpx
from litellm.types.llms.openai import OpenAIImageGenerationOptionalParams
from litellm.types.utils import ImageObject, ImageResponse
from .transformation import FalAIBaseConfig
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class FalAIFluxProV11UltraConfig(FalAIBaseConfig):
"""
Configuration for Fal AI Flux Pro v1.1-ultra model.
FLUX Pro v1.1-ultra is a high-quality text-to-image model with enhanced detail
and support for image prompts.
Model endpoint: fal-ai/flux-pro/v1.1-ultra
Documentation: https://fal.ai/models/fal-ai/flux-pro/v1.1-ultra
"""
IMAGE_GENERATION_ENDPOINT: str = "fal-ai/flux-pro/v1.1-ultra"
def get_supported_openai_params(
self, model: str
) -> List[OpenAIImageGenerationOptionalParams]:
"""
Get supported OpenAI parameters for Flux Pro v1.1-ultra.
"""
return [
"n",
"response_format",
"size",
]
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool,
) -> dict:
"""
Map OpenAI parameters to Flux Pro v1.1-ultra parameters.
Mappings:
- n -> num_images (1-4, default 1)
- response_format -> output_format (jpeg or png)
- size -> aspect_ratio (21:9, 16:9, 4:3, 3:2, 1:1, 2:3, 3:4, 9:16, 9:21)
"""
supported_params = self.get_supported_openai_params(model)
# Map OpenAI params to Flux Pro v1.1-ultra params
param_mapping = {
"n": "num_images",
"response_format": "output_format",
"size": "aspect_ratio",
}
for k in non_default_params.keys():
if k not in optional_params.keys():
if k in supported_params:
# Use mapped parameter name if exists
mapped_key = param_mapping.get(k, k)
mapped_value = non_default_params[k]
# Transform specific parameters
if k == "response_format":
# Map OpenAI response formats to image formats
if mapped_value in ["b64_json", "url"]:
mapped_value = "jpeg"
elif k == "size":
# Map OpenAI size format to Flux aspect ratio
mapped_value = self._map_aspect_ratio(mapped_value)
optional_params[mapped_key] = mapped_value
elif drop_params:
pass
else:
raise ValueError(
f"Parameter {k} is not supported for model {model}. Supported parameters are {supported_params}. Set drop_params=True to drop unsupported parameters."
)
return optional_params
def _map_aspect_ratio(self, size: str) -> str:
"""
Map OpenAI size format to Flux Pro aspect ratio format.
OpenAI format: "1024x1024", "1792x1024", etc.
Flux format: "21:9", "16:9", "4:3", "3:2", "1:1", "2:3", "3:4", "9:16", "9:21"
Default: "16:9"
"""
# Map common OpenAI sizes to Flux aspect ratios
size_to_aspect_ratio = {
"1024x1024": "1:1",
"512x512": "1:1",
"1792x1024": "16:9",
"1024x1792": "9:16",
"1024x768": "4:3",
"768x1024": "3:4",
"1536x1024": "3:2",
"1024x1536": "2:3",
"2048x876": "21:9",
"876x2048": "9:21",
}
if size in size_to_aspect_ratio:
return size_to_aspect_ratio[size]
# Parse custom size format "WIDTHxHEIGHT" and calculate aspect ratio
if "x" in size:
try:
width_str, height_str = size.split("x")
width = int(width_str)
height = int(height_str)
# Calculate aspect ratio and find closest match
ratio = width / height
# Map to closest supported aspect ratio
if 0.95 <= ratio <= 1.05: # Close to 1:1
return "1:1"
elif ratio >= 2.3: # Close to 21:9
return "21:9"
elif 1.7 <= ratio < 2.3: # Close to 16:9
return "16:9"
elif 1.3 <= ratio < 1.7: # Close to 4:3
return "4:3"
elif 1.4 <= ratio < 1.6: # Close to 3:2
return "3:2"
elif 0.6 <= ratio < 0.7: # Close to 3:4
return "3:4"
elif 0.65 <= ratio < 0.75: # Close to 2:3
return "2:3"
elif 0.5 <= ratio < 0.6: # Close to 9:16
return "9:16"
elif ratio < 0.5: # Close to 9:21
return "9:21"
except (ValueError, AttributeError, ZeroDivisionError):
pass
# Default to 16:9
return "16:9"
def transform_image_generation_request(
self,
model: str,
prompt: str,
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transform the image generation request to Flux Pro v1.1-ultra request body.
Required parameters:
- prompt: The prompt to generate an image from
Optional parameters:
- num_images: Number of images (1-4, default: 1)
- aspect_ratio: Aspect ratio (default: "16:9")
- raw: Generate less processed images (default: false)
- output_format: "jpeg" or "png" (default: "jpeg")
- image_url: Image URL for image-to-image generation
- sync_mode: Return data URI (default: false)
- safety_tolerance: Safety level "1"-"6" (default: "2")
- enable_safety_checker: Enable safety checker (default: true)
- seed: Random seed for reproducibility
- image_prompt_strength: Strength of image prompt 0-1 (default: 0.1)
- enhance_prompt: Enhance prompt for better results (default: false)
"""
flux_pro_request_body = {
"prompt": prompt,
**optional_params,
}
return flux_pro_request_body
def transform_image_generation_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ImageResponse,
logging_obj: LiteLLMLoggingObj,
request_data: dict,
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
"""
Transform the Flux Pro v1.1-ultra response to litellm ImageResponse format.
Expected response format:
{
"images": [
{
"url": "https://...",
"width": 1024,
"height": 768,
"content_type": "image/jpeg"
}
],
"timings": {"inference": 2.5, ...},
"seed": 42,
"has_nsfw_concepts": [false],
"prompt": "original prompt"
}
"""
try:
response_data = raw_response.json()
except Exception as e:
raise self.get_error_class(
error_message=f"Error transforming image generation response: {e}",
status_code=raw_response.status_code,
headers=raw_response.headers,
)
if not model_response.data:
model_response.data = []
# Handle Flux Pro v1.1-ultra response format
images = response_data.get("images", [])
if isinstance(images, list):
for image_data in images:
if isinstance(image_data, dict):
model_response.data.append(
ImageObject(
url=image_data.get("url", None),
b64_json=None, # Flux Pro returns URLs only
)
)
elif isinstance(image_data, str):
# If images is just a list of URLs
model_response.data.append(
ImageObject(
url=image_data,
b64_json=None,
)
)
# Add additional metadata from Flux Pro response
if hasattr(model_response, "_hidden_params"):
if "seed" in response_data:
model_response._hidden_params["seed"] = response_data["seed"]
if "timings" in response_data:
model_response._hidden_params["timings"] = response_data["timings"]
if "has_nsfw_concepts" in response_data:
model_response._hidden_params["has_nsfw_concepts"] = response_data[
"has_nsfw_concepts"
]
return model_response

242
litellm/llms/fal_ai/image_generation/imagen4_transformation.py Normal file
View file

@ -0,0 +1,242 @@
from typing import TYPE_CHECKING, Any, List, Optional
import httpx
from litellm.types.llms.openai import OpenAIImageGenerationOptionalParams
from litellm.types.utils import ImageObject, ImageResponse
from .transformation import FalAIBaseConfig
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class FalAIImagen4Config(FalAIBaseConfig):
"""
Configuration for Fal AI Imagen4 model.
Google's highest quality image generation model available through Fal AI.
Model variants:
- fal-ai/imagen4/preview (Standard): $0.05 per image
- fal-ai/imagen4/preview/fast (Fast): $0.04 per image
- fal-ai/imagen4/preview/ultra (Ultra): $0.06 per image
Documentation: https://fal.ai/models/fal-ai/imagen4/preview
"""
IMAGE_GENERATION_ENDPOINT: str = "fal-ai/imagen4/preview"
def get_supported_openai_params(
self, model: str
) -> List[OpenAIImageGenerationOptionalParams]:
"""
Get supported OpenAI parameters for Imagen4.
"""
return [
"n",
"response_format",
"size",
]
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool,
) -> dict:
"""
Map OpenAI parameters to Imagen4 parameters.
Mappings:
- n -> num_images (1-4, default 1)
- size -> aspect_ratio (1:1, 16:9, 9:16, 3:4, 4:3)
- response_format -> ignored (Imagen4 returns URLs)
"""
supported_params = self.get_supported_openai_params(model)
# Map OpenAI params to Imagen4 params
param_mapping = {
"n": "num_images",
"size": "aspect_ratio",
}
for k in non_default_params.keys():
if k not in optional_params.keys():
if k in supported_params:
# Use mapped parameter name if exists
mapped_key = param_mapping.get(k, k)
mapped_value = non_default_params[k]
# Transform specific parameters
if k == "response_format":
# Imagen4 always returns URLs, so we can ignore this
continue
elif k == "size":
# Map OpenAI size format to Imagen4 aspect ratio
mapped_value = self._map_aspect_ratio(mapped_value)
optional_params[mapped_key] = mapped_value
elif drop_params:
pass
else:
raise ValueError(
f"Parameter {k} is not supported for model {model}. Supported parameters are {supported_params}. Set drop_params=True to drop unsupported parameters."
)
return optional_params
def _map_aspect_ratio(self, size: str) -> str:
"""
Map OpenAI size format to Imagen4 aspect ratio format.
OpenAI format: "1024x1024", "1792x1024", etc.
Imagen4 format: "1:1", "16:9", "9:16", "3:4", "4:3"
Available aspect ratios:
- 1:1 (default)
- 16:9
- 9:16
- 3:4
- 4:3
"""
# Map common OpenAI sizes to Imagen4 aspect ratios
size_to_aspect_ratio = {
"1024x1024": "1:1",
"512x512": "1:1",
"1792x1024": "16:9",
"1024x1792": "9:16",
"1024x768": "4:3",
"768x1024": "3:4",
}
if size in size_to_aspect_ratio:
return size_to_aspect_ratio[size]
# Parse custom size format "WIDTHxHEIGHT" and calculate aspect ratio
if "x" in size:
try:
width_str, height_str = size.split("x")
width = int(width_str)
height = int(height_str)
# Calculate aspect ratio and find closest match
ratio = width / height
# Map to closest supported aspect ratio
if 0.95 <= ratio <= 1.05: # Close to 1:1
return "1:1"
elif ratio >= 1.7: # Close to 16:9
return "16:9"
elif ratio <= 0.6: # Close to 9:16
return "9:16"
elif ratio >= 1.2: # Close to 4:3
return "4:3"
elif ratio <= 0.8: # Close to 3:4
return "3:4"
except (ValueError, AttributeError, ZeroDivisionError):
pass
# Default to 1:1
return "1:1"
def transform_image_generation_request(
self,
model: str,
prompt: str,
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transform the image generation request to Imagen4 request body.
Required parameters:
- prompt: The text prompt describing what you want to see
Optional parameters:
- aspect_ratio: "1:1", "16:9", "9:16", "3:4", "4:3" (default: "1:1")
- num_images: Number of images (1-4, default: 1)
- resolution: "1K" or "2K" (default: "1K")
- seed: Random seed for reproducibility
- negative_prompt: Description of what to discourage (default: "")
"""
imagen4_request_body = {
"prompt": prompt,
**optional_params,
}
return imagen4_request_body
def transform_image_generation_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ImageResponse,
logging_obj: LiteLLMLoggingObj,
request_data: dict,
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
"""
Transform the Imagen4 response to litellm ImageResponse format.
Expected response format:
{
"images": [
{
"url": "https://...",
"content_type": "image/png",
"file_name": "z9RV14K95DvU.png",
"file_size": 4404019
}
],
"seed": 42
}
"""
try:
response_data = raw_response.json()
except Exception as e:
raise self.get_error_class(
error_message=f"Error transforming image generation response: {e}",
status_code=raw_response.status_code,
headers=raw_response.headers,
)
if not model_response.data:
model_response.data = []
# Handle Imagen4 response format
images = response_data.get("images", [])
if isinstance(images, list):
for image_data in images:
if isinstance(image_data, dict):
model_response.data.append(
ImageObject(
url=image_data.get("url", None),
b64_json=None, # Imagen4 returns URLs only
)
)
elif isinstance(image_data, str):
# If images is just a list of URLs
model_response.data.append(
ImageObject(
url=image_data,
b64_json=None,
)
)
# Add seed metadata from Imagen4 response
if hasattr(model_response, "_hidden_params"):
if "seed" in response_data:
model_response._hidden_params["seed"] = response_data["seed"]
return model_response

226
litellm/llms/fal_ai/image_generation/recraft_v3_transformation.py Normal file
View file

@ -0,0 +1,226 @@
from typing import TYPE_CHECKING, Any, List, Optional
import httpx
from litellm.types.llms.openai import OpenAIImageGenerationOptionalParams
from litellm.types.utils import ImageObject, ImageResponse
from .transformation import FalAIBaseConfig
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class FalAIRecraftV3Config(FalAIBaseConfig):
"""
Configuration for Fal AI Recraft v3 Text-to-Image model.
Recraft v3 is a text-to-image model with multiple style options including
realistic images, digital illustrations, and vector illustrations.
Model endpoint: fal-ai/recraft/v3/text-to-image
Documentation: https://fal.ai/models/fal-ai/recraft/v3/text-to-image
"""
IMAGE_GENERATION_ENDPOINT: str = "fal-ai/recraft/v3/text-to-image"
def get_supported_openai_params(
self, model: str
) -> List[OpenAIImageGenerationOptionalParams]:
"""
Get supported OpenAI parameters for Recraft v3.
"""
return [
"n",
"response_format",
"size",
]
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool,
) -> dict:
"""
Map OpenAI parameters to Recraft v3 parameters.
Mappings:
- size -> image_size (can be preset or custom width/height)
- response_format -> ignored (Recraft returns URLs)
- n -> ignored (Recraft doesn't support multiple images)
"""
supported_params = self.get_supported_openai_params(model)
# Map OpenAI params to Recraft v3 params
param_mapping = {
"size": "image_size",
}
for k in non_default_params.keys():
if k not in optional_params.keys():
if k in supported_params:
# Use mapped parameter name if exists
mapped_key = param_mapping.get(k, k)
mapped_value = non_default_params[k]
# Transform specific parameters
if k == "response_format":
# Recraft always returns URLs, so we can ignore this
continue
elif k == "n":
# Recraft doesn't support multiple images, ignore
continue
elif k == "size":
# Map OpenAI size format to Recraft image_size
mapped_value = self._map_image_size(mapped_value)
optional_params[mapped_key] = mapped_value
elif drop_params:
pass
else:
raise ValueError(
f"Parameter {k} is not supported for model {model}. Supported parameters are {supported_params}. Set drop_params=True to drop unsupported parameters."
)
return optional_params
def _map_image_size(self, size: str) -> Any:
"""
Map OpenAI size format to Recraft v3 image_size format.
OpenAI format: "1024x1024", "1792x1024", etc.
Recraft format: Can be preset strings or {"width": int, "height": int}
Available presets:
- square_hd (default)
- square
- portrait_4_3
- portrait_16_9
- landscape_4_3
- landscape_16_9
"""
# Map common OpenAI sizes to Recraft presets
size_mapping = {
"1024x1024": "square_hd",
"512x512": "square",
"768x1024": "portrait_4_3",
"576x1024": "portrait_16_9",
"1024x768": "landscape_4_3",
"1024x576": "landscape_16_9",
}
if size in size_mapping:
return size_mapping[size]
# Parse custom size format "WIDTHxHEIGHT"
if "x" in size:
try:
width, height = size.split("x")
return {
"width": int(width),
"height": int(height),
}
except (ValueError, AttributeError):
pass
# Default to square_hd
return "square_hd"
def transform_image_generation_request(
self,
model: str,
prompt: str,
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transform the image generation request to Recraft v3 request body.
Required parameters:
- prompt: Text prompt (max 1000 characters)
Optional parameters:
- image_size: Preset or {"width": int, "height": int} (default: "square_hd")
- style: Style preset (default: "realistic_image")
Options: "any", "realistic_image", "digital_illustration", "vector_illustration", etc.
- colors: Array of RGB color objects [{"r": 0-255, "g": 0-255, "b": 0-255}]
- enable_safety_checker: Enable safety checker (default: false)
- style_id: UUID for custom style reference
Note: Vector illustrations cost 2X as much.
"""
recraft_request_body = {
"prompt": prompt,
**optional_params,
}
return recraft_request_body
def transform_image_generation_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ImageResponse,
logging_obj: LiteLLMLoggingObj,
request_data: dict,
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
"""
Transform the Recraft v3 response to litellm ImageResponse format.
Expected response format:
{
"images": [
{
"url": "https://...",
"content_type": "image/webp",
"file_name": "...",
"file_size": 123456
}
]
}
"""
try:
response_data = raw_response.json()
except Exception as e:
raise self.get_error_class(
error_message=f"Error transforming image generation response: {e}",
status_code=raw_response.status_code,
headers=raw_response.headers,
)
if not model_response.data:
model_response.data = []
# Handle Recraft v3 response format
images = response_data.get("images", [])
if isinstance(images, list):
for image_data in images:
if isinstance(image_data, dict):
model_response.data.append(
ImageObject(
url=image_data.get("url", None),
b64_json=None, # Recraft returns URLs only
)
)
elif isinstance(image_data, str):
# If images is just a list of URLs
model_response.data.append(
ImageObject(
url=image_data,
b64_json=None,
)
)
return model_response

281
litellm/llms/fal_ai/image_generation/stable_diffusion_transformation.py Normal file
View file

@ -0,0 +1,281 @@
from typing import TYPE_CHECKING, Any, List, Optional
import httpx
from litellm.types.llms.openai import OpenAIImageGenerationOptionalParams
from litellm.types.utils import ImageObject, ImageResponse
from .transformation import FalAIBaseConfig
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class FalAIStableDiffusionConfig(FalAIBaseConfig):
"""
Configuration for Fal AI Stable Diffusion models.
Supports Stable Diffusion v3.5 variants and other Stable Diffusion models on Fal AI.
Example models:
- fal-ai/stable-diffusion-v35-medium
- fal-ai/stable-diffusion-v35-large
Documentation: https://fal.ai/models/fal-ai/stable-diffusion-v35-medium
"""
IMAGE_GENERATION_ENDPOINT: str = "" # Will be set from model name
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
"""
Get the complete url for the request.
For Stable Diffusion models, extract the endpoint from the model name.
"""
from litellm.secret_managers.main import get_secret_str
complete_url: str = (
api_base
or get_secret_str("FAL_AI_API_BASE")
or self.DEFAULT_BASE_URL
)
complete_url = complete_url.rstrip("/")
# Extract endpoint from model name
# e.g., "fal-ai/stable-diffusion-v35-medium" or "stable-diffusion-v35-medium"
endpoint = model
if "/" in model and not model.startswith("fal-ai/"):
# If model is like "custom/stable-diffusion-v35-medium", use full path
endpoint = model
elif not model.startswith("fal-ai/"):
# If model is just "stable-diffusion-v35-medium", prepend fal-ai
endpoint = f"fal-ai/{model}"
complete_url = f"{complete_url}/{endpoint}"
return complete_url
def get_supported_openai_params(
self, model: str
) -> List[OpenAIImageGenerationOptionalParams]:
"""
Get supported OpenAI parameters for Stable Diffusion models.
"""
return [
"n",
"response_format",
"size",
]
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool,
) -> dict:
"""
Map OpenAI parameters to Stable Diffusion parameters.
Mappings:
- n -> num_images (1-4, default 1)
- response_format -> output_format (jpeg or png)
- size -> image_size (can be preset or custom width/height)
"""
supported_params = self.get_supported_openai_params(model)
# Map OpenAI params to Stable Diffusion params
param_mapping = {
"n": "num_images",
"response_format": "output_format",
"size": "image_size",
}
for k in non_default_params.keys():
if k not in optional_params.keys():
if k in supported_params:
# Use mapped parameter name if exists
mapped_key = param_mapping.get(k, k)
mapped_value = non_default_params[k]
# Transform specific parameters
if k == "response_format":
# Map OpenAI response formats to image formats
if mapped_value in ["b64_json", "url"]:
mapped_value = "jpeg"
elif k == "size":
# Map OpenAI size format to Stable Diffusion image_size
mapped_value = self._map_image_size(mapped_value)
optional_params[mapped_key] = mapped_value
elif drop_params:
pass
else:
raise ValueError(
f"Parameter {k} is not supported for model {model}. Supported parameters are {supported_params}. Set drop_params=True to drop unsupported parameters."
)
return optional_params
def _map_image_size(self, size: str) -> Any:
"""
Map OpenAI size format to Stable Diffusion image_size format.
OpenAI format: "1024x1024", "1792x1024", etc.
Stable Diffusion format: Can be preset strings or {"width": int, "height": int}
Available presets:
- square_hd
- square
- portrait_4_3
- portrait_16_9
- landscape_4_3 (default)
- landscape_16_9
"""
# Map common OpenAI sizes to Stable Diffusion presets
size_mapping = {
"1024x1024": "square_hd",
"512x512": "square",
"768x1024": "portrait_4_3",
"576x1024": "portrait_16_9",
"1024x768": "landscape_4_3",
"1024x576": "landscape_16_9",
}
if size in size_mapping:
return size_mapping[size]
# Parse custom size format "WIDTHxHEIGHT"
if "x" in size:
try:
width, height = size.split("x")
return {
"width": int(width),
"height": int(height),
}
except (ValueError, AttributeError):
pass
# Default to landscape_4_3
return "landscape_4_3"
def transform_image_generation_request(
self,
model: str,
prompt: str,
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transform the image generation request to Stable Diffusion request body.
Required parameters:
- prompt: The prompt to generate an image from
Optional parameters:
- num_images: Number of images (1-4, default: 1)
- image_size: Size preset or {"width": int, "height": int} (default: landscape_4_3)
- output_format: "jpeg" or "png" (default: jpeg)
- sync_mode: Wait for image upload before returning (default: false)
- guidance_scale: CFG scale 0-20 (default: 4.5)
- num_inference_steps: Inference steps 1-50 (default: 40)
- seed: Random seed for reproducibility
- negative_prompt: Negative prompt string (default: "")
- enable_safety_checker: Enable safety checker (default: true)
"""
stable_diffusion_request_body = {
"prompt": prompt,
**optional_params,
}
return stable_diffusion_request_body
def transform_image_generation_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ImageResponse,
logging_obj: LiteLLMLoggingObj,
request_data: dict,
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
"""
Transform the Stable Diffusion response to litellm ImageResponse format.
Expected response format:
{
"images": [
{
"url": "https://...",
"width": 1024,
"height": 768,
"content_type": "image/jpeg"
}
],
"timings": {"inference": 2.5, ...},
"seed": 42,
"has_nsfw_concepts": [false],
"prompt": "original prompt"
}
"""
try:
response_data = raw_response.json()
except Exception as e:
raise self.get_error_class(
error_message=f"Error transforming image generation response: {e}",
status_code=raw_response.status_code,
headers=raw_response.headers,
)
if not model_response.data:
model_response.data = []
# Handle Stable Diffusion response format
images = response_data.get("images", [])
if isinstance(images, list):
for image_data in images:
if isinstance(image_data, dict):
model_response.data.append(
ImageObject(
url=image_data.get("url", None),
b64_json=None, # Stable Diffusion returns URLs only
)
)
elif isinstance(image_data, str):
# If images is just a list of URLs
model_response.data.append(
ImageObject(
url=image_data,
b64_json=None,
)
)
# Add additional metadata from Stable Diffusion response
if hasattr(model_response, "_hidden_params"):
if "seed" in response_data:
model_response._hidden_params["seed"] = response_data["seed"]
if "timings" in response_data:
model_response._hidden_params["timings"] = response_data["timings"]
if "has_nsfw_concepts" in response_data:
model_response._hidden_params["has_nsfw_concepts"] = response_data[
"has_nsfw_concepts"
]
return model_response

176
litellm/llms/fal_ai/image_generation/transformation.py Normal file
View file

@ -0,0 +1,176 @@
from typing import TYPE_CHECKING, Any, List, Optional
import httpx
from litellm.llms.base_llm.image_generation.transformation import (
BaseImageGenerationConfig,
)
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import (
AllMessageValues,
OpenAIImageGenerationOptionalParams,
)
from litellm.types.utils import ImageObject, ImageResponse
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
LiteLLMLoggingObj = _LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class FalAIBaseConfig(BaseImageGenerationConfig):
"""
Base configuration for Fal AI image generation models.
Handles common functionality like URL construction and authentication.
"""
DEFAULT_BASE_URL: str = "https://fal.run"
IMAGE_GENERATION_ENDPOINT: str = ""
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
"""
Get the complete url for the request
Some providers need `model` in `api_base`
"""
complete_url: str = (
api_base
or get_secret_str("FAL_AI_API_BASE")
or self.DEFAULT_BASE_URL
)
complete_url = complete_url.rstrip("/")
if self.IMAGE_GENERATION_ENDPOINT:
complete_url = f"{complete_url}/{self.IMAGE_GENERATION_ENDPOINT}"
return complete_url
def validate_environment(
self,
headers: dict,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
final_api_key: Optional[str] = (
api_key or
get_secret_str("FAL_AI_API_KEY")
)
if not final_api_key:
raise ValueError("FAL_AI_API_KEY is not set")
headers["Authorization"] = f"Key {final_api_key}"
return headers
def transform_image_generation_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ImageResponse,
logging_obj: LiteLLMLoggingObj,
request_data: dict,
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ImageResponse:
"""
Transform the image generation response to the litellm image response
"""
try:
response_data = raw_response.json()
except Exception as e:
raise self.get_error_class(
error_message=f"Error transforming image generation response: {e}",
status_code=raw_response.status_code,
headers=raw_response.headers,
)
if not model_response.data:
model_response.data = []
# Handle fal.ai response format
images = response_data.get("images", [])
if isinstance(images, list):
for image_data in images:
if isinstance(image_data, dict):
model_response.data.append(ImageObject(
url=image_data.get("url", None),
b64_json=image_data.get("b64_json", None),
))
elif isinstance(image_data, str):
# If images is just a list of URLs
model_response.data.append(ImageObject(
url=image_data,
b64_json=None,
))
return model_response
class FalAIImageGenerationConfig(FalAIBaseConfig):
"""
Default Fal AI image generation configuration for generic models.
"""
def get_supported_openai_params(
self, model: str
) -> List[OpenAIImageGenerationOptionalParams]:
"""
Get supported OpenAI parameters for fal.ai image generation
"""
return [
"n",
"response_format",
"size",
]
def map_openai_params(
self,
non_default_params: dict,
optional_params: dict,
model: str,
drop_params: bool,
) -> dict:
supported_params = self.get_supported_openai_params(model)
for k in non_default_params.keys():
if k not in optional_params.keys():
if k in supported_params:
optional_params[k] = non_default_params[k]
elif drop_params:
pass
else:
raise ValueError(
f"Parameter {k} is not supported for model {model}. Supported parameters are {supported_params}. Set drop_params=True to drop unsupported parameters."
)
return optional_params
def transform_image_generation_request(
self,
model: str,
prompt: str,
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transform the image generation request to the fal.ai image generation request body
"""
fal_ai_image_generation_request_body = {
"prompt": prompt,
**optional_params,
}
return fal_ai_image_generation_request_body

328
litellm/llms/oci/chat/transformation.py
View file

@ -2,7 +2,8 @@ import base64
import datetime
import hashlib
import json
from typing import TYPE_CHECKING, Any, AsyncIterator, Dict, List, Optional, Tuple, Union
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, AsyncIterator, Dict, List, Optional, Protocol, Tuple, Union
from urllib.parse import urlparse
import httpx
@ -62,6 +63,47 @@ else:
LiteLLMLoggingObj = Any
class OCISignerProtocol(Protocol):
"""
Protocol for OCI request signers (e.g., oci.signer.Signer).
This protocol defines the interface expected for OCI SDK signer objects.
Compatible with the OCI Python SDK's Signer class.
See: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html
"""
def do_request_sign(self, request: Any, *, enforce_content_headers: bool = False) -> None:
"""
Sign an HTTP request by adding authentication headers.
Args:
request: Request object with method, url, headers, body, and path_url attributes
enforce_content_headers: Whether to enforce content-type and content-length headers
"""
...
@dataclass
class OCIRequestWrapper:
"""
Wrapper for HTTP requests compatible with OCI signer interface.
This class wraps request data in a format compatible with OCI SDK signers,
which expect objects with method, url, headers, body, and path_url attributes.
"""
method: str
url: str
headers: dict
body: bytes
@property
def path_url(self) -> str:
"""Returns the path + query string for OCI signing."""
parsed_url = urlparse(self.url)
return parsed_url.path + ("?" + parsed_url.query if parsed_url.query else "")
def sha256_base64(data: bytes) -> str:
digest = hashlib.sha256(data).digest()
return base64.b64encode(digest).decode()
@ -228,29 +270,89 @@ class OCIChatConfig(BaseConfig):
return adapted_params
def sign_request(
def _sign_with_oci_signer(
self,
headers: dict,
optional_params: dict,
request_data: dict,
api_base: str,
api_key: Optional[str] = None,
model: Optional[str] = None,
stream: Optional[bool] = None,
fake_stream: Optional[bool] = None,
) -> Tuple[dict, Optional[bytes]]:
) -> Tuple[dict, bytes]:
"""
Some providers like Bedrock require signing the request. The sign request funtion needs access to `request_data` and `complete_url`
Args:
headers: dict
optional_params: dict
request_data: dict - the request body being sent in http request
api_base: str - the complete url being sent in http request
Returns:
dict - the signed headers
"""
import json
Sign request using OCI SDK Signer object.
Args:
headers: Request headers to be signed
optional_params: Optional parameters including oci_signer
request_data: The request body dict to be sent in HTTP request
api_base: The complete URL for the HTTP request
Returns:
Tuple of (signed_headers, encoded_body)
Raises:
OCIError: If signing fails
ValueError: If HTTP method is unsupported
"""
oci_signer = optional_params.get("oci_signer")
body = json.dumps(request_data).encode("utf-8")
method = str(optional_params.get("method", "POST")).upper()
if method not in ["POST", "GET", "PUT", "DELETE", "PATCH"]:
raise ValueError(f"Unsupported HTTP method: {method}")
prepared_headers = headers.copy()
prepared_headers.setdefault("content-type", "application/json")
prepared_headers.setdefault("content-length", str(len(body)))
request_wrapper = OCIRequestWrapper(
method=method,
url=api_base,
headers=prepared_headers,
body=body
)
if oci_signer is None:
raise ValueError("oci_signer cannot be None when calling _sign_with_oci_signer")
try:
oci_signer.do_request_sign(request_wrapper, enforce_content_headers=True)
except Exception as e:
raise OCIError(
status_code=500,
message=(
f"Failed to sign request with provided oci_signer: {str(e)}. "
"The signer must implement the OCI SDK Signer interface with a "
"do_request_sign(request, enforce_content_headers=True) method. "
"See: https://docs.oracle.com/en-us/iaas/tools/python/latest/api/signing.html"
)
) from e
headers.update(request_wrapper.headers)
return headers, body
def _sign_with_manual_credentials(
self,
headers: dict,
optional_params: dict,
request_data: dict,
api_base: str,
) -> Tuple[dict, None]:
"""
Sign request using manual OCI credentials.
Args:
headers: Request headers to be signed
optional_params: Optional parameters including OCI credentials
request_data: The request body dict to be sent in HTTP request
api_base: The complete URL for the HTTP request
Returns:
Tuple of (signed_headers, None)
Raises:
Exception: If required credentials are missing
ImportError: If cryptography package is not installed
"""
oci_region = optional_params.get("oci_region", "us-ashburn-1")
api_base = (
api_base
@ -355,6 +457,69 @@ class OCIChatConfig(BaseConfig):
return headers, None
def sign_request(
self,
headers: dict,
optional_params: dict,
request_data: dict,
api_base: str,
api_key: Optional[str] = None,
model: Optional[str] = None,
stream: Optional[bool] = None,
fake_stream: Optional[bool] = None,
) -> Tuple[dict, Optional[bytes]]:
"""
Sign the OCI request by adding authentication headers.
Supports two signing modes:
1. OCI SDK Signer: Use an oci_signer object to sign the request
2. Manual Signing: Use OCI credentials to manually sign the request
Args:
headers: Request headers to be signed
optional_params: Optional parameters including auth credentials or oci_signer
request_data: The request body dict to be sent in HTTP request
api_base: The complete URL for the HTTP request
api_key: Optional API key (not used for OCI)
model: Optional model name
stream: Optional streaming flag
fake_stream: Optional fake streaming flag
Returns:
Tuple of (signed_headers, encoded_body):
- If oci_signer is provided: Returns (headers, body) where body is the encoded JSON
- If manual credentials are provided: Returns (headers, None) as body is not returned
for the manual signing path
Raises:
OCIError: If signing fails with oci_signer
Exception: If required credentials are missing
ImportError: If cryptography package is not installed (manual signing only)
Example:
>>> from oci.signer import Signer
>>> signer = Signer(
... tenancy="ocid1.tenancy.oc1..",
... user="ocid1.user.oc1..",
... fingerprint="xx:xx:xx",
... private_key_file_location="~/.oci/key.pem"
... )
>>> headers, body = config.sign_request(
... headers={},
... optional_params={"oci_signer": signer},
... request_data={"message": "Hello"},
... api_base="https://inference.generativeai.us-ashburn-1.oci.oraclecloud.com/..."
... )
"""
oci_signer = optional_params.get("oci_signer")
# If a signer is provided, use it for request signing
if oci_signer is not None:
return self._sign_with_oci_signer(headers, optional_params, request_data, api_base)
# Standard manual credential signing
return self._sign_with_manual_credentials(headers, optional_params, request_data, api_base)
def validate_environment(
self,
headers: dict,
@ -365,36 +530,67 @@ class OCIChatConfig(BaseConfig):
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
"""
Validate the OCI environment and credentials.
Supports two authentication modes:
1. OCI SDK Signer: Pass an oci_signer object (e.g., oci.signer.Signer)
2. Manual Credentials: Pass oci_user, oci_fingerprint, oci_tenancy, and oci_key/oci_key_file
Args:
headers: Request headers to populate
model: Model name
messages: List of chat messages
optional_params: Optional parameters including authentication credentials
litellm_params: LiteLLM parameters
api_key: Optional API key (not used for OCI)
api_base: Optional API base URL
Returns:
Updated headers dict
Raises:
Exception: If required parameters are missing or invalid
"""
oci_signer = optional_params.get("oci_signer")
oci_region = optional_params.get("oci_region", "us-ashburn-1")
# Determine api_base
api_base = (
api_base
or litellm.api_base
or f"https://inference.generativeai.{oci_region}.oci.oraclecloud.com"
)
oci_user = optional_params.get("oci_user")
oci_fingerprint = optional_params.get("oci_fingerprint")
oci_tenancy = optional_params.get("oci_tenancy")
oci_key = optional_params.get("oci_key")
oci_key_file = optional_params.get("oci_key_file")
oci_compartment_id = optional_params.get("oci_compartment_id")
if (
not oci_user
or not oci_fingerprint
or not oci_tenancy
or not (oci_key or oci_key_file)
or not oci_compartment_id
):
raise Exception(
"Missing required parameters: oci_user, oci_fingerprint, oci_tenancy, oci_compartment_id "
"and at least one of oci_key or oci_key_file."
)
if not api_base:
raise Exception(
"Either `api_base` must be provided or `litellm.api_base` must be set. Alternatively, you can set the `oci_region` optional parameter to use the default OCI region."
"Either `api_base` must be provided or `litellm.api_base` must be set. "
"Alternatively, you can set the `oci_region` optional parameter to use the default OCI region."
)
# Validate credentials only if signer is not provided
if oci_signer is None:
oci_user = optional_params.get("oci_user")
oci_fingerprint = optional_params.get("oci_fingerprint")
oci_tenancy = optional_params.get("oci_tenancy")
oci_key = optional_params.get("oci_key")
oci_key_file = optional_params.get("oci_key_file")
oci_compartment_id = optional_params.get("oci_compartment_id")
if (
not oci_user
or not oci_fingerprint
or not oci_tenancy
or not (oci_key or oci_key_file)
or not oci_compartment_id
):
raise Exception(
"Missing required parameters: oci_user, oci_fingerprint, oci_tenancy, oci_compartment_id "
"and at least one of oci_key or oci_key_file. "
"Alternatively, provide an oci_signer object from the OCI SDK."
)
# Common header setup
headers.update(
{
"content-type": "application/json",
@ -442,12 +638,12 @@ class OCIChatConfig(BaseConfig):
for openai_key, oci_key in open_ai_to_oci_param_map.items():
if oci_key and openai_key in optional_params:
selected_params[oci_key] = optional_params[openai_key] # type: ignore[index]
# Also check for already-mapped OCI params (for backward compatibility)
for oci_value in open_ai_to_oci_param_map.values():
if oci_value and oci_value in optional_params and oci_value not in selected_params:
selected_params[oci_value] = optional_params[oci_value] # type: ignore[index]
if "tools" in selected_params:
if vendor == OCIVendors.COHERE:
selected_params["tools"] = self.adapt_tool_definitions_to_cohere_standard( # type: ignore[assignment]
@ -465,7 +661,7 @@ class OCIChatConfig(BaseConfig):
for msg in messages[:-1]: # All messages except the last one
role = msg.get("role")
content = msg.get("content")
if isinstance(content, list):
# Extract text from content array
text_content = ""
@ -473,11 +669,11 @@ class OCIChatConfig(BaseConfig):
if isinstance(content_item, dict) and content_item.get("type") == "text":
text_content += content_item.get("text", "")
content = text_content
# Ensure content is a string
if not isinstance(content, str):
content = str(content) if content is not None else ""
# Handle tool calls
tool_calls: Optional[List[CohereToolCall]] = None
if role == "assistant" and "tool_calls" in msg and msg.get("tool_calls"): # type: ignore[union-attr,typeddict-item]
@ -492,12 +688,12 @@ class OCIChatConfig(BaseConfig):
arguments = {}
else:
arguments = raw_arguments
tool_calls.append(CohereToolCall(
name=str(tool_call.get("function", {}).get("name", "")),
parameters=arguments
))
if role == "user":
chat_history.append(CohereMessage(role="USER", message=content))
elif role == "assistant":
@ -505,11 +701,11 @@ class OCIChatConfig(BaseConfig):
elif role == "tool":
# Tool messages need special handling
chat_history.append(CohereMessage(
role="TOOL",
role="TOOL",
message=content,
toolCalls=None # Tool messages don't have tool calls
))
return chat_history
def adapt_tool_definitions_to_cohere_standard(self, tools: List[Dict[str, Any]]) -> List[CohereTool]:
@ -519,7 +715,7 @@ class OCIChatConfig(BaseConfig):
function_def = tool.get("function", {})
parameters = function_def.get("parameters", {}).get("properties", {})
required = function_def.get("parameters", {}).get("required", [])
parameter_definitions = {}
for param_name, param_schema in parameters.items():
parameter_definitions[param_name] = CohereParameterDefinition(
@ -527,13 +723,13 @@ class OCIChatConfig(BaseConfig):
type=param_schema.get("type", "string"),
isRequired=param_name in required
)
cohere_tools.append(CohereTool(
name=function_def.get("name", ""),
description=function_def.get("description", ""),
parameterDefinitions=parameter_definitions
))
return cohere_tools
def _extract_text_content(self, content: Any) -> str:
@ -586,7 +782,7 @@ class OCIChatConfig(BaseConfig):
user_messages = [msg for msg in messages if msg.get("role") == "user"]
if not user_messages:
raise Exception("No user message found for Cohere model")
# Create Cohere-specific chat request
chat_request = CohereChatRequest(
@ -595,7 +791,7 @@ class OCIChatConfig(BaseConfig):
chatHistory=self.adapt_messages_to_cohere_standard(messages),
**self._get_optional_params(OCIVendors.COHERE, optional_params)
)
data = OCICompletionPayload(
compartmentId=oci_compartment_id,
servingMode=servingMode,
@ -616,24 +812,24 @@ class OCIChatConfig(BaseConfig):
return data.model_dump(exclude_none=True)
def _handle_cohere_response(
self,
json_response: dict,
model: str,
self,
json_response: dict,
model: str,
model_response: ModelResponse
) -> ModelResponse:
"""Handle Cohere-specific response format."""
cohere_response = CohereChatResult(**json_response)
# Cohere response format (uses camelCase)
model_id = model
# Set basic response info
model_response.model = model_id
model_response.created = int(datetime.datetime.now().timestamp())
# Extract the response text
response_text = cohere_response.chatResponse.text
oci_finish_reason = cohere_response.chatResponse.finishReason
# Map finish reason
if oci_finish_reason == "COMPLETE":
finish_reason = "stop"
@ -641,7 +837,7 @@ class OCIChatConfig(BaseConfig):
finish_reason = "length"
else:
finish_reason = "stop"
# Handle tool calls
tool_calls: Optional[List[Dict[str, Any]]] = None
if cohere_response.chatResponse.toolCalls:
@ -655,7 +851,7 @@ class OCIChatConfig(BaseConfig):
"arguments": json.dumps(tool_call.parameters)
}
})
# Create choice
from litellm.types.utils import Choices
choice = Choices(
@ -668,7 +864,7 @@ class OCIChatConfig(BaseConfig):
finish_reason=finish_reason
)
model_response.choices = [choice]
# Extract usage info
usage_info = cohere_response.chatResponse.usage
from litellm.types.utils import Usage
@ -677,13 +873,13 @@ class OCIChatConfig(BaseConfig):
completion_tokens=usage_info.completionTokens, # type: ignore[union-attr]
total_tokens=usage_info.totalTokens # type: ignore[union-attr]
)
return model_response
def _handle_generic_response(
self,
json: dict,
model: str,
self,
json: dict,
model: str,
model_response: ModelResponse,
raw_response: httpx.Response
) -> ModelResponse:
@ -695,7 +891,7 @@ class OCIChatConfig(BaseConfig):
message=f"Response cannot be casted to OCICompletionResponse: {str(e)}",
status_code=raw_response.status_code,
)
iso_str = completion_response.chatResponse.timeCreated
dt = datetime.datetime.fromisoformat(iso_str.replace("Z", "+00:00"))
model_response.created = int(dt.timestamp())
@ -751,7 +947,7 @@ class OCIChatConfig(BaseConfig):
)
vendor = get_vendor_from_model(model)
# Handle response based on vendor type
if vendor == OCIVendors.COHERE:
model_response = self._handle_cohere_response(json, model, model_response)
@ -1080,7 +1276,7 @@ class OCIStreamWrapper(CustomStreamWrapper):
if not chunk.startswith("data:"):
raise ValueError(f"Chunk does not start with 'data:': {chunk}")
dict_chunk = json.loads(chunk[5:]) # Remove 'data: ' prefix and parse JSON
# Check if this is a Cohere stream chunk
if "apiFormat" in dict_chunk and dict_chunk.get("apiFormat") == "COHERE":
return self._handle_cohere_stream_chunk(dict_chunk)

2
litellm/llms/openai/openai.py
View file

@ -1203,7 +1203,6 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
) -> EmbeddingResponse:
super().embedding()
try:
model = model
data = {"model": model, "input": input, **optional_params}
max_retries = max_retries or litellm.DEFAULT_MAX_RETRIES
if not isinstance(max_retries, int):
@ -1332,7 +1331,6 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM):
) -> ImageResponse:
data = {}
try:
model = model
data = {"model": model, "prompt": prompt, **optional_params}
max_retries = data.pop("max_retries", 2)
if not isinstance(max_retries, int):

9
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -220,11 +220,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
def _supports_penalty_parameters(self, model: str) -> bool:
unsupported_models = ["gemini-2.5-pro-preview-06-05"]
for pattern in unsupported_models:
if model in pattern:
return False
if model in unsupported_models:
return False
return True
def get_supported_openai_params(self, model: str) -> List[str]:
@ -416,7 +413,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
# Handle tools with 'type' field (OpenAI spec compliance) Ignore this field -> https://github.com/BerriAI/litellm/issues/14644#issuecomment-3342061838
if "type" in tool:
del tool["type"] # type: ignore
tool = {k: tool[k] for k in tool if k != "type"}
tool_name = list(tool.keys())[0] if len(tool.keys()) == 1 else None
if tool_name and (

34
litellm/main.py
View file

@ -5298,11 +5298,15 @@ def moderation(
or get_secret_str("OPENAI_API_KEY")
)
# Extract api_base from kwargs
api_base = kwargs.get("api_base", None)
openai_client = kwargs.get("client", None)
if openai_client is None:
openai_client = openai.OpenAI(
api_key=api_key,
)
if api_base is not None:
openai_client = openai.OpenAI(api_key=api_key, base_url=api_base)
else:
openai_client = openai.OpenAI(api_key=api_key)
if model is not None:
response = openai_client.moderations.create(input=input, model=model)
@ -5332,21 +5336,11 @@ async def amoderation(
or litellm.openai_key
or get_secret_str("OPENAI_API_KEY")
)
openai_client = kwargs.get("client", None)
if openai_client is None or not isinstance(openai_client, AsyncOpenAI):
# call helper to get OpenAI client
# _get_openai_client maintains in-memory caching logic for OpenAI clients
_openai_client: AsyncOpenAI = openai_chat_completions._get_openai_client( # type: ignore
is_async=True,
api_key=api_key,
)
else:
_openai_client = openai_client
optional_params = GenericLiteLLMParams(**kwargs)
litellm_logging_obj: Optional[LiteLLMLoggingObj] = kwargs.get(
"litellm_logging_obj", None
)
_dynamic_api_base = None
try:
(
model,
@ -5363,6 +5357,18 @@ async def amoderation(
# `model` is optional field for moderation - get_llm_provider will throw BadRequestError if model is not set / not recognized
pass
openai_client = kwargs.get("client", None)
if openai_client is None or not isinstance(openai_client, AsyncOpenAI):
# call helper to get OpenAI client
# _get_openai_client maintains in-memory caching logic for OpenAI clients
_openai_client: AsyncOpenAI = openai_chat_completions._get_openai_client( # type: ignore
is_async=True,
api_key=api_key,
api_base=optional_params.api_base or _dynamic_api_base,
)
else:
_openai_client = openai_client
# update litellm_logging_obj with environment variables
custom_llm_provider = custom_llm_provider or litellm.LlmProviders.OPENAI.value
if litellm_logging_obj is not None:

126
litellm/model_prices_and_context_window_backup.json
View file

@ -1106,6 +1106,7 @@
"supports_vision": true
},
"azure/eu/gpt-4o-2024-08-06": {
"deprecation_date": "2026-02-27",
"cache_read_input_token_cost": 1.375e-06,
"input_cost_per_token": 2.75e-06,
"litellm_provider": "azure",
@ -1122,6 +1123,7 @@
"supports_vision": true
},
"azure/eu/gpt-4o-2024-11-20": {
"deprecation_date": "2026-03-01",
"cache_creation_input_token_cost": 1.38e-06,
"input_cost_per_token": 2.75e-06,
"litellm_provider": "azure",
@ -1280,7 +1282,7 @@
},
"azure/global-standard/gpt-4o-2024-08-06": {
"cache_read_input_token_cost": 1.25e-06,
"deprecation_date": "2025-08-20",
"deprecation_date": "2026-02-27",
"input_cost_per_token": 2.5e-06,
"litellm_provider": "azure",
"max_input_tokens": 128000,
@ -1297,7 +1299,7 @@
},
"azure/global-standard/gpt-4o-2024-11-20": {
"cache_read_input_token_cost": 1.25e-06,
"deprecation_date": "2025-12-20",
"deprecation_date": "2026-03-01",
"input_cost_per_token": 2.5e-06,
"litellm_provider": "azure",
"max_input_tokens": 128000,
@ -1326,6 +1328,7 @@
"supports_vision": true
},
"azure/global/gpt-4o-2024-08-06": {
"deprecation_date": "2026-02-27",
"cache_read_input_token_cost": 1.25e-06,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "azure",
@ -1342,6 +1345,7 @@
"supports_vision": true
},
"azure/global/gpt-4o-2024-11-20": {
"deprecation_date": "2026-03-01",
"cache_read_input_token_cost": 1.25e-06,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "azure",
@ -1625,6 +1629,7 @@
"supports_web_search": false
},
"azure/gpt-4.1-2025-04-14": {
"deprecation_date": "2026-11-04",
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 2e-06,
"input_cost_per_token_batches": 1e-06,
@ -1691,6 +1696,7 @@
"supports_web_search": false
},
"azure/gpt-4.1-mini-2025-04-14": {
"deprecation_date": "2026-11-04",
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 4e-07,
"input_cost_per_token_batches": 2e-07,
@ -1756,6 +1762,7 @@
"supports_vision": true
},
"azure/gpt-4.1-nano-2025-04-14": {
"deprecation_date": "2026-11-04",
"cache_read_input_token_cost": 2.5e-08,
"input_cost_per_token": 1e-07,
"input_cost_per_token_batches": 5e-08,
@ -1837,6 +1844,7 @@
"supports_vision": true
},
"azure/gpt-4o-2024-08-06": {
"deprecation_date": "2026-02-27",
"cache_read_input_token_cost": 1.25e-06,
"input_cost_per_token": 2.5e-06,
"litellm_provider": "azure",
@ -1853,6 +1861,7 @@
"supports_vision": true
},
"azure/gpt-4o-2024-11-20": {
"deprecation_date": "2026-03-01",
"cache_read_input_token_cost": 1.25e-06,
"input_cost_per_token": 2.75e-06,
"litellm_provider": "azure",
@ -2604,6 +2613,7 @@
"supports_vision": true
},
"azure/o3-2025-04-16": {
"deprecation_date": "2026-04-16",
"cache_read_input_token_cost": 2.5e-06,
"input_cost_per_token": 1e-05,
"litellm_provider": "azure",
@ -2832,6 +2842,7 @@
"output_cost_per_token": 0.0
},
"azure/text-embedding-3-small": {
"deprecation_date": "2026-04-30",
"input_cost_per_token": 2e-08,
"litellm_provider": "azure",
"max_input_tokens": 8191,
@ -2870,6 +2881,7 @@
"mode": "audio_speech"
},
"azure/us/gpt-4o-2024-08-06": {
"deprecation_date": "2026-02-27",
"cache_read_input_token_cost": 1.375e-06,
"input_cost_per_token": 2.75e-06,
"litellm_provider": "azure",
@ -2886,6 +2898,7 @@
"supports_vision": true
},
"azure/us/gpt-4o-2024-11-20": {
"deprecation_date": "2026-03-01",
"cache_creation_input_token_cost": 1.38e-06,
"input_cost_per_token": 2.75e-06,
"litellm_provider": "azure",
@ -3385,7 +3398,7 @@
"source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/microsoft/",
"supports_reasoning": true,
"supports_tool_choice": true
},
},
"azure_ai/cohere-rerank-v3-english": {
"input_cost_per_query": 0.002,
"input_cost_per_token": 0.0,
@ -4911,7 +4924,7 @@
"cache_creation_input_token_cost": 3.75e-06,
"cache_creation_input_token_cost_above_1hr": 6e-06,
"cache_read_input_token_cost": 3e-07,
"deprecation_date": "2026-02-01",
"deprecation_date": "2026-02-19",
"input_cost_per_token": 3e-06,
"litellm_provider": "anthropic",
"max_input_tokens": 200000,
@ -4968,7 +4981,6 @@
"cache_creation_input_token_cost": 3e-07,
"cache_creation_input_token_cost_above_1hr": 6e-06,
"cache_read_input_token_cost": 3e-08,
"deprecation_date": "2025-03-01",
"input_cost_per_token": 2.5e-07,
"litellm_provider": "anthropic",
"max_input_tokens": 200000,
@ -4988,7 +5000,7 @@
"cache_creation_input_token_cost": 1.875e-05,
"cache_creation_input_token_cost_above_1hr": 6e-06,
"cache_read_input_token_cost": 1.5e-06,
"deprecation_date": "2025-03-01",
"deprecation_date": "2026-05-01",
"input_cost_per_token": 1.5e-05,
"litellm_provider": "anthropic",
"max_input_tokens": 200000,
@ -5172,6 +5184,7 @@
"cache_creation_input_token_cost_above_1hr": 3e-05,
"cache_read_input_token_cost": 1.5e-06,
"input_cost_per_token": 1.5e-05,
"deprecation_date": "2026-08-05",
"litellm_provider": "anthropic",
"max_input_tokens": 200000,
"max_output_tokens": 32000,
@ -5199,6 +5212,7 @@
"cache_creation_input_token_cost_above_1hr": 3e-05,
"cache_read_input_token_cost": 1.5e-06,
"input_cost_per_token": 1.5e-05,
"deprecation_date": "2026-05-14",
"litellm_provider": "anthropic",
"max_input_tokens": 200000,
"max_output_tokens": 32000,
@ -5222,6 +5236,7 @@
"tool_use_system_prompt_tokens": 159
},
"claude-sonnet-4-20250514": {
"deprecation_date": "2026-05-14",
"cache_creation_input_token_cost": 3.75e-06,
"cache_creation_input_token_cost_above_1hr": 6e-06,
"cache_read_input_token_cost": 3e-07,
@ -7753,7 +7768,7 @@
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
},
"dolphin": {
"input_cost_per_token": 5e-07,
"litellm_provider": "nlp_cloud",
@ -7847,7 +7862,7 @@
"input_cost_per_query": 5e-03,
"litellm_provider": "perplexity",
"mode": "search"
},
},
"elevenlabs/scribe_v1": {
"input_cost_per_second": 6.11e-05,
"litellm_provider": "elevenlabs",
@ -7993,6 +8008,7 @@
"cache_creation_input_token_cost": 1.375e-06,
"cache_read_input_token_cost": 1.1e-07,
"input_cost_per_token": 1.1e-06,
"deprecation_date": "2026-10-15",
"litellm_provider": "bedrock",
"max_input_tokens": 200000,
"max_output_tokens": 8192,
@ -8243,6 +8259,46 @@
"supports_function_calling": true,
"supports_tool_choice": false
},
"fal_ai/bria/text-to-image/3.2": {
"litellm_provider": "fal_ai",
"mode": "image_generation",
"output_cost_per_image": 0.0398,
"supported_endpoints": [
"/v1/images/generations"
]
},
"fal_ai/fal-ai/flux-pro/v1.1-ultra": {
"litellm_provider": "fal_ai",
"mode": "image_generation",
"output_cost_per_image": 0.0398,
"supported_endpoints": [
"/v1/images/generations"
]
},
"fal_ai/fal-ai/imagen4/preview": {
"litellm_provider": "fal_ai",
"mode": "image_generation",
"output_cost_per_image": 0.0398,
"supported_endpoints": [
"/v1/images/generations"
]
},
"fal_ai/fal-ai/recraft/v3/text-to-image": {
"litellm_provider": "fal_ai",
"mode": "image_generation",
"output_cost_per_image": 0.0398,
"supported_endpoints": [
"/v1/images/generations"
]
},
"fal_ai/fal-ai/stable-diffusion-v35-medium": {
"litellm_provider": "fal_ai",
"mode": "image_generation",
"output_cost_per_image": 0.0398,
"supported_endpoints": [
"/v1/images/generations"
]
},
"featherless_ai/featherless-ai/Qwerky-72B": {
"litellm_provider": "featherless_ai",
"max_input_tokens": 32768,
@ -10453,6 +10509,18 @@
"supports_tool_choice": true,
"supports_vision": true
},
"gemini/gemini-embedding-001": {
"input_cost_per_token": 1.5e-07,
"litellm_provider": "gemini",
"max_input_tokens": 2048,
"max_tokens": 2048,
"mode": "embedding",
"output_cost_per_token": 0,
"output_vector_size": 3072,
"rpm": 10000,
"source": "https://ai.google.dev/gemini-api/docs/embeddings#model-versions",
"tpm": 10000000
},
"gemini/gemini-1.5-flash": {
"input_cost_per_token": 7.5e-08,
"input_cost_per_token_above_128k_tokens": 1.5e-07,
@ -12397,6 +12465,7 @@
"supports_tool_choice": true
},
"gpt-3.5-turbo-1106": {
"deprecation_date": "2026-09-28",
"input_cost_per_token": 1e-06,
"litellm_provider": "openai",
"max_input_tokens": 16385,
@ -12466,6 +12535,7 @@
"supports_tool_choice": true
},
"gpt-4-0125-preview": {
"deprecation_date": "2026-03-26",
"input_cost_per_token": 1e-05,
"litellm_provider": "openai",
"max_input_tokens": 128000,
@ -12506,6 +12576,7 @@
"supports_tool_choice": true
},
"gpt-4-1106-preview": {
"deprecation_date": "2026-03-26",
"input_cost_per_token": 1e-05,
"litellm_provider": "openai",
"max_input_tokens": 128000,
@ -15830,6 +15901,20 @@
"max_tokens": 8192,
"mode": "embedding"
},
"mistral/codestral-embed": {
"input_cost_per_token": 0.15e-06,
"litellm_provider": "mistral",
"max_input_tokens": 8192,
"max_tokens": 8192,
"mode": "embedding"
},
"mistral/codestral-embed-2505": {
"input_cost_per_token": 0.15e-06,
"litellm_provider": "mistral",
"max_input_tokens": 8192,
"max_tokens": 8192,
"mode": "embedding"
},
"mistral/mistral-large-2402": {
"input_cost_per_token": 4e-06,
"litellm_provider": "mistral",
@ -16563,6 +16648,7 @@
"supports_vision": true
},
"o1-mini-2024-09-12": {
"deprecation_date": "2025-10-27",
"cache_read_input_token_cost": 1.5e-06,
"input_cost_per_token": 3e-06,
"litellm_provider": "openai",
@ -17733,24 +17819,24 @@
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"openrouter/anthropic/claude-sonnet-4.5": {
"input_cost_per_image": 0.0048,
"input_cost_per_token": 3e-06,
"input_cost_per_token_above_200k_tokens": 6e-06,
"output_cost_per_token_above_200k_tokens": 2.25e-05,
"openrouter/anthropic/claude-haiku-4.5": {
"cache_creation_input_token_cost": 1.25e-06,
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "openrouter",
"max_input_tokens": 1000000,
"max_output_tokens": 1000000,
"max_tokens": 1000000,
"max_input_tokens": 200000,
"max_output_tokens": 200000,
"max_tokens": 200000,
"mode": "chat",
"output_cost_per_token": 1.5e-05,
"output_cost_per_token": 5e-06,
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true,
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
"tool_use_system_prompt_tokens": 346
},
"openrouter/bytedance/ui-tars-1.5-7b": {
"input_cost_per_token": 1e-07,
@ -19166,7 +19252,7 @@
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
},
"qwen.qwen3-32b-v1:0": {
"input_cost_per_token": 1.5e-07,
"litellm_provider": "bedrock_converse",
@ -19178,7 +19264,7 @@
"supports_function_calling": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
},
"recraft/recraftv2": {
"litellm_provider": "recraft",
"mode": "image_generation",

65
litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py
View file

@ -20,6 +20,55 @@ router = APIRouter(
)
def get_request_base_url(request: Request) -> str:
"""
Get the base URL for the request, considering X-Forwarded-* headers.
When behind a proxy (like nginx), the proxy may set:
- X-Forwarded-Proto: The original protocol (http/https)
- X-Forwarded-Host: The original host (may include port)
- X-Forwarded-Port: The original port (if not in Host header)
Args:
request: FastAPI Request object
Returns:
The reconstructed base URL (e.g., "https://proxy.example.com")
"""
base_url = str(request.base_url).rstrip("/")
parsed = urlparse(base_url)
# Get forwarded headers
x_forwarded_proto = request.headers.get("X-Forwarded-Proto")
x_forwarded_host = request.headers.get("X-Forwarded-Host")
x_forwarded_port = request.headers.get("X-Forwarded-Port")
# Start with the original scheme
scheme = x_forwarded_proto if x_forwarded_proto else parsed.scheme
# Handle host and port
if x_forwarded_host:
# X-Forwarded-Host may already include port (e.g., "example.com:8080")
if ":" in x_forwarded_host and not x_forwarded_host.startswith("["):
# Host includes port
netloc = x_forwarded_host
elif x_forwarded_port:
# Port is separate
netloc = f"{x_forwarded_host}:{x_forwarded_port}"
else:
# Just host, no explicit port
netloc = x_forwarded_host
else:
# No X-Forwarded-Host, use original netloc
netloc = parsed.netloc
if x_forwarded_port and ":" not in netloc:
# Add forwarded port if not already in netloc
netloc = f"{netloc}:{x_forwarded_port}"
# Reconstruct the URL
return urlunparse((scheme, netloc, parsed.path, "", "", ""))
def encode_state_with_base_url(
base_url: str,
original_state: str,
@ -107,7 +156,9 @@ async def authorize(
# Parse it to remove any existing query
parsed = urlparse(redirect_uri)
base_url = urlunparse(parsed._replace(query=""))
request_base_url = str(request.base_url).rstrip("/")
# Get the correct base URL considering X-Forwarded-* headers
request_base_url = get_request_base_url(request)
# Encode the base_url, original state, PKCE params, and client redirect_uri in encrypted state
encoded_state = encode_state_with_base_url(
@ -177,7 +228,8 @@ async def token_endpoint(
if mcp_server.token_url is None:
raise HTTPException(status_code=400, detail="MCP server token url is not set")
proxy_base_url = str(request.base_url).rstrip("/")
# Get the correct base URL considering X-Forwarded-* headers
proxy_base_url = get_request_base_url(request)
# Build token request data
token_data = {
@ -251,7 +303,8 @@ async def callback(code: str, state: str):
async def oauth_protected_resource_mcp(
request: Request, mcp_server_name: Optional[str] = None
):
request_base_url = str(request.base_url).rstrip("/")
# Get the correct base URL considering X-Forwarded-* headers
request_base_url = get_request_base_url(request)
return {
"authorization_servers": [
(
@ -273,7 +326,8 @@ async def oauth_protected_resource_mcp(
async def oauth_authorization_server_mcp(
request: Request, mcp_server_name: Optional[str] = None
):
request_base_url = str(request.base_url).rstrip("/")
# Get the correct base URL considering X-Forwarded-* headers
request_base_url = get_request_base_url(request)
authorization_endpoint = (
f"{request_base_url}/{mcp_server_name}/authorize"
@ -320,7 +374,8 @@ async def register_client(request: Request, mcp_server_name: Optional[str] = Non
global_mcp_server_manager,
)
request_base_url = str(request.base_url).rstrip("/")
# Get the correct base URL considering X-Forwarded-* headers
request_base_url = get_request_base_url(request)
request_data = await _read_request_body(request=request)
data: dict = {**request_data}

1
litellm/proxy/_experimental/out/_next/static/chunks/1162-278deed893787c5d.js Normal file
View file

File diff suppressed because one or more lines are too long

0
litellm/proxy/_experimental/out/_next/static/chunks/131-114a46ffd6544611.js → litellm/proxy/_experimental/out/_next/static/chunks/131-430e945bc56307dc.js
View file

0
litellm/proxy/_experimental/out/_next/static/chunks/1739-fc28b5c8fce919b7.js → litellm/proxy/_experimental/out/_next/static/chunks/1739-b8a3795ea80fe5e5.js
View file

1
litellm/proxy/_experimental/out/_next/static/chunks/1853-fa2eb7102429db88.js
View file

File diff suppressed because one or more lines are too long

0
litellm/proxy/_experimental/out/_next/static/chunks/2004-08bb1f463a2409e4.js → litellm/proxy/_experimental/out/_next/static/chunks/2004-836e8f164e69b068.js
View file

0
litellm/proxy/_experimental/out/_next/static/chunks/2012-349484b0c5e31882.js → litellm/proxy/_experimental/out/_next/static/chunks/2012-0bbf11e40fd2af7d.js
View file

0
litellm/proxy/_experimental/out/_next/static/chunks/2019-48d89281c5103ab7.js → litellm/proxy/_experimental/out/_next/static/chunks/2019-c448ee48f8526098.js
View file

0
litellm/proxy/_experimental/out/_next/static/chunks/2162-246f81f806dcd551.js → litellm/proxy/_experimental/out/_next/static/chunks/2162-52d2743e52b9b773.js
View file

2
litellm/proxy/_experimental/out/_next/static/chunks/2202-5828413daf645b2b.js → litellm/proxy/_experimental/out/_next/static/chunks/2202-902e08da4e70850b.js
View file

File diff suppressed because one or more lines are too long

0
litellm/proxy/_experimental/out/_next/static/chunks/2273-28dec6d6da007fff.js → litellm/proxy/_experimental/out/_next/static/chunks/2273-8b9ed1c724f4f8b6.js
View file

0
litellm/proxy/_experimental/out/_next/static/chunks/2306-8bb763a915ba9145.js → litellm/proxy/_experimental/out/_next/static/chunks/2306-dfb3df53f3a9cd88.js
View file

1
litellm/proxy/_experimental/out/_next/static/chunks/2945-00442a283770ee5d.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/302-b44f49193a517791.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/3298-f271bfaa40544a40.js
View file

File diff suppressed because one or more lines are too long

0
litellm/proxy/_experimental/out/_next/static/chunks/3801-e8b8c1bf8b6a231c.js → litellm/proxy/_experimental/out/_next/static/chunks/3801-12f0e220f3db9b4b.js
View file

1
litellm/proxy/_experimental/out/_next/static/chunks/3866-e3419825a249263f.js
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/4138-c37bac62b3dd3750.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/4193-486afc1bc957d491.js Normal file
View file

File diff suppressed because one or more lines are too long

1
litellm/proxy/_experimental/out/_next/static/chunks/4238-943dbcbd28576cd0.js Normal file
View file

File diff suppressed because one or more lines are too long

2
litellm/proxy/_experimental/out/_next/static/chunks/4292-8682ab62a15b3165.js → litellm/proxy/_experimental/out/_next/static/chunks/4292-fce839d666533ad3.js
View file

File diff suppressed because one or more lines are too long

0
litellm/proxy/_experimental/out/_next/static/chunks/4696-6ed186a88283bc88.js → litellm/proxy/_experimental/out/_next/static/chunks/4696-42f889dd0e9f1f8b.js
View file

1
litellm/proxy/_experimental/out/_next/static/chunks/5030-d46e46be47c1567d.js Normal file
View file

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show more