mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-07 02:59:05 +00:00
Mirror of https://github.com/BerriAI/litellm.git
9a0658084b
This commit fixes two critical test failures and two test isolation issues in the SSL configuration tests. ## Critical Test Failures Fixed ### 1. test_get_ssl_configuration **Problem:** Test was failing with assertion error that ssl.create_default_context was never called (expected 1 call, got 0). **Root Cause:** The get_ssl_configuration() function uses a caching mechanism (_ssl_context_cache) to avoid creating duplicate SSL contexts with the same configuration. When tests run in sequence, a previous test may have created an SSL context with the same configuration (same cafile, ssl_security_level, ssl_ecdh_curve). When this test runs, it retrieves the cached context instead of creating a new one, so ssl.create_default_context() is never called, causing the mock assertion to fail. **Fix:** Clear the SSL context cache at the start of the test to ensure a fresh context is created, allowing the mock to be called and verified. ### 2. test_ssl_ecdh_curve **Problem:** Test was failing with assertion error that set_ecdh_curve was never called (expected 1 call, got 0). **Root Cause:** Same caching issue as above. Additionally, the test needed to use a real SSLContext instance instead of a MagicMock because _create_ssl_context calls methods like set_ciphers() and minimum_version that require a real context. **Fix:** - Clear the SSL context cache at the start of the test - Use a real SSLContext instance and patch set_ecdh_curve on it specifically - Added explanatory comment about why a real context is needed ## Test Isolation Issues Fixed ### 3. test_ssl_security_level **Problem:** Test was failing because it expected LiteLLMAiohttpTransport but got httpx.AsyncHTTPTransport instead. **Root Cause:** Test isolation issue. Other tests in the file (test_force_ipv4_transport, test_aiohttp_disabled_transport) set litellm.disable_aiohttp_transport = True but don't restore the original value. When this test runs after those tests, aiohttp transport is disabled, causing it to use httpx transport instead. **Fix:** Explicitly enable aiohttp transport at the start of the test and restore the original value in a finally block, ensuring the test works regardless of test execution order. ### 4. test_ssl_verification_with_aiohttp_transport **Problem:** Same as above - expected LiteLLMAiohttpTransport but got httpx.AsyncHTTPTransport. **Root Cause:** Same test isolation issue - aiohttp transport disabled by previous tests. **Fix:** Same approach - explicitly enable aiohttp transport and restore original value in finally block. ## Why These Fixes Work 1. **Cache clearing:** By clearing _ssl_context_cache before each test, we ensure that get_ssl_configuration() creates a fresh SSL context, allowing mocks to be properly called and verified. 2. **Test isolation:** By saving and restoring the disable_aiohttp_transport setting, tests are independent of each other and work correctly regardless of execution order. These are minimal, targeted fixes that address the root causes without modifying production code or affecting other functionality. |
||
|---|---|---|
| .circleci | ||
| .devcontainer | ||
| .github | ||
| ci_cd | ||
| cookbook | ||
| db_scripts | ||
| deploy | ||
| dist | ||
| docker | ||
| docs/my-website | ||
| enterprise | ||
| litellm | ||
| litellm-js | ||
| litellm-proxy-extras | ||
| scripts | ||
| tests | ||
| ui/litellm-dashboard | ||
| .dockerignore | ||
| .env.example | ||
| .flake8 | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| AGENTS.md | ||
| batch_small.jsonl | ||
| CLAUDE.md | ||
| codecov.yaml | ||
| CONTRIBUTING.md | ||
| docker-compose.yml | ||
| Dockerfile | ||
| GEMINI.md | ||
| index.yaml | ||
| LICENSE | ||
| Makefile | ||
| mcp_servers.json | ||
| model_prices_and_context_window.json | ||
| package-lock.json | ||
| package.json | ||
| poetry.lock | ||
| prometheus.yml | ||
| provider_endpoints_support.json | ||
| proxy_server_config.yaml | ||
| pyproject.toml | ||
| pyrightconfig.json | ||
| README.md | ||
| render.yaml | ||
| requirements.txt | ||
| ruff.toml | ||
| schema.prisma | ||
| security.md | ||
| test_pydantic_fields.py | ||
🚅 LiteLLM
Call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]
LiteLLM Proxy Server (LLM Gateway) | Hosted Proxy (Preview) | Enterprise Tier
LiteLLM manages:
- Translate inputs to provider's
completion,embedding, andimage_generationendpoints - Consistent output, text responses will always be available at
['choices'][0]['message']['content'] - Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router
- Set Budgets & Rate limits per project, api key, model LiteLLM Proxy Server (LLM Gateway)
LiteLLM Performance: 8ms P95 latency at 1k RPS (See benchmarks here)
Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers
🚨 Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here
Support for more providers. Missing a provider or LLM Platform, raise a feature request.
Usage (Docs)
Important
LiteLLM v1.0.0 now requires
openai>=1.0.0. Migration guide here LiteLLM v1.40.14+ now requirespydantic>=2.0.0. No changes required.
pip install litellm
from litellm import completion
import os
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="openai/gpt-4o", messages=messages)
# anthropic call
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=messages)
print(response)
Response (OpenAI Format)
{
"id": "chatcmpl-1214900a-6cdd-4148-b663-b5e2f642b4de",
"created": 1751494488,
"model": "claude-sonnet-4-20250514",
"object": "chat.completion",
"system_fingerprint": null,
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "Hello! I'm doing well, thank you for asking. I'm here and ready to help with whatever you'd like to discuss or work on. How are you doing today?",
"role": "assistant",
"tool_calls": null,
"function_call": null
}
}
],
"usage": {
"completion_tokens": 39,
"prompt_tokens": 13,
"total_tokens": 52,
"completion_tokens_details": null,
"prompt_tokens_details": {
"audio_tokens": null,
"cached_tokens": 0
},
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 0
}
}
Call any model supported by a provider, with model=<provider_name>/<model_name>. There might be provider-specific details here, so refer to provider docs for more information
Async (Docs)
from litellm import acompletion
import asyncio
async def test_get_response():
user_message = "Hello, how are you?"
messages = [{"content": user_message, "role": "user"}]
response = await acompletion(model="openai/gpt-4o", messages=messages)
return response
response = asyncio.run(test_get_response())
print(response)
Streaming (Docs)
LiteLLM supports streaming the model response back, pass stream=True to get a streaming iterator in response.
Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
from litellm import completion
messages = [{"content": "Hello, how are you?", "role": "user"}]
# gpt-4o
response = completion(model="openai/gpt-4o", messages=messages, stream=True)
for part in response:
print(part.choices[0].delta.content or "")
# claude sonnet 4
response = completion('anthropic/claude-sonnet-4-20250514', messages, stream=True)
for part in response:
print(part)
Response chunk (OpenAI Format)
{
"id": "chatcmpl-fe575c37-5004-4926-ae5e-bfbc31f356ca",
"created": 1751494808,
"model": "claude-sonnet-4-20250514",
"object": "chat.completion.chunk",
"system_fingerprint": null,
"choices": [
{
"finish_reason": null,
"index": 0,
"delta": {
"provider_specific_fields": null,
"content": "Hello",
"role": "assistant",
"function_call": null,
"tool_calls": null,
"audio": null
},
"logprobs": null
}
],
"provider_specific_fields": null,
"stream_options": null,
"citations": null
}
Logging Observability (Docs)
LiteLLM exposes pre defined callbacks to send data to Lunary, MLflow, Langfuse, DynamoDB, s3 Buckets, Helicone, Promptlayer, Traceloop, Athina, Slack
from litellm import completion
## set env variables for logging tools (when using MLflow, no API key set up is required)
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key"
os.environ["HELICONE_API_KEY"] = "your-helicone-auth-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["ATHINA_API_KEY"] = "your-athina-api-key"
os.environ["OPENAI_API_KEY"] = "your-openai-key"
# set callbacks
litellm.success_callback = ["lunary", "mlflow", "langfuse", "athina", "helicone"] # log input/output to lunary, langfuse, supabase, athina, helicone etc
#openai call
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])
LiteLLM Proxy Server (LLM Gateway) - (Docs)
Track spend + Load Balance across multiple projects
The proxy provides:
📖 Proxy Endpoints - Swagger Docs
Quick Start Proxy - CLI
pip install 'litellm[proxy]'
Step 1: Start litellm proxy
$ litellm --model huggingface/bigcode/starcoder
#INFO: Proxy running on http://0.0.0.0:4000
Step 2: Make ChatCompletions Request to Proxy
Important
import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:4000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
}
])
print(response)
Proxy Key Management (Docs)
Connect the proxy with a Postgres DB to create proxy keys
# Get the code
git clone https://github.com/BerriAI/litellm
# Go to folder
cd litellm
# Add the master key - you can change this after setup
echo 'LITELLM_MASTER_KEY="sk-1234"' > .env
# Add the litellm salt key - you cannot change this after adding a model
# It is used to encrypt / decrypt your LLM API Key credentials
# We recommend - https://1password.com/password-generator/
# password generator to get a random hash for litellm salt key
echo 'LITELLM_SALT_KEY="sk-1234"' >> .env
source .env
# Start
docker compose up
UI on /ui on your proxy server
Set budgets and rate limits across multiple projects
POST /key/generate
Request
curl 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data-raw '{"models": ["gpt-3.5-turbo", "gpt-4", "claude-2"], "duration": "20m","metadata": {"user": "ishaan@berri.ai", "team": "core-infra"}}'
Expected Response
{
"key": "sk-kdEXbIqZRwEeEiHwdg7sFA", # Bearer token
"expires": "2023-11-19T01:38:25.838000+00:00" # datetime object
}
Supported Providers (Website Supported Models | Docs)
Run in Developer mode
Services
- Setup .env file in root
- Run dependant services
docker-compose up db prometheus
Backend
- (In root) create virtual environment
python -m venv .venv - Activate virtual environment
source .venv/bin/activate - Install dependencies
pip install -e ".[all]" - Start proxy backend
python litellm/proxy_cli.py
Frontend
- Navigate to
ui/litellm-dashboard - Install dependencies
npm install - Run
npm run devto start the dashboard
Enterprise
For companies that need better security, user management and professional support
This covers:
- ✅ Features under the LiteLLM Commercial License:
- ✅ Feature Prioritization
- ✅ Custom Integrations
- ✅ Professional Support - Dedicated discord + slack
- ✅ Custom SLAs
- ✅ Secure access with Single Sign-On
Contributing
We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.
Quick Start for Contributors
This requires poetry to be installed.
git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev # Install development dependencies
make format # Format your code
make lint # Run all linting checks
make test-unit # Run unit tests
make format-check # Check formatting only
For detailed contributing guidelines, see CONTRIBUTING.md.
Code Quality / Linting
LiteLLM follows the Google Python Style Guide.
Our automated checks include:
- Black for code formatting
- Ruff for linting and code quality
- MyPy for type checking
- Circular import detection
- Import safety checks
All these checks must pass before your PR can be merged.
Support / talk with founders
- Schedule Demo 👋
- Community Discord 💭
- Community Slack 💭
- Our numbers 📞 +1 (770) 8783-106 / +1 (412) 618-6238
- Our emails ✉️ ishaan@berri.ai / krrish@berri.ai
Why did we build this
- Need for simplicity: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI and Cohere.