-
- 
-
@@ -40,7 +37,7 @@ LiteLLM manages:
[**Jump to LiteLLM Proxy (LLM Gateway) Docs**](https://github.com/BerriAI/litellm?tab=readme-ov-file#openai-proxy---docs) + Use Vector Stores with any LiteLLM supported model +
+ + +LiteLLM integrates with vector stores, allowing your models to access your organization's data for more accurate and contextually relevant responses. + +## Supported Vector Stores +- [Bedrock Knowledge Bases](https://aws.amazon.com/bedrock/knowledge-bases/) + +## Quick Start + +In order to use a vector store with LiteLLM, you need to + +- Initialize litellm.vector_store_registry +- Pass tools with vector_store_ids to the completion request. Where `vector_store_ids` is a list of vector store ids you initialized in litellm.vector_store_registry + +### LiteLLM Python SDK + +LiteLLM's allows you to use vector stores in the [OpenAI API spec](https://platform.openai.com/docs/api-reference/chat/create) by passing a tool with vector_store_ids you want to use + +```python showLineNumbers title="Basic Bedrock Knowledge Base Usage" +import os +import litellm + +from litellm.vector_stores.vector_store_registry import VectorStoreRegistry, LiteLLM_ManagedVectorStore + +# Init vector store registry +litellm.vector_store_registry = VectorStoreRegistry( + vector_stores=[ + LiteLLM_ManagedVectorStore( + vector_store_id="T37J8R4WTM", + custom_llm_provider="bedrock" + ) + ] +) + + +# Make a completion request with vector_store_ids parameter +response = await litellm.acompletion( + model="anthropic/claude-3-5-sonnet", + messages=[{"role": "user", "content": "What is litellm?"}], + tools=[ + { + "type": "file_search", + "vector_store_ids": ["T37J8R4WTM"] + } + ], +) + +print(response.choices[0].message.content) +``` + +### LiteLLM Proxy + +#### 1. Configure your vector_store_registry + +In order to use a vector store with LiteLLM, you need to configure your vector_store_registry. This tells litellm which vector stores to use and api provider to use for the vector store. + ++ LiteLLM Logs Page: Vector Store Usage +
+ + +### Listing available vector stores + +You can list all available vector stores using the /vector_store/list endpoint + +**Request:** +```bash showLineNumbers title="List all available vector stores" +curl -X GET "http://localhost:4000/vector_store/list" \ + -H "Authorization: Bearer $LITELLM_API_KEY" +``` + +**Response:** + +The response will be a list of all vector stores that are available to use with LiteLLM. + +```json +{ + "object": "list", + "data": [ + { + "vector_store_id": "T37J8R4WTM", + "custom_llm_provider": "bedrock", + "vector_store_name": "bedrock-litellm-website-knowledgebase", + "vector_store_description": "Bedrock vector store for the Litellm website knowledgebase", + "vector_store_metadata": { + "source": "https://www.litellm.com/docs" + }, + "created_at": "2023-05-03T18:21:36.462Z", + "updated_at": "2023-05-03T18:21:36.462Z", + "litellm_credential_name": "bedrock_credentials" + } + ], + "total_count": 1, + "current_page": 1, + "total_pages": 1 +} +``` + + +### Always on for a model + +**Use this if you want vector stores to be used by default for a specific model.** + +In this config, we add `vector_store_ids` to the claude-3-5-sonnet-with-vector-store model. This means that any request to the claude-3-5-sonnet-with-vector-store model will always use the vector store with the id `T37J8R4WTM` defined in the `vector_store_registry`. + +```yaml showLineNumbers title="Always on for a model" +model_list: + - model_name: claude-3-5-sonnet-with-vector-store + litellm_params: + model: anthropic/claude-3-5-sonnet + vector_store_ids: ["T37J8R4WTM"] + +vector_store_registry: + - vector_store_name: "bedrock-litellm-website-knowledgebase" + litellm_params: + vector_store_id: "T37J8R4WTM" + custom_llm_provider: "bedrock" + vector_store_description: "Bedrock vector store for the Litellm website knowledgebase" + vector_store_metadata: + source: "https://www.litellm.com/docs" +``` + +## How It Works + +If your request includes a `vector_store_ids` parameter where any of the vector store ids are found in the `vector_store_registry`, LiteLLM will automatically use the vector store for the request. + +1. You make a completion request with the `vector_store_ids` parameter and any of the vector store ids are found in the `litellm.vector_store_registry` +2. LiteLLM automatically: + - Uses your last message as the query to retrieve relevant information from the Knowledge Base + - Adds the retrieved context to your conversation + - Sends the augmented messages to the model + +#### Example Transformation + +When you pass `vector_store_ids=["YOUR_KNOWLEDGE_BASE_ID"]`, your request flows through these steps: + +**1. Original Request to LiteLLM:** +```json +{ + "model": "anthropic/claude-3-5-sonnet", + "messages": [ + {"role": "user", "content": "What is litellm?"} + ], + "vector_store_ids": ["YOUR_KNOWLEDGE_BASE_ID"] +} +``` + +**2. Request to AWS Bedrock Knowledge Base:** +```json +{ + "retrievalQuery": { + "text": "What is litellm?" + } +} +``` +This is sent to: `https://bedrock-agent-runtime.{aws_region}.amazonaws.com/knowledgebases/YOUR_KNOWLEDGE_BASE_ID/retrieve` + +**3. Final Request to LiteLLM:** +```json +{ + "model": "anthropic/claude-3-5-sonnet", + "messages": [ + {"role": "user", "content": "What is litellm?"}, + {"role": "user", "content": "Context: \n\nLiteLLM is an open-source SDK to simplify LLM API calls across providers (OpenAI, Claude, etc). It provides a standardized interface with robust error handling, streaming, and observability tools."} + ] +} +``` + +This process happens automatically whenever you include the `vector_store_ids` parameter in your request. + +## API Reference + +### LiteLLM Completion Knowledge Base Parameters + +When using the Knowledge Base integration with LiteLLM, you can include the following parameters: + +| Parameter | Type | Description | +|-----------|------|-------------| +| `vector_store_ids` | List[str] | List of Knowledge Base IDs to query | + +### VectorStoreRegistry + +The `VectorStoreRegistry` is a central component for managing vector stores in LiteLLM. It acts as a registry where you can configure and access your vector stores. + +#### What is VectorStoreRegistry? + +`VectorStoreRegistry` is a class that: +- Maintains a collection of vector stores that LiteLLM can use +- Allows you to register vector stores with their credentials and metadata +- Makes vector stores accessible via their IDs in your completion requests + +#### Using VectorStoreRegistry in Python + +```python +from litellm.vector_stores.vector_store_registry import VectorStoreRegistry, LiteLLM_ManagedVectorStore + +# Initialize the vector store registry with one or more vector stores +litellm.vector_store_registry = VectorStoreRegistry( + vector_stores=[ + LiteLLM_ManagedVectorStore( + vector_store_id="YOUR_VECTOR_STORE_ID", # Required: Unique ID for referencing this store + custom_llm_provider="bedrock" # Required: Provider (e.g., "bedrock") + ) + ] +) +``` + +#### LiteLLM_ManagedVectorStore Parameters + +Each vector store in the registry is configured using a `LiteLLM_ManagedVectorStore` object with these parameters: + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `vector_store_id` | str | Yes | Unique identifier for the vector store | +| `custom_llm_provider` | str | Yes | The provider of the vector store (e.g., "bedrock") | +| `vector_store_name` | str | No | A friendly name for the vector store | +| `vector_store_description` | str | No | Description of what the vector store contains | +| `vector_store_metadata` | dict or str | No | Additional metadata about the vector store | +| `litellm_credential_name` | str | No | Name of the credentials to use for this vector store | + +#### Configuring VectorStoreRegistry in config.yaml + +For the LiteLLM Proxy, you can configure the same registry in your `config.yaml` file: + +```yaml showLineNumbers title="Vector store configuration in config.yaml" +vector_store_registry: + - vector_store_name: "bedrock-litellm-website-knowledgebase" # Optional friendly name + litellm_params: + vector_store_id: "T37J8R4WTM" # Required: Unique ID + custom_llm_provider: "bedrock" # Required: Provider + vector_store_description: "Bedrock vector store for the Litellm website knowledgebase" + vector_store_metadata: + source: "https://www.litellm.com/docs" +``` + +The `litellm_params` section accepts all the same parameters as the `LiteLLM_ManagedVectorStore` constructor in the Python SDK. + + diff --git a/docs/my-website/docs/completion/prompt_caching.md b/docs/my-website/docs/completion/prompt_caching.md index 5c795778ef9..9447a11d527 100644 --- a/docs/my-website/docs/completion/prompt_caching.md +++ b/docs/my-website/docs/completion/prompt_caching.md @@ -3,7 +3,13 @@ import TabItem from '@theme/TabItem'; # Prompt Caching -For OpenAI + Anthropic + Deepseek, LiteLLM follows the OpenAI prompt caching usage object format: +Supported Providers: +- OpenAI (`openai/`) +- Anthropic API (`anthropic/`) +- Bedrock (`bedrock/`, `bedrock/invoke/`, `bedrock/converse`) ([All models bedrock supports prompt caching on](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html)) +- Deepseek API (`deepseek/`) + +For the supported providers, LiteLLM follows the OpenAI prompt caching usage object format: ```bash "usage": { @@ -499,4 +505,4 @@ curl -L -X GET 'http://0.0.0.0:4000/v1/model/info' \ -This checks our maintained [model info/cost map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) \ No newline at end of file +This checks our maintained [model info/cost map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) diff --git a/docs/my-website/docs/completion/reliable_completions.md b/docs/my-website/docs/completion/reliable_completions.md index 94102e1944e..f38917fe53d 100644 --- a/docs/my-website/docs/completion/reliable_completions.md +++ b/docs/my-website/docs/completion/reliable_completions.md @@ -46,7 +46,7 @@ from litellm import completion fallback_dict = {"gpt-3.5-turbo": "gpt-3.5-turbo-16k"} messages = [{"content": "how does a court case get to the Supreme Court?" * 500, "role": "user"}] -completion(model="gpt-3.5-turbo", messages=messages, context_window_fallback_dict=ctx_window_fallback_dict) +completion(model="gpt-3.5-turbo", messages=messages, context_window_fallback_dict=fallback_dict) ``` ### Fallbacks - Switch Models/API Keys/API Bases (SDK) diff --git a/docs/my-website/docs/completion/vision.md b/docs/my-website/docs/completion/vision.md index 0880d0ec492..76700084868 100644 --- a/docs/my-website/docs/completion/vision.md +++ b/docs/my-website/docs/completion/vision.md @@ -118,9 +118,11 @@ response = client.chat.completions.create( Use `litellm.supports_vision(model="")` -> returns `True` if model supports `vision` and `False` if not ```python -assert litellm.supports_vision(model="gpt-4-vision-preview") == True -assert litellm.supports_vision(model="gemini-1.0-pro-vision") == True -assert litellm.supports_vision(model="gpt-3.5-turbo") == False +assert litellm.supports_vision(model="openai/gpt-4-vision-preview") == True +assert litellm.supports_vision(model="vertex_ai/gemini-1.0-pro-vision") == True +assert litellm.supports_vision(model="openai/gpt-3.5-turbo") == False +assert litellm.supports_vision(model="xai/grok-2-vision-latest") == True +assert litellm.supports_vision(model="xai/grok-2-latest") == False ``` @@ -187,4 +189,138 @@ Expected Response ``` - \ No newline at end of file + + + +## Explicitly specify image type + +If you have images without a mime-type, or if litellm is incorrectly inferring the mime type of your image (e.g. calling `gs://` url's with vertex ai), you can set this explicitly via the `format` param. + +```python +"image_url": { + "url": "gs://my-gs-image", + "format": "image/jpeg" +} +``` + +LiteLLM will use this for any API endpoint, which supports specifying mime-type (e.g. anthropic/bedrock/vertex ai). + +For others (e.g. openai), it will be ignored. + ++ LiteLLM MCP Architecture: Use MCP tools with all LiteLLM supported models +
+ +#### How it works + +LiteLLM exposes the following MCP endpoints: + +- `/mcp/tools/list` - List all available tools +- `/mcp/tools/call` - Call a specific tool with the provided arguments + +When MCP clients connect to LiteLLM they can follow this workflow: + +1. Connect to the LiteLLM MCP server +2. List all available tools on LiteLLM +3. Client makes LLM API request with tool call(s) +4. LLM API returns which tools to call and with what arguments +5. MCP client makes MCP tool calls to LiteLLM +6. LiteLLM makes the tool calls to the appropriate MCP server +7. LiteLLM returns the tool call results to the MCP client + +#### Usage + +#### 1. Define your tools on under `mcp_servers` in your config.yaml file. + +LiteLLM allows you to define your tools on the `mcp_servers` section in your config.yaml file. All tools listed here will be available to MCP clients (when they connect to LiteLLM and call `list_tools`). + +```yaml title="config.yaml" showLineNumbers +model_list: + - model_name: gpt-4o + litellm_params: + model: openai/gpt-4o + api_key: sk-xxxxxxx + +mcp_servers: + zapier_mcp: + url: "https://actions.zapier.com/mcp/sk-akxxxxx/sse" + fetch: + url: "http://localhost:8000/sse" +``` + + +#### 2. Start LiteLLM Gateway + +
-You need to tell LiteLLM when you're calling Huggingface.
-This is done by adding the "huggingface/" prefix to `model`, example `completion(model="huggingface/
+
+
+### Basic Completion
+After you have [deployed your Hugging Face Inference Endpoint](https://endpoints.huggingface.co/new) on dedicated infrastructure, you can run inference on it by providing the endpoint base URL in `api_base`, and indicating `huggingface/tgi` as the model name.
+
+```python
+import os
+from litellm import completion
+
+os.environ["HF_TOKEN"] = "hf_xxxxxx"
+
+response = completion(
+ model="huggingface/tgi",
+ messages=[{"content": "Hello, how are you?", "role": "user"}],
+ api_base="https://my-endpoint.endpoints.huggingface.cloud/v1/"
+)
+print(response)
+```
+
+### Streaming
+
+```python
+import os
+from litellm import completion
+
+os.environ["HF_TOKEN"] = "hf_xxxxxx"
+
+response = completion(
+ model="huggingface/tgi",
+ messages=[{"content": "Hello, how are you?", "role": "user"}],
+ api_base="https://my-endpoint.endpoints.huggingface.cloud/v1/",
stream=True
)
-print(response)
-```
-
-
-
-
-You need to tell LiteLLM when you're calling Huggingface.
-This is done by adding the "huggingface/" prefix to `model`, example `completion(model="huggingface/+Each instance writes updates to redis +
+ + +### Stage 2. A single instance flushes the redis queue to the DB + +A single instance will acquire a lock on the DB and flush all elements in the redis queue to the DB. + +- 1 instance will attempt to acquire the lock for the DB update job +- The status of the lock is stored in redis +- If the instance acquires the lock to write to DB + - It will read all updates from redis + - Aggregate all updates into 1 transaction + - Write updates to DB + - Release the lock +- Note: Only 1 instance can acquire the lock at a time, this limits the number of instances that can write to the DB at once + + ++A single instance flushes the redis queue to the DB +
+ + +## Usage + +### Required components + +- Redis +- Postgres + +### Setup on LiteLLM config + +You can enable using the redis buffer by setting `use_redis_transaction_buffer: true` in the `general_settings` section of your `proxy_config.yaml` file. + +Note: This setup requires litellm to be connected to a redis instance. + +```yaml showLineNumbers title="litellm proxy_config.yaml" +general_settings: + use_redis_transaction_buffer: true + +litellm_settings: + cache: True + cache_params: + type: redis + supported_call_types: [] # Optional: Set cache for proxy, but not on the actual llm api call +``` + +## Monitoring + +LiteLLM emits the following prometheus metrics to monitor the health/status of the in memory buffer and redis buffer. + + +| Metric Name | Description | Storage Type | +|-----------------------------------------------------|-----------------------------------------------------------------------------|--------------| +| `litellm_pod_lock_manager_size` | Indicates which pod has the lock to write updates to the database. | Redis | +| `litellm_in_memory_daily_spend_update_queue_size` | Number of items in the in-memory daily spend update queue. These are the aggregate spend logs for each user. | In-Memory | +| `litellm_redis_daily_spend_update_queue_size` | Number of items in the Redis daily spend update queue. These are the aggregate spend logs for each user. | Redis | +| `litellm_in_memory_spend_update_queue_size` | In-memory aggregate spend values for keys, users, teams, team members, etc.| In-Memory | +| `litellm_redis_spend_update_queue_size` | Redis aggregate spend values for keys, users, teams, etc. | Redis | + diff --git a/docs/my-website/docs/proxy/db_info.md b/docs/my-website/docs/proxy/db_info.md index 1b87aa1e548..946089bf147 100644 --- a/docs/my-website/docs/proxy/db_info.md +++ b/docs/my-website/docs/proxy/db_info.md @@ -46,18 +46,17 @@ You can see the full DB Schema [here](https://github.com/BerriAI/litellm/blob/ma | Table Name | Description | Row Insert Frequency | |------------|-------------|---------------------| -| LiteLLM_SpendLogs | Detailed logs of all API requests. Records token usage, spend, and timing information. Tracks which models and keys were used. | **High - every LLM API request** | -| LiteLLM_ErrorLogs | Captures failed requests and errors. Stores exception details and request information. Helps with debugging and monitoring. | **Medium - on errors only** | +| LiteLLM_SpendLogs | Detailed logs of all API requests. Records token usage, spend, and timing information. Tracks which models and keys were used. | **High - every LLM API request - Success or Failure** | | LiteLLM_AuditLog | Tracks changes to system configuration. Records who made changes and what was modified. Maintains history of updates to teams, users, and models. | **Off by default**, **High - when enabled** | -## Disable `LiteLLM_SpendLogs` & `LiteLLM_ErrorLogs` +## Disable `LiteLLM_SpendLogs` You can disable spend_logs and error_logs by setting `disable_spend_logs` and `disable_error_logs` to `True` on the `general_settings` section of your proxy_config.yaml file. ```yaml general_settings: disable_spend_logs: True # Disable writing spend logs to DB - disable_error_logs: True # Disable writing error logs to DB + disable_error_logs: True # Only disable writing error logs to DB, regular spend logs will still be written unless `disable_spend_logs: True` ``` ### What is the impact of disabling these logs? diff --git a/docs/my-website/docs/proxy/deploy.md b/docs/my-website/docs/proxy/deploy.md index 011778f5841..511a9dda087 100644 --- a/docs/my-website/docs/proxy/deploy.md +++ b/docs/my-website/docs/proxy/deploy.md @@ -22,7 +22,7 @@ echo 'LITELLM_MASTER_KEY="sk-1234"' > .env # Add the litellm salt key - you cannot change this after adding a model # It is used to encrypt / decrypt your LLM API Key credentials -# We recommned - https://1password.com/password-generator/ +# We recommend - https://1password.com/password-generator/ # password generator to get a random hash for litellm salt key echo 'LITELLM_SALT_KEY="sk-1234"' >> .env @@ -125,7 +125,7 @@ CMD ["--port", "4000", "--config", "config.yaml", "--detailed_debug"] ### Build from litellm `pip` package -Follow these instructons to build a docker container from the litellm pip package. If your company has a strict requirement around security / building images you can follow these steps. +Follow these instructions to build a docker container from the litellm pip package. If your company has a strict requirement around security / building images you can follow these steps. Dockerfile @@ -999,7 +999,7 @@ services: - "4000:4000" # Map the container port to the host, change the host port if necessary volumes: - ./litellm-config.yaml:/app/config.yaml # Mount the local configuration file - # You can change the port or number of workers as per your requirements or pass any new supported CLI augument. Make sure the port passed here matches with the container port defined above in `ports` value + # You can change the port or number of workers as per your requirements or pass any new supported CLI argument. Make sure the port passed here matches with the container port defined above in `ports` value command: [ "--config", "/app/config.yaml", "--port", "4000", "--num_workers", "8" ] # ...rest of your docker-compose config if any @@ -1049,3 +1049,25 @@ export DATABASE_SCHEMA="schema-name" # skip to use the default "public" schema litellm --config /path/to/config.yaml --iam_token_db_auth ``` +### ✨ Blocking web crawlers + +Note: This is an [enterprise only feature](https://docs.litellm.ai/docs/enterprise). + +To block web crawlers from indexing the proxy server endpoints, set the `block_robots` setting to `true` in your `litellm_config.yaml` file. + +```yaml showLineNumbers title="litellm_config.yaml" +general_settings: + block_robots: true +``` + +#### How it works + +When this is enabled, the `/robots.txt` endpoint will return a 200 status code with the following content: + +```shell showLineNumbers title="robots.txt" +User-agent: * +Disallow: / +``` + + + diff --git a/docs/my-website/docs/proxy/email.md b/docs/my-website/docs/proxy/email.md index a3f3a41694e..4eb35367dbe 100644 --- a/docs/my-website/docs/proxy/email.md +++ b/docs/my-website/docs/proxy/email.md @@ -1,35 +1,130 @@ import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; # Email Notifications -Send an Email to your users when: -- A Proxy API Key is created for them -- Their API Key crosses it's Budget -- All Team members of a LiteLLM Team -> when the team crosses it's budget ++ LiteLLM Email Notifications +
-+ Send LiteLLM logs to a custom API endpoint +
+ :::info This is an Enterprise only feature [Get Started with Enterprise here](https://github.com/BerriAI/litellm/tree/main/enterprise) ::: +| Property | Details | +|----------|---------| +| Description | Log LLM Input/Output to a custom API endpoint | +| Logged Payload | `List[StandardLoggingPayload]` LiteLLM logs a list of [`StandardLoggingPayload` objects](https://docs.litellm.ai/docs/proxy/logging_spec) to your endpoint | + + + Use this if you: - Want to use custom callbacks written in a non Python programming language - Want your callbacks to run on a different microservice -#### Step 1. Create your generic logging API endpoint +#### Usage -Set up a generic API endpoint that can receive data in JSON format. The data will be included within a "data" field. +1. Set `success_callback: ["generic_api"]` on litellm config.yaml -Your server should support the following Request format: - -```shell -curl --location https://your-domain.com/log-event \ - --request POST \ - --header "Content-Type: application/json" \ - --data '{ - "data": { - "id": "chatcmpl-8sgE89cEQ4q9biRtxMvDfQU1O82PT", - "call_type": "acompletion", - "cache_hit": "None", - "startTime": "2024-02-15 16:18:44.336280", - "endTime": "2024-02-15 16:18:45.045539", - "model": "gpt-3.5-turbo", - "user": "ishaan-2", - "modelParameters": "{'temperature': 0.7, 'max_tokens': 10, 'user': 'ishaan-2', 'extra_body': {}}", - "messages": "[{'role': 'user', 'content': 'This is a test'}]", - "response": "ModelResponse(id='chatcmpl-8sgE89cEQ4q9biRtxMvDfQU1O82PT', choices=[Choices(finish_reason='length', index=0, message=Message(content='Great! How can I assist you with this test', role='assistant'))], created=1708042724, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=Usage(completion_tokens=10, prompt_tokens=11, total_tokens=21))", - "usage": "Usage(completion_tokens=10, prompt_tokens=11, total_tokens=21)", - "metadata": "{}", - "cost": "3.65e-05" - } - }' -``` - -Reference FastAPI Python Server - -Here's a reference FastAPI Server that is compatible with LiteLLM Proxy: - -```python -# this is an example endpoint to receive data from litellm -from fastapi import FastAPI, HTTPException, Request - -app = FastAPI() - - -@app.post("/log-event") -async def log_event(request: Request): - try: - print("Received /log-event request") - # Assuming the incoming request has JSON data - data = await request.json() - print("Received request data:") - print(data) - - # Your additional logic can go here - # For now, just printing the received data - - return {"message": "Request received successfully"} - except Exception as e: - print(f"Error processing request: {str(e)}") - import traceback - - traceback.print_exc() - raise HTTPException(status_code=500, detail="Internal Server Error") - - -if __name__ == "__main__": - import uvicorn - uvicorn.run(app, host="127.0.0.1", port=4000) -``` - -#### Step 2. Set your `GENERIC_LOGGER_ENDPOINT` to the endpoint + route we should send callback logs to - -```shell -os.environ["GENERIC_LOGGER_ENDPOINT"] = "http://localhost:4000/log-event" -``` - -#### Step 3. Create a `config.yaml` file and set `litellm_settings`: `success_callback` = ["generic"] - -Example litellm proxy config.yaml - -```yaml +```yaml showLineNumbers title="litellm config.yaml" model_list: - - model_name: gpt-3.5-turbo + - model_name: openai/gpt-4o litellm_params: - model: gpt-3.5-turbo + model: openai/gpt-4o + api_key: os.environ/OPENAI_API_KEY + litellm_settings: - success_callback: ["generic"] + success_callback: ["generic_api"] ``` -Start the LiteLLM Proxy and make a test request to verify the logs reached your callback API +2. Set Environment Variables for the custom API endpoint + +| Environment Variable | Details | Required | +|----------|---------|----------| +| `GENERIC_LOGGER_ENDPOINT` | The endpoint + route we should send callback logs to | Yes | +| `GENERIC_LOGGER_HEADERS` | Optional: Set headers to be sent to the custom API endpoint | No, this is optional | + +```shell showLineNumbers title=".env" +GENERIC_LOGGER_ENDPOINT="https://webhook-test.com/30343bc33591bc5e6dc44217ceae3e0a" + + +# Optional: Set headers to be sent to the custom API endpoint +GENERIC_LOGGER_HEADERS="Authorization=Bearer+ Use Google ADK with LiteLLM Python SDK, LiteLLM Proxy +
+ + +This tutorial shows you how to create intelligent agents using Agent Development Kit (ADK) with support for multiple Large Language Model (LLM) providers with LiteLLM. + + + +## Overview + +ADK (Agent Development Kit) allows you to build intelligent agents powered by LLMs. By integrating with LiteLLM, you can: + +- Use multiple LLM providers (OpenAI, Anthropic, Google, etc.) +- Switch easily between models from different providers +- Connect to a LiteLLM proxy for centralized model management + +## Prerequisites + +- Python environment setup +- API keys for model providers (OpenAI, Anthropic, Google AI Studio) +- Basic understanding of LLMs and agent concepts + +## Installation + +```bash showLineNumbers title="Install dependencies" +pip install google-adk litellm +``` + +## 1. Setting Up Environment + +First, import the necessary libraries and set up your API keys: + +```python showLineNumbers title="Setup environment and API keys" +import os +import asyncio +from google.adk.agents import Agent +from google.adk.models.lite_llm import LiteLlm # For multi-model support +from google.adk.sessions import InMemorySessionService +from google.adk.runners import Runner +from google.genai import types +import litellm # Import for proxy configuration + +# Set your API keys +os.environ["GOOGLE_API_KEY"] = "your-google-api-key" # For Gemini models +os.environ["OPENAI_API_KEY"] = "your-openai-api-key" # For OpenAI models +os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key" # For Claude models + +# Define model constants for cleaner code +MODEL_GEMINI_PRO = "gemini-1.5-pro" +MODEL_GPT_4O = "openai/gpt-4o" +MODEL_CLAUDE_SONNET = "anthropic/claude-3-sonnet-20240229" +``` + +## 2. Define a Simple Tool + +Create a tool that your agent can use: + +```python showLineNumbers title="Weather tool implementation" +def get_weather(city: str) -> dict: + """Retrieves the current weather report for a specified city. + + Args: + city (str): The name of the city (e.g., "New York", "London", "Tokyo"). + + Returns: + dict: A dictionary containing the weather information. + Includes a 'status' key ('success' or 'error'). + If 'success', includes a 'report' key with weather details. + If 'error', includes an 'error_message' key. + """ + print(f"Tool: get_weather called for city: {city}") + + # Mock weather data + mock_weather_db = { + "newyork": {"status": "success", "report": "The weather in New York is sunny with a temperature of 25°C."}, + "london": {"status": "success", "report": "It's cloudy in London with a temperature of 15°C."}, + "tokyo": {"status": "success", "report": "Tokyo is experiencing light rain and a temperature of 18°C."}, + } + + city_normalized = city.lower().replace(" ", "") + + if city_normalized in mock_weather_db: + return mock_weather_db[city_normalized] + else: + return {"status": "error", "error_message": f"Sorry, I don't have weather information for '{city}'."} +``` + +## 3. Helper Function for Agent Interaction + +Create a helper function to facilitate agent interaction: + +```python showLineNumbers title="Agent interaction helper function" +async def call_agent_async(query: str, runner, user_id, session_id): + """Sends a query to the agent and prints the final response.""" + print(f"\n>>> User Query: {query}") + + # Prepare the user's message in ADK format + content = types.Content(role='user', parts=[types.Part(text=query)]) + + final_response_text = "Agent did not produce a final response." + + # Execute the agent and find the final response + async for event in runner.run_async( + user_id=user_id, + session_id=session_id, + new_message=content + ): + if event.is_final_response(): + if event.content and event.content.parts: + final_response_text = event.content.parts[0].text + break + + print(f"<<< Agent Response: {final_response_text}") +``` + +## 4. Using Different Model Providers with ADK + +### 4.1 Using OpenAI Models + +```python showLineNumbers title="OpenAI model implementation" +# Create an agent powered by OpenAI's GPT model +weather_agent_gpt = Agent( + name="weather_agent_gpt", + model=LiteLlm(model=MODEL_GPT_4O), # Use OpenAI's GPT model + description="Provides weather information using OpenAI's GPT.", + instruction="You are a helpful weather assistant powered by GPT-4o. " + "Use the 'get_weather' tool for city weather requests. " + "Present information clearly.", + tools=[get_weather], +) + +# Set up session and runner +session_service_gpt = InMemorySessionService() +session_gpt = session_service_gpt.create_session( + app_name="weather_app", + user_id="user_1", + session_id="session_gpt" +) + +runner_gpt = Runner( + agent=weather_agent_gpt, + app_name="weather_app", + session_service=session_service_gpt +) + +# Test the GPT agent +async def test_gpt_agent(): + print("\n--- Testing GPT Agent ---") + await call_agent_async( + "What's the weather in London?", + runner=runner_gpt, + user_id="user_1", + session_id="session_gpt" + ) + +# Execute the conversation with the GPT agent +await test_gpt_agent() + +# Or if running as a standard Python script: +# if __name__ == "__main__": +# asyncio.run(test_gpt_agent()) +``` + +### 4.2 Using Anthropic Models + +```python showLineNumbers title="Anthropic model implementation" +# Create an agent powered by Anthropic's Claude model +weather_agent_claude = Agent( + name="weather_agent_claude", + model=LiteLlm(model=MODEL_CLAUDE_SONNET), # Use Anthropic's Claude model + description="Provides weather information using Anthropic's Claude.", + instruction="You are a helpful weather assistant powered by Claude Sonnet. " + "Use the 'get_weather' tool for city weather requests. " + "Present information clearly.", + tools=[get_weather], +) + +# Set up session and runner +session_service_claude = InMemorySessionService() +session_claude = session_service_claude.create_session( + app_name="weather_app", + user_id="user_1", + session_id="session_claude" +) + +runner_claude = Runner( + agent=weather_agent_claude, + app_name="weather_app", + session_service=session_service_claude +) + +# Test the Claude agent +async def test_claude_agent(): + print("\n--- Testing Claude Agent ---") + await call_agent_async( + "What's the weather in Tokyo?", + runner=runner_claude, + user_id="user_1", + session_id="session_claude" + ) + +# Execute the conversation with the Claude agent +await test_claude_agent() + +# Or if running as a standard Python script: +# if __name__ == "__main__": +# asyncio.run(test_claude_agent()) +``` + +### 4.3 Using Google's Gemini Models + +```python showLineNumbers title="Gemini model implementation" +# Create an agent powered by Google's Gemini model +weather_agent_gemini = Agent( + name="weather_agent_gemini", + model=MODEL_GEMINI_PRO, # Use Gemini model directly (no LiteLlm wrapper needed) + description="Provides weather information using Google's Gemini.", + instruction="You are a helpful weather assistant powered by Gemini Pro. " + "Use the 'get_weather' tool for city weather requests. " + "Present information clearly.", + tools=[get_weather], +) + +# Set up session and runner +session_service_gemini = InMemorySessionService() +session_gemini = session_service_gemini.create_session( + app_name="weather_app", + user_id="user_1", + session_id="session_gemini" +) + +runner_gemini = Runner( + agent=weather_agent_gemini, + app_name="weather_app", + session_service=session_service_gemini +) + +# Test the Gemini agent +async def test_gemini_agent(): + print("\n--- Testing Gemini Agent ---") + await call_agent_async( + "What's the weather in New York?", + runner=runner_gemini, + user_id="user_1", + session_id="session_gemini" + ) + +# Execute the conversation with the Gemini agent +await test_gemini_agent() + +# Or if running as a standard Python script: +# if __name__ == "__main__": +# asyncio.run(test_gemini_agent()) +``` + +## 5. Using LiteLLM Proxy with ADK + +LiteLLM proxy provides a unified API endpoint for multiple models, simplifying deployment and centralized management. + +Required settings for using litellm proxy + +| Variable | Description | +|----------|-------------| +| `LITELLM_PROXY_API_KEY` | The API key for the LiteLLM proxy | +| `LITELLM_PROXY_API_BASE` | The base URL for the LiteLLM proxy | +| `USE_LITELLM_PROXY` or `litellm.use_litellm_proxy` | When set to True, your request will be sent to litellm proxy. | + +```python showLineNumbers title="LiteLLM proxy integration" +# Set your LiteLLM Proxy credentials as environment variables +os.environ["LITELLM_PROXY_API_KEY"] = "your-litellm-proxy-api-key" +os.environ["LITELLM_PROXY_API_BASE"] = "your-litellm-proxy-url" # e.g., "http://localhost:4000" +# Enable the use_litellm_proxy flag +litellm.use_litellm_proxy = True + +# Create a proxy-enabled agent (using environment variables) +weather_agent_proxy_env = Agent( + name="weather_agent_proxy_env", + model=LiteLlm(model="gpt-4o"), # this will call the `gpt-4o` model on LiteLLM proxy + description="Provides weather information using a model from LiteLLM proxy.", + instruction="You are a helpful weather assistant. " + "Use the 'get_weather' tool for city weather requests. " + "Present information clearly.", + tools=[get_weather], +) + +# Set up session and runner +session_service_proxy_env = InMemorySessionService() +session_proxy_env = session_service_proxy_env.create_session( + app_name="weather_app", + user_id="user_1", + session_id="session_proxy_env" +) + +runner_proxy_env = Runner( + agent=weather_agent_proxy_env, + app_name="weather_app", + session_service=session_service_proxy_env +) + +# Test the proxy-enabled agent (environment variables method) +async def test_proxy_env_agent(): + print("\n--- Testing Proxy-enabled Agent (Environment Variables) ---") + await call_agent_async( + "What's the weather in London?", + runner=runner_proxy_env, + user_id="user_1", + session_id="session_proxy_env" + ) + +# Execute the conversation +await test_proxy_env_agent() +``` diff --git a/docs/my-website/docs/tutorials/gradio_integration.md b/docs/my-website/docs/tutorials/gradio_integration.md index 4854fc2ac95..021815d9372 100644 --- a/docs/my-website/docs/tutorials/gradio_integration.md +++ b/docs/my-website/docs/tutorials/gradio_integration.md @@ -30,7 +30,7 @@ def inference(message, history): yield partial_message except Exception as e: print("Exception encountered:", str(e)) - yield f"An Error occured please 'Clear' the error and try your question again" + yield f"An Error occurred please 'Clear' the error and try your question again" ``` ### Define Chat Interface diff --git a/docs/my-website/docs/tutorials/instructor.md b/docs/my-website/docs/tutorials/instructor.md index d972aff9151..073215b47be 100644 --- a/docs/my-website/docs/tutorials/instructor.md +++ b/docs/my-website/docs/tutorials/instructor.md @@ -1,80 +1,73 @@ -# Instructor - Function Calling +# Instructor -Use LiteLLM with [jxnl's instructor library](https://github.com/jxnl/instructor) for function calling in prod. +Combine LiteLLM with [jxnl's instructor library](https://github.com/jxnl/instructor) for more robust structured outputs. Outputs are automatically validated into Pydantic types and validation errors are provided back to the model to increase the chance of a successful response in the retries. -## Usage +## Usage (Sync) ```python -import os - import instructor from litellm import completion from pydantic import BaseModel -os.environ["LITELLM_LOG"] = "DEBUG" # 👈 print DEBUG LOGS client = instructor.from_litellm(completion) -# import dotenv -# dotenv.load_dotenv() - -class UserDetail(BaseModel): +class User(BaseModel): name: str age: int -user = client.chat.completions.create( - model="gpt-4o-mini", - response_model=UserDetail, - messages=[ - {"role": "user", "content": "Extract Jason is 25 years old"}, - ], -) +def extract_user(text: str): + return client.chat.completions.create( + model="gpt-4o-mini", + response_model=User, + messages=[ + {"role": "user", "content": text}, + ], + max_retries=3, + ) -assert isinstance(user, UserDetail) +user = extract_user("Jason is 25 years old") + +assert isinstance(user, User) assert user.name == "Jason" assert user.age == 25 - -print(f"user: {user}") +print(f"{user=}") ``` -## Async Calls +## Usage (Async) ```python import asyncio + import instructor -from litellm import Router +from litellm import acompletion from pydantic import BaseModel -aclient = instructor.patch( - Router( - model_list=[ - { - "model_name": "gpt-4o-mini", - "litellm_params": {"model": "gpt-4o-mini"}, - } - ], - default_litellm_params={"acompletion": True}, # 👈 IMPORTANT - tells litellm to route to async completion function. - ) -) + +client = instructor.from_litellm(acompletion) -class UserExtract(BaseModel): +class User(BaseModel): name: str age: int -async def main(): - model = await aclient.chat.completions.create( +async def extract(text: str) -> User: + return await client.chat.completions.create( model="gpt-4o-mini", - response_model=UserExtract, + response_model=User, messages=[ - {"role": "user", "content": "Extract jason is 25 years old"}, + {"role": "user", "content": text}, ], + max_retries=3, ) - print(f"model: {model}") +user = asyncio.run(extract("Alice is 30 years old")) -asyncio.run(main()) -``` \ No newline at end of file +assert isinstance(user, User) +assert user.name == "Alice" +assert user.age == 30 +print(f"{user=}") +``` diff --git a/docs/my-website/docs/tutorials/litellm_proxy_aporia.md b/docs/my-website/docs/tutorials/litellm_proxy_aporia.md index 3b5bada2bce..143512f99c2 100644 --- a/docs/my-website/docs/tutorials/litellm_proxy_aporia.md +++ b/docs/my-website/docs/tutorials/litellm_proxy_aporia.md @@ -2,9 +2,9 @@ import Image from '@theme/IdealImage'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Use LiteLLM AI Gateway with Aporia Guardrails +# Aporia Guardrails with LiteLLM Gateway -In this tutorial we will use LiteLLM Proxy with Aporia to detect PII in requests and profanity in responses +In this tutorial we will use LiteLLM AI Gateway with Aporia to detect PII in requests and profanity in responses ## 1. Setup guardrails on Aporia diff --git a/docs/my-website/docs/tutorials/lm_evaluation_harness.md b/docs/my-website/docs/tutorials/lm_evaluation_harness.md index c28f2dac775..01fdb4b304c 100644 --- a/docs/my-website/docs/tutorials/lm_evaluation_harness.md +++ b/docs/my-website/docs/tutorials/lm_evaluation_harness.md @@ -39,7 +39,7 @@ pip install openai==0.28.01 **Step 3: Set OpenAI API Base & Key** ```shell -$ export OPENAI_API_BASE=http://0.0.0.0:8000 +$ export OPENAI_BASE_URL=http://0.0.0.0:8000 ``` LM Harness requires you to set an OpenAI API key `OPENAI_API_SECRET_KEY` for running benchmarks @@ -74,7 +74,7 @@ $ litellm --model huggingface/bigcode/starcoder **Step 2: Set OpenAI API Base & Key** ```shell -$ export OPENAI_API_BASE=http://0.0.0.0:8000 +$ export OPENAI_BASE_URL=http://0.0.0.0:8000 ``` Set this to anything since the proxy has the credentials @@ -93,12 +93,12 @@ cd FastEval **Set API Base on FastEval** -On FastEval make the following **2 line code change** to set `OPENAI_API_BASE` +On FastEval make the following **2 line code change** to set `OPENAI_BASE_URL` https://github.com/FastEval/FastEval/pull/90/files ```python try: - api_base = os.environ["OPENAI_API_BASE"] #changed: read api base from .env + api_base = os.environ["OPENAI_BASE_URL"] #changed: read api base from .env if api_base == None: api_base = "https://api.openai.com/v1" response = await self.reply_two_attempts_with_different_max_new_tokens( @@ -130,7 +130,7 @@ $ litellm --model huggingface/bigcode/starcoder **Step 2: Set OpenAI API Base & Key** ```shell -$ export OPENAI_API_BASE=http://0.0.0.0:8000 +$ export OPENAI_BASE_URL=http://0.0.0.0:8000 ``` **Step 3 Run with FLASK** diff --git a/docs/my-website/docs/tutorials/msft_sso.md b/docs/my-website/docs/tutorials/msft_sso.md new file mode 100644 index 00000000000..f7ad6440f2e --- /dev/null +++ b/docs/my-website/docs/tutorials/msft_sso.md @@ -0,0 +1,162 @@ +import Image from '@theme/IdealImage'; + +# Microsoft SSO: Sync Groups, Members with LiteLLM + +Sync Microsoft SSO Groups, Members with LiteLLM Teams. + ++ Expose and use MCP servers through LiteLLM +
+ +## UI view total usage after 1M+ logs + +This release brings the ability to view total usage analytics even after exceeding 1M+ logs in your database. We've implemented a scalable architecture that stores only aggregate usage data, resulting in significantly more efficient queries and reduced database CPU utilization. + + ++ View total usage after 1M+ logs +
+ + +- How this works: + - We now aggregate usage data into a dedicated DailyUserSpend table, significantly reducing query load and CPU usage even beyond 1M+ logs. + +- Daily Spend Breakdown API: + + - Retrieve granular daily usage data (by model, provider, and API key) with a single endpoint. + Example Request: + + ```shell title="Daily Spend Breakdown API" showLineNumbers + curl -L -X GET 'http://localhost:4000/user/daily/activity?start_date=2025-03-20&end_date=2025-03-27' \ + -H 'Authorization: Bearer sk-...' + ``` + + ```json title="Daily Spend Breakdown API Response" showLineNumbers + { + "results": [ + { + "date": "2025-03-27", + "metrics": { + "spend": 0.0177072, + "prompt_tokens": 111, + "completion_tokens": 1711, + "total_tokens": 1822, + "api_requests": 11 + }, + "breakdown": { + "models": { + "gpt-4o-mini": { + "spend": 1.095e-05, + "prompt_tokens": 37, + "completion_tokens": 9, + "total_tokens": 46, + "api_requests": 1 + }, + "providers": { "openai": { ... }, "azure_ai": { ... } }, + "api_keys": { "3126b6eaf1...": { ... } } + } + } + ], + "metadata": { + "total_spend": 0.7274667, + "total_prompt_tokens": 280990, + "total_completion_tokens": 376674, + "total_api_requests": 14 + } + } + ``` + + + + +## New Models / Updated Models +- Support for Vertex AI gemini-2.0-flash-lite & Google AI Studio gemini-2.0-flash-lite [PR](https://github.com/BerriAI/litellm/pull/9523) +- Support for Vertex AI Fine-Tuned LLMs [PR](https://github.com/BerriAI/litellm/pull/9542) +- Nova Canvas image generation support [PR](https://github.com/BerriAI/litellm/pull/9525) +- OpenAI gpt-4o-transcribe support [PR](https://github.com/BerriAI/litellm/pull/9517) +- Added new Vertex AI text embedding model [PR](https://github.com/BerriAI/litellm/pull/9476) + +## LLM Translation +- OpenAI Web Search Tool Call Support [PR](https://github.com/BerriAI/litellm/pull/9465) +- Vertex AI topLogprobs support [PR](https://github.com/BerriAI/litellm/pull/9518) +- Support for sending images and video to Vertex AI multimodal embedding [Doc](https://docs.litellm.ai/docs/providers/vertex#multi-modal-embeddings) +- Support litellm.api_base for Vertex AI + Gemini across completion, embedding, image_generation [PR](https://github.com/BerriAI/litellm/pull/9516) +- Bug fix for returning `response_cost` when using litellm python SDK with LiteLLM Proxy [PR](https://github.com/BerriAI/litellm/commit/6fd18651d129d606182ff4b980e95768fc43ca3d) +- Support for `max_completion_tokens` on Mistral API [PR](https://github.com/BerriAI/litellm/pull/9606) +- Refactored Vertex AI passthrough routes - fixes unpredictable behaviour with auto-setting default_vertex_region on router model add [PR](https://github.com/BerriAI/litellm/pull/9467) + +## Spend Tracking Improvements +- Log 'api_base' on spend logs [PR](https://github.com/BerriAI/litellm/pull/9509) +- Support for Gemini audio token cost tracking [PR](https://github.com/BerriAI/litellm/pull/9535) +- Fixed OpenAI audio input token cost tracking [PR](https://github.com/BerriAI/litellm/pull/9535) + +## UI + +### Model Management +- Allowed team admins to add/update/delete models on UI [PR](https://github.com/BerriAI/litellm/pull/9572) +- Added render supports_web_search on model hub [PR](https://github.com/BerriAI/litellm/pull/9469) + +### Request Logs +- Show API base and model ID on request logs [PR](https://github.com/BerriAI/litellm/pull/9572) +- Allow viewing keyinfo on request logs [PR](https://github.com/BerriAI/litellm/pull/9568) + +### Usage Tab +- Added Daily User Spend Aggregate view - allows UI Usage tab to work > 1m rows [PR](https://github.com/BerriAI/litellm/pull/9538) +- Connected UI to "LiteLLM_DailyUserSpend" spend table [PR](https://github.com/BerriAI/litellm/pull/9603) + +## Logging Integrations +- Fixed StandardLoggingPayload for GCS Pub Sub Logging Integration [PR](https://github.com/BerriAI/litellm/pull/9508) +- Track `litellm_model_name` on `StandardLoggingPayload` [Docs](https://docs.litellm.ai/docs/proxy/logging_spec#standardlogginghiddenparams) + +## Performance / Reliability Improvements +- LiteLLM Redis semantic caching implementation [PR](https://github.com/BerriAI/litellm/pull/9356) +- Gracefully handle exceptions when DB is having an outage [PR](https://github.com/BerriAI/litellm/pull/9533) +- Allow Pods to startup + passing /health/readiness when allow_requests_on_db_unavailable: True and DB is down [PR](https://github.com/BerriAI/litellm/pull/9569) + + +## General Improvements +- Support for exposing MCP tools on litellm proxy [PR](https://github.com/BerriAI/litellm/pull/9426) +- Support discovering Gemini, Anthropic, xAI models by calling their /v1/model endpoint [PR](https://github.com/BerriAI/litellm/pull/9530) +- Fixed route check for non-proxy admins on JWT auth [PR](https://github.com/BerriAI/litellm/pull/9454) +- Added baseline Prisma database migrations [PR](https://github.com/BerriAI/litellm/pull/9565) +- View all wildcard models on /model/info [PR](https://github.com/BerriAI/litellm/pull/9572) + + +## Security +- Bumped next from 14.2.21 to 14.2.25 in UI dashboard [PR](https://github.com/BerriAI/litellm/pull/9458) + +## Complete Git Diff + +[Here's the complete git diff](https://github.com/BerriAI/litellm/compare/v1.63.14-stable.patch1...v1.65.0-stable) diff --git a/docs/my-website/release_notes/v1.65.0/index.md b/docs/my-website/release_notes/v1.65.0/index.md new file mode 100644 index 00000000000..84276c997da --- /dev/null +++ b/docs/my-website/release_notes/v1.65.0/index.md @@ -0,0 +1,34 @@ +--- +title: v1.65.0 - Team Model Add - update +slug: v1.65.0 +date: 2025-03-28T10:00:00 +authors: + - name: Krrish Dholakia + title: CEO, LiteLLM + url: https://www.linkedin.com/in/krish-d/ + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8 + - name: Ishaan Jaffer + title: CTO, LiteLLM + url: https://www.linkedin.com/in/reffajnaahsi/ + image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg +tags: [management endpoints, team models, ui] +hide_table_of_contents: false +--- + +import Image from '@theme/IdealImage'; + +v1.65.0 updates the `/model/new` endpoint to prevent non-team admins from creating team models. + +This means that only proxy admins or team admins can create team models. + +## Additional Changes + +- Allows team admins to call `/model/update` to update team models. +- Allows team admins to call `/model/delete` to delete team models. +- Introduces new `user_models_only` param to `/v2/model/info` - only return models added by this user. + + +These changes enable team admins to add and manage models for their team on the LiteLLM UI + API. + + ++ Auto-sync groups and members from Azure Entra ID to LiteLLM +
+ +This release adds support for auto-syncing groups and members on Microsoft Entra ID with LiteLLM. This means that LiteLLM proxy administrators can spend less time managing teams and members and LiteLLM handles the following: + +- Auto-create teams that exist on Microsoft Entra ID +- Sync team members on Microsoft Entra ID with LiteLLM teams + +Get started with this [here](https://docs.litellm.ai/docs/tutorials/msft_sso) + + +## New Models / Updated Models + +- **xAI** + 1. Added reasoning_effort support for `xai/grok-3-mini-beta` [Get Started](https://docs.litellm.ai/docs/providers/xai#reasoning-usage) + 2. Added cost tracking for `xai/grok-3` models [PR](https://github.com/BerriAI/litellm/pull/9920) + +- **Hugging Face** + 1. Added inference providers support [Get Started](https://docs.litellm.ai/docs/providers/huggingface#serverless-inference-providers) + +- **Azure** + 1. Added azure/gpt-4o-realtime-audio cost tracking [PR](https://github.com/BerriAI/litellm/pull/9893) + +- **VertexAI** + 1. Added enterpriseWebSearch tool support [Get Started](https://docs.litellm.ai/docs/providers/vertex#grounding---web-search) + 2. Moved to only passing keys accepted by the Vertex AI response schema [PR](https://github.com/BerriAI/litellm/pull/8992) + +- **Google AI Studio** + 1. Added cost tracking for `gemini-2.5-pro` [PR](https://github.com/BerriAI/litellm/pull/9837) + 2. Fixed pricing for 'gemini/gemini-2.5-pro-preview-03-25' [PR](https://github.com/BerriAI/litellm/pull/9896) + 3. Fixed handling file_data being passed in [PR](https://github.com/BerriAI/litellm/pull/9786) + +- **Azure** + 1. Updated Azure Phi-4 pricing [PR](https://github.com/BerriAI/litellm/pull/9862) + 2. Added azure/gpt-4o-realtime-audio cost tracking [PR](https://github.com/BerriAI/litellm/pull/9893) + +- **Databricks** + 1. Removed reasoning_effort from parameters [PR](https://github.com/BerriAI/litellm/pull/9811) + 2. Fixed custom endpoint check for Databricks [PR](https://github.com/BerriAI/litellm/pull/9925) + +- **General** + 1. Added litellm.supports_reasoning() util to track if an llm supports reasoning [Get Started](https://docs.litellm.ai/docs/providers/anthropic#reasoning) + 2. Function Calling - Handle pydantic base model in message tool calls, handle tools = [], and support fake streaming on tool calls for meta.llama3-3-70b-instruct-v1:0 [PR](https://github.com/BerriAI/litellm/pull/9774) + 3. LiteLLM Proxy - Allow passing `thinking` param to litellm proxy via client sdk [PR](https://github.com/BerriAI/litellm/pull/9386) + 4. Fixed correctly translating 'thinking' param for litellm [PR](https://github.com/BerriAI/litellm/pull/9904) + + +## Spend Tracking Improvements +- **OpenAI, Azure** + 1. Realtime API Cost tracking with token usage metrics in spend logs [Get Started](https://docs.litellm.ai/docs/realtime) +- **Anthropic** + 1. Fixed Claude Haiku cache read pricing per token [PR](https://github.com/BerriAI/litellm/pull/9834) + 2. Added cost tracking for Claude responses with base_model [PR](https://github.com/BerriAI/litellm/pull/9897) + 3. Fixed Anthropic prompt caching cost calculation and trimmed logged message in db [PR](https://github.com/BerriAI/litellm/pull/9838) +- **General** + 1. Added token tracking and log usage object in spend logs [PR](https://github.com/BerriAI/litellm/pull/9843) + 2. Handle custom pricing at deployment level [PR](https://github.com/BerriAI/litellm/pull/9855) + + +## Management Endpoints / UI + +- **Test Key Tab** + 1. Added rendering of Reasoning content, ttft, usage metrics on test key page [PR](https://github.com/BerriAI/litellm/pull/9931) + ++ View input, output, reasoning tokens, ttft metrics. +
+- **Tag / Policy Management** + 1. Added Tag/Policy Management. Create routing rules based on request metadata. This allows you to enforce that requests with `tags="private"` only go to specific models. [Get Started](https://docs.litellm.ai/docs/tutorials/tag_management) + ++ Create and manage tags. +
+- **Redesigned Login Screen** + 1. Polished login screen [PR](https://github.com/BerriAI/litellm/pull/9778) +- **Microsoft SSO Auto-Sync** + 1. Added debug route to allow admins to debug SSO JWT fields [PR](https://github.com/BerriAI/litellm/pull/9835) + 2. Added ability to use MSFT Graph API to assign users to teams [PR](https://github.com/BerriAI/litellm/pull/9865) + 3. Connected litellm to Azure Entra ID Enterprise Application [PR](https://github.com/BerriAI/litellm/pull/9872) + 4. Added ability for admins to set `default_team_params` for when litellm SSO creates default teams [PR](https://github.com/BerriAI/litellm/pull/9895) + 5. Fixed MSFT SSO to use correct field for user email [PR](https://github.com/BerriAI/litellm/pull/9886) + 6. Added UI support for setting Default Team setting when litellm SSO auto creates teams [PR](https://github.com/BerriAI/litellm/pull/9918) +- **UI Bug Fixes** + 1. Prevented team, key, org, model numerical values changing on scrolling [PR](https://github.com/BerriAI/litellm/pull/9776) + 2. Instantly reflect key and team updates in UI [PR](https://github.com/BerriAI/litellm/pull/9825) + +## Logging / Guardrail Improvements + +- **Prometheus** + 1. Emit Key and Team Budget metrics on a cron job schedule [Get Started](https://docs.litellm.ai/docs/proxy/prometheus#initialize-budget-metrics-on-startup) + +## Security Fixes + +- Fixed [CVE-2025-0330](https://www.cve.org/CVERecord?id=CVE-2025-0330) - Leakage of Langfuse API keys in team exception handling [PR](https://github.com/BerriAI/litellm/pull/9830) +- Fixed [CVE-2024-6825](https://www.cve.org/CVERecord?id=CVE-2024-6825) - Remote code execution in post call rules [PR](https://github.com/BerriAI/litellm/pull/9826) + +## Helm + +- Added service annotations to litellm-helm chart [PR](https://github.com/BerriAI/litellm/pull/9840) +- Added extraEnvVars to the helm deployment [PR](https://github.com/BerriAI/litellm/pull/9292) + +## Demo + +Try this on the demo instance [today](https://docs.litellm.ai/docs/proxy/demo) + +## Complete Git Diff + +See the complete git diff since v1.65.4-stable, [here](https://github.com/BerriAI/litellm/releases/tag/v1.66.0-stable) + + diff --git a/docs/my-website/release_notes/v1.67.0-stable/index.md b/docs/my-website/release_notes/v1.67.0-stable/index.md new file mode 100644 index 00000000000..cb7938fce57 --- /dev/null +++ b/docs/my-website/release_notes/v1.67.0-stable/index.md @@ -0,0 +1,153 @@ +--- +title: v1.67.0-stable - SCIM Integration +slug: v1.67.0-stable +date: 2025-04-19T10:00:00 +authors: + - name: Krrish Dholakia + title: CEO, LiteLLM + url: https://www.linkedin.com/in/krish-d/ + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8 + - name: Ishaan Jaffer + title: CTO, LiteLLM + url: https://www.linkedin.com/in/reffajnaahsi/ + image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg + +tags: ["sso", "unified_file_id", "cost_tracking", "security"] +hide_table_of_contents: false +--- +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Key Highlights + +- **SCIM Integration**: Enables identity providers (Okta, Azure AD, OneLogin, etc.) to automate user and team (group) provisioning, updates, and deprovisioning +- **Team and Tag based usage tracking**: You can now see usage and spend by team and tag at 1M+ spend logs. +- **Unified Responses API**: Support for calling Anthropic, Gemini, Groq, etc. via OpenAI's new Responses API. + +Let's dive in. + +## SCIM Integration + +The request you would send to LiteLLM /chat/completions endpoint.
+