diff --git a/docs/my-website/docs/anthropic_unified.md b/docs/my-website/docs/anthropic_unified.md new file mode 100644 index 00000000000..71b9203399b --- /dev/null +++ b/docs/my-website/docs/anthropic_unified.md @@ -0,0 +1,92 @@ +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# [BETA] `/v1/messages` + +LiteLLM provides a BETA endpoint in the spec of Anthropic's `/v1/messages` endpoint. + +This currently just supports the Anthropic API. + +| Feature | Supported | Notes | +|-------|-------|-------| +| Cost Tracking | ✅ | | +| Logging | ✅ | works across all integrations | +| End-user Tracking | ✅ | | +| Streaming | ✅ | | +| Fallbacks | ✅ | between anthropic models | +| Loadbalancing | ✅ | between anthropic models | + +Planned improvement: +- Vertex AI Anthropic support +- Bedrock Anthropic support + +## Usage + + + + +1. Setup config.yaml + +```yaml +model_list: + - model_name: anthropic-claude + litellm_params: + model: claude-3-7-sonnet-latest +``` + +2. Start proxy + +```bash +litellm --config /path/to/config.yaml +``` + +3. Test it! + +```bash +curl -L -X POST 'http://0.0.0.0:4000/v1/messages' \ +-H 'content-type: application/json' \ +-H 'x-api-key: $LITELLM_API_KEY' \ +-H 'anthropic-version: 2023-06-01' \ +-d '{ + "model": "anthropic-claude", + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "List 5 important events in the XIX century" + } + ] + } + ], + "max_tokens": 4096 +}' +``` + + + +```python +from litellm.llms.anthropic.experimental_pass_through.messages.handler import anthropic_messages +import asyncio +import os + +# set env +os.environ["ANTHROPIC_API_KEY"] = "my-api-key" + +messages = [{"role": "user", "content": "Hello, can you tell me a short joke?"}] + +# Call the handler +async def call(): + response = await anthropic_messages( + messages=messages, + api_key=api_key, + model="claude-3-haiku-20240307", + max_tokens=100, + ) + +asyncio.run(call()) +``` + + + \ No newline at end of file diff --git a/docs/my-website/docs/completion/vision.md b/docs/my-website/docs/completion/vision.md index efb988b76f7..1e18109b3b9 100644 --- a/docs/my-website/docs/completion/vision.md +++ b/docs/my-website/docs/completion/vision.md @@ -189,4 +189,138 @@ Expected Response ``` - \ No newline at end of file + + + +## Explicitly specify image type + +If you have images without a mime-type, or if litellm is incorrectly inferring the mime type of your image (e.g. calling `gs://` url's with vertex ai), you can set this explicity via the `format` param. + +```python +"image_url": { + "url": "gs://my-gs-image", + "format": "image/jpeg" +} +``` + +LiteLLM will use this for any API endpoint, which supports specifying mime-type (e.g. anthropic/bedrock/vertex ai). + +For others (e.g. openai), it will be ignored. + + + + +```python +import os +from litellm import completion + +os.environ["ANTHROPIC_API_KEY"] = "your-api-key" + +# openai call +response = completion( + model = "claude-3-7-sonnet-latest", + messages=[ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "What’s in this image?" + }, + { + "type": "image_url", + "image_url": { + "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", + "format": "image/jpeg" + } + } + ] + } + ], +) + +``` + + + + +1. Define vision models on config.yaml + +```yaml +model_list: + - model_name: gpt-4-vision-preview # OpenAI gpt-4-vision-preview + litellm_params: + model: openai/gpt-4-vision-preview + api_key: os.environ/OPENAI_API_KEY + - model_name: llava-hf # Custom OpenAI compatible model + litellm_params: + model: openai/llava-hf/llava-v1.6-vicuna-7b-hf + api_base: http://localhost:8000 + api_key: fake-key + model_info: + supports_vision: True # set supports_vision to True so /model/info returns this attribute as True + +``` + +2. Run proxy server + +```bash +litellm --config config.yaml +``` + +3. Test it using the OpenAI Python SDK + + +```python +import os +from openai import OpenAI + +client = OpenAI( + api_key="sk-1234", # your litellm proxy api key +) + +response = client.chat.completions.create( + model = "gpt-4-vision-preview", # use model="llava-hf" to test your custom OpenAI endpoint + messages=[ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "What’s in this image?" + }, + { + "type": "image_url", + "image_url": { + "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", + "format": "image/jpeg" + } + } + ] + } + ], +) + +``` + + + + + + + + + +## Spec + +``` +"image_url": str + +OR + +"image_url": { + "url": "url OR base64 encoded str", + "detail": "openai-only param", + "format": "specify mime-type of image" +} +``` \ No newline at end of file diff --git a/docs/my-website/docs/providers/bedrock.md b/docs/my-website/docs/providers/bedrock.md index fccce12c8ee..bd2d4be1a4e 100644 --- a/docs/my-website/docs/providers/bedrock.md +++ b/docs/my-website/docs/providers/bedrock.md @@ -286,9 +286,12 @@ print(response) -## Usage - Function Calling +## Usage - Function Calling / Tool calling -LiteLLM uses Bedrock's Converse API for making tool calls +LiteLLM supports tool calling via Bedrock's Converse and Invoke API's. + + + ```python from litellm import completion @@ -333,6 +336,69 @@ assert isinstance( response.choices[0].message.tool_calls[0].function.arguments, str ) ``` + + + +1. Setup config.yaml + +```yaml +model_list: + - model_name: bedrock-claude-3-7 + litellm_params: + model: bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0 # for bedrock invoke, specify `bedrock/invoke/` +``` + +2. Start proxy + +```bash +litellm --config /path/to/config.yaml +``` + +3. Test it! + +```bash +curl http://0.0.0.0:4000/v1/chat/completions \ +-H "Content-Type: application/json" \ +-H "Authorization: Bearer $LITELLM_API_KEY" \ +-d '{ + "model": "bedrock-claude-3-7", + "messages": [ + { + "role": "user", + "content": "What'\''s the weather like in Boston today?" + } + ], + "tools": [ + { + "type": "function", + "function": { + "name": "get_current_weather", + "description": "Get the current weather in a given location", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "The city and state, e.g. San Francisco, CA" + }, + "unit": { + "type": "string", + "enum": ["celsius", "fahrenheit"] + } + }, + "required": ["location"] + } + } + } + ], + "tool_choice": "auto" +}' + +``` + + + + ## Usage - Vision @@ -492,6 +558,111 @@ Same as [Anthropic API response](../providers/anthropic#usage---thinking--reason ``` +## Usage - Structured Output / JSON mode + + + + +```python +from litellm import completion +import os +from pydantic import BaseModel + +# set env +os.environ["AWS_ACCESS_KEY_ID"] = "" +os.environ["AWS_SECRET_ACCESS_KEY"] = "" +os.environ["AWS_REGION_NAME"] = "" + +class CalendarEvent(BaseModel): + name: str + date: str + participants: list[str] + +class EventsList(BaseModel): + events: list[CalendarEvent] + +response = completion( + model="bedrock/anthropic.claude-3-7-sonnet-20250219-v1:0", # specify invoke via `bedrock/invoke/anthropic.claude-3-7-sonnet-20250219-v1:0` + response_format=EventsList, + messages=[ + {"role": "system", "content": "You are a helpful assistant designed to output JSON."}, + {"role": "user", "content": "Who won the world series in 2020?"} + ], +) +print(response.choices[0].message.content) +``` + + + +1. Setup config.yaml + +```yaml +model_list: + - model_name: bedrock-claude-3-7 + litellm_params: + model: bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0 # specify invoke via `bedrock/invoke/` + aws_access_key_id: os.environ/CUSTOM_AWS_ACCESS_KEY_ID + aws_secret_access_key: os.environ/CUSTOM_AWS_SECRET_ACCESS_KEY + aws_region_name: os.environ/CUSTOM_AWS_REGION_NAME +``` + +2. Start proxy + +```bash +litellm --config /path/to/config.yaml +``` + +3. Test it! + +```bash +curl http://0.0.0.0:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $LITELLM_KEY" \ + -d '{ + "model": "bedrock-claude-3-7", + "messages": [ + { + "role": "system", + "content": "You are a helpful assistant designed to output JSON." + }, + { + "role": "user", + "content": "Who won the worlde series in 2020?" + } + ], + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "math_reasoning", + "description": "reason about maths", + "schema": { + "type": "object", + "properties": { + "steps": { + "type": "array", + "items": { + "type": "object", + "properties": { + "explanation": { "type": "string" }, + "output": { "type": "string" } + }, + "required": ["explanation", "output"], + "additionalProperties": false + } + }, + "final_answer": { "type": "string" } + }, + "required": ["steps", "final_answer"], + "additionalProperties": false + }, + "strict": true + } + } + }' +``` + + + ## Usage - Bedrock Guardrails Example of using [Bedrock Guardrails with LiteLLM](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-use-converse-api.html) diff --git a/docs/my-website/docs/proxy/logging_spec.md b/docs/my-website/docs/proxy/logging_spec.md index 86ba907373d..7da937e565a 100644 --- a/docs/my-website/docs/proxy/logging_spec.md +++ b/docs/my-website/docs/proxy/logging_spec.md @@ -78,6 +78,7 @@ Inherits from `StandardLoggingUserAPIKeyMetadata` and adds: | `api_base` | `Optional[str]` | Optional API base URL | | `response_cost` | `Optional[str]` | Optional response cost | | `additional_headers` | `Optional[StandardLoggingAdditionalHeaders]` | Additional headers | +| `batch_models` | `Optional[List[str]]` | Only set for Batches API. Lists the models used for cost calculation | ## StandardLoggingModelInformation diff --git a/docs/my-website/release_notes/v1.61.20-stable/index.md b/docs/my-website/release_notes/v1.61.20-stable/index.md index 4dda52e6320..132c1aa3183 100644 --- a/docs/my-website/release_notes/v1.61.20-stable/index.md +++ b/docs/my-website/release_notes/v1.61.20-stable/index.md @@ -20,12 +20,6 @@ import Image from '@theme/IdealImage'; # v1.61.20-stable -:::info - -`v1.61.20-stable` will be live on 2025-02-04. - -::: - These are the changes since `v1.61.13-stable`. This release is primarily focused on: diff --git a/docs/my-website/release_notes/v1.63.2-stable/index.md b/docs/my-website/release_notes/v1.63.2-stable/index.md new file mode 100644 index 00000000000..427d9e6243b --- /dev/null +++ b/docs/my-website/release_notes/v1.63.2-stable/index.md @@ -0,0 +1,92 @@ +--- +title: v1.63.2-stable +slug: v1.63.2-stable +date: 2025-03-08T10:00:00 +authors: + - name: Krrish Dholakia + title: CEO, LiteLLM + url: https://www.linkedin.com/in/krish-d/ + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1743638400&v=beta&t=39KOXMUFedvukiWWVPHf3qI45fuQD7lNglICwN31DrI + - name: Ishaan Jaffer + title: CTO, LiteLLM + url: https://www.linkedin.com/in/reffajnaahsi/ + image_url: https://media.licdn.com/dms/image/v2/D4D03AQGiM7ZrUwqu_Q/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1675971026692?e=1741824000&v=beta&t=eQnRdXPJo4eiINWTZARoYTfqh064pgZ-E21pQTSy8jc +tags: [llm translation, thinking, reasoning_content, claude-3-7-sonnet] +hide_table_of_contents: false +--- + + +These are the changes since `v1.61.20-stable`. + +This release is primarily focused on: +- LLM Translation improvements (more `thinking` content improvements) +- UI improvements (Error logs now shown on UI) + +## Demo Instance + +Here's a Demo Instance to test changes: +- Instance: https://demo.litellm.ai/ +- Login Credentials: + - Username: admin + - Password: sk-1234 + + +## New Models / Updated Models + +1. Add `supports_pdf_input` for specific Bedrock Claude models [PR](https://github.com/BerriAI/litellm/commit/f63cf0030679fe1a43d03fb196e815a0f28dae92) +2. Add pricing for amazon `eu` models [PR](https://github.com/BerriAI/litellm/commits/main/model_prices_and_context_window.json) +3. Fix Azure O1 mini pricing [PR](https://github.com/BerriAI/litellm/commit/52de1949ef2f76b8572df751f9c868a016d4832c) + +## LLM Translation + +1. Support `/openai/` passthrough for Assistant endpoints. [Get Started](https://docs.litellm.ai/docs/pass_through/openai_passthrough) +2. Bedrock Claude - fix tool calling transformation on invoke route. [Get Started](../../docs/providers/bedrock#usage---function-calling--tool-calling) +3. Bedrock Claude - response_format support for claude on invoke route. [Get Started](../../docs/providers/bedrock#usage---structured-output--json-mode) +4. Bedrock - pass `description` if set in response_format. [Get Started](../../docs/providers/bedrock#usage---structured-output--json-mode) +5. Bedrock - Fix passing response_format: {"type": "text"}. [PR](https://github.com/BerriAI/litellm/commit/c84b489d5897755139aa7d4e9e54727ebe0fa540) +6. OpenAI - Handle sending image_url as str to openai. [Get Started](https://docs.litellm.ai/docs/completion/vision) +7. Deepseek - return 'reasoning_content' missing on streaming. [Get Started](https://docs.litellm.ai/docs/reasoning_content) +8. Caching - Support caching on reasoning content. [Get Started](https://docs.litellm.ai/docs/proxy/caching) +9. Bedrock - handle thinking blocks in assistant message. [Get Started](https://docs.litellm.ai/docs/providers/bedrock#usage---thinking--reasoning-content) +10. Anthropic - Return `signature` on streaming. [Get Started](https://docs.litellm.ai/docs/providers/bedrock#usage---thinking--reasoning-content) +- Note: We've also migrated from `signature_delta` to `signature`. [Read more](https://docs.litellm.ai/release_notes/v1.63.0) +11. Support format param for specifying image type. [Get Started](../../docs/completion/vision.md#explicitly-specify-image-type) +12. Anthropic - `/v1/messages` endpoint - `thinking` param support. [Get Started](../../docs/anthropic_unified.md) +- Note: this refactors the [BETA] unified `/v1/messages` endpoint, to just work for the Anthropic API. +13. Vertex AI - handle $id in response schema when calling vertex ai. [Get Started](https://docs.litellm.ai/docs/providers/vertex#json-schema) + +## Spend Tracking Improvements + +1. Batches API - Fix cost calculation to run on retrieve_batch. [Get Started](https://docs.litellm.ai/docs/batches) +2. Batches API - Log batch models in spend logs / standard logging payload. [ADD DOCS - Add to standard logging doc](https://docs.litellm.ai/docs/proxy/logging_spec) + +## Management Endpoints / UI + +1. Virtual Keys Page + - Allow team/org filters to be searchable on the Create Key Page + - Add created_by and updated_by fields to Keys table + - Show 'user_email' on key table + - Show 100 Keys Per Page, Use full height, increase width of key alias +2. Logs Page + - Show Error Logs on LiteLLM UI + - Allow Internal Users to View their own logs +3. Internal Users Page + - Allow admin to control default model access for internal users +7. Fix session handling with cookies + +## Logging / Guardrail Integrations + +1. Fix prometheus metrics w/ custom metrics, when keys containing team_id make requests. [PR](https://github.com/BerriAI/litellm/pull/8935) + +## Performance / Loadbalancing / Reliability improvements + +1. Cooldowns - Support cooldowns on models called with client side credentials. [Get Started](https://docs.litellm.ai/docs/proxy/clientside_auth#pass-user-llm-api-keys--api-base) +2. Tag-based Routing - ensures tag-based routing across all endpoints (`/embeddings`, `/image_generation`, etc.). [Get Started](https://docs.litellm.ai/docs/proxy/tag_routing) + +## General Proxy Improvements + +1. Raise BadRequestError when unknown model passed in request +2. Enforce model access restrictions on Azure OpenAI proxy route +3. Reliability fix - Handle emoji’s in text - fix orjson error +4. Model Access Patch - don't overwrite litellm.anthropic_models when running auth checks +5. Enable setting timezone information in docker image diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 9c3b227ebe5..51e0aa774bc 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -288,6 +288,7 @@ const sidebars = { }, "text_completion", "embedding/supported_embedding", + "anthropic_unified", { type: "category", label: "Image",