mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
Merge upstream/main into litellm_fix_ci_failures
Resolves merge conflicts from 121 commits of upstream drift. Took upstream/main versions for all 6 conflicted files: - litellm/a2a_protocol/main.py - litellm/batches/main.py - litellm/llms/openrouter/image_edit/transformation.py - litellm/proxy/guardrails/guardrail_hooks/azure/prompt_shield.py - litellm/proxy/guardrails/guardrail_hooks/azure/text_moderation.py - litellm/proxy/management_endpoints/cost_tracking_settings.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Harshit28j
commit
8bcbd4a7af
753 changed files with 17468 additions and 6930 deletions
|
|
@ -212,6 +212,8 @@ When opening issues or pull requests, follow these templates:
|
|||
|
||||
Using helpers like `supports_reasoning` (which read from `model_prices_and_context_window.json` / `get_model_info`) allows future model updates to "just work" without code changes.
|
||||
|
||||
9. **Never close HTTP/SDK clients on cache eviction**: Do not add `close()`, `aclose()`, or `create_task(close_fn())` inside `LLMClientCache._remove_key()` or any cache eviction path. Evicted clients may still be held by in-flight requests; closing them causes `RuntimeError: Cannot send a request, as the client has been closed.` in production after the cache TTL (1 hour) expires. Connection cleanup is handled at shutdown by `close_litellm_async_clients()`. See PR #22247 for the full incident history.
|
||||
|
||||
## HELPFUL RESOURCES
|
||||
|
||||
- Main documentation: https://docs.litellm.ai/
|
||||
|
|
|
|||
|
|
@ -116,6 +116,9 @@ LiteLLM is a unified interface for 100+ LLM providers with two main components:
|
|||
- Optional features enabled via environment variables
|
||||
- Separate licensing and authentication for enterprise features
|
||||
|
||||
### HTTP Client Cache Safety
|
||||
- **Never close HTTP/SDK clients on cache eviction.** `LLMClientCache._remove_key()` must not call `close()`/`aclose()` on evicted clients — they may still be used by in-flight requests. Doing so causes `RuntimeError: Cannot send a request, as the client has been closed.` after the 1-hour TTL expires. Cleanup happens at shutdown via `close_litellm_async_clients()`.
|
||||
|
||||
### Troubleshooting: DB schema out of sync after proxy restart
|
||||
`litellm-proxy-extras` runs `prisma migrate deploy` on startup using **its own** bundled migration files, which may lag behind schema changes in the current worktree. Symptoms: `Unknown column`, `Invalid prisma invocation`, or missing data on new fields.
|
||||
|
||||
|
|
|
|||
97
docs/my-website/blog/gpt_5_4/index.md
Normal file
97
docs/my-website/blog/gpt_5_4/index.md
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
---
|
||||
slug: gpt_5_4
|
||||
title: "Day 0 Support: GPT-5.4"
|
||||
date: 2026-03-05T10:00:00
|
||||
authors:
|
||||
- name: Sameer Kankute
|
||||
title: SWE @ LiteLLM (LLM Translation)
|
||||
url: https://www.linkedin.com/in/sameer-kankute/
|
||||
image_url: https://pbs.twimg.com/profile_images/2001352686994907136/ONgNuSk5_400x400.jpg
|
||||
- name: Krrish Dholakia
|
||||
title: "CEO, LiteLLM"
|
||||
url: https://www.linkedin.com/in/krish-d/
|
||||
image_url: https://pbs.twimg.com/profile_images/1298587542745358340/DZv3Oj-h_400x400.jpg
|
||||
- name: Ishaan Jaff
|
||||
title: "CTO, LiteLLM"
|
||||
url: https://www.linkedin.com/in/reffajnaahsi/
|
||||
image_url: https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
|
||||
description: "GPT-5.4 model support in LiteLLM"
|
||||
tags: [openai, gpt-5.4, completion]
|
||||
hide_table_of_contents: false
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
LiteLLM now supports fully GPT-5.4!
|
||||
|
||||
## Docker Image
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="proxy" label="LiteLLM Proxy">
|
||||
|
||||
**1. Setup config.yaml**
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-5.4
|
||||
litellm_params:
|
||||
model: openai/gpt-5.4
|
||||
api_key: os.environ/OPENAI_API_KEY
|
||||
```
|
||||
|
||||
**2. Start the proxy**
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
-p 4000:4000 \
|
||||
-e OPENAI_API_KEY=$OPENAI_API_KEY \
|
||||
-v $(pwd)/config.yaml:/app/config.yaml \
|
||||
ghcr.io/berriai/litellm:v1.81.14-stable.gpt-5.4_patch \
|
||||
--config /app/config.yaml
|
||||
```
|
||||
|
||||
**3. Test it**
|
||||
|
||||
```bash
|
||||
curl -X POST "http://0.0.0.0:4000/chat/completions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $LITELLM_KEY" \
|
||||
-d '{
|
||||
"model": "gpt-5.4",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Write a Python function to check if a number is prime."}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="sdk" label="LiteLLM SDK">
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
|
||||
response = completion(
|
||||
model="openai/gpt-5.4",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function to check if a number is prime."}
|
||||
],
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Notes
|
||||
|
||||
- Restart your container to get the cost tracking for this model.
|
||||
- Use `/responses` for better model performance.
|
||||
- GPT-5.4 supports reasoning, function calling, vision, and tool-use — see the [OpenAI provider docs](../../docs/providers/openai) for advanced usage.
|
||||
252
docs/my-website/docs/a2a_agent_headers.md
Normal file
252
docs/my-website/docs/a2a_agent_headers.md
Normal file
|
|
@ -0,0 +1,252 @@
|
|||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# A2A Agent Authentication Headers
|
||||
|
||||
Forward authentication credentials (Bearer tokens, API keys, etc.) from clients to backend A2A agents.
|
||||
|
||||
## Overview
|
||||
|
||||
When LiteLLM proxies a request to a backend A2A agent, the agent may require its own authentication headers. There are three ways to supply them:
|
||||
|
||||
| Method | Who configures | How it works |
|
||||
|---|---|---|
|
||||
| **Static headers** | Admin (UI / API) | Always sent, regardless of client request |
|
||||
| **Forward client headers** | Admin (UI / API) | Header names to extract from client request and forward |
|
||||
| **Convention-based** | Client (no admin config) | Client sends `x-a2a-{agent_name}-{header}` — automatically routed |
|
||||
|
||||
All three methods can be combined. **Static headers always win** on key conflicts.
|
||||
|
||||
---
|
||||
|
||||
## Method 1 — Static Headers
|
||||
|
||||
Admin-configured headers that are always sent to the backend agent. Use this for server-to-server tokens or internal credentials that clients should never see or override.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="ui" label="UI">
|
||||
|
||||
1. Go to **Agents** in the LiteLLM dashboard.
|
||||
2. Create or edit an agent.
|
||||
3. Open the **Authentication Headers** panel.
|
||||
4. Under **Static Headers**, click **Add Static Header** and fill in the header name and value.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="api" label="REST API">
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:4000/v1/agents \
|
||||
-H "Authorization: Bearer sk-admin" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"agent_name": "my-agent",
|
||||
"agent_card_params": { ... },
|
||||
"static_headers": {
|
||||
"Authorization": "Bearer internal-server-token",
|
||||
"X-Internal-Service": "litellm-proxy"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
To update an existing agent:
|
||||
|
||||
```bash
|
||||
curl -X PATCH http://localhost:4000/v1/agents/{agent_id} \
|
||||
-H "Authorization: Bearer sk-admin" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"static_headers": {
|
||||
"Authorization": "Bearer new-token"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
**Client call — no special headers needed:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:4000/a2a/my-agent \
|
||||
-H "Authorization: Bearer sk-client-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0", "id": "1", "method": "message/send",
|
||||
"params": { "message": { "role": "user", "parts": [{"kind": "text", "text": "Hello"}], "messageId": "msg-1" } }
|
||||
}'
|
||||
```
|
||||
|
||||
The backend agent receives `Authorization: Bearer internal-server-token` without the client ever knowing the value.
|
||||
|
||||
---
|
||||
|
||||
## Method 2 — Forward Client Headers
|
||||
|
||||
Admin specifies a list of header **names**. When the client sends a request that includes those headers, LiteLLM extracts their values and forwards them to the backend agent. The client controls the values; the admin controls which headers are eligible to be forwarded.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="ui" label="UI">
|
||||
|
||||
1. Go to **Agents** in the LiteLLM dashboard.
|
||||
2. Create or edit an agent.
|
||||
3. Open the **Authentication Headers** panel.
|
||||
4. Under **Forward Client Headers**, type header names and press **Enter** (e.g. `x-api-key`, `Authorization`).
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="api" label="REST API">
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:4000/v1/agents \
|
||||
-H "Authorization: Bearer sk-admin" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"agent_name": "my-agent",
|
||||
"agent_card_params": { ... },
|
||||
"extra_headers": ["x-api-key", "x-user-token"]
|
||||
}'
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
**Client call — include the forwarded headers:**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:4000/a2a/my-agent \
|
||||
-H "Authorization: Bearer sk-client-key" \
|
||||
-H "x-api-key: user-secret-value" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ ... }'
|
||||
```
|
||||
|
||||
The backend agent receives `x-api-key: user-secret-value`.
|
||||
|
||||
:::note
|
||||
Header name matching is **case-insensitive**. If the client sends `X-API-Key` and `extra_headers` lists `x-api-key`, they match.
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Method 3 — Convention-Based Forwarding
|
||||
|
||||
Clients can forward headers to a specific agent without any admin pre-configuration by using the naming convention:
|
||||
|
||||
```
|
||||
x-a2a-{agent_name_or_id}-{header_name}: value
|
||||
```
|
||||
|
||||
LiteLLM parses these headers automatically and routes them to the matching agent only.
|
||||
|
||||
**Examples:**
|
||||
|
||||
| Client header sent | Agent name/ID | Forwarded as |
|
||||
|---|---|---|
|
||||
| `x-a2a-my-agent-authorization: Bearer tok` | `my-agent` | `authorization: Bearer tok` |
|
||||
| `x-a2a-my-agent-x-api-key: secret` | `my-agent` | `x-api-key: secret` |
|
||||
| `x-a2a-abc123-authorization: Bearer tok` | agent ID `abc123` | `authorization: Bearer tok` |
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:4000/a2a/my-agent \
|
||||
-H "Authorization: Bearer sk-client-key" \
|
||||
-H "x-a2a-my-agent-authorization: Bearer agent-specific-token" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ ... }'
|
||||
```
|
||||
|
||||
The `x-a2a-other-agent-authorization` header sent in the same request is **not** forwarded to `my-agent` — it is silently ignored.
|
||||
|
||||
:::tip Matches both agent name and agent ID
|
||||
Both the human-readable name (e.g. `my-agent`) and the UUID (e.g. `abc123-...`) are valid. Use whichever is convenient for the client.
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Merge Precedence
|
||||
|
||||
When multiple methods supply the same header name, **static headers win**:
|
||||
|
||||
```
|
||||
dynamic (forwarded/convention) → merged ← static (overlays, wins)
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
| Source | `Authorization` value |
|
||||
|---|---|
|
||||
| Client sends (via `extra_headers` or convention) | `Bearer client-token` |
|
||||
| Admin-configured `static_headers` | `Bearer server-token` |
|
||||
| **What the backend agent receives** | **`Bearer server-token`** |
|
||||
|
||||
This ensures admin-controlled credentials cannot be overridden by client requests.
|
||||
|
||||
---
|
||||
|
||||
## Combining All Three Methods
|
||||
|
||||
```bash
|
||||
# Register agent with static + forwarded headers
|
||||
curl -X POST http://localhost:4000/v1/agents \
|
||||
-H "Authorization: Bearer sk-admin" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"agent_name": "my-agent",
|
||||
"agent_card_params": { ... },
|
||||
"static_headers": {
|
||||
"X-Internal-Token": "secret123"
|
||||
},
|
||||
"extra_headers": ["x-user-id"]
|
||||
}'
|
||||
|
||||
# Client call using all three mechanisms
|
||||
curl -X POST http://localhost:4000/a2a/my-agent \
|
||||
-H "Authorization: Bearer sk-client-key" \
|
||||
-H "x-user-id: user-42" \
|
||||
-H "x-a2a-my-agent-x-request-id: req-abc" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{ ... }'
|
||||
```
|
||||
|
||||
The backend agent receives:
|
||||
|
||||
```
|
||||
X-Internal-Token: secret123 ← static header (always)
|
||||
x-user-id: user-42 ← forwarded (in extra_headers)
|
||||
x-request-id: req-abc ← convention-based (x-a2a-my-agent-*)
|
||||
X-LiteLLM-Trace-Id: <uuid> ← LiteLLM internal
|
||||
X-LiteLLM-Agent-Id: <agent-id> ← LiteLLM internal
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Header Isolation
|
||||
|
||||
Each agent invocation uses an isolated HTTP connection. Headers configured for agent A are **never** sent to agent B, even if both agents are running and receiving requests simultaneously.
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### `POST /v1/agents` / `PATCH /v1/agents/{agent_id}`
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `static_headers` | `object` | `{"Header-Name": "value"}` — always forwarded |
|
||||
| `extra_headers` | `string[]` | Header names to extract from client request and forward |
|
||||
|
||||
### Agent Response
|
||||
|
||||
Both fields are returned in `GET /v1/agents` and `GET /v1/agents/{agent_id}`:
|
||||
|
||||
```json
|
||||
{
|
||||
"agent_id": "...",
|
||||
"agent_name": "my-agent",
|
||||
"static_headers": { "X-Internal-Token": "secret123" },
|
||||
"extra_headers": ["x-user-id"],
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
:::caution
|
||||
`static_headers` values are stored in the database and returned by the API. Treat them as you would any credential — do not store sensitive long-lived tokens here if your API is publicly accessible. Consider using short-lived tokens or environment-injected secrets instead.
|
||||
:::
|
||||
|
|
@ -0,0 +1,120 @@
|
|||
# v1/messages → /responses Parameter Mapping
|
||||
|
||||
When you send a request to `/v1/messages` targeting an OpenAI or Azure model, LiteLLM internally routes it through the OpenAI Responses API. This page documents exactly how every parameter gets translated in both directions.
|
||||
|
||||
The transformation lives in `litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py`.
|
||||
|
||||
|
||||
## Request: Anthropic → Responses API
|
||||
|
||||
### Top-level parameters
|
||||
|
||||
| Anthropic (`/v1/messages`) | Responses API | Notes |
|
||||
|---|---|---|
|
||||
| `model` | `model` | Passed through as-is |
|
||||
| `messages` | `input` | Structurally transformed — see the messages section below |
|
||||
| `system` (string) | `instructions` | Passed as a plain string |
|
||||
| `system` (list of content blocks) | `instructions` | Text blocks are joined with `\n`; non-text blocks are ignored |
|
||||
| `max_tokens` | `max_output_tokens` | Renamed |
|
||||
| `temperature` | `temperature` | Passed through as-is |
|
||||
| `top_p` | `top_p` | Passed through as-is |
|
||||
| `tools` | `tools` | Format-translated — see the tools section below |
|
||||
| `tool_choice` | `tool_choice` | Type-remapped — see the tool_choice section below |
|
||||
| `thinking` | `reasoning` | Budget tokens mapped to effort level — see the thinking section below |
|
||||
| `output_format` or `output_config.format` | `text` | Wrapped as `{"format": {"type": "json_schema", "name": "structured_output", "schema": ..., "strict": true}}` |
|
||||
| `context_management` | `context_management` | Converted from Anthropic dict to OpenAI array format — see the context_management section below |
|
||||
| `metadata.user_id` | `user` | Extracted from the metadata object and truncated to 64 characters |
|
||||
| `stop_sequences` | ❌ Not mapped | Dropped silently |
|
||||
| `top_k` | ❌ Not mapped | Dropped silently |
|
||||
| `speed` | ❌ Not mapped | Only used to set Anthropic beta headers on the native path |
|
||||
|
||||
|
||||
### How messages get converted
|
||||
|
||||
Each Anthropic message is expanded into one or more Responses API input items. The key difference is that `tool_result` and `tool_use` blocks become **top-level items** in the input array rather than being nested inside a message.
|
||||
|
||||
| Anthropic message | Responses API input item |
|
||||
|---|---|
|
||||
| `user` role, string content | `{"type": "message", "role": "user", "content": [{"type": "input_text", "text": "..."}]}` |
|
||||
| `user` role, `{"type": "text"}` block | `{"type": "input_text", "text": "..."}` inside a user message |
|
||||
| `user` role, `{"type": "image", "source": {"type": "base64"}}` | `{"type": "input_image", "image_url": "data:<media_type>;base64,<data>"}` inside a user message |
|
||||
| `user` role, `{"type": "image", "source": {"type": "url"}}` | `{"type": "input_image", "image_url": "<url>"}` inside a user message |
|
||||
| `user` role, `{"type": "tool_result"}` block | Top-level `{"type": "function_call_output", "call_id": "...", "output": "..."}` — pulled out of the message entirely |
|
||||
| `assistant` role, string content | `{"type": "message", "role": "assistant", "content": [{"type": "output_text", "text": "..."}]}` |
|
||||
| `assistant` role, `{"type": "text"}` block | `{"type": "output_text", "text": "..."}` inside an assistant message |
|
||||
| `assistant` role, `{"type": "tool_use"}` block | Top-level `{"type": "function_call", "call_id": "<id>", "name": "...", "arguments": "<JSON string>"}` — pulled out of the message entirely |
|
||||
| `assistant` role, `{"type": "thinking"}` block | `{"type": "output_text", "text": "<thinking text>"}` inside an assistant message |
|
||||
|
||||
|
||||
### tools
|
||||
|
||||
| Anthropic tool | Responses API tool |
|
||||
|---|---|
|
||||
| Any tool where `type` starts with `"web_search"` or `name == "web_search"` | `{"type": "web_search_preview"}` |
|
||||
| All other tools | `{"type": "function", "name": "...", "description": "...", "parameters": <input_schema>}` |
|
||||
|
||||
|
||||
### tool_choice
|
||||
|
||||
| Anthropic `tool_choice.type` | Responses API `tool_choice` |
|
||||
|---|---|
|
||||
| `"auto"` | `{"type": "auto"}` |
|
||||
| `"any"` | `{"type": "required"}` |
|
||||
| `"tool"` | `{"type": "function", "name": "<tool name>"}` |
|
||||
|
||||
|
||||
### thinking → reasoning
|
||||
|
||||
The `budget_tokens` value is mapped to a string effort level. `summary` is always set to `"detailed"`.
|
||||
|
||||
| `thinking.budget_tokens` | `reasoning.effort` |
|
||||
|---|---|
|
||||
| >= 10000 | `"high"` |
|
||||
| >= 5000 | `"medium"` |
|
||||
| >= 2000 | `"low"` |
|
||||
| < 2000 | `"minimal"` |
|
||||
|
||||
If `thinking.type` is anything other than `"enabled"`, the `reasoning` field is not sent at all.
|
||||
|
||||
|
||||
### context_management
|
||||
|
||||
Anthropic uses a nested dict with an `edits` array. OpenAI uses a flat array of compaction objects.
|
||||
|
||||
```
|
||||
Anthropic input:
|
||||
{
|
||||
"edits": [
|
||||
{
|
||||
"type": "compact_20260112",
|
||||
"trigger": {"type": "input_tokens", "value": 150000}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Responses API output:
|
||||
[
|
||||
{"type": "compaction", "compact_threshold": 150000}
|
||||
]
|
||||
```
|
||||
|
||||
|
||||
## Response: Responses API → Anthropic
|
||||
|
||||
When the Responses API reply comes back, LiteLLM converts it into an Anthropic `AnthropicMessagesResponse`.
|
||||
|
||||
| Responses API field | Anthropic response field | Notes |
|
||||
|---|---|---|
|
||||
| `response.id` | `id` | |
|
||||
| `response.model` | `model` | Falls back to `"unknown-model"` if missing |
|
||||
| `ResponseReasoningItem` — `summary[*].text` | `content` block `{"type": "thinking", "thinking": "..."}` | Each non-empty summary text becomes a thinking block |
|
||||
| `ResponseOutputMessage` — `content[*]` where `type == "output_text"` | `content` block `{"type": "text", "text": "..."}` | |
|
||||
| `ResponseFunctionToolCall` — `{call_id, name, arguments}` | `content` block `{"type": "tool_use", "id": "...", "name": "...", "input": {...}}` | `arguments` is JSON-parsed back into a dict |
|
||||
| Any `function_call` present in output | `stop_reason: "tool_use"` | |
|
||||
| `response.status == "incomplete"` | `stop_reason: "max_tokens"` | Takes precedence over the default |
|
||||
| Everything else | `stop_reason: "end_turn"` | Default |
|
||||
| `response.usage.input_tokens` | `usage.input_tokens` | |
|
||||
| `response.usage.output_tokens` | `usage.output_tokens` | |
|
||||
| *(hardcoded)* | `type: "message"` | Always set |
|
||||
| *(hardcoded)* | `role: "assistant"` | Always set |
|
||||
| *(hardcoded)* | `stop_sequence: null` | Always null on this path |
|
||||
|
|
@ -704,6 +704,63 @@ asyncio.run(main())
|
|||
|
||||
[Learn more about customer management →](./proxy/customers)
|
||||
|
||||
## Calling the Proxy's /v1/responses Endpoint
|
||||
|
||||
When calling your LiteLLM Proxy's `/v1/responses` endpoint to use MCP tools, **always use `server_url: "litellm_proxy"`** in the tools array. This tells the proxy to use its configured MCP servers.
|
||||
|
||||
:::important Do not use the full proxy URL
|
||||
Using `server_url: "https://your-proxy.com/mcp"` is incorrect when the request is already going to the proxy. The proxy needs the literal value `litellm_proxy` to route to its configured MCP servers.
|
||||
:::
|
||||
|
||||
```bash title="Correct: Using litellm_proxy" showLineNumbers
|
||||
curl --location 'https://your-proxy.com/v1/responses' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--header "Authorization: Bearer $LITELLM_API_KEY" \
|
||||
--data '{
|
||||
"model": "gpt-4",
|
||||
"tools": [
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy",
|
||||
"require_approval": "never"
|
||||
}
|
||||
],
|
||||
"input": "Run available tools",
|
||||
"tool_choice": "required"
|
||||
}'
|
||||
```
|
||||
|
||||
### Sending Custom Headers to MCP Servers
|
||||
|
||||
To pass custom headers (e.g., API keys, auth tokens) to specific MCP servers, use either:
|
||||
|
||||
**Option 1: Request headers** – Add `x-mcp-{server_alias}-{header_name}` to your request headers. The proxy forwards these to the matching MCP server.
|
||||
|
||||
```bash
|
||||
# Send Authorization header to the "weather2" MCP server
|
||||
--header 'x-mcp-weather2-authorization: Bearer your-token'
|
||||
|
||||
# Send custom header to the "github" MCP server
|
||||
--header 'x-mcp-github-x-api-key: your-api-key'
|
||||
```
|
||||
|
||||
**Option 2: Headers in tool config** – Include a `headers` object in the tool definition. These are merged with request headers.
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "litellm_proxy",
|
||||
"require_approval": "never",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY",
|
||||
"x-mcp-servers": "Zapier_MCP,dev-group",
|
||||
"x-mcp-weather2-authorization": "Bearer your-weather-api-token"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Using your MCP with client side credentials
|
||||
|
||||
Use this if you want to pass a client side authentication token to LiteLLM to then pass to your MCP to auth to your MCP.
|
||||
|
|
|
|||
|
|
@ -323,7 +323,7 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
|
|||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "<your-litellm-proxy-base-url>/dev_group/mcp",
|
||||
"server_url": "litellm_proxy",
|
||||
"require_approval": "never",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY"
|
||||
|
|
@ -335,7 +335,7 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
|
|||
}'
|
||||
```
|
||||
|
||||
This example uses URL namespacing to access all servers in the "dev_group" access group.
|
||||
This example uses the `x-mcp-servers` header to access all servers in the "dev_group" access group. Use `server_url: "litellm_proxy"` when calling the proxy's `/v1/responses` endpoint—do not use the full proxy URL.
|
||||
|
||||
</TabItem>
|
||||
|
||||
|
|
@ -423,7 +423,7 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
|
|||
{
|
||||
"type": "mcp",
|
||||
"server_label": "litellm",
|
||||
"server_url": "<your-litellm-proxy-base-url>/mcp/",
|
||||
"server_url": "litellm_proxy",
|
||||
"require_approval": "never",
|
||||
"headers": {
|
||||
"x-litellm-api-key": "Bearer YOUR_LITELLM_API_KEY",
|
||||
|
|
@ -436,7 +436,7 @@ curl --location '<your-litellm-proxy-base-url>/v1/responses' \
|
|||
}'
|
||||
```
|
||||
|
||||
This configuration restricts the request to only use tools from the specified MCP servers.
|
||||
This configuration restricts the request to only use tools from the specified MCP servers. Use `server_url: "litellm_proxy"` when calling the proxy's `/v1/responses` endpoint.
|
||||
|
||||
</TabItem>
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,32 @@
|
|||
|
||||
Azure Model Router is a feature in Azure AI Foundry that automatically routes your requests to the best available model based on your requirements. This allows you to use a single endpoint that intelligently selects the optimal model for each request.
|
||||
|
||||
## Quick Start
|
||||
|
||||
**Model pattern**: `azure_ai/model_router/<deployment-name>`
|
||||
|
||||
```python
|
||||
import litellm
|
||||
|
||||
response = litellm.completion(
|
||||
model="azure_ai/model_router/model-router", # Replace with your deployment name
|
||||
messages=[{"role": "user", "content": "Hello!"}],
|
||||
api_base="https://your-endpoint.cognitiveservices.azure.com/openai/v1/",
|
||||
api_key="your-api-key",
|
||||
)
|
||||
```
|
||||
|
||||
**Proxy config** (`config.yaml`):
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: model-router
|
||||
litellm_params:
|
||||
model: azure_ai/model_router/model-router
|
||||
api_base: https://your-endpoint.cognitiveservices.azure.com/openai/deployments/model-router/chat/completions?api-version=2025-01-01-preview
|
||||
api_key: your-api-key
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Automatic Model Selection**: Azure Model Router dynamically selects the best model for your request
|
||||
|
|
@ -229,19 +255,51 @@ Cost is tracked based on the actual model used (e.g., `gpt-4.1-nano`), plus a fl
|
|||
|
||||
## Cost Tracking
|
||||
|
||||
LiteLLM automatically handles cost tracking for Azure Model Router by:
|
||||
LiteLLM automatically handles cost tracking for Azure Model Router. Understanding how this works helps you interpret spend and debug billing.
|
||||
|
||||
1. **Detecting the actual model**: When Azure Model Router routes your request to a specific model (e.g., `gpt-4.1-nano-2025-04-14`), LiteLLM extracts this from the response
|
||||
2. **Calculating accurate costs**: Costs are calculated based on:
|
||||
- The actual model used (e.g., `gpt-4.1-nano` token costs)
|
||||
- Plus a flat infrastructure cost of **$0.14 per million input tokens** for using the Model Router
|
||||
3. **Streaming support**: Cost tracking works correctly for both streaming and non-streaming requests
|
||||
### How LiteLLM Calculates Cost
|
||||
|
||||
When you use Azure Model Router, LiteLLM computes **two cost components**:
|
||||
|
||||
| Component | Description | When Applied |
|
||||
|-----------|-------------|--------------|
|
||||
| **Model Cost** | Token-based cost for the actual model that handled the request (e.g., `gpt-5-nano`, `gpt-4.1-nano`) | Always, when Azure returns the model in the response |
|
||||
| **Router Flat Cost** | $0.14 per million input tokens (Azure AI Foundry infrastructure fee) | When the **request** was made via a model router endpoint |
|
||||
|
||||
### Cost Calculation Flow
|
||||
|
||||
1. **Request model detection**: LiteLLM records the model you requested (e.g., `azure_ai/model_router/model-router`). If it contains `model_router` or `model-router`, the request is treated as a router request.
|
||||
|
||||
2. **Response model extraction**: Azure returns the actual model used in the response (e.g., `gpt-5-nano-2025-08-07`). LiteLLM uses this for the model cost lookup.
|
||||
|
||||
3. **Model cost**: LiteLLM looks up the response model in its pricing table and computes cost from prompt tokens and completion tokens.
|
||||
|
||||
4. **Router flat cost**: Because the original request was to a model router, LiteLLM adds the flat cost ($0.14 per M input tokens) on top of the model cost.
|
||||
|
||||
5. **Total cost**: `Total = Model Cost + Router Flat Cost`
|
||||
|
||||
### Configuration Requirements
|
||||
|
||||
For cost tracking to work correctly:
|
||||
|
||||
- **Use the full pattern**: `azure_ai/model_router/<deployment-name>` (e.g., `azure_ai/model_router/model-router`)
|
||||
- **Proxy config**: When using the LiteLLM proxy, set `model` in `litellm_params` to the full pattern so the request model is correctly identified as a router
|
||||
|
||||
```yaml
|
||||
# proxy_server_config.yaml
|
||||
model_list:
|
||||
- model_name: model-router
|
||||
litellm_params:
|
||||
model: azure_ai/model_router/model-router # Required for router cost detection
|
||||
api_base: https://your-endpoint.cognitiveservices.azure.com/openai/deployments/model-router/chat/completions?api-version=2025-01-01-preview
|
||||
api_key: your-api-key
|
||||
```
|
||||
|
||||
### Cost Breakdown
|
||||
|
||||
When you use Azure Model Router, the total cost includes:
|
||||
|
||||
- **Model Cost**: Based on the actual model that handled your request (e.g., `gpt-4.1-nano`)
|
||||
- **Model Cost**: Based on the actual model that handled your request (e.g., `gpt-5-nano`, `gpt-4.1-nano`)
|
||||
- **Router Flat Cost**: $0.14 per million input tokens (Azure AI Foundry infrastructure fee)
|
||||
|
||||
### Example Response with Cost
|
||||
|
|
|
|||
157
docs/my-website/docs/providers/bedrock_mantle.md
Normal file
157
docs/my-website/docs/providers/bedrock_mantle.md
Normal file
|
|
@ -0,0 +1,157 @@
|
|||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# Amazon Bedrock Mantle
|
||||
|
||||
[Amazon Bedrock Mantle](https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-mantle.html) is Amazon Bedrock's distributed inference engine (Project Mantle) that exposes an **OpenAI-compatible API** for Bedrock-hosted models.
|
||||
|
||||
Use this provider to call Bedrock Mantle models with accurate **AWS Bedrock pricing** instead of OpenAI pricing.
|
||||
|
||||
:::tip
|
||||
|
||||
**We support ALL Bedrock Mantle models, just set `model=bedrock_mantle/<model-id>` as a prefix when sending litellm requests**
|
||||
|
||||
:::
|
||||
|
||||
## API Key
|
||||
|
||||
```python
|
||||
# env variable
|
||||
os.environ['BEDROCK_MANTLE_API_KEY'] = "your-aws-bedrock-api-key"
|
||||
|
||||
# optional: override region (defaults to us-east-1)
|
||||
os.environ['BEDROCK_MANTLE_REGION'] = "us-east-1" # or use AWS_REGION
|
||||
```
|
||||
|
||||
## Supported Models
|
||||
|
||||
| Model | Context Window | Input (per 1M tokens) | Output (per 1M tokens) |
|
||||
|-------|---------------|----------------------|------------------------|
|
||||
| `openai.gpt-oss-120b` | 131K | $0.15 | $0.60 |
|
||||
| `openai.gpt-oss-20b` | 131K | $0.075 | $0.30 |
|
||||
| `openai.gpt-oss-safeguard-120b` | 131K | $0.15 | $0.60 |
|
||||
| `openai.gpt-oss-safeguard-20b` | 131K | $0.075 | $0.30 |
|
||||
|
||||
## Sample Usage
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="sdk" label="SDK">
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
import os
|
||||
|
||||
os.environ['BEDROCK_MANTLE_API_KEY'] = "your-bedrock-api-key"
|
||||
|
||||
response = completion(
|
||||
model="bedrock_mantle/openai.gpt-oss-120b",
|
||||
messages=[{"role": "user", "content": "hello from litellm"}],
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="streaming" label="Streaming">
|
||||
|
||||
```python
|
||||
from litellm import completion
|
||||
import os
|
||||
|
||||
os.environ['BEDROCK_MANTLE_API_KEY'] = "your-bedrock-api-key"
|
||||
|
||||
response = completion(
|
||||
model="bedrock_mantle/openai.gpt-oss-120b",
|
||||
messages=[{"role": "user", "content": "hello from litellm"}],
|
||||
stream=True,
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
print(chunk)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="async" label="Async">
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from litellm import acompletion
|
||||
import os
|
||||
|
||||
os.environ['BEDROCK_MANTLE_API_KEY'] = "your-bedrock-api-key"
|
||||
|
||||
async def main():
|
||||
response = await acompletion(
|
||||
model="bedrock_mantle/openai.gpt-oss-120b",
|
||||
messages=[{"role": "user", "content": "hello from litellm"}],
|
||||
)
|
||||
print(response)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Region Configuration
|
||||
|
||||
The API base URL is `https://bedrock-mantle.{region}.api.aws/v1`. Region is resolved in this order:
|
||||
|
||||
1. `BEDROCK_MANTLE_REGION` env var
|
||||
2. `AWS_REGION` env var
|
||||
3. Default: `us-east-1`
|
||||
|
||||
**Supported regions:** `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, `eu-west-2`, `eu-central-1`, `eu-south-1`, `eu-north-1`, `ap-northeast-1`, `ap-south-1`, `ap-southeast-3`, `sa-east-1`
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ['BEDROCK_MANTLE_REGION'] = "eu-west-1"
|
||||
|
||||
# or pass api_base directly
|
||||
response = completion(
|
||||
model="bedrock_mantle/openai.gpt-oss-120b",
|
||||
messages=[{"role": "user", "content": "hello"}],
|
||||
api_base="https://bedrock-mantle.eu-west-1.api.aws/v1",
|
||||
)
|
||||
```
|
||||
|
||||
## Usage with LiteLLM Proxy
|
||||
|
||||
### 1. Set Bedrock Mantle models on config.yaml
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: gpt-oss-120b
|
||||
litellm_params:
|
||||
model: bedrock_mantle/openai.gpt-oss-120b
|
||||
api_key: os.environ/BEDROCK_MANTLE_API_KEY
|
||||
# optional region override:
|
||||
api_base: "https://bedrock-mantle.us-east-1.api.aws/v1"
|
||||
|
||||
- model_name: gpt-oss-20b
|
||||
litellm_params:
|
||||
model: bedrock_mantle/openai.gpt-oss-20b
|
||||
api_key: os.environ/BEDROCK_MANTLE_API_KEY
|
||||
```
|
||||
|
||||
### 2. Start the proxy
|
||||
|
||||
```shell
|
||||
litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
### 3. Send a request
|
||||
|
||||
```python
|
||||
import openai
|
||||
|
||||
client = openai.OpenAI(
|
||||
api_key="anything",
|
||||
base_url="http://0.0.0.0:4000",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="gpt-oss-120b",
|
||||
messages=[{"role": "user", "content": "hello from litellm"}],
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
|
@ -4,12 +4,12 @@ Use ChatGPT Pro/Max subscription models through LiteLLM with OAuth device flow a
|
|||
|
||||
| Property | Details |
|
||||
|-------|-------|
|
||||
| Description | ChatGPT subscription access (Codex + GPT-5.2 family) via ChatGPT backend API |
|
||||
| Description | ChatGPT subscription access (Codex + GPT-5.3/5.4 family) via ChatGPT backend API |
|
||||
| Provider Route on LiteLLM | `chatgpt/` |
|
||||
| Supported Endpoints | `/responses`, `/chat/completions` (bridged to Responses for supported models) |
|
||||
| API Reference | https://chatgpt.com |
|
||||
|
||||
ChatGPT subscription access is native to the Responses API. Chat Completions requests are bridged to Responses for supported models (for example `chatgpt/gpt-5.2`).
|
||||
ChatGPT subscription access is native to the Responses API. Chat Completions requests are bridged to Responses for supported models (for example `chatgpt/gpt-5.4`).
|
||||
|
||||
Notes:
|
||||
- The ChatGPT subscription backend rejects token limit fields (`max_tokens`, `max_output_tokens`, `max_completion_tokens`) and `metadata`. LiteLLM strips these fields for this provider.
|
||||
|
|
@ -31,7 +31,7 @@ ChatGPT subscription access uses an OAuth device code flow:
|
|||
import litellm
|
||||
|
||||
response = litellm.responses(
|
||||
model="chatgpt/gpt-5.2-codex",
|
||||
model="chatgpt/gpt-5.3-codex",
|
||||
input="Write a Python hello world"
|
||||
)
|
||||
|
||||
|
|
@ -44,7 +44,7 @@ print(response)
|
|||
import litellm
|
||||
|
||||
response = litellm.completion(
|
||||
model="chatgpt/gpt-5.2",
|
||||
model="chatgpt/gpt-5.4",
|
||||
messages=[{"role": "user", "content": "Write a Python hello world"}]
|
||||
)
|
||||
|
||||
|
|
@ -55,16 +55,36 @@ print(response)
|
|||
|
||||
```yaml showLineNumbers title="config.yaml"
|
||||
model_list:
|
||||
- model_name: chatgpt/gpt-5.2
|
||||
- model_name: chatgpt/gpt-5.4
|
||||
model_info:
|
||||
mode: responses
|
||||
litellm_params:
|
||||
model: chatgpt/gpt-5.2
|
||||
- model_name: chatgpt/gpt-5.2-codex
|
||||
model: chatgpt/gpt-5.4
|
||||
- model_name: chatgpt/gpt-5.4-pro
|
||||
model_info:
|
||||
mode: responses
|
||||
litellm_params:
|
||||
model: chatgpt/gpt-5.2-codex
|
||||
model: chatgpt/gpt-5.4-pro
|
||||
- model_name: chatgpt/gpt-5.3-codex
|
||||
model_info:
|
||||
mode: responses
|
||||
litellm_params:
|
||||
model: chatgpt/gpt-5.3-codex
|
||||
- model_name: chatgpt/gpt-5.3-codex-spark
|
||||
model_info:
|
||||
mode: responses
|
||||
litellm_params:
|
||||
model: chatgpt/gpt-5.3-codex-spark
|
||||
- model_name: chatgpt/gpt-5.3-instant
|
||||
model_info:
|
||||
mode: responses
|
||||
litellm_params:
|
||||
model: chatgpt/gpt-5.3-instant
|
||||
- model_name: chatgpt/gpt-5.3-chat-latest
|
||||
model_info:
|
||||
mode: responses
|
||||
litellm_params:
|
||||
model: chatgpt/gpt-5.3-chat-latest
|
||||
```
|
||||
|
||||
```bash showLineNumbers title="Start LiteLLM Proxy"
|
||||
|
|
|
|||
|
|
@ -192,8 +192,12 @@ os.environ["OPENAI_BASE_URL"] = "https://your_host/v1" # OPTIONAL
|
|||
| gpt-5.2-2025-12-11 | `response = completion(model="gpt-5.2-2025-12-11", messages=messages)` |
|
||||
| gpt-5.2-chat-latest | `response = completion(model="gpt-5.2-chat-latest", messages=messages)` |
|
||||
| gpt-5.3-chat-latest | `response = completion(model="gpt-5.3-chat-latest", messages=messages)` |
|
||||
| gpt-5.4 | `response = completion(model="gpt-5.4", messages=messages)` |
|
||||
| gpt-5.4-2026-03-05 | `response = completion(model="gpt-5.4-2026-03-05", messages=messages)` |
|
||||
| gpt-5.2-pro | `response = completion(model="gpt-5.2-pro", messages=messages)` |
|
||||
| gpt-5.2-pro-2025-12-11 | `response = completion(model="gpt-5.2-pro-2025-12-11", messages=messages)` |
|
||||
| gpt-5.4-pro | `response = completion(model="gpt-5.4-pro", messages=messages)` |
|
||||
| gpt-5.4-pro-2026-03-05 | `response = completion(model="gpt-5.4-pro-2026-03-05", messages=messages)` |
|
||||
| gpt-5.1 | `response = completion(model="gpt-5.1", messages=messages)` |
|
||||
| gpt-5.1-codex | `response = completion(model="gpt-5.1-codex", messages=messages)` |
|
||||
| gpt-5.1-codex-mini | `response = completion(model="gpt-5.1-codex-mini", messages=messages)` |
|
||||
|
|
|
|||
|
|
@ -1687,6 +1687,20 @@ litellm.vertex_location = "us-central1 # Your Location
|
|||
| gemini-2.5-flash-lite-preview-09-2025 | `completion('gemini-2.5-flash-lite-preview-09-2025', messages)`, `completion('vertex_ai/gemini-2.5-flash-lite-preview-09-2025', messages)` |
|
||||
| gemini-3.1-flash-lite-preview | `completion('gemini-3.1-flash-lite-preview', messages)`, `completion('vertex_ai/gemini-3.1-flash-lite-preview', messages)` |
|
||||
|
||||
## PayGo / Priority Cost Tracking
|
||||
|
||||
LiteLLM automatically tracks spend for Vertex AI Gemini models using the correct pricing tier based on the response's `usageMetadata.trafficType`:
|
||||
|
||||
| Vertex AI `trafficType` | LiteLLM `service_tier` | Pricing applied |
|
||||
|-------------------------|-------------------------|-----------------|
|
||||
| `ON_DEMAND_PRIORITY` | `priority` | PayGo / priority pricing (`input_cost_per_token_priority`, `output_cost_per_token_priority`) |
|
||||
| `ON_DEMAND` | standard | Default on-demand pricing |
|
||||
| `FLEX` / `BATCH` | `flex` | Batch/flex pricing |
|
||||
|
||||
When you use [Vertex AI PayGo](https://cloud.google.com/vertex-ai/generative-ai/pricing) (on-demand priority) or batch workloads, LiteLLM reads `trafficType` from the response and applies the matching cost per token from the [model cost map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json). No configuration is required — spend tracking works out of the box for both standard and PayGo requests.
|
||||
|
||||
See [Spend Tracking](../proxy/cost_tracking.md) for general cost tracking setup.
|
||||
|
||||
## Private Service Connect (PSC) Endpoints
|
||||
|
||||
LiteLLM supports Vertex AI models deployed to Private Service Connect (PSC) endpoints, allowing you to use custom `api_base` URLs for private deployments.
|
||||
|
|
|
|||
|
|
@ -8,6 +8,8 @@ Track spend for keys, users, and teams across 100+ LLMs.
|
|||
|
||||
LiteLLM automatically tracks spend for all known models. See our [model cost map](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
|
||||
|
||||
Provider-specific cost tracking (e.g., [Vertex AI PayGo / priority pricing](../providers/vertex.md#paygo--priority-cost-tracking), [Bedrock service tiers](../providers/bedrock.md#usage---service-tier), [Azure base model mapping](./custom_pricing.md#set-base_model-for-cost-tracking-eg-azure-deployments)) is applied automatically when the response includes tier metadata.
|
||||
|
||||
:::tip Keep Pricing Data Updated
|
||||
[Sync model pricing data from GitHub](./sync_models_github.md) to ensure accurate cost tracking.
|
||||
:::
|
||||
|
|
|
|||
|
|
@ -104,9 +104,18 @@ There are other keys you can use to specify costs for different scenarios and mo
|
|||
- `input_cost_per_video_per_second` - Cost per second of video input
|
||||
- `input_cost_per_video_per_second_above_128k_tokens` - Video cost for large contexts
|
||||
- `input_cost_per_character` - Character-based pricing for some providers
|
||||
- `input_cost_per_token_priority` / `output_cost_per_token_priority` - Priority/PayGo pricing (Vertex AI Gemini, Bedrock)
|
||||
- `input_cost_per_token_flex` / `output_cost_per_token_flex` - Batch/flex pricing
|
||||
|
||||
These keys evolve based on how new models handle multimodality. The latest version can be found at [https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json).
|
||||
|
||||
### Service Tier / PayGo Pricing (Vertex AI, Bedrock)
|
||||
|
||||
For providers that support multiple pricing tiers (e.g., Vertex AI PayGo, Bedrock service tiers), LiteLLM automatically applies the correct cost based on the response:
|
||||
|
||||
- **Vertex AI Gemini**: Uses `usageMetadata.trafficType` (`ON_DEMAND_PRIORITY` → priority, `FLEX`/`BATCH` → flex). See [Vertex AI - PayGo / Priority Cost Tracking](../providers/vertex.md#paygo--priority-cost-tracking).
|
||||
- **Bedrock**: Uses `serviceTier` from the response. See [Bedrock - Usage - Service Tier](../providers/bedrock.md#usage---service-tier).
|
||||
|
||||
## Zero-Cost Models (Bypass Budget Checks)
|
||||
|
||||
**Use Case**: You have on-premises or free models that should be accessible even when users exceed their budget limits.
|
||||
|
|
|
|||
|
|
@ -112,6 +112,8 @@ general_settings:
|
|||
forward_llm_provider_auth_headers: true # Enable BYOK
|
||||
```
|
||||
|
||||
For **Claude Code** with `/login` and your own Anthropic key, see [Claude Code BYOK](../tutorials/claude_code_byok.md). Use `ANTHROPIC_CUSTOM_HEADERS="x-litellm-api-key: sk-12345"` to pass your LiteLLM key while your Anthropic key (from `/login`) is forwarded as `x-api-key`.
|
||||
|
||||
Client request:
|
||||
```bash
|
||||
curl -X POST "http://localhost:4000/v1/messages" \
|
||||
|
|
|
|||
123
docs/my-website/docs/tutorials/claude_code_byok.md
Normal file
123
docs/my-website/docs/tutorials/claude_code_byok.md
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
# Claude Code with Bring Your Own Key (BYOK)
|
||||
|
||||
Use Claude Code with your own Anthropic API key through the LiteLLM proxy. When you use Claude's `/login` with your Anthropic account, your API key is sent as `x-api-key`. With BYOK enabled, LiteLLM forwards your key to Anthropic instead of using proxy-configured keys — so you pay Anthropic directly while still benefiting from LiteLLM's routing, logging, and guardrails.
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Claude Code `/login`** — You sign in with your Anthropic account; Claude Code sends your Anthropic API key as `x-api-key`.
|
||||
2. **LiteLLM authentication** — You pass your LiteLLM proxy key via `ANTHROPIC_CUSTOM_HEADERS` so the proxy can authenticate and track your usage.
|
||||
3. **Key forwarding** — With `forward_llm_provider_auth_headers: true`, LiteLLM forwards your `x-api-key` to Anthropic, giving it precedence over any proxy-configured keys.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) installed
|
||||
- Anthropic API key (from [console.anthropic.com](https://console.anthropic.com))
|
||||
- LiteLLM proxy with a virtual key for authentication
|
||||
|
||||
## Step 1: Configure LiteLLM Proxy
|
||||
|
||||
Enable forwarding of LLM provider auth headers so your Anthropic key takes precedence:
|
||||
|
||||
```yaml title="config.yaml"
|
||||
model_list:
|
||||
- model_name: claude-sonnet-4-5
|
||||
litellm_params:
|
||||
model: anthropic/claude-sonnet-4-5
|
||||
# No api_key needed — client's key will be used
|
||||
|
||||
litellm_settings:
|
||||
forward_llm_provider_auth_headers: true # Required for BYOK
|
||||
```
|
||||
|
||||
:::info Why `forward_llm_provider_auth_headers`?
|
||||
|
||||
By default, LiteLLM strips `x-api-key` from client requests for security. Setting this to `true` allows client-provided provider keys (like your Anthropic key from `/login`) to be forwarded to Anthropic, overriding any proxy-configured keys.
|
||||
|
||||
:::
|
||||
|
||||
## Step 2: Create a LiteLLM Virtual Key
|
||||
|
||||
Create a virtual key in the LiteLLM UI or via API.
|
||||
```bash
|
||||
# Example: Create key via API
|
||||
curl -X POST "http://localhost:4000/key/generate" \
|
||||
-H "Authorization: Bearer sk-your-master-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"key_alias": "claude-code-byok", "models": ["claude-sonnet-4-5"]}'
|
||||
```
|
||||
|
||||
## Step 3: Configure Claude Code
|
||||
|
||||
Set environment variables so Claude Code uses LiteLLM and sends your LiteLLM key for proxy auth:
|
||||
|
||||
```bash
|
||||
# Point Claude Code to your LiteLLM proxy
|
||||
export ANTHROPIC_BASE_URL="http://localhost:4000"
|
||||
|
||||
# Model name from your config
|
||||
export ANTHROPIC_MODEL="claude-sonnet-4-5"
|
||||
|
||||
# LiteLLM proxy auth — this is added to every request
|
||||
# Use x-litellm-api-key so the proxy authenticates you; your Anthropic key goes via x-api-key from /login
|
||||
export ANTHROPIC_CUSTOM_HEADERS="x-litellm-api-key: sk-12345"
|
||||
```
|
||||
|
||||
Replace `sk-12345` with your actual LiteLLM virtual key.
|
||||
|
||||
:::tip Multiple headers
|
||||
|
||||
For multiple headers, use newline-separated values:
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_CUSTOM_HEADERS="x-litellm-api-key: sk-12345
|
||||
x-litellm-user-id: my-user-id"
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
## Step 4: Sign In with Claude Code
|
||||
|
||||
1. Launch Claude Code:
|
||||
|
||||
```bash
|
||||
claude
|
||||
```
|
||||
|
||||
2. Use **`/login`** and sign in with your Anthropic account (or use your API key directly).
|
||||
|
||||
3. Claude Code will send:
|
||||
- `x-api-key`: Your Anthropic API key (from `/login`)
|
||||
- `x-litellm-api-key`: Your LiteLLM key (from `ANTHROPIC_CUSTOM_HEADERS`)
|
||||
|
||||
4. LiteLLM authenticates you via `x-litellm-api-key`, then forwards `x-api-key` to Anthropic. Your Anthropic key takes precedence over any proxy-configured key.
|
||||
|
||||
## Summary
|
||||
|
||||
| Header | Source | Purpose |
|
||||
|--------|--------|---------|
|
||||
| `x-api-key` | Claude Code `/login` (Anthropic key) | Sent to Anthropic for API calls |
|
||||
| `x-litellm-api-key` | `ANTHROPIC_CUSTOM_HEADERS` | Proxy authentication, tracking, rate limits |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Requests fail with "invalid x-api-key"
|
||||
|
||||
- Ensure `forward_llm_provider_auth_headers: true` is set in `litellm_settings` (or `general_settings`).
|
||||
- Restart the LiteLLM proxy after changing the config.
|
||||
- Verify you completed `/login` in Claude Code so your Anthropic key is being sent.
|
||||
|
||||
### Proxy returns 401
|
||||
|
||||
- Check that `ANTHROPIC_CUSTOM_HEADERS` includes `x-litellm-api-key: <your-key>`.
|
||||
- Ensure the LiteLLM key is valid and has access to the model.
|
||||
|
||||
### Proxy key is used instead of my Anthropic key
|
||||
|
||||
- Confirm `forward_llm_provider_auth_headers: true` is in your config.
|
||||
- The setting can be in `litellm_settings` or `general_settings` depending on your config structure.
|
||||
- Enable debug logging: `LITELLM_LOG=DEBUG` to see which key is being forwarded.
|
||||
|
||||
## Related
|
||||
|
||||
- [Forward Client Headers](./../proxy/forward_client_headers.md) — Full BYOK and header forwarding docs
|
||||
- [Claude Code Max Subscription](./claude_code_max_subscription.md) — Using Claude Code with OAuth/Max subscription through LiteLLM
|
||||
BIN
docs/my-website/img/claude_code_byok_screenshot.png
Normal file
BIN
docs/my-website/img/claude_code_byok_screenshot.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
|
|
@ -154,6 +154,7 @@ const sidebars = {
|
|||
items: [
|
||||
"tutorials/claude_responses_api",
|
||||
"tutorials/claude_code_max_subscription",
|
||||
"tutorials/claude_code_byok",
|
||||
"tutorials/claude_code_customer_tracking",
|
||||
"tutorials/claude_code_prompt_cache_routing",
|
||||
"tutorials/claude_code_websearch",
|
||||
|
|
@ -538,6 +539,7 @@ const sidebars = {
|
|||
items: [
|
||||
"a2a",
|
||||
"a2a_invoking_agents",
|
||||
"a2a_agent_headers",
|
||||
"a2a_cost_tracking",
|
||||
"a2a_agent_permissions"
|
||||
],
|
||||
|
|
@ -624,6 +626,7 @@ const sidebars = {
|
|||
items: [
|
||||
"anthropic_unified/index",
|
||||
"anthropic_unified/structured_output",
|
||||
"anthropic_unified/messages_to_responses_mapping",
|
||||
]
|
||||
},
|
||||
"anthropic_count_tokens",
|
||||
|
|
@ -795,6 +798,7 @@ const sidebars = {
|
|||
"providers/bedrock_realtime_with_audio",
|
||||
"providers/aws_polly",
|
||||
"providers/bedrock_vector_store",
|
||||
"providers/bedrock_mantle",
|
||||
]
|
||||
},
|
||||
"providers/litellm_proxy",
|
||||
|
|
|
|||
|
|
@ -593,6 +593,7 @@ minimax_models: Set = set()
|
|||
aws_polly_models: Set = set()
|
||||
gigachat_models: Set = set()
|
||||
llamagate_models: Set = set()
|
||||
bedrock_mantle_models: Set = set()
|
||||
|
||||
|
||||
def is_bedrock_pricing_only_model(key: str) -> bool:
|
||||
|
|
@ -855,6 +856,8 @@ def add_known_models(model_cost_map: Optional[Dict] = None):
|
|||
gigachat_models.add(key)
|
||||
elif value.get("litellm_provider") == "llamagate":
|
||||
llamagate_models.add(key)
|
||||
elif value.get("litellm_provider") == "bedrock_mantle":
|
||||
bedrock_mantle_models.add(key)
|
||||
|
||||
|
||||
add_known_models()
|
||||
|
|
@ -962,6 +965,7 @@ model_list = list(
|
|||
| ovhcloud_models
|
||||
| lemonade_models
|
||||
| docker_model_runner_models
|
||||
| bedrock_mantle_models
|
||||
| set(clarifai_models)
|
||||
)
|
||||
|
||||
|
|
@ -1065,6 +1069,7 @@ models_by_provider: dict = {
|
|||
"aws_polly": aws_polly_models,
|
||||
"gigachat": gigachat_models,
|
||||
"llamagate": llamagate_models,
|
||||
"bedrock_mantle": bedrock_mantle_models
|
||||
}
|
||||
|
||||
# mapping for those models which have larger equivalents
|
||||
|
|
@ -1426,6 +1431,7 @@ if TYPE_CHECKING:
|
|||
from .llms.topaz.image_variations.transformation import TopazImageVariationConfig as TopazImageVariationConfig
|
||||
from litellm.llms.openai.completion.transformation import OpenAITextCompletionConfig as OpenAITextCompletionConfig
|
||||
from .llms.groq.chat.transformation import GroqChatConfig as GroqChatConfig
|
||||
from .llms.bedrock_mantle.chat.transformation import BedrockMantleChatConfig as BedrockMantleChatConfig
|
||||
from .llms.a2a.chat.transformation import A2AConfig as A2AConfig
|
||||
from .llms.voyage.embedding.transformation import VoyageEmbeddingConfig as VoyageEmbeddingConfig
|
||||
from .llms.voyage.embedding.transformation_contextual import VoyageContextualEmbeddingConfig as VoyageContextualEmbeddingConfig
|
||||
|
|
|
|||
|
|
@ -214,6 +214,7 @@ LLM_CONFIG_NAMES = (
|
|||
"TopazImageVariationConfig",
|
||||
"OpenAITextCompletionConfig",
|
||||
"GroqChatConfig",
|
||||
"BedrockMantleChatConfig",
|
||||
"A2AConfig",
|
||||
"GenAIHubOrchestrationConfig",
|
||||
"VoyageEmbeddingConfig",
|
||||
|
|
@ -858,6 +859,7 @@ _LLM_CONFIGS_IMPORT_MAP = {
|
|||
"OpenAITextCompletionConfig",
|
||||
),
|
||||
"GroqChatConfig": (".llms.groq.chat.transformation", "GroqChatConfig"),
|
||||
"BedrockMantleChatConfig": (".llms.bedrock_mantle.chat.transformation", "BedrockMantleChatConfig"),
|
||||
"A2AConfig": (".llms.a2a.chat.transformation", "A2AConfig"),
|
||||
"GenAIHubOrchestrationConfig": (
|
||||
".llms.sap.chat.transformation",
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import datetime
|
|||
import uuid
|
||||
from typing import TYPE_CHECKING, Any, AsyncIterator, Coroutine, Dict, Optional, Union
|
||||
|
||||
import httpx
|
||||
import litellm
|
||||
from litellm._logging import verbose_logger, verbose_proxy_logger
|
||||
from litellm.a2a_protocol.streaming_iterator import A2AStreamingIterator
|
||||
|
|
@ -162,29 +163,47 @@ async def _send_message_via_completion_bridge(
|
|||
return LiteLLMSendMessageResponse.from_dict(response_dict)
|
||||
|
||||
|
||||
async def _create_a2a_client_with_headers(
|
||||
async def _execute_a2a_send_with_retry(
|
||||
a2a_client: Any,
|
||||
request: Any,
|
||||
agent_card: Any,
|
||||
card_url: Optional[str],
|
||||
api_base: Optional[str],
|
||||
trace_id: Optional[str],
|
||||
agent_id: Optional[str],
|
||||
) -> tuple:
|
||||
"""Create an A2A client with LiteLLM trace headers."""
|
||||
if api_base is None:
|
||||
raise ValueError("Either a2a_client or api_base is required for standard A2A flow")
|
||||
trace_id = trace_id or str(uuid.uuid4())
|
||||
extra_headers: Dict[str, str] = {"X-LiteLLM-Trace-Id": trace_id}
|
||||
if agent_id:
|
||||
extra_headers["X-LiteLLM-Agent-Id"] = agent_id
|
||||
return await create_a2a_client(base_url=api_base, extra_headers=extra_headers), trace_id
|
||||
|
||||
|
||||
def _set_message_context_id(message: Any, context_id: str) -> None:
|
||||
"""Set context_id on an A2A message if not already set."""
|
||||
if isinstance(message, dict):
|
||||
if message.get("context_id") is None:
|
||||
message["context_id"] = context_id
|
||||
else:
|
||||
if getattr(message, "context_id", None) is None:
|
||||
message.context_id = context_id
|
||||
agent_name: Optional[str],
|
||||
) -> Any:
|
||||
"""Send an A2A message with retry logic for localhost URL errors."""
|
||||
a2a_response = None
|
||||
for _ in range(2): # max 2 attempts: original + 1 retry
|
||||
try:
|
||||
a2a_response = await a2a_client.send_message(request)
|
||||
break # success, exit retry loop
|
||||
except A2ALocalhostURLError as e:
|
||||
a2a_client = handle_a2a_localhost_retry(
|
||||
error=e,
|
||||
agent_card=agent_card,
|
||||
a2a_client=a2a_client,
|
||||
is_streaming=False,
|
||||
)
|
||||
card_url = agent_card.url if agent_card else None
|
||||
except Exception as e:
|
||||
try:
|
||||
map_a2a_exception(e, card_url, api_base, model=agent_name)
|
||||
except A2ALocalhostURLError as localhost_err:
|
||||
a2a_client = handle_a2a_localhost_retry(
|
||||
error=localhost_err,
|
||||
agent_card=agent_card,
|
||||
a2a_client=a2a_client,
|
||||
is_streaming=False,
|
||||
)
|
||||
card_url = agent_card.url if agent_card else None
|
||||
continue
|
||||
except Exception:
|
||||
raise
|
||||
if a2a_response is None:
|
||||
raise RuntimeError(
|
||||
"A2A send_message failed: no response received after retry attempts."
|
||||
)
|
||||
return a2a_response
|
||||
|
||||
|
||||
@client
|
||||
|
|
@ -194,6 +213,7 @@ async def asend_message(
|
|||
api_base: Optional[str] = None,
|
||||
litellm_params: Optional[Dict[str, Any]] = None,
|
||||
agent_id: Optional[str] = None,
|
||||
agent_extra_headers: Optional[Dict[str, str]] = None,
|
||||
**kwargs: Any,
|
||||
) -> LiteLLMSendMessageResponse:
|
||||
"""
|
||||
|
|
@ -270,8 +290,19 @@ async def asend_message(
|
|||
|
||||
# Create A2A client if not provided but api_base is available
|
||||
if a2a_client is None:
|
||||
a2a_client, trace_id = await _create_a2a_client_with_headers(
|
||||
api_base=api_base, trace_id=trace_id, agent_id=agent_id
|
||||
if api_base is None:
|
||||
raise ValueError(
|
||||
"Either a2a_client or api_base is required for standard A2A flow"
|
||||
)
|
||||
trace_id = trace_id or str(uuid.uuid4())
|
||||
extra_headers: Dict[str, str] = {"X-LiteLLM-Trace-Id": trace_id}
|
||||
if agent_id:
|
||||
extra_headers["X-LiteLLM-Agent-Id"] = agent_id
|
||||
# Overlay agent-level headers (agent headers take precedence over LiteLLM internal ones)
|
||||
if agent_extra_headers:
|
||||
extra_headers.update(agent_extra_headers)
|
||||
a2a_client = await create_a2a_client(
|
||||
base_url=api_base, extra_headers=extra_headers
|
||||
)
|
||||
|
||||
# Type assertion: a2a_client is guaranteed to be non-None here
|
||||
|
|
@ -288,46 +319,25 @@ async def asend_message(
|
|||
card_url = getattr(agent_card, "url", None) if agent_card else None
|
||||
|
||||
context_id = trace_id or str(uuid.uuid4())
|
||||
_set_message_context_id(request.params.message, context_id)
|
||||
message = request.params.message
|
||||
if isinstance(message, dict):
|
||||
if message.get("context_id") is None:
|
||||
message["context_id"] = context_id
|
||||
else:
|
||||
if getattr(message, "context_id", None) is None:
|
||||
message.context_id = context_id
|
||||
|
||||
# Retry loop: if connection fails due to localhost URL in agent card, retry with fixed URL
|
||||
a2a_response = None
|
||||
for _ in range(2): # max 2 attempts: original + 1 retry
|
||||
try:
|
||||
a2a_response = await a2a_client.send_message(request)
|
||||
break # success, exit retry loop
|
||||
except A2ALocalhostURLError as e:
|
||||
# Localhost URL error - fix and retry
|
||||
a2a_client = handle_a2a_localhost_retry(
|
||||
error=e,
|
||||
agent_card=agent_card,
|
||||
a2a_client=a2a_client,
|
||||
is_streaming=False,
|
||||
)
|
||||
card_url = agent_card.url if agent_card else None
|
||||
except Exception as e:
|
||||
# Map exception - will raise A2ALocalhostURLError if applicable
|
||||
try:
|
||||
map_a2a_exception(e, card_url, api_base, model=agent_name)
|
||||
except A2ALocalhostURLError as localhost_err:
|
||||
# Localhost URL error - fix and retry
|
||||
a2a_client = handle_a2a_localhost_retry(
|
||||
error=localhost_err,
|
||||
agent_card=agent_card,
|
||||
a2a_client=a2a_client,
|
||||
is_streaming=False,
|
||||
)
|
||||
card_url = agent_card.url if agent_card else None
|
||||
continue
|
||||
except Exception:
|
||||
# Re-raise the mapped exception
|
||||
raise
|
||||
a2a_response = await _execute_a2a_send_with_retry(
|
||||
a2a_client=a2a_client,
|
||||
request=request,
|
||||
agent_card=agent_card,
|
||||
card_url=card_url,
|
||||
api_base=api_base,
|
||||
agent_name=agent_name,
|
||||
)
|
||||
|
||||
verbose_logger.info(f"A2A send_message completed, request_id={request.id}")
|
||||
|
||||
# a2a_response is guaranteed to be set if we reach here (loop breaks on success or raises)
|
||||
assert a2a_response is not None
|
||||
|
||||
# Wrap in LiteLLM response type for _hidden_params support
|
||||
response = LiteLLMSendMessageResponse.from_a2a_response(a2a_response)
|
||||
|
||||
|
|
@ -437,6 +447,7 @@ async def asend_message_streaming(
|
|||
agent_id: Optional[str] = None,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
proxy_server_request: Optional[Dict[str, Any]] = None,
|
||||
agent_extra_headers: Optional[Dict[str, str]] = None,
|
||||
) -> AsyncIterator[Any]:
|
||||
"""
|
||||
Async: Send a streaming message to an A2A agent.
|
||||
|
|
@ -518,7 +529,17 @@ async def asend_message_streaming(
|
|||
raise ValueError(
|
||||
"Either a2a_client or api_base is required for standard A2A flow"
|
||||
)
|
||||
a2a_client = await create_a2a_client(base_url=api_base)
|
||||
# Mirror the non-streaming path: always include trace and agent-id headers
|
||||
streaming_extra_headers: Dict[str, str] = {
|
||||
"X-LiteLLM-Trace-Id": str(request.id),
|
||||
}
|
||||
if agent_id:
|
||||
streaming_extra_headers["X-LiteLLM-Agent-Id"] = agent_id
|
||||
if agent_extra_headers:
|
||||
streaming_extra_headers.update(agent_extra_headers)
|
||||
a2a_client = await create_a2a_client(
|
||||
base_url=api_base, extra_headers=streaming_extra_headers
|
||||
)
|
||||
|
||||
# Type assertion: a2a_client is guaranteed to be non-None here
|
||||
assert a2a_client is not None
|
||||
|
|
@ -632,17 +653,17 @@ async def create_a2a_client(
|
|||
|
||||
verbose_logger.info(f"Creating A2A client for {base_url}")
|
||||
|
||||
# Use LiteLLM's cached httpx client
|
||||
http_handler = get_async_httpx_client(
|
||||
llm_provider=httpxSpecialProvider.A2A,
|
||||
params={"timeout": timeout},
|
||||
# Always create a fresh httpx client per A2A call so that per-agent auth
|
||||
# headers (extra_headers) are never shared across agents or requests.
|
||||
# Mutating a cached shared client would cause headers from one agent to
|
||||
# bleed into requests made to a different agent.
|
||||
httpx_client = httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(timeout),
|
||||
headers=extra_headers or {},
|
||||
)
|
||||
httpx_client = http_handler.client
|
||||
|
||||
if extra_headers:
|
||||
httpx_client.headers.update(extra_headers)
|
||||
verbose_proxy_logger.debug(
|
||||
f"A2A client created with extra_headers={extra_headers}"
|
||||
f"A2A client created with extra_headers={list(extra_headers.keys())}"
|
||||
)
|
||||
|
||||
# Resolve agent card
|
||||
|
|
|
|||
|
|
@ -33,6 +33,7 @@ from litellm.secret_managers.main import get_secret_str
|
|||
from litellm.types.llms.openai import (
|
||||
CancelBatchRequest,
|
||||
CreateBatchRequest,
|
||||
FileExpiresAfter,
|
||||
RetrieveBatchRequest,
|
||||
)
|
||||
from litellm.types.router import GenericLiteLLMParams
|
||||
|
|
@ -219,7 +220,7 @@ def create_batch( # noqa: PLR0915
|
|||
extra_body=extra_body,
|
||||
)
|
||||
if output_expires_after is not None:
|
||||
_create_batch_request["output_expires_after"] = output_expires_after # type: ignore[typeddict-item]
|
||||
_create_batch_request["output_expires_after"] = cast(FileExpiresAfter, output_expires_after)
|
||||
if model is not None:
|
||||
provider_config = ProviderConfigManager.get_provider_batches_config(
|
||||
model=model,
|
||||
|
|
|
|||
|
|
@ -3,36 +3,21 @@ Add the event loop to the cache key, to prevent event loop closed errors.
|
|||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import Set
|
||||
|
||||
from .in_memory_cache import InMemoryCache
|
||||
|
||||
|
||||
class LLMClientCache(InMemoryCache):
|
||||
# Background tasks must be stored to prevent garbage collection, which would
|
||||
# trigger "coroutine was never awaited" warnings. See:
|
||||
# https://docs.python.org/3/library/asyncio-task.html#creating-tasks
|
||||
# Intentionally shared across all instances as a global task registry.
|
||||
_background_tasks: Set[asyncio.Task] = set()
|
||||
"""Cache for LLM HTTP clients (OpenAI, Azure, httpx, etc.).
|
||||
|
||||
def _remove_key(self, key: str) -> None:
|
||||
"""Close async clients before evicting them to prevent connection pool leaks."""
|
||||
value = self.cache_dict.get(key)
|
||||
super()._remove_key(key)
|
||||
if value is not None:
|
||||
close_fn = getattr(value, "aclose", None) or getattr(value, "close", None)
|
||||
if close_fn and asyncio.iscoroutinefunction(close_fn):
|
||||
try:
|
||||
task = asyncio.get_running_loop().create_task(close_fn())
|
||||
self._background_tasks.add(task)
|
||||
task.add_done_callback(self._background_tasks.discard)
|
||||
except RuntimeError:
|
||||
pass
|
||||
elif close_fn and callable(close_fn):
|
||||
try:
|
||||
close_fn()
|
||||
except Exception:
|
||||
pass
|
||||
IMPORTANT: This cache intentionally does NOT close clients on eviction.
|
||||
Evicted clients may still be in use by in-flight requests. Closing them
|
||||
eagerly causes ``RuntimeError: Cannot send a request, as the client has
|
||||
been closed.`` errors in production after the TTL (1 hour) expires.
|
||||
|
||||
Clients that are no longer referenced will be garbage-collected normally.
|
||||
For explicit shutdown cleanup, use ``close_litellm_async_clients()``.
|
||||
"""
|
||||
|
||||
def update_cache_key_with_event_loop(self, key):
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -1028,12 +1028,16 @@ class OpenAiResponsesToChatCompletionStreamIterator(BaseModelResponseIterator):
|
|||
if provider_specific_fields:
|
||||
tool_call_chunk.provider_specific_fields = provider_specific_fields # type: ignore
|
||||
|
||||
# Do NOT emit finish_reason here — response.completed handles the terminal
|
||||
# finish_reason. Emitting "tool_calls" here would prematurely terminate
|
||||
# the stream before subsequent tool calls arrive (same fix as #17246 for
|
||||
# the message-type branch).
|
||||
return ModelResponseStream(
|
||||
choices=[
|
||||
StreamingChoices(
|
||||
index=0,
|
||||
delta=Delta(tool_calls=[tool_call_chunk]),
|
||||
finish_reason="tool_calls",
|
||||
delta=Delta(),
|
||||
finish_reason=None,
|
||||
)
|
||||
]
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1242,6 +1242,11 @@ X_LITELLM_DISABLE_CALLBACKS = "x-litellm-disable-callbacks"
|
|||
LITELLM_METADATA_FIELD = "litellm_metadata"
|
||||
OLD_LITELLM_METADATA_FIELD = "metadata"
|
||||
LITELLM_TRUNCATED_PAYLOAD_FIELD = "litellm_truncated"
|
||||
LITELLM_TRUNCATION_DB_SAFEGUARD_NOTE = (
|
||||
"Truncation is a DB storage safeguard. "
|
||||
"Full, untruncated data is logged to logging callbacks (OTEL, Datadog, etc.). "
|
||||
"To increase the truncation limit, set `MAX_STRING_LENGTH_PROMPT_IN_DB` in your env."
|
||||
)
|
||||
|
||||
########################### LiteLLM Proxy Specific Constants ###########################
|
||||
########################################################################################
|
||||
|
|
|
|||
|
|
@ -272,6 +272,8 @@ def cost_per_token( # noqa: PLR0915
|
|||
### SERVICE TIER ###
|
||||
service_tier: Optional[str] = None, # for OpenAI service tier pricing
|
||||
response: Optional[Any] = None,
|
||||
### REQUEST MODEL ###
|
||||
request_model: Optional[str] = None, # original request model for router detection
|
||||
) -> Tuple[float, float]: # type: ignore
|
||||
"""
|
||||
Calculates the cost per token for a given model, prompt tokens, and completion tokens.
|
||||
|
|
@ -520,7 +522,7 @@ def cost_per_token( # noqa: PLR0915
|
|||
return dashscope_cost_per_token(model=model, usage=usage_block)
|
||||
elif custom_llm_provider == "azure_ai":
|
||||
return azure_ai_cost_per_token(
|
||||
model=model, usage=usage_block, response_time_ms=response_time_ms
|
||||
model=model, usage=usage_block, response_time_ms=response_time_ms, request_model=request_model
|
||||
)
|
||||
else:
|
||||
model_info = _cached_get_model_info_helper(
|
||||
|
|
@ -1457,6 +1459,11 @@ def completion_cost( # noqa: PLR0915
|
|||
text=completion_string
|
||||
)
|
||||
|
||||
# Get the original request model for router detection
|
||||
request_model_for_cost = None
|
||||
if litellm_logging_obj is not None:
|
||||
request_model_for_cost = litellm_logging_obj.model
|
||||
|
||||
(
|
||||
prompt_tokens_cost_usd_dollar,
|
||||
completion_tokens_cost_usd_dollar,
|
||||
|
|
@ -1479,6 +1486,7 @@ def completion_cost( # noqa: PLR0915
|
|||
rerank_billed_units=rerank_billed_units,
|
||||
service_tier=service_tier,
|
||||
response=completion_response,
|
||||
request_model=request_model_for_cost,
|
||||
)
|
||||
|
||||
# Get additional costs from provider (e.g., routing fees, infrastructure costs)
|
||||
|
|
|
|||
|
|
@ -126,6 +126,19 @@ async def acreate_fine_tuning_job(
|
|||
raise e
|
||||
|
||||
|
||||
def _resolve_fine_tuning_timeout(
|
||||
timeout: Any,
|
||||
custom_llm_provider: str,
|
||||
) -> Union[float, httpx.Timeout]:
|
||||
"""Normalise a raw timeout value to a float (seconds) or httpx.Timeout for fine-tuning calls."""
|
||||
timeout = timeout or 600.0
|
||||
if isinstance(timeout, httpx.Timeout):
|
||||
if not supports_httpx_timeout(custom_llm_provider):
|
||||
return float(timeout.read or 600)
|
||||
return timeout
|
||||
return float(timeout)
|
||||
|
||||
|
||||
@client
|
||||
def create_fine_tuning_job(
|
||||
model: str,
|
||||
|
|
@ -164,21 +177,10 @@ def create_fine_tuning_job(
|
|||
_oai_hyperparameters: Hyperparameters = Hyperparameters(
|
||||
**hyperparameters
|
||||
) # Typed Hyperparameters for OpenAI Spec
|
||||
### TIMEOUT LOGIC ###
|
||||
timeout = optional_params.timeout or kwargs.get("request_timeout", 600) or 600
|
||||
# set timeout for 10 minutes by default
|
||||
|
||||
if (
|
||||
timeout is not None
|
||||
and isinstance(timeout, httpx.Timeout)
|
||||
and supports_httpx_timeout(custom_llm_provider) is False
|
||||
):
|
||||
read_timeout = timeout.read or 600
|
||||
timeout = read_timeout # default 10 min timeout
|
||||
elif timeout is not None and not isinstance(timeout, httpx.Timeout):
|
||||
timeout = float(timeout) # type: ignore
|
||||
elif timeout is None:
|
||||
timeout = 600.0
|
||||
timeout = _resolve_fine_tuning_timeout(
|
||||
optional_params.timeout or kwargs.get("request_timeout", 600),
|
||||
custom_llm_provider,
|
||||
)
|
||||
|
||||
# OpenAI
|
||||
if custom_llm_provider == "openai":
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ server-side using litellm router's search tools.
|
|||
"""
|
||||
|
||||
import asyncio
|
||||
import math
|
||||
from typing import Any, Dict, List, Optional, Tuple, Union, cast
|
||||
|
||||
import litellm
|
||||
|
|
@ -481,6 +482,56 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
response_format=response_format,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _resolve_max_tokens(
|
||||
optional_params: Dict,
|
||||
kwargs: Dict,
|
||||
) -> int:
|
||||
"""Extract max_tokens and validate against thinking.budget_tokens.
|
||||
|
||||
Anthropic API requires ``max_tokens > thinking.budget_tokens``.
|
||||
If the constraint is violated, auto-adjust to ``budget_tokens + 1024``.
|
||||
"""
|
||||
max_tokens: int = optional_params.get(
|
||||
"max_tokens",
|
||||
kwargs.get("max_tokens", 1024),
|
||||
)
|
||||
thinking_param = optional_params.get("thinking")
|
||||
if thinking_param and isinstance(thinking_param, dict):
|
||||
budget_tokens = thinking_param.get("budget_tokens")
|
||||
if (
|
||||
budget_tokens is not None
|
||||
and isinstance(budget_tokens, (int, float))
|
||||
and math.isfinite(budget_tokens)
|
||||
and budget_tokens > 0
|
||||
):
|
||||
if max_tokens <= budget_tokens:
|
||||
adjusted = math.ceil(budget_tokens) + 1024
|
||||
verbose_logger.debug(
|
||||
"WebSearchInterception: max_tokens=%s <= thinking.budget_tokens=%s, "
|
||||
"adjusting to %s to satisfy Anthropic API constraint",
|
||||
max_tokens, budget_tokens, adjusted,
|
||||
)
|
||||
max_tokens = adjusted
|
||||
return max_tokens
|
||||
|
||||
@staticmethod
|
||||
def _prepare_followup_kwargs(kwargs: Dict) -> Dict:
|
||||
"""Build kwargs for the follow-up call, excluding internal keys.
|
||||
|
||||
``litellm_logging_obj`` MUST be excluded so the follow-up call creates
|
||||
its own ``Logging`` instance via ``function_setup``. Reusing the
|
||||
initial call's logging object triggers the dedup flag
|
||||
(``has_logged_async_success``) which silently prevents the initial
|
||||
call's spend from being recorded — the root cause of the
|
||||
SpendLog / AWS billing mismatch.
|
||||
"""
|
||||
_internal_keys = {'litellm_logging_obj'}
|
||||
return {
|
||||
k: v for k, v in kwargs.items()
|
||||
if not k.startswith('_websearch_interception') and k not in _internal_keys
|
||||
}
|
||||
|
||||
async def _execute_agentic_loop(
|
||||
self,
|
||||
model: str,
|
||||
|
|
@ -504,7 +555,7 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
)
|
||||
search_tasks.append(self._execute_search(query))
|
||||
else:
|
||||
verbose_logger.warning(
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Tool call {tool_call['id']} has no query"
|
||||
)
|
||||
# Add empty result for tools without query
|
||||
|
|
@ -531,7 +582,7 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
final_search_results.append(cast(str, result))
|
||||
else:
|
||||
# Should never happen, but handle for type safety
|
||||
verbose_logger.warning(
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Unexpected result type {type(result)} at index {i}"
|
||||
)
|
||||
final_search_results.append(str(result))
|
||||
|
|
@ -557,13 +608,18 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
f"WebSearchInterception: Last message (tool_result): {user_message}"
|
||||
)
|
||||
|
||||
# Correlation context for structured logging
|
||||
_call_id = (
|
||||
getattr(logging_obj, "litellm_call_id", None)
|
||||
or kwargs.get("litellm_call_id", "unknown")
|
||||
)
|
||||
|
||||
full_model_name = model # safe default before try block
|
||||
|
||||
# Use anthropic_messages.acreate for follow-up request
|
||||
try:
|
||||
# Extract max_tokens from optional params or kwargs
|
||||
# max_tokens is a required parameter for anthropic_messages.acreate()
|
||||
max_tokens = anthropic_messages_optional_request_params.get(
|
||||
"max_tokens",
|
||||
kwargs.get("max_tokens", 1024) # Default to 1024 if not found
|
||||
max_tokens = self._resolve_max_tokens(
|
||||
anthropic_messages_optional_request_params, kwargs
|
||||
)
|
||||
|
||||
verbose_logger.debug(
|
||||
|
|
@ -576,16 +632,10 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
if k != 'max_tokens'
|
||||
}
|
||||
|
||||
# Remove internal websearch interception flags from kwargs before follow-up request
|
||||
# These flags are used internally and should not be passed to the LLM provider
|
||||
kwargs_for_followup = {
|
||||
k: v for k, v in kwargs.items()
|
||||
if not k.startswith('_websearch_interception')
|
||||
}
|
||||
kwargs_for_followup = self._prepare_followup_kwargs(kwargs)
|
||||
|
||||
# Get model from logging_obj.model_call_details["agentic_loop_params"]
|
||||
# This preserves the full model name with provider prefix (e.g., "bedrock/invoke/...")
|
||||
full_model_name = model
|
||||
if logging_obj is not None:
|
||||
agentic_params = logging_obj.model_call_details.get("agentic_loop_params", {})
|
||||
full_model_name = agentic_params.get("model", model)
|
||||
|
|
@ -609,7 +659,10 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
return final_response
|
||||
except Exception as e:
|
||||
verbose_logger.exception(
|
||||
f"WebSearchInterception: Follow-up request failed: {str(e)}"
|
||||
"WebSearchInterception: Follow-up request failed "
|
||||
"[call_id=%s model=%s messages=%d searches=%d]: %s",
|
||||
_call_id, full_model_name, len(follow_up_messages),
|
||||
len(final_search_results), str(e),
|
||||
)
|
||||
raise
|
||||
|
||||
|
|
@ -620,7 +673,7 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
try:
|
||||
from litellm.proxy.proxy_server import llm_router
|
||||
except ImportError:
|
||||
verbose_logger.warning(
|
||||
verbose_logger.debug(
|
||||
"WebSearchInterception: Could not import llm_router from proxy_server, "
|
||||
"falling back to direct litellm.asearch() with perplexity"
|
||||
)
|
||||
|
|
@ -643,7 +696,7 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
f"with provider '{search_provider}'"
|
||||
)
|
||||
else:
|
||||
verbose_logger.warning(
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Search tool '{self.search_tool_name}' not found in router, "
|
||||
"falling back to first available or perplexity"
|
||||
)
|
||||
|
|
@ -717,7 +770,7 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
)
|
||||
search_tasks.append(self._execute_search(query))
|
||||
else:
|
||||
verbose_logger.warning(
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Tool call {tool_call.get('id')} has no query"
|
||||
)
|
||||
# Add empty result for tools without query
|
||||
|
|
@ -742,7 +795,7 @@ class WebSearchInterceptionLogger(CustomLogger):
|
|||
elif isinstance(result, str):
|
||||
final_search_results.append(cast(str, result))
|
||||
else:
|
||||
verbose_logger.warning(
|
||||
verbose_logger.debug(
|
||||
f"WebSearchInterception: Unexpected result type {type(result)} at index {i}"
|
||||
)
|
||||
final_search_results.append(str(result))
|
||||
|
|
|
|||
|
|
@ -561,6 +561,13 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
|
|||
) = litellm.GroqChatConfig()._get_openai_compatible_provider_info(
|
||||
api_base, api_key
|
||||
)
|
||||
elif custom_llm_provider == "bedrock_mantle":
|
||||
(
|
||||
api_base,
|
||||
dynamic_api_key,
|
||||
) = litellm.BedrockMantleChatConfig()._get_openai_compatible_provider_info(
|
||||
api_base, api_key
|
||||
)
|
||||
elif custom_llm_provider == "nvidia_nim":
|
||||
# nvidia_nim is openai compatible, we just need to set this to custom_openai and have the api_base be https://api.endpoints.anyscale.com/v1
|
||||
api_base = (
|
||||
|
|
|
|||
|
|
@ -88,6 +88,8 @@ def get_supported_openai_params( # noqa: PLR0915
|
|||
return litellm.VolcEngineConfig().get_supported_openai_params(model=model)
|
||||
elif custom_llm_provider == "groq":
|
||||
return litellm.GroqChatConfig().get_supported_openai_params(model=model)
|
||||
elif custom_llm_provider == "bedrock_mantle":
|
||||
return litellm.BedrockMantleChatConfig().get_supported_openai_params(model=model)
|
||||
elif custom_llm_provider == "hosted_vllm":
|
||||
return litellm.HostedVLLMChatConfig().get_supported_openai_params(model=model)
|
||||
elif custom_llm_provider == "vllm":
|
||||
|
|
|
|||
|
|
@ -194,6 +194,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
|
|||
"web_search_options",
|
||||
"speed",
|
||||
"context_management",
|
||||
"cache_control",
|
||||
]
|
||||
|
||||
if (
|
||||
|
|
@ -1106,6 +1107,9 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
|
|||
elif param == "speed" and isinstance(value, str):
|
||||
# Pass through Anthropic-specific speed parameter for fast mode
|
||||
optional_params["speed"] = value
|
||||
elif param == "cache_control" and isinstance(value, dict):
|
||||
# Pass through top-level cache_control for automatic prompt caching
|
||||
optional_params["cache_control"] = value
|
||||
|
||||
## handle thinking tokens
|
||||
self.update_optional_params_with_thinking_tokens(
|
||||
|
|
|
|||
|
|
@ -81,7 +81,7 @@ class AzureBatchesAPI(BaseAzureLLM):
|
|||
retrieve_batch_data: RetrieveBatchRequest,
|
||||
client: Union[AsyncAzureOpenAI, AsyncOpenAI],
|
||||
) -> LiteLLMBatch:
|
||||
response = await client.batches.retrieve(**retrieve_batch_data)
|
||||
response = await client.batches.retrieve(**retrieve_batch_data) # type: ignore[arg-type]
|
||||
return LiteLLMBatch(**response.model_dump())
|
||||
|
||||
def retrieve_batch(
|
||||
|
|
|
|||
|
|
@ -28,8 +28,8 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
|
|||
def get_supported_openai_params(self, model: str) -> List[str]:
|
||||
"""Get supported parameters for Azure OpenAI GPT-5 models.
|
||||
|
||||
Azure OpenAI GPT-5.2 models support logprobs, unlike OpenAI's GPT-5.
|
||||
This overrides the parent class to add logprobs support back for gpt-5.2.
|
||||
Azure OpenAI GPT-5.2/5.4 models support logprobs, unlike OpenAI's GPT-5.
|
||||
This overrides the parent class to add logprobs support back for gpt-5.2+.
|
||||
|
||||
Reference:
|
||||
- Tested with Azure OpenAI GPT-5.2 (api-version: 2025-01-01-preview)
|
||||
|
|
@ -43,10 +43,10 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
|
|||
if "tool_choice" not in params:
|
||||
params.append("tool_choice")
|
||||
|
||||
# Only gpt-5.2 has been verified to support logprobs on Azure.
|
||||
# Only gpt-5.2+ has been verified to support logprobs on Azure.
|
||||
# The base OpenAI class includes logprobs for gpt-5.1+, but Azure
|
||||
# hasn't verified support for gpt-5.1, so remove them unless gpt-5.2.
|
||||
if self.is_model_gpt_5_1_model(model) and not self.is_model_gpt_5_2_model(model):
|
||||
# hasn't verified support for gpt-5.1, so remove them unless gpt-5.2/5.4+.
|
||||
if self._supports_reasoning_effort_level(model, "none") and not self.is_model_gpt_5_2_model(model):
|
||||
params = [p for p in params if p not in ["logprobs", "top_logprobs"]]
|
||||
elif self.is_model_gpt_5_2_model(model):
|
||||
azure_supported_params = ["logprobs", "top_logprobs"]
|
||||
|
|
@ -67,11 +67,11 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
|
|||
or optional_params.get("reasoning_effort")
|
||||
)
|
||||
|
||||
# gpt-5.1 supports reasoning_effort='none', but other gpt-5 models don't
|
||||
# gpt-5.1/5.2/5.4 support reasoning_effort='none', but other gpt-5 models don't
|
||||
# See: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/reasoning
|
||||
is_gpt_5_1 = self.is_model_gpt_5_1_model(model)
|
||||
supports_none = self._supports_reasoning_effort_level(model, "none")
|
||||
|
||||
if reasoning_effort_value == "none" and not is_gpt_5_1:
|
||||
if reasoning_effort_value == "none" and not supports_none:
|
||||
if litellm.drop_params is True or (
|
||||
drop_params is not None and drop_params is True
|
||||
):
|
||||
|
|
@ -101,8 +101,8 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
|
|||
drop_params=drop_params,
|
||||
)
|
||||
|
||||
# Only drop reasoning_effort='none' for non-gpt-5.1 models
|
||||
if result.get("reasoning_effort") == "none" and not is_gpt_5_1:
|
||||
# Only drop reasoning_effort='none' for models that don't support it
|
||||
if result.get("reasoning_effort") == "none" and not supports_none:
|
||||
result.pop("reasoning_effort")
|
||||
|
||||
return result
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
"""
|
||||
Azure Anthropic messages transformation config - extends AnthropicMessagesConfig with Azure authentication
|
||||
"""
|
||||
from typing import TYPE_CHECKING, Any, List, Optional, Tuple
|
||||
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple
|
||||
|
||||
from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
|
||||
AnthropicMessagesConfig,
|
||||
|
|
@ -114,3 +114,53 @@ class AzureAnthropicMessagesConfig(AnthropicMessagesConfig):
|
|||
|
||||
return api_base
|
||||
|
||||
def _remove_scope_from_cache_control(
|
||||
self, anthropic_messages_request: Dict
|
||||
) -> None:
|
||||
"""
|
||||
Remove `scope` field from cache_control for Azure AI Foundry.
|
||||
|
||||
Azure AI Foundry's Anthropic endpoint does not support the `scope` field
|
||||
(e.g., "global" for cross-request caching). Only `type` and `ttl` are supported.
|
||||
|
||||
Processes both `system` and `messages` content blocks.
|
||||
"""
|
||||
def _sanitize(cache_control: Any) -> None:
|
||||
if isinstance(cache_control, dict):
|
||||
cache_control.pop("scope", None)
|
||||
|
||||
def _process_content_list(content: list) -> None:
|
||||
for item in content:
|
||||
if isinstance(item, dict) and "cache_control" in item:
|
||||
_sanitize(item["cache_control"])
|
||||
|
||||
if "system" in anthropic_messages_request:
|
||||
system = anthropic_messages_request["system"]
|
||||
if isinstance(system, list):
|
||||
_process_content_list(system)
|
||||
|
||||
if "messages" in anthropic_messages_request:
|
||||
for message in anthropic_messages_request["messages"]:
|
||||
if isinstance(message, dict) and "content" in message:
|
||||
content = message["content"]
|
||||
if isinstance(content, list):
|
||||
_process_content_list(content)
|
||||
|
||||
def transform_anthropic_messages_request(
|
||||
self,
|
||||
model: str,
|
||||
messages: List[Dict],
|
||||
anthropic_messages_optional_request_params: Dict,
|
||||
litellm_params: GenericLiteLLMParams,
|
||||
headers: dict,
|
||||
) -> Dict:
|
||||
anthropic_messages_request = super().transform_anthropic_messages_request(
|
||||
model=model,
|
||||
messages=messages,
|
||||
anthropic_messages_optional_request_params=anthropic_messages_optional_request_params,
|
||||
litellm_params=litellm_params,
|
||||
headers=headers,
|
||||
)
|
||||
self._remove_scope_from_cache_control(anthropic_messages_request)
|
||||
return anthropic_messages_request
|
||||
|
||||
|
|
|
|||
|
|
@ -61,7 +61,10 @@ def calculate_azure_model_router_flat_cost(model: str, prompt_tokens: int) -> fl
|
|||
|
||||
|
||||
def cost_per_token(
|
||||
model: str, usage: Usage, response_time_ms: Optional[float] = 0.0
|
||||
model: str,
|
||||
usage: Usage,
|
||||
response_time_ms: Optional[float] = 0.0,
|
||||
request_model: Optional[str] = None,
|
||||
) -> Tuple[float, float]:
|
||||
"""
|
||||
Calculate the cost per token for Azure AI models.
|
||||
|
|
@ -71,9 +74,10 @@ def cost_per_token(
|
|||
- Plus the cost of the actual model used (handled by generic_cost_per_token)
|
||||
|
||||
Args:
|
||||
model: str, the model name without provider prefix
|
||||
model: str, the model name without provider prefix (from response)
|
||||
usage: LiteLLM Usage block
|
||||
response_time_ms: Optional response time in milliseconds
|
||||
request_model: Optional[str], the original request model name (to detect router usage)
|
||||
|
||||
Returns:
|
||||
Tuple[float, float] - prompt_cost_in_usd, completion_cost_in_usd
|
||||
|
|
@ -84,7 +88,13 @@ def cost_per_token(
|
|||
"""
|
||||
prompt_cost = 0.0
|
||||
completion_cost = 0.0
|
||||
|
||||
|
||||
# Determine if this was a model router request
|
||||
# Check both the response model and the request model
|
||||
is_router_request = _is_azure_model_router(model) or (
|
||||
request_model is not None and _is_azure_model_router(request_model)
|
||||
)
|
||||
|
||||
# Calculate base cost using generic cost calculator
|
||||
# This may raise an exception if the model is not in the cost map
|
||||
try:
|
||||
|
|
@ -103,19 +113,21 @@ def cost_per_token(
|
|||
verbose_logger.debug(
|
||||
f"Azure AI Model Router: model '{model}' not in cost map, calculating routing flat cost only. Error: {e}"
|
||||
)
|
||||
|
||||
|
||||
# Add flat cost for Azure Model Router
|
||||
# The flat cost is defined in model_prices_and_context_window.json for azure_ai/model_router
|
||||
if _is_azure_model_router(model):
|
||||
router_flat_cost = calculate_azure_model_router_flat_cost(model, usage.prompt_tokens)
|
||||
|
||||
if is_router_request:
|
||||
# Use the request model for flat cost calculation if available, otherwise use response model
|
||||
router_model_for_calc = request_model if request_model else model
|
||||
router_flat_cost = calculate_azure_model_router_flat_cost(router_model_for_calc, usage.prompt_tokens)
|
||||
|
||||
if router_flat_cost > 0:
|
||||
verbose_logger.debug(
|
||||
f"Azure AI Model Router flat cost: ${router_flat_cost:.6f} "
|
||||
f"({usage.prompt_tokens} tokens × ${router_flat_cost / usage.prompt_tokens:.9f}/token)"
|
||||
)
|
||||
|
||||
|
||||
# Add flat cost to prompt cost
|
||||
prompt_cost += router_flat_cost
|
||||
|
||||
|
||||
return prompt_cost, completion_cost
|
||||
|
|
|
|||
|
|
@ -6,7 +6,10 @@ from litellm.llms.anthropic.chat.transformation import AnthropicConfig
|
|||
from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation import (
|
||||
AmazonInvokeConfig,
|
||||
)
|
||||
from litellm.llms.bedrock.common_utils import get_anthropic_beta_from_headers
|
||||
from litellm.llms.bedrock.common_utils import (
|
||||
get_anthropic_beta_from_headers,
|
||||
remove_custom_field_from_tools,
|
||||
)
|
||||
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
from litellm.types.utils import ModelResponse
|
||||
|
|
@ -108,6 +111,12 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
|
|||
if "anthropic_version" not in _anthropic_request:
|
||||
_anthropic_request["anthropic_version"] = self.anthropic_version
|
||||
|
||||
# Remove `custom` field from tools (Bedrock doesn't support it)
|
||||
# Claude Code sends `custom: {defer_loading: true}` on tool definitions,
|
||||
# which causes Bedrock to reject the request with "Extra inputs are not permitted"
|
||||
# Ref: https://github.com/BerriAI/litellm/issues/22847
|
||||
remove_custom_field_from_tools(_anthropic_request)
|
||||
|
||||
tools = optional_params.get("tools")
|
||||
tool_search_used = self.is_tool_search_used(tools)
|
||||
programmatic_tool_calling_used = self.is_programmatic_tool_calling_used(tools)
|
||||
|
|
|
|||
|
|
@ -49,6 +49,27 @@ def get_cached_model_info():
|
|||
return _get_model_info
|
||||
|
||||
|
||||
def remove_custom_field_from_tools(request_body: dict) -> None:
|
||||
"""
|
||||
Remove ``custom`` field from each tool in the request body.
|
||||
|
||||
Claude Code (v2.1.69+) sends ``custom: {defer_loading: true}`` on tool
|
||||
definitions, which Anthropic's API accepts but Bedrock rejects with
|
||||
``"Extra inputs are not permitted"``.
|
||||
|
||||
Args:
|
||||
request_body: The request dictionary to modify in-place.
|
||||
|
||||
Ref: https://github.com/BerriAI/litellm/issues/22847
|
||||
"""
|
||||
tools = request_body.get("tools")
|
||||
if not tools or not isinstance(tools, list):
|
||||
return
|
||||
for tool in tools:
|
||||
if isinstance(tool, dict):
|
||||
tool.pop("custom", None)
|
||||
|
||||
|
||||
class AmazonBedrockGlobalConfig:
|
||||
def __init__(self):
|
||||
pass
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation
|
|||
from litellm.llms.bedrock.common_utils import (
|
||||
get_anthropic_beta_from_headers,
|
||||
is_claude_4_5_on_bedrock,
|
||||
remove_custom_field_from_tools,
|
||||
)
|
||||
from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER
|
||||
from litellm.types.llms.openai import AllMessageValues
|
||||
|
|
@ -118,10 +119,13 @@ class AmazonAnthropicClaudeMessagesConfig(
|
|||
self, anthropic_messages_request: Dict, model: Optional[str] = None
|
||||
) -> None:
|
||||
"""
|
||||
Remove `ttl` field from cache_control in messages.
|
||||
Bedrock doesn't support the ttl field in cache_control.
|
||||
Remove unsupported fields from cache_control for Bedrock.
|
||||
|
||||
Update: Bedock supports `5m` and `1h` for Claude 4.5 models.
|
||||
Bedrock only supports `type` and `ttl` in cache_control. It does NOT support:
|
||||
- `scope` (e.g., "global") - always removed
|
||||
- `ttl` - removed for older models; Claude 4.5+ supports "5m" and "1h"
|
||||
|
||||
Processes both `system` and `messages` content blocks.
|
||||
|
||||
Args:
|
||||
anthropic_messages_request: The request dictionary to modify in-place
|
||||
|
|
@ -131,23 +135,36 @@ class AmazonAnthropicClaudeMessagesConfig(
|
|||
if model:
|
||||
is_claude_4_5 = self._is_claude_4_5_on_bedrock(model)
|
||||
|
||||
def _sanitize_cache_control(cache_control: dict) -> None:
|
||||
if not isinstance(cache_control, dict):
|
||||
return
|
||||
# Bedrock doesn't support scope (e.g., "global" for cross-request caching)
|
||||
cache_control.pop("scope", None)
|
||||
# Remove ttl for models that don't support it
|
||||
if "ttl" in cache_control:
|
||||
ttl = cache_control["ttl"]
|
||||
if is_claude_4_5 and ttl in ["5m", "1h"]:
|
||||
return
|
||||
cache_control.pop("ttl", None)
|
||||
|
||||
def _process_content_list(content: list) -> None:
|
||||
for item in content:
|
||||
if isinstance(item, dict) and "cache_control" in item:
|
||||
_sanitize_cache_control(item["cache_control"])
|
||||
|
||||
# Process system (list of content blocks)
|
||||
if "system" in anthropic_messages_request:
|
||||
system = anthropic_messages_request["system"]
|
||||
if isinstance(system, list):
|
||||
_process_content_list(system)
|
||||
|
||||
# Process messages
|
||||
if "messages" in anthropic_messages_request:
|
||||
for message in anthropic_messages_request["messages"]:
|
||||
if isinstance(message, dict) and "content" in message:
|
||||
content = message["content"]
|
||||
if isinstance(content, list):
|
||||
for item in content:
|
||||
if isinstance(item, dict) and "cache_control" in item:
|
||||
cache_control = item["cache_control"]
|
||||
if (
|
||||
isinstance(cache_control, dict)
|
||||
and "ttl" in cache_control
|
||||
):
|
||||
ttl = cache_control["ttl"]
|
||||
if is_claude_4_5 and ttl in ["5m", "1h"]:
|
||||
continue
|
||||
|
||||
cache_control.pop("ttl", None)
|
||||
_process_content_list(content)
|
||||
|
||||
def _supports_extended_thinking_on_bedrock(self, model: str) -> bool:
|
||||
"""
|
||||
|
|
@ -402,6 +419,12 @@ class AmazonAnthropicClaudeMessagesConfig(
|
|||
anthropic_messages_request=anthropic_messages_request,
|
||||
)
|
||||
|
||||
# 5a. Remove `custom` field from tools (Bedrock doesn't support it)
|
||||
# Claude Code sends `custom: {defer_loading: true}` on tool definitions,
|
||||
# which causes Bedrock to reject the request with "Extra inputs are not permitted"
|
||||
# Ref: https://github.com/BerriAI/litellm/issues/22847
|
||||
remove_custom_field_from_tools(anthropic_messages_request)
|
||||
|
||||
# 6. AUTO-INJECT beta headers based on features used
|
||||
anthropic_model_info = AnthropicModelInfo()
|
||||
tools = anthropic_messages_optional_request_params.get("tools")
|
||||
|
|
|
|||
80
litellm/llms/bedrock_mantle/chat/transformation.py
Normal file
80
litellm/llms/bedrock_mantle/chat/transformation.py
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
"""
|
||||
Amazon Bedrock Mantle - OpenAI-compatible inference engine in Amazon Bedrock.
|
||||
|
||||
API docs: https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-mantle.html
|
||||
|
||||
Base URL: https://bedrock-mantle.{region}.api.aws/v1
|
||||
Auth: AWS Bedrock API key as Bearer token (set via BEDROCK_MANTLE_API_KEY env var)
|
||||
or region-aware key via BEDROCK_MANTLE_{REGION}_API_KEY.
|
||||
"""
|
||||
|
||||
from typing import Iterator, AsyncIterator, Any, Optional, Tuple, Union
|
||||
|
||||
import litellm
|
||||
from litellm._logging import verbose_logger
|
||||
from litellm.secret_managers.main import get_secret_str
|
||||
|
||||
from ...openai_like.chat.transformation import OpenAILikeChatConfig
|
||||
|
||||
|
||||
BEDROCK_MANTLE_DEFAULT_REGION = "us-east-1"
|
||||
|
||||
|
||||
class BedrockMantleChatConfig(OpenAILikeChatConfig):
|
||||
"""
|
||||
Transformation config for Amazon Bedrock Mantle OpenAI-compatible API.
|
||||
"""
|
||||
|
||||
@property
|
||||
def custom_llm_provider(self) -> Optional[str]:
|
||||
return "bedrock_mantle"
|
||||
|
||||
@classmethod
|
||||
def get_config(cls):
|
||||
return super().get_config()
|
||||
|
||||
def _get_openai_compatible_provider_info(
|
||||
self, api_base: Optional[str], api_key: Optional[str]
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
region = (
|
||||
get_secret_str("BEDROCK_MANTLE_REGION")
|
||||
or get_secret_str("AWS_REGION")
|
||||
or BEDROCK_MANTLE_DEFAULT_REGION
|
||||
)
|
||||
api_base = (
|
||||
api_base
|
||||
or get_secret_str("BEDROCK_MANTLE_API_BASE")
|
||||
or f"https://bedrock-mantle.{region}.api.aws/v1"
|
||||
)
|
||||
dynamic_api_key = api_key or get_secret_str("BEDROCK_MANTLE_API_KEY")
|
||||
return api_base, dynamic_api_key
|
||||
|
||||
def get_supported_openai_params(self, model: str) -> list:
|
||||
base_params = super().get_supported_openai_params(model)
|
||||
try:
|
||||
if litellm.supports_reasoning(
|
||||
model=model, custom_llm_provider=self.custom_llm_provider
|
||||
):
|
||||
if "reasoning_effort" not in base_params:
|
||||
base_params.append("reasoning_effort")
|
||||
except Exception as e:
|
||||
verbose_logger.debug(
|
||||
f"BedrockMantleChatConfig: error checking reasoning support: {e}"
|
||||
)
|
||||
return base_params
|
||||
|
||||
def get_model_response_iterator(
|
||||
self,
|
||||
streaming_response: Union[Iterator[str], AsyncIterator[str], Any],
|
||||
sync_stream: bool,
|
||||
json_mode: Optional[bool] = False,
|
||||
) -> Any:
|
||||
from litellm.llms.openai.chat.gpt_transformation import (
|
||||
OpenAIChatCompletionStreamingHandler,
|
||||
)
|
||||
|
||||
return OpenAIChatCompletionStreamingHandler(
|
||||
streaming_response=streaming_response,
|
||||
sync_stream=sync_stream,
|
||||
json_mode=json_mode,
|
||||
)
|
||||
|
|
@ -4454,8 +4454,11 @@ class BaseLLMHTTPHandler:
|
|||
return agentic_response
|
||||
|
||||
except Exception as e:
|
||||
_call_id = getattr(logging_obj, "litellm_call_id", "unknown")
|
||||
verbose_logger.exception(
|
||||
f"LiteLLM.AgenticHookError: Exception in agentic completion hooks: {str(e)}"
|
||||
"LiteLLM.AgenticHookError: Exception in agentic completion hooks "
|
||||
"[call_id=%s model=%s]: %s",
|
||||
_call_id, model, str(e),
|
||||
)
|
||||
|
||||
# Check if we need to convert response to fake stream
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@
|
|||
from typing import Optional
|
||||
|
||||
import litellm
|
||||
from litellm.utils import _supports_factory
|
||||
|
||||
from .gpt_transformation import OpenAIGPTConfig
|
||||
|
||||
|
|
@ -40,41 +41,25 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
|
|||
"""Check if the model is specifically a GPT-5 Codex variant."""
|
||||
return "gpt-5-codex" in model
|
||||
|
||||
@classmethod
|
||||
def is_model_gpt_5_1_codex_max_model(cls, model: str) -> bool:
|
||||
"""Check if the model is the gpt-5.1-codex-max variant."""
|
||||
model_name = model.split("/")[-1] # handle provider prefixes
|
||||
return model_name == "gpt-5.1-codex-max"
|
||||
|
||||
@classmethod
|
||||
def is_model_gpt_5_1_model(cls, model: str) -> bool:
|
||||
"""Check if the model is a gpt-5.1 or gpt-5.2 chat variant.
|
||||
|
||||
gpt-5.1/5.2 support temperature when reasoning_effort="none",
|
||||
unlike base gpt-5 which only supports temperature=1. Excludes
|
||||
pro variants which keep stricter knobs and gpt-5.2-chat variants
|
||||
which only support temperature=1.
|
||||
"""
|
||||
model_name = model.split("/")[-1]
|
||||
is_gpt_5_1 = model_name.startswith("gpt-5.1")
|
||||
is_gpt_5_2 = (
|
||||
model_name.startswith("gpt-5.2")
|
||||
and "pro" not in model_name
|
||||
and not model_name.startswith("gpt-5.2-chat")
|
||||
)
|
||||
return is_gpt_5_1 or is_gpt_5_2
|
||||
|
||||
@classmethod
|
||||
def is_model_gpt_5_2_pro_model(cls, model: str) -> bool:
|
||||
"""Check if the model is the gpt-5.2-pro snapshot/alias."""
|
||||
model_name = model.split("/")[-1]
|
||||
return model_name.startswith("gpt-5.2-pro")
|
||||
|
||||
@classmethod
|
||||
def is_model_gpt_5_2_model(cls, model: str) -> bool:
|
||||
"""Check if the model is a gpt-5.2 variant (including pro)."""
|
||||
model_name = model.split("/")[-1]
|
||||
return model_name.startswith("gpt-5.2")
|
||||
return model_name.startswith("gpt-5.2") or model_name.startswith("gpt-5.4")
|
||||
|
||||
@classmethod
|
||||
def _supports_reasoning_effort_level(cls, model: str, level: str) -> bool:
|
||||
"""Check if the model supports a specific reasoning_effort level.
|
||||
|
||||
Looks up ``supports_{level}_reasoning_effort`` in the model map via
|
||||
the shared ``_supports_factory`` helper.
|
||||
Returns False for unknown models (safe fallback).
|
||||
"""
|
||||
return _supports_factory(
|
||||
model=model,
|
||||
custom_llm_provider=None,
|
||||
key=f"supports_{level}_reasoning_effort",
|
||||
)
|
||||
|
||||
def get_supported_openai_params(self, model: str) -> list:
|
||||
if self.is_model_gpt_5_search_model(model):
|
||||
|
|
@ -114,7 +99,7 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
|
|||
]
|
||||
|
||||
# gpt-5.1/5.2 support logprobs, top_p, top_logprobs when reasoning_effort="none"
|
||||
if not self.is_model_gpt_5_1_model(model):
|
||||
if not self._supports_reasoning_effort_level(model, "none"):
|
||||
non_supported_params.extend(["logprobs", "top_p", "top_logprobs"])
|
||||
|
||||
return [
|
||||
|
|
@ -147,16 +132,13 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
|
|||
or optional_params.get("reasoning_effort")
|
||||
)
|
||||
if reasoning_effort is not None and reasoning_effort == "xhigh":
|
||||
if not (
|
||||
self.is_model_gpt_5_1_codex_max_model(model)
|
||||
or self.is_model_gpt_5_2_model(model)
|
||||
):
|
||||
if not self._supports_reasoning_effort_level(model, "xhigh"):
|
||||
if litellm.drop_params or drop_params:
|
||||
non_default_params.pop("reasoning_effort", None)
|
||||
else:
|
||||
raise litellm.utils.UnsupportedParamsError(
|
||||
message=(
|
||||
"reasoning_effort='xhigh' is only supported for gpt-5.1-codex-max and gpt-5.2 models."
|
||||
"reasoning_effort='xhigh' is only supported for gpt-5.1-codex-max, gpt-5.2, and gpt-5.4+ models."
|
||||
),
|
||||
status_code=400,
|
||||
)
|
||||
|
|
@ -171,7 +153,8 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
|
|||
)
|
||||
|
||||
# gpt-5.1/5.2 support logprobs, top_p, top_logprobs only when reasoning_effort="none"
|
||||
if self.is_model_gpt_5_1_model(model):
|
||||
supports_none = self._supports_reasoning_effort_level(model, "none")
|
||||
if supports_none:
|
||||
sampling_params = ["logprobs", "top_logprobs", "top_p"]
|
||||
has_sampling = any(p in non_default_params for p in sampling_params)
|
||||
if has_sampling and reasoning_effort not in (None, "none"):
|
||||
|
|
@ -181,7 +164,7 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
|
|||
else:
|
||||
raise litellm.utils.UnsupportedParamsError(
|
||||
message=(
|
||||
"gpt-5.1/5.2 only support logprobs, top_p, top_logprobs when "
|
||||
"gpt-5.1/5.2/5.4 only support logprobs, top_p, top_logprobs when "
|
||||
"reasoning_effort='none'. Current reasoning_effort='{}'. "
|
||||
"To drop unsupported params set `litellm.drop_params = True`"
|
||||
).format(reasoning_effort),
|
||||
|
|
@ -191,10 +174,8 @@ class OpenAIGPT5Config(OpenAIGPTConfig):
|
|||
if "temperature" in non_default_params:
|
||||
temperature_value: Optional[float] = non_default_params.pop("temperature")
|
||||
if temperature_value is not None:
|
||||
is_gpt_5_1 = self.is_model_gpt_5_1_model(model)
|
||||
|
||||
# gpt-5.1 supports any temperature when reasoning_effort="none" (or not specified, as it defaults to "none")
|
||||
if is_gpt_5_1 and (reasoning_effort == "none" or reasoning_effort is None):
|
||||
# models supporting reasoning_effort="none" also support flexible temperature
|
||||
if supports_none and (reasoning_effort == "none" or reasoning_effort is None):
|
||||
optional_params["temperature"] = temperature_value
|
||||
elif temperature_value == 1:
|
||||
optional_params["temperature"] = temperature_value
|
||||
|
|
|
|||
|
|
@ -131,7 +131,10 @@ class OpenAIOSeriesConfig(OpenAIGPTConfig):
|
|||
|
||||
def is_model_o_series_model(self, model: str) -> bool:
|
||||
model = model.split("/")[-1] # could be "openai/o3" or "o3"
|
||||
return model.startswith(("o1", "o3", "o4")) and model in litellm.open_ai_chat_completion_models
|
||||
return (
|
||||
len(model) > 1 and model[0] == "o" and model[1].isdigit()
|
||||
and model in litellm.open_ai_chat_completion_models
|
||||
)
|
||||
|
||||
@overload
|
||||
def _transform_messages(
|
||||
|
|
|
|||
|
|
@ -1984,7 +1984,7 @@ class OpenAIBatchesAPI(BaseLLM):
|
|||
openai_client: AsyncOpenAI,
|
||||
) -> LiteLLMBatch:
|
||||
verbose_logger.debug("retrieving batch, args= %s", retrieve_batch_data)
|
||||
response = await openai_client.batches.retrieve(**retrieve_batch_data)
|
||||
response = await openai_client.batches.retrieve(**retrieve_batch_data) # type: ignore[arg-type]
|
||||
return LiteLLMBatch(**response.model_dump())
|
||||
|
||||
def retrieve_batch(
|
||||
|
|
@ -2020,7 +2020,7 @@ class OpenAIBatchesAPI(BaseLLM):
|
|||
return self.aretrieve_batch( # type: ignore
|
||||
retrieve_batch_data=retrieve_batch_data, openai_client=openai_client
|
||||
)
|
||||
response = cast(OpenAI, openai_client).batches.retrieve(**retrieve_batch_data)
|
||||
response = cast(OpenAI, openai_client).batches.retrieve(**retrieve_batch_data) # type: ignore[arg-type]
|
||||
return LiteLLMBatch(**response.model_dump())
|
||||
|
||||
async def acancel_batch(
|
||||
|
|
|
|||
|
|
@ -91,9 +91,9 @@ class OpenRouterImageEditConfig(BaseImageEditConfig):
|
|||
if key == "size":
|
||||
if "image_config" not in mapped_params:
|
||||
mapped_params["image_config"] = {}
|
||||
mapped_params["image_config"]["aspect_ratio"] = self._map_size_to_aspect_ratio(str(value))
|
||||
mapped_params["image_config"]["aspect_ratio"] = self._map_size_to_aspect_ratio(cast(str, value))
|
||||
elif key == "quality":
|
||||
image_size = self._map_quality_to_image_size(str(value))
|
||||
image_size = self._map_quality_to_image_size(cast(str, value))
|
||||
if image_size:
|
||||
if "image_config" not in mapped_params:
|
||||
mapped_params["image_config"] = {}
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ Calls SearchAPI.io's Google Search API endpoint.
|
|||
|
||||
SearchAPI.io API Reference: https://www.searchapi.io/docs/google
|
||||
"""
|
||||
from typing import Dict, List, Literal, Optional, TypedDict, Union
|
||||
from typing import Dict, List, Literal, Optional, TypedDict, Union, cast
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import httpx
|
||||
|
|
@ -164,7 +164,7 @@ class SearchAPIConfig(BaseSearchConfig):
|
|||
|
||||
if "country" in optional_params:
|
||||
# Map to gl parameter
|
||||
result_data["gl"] = optional_params["country"].lower()
|
||||
result_data["gl"] = cast(str, optional_params["country"]).lower()
|
||||
|
||||
# Pass through all other SearchAPI.io-specific parameters
|
||||
for param, value in optional_params.items():
|
||||
|
|
|
|||
|
|
@ -595,6 +595,8 @@ def _transform_request_body(
|
|||
safety_settings: Optional[List[SafetSettingsConfig]] = optional_params.pop(
|
||||
"safety_settings", None
|
||||
) # type: ignore
|
||||
# Drop output_config as it's not supported by Vertex AI
|
||||
optional_params.pop("output_config", None)
|
||||
config_fields = GenerationConfig.__annotations__.keys()
|
||||
|
||||
# If the LiteLLM client sends Gemini-supported parameter "labels", add it
|
||||
|
|
|
|||
|
|
@ -800,9 +800,10 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
|
|||
GeminiThinkingConfig with thinkingLevel and includeThoughts
|
||||
"""
|
||||
# Check if this is gemini-3-flash which supports MINIMAL thinking level
|
||||
# Covers gemini-3-flash, gemini-3-flash-preview, gemini-3.1-flash, gemini-3.1-flash-lite-preview, etc.
|
||||
is_gemini3flash = model and (
|
||||
"gemini-3-flash-preview" in model.lower()
|
||||
or "gemini-3-flash" in model.lower()
|
||||
"gemini-3-flash" in model.lower()
|
||||
or "gemini-3.1-flash" in model.lower()
|
||||
)
|
||||
is_gemini31pro = model and (
|
||||
"gemini-3.1-pro-preview" in model.lower()
|
||||
|
|
|
|||
|
|
@ -152,4 +152,8 @@ class VertexAIPartnerModelsAnthropicMessagesConfig(AnthropicMessagesConfig, Vert
|
|||
"output_format", None
|
||||
) # do not pass output_format in request body to vertex ai - vertex ai does not support output_format as yet
|
||||
|
||||
anthropic_messages_request.pop(
|
||||
"output_config", None
|
||||
) # do not pass output_config in request body to vertex ai - vertex ai does not support output_config
|
||||
|
||||
return anthropic_messages_request
|
||||
|
|
|
|||
|
|
@ -107,6 +107,9 @@ class VertexAIAnthropicConfig(AnthropicConfig):
|
|||
|
||||
# VertexAI doesn't support output_format parameter, remove it if present
|
||||
data.pop("output_format", None)
|
||||
|
||||
# VertexAI doesn't support output_config parameter, remove it if present
|
||||
data.pop("output_config", None)
|
||||
|
||||
tools = optional_params.get("tools")
|
||||
tool_search_used = self.is_tool_search_used(tools)
|
||||
|
|
|
|||
|
|
@ -2241,6 +2241,32 @@ def completion( # type: ignore # noqa: PLR0915
|
|||
logging_obj=logging, # model call logging done inside the class as we make need to modify I/O to fit aleph alpha's requirements
|
||||
client=client,
|
||||
)
|
||||
elif custom_llm_provider == "bedrock_mantle":
|
||||
api_base = api_base or litellm.api_base or get_secret("BEDROCK_MANTLE_API_BASE")
|
||||
api_key = api_key or litellm.api_key or get_secret("BEDROCK_MANTLE_API_KEY")
|
||||
headers = headers or litellm.headers
|
||||
config = litellm.BedrockMantleChatConfig.get_config()
|
||||
for k, v in config.items():
|
||||
if k not in optional_params:
|
||||
optional_params[k] = v
|
||||
response = base_llm_http_handler.completion(
|
||||
model=model,
|
||||
stream=stream,
|
||||
messages=messages,
|
||||
acompletion=acompletion,
|
||||
api_base=api_base,
|
||||
model_response=model_response,
|
||||
optional_params=optional_params,
|
||||
litellm_params=litellm_params,
|
||||
shared_session=shared_session,
|
||||
custom_llm_provider=custom_llm_provider,
|
||||
timeout=timeout,
|
||||
headers=headers,
|
||||
encoding=_get_encoding(),
|
||||
api_key=api_key,
|
||||
logging_obj=logging,
|
||||
client=client,
|
||||
)
|
||||
elif custom_llm_provider == "a2a":
|
||||
# A2A (Agent-to-Agent) Protocol
|
||||
# Resolve agent configuration from registry if model format is "a2a/<agent-name>"
|
||||
|
|
|
|||
|
|
@ -1239,7 +1239,7 @@
|
|||
"supports_vision": true,
|
||||
"tool_use_system_prompt_tokens": 346
|
||||
},
|
||||
"apac.anthropic.claude-sonnet-4-6": {
|
||||
"au.anthropic.claude-sonnet-4-6": {
|
||||
"cache_creation_input_token_cost": 4.125e-06,
|
||||
"cache_creation_input_token_cost_above_200k_tokens": 8.25e-06,
|
||||
"cache_read_input_token_cost": 3.3e-07,
|
||||
|
|
@ -5817,6 +5817,15 @@
|
|||
],
|
||||
"source": "https://devblogs.microsoft.com/foundry/whats-new-in-azure-ai-foundry-august-2025/#mistral-document-ai-(ocr)-%E2%80%94-serverless-in-foundry"
|
||||
},
|
||||
"azure_ai/mistral-document-ai-2512": {
|
||||
"litellm_provider": "azure_ai",
|
||||
"ocr_cost_per_page": 0.003,
|
||||
"mode": "ocr",
|
||||
"supported_endpoints": [
|
||||
"/v1/ocr"
|
||||
],
|
||||
"source": "https://azure.microsoft.com/en-us/pricing/details/ai-foundry-models/"
|
||||
},
|
||||
"azure_ai/doc-intelligence/prebuilt-read": {
|
||||
"litellm_provider": "azure_ai",
|
||||
"ocr_cost_per_page": 0.0015,
|
||||
|
|
@ -18428,6 +18437,93 @@
|
|||
"max_tokens": 8191,
|
||||
"mode": "embedding"
|
||||
},
|
||||
"chatgpt/gpt-5.4": {
|
||||
"litellm_provider": "chatgpt",
|
||||
"max_input_tokens": 1050000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"chatgpt/gpt-5.4-pro": {
|
||||
"litellm_provider": "chatgpt",
|
||||
"max_input_tokens": 1050000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"supported_endpoints": [
|
||||
"/v1/responses"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"chatgpt/gpt-5.3-codex": {
|
||||
"litellm_provider": "chatgpt",
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"supported_endpoints": [
|
||||
"/v1/responses"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"chatgpt/gpt-5.3-codex-spark": {
|
||||
"litellm_provider": "chatgpt",
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"supported_endpoints": [
|
||||
"/v1/responses"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"chatgpt/gpt-5.3-instant": {
|
||||
"litellm_provider": "chatgpt",
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 64000,
|
||||
"max_tokens": 64000,
|
||||
"mode": "responses",
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"chatgpt/gpt-5.3-chat-latest": {
|
||||
"litellm_provider": "chatgpt",
|
||||
"max_input_tokens": 128000,
|
||||
"max_output_tokens": 64000,
|
||||
"max_tokens": 64000,
|
||||
"mode": "responses",
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_vision": true
|
||||
},
|
||||
"chatgpt/gpt-5.2-codex": {
|
||||
"litellm_provider": "chatgpt",
|
||||
"max_input_tokens": 128000,
|
||||
|
|
@ -20497,7 +20593,9 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5.1": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -20533,7 +20631,10 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": true,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5.1-2025-11-13": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -20569,7 +20670,10 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": true,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5.1-chat-latest": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -20604,7 +20708,10 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": false,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": true,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5.2": {
|
||||
"cache_read_input_token_cost": 1.75e-07,
|
||||
|
|
@ -20641,7 +20748,10 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": true,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5.2-2025-12-11": {
|
||||
"cache_read_input_token_cost": 1.75e-07,
|
||||
|
|
@ -20678,7 +20788,10 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": true,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5.2-chat-latest": {
|
||||
"cache_read_input_token_cost": 1.75e-07,
|
||||
|
|
@ -20712,7 +20825,10 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5.3-chat-latest": {
|
||||
"cache_read_input_token_cost": 1.75e-07,
|
||||
|
|
@ -20746,7 +20862,10 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5.2-pro": {
|
||||
"input_cost_per_token": 2.1e-05,
|
||||
|
|
@ -20777,7 +20896,9 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5.2-pro-2025-12-11": {
|
||||
"input_cost_per_token": 2.1e-05,
|
||||
|
|
@ -20808,8 +20929,226 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5.4": {
|
||||
"cache_read_input_token_cost": 2.5e-07,
|
||||
"cache_read_input_token_cost_priority": 5e-07,
|
||||
"input_cost_per_token": 2.5e-06,
|
||||
"input_cost_per_token_priority": 5e-06,
|
||||
"litellm_provider": "openai",
|
||||
"max_input_tokens": 1050000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"output_cost_per_token": 1.5e-05,
|
||||
"output_cost_per_token_priority": 2.25e-05,
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/batch",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": true,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5.4-2026-03-05": {
|
||||
"cache_read_input_token_cost": 2.5e-07,
|
||||
"cache_read_input_token_cost_priority": 5e-07,
|
||||
"input_cost_per_token": 2.5e-06,
|
||||
"input_cost_per_token_priority": 5e-06,
|
||||
"litellm_provider": "openai",
|
||||
"max_input_tokens": 1050000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"output_cost_per_token": 1.5e-05,
|
||||
"output_cost_per_token_priority": 2.25e-05,
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/batch",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": true,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5.4-pro": {
|
||||
"cache_read_input_token_cost": 2e-06,
|
||||
"input_cost_per_token": 2e-05,
|
||||
"litellm_provider": "openai",
|
||||
"max_input_tokens": 1050000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"output_cost_per_token": 0.00012,
|
||||
"supported_endpoints": [
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
},
|
||||
"gpt-5.4-pro-2026-03-05": {
|
||||
"cache_read_input_token_cost": 2e-06,
|
||||
"input_cost_per_token": 2e-05,
|
||||
"litellm_provider": "openai",
|
||||
"max_input_tokens": 1050000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"output_cost_per_token": 0.00012,
|
||||
"supported_endpoints": [
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
},
|
||||
"gpt-5.4-pro": {
|
||||
"cache_read_input_token_cost": 3e-06,
|
||||
"cache_read_input_token_cost_priority": 6e-06,
|
||||
"input_cost_per_token": 3e-05,
|
||||
"input_cost_per_token_priority": 6e-05,
|
||||
"litellm_provider": "openai",
|
||||
"max_input_tokens": 1050000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"output_cost_per_token": 1.8e-04,
|
||||
"output_cost_per_token_priority": 2.7e-04,
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/batch",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": false,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5.4-pro-2026-03-05": {
|
||||
"cache_read_input_token_cost": 3e-06,
|
||||
"cache_read_input_token_cost_priority": 6e-06,
|
||||
"input_cost_per_token": 3e-05,
|
||||
"input_cost_per_token_priority": 6e-05,
|
||||
"litellm_provider": "openai",
|
||||
"max_input_tokens": 1050000,
|
||||
"max_output_tokens": 128000,
|
||||
"max_tokens": 128000,
|
||||
"mode": "responses",
|
||||
"output_cost_per_token": 1.8e-04,
|
||||
"output_cost_per_token_priority": 2.7e-04,
|
||||
"supported_endpoints": [
|
||||
"/v1/chat/completions",
|
||||
"/v1/batch",
|
||||
"/v1/responses"
|
||||
],
|
||||
"supported_modalities": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"supported_output_modalities": [
|
||||
"text"
|
||||
],
|
||||
"supports_function_calling": true,
|
||||
"supports_native_streaming": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_pdf_input": true,
|
||||
"supports_prompt_caching": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": false,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5-pro": {
|
||||
"input_cost_per_token": 1.5e-05,
|
||||
"input_cost_per_token_batches": 7.5e-06,
|
||||
|
|
@ -20841,7 +21180,9 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-pro-2025-10-06": {
|
||||
"input_cost_per_token": 1.5e-05,
|
||||
|
|
@ -20874,7 +21215,9 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-2025-08-07": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -20913,7 +21256,9 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-chat": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -20945,7 +21290,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": false,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-chat-latest": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -20977,7 +21324,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": false,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-codex": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -21007,7 +21356,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": false,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5.1-codex": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -21040,7 +21391,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": false,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5.1-codex-max": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -21070,7 +21423,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": false,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5.1-codex-mini": {
|
||||
"cache_read_input_token_cost": 2.5e-08,
|
||||
|
|
@ -21103,7 +21458,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": false,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5.2-codex": {
|
||||
"cache_read_input_token_cost": 1.75e-07,
|
||||
|
|
@ -21136,7 +21493,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": false,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": true
|
||||
},
|
||||
"gpt-5.3-codex": {
|
||||
"cache_read_input_token_cost": 1.75e-07,
|
||||
|
|
@ -21169,7 +21528,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": false,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-mini": {
|
||||
"cache_read_input_token_cost": 2.5e-08,
|
||||
|
|
@ -21208,7 +21569,9 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-mini-2025-08-07": {
|
||||
"cache_read_input_token_cost": 2.5e-08,
|
||||
|
|
@ -21247,7 +21610,9 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_service_tier": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-nano": {
|
||||
"cache_read_input_token_cost": 5e-09,
|
||||
|
|
@ -21283,7 +21648,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-nano-2025-08-07": {
|
||||
"cache_read_input_token_cost": 5e-09,
|
||||
|
|
@ -21318,7 +21685,9 @@
|
|||
"supports_response_schema": true,
|
||||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true
|
||||
"supports_vision": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-image-1": {
|
||||
"cache_read_input_image_token_cost": 2.5e-06,
|
||||
|
|
@ -38478,7 +38847,9 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-5-search-api-2025-10-14": {
|
||||
"cache_read_input_token_cost": 1.25e-07,
|
||||
|
|
@ -38497,7 +38868,9 @@
|
|||
"supports_system_messages": true,
|
||||
"supports_tool_choice": true,
|
||||
"supports_vision": true,
|
||||
"supports_web_search": true
|
||||
"supports_web_search": true,
|
||||
"supports_none_reasoning_effort": false,
|
||||
"supports_xhigh_reasoning_effort": false
|
||||
},
|
||||
"gpt-realtime-mini-2025-10-06": {
|
||||
"cache_creation_input_audio_token_cost": 3e-07,
|
||||
|
|
@ -39140,5 +39513,59 @@
|
|||
"metadata": {
|
||||
"notes": "DuckDuckGo Instant Answer API is free and does not require an API key."
|
||||
}
|
||||
},
|
||||
"bedrock_mantle/openai.gpt-oss-120b": {
|
||||
"input_cost_per_token": 1.5e-07,
|
||||
"output_cost_per_token": 6e-07,
|
||||
"litellm_provider": "bedrock_mantle",
|
||||
"max_input_tokens": 131072,
|
||||
"max_output_tokens": 32768,
|
||||
"max_tokens": 32768,
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true
|
||||
},
|
||||
"bedrock_mantle/openai.gpt-oss-20b": {
|
||||
"input_cost_per_token": 7.5e-08,
|
||||
"output_cost_per_token": 3e-07,
|
||||
"litellm_provider": "bedrock_mantle",
|
||||
"max_input_tokens": 131072,
|
||||
"max_output_tokens": 32768,
|
||||
"max_tokens": 32768,
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_parallel_function_calling": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true
|
||||
},
|
||||
"bedrock_mantle/openai.gpt-oss-safeguard-120b": {
|
||||
"input_cost_per_token": 1.5e-07,
|
||||
"output_cost_per_token": 6e-07,
|
||||
"litellm_provider": "bedrock_mantle",
|
||||
"max_input_tokens": 131072,
|
||||
"max_output_tokens": 65536,
|
||||
"max_tokens": 65536,
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true
|
||||
},
|
||||
"bedrock_mantle/openai.gpt-oss-safeguard-20b": {
|
||||
"input_cost_per_token": 7.5e-08,
|
||||
"output_cost_per_token": 3e-07,
|
||||
"litellm_provider": "bedrock_mantle",
|
||||
"max_input_tokens": 131072,
|
||||
"max_output_tokens": 65536,
|
||||
"max_tokens": 65536,
|
||||
"mode": "chat",
|
||||
"supports_function_calling": true,
|
||||
"supports_reasoning": true,
|
||||
"supports_response_schema": true,
|
||||
"supports_tool_choice": true
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -289,10 +289,10 @@ def llm_passthrough_route(
|
|||
request = client.client.build_request(
|
||||
method=method,
|
||||
url=updated_url,
|
||||
content=signed_json_body,
|
||||
data=data if signed_json_body is None else None,
|
||||
content=signed_json_body if signed_json_body is not None else content,
|
||||
data=data if (signed_json_body is None and content is None) else None,
|
||||
files=files,
|
||||
json=json if signed_json_body is None else None,
|
||||
json=json if (signed_json_body is None and content is None) else None,
|
||||
params=params,
|
||||
headers=headers,
|
||||
cookies=cookies,
|
||||
|
|
@ -410,8 +410,9 @@ async def _async_streaming(
|
|||
litellm_logging_obj: "LiteLLMLoggingObj",
|
||||
provider_config: "BasePassthroughConfig",
|
||||
):
|
||||
iter_response = await response
|
||||
try:
|
||||
iter_response = await response
|
||||
iter_response.raise_for_status()
|
||||
raw_bytes: List[bytes] = []
|
||||
|
||||
async for chunk in iter_response.aiter_bytes(): # type: ignore
|
||||
|
|
@ -425,5 +426,9 @@ async def _async_streaming(
|
|||
provider_config=provider_config,
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
raise e
|
||||
except Exception:
|
||||
try:
|
||||
await iter_response.aclose()
|
||||
except Exception:
|
||||
pass
|
||||
raise
|
||||
|
|
|
|||
|
|
@ -124,6 +124,8 @@ def extract_parameters(operation: Dict[str, Any]) -> tuple:
|
|||
# OpenAPI 3.x and 2.x parameters
|
||||
if "parameters" in operation:
|
||||
for param in operation["parameters"]:
|
||||
if "name" not in param:
|
||||
continue
|
||||
param_name = param["name"]
|
||||
if param.get("in") == "path":
|
||||
path_params.append(param_name)
|
||||
|
|
@ -147,6 +149,8 @@ def build_input_schema(operation: Dict[str, Any]) -> Dict[str, Any]:
|
|||
# Process parameters
|
||||
if "parameters" in operation:
|
||||
for param in operation["parameters"]:
|
||||
if "name" not in param:
|
||||
continue
|
||||
param_name = param["name"]
|
||||
param_schema = param.get("schema", {})
|
||||
param_type = param_schema.get("type", "string")
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -1,32 +1,29 @@
|
|||
1:"$Sreact.fragment"
|
||||
2:I[347257,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"ClientPageRoot"]
|
||||
3:I[952683,["/litellm-asset-prefix/_next/static/chunks/26adfa4e8ffc85c7.js","/litellm-asset-prefix/_next/static/chunks/e8ed72789c2b42ff.js","/litellm-asset-prefix/_next/static/chunks/61f59596bf7f0628.js","/litellm-asset-prefix/_next/static/chunks/9f5ccd929375c1d6.js","/litellm-asset-prefix/_next/static/chunks/30539b80ac15aad2.js","/litellm-asset-prefix/_next/static/chunks/142704439974f6b3.js","/litellm-asset-prefix/_next/static/chunks/8b4fd43197a5dfc8.js","/litellm-asset-prefix/_next/static/chunks/1fe0596a309ad6cf.js","/litellm-asset-prefix/_next/static/chunks/7b788dd93ad868b3.js","/litellm-asset-prefix/_next/static/chunks/c979f252fa5ee77c.js","/litellm-asset-prefix/_next/static/chunks/50febaabfb896c46.js","/litellm-asset-prefix/_next/static/chunks/c4cc89d1b0a147d5.js","/litellm-asset-prefix/_next/static/chunks/fe5201571c777f09.js","/litellm-asset-prefix/_next/static/chunks/c748f222de4766db.js","/litellm-asset-prefix/_next/static/chunks/5f9c3b92a016f382.js","/litellm-asset-prefix/_next/static/chunks/fe750aa0bf04912c.js","/litellm-asset-prefix/_next/static/chunks/7d4cded1a1238581.js","/litellm-asset-prefix/_next/static/chunks/d64d74932cb225a3.js","/litellm-asset-prefix/_next/static/chunks/9ffb8ddd0c9a7c31.js","/litellm-asset-prefix/_next/static/chunks/c4fe728e74b52958.js","/litellm-asset-prefix/_next/static/chunks/1532edb438ed84bb.js","/litellm-asset-prefix/_next/static/chunks/d0d828f9a0668699.js","/litellm-asset-prefix/_next/static/chunks/88c74f8b4b20d25a.js","/litellm-asset-prefix/_next/static/chunks/25c5a08661ac2ec3.js","/litellm-asset-prefix/_next/static/chunks/134f728fa7099e3e.js","/litellm-asset-prefix/_next/static/chunks/9d6e5aad99b19216.js","/litellm-asset-prefix/_next/static/chunks/e8718f949e42598e.js","/litellm-asset-prefix/_next/static/chunks/6a51328383335d1e.js","/litellm-asset-prefix/_next/static/chunks/3b30ab8eaa03bc21.js","/litellm-asset-prefix/_next/static/chunks/99cf9cf99df5ccfc.js","/litellm-asset-prefix/_next/static/chunks/6a4eede876bb5c8f.js","/litellm-asset-prefix/_next/static/chunks/7e417dd24c8becd0.js","/litellm-asset-prefix/_next/static/chunks/c7b74067c01ee971.js","/litellm-asset-prefix/_next/static/chunks/a85adee4198d5478.js","/litellm-asset-prefix/_next/static/chunks/bff4459534c52f16.js","/litellm-asset-prefix/_next/static/chunks/39eb5927f56ccc0e.js","/litellm-asset-prefix/_next/static/chunks/ebb2b2d8175d3d2f.js","/litellm-asset-prefix/_next/static/chunks/5a5488ab3db0c3de.js","/litellm-asset-prefix/_next/static/chunks/0a65da2cd24e2ab6.js","/litellm-asset-prefix/_next/static/chunks/82dbdb17d57c6737.js","/litellm-asset-prefix/_next/static/chunks/b31a08272a82b84e.js","/litellm-asset-prefix/_next/static/chunks/496b84010c33cf69.js","/litellm-asset-prefix/_next/static/chunks/eba9ac65320061b1.js","/litellm-asset-prefix/_next/static/chunks/0eda6dc5d5f35d92.js","/litellm-asset-prefix/_next/static/chunks/96aed36e606e8582.js","/litellm-asset-prefix/_next/static/chunks/ad426ab08aee6c64.js","/litellm-asset-prefix/_next/static/chunks/2971c4658f1bcd7d.js","/litellm-asset-prefix/_next/static/chunks/06ebe9b0e9cdf241.js","/litellm-asset-prefix/_next/static/chunks/6c4c97f1ea6e7d77.js","/litellm-asset-prefix/_next/static/chunks/165d00848f04c4c9.js","/litellm-asset-prefix/_next/static/chunks/bb0a6e4a3a18721a.js","/litellm-asset-prefix/_next/static/chunks/4980372eaa37b78b.js","/litellm-asset-prefix/_next/static/chunks/8ab6e2ca95000c8e.js","/litellm-asset-prefix/_next/static/chunks/06cdd9bb80c63794.js","/litellm-asset-prefix/_next/static/chunks/66a190706fc6c35a.js","/litellm-asset-prefix/_next/static/chunks/54e29148cb2f2582.js","/litellm-asset-prefix/_next/static/chunks/fd9de419c8c0222e.js"],"default"]
|
||||
1c:I[897367,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"OutletBoundary"]
|
||||
1d:"$Sreact.suspense"
|
||||
3:I[952683,["/litellm-asset-prefix/_next/static/chunks/a7f104aa2cc7f3f0.js","/litellm-asset-prefix/_next/static/chunks/e8ed72789c2b42ff.js","/litellm-asset-prefix/_next/static/chunks/db1ebd02d726c50f.js","/litellm-asset-prefix/_next/static/chunks/142704439974f6b3.js","/litellm-asset-prefix/_next/static/chunks/30539b80ac15aad2.js","/litellm-asset-prefix/_next/static/chunks/e55394619917c445.js","/litellm-asset-prefix/_next/static/chunks/df6665addf8c6036.js","/litellm-asset-prefix/_next/static/chunks/1fe0596a309ad6cf.js","/litellm-asset-prefix/_next/static/chunks/9d9e6235c06ebb90.js","/litellm-asset-prefix/_next/static/chunks/403c4d96324c23a6.js","/litellm-asset-prefix/_next/static/chunks/308c947b873bf49b.js","/litellm-asset-prefix/_next/static/chunks/88c74f8b4b20d25a.js","/litellm-asset-prefix/_next/static/chunks/2d471965761a22ff.js","/litellm-asset-prefix/_next/static/chunks/6eb81672801a9dec.js","/litellm-asset-prefix/_next/static/chunks/3b30ab8eaa03bc21.js","/litellm-asset-prefix/_next/static/chunks/7b788dd93ad868b3.js","/litellm-asset-prefix/_next/static/chunks/5f9c3b92a016f382.js","/litellm-asset-prefix/_next/static/chunks/38605193023f8e1c.js","/litellm-asset-prefix/_next/static/chunks/d0d828f9a0668699.js","/litellm-asset-prefix/_next/static/chunks/4d3d997560b322ca.js","/litellm-asset-prefix/_next/static/chunks/4e4d0f466b5c1780.js","/litellm-asset-prefix/_next/static/chunks/d64d74932cb225a3.js","/litellm-asset-prefix/_next/static/chunks/3fb7a83546b6aa35.js","/litellm-asset-prefix/_next/static/chunks/134f728fa7099e3e.js","/litellm-asset-prefix/_next/static/chunks/9d6e5aad99b19216.js","/litellm-asset-prefix/_next/static/chunks/bb02cdce134f811b.js","/litellm-asset-prefix/_next/static/chunks/0f93c304f38c63d0.js","/litellm-asset-prefix/_next/static/chunks/130b80d41c79d98e.js","/litellm-asset-prefix/_next/static/chunks/348b31083769a7c4.js","/litellm-asset-prefix/_next/static/chunks/759173c2a452c43c.js","/litellm-asset-prefix/_next/static/chunks/4980372eaa37b78b.js","/litellm-asset-prefix/_next/static/chunks/c81bdad246412686.js","/litellm-asset-prefix/_next/static/chunks/b6cdb9a433f054f3.js","/litellm-asset-prefix/_next/static/chunks/a85adee4198d5478.js","/litellm-asset-prefix/_next/static/chunks/fcdf7322b0aa3e2e.js","/litellm-asset-prefix/_next/static/chunks/0adb91ab5f3140d5.js","/litellm-asset-prefix/_next/static/chunks/675f6c62ddae3031.js","/litellm-asset-prefix/_next/static/chunks/7e417dd24c8becd0.js","/litellm-asset-prefix/_next/static/chunks/0a65da2cd24e2ab6.js","/litellm-asset-prefix/_next/static/chunks/86c1e01d849eed8e.js","/litellm-asset-prefix/_next/static/chunks/eba9ac65320061b1.js","/litellm-asset-prefix/_next/static/chunks/85238af541b170ca.js","/litellm-asset-prefix/_next/static/chunks/8c09c924a98654d1.js","/litellm-asset-prefix/_next/static/chunks/b13b0ddeb85b5333.js","/litellm-asset-prefix/_next/static/chunks/2971c4658f1bcd7d.js","/litellm-asset-prefix/_next/static/chunks/496b84010c33cf69.js","/litellm-asset-prefix/_next/static/chunks/06ebe9b0e9cdf241.js","/litellm-asset-prefix/_next/static/chunks/6c4c97f1ea6e7d77.js","/litellm-asset-prefix/_next/static/chunks/e8718f949e42598e.js","/litellm-asset-prefix/_next/static/chunks/99cf9cf99df5ccfc.js","/litellm-asset-prefix/_next/static/chunks/bb0a6e4a3a18721a.js","/litellm-asset-prefix/_next/static/chunks/cfca40b1a4a490bf.js","/litellm-asset-prefix/_next/static/chunks/54e29148cb2f2582.js","/litellm-asset-prefix/_next/static/chunks/37c03dc421ba81f8.js"],"default"]
|
||||
19:I[897367,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"OutletBoundary"]
|
||||
1a:"$Sreact.suspense"
|
||||
:HL["/litellm-asset-prefix/_next/static/chunks/3f3fa56b5786d58c.css","style"]
|
||||
0:{"buildId":"U_YrOOnSehrpkdU42KJ-W","rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/3f3fa56b5786d58c.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/61f59596bf7f0628.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/9f5ccd929375c1d6.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/30539b80ac15aad2.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/142704439974f6b3.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/8b4fd43197a5dfc8.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/1fe0596a309ad6cf.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/7b788dd93ad868b3.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/c979f252fa5ee77c.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/50febaabfb896c46.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/c4cc89d1b0a147d5.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/fe5201571c777f09.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/c748f222de4766db.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/5f9c3b92a016f382.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/fe750aa0bf04912c.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/7d4cded1a1238581.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/d64d74932cb225a3.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/9ffb8ddd0c9a7c31.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/c4fe728e74b52958.js","async":true}],["$","script","script-18",{"src":"/litellm-asset-prefix/_next/static/chunks/1532edb438ed84bb.js","async":true}],["$","script","script-19",{"src":"/litellm-asset-prefix/_next/static/chunks/d0d828f9a0668699.js","async":true}],["$","script","script-20",{"src":"/litellm-asset-prefix/_next/static/chunks/88c74f8b4b20d25a.js","async":true}],["$","script","script-21",{"src":"/litellm-asset-prefix/_next/static/chunks/25c5a08661ac2ec3.js","async":true}],["$","script","script-22",{"src":"/litellm-asset-prefix/_next/static/chunks/134f728fa7099e3e.js","async":true}],["$","script","script-23",{"src":"/litellm-asset-prefix/_next/static/chunks/9d6e5aad99b19216.js","async":true}],["$","script","script-24",{"src":"/litellm-asset-prefix/_next/static/chunks/e8718f949e42598e.js","async":true}],["$","script","script-25",{"src":"/litellm-asset-prefix/_next/static/chunks/6a51328383335d1e.js","async":true}],["$","script","script-26",{"src":"/litellm-asset-prefix/_next/static/chunks/3b30ab8eaa03bc21.js","async":true}],["$","script","script-27",{"src":"/litellm-asset-prefix/_next/static/chunks/99cf9cf99df5ccfc.js","async":true}],["$","script","script-28",{"src":"/litellm-asset-prefix/_next/static/chunks/6a4eede876bb5c8f.js","async":true}],["$","script","script-29",{"src":"/litellm-asset-prefix/_next/static/chunks/7e417dd24c8becd0.js","async":true}],["$","script","script-30",{"src":"/litellm-asset-prefix/_next/static/chunks/c7b74067c01ee971.js","async":true}],["$","script","script-31",{"src":"/litellm-asset-prefix/_next/static/chunks/a85adee4198d5478.js","async":true}],["$","script","script-32",{"src":"/litellm-asset-prefix/_next/static/chunks/bff4459534c52f16.js","async":true}],["$","script","script-33",{"src":"/litellm-asset-prefix/_next/static/chunks/39eb5927f56ccc0e.js","async":true}],"$L6","$L7","$L8","$L9","$La","$Lb","$Lc","$Ld","$Le","$Lf","$L10","$L11","$L12","$L13","$L14","$L15","$L16","$L17","$L18","$L19","$L1a"],"$L1b"]}],"loading":null,"isPartial":false}
|
||||
0:{"buildId":"cbFGTIkRGp63usVNisey9","rsc":["$","$1","c",{"children":[["$","$L2",null,{"Component":"$3","serverProvidedParams":{"searchParams":{},"params":{},"promises":["$@4","$@5"]}}],[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/3f3fa56b5786d58c.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/db1ebd02d726c50f.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/142704439974f6b3.js","async":true}],["$","script","script-2",{"src":"/litellm-asset-prefix/_next/static/chunks/30539b80ac15aad2.js","async":true}],["$","script","script-3",{"src":"/litellm-asset-prefix/_next/static/chunks/e55394619917c445.js","async":true}],["$","script","script-4",{"src":"/litellm-asset-prefix/_next/static/chunks/df6665addf8c6036.js","async":true}],["$","script","script-5",{"src":"/litellm-asset-prefix/_next/static/chunks/1fe0596a309ad6cf.js","async":true}],["$","script","script-6",{"src":"/litellm-asset-prefix/_next/static/chunks/9d9e6235c06ebb90.js","async":true}],["$","script","script-7",{"src":"/litellm-asset-prefix/_next/static/chunks/403c4d96324c23a6.js","async":true}],["$","script","script-8",{"src":"/litellm-asset-prefix/_next/static/chunks/308c947b873bf49b.js","async":true}],["$","script","script-9",{"src":"/litellm-asset-prefix/_next/static/chunks/88c74f8b4b20d25a.js","async":true}],["$","script","script-10",{"src":"/litellm-asset-prefix/_next/static/chunks/2d471965761a22ff.js","async":true}],["$","script","script-11",{"src":"/litellm-asset-prefix/_next/static/chunks/6eb81672801a9dec.js","async":true}],["$","script","script-12",{"src":"/litellm-asset-prefix/_next/static/chunks/3b30ab8eaa03bc21.js","async":true}],["$","script","script-13",{"src":"/litellm-asset-prefix/_next/static/chunks/7b788dd93ad868b3.js","async":true}],["$","script","script-14",{"src":"/litellm-asset-prefix/_next/static/chunks/5f9c3b92a016f382.js","async":true}],["$","script","script-15",{"src":"/litellm-asset-prefix/_next/static/chunks/38605193023f8e1c.js","async":true}],["$","script","script-16",{"src":"/litellm-asset-prefix/_next/static/chunks/d0d828f9a0668699.js","async":true}],["$","script","script-17",{"src":"/litellm-asset-prefix/_next/static/chunks/4d3d997560b322ca.js","async":true}],["$","script","script-18",{"src":"/litellm-asset-prefix/_next/static/chunks/4e4d0f466b5c1780.js","async":true}],["$","script","script-19",{"src":"/litellm-asset-prefix/_next/static/chunks/d64d74932cb225a3.js","async":true}],["$","script","script-20",{"src":"/litellm-asset-prefix/_next/static/chunks/3fb7a83546b6aa35.js","async":true}],["$","script","script-21",{"src":"/litellm-asset-prefix/_next/static/chunks/134f728fa7099e3e.js","async":true}],["$","script","script-22",{"src":"/litellm-asset-prefix/_next/static/chunks/9d6e5aad99b19216.js","async":true}],["$","script","script-23",{"src":"/litellm-asset-prefix/_next/static/chunks/bb02cdce134f811b.js","async":true}],["$","script","script-24",{"src":"/litellm-asset-prefix/_next/static/chunks/0f93c304f38c63d0.js","async":true}],["$","script","script-25",{"src":"/litellm-asset-prefix/_next/static/chunks/130b80d41c79d98e.js","async":true}],["$","script","script-26",{"src":"/litellm-asset-prefix/_next/static/chunks/348b31083769a7c4.js","async":true}],["$","script","script-27",{"src":"/litellm-asset-prefix/_next/static/chunks/759173c2a452c43c.js","async":true}],["$","script","script-28",{"src":"/litellm-asset-prefix/_next/static/chunks/4980372eaa37b78b.js","async":true}],["$","script","script-29",{"src":"/litellm-asset-prefix/_next/static/chunks/c81bdad246412686.js","async":true}],["$","script","script-30",{"src":"/litellm-asset-prefix/_next/static/chunks/b6cdb9a433f054f3.js","async":true}],["$","script","script-31",{"src":"/litellm-asset-prefix/_next/static/chunks/a85adee4198d5478.js","async":true}],["$","script","script-32",{"src":"/litellm-asset-prefix/_next/static/chunks/fcdf7322b0aa3e2e.js","async":true}],["$","script","script-33",{"src":"/litellm-asset-prefix/_next/static/chunks/0adb91ab5f3140d5.js","async":true}],"$L6","$L7","$L8","$L9","$La","$Lb","$Lc","$Ld","$Le","$Lf","$L10","$L11","$L12","$L13","$L14","$L15","$L16","$L17"],"$L18"]}],"loading":null,"isPartial":false}
|
||||
4:{}
|
||||
5:"$0:rsc:props:children:0:props:serverProvidedParams:params"
|
||||
6:["$","script","script-34",{"src":"/litellm-asset-prefix/_next/static/chunks/ebb2b2d8175d3d2f.js","async":true}]
|
||||
7:["$","script","script-35",{"src":"/litellm-asset-prefix/_next/static/chunks/5a5488ab3db0c3de.js","async":true}]
|
||||
6:["$","script","script-34",{"src":"/litellm-asset-prefix/_next/static/chunks/675f6c62ddae3031.js","async":true}]
|
||||
7:["$","script","script-35",{"src":"/litellm-asset-prefix/_next/static/chunks/7e417dd24c8becd0.js","async":true}]
|
||||
8:["$","script","script-36",{"src":"/litellm-asset-prefix/_next/static/chunks/0a65da2cd24e2ab6.js","async":true}]
|
||||
9:["$","script","script-37",{"src":"/litellm-asset-prefix/_next/static/chunks/82dbdb17d57c6737.js","async":true}]
|
||||
a:["$","script","script-38",{"src":"/litellm-asset-prefix/_next/static/chunks/b31a08272a82b84e.js","async":true}]
|
||||
b:["$","script","script-39",{"src":"/litellm-asset-prefix/_next/static/chunks/496b84010c33cf69.js","async":true}]
|
||||
c:["$","script","script-40",{"src":"/litellm-asset-prefix/_next/static/chunks/eba9ac65320061b1.js","async":true}]
|
||||
d:["$","script","script-41",{"src":"/litellm-asset-prefix/_next/static/chunks/0eda6dc5d5f35d92.js","async":true}]
|
||||
e:["$","script","script-42",{"src":"/litellm-asset-prefix/_next/static/chunks/96aed36e606e8582.js","async":true}]
|
||||
f:["$","script","script-43",{"src":"/litellm-asset-prefix/_next/static/chunks/ad426ab08aee6c64.js","async":true}]
|
||||
10:["$","script","script-44",{"src":"/litellm-asset-prefix/_next/static/chunks/2971c4658f1bcd7d.js","async":true}]
|
||||
11:["$","script","script-45",{"src":"/litellm-asset-prefix/_next/static/chunks/06ebe9b0e9cdf241.js","async":true}]
|
||||
12:["$","script","script-46",{"src":"/litellm-asset-prefix/_next/static/chunks/6c4c97f1ea6e7d77.js","async":true}]
|
||||
13:["$","script","script-47",{"src":"/litellm-asset-prefix/_next/static/chunks/165d00848f04c4c9.js","async":true}]
|
||||
9:["$","script","script-37",{"src":"/litellm-asset-prefix/_next/static/chunks/86c1e01d849eed8e.js","async":true}]
|
||||
a:["$","script","script-38",{"src":"/litellm-asset-prefix/_next/static/chunks/eba9ac65320061b1.js","async":true}]
|
||||
b:["$","script","script-39",{"src":"/litellm-asset-prefix/_next/static/chunks/85238af541b170ca.js","async":true}]
|
||||
c:["$","script","script-40",{"src":"/litellm-asset-prefix/_next/static/chunks/8c09c924a98654d1.js","async":true}]
|
||||
d:["$","script","script-41",{"src":"/litellm-asset-prefix/_next/static/chunks/b13b0ddeb85b5333.js","async":true}]
|
||||
e:["$","script","script-42",{"src":"/litellm-asset-prefix/_next/static/chunks/2971c4658f1bcd7d.js","async":true}]
|
||||
f:["$","script","script-43",{"src":"/litellm-asset-prefix/_next/static/chunks/496b84010c33cf69.js","async":true}]
|
||||
10:["$","script","script-44",{"src":"/litellm-asset-prefix/_next/static/chunks/06ebe9b0e9cdf241.js","async":true}]
|
||||
11:["$","script","script-45",{"src":"/litellm-asset-prefix/_next/static/chunks/6c4c97f1ea6e7d77.js","async":true}]
|
||||
12:["$","script","script-46",{"src":"/litellm-asset-prefix/_next/static/chunks/e8718f949e42598e.js","async":true}]
|
||||
13:["$","script","script-47",{"src":"/litellm-asset-prefix/_next/static/chunks/99cf9cf99df5ccfc.js","async":true}]
|
||||
14:["$","script","script-48",{"src":"/litellm-asset-prefix/_next/static/chunks/bb0a6e4a3a18721a.js","async":true}]
|
||||
15:["$","script","script-49",{"src":"/litellm-asset-prefix/_next/static/chunks/4980372eaa37b78b.js","async":true}]
|
||||
16:["$","script","script-50",{"src":"/litellm-asset-prefix/_next/static/chunks/8ab6e2ca95000c8e.js","async":true}]
|
||||
17:["$","script","script-51",{"src":"/litellm-asset-prefix/_next/static/chunks/06cdd9bb80c63794.js","async":true}]
|
||||
18:["$","script","script-52",{"src":"/litellm-asset-prefix/_next/static/chunks/66a190706fc6c35a.js","async":true}]
|
||||
19:["$","script","script-53",{"src":"/litellm-asset-prefix/_next/static/chunks/54e29148cb2f2582.js","async":true}]
|
||||
1a:["$","script","script-54",{"src":"/litellm-asset-prefix/_next/static/chunks/fd9de419c8c0222e.js","async":true}]
|
||||
1b:["$","$L1c",null,{"children":["$","$1d",null,{"name":"Next.MetadataOutlet","children":"$@1e"}]}]
|
||||
1e:null
|
||||
15:["$","script","script-49",{"src":"/litellm-asset-prefix/_next/static/chunks/cfca40b1a4a490bf.js","async":true}]
|
||||
16:["$","script","script-50",{"src":"/litellm-asset-prefix/_next/static/chunks/54e29148cb2f2582.js","async":true}]
|
||||
17:["$","script","script-51",{"src":"/litellm-asset-prefix/_next/static/chunks/37c03dc421ba81f8.js","async":true}]
|
||||
18:["$","$L19",null,{"children":["$","$1a",null,{"name":"Next.MetadataOutlet","children":"$@1b"}]}]
|
||||
1b:null
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -3,4 +3,4 @@
|
|||
3:I[897367,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"MetadataBoundary"]
|
||||
4:"$Sreact.suspense"
|
||||
5:I[27201,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"IconMark"]
|
||||
0:{"buildId":"U_YrOOnSehrpkdU42KJ-W","rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.1d32c690.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"./favicon.ico"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"loading":null,"isPartial":false}
|
||||
0:{"buildId":"cbFGTIkRGp63usVNisey9","rsc":["$","$1","h",{"children":[null,["$","$L2",null,{"children":[["$","meta","0",{"charSet":"utf-8"}],["$","meta","1",{"name":"viewport","content":"width=device-width, initial-scale=1"}]]}],["$","div",null,{"hidden":true,"children":["$","$L3",null,{"children":["$","$4",null,{"name":"Next.Metadata","children":[["$","title","0",{"children":"LiteLLM Dashboard"}],["$","meta","1",{"name":"description","content":"LiteLLM Proxy Admin UI"}],["$","link","2",{"rel":"icon","href":"/favicon.ico?favicon.1d32c690.ico","sizes":"48x48","type":"image/x-icon"}],["$","link","3",{"rel":"icon","href":"./favicon.ico"}],["$","$L5","4",{}]]}]}]}],["$","meta",null,{"name":"next-size-adjust","content":""}]]}],"loading":null,"isPartial":false}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
1:"$Sreact.fragment"
|
||||
2:I[71195,["/litellm-asset-prefix/_next/static/chunks/26adfa4e8ffc85c7.js","/litellm-asset-prefix/_next/static/chunks/e8ed72789c2b42ff.js"],"default"]
|
||||
3:I[339756,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"default"]
|
||||
4:I[837457,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"default"]
|
||||
2:I[867271,["/litellm-asset-prefix/_next/static/chunks/a7f104aa2cc7f3f0.js","/litellm-asset-prefix/_next/static/chunks/e8ed72789c2b42ff.js"],"default"]
|
||||
3:I[71195,["/litellm-asset-prefix/_next/static/chunks/a7f104aa2cc7f3f0.js","/litellm-asset-prefix/_next/static/chunks/e8ed72789c2b42ff.js"],"default"]
|
||||
4:I[339756,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"default"]
|
||||
5:I[837457,["/litellm-asset-prefix/_next/static/chunks/d96012bcfc98706a.js","/litellm-asset-prefix/_next/static/chunks/dbca964212122d58.js"],"default"]
|
||||
:HL["/litellm-asset-prefix/_next/static/chunks/4e20891f2fd03463.css","style"]
|
||||
:HL["/litellm-asset-prefix/_next/static/chunks/6fabf2cec1bd2d6e.css","style"]
|
||||
0:{"buildId":"U_YrOOnSehrpkdU42KJ-W","rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/4e20891f2fd03463.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/6fabf2cec1bd2d6e.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/26adfa4e8ffc85c7.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/e8ed72789c2b42ff.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"parallelRouterKey":"children","template":["$","$L4",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]]}],"loading":null,"isPartial":false}
|
||||
:HL["/litellm-asset-prefix/_next/static/chunks/7936c9bd377ea4bf.css","style"]
|
||||
0:{"buildId":"cbFGTIkRGp63usVNisey9","rsc":["$","$1","c",{"children":[[["$","link","0",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/4e20891f2fd03463.css","precedence":"next"}],["$","link","1",{"rel":"stylesheet","href":"/litellm-asset-prefix/_next/static/chunks/7936c9bd377ea4bf.css","precedence":"next"}],["$","script","script-0",{"src":"/litellm-asset-prefix/_next/static/chunks/a7f104aa2cc7f3f0.js","async":true}],["$","script","script-1",{"src":"/litellm-asset-prefix/_next/static/chunks/e8ed72789c2b42ff.js","async":true}]],["$","html",null,{"lang":"en","children":["$","body",null,{"className":"inter_5972bc34-module__OU16Qa__className","children":["$","$L2",null,{"children":["$","$L3",null,{"children":["$","$L4",null,{"parallelRouterKey":"children","template":["$","$L5",null,{}],"notFound":[[["$","title",null,{"children":"404: This page could not be found."}],["$","div",null,{"style":{"fontFamily":"system-ui,\"Segoe UI\",Roboto,Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\"","height":"100vh","textAlign":"center","display":"flex","flexDirection":"column","alignItems":"center","justifyContent":"center"},"children":["$","div",null,{"children":[["$","style",null,{"dangerouslySetInnerHTML":{"__html":"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}"}}],["$","h1",null,{"className":"next-error-h1","style":{"display":"inline-block","margin":"0 20px 0 0","padding":"0 23px 0 0","fontSize":24,"fontWeight":500,"verticalAlign":"top","lineHeight":"49px"},"children":404}],["$","div",null,{"style":{"display":"inline-block"},"children":["$","h2",null,{"style":{"fontSize":14,"fontWeight":400,"lineHeight":"49px","margin":0},"children":"This page could not be found."}]}]]}]}]],[]]}]}]}]}]}]]}],"loading":null,"isPartial":false}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
:HL["/litellm-asset-prefix/_next/static/chunks/4e20891f2fd03463.css","style"]
|
||||
:HL["/litellm-asset-prefix/_next/static/chunks/6fabf2cec1bd2d6e.css","style"]
|
||||
:HL["/litellm-asset-prefix/_next/static/chunks/7936c9bd377ea4bf.css","style"]
|
||||
:HL["/litellm-asset-prefix/_next/static/media/83afe278b6a6bb3c-s.p.3a6ba036.woff2","font",{"crossOrigin":"","type":"font/woff2"}]
|
||||
:HL["/litellm-asset-prefix/_next/static/chunks/3f3fa56b5786d58c.css","style"]
|
||||
0:{"buildId":"U_YrOOnSehrpkdU42KJ-W","tree":{"name":"","paramType":null,"paramKey":"","hasRuntimePrefetch":false,"slots":{"children":{"name":"__PAGE__","paramType":null,"paramKey":"__PAGE__","hasRuntimePrefetch":false,"slots":null,"isRootLayout":false}},"isRootLayout":true},"staleTime":300}
|
||||
0:{"buildId":"cbFGTIkRGp63usVNisey9","tree":{"name":"","paramType":null,"paramKey":"","hasRuntimePrefetch":false,"slots":{"children":{"name":"__PAGE__","paramType":null,"paramKey":"__PAGE__","hasRuntimePrefetch":false,"slots":null,"isRootLayout":false}},"isRootLayout":true},"staleTime":300}
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1 @@
|
|||
(globalThis.TURBOPACK||(globalThis.TURBOPACK=[])).push(["object"==typeof document?document.currentScript:void 0,233525,(e,c,u)=>{"use strict";Object.defineProperty(u,"__esModule",{value:!0}),Object.defineProperty(u,"warnOnce",{enumerable:!0,get:function(){return r}});let r=e=>{}},349356,e=>{e.v({AElig:"Æ",AMP:"&",Aacute:"Á",Acirc:"Â",Agrave:"À",Aring:"Å",Atilde:"Ã",Auml:"Ä",COPY:"©",Ccedil:"Ç",ETH:"Ð",Eacute:"É",Ecirc:"Ê",Egrave:"È",Euml:"Ë",GT:">",Iacute:"Í",Icirc:"Î",Igrave:"Ì",Iuml:"Ï",LT:"<",Ntilde:"Ñ",Oacute:"Ó",Ocirc:"Ô",Ograve:"Ò",Oslash:"Ø",Otilde:"Õ",Ouml:"Ö",QUOT:'"',REG:"®",THORN:"Þ",Uacute:"Ú",Ucirc:"Û",Ugrave:"Ù",Uuml:"Ü",Yacute:"Ý",aacute:"á",acirc:"â",acute:"´",aelig:"æ",agrave:"à",amp:"&",aring:"å",atilde:"ã",auml:"ä",brvbar:"¦",ccedil:"ç",cedil:"¸",cent:"¢",copy:"©",curren:"¤",deg:"°",divide:"÷",eacute:"é",ecirc:"ê",egrave:"è",eth:"ð",euml:"ë",frac12:"½",frac14:"¼",frac34:"¾",gt:">",iacute:"í",icirc:"î",iexcl:"¡",igrave:"ì",iquest:"¿",iuml:"ï",laquo:"«",lt:"<",macr:"¯",micro:"µ",middot:"·",nbsp:" ",not:"¬",ntilde:"ñ",oacute:"ó",ocirc:"ô",ograve:"ò",ordf:"ª",ordm:"º",oslash:"ø",otilde:"õ",ouml:"ö",para:"¶",plusmn:"±",pound:"£",quot:'"',raquo:"»",reg:"®",sect:"§",shy:"",sup1:"¹",sup2:"²",sup3:"³",szlig:"ß",thorn:"þ",times:"×",uacute:"ú",ucirc:"û",ugrave:"ù",uml:"¨",uuml:"ü",yacute:"ý",yen:"¥",yuml:"ÿ"})},137429,e=>{e.v({0:"<22>",128:"€",130:"‚",131:"ƒ",132:"„",133:"…",134:"†",135:"‡",136:"ˆ",137:"‰",138:"Š",139:"‹",140:"Œ",142:"Ž",145:"‘",146:"’",147:"“",148:"”",149:"•",150:"–",151:"—",152:"˜",153:"™",154:"š",155:"›",156:"œ",158:"ž",159:"Ÿ"})},928685,e=>{"use strict";var c=e.i(38953);e.s(["SearchOutlined",()=>c.default])},86408,e=>{"use strict";var c=e.i(843476),u=e.i(271645),r=e.i(618566),t=e.i(934879);function a(){let e=(0,r.useSearchParams)().get("key"),[a,i]=(0,u.useState)(null);return console.log("PublicModelHubTable accessToken:",a),(0,u.useEffect)(()=>{e&&i(e)},[e]),(0,c.jsx)(t.default,{accessToken:a,publicPage:!0,premiumUser:!1,userRole:null})}function i(){return(0,c.jsx)(u.Suspense,{fallback:(0,c.jsx)("div",{className:"flex items-center justify-center min-h-screen",children:"Loading..."}),children:(0,c.jsx)(a,{})})}e.s(["default",()=>i])}]);
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -1 +0,0 @@
|
|||
(globalThis.TURBOPACK||(globalThis.TURBOPACK=[])).push(["object"==typeof document?document.currentScript:void 0,56456,e=>{"use strict";var s=e.i(739295);e.s(["LoadingOutlined",()=>s.default])},566606,e=>{"use strict";var s=e.i(843476),i=e.i(271645),r=e.i(618566),t=e.i(317751),a=e.i(912598),n=e.i(947293),l=e.i(764205),o=e.i(954616),d=e.i(266027),u=e.i(612256);let c=(0,e.i(243652).createQueryKeys)("onboarding");var m=e.i(482725),p=e.i(56456);function x(){return(0,s.jsx)("div",{className:"mx-auto w-full max-w-md mt-10 flex justify-center",children:(0,s.jsx)(m.Spin,{indicator:(0,s.jsx)(p.LoadingOutlined,{spin:!0}),size:"large"})})}var h=e.i(560445),g=e.i(464571);function j(){return(0,s.jsxs)("div",{className:"mx-auto w-full max-w-md mt-10",children:[(0,s.jsx)(h.Alert,{type:"error",message:"Failed to load invitation",description:"The invitation link may be invalid or expired.",showIcon:!0}),(0,s.jsx)("div",{className:"mt-4",children:(0,s.jsx)(g.Button,{href:"/ui/login",children:"Back to Login"})})]})}var y=e.i(175712),w=e.i(808613),f=e.i(311451),v=e.i(898586);function b({variant:e,userEmail:r,isPending:t,claimError:a,onSubmit:n}){let[l]=w.Form.useForm();return i.default.useEffect(()=>{r&&l.setFieldValue("user_email",r)},[r,l]),(0,s.jsx)("div",{className:"mx-auto w-full max-w-md mt-10",children:(0,s.jsxs)(y.Card,{children:[(0,s.jsx)(v.Typography.Title,{level:5,className:"text-center mb-5",children:"🚅 LiteLLM"}),(0,s.jsx)(v.Typography.Title,{level:3,children:"reset_password"===e?"Reset Password":"Sign Up"}),(0,s.jsx)(v.Typography.Text,{children:"reset_password"===e?"Reset your password to access Admin UI.":"Claim your user account to login to Admin UI."}),"signup"===e&&(0,s.jsx)(h.Alert,{className:"mt-4",type:"info",message:"SSO",description:(0,s.jsxs)("div",{className:"flex justify-between items-center",children:[(0,s.jsx)("span",{children:"SSO is under the Enterprise Tier."}),(0,s.jsx)(g.Button,{type:"primary",size:"small",href:"https://forms.gle/W3U4PZpJGFHWtHyA9",target:"_blank",children:"Get Free Trial"})]}),showIcon:!0}),(0,s.jsxs)(w.Form,{className:"mt-10 mb-5",layout:"vertical",form:l,onFinish:e=>n({password:e.password}),children:[(0,s.jsx)(w.Form.Item,{label:"Email Address",name:"user_email",children:(0,s.jsx)(f.Input,{type:"email",disabled:!0})}),(0,s.jsx)(w.Form.Item,{label:"Password",name:"password",rules:[{required:!0,message:"password required to sign up"}],help:"reset_password"===e?"Enter your new password":"Create a password for your account",children:(0,s.jsx)(f.Input.Password,{})}),a&&(0,s.jsx)(h.Alert,{type:"error",message:a,showIcon:!0,className:"mb-4"}),(0,s.jsx)("div",{className:"mt-10",children:(0,s.jsx)(g.Button,{htmlType:"submit",loading:t,children:"reset_password"===e?"Reset Password":"Sign Up"})})]})]})})}function S({variant:e}){let t=(0,r.useSearchParams)().get("invitation_id"),[a,m]=i.default.useState(null),{data:p,isLoading:h,isError:g}=(e=>{let{isLoading:s}=(0,u.useUIConfig)();return(0,d.useQuery)({queryKey:c.detail(e??""),queryFn:async()=>{if(!e)throw Error("inviteId is required");return(0,l.getOnboardingCredentials)(e)},enabled:!!e&&!s})})(t),{mutate:y,isPending:w}=(0,o.useMutation)({mutationFn:async({accessToken:e,inviteId:s,userId:i,password:r})=>await (0,l.claimOnboardingToken)(e,s,i,r)}),f=p?.token?(0,n.jwtDecode)(p.token):null,v=f?.user_email??"",S=f?.user_id??null,T=f?.key??null,F=p?.token??null;return h?(0,s.jsx)(x,{}):g?(0,s.jsx)(j,{}):(0,s.jsx)(b,{variant:e,userEmail:v,isPending:w,claimError:a,onSubmit:e=>{T&&F&&S&&t&&(m(null),y({accessToken:T,inviteId:t,userId:S,password:e.password},{onSuccess:()=>{document.cookie=`token=${F}; path=/; SameSite=Lax`;let e=(0,l.getProxyBaseUrl)();window.location.href=e?`${e}/ui/?login=success`:"/ui/?login=success"},onError:e=>{m(e.message||"Failed to submit. Please try again.")}}))}})}let T=new t.QueryClient;function F(){let e=(0,r.useSearchParams)().get("action");return(0,s.jsx)(S,{variant:"reset_password"===e?"reset_password":"signup"})}function P(){return(0,s.jsx)(a.QueryClientProvider,{client:T,children:(0,s.jsx)(i.Suspense,{fallback:(0,s.jsx)("div",{className:"flex items-center justify-center min-h-screen",children:"Loading..."}),children:(0,s.jsx)(F,{})})})}e.s(["default",()=>P],566606)}]);
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Reference in a new issue