mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
docs add /invoke and /converse routes
This commit is contained in:
Ishaan Jaffer
parent
8c5d6891d4
commit
79516563a9
3 changed files with 294 additions and 0 deletions
149
docs/my-website/docs/bedrock_converse.md
Normal file
149
docs/my-website/docs/bedrock_converse.md
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
# /converse
|
||||
|
||||
Call Bedrock's `/converse` endpoint through LiteLLM Proxy.
|
||||
|
||||
| Feature | Supported |
|
||||
|---------|-----------|
|
||||
| Cost Tracking | ✅ |
|
||||
| Logging | ✅ |
|
||||
| Streaming | ✅ via `/converse-stream` |
|
||||
| Load Balancing | ✅ |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Setup config.yaml
|
||||
|
||||
```yaml showLineNumbers
|
||||
model_list:
|
||||
- model_name: my-bedrock-model
|
||||
litellm_params:
|
||||
model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0
|
||||
aws_region_name: us-west-2
|
||||
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID # reads from environment
|
||||
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
|
||||
custom_llm_provider: bedrock
|
||||
```
|
||||
|
||||
Set AWS credentials in your environment:
|
||||
|
||||
```bash showLineNumbers
|
||||
export AWS_ACCESS_KEY_ID="your-access-key"
|
||||
export AWS_SECRET_ACCESS_KEY="your-secret-key"
|
||||
```
|
||||
|
||||
### 2. Start Proxy
|
||||
|
||||
```bash showLineNumbers
|
||||
litellm --config config.yaml
|
||||
|
||||
# RUNNING on http://0.0.0.0:4000
|
||||
```
|
||||
|
||||
### 3. Call /converse endpoint
|
||||
|
||||
```bash showLineNumbers
|
||||
curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/converse' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"text": "Hello, how are you?"}]
|
||||
}
|
||||
],
|
||||
"inferenceConfig": {
|
||||
"temperature": 0.5,
|
||||
"maxTokens": 100
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
## Streaming
|
||||
|
||||
For streaming responses, use `/converse-stream`:
|
||||
|
||||
```bash showLineNumbers
|
||||
curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/converse-stream' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"text": "Tell me a short story"}]
|
||||
}
|
||||
],
|
||||
"inferenceConfig": {
|
||||
"temperature": 0.7,
|
||||
"maxTokens": 200
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
## Load Balancing
|
||||
|
||||
Define multiple deployments with the same `model_name` for automatic load balancing:
|
||||
|
||||
```yaml showLineNumbers
|
||||
model_list:
|
||||
# Deployment 1 - us-west-2
|
||||
- model_name: my-bedrock-model
|
||||
litellm_params:
|
||||
model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0
|
||||
aws_region_name: us-west-2
|
||||
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
|
||||
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
|
||||
custom_llm_provider: bedrock
|
||||
|
||||
# Deployment 2 - us-east-1
|
||||
- model_name: my-bedrock-model
|
||||
litellm_params:
|
||||
model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0
|
||||
aws_region_name: us-east-1
|
||||
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
|
||||
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
|
||||
custom_llm_provider: bedrock
|
||||
```
|
||||
|
||||
The proxy automatically distributes requests across both regions.
|
||||
|
||||
## Using boto3 SDK
|
||||
|
||||
```python showLineNumbers
|
||||
import boto3
|
||||
import json
|
||||
|
||||
bedrock_runtime = boto3.client(
|
||||
service_name='bedrock-runtime',
|
||||
region_name='us-west-2',
|
||||
endpoint_url='http://0.0.0.0:4000/bedrock'
|
||||
)
|
||||
|
||||
def add_custom_headers(request, **kwargs):
|
||||
request.headers.update({'litellm_user_api_key': 'Bearer sk-1234'})
|
||||
|
||||
bedrock_runtime.meta.events.register('before-send.*.*', add_custom_headers)
|
||||
|
||||
response = bedrock_runtime.converse(
|
||||
modelId='my-bedrock-model',
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"text": "Hello, how are you?"}]
|
||||
}
|
||||
],
|
||||
inferenceConfig={
|
||||
"temperature": 0.5,
|
||||
"maxTokens": 100
|
||||
}
|
||||
)
|
||||
|
||||
print(response['output']['message']['content'][0]['text'])
|
||||
```
|
||||
|
||||
## More Info
|
||||
|
||||
For complete documentation including Guardrails, Knowledge Bases, and Agents, see:
|
||||
- [Full Bedrock Passthrough Docs](./pass_through/bedrock)
|
||||
|
||||
143
docs/my-website/docs/bedrock_invoke.md
Normal file
143
docs/my-website/docs/bedrock_invoke.md
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
# /invoke
|
||||
|
||||
Call Bedrock's `/invoke` endpoint through LiteLLM Proxy.
|
||||
|
||||
| Feature | Supported |
|
||||
|---------|-----------|
|
||||
| Cost Tracking | ✅ |
|
||||
| Logging | ✅ |
|
||||
| Streaming | ✅ via `/invoke-with-response-stream` |
|
||||
| Load Balancing | ✅ |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Setup config.yaml
|
||||
|
||||
```yaml showLineNumbers
|
||||
model_list:
|
||||
- model_name: my-bedrock-model
|
||||
litellm_params:
|
||||
model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0
|
||||
aws_region_name: us-west-2
|
||||
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID # reads from environment
|
||||
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
|
||||
custom_llm_provider: bedrock
|
||||
```
|
||||
|
||||
Set AWS credentials in your environment:
|
||||
|
||||
```bash showLineNumbers
|
||||
export AWS_ACCESS_KEY_ID="your-access-key"
|
||||
export AWS_SECRET_ACCESS_KEY="your-secret-key"
|
||||
```
|
||||
|
||||
### 2. Start Proxy
|
||||
|
||||
```bash showLineNumbers
|
||||
litellm --config config.yaml
|
||||
|
||||
# RUNNING on http://0.0.0.0:4000
|
||||
```
|
||||
|
||||
### 3. Call /invoke endpoint
|
||||
|
||||
```bash showLineNumbers
|
||||
curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/invoke' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"max_tokens": 100,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Hello, how are you?"
|
||||
}
|
||||
],
|
||||
"anthropic_version": "bedrock-2023-05-31"
|
||||
}'
|
||||
```
|
||||
|
||||
## Streaming
|
||||
|
||||
For streaming responses, use `/invoke-with-response-stream`:
|
||||
|
||||
```bash showLineNumbers
|
||||
curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/invoke-with-response-stream' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"max_tokens": 100,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Tell me a short story"
|
||||
}
|
||||
],
|
||||
"anthropic_version": "bedrock-2023-05-31"
|
||||
}'
|
||||
```
|
||||
|
||||
## Load Balancing
|
||||
|
||||
Define multiple deployments with the same `model_name` for automatic load balancing:
|
||||
|
||||
```yaml showLineNumbers
|
||||
model_list:
|
||||
# Deployment 1 - us-west-2
|
||||
- model_name: my-bedrock-model
|
||||
litellm_params:
|
||||
model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0
|
||||
aws_region_name: us-west-2
|
||||
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
|
||||
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
|
||||
custom_llm_provider: bedrock
|
||||
|
||||
# Deployment 2 - us-east-1
|
||||
- model_name: my-bedrock-model
|
||||
litellm_params:
|
||||
model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0
|
||||
aws_region_name: us-east-1
|
||||
aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
|
||||
aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
|
||||
custom_llm_provider: bedrock
|
||||
```
|
||||
|
||||
The proxy automatically distributes requests across both regions.
|
||||
|
||||
## Using boto3 SDK
|
||||
|
||||
```python showLineNumbers
|
||||
import boto3
|
||||
import json
|
||||
|
||||
bedrock_runtime = boto3.client(
|
||||
service_name='bedrock-runtime',
|
||||
region_name='us-west-2',
|
||||
endpoint_url='http://0.0.0.0:4000/bedrock'
|
||||
)
|
||||
|
||||
def add_custom_headers(request, **kwargs):
|
||||
request.headers.update({'litellm_user_api_key': 'Bearer sk-1234'})
|
||||
|
||||
bedrock_runtime.meta.events.register('before-send.*.*', add_custom_headers)
|
||||
|
||||
response = bedrock_runtime.invoke_model(
|
||||
modelId='my-bedrock-model',
|
||||
contentType='application/json',
|
||||
accept='application/json',
|
||||
body=json.dumps({
|
||||
"max_tokens": 100,
|
||||
"messages": [{"role": "user", "content": "Hello"}],
|
||||
"anthropic_version": "bedrock-2023-05-31"
|
||||
})
|
||||
)
|
||||
|
||||
response_body = json.loads(response['body'].read())
|
||||
print(response_body['content'][0]['text'])
|
||||
```
|
||||
|
||||
## More Info
|
||||
|
||||
For complete documentation including Guardrails, Knowledge Bases, and Agents, see:
|
||||
- [Full Bedrock Passthrough Docs](./pass_through/bedrock)
|
||||
|
||||
|
|
@ -347,6 +347,8 @@ const sidebars = {
|
|||
]
|
||||
},
|
||||
"moderation",
|
||||
"bedrock_invoke",
|
||||
"bedrock_converse",
|
||||
"ocr",
|
||||
{
|
||||
type: "category",
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue