docs(bedrock.md): adding docs for calling bedrock models on proxy via config.yaml

This commit is contained in:
Krrish Dholakia 2024-03-14 14:50:13 -07:00
parent 1ba21a8c58
commit 6f1eb038bc

View file

@ -4,7 +4,6 @@ import TabItem from '@theme/TabItem';
# AWS Bedrock
Anthropic, Amazon Titan, A121 LLMs are Supported on Bedrock
## Pre-Requisites
LiteLLM requires `boto3` to be installed on your system for Bedrock requests
```shell
pip install boto3>=1.28.57
@ -51,11 +50,25 @@ export AWS_REGION_NAME=""
### 2. Start the proxy
<Tabs>
<TabItem value="cli" label="CLI">
```bash
$ litellm --model anthropic.claude-3-sonnet-20240229-v1:0
# Server running on http://0.0.0.0:4000
```
</TabItem>
<TabItem value="config" label="config.yaml">
```yaml
model_list:
- model_name: bedrock-claude-v1
litellm_params:
model: bedrock/anthropic.claude-instant-v1
```
</TabItem>
</Tabs>
### 3. Test it
@ -67,7 +80,7 @@ $ litellm --model anthropic.claude-3-sonnet-20240229-v1:0
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--data ' {
"model": "gpt-3.5-turbo",
"model": "bedrock-claude-v1",
"messages": [
{
"role": "user",
@ -88,7 +101,7 @@ client = openai.OpenAI(
)
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
response = client.chat.completions.create(model="bedrock-claude-v1", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
@ -112,7 +125,7 @@ from langchain.schema import HumanMessage, SystemMessage
chat = ChatOpenAI(
openai_api_base="http://0.0.0.0:4000", # set openai_api_base to the LiteLLM Proxy
model = "gpt-3.5-turbo",
model = "bedrock-claude-v1",
temperature=0.1
)