scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-08 03:08:45 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 115

docs bedrock streaming

Browse source
This commit is contained in:
ishaan-jaff 2023-09-16 10:01:19 -07:00
parent c714372b9d
commit 58c9748a14
1 changed files with 51 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

55
docs/my-website/docs/providers/bedrock.md
View file

@ -1,13 +1,13 @@
# AWS Bedrock
### API KEYS
## API KEYS
```python
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""
```
### Usage
## Usage
```python
import os
from litellm import completion
@ -24,7 +24,7 @@ response = completion(
)
```
### Supported AWS Bedrock Models
## Supported AWS Bedrock Models
Here's an example of using a bedrock model with LiteLLM
| Model Name | Function Call | Required OS Variables |
@ -33,7 +33,54 @@ Here's an example of using a bedrock model with LiteLLM
| AI21 J2-Ultra | `completion(model='bedrock/ai21.j2-ultra', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']`, `os.environ['AWS_REGION_NAME']` |
| AI21 J2-Mid | `completion(model='bedrock/ai21.j2-mid', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']`, `os.environ['AWS_REGION_NAME']` |
### Troubleshooting
## Streaming
Bedrock currently supports treaming for:
* `bedrock/amazon.titan-tg1-large`
Example Usage
```python
import os
from litellm import completion
os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""
response = completion(
model="bedrock/amazon.titan-tg1-large",
messages=[{ "content": "Hello, how are you?","role": "user"}],
temperature=0.2,
max_tokens=80,
stream=True
)
for chunk in response:
print(chunk)
```
### Example Streaming Output Chunk
```json
{
"choices": [
{
"finish_reason": null,
"index": 0,
"delta": {
"content": "ase can appeal the case to a higher federal court. If a higher federal court rules in a way that conflicts with a ruling from a lower federal court or conflicts with a ruling from a higher state court, the parties involved in the case can appeal the case to the Supreme Court. In order to appeal a case to the Sup"
}
}
],
"created": null,
"model": "amazon.titan-tg1-large",
"usage": {
"prompt_tokens": null,
"completion_tokens": null,
"total_tokens": null
}
}
```
## Troubleshooting
If creating a boto3 bedrock client fails with `Unknown service: 'bedrock'`
Try re installing boto3 using the following commands
```shell