diff --git a/docs/my-website/docs/providers/bedrock.md b/docs/my-website/docs/providers/bedrock.md index 08b72a484dd..31e34f40892 100644 --- a/docs/my-website/docs/providers/bedrock.md +++ b/docs/my-website/docs/providers/bedrock.md @@ -1,13 +1,13 @@ # AWS Bedrock -### API KEYS +## API KEYS ```python os.environ["AWS_ACCESS_KEY_ID"] = "" os.environ["AWS_SECRET_ACCESS_KEY"] = "" os.environ["AWS_REGION_NAME"] = "" ``` -### Usage +## Usage ```python import os from litellm import completion @@ -24,7 +24,7 @@ response = completion( ) ``` -### Supported AWS Bedrock Models +## Supported AWS Bedrock Models Here's an example of using a bedrock model with LiteLLM | Model Name | Function Call | Required OS Variables | @@ -33,7 +33,54 @@ Here's an example of using a bedrock model with LiteLLM | AI21 J2-Ultra | `completion(model='bedrock/ai21.j2-ultra', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']`, `os.environ['AWS_REGION_NAME']` | | AI21 J2-Mid | `completion(model='bedrock/ai21.j2-mid', messages=messages)` | `os.environ['AWS_ACCESS_KEY_ID']`, `os.environ['AWS_SECRET_ACCESS_KEY']`, `os.environ['AWS_REGION_NAME']` | -### Troubleshooting +## Streaming +Bedrock currently supports treaming for: +* `bedrock/amazon.titan-tg1-large` + +Example Usage +```python +import os +from litellm import completion + +os.environ["AWS_ACCESS_KEY_ID"] = "" +os.environ["AWS_SECRET_ACCESS_KEY"] = "" +os.environ["AWS_REGION_NAME"] = "" + +response = completion( + model="bedrock/amazon.titan-tg1-large", + messages=[{ "content": "Hello, how are you?","role": "user"}], + temperature=0.2, + max_tokens=80, + stream=True +) + +for chunk in response: + print(chunk) +``` + +### Example Streaming Output Chunk +```json +{ + "choices": [ + { + "finish_reason": null, + "index": 0, + "delta": { + "content": "ase can appeal the case to a higher federal court. If a higher federal court rules in a way that conflicts with a ruling from a lower federal court or conflicts with a ruling from a higher state court, the parties involved in the case can appeal the case to the Supreme Court. In order to appeal a case to the Sup" + } + } + ], + "created": null, + "model": "amazon.titan-tg1-large", + "usage": { + "prompt_tokens": null, + "completion_tokens": null, + "total_tokens": null + } +} +``` + +## Troubleshooting If creating a boto3 bedrock client fails with `Unknown service: 'bedrock'` Try re installing boto3 using the following commands ```shell