diff --git a/docs/my-website/docs/bedrock_converse.md b/docs/my-website/docs/bedrock_converse.md new file mode 100644 index 00000000000..4458b57a51d --- /dev/null +++ b/docs/my-website/docs/bedrock_converse.md @@ -0,0 +1,149 @@ +# /converse + +Call Bedrock's `/converse` endpoint through LiteLLM Proxy. + +| Feature | Supported | +|---------|-----------| +| Cost Tracking | ✅ | +| Logging | ✅ | +| Streaming | ✅ via `/converse-stream` | +| Load Balancing | ✅ | + +## Quick Start + +### 1. Setup config.yaml + +```yaml showLineNumbers +model_list: + - model_name: my-bedrock-model + litellm_params: + model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0 + aws_region_name: us-west-2 + aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID # reads from environment + aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY + custom_llm_provider: bedrock +``` + +Set AWS credentials in your environment: + +```bash showLineNumbers +export AWS_ACCESS_KEY_ID="your-access-key" +export AWS_SECRET_ACCESS_KEY="your-secret-key" +``` + +### 2. Start Proxy + +```bash showLineNumbers +litellm --config config.yaml + +# RUNNING on http://0.0.0.0:4000 +``` + +### 3. Call /converse endpoint + +```bash showLineNumbers +curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/converse' \ +-H 'Authorization: Bearer sk-1234' \ +-H 'Content-Type: application/json' \ +-d '{ + "messages": [ + { + "role": "user", + "content": [{"text": "Hello, how are you?"}] + } + ], + "inferenceConfig": { + "temperature": 0.5, + "maxTokens": 100 + } +}' +``` + +## Streaming + +For streaming responses, use `/converse-stream`: + +```bash showLineNumbers +curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/converse-stream' \ +-H 'Authorization: Bearer sk-1234' \ +-H 'Content-Type: application/json' \ +-d '{ + "messages": [ + { + "role": "user", + "content": [{"text": "Tell me a short story"}] + } + ], + "inferenceConfig": { + "temperature": 0.7, + "maxTokens": 200 + } +}' +``` + +## Load Balancing + +Define multiple deployments with the same `model_name` for automatic load balancing: + +```yaml showLineNumbers +model_list: + # Deployment 1 - us-west-2 + - model_name: my-bedrock-model + litellm_params: + model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0 + aws_region_name: us-west-2 + aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID + aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY + custom_llm_provider: bedrock + + # Deployment 2 - us-east-1 + - model_name: my-bedrock-model + litellm_params: + model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0 + aws_region_name: us-east-1 + aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID + aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY + custom_llm_provider: bedrock +``` + +The proxy automatically distributes requests across both regions. + +## Using boto3 SDK + +```python showLineNumbers +import boto3 +import json + +bedrock_runtime = boto3.client( + service_name='bedrock-runtime', + region_name='us-west-2', + endpoint_url='http://0.0.0.0:4000/bedrock' +) + +def add_custom_headers(request, **kwargs): + request.headers.update({'litellm_user_api_key': 'Bearer sk-1234'}) + +bedrock_runtime.meta.events.register('before-send.*.*', add_custom_headers) + +response = bedrock_runtime.converse( + modelId='my-bedrock-model', + messages=[ + { + "role": "user", + "content": [{"text": "Hello, how are you?"}] + } + ], + inferenceConfig={ + "temperature": 0.5, + "maxTokens": 100 + } +) + +print(response['output']['message']['content'][0]['text']) +``` + +## More Info + +For complete documentation including Guardrails, Knowledge Bases, and Agents, see: +- [Full Bedrock Passthrough Docs](./pass_through/bedrock) + diff --git a/docs/my-website/docs/bedrock_invoke.md b/docs/my-website/docs/bedrock_invoke.md new file mode 100644 index 00000000000..6d86778bc47 --- /dev/null +++ b/docs/my-website/docs/bedrock_invoke.md @@ -0,0 +1,143 @@ +# /invoke + +Call Bedrock's `/invoke` endpoint through LiteLLM Proxy. + +| Feature | Supported | +|---------|-----------| +| Cost Tracking | ✅ | +| Logging | ✅ | +| Streaming | ✅ via `/invoke-with-response-stream` | +| Load Balancing | ✅ | + +## Quick Start + +### 1. Setup config.yaml + +```yaml showLineNumbers +model_list: + - model_name: my-bedrock-model + litellm_params: + model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0 + aws_region_name: us-west-2 + aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID # reads from environment + aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY + custom_llm_provider: bedrock +``` + +Set AWS credentials in your environment: + +```bash showLineNumbers +export AWS_ACCESS_KEY_ID="your-access-key" +export AWS_SECRET_ACCESS_KEY="your-secret-key" +``` + +### 2. Start Proxy + +```bash showLineNumbers +litellm --config config.yaml + +# RUNNING on http://0.0.0.0:4000 +``` + +### 3. Call /invoke endpoint + +```bash showLineNumbers +curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/invoke' \ +-H 'Authorization: Bearer sk-1234' \ +-H 'Content-Type: application/json' \ +-d '{ + "max_tokens": 100, + "messages": [ + { + "role": "user", + "content": "Hello, how are you?" + } + ], + "anthropic_version": "bedrock-2023-05-31" +}' +``` + +## Streaming + +For streaming responses, use `/invoke-with-response-stream`: + +```bash showLineNumbers +curl -X POST 'http://0.0.0.0:4000/bedrock/model/my-bedrock-model/invoke-with-response-stream' \ +-H 'Authorization: Bearer sk-1234' \ +-H 'Content-Type: application/json' \ +-d '{ + "max_tokens": 100, + "messages": [ + { + "role": "user", + "content": "Tell me a short story" + } + ], + "anthropic_version": "bedrock-2023-05-31" +}' +``` + +## Load Balancing + +Define multiple deployments with the same `model_name` for automatic load balancing: + +```yaml showLineNumbers +model_list: + # Deployment 1 - us-west-2 + - model_name: my-bedrock-model + litellm_params: + model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0 + aws_region_name: us-west-2 + aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID + aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY + custom_llm_provider: bedrock + + # Deployment 2 - us-east-1 + - model_name: my-bedrock-model + litellm_params: + model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0 + aws_region_name: us-east-1 + aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID + aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY + custom_llm_provider: bedrock +``` + +The proxy automatically distributes requests across both regions. + +## Using boto3 SDK + +```python showLineNumbers +import boto3 +import json + +bedrock_runtime = boto3.client( + service_name='bedrock-runtime', + region_name='us-west-2', + endpoint_url='http://0.0.0.0:4000/bedrock' +) + +def add_custom_headers(request, **kwargs): + request.headers.update({'litellm_user_api_key': 'Bearer sk-1234'}) + +bedrock_runtime.meta.events.register('before-send.*.*', add_custom_headers) + +response = bedrock_runtime.invoke_model( + modelId='my-bedrock-model', + contentType='application/json', + accept='application/json', + body=json.dumps({ + "max_tokens": 100, + "messages": [{"role": "user", "content": "Hello"}], + "anthropic_version": "bedrock-2023-05-31" + }) +) + +response_body = json.loads(response['body'].read()) +print(response_body['content'][0]['text']) +``` + +## More Info + +For complete documentation including Guardrails, Knowledge Bases, and Agents, see: +- [Full Bedrock Passthrough Docs](./pass_through/bedrock) + diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 71f730c4542..e5d1ab81ac4 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -347,6 +347,8 @@ const sidebars = { ] }, "moderation", + "bedrock_invoke", + "bedrock_converse", "ocr", { type: "category",