diff --git a/litellm-rust/INTERVIEW_TASK.md b/litellm-rust/INTERVIEW_TASK.md index ccc3ffd5611..197390a50a2 100644 --- a/litellm-rust/INTERVIEW_TASK.md +++ b/litellm-rust/INTERVIEW_TASK.md @@ -1,75 +1,87 @@ -# Bedrock Messages Interview Task +# Task: serve AWS Bedrock on `POST /v1/messages` -Complete the AWS Bedrock implementation behind the Anthropic-compatible -`POST /v1/messages` route. The gateway plumbing, provider registration, bearer -authentication setup, SigV4 support scaffolding, streaming response plumbing, -test harness, and live harness are already present. The remaining work is to -complete the shared Messages abstractions, Bedrock request and response -transforms, and Bedrock EventStream decoding. +The gateway in this workspace already serves the Anthropic Messages API for +Anthropic and Azure. Your job is to make it serve AWS Bedrock too, well enough +that the Claude Code CLI can point `ANTHROPIC_BASE_URL` at this gateway and +complete a turn against a Bedrock-hosted Claude -## Given +Claude Code always sends `stream: true`, so streaming is part of the job, not a +stretch goal -The gateway is configured to use: +## What you are given -- `AWS_BEARER_TOKEN_BEDROCK` for the live bearer credential -- Region `us-west-2` -- Model `us.anthropic.claude-sonnet-4-5-20250929-v1:0` +Credentials and target, already in your environment: -Claude 3.5 and Claude 3.7 Bedrock model identifiers are end of life and should -not be used. +- `AWS_BEARER_TOKEN_BEDROCK`, a Bedrock API key. Bedrock accepts it as a plain + `Authorization: Bearer ` header, so you do not need to implement SigV4 +- Region `us-west-2`, model `us.anthropic.claude-sonnet-4-5-20250929-v1:0`. + The Claude 3.5 and 3.7 Bedrock ids are end of life and return 404 -Bedrock streaming responses use -`application/vnd.amazon.eventstream`, not ordinary SSE. The EventStream -payload contains an object shaped like -`{"bytes": ""}`. The -`aws-smithy-eventstream::MessageFrameDecoder` type is available for parsing -the binary framing. +Already wired up, so you should not need to touch it: -## Running the gateway and live harness +- provider registration, the `BEDROCK_MODEL` startup config, the bearer and + SigV4 auth paths in `crates/ai-gateway/src/messages/{prepare,handler}.rs`, + and the streaming plumbing in `crates/ai-gateway/src/routes/messages/mod.rs` + that routes a Bedrock response through a normalizing SSE stream +- `bedrock_messages_harness.sh`, a one-command live check against real Bedrock +- failing tests that specify the behavior you need to produce -From `litellm-rust/`, start the gateway with: +Two facts about Bedrock that would otherwise cost you an hour of reading: + +- non-streaming requests go to `POST /model/{modelId}/invoke`; streaming goes + to `POST /model/{modelId}/invoke-with-response-stream`. The model id is in + the path, not the body +- the streaming response is `application/vnd.amazon.eventstream`, a binary + framing format, not SSE. Each frame's payload is + `{"bytes": ""}`. + `aws_smithy_eventstream::frame::MessageFrameDecoder` parses the framing for + you and is already a dependency + +## What is yours to write + +Everything marked `todo!()`, plus whatever the compiler and the failing tests +tell you is missing: + +- `crates/core/src/providers/bedrock/messages/transformation.rs`, the Bedrock + provider config: URL construction, region resolution, the request transform, + and the response transform. `providers/anthropic/messages/transformation.rs` + and `providers/azure_ai/messages/transformation.rs` are the two existing + implementations of the same trait; read them first +- whatever the shared contract in `crates/core/src/messages/transformation.rs` + needs in order to express Bedrock. It currently cannot express everything + Bedrock requires, and part of this task is deciding how to extend it +- the EventStream decode loop in `crates/ai-gateway/src/routes/messages/mod.rs` + +## Running it ```bash -cargo run -p litellm-ai-gateway --features server +cd litellm-rust +cargo run -p litellm-ai-gateway --features server # BEDROCK_MODEL, PORT, LITELLM_MASTER_KEY from env +./bedrock_messages_harness.sh # in another shell ``` -In another shell, with `AWS_BEARER_TOKEN_BEDROCK` available in the -environment, run: +The harness runs four scenarios: a non-streaming turn, a streaming turn, a +tool_use round trip, and an invalid model id -```bash -./bedrock_messages_harness.sh -``` +## Definition of done -The harness exercises simple non-streaming output, streaming output, tool use, -and an invalid-model response. +1. all four harness scenarios behave correctly against live Bedrock +2. the Bedrock tests in core and in the messages route pass +3. `cargo test --workspace` is green, including the roughly 150 tests that + existed before you started +4. `cargo fmt --check` and + `cargo clippy -p litellm-ai-gateway --all-targets --features server -- -D warnings` + are clean +5. Claude Code, pointed at the gateway, completes a turn -## Tests and checks +To point Claude Code at the gateway, set `ANTHROPIC_BASE_URL` to the gateway's +address, `ANTHROPIC_AUTH_TOKEN` to the gateway master key, and `ANTHROPIC_MODEL` +to the Bedrock model id, with `ANTHROPIC_API_KEY` unset -Format and compile the workspace with: +## Not in scope -```bash -cargo fmt --check -cargo build -``` +`POST /v1/messages/count_tokens`, multi-deployment routing, and Python-side +integration. If you find yourself in any of those, come up for air -Run the existing test suite with: - -```bash -cargo test --workspace -``` - -The Bedrock-specific tests in the core transformation and messages route -modules describe the expected behavior. They are intentionally failing until -the implementation is completed. - -## Required scenarios - -Make all of these scenarios pass: - -1. Non-streaming Bedrock InvokeModel requests use the correct encoded model - path, request body shape, authentication headers, and response mapping -2. Streaming Bedrock EventStream frames, including frames split across - transport chunks, become normalized Anthropic SSE -3. Tool-use requests and responses round trip correctly -4. Bedrock HTTP errors and EventStream exception frames map to the gateway's - expected error shapes +Read `CLAUDE.md` and `AGENTS.md` in this directory before you start. They are +the standards this code is held to diff --git a/litellm-rust/crates/core/src/providers/bedrock/messages/transformation.rs b/litellm-rust/crates/core/src/providers/bedrock/messages/transformation.rs index 6c5407fae4c..52306ce8f13 100644 --- a/litellm-rust/crates/core/src/providers/bedrock/messages/transformation.rs +++ b/litellm-rust/crates/core/src/providers/bedrock/messages/transformation.rs @@ -23,7 +23,7 @@ impl AnthropicMessagesProviderConfig for BedrockMessagesConfig { env_lookup: &dyn Fn(&str) -> Option, ) -> CoreResult { let _ = (api_base, model, env_lookup); - todo!("extend the shared URL contract for streaming, then implement Bedrock URLs") + todo!("Bedrock InvokeModel URLs carry the model id and the streaming mode in the path") } fn signing_region( @@ -32,7 +32,7 @@ impl AnthropicMessagesProviderConfig for BedrockMessagesConfig { env_lookup: &dyn Fn(&str) -> Option, ) -> Option { let _ = (api_base, env_lookup); - todo!("implement Bedrock region precedence and endpoint parsing") + todo!("resolve the AWS region the request is signed and routed for") } fn resolve_api_key( @@ -44,7 +44,9 @@ impl AnthropicMessagesProviderConfig for BedrockMessagesConfig { } fn auth_strategy(&self) -> MessagesAuthStrategy { - todo!("extend MessagesAuthStrategy for Bedrock bearer and SigV4 authentication") + todo!( + "Bedrock authenticates with a bearer token or a signed request, not a static api key header" + ) } fn transform_request(