From 0ac7a383db34a4c3775938e8fe8fe4910a67e88e Mon Sep 17 00:00:00 2001 From: Nate Armstrong Date: Mon, 19 Oct 2026 12:55:26 -0700 Subject: [PATCH] feat(loadtest): chat completions parity check between the python proxy and the gateway (#47362) * feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough Adds azure/ chat completions, streaming and not, on the same passthrough the openai/ route uses. The request goes to the deployment path with the api-version query, or to /openai/v1/ for the v1, latest and preview versions, with api_base, api_key and api_version falling back to the AZURE_* variables and os.environ references resolved as the proxy does. A key is sent as api-key, and an azure_ad_token or an Entra ID token from auth-azure goes as a Bearer token. The body follows the proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the deployment name and reading model capabilities from the cost map, so an unsupported param is refused unless drop_params allows it to be dropped. Spend is priced as the proxy prices Azure, by azure/ when one is set and otherwise by the model the response names, and a content filter error keeps Azure's inner error for the client * feat(gateway): Anthropic and Bedrock chat completions streaming Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/ deployments, which were refused with 501 before. The request is the non-streaming one with "stream": true added for Anthropic and sent to /converse-stream for Bedrock, and the provider's events are decoded the way the proxy's chunk parsers decode them: text, tool call deltas with empty arguments sent as "{}", reasoning_content and thinking_blocks with their index and signature, citations and the mapped finish_reason. A wrapper in the shape of the proxy's stream wrapper gives every chunk one id and the client's model name, sends role on the first chunk, adds the empty finish chunk, strips usage from the chunks and, when stream_options.include_usage is set, ends with the usage chunk before [DONE]. The gateway waits for the first chunk before answering, so an upstream error up to there is retried or falls back to another deployment, and an error after it ends the stream with the OpenAI error frame. Usage, cache reads and writes included, is handed to the meter through the response whether or not the client asked for it, and dropping the client stream drops the upstream request * feat(gateway): full openai chat request translation for anthropic and bedrock (#47088) * feat(gateway): share router cooldowns and follow proxy key invalidations through Redis Gateway replicas now share state through the coordination Redis the Python proxy uses (general_settings.coordination_redis, a redis response cache, or REDIS_*): - A deployment one replica cools down is written under Python's deployment::cooldown key and applied by every other replica's background poll (one MGET a second), so the fleet stops sending to a failing deployment together. allowed_fails counts failures across replicas, as Python's v1 rule does; the v2 failure-rate rule stays per replica, as in Python. Deployments are named by model_info.id, or an id derived from the config that is the same on every replica. - The virtual key cache follows the proxy's litellm_proxy.auth_cache_invalidation channel, so a key blocked, deleted or edited through the admin API (or its team or user) is re-read on every replica at once instead of after up to 60 s. general_settings.user_api_key_cache_ttl sets the key TTL. Nothing is added to the path of a successful request. Without Redis, or while it is down, each replica keeps its own state as before. Redis Cluster and Sentinel are refused at startup for now. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): hooks and observers around inference requests, with usage and cost Adds the litellm-gateway-hooks crate: request facts built only when something reads them, on-path hooks that declare what they read and what they may change and run in config order, off-path observers on bounded queues, a usage tap for SSE and JSON responses, and per-deployment prices from the model cost map overlaid by config prices. The messages and chat handlers run through it when anything is installed; with nothing installed requests take the old path. The key store now also returns the key alias, user, team alias and key and team metadata, and router attempts carry the deployment id. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): write spend to the proxy's database from a native observer Adds a spend writer observer that records every gateway request in the proxy's own Postgres tables with the same statements the Python proxy uses: spend logs, key/user/team/team member/end user spend, and the daily user, team and end-user activity tables. Writes are batched on the proxy's timers and are additive, so replicas and Python proxies can write side by side. The database pool is now shared between virtual keys and the writer. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): enforce key, team, user, member and end-user budgets on shared spend counters Adds a built-in budgets hook that runs the proxy's budget checks, in its order and with its messages, on the proxy's Redis spend:* counters. Each request reads and seeds the counters, checks them and reserves its maximum cost in one Redis script, then settles the reservation to its real cost off the request path, so replicas and Python proxies sharing the Redis enforce one budget. Without Redis the counters are kept per replica. The key store now reads user, team member and end-user budgets, and auth no longer checks budgets against stored spend. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): enforce key, user, team, member and end-user rate limits on shared counters Adds a built-in rate limits hook with the proxy's v3 limiter semantics: requests per window, tokens per window and in-flight requests per key, on the proxy's own {scope:id}:window|requests|tokens|max_parallel_requests Redis keys and with its scripts, so replicas and Python proxies sharing the Redis enforce one limit. Tokens are reserved from the proxy's estimate and settled to usage when the response ends. Refusals carry the proxy's message, body and headers on each endpoint, and complete responses carry its x-ratelimit-* headers. Without Redis the counters are kept per replica. The key store now reads key, user, team and member rpm/tpm limits and the key's max_parallel_requests, and end users' budget limits are read with their budgets; keys with plain rate limits are no longer refused. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): configurable header, spend metadata and provider metadata hooks Adds four built-in hooks a config lists under litellm_settings.hooks and runs in that order: forward_headers copies named client headers to the provider, strip_headers removes headers from the provider request (including the gateway's default pass-through), spend_metadata adds header and key-metadata values to the spend log's spend_logs_metadata, and provider_metadata sets Messages body fields and headers from templates over request facts. Hooks' upstream header edits now reach the provider on both endpoints, and body edits can set nested fields. Credentials and signer headers are never forwarded or set, and a misconfigured hook stops the gateway at startup. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): model capacity hook, the proxy's dynamic rate limiter Adds the model_capacity hook, turned on by listing dynamic_rate_limiter_v3 (or model_capacity) in litellm_settings.callbacks. It caps each model group at the sum of its deployments' rpm and tpm across every replica, and with priority_reservation holds each priority's pool to its share once the group's saturation reaches the threshold, counting it below. It uses the proxy's model_saturation_check and priority_model keys and scripts, its refusal messages and headers, and adds a successful response's tokens to the group and the pool admission chose. The rate limit counters gain a general check-and-add over request and token pairs, and the throttling refusal shape is shared. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * test(gateway-auth): expect the narrowed unsupported rate limit message Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): prometheus observer with the proxy's metric and label names Adds the prometheus observer, turned on by listing prometheus in litellm_settings.callbacks, success_callback or failure_callback. Each finished request updates the proxy's request, spend, token, latency and deployment metrics under its names and labels, and /metrics serves them after the gateway's own. prometheus_metrics_config, the exclude lists and prometheus_latency_buckets are read as the proxy reads them. The metrics the gateway already served under the same names come from the observer alone when it is on. Outcomes now carry when the attempt was sent upstream, for the LLM API latency and time to first token. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): s3_v2 observer writing the proxy's logging payload to S3 Adds the s3_v2 observer, turned on by listing s3_v2 (or s3_log) in litellm_settings.callbacks, success_callback or failure_callback. Each finished request becomes the proxy's StandardLoggingPayload, with the messages from the request body and the response as the client got it, streams put back together. It is queued and written every flush interval under the proxy's key layout, with its Content-MD5, Content-Disposition and encryption headers, one PUT per request or one NDJSON object per folder with s3_batch_file_upload. s3_callback_params is read as the proxy reads it, os.environ references included. Uploads are retried, dropped and kept for the next flush by the proxy's rules, and what is queued is written at shutdown. auth-aws gains an S3 signer that signs the path as sent and every header, as botocore does. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough Adds azure/ chat completions, streaming and not, on the same passthrough the openai/ route uses. The request goes to the deployment path with the api-version query, or to /openai/v1/ for the v1, latest and preview versions, with api_base, api_key and api_version falling back to the AZURE_* variables and os.environ references resolved as the proxy does. A key is sent as api-key, and an azure_ad_token or an Entra ID token from auth-azure goes as a Bearer token. The body follows the proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the deployment name and reading model capabilities from the cost map, so an unsupported param is refused unless drop_params allows it to be dropped. Spend is priced as the proxy prices Azure, by azure/ when one is set and otherwise by the model the response names, and a content filter error keeps Azure's inner error for the client * feat(gateway): Anthropic and Bedrock chat completions streaming Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/ deployments, which were refused with 501 before. The request is the non-streaming one with "stream": true added for Anthropic and sent to /converse-stream for Bedrock, and the provider's events are decoded the way the proxy's chunk parsers decode them: text, tool call deltas with empty arguments sent as "{}", reasoning_content and thinking_blocks with their index and signature, citations and the mapped finish_reason. A wrapper in the shape of the proxy's stream wrapper gives every chunk one id and the client's model name, sends role on the first chunk, adds the empty finish chunk, strips usage from the chunks and, when stream_options.include_usage is set, ends with the usage chunk before [DONE]. The gateway waits for the first chunk before answering, so an upstream error up to there is retried or falls back to another deployment, and an error after it ends the stream with the OpenAI error frame. Usage, cache reads and writes included, is handed to the meter through the response whether or not the client asked for it, and dropping the client stream drops the upstream request * feat(gateway): full OpenAI chat request translation for Anthropic and Bedrock Anthropic and Bedrock Converse deployments now take the whole OpenAI chat request the Python proxy accepts instead of a narrow allowlist. Messages translate as Python builds them: system as a string, parts or several messages, text, image_url (URL or data URL), file and document parts, assistant tool_calls, the tool and function roles, consecutive role merging, empty content, cache_control on Anthropic and cachePoint on Bedrock. Params map per provider: max_tokens and max_completion_tokens with Anthropic's model default, temperature, top_p, stop, tools and tool_choice, parallel_tool_calls, response_format as a json tool or native output format, reasoning_effort and thinking, user as metadata, with n and any other unsupported param refused with a 400 unless drop_params is set globally, on the deployment or on the request, and allowed_openai_params and additional_drop_params honoured. Responses carry tool calls, thinking blocks and response_format answers back in OpenAI shape, streamed and not. The route downloads media URLs a provider cannot take by reference, merges the beta headers a request needs with forwarded ones, and the gateway hands the cost map to both providers for the capability rules that read it. * fix(rust-bridge): read Python's model_cost for chat translation capabilities Chat translation for Anthropic and Bedrock reads capability flags from the cost map. The Python bridge passed none, so every SDK chat call through the Rust path failed. It now reads litellm.model_cost live, so lookups match Python's, runtime-registered models included. ModelInfo::entry returns a Cow so a map read from Python can hand over a copy. Two tests sent temperature != 1 to a model that removed sampling params, which Python itself refuses; they now send temperature=1. * fix(model-catalog): read token counts the cost map writes as whole floats The published cost map now has prompt_cache_min_tokens: 4096.0, which the typed u64 fields rejected, so the gateway dropped the whole map at startup and priced every request at zero. Counts accept any whole non-negative number, as Python reads any JSON number, and a test keeps the checked-in map parsing. * feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough (#47074) Adds azure/ chat completions, streaming and not, on the same passthrough the openai/ route uses. The request goes to the deployment path with the api-version query, or to /openai/v1/ for the v1, latest and preview versions, with api_base, api_key and api_version falling back to the AZURE_* variables and os.environ references resolved as the proxy does. A key is sent as api-key, and an azure_ad_token or an Entra ID token from auth-azure goes as a Bearer token. The body follows the proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the deployment name and reading model capabilities from the cost map, so an unsupported param is refused unless drop_params allows it to be dropped. Spend is priced as the proxy prices Azure, by azure/ when one is set and otherwise by the model the response names, and a content filter error keeps Azure's inner error for the client * feat(gateway): Anthropic and Bedrock chat completions streaming (#47077) Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/ deployments, which were refused with 501 before. The request is the non-streaming one with "stream": true added for Anthropic and sent to /converse-stream for Bedrock, and the provider's events are decoded the way the proxy's chunk parsers decode them: text, tool call deltas with empty arguments sent as "{}", reasoning_content and thinking_blocks with their index and signature, citations and the mapped finish_reason. A wrapper in the shape of the proxy's stream wrapper gives every chunk one id and the client's model name, sends role on the first chunk, adds the empty finish chunk, strips usage from the chunks and, when stream_options.include_usage is set, ends with the usage chunk before [DONE]. The gateway waits for the first chunk before answering, so an upstream error up to there is retried or falls back to another deployment, and an error after it ends the stream with the OpenAI error frame. Usage, cache reads and writes included, is handed to the meter through the response whether or not the client asked for it, and dropping the client stream drops the upstream request --------- Co-authored-by: Claude Opus 5.5 (1M context) * feat(gateway): gemini chat completions on google ai studio and vertex ai (#47098) * feat(gateway): Gemini chat completions on Google AI Studio and Vertex AI gemini/ deployments and vertex_ai/ deployments of Gemini models now serve /v1/chat/completions, streamed and not, instead of being refused. Both surfaces share one translation ported from the Python VertexGeminiConfig. Params map as Python maps them: temperature, top_p, max_tokens and max_completion_tokens, n, stop, penalties where the model takes them, seed, logprobs, tools with function declarations and the built in search, code execution, url context, maps and computer use tools, tool_choice, response_format as a response schema, reasoning_effort and thinking as a thinking config, modalities, audio and labels. Messages become contents with a system instruction when the cost map allows it, images, files, audio, function calls and responses, and thought signatures carried back on tool call ids AI Studio sends the key in x-goog-api-key from the deployment or GOOGLE_API_KEY, GEMINI_API_KEY or PALM_API_KEY, and calls v1alpha for Gemini 3 and v1beta otherwise. Vertex AI calls the project and location URL with a forwarded bearer or a Google access token from vertex_credentials, the environment or ADC, and grafts the path onto a custom api_base. Responses carry text, reasoning, tool calls, grounding annotations, logprobs and the vertex_ai metadata, a blocked prompt or flagged answer comes back as content_filter, and usage counts thoughts and cached tokens as Python does so cost is recorded. The gateway resolves os.environ references in the deployment key, base and Vertex params, passes the deployment Vertex params to the call, and loads the cost map for gemini and vertex_ai deployments. Claude and partner models on Vertex AI stay refused * test(gateway): gemini cost map test double returns a Cow entry * feat(gateway): native chat for openai-compatible providers (#47099) Serve chat completions for hosted_vllm, nvidia_nim, groq, together_ai, fireworks_ai, xai, mistral, deepseek, openrouter, cerebras and perplexity through the OpenAI passthrough, with each provider's Python config ported as a request transform in llms and its response and stream fixups applied on the way back. Streams ask for stream_options.include_usage on the client's behalf where the provider takes it, record the usage through ReportedUsage for the spend hooks, and keep it from the client unless the client asked for it. * feat(gateway): claude chat completions on vertex ai (#47101) vertex_ai/ deployments of Claude models now serve /v1/chat/completions, streamed and not, instead of being refused. The request is the Anthropic chat translation built for the vertex_ai host, as Python's VertexAIAnthropicConfig builds it on top of AnthropicConfig: capabilities read the vertex_ai/ cost map entry first and native structured output reads only that entry, service_tier and speed are refused or dropped with drop_params, and the Anthropic API betas are not added. The body drops model, carries anthropic_version vertex-2023-10-16, folds output_format into output_config and keeps effort only for models that take it, and names its context management and caller betas in anthropic_beta and the anthropic-beta header, filtered to what Vertex accepts, with the web search beta alone when there are none. The call goes to the publisher rawPredict URL for the project and location, regional or global, or a custom api_base or VERTEXAI_API_BASE, with a Google access token from vertex_credentials, the environment or ADC. Streams call streamRawPredict with stream true, alt=sse on a custom base, decode the Anthropic events as the anthropic/ path does and hand usage to the hooks through ReportedUsage. Every remote image is downloaded first, since Vertex takes none by URL, and the response names the requested model. * fix(gateway): find anthropic stream usage by event name (#47206) The stream meter kept a data line only when "message_start" or "message_delta" appeared in its first 96 bytes. An upstream that puts "type" after the message object (JSON key order is not fixed) had every message_start skipped, so input tokens were recorded as 0 and spend, budgets, rate limits, metrics and S3 logs undercounted. Anthropic streams name every event, so the event: line now decides; the byte-window check remains for streams without event lines. * fix(gateway): stop hooks pinning connection read buffers for a stream's life (#47220) The s3 logger asks for the response text, and the tap kept every response chunk as a Bytes slice of the upstream connection's read buffer, so each open stream held a buffer per chunk until it ended (about 460 KB per open 10k-token stream locally). Chunks are now copied into one owned buffer. The same applies on the request side: the facts kept the raw request slice and a clone of the client headers, both slices of the client connection's read buffer. The body is now released once hooks have run (an owned copy is kept only when an observer reads it) and the headers are copied. * feat(loadtest): full-feature mode with recorded-data verification (#47207) * feat(loadtest): full-feature mode with recorded-data verification genconfig -features full turns on what both gateways do per request: owners with rpm/tpm/budgets/max_parallel, deployment capacity, the dynamic_rate_limiter_v3, prometheus and s3_v2 callbacks, and the four request hooks with Python stand-ins (custom_callbacks.py). -key-sets gives each gateway its own keys, users and teams in a shared database. verify checks spend logs, owner spend, daily tables, Prometheus counters, S3 logs and the headers the provider received against what loadgen sent, and compares per-deployment prices across sides. s3stub serves S3 for the logs. mockllm takes in= for exact input tokens, returns unique response ids, and counts chosen request headers; loadgen takes -keys and -header. * feat(loadtest): s3stub -max-string keeps long runs' logs in memory * feat(loadtest): bifrost as a third side of the full-feature run (#47236) genconfig writes a "bifrost" key set as a Bifrost config: a custom provider per deployment, a routing rule per model group, virtual keys, and with -features full key/team/customer budgets and rate limits, model configs for deployment limits, the logs store with S3 payloads, the telemetry plugin and header forwarding/stripping/logging. A pricing file copies the cost map's prices per deployment. mockllm answers Bedrock Converse (streamed and not), serves a self-signed TLS listener for Bifrost's HTTPS-only Bedrock provider, and echoes the request's model, which Bifrost prices by. s3stub answers HeadBucket. loadgen takes -chat-path and -messages-path for Bifrost's /anthropic/v1/messages. verify reads a side named bifrost in Bifrost's schema and reports what it has no counterpart for (users, members, daily tables, provider_metadata, cross-replica state) as n/a; with three sides it compares every pair. Claude-Session: https://claude.ai/code/session_01XF7rGf49TWYH5Hqkb14eSL Co-authored-by: Claude Opus 5.5 (1M context) * feat(loadtest): chat completions parity check between the python proxy and the gateway chatparity sends the same /v1/chat/completions requests through the Python proxy and the Rust gateway, both loading one config pointed at its mock provider, and diffs what each provider was sent (path, headers, body) and what the client got back (status, body, the assembled stream, and the stream chunk by chunk as information). The mock answers OpenAI-compatible, Azure, Anthropic, Bedrock Converse (event-stream framing), Gemini, Vertex Gemini and Vertex Claude the same way for both proxies, and issues Vertex access tokens for a throwaway service account. Cases cover 19 deployments and 15 request scenarios: text, multi-turn, tools, tool results, response_format, images, reasoning_effort, cache_control, unsupported params refused and dropped, max_completion_tokens, and streams with and without include_usage. scripts/chat-parity.sh builds and starts everything locally and writes report.md plus one capture per case. Reviewed differences go in cmd/chatparity/known.json with a reason; the run fails on the rest. --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .../docs/customer-proxy/chat-parity.svg | 41 ++ litellm-rust/loadtest/README.md | 47 ++ litellm-rust/loadtest/cmd/chatparity/cases.go | 190 ++++++++ .../cmd/chatparity/chatparity_test.go | 112 +++++ litellm-rust/loadtest/cmd/chatparity/diff.go | 339 +++++++++++++ .../loadtest/cmd/chatparity/known.json | 1 + litellm-rust/loadtest/cmd/chatparity/main.go | 289 +++++++++++ litellm-rust/loadtest/cmd/chatparity/mock.go | 461 ++++++++++++++++++ litellm-rust/loadtest/scripts/chat-parity.sh | 49 ++ 9 files changed, 1529 insertions(+) create mode 100644 litellm-rust/docs/customer-proxy/chat-parity.svg create mode 100644 litellm-rust/loadtest/cmd/chatparity/cases.go create mode 100644 litellm-rust/loadtest/cmd/chatparity/chatparity_test.go create mode 100644 litellm-rust/loadtest/cmd/chatparity/diff.go create mode 100644 litellm-rust/loadtest/cmd/chatparity/known.json create mode 100644 litellm-rust/loadtest/cmd/chatparity/main.go create mode 100644 litellm-rust/loadtest/cmd/chatparity/mock.go create mode 100755 litellm-rust/loadtest/scripts/chat-parity.sh diff --git a/litellm-rust/docs/customer-proxy/chat-parity.svg b/litellm-rust/docs/customer-proxy/chat-parity.svg new file mode 100644 index 00000000000..b741dc57b2b --- /dev/null +++ b/litellm-rust/docs/customer-proxy/chat-parity.svg @@ -0,0 +1,41 @@ + + + + + + + + chatparity: one request, two proxies, one recording mock + + + chatparity run + 340 cases: + 19 deployments x + 15 scenarios x mode + + + Python proxy + litellm --config config.yaml + + + Rust gateway + same config.yaml + + + mock provider (in chatparity) + OpenAI-compatible, Azure, Anthropic, + Bedrock Converse, Gemini, Vertex + records every request it gets + + + + + + 1. send + 2. send + upstream A + upstream B + + + diff upstream A vs B (path, headers, body) and response A vs B (status, body, assembled stream) → report.md + diff --git a/litellm-rust/loadtest/README.md b/litellm-rust/loadtest/README.md index 7a03f1c3fb2..9c96c0ce4d0 100644 --- a/litellm-rust/loadtest/README.md +++ b/litellm-rust/loadtest/README.md @@ -12,6 +12,7 @@ machines. | `summarize` | Merges result files and prints proxy overhead (observed minus planned time) by stage, class and context size. | | `procmon` | Samples the gateway process's CPU, RSS, fds and TCP connections from `/proc`. | | `verify` | Checks what each gateway recorded for a run against what loadgen sent: spend logs, key/team/user/member spend, daily tables, Prometheus counters, S3 request logs and the headers the provider received. See [Full-feature runs](#full-feature-runs). | +| `chatparity` | Sends the same chat completions requests through the Python proxy and the gateway, both pointed at its recording mock provider, and reports every difference in what the provider was sent and what the client got. See [Chat parity](#chat-parity). | | `s3stub` | In-memory path-style S3 (PUT, GET, ListObjectsV2) for the `s3_v2` logs of a full-feature run. | Workloads live in `workloads/`: `smoke`, `realistic-ramp`, `long-streams`, `large-context`, @@ -185,3 +186,49 @@ carry static dummy AWS keys, so every request is SigV4-signed, and mockllm answe `/model//invoke` and `/invoke-with-response-stream` with AWS event-stream framing that the gateway re-encodes as Anthropic SSE. Bedrock hits show up as `bedrock--` in the mock's `/stats`. +## Chat parity + +`scripts/chat-parity.sh` checks the gateway's `/v1/chat/completions` against the Python proxy's. +It writes one config, starts both proxies on it, and sends every case through each one in turn. +Both talk to `chatparity`'s mock, which answers like each provider's API (OpenAI-compatible, Azure, +Anthropic, Bedrock Converse, Gemini, Vertex Gemini and Vertex Claude) and records every request it +receives. + +```sh +scripts/chat-parity.sh # every case, report in out/chat-parity/report.md +FILTER=bedrock-env/ scripts/chat-parity.sh # one deployment, scenario or mode +GATEWAY= PYTHON=/bin/python scripts/chat-parity.sh +``` + +A case is one deployment, one request scenario and one mode (`//`). +There are 19 deployments (every chat provider the gateway serves natively) and 15 scenarios: plain +text, multi-turn with a system message, tools, a tool result round trip, `response_format` +(JSON schema and JSON object), inline and remote images, `reasoning_effort`, `cache_control`, +unsupported params refused and then dropped, `max_completion_tokens`, and streams with and without +`include_usage`. The mock gives both proxies the same answer for the same request (text, a tool +call, thinking or JSON, picked by a `[[parity reply=...]]` marker in the prompt), so any difference +in what the client gets comes from the proxies. + +For each case it compares: + +- **upstream**: the method, path, query, headers and JSON body the provider received. Transport + headers (user agent, content length, SDK telemetry) are ignored, and a SigV4 signature is reduced + to its region and service. +- **response**: the status, content type and JSON body the client got. Ids and timestamps the + proxies generate are compared by prefix only (`chatcmpl-*`). For a stream, this is what a client + library would assemble: content, reasoning, tool calls, finish reasons, usage, the models named, + whether one id spans the stream, and `[DONE]`. +- **chunks**: the stream chunk by chunk. This is informational only, since the same content can be + split into chunks differently. + +`out/chat-parity/report.md` lists every open difference, and `cases/.json` holds the request, +both responses and everything each proxy sent upstream. Once a difference has been reviewed and +accepted, add it to `cmd/chatparity/known.json` (`{"case": "*/stream", "where": "upstream", "path": +"*.accept", "reason": "..."}`, where `*` matches anything). The run fails only on differences that +aren't listed there. + +The Python side needs the proxy and `google-auth` installed (`uv pip install -e '.[proxy]' +google-auth`). Both price from the repo's `model_prices_and_context_window.json`, so parameter +support and defaults come from the same data. `bedrock` carries its AWS settings in the config; +`bedrock-env` uses `api_base` with credentials from the environment, which the script sets to dummy +values that only ever reach the mock. diff --git a/litellm-rust/loadtest/cmd/chatparity/cases.go b/litellm-rust/loadtest/cmd/chatparity/cases.go new file mode 100644 index 00000000000..6f197d865cb --- /dev/null +++ b/litellm-rust/loadtest/cmd/chatparity/cases.go @@ -0,0 +1,190 @@ +package main + +import ( + "fmt" + "strings" +) + +// deployment is one model group in the shared config, pointing at the mock. +type deployment struct { + name string // model group name the client asks for + family string // which scenarios apply: openai, claude, gemini or compat + params map[string]any // litellm_params, with {mock} replaced by the mock's URL +} + +func deployments(credentials string) []deployment { + compat := func(name, model string) deployment { + return deployment{name, "compat", map[string]any{"model": model, "api_base": "{mock}/v1", "api_key": "sk-parity-upstream"}} + } + vertex := map[string]any{"api_base": "{mock}", "vertex_project": "parity-project", "vertex_credentials": credentials} + with := func(base map[string]any, extra map[string]any) map[string]any { + out := map[string]any{} + for k, v := range base { + out[k] = v + } + for k, v := range extra { + out[k] = v + } + return out + } + return []deployment{ + {"openai", "openai", map[string]any{"model": "openai/gpt-4o", "api_base": "{mock}/v1", "api_key": "sk-parity-upstream"}}, + {"azure", "openai", map[string]any{"model": "azure/gpt-4o", "api_base": "{mock}", "api_key": "sk-parity-upstream", + "api_version": "2025-02-01-preview"}}, + {"anthropic", "claude", map[string]any{"model": "anthropic/claude-sonnet-4-5-20250929", "api_base": "{mock}", + "api_key": "sk-parity-upstream"}}, + {"bedrock", "claude", map[string]any{"model": "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "aws_access_key_id": "AKIAPARITYPARITY", "aws_secret_access_key": "parity-secret", "aws_region_name": "us-east-1", + "aws_bedrock_runtime_endpoint": "{mock}"}}, + // The same model with the endpoint as api_base and credentials from the environment + // (chat-parity.sh sets them), for a host that does not read AWS settings from the config. + {"bedrock-env", "claude", map[string]any{"model": "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", "api_base": "{mock}"}}, + {"vertex-claude", "claude", with(vertex, map[string]any{"model": "vertex_ai/claude-sonnet-4-5@20250929", + "vertex_location": "us-east5"})}, + {"gemini", "gemini", map[string]any{"model": "gemini/gemini-2.5-flash", "api_base": "{mock}", "api_key": "parity-google-key"}}, + {"vertex-gemini", "gemini", with(vertex, map[string]any{"model": "vertex_ai/gemini-2.5-flash", "vertex_location": "us-central1"})}, + compat("groq", "groq/llama-3.3-70b-versatile"), + compat("mistral", "mistral/mistral-large-latest"), + compat("deepseek", "deepseek/deepseek-chat"), + compat("together", "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo"), + compat("fireworks", "fireworks_ai/accounts/fireworks/models/llama-v3p3-70b-instruct"), + compat("xai", "xai/grok-3"), + compat("openrouter", "openrouter/openai/gpt-4o"), + compat("cerebras", "cerebras/llama3.1-8b"), + compat("perplexity", "perplexity/sonar"), + compat("hosted-vllm", "hosted_vllm/Qwen/Qwen3-8B"), + compat("nvidia-nim", "nvidia_nim/meta/llama-3.3-70b-instruct"), + } +} + +// scenario is one shape of chat request, sent to every deployment whose family it lists. +type scenario struct { + name string + families string // space-separated families, or "all" + streams []bool // which of non-streaming and streaming to send + body func(mock string) map[string]any +} + +var both = []bool{false, true} + +func user(text string) map[string]any { return map[string]any{"role": "user", "content": text} } + +var weatherTool = map[string]any{"type": "function", "function": map[string]any{ + "name": "get_weather", "description": "Current weather for a city", + "parameters": map[string]any{"type": "object", "properties": map[string]any{"city": map[string]any{"type": "string"}}, + "required": []any{"city"}}, +}} + +const pngDataURL = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==" + +var scenarios = []scenario{ + {"basic", "all", both, func(string) map[string]any { + return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, "temperature": 0.2} + }}, + {"include-usage", "all", []bool{true}, func(string) map[string]any { + return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, + "stream_options": map[string]any{"include_usage": true}} + }}, + {"system-multi-turn", "all", both, func(string) map[string]any { + return map[string]any{"messages": []any{ + map[string]any{"role": "system", "content": "You are terse."}, + user("Hi."), + map[string]any{"role": "assistant", "content": "Hello."}, + user("Say hello again."), + }, "max_tokens": 50, "top_p": 0.9, "stop": []any{"END"}, "user": "parity-user"} + }}, + {"tools", "all", both, func(string) map[string]any { + return map[string]any{"messages": []any{user("Weather in Paris? [[parity reply=tool]]")}, + "tools": []any{weatherTool}, "tool_choice": "auto", "max_tokens": 100} + }}, + {"tools-include-usage", "all", []bool{true}, func(string) map[string]any { + return map[string]any{"messages": []any{user("Weather in Paris? [[parity reply=tool]]")}, + "tools": []any{weatherTool}, "tool_choice": "required", "max_tokens": 100, + "stream_options": map[string]any{"include_usage": true}} + }}, + {"tool-result", "all", both, func(string) map[string]any { + return map[string]any{"messages": []any{ + user("Weather in Paris?"), + map[string]any{"role": "assistant", "content": nil, "tool_calls": []any{map[string]any{"id": "call_prev_1", + "type": "function", "function": map[string]any{"name": "get_weather", "arguments": `{"city": "Paris"}`}}}}, + map[string]any{"role": "tool", "tool_call_id": "call_prev_1", "content": "18C and sunny"}, + }, "tools": []any{weatherTool}, "max_tokens": 100} + }}, + {"json-schema", "all", both, func(string) map[string]any { + return map[string]any{"messages": []any{user("Answer as JSON. [[parity reply=json]]")}, "max_tokens": 100, + "response_format": map[string]any{"type": "json_schema", "json_schema": map[string]any{"name": "answer", "strict": true, + "schema": map[string]any{"type": "object", "properties": map[string]any{"answer": map[string]any{"type": "integer"}}, + "required": []any{"answer"}, "additionalProperties": false}}}} + }}, + {"json-object", "openai gemini compat", []bool{false}, func(string) map[string]any { + return map[string]any{"messages": []any{user("Answer as JSON. [[parity reply=json]]")}, "max_tokens": 100, + "response_format": map[string]any{"type": "json_object"}} + }}, + {"image-data-url", "openai claude gemini", both, func(string) map[string]any { + return map[string]any{"messages": []any{map[string]any{"role": "user", "content": []any{ + map[string]any{"type": "text", "text": "What is in this image?"}, + map[string]any{"type": "image_url", "image_url": map[string]any{"url": pngDataURL}}, + }}}, "max_tokens": 50} + }}, + {"image-remote-url", "openai claude gemini", []bool{false}, func(mock string) map[string]any { + return map[string]any{"messages": []any{map[string]any{"role": "user", "content": []any{ + map[string]any{"type": "text", "text": "What is in this image?"}, + map[string]any{"type": "image_url", "image_url": map[string]any{"url": mock + "/img.png"}}, + }}}, "max_tokens": 50} + }}, + {"reasoning-effort", "claude gemini", both, func(string) map[string]any { + return map[string]any{"messages": []any{user("Think, then say hello. [[parity reply=thinking]]")}, + "reasoning_effort": "low", "max_tokens": 4096} + }}, + {"cache-control", "claude", []bool{false}, func(string) map[string]any { + return map[string]any{"messages": []any{ + map[string]any{"role": "system", "content": []any{map[string]any{"type": "text", "text": "Long shared prefix.", + "cache_control": map[string]any{"type": "ephemeral"}}}}, + user("Say hello."), + }, "max_tokens": 50} + }}, + {"unsupported-param", "all", []bool{false}, func(string) map[string]any { + return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, "n": 2, "logit_bias": map[string]any{"50256": -100}} + }}, + {"drop-params", "all", []bool{false}, func(string) map[string]any { + return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, "n": 2, + "logit_bias": map[string]any{"50256": -100}, "drop_params": true} + }}, + {"max-completion-tokens", "all", []bool{false}, func(string) map[string]any { + return map[string]any{"messages": []any{user("Say hello.")}, "max_completion_tokens": 60, "seed": 7} + }}, +} + +// testCase is one request, sent once to each proxy. +type testCase struct { + Name string `json:"name"` + Model string `json:"model"` + Stream bool `json:"stream"` + Body map[string]any `json:"body"` +} + +func buildCases(deps []deployment, mockURL, filter string) []testCase { + var cases []testCase + for _, d := range deps { + for _, s := range scenarios { + if s.families != "all" && !strings.Contains(" "+s.families+" ", " "+d.family+" ") { + continue + } + for _, stream := range s.streams { + body := s.body(mockURL) + body["model"] = d.name + mode := "unary" + if stream { + body["stream"] = true + mode = "stream" + } + name := fmt.Sprintf("%s/%s/%s", d.name, s.name, mode) + if filter != "" && !strings.Contains(name, filter) { + continue + } + cases = append(cases, testCase{name, d.name, stream, body}) + } + } + } + return cases +} diff --git a/litellm-rust/loadtest/cmd/chatparity/chatparity_test.go b/litellm-rust/loadtest/cmd/chatparity/chatparity_test.go new file mode 100644 index 00000000000..ba260bac16a --- /dev/null +++ b/litellm-rust/loadtest/cmd/chatparity/chatparity_test.go @@ -0,0 +1,112 @@ +package main + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func decode(t *testing.T, text string) any { + t.Helper() + var v any + if err := json.Unmarshal([]byte(text), &v); err != nil { + t.Fatal(err) + } + return v +} + +func TestDiffValuesReportsMissingAndChangedPaths(t *testing.T) { + var diffs []difference + diffValues("response", "", decode(t, `{"a": 1, "b": [1, 2], "c": {"d": "x"}}`), + decode(t, `{"a": 1, "b": [1], "c": {"d": "y"}, "e": true}`), &diffs) + got := map[string]bool{} + for _, d := range diffs { + got[d.Path] = true + } + for _, want := range []string{".b[1]", ".c.d", ".e"} { + if !got[want] { + t.Errorf("missing %s in %+v", want, diffs) + } + } + if len(diffs) != 3 { + t.Errorf("want 3 differences, got %+v", diffs) + } +} + +func TestNormalizeIDsKeepsMockIDsAndPrefixesOfGeneratedOnes(t *testing.T) { + got := normalizeIDs(decode(t, `{"id": "chatcmpl-123", "created": 5, "choices": [{"message": {"tool_calls": [{"id": "call_parity_1"}, {"id": "call_abc"}]}}]}`)) + want := decode(t, `{"id": "chatcmpl-*", "created": "", "choices": [{"message": {"tool_calls": [{"id": "call_parity_1"}, {"id": "call_*"}]}}]}`) + if !equal(got, want) { + t.Errorf("got %v", got) + } +} + +func TestAssembleFoldsContentToolCallsAndUsage(t *testing.T) { + s := parseSSE([]byte(strings.Join([]string{ + `data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"role":"assistant","content":"Hel"}}]}`, + `data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"content":"lo"}}]}`, + `data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_parity_1","type":"function","function":{"name":"f","arguments":"{\"a\""}}]}}]}`, + `data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":": 1}"}}]},"finish_reason":"tool_calls"}]}`, + `data: {"id":"chatcmpl-1","model":"m","choices":[],"usage":{"total_tokens":3}}`, + `data: [DONE]`, + }, "\n\n"))) + got := assemble(s) + if got["content"] != "Hello" || got["done"] != true || got["one_id_per_stream"] != true { + t.Errorf("got %v", got) + } + calls := got["tool_calls"].([]any) + if len(calls) != 1 || calls[0].(map[string]any)["arguments"] != `{"a": 1}` { + t.Errorf("tool calls %v", calls) + } + if !equal(got["finish_reasons"], []any{"tool_calls"}) || !equal(got["usage"], map[string]any{"total_tokens": 3.0}) { + t.Errorf("got %v", got) + } +} + +func TestNormalizeUpstreamDropsTransportHeadersAndTheSignature(t *testing.T) { + got := normalizeUpstream([]upstream{{Method: "POST", Path: "/p", Headers: map[string]string{ + "user-agent": "x", + "x-amz-date": "20261019T000000Z", + "authorization": "AWS4-HMAC-SHA256 Credential=AKIA/20261019/us-east-1/bedrock/aws4_request, SignedHeaders=host, Signature=abc", + "anthropic-version": "2023-06-01", + }}}) + headers := got.([]any)[0].(map[string]any)["headers"] + want := map[string]any{"authorization": "AWS4-HMAC-SHA256 region=us-east-1 service=bedrock", "anthropic-version": "2023-06-01"} + if !equal(headers, want) { + t.Errorf("got %v", headers) + } +} + +func TestMarkKnownMatchesCaseWhereAndPathGlobs(t *testing.T) { + diffs := []difference{{Where: "upstream", Path: "[0].headers.accept"}, {Where: "response", Path: ".body.model"}} + markKnown("openai/basic/unary", diffs, []known{{Case: "*/basic/*", Where: "upstream", Path: "*.accept", Reason: "r"}}) + if diffs[0].Known != "r" || diffs[1].Known != "" { + t.Errorf("got %+v", diffs) + } +} + +func TestMockAnswersAnthropicJSONModeThroughTheForcedTool(t *testing.T) { + m := &mock{} + body := `{"model":"c","tools":[{"name":"json_tool_call"}],"messages":[{"role":"user","content":"x"}]}` + w := httptest.NewRecorder() + m.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/messages", strings.NewReader(body))) + got := decode(t, w.Body.String()) + if dig(got, "content", 0, "name") != "json_tool_call" || dig(got, "content", 0, "input", "answer") != 42.0 { + t.Errorf("got %v", got) + } + if recorded := m.take(); len(recorded) != 1 || recorded[0].Path != "/v1/messages" { + t.Errorf("recorded %+v", recorded) + } +} + +func TestEveryCaseNameIsUnique(t *testing.T) { + seen := map[string]bool{} + for _, c := range buildCases(deployments(""), "http://mock", "") { + if seen[c.Name] { + t.Errorf("duplicate case %s", c.Name) + } + seen[c.Name] = true + } +} diff --git a/litellm-rust/loadtest/cmd/chatparity/diff.go b/litellm-rust/loadtest/cmd/chatparity/diff.go new file mode 100644 index 00000000000..6e86bf98c8e --- /dev/null +++ b/litellm-rust/loadtest/cmd/chatparity/diff.go @@ -0,0 +1,339 @@ +package main + +import ( + "bufio" + "bytes" + "encoding/json" + "fmt" + "regexp" + "sort" + "strings" +) + +// difference is one place where the two proxies disagree. Where is "upstream" (what the +// provider was sent), "response" (what the client got, or the assembled stream) or "chunks" +// (the stream chunk by chunk, informational: boundaries may differ for the same content). +type difference struct { + Where string `json:"where"` + Path string `json:"path"` + Python any `json:"python"` + Rust any `json:"rust"` + Known string `json:"known,omitempty"` // the reason, when the difference is listed as known +} + +const absent = "" + +// diffValues appends every path where a and b differ. +func diffValues(where, path string, a, b any, out *[]difference) { + switch av := a.(type) { + case map[string]any: + bv, ok := b.(map[string]any) + if !ok { + break + } + keys := map[string]bool{} + for k := range av { + keys[k] = true + } + for k := range bv { + keys[k] = true + } + for _, k := range sortedKeys(keys) { + x, xok := av[k] + y, yok := bv[k] + switch { + case !xok: + *out = append(*out, difference{Where: where, Path: path + "." + k, Python: absent, Rust: y}) + case !yok: + *out = append(*out, difference{Where: where, Path: path + "." + k, Python: x, Rust: absent}) + default: + diffValues(where, path+"."+k, x, y, out) + } + } + return + case []any: + bv, ok := b.([]any) + if !ok { + break + } + for i := 0; i < len(av) || i < len(bv); i++ { + p := fmt.Sprintf("%s[%d]", path, i) + switch { + case i >= len(av): + *out = append(*out, difference{Where: where, Path: p, Python: absent, Rust: bv[i]}) + case i >= len(bv): + *out = append(*out, difference{Where: where, Path: p, Python: av[i], Rust: absent}) + default: + diffValues(where, p, av[i], bv[i], out) + } + } + return + } + if !equal(a, b) { + *out = append(*out, difference{Where: where, Path: path, Python: a, Rust: b}) + } +} + +func equal(a, b any) bool { + x, _ := json.Marshal(a) + y, _ := json.Marshal(b) + return bytes.Equal(x, y) +} + +// Headers that differ for reasons unrelated to translation: client libraries, transport, +// tracing, and request signing that depends on the clock. +var ignoredHeaders = regexp.MustCompile(`^(user-agent|content-length|accept-encoding|connection|host|traceparent|tracestate|` + + `x-stainless-.*|amz-sdk-.*|x-amzn-trace-id|x-amz-date|x-amz-content-sha256|x-goog-api-client|x-litellm-.*)$`) + +var sigV4 = regexp.MustCompile(`^AWS4-HMAC-SHA256 Credential=[^/]+/\d{8}/([^/]+)/([^/]+)/aws4_request`) + +func normalizeUpstream(requests []upstream) any { + out := make([]any, len(requests)) + for i, r := range requests { + headers := map[string]any{} + for name, value := range r.Headers { + if ignoredHeaders.MatchString(name) { + continue + } + if name == "authorization" { + if m := sigV4.FindStringSubmatch(value); m != nil { + value = fmt.Sprintf("AWS4-HMAC-SHA256 region=%s service=%s", m[1], m[2]) + } + } + headers[name] = value + } + entry := map[string]any{"method": r.Method, "path": r.Path, "headers": headers, "body": r.Body} + if r.Query != "" { + entry["query"] = r.Query + } + out[i] = entry + } + return out +} + +// normalizeIDs replaces ids and timestamps the proxies generate with a placeholder that keeps +// their prefix (`chatcmpl-*`, `call_*`), so only their shape is compared. Ids the mock chose +// contain "parity" and are compared exactly. +func normalizeIDs(v any) any { + switch t := v.(type) { + case map[string]any: + out := map[string]any{} + for k, x := range t { + switch { + case k == "created": + out[k] = "" + case k == "id": + out[k] = generated(x) + default: + out[k] = normalizeIDs(x) + } + } + return out + case []any: + out := make([]any, len(t)) + for i, x := range t { + out[i] = normalizeIDs(x) + } + return out + } + return v +} + +func generated(v any) any { + s, ok := v.(string) + if !ok || strings.Contains(s, "parity") { + return v + } + if i := strings.IndexAny(s, "-_"); i >= 0 { + return s[:i+1] + "*" + } + return "*" +} + +// stream is a parsed SSE response. +type stream struct { + Chunks []any `json:"chunks"` + Done bool `json:"done"` + Other []any `json:"other,omitempty"` // lines that are neither JSON data nor [DONE] +} + +func parseSSE(body []byte) stream { + var s stream + scanner := bufio.NewScanner(bytes.NewReader(body)) + scanner.Buffer(make([]byte, 1<<20), 16<<20) + for scanner.Scan() { + line := scanner.Text() + if line == "" || strings.HasPrefix(line, ":") { + continue + } + data, ok := strings.CutPrefix(line, "data:") + if !ok { + s.Other = append(s.Other, line) + continue + } + data = strings.TrimSpace(data) + if data == "[DONE]" { + s.Done = true + continue + } + var chunk any + if json.Unmarshal([]byte(data), &chunk) != nil { + s.Other = append(s.Other, line) + continue + } + s.Chunks = append(s.Chunks, chunk) + } + return s +} + +// assemble folds a stream's chunks into what a client library would build from them, which is +// what must agree; how the content was split into chunks may differ. +func assemble(s stream) map[string]any { + var content, reasoning strings.Builder + type call struct{ id, kind, name, args any } + calls := map[float64]*call{} + var order []float64 + ids := map[string]bool{} + models := map[string]bool{} + var roles, finishes []any + var usage, errorFrame any + extra := map[string]bool{} + sawContent, sawReasoning := false, false + for _, c := range s.Chunks { + chunk, _ := c.(map[string]any) + if e, ok := chunk["error"]; ok { + errorFrame = e + continue + } + if id, ok := chunk["id"].(string); ok { + ids[id] = true + } + if m, ok := chunk["model"].(string); ok { + models[m] = true + } + if u, ok := chunk["usage"]; ok && u != nil { + usage = u + } + for k := range chunk { + switch k { + case "id", "object", "created", "model", "choices", "usage", "system_fingerprint": + default: + extra[k] = true + } + } + choices, _ := chunk["choices"].([]any) + for _, ch := range choices { + choice, _ := ch.(map[string]any) + if f := choice["finish_reason"]; f != nil { + finishes = append(finishes, f) + } + delta, _ := choice["delta"].(map[string]any) + for k, v := range delta { + switch k { + case "role": + if v != nil { + roles = append(roles, v) + } + case "content": + if t, ok := v.(string); ok { + content.WriteString(t) + sawContent = true + } + case "reasoning_content": + if t, ok := v.(string); ok { + reasoning.WriteString(t) + sawReasoning = true + } + case "tool_calls": + list, _ := v.([]any) + for _, item := range list { + tc, _ := item.(map[string]any) + index, _ := tc["index"].(float64) + entry := calls[index] + if entry == nil { + entry = &call{} + calls[index] = entry + order = append(order, index) + } + if x := tc["id"]; x != nil { + entry.id = generated(x) + } + if x := tc["type"]; x != nil { + entry.kind = x + } + fn, _ := tc["function"].(map[string]any) + if x := fn["name"]; x != nil { + entry.name = x + } + if x, ok := fn["arguments"].(string); ok { + prev, _ := entry.args.(string) + entry.args = prev + x + } + } + default: + if v != nil { + extra["delta."+k] = true + } + } + } + } + } + sort.Float64s(order) + var toolCalls []any + for _, i := range order { + c := calls[i] + toolCalls = append(toolCalls, map[string]any{"index": i, "id": c.id, "type": c.kind, "name": c.name, "arguments": c.args}) + } + out := map[string]any{ + "one_id_per_stream": len(ids) <= 1, + "id_prefix": nil, + "models": sortedKeys(models), + "roles": roles, + "finish_reasons": finishes, + "usage": usage, + "done": s.Done, + "tool_calls": toolCalls, + "other_fields": sortedKeys(extra), + "non_data_lines": s.Other, + } + for id := range ids { + out["id_prefix"] = generated(id) + } + if sawContent { + out["content"] = content.String() + } + if sawReasoning { + out["reasoning_content"] = reasoning.String() + } + if errorFrame != nil { + out["error"] = errorFrame + } + return out +} + +// known is a difference the team has looked at and accepted, with the reason. +type known struct { + Case string `json:"case"` // glob over case names; * matches anything + Where string `json:"where"` // upstream, response or chunks; empty matches all + Path string `json:"path"` // glob over the difference's path + Reason string `json:"reason"` +} + +func glob(pattern string) *regexp.Regexp { + parts := strings.Split(pattern, "*") + for i, p := range parts { + parts[i] = regexp.QuoteMeta(p) + } + return regexp.MustCompile("^" + strings.Join(parts, ".*") + "$") +} + +func markKnown(name string, diffs []difference, list []known) { + for i := range diffs { + for _, k := range list { + if (k.Where == "" || k.Where == diffs[i].Where) && glob(k.Case).MatchString(name) && glob(k.Path).MatchString(diffs[i].Path) { + diffs[i].Known = k.Reason + break + } + } + } +} diff --git a/litellm-rust/loadtest/cmd/chatparity/known.json b/litellm-rust/loadtest/cmd/chatparity/known.json new file mode 100644 index 00000000000..fe51488c706 --- /dev/null +++ b/litellm-rust/loadtest/cmd/chatparity/known.json @@ -0,0 +1 @@ +[] diff --git a/litellm-rust/loadtest/cmd/chatparity/main.go b/litellm-rust/loadtest/cmd/chatparity/main.go new file mode 100644 index 00000000000..e26468dac46 --- /dev/null +++ b/litellm-rust/loadtest/cmd/chatparity/main.go @@ -0,0 +1,289 @@ +// chatparity sends the same chat completions requests through the Python proxy and the Rust +// gateway, both pointed at one recording mock provider, and reports every difference in what the +// provider was sent and what the client got back. +// +// chatparity config -mock http://127.0.0.1:9100 -out out # config.yaml both proxies load +// chatparity run -mock-addr 127.0.0.1:9100 -python http://127.0.0.1:4001 \ +// -rust http://127.0.0.1:4002 -out out [-known known.json] [-filter anthropic/] +// +// The mock answers each provider's API (OpenAI-compatible, Azure, Anthropic, Bedrock Converse, +// Gemini, Vertex) the same way for both proxies, so differences in the client's answer come from +// the proxies' translation. scripts/chat-parity.sh builds and starts everything locally. +package main + +import ( + "bytes" + "crypto/rand" + "crypto/rsa" + "crypto/x509" + "encoding/json" + "encoding/pem" + "flag" + "fmt" + "io" + "log" + "net" + "net/http" + "net/url" + "os" + "path/filepath" + "strings" + "time" +) + +const masterKey = "sk-parity-master" + +func main() { + log.SetFlags(0) + if len(os.Args) < 2 { + log.Fatal("usage: chatparity config|run ...") + } + switch os.Args[1] { + case "config": + configCmd(os.Args[2:]) + case "run": + runCmd(os.Args[2:]) + default: + log.Fatalf("unknown subcommand %q", os.Args[1]) + } +} + +// configCmd writes the one config both proxies load: every deployment pointed at the mock, and +// a throwaway service account whose token endpoint is the mock, for the Vertex deployments. +func configCmd(args []string) { + fs := flag.NewFlagSet("config", flag.ExitOnError) + mockURL := fs.String("mock", "http://127.0.0.1:9100", "URL the proxies reach the mock at") + out := fs.String("out", "out", "directory to write config.yaml to") + fs.Parse(args) + must(os.MkdirAll(*out, 0o755)) + var models []any + for _, d := range deployments(serviceAccount(*mockURL)) { + params := map[string]any{} + for k, v := range d.params { + if s, ok := v.(string); ok { + v = strings.ReplaceAll(s, "{mock}", *mockURL) + } + params[k] = v + } + models = append(models, map[string]any{"model_name": d.name, "litellm_params": params}) + } + // JSON is YAML, and both proxies read it as their config.yaml. + config := map[string]any{ + "model_list": models, + "general_settings": map[string]any{"master_key": masterKey}, + "litellm_settings": map[string]any{"num_retries": 0, "user_url_allowed_hosts": []any{mockHost(*mockURL)}}, + } + b, _ := json.MarshalIndent(config, "", " ") + must(os.WriteFile(filepath.Join(*out, "config.yaml"), b, 0o644)) + fmt.Println(filepath.Join(*out, "config.yaml")) +} + +// mockHost is the mock's host:port, allowed as a media URL host so remote image cases reach it. +func mockHost(mockURL string) string { + u, err := url.Parse(mockURL) + must(err) + return u.Host +} + +func serviceAccount(mockURL string) string { + key, err := rsa.GenerateKey(rand.Reader, 2048) + must(err) + der, err := x509.MarshalPKCS8PrivateKey(key) + must(err) + b, _ := json.Marshal(map[string]any{ + "type": "service_account", "project_id": "parity-project", "private_key_id": "parity", + "private_key": string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der})), + "client_email": "parity@parity-project.iam.gserviceaccount.com", "client_id": "1", + "token_uri": mockURL + "/token", + }) + return string(b) +} + +// exchange is one side's request and what came of it. +type exchange struct { + Status int `json:"status"` + ContentType string `json:"content_type"` + Body any `json:"body"` + Upstream []upstream `json:"upstream"` + Error string `json:"error,omitempty"` +} + +type outcome struct { + Case string `json:"case"` + Python exchange `json:"python"` + Rust exchange `json:"rust"` + Differences []difference `json:"differences"` +} + +func runCmd(args []string) { + fs := flag.NewFlagSet("run", flag.ExitOnError) + mockAddr := fs.String("mock-addr", "127.0.0.1:9100", "address the mock listens on") + mockURL := fs.String("mock", "", "URL the proxies reach the mock at (default http://)") + python := fs.String("python", "http://127.0.0.1:4001", "Python proxy base URL") + rust := fs.String("rust", "http://127.0.0.1:4002", "Rust gateway base URL") + out := fs.String("out", "out", "directory for report.md, results.json and per-case captures") + knownPath := fs.String("known", "", "JSON list of accepted differences") + filter := fs.String("filter", "", "only run cases whose name contains this") + fs.Parse(args) + if *mockURL == "" { + *mockURL = "http://" + *mockAddr + } + var accepted []known + if *knownPath != "" { + b, err := os.ReadFile(*knownPath) + must(err) + must(json.Unmarshal(b, &accepted)) + } + + m := &mock{} + listener, err := net.Listen("tcp", *mockAddr) + must(err) + go http.Serve(listener, m) + + client := &http.Client{Timeout: 120 * time.Second} + cases := buildCases(deployments(""), *mockURL, *filter) + must(os.MkdirAll(filepath.Join(*out, "cases"), 0o755)) + var outcomes []outcome + for _, c := range cases { + o := outcome{Case: c.Name} + o.Python = send(client, m, *python, c) + o.Rust = send(client, m, *rust, c) + o.Differences = compare(c, o.Python, o.Rust) + markKnown(c.Name, o.Differences, accepted) + outcomes = append(outcomes, o) + b, _ := json.MarshalIndent(map[string]any{"request": c.Body, "outcome": o}, "", " ") + must(os.WriteFile(filepath.Join(*out, "cases", strings.ReplaceAll(c.Name, "/", "__")+".json"), b, 0o644)) + fmt.Printf("%-55s %s\n", c.Name, verdict(o.Differences)) + } + b, _ := json.MarshalIndent(outcomes, "", " ") + must(os.WriteFile(filepath.Join(*out, "results.json"), b, 0o644)) + failing := writeReport(filepath.Join(*out, "report.md"), outcomes) + fmt.Printf("\n%d cases, %d with unexplained differences; report in %s\n", len(outcomes), failing, filepath.Join(*out, "report.md")) + if failing > 0 { + os.Exit(1) + } +} + +func send(client *http.Client, m *mock, base string, c testCase) exchange { + m.take() + body, _ := json.Marshal(c.Body) + req, _ := http.NewRequest(http.MethodPost, strings.TrimRight(base, "/")+"/v1/chat/completions", bytes.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Authorization", "Bearer "+masterKey) + resp, err := client.Do(req) + if err != nil { + return exchange{Error: err.Error(), Upstream: m.take()} + } + defer resp.Body.Close() + raw, _ := io.ReadAll(resp.Body) + x := exchange{Status: resp.StatusCode, ContentType: resp.Header.Get("Content-Type"), Upstream: m.take()} + if strings.HasPrefix(x.ContentType, "text/event-stream") { + x.Body = parseSSE(raw) + } else { + var parsed any + if json.Unmarshal(raw, &parsed) == nil { + x.Body = parsed + } else { + x.Body = string(raw) + } + } + return x +} + +func compare(c testCase, py, rs exchange) []difference { + var diffs []difference + diffValues("upstream", "", normalizeUpstream(py.Upstream), normalizeUpstream(rs.Upstream), &diffs) + top := func(x exchange) map[string]any { + return map[string]any{"status": x.Status, "content_type": mediaType(x.ContentType), "transport_error": x.Error} + } + diffValues("response", "", top(py), top(rs), &diffs) + ps, pok := py.Body.(stream) + rsm, rok := rs.Body.(stream) + switch { + case pok && rok: + diffValues("response", ".stream", assemble(ps), assemble(rsm), &diffs) + diffValues("chunks", "", normalizeIDs(ps.Chunks), normalizeIDs(rsm.Chunks), &diffs) + case !pok && !rok: + diffValues("response", ".body", normalizeIDs(py.Body), normalizeIDs(rs.Body), &diffs) + } + return diffs +} + +func mediaType(contentType string) string { + media, _, _ := strings.Cut(contentType, ";") + return strings.TrimSpace(media) +} + +func counts(diffs []difference) (open, accepted, chunks int) { + for _, d := range diffs { + switch { + case d.Where == "chunks": + chunks++ + case d.Known != "": + accepted++ + default: + open++ + } + } + return +} + +func verdict(diffs []difference) string { + open, accepted, chunks := counts(diffs) + if open == 0 { + return fmt.Sprintf("ok (%d known, %d chunk-level)", accepted, chunks) + } + return fmt.Sprintf("DIFF %d (%d known, %d chunk-level)", open, accepted, chunks) +} + +// writeReport writes a summary table and every case's open differences, and returns how many +// cases have any. +func writeReport(path string, outcomes []outcome) int { + var b strings.Builder + failing := 0 + b.WriteString("# Chat completions parity: Python proxy vs Rust gateway\n\n") + b.WriteString("Open differences fail the run. Known ones are listed in the known-differences file with a reason. ") + b.WriteString("Chunk-level differences compare streams chunk by chunk and are informational: the assembled stream is what must agree.\n\n") + b.WriteString("| case | python | rust | open | known | chunk-level |\n|---|---|---|---|---|---|\n") + for _, o := range outcomes { + open, accepted, chunks := counts(o.Differences) + if open > 0 { + failing++ + } + fmt.Fprintf(&b, "| %s | %d | %d | %d | %d | %d |\n", o.Case, o.Python.Status, o.Rust.Status, open, accepted, chunks) + } + b.WriteString("\n## Open differences\n") + for _, o := range outcomes { + var lines []string + for _, d := range o.Differences { + if d.Where == "chunks" || d.Known != "" { + continue + } + lines = append(lines, fmt.Sprintf("| %s | `%s` | %s | %s |", d.Where, d.Path, cell(d.Python), cell(d.Rust))) + } + if len(lines) == 0 { + continue + } + fmt.Fprintf(&b, "\n### %s\n\n| where | path | python | rust |\n|---|---|---|---|\n%s\n", o.Case, strings.Join(lines, "\n")) + } + must(os.WriteFile(path, []byte(b.String()), 0o644)) + return failing +} + +func cell(v any) string { + s, ok := v.(string) + if !ok { + raw, _ := json.Marshal(v) + s = string(raw) + } + if len(s) > 300 { + s = s[:300] + "…" + } + return "`" + strings.NewReplacer("|", "\\|", "\n", " ", "`", "'").Replace(s) + "`" +} + +func must(err error) { + if err != nil { + log.Fatal(err) + } +} diff --git a/litellm-rust/loadtest/cmd/chatparity/mock.go b/litellm-rust/loadtest/cmd/chatparity/mock.go new file mode 100644 index 00000000000..9fd2287d4c7 --- /dev/null +++ b/litellm-rust/loadtest/cmd/chatparity/mock.go @@ -0,0 +1,461 @@ +package main + +import ( + "bytes" + "encoding/base64" + "encoding/binary" + "encoding/json" + "hash/crc32" + "io" + "net/http" + "regexp" + "sort" + "strings" + "sync" +) + +// upstream is one request a proxy sent to the mock provider. +type upstream struct { + Method string `json:"method"` + Path string `json:"path"` + Query string `json:"query,omitempty"` + Headers map[string]string `json:"headers"` + Body any `json:"body,omitempty"` +} + +// mock answers as each provider would, the same way for both proxies, and records what it was +// sent. Cases run one at a time, so everything that arrives belongs to the current slot. +type mock struct { + mu sync.Mutex + current []upstream +} + +func (m *mock) take() []upstream { + m.mu.Lock() + defer m.mu.Unlock() + got := m.current + m.current = nil + return got +} + +func (m *mock) record(r *http.Request, body []byte) { + headers := map[string]string{} + for name, values := range r.Header { + headers[strings.ToLower(name)] = strings.Join(values, ", ") + } + var parsed any + if len(body) > 0 && json.Unmarshal(body, &parsed) != nil { + parsed = string(body) + } + m.mu.Lock() + m.current = append(m.current, upstream{r.Method, r.URL.Path, r.URL.RawQuery, headers, parsed}) + m.mu.Unlock() +} + +// The fixed answer parts. Everything the mock returns is deterministic, so any difference in +// what the clients get comes from the proxies. +const ( + answerText = "Hello from the parity mock." + reasoningText = "The user wants a greeting." + signature = "sig_parity" + toolID = "call_parity_1" + toolArgs = `{"city": "Paris"}` + jsonAnswer = `{"answer": 42}` + inputTokens = 21 + outputTokens = 9 + cacheRead = 3 + cacheWrite = 2 + reasoningCount = 4 +) + +// pixel is a 1x1 PNG, served at /img.png for remote image cases. +var pixel, _ = base64.StdEncoding.DecodeString("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==") + +var directive = regexp.MustCompile(`\[\[parity reply=(\w+)\]\]`) + +// reply is what a case asks the provider to answer with: text, tool, thinking or json. +type reply struct { + kind string + tool string // the tool to call: the request's first tool, or the proxy's JSON-mode tool +} + +func replyFor(body []byte, firstTool string) reply { + kind := "text" + if m := directive.FindSubmatch(body); m != nil { + kind = string(m[1]) + } + // Anthropic-style JSON mode turns response_format into a forced tool; answer through it. + if bytes.Contains(body, []byte(`"json_tool_call"`)) { + return reply{kind: "tool", tool: "json_tool_call"} + } + return reply{kind: kind, tool: firstTool} +} + +func (r reply) text() string { + if r.kind == "json" { + return jsonAnswer + } + return answerText +} + +func (m *mock) ServeHTTP(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + path := r.URL.Path + switch { + case path == "/img.png": + w.Header().Set("Content-Type", "image/png") + w.Write(pixel) + return + case path == "/token": + writeJSON(w, map[string]any{"access_token": "ya29.parity", "expires_in": 3600, "token_type": "Bearer"}) + return + } + m.record(r, body) + var req map[string]any + json.Unmarshal(body, &req) + stream, _ := req["stream"].(bool) + switch { + case strings.HasSuffix(path, "/chat/completions"): + openAI(w, req, replyFor(body, str(dig(req, "tools", 0, "function", "name"))), stream) + case strings.HasSuffix(path, "/messages"), strings.HasSuffix(path, ":rawPredict"): + anthropic(w, replyFor(body, str(dig(req, "tools", 0, "name"))), stream) + case strings.HasSuffix(path, ":streamRawPredict"): + anthropic(w, replyFor(body, str(dig(req, "tools", 0, "name"))), true) + case strings.HasSuffix(path, "/converse"): + bedrock(w, replyFor(body, str(dig(req, "toolConfig", "tools", 0, "toolSpec", "name"))), false) + case strings.HasSuffix(path, "/converse-stream"): + bedrock(w, replyFor(body, str(dig(req, "toolConfig", "tools", 0, "toolSpec", "name"))), true) + case strings.HasSuffix(path, ":generateContent"): + gemini(w, replyFor(body, str(dig(req, "tools", 0, "functionDeclarations", 0, "name"))), false) + case strings.HasSuffix(path, ":streamGenerateContent"): + gemini(w, replyFor(body, str(dig(req, "tools", 0, "functionDeclarations", 0, "name"))), true) + default: + w.WriteHeader(http.StatusNotFound) + writeJSON(w, map[string]any{"error": map[string]any{"message": "parity mock: no route for " + path}}) + } +} + +func openAI(w http.ResponseWriter, req map[string]any, rep reply, stream bool) { + model, _ := req["model"].(string) + usage := map[string]any{ + "prompt_tokens": inputTokens, "completion_tokens": outputTokens, "total_tokens": inputTokens + outputTokens, + "prompt_tokens_details": map[string]any{"cached_tokens": cacheRead}, + } + message := map[string]any{"role": "assistant", "content": rep.text()} + finish := "stop" + if rep.kind == "tool" { + message["content"] = nil + message["tool_calls"] = []any{map[string]any{"id": toolID, "type": "function", + "function": map[string]any{"name": rep.tool, "arguments": toolArgs}}} + finish = "tool_calls" + } + if rep.kind == "thinking" { + message["reasoning_content"] = reasoningText + } + if !stream { + writeJSON(w, map[string]any{"id": "chatcmpl-parity", "object": "chat.completion", "created": 1700000000, + "model": model, "choices": []any{map[string]any{"index": 0, "message": message, "finish_reason": finish}}, + "usage": usage}) + return + } + s := newSSE(w, "text/event-stream") + chunk := func(delta map[string]any, finish any) map[string]any { + return map[string]any{"id": "chatcmpl-parity", "object": "chat.completion.chunk", "created": 1700000000, + "model": model, "choices": []any{map[string]any{"index": 0, "delta": delta, "finish_reason": finish}}} + } + s.data(chunk(map[string]any{"role": "assistant", "content": ""}, nil)) + if rep.kind == "thinking" { + s.data(chunk(map[string]any{"reasoning_content": reasoningText}, nil)) + } + if rep.kind == "tool" { + s.data(chunk(map[string]any{"tool_calls": []any{map[string]any{"index": 0, "id": toolID, "type": "function", + "function": map[string]any{"name": rep.tool, "arguments": ""}}}}, nil)) + for _, part := range halves(toolArgs) { + s.data(chunk(map[string]any{"tool_calls": []any{map[string]any{"index": 0, + "function": map[string]any{"arguments": part}}}}, nil)) + } + } else { + for _, part := range halves(rep.text()) { + s.data(chunk(map[string]any{"content": part}, nil)) + } + } + s.data(chunk(map[string]any{}, finish)) + if b, _ := dig(req, "stream_options", "include_usage").(bool); b { + last := chunk(nil, nil) + last["choices"] = []any{} + last["usage"] = usage + s.data(last) + } + s.raw("data: [DONE]\n\n") +} + +func anthropic(w http.ResponseWriter, rep reply, stream bool) { + var blocks []map[string]any + stop := "end_turn" + if rep.kind == "thinking" { + blocks = append(blocks, map[string]any{"type": "thinking", "thinking": reasoningText, "signature": signature}) + } + if rep.kind == "tool" { + var input any + json.Unmarshal([]byte(toolArgs), &input) + if rep.tool == "json_tool_call" { + json.Unmarshal([]byte(jsonAnswer), &input) + } + blocks = append(blocks, map[string]any{"type": "tool_use", "id": toolID, "name": rep.tool, "input": input}) + stop = "tool_use" + } else { + blocks = append(blocks, map[string]any{"type": "text", "text": rep.text()}) + } + usage := map[string]any{"input_tokens": inputTokens, "output_tokens": outputTokens, + "cache_read_input_tokens": cacheRead, "cache_creation_input_tokens": cacheWrite} + if !stream { + content := make([]any, len(blocks)) + for i, b := range blocks { + content[i] = b + } + writeJSON(w, map[string]any{"id": "msg_parity", "type": "message", "role": "assistant", + "model": "claude-parity", "content": content, "stop_reason": stop, "stop_sequence": nil, "usage": usage}) + return + } + s := newSSE(w, "text/event-stream") + s.event("message_start", map[string]any{"type": "message_start", "message": map[string]any{ + "id": "msg_parity", "type": "message", "role": "assistant", "model": "claude-parity", "content": []any{}, + "stop_reason": nil, "stop_sequence": nil, "usage": map[string]any{"input_tokens": inputTokens, "output_tokens": 1, + "cache_read_input_tokens": cacheRead, "cache_creation_input_tokens": cacheWrite}}}) + for i, b := range blocks { + start := map[string]any{} + for k, v := range b { + start[k] = v + } + var deltas []map[string]any + switch b["type"] { + case "thinking": + start["thinking"], start["signature"] = "", "" + for _, part := range halves(reasoningText) { + deltas = append(deltas, map[string]any{"type": "thinking_delta", "thinking": part}) + } + deltas = append(deltas, map[string]any{"type": "signature_delta", "signature": signature}) + case "tool_use": + start["input"] = map[string]any{} + input, _ := json.Marshal(b["input"]) + for _, part := range halves(string(input)) { + deltas = append(deltas, map[string]any{"type": "input_json_delta", "partial_json": part}) + } + default: + start["text"] = "" + for _, part := range halves(rep.text()) { + deltas = append(deltas, map[string]any{"type": "text_delta", "text": part}) + } + } + s.event("content_block_start", map[string]any{"type": "content_block_start", "index": i, "content_block": start}) + for _, d := range deltas { + s.event("content_block_delta", map[string]any{"type": "content_block_delta", "index": i, "delta": d}) + } + s.event("content_block_stop", map[string]any{"type": "content_block_stop", "index": i}) + } + s.event("message_delta", map[string]any{"type": "message_delta", + "delta": map[string]any{"stop_reason": stop, "stop_sequence": nil}, "usage": map[string]any{"output_tokens": outputTokens}}) + s.event("message_stop", map[string]any{"type": "message_stop"}) +} + +func bedrock(w http.ResponseWriter, rep reply, stream bool) { + stop := "end_turn" + var input any + json.Unmarshal([]byte(toolArgs), &input) + if rep.tool == "json_tool_call" { + json.Unmarshal([]byte(jsonAnswer), &input) + } + usage := map[string]any{"inputTokens": inputTokens, "outputTokens": outputTokens, "totalTokens": inputTokens + outputTokens, + "cacheReadInputTokens": cacheRead, "cacheWriteInputTokens": cacheWrite} + if rep.kind == "tool" { + stop = "tool_use" + } + if !stream { + var content []any + if rep.kind == "thinking" { + content = append(content, map[string]any{"reasoningContent": map[string]any{ + "reasoningText": map[string]any{"text": reasoningText, "signature": signature}}}) + } + if rep.kind == "tool" { + content = append(content, map[string]any{"toolUse": map[string]any{"toolUseId": toolID, "name": rep.tool, "input": input}}) + } else { + content = append(content, map[string]any{"text": rep.text()}) + } + writeJSON(w, map[string]any{"output": map[string]any{"message": map[string]any{"role": "assistant", "content": content}}, + "stopReason": stop, "usage": usage, "metrics": map[string]any{"latencyMs": 1}}) + return + } + w.Header().Set("Content-Type", "application/vnd.amazon.eventstream") + w.WriteHeader(http.StatusOK) + frame := func(event string, payload any) { + b, _ := json.Marshal(payload) + w.Write(eventStreamFrame(event, b)) + if f, ok := w.(http.Flusher); ok { + f.Flush() + } + } + frame("messageStart", map[string]any{"role": "assistant"}) + index := 0 + if rep.kind == "thinking" { + for _, part := range halves(reasoningText) { + frame("contentBlockDelta", map[string]any{"contentBlockIndex": index, + "delta": map[string]any{"reasoningContent": map[string]any{"text": part}}}) + } + frame("contentBlockDelta", map[string]any{"contentBlockIndex": index, + "delta": map[string]any{"reasoningContent": map[string]any{"signature": signature}}}) + frame("contentBlockStop", map[string]any{"contentBlockIndex": index}) + index++ + } + if rep.kind == "tool" { + frame("contentBlockStart", map[string]any{"contentBlockIndex": index, + "start": map[string]any{"toolUse": map[string]any{"toolUseId": toolID, "name": rep.tool}}}) + args, _ := json.Marshal(input) + for _, part := range halves(string(args)) { + frame("contentBlockDelta", map[string]any{"contentBlockIndex": index, + "delta": map[string]any{"toolUse": map[string]any{"input": part}}}) + } + } else { + for _, part := range halves(rep.text()) { + frame("contentBlockDelta", map[string]any{"contentBlockIndex": index, "delta": map[string]any{"text": part}}) + } + } + frame("contentBlockStop", map[string]any{"contentBlockIndex": index}) + frame("messageStop", map[string]any{"stopReason": stop}) + frame("metadata", map[string]any{"usage": usage, "metrics": map[string]any{"latencyMs": 1}}) +} + +func gemini(w http.ResponseWriter, rep reply, stream bool) { + var parts []any + if rep.kind == "thinking" { + parts = append(parts, map[string]any{"text": reasoningText, "thought": true}) + } + if rep.kind == "tool" { + var args any + json.Unmarshal([]byte(toolArgs), &args) + parts = append(parts, map[string]any{"functionCall": map[string]any{"name": rep.tool, "args": args}}) + } else { + parts = append(parts, map[string]any{"text": rep.text()}) + } + usage := map[string]any{"promptTokenCount": inputTokens, "candidatesTokenCount": outputTokens, + "totalTokenCount": inputTokens + outputTokens, "cachedContentTokenCount": cacheRead} + if rep.kind == "thinking" { + usage["thoughtsTokenCount"] = reasoningCount + usage["totalTokenCount"] = inputTokens + outputTokens + reasoningCount + } + candidate := func(parts []any, finish any) map[string]any { + c := map[string]any{"content": map[string]any{"role": "model", "parts": parts}, "index": 0} + if finish != nil { + c["finishReason"] = finish + } + return c + } + if !stream { + writeJSON(w, map[string]any{"candidates": []any{candidate(parts, "STOP")}, "usageMetadata": usage, + "modelVersion": "gemini-parity", "responseId": "resp_parity"}) + return + } + s := newSSE(w, "text/event-stream") + for i, part := range parts { + var finish any + if i == len(parts)-1 { + finish = "STOP" + } + event := map[string]any{"candidates": []any{candidate([]any{part}, finish)}, "modelVersion": "gemini-parity", + "responseId": "resp_parity"} + if finish != nil { + event["usageMetadata"] = usage + } + s.data(event) + } +} + +type sse struct{ w http.ResponseWriter } + +func newSSE(w http.ResponseWriter, contentType string) sse { + w.Header().Set("Content-Type", contentType) + w.WriteHeader(http.StatusOK) + return sse{w} +} + +func (s sse) raw(text string) { + io.WriteString(s.w, text) + if f, ok := s.w.(http.Flusher); ok { + f.Flush() + } +} + +func (s sse) data(v any) { + b, _ := json.Marshal(v) + s.raw("data: " + string(b) + "\n\n") +} + +func (s sse) event(name string, v any) { + b, _ := json.Marshal(v) + s.raw("event: " + name + "\ndata: " + string(b) + "\n\n") +} + +// eventStreamFrame encodes one AWS event-stream message, as mockllm does. +func eventStreamFrame(eventType string, payload []byte) []byte { + var headers bytes.Buffer + for _, h := range [][2]string{{":event-type", eventType}, {":content-type", "application/json"}, {":message-type", "event"}} { + headers.WriteByte(byte(len(h[0]))) + headers.WriteString(h[0]) + headers.WriteByte(7) + binary.Write(&headers, binary.BigEndian, uint16(len(h[1]))) + headers.WriteString(h[1]) + } + total := 12 + headers.Len() + len(payload) + 4 + frame := make([]byte, 0, total) + frame = binary.BigEndian.AppendUint32(frame, uint32(total)) + frame = binary.BigEndian.AppendUint32(frame, uint32(headers.Len())) + frame = binary.BigEndian.AppendUint32(frame, crc32.ChecksumIEEE(frame)) + frame = append(frame, headers.Bytes()...) + frame = append(frame, payload...) + return binary.BigEndian.AppendUint32(frame, crc32.ChecksumIEEE(frame)) +} + +// halves splits text in two, so every stream carries more than one delta per block. +func halves(text string) []string { + mid := len(text) / 2 + return []string{text[:mid], text[mid:]} +} + +func writeJSON(w http.ResponseWriter, v any) { + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(v) +} + +// dig walks maps by key and slices by index, returning nil when anything is missing. +func dig(v any, path ...any) any { + for _, step := range path { + switch s := step.(type) { + case string: + m, ok := v.(map[string]any) + if !ok { + return nil + } + v = m[s] + case int: + a, ok := v.([]any) + if !ok || s >= len(a) { + return nil + } + v = a[s] + } + } + return v +} + +func str(v any) string { + s, _ := v.(string) + return s +} + +func sortedKeys[V any](m map[string]V) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} diff --git a/litellm-rust/loadtest/scripts/chat-parity.sh b/litellm-rust/loadtest/scripts/chat-parity.sh new file mode 100755 index 00000000000..56866d5dab1 --- /dev/null +++ b/litellm-rust/loadtest/scripts/chat-parity.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# Runs the chat completions parity check locally: writes one config pointed at chatparity's mock, +# starts the Python proxy and the Rust gateway on it, sends every case through both and writes +# out/report.md. Exits non-zero when there are unexplained differences. +# +# scripts/chat-parity.sh # every case +# FILTER=anthropic/ scripts/chat-parity.sh +# +# Needs: go, cargo, and a Python environment with the proxy and google-auth installed +# (`uv pip install -e '.[proxy]' google-auth` from the repo root). PYTHON picks the +# interpreter; GATEWAY skips the cargo build. +set -euo pipefail + +here="$(cd "$(dirname "$0")/.." && pwd)" +repo="$(cd "$here/../.." && pwd)" +out="${OUT:-$here/out/chat-parity}" +mock_port="${MOCK_PORT:-9100}" python_port="${PYTHON_PORT:-4001}" rust_port="${RUST_PORT:-4002}" +python="${PYTHON:-python3}" +mkdir -p "$out" + +(cd "$here" && go build -o "$out/chatparity" ./cmd/chatparity) +if [[ -z "${GATEWAY:-}" ]]; then + (cd "$repo/litellm-rust" && cargo build -p litellm-gateway) + GATEWAY="${CARGO_TARGET_DIR:-$repo/litellm-rust/target}/debug/litellm-gateway" +fi +"$out/chatparity" config -mock "http://127.0.0.1:$mock_port" -out "$out" >/dev/null + +# Credentials for the bedrock-env deployment, which takes them from the environment. They only +# ever reach the mock. +export AWS_ACCESS_KEY_ID=AKIAPARITYPARITY AWS_SECRET_ACCESS_KEY=parity-secret AWS_REGION_NAME=us-east-1 AWS_REGION=us-east-1 +unset AWS_SESSION_TOKEN AWS_PROFILE + +pids=() +trap 'kill "${pids[@]}" 2>/dev/null || true' EXIT +# Both price from the repo's cost map, so parameter support and defaults come from the same data. +LITELLM_LOCAL_MODEL_COST_MAP=True "$python" -m litellm.proxy.proxy_cli --config "$out/config.yaml" \ + --port "$python_port" >"$out/python.log" 2>&1 & +pids+=($!) +LITELLM_MODEL_COST_MAP_PATH="$repo/model_prices_and_context_window.json" LITELLM_CONFIG="$out/config.yaml" \ + HOST=127.0.0.1 PORT="$rust_port" "$GATEWAY" >"$out/rust.log" 2>&1 & +pids+=($!) + +for url in "http://127.0.0.1:$python_port/health/liveliness" "http://127.0.0.1:$rust_port/health/liveliness"; do + for _ in $(seq 120); do curl -sf "$url" >/dev/null && break; sleep 1; done + curl -sf "$url" >/dev/null || { echo "not up: $url (see $out/*.log)" >&2; exit 1; } +done + +"$out/chatparity" run -mock-addr "127.0.0.1:$mock_port" -python "http://127.0.0.1:$python_port" \ + -rust "http://127.0.0.1:$rust_port" -out "$out" -known "$here/cmd/chatparity/known.json" -filter "${FILTER:-}"