diff --git a/litellm-rust/docs/customer-proxy/chat-parity.svg b/litellm-rust/docs/customer-proxy/chat-parity.svg
new file mode 100644
index 00000000000..b741dc57b2b
--- /dev/null
+++ b/litellm-rust/docs/customer-proxy/chat-parity.svg
@@ -0,0 +1,41 @@
+
diff --git a/litellm-rust/loadtest/README.md b/litellm-rust/loadtest/README.md
index 7a03f1c3fb2..9c96c0ce4d0 100644
--- a/litellm-rust/loadtest/README.md
+++ b/litellm-rust/loadtest/README.md
@@ -12,6 +12,7 @@ machines.
| `summarize` | Merges result files and prints proxy overhead (observed minus planned time) by stage, class and context size. |
| `procmon` | Samples the gateway process's CPU, RSS, fds and TCP connections from `/proc`. |
| `verify` | Checks what each gateway recorded for a run against what loadgen sent: spend logs, key/team/user/member spend, daily tables, Prometheus counters, S3 request logs and the headers the provider received. See [Full-feature runs](#full-feature-runs). |
+| `chatparity` | Sends the same chat completions requests through the Python proxy and the gateway, both pointed at its recording mock provider, and reports every difference in what the provider was sent and what the client got. See [Chat parity](#chat-parity). |
| `s3stub` | In-memory path-style S3 (PUT, GET, ListObjectsV2) for the `s3_v2` logs of a full-feature run. |
Workloads live in `workloads/`: `smoke`, `realistic-ramp`, `long-streams`, `large-context`,
@@ -185,3 +186,49 @@ carry static dummy AWS keys, so every request is SigV4-signed, and mockllm answe
`/model//invoke` and `/invoke-with-response-stream` with AWS event-stream framing that the
gateway re-encodes as Anthropic SSE. Bedrock hits show up as `bedrock--` in the mock's `/stats`.
+## Chat parity
+
+`scripts/chat-parity.sh` checks the gateway's `/v1/chat/completions` against the Python proxy's.
+It writes one config, starts both proxies on it, and sends every case through each one in turn.
+Both talk to `chatparity`'s mock, which answers like each provider's API (OpenAI-compatible, Azure,
+Anthropic, Bedrock Converse, Gemini, Vertex Gemini and Vertex Claude) and records every request it
+receives.
+
+```sh
+scripts/chat-parity.sh # every case, report in out/chat-parity/report.md
+FILTER=bedrock-env/ scripts/chat-parity.sh # one deployment, scenario or mode
+GATEWAY= PYTHON=/bin/python scripts/chat-parity.sh
+```
+
+A case is one deployment, one request scenario and one mode (`//`).
+There are 19 deployments (every chat provider the gateway serves natively) and 15 scenarios: plain
+text, multi-turn with a system message, tools, a tool result round trip, `response_format`
+(JSON schema and JSON object), inline and remote images, `reasoning_effort`, `cache_control`,
+unsupported params refused and then dropped, `max_completion_tokens`, and streams with and without
+`include_usage`. The mock gives both proxies the same answer for the same request (text, a tool
+call, thinking or JSON, picked by a `[[parity reply=...]]` marker in the prompt), so any difference
+in what the client gets comes from the proxies.
+
+For each case it compares:
+
+- **upstream**: the method, path, query, headers and JSON body the provider received. Transport
+ headers (user agent, content length, SDK telemetry) are ignored, and a SigV4 signature is reduced
+ to its region and service.
+- **response**: the status, content type and JSON body the client got. Ids and timestamps the
+ proxies generate are compared by prefix only (`chatcmpl-*`). For a stream, this is what a client
+ library would assemble: content, reasoning, tool calls, finish reasons, usage, the models named,
+ whether one id spans the stream, and `[DONE]`.
+- **chunks**: the stream chunk by chunk. This is informational only, since the same content can be
+ split into chunks differently.
+
+`out/chat-parity/report.md` lists every open difference, and `cases/.json` holds the request,
+both responses and everything each proxy sent upstream. Once a difference has been reviewed and
+accepted, add it to `cmd/chatparity/known.json` (`{"case": "*/stream", "where": "upstream", "path":
+"*.accept", "reason": "..."}`, where `*` matches anything). The run fails only on differences that
+aren't listed there.
+
+The Python side needs the proxy and `google-auth` installed (`uv pip install -e '.[proxy]'
+google-auth`). Both price from the repo's `model_prices_and_context_window.json`, so parameter
+support and defaults come from the same data. `bedrock` carries its AWS settings in the config;
+`bedrock-env` uses `api_base` with credentials from the environment, which the script sets to dummy
+values that only ever reach the mock.
diff --git a/litellm-rust/loadtest/cmd/chatparity/cases.go b/litellm-rust/loadtest/cmd/chatparity/cases.go
new file mode 100644
index 00000000000..6f197d865cb
--- /dev/null
+++ b/litellm-rust/loadtest/cmd/chatparity/cases.go
@@ -0,0 +1,190 @@
+package main
+
+import (
+ "fmt"
+ "strings"
+)
+
+// deployment is one model group in the shared config, pointing at the mock.
+type deployment struct {
+ name string // model group name the client asks for
+ family string // which scenarios apply: openai, claude, gemini or compat
+ params map[string]any // litellm_params, with {mock} replaced by the mock's URL
+}
+
+func deployments(credentials string) []deployment {
+ compat := func(name, model string) deployment {
+ return deployment{name, "compat", map[string]any{"model": model, "api_base": "{mock}/v1", "api_key": "sk-parity-upstream"}}
+ }
+ vertex := map[string]any{"api_base": "{mock}", "vertex_project": "parity-project", "vertex_credentials": credentials}
+ with := func(base map[string]any, extra map[string]any) map[string]any {
+ out := map[string]any{}
+ for k, v := range base {
+ out[k] = v
+ }
+ for k, v := range extra {
+ out[k] = v
+ }
+ return out
+ }
+ return []deployment{
+ {"openai", "openai", map[string]any{"model": "openai/gpt-4o", "api_base": "{mock}/v1", "api_key": "sk-parity-upstream"}},
+ {"azure", "openai", map[string]any{"model": "azure/gpt-4o", "api_base": "{mock}", "api_key": "sk-parity-upstream",
+ "api_version": "2025-02-01-preview"}},
+ {"anthropic", "claude", map[string]any{"model": "anthropic/claude-sonnet-4-5-20250929", "api_base": "{mock}",
+ "api_key": "sk-parity-upstream"}},
+ {"bedrock", "claude", map[string]any{"model": "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
+ "aws_access_key_id": "AKIAPARITYPARITY", "aws_secret_access_key": "parity-secret", "aws_region_name": "us-east-1",
+ "aws_bedrock_runtime_endpoint": "{mock}"}},
+ // The same model with the endpoint as api_base and credentials from the environment
+ // (chat-parity.sh sets them), for a host that does not read AWS settings from the config.
+ {"bedrock-env", "claude", map[string]any{"model": "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", "api_base": "{mock}"}},
+ {"vertex-claude", "claude", with(vertex, map[string]any{"model": "vertex_ai/claude-sonnet-4-5@20250929",
+ "vertex_location": "us-east5"})},
+ {"gemini", "gemini", map[string]any{"model": "gemini/gemini-2.5-flash", "api_base": "{mock}", "api_key": "parity-google-key"}},
+ {"vertex-gemini", "gemini", with(vertex, map[string]any{"model": "vertex_ai/gemini-2.5-flash", "vertex_location": "us-central1"})},
+ compat("groq", "groq/llama-3.3-70b-versatile"),
+ compat("mistral", "mistral/mistral-large-latest"),
+ compat("deepseek", "deepseek/deepseek-chat"),
+ compat("together", "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo"),
+ compat("fireworks", "fireworks_ai/accounts/fireworks/models/llama-v3p3-70b-instruct"),
+ compat("xai", "xai/grok-3"),
+ compat("openrouter", "openrouter/openai/gpt-4o"),
+ compat("cerebras", "cerebras/llama3.1-8b"),
+ compat("perplexity", "perplexity/sonar"),
+ compat("hosted-vllm", "hosted_vllm/Qwen/Qwen3-8B"),
+ compat("nvidia-nim", "nvidia_nim/meta/llama-3.3-70b-instruct"),
+ }
+}
+
+// scenario is one shape of chat request, sent to every deployment whose family it lists.
+type scenario struct {
+ name string
+ families string // space-separated families, or "all"
+ streams []bool // which of non-streaming and streaming to send
+ body func(mock string) map[string]any
+}
+
+var both = []bool{false, true}
+
+func user(text string) map[string]any { return map[string]any{"role": "user", "content": text} }
+
+var weatherTool = map[string]any{"type": "function", "function": map[string]any{
+ "name": "get_weather", "description": "Current weather for a city",
+ "parameters": map[string]any{"type": "object", "properties": map[string]any{"city": map[string]any{"type": "string"}},
+ "required": []any{"city"}},
+}}
+
+const pngDataURL = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
+
+var scenarios = []scenario{
+ {"basic", "all", both, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, "temperature": 0.2}
+ }},
+ {"include-usage", "all", []bool{true}, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50,
+ "stream_options": map[string]any{"include_usage": true}}
+ }},
+ {"system-multi-turn", "all", both, func(string) map[string]any {
+ return map[string]any{"messages": []any{
+ map[string]any{"role": "system", "content": "You are terse."},
+ user("Hi."),
+ map[string]any{"role": "assistant", "content": "Hello."},
+ user("Say hello again."),
+ }, "max_tokens": 50, "top_p": 0.9, "stop": []any{"END"}, "user": "parity-user"}
+ }},
+ {"tools", "all", both, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Weather in Paris? [[parity reply=tool]]")},
+ "tools": []any{weatherTool}, "tool_choice": "auto", "max_tokens": 100}
+ }},
+ {"tools-include-usage", "all", []bool{true}, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Weather in Paris? [[parity reply=tool]]")},
+ "tools": []any{weatherTool}, "tool_choice": "required", "max_tokens": 100,
+ "stream_options": map[string]any{"include_usage": true}}
+ }},
+ {"tool-result", "all", both, func(string) map[string]any {
+ return map[string]any{"messages": []any{
+ user("Weather in Paris?"),
+ map[string]any{"role": "assistant", "content": nil, "tool_calls": []any{map[string]any{"id": "call_prev_1",
+ "type": "function", "function": map[string]any{"name": "get_weather", "arguments": `{"city": "Paris"}`}}}},
+ map[string]any{"role": "tool", "tool_call_id": "call_prev_1", "content": "18C and sunny"},
+ }, "tools": []any{weatherTool}, "max_tokens": 100}
+ }},
+ {"json-schema", "all", both, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Answer as JSON. [[parity reply=json]]")}, "max_tokens": 100,
+ "response_format": map[string]any{"type": "json_schema", "json_schema": map[string]any{"name": "answer", "strict": true,
+ "schema": map[string]any{"type": "object", "properties": map[string]any{"answer": map[string]any{"type": "integer"}},
+ "required": []any{"answer"}, "additionalProperties": false}}}}
+ }},
+ {"json-object", "openai gemini compat", []bool{false}, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Answer as JSON. [[parity reply=json]]")}, "max_tokens": 100,
+ "response_format": map[string]any{"type": "json_object"}}
+ }},
+ {"image-data-url", "openai claude gemini", both, func(string) map[string]any {
+ return map[string]any{"messages": []any{map[string]any{"role": "user", "content": []any{
+ map[string]any{"type": "text", "text": "What is in this image?"},
+ map[string]any{"type": "image_url", "image_url": map[string]any{"url": pngDataURL}},
+ }}}, "max_tokens": 50}
+ }},
+ {"image-remote-url", "openai claude gemini", []bool{false}, func(mock string) map[string]any {
+ return map[string]any{"messages": []any{map[string]any{"role": "user", "content": []any{
+ map[string]any{"type": "text", "text": "What is in this image?"},
+ map[string]any{"type": "image_url", "image_url": map[string]any{"url": mock + "/img.png"}},
+ }}}, "max_tokens": 50}
+ }},
+ {"reasoning-effort", "claude gemini", both, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Think, then say hello. [[parity reply=thinking]]")},
+ "reasoning_effort": "low", "max_tokens": 4096}
+ }},
+ {"cache-control", "claude", []bool{false}, func(string) map[string]any {
+ return map[string]any{"messages": []any{
+ map[string]any{"role": "system", "content": []any{map[string]any{"type": "text", "text": "Long shared prefix.",
+ "cache_control": map[string]any{"type": "ephemeral"}}}},
+ user("Say hello."),
+ }, "max_tokens": 50}
+ }},
+ {"unsupported-param", "all", []bool{false}, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, "n": 2, "logit_bias": map[string]any{"50256": -100}}
+ }},
+ {"drop-params", "all", []bool{false}, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, "n": 2,
+ "logit_bias": map[string]any{"50256": -100}, "drop_params": true}
+ }},
+ {"max-completion-tokens", "all", []bool{false}, func(string) map[string]any {
+ return map[string]any{"messages": []any{user("Say hello.")}, "max_completion_tokens": 60, "seed": 7}
+ }},
+}
+
+// testCase is one request, sent once to each proxy.
+type testCase struct {
+ Name string `json:"name"`
+ Model string `json:"model"`
+ Stream bool `json:"stream"`
+ Body map[string]any `json:"body"`
+}
+
+func buildCases(deps []deployment, mockURL, filter string) []testCase {
+ var cases []testCase
+ for _, d := range deps {
+ for _, s := range scenarios {
+ if s.families != "all" && !strings.Contains(" "+s.families+" ", " "+d.family+" ") {
+ continue
+ }
+ for _, stream := range s.streams {
+ body := s.body(mockURL)
+ body["model"] = d.name
+ mode := "unary"
+ if stream {
+ body["stream"] = true
+ mode = "stream"
+ }
+ name := fmt.Sprintf("%s/%s/%s", d.name, s.name, mode)
+ if filter != "" && !strings.Contains(name, filter) {
+ continue
+ }
+ cases = append(cases, testCase{name, d.name, stream, body})
+ }
+ }
+ }
+ return cases
+}
diff --git a/litellm-rust/loadtest/cmd/chatparity/chatparity_test.go b/litellm-rust/loadtest/cmd/chatparity/chatparity_test.go
new file mode 100644
index 00000000000..ba260bac16a
--- /dev/null
+++ b/litellm-rust/loadtest/cmd/chatparity/chatparity_test.go
@@ -0,0 +1,112 @@
+package main
+
+import (
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+)
+
+func decode(t *testing.T, text string) any {
+ t.Helper()
+ var v any
+ if err := json.Unmarshal([]byte(text), &v); err != nil {
+ t.Fatal(err)
+ }
+ return v
+}
+
+func TestDiffValuesReportsMissingAndChangedPaths(t *testing.T) {
+ var diffs []difference
+ diffValues("response", "", decode(t, `{"a": 1, "b": [1, 2], "c": {"d": "x"}}`),
+ decode(t, `{"a": 1, "b": [1], "c": {"d": "y"}, "e": true}`), &diffs)
+ got := map[string]bool{}
+ for _, d := range diffs {
+ got[d.Path] = true
+ }
+ for _, want := range []string{".b[1]", ".c.d", ".e"} {
+ if !got[want] {
+ t.Errorf("missing %s in %+v", want, diffs)
+ }
+ }
+ if len(diffs) != 3 {
+ t.Errorf("want 3 differences, got %+v", diffs)
+ }
+}
+
+func TestNormalizeIDsKeepsMockIDsAndPrefixesOfGeneratedOnes(t *testing.T) {
+ got := normalizeIDs(decode(t, `{"id": "chatcmpl-123", "created": 5, "choices": [{"message": {"tool_calls": [{"id": "call_parity_1"}, {"id": "call_abc"}]}}]}`))
+ want := decode(t, `{"id": "chatcmpl-*", "created": "", "choices": [{"message": {"tool_calls": [{"id": "call_parity_1"}, {"id": "call_*"}]}}]}`)
+ if !equal(got, want) {
+ t.Errorf("got %v", got)
+ }
+}
+
+func TestAssembleFoldsContentToolCallsAndUsage(t *testing.T) {
+ s := parseSSE([]byte(strings.Join([]string{
+ `data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"role":"assistant","content":"Hel"}}]}`,
+ `data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"content":"lo"}}]}`,
+ `data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_parity_1","type":"function","function":{"name":"f","arguments":"{\"a\""}}]}}]}`,
+ `data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":": 1}"}}]},"finish_reason":"tool_calls"}]}`,
+ `data: {"id":"chatcmpl-1","model":"m","choices":[],"usage":{"total_tokens":3}}`,
+ `data: [DONE]`,
+ }, "\n\n")))
+ got := assemble(s)
+ if got["content"] != "Hello" || got["done"] != true || got["one_id_per_stream"] != true {
+ t.Errorf("got %v", got)
+ }
+ calls := got["tool_calls"].([]any)
+ if len(calls) != 1 || calls[0].(map[string]any)["arguments"] != `{"a": 1}` {
+ t.Errorf("tool calls %v", calls)
+ }
+ if !equal(got["finish_reasons"], []any{"tool_calls"}) || !equal(got["usage"], map[string]any{"total_tokens": 3.0}) {
+ t.Errorf("got %v", got)
+ }
+}
+
+func TestNormalizeUpstreamDropsTransportHeadersAndTheSignature(t *testing.T) {
+ got := normalizeUpstream([]upstream{{Method: "POST", Path: "/p", Headers: map[string]string{
+ "user-agent": "x",
+ "x-amz-date": "20261019T000000Z",
+ "authorization": "AWS4-HMAC-SHA256 Credential=AKIA/20261019/us-east-1/bedrock/aws4_request, SignedHeaders=host, Signature=abc",
+ "anthropic-version": "2023-06-01",
+ }}})
+ headers := got.([]any)[0].(map[string]any)["headers"]
+ want := map[string]any{"authorization": "AWS4-HMAC-SHA256 region=us-east-1 service=bedrock", "anthropic-version": "2023-06-01"}
+ if !equal(headers, want) {
+ t.Errorf("got %v", headers)
+ }
+}
+
+func TestMarkKnownMatchesCaseWhereAndPathGlobs(t *testing.T) {
+ diffs := []difference{{Where: "upstream", Path: "[0].headers.accept"}, {Where: "response", Path: ".body.model"}}
+ markKnown("openai/basic/unary", diffs, []known{{Case: "*/basic/*", Where: "upstream", Path: "*.accept", Reason: "r"}})
+ if diffs[0].Known != "r" || diffs[1].Known != "" {
+ t.Errorf("got %+v", diffs)
+ }
+}
+
+func TestMockAnswersAnthropicJSONModeThroughTheForcedTool(t *testing.T) {
+ m := &mock{}
+ body := `{"model":"c","tools":[{"name":"json_tool_call"}],"messages":[{"role":"user","content":"x"}]}`
+ w := httptest.NewRecorder()
+ m.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/messages", strings.NewReader(body)))
+ got := decode(t, w.Body.String())
+ if dig(got, "content", 0, "name") != "json_tool_call" || dig(got, "content", 0, "input", "answer") != 42.0 {
+ t.Errorf("got %v", got)
+ }
+ if recorded := m.take(); len(recorded) != 1 || recorded[0].Path != "/v1/messages" {
+ t.Errorf("recorded %+v", recorded)
+ }
+}
+
+func TestEveryCaseNameIsUnique(t *testing.T) {
+ seen := map[string]bool{}
+ for _, c := range buildCases(deployments(""), "http://mock", "") {
+ if seen[c.Name] {
+ t.Errorf("duplicate case %s", c.Name)
+ }
+ seen[c.Name] = true
+ }
+}
diff --git a/litellm-rust/loadtest/cmd/chatparity/diff.go b/litellm-rust/loadtest/cmd/chatparity/diff.go
new file mode 100644
index 00000000000..6e86bf98c8e
--- /dev/null
+++ b/litellm-rust/loadtest/cmd/chatparity/diff.go
@@ -0,0 +1,339 @@
+package main
+
+import (
+ "bufio"
+ "bytes"
+ "encoding/json"
+ "fmt"
+ "regexp"
+ "sort"
+ "strings"
+)
+
+// difference is one place where the two proxies disagree. Where is "upstream" (what the
+// provider was sent), "response" (what the client got, or the assembled stream) or "chunks"
+// (the stream chunk by chunk, informational: boundaries may differ for the same content).
+type difference struct {
+ Where string `json:"where"`
+ Path string `json:"path"`
+ Python any `json:"python"`
+ Rust any `json:"rust"`
+ Known string `json:"known,omitempty"` // the reason, when the difference is listed as known
+}
+
+const absent = ""
+
+// diffValues appends every path where a and b differ.
+func diffValues(where, path string, a, b any, out *[]difference) {
+ switch av := a.(type) {
+ case map[string]any:
+ bv, ok := b.(map[string]any)
+ if !ok {
+ break
+ }
+ keys := map[string]bool{}
+ for k := range av {
+ keys[k] = true
+ }
+ for k := range bv {
+ keys[k] = true
+ }
+ for _, k := range sortedKeys(keys) {
+ x, xok := av[k]
+ y, yok := bv[k]
+ switch {
+ case !xok:
+ *out = append(*out, difference{Where: where, Path: path + "." + k, Python: absent, Rust: y})
+ case !yok:
+ *out = append(*out, difference{Where: where, Path: path + "." + k, Python: x, Rust: absent})
+ default:
+ diffValues(where, path+"."+k, x, y, out)
+ }
+ }
+ return
+ case []any:
+ bv, ok := b.([]any)
+ if !ok {
+ break
+ }
+ for i := 0; i < len(av) || i < len(bv); i++ {
+ p := fmt.Sprintf("%s[%d]", path, i)
+ switch {
+ case i >= len(av):
+ *out = append(*out, difference{Where: where, Path: p, Python: absent, Rust: bv[i]})
+ case i >= len(bv):
+ *out = append(*out, difference{Where: where, Path: p, Python: av[i], Rust: absent})
+ default:
+ diffValues(where, p, av[i], bv[i], out)
+ }
+ }
+ return
+ }
+ if !equal(a, b) {
+ *out = append(*out, difference{Where: where, Path: path, Python: a, Rust: b})
+ }
+}
+
+func equal(a, b any) bool {
+ x, _ := json.Marshal(a)
+ y, _ := json.Marshal(b)
+ return bytes.Equal(x, y)
+}
+
+// Headers that differ for reasons unrelated to translation: client libraries, transport,
+// tracing, and request signing that depends on the clock.
+var ignoredHeaders = regexp.MustCompile(`^(user-agent|content-length|accept-encoding|connection|host|traceparent|tracestate|` +
+ `x-stainless-.*|amz-sdk-.*|x-amzn-trace-id|x-amz-date|x-amz-content-sha256|x-goog-api-client|x-litellm-.*)$`)
+
+var sigV4 = regexp.MustCompile(`^AWS4-HMAC-SHA256 Credential=[^/]+/\d{8}/([^/]+)/([^/]+)/aws4_request`)
+
+func normalizeUpstream(requests []upstream) any {
+ out := make([]any, len(requests))
+ for i, r := range requests {
+ headers := map[string]any{}
+ for name, value := range r.Headers {
+ if ignoredHeaders.MatchString(name) {
+ continue
+ }
+ if name == "authorization" {
+ if m := sigV4.FindStringSubmatch(value); m != nil {
+ value = fmt.Sprintf("AWS4-HMAC-SHA256 region=%s service=%s", m[1], m[2])
+ }
+ }
+ headers[name] = value
+ }
+ entry := map[string]any{"method": r.Method, "path": r.Path, "headers": headers, "body": r.Body}
+ if r.Query != "" {
+ entry["query"] = r.Query
+ }
+ out[i] = entry
+ }
+ return out
+}
+
+// normalizeIDs replaces ids and timestamps the proxies generate with a placeholder that keeps
+// their prefix (`chatcmpl-*`, `call_*`), so only their shape is compared. Ids the mock chose
+// contain "parity" and are compared exactly.
+func normalizeIDs(v any) any {
+ switch t := v.(type) {
+ case map[string]any:
+ out := map[string]any{}
+ for k, x := range t {
+ switch {
+ case k == "created":
+ out[k] = ""
+ case k == "id":
+ out[k] = generated(x)
+ default:
+ out[k] = normalizeIDs(x)
+ }
+ }
+ return out
+ case []any:
+ out := make([]any, len(t))
+ for i, x := range t {
+ out[i] = normalizeIDs(x)
+ }
+ return out
+ }
+ return v
+}
+
+func generated(v any) any {
+ s, ok := v.(string)
+ if !ok || strings.Contains(s, "parity") {
+ return v
+ }
+ if i := strings.IndexAny(s, "-_"); i >= 0 {
+ return s[:i+1] + "*"
+ }
+ return "*"
+}
+
+// stream is a parsed SSE response.
+type stream struct {
+ Chunks []any `json:"chunks"`
+ Done bool `json:"done"`
+ Other []any `json:"other,omitempty"` // lines that are neither JSON data nor [DONE]
+}
+
+func parseSSE(body []byte) stream {
+ var s stream
+ scanner := bufio.NewScanner(bytes.NewReader(body))
+ scanner.Buffer(make([]byte, 1<<20), 16<<20)
+ for scanner.Scan() {
+ line := scanner.Text()
+ if line == "" || strings.HasPrefix(line, ":") {
+ continue
+ }
+ data, ok := strings.CutPrefix(line, "data:")
+ if !ok {
+ s.Other = append(s.Other, line)
+ continue
+ }
+ data = strings.TrimSpace(data)
+ if data == "[DONE]" {
+ s.Done = true
+ continue
+ }
+ var chunk any
+ if json.Unmarshal([]byte(data), &chunk) != nil {
+ s.Other = append(s.Other, line)
+ continue
+ }
+ s.Chunks = append(s.Chunks, chunk)
+ }
+ return s
+}
+
+// assemble folds a stream's chunks into what a client library would build from them, which is
+// what must agree; how the content was split into chunks may differ.
+func assemble(s stream) map[string]any {
+ var content, reasoning strings.Builder
+ type call struct{ id, kind, name, args any }
+ calls := map[float64]*call{}
+ var order []float64
+ ids := map[string]bool{}
+ models := map[string]bool{}
+ var roles, finishes []any
+ var usage, errorFrame any
+ extra := map[string]bool{}
+ sawContent, sawReasoning := false, false
+ for _, c := range s.Chunks {
+ chunk, _ := c.(map[string]any)
+ if e, ok := chunk["error"]; ok {
+ errorFrame = e
+ continue
+ }
+ if id, ok := chunk["id"].(string); ok {
+ ids[id] = true
+ }
+ if m, ok := chunk["model"].(string); ok {
+ models[m] = true
+ }
+ if u, ok := chunk["usage"]; ok && u != nil {
+ usage = u
+ }
+ for k := range chunk {
+ switch k {
+ case "id", "object", "created", "model", "choices", "usage", "system_fingerprint":
+ default:
+ extra[k] = true
+ }
+ }
+ choices, _ := chunk["choices"].([]any)
+ for _, ch := range choices {
+ choice, _ := ch.(map[string]any)
+ if f := choice["finish_reason"]; f != nil {
+ finishes = append(finishes, f)
+ }
+ delta, _ := choice["delta"].(map[string]any)
+ for k, v := range delta {
+ switch k {
+ case "role":
+ if v != nil {
+ roles = append(roles, v)
+ }
+ case "content":
+ if t, ok := v.(string); ok {
+ content.WriteString(t)
+ sawContent = true
+ }
+ case "reasoning_content":
+ if t, ok := v.(string); ok {
+ reasoning.WriteString(t)
+ sawReasoning = true
+ }
+ case "tool_calls":
+ list, _ := v.([]any)
+ for _, item := range list {
+ tc, _ := item.(map[string]any)
+ index, _ := tc["index"].(float64)
+ entry := calls[index]
+ if entry == nil {
+ entry = &call{}
+ calls[index] = entry
+ order = append(order, index)
+ }
+ if x := tc["id"]; x != nil {
+ entry.id = generated(x)
+ }
+ if x := tc["type"]; x != nil {
+ entry.kind = x
+ }
+ fn, _ := tc["function"].(map[string]any)
+ if x := fn["name"]; x != nil {
+ entry.name = x
+ }
+ if x, ok := fn["arguments"].(string); ok {
+ prev, _ := entry.args.(string)
+ entry.args = prev + x
+ }
+ }
+ default:
+ if v != nil {
+ extra["delta."+k] = true
+ }
+ }
+ }
+ }
+ }
+ sort.Float64s(order)
+ var toolCalls []any
+ for _, i := range order {
+ c := calls[i]
+ toolCalls = append(toolCalls, map[string]any{"index": i, "id": c.id, "type": c.kind, "name": c.name, "arguments": c.args})
+ }
+ out := map[string]any{
+ "one_id_per_stream": len(ids) <= 1,
+ "id_prefix": nil,
+ "models": sortedKeys(models),
+ "roles": roles,
+ "finish_reasons": finishes,
+ "usage": usage,
+ "done": s.Done,
+ "tool_calls": toolCalls,
+ "other_fields": sortedKeys(extra),
+ "non_data_lines": s.Other,
+ }
+ for id := range ids {
+ out["id_prefix"] = generated(id)
+ }
+ if sawContent {
+ out["content"] = content.String()
+ }
+ if sawReasoning {
+ out["reasoning_content"] = reasoning.String()
+ }
+ if errorFrame != nil {
+ out["error"] = errorFrame
+ }
+ return out
+}
+
+// known is a difference the team has looked at and accepted, with the reason.
+type known struct {
+ Case string `json:"case"` // glob over case names; * matches anything
+ Where string `json:"where"` // upstream, response or chunks; empty matches all
+ Path string `json:"path"` // glob over the difference's path
+ Reason string `json:"reason"`
+}
+
+func glob(pattern string) *regexp.Regexp {
+ parts := strings.Split(pattern, "*")
+ for i, p := range parts {
+ parts[i] = regexp.QuoteMeta(p)
+ }
+ return regexp.MustCompile("^" + strings.Join(parts, ".*") + "$")
+}
+
+func markKnown(name string, diffs []difference, list []known) {
+ for i := range diffs {
+ for _, k := range list {
+ if (k.Where == "" || k.Where == diffs[i].Where) && glob(k.Case).MatchString(name) && glob(k.Path).MatchString(diffs[i].Path) {
+ diffs[i].Known = k.Reason
+ break
+ }
+ }
+ }
+}
diff --git a/litellm-rust/loadtest/cmd/chatparity/known.json b/litellm-rust/loadtest/cmd/chatparity/known.json
new file mode 100644
index 00000000000..fe51488c706
--- /dev/null
+++ b/litellm-rust/loadtest/cmd/chatparity/known.json
@@ -0,0 +1 @@
+[]
diff --git a/litellm-rust/loadtest/cmd/chatparity/main.go b/litellm-rust/loadtest/cmd/chatparity/main.go
new file mode 100644
index 00000000000..e26468dac46
--- /dev/null
+++ b/litellm-rust/loadtest/cmd/chatparity/main.go
@@ -0,0 +1,289 @@
+// chatparity sends the same chat completions requests through the Python proxy and the Rust
+// gateway, both pointed at one recording mock provider, and reports every difference in what the
+// provider was sent and what the client got back.
+//
+// chatparity config -mock http://127.0.0.1:9100 -out out # config.yaml both proxies load
+// chatparity run -mock-addr 127.0.0.1:9100 -python http://127.0.0.1:4001 \
+// -rust http://127.0.0.1:4002 -out out [-known known.json] [-filter anthropic/]
+//
+// The mock answers each provider's API (OpenAI-compatible, Azure, Anthropic, Bedrock Converse,
+// Gemini, Vertex) the same way for both proxies, so differences in the client's answer come from
+// the proxies' translation. scripts/chat-parity.sh builds and starts everything locally.
+package main
+
+import (
+ "bytes"
+ "crypto/rand"
+ "crypto/rsa"
+ "crypto/x509"
+ "encoding/json"
+ "encoding/pem"
+ "flag"
+ "fmt"
+ "io"
+ "log"
+ "net"
+ "net/http"
+ "net/url"
+ "os"
+ "path/filepath"
+ "strings"
+ "time"
+)
+
+const masterKey = "sk-parity-master"
+
+func main() {
+ log.SetFlags(0)
+ if len(os.Args) < 2 {
+ log.Fatal("usage: chatparity config|run ...")
+ }
+ switch os.Args[1] {
+ case "config":
+ configCmd(os.Args[2:])
+ case "run":
+ runCmd(os.Args[2:])
+ default:
+ log.Fatalf("unknown subcommand %q", os.Args[1])
+ }
+}
+
+// configCmd writes the one config both proxies load: every deployment pointed at the mock, and
+// a throwaway service account whose token endpoint is the mock, for the Vertex deployments.
+func configCmd(args []string) {
+ fs := flag.NewFlagSet("config", flag.ExitOnError)
+ mockURL := fs.String("mock", "http://127.0.0.1:9100", "URL the proxies reach the mock at")
+ out := fs.String("out", "out", "directory to write config.yaml to")
+ fs.Parse(args)
+ must(os.MkdirAll(*out, 0o755))
+ var models []any
+ for _, d := range deployments(serviceAccount(*mockURL)) {
+ params := map[string]any{}
+ for k, v := range d.params {
+ if s, ok := v.(string); ok {
+ v = strings.ReplaceAll(s, "{mock}", *mockURL)
+ }
+ params[k] = v
+ }
+ models = append(models, map[string]any{"model_name": d.name, "litellm_params": params})
+ }
+ // JSON is YAML, and both proxies read it as their config.yaml.
+ config := map[string]any{
+ "model_list": models,
+ "general_settings": map[string]any{"master_key": masterKey},
+ "litellm_settings": map[string]any{"num_retries": 0, "user_url_allowed_hosts": []any{mockHost(*mockURL)}},
+ }
+ b, _ := json.MarshalIndent(config, "", " ")
+ must(os.WriteFile(filepath.Join(*out, "config.yaml"), b, 0o644))
+ fmt.Println(filepath.Join(*out, "config.yaml"))
+}
+
+// mockHost is the mock's host:port, allowed as a media URL host so remote image cases reach it.
+func mockHost(mockURL string) string {
+ u, err := url.Parse(mockURL)
+ must(err)
+ return u.Host
+}
+
+func serviceAccount(mockURL string) string {
+ key, err := rsa.GenerateKey(rand.Reader, 2048)
+ must(err)
+ der, err := x509.MarshalPKCS8PrivateKey(key)
+ must(err)
+ b, _ := json.Marshal(map[string]any{
+ "type": "service_account", "project_id": "parity-project", "private_key_id": "parity",
+ "private_key": string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der})),
+ "client_email": "parity@parity-project.iam.gserviceaccount.com", "client_id": "1",
+ "token_uri": mockURL + "/token",
+ })
+ return string(b)
+}
+
+// exchange is one side's request and what came of it.
+type exchange struct {
+ Status int `json:"status"`
+ ContentType string `json:"content_type"`
+ Body any `json:"body"`
+ Upstream []upstream `json:"upstream"`
+ Error string `json:"error,omitempty"`
+}
+
+type outcome struct {
+ Case string `json:"case"`
+ Python exchange `json:"python"`
+ Rust exchange `json:"rust"`
+ Differences []difference `json:"differences"`
+}
+
+func runCmd(args []string) {
+ fs := flag.NewFlagSet("run", flag.ExitOnError)
+ mockAddr := fs.String("mock-addr", "127.0.0.1:9100", "address the mock listens on")
+ mockURL := fs.String("mock", "", "URL the proxies reach the mock at (default http://)")
+ python := fs.String("python", "http://127.0.0.1:4001", "Python proxy base URL")
+ rust := fs.String("rust", "http://127.0.0.1:4002", "Rust gateway base URL")
+ out := fs.String("out", "out", "directory for report.md, results.json and per-case captures")
+ knownPath := fs.String("known", "", "JSON list of accepted differences")
+ filter := fs.String("filter", "", "only run cases whose name contains this")
+ fs.Parse(args)
+ if *mockURL == "" {
+ *mockURL = "http://" + *mockAddr
+ }
+ var accepted []known
+ if *knownPath != "" {
+ b, err := os.ReadFile(*knownPath)
+ must(err)
+ must(json.Unmarshal(b, &accepted))
+ }
+
+ m := &mock{}
+ listener, err := net.Listen("tcp", *mockAddr)
+ must(err)
+ go http.Serve(listener, m)
+
+ client := &http.Client{Timeout: 120 * time.Second}
+ cases := buildCases(deployments(""), *mockURL, *filter)
+ must(os.MkdirAll(filepath.Join(*out, "cases"), 0o755))
+ var outcomes []outcome
+ for _, c := range cases {
+ o := outcome{Case: c.Name}
+ o.Python = send(client, m, *python, c)
+ o.Rust = send(client, m, *rust, c)
+ o.Differences = compare(c, o.Python, o.Rust)
+ markKnown(c.Name, o.Differences, accepted)
+ outcomes = append(outcomes, o)
+ b, _ := json.MarshalIndent(map[string]any{"request": c.Body, "outcome": o}, "", " ")
+ must(os.WriteFile(filepath.Join(*out, "cases", strings.ReplaceAll(c.Name, "/", "__")+".json"), b, 0o644))
+ fmt.Printf("%-55s %s\n", c.Name, verdict(o.Differences))
+ }
+ b, _ := json.MarshalIndent(outcomes, "", " ")
+ must(os.WriteFile(filepath.Join(*out, "results.json"), b, 0o644))
+ failing := writeReport(filepath.Join(*out, "report.md"), outcomes)
+ fmt.Printf("\n%d cases, %d with unexplained differences; report in %s\n", len(outcomes), failing, filepath.Join(*out, "report.md"))
+ if failing > 0 {
+ os.Exit(1)
+ }
+}
+
+func send(client *http.Client, m *mock, base string, c testCase) exchange {
+ m.take()
+ body, _ := json.Marshal(c.Body)
+ req, _ := http.NewRequest(http.MethodPost, strings.TrimRight(base, "/")+"/v1/chat/completions", bytes.NewReader(body))
+ req.Header.Set("Content-Type", "application/json")
+ req.Header.Set("Authorization", "Bearer "+masterKey)
+ resp, err := client.Do(req)
+ if err != nil {
+ return exchange{Error: err.Error(), Upstream: m.take()}
+ }
+ defer resp.Body.Close()
+ raw, _ := io.ReadAll(resp.Body)
+ x := exchange{Status: resp.StatusCode, ContentType: resp.Header.Get("Content-Type"), Upstream: m.take()}
+ if strings.HasPrefix(x.ContentType, "text/event-stream") {
+ x.Body = parseSSE(raw)
+ } else {
+ var parsed any
+ if json.Unmarshal(raw, &parsed) == nil {
+ x.Body = parsed
+ } else {
+ x.Body = string(raw)
+ }
+ }
+ return x
+}
+
+func compare(c testCase, py, rs exchange) []difference {
+ var diffs []difference
+ diffValues("upstream", "", normalizeUpstream(py.Upstream), normalizeUpstream(rs.Upstream), &diffs)
+ top := func(x exchange) map[string]any {
+ return map[string]any{"status": x.Status, "content_type": mediaType(x.ContentType), "transport_error": x.Error}
+ }
+ diffValues("response", "", top(py), top(rs), &diffs)
+ ps, pok := py.Body.(stream)
+ rsm, rok := rs.Body.(stream)
+ switch {
+ case pok && rok:
+ diffValues("response", ".stream", assemble(ps), assemble(rsm), &diffs)
+ diffValues("chunks", "", normalizeIDs(ps.Chunks), normalizeIDs(rsm.Chunks), &diffs)
+ case !pok && !rok:
+ diffValues("response", ".body", normalizeIDs(py.Body), normalizeIDs(rs.Body), &diffs)
+ }
+ return diffs
+}
+
+func mediaType(contentType string) string {
+ media, _, _ := strings.Cut(contentType, ";")
+ return strings.TrimSpace(media)
+}
+
+func counts(diffs []difference) (open, accepted, chunks int) {
+ for _, d := range diffs {
+ switch {
+ case d.Where == "chunks":
+ chunks++
+ case d.Known != "":
+ accepted++
+ default:
+ open++
+ }
+ }
+ return
+}
+
+func verdict(diffs []difference) string {
+ open, accepted, chunks := counts(diffs)
+ if open == 0 {
+ return fmt.Sprintf("ok (%d known, %d chunk-level)", accepted, chunks)
+ }
+ return fmt.Sprintf("DIFF %d (%d known, %d chunk-level)", open, accepted, chunks)
+}
+
+// writeReport writes a summary table and every case's open differences, and returns how many
+// cases have any.
+func writeReport(path string, outcomes []outcome) int {
+ var b strings.Builder
+ failing := 0
+ b.WriteString("# Chat completions parity: Python proxy vs Rust gateway\n\n")
+ b.WriteString("Open differences fail the run. Known ones are listed in the known-differences file with a reason. ")
+ b.WriteString("Chunk-level differences compare streams chunk by chunk and are informational: the assembled stream is what must agree.\n\n")
+ b.WriteString("| case | python | rust | open | known | chunk-level |\n|---|---|---|---|---|---|\n")
+ for _, o := range outcomes {
+ open, accepted, chunks := counts(o.Differences)
+ if open > 0 {
+ failing++
+ }
+ fmt.Fprintf(&b, "| %s | %d | %d | %d | %d | %d |\n", o.Case, o.Python.Status, o.Rust.Status, open, accepted, chunks)
+ }
+ b.WriteString("\n## Open differences\n")
+ for _, o := range outcomes {
+ var lines []string
+ for _, d := range o.Differences {
+ if d.Where == "chunks" || d.Known != "" {
+ continue
+ }
+ lines = append(lines, fmt.Sprintf("| %s | `%s` | %s | %s |", d.Where, d.Path, cell(d.Python), cell(d.Rust)))
+ }
+ if len(lines) == 0 {
+ continue
+ }
+ fmt.Fprintf(&b, "\n### %s\n\n| where | path | python | rust |\n|---|---|---|---|\n%s\n", o.Case, strings.Join(lines, "\n"))
+ }
+ must(os.WriteFile(path, []byte(b.String()), 0o644))
+ return failing
+}
+
+func cell(v any) string {
+ s, ok := v.(string)
+ if !ok {
+ raw, _ := json.Marshal(v)
+ s = string(raw)
+ }
+ if len(s) > 300 {
+ s = s[:300] + "…"
+ }
+ return "`" + strings.NewReplacer("|", "\\|", "\n", " ", "`", "'").Replace(s) + "`"
+}
+
+func must(err error) {
+ if err != nil {
+ log.Fatal(err)
+ }
+}
diff --git a/litellm-rust/loadtest/cmd/chatparity/mock.go b/litellm-rust/loadtest/cmd/chatparity/mock.go
new file mode 100644
index 00000000000..9fd2287d4c7
--- /dev/null
+++ b/litellm-rust/loadtest/cmd/chatparity/mock.go
@@ -0,0 +1,461 @@
+package main
+
+import (
+ "bytes"
+ "encoding/base64"
+ "encoding/binary"
+ "encoding/json"
+ "hash/crc32"
+ "io"
+ "net/http"
+ "regexp"
+ "sort"
+ "strings"
+ "sync"
+)
+
+// upstream is one request a proxy sent to the mock provider.
+type upstream struct {
+ Method string `json:"method"`
+ Path string `json:"path"`
+ Query string `json:"query,omitempty"`
+ Headers map[string]string `json:"headers"`
+ Body any `json:"body,omitempty"`
+}
+
+// mock answers as each provider would, the same way for both proxies, and records what it was
+// sent. Cases run one at a time, so everything that arrives belongs to the current slot.
+type mock struct {
+ mu sync.Mutex
+ current []upstream
+}
+
+func (m *mock) take() []upstream {
+ m.mu.Lock()
+ defer m.mu.Unlock()
+ got := m.current
+ m.current = nil
+ return got
+}
+
+func (m *mock) record(r *http.Request, body []byte) {
+ headers := map[string]string{}
+ for name, values := range r.Header {
+ headers[strings.ToLower(name)] = strings.Join(values, ", ")
+ }
+ var parsed any
+ if len(body) > 0 && json.Unmarshal(body, &parsed) != nil {
+ parsed = string(body)
+ }
+ m.mu.Lock()
+ m.current = append(m.current, upstream{r.Method, r.URL.Path, r.URL.RawQuery, headers, parsed})
+ m.mu.Unlock()
+}
+
+// The fixed answer parts. Everything the mock returns is deterministic, so any difference in
+// what the clients get comes from the proxies.
+const (
+ answerText = "Hello from the parity mock."
+ reasoningText = "The user wants a greeting."
+ signature = "sig_parity"
+ toolID = "call_parity_1"
+ toolArgs = `{"city": "Paris"}`
+ jsonAnswer = `{"answer": 42}`
+ inputTokens = 21
+ outputTokens = 9
+ cacheRead = 3
+ cacheWrite = 2
+ reasoningCount = 4
+)
+
+// pixel is a 1x1 PNG, served at /img.png for remote image cases.
+var pixel, _ = base64.StdEncoding.DecodeString("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==")
+
+var directive = regexp.MustCompile(`\[\[parity reply=(\w+)\]\]`)
+
+// reply is what a case asks the provider to answer with: text, tool, thinking or json.
+type reply struct {
+ kind string
+ tool string // the tool to call: the request's first tool, or the proxy's JSON-mode tool
+}
+
+func replyFor(body []byte, firstTool string) reply {
+ kind := "text"
+ if m := directive.FindSubmatch(body); m != nil {
+ kind = string(m[1])
+ }
+ // Anthropic-style JSON mode turns response_format into a forced tool; answer through it.
+ if bytes.Contains(body, []byte(`"json_tool_call"`)) {
+ return reply{kind: "tool", tool: "json_tool_call"}
+ }
+ return reply{kind: kind, tool: firstTool}
+}
+
+func (r reply) text() string {
+ if r.kind == "json" {
+ return jsonAnswer
+ }
+ return answerText
+}
+
+func (m *mock) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+ body, _ := io.ReadAll(r.Body)
+ path := r.URL.Path
+ switch {
+ case path == "/img.png":
+ w.Header().Set("Content-Type", "image/png")
+ w.Write(pixel)
+ return
+ case path == "/token":
+ writeJSON(w, map[string]any{"access_token": "ya29.parity", "expires_in": 3600, "token_type": "Bearer"})
+ return
+ }
+ m.record(r, body)
+ var req map[string]any
+ json.Unmarshal(body, &req)
+ stream, _ := req["stream"].(bool)
+ switch {
+ case strings.HasSuffix(path, "/chat/completions"):
+ openAI(w, req, replyFor(body, str(dig(req, "tools", 0, "function", "name"))), stream)
+ case strings.HasSuffix(path, "/messages"), strings.HasSuffix(path, ":rawPredict"):
+ anthropic(w, replyFor(body, str(dig(req, "tools", 0, "name"))), stream)
+ case strings.HasSuffix(path, ":streamRawPredict"):
+ anthropic(w, replyFor(body, str(dig(req, "tools", 0, "name"))), true)
+ case strings.HasSuffix(path, "/converse"):
+ bedrock(w, replyFor(body, str(dig(req, "toolConfig", "tools", 0, "toolSpec", "name"))), false)
+ case strings.HasSuffix(path, "/converse-stream"):
+ bedrock(w, replyFor(body, str(dig(req, "toolConfig", "tools", 0, "toolSpec", "name"))), true)
+ case strings.HasSuffix(path, ":generateContent"):
+ gemini(w, replyFor(body, str(dig(req, "tools", 0, "functionDeclarations", 0, "name"))), false)
+ case strings.HasSuffix(path, ":streamGenerateContent"):
+ gemini(w, replyFor(body, str(dig(req, "tools", 0, "functionDeclarations", 0, "name"))), true)
+ default:
+ w.WriteHeader(http.StatusNotFound)
+ writeJSON(w, map[string]any{"error": map[string]any{"message": "parity mock: no route for " + path}})
+ }
+}
+
+func openAI(w http.ResponseWriter, req map[string]any, rep reply, stream bool) {
+ model, _ := req["model"].(string)
+ usage := map[string]any{
+ "prompt_tokens": inputTokens, "completion_tokens": outputTokens, "total_tokens": inputTokens + outputTokens,
+ "prompt_tokens_details": map[string]any{"cached_tokens": cacheRead},
+ }
+ message := map[string]any{"role": "assistant", "content": rep.text()}
+ finish := "stop"
+ if rep.kind == "tool" {
+ message["content"] = nil
+ message["tool_calls"] = []any{map[string]any{"id": toolID, "type": "function",
+ "function": map[string]any{"name": rep.tool, "arguments": toolArgs}}}
+ finish = "tool_calls"
+ }
+ if rep.kind == "thinking" {
+ message["reasoning_content"] = reasoningText
+ }
+ if !stream {
+ writeJSON(w, map[string]any{"id": "chatcmpl-parity", "object": "chat.completion", "created": 1700000000,
+ "model": model, "choices": []any{map[string]any{"index": 0, "message": message, "finish_reason": finish}},
+ "usage": usage})
+ return
+ }
+ s := newSSE(w, "text/event-stream")
+ chunk := func(delta map[string]any, finish any) map[string]any {
+ return map[string]any{"id": "chatcmpl-parity", "object": "chat.completion.chunk", "created": 1700000000,
+ "model": model, "choices": []any{map[string]any{"index": 0, "delta": delta, "finish_reason": finish}}}
+ }
+ s.data(chunk(map[string]any{"role": "assistant", "content": ""}, nil))
+ if rep.kind == "thinking" {
+ s.data(chunk(map[string]any{"reasoning_content": reasoningText}, nil))
+ }
+ if rep.kind == "tool" {
+ s.data(chunk(map[string]any{"tool_calls": []any{map[string]any{"index": 0, "id": toolID, "type": "function",
+ "function": map[string]any{"name": rep.tool, "arguments": ""}}}}, nil))
+ for _, part := range halves(toolArgs) {
+ s.data(chunk(map[string]any{"tool_calls": []any{map[string]any{"index": 0,
+ "function": map[string]any{"arguments": part}}}}, nil))
+ }
+ } else {
+ for _, part := range halves(rep.text()) {
+ s.data(chunk(map[string]any{"content": part}, nil))
+ }
+ }
+ s.data(chunk(map[string]any{}, finish))
+ if b, _ := dig(req, "stream_options", "include_usage").(bool); b {
+ last := chunk(nil, nil)
+ last["choices"] = []any{}
+ last["usage"] = usage
+ s.data(last)
+ }
+ s.raw("data: [DONE]\n\n")
+}
+
+func anthropic(w http.ResponseWriter, rep reply, stream bool) {
+ var blocks []map[string]any
+ stop := "end_turn"
+ if rep.kind == "thinking" {
+ blocks = append(blocks, map[string]any{"type": "thinking", "thinking": reasoningText, "signature": signature})
+ }
+ if rep.kind == "tool" {
+ var input any
+ json.Unmarshal([]byte(toolArgs), &input)
+ if rep.tool == "json_tool_call" {
+ json.Unmarshal([]byte(jsonAnswer), &input)
+ }
+ blocks = append(blocks, map[string]any{"type": "tool_use", "id": toolID, "name": rep.tool, "input": input})
+ stop = "tool_use"
+ } else {
+ blocks = append(blocks, map[string]any{"type": "text", "text": rep.text()})
+ }
+ usage := map[string]any{"input_tokens": inputTokens, "output_tokens": outputTokens,
+ "cache_read_input_tokens": cacheRead, "cache_creation_input_tokens": cacheWrite}
+ if !stream {
+ content := make([]any, len(blocks))
+ for i, b := range blocks {
+ content[i] = b
+ }
+ writeJSON(w, map[string]any{"id": "msg_parity", "type": "message", "role": "assistant",
+ "model": "claude-parity", "content": content, "stop_reason": stop, "stop_sequence": nil, "usage": usage})
+ return
+ }
+ s := newSSE(w, "text/event-stream")
+ s.event("message_start", map[string]any{"type": "message_start", "message": map[string]any{
+ "id": "msg_parity", "type": "message", "role": "assistant", "model": "claude-parity", "content": []any{},
+ "stop_reason": nil, "stop_sequence": nil, "usage": map[string]any{"input_tokens": inputTokens, "output_tokens": 1,
+ "cache_read_input_tokens": cacheRead, "cache_creation_input_tokens": cacheWrite}}})
+ for i, b := range blocks {
+ start := map[string]any{}
+ for k, v := range b {
+ start[k] = v
+ }
+ var deltas []map[string]any
+ switch b["type"] {
+ case "thinking":
+ start["thinking"], start["signature"] = "", ""
+ for _, part := range halves(reasoningText) {
+ deltas = append(deltas, map[string]any{"type": "thinking_delta", "thinking": part})
+ }
+ deltas = append(deltas, map[string]any{"type": "signature_delta", "signature": signature})
+ case "tool_use":
+ start["input"] = map[string]any{}
+ input, _ := json.Marshal(b["input"])
+ for _, part := range halves(string(input)) {
+ deltas = append(deltas, map[string]any{"type": "input_json_delta", "partial_json": part})
+ }
+ default:
+ start["text"] = ""
+ for _, part := range halves(rep.text()) {
+ deltas = append(deltas, map[string]any{"type": "text_delta", "text": part})
+ }
+ }
+ s.event("content_block_start", map[string]any{"type": "content_block_start", "index": i, "content_block": start})
+ for _, d := range deltas {
+ s.event("content_block_delta", map[string]any{"type": "content_block_delta", "index": i, "delta": d})
+ }
+ s.event("content_block_stop", map[string]any{"type": "content_block_stop", "index": i})
+ }
+ s.event("message_delta", map[string]any{"type": "message_delta",
+ "delta": map[string]any{"stop_reason": stop, "stop_sequence": nil}, "usage": map[string]any{"output_tokens": outputTokens}})
+ s.event("message_stop", map[string]any{"type": "message_stop"})
+}
+
+func bedrock(w http.ResponseWriter, rep reply, stream bool) {
+ stop := "end_turn"
+ var input any
+ json.Unmarshal([]byte(toolArgs), &input)
+ if rep.tool == "json_tool_call" {
+ json.Unmarshal([]byte(jsonAnswer), &input)
+ }
+ usage := map[string]any{"inputTokens": inputTokens, "outputTokens": outputTokens, "totalTokens": inputTokens + outputTokens,
+ "cacheReadInputTokens": cacheRead, "cacheWriteInputTokens": cacheWrite}
+ if rep.kind == "tool" {
+ stop = "tool_use"
+ }
+ if !stream {
+ var content []any
+ if rep.kind == "thinking" {
+ content = append(content, map[string]any{"reasoningContent": map[string]any{
+ "reasoningText": map[string]any{"text": reasoningText, "signature": signature}}})
+ }
+ if rep.kind == "tool" {
+ content = append(content, map[string]any{"toolUse": map[string]any{"toolUseId": toolID, "name": rep.tool, "input": input}})
+ } else {
+ content = append(content, map[string]any{"text": rep.text()})
+ }
+ writeJSON(w, map[string]any{"output": map[string]any{"message": map[string]any{"role": "assistant", "content": content}},
+ "stopReason": stop, "usage": usage, "metrics": map[string]any{"latencyMs": 1}})
+ return
+ }
+ w.Header().Set("Content-Type", "application/vnd.amazon.eventstream")
+ w.WriteHeader(http.StatusOK)
+ frame := func(event string, payload any) {
+ b, _ := json.Marshal(payload)
+ w.Write(eventStreamFrame(event, b))
+ if f, ok := w.(http.Flusher); ok {
+ f.Flush()
+ }
+ }
+ frame("messageStart", map[string]any{"role": "assistant"})
+ index := 0
+ if rep.kind == "thinking" {
+ for _, part := range halves(reasoningText) {
+ frame("contentBlockDelta", map[string]any{"contentBlockIndex": index,
+ "delta": map[string]any{"reasoningContent": map[string]any{"text": part}}})
+ }
+ frame("contentBlockDelta", map[string]any{"contentBlockIndex": index,
+ "delta": map[string]any{"reasoningContent": map[string]any{"signature": signature}}})
+ frame("contentBlockStop", map[string]any{"contentBlockIndex": index})
+ index++
+ }
+ if rep.kind == "tool" {
+ frame("contentBlockStart", map[string]any{"contentBlockIndex": index,
+ "start": map[string]any{"toolUse": map[string]any{"toolUseId": toolID, "name": rep.tool}}})
+ args, _ := json.Marshal(input)
+ for _, part := range halves(string(args)) {
+ frame("contentBlockDelta", map[string]any{"contentBlockIndex": index,
+ "delta": map[string]any{"toolUse": map[string]any{"input": part}}})
+ }
+ } else {
+ for _, part := range halves(rep.text()) {
+ frame("contentBlockDelta", map[string]any{"contentBlockIndex": index, "delta": map[string]any{"text": part}})
+ }
+ }
+ frame("contentBlockStop", map[string]any{"contentBlockIndex": index})
+ frame("messageStop", map[string]any{"stopReason": stop})
+ frame("metadata", map[string]any{"usage": usage, "metrics": map[string]any{"latencyMs": 1}})
+}
+
+func gemini(w http.ResponseWriter, rep reply, stream bool) {
+ var parts []any
+ if rep.kind == "thinking" {
+ parts = append(parts, map[string]any{"text": reasoningText, "thought": true})
+ }
+ if rep.kind == "tool" {
+ var args any
+ json.Unmarshal([]byte(toolArgs), &args)
+ parts = append(parts, map[string]any{"functionCall": map[string]any{"name": rep.tool, "args": args}})
+ } else {
+ parts = append(parts, map[string]any{"text": rep.text()})
+ }
+ usage := map[string]any{"promptTokenCount": inputTokens, "candidatesTokenCount": outputTokens,
+ "totalTokenCount": inputTokens + outputTokens, "cachedContentTokenCount": cacheRead}
+ if rep.kind == "thinking" {
+ usage["thoughtsTokenCount"] = reasoningCount
+ usage["totalTokenCount"] = inputTokens + outputTokens + reasoningCount
+ }
+ candidate := func(parts []any, finish any) map[string]any {
+ c := map[string]any{"content": map[string]any{"role": "model", "parts": parts}, "index": 0}
+ if finish != nil {
+ c["finishReason"] = finish
+ }
+ return c
+ }
+ if !stream {
+ writeJSON(w, map[string]any{"candidates": []any{candidate(parts, "STOP")}, "usageMetadata": usage,
+ "modelVersion": "gemini-parity", "responseId": "resp_parity"})
+ return
+ }
+ s := newSSE(w, "text/event-stream")
+ for i, part := range parts {
+ var finish any
+ if i == len(parts)-1 {
+ finish = "STOP"
+ }
+ event := map[string]any{"candidates": []any{candidate([]any{part}, finish)}, "modelVersion": "gemini-parity",
+ "responseId": "resp_parity"}
+ if finish != nil {
+ event["usageMetadata"] = usage
+ }
+ s.data(event)
+ }
+}
+
+type sse struct{ w http.ResponseWriter }
+
+func newSSE(w http.ResponseWriter, contentType string) sse {
+ w.Header().Set("Content-Type", contentType)
+ w.WriteHeader(http.StatusOK)
+ return sse{w}
+}
+
+func (s sse) raw(text string) {
+ io.WriteString(s.w, text)
+ if f, ok := s.w.(http.Flusher); ok {
+ f.Flush()
+ }
+}
+
+func (s sse) data(v any) {
+ b, _ := json.Marshal(v)
+ s.raw("data: " + string(b) + "\n\n")
+}
+
+func (s sse) event(name string, v any) {
+ b, _ := json.Marshal(v)
+ s.raw("event: " + name + "\ndata: " + string(b) + "\n\n")
+}
+
+// eventStreamFrame encodes one AWS event-stream message, as mockllm does.
+func eventStreamFrame(eventType string, payload []byte) []byte {
+ var headers bytes.Buffer
+ for _, h := range [][2]string{{":event-type", eventType}, {":content-type", "application/json"}, {":message-type", "event"}} {
+ headers.WriteByte(byte(len(h[0])))
+ headers.WriteString(h[0])
+ headers.WriteByte(7)
+ binary.Write(&headers, binary.BigEndian, uint16(len(h[1])))
+ headers.WriteString(h[1])
+ }
+ total := 12 + headers.Len() + len(payload) + 4
+ frame := make([]byte, 0, total)
+ frame = binary.BigEndian.AppendUint32(frame, uint32(total))
+ frame = binary.BigEndian.AppendUint32(frame, uint32(headers.Len()))
+ frame = binary.BigEndian.AppendUint32(frame, crc32.ChecksumIEEE(frame))
+ frame = append(frame, headers.Bytes()...)
+ frame = append(frame, payload...)
+ return binary.BigEndian.AppendUint32(frame, crc32.ChecksumIEEE(frame))
+}
+
+// halves splits text in two, so every stream carries more than one delta per block.
+func halves(text string) []string {
+ mid := len(text) / 2
+ return []string{text[:mid], text[mid:]}
+}
+
+func writeJSON(w http.ResponseWriter, v any) {
+ w.Header().Set("Content-Type", "application/json")
+ json.NewEncoder(w).Encode(v)
+}
+
+// dig walks maps by key and slices by index, returning nil when anything is missing.
+func dig(v any, path ...any) any {
+ for _, step := range path {
+ switch s := step.(type) {
+ case string:
+ m, ok := v.(map[string]any)
+ if !ok {
+ return nil
+ }
+ v = m[s]
+ case int:
+ a, ok := v.([]any)
+ if !ok || s >= len(a) {
+ return nil
+ }
+ v = a[s]
+ }
+ }
+ return v
+}
+
+func str(v any) string {
+ s, _ := v.(string)
+ return s
+}
+
+func sortedKeys[V any](m map[string]V) []string {
+ keys := make([]string, 0, len(m))
+ for k := range m {
+ keys = append(keys, k)
+ }
+ sort.Strings(keys)
+ return keys
+}
diff --git a/litellm-rust/loadtest/scripts/chat-parity.sh b/litellm-rust/loadtest/scripts/chat-parity.sh
new file mode 100755
index 00000000000..56866d5dab1
--- /dev/null
+++ b/litellm-rust/loadtest/scripts/chat-parity.sh
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# Runs the chat completions parity check locally: writes one config pointed at chatparity's mock,
+# starts the Python proxy and the Rust gateway on it, sends every case through both and writes
+# out/report.md. Exits non-zero when there are unexplained differences.
+#
+# scripts/chat-parity.sh # every case
+# FILTER=anthropic/ scripts/chat-parity.sh
+#
+# Needs: go, cargo, and a Python environment with the proxy and google-auth installed
+# (`uv pip install -e '.[proxy]' google-auth` from the repo root). PYTHON picks the
+# interpreter; GATEWAY skips the cargo build.
+set -euo pipefail
+
+here="$(cd "$(dirname "$0")/.." && pwd)"
+repo="$(cd "$here/../.." && pwd)"
+out="${OUT:-$here/out/chat-parity}"
+mock_port="${MOCK_PORT:-9100}" python_port="${PYTHON_PORT:-4001}" rust_port="${RUST_PORT:-4002}"
+python="${PYTHON:-python3}"
+mkdir -p "$out"
+
+(cd "$here" && go build -o "$out/chatparity" ./cmd/chatparity)
+if [[ -z "${GATEWAY:-}" ]]; then
+ (cd "$repo/litellm-rust" && cargo build -p litellm-gateway)
+ GATEWAY="${CARGO_TARGET_DIR:-$repo/litellm-rust/target}/debug/litellm-gateway"
+fi
+"$out/chatparity" config -mock "http://127.0.0.1:$mock_port" -out "$out" >/dev/null
+
+# Credentials for the bedrock-env deployment, which takes them from the environment. They only
+# ever reach the mock.
+export AWS_ACCESS_KEY_ID=AKIAPARITYPARITY AWS_SECRET_ACCESS_KEY=parity-secret AWS_REGION_NAME=us-east-1 AWS_REGION=us-east-1
+unset AWS_SESSION_TOKEN AWS_PROFILE
+
+pids=()
+trap 'kill "${pids[@]}" 2>/dev/null || true' EXIT
+# Both price from the repo's cost map, so parameter support and defaults come from the same data.
+LITELLM_LOCAL_MODEL_COST_MAP=True "$python" -m litellm.proxy.proxy_cli --config "$out/config.yaml" \
+ --port "$python_port" >"$out/python.log" 2>&1 &
+pids+=($!)
+LITELLM_MODEL_COST_MAP_PATH="$repo/model_prices_and_context_window.json" LITELLM_CONFIG="$out/config.yaml" \
+ HOST=127.0.0.1 PORT="$rust_port" "$GATEWAY" >"$out/rust.log" 2>&1 &
+pids+=($!)
+
+for url in "http://127.0.0.1:$python_port/health/liveliness" "http://127.0.0.1:$rust_port/health/liveliness"; do
+ for _ in $(seq 120); do curl -sf "$url" >/dev/null && break; sleep 1; done
+ curl -sf "$url" >/dev/null || { echo "not up: $url (see $out/*.log)" >&2; exit 1; }
+done
+
+"$out/chatparity" run -mock-addr "127.0.0.1:$mock_port" -python "http://127.0.0.1:$python_port" \
+ -rust "http://127.0.0.1:$rust_port" -out "$out" -known "$here/cmd/chatparity/known.json" -filter "${FILTER:-}"