mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-10 08:28:25 +00:00
feat(loadtest): chat completions parity check between the python proxy and the gateway (#47362)
* feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough
Adds azure/<deployment> chat completions, streaming and not, on the same
passthrough the openai/ route uses. The request goes to the deployment
path with the api-version query, or to /openai/v1/ for the v1, latest and
preview versions, with api_base, api_key and api_version falling back to
the AZURE_* variables and os.environ references resolved as the proxy
does. A key is sent as api-key, and an azure_ad_token or an Entra ID
token from auth-azure goes as a Bearer token. The body follows the
proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the
deployment name and reading model capabilities from the cost map, so an
unsupported param is refused unless drop_params allows it to be dropped.
Spend is priced as the proxy prices Azure, by azure/<base_model> when one
is set and otherwise by the model the response names, and a content
filter error keeps Azure's inner error for the client
* feat(gateway): Anthropic and Bedrock chat completions streaming
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
* feat(gateway): full openai chat request translation for anthropic and bedrock (#47088)
* feat(gateway): share router cooldowns and follow proxy key invalidations through Redis
Gateway replicas now share state through the coordination Redis the
Python proxy uses (general_settings.coordination_redis, a redis
response cache, or REDIS_*):
- A deployment one replica cools down is written under Python's
deployment:<id>:cooldown key and applied by every other replica's
background poll (one MGET a second), so the fleet stops sending to a
failing deployment together. allowed_fails counts failures across
replicas, as Python's v1 rule does; the v2 failure-rate rule stays
per replica, as in Python. Deployments are named by model_info.id, or
an id derived from the config that is the same on every replica.
- The virtual key cache follows the proxy's
litellm_proxy.auth_cache_invalidation channel, so a key blocked,
deleted or edited through the admin API (or its team or user) is
re-read on every replica at once instead of after up to 60 s.
general_settings.user_api_key_cache_ttl sets the key TTL.
Nothing is added to the path of a successful request. Without Redis,
or while it is down, each replica keeps its own state as before.
Redis Cluster and Sentinel are refused at startup for now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): hooks and observers around inference requests, with usage and cost
Adds the litellm-gateway-hooks crate: request facts built only when something
reads them, on-path hooks that declare what they read and what they may
change and run in config order, off-path observers on bounded queues, a
usage tap for SSE and JSON responses, and per-deployment prices from the
model cost map overlaid by config prices. The messages and chat handlers run
through it when anything is installed; with nothing installed requests take
the old path.
The key store now also returns the key alias, user, team alias and key and
team metadata, and router attempts carry the deployment id.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): write spend to the proxy's database from a native observer
Adds a spend writer observer that records every gateway request in the
proxy's own Postgres tables with the same statements the Python proxy
uses: spend logs, key/user/team/team member/end user spend, and the
daily user, team and end-user activity tables. Writes are batched on the
proxy's timers and are additive, so replicas and Python proxies can
write side by side. The database pool is now shared between virtual
keys and the writer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): enforce key, team, user, member and end-user budgets on shared spend counters
Adds a built-in budgets hook that runs the proxy's budget checks, in its
order and with its messages, on the proxy's Redis spend:* counters. Each
request reads and seeds the counters, checks them and reserves its
maximum cost in one Redis script, then settles the reservation to its
real cost off the request path, so replicas and Python proxies sharing
the Redis enforce one budget. Without Redis the counters are kept per
replica. The key store now reads user, team member and end-user budgets,
and auth no longer checks budgets against stored spend.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): enforce key, user, team, member and end-user rate limits on shared counters
Adds a built-in rate limits hook with the proxy's v3 limiter semantics:
requests per window, tokens per window and in-flight requests per key,
on the proxy's own {scope:id}:window|requests|tokens|max_parallel_requests
Redis keys and with its scripts, so replicas and Python proxies sharing
the Redis enforce one limit. Tokens are reserved from the proxy's
estimate and settled to usage when the response ends. Refusals carry the
proxy's message, body and headers on each endpoint, and complete
responses carry its x-ratelimit-* headers. Without Redis the counters are
kept per replica. The key store now reads key, user, team and member
rpm/tpm limits and the key's max_parallel_requests, and end users' budget
limits are read with their budgets; keys with plain rate limits are no
longer refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): configurable header, spend metadata and provider metadata hooks
Adds four built-in hooks a config lists under litellm_settings.hooks and
runs in that order: forward_headers copies named client headers to the
provider, strip_headers removes headers from the provider request
(including the gateway's default pass-through), spend_metadata adds
header and key-metadata values to the spend log's spend_logs_metadata,
and provider_metadata sets Messages body fields and headers from
templates over request facts. Hooks' upstream header edits now reach the
provider on both endpoints, and body edits can set nested fields.
Credentials and signer headers are never forwarded or set, and a
misconfigured hook stops the gateway at startup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): model capacity hook, the proxy's dynamic rate limiter
Adds the model_capacity hook, turned on by listing dynamic_rate_limiter_v3
(or model_capacity) in litellm_settings.callbacks. It caps each model
group at the sum of its deployments' rpm and tpm across every replica,
and with priority_reservation holds each priority's pool to its share
once the group's saturation reaches the threshold, counting it below.
It uses the proxy's model_saturation_check and priority_model keys and
scripts, its refusal messages and headers, and adds a successful
response's tokens to the group and the pool admission chose. The rate
limit counters gain a general check-and-add over request and token
pairs, and the throttling refusal shape is shared.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* test(gateway-auth): expect the narrowed unsupported rate limit message
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): prometheus observer with the proxy's metric and label names
Adds the prometheus observer, turned on by listing prometheus in
litellm_settings.callbacks, success_callback or failure_callback. Each
finished request updates the proxy's request, spend, token, latency and
deployment metrics under its names and labels, and /metrics serves them
after the gateway's own. prometheus_metrics_config, the exclude lists
and prometheus_latency_buckets are read as the proxy reads them. The
metrics the gateway already served under the same names come from the
observer alone when it is on. Outcomes now carry when the attempt was
sent upstream, for the LLM API latency and time to first token.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): s3_v2 observer writing the proxy's logging payload to S3
Adds the s3_v2 observer, turned on by listing s3_v2 (or s3_log) in
litellm_settings.callbacks, success_callback or failure_callback. Each
finished request becomes the proxy's StandardLoggingPayload, with the
messages from the request body and the response as the client got it,
streams put back together. It is queued and written every flush interval
under the proxy's key layout, with its Content-MD5, Content-Disposition
and encryption headers, one PUT per request or one NDJSON object per
folder with s3_batch_file_upload. s3_callback_params is read as the
proxy reads it, os.environ references included. Uploads are retried,
dropped and kept for the next flush by the proxy's rules, and what is
queued is written at shutdown. auth-aws gains an S3 signer that signs
the path as sent and every header, as botocore does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5
* feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough
Adds azure/<deployment> chat completions, streaming and not, on the same
passthrough the openai/ route uses. The request goes to the deployment
path with the api-version query, or to /openai/v1/ for the v1, latest and
preview versions, with api_base, api_key and api_version falling back to
the AZURE_* variables and os.environ references resolved as the proxy
does. A key is sent as api-key, and an azure_ad_token or an Entra ID
token from auth-azure goes as a Bearer token. The body follows the
proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the
deployment name and reading model capabilities from the cost map, so an
unsupported param is refused unless drop_params allows it to be dropped.
Spend is priced as the proxy prices Azure, by azure/<base_model> when one
is set and otherwise by the model the response names, and a content
filter error keeps Azure's inner error for the client
* feat(gateway): Anthropic and Bedrock chat completions streaming
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
* feat(gateway): full OpenAI chat request translation for Anthropic and Bedrock
Anthropic and Bedrock Converse deployments now take the whole OpenAI chat request the
Python proxy accepts instead of a narrow allowlist. Messages translate as Python builds
them: system as a string, parts or several messages, text, image_url (URL or data URL),
file and document parts, assistant tool_calls, the tool and function roles, consecutive
role merging, empty content, cache_control on Anthropic and cachePoint on Bedrock.
Params map per provider: max_tokens and max_completion_tokens with Anthropic's model
default, temperature, top_p, stop, tools and tool_choice, parallel_tool_calls,
response_format as a json tool or native output format, reasoning_effort and thinking,
user as metadata, with n and any other unsupported param refused with a 400 unless
drop_params is set globally, on the deployment or on the request, and
allowed_openai_params and additional_drop_params honoured. Responses carry tool calls,
thinking blocks and response_format answers back in OpenAI shape, streamed and not. The
route downloads media URLs a provider cannot take by reference, merges the beta headers
a request needs with forwarded ones, and the gateway hands the cost map to both
providers for the capability rules that read it.
* fix(rust-bridge): read Python's model_cost for chat translation capabilities
Chat translation for Anthropic and Bedrock reads capability flags from the
cost map. The Python bridge passed none, so every SDK chat call through the
Rust path failed. It now reads litellm.model_cost live, so lookups match
Python's, runtime-registered models included. ModelInfo::entry returns a Cow
so a map read from Python can hand over a copy.
Two tests sent temperature != 1 to a model that removed sampling params,
which Python itself refuses; they now send temperature=1.
* fix(model-catalog): read token counts the cost map writes as whole floats
The published cost map now has prompt_cache_min_tokens: 4096.0, which the
typed u64 fields rejected, so the gateway dropped the whole map at startup
and priced every request at zero. Counts accept any whole non-negative
number, as Python reads any JSON number, and a test keeps the checked-in
map parsing.
* feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough (#47074)
Adds azure/<deployment> chat completions, streaming and not, on the same
passthrough the openai/ route uses. The request goes to the deployment
path with the api-version query, or to /openai/v1/ for the v1, latest and
preview versions, with api_base, api_key and api_version falling back to
the AZURE_* variables and os.environ references resolved as the proxy
does. A key is sent as api-key, and an azure_ad_token or an Entra ID
token from auth-azure goes as a Bearer token. The body follows the
proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the
deployment name and reading model capabilities from the cost map, so an
unsupported param is refused unless drop_params allows it to be dropped.
Spend is priced as the proxy prices Azure, by azure/<base_model> when one
is set and otherwise by the model the response names, and a content
filter error keeps Azure's inner error for the client
* feat(gateway): Anthropic and Bedrock chat completions streaming (#47077)
Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/
deployments, which were refused with 501 before. The request is the
non-streaming one with "stream": true added for Anthropic and sent to
/converse-stream for Bedrock, and the provider's events are decoded the
way the proxy's chunk parsers decode them: text, tool call deltas with
empty arguments sent as "{}", reasoning_content and thinking_blocks with
their index and signature, citations and the mapped finish_reason. A
wrapper in the shape of the proxy's stream wrapper gives every chunk one
id and the client's model name, sends role on the first chunk, adds the
empty finish chunk, strips usage from the chunks and, when
stream_options.include_usage is set, ends with the usage chunk before
[DONE]. The gateway waits for the first chunk before answering, so an
upstream error up to there is retried or falls back to another
deployment, and an error after it ends the stream with the OpenAI error
frame. Usage, cache reads and writes included, is handed to the meter
through the response whether or not the client asked for it, and
dropping the client stream drops the upstream request
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(gateway): gemini chat completions on google ai studio and vertex ai (#47098)
* feat(gateway): Gemini chat completions on Google AI Studio and Vertex AI
gemini/ deployments and vertex_ai/ deployments of Gemini models now serve
/v1/chat/completions, streamed and not, instead of being refused. Both surfaces share
one translation ported from the Python VertexGeminiConfig. Params map as Python maps
them: temperature, top_p, max_tokens and max_completion_tokens, n, stop, penalties
where the model takes them, seed, logprobs, tools with function declarations and the
built in search, code execution, url context, maps and computer use tools,
tool_choice, response_format as a response schema, reasoning_effort and thinking as a
thinking config, modalities, audio and labels. Messages become contents with a system
instruction when the cost map allows it, images, files, audio, function calls and
responses, and thought signatures carried back on tool call ids
AI Studio sends the key in x-goog-api-key from the deployment or GOOGLE_API_KEY,
GEMINI_API_KEY or PALM_API_KEY, and calls v1alpha for Gemini 3 and v1beta otherwise.
Vertex AI calls the project and location URL with a forwarded bearer or a Google
access token from vertex_credentials, the environment or ADC, and grafts the path onto
a custom api_base. Responses carry text, reasoning, tool calls, grounding annotations,
logprobs and the vertex_ai metadata, a blocked prompt or flagged answer comes back as
content_filter, and usage counts thoughts and cached tokens as Python does so cost is
recorded. The gateway resolves os.environ references in the deployment key, base and
Vertex params, passes the deployment Vertex params to the call, and loads the cost map
for gemini and vertex_ai deployments. Claude and partner models on Vertex AI stay
refused
* test(gateway): gemini cost map test double returns a Cow entry
* feat(gateway): native chat for openai-compatible providers (#47099)
Serve chat completions for hosted_vllm, nvidia_nim, groq, together_ai,
fireworks_ai, xai, mistral, deepseek, openrouter, cerebras and perplexity
through the OpenAI passthrough, with each provider's Python config ported
as a request transform in llms and its response and stream fixups applied
on the way back.
Streams ask for stream_options.include_usage on the client's behalf where
the provider takes it, record the usage through ReportedUsage for the spend
hooks, and keep it from the client unless the client asked for it.
* feat(gateway): claude chat completions on vertex ai (#47101)
vertex_ai/ deployments of Claude models now serve /v1/chat/completions,
streamed and not, instead of being refused. The request is the Anthropic
chat translation built for the vertex_ai host, as Python's
VertexAIAnthropicConfig builds it on top of AnthropicConfig: capabilities
read the vertex_ai/ cost map entry first and native structured output
reads only that entry, service_tier and speed are refused or dropped with
drop_params, and the Anthropic API betas are not added. The body drops
model, carries anthropic_version vertex-2023-10-16, folds output_format
into output_config and keeps effort only for models that take it, and
names its context management and caller betas in anthropic_beta and the
anthropic-beta header, filtered to what Vertex accepts, with the web
search beta alone when there are none.
The call goes to the publisher rawPredict URL for the project and
location, regional or global, or a custom api_base or VERTEXAI_API_BASE,
with a Google access token from vertex_credentials, the environment or
ADC. Streams call streamRawPredict with stream true, alt=sse on a custom
base, decode the Anthropic events as the anthropic/ path does and hand
usage to the hooks through ReportedUsage. Every remote image is
downloaded first, since Vertex takes none by URL, and the response names
the requested model.
* fix(gateway): find anthropic stream usage by event name (#47206)
The stream meter kept a data line only when "message_start" or
"message_delta" appeared in its first 96 bytes. An upstream that puts
"type" after the message object (JSON key order is not fixed) had every
message_start skipped, so input tokens were recorded as 0 and spend,
budgets, rate limits, metrics and S3 logs undercounted.
Anthropic streams name every event, so the event: line now decides; the
byte-window check remains for streams without event lines.
* fix(gateway): stop hooks pinning connection read buffers for a stream's life (#47220)
The s3 logger asks for the response text, and the tap kept every
response chunk as a Bytes slice of the upstream connection's read
buffer, so each open stream held a buffer per chunk until it ended
(about 460 KB per open 10k-token stream locally). Chunks are now copied
into one owned buffer.
The same applies on the request side: the facts kept the raw request
slice and a clone of the client headers, both slices of the client
connection's read buffer. The body is now released once hooks have run
(an owned copy is kept only when an observer reads it) and the headers
are copied.
* feat(loadtest): full-feature mode with recorded-data verification (#47207)
* feat(loadtest): full-feature mode with recorded-data verification
genconfig -features full turns on what both gateways do per request:
owners with rpm/tpm/budgets/max_parallel, deployment capacity, the
dynamic_rate_limiter_v3, prometheus and s3_v2 callbacks, and the four
request hooks with Python stand-ins (custom_callbacks.py). -key-sets
gives each gateway its own keys, users and teams in a shared database.
verify checks spend logs, owner spend, daily tables, Prometheus counters,
S3 logs and the headers the provider received against what loadgen sent,
and compares per-deployment prices across sides. s3stub serves S3 for
the logs. mockllm takes in= for exact input tokens, returns unique
response ids, and counts chosen request headers; loadgen takes -keys
and -header.
* feat(loadtest): s3stub -max-string keeps long runs' logs in memory
* feat(loadtest): bifrost as a third side of the full-feature run (#47236)
genconfig writes a "bifrost" key set as a Bifrost config: a custom provider
per deployment, a routing rule per model group, virtual keys, and with
-features full key/team/customer budgets and rate limits, model configs for
deployment limits, the logs store with S3 payloads, the telemetry plugin and
header forwarding/stripping/logging. A pricing file copies the cost map's
prices per deployment.
mockllm answers Bedrock Converse (streamed and not), serves a self-signed
TLS listener for Bifrost's HTTPS-only Bedrock provider, and echoes the
request's model, which Bifrost prices by. s3stub answers HeadBucket.
loadgen takes -chat-path and -messages-path for Bifrost's
/anthropic/v1/messages.
verify reads a side named bifrost in Bifrost's schema and reports what it
has no counterpart for (users, members, daily tables, provider_metadata,
cross-replica state) as n/a; with three sides it compares every pair.
Claude-Session: https://claude.ai/code/session_01XF7rGf49TWYH5Hqkb14eSL
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* feat(loadtest): chat completions parity check between the python proxy and the gateway
chatparity sends the same /v1/chat/completions requests through the Python
proxy and the Rust gateway, both loading one config pointed at its mock
provider, and diffs what each provider was sent (path, headers, body) and
what the client got back (status, body, the assembled stream, and the
stream chunk by chunk as information).
The mock answers OpenAI-compatible, Azure, Anthropic, Bedrock Converse
(event-stream framing), Gemini, Vertex Gemini and Vertex Claude the same
way for both proxies, and issues Vertex access tokens for a throwaway
service account. Cases cover 19 deployments and 15 request scenarios:
text, multi-turn, tools, tool results, response_format, images,
reasoning_effort, cache_control, unsupported params refused and dropped,
max_completion_tokens, and streams with and without include_usage.
scripts/chat-parity.sh builds and starts everything locally and writes
report.md plus one capture per case. Reviewed differences go in
cmd/chatparity/known.json with a reason; the run fails on the rest.
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Nate Armstrong
GitHub
parent
6f345e3eb9
commit
0ac7a383db
9 changed files with 1529 additions and 0 deletions
41
litellm-rust/docs/customer-proxy/chat-parity.svg
Normal file
41
litellm-rust/docs/customer-proxy/chat-parity.svg
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 880 340" font-family="-apple-system, Segoe UI, Helvetica, Arial, sans-serif" font-size="13">
|
||||
<rect width="880" height="340" fill="#ffffff"/>
|
||||
<defs>
|
||||
<marker id="a" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
|
||||
<path d="M0 0 L10 5 L0 10 z" fill="#555"/>
|
||||
</marker>
|
||||
</defs>
|
||||
<text x="440" y="26" text-anchor="middle" font-size="15" font-weight="600" fill="#222">chatparity: one request, two proxies, one recording mock</text>
|
||||
|
||||
<rect x="20" y="120" width="170" height="100" rx="8" fill="#eef2ff" stroke="#6366f1"/>
|
||||
<text x="105" y="148" text-anchor="middle" font-weight="600" fill="#222">chatparity run</text>
|
||||
<text x="105" y="170" text-anchor="middle" fill="#444">340 cases:</text>
|
||||
<text x="105" y="188" text-anchor="middle" fill="#444">19 deployments x</text>
|
||||
<text x="105" y="206" text-anchor="middle" fill="#444">15 scenarios x mode</text>
|
||||
|
||||
<rect x="300" y="60" width="200" height="70" rx="8" fill="#fef3c7" stroke="#d97706"/>
|
||||
<text x="400" y="90" text-anchor="middle" font-weight="600" fill="#222">Python proxy</text>
|
||||
<text x="400" y="110" text-anchor="middle" fill="#444">litellm --config config.yaml</text>
|
||||
|
||||
<rect x="300" y="210" width="200" height="70" rx="8" fill="#dcfce7" stroke="#16a34a"/>
|
||||
<text x="400" y="240" text-anchor="middle" font-weight="600" fill="#222">Rust gateway</text>
|
||||
<text x="400" y="260" text-anchor="middle" fill="#444">same config.yaml</text>
|
||||
|
||||
<rect x="610" y="120" width="250" height="100" rx="8" fill="#f3f4f6" stroke="#6b7280"/>
|
||||
<text x="735" y="146" text-anchor="middle" font-weight="600" fill="#222">mock provider (in chatparity)</text>
|
||||
<text x="735" y="166" text-anchor="middle" fill="#444">OpenAI-compatible, Azure, Anthropic,</text>
|
||||
<text x="735" y="184" text-anchor="middle" fill="#444">Bedrock Converse, Gemini, Vertex</text>
|
||||
<text x="735" y="204" text-anchor="middle" fill="#444">records every request it gets</text>
|
||||
|
||||
<path d="M190 150 L298 100" stroke="#555" fill="none" marker-end="url(#a)"/>
|
||||
<path d="M190 190 L298 240" stroke="#555" fill="none" marker-end="url(#a)"/>
|
||||
<path d="M500 95 L608 150" stroke="#555" fill="none" marker-end="url(#a)"/>
|
||||
<path d="M500 245 L608 190" stroke="#555" fill="none" marker-end="url(#a)"/>
|
||||
<text x="225" y="112" fill="#555">1. send</text>
|
||||
<text x="225" y="240" fill="#555">2. send</text>
|
||||
<text x="560" y="110" fill="#555">upstream A</text>
|
||||
<text x="560" y="240" fill="#555">upstream B</text>
|
||||
|
||||
<rect x="20" y="290" width="840" height="36" rx="6" fill="#fff7ed" stroke="#ea580c"/>
|
||||
<text x="440" y="313" text-anchor="middle" fill="#222">diff upstream A vs B (path, headers, body) and response A vs B (status, body, assembled stream) → report.md</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.7 KiB |
|
|
@ -12,6 +12,7 @@ machines.
|
|||
| `summarize` | Merges result files and prints proxy overhead (observed minus planned time) by stage, class and context size. |
|
||||
| `procmon` | Samples the gateway process's CPU, RSS, fds and TCP connections from `/proc`. |
|
||||
| `verify` | Checks what each gateway recorded for a run against what loadgen sent: spend logs, key/team/user/member spend, daily tables, Prometheus counters, S3 request logs and the headers the provider received. See [Full-feature runs](#full-feature-runs). |
|
||||
| `chatparity` | Sends the same chat completions requests through the Python proxy and the gateway, both pointed at its recording mock provider, and reports every difference in what the provider was sent and what the client got. See [Chat parity](#chat-parity). |
|
||||
| `s3stub` | In-memory path-style S3 (PUT, GET, ListObjectsV2) for the `s3_v2` logs of a full-feature run. |
|
||||
|
||||
Workloads live in `workloads/`: `smoke`, `realistic-ramp`, `long-streams`, `large-context`,
|
||||
|
|
@ -185,3 +186,49 @@ carry static dummy AWS keys, so every request is SigV4-signed, and mockllm answe
|
|||
`/model/<id>/invoke` and `/invoke-with-response-stream` with AWS event-stream framing that the
|
||||
gateway re-encodes as Anthropic SSE. Bedrock hits show up as `bedrock-<g>-<i>` in the mock's `/stats`.
|
||||
|
||||
## Chat parity
|
||||
|
||||
`scripts/chat-parity.sh` checks the gateway's `/v1/chat/completions` against the Python proxy's.
|
||||
It writes one config, starts both proxies on it, and sends every case through each one in turn.
|
||||
Both talk to `chatparity`'s mock, which answers like each provider's API (OpenAI-compatible, Azure,
|
||||
Anthropic, Bedrock Converse, Gemini, Vertex Gemini and Vertex Claude) and records every request it
|
||||
receives.
|
||||
|
||||
```sh
|
||||
scripts/chat-parity.sh # every case, report in out/chat-parity/report.md
|
||||
FILTER=bedrock-env/ scripts/chat-parity.sh # one deployment, scenario or mode
|
||||
GATEWAY=<built binary> PYTHON=<venv>/bin/python scripts/chat-parity.sh
|
||||
```
|
||||
|
||||
A case is one deployment, one request scenario and one mode (`<deployment>/<scenario>/<unary|stream>`).
|
||||
There are 19 deployments (every chat provider the gateway serves natively) and 15 scenarios: plain
|
||||
text, multi-turn with a system message, tools, a tool result round trip, `response_format`
|
||||
(JSON schema and JSON object), inline and remote images, `reasoning_effort`, `cache_control`,
|
||||
unsupported params refused and then dropped, `max_completion_tokens`, and streams with and without
|
||||
`include_usage`. The mock gives both proxies the same answer for the same request (text, a tool
|
||||
call, thinking or JSON, picked by a `[[parity reply=...]]` marker in the prompt), so any difference
|
||||
in what the client gets comes from the proxies.
|
||||
|
||||
For each case it compares:
|
||||
|
||||
- **upstream**: the method, path, query, headers and JSON body the provider received. Transport
|
||||
headers (user agent, content length, SDK telemetry) are ignored, and a SigV4 signature is reduced
|
||||
to its region and service.
|
||||
- **response**: the status, content type and JSON body the client got. Ids and timestamps the
|
||||
proxies generate are compared by prefix only (`chatcmpl-*`). For a stream, this is what a client
|
||||
library would assemble: content, reasoning, tool calls, finish reasons, usage, the models named,
|
||||
whether one id spans the stream, and `[DONE]`.
|
||||
- **chunks**: the stream chunk by chunk. This is informational only, since the same content can be
|
||||
split into chunks differently.
|
||||
|
||||
`out/chat-parity/report.md` lists every open difference, and `cases/<case>.json` holds the request,
|
||||
both responses and everything each proxy sent upstream. Once a difference has been reviewed and
|
||||
accepted, add it to `cmd/chatparity/known.json` (`{"case": "*/stream", "where": "upstream", "path":
|
||||
"*.accept", "reason": "..."}`, where `*` matches anything). The run fails only on differences that
|
||||
aren't listed there.
|
||||
|
||||
The Python side needs the proxy and `google-auth` installed (`uv pip install -e '.[proxy]'
|
||||
google-auth`). Both price from the repo's `model_prices_and_context_window.json`, so parameter
|
||||
support and defaults come from the same data. `bedrock` carries its AWS settings in the config;
|
||||
`bedrock-env` uses `api_base` with credentials from the environment, which the script sets to dummy
|
||||
values that only ever reach the mock.
|
||||
|
|
|
|||
190
litellm-rust/loadtest/cmd/chatparity/cases.go
Normal file
190
litellm-rust/loadtest/cmd/chatparity/cases.go
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// deployment is one model group in the shared config, pointing at the mock.
|
||||
type deployment struct {
|
||||
name string // model group name the client asks for
|
||||
family string // which scenarios apply: openai, claude, gemini or compat
|
||||
params map[string]any // litellm_params, with {mock} replaced by the mock's URL
|
||||
}
|
||||
|
||||
func deployments(credentials string) []deployment {
|
||||
compat := func(name, model string) deployment {
|
||||
return deployment{name, "compat", map[string]any{"model": model, "api_base": "{mock}/v1", "api_key": "sk-parity-upstream"}}
|
||||
}
|
||||
vertex := map[string]any{"api_base": "{mock}", "vertex_project": "parity-project", "vertex_credentials": credentials}
|
||||
with := func(base map[string]any, extra map[string]any) map[string]any {
|
||||
out := map[string]any{}
|
||||
for k, v := range base {
|
||||
out[k] = v
|
||||
}
|
||||
for k, v := range extra {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
return []deployment{
|
||||
{"openai", "openai", map[string]any{"model": "openai/gpt-4o", "api_base": "{mock}/v1", "api_key": "sk-parity-upstream"}},
|
||||
{"azure", "openai", map[string]any{"model": "azure/gpt-4o", "api_base": "{mock}", "api_key": "sk-parity-upstream",
|
||||
"api_version": "2025-02-01-preview"}},
|
||||
{"anthropic", "claude", map[string]any{"model": "anthropic/claude-sonnet-4-5-20250929", "api_base": "{mock}",
|
||||
"api_key": "sk-parity-upstream"}},
|
||||
{"bedrock", "claude", map[string]any{"model": "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0",
|
||||
"aws_access_key_id": "AKIAPARITYPARITY", "aws_secret_access_key": "parity-secret", "aws_region_name": "us-east-1",
|
||||
"aws_bedrock_runtime_endpoint": "{mock}"}},
|
||||
// The same model with the endpoint as api_base and credentials from the environment
|
||||
// (chat-parity.sh sets them), for a host that does not read AWS settings from the config.
|
||||
{"bedrock-env", "claude", map[string]any{"model": "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0", "api_base": "{mock}"}},
|
||||
{"vertex-claude", "claude", with(vertex, map[string]any{"model": "vertex_ai/claude-sonnet-4-5@20250929",
|
||||
"vertex_location": "us-east5"})},
|
||||
{"gemini", "gemini", map[string]any{"model": "gemini/gemini-2.5-flash", "api_base": "{mock}", "api_key": "parity-google-key"}},
|
||||
{"vertex-gemini", "gemini", with(vertex, map[string]any{"model": "vertex_ai/gemini-2.5-flash", "vertex_location": "us-central1"})},
|
||||
compat("groq", "groq/llama-3.3-70b-versatile"),
|
||||
compat("mistral", "mistral/mistral-large-latest"),
|
||||
compat("deepseek", "deepseek/deepseek-chat"),
|
||||
compat("together", "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo"),
|
||||
compat("fireworks", "fireworks_ai/accounts/fireworks/models/llama-v3p3-70b-instruct"),
|
||||
compat("xai", "xai/grok-3"),
|
||||
compat("openrouter", "openrouter/openai/gpt-4o"),
|
||||
compat("cerebras", "cerebras/llama3.1-8b"),
|
||||
compat("perplexity", "perplexity/sonar"),
|
||||
compat("hosted-vllm", "hosted_vllm/Qwen/Qwen3-8B"),
|
||||
compat("nvidia-nim", "nvidia_nim/meta/llama-3.3-70b-instruct"),
|
||||
}
|
||||
}
|
||||
|
||||
// scenario is one shape of chat request, sent to every deployment whose family it lists.
|
||||
type scenario struct {
|
||||
name string
|
||||
families string // space-separated families, or "all"
|
||||
streams []bool // which of non-streaming and streaming to send
|
||||
body func(mock string) map[string]any
|
||||
}
|
||||
|
||||
var both = []bool{false, true}
|
||||
|
||||
func user(text string) map[string]any { return map[string]any{"role": "user", "content": text} }
|
||||
|
||||
var weatherTool = map[string]any{"type": "function", "function": map[string]any{
|
||||
"name": "get_weather", "description": "Current weather for a city",
|
||||
"parameters": map[string]any{"type": "object", "properties": map[string]any{"city": map[string]any{"type": "string"}},
|
||||
"required": []any{"city"}},
|
||||
}}
|
||||
|
||||
const pngDataURL = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
|
||||
|
||||
var scenarios = []scenario{
|
||||
{"basic", "all", both, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, "temperature": 0.2}
|
||||
}},
|
||||
{"include-usage", "all", []bool{true}, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50,
|
||||
"stream_options": map[string]any{"include_usage": true}}
|
||||
}},
|
||||
{"system-multi-turn", "all", both, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{
|
||||
map[string]any{"role": "system", "content": "You are terse."},
|
||||
user("Hi."),
|
||||
map[string]any{"role": "assistant", "content": "Hello."},
|
||||
user("Say hello again."),
|
||||
}, "max_tokens": 50, "top_p": 0.9, "stop": []any{"END"}, "user": "parity-user"}
|
||||
}},
|
||||
{"tools", "all", both, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Weather in Paris? [[parity reply=tool]]")},
|
||||
"tools": []any{weatherTool}, "tool_choice": "auto", "max_tokens": 100}
|
||||
}},
|
||||
{"tools-include-usage", "all", []bool{true}, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Weather in Paris? [[parity reply=tool]]")},
|
||||
"tools": []any{weatherTool}, "tool_choice": "required", "max_tokens": 100,
|
||||
"stream_options": map[string]any{"include_usage": true}}
|
||||
}},
|
||||
{"tool-result", "all", both, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{
|
||||
user("Weather in Paris?"),
|
||||
map[string]any{"role": "assistant", "content": nil, "tool_calls": []any{map[string]any{"id": "call_prev_1",
|
||||
"type": "function", "function": map[string]any{"name": "get_weather", "arguments": `{"city": "Paris"}`}}}},
|
||||
map[string]any{"role": "tool", "tool_call_id": "call_prev_1", "content": "18C and sunny"},
|
||||
}, "tools": []any{weatherTool}, "max_tokens": 100}
|
||||
}},
|
||||
{"json-schema", "all", both, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Answer as JSON. [[parity reply=json]]")}, "max_tokens": 100,
|
||||
"response_format": map[string]any{"type": "json_schema", "json_schema": map[string]any{"name": "answer", "strict": true,
|
||||
"schema": map[string]any{"type": "object", "properties": map[string]any{"answer": map[string]any{"type": "integer"}},
|
||||
"required": []any{"answer"}, "additionalProperties": false}}}}
|
||||
}},
|
||||
{"json-object", "openai gemini compat", []bool{false}, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Answer as JSON. [[parity reply=json]]")}, "max_tokens": 100,
|
||||
"response_format": map[string]any{"type": "json_object"}}
|
||||
}},
|
||||
{"image-data-url", "openai claude gemini", both, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{map[string]any{"role": "user", "content": []any{
|
||||
map[string]any{"type": "text", "text": "What is in this image?"},
|
||||
map[string]any{"type": "image_url", "image_url": map[string]any{"url": pngDataURL}},
|
||||
}}}, "max_tokens": 50}
|
||||
}},
|
||||
{"image-remote-url", "openai claude gemini", []bool{false}, func(mock string) map[string]any {
|
||||
return map[string]any{"messages": []any{map[string]any{"role": "user", "content": []any{
|
||||
map[string]any{"type": "text", "text": "What is in this image?"},
|
||||
map[string]any{"type": "image_url", "image_url": map[string]any{"url": mock + "/img.png"}},
|
||||
}}}, "max_tokens": 50}
|
||||
}},
|
||||
{"reasoning-effort", "claude gemini", both, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Think, then say hello. [[parity reply=thinking]]")},
|
||||
"reasoning_effort": "low", "max_tokens": 4096}
|
||||
}},
|
||||
{"cache-control", "claude", []bool{false}, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{
|
||||
map[string]any{"role": "system", "content": []any{map[string]any{"type": "text", "text": "Long shared prefix.",
|
||||
"cache_control": map[string]any{"type": "ephemeral"}}}},
|
||||
user("Say hello."),
|
||||
}, "max_tokens": 50}
|
||||
}},
|
||||
{"unsupported-param", "all", []bool{false}, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, "n": 2, "logit_bias": map[string]any{"50256": -100}}
|
||||
}},
|
||||
{"drop-params", "all", []bool{false}, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Say hello.")}, "max_tokens": 50, "n": 2,
|
||||
"logit_bias": map[string]any{"50256": -100}, "drop_params": true}
|
||||
}},
|
||||
{"max-completion-tokens", "all", []bool{false}, func(string) map[string]any {
|
||||
return map[string]any{"messages": []any{user("Say hello.")}, "max_completion_tokens": 60, "seed": 7}
|
||||
}},
|
||||
}
|
||||
|
||||
// testCase is one request, sent once to each proxy.
|
||||
type testCase struct {
|
||||
Name string `json:"name"`
|
||||
Model string `json:"model"`
|
||||
Stream bool `json:"stream"`
|
||||
Body map[string]any `json:"body"`
|
||||
}
|
||||
|
||||
func buildCases(deps []deployment, mockURL, filter string) []testCase {
|
||||
var cases []testCase
|
||||
for _, d := range deps {
|
||||
for _, s := range scenarios {
|
||||
if s.families != "all" && !strings.Contains(" "+s.families+" ", " "+d.family+" ") {
|
||||
continue
|
||||
}
|
||||
for _, stream := range s.streams {
|
||||
body := s.body(mockURL)
|
||||
body["model"] = d.name
|
||||
mode := "unary"
|
||||
if stream {
|
||||
body["stream"] = true
|
||||
mode = "stream"
|
||||
}
|
||||
name := fmt.Sprintf("%s/%s/%s", d.name, s.name, mode)
|
||||
if filter != "" && !strings.Contains(name, filter) {
|
||||
continue
|
||||
}
|
||||
cases = append(cases, testCase{name, d.name, stream, body})
|
||||
}
|
||||
}
|
||||
}
|
||||
return cases
|
||||
}
|
||||
112
litellm-rust/loadtest/cmd/chatparity/chatparity_test.go
Normal file
112
litellm-rust/loadtest/cmd/chatparity/chatparity_test.go
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func decode(t *testing.T, text string) any {
|
||||
t.Helper()
|
||||
var v any
|
||||
if err := json.Unmarshal([]byte(text), &v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func TestDiffValuesReportsMissingAndChangedPaths(t *testing.T) {
|
||||
var diffs []difference
|
||||
diffValues("response", "", decode(t, `{"a": 1, "b": [1, 2], "c": {"d": "x"}}`),
|
||||
decode(t, `{"a": 1, "b": [1], "c": {"d": "y"}, "e": true}`), &diffs)
|
||||
got := map[string]bool{}
|
||||
for _, d := range diffs {
|
||||
got[d.Path] = true
|
||||
}
|
||||
for _, want := range []string{".b[1]", ".c.d", ".e"} {
|
||||
if !got[want] {
|
||||
t.Errorf("missing %s in %+v", want, diffs)
|
||||
}
|
||||
}
|
||||
if len(diffs) != 3 {
|
||||
t.Errorf("want 3 differences, got %+v", diffs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeIDsKeepsMockIDsAndPrefixesOfGeneratedOnes(t *testing.T) {
|
||||
got := normalizeIDs(decode(t, `{"id": "chatcmpl-123", "created": 5, "choices": [{"message": {"tool_calls": [{"id": "call_parity_1"}, {"id": "call_abc"}]}}]}`))
|
||||
want := decode(t, `{"id": "chatcmpl-*", "created": "<timestamp>", "choices": [{"message": {"tool_calls": [{"id": "call_parity_1"}, {"id": "call_*"}]}}]}`)
|
||||
if !equal(got, want) {
|
||||
t.Errorf("got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssembleFoldsContentToolCallsAndUsage(t *testing.T) {
|
||||
s := parseSSE([]byte(strings.Join([]string{
|
||||
`data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"role":"assistant","content":"Hel"}}]}`,
|
||||
`data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"content":"lo"}}]}`,
|
||||
`data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_parity_1","type":"function","function":{"name":"f","arguments":"{\"a\""}}]}}]}`,
|
||||
`data: {"id":"chatcmpl-1","model":"m","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":": 1}"}}]},"finish_reason":"tool_calls"}]}`,
|
||||
`data: {"id":"chatcmpl-1","model":"m","choices":[],"usage":{"total_tokens":3}}`,
|
||||
`data: [DONE]`,
|
||||
}, "\n\n")))
|
||||
got := assemble(s)
|
||||
if got["content"] != "Hello" || got["done"] != true || got["one_id_per_stream"] != true {
|
||||
t.Errorf("got %v", got)
|
||||
}
|
||||
calls := got["tool_calls"].([]any)
|
||||
if len(calls) != 1 || calls[0].(map[string]any)["arguments"] != `{"a": 1}` {
|
||||
t.Errorf("tool calls %v", calls)
|
||||
}
|
||||
if !equal(got["finish_reasons"], []any{"tool_calls"}) || !equal(got["usage"], map[string]any{"total_tokens": 3.0}) {
|
||||
t.Errorf("got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeUpstreamDropsTransportHeadersAndTheSignature(t *testing.T) {
|
||||
got := normalizeUpstream([]upstream{{Method: "POST", Path: "/p", Headers: map[string]string{
|
||||
"user-agent": "x",
|
||||
"x-amz-date": "20261019T000000Z",
|
||||
"authorization": "AWS4-HMAC-SHA256 Credential=AKIA/20261019/us-east-1/bedrock/aws4_request, SignedHeaders=host, Signature=abc",
|
||||
"anthropic-version": "2023-06-01",
|
||||
}}})
|
||||
headers := got.([]any)[0].(map[string]any)["headers"]
|
||||
want := map[string]any{"authorization": "AWS4-HMAC-SHA256 region=us-east-1 service=bedrock", "anthropic-version": "2023-06-01"}
|
||||
if !equal(headers, want) {
|
||||
t.Errorf("got %v", headers)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMarkKnownMatchesCaseWhereAndPathGlobs(t *testing.T) {
|
||||
diffs := []difference{{Where: "upstream", Path: "[0].headers.accept"}, {Where: "response", Path: ".body.model"}}
|
||||
markKnown("openai/basic/unary", diffs, []known{{Case: "*/basic/*", Where: "upstream", Path: "*.accept", Reason: "r"}})
|
||||
if diffs[0].Known != "r" || diffs[1].Known != "" {
|
||||
t.Errorf("got %+v", diffs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMockAnswersAnthropicJSONModeThroughTheForcedTool(t *testing.T) {
|
||||
m := &mock{}
|
||||
body := `{"model":"c","tools":[{"name":"json_tool_call"}],"messages":[{"role":"user","content":"x"}]}`
|
||||
w := httptest.NewRecorder()
|
||||
m.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/v1/messages", strings.NewReader(body)))
|
||||
got := decode(t, w.Body.String())
|
||||
if dig(got, "content", 0, "name") != "json_tool_call" || dig(got, "content", 0, "input", "answer") != 42.0 {
|
||||
t.Errorf("got %v", got)
|
||||
}
|
||||
if recorded := m.take(); len(recorded) != 1 || recorded[0].Path != "/v1/messages" {
|
||||
t.Errorf("recorded %+v", recorded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEveryCaseNameIsUnique(t *testing.T) {
|
||||
seen := map[string]bool{}
|
||||
for _, c := range buildCases(deployments(""), "http://mock", "") {
|
||||
if seen[c.Name] {
|
||||
t.Errorf("duplicate case %s", c.Name)
|
||||
}
|
||||
seen[c.Name] = true
|
||||
}
|
||||
}
|
||||
339
litellm-rust/loadtest/cmd/chatparity/diff.go
Normal file
339
litellm-rust/loadtest/cmd/chatparity/diff.go
Normal file
|
|
@ -0,0 +1,339 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// difference is one place where the two proxies disagree. Where is "upstream" (what the
|
||||
// provider was sent), "response" (what the client got, or the assembled stream) or "chunks"
|
||||
// (the stream chunk by chunk, informational: boundaries may differ for the same content).
|
||||
type difference struct {
|
||||
Where string `json:"where"`
|
||||
Path string `json:"path"`
|
||||
Python any `json:"python"`
|
||||
Rust any `json:"rust"`
|
||||
Known string `json:"known,omitempty"` // the reason, when the difference is listed as known
|
||||
}
|
||||
|
||||
const absent = "<absent>"
|
||||
|
||||
// diffValues appends every path where a and b differ.
|
||||
func diffValues(where, path string, a, b any, out *[]difference) {
|
||||
switch av := a.(type) {
|
||||
case map[string]any:
|
||||
bv, ok := b.(map[string]any)
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
keys := map[string]bool{}
|
||||
for k := range av {
|
||||
keys[k] = true
|
||||
}
|
||||
for k := range bv {
|
||||
keys[k] = true
|
||||
}
|
||||
for _, k := range sortedKeys(keys) {
|
||||
x, xok := av[k]
|
||||
y, yok := bv[k]
|
||||
switch {
|
||||
case !xok:
|
||||
*out = append(*out, difference{Where: where, Path: path + "." + k, Python: absent, Rust: y})
|
||||
case !yok:
|
||||
*out = append(*out, difference{Where: where, Path: path + "." + k, Python: x, Rust: absent})
|
||||
default:
|
||||
diffValues(where, path+"."+k, x, y, out)
|
||||
}
|
||||
}
|
||||
return
|
||||
case []any:
|
||||
bv, ok := b.([]any)
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
for i := 0; i < len(av) || i < len(bv); i++ {
|
||||
p := fmt.Sprintf("%s[%d]", path, i)
|
||||
switch {
|
||||
case i >= len(av):
|
||||
*out = append(*out, difference{Where: where, Path: p, Python: absent, Rust: bv[i]})
|
||||
case i >= len(bv):
|
||||
*out = append(*out, difference{Where: where, Path: p, Python: av[i], Rust: absent})
|
||||
default:
|
||||
diffValues(where, p, av[i], bv[i], out)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
if !equal(a, b) {
|
||||
*out = append(*out, difference{Where: where, Path: path, Python: a, Rust: b})
|
||||
}
|
||||
}
|
||||
|
||||
func equal(a, b any) bool {
|
||||
x, _ := json.Marshal(a)
|
||||
y, _ := json.Marshal(b)
|
||||
return bytes.Equal(x, y)
|
||||
}
|
||||
|
||||
// Headers that differ for reasons unrelated to translation: client libraries, transport,
|
||||
// tracing, and request signing that depends on the clock.
|
||||
var ignoredHeaders = regexp.MustCompile(`^(user-agent|content-length|accept-encoding|connection|host|traceparent|tracestate|` +
|
||||
`x-stainless-.*|amz-sdk-.*|x-amzn-trace-id|x-amz-date|x-amz-content-sha256|x-goog-api-client|x-litellm-.*)$`)
|
||||
|
||||
var sigV4 = regexp.MustCompile(`^AWS4-HMAC-SHA256 Credential=[^/]+/\d{8}/([^/]+)/([^/]+)/aws4_request`)
|
||||
|
||||
func normalizeUpstream(requests []upstream) any {
|
||||
out := make([]any, len(requests))
|
||||
for i, r := range requests {
|
||||
headers := map[string]any{}
|
||||
for name, value := range r.Headers {
|
||||
if ignoredHeaders.MatchString(name) {
|
||||
continue
|
||||
}
|
||||
if name == "authorization" {
|
||||
if m := sigV4.FindStringSubmatch(value); m != nil {
|
||||
value = fmt.Sprintf("AWS4-HMAC-SHA256 region=%s service=%s", m[1], m[2])
|
||||
}
|
||||
}
|
||||
headers[name] = value
|
||||
}
|
||||
entry := map[string]any{"method": r.Method, "path": r.Path, "headers": headers, "body": r.Body}
|
||||
if r.Query != "" {
|
||||
entry["query"] = r.Query
|
||||
}
|
||||
out[i] = entry
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// normalizeIDs replaces ids and timestamps the proxies generate with a placeholder that keeps
|
||||
// their prefix (`chatcmpl-*`, `call_*`), so only their shape is compared. Ids the mock chose
|
||||
// contain "parity" and are compared exactly.
|
||||
func normalizeIDs(v any) any {
|
||||
switch t := v.(type) {
|
||||
case map[string]any:
|
||||
out := map[string]any{}
|
||||
for k, x := range t {
|
||||
switch {
|
||||
case k == "created":
|
||||
out[k] = "<timestamp>"
|
||||
case k == "id":
|
||||
out[k] = generated(x)
|
||||
default:
|
||||
out[k] = normalizeIDs(x)
|
||||
}
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
out := make([]any, len(t))
|
||||
for i, x := range t {
|
||||
out[i] = normalizeIDs(x)
|
||||
}
|
||||
return out
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func generated(v any) any {
|
||||
s, ok := v.(string)
|
||||
if !ok || strings.Contains(s, "parity") {
|
||||
return v
|
||||
}
|
||||
if i := strings.IndexAny(s, "-_"); i >= 0 {
|
||||
return s[:i+1] + "*"
|
||||
}
|
||||
return "*"
|
||||
}
|
||||
|
||||
// stream is a parsed SSE response.
|
||||
type stream struct {
|
||||
Chunks []any `json:"chunks"`
|
||||
Done bool `json:"done"`
|
||||
Other []any `json:"other,omitempty"` // lines that are neither JSON data nor [DONE]
|
||||
}
|
||||
|
||||
func parseSSE(body []byte) stream {
|
||||
var s stream
|
||||
scanner := bufio.NewScanner(bytes.NewReader(body))
|
||||
scanner.Buffer(make([]byte, 1<<20), 16<<20)
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
if line == "" || strings.HasPrefix(line, ":") {
|
||||
continue
|
||||
}
|
||||
data, ok := strings.CutPrefix(line, "data:")
|
||||
if !ok {
|
||||
s.Other = append(s.Other, line)
|
||||
continue
|
||||
}
|
||||
data = strings.TrimSpace(data)
|
||||
if data == "[DONE]" {
|
||||
s.Done = true
|
||||
continue
|
||||
}
|
||||
var chunk any
|
||||
if json.Unmarshal([]byte(data), &chunk) != nil {
|
||||
s.Other = append(s.Other, line)
|
||||
continue
|
||||
}
|
||||
s.Chunks = append(s.Chunks, chunk)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// assemble folds a stream's chunks into what a client library would build from them, which is
|
||||
// what must agree; how the content was split into chunks may differ.
|
||||
func assemble(s stream) map[string]any {
|
||||
var content, reasoning strings.Builder
|
||||
type call struct{ id, kind, name, args any }
|
||||
calls := map[float64]*call{}
|
||||
var order []float64
|
||||
ids := map[string]bool{}
|
||||
models := map[string]bool{}
|
||||
var roles, finishes []any
|
||||
var usage, errorFrame any
|
||||
extra := map[string]bool{}
|
||||
sawContent, sawReasoning := false, false
|
||||
for _, c := range s.Chunks {
|
||||
chunk, _ := c.(map[string]any)
|
||||
if e, ok := chunk["error"]; ok {
|
||||
errorFrame = e
|
||||
continue
|
||||
}
|
||||
if id, ok := chunk["id"].(string); ok {
|
||||
ids[id] = true
|
||||
}
|
||||
if m, ok := chunk["model"].(string); ok {
|
||||
models[m] = true
|
||||
}
|
||||
if u, ok := chunk["usage"]; ok && u != nil {
|
||||
usage = u
|
||||
}
|
||||
for k := range chunk {
|
||||
switch k {
|
||||
case "id", "object", "created", "model", "choices", "usage", "system_fingerprint":
|
||||
default:
|
||||
extra[k] = true
|
||||
}
|
||||
}
|
||||
choices, _ := chunk["choices"].([]any)
|
||||
for _, ch := range choices {
|
||||
choice, _ := ch.(map[string]any)
|
||||
if f := choice["finish_reason"]; f != nil {
|
||||
finishes = append(finishes, f)
|
||||
}
|
||||
delta, _ := choice["delta"].(map[string]any)
|
||||
for k, v := range delta {
|
||||
switch k {
|
||||
case "role":
|
||||
if v != nil {
|
||||
roles = append(roles, v)
|
||||
}
|
||||
case "content":
|
||||
if t, ok := v.(string); ok {
|
||||
content.WriteString(t)
|
||||
sawContent = true
|
||||
}
|
||||
case "reasoning_content":
|
||||
if t, ok := v.(string); ok {
|
||||
reasoning.WriteString(t)
|
||||
sawReasoning = true
|
||||
}
|
||||
case "tool_calls":
|
||||
list, _ := v.([]any)
|
||||
for _, item := range list {
|
||||
tc, _ := item.(map[string]any)
|
||||
index, _ := tc["index"].(float64)
|
||||
entry := calls[index]
|
||||
if entry == nil {
|
||||
entry = &call{}
|
||||
calls[index] = entry
|
||||
order = append(order, index)
|
||||
}
|
||||
if x := tc["id"]; x != nil {
|
||||
entry.id = generated(x)
|
||||
}
|
||||
if x := tc["type"]; x != nil {
|
||||
entry.kind = x
|
||||
}
|
||||
fn, _ := tc["function"].(map[string]any)
|
||||
if x := fn["name"]; x != nil {
|
||||
entry.name = x
|
||||
}
|
||||
if x, ok := fn["arguments"].(string); ok {
|
||||
prev, _ := entry.args.(string)
|
||||
entry.args = prev + x
|
||||
}
|
||||
}
|
||||
default:
|
||||
if v != nil {
|
||||
extra["delta."+k] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Float64s(order)
|
||||
var toolCalls []any
|
||||
for _, i := range order {
|
||||
c := calls[i]
|
||||
toolCalls = append(toolCalls, map[string]any{"index": i, "id": c.id, "type": c.kind, "name": c.name, "arguments": c.args})
|
||||
}
|
||||
out := map[string]any{
|
||||
"one_id_per_stream": len(ids) <= 1,
|
||||
"id_prefix": nil,
|
||||
"models": sortedKeys(models),
|
||||
"roles": roles,
|
||||
"finish_reasons": finishes,
|
||||
"usage": usage,
|
||||
"done": s.Done,
|
||||
"tool_calls": toolCalls,
|
||||
"other_fields": sortedKeys(extra),
|
||||
"non_data_lines": s.Other,
|
||||
}
|
||||
for id := range ids {
|
||||
out["id_prefix"] = generated(id)
|
||||
}
|
||||
if sawContent {
|
||||
out["content"] = content.String()
|
||||
}
|
||||
if sawReasoning {
|
||||
out["reasoning_content"] = reasoning.String()
|
||||
}
|
||||
if errorFrame != nil {
|
||||
out["error"] = errorFrame
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// known is a difference the team has looked at and accepted, with the reason.
|
||||
type known struct {
|
||||
Case string `json:"case"` // glob over case names; * matches anything
|
||||
Where string `json:"where"` // upstream, response or chunks; empty matches all
|
||||
Path string `json:"path"` // glob over the difference's path
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
func glob(pattern string) *regexp.Regexp {
|
||||
parts := strings.Split(pattern, "*")
|
||||
for i, p := range parts {
|
||||
parts[i] = regexp.QuoteMeta(p)
|
||||
}
|
||||
return regexp.MustCompile("^" + strings.Join(parts, ".*") + "$")
|
||||
}
|
||||
|
||||
func markKnown(name string, diffs []difference, list []known) {
|
||||
for i := range diffs {
|
||||
for _, k := range list {
|
||||
if (k.Where == "" || k.Where == diffs[i].Where) && glob(k.Case).MatchString(name) && glob(k.Path).MatchString(diffs[i].Path) {
|
||||
diffs[i].Known = k.Reason
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
1
litellm-rust/loadtest/cmd/chatparity/known.json
Normal file
1
litellm-rust/loadtest/cmd/chatparity/known.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
[]
|
||||
289
litellm-rust/loadtest/cmd/chatparity/main.go
Normal file
289
litellm-rust/loadtest/cmd/chatparity/main.go
Normal file
|
|
@ -0,0 +1,289 @@
|
|||
// chatparity sends the same chat completions requests through the Python proxy and the Rust
|
||||
// gateway, both pointed at one recording mock provider, and reports every difference in what the
|
||||
// provider was sent and what the client got back.
|
||||
//
|
||||
// chatparity config -mock http://127.0.0.1:9100 -out out # config.yaml both proxies load
|
||||
// chatparity run -mock-addr 127.0.0.1:9100 -python http://127.0.0.1:4001 \
|
||||
// -rust http://127.0.0.1:4002 -out out [-known known.json] [-filter anthropic/]
|
||||
//
|
||||
// The mock answers each provider's API (OpenAI-compatible, Azure, Anthropic, Bedrock Converse,
|
||||
// Gemini, Vertex) the same way for both proxies, so differences in the client's answer come from
|
||||
// the proxies' translation. scripts/chat-parity.sh builds and starts everything locally.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/rand"
|
||||
"crypto/rsa"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const masterKey = "sk-parity-master"
|
||||
|
||||
func main() {
|
||||
log.SetFlags(0)
|
||||
if len(os.Args) < 2 {
|
||||
log.Fatal("usage: chatparity config|run ...")
|
||||
}
|
||||
switch os.Args[1] {
|
||||
case "config":
|
||||
configCmd(os.Args[2:])
|
||||
case "run":
|
||||
runCmd(os.Args[2:])
|
||||
default:
|
||||
log.Fatalf("unknown subcommand %q", os.Args[1])
|
||||
}
|
||||
}
|
||||
|
||||
// configCmd writes the one config both proxies load: every deployment pointed at the mock, and
|
||||
// a throwaway service account whose token endpoint is the mock, for the Vertex deployments.
|
||||
func configCmd(args []string) {
|
||||
fs := flag.NewFlagSet("config", flag.ExitOnError)
|
||||
mockURL := fs.String("mock", "http://127.0.0.1:9100", "URL the proxies reach the mock at")
|
||||
out := fs.String("out", "out", "directory to write config.yaml to")
|
||||
fs.Parse(args)
|
||||
must(os.MkdirAll(*out, 0o755))
|
||||
var models []any
|
||||
for _, d := range deployments(serviceAccount(*mockURL)) {
|
||||
params := map[string]any{}
|
||||
for k, v := range d.params {
|
||||
if s, ok := v.(string); ok {
|
||||
v = strings.ReplaceAll(s, "{mock}", *mockURL)
|
||||
}
|
||||
params[k] = v
|
||||
}
|
||||
models = append(models, map[string]any{"model_name": d.name, "litellm_params": params})
|
||||
}
|
||||
// JSON is YAML, and both proxies read it as their config.yaml.
|
||||
config := map[string]any{
|
||||
"model_list": models,
|
||||
"general_settings": map[string]any{"master_key": masterKey},
|
||||
"litellm_settings": map[string]any{"num_retries": 0, "user_url_allowed_hosts": []any{mockHost(*mockURL)}},
|
||||
}
|
||||
b, _ := json.MarshalIndent(config, "", " ")
|
||||
must(os.WriteFile(filepath.Join(*out, "config.yaml"), b, 0o644))
|
||||
fmt.Println(filepath.Join(*out, "config.yaml"))
|
||||
}
|
||||
|
||||
// mockHost is the mock's host:port, allowed as a media URL host so remote image cases reach it.
|
||||
func mockHost(mockURL string) string {
|
||||
u, err := url.Parse(mockURL)
|
||||
must(err)
|
||||
return u.Host
|
||||
}
|
||||
|
||||
func serviceAccount(mockURL string) string {
|
||||
key, err := rsa.GenerateKey(rand.Reader, 2048)
|
||||
must(err)
|
||||
der, err := x509.MarshalPKCS8PrivateKey(key)
|
||||
must(err)
|
||||
b, _ := json.Marshal(map[string]any{
|
||||
"type": "service_account", "project_id": "parity-project", "private_key_id": "parity",
|
||||
"private_key": string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der})),
|
||||
"client_email": "parity@parity-project.iam.gserviceaccount.com", "client_id": "1",
|
||||
"token_uri": mockURL + "/token",
|
||||
})
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// exchange is one side's request and what came of it.
|
||||
type exchange struct {
|
||||
Status int `json:"status"`
|
||||
ContentType string `json:"content_type"`
|
||||
Body any `json:"body"`
|
||||
Upstream []upstream `json:"upstream"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type outcome struct {
|
||||
Case string `json:"case"`
|
||||
Python exchange `json:"python"`
|
||||
Rust exchange `json:"rust"`
|
||||
Differences []difference `json:"differences"`
|
||||
}
|
||||
|
||||
func runCmd(args []string) {
|
||||
fs := flag.NewFlagSet("run", flag.ExitOnError)
|
||||
mockAddr := fs.String("mock-addr", "127.0.0.1:9100", "address the mock listens on")
|
||||
mockURL := fs.String("mock", "", "URL the proxies reach the mock at (default http://<mock-addr>)")
|
||||
python := fs.String("python", "http://127.0.0.1:4001", "Python proxy base URL")
|
||||
rust := fs.String("rust", "http://127.0.0.1:4002", "Rust gateway base URL")
|
||||
out := fs.String("out", "out", "directory for report.md, results.json and per-case captures")
|
||||
knownPath := fs.String("known", "", "JSON list of accepted differences")
|
||||
filter := fs.String("filter", "", "only run cases whose name contains this")
|
||||
fs.Parse(args)
|
||||
if *mockURL == "" {
|
||||
*mockURL = "http://" + *mockAddr
|
||||
}
|
||||
var accepted []known
|
||||
if *knownPath != "" {
|
||||
b, err := os.ReadFile(*knownPath)
|
||||
must(err)
|
||||
must(json.Unmarshal(b, &accepted))
|
||||
}
|
||||
|
||||
m := &mock{}
|
||||
listener, err := net.Listen("tcp", *mockAddr)
|
||||
must(err)
|
||||
go http.Serve(listener, m)
|
||||
|
||||
client := &http.Client{Timeout: 120 * time.Second}
|
||||
cases := buildCases(deployments(""), *mockURL, *filter)
|
||||
must(os.MkdirAll(filepath.Join(*out, "cases"), 0o755))
|
||||
var outcomes []outcome
|
||||
for _, c := range cases {
|
||||
o := outcome{Case: c.Name}
|
||||
o.Python = send(client, m, *python, c)
|
||||
o.Rust = send(client, m, *rust, c)
|
||||
o.Differences = compare(c, o.Python, o.Rust)
|
||||
markKnown(c.Name, o.Differences, accepted)
|
||||
outcomes = append(outcomes, o)
|
||||
b, _ := json.MarshalIndent(map[string]any{"request": c.Body, "outcome": o}, "", " ")
|
||||
must(os.WriteFile(filepath.Join(*out, "cases", strings.ReplaceAll(c.Name, "/", "__")+".json"), b, 0o644))
|
||||
fmt.Printf("%-55s %s\n", c.Name, verdict(o.Differences))
|
||||
}
|
||||
b, _ := json.MarshalIndent(outcomes, "", " ")
|
||||
must(os.WriteFile(filepath.Join(*out, "results.json"), b, 0o644))
|
||||
failing := writeReport(filepath.Join(*out, "report.md"), outcomes)
|
||||
fmt.Printf("\n%d cases, %d with unexplained differences; report in %s\n", len(outcomes), failing, filepath.Join(*out, "report.md"))
|
||||
if failing > 0 {
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func send(client *http.Client, m *mock, base string, c testCase) exchange {
|
||||
m.take()
|
||||
body, _ := json.Marshal(c.Body)
|
||||
req, _ := http.NewRequest(http.MethodPost, strings.TrimRight(base, "/")+"/v1/chat/completions", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+masterKey)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return exchange{Error: err.Error(), Upstream: m.take()}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
raw, _ := io.ReadAll(resp.Body)
|
||||
x := exchange{Status: resp.StatusCode, ContentType: resp.Header.Get("Content-Type"), Upstream: m.take()}
|
||||
if strings.HasPrefix(x.ContentType, "text/event-stream") {
|
||||
x.Body = parseSSE(raw)
|
||||
} else {
|
||||
var parsed any
|
||||
if json.Unmarshal(raw, &parsed) == nil {
|
||||
x.Body = parsed
|
||||
} else {
|
||||
x.Body = string(raw)
|
||||
}
|
||||
}
|
||||
return x
|
||||
}
|
||||
|
||||
func compare(c testCase, py, rs exchange) []difference {
|
||||
var diffs []difference
|
||||
diffValues("upstream", "", normalizeUpstream(py.Upstream), normalizeUpstream(rs.Upstream), &diffs)
|
||||
top := func(x exchange) map[string]any {
|
||||
return map[string]any{"status": x.Status, "content_type": mediaType(x.ContentType), "transport_error": x.Error}
|
||||
}
|
||||
diffValues("response", "", top(py), top(rs), &diffs)
|
||||
ps, pok := py.Body.(stream)
|
||||
rsm, rok := rs.Body.(stream)
|
||||
switch {
|
||||
case pok && rok:
|
||||
diffValues("response", ".stream", assemble(ps), assemble(rsm), &diffs)
|
||||
diffValues("chunks", "", normalizeIDs(ps.Chunks), normalizeIDs(rsm.Chunks), &diffs)
|
||||
case !pok && !rok:
|
||||
diffValues("response", ".body", normalizeIDs(py.Body), normalizeIDs(rs.Body), &diffs)
|
||||
}
|
||||
return diffs
|
||||
}
|
||||
|
||||
func mediaType(contentType string) string {
|
||||
media, _, _ := strings.Cut(contentType, ";")
|
||||
return strings.TrimSpace(media)
|
||||
}
|
||||
|
||||
func counts(diffs []difference) (open, accepted, chunks int) {
|
||||
for _, d := range diffs {
|
||||
switch {
|
||||
case d.Where == "chunks":
|
||||
chunks++
|
||||
case d.Known != "":
|
||||
accepted++
|
||||
default:
|
||||
open++
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func verdict(diffs []difference) string {
|
||||
open, accepted, chunks := counts(diffs)
|
||||
if open == 0 {
|
||||
return fmt.Sprintf("ok (%d known, %d chunk-level)", accepted, chunks)
|
||||
}
|
||||
return fmt.Sprintf("DIFF %d (%d known, %d chunk-level)", open, accepted, chunks)
|
||||
}
|
||||
|
||||
// writeReport writes a summary table and every case's open differences, and returns how many
|
||||
// cases have any.
|
||||
func writeReport(path string, outcomes []outcome) int {
|
||||
var b strings.Builder
|
||||
failing := 0
|
||||
b.WriteString("# Chat completions parity: Python proxy vs Rust gateway\n\n")
|
||||
b.WriteString("Open differences fail the run. Known ones are listed in the known-differences file with a reason. ")
|
||||
b.WriteString("Chunk-level differences compare streams chunk by chunk and are informational: the assembled stream is what must agree.\n\n")
|
||||
b.WriteString("| case | python | rust | open | known | chunk-level |\n|---|---|---|---|---|---|\n")
|
||||
for _, o := range outcomes {
|
||||
open, accepted, chunks := counts(o.Differences)
|
||||
if open > 0 {
|
||||
failing++
|
||||
}
|
||||
fmt.Fprintf(&b, "| %s | %d | %d | %d | %d | %d |\n", o.Case, o.Python.Status, o.Rust.Status, open, accepted, chunks)
|
||||
}
|
||||
b.WriteString("\n## Open differences\n")
|
||||
for _, o := range outcomes {
|
||||
var lines []string
|
||||
for _, d := range o.Differences {
|
||||
if d.Where == "chunks" || d.Known != "" {
|
||||
continue
|
||||
}
|
||||
lines = append(lines, fmt.Sprintf("| %s | `%s` | %s | %s |", d.Where, d.Path, cell(d.Python), cell(d.Rust)))
|
||||
}
|
||||
if len(lines) == 0 {
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(&b, "\n### %s\n\n| where | path | python | rust |\n|---|---|---|---|\n%s\n", o.Case, strings.Join(lines, "\n"))
|
||||
}
|
||||
must(os.WriteFile(path, []byte(b.String()), 0o644))
|
||||
return failing
|
||||
}
|
||||
|
||||
func cell(v any) string {
|
||||
s, ok := v.(string)
|
||||
if !ok {
|
||||
raw, _ := json.Marshal(v)
|
||||
s = string(raw)
|
||||
}
|
||||
if len(s) > 300 {
|
||||
s = s[:300] + "…"
|
||||
}
|
||||
return "`" + strings.NewReplacer("|", "\\|", "\n", " ", "`", "'").Replace(s) + "`"
|
||||
}
|
||||
|
||||
func must(err error) {
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
461
litellm-rust/loadtest/cmd/chatparity/mock.go
Normal file
461
litellm-rust/loadtest/cmd/chatparity/mock.go
Normal file
|
|
@ -0,0 +1,461 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/base64"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"hash/crc32"
|
||||
"io"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// upstream is one request a proxy sent to the mock provider.
|
||||
type upstream struct {
|
||||
Method string `json:"method"`
|
||||
Path string `json:"path"`
|
||||
Query string `json:"query,omitempty"`
|
||||
Headers map[string]string `json:"headers"`
|
||||
Body any `json:"body,omitempty"`
|
||||
}
|
||||
|
||||
// mock answers as each provider would, the same way for both proxies, and records what it was
|
||||
// sent. Cases run one at a time, so everything that arrives belongs to the current slot.
|
||||
type mock struct {
|
||||
mu sync.Mutex
|
||||
current []upstream
|
||||
}
|
||||
|
||||
func (m *mock) take() []upstream {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
got := m.current
|
||||
m.current = nil
|
||||
return got
|
||||
}
|
||||
|
||||
func (m *mock) record(r *http.Request, body []byte) {
|
||||
headers := map[string]string{}
|
||||
for name, values := range r.Header {
|
||||
headers[strings.ToLower(name)] = strings.Join(values, ", ")
|
||||
}
|
||||
var parsed any
|
||||
if len(body) > 0 && json.Unmarshal(body, &parsed) != nil {
|
||||
parsed = string(body)
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.current = append(m.current, upstream{r.Method, r.URL.Path, r.URL.RawQuery, headers, parsed})
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
// The fixed answer parts. Everything the mock returns is deterministic, so any difference in
|
||||
// what the clients get comes from the proxies.
|
||||
const (
|
||||
answerText = "Hello from the parity mock."
|
||||
reasoningText = "The user wants a greeting."
|
||||
signature = "sig_parity"
|
||||
toolID = "call_parity_1"
|
||||
toolArgs = `{"city": "Paris"}`
|
||||
jsonAnswer = `{"answer": 42}`
|
||||
inputTokens = 21
|
||||
outputTokens = 9
|
||||
cacheRead = 3
|
||||
cacheWrite = 2
|
||||
reasoningCount = 4
|
||||
)
|
||||
|
||||
// pixel is a 1x1 PNG, served at /img.png for remote image cases.
|
||||
var pixel, _ = base64.StdEncoding.DecodeString("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==")
|
||||
|
||||
var directive = regexp.MustCompile(`\[\[parity reply=(\w+)\]\]`)
|
||||
|
||||
// reply is what a case asks the provider to answer with: text, tool, thinking or json.
|
||||
type reply struct {
|
||||
kind string
|
||||
tool string // the tool to call: the request's first tool, or the proxy's JSON-mode tool
|
||||
}
|
||||
|
||||
func replyFor(body []byte, firstTool string) reply {
|
||||
kind := "text"
|
||||
if m := directive.FindSubmatch(body); m != nil {
|
||||
kind = string(m[1])
|
||||
}
|
||||
// Anthropic-style JSON mode turns response_format into a forced tool; answer through it.
|
||||
if bytes.Contains(body, []byte(`"json_tool_call"`)) {
|
||||
return reply{kind: "tool", tool: "json_tool_call"}
|
||||
}
|
||||
return reply{kind: kind, tool: firstTool}
|
||||
}
|
||||
|
||||
func (r reply) text() string {
|
||||
if r.kind == "json" {
|
||||
return jsonAnswer
|
||||
}
|
||||
return answerText
|
||||
}
|
||||
|
||||
func (m *mock) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
path := r.URL.Path
|
||||
switch {
|
||||
case path == "/img.png":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
w.Write(pixel)
|
||||
return
|
||||
case path == "/token":
|
||||
writeJSON(w, map[string]any{"access_token": "ya29.parity", "expires_in": 3600, "token_type": "Bearer"})
|
||||
return
|
||||
}
|
||||
m.record(r, body)
|
||||
var req map[string]any
|
||||
json.Unmarshal(body, &req)
|
||||
stream, _ := req["stream"].(bool)
|
||||
switch {
|
||||
case strings.HasSuffix(path, "/chat/completions"):
|
||||
openAI(w, req, replyFor(body, str(dig(req, "tools", 0, "function", "name"))), stream)
|
||||
case strings.HasSuffix(path, "/messages"), strings.HasSuffix(path, ":rawPredict"):
|
||||
anthropic(w, replyFor(body, str(dig(req, "tools", 0, "name"))), stream)
|
||||
case strings.HasSuffix(path, ":streamRawPredict"):
|
||||
anthropic(w, replyFor(body, str(dig(req, "tools", 0, "name"))), true)
|
||||
case strings.HasSuffix(path, "/converse"):
|
||||
bedrock(w, replyFor(body, str(dig(req, "toolConfig", "tools", 0, "toolSpec", "name"))), false)
|
||||
case strings.HasSuffix(path, "/converse-stream"):
|
||||
bedrock(w, replyFor(body, str(dig(req, "toolConfig", "tools", 0, "toolSpec", "name"))), true)
|
||||
case strings.HasSuffix(path, ":generateContent"):
|
||||
gemini(w, replyFor(body, str(dig(req, "tools", 0, "functionDeclarations", 0, "name"))), false)
|
||||
case strings.HasSuffix(path, ":streamGenerateContent"):
|
||||
gemini(w, replyFor(body, str(dig(req, "tools", 0, "functionDeclarations", 0, "name"))), true)
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
writeJSON(w, map[string]any{"error": map[string]any{"message": "parity mock: no route for " + path}})
|
||||
}
|
||||
}
|
||||
|
||||
func openAI(w http.ResponseWriter, req map[string]any, rep reply, stream bool) {
|
||||
model, _ := req["model"].(string)
|
||||
usage := map[string]any{
|
||||
"prompt_tokens": inputTokens, "completion_tokens": outputTokens, "total_tokens": inputTokens + outputTokens,
|
||||
"prompt_tokens_details": map[string]any{"cached_tokens": cacheRead},
|
||||
}
|
||||
message := map[string]any{"role": "assistant", "content": rep.text()}
|
||||
finish := "stop"
|
||||
if rep.kind == "tool" {
|
||||
message["content"] = nil
|
||||
message["tool_calls"] = []any{map[string]any{"id": toolID, "type": "function",
|
||||
"function": map[string]any{"name": rep.tool, "arguments": toolArgs}}}
|
||||
finish = "tool_calls"
|
||||
}
|
||||
if rep.kind == "thinking" {
|
||||
message["reasoning_content"] = reasoningText
|
||||
}
|
||||
if !stream {
|
||||
writeJSON(w, map[string]any{"id": "chatcmpl-parity", "object": "chat.completion", "created": 1700000000,
|
||||
"model": model, "choices": []any{map[string]any{"index": 0, "message": message, "finish_reason": finish}},
|
||||
"usage": usage})
|
||||
return
|
||||
}
|
||||
s := newSSE(w, "text/event-stream")
|
||||
chunk := func(delta map[string]any, finish any) map[string]any {
|
||||
return map[string]any{"id": "chatcmpl-parity", "object": "chat.completion.chunk", "created": 1700000000,
|
||||
"model": model, "choices": []any{map[string]any{"index": 0, "delta": delta, "finish_reason": finish}}}
|
||||
}
|
||||
s.data(chunk(map[string]any{"role": "assistant", "content": ""}, nil))
|
||||
if rep.kind == "thinking" {
|
||||
s.data(chunk(map[string]any{"reasoning_content": reasoningText}, nil))
|
||||
}
|
||||
if rep.kind == "tool" {
|
||||
s.data(chunk(map[string]any{"tool_calls": []any{map[string]any{"index": 0, "id": toolID, "type": "function",
|
||||
"function": map[string]any{"name": rep.tool, "arguments": ""}}}}, nil))
|
||||
for _, part := range halves(toolArgs) {
|
||||
s.data(chunk(map[string]any{"tool_calls": []any{map[string]any{"index": 0,
|
||||
"function": map[string]any{"arguments": part}}}}, nil))
|
||||
}
|
||||
} else {
|
||||
for _, part := range halves(rep.text()) {
|
||||
s.data(chunk(map[string]any{"content": part}, nil))
|
||||
}
|
||||
}
|
||||
s.data(chunk(map[string]any{}, finish))
|
||||
if b, _ := dig(req, "stream_options", "include_usage").(bool); b {
|
||||
last := chunk(nil, nil)
|
||||
last["choices"] = []any{}
|
||||
last["usage"] = usage
|
||||
s.data(last)
|
||||
}
|
||||
s.raw("data: [DONE]\n\n")
|
||||
}
|
||||
|
||||
func anthropic(w http.ResponseWriter, rep reply, stream bool) {
|
||||
var blocks []map[string]any
|
||||
stop := "end_turn"
|
||||
if rep.kind == "thinking" {
|
||||
blocks = append(blocks, map[string]any{"type": "thinking", "thinking": reasoningText, "signature": signature})
|
||||
}
|
||||
if rep.kind == "tool" {
|
||||
var input any
|
||||
json.Unmarshal([]byte(toolArgs), &input)
|
||||
if rep.tool == "json_tool_call" {
|
||||
json.Unmarshal([]byte(jsonAnswer), &input)
|
||||
}
|
||||
blocks = append(blocks, map[string]any{"type": "tool_use", "id": toolID, "name": rep.tool, "input": input})
|
||||
stop = "tool_use"
|
||||
} else {
|
||||
blocks = append(blocks, map[string]any{"type": "text", "text": rep.text()})
|
||||
}
|
||||
usage := map[string]any{"input_tokens": inputTokens, "output_tokens": outputTokens,
|
||||
"cache_read_input_tokens": cacheRead, "cache_creation_input_tokens": cacheWrite}
|
||||
if !stream {
|
||||
content := make([]any, len(blocks))
|
||||
for i, b := range blocks {
|
||||
content[i] = b
|
||||
}
|
||||
writeJSON(w, map[string]any{"id": "msg_parity", "type": "message", "role": "assistant",
|
||||
"model": "claude-parity", "content": content, "stop_reason": stop, "stop_sequence": nil, "usage": usage})
|
||||
return
|
||||
}
|
||||
s := newSSE(w, "text/event-stream")
|
||||
s.event("message_start", map[string]any{"type": "message_start", "message": map[string]any{
|
||||
"id": "msg_parity", "type": "message", "role": "assistant", "model": "claude-parity", "content": []any{},
|
||||
"stop_reason": nil, "stop_sequence": nil, "usage": map[string]any{"input_tokens": inputTokens, "output_tokens": 1,
|
||||
"cache_read_input_tokens": cacheRead, "cache_creation_input_tokens": cacheWrite}}})
|
||||
for i, b := range blocks {
|
||||
start := map[string]any{}
|
||||
for k, v := range b {
|
||||
start[k] = v
|
||||
}
|
||||
var deltas []map[string]any
|
||||
switch b["type"] {
|
||||
case "thinking":
|
||||
start["thinking"], start["signature"] = "", ""
|
||||
for _, part := range halves(reasoningText) {
|
||||
deltas = append(deltas, map[string]any{"type": "thinking_delta", "thinking": part})
|
||||
}
|
||||
deltas = append(deltas, map[string]any{"type": "signature_delta", "signature": signature})
|
||||
case "tool_use":
|
||||
start["input"] = map[string]any{}
|
||||
input, _ := json.Marshal(b["input"])
|
||||
for _, part := range halves(string(input)) {
|
||||
deltas = append(deltas, map[string]any{"type": "input_json_delta", "partial_json": part})
|
||||
}
|
||||
default:
|
||||
start["text"] = ""
|
||||
for _, part := range halves(rep.text()) {
|
||||
deltas = append(deltas, map[string]any{"type": "text_delta", "text": part})
|
||||
}
|
||||
}
|
||||
s.event("content_block_start", map[string]any{"type": "content_block_start", "index": i, "content_block": start})
|
||||
for _, d := range deltas {
|
||||
s.event("content_block_delta", map[string]any{"type": "content_block_delta", "index": i, "delta": d})
|
||||
}
|
||||
s.event("content_block_stop", map[string]any{"type": "content_block_stop", "index": i})
|
||||
}
|
||||
s.event("message_delta", map[string]any{"type": "message_delta",
|
||||
"delta": map[string]any{"stop_reason": stop, "stop_sequence": nil}, "usage": map[string]any{"output_tokens": outputTokens}})
|
||||
s.event("message_stop", map[string]any{"type": "message_stop"})
|
||||
}
|
||||
|
||||
func bedrock(w http.ResponseWriter, rep reply, stream bool) {
|
||||
stop := "end_turn"
|
||||
var input any
|
||||
json.Unmarshal([]byte(toolArgs), &input)
|
||||
if rep.tool == "json_tool_call" {
|
||||
json.Unmarshal([]byte(jsonAnswer), &input)
|
||||
}
|
||||
usage := map[string]any{"inputTokens": inputTokens, "outputTokens": outputTokens, "totalTokens": inputTokens + outputTokens,
|
||||
"cacheReadInputTokens": cacheRead, "cacheWriteInputTokens": cacheWrite}
|
||||
if rep.kind == "tool" {
|
||||
stop = "tool_use"
|
||||
}
|
||||
if !stream {
|
||||
var content []any
|
||||
if rep.kind == "thinking" {
|
||||
content = append(content, map[string]any{"reasoningContent": map[string]any{
|
||||
"reasoningText": map[string]any{"text": reasoningText, "signature": signature}}})
|
||||
}
|
||||
if rep.kind == "tool" {
|
||||
content = append(content, map[string]any{"toolUse": map[string]any{"toolUseId": toolID, "name": rep.tool, "input": input}})
|
||||
} else {
|
||||
content = append(content, map[string]any{"text": rep.text()})
|
||||
}
|
||||
writeJSON(w, map[string]any{"output": map[string]any{"message": map[string]any{"role": "assistant", "content": content}},
|
||||
"stopReason": stop, "usage": usage, "metrics": map[string]any{"latencyMs": 1}})
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/vnd.amazon.eventstream")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
frame := func(event string, payload any) {
|
||||
b, _ := json.Marshal(payload)
|
||||
w.Write(eventStreamFrame(event, b))
|
||||
if f, ok := w.(http.Flusher); ok {
|
||||
f.Flush()
|
||||
}
|
||||
}
|
||||
frame("messageStart", map[string]any{"role": "assistant"})
|
||||
index := 0
|
||||
if rep.kind == "thinking" {
|
||||
for _, part := range halves(reasoningText) {
|
||||
frame("contentBlockDelta", map[string]any{"contentBlockIndex": index,
|
||||
"delta": map[string]any{"reasoningContent": map[string]any{"text": part}}})
|
||||
}
|
||||
frame("contentBlockDelta", map[string]any{"contentBlockIndex": index,
|
||||
"delta": map[string]any{"reasoningContent": map[string]any{"signature": signature}}})
|
||||
frame("contentBlockStop", map[string]any{"contentBlockIndex": index})
|
||||
index++
|
||||
}
|
||||
if rep.kind == "tool" {
|
||||
frame("contentBlockStart", map[string]any{"contentBlockIndex": index,
|
||||
"start": map[string]any{"toolUse": map[string]any{"toolUseId": toolID, "name": rep.tool}}})
|
||||
args, _ := json.Marshal(input)
|
||||
for _, part := range halves(string(args)) {
|
||||
frame("contentBlockDelta", map[string]any{"contentBlockIndex": index,
|
||||
"delta": map[string]any{"toolUse": map[string]any{"input": part}}})
|
||||
}
|
||||
} else {
|
||||
for _, part := range halves(rep.text()) {
|
||||
frame("contentBlockDelta", map[string]any{"contentBlockIndex": index, "delta": map[string]any{"text": part}})
|
||||
}
|
||||
}
|
||||
frame("contentBlockStop", map[string]any{"contentBlockIndex": index})
|
||||
frame("messageStop", map[string]any{"stopReason": stop})
|
||||
frame("metadata", map[string]any{"usage": usage, "metrics": map[string]any{"latencyMs": 1}})
|
||||
}
|
||||
|
||||
func gemini(w http.ResponseWriter, rep reply, stream bool) {
|
||||
var parts []any
|
||||
if rep.kind == "thinking" {
|
||||
parts = append(parts, map[string]any{"text": reasoningText, "thought": true})
|
||||
}
|
||||
if rep.kind == "tool" {
|
||||
var args any
|
||||
json.Unmarshal([]byte(toolArgs), &args)
|
||||
parts = append(parts, map[string]any{"functionCall": map[string]any{"name": rep.tool, "args": args}})
|
||||
} else {
|
||||
parts = append(parts, map[string]any{"text": rep.text()})
|
||||
}
|
||||
usage := map[string]any{"promptTokenCount": inputTokens, "candidatesTokenCount": outputTokens,
|
||||
"totalTokenCount": inputTokens + outputTokens, "cachedContentTokenCount": cacheRead}
|
||||
if rep.kind == "thinking" {
|
||||
usage["thoughtsTokenCount"] = reasoningCount
|
||||
usage["totalTokenCount"] = inputTokens + outputTokens + reasoningCount
|
||||
}
|
||||
candidate := func(parts []any, finish any) map[string]any {
|
||||
c := map[string]any{"content": map[string]any{"role": "model", "parts": parts}, "index": 0}
|
||||
if finish != nil {
|
||||
c["finishReason"] = finish
|
||||
}
|
||||
return c
|
||||
}
|
||||
if !stream {
|
||||
writeJSON(w, map[string]any{"candidates": []any{candidate(parts, "STOP")}, "usageMetadata": usage,
|
||||
"modelVersion": "gemini-parity", "responseId": "resp_parity"})
|
||||
return
|
||||
}
|
||||
s := newSSE(w, "text/event-stream")
|
||||
for i, part := range parts {
|
||||
var finish any
|
||||
if i == len(parts)-1 {
|
||||
finish = "STOP"
|
||||
}
|
||||
event := map[string]any{"candidates": []any{candidate([]any{part}, finish)}, "modelVersion": "gemini-parity",
|
||||
"responseId": "resp_parity"}
|
||||
if finish != nil {
|
||||
event["usageMetadata"] = usage
|
||||
}
|
||||
s.data(event)
|
||||
}
|
||||
}
|
||||
|
||||
type sse struct{ w http.ResponseWriter }
|
||||
|
||||
func newSSE(w http.ResponseWriter, contentType string) sse {
|
||||
w.Header().Set("Content-Type", contentType)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return sse{w}
|
||||
}
|
||||
|
||||
func (s sse) raw(text string) {
|
||||
io.WriteString(s.w, text)
|
||||
if f, ok := s.w.(http.Flusher); ok {
|
||||
f.Flush()
|
||||
}
|
||||
}
|
||||
|
||||
func (s sse) data(v any) {
|
||||
b, _ := json.Marshal(v)
|
||||
s.raw("data: " + string(b) + "\n\n")
|
||||
}
|
||||
|
||||
func (s sse) event(name string, v any) {
|
||||
b, _ := json.Marshal(v)
|
||||
s.raw("event: " + name + "\ndata: " + string(b) + "\n\n")
|
||||
}
|
||||
|
||||
// eventStreamFrame encodes one AWS event-stream message, as mockllm does.
|
||||
func eventStreamFrame(eventType string, payload []byte) []byte {
|
||||
var headers bytes.Buffer
|
||||
for _, h := range [][2]string{{":event-type", eventType}, {":content-type", "application/json"}, {":message-type", "event"}} {
|
||||
headers.WriteByte(byte(len(h[0])))
|
||||
headers.WriteString(h[0])
|
||||
headers.WriteByte(7)
|
||||
binary.Write(&headers, binary.BigEndian, uint16(len(h[1])))
|
||||
headers.WriteString(h[1])
|
||||
}
|
||||
total := 12 + headers.Len() + len(payload) + 4
|
||||
frame := make([]byte, 0, total)
|
||||
frame = binary.BigEndian.AppendUint32(frame, uint32(total))
|
||||
frame = binary.BigEndian.AppendUint32(frame, uint32(headers.Len()))
|
||||
frame = binary.BigEndian.AppendUint32(frame, crc32.ChecksumIEEE(frame))
|
||||
frame = append(frame, headers.Bytes()...)
|
||||
frame = append(frame, payload...)
|
||||
return binary.BigEndian.AppendUint32(frame, crc32.ChecksumIEEE(frame))
|
||||
}
|
||||
|
||||
// halves splits text in two, so every stream carries more than one delta per block.
|
||||
func halves(text string) []string {
|
||||
mid := len(text) / 2
|
||||
return []string{text[:mid], text[mid:]}
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
|
||||
// dig walks maps by key and slices by index, returning nil when anything is missing.
|
||||
func dig(v any, path ...any) any {
|
||||
for _, step := range path {
|
||||
switch s := step.(type) {
|
||||
case string:
|
||||
m, ok := v.(map[string]any)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
v = m[s]
|
||||
case int:
|
||||
a, ok := v.([]any)
|
||||
if !ok || s >= len(a) {
|
||||
return nil
|
||||
}
|
||||
v = a[s]
|
||||
}
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func str(v any) string {
|
||||
s, _ := v.(string)
|
||||
return s
|
||||
}
|
||||
|
||||
func sortedKeys[V any](m map[string]V) []string {
|
||||
keys := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
return keys
|
||||
}
|
||||
49
litellm-rust/loadtest/scripts/chat-parity.sh
Executable file
49
litellm-rust/loadtest/scripts/chat-parity.sh
Executable file
|
|
@ -0,0 +1,49 @@
|
|||
#!/usr/bin/env bash
|
||||
# Runs the chat completions parity check locally: writes one config pointed at chatparity's mock,
|
||||
# starts the Python proxy and the Rust gateway on it, sends every case through both and writes
|
||||
# out/report.md. Exits non-zero when there are unexplained differences.
|
||||
#
|
||||
# scripts/chat-parity.sh # every case
|
||||
# FILTER=anthropic/ scripts/chat-parity.sh
|
||||
#
|
||||
# Needs: go, cargo, and a Python environment with the proxy and google-auth installed
|
||||
# (`uv pip install -e '.[proxy]' google-auth` from the repo root). PYTHON picks the
|
||||
# interpreter; GATEWAY skips the cargo build.
|
||||
set -euo pipefail
|
||||
|
||||
here="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
repo="$(cd "$here/../.." && pwd)"
|
||||
out="${OUT:-$here/out/chat-parity}"
|
||||
mock_port="${MOCK_PORT:-9100}" python_port="${PYTHON_PORT:-4001}" rust_port="${RUST_PORT:-4002}"
|
||||
python="${PYTHON:-python3}"
|
||||
mkdir -p "$out"
|
||||
|
||||
(cd "$here" && go build -o "$out/chatparity" ./cmd/chatparity)
|
||||
if [[ -z "${GATEWAY:-}" ]]; then
|
||||
(cd "$repo/litellm-rust" && cargo build -p litellm-gateway)
|
||||
GATEWAY="${CARGO_TARGET_DIR:-$repo/litellm-rust/target}/debug/litellm-gateway"
|
||||
fi
|
||||
"$out/chatparity" config -mock "http://127.0.0.1:$mock_port" -out "$out" >/dev/null
|
||||
|
||||
# Credentials for the bedrock-env deployment, which takes them from the environment. They only
|
||||
# ever reach the mock.
|
||||
export AWS_ACCESS_KEY_ID=AKIAPARITYPARITY AWS_SECRET_ACCESS_KEY=parity-secret AWS_REGION_NAME=us-east-1 AWS_REGION=us-east-1
|
||||
unset AWS_SESSION_TOKEN AWS_PROFILE
|
||||
|
||||
pids=()
|
||||
trap 'kill "${pids[@]}" 2>/dev/null || true' EXIT
|
||||
# Both price from the repo's cost map, so parameter support and defaults come from the same data.
|
||||
LITELLM_LOCAL_MODEL_COST_MAP=True "$python" -m litellm.proxy.proxy_cli --config "$out/config.yaml" \
|
||||
--port "$python_port" >"$out/python.log" 2>&1 &
|
||||
pids+=($!)
|
||||
LITELLM_MODEL_COST_MAP_PATH="$repo/model_prices_and_context_window.json" LITELLM_CONFIG="$out/config.yaml" \
|
||||
HOST=127.0.0.1 PORT="$rust_port" "$GATEWAY" >"$out/rust.log" 2>&1 &
|
||||
pids+=($!)
|
||||
|
||||
for url in "http://127.0.0.1:$python_port/health/liveliness" "http://127.0.0.1:$rust_port/health/liveliness"; do
|
||||
for _ in $(seq 120); do curl -sf "$url" >/dev/null && break; sleep 1; done
|
||||
curl -sf "$url" >/dev/null || { echo "not up: $url (see $out/*.log)" >&2; exit 1; }
|
||||
done
|
||||
|
||||
"$out/chatparity" run -mock-addr "127.0.0.1:$mock_port" -python "http://127.0.0.1:$python_port" \
|
||||
-rust "http://127.0.0.1:$rust_port" -out "$out" -known "$here/cmd/chatparity/known.json" -filter "${FILTER:-}"
|
||||
Loading…
Add table
Reference in a new issue