From 6f345e3eb9e394456510d8fd189762611fc81d4f Mon Sep 17 00:00:00 2001 From: Nate Armstrong Date: Mon, 19 Oct 2026 12:47:12 -0700 Subject: [PATCH] feat(gateway): land the chat provider series and full-feature load test on the poc branch (#47388) * feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough Adds azure/ chat completions, streaming and not, on the same passthrough the openai/ route uses. The request goes to the deployment path with the api-version query, or to /openai/v1/ for the v1, latest and preview versions, with api_base, api_key and api_version falling back to the AZURE_* variables and os.environ references resolved as the proxy does. A key is sent as api-key, and an azure_ad_token or an Entra ID token from auth-azure goes as a Bearer token. The body follows the proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the deployment name and reading model capabilities from the cost map, so an unsupported param is refused unless drop_params allows it to be dropped. Spend is priced as the proxy prices Azure, by azure/ when one is set and otherwise by the model the response names, and a content filter error keeps Azure's inner error for the client * feat(gateway): Anthropic and Bedrock chat completions streaming Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/ deployments, which were refused with 501 before. The request is the non-streaming one with "stream": true added for Anthropic and sent to /converse-stream for Bedrock, and the provider's events are decoded the way the proxy's chunk parsers decode them: text, tool call deltas with empty arguments sent as "{}", reasoning_content and thinking_blocks with their index and signature, citations and the mapped finish_reason. A wrapper in the shape of the proxy's stream wrapper gives every chunk one id and the client's model name, sends role on the first chunk, adds the empty finish chunk, strips usage from the chunks and, when stream_options.include_usage is set, ends with the usage chunk before [DONE]. The gateway waits for the first chunk before answering, so an upstream error up to there is retried or falls back to another deployment, and an error after it ends the stream with the OpenAI error frame. Usage, cache reads and writes included, is handed to the meter through the response whether or not the client asked for it, and dropping the client stream drops the upstream request * feat(gateway): full openai chat request translation for anthropic and bedrock (#47088) * feat(gateway): share router cooldowns and follow proxy key invalidations through Redis Gateway replicas now share state through the coordination Redis the Python proxy uses (general_settings.coordination_redis, a redis response cache, or REDIS_*): - A deployment one replica cools down is written under Python's deployment::cooldown key and applied by every other replica's background poll (one MGET a second), so the fleet stops sending to a failing deployment together. allowed_fails counts failures across replicas, as Python's v1 rule does; the v2 failure-rate rule stays per replica, as in Python. Deployments are named by model_info.id, or an id derived from the config that is the same on every replica. - The virtual key cache follows the proxy's litellm_proxy.auth_cache_invalidation channel, so a key blocked, deleted or edited through the admin API (or its team or user) is re-read on every replica at once instead of after up to 60 s. general_settings.user_api_key_cache_ttl sets the key TTL. Nothing is added to the path of a successful request. Without Redis, or while it is down, each replica keeps its own state as before. Redis Cluster and Sentinel are refused at startup for now. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): hooks and observers around inference requests, with usage and cost Adds the litellm-gateway-hooks crate: request facts built only when something reads them, on-path hooks that declare what they read and what they may change and run in config order, off-path observers on bounded queues, a usage tap for SSE and JSON responses, and per-deployment prices from the model cost map overlaid by config prices. The messages and chat handlers run through it when anything is installed; with nothing installed requests take the old path. The key store now also returns the key alias, user, team alias and key and team metadata, and router attempts carry the deployment id. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): write spend to the proxy's database from a native observer Adds a spend writer observer that records every gateway request in the proxy's own Postgres tables with the same statements the Python proxy uses: spend logs, key/user/team/team member/end user spend, and the daily user, team and end-user activity tables. Writes are batched on the proxy's timers and are additive, so replicas and Python proxies can write side by side. The database pool is now shared between virtual keys and the writer. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): enforce key, team, user, member and end-user budgets on shared spend counters Adds a built-in budgets hook that runs the proxy's budget checks, in its order and with its messages, on the proxy's Redis spend:* counters. Each request reads and seeds the counters, checks them and reserves its maximum cost in one Redis script, then settles the reservation to its real cost off the request path, so replicas and Python proxies sharing the Redis enforce one budget. Without Redis the counters are kept per replica. The key store now reads user, team member and end-user budgets, and auth no longer checks budgets against stored spend. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): enforce key, user, team, member and end-user rate limits on shared counters Adds a built-in rate limits hook with the proxy's v3 limiter semantics: requests per window, tokens per window and in-flight requests per key, on the proxy's own {scope:id}:window|requests|tokens|max_parallel_requests Redis keys and with its scripts, so replicas and Python proxies sharing the Redis enforce one limit. Tokens are reserved from the proxy's estimate and settled to usage when the response ends. Refusals carry the proxy's message, body and headers on each endpoint, and complete responses carry its x-ratelimit-* headers. Without Redis the counters are kept per replica. The key store now reads key, user, team and member rpm/tpm limits and the key's max_parallel_requests, and end users' budget limits are read with their budgets; keys with plain rate limits are no longer refused. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): configurable header, spend metadata and provider metadata hooks Adds four built-in hooks a config lists under litellm_settings.hooks and runs in that order: forward_headers copies named client headers to the provider, strip_headers removes headers from the provider request (including the gateway's default pass-through), spend_metadata adds header and key-metadata values to the spend log's spend_logs_metadata, and provider_metadata sets Messages body fields and headers from templates over request facts. Hooks' upstream header edits now reach the provider on both endpoints, and body edits can set nested fields. Credentials and signer headers are never forwarded or set, and a misconfigured hook stops the gateway at startup. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): model capacity hook, the proxy's dynamic rate limiter Adds the model_capacity hook, turned on by listing dynamic_rate_limiter_v3 (or model_capacity) in litellm_settings.callbacks. It caps each model group at the sum of its deployments' rpm and tpm across every replica, and with priority_reservation holds each priority's pool to its share once the group's saturation reaches the threshold, counting it below. It uses the proxy's model_saturation_check and priority_model keys and scripts, its refusal messages and headers, and adds a successful response's tokens to the group and the pool admission chose. The rate limit counters gain a general check-and-add over request and token pairs, and the throttling refusal shape is shared. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * test(gateway-auth): expect the narrowed unsupported rate limit message Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): prometheus observer with the proxy's metric and label names Adds the prometheus observer, turned on by listing prometheus in litellm_settings.callbacks, success_callback or failure_callback. Each finished request updates the proxy's request, spend, token, latency and deployment metrics under its names and labels, and /metrics serves them after the gateway's own. prometheus_metrics_config, the exclude lists and prometheus_latency_buckets are read as the proxy reads them. The metrics the gateway already served under the same names come from the observer alone when it is on. Outcomes now carry when the attempt was sent upstream, for the LLM API latency and time to first token. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): s3_v2 observer writing the proxy's logging payload to S3 Adds the s3_v2 observer, turned on by listing s3_v2 (or s3_log) in litellm_settings.callbacks, success_callback or failure_callback. Each finished request becomes the proxy's StandardLoggingPayload, with the messages from the request body and the response as the client got it, streams put back together. It is queued and written every flush interval under the proxy's key layout, with its Content-MD5, Content-Disposition and encryption headers, one PUT per request or one NDJSON object per folder with s3_batch_file_upload. s3_callback_params is read as the proxy reads it, os.environ references included. Uploads are retried, dropped and kept for the next flush by the proxy's rules, and what is queued is written at shutdown. auth-aws gains an S3 signer that signs the path as sent and every header, as botocore does. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0138knkqKpkoSr6PApenZan5 * feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough Adds azure/ chat completions, streaming and not, on the same passthrough the openai/ route uses. The request goes to the deployment path with the api-version query, or to /openai/v1/ for the v1, latest and preview versions, with api_base, api_key and api_version falling back to the AZURE_* variables and os.environ references resolved as the proxy does. A key is sent as api-key, and an azure_ad_token or an Entra ID token from auth-azure goes as a Bearer token. The body follows the proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the deployment name and reading model capabilities from the cost map, so an unsupported param is refused unless drop_params allows it to be dropped. Spend is priced as the proxy prices Azure, by azure/ when one is set and otherwise by the model the response names, and a content filter error keeps Azure's inner error for the client * feat(gateway): Anthropic and Bedrock chat completions streaming Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/ deployments, which were refused with 501 before. The request is the non-streaming one with "stream": true added for Anthropic and sent to /converse-stream for Bedrock, and the provider's events are decoded the way the proxy's chunk parsers decode them: text, tool call deltas with empty arguments sent as "{}", reasoning_content and thinking_blocks with their index and signature, citations and the mapped finish_reason. A wrapper in the shape of the proxy's stream wrapper gives every chunk one id and the client's model name, sends role on the first chunk, adds the empty finish chunk, strips usage from the chunks and, when stream_options.include_usage is set, ends with the usage chunk before [DONE]. The gateway waits for the first chunk before answering, so an upstream error up to there is retried or falls back to another deployment, and an error after it ends the stream with the OpenAI error frame. Usage, cache reads and writes included, is handed to the meter through the response whether or not the client asked for it, and dropping the client stream drops the upstream request * feat(gateway): full OpenAI chat request translation for Anthropic and Bedrock Anthropic and Bedrock Converse deployments now take the whole OpenAI chat request the Python proxy accepts instead of a narrow allowlist. Messages translate as Python builds them: system as a string, parts or several messages, text, image_url (URL or data URL), file and document parts, assistant tool_calls, the tool and function roles, consecutive role merging, empty content, cache_control on Anthropic and cachePoint on Bedrock. Params map per provider: max_tokens and max_completion_tokens with Anthropic's model default, temperature, top_p, stop, tools and tool_choice, parallel_tool_calls, response_format as a json tool or native output format, reasoning_effort and thinking, user as metadata, with n and any other unsupported param refused with a 400 unless drop_params is set globally, on the deployment or on the request, and allowed_openai_params and additional_drop_params honoured. Responses carry tool calls, thinking blocks and response_format answers back in OpenAI shape, streamed and not. The route downloads media URLs a provider cannot take by reference, merges the beta headers a request needs with forwarded ones, and the gateway hands the cost map to both providers for the capability rules that read it. * fix(rust-bridge): read Python's model_cost for chat translation capabilities Chat translation for Anthropic and Bedrock reads capability flags from the cost map. The Python bridge passed none, so every SDK chat call through the Rust path failed. It now reads litellm.model_cost live, so lookups match Python's, runtime-registered models included. ModelInfo::entry returns a Cow so a map read from Python can hand over a copy. Two tests sent temperature != 1 to a model that removed sampling params, which Python itself refuses; they now send temperature=1. * fix(model-catalog): read token counts the cost map writes as whole floats The published cost map now has prompt_cache_min_tokens: 4096.0, which the typed u64 fields rejected, so the gateway dropped the whole map at startup and priced every request at zero. Counts accept any whole non-negative number, as Python reads any JSON number, and a test keeps the checked-in map parsing. * feat(gateway): Azure OpenAI chat completions through the OpenAI passthrough (#47074) Adds azure/ chat completions, streaming and not, on the same passthrough the openai/ route uses. The request goes to the deployment path with the api-version query, or to /openai/v1/ for the v1, latest and preview versions, with api_base, api_key and api_version falling back to the AZURE_* variables and os.environ references resolved as the proxy does. A key is sent as api-key, and an azure_ad_token or an Entra ID token from auth-azure goes as a Bearer token. The body follows the proxy's gpt, o-series and gpt-5 param rules, picked by base_model or the deployment name and reading model capabilities from the cost map, so an unsupported param is refused unless drop_params allows it to be dropped. Spend is priced as the proxy prices Azure, by azure/ when one is set and otherwise by the model the response names, and a content filter error keeps Azure's inner error for the client * feat(gateway): Anthropic and Bedrock chat completions streaming (#47077) Adds "stream": true to /v1/chat/completions for anthropic/ and bedrock/ deployments, which were refused with 501 before. The request is the non-streaming one with "stream": true added for Anthropic and sent to /converse-stream for Bedrock, and the provider's events are decoded the way the proxy's chunk parsers decode them: text, tool call deltas with empty arguments sent as "{}", reasoning_content and thinking_blocks with their index and signature, citations and the mapped finish_reason. A wrapper in the shape of the proxy's stream wrapper gives every chunk one id and the client's model name, sends role on the first chunk, adds the empty finish chunk, strips usage from the chunks and, when stream_options.include_usage is set, ends with the usage chunk before [DONE]. The gateway waits for the first chunk before answering, so an upstream error up to there is retried or falls back to another deployment, and an error after it ends the stream with the OpenAI error frame. Usage, cache reads and writes included, is handed to the meter through the response whether or not the client asked for it, and dropping the client stream drops the upstream request --------- Co-authored-by: Claude Opus 5.5 (1M context) * feat(gateway): gemini chat completions on google ai studio and vertex ai (#47098) * feat(gateway): Gemini chat completions on Google AI Studio and Vertex AI gemini/ deployments and vertex_ai/ deployments of Gemini models now serve /v1/chat/completions, streamed and not, instead of being refused. Both surfaces share one translation ported from the Python VertexGeminiConfig. Params map as Python maps them: temperature, top_p, max_tokens and max_completion_tokens, n, stop, penalties where the model takes them, seed, logprobs, tools with function declarations and the built in search, code execution, url context, maps and computer use tools, tool_choice, response_format as a response schema, reasoning_effort and thinking as a thinking config, modalities, audio and labels. Messages become contents with a system instruction when the cost map allows it, images, files, audio, function calls and responses, and thought signatures carried back on tool call ids AI Studio sends the key in x-goog-api-key from the deployment or GOOGLE_API_KEY, GEMINI_API_KEY or PALM_API_KEY, and calls v1alpha for Gemini 3 and v1beta otherwise. Vertex AI calls the project and location URL with a forwarded bearer or a Google access token from vertex_credentials, the environment or ADC, and grafts the path onto a custom api_base. Responses carry text, reasoning, tool calls, grounding annotations, logprobs and the vertex_ai metadata, a blocked prompt or flagged answer comes back as content_filter, and usage counts thoughts and cached tokens as Python does so cost is recorded. The gateway resolves os.environ references in the deployment key, base and Vertex params, passes the deployment Vertex params to the call, and loads the cost map for gemini and vertex_ai deployments. Claude and partner models on Vertex AI stay refused * test(gateway): gemini cost map test double returns a Cow entry * feat(gateway): native chat for openai-compatible providers (#47099) Serve chat completions for hosted_vllm, nvidia_nim, groq, together_ai, fireworks_ai, xai, mistral, deepseek, openrouter, cerebras and perplexity through the OpenAI passthrough, with each provider's Python config ported as a request transform in llms and its response and stream fixups applied on the way back. Streams ask for stream_options.include_usage on the client's behalf where the provider takes it, record the usage through ReportedUsage for the spend hooks, and keep it from the client unless the client asked for it. * feat(gateway): claude chat completions on vertex ai (#47101) vertex_ai/ deployments of Claude models now serve /v1/chat/completions, streamed and not, instead of being refused. The request is the Anthropic chat translation built for the vertex_ai host, as Python's VertexAIAnthropicConfig builds it on top of AnthropicConfig: capabilities read the vertex_ai/ cost map entry first and native structured output reads only that entry, service_tier and speed are refused or dropped with drop_params, and the Anthropic API betas are not added. The body drops model, carries anthropic_version vertex-2023-10-16, folds output_format into output_config and keeps effort only for models that take it, and names its context management and caller betas in anthropic_beta and the anthropic-beta header, filtered to what Vertex accepts, with the web search beta alone when there are none. The call goes to the publisher rawPredict URL for the project and location, regional or global, or a custom api_base or VERTEXAI_API_BASE, with a Google access token from vertex_credentials, the environment or ADC. Streams call streamRawPredict with stream true, alt=sse on a custom base, decode the Anthropic events as the anthropic/ path does and hand usage to the hooks through ReportedUsage. Every remote image is downloaded first, since Vertex takes none by URL, and the response names the requested model. * fix(gateway): find anthropic stream usage by event name (#47206) The stream meter kept a data line only when "message_start" or "message_delta" appeared in its first 96 bytes. An upstream that puts "type" after the message object (JSON key order is not fixed) had every message_start skipped, so input tokens were recorded as 0 and spend, budgets, rate limits, metrics and S3 logs undercounted. Anthropic streams name every event, so the event: line now decides; the byte-window check remains for streams without event lines. * fix(gateway): stop hooks pinning connection read buffers for a stream's life (#47220) The s3 logger asks for the response text, and the tap kept every response chunk as a Bytes slice of the upstream connection's read buffer, so each open stream held a buffer per chunk until it ended (about 460 KB per open 10k-token stream locally). Chunks are now copied into one owned buffer. The same applies on the request side: the facts kept the raw request slice and a clone of the client headers, both slices of the client connection's read buffer. The body is now released once hooks have run (an owned copy is kept only when an observer reads it) and the headers are copied. * feat(loadtest): full-feature mode with recorded-data verification (#47207) * feat(loadtest): full-feature mode with recorded-data verification genconfig -features full turns on what both gateways do per request: owners with rpm/tpm/budgets/max_parallel, deployment capacity, the dynamic_rate_limiter_v3, prometheus and s3_v2 callbacks, and the four request hooks with Python stand-ins (custom_callbacks.py). -key-sets gives each gateway its own keys, users and teams in a shared database. verify checks spend logs, owner spend, daily tables, Prometheus counters, S3 logs and the headers the provider received against what loadgen sent, and compares per-deployment prices across sides. s3stub serves S3 for the logs. mockllm takes in= for exact input tokens, returns unique response ids, and counts chosen request headers; loadgen takes -keys and -header. * feat(loadtest): s3stub -max-string keeps long runs' logs in memory * feat(loadtest): bifrost as a third side of the full-feature run (#47236) genconfig writes a "bifrost" key set as a Bifrost config: a custom provider per deployment, a routing rule per model group, virtual keys, and with -features full key/team/customer budgets and rate limits, model configs for deployment limits, the logs store with S3 payloads, the telemetry plugin and header forwarding/stripping/logging. A pricing file copies the cost map's prices per deployment. mockllm answers Bedrock Converse (streamed and not), serves a self-signed TLS listener for Bifrost's HTTPS-only Bedrock provider, and echoes the request's model, which Bifrost prices by. s3stub answers HeadBucket. loadgen takes -chat-path and -messages-path for Bifrost's /anthropic/v1/messages. verify reads a side named bifrost in Bifrost's schema and reports what it has no counterpart for (users, members, daily tables, provider_metadata, cross-replica state) as n/a; with three sides it compares every pair. Claude-Session: https://claude.ai/code/session_01XF7rGf49TWYH5Hqkb14eSL Co-authored-by: Claude Opus 5.5 (1M context) * build(deps): bump aiohttp to 3.14.5 to clear osv-scan (#47369) Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- litellm-rust/Cargo.lock | 3 + .../crates/core-utils/src/core_helpers.rs | 27 + .../core-utils/src/get_llm_provider_logic.rs | 20 + .../crates/gateway-hooks/src/facts.rs | 38 +- .../crates/gateway-hooks/src/pipeline.rs | 67 +- .../crates/gateway-hooks/src/usage.rs | 73 +- .../gateway-inference/src/chat_completions.rs | 90 +- .../crates/gateway-inference/src/hooks.rs | 39 +- .../crates/gateway-inference/src/lib.rs | 12 +- .../crates/gateway-inference/tests/routes.rs | 2 + litellm-rust/crates/gateway/src/hooks.rs | 50 +- .../crates/gateway/src/rate_limits.rs | 2 +- .../crates/gateway/src/s3_log/payload.rs | 2 +- .../crates/gateway/tests/chat_translation.rs | 722 ++++++ .../crates/gateway/tests/gemini_chat.rs | 559 +++++ .../gateway/tests/openai_compat_chat.rs | 442 ++++ .../gateway/tests/vertex_claude_chat.rs | 624 +++++ litellm-rust/crates/inference-chat/Cargo.toml | 5 +- .../crates/inference-chat/src/azure.rs | 1 + .../crates/inference-chat/src/common_utils.rs | 58 +- .../crates/inference-chat/src/handler.rs | 6 +- litellm-rust/crates/inference-chat/src/lib.rs | 16 +- .../crates/inference-chat/src/media.rs | 158 ++ .../crates/inference-chat/src/openai.rs | 42 +- .../inference-chat/src/openai_compatible.rs | 416 ++++ .../crates/inference-chat/src/prepare.rs | 78 +- .../crates/inference-chat/src/route.rs | 2 + .../crates/inference-chat/src/stream.rs | 9 +- .../crates/inference-chat/src/types.rs | 27 +- .../crates/inference-chat/tests/caching.rs | 6 + .../inference-chat/tests/chat_completions.rs | 6 +- .../src/formats/chat_completions/response.rs | 21 + .../src/providers/vertex_ai/AGENTS.md | 1 + .../src/providers/vertex_ai/constants.rs | 10 + litellm-rust/crates/llms/Cargo.toml | 2 + .../crates/llms/src/anthropic/chat/AGENTS.md | 1 + .../crates/llms/src/anthropic/chat/handler.rs | 87 +- .../llms/src/anthropic/chat/messages.rs | 777 ++++++ .../crates/llms/src/anthropic/chat/mod.rs | 2 + .../crates/llms/src/anthropic/chat/params.rs | 1415 +++++++++++ .../llms/src/anthropic/chat/transformation.rs | 1650 +++++++++---- .../llms/src/azure/chat/transformation.rs | 104 +- .../crates/llms/src/base_llm/chat/media.rs | 209 ++ .../crates/llms/src/base_llm/chat/mod.rs | 3 + .../crates/llms/src/base_llm/chat/params.rs | 434 ++++ .../llms/src/base_llm/chat/response_format.rs | 445 ++++ .../llms/src/base_llm/chat/streaming.rs | 8 + .../llms/src/base_llm/chat/transformation.rs | 23 +- .../crates/llms/src/base_llm/model_info.rs | 14 +- .../src/bedrock/chat/converse_messages.rs | 1343 ++++++++++ .../llms/src/bedrock/chat/converse_params.rs | 1039 ++++++++ .../bedrock/chat/converse_transformation.rs | 1155 ++++++--- .../llms/src/bedrock/chat/invoke_handler.rs | 83 +- .../crates/llms/src/bedrock/chat/mod.rs | 2 + .../crates/llms/src/cerebras/chat/AGENTS.md | 10 + .../crates/llms/src/cerebras/chat/mod.rs | 1 + .../llms/src/cerebras/chat/transformation.rs | 163 ++ litellm-rust/crates/llms/src/cerebras/mod.rs | 1 + .../crates/llms/src/deepseek/chat/AGENTS.md | 11 + .../crates/llms/src/deepseek/chat/mod.rs | 1 + .../llms/src/deepseek/chat/transformation.rs | 352 +++ litellm-rust/crates/llms/src/deepseek/mod.rs | 1 + .../llms/src/fireworks_ai/chat/AGENTS.md | 12 + .../crates/llms/src/fireworks_ai/chat/mod.rs | 1 + .../src/fireworks_ai/chat/transformation.rs | 680 ++++++ .../crates/llms/src/fireworks_ai/mod.rs | 1 + .../crates/llms/src/gemini/chat/AGENTS.md | 11 + .../crates/llms/src/gemini/chat/mod.rs | 1 + .../llms/src/gemini/chat/transformation.rs | 252 ++ litellm-rust/crates/llms/src/gemini/mod.rs | 1 + .../crates/llms/src/groq/chat/AGENTS.md | 12 + litellm-rust/crates/llms/src/groq/chat/mod.rs | 1 + .../llms/src/groq/chat/transformation.rs | 368 +++ litellm-rust/crates/llms/src/groq/mod.rs | 1 + .../llms/src/hosted_vllm/chat/AGENTS.md | 11 + .../crates/llms/src/hosted_vllm/chat/mod.rs | 1 + .../src/hosted_vllm/chat/transformation.rs | 385 +++ .../crates/llms/src/hosted_vllm/mod.rs | 1 + litellm-rust/crates/llms/src/lib.rs | 9 + .../crates/llms/src/mistral/chat/AGENTS.md | 11 + .../crates/llms/src/mistral/chat/mod.rs | 1 + .../llms/src/mistral/chat/transformation.rs | 605 +++++ litellm-rust/crates/llms/src/mistral/mod.rs | 1 + .../crates/llms/src/nvidia_nim/chat/AGENTS.md | 11 + .../crates/llms/src/nvidia_nim/chat/mod.rs | 1 + .../src/nvidia_nim/chat/transformation.rs | 196 ++ .../crates/llms/src/nvidia_nim/mod.rs | 1 + .../crates/llms/src/openai/chat/AGENTS.md | 1 + .../crates/llms/src/openai/chat/compatible.rs | 985 ++++++++ .../crates/llms/src/openai/chat/mod.rs | 1 + .../src/openai_like/chat/transformation.rs | 25 +- .../crates/llms/src/openrouter/chat/AGENTS.md | 11 + .../crates/llms/src/openrouter/chat/mod.rs | 1 + .../src/openrouter/chat/transformation.rs | 231 ++ .../crates/llms/src/openrouter/mod.rs | 1 + .../crates/llms/src/perplexity/chat/AGENTS.md | 10 + .../crates/llms/src/perplexity/chat/mod.rs | 1 + .../src/perplexity/chat/transformation.rs | 283 +++ .../crates/llms/src/perplexity/mod.rs | 1 + .../llms/src/together_ai/chat/AGENTS.md | 11 + .../crates/llms/src/together_ai/chat/mod.rs | 1 + .../src/together_ai/chat/transformation.rs | 324 +++ .../crates/llms/src/together_ai/mod.rs | 1 + .../crates/llms/src/vertex_ai/common_utils.rs | 80 + .../llms/src/vertex_ai/gemini/AGENTS.md | 13 + .../llms/src/vertex_ai/gemini/handler.rs | 266 ++ .../crates/llms/src/vertex_ai/gemini/mod.rs | 4 + .../llms/src/vertex_ai/gemini/schema.rs | 461 ++++ .../src/vertex_ai/gemini/transformation.rs | 1538 ++++++++++++ .../vertex_and_google_ai_studio_gemini.rs | 2174 +++++++++++++++++ litellm-rust/crates/llms/src/vertex_ai/mod.rs | 2 + .../anthropic/AGENTS.md | 12 + .../vertex_ai_partner_models/anthropic/mod.rs | 1 + .../anthropic/transformation.rs | 847 +++++++ .../vertex_ai/vertex_ai_partner_models/mod.rs | 1 + .../crates/llms/src/xai/chat/AGENTS.md | 11 + litellm-rust/crates/llms/src/xai/chat/mod.rs | 1 + .../llms/src/xai/chat/transformation.rs | 272 +++ litellm-rust/crates/llms/src/xai/mod.rs | 1 + .../src/routes/chat_completions/host.rs | 7 +- .../src/routes/chat_completions/mod.rs | 1 + .../src/routes/chat_completions/model_cost.rs | 88 + litellm-rust/crates/python-compat/src/json.rs | 44 +- .../docs/customer-proxy/gemini-chat.svg | 95 + .../customer-proxy/openai-compat-chat.svg | 98 + .../customer-proxy/vertex-claude-chat.svg | 98 + litellm-rust/loadtest/Dockerfile | 2 +- litellm-rust/loadtest/README.md | 72 +- .../loadtest/cmd/genconfig/bifrost.go | 261 ++ .../cmd/genconfig/custom_callbacks.py | 123 + litellm-rust/loadtest/cmd/genconfig/full.go | 174 ++ litellm-rust/loadtest/cmd/genconfig/main.go | 107 +- litellm-rust/loadtest/cmd/loadgen/body.go | 2 +- litellm-rust/loadtest/cmd/loadgen/main.go | 39 +- litellm-rust/loadtest/cmd/mockllm/main.go | 174 +- .../loadtest/cmd/mockllm/main_test.go | 27 + litellm-rust/loadtest/cmd/s3stub/main.go | 164 ++ litellm-rust/loadtest/cmd/s3stub/main_test.go | 15 + litellm-rust/loadtest/cmd/verify/bifrost.go | 190 ++ litellm-rust/loadtest/cmd/verify/compare.go | 83 + litellm-rust/loadtest/cmd/verify/db.go | 183 ++ litellm-rust/loadtest/cmd/verify/main.go | 262 ++ litellm-rust/loadtest/cmd/verify/metrics.go | 171 ++ litellm-rust/loadtest/cmd/verify/mock.go | 69 + litellm-rust/loadtest/cmd/verify/s3.go | 161 ++ .../loadtest/cmd/verify/verify_test.go | 162 ++ .../chat_completions/test_callbacks.py | 4 +- .../chat_completions/test_requests.py | 3 +- uv.lock | 475 ++-- 149 files changed, 25167 insertions(+), 1304 deletions(-) create mode 100644 litellm-rust/crates/gateway/tests/chat_translation.rs create mode 100644 litellm-rust/crates/gateway/tests/gemini_chat.rs create mode 100644 litellm-rust/crates/gateway/tests/openai_compat_chat.rs create mode 100644 litellm-rust/crates/gateway/tests/vertex_claude_chat.rs create mode 100644 litellm-rust/crates/inference-chat/src/media.rs create mode 100644 litellm-rust/crates/inference-chat/src/openai_compatible.rs create mode 100644 litellm-rust/crates/llms/src/anthropic/chat/messages.rs create mode 100644 litellm-rust/crates/llms/src/anthropic/chat/params.rs create mode 100644 litellm-rust/crates/llms/src/base_llm/chat/media.rs create mode 100644 litellm-rust/crates/llms/src/base_llm/chat/params.rs create mode 100644 litellm-rust/crates/llms/src/base_llm/chat/response_format.rs create mode 100644 litellm-rust/crates/llms/src/bedrock/chat/converse_messages.rs create mode 100644 litellm-rust/crates/llms/src/bedrock/chat/converse_params.rs create mode 100644 litellm-rust/crates/llms/src/cerebras/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/cerebras/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/cerebras/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/cerebras/mod.rs create mode 100644 litellm-rust/crates/llms/src/deepseek/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/deepseek/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/deepseek/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/fireworks_ai/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/fireworks_ai/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/fireworks_ai/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/fireworks_ai/mod.rs create mode 100644 litellm-rust/crates/llms/src/gemini/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/gemini/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/gemini/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/gemini/mod.rs create mode 100644 litellm-rust/crates/llms/src/groq/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/groq/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/groq/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/groq/mod.rs create mode 100644 litellm-rust/crates/llms/src/hosted_vllm/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/hosted_vllm/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/hosted_vllm/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/hosted_vllm/mod.rs create mode 100644 litellm-rust/crates/llms/src/mistral/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/mistral/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/mistral/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/nvidia_nim/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/nvidia_nim/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/nvidia_nim/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/nvidia_nim/mod.rs create mode 100644 litellm-rust/crates/llms/src/openai/chat/compatible.rs create mode 100644 litellm-rust/crates/llms/src/openrouter/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/openrouter/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/openrouter/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/perplexity/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/perplexity/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/perplexity/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/perplexity/mod.rs create mode 100644 litellm-rust/crates/llms/src/together_ai/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/together_ai/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/together_ai/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/together_ai/mod.rs create mode 100644 litellm-rust/crates/llms/src/vertex_ai/gemini/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/vertex_ai/gemini/handler.rs create mode 100644 litellm-rust/crates/llms/src/vertex_ai/gemini/mod.rs create mode 100644 litellm-rust/crates/llms/src/vertex_ai/gemini/schema.rs create mode 100644 litellm-rust/crates/llms/src/vertex_ai/gemini/transformation.rs create mode 100644 litellm-rust/crates/llms/src/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.rs create mode 100644 litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/mod.rs create mode 100644 litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/transformation.rs create mode 100644 litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/mod.rs create mode 100644 litellm-rust/crates/llms/src/xai/chat/AGENTS.md create mode 100644 litellm-rust/crates/llms/src/xai/chat/mod.rs create mode 100644 litellm-rust/crates/llms/src/xai/chat/transformation.rs create mode 100644 litellm-rust/crates/llms/src/xai/mod.rs create mode 100644 litellm-rust/crates/python-bridge/src/routes/chat_completions/model_cost.rs create mode 100644 litellm-rust/docs/customer-proxy/gemini-chat.svg create mode 100644 litellm-rust/docs/customer-proxy/openai-compat-chat.svg create mode 100644 litellm-rust/docs/customer-proxy/vertex-claude-chat.svg create mode 100644 litellm-rust/loadtest/cmd/genconfig/bifrost.go create mode 100644 litellm-rust/loadtest/cmd/genconfig/custom_callbacks.py create mode 100644 litellm-rust/loadtest/cmd/genconfig/full.go create mode 100644 litellm-rust/loadtest/cmd/mockllm/main_test.go create mode 100644 litellm-rust/loadtest/cmd/s3stub/main.go create mode 100644 litellm-rust/loadtest/cmd/s3stub/main_test.go create mode 100644 litellm-rust/loadtest/cmd/verify/bifrost.go create mode 100644 litellm-rust/loadtest/cmd/verify/compare.go create mode 100644 litellm-rust/loadtest/cmd/verify/db.go create mode 100644 litellm-rust/loadtest/cmd/verify/main.go create mode 100644 litellm-rust/loadtest/cmd/verify/metrics.go create mode 100644 litellm-rust/loadtest/cmd/verify/mock.go create mode 100644 litellm-rust/loadtest/cmd/verify/s3.go create mode 100644 litellm-rust/loadtest/cmd/verify/verify_test.go diff --git a/litellm-rust/Cargo.lock b/litellm-rust/Cargo.lock index 4b741c993e4..1c4e6553b66 100644 --- a/litellm-rust/Cargo.lock +++ b/litellm-rust/Cargo.lock @@ -4055,6 +4055,7 @@ dependencies = [ name = "litellm-inference-chat" version = "0.1.0" dependencies = [ + "base64 0.22.1", "bytes", "futures-util", "litellm-auth", @@ -4235,11 +4236,13 @@ dependencies = [ "serde_json", "serde_path_to_error", "serde_with", + "sha2 0.10.9", "strum", "thiserror 2.0.19", "time", "tokio", "url", + "uuid", ] [[package]] diff --git a/litellm-rust/crates/core-utils/src/core_helpers.rs b/litellm-rust/crates/core-utils/src/core_helpers.rs index 94268f4b452..bf32b89066d 100644 --- a/litellm-rust/crates/core-utils/src/core_helpers.rs +++ b/litellm-rust/crates/core-utils/src/core_helpers.rs @@ -21,6 +21,28 @@ const FINISH_REASONS: &[(&str, &str)] = &[ ("content_filter", "content_filter"), ("stop", "stop"), ("length", "length"), + ("STOP", "stop"), + ("MAX_TOKENS", "length"), + ("CONTINUATION", "length"), + ("SAFETY", "content_filter"), + ("RECITATION", "content_filter"), + ("FINISH_REASON_UNSPECIFIED", "stop"), + ("MALFORMED_FUNCTION_CALL", "stop"), + ("LANGUAGE", "content_filter"), + ("OTHER", "content_filter"), + ("BLOCKLIST", "content_filter"), + ("PROHIBITED_CONTENT", "content_filter"), + ("SPII", "content_filter"), + ("IMAGE_SAFETY", "content_filter"), + ("IMAGE_PROHIBITED_CONTENT", "content_filter"), + ("TOO_MANY_TOOL_CALLS", "stop"), + ("MALFORMED_RESPONSE", "stop"), + ("NO_IMAGE", "content_filter"), + ("IMAGE_RECITATION", "content_filter"), + ("IMAGE_OTHER", "content_filter"), + ("ESCALATION", "content_filter"), + ("UNEXPECTED_TOOL_CALL", "stop"), + ("MISSING_THOUGHT_SIGNATURE", "stop"), ]; pub fn finish_reason_for(provider_reason: &str) -> &'static str { @@ -84,6 +106,11 @@ mod tests { #[case::guardrail_intervened("guardrail_intervened", "content_filter")] #[case::content_filtered("content_filtered", "content_filter")] #[case::content_filter("content_filter", "content_filter")] + #[case::gemini_stop("STOP", "stop")] + #[case::gemini_max_tokens("MAX_TOKENS", "length")] + #[case::gemini_safety("SAFETY", "content_filter")] + #[case::gemini_malformed_call("MALFORMED_FUNCTION_CALL", "stop")] + #[case::gemini_other("OTHER", "content_filter")] fn maps_every_reason_the_route_can_observe(#[case] reason: &str, #[case] expected: &str) { assert_eq!(finish_reason_for(reason), expected); } diff --git a/litellm-rust/crates/core-utils/src/get_llm_provider_logic.rs b/litellm-rust/crates/core-utils/src/get_llm_provider_logic.rs index eb352ea3ba1..4507ac20093 100644 --- a/litellm-rust/crates/core-utils/src/get_llm_provider_logic.rs +++ b/litellm-rust/crates/core-utils/src/get_llm_provider_logic.rs @@ -21,28 +21,48 @@ pub enum LlmProviders { Bedrock, #[strum(serialize = "bedrock_mantle")] BedrockMantle, + #[strum(serialize = "cerebras")] + Cerebras, #[strum(serialize = "cohere")] Cohere, #[strum(serialize = "deepseek")] Deepseek, #[strum(serialize = "edenai")] Edenai, + #[strum(serialize = "fireworks_ai")] + FireworksAi, + /// Google AI Studio (`gemini/`), the Gemini API addressed with an API key. + #[strum(serialize = "gemini")] + Gemini, + #[strum(serialize = "groq")] + Groq, + /// A vLLM server (`hosted_vllm/`) at the deployment's `api_base`. + #[strum(serialize = "hosted_vllm")] + HostedVllm, #[strum(serialize = "minimax")] Minimax, #[strum(serialize = "mistral")] Mistral, + #[strum(serialize = "nvidia_nim")] + NvidiaNim, #[strum(serialize = "openai")] Openai, #[strum(serialize = "openai_like")] OpenaiLike, #[strum(serialize = "openrouter")] Openrouter, + #[strum(serialize = "perplexity")] + Perplexity, #[strum(serialize = "reducto")] Reducto, #[strum(serialize = "tencent")] Tencent, + #[strum(serialize = "together_ai")] + TogetherAi, #[strum(serialize = "vertex_ai")] VertexAi, + #[strum(serialize = "xai")] + Xai, } pub fn get_custom_llm_provider<'a>( diff --git a/litellm-rust/crates/gateway-hooks/src/facts.rs b/litellm-rust/crates/gateway-hooks/src/facts.rs index 3b05ce4bb91..2b8d8dc9e11 100644 --- a/litellm-rust/crates/gateway-hooks/src/facts.rs +++ b/litellm-rust/crates/gateway-hooks/src/facts.rs @@ -1,6 +1,6 @@ use std::{ ops::{BitOr, BitOrAssign}, - sync::{Arc, OnceLock}, + sync::{Arc, Mutex, OnceLock}, time::{Duration, SystemTime}, }; @@ -34,7 +34,8 @@ impl Facts { pub const TIMINGS: Self = Self(1 << 7); /// Values earlier hooks attached for observers. pub const ANNOTATIONS: Self = Self(1 << 8); - /// The raw request bytes, already held for retries. + /// The raw request bytes, already held for retries. Hooks read them while the request is + /// admitted; only an observer that asks for them keeps a copy for the rest of the request. pub const REQUEST_BODY: Self = Self(1 << 9); /// The response text. The only fact that makes the gateway keep a stream's bytes. pub const RESPONSE_BODY: Self = Self(1 << 10); @@ -121,7 +122,11 @@ pub struct RequestFacts { /// The output budget the request set, if any. pub max_tokens: Option, pub started_at: SystemTime, - body: Bytes, + /// The request bytes while hooks run. Afterwards a compact copy when an observer reads them, + /// else empty: the raw bytes are a slice of the connection's read buffer, and holding them + /// would hold the whole buffer for as long as the response streams. + body: Mutex, + body_len: usize, input_tokens: OnceLock, } @@ -148,21 +153,38 @@ impl RequestFacts { stream, max_tokens, started_at: SystemTime::now(), - body, + body_len: body.len(), + body: Mutex::new(body), input_tokens: OnceLock::new(), } } - /// The raw request bytes. Shared, never copied. - pub fn body(&self) -> &Bytes { - &self.body + /// The request bytes: in full while hooks run, afterwards only for observers that read + /// [`Facts::REQUEST_BODY`], empty for everyone else. + pub fn body(&self) -> Bytes { + self.body + .lock() + .unwrap_or_else(|poison| poison.into_inner()) + .clone() + } + + /// Lets go of the raw bytes once hooks have run, keeping an owned copy when `keep`. + pub(crate) fn release_body(&self, keep: bool) { + let mut body = self + .body + .lock() + .unwrap_or_else(|poison| poison.into_inner()); + *body = match keep { + true => Bytes::copy_from_slice(&body), + false => Bytes::new(), + }; } /// The proxy's cheap input estimate, a token per four bytes of request, made once. pub fn input_tokens(&self) -> u64 { *self .input_tokens - .get_or_init(|| (self.body.len() as u64).div_ceil(4).max(1)) + .get_or_init(|| (self.body_len as u64).div_ceil(4).max(1)) } } diff --git a/litellm-rust/crates/gateway-hooks/src/pipeline.rs b/litellm-rust/crates/gateway-hooks/src/pipeline.rs index db18be8b9a1..64afeeb0838 100644 --- a/litellm-rust/crates/gateway-hooks/src/pipeline.rs +++ b/litellm-rust/crates/gateway-hooks/src/pipeline.rs @@ -63,6 +63,21 @@ impl Pipeline { /// Runs every hook in order. When one refuses, the hooks before it are told the request /// ended with that refusal, and so are the observers. pub async fn admit(&self, request: Arc) -> Result { + let admitted = self.run_hooks(request).await; + // Refused or not, the request's facts outlive this call (in observer records, or for + // the whole response), and should not keep the connection's read buffer with them. + let request = match &admitted { + Ok(admission) => &admission.request, + Err((request, _)) => request, + }; + request.release_body(self.observers.reads().contains(Facts::REQUEST_BODY)); + admitted.map_err(|(_, refusal)| refusal) + } + + async fn run_hooks( + &self, + request: Arc, + ) -> Result, Refusal)> { let mut admission = Admission { request, edit: RequestEdit::default(), @@ -84,8 +99,9 @@ impl Pipeline { ended_at: SystemTime::now(), interrupted: false, }; + let request = Arc::clone(&admission.request); self.finish(admission, outcome, None); - return Err(refusal); + return Err((request, refusal)); } } } @@ -304,6 +320,55 @@ mod tests { ); } + struct Bodies; + + impl Observer for Bodies { + fn name(&self) -> &'static str { + "bodies" + } + + fn reads(&self) -> Facts { + Facts::REQUEST_BODY + } + + fn observe<'a>(&'a self, _record: &'a Record) -> BoxFuture<'a, ()> { + Box::pin(async {}) + } + } + + /// Hooks see the request as sent; afterwards the facts hold no slice of the read buffer, + /// only a copy when an observer reads the body. + #[rstest::rstest] + #[case::no_observer_reads_it(Vec::new(), b"")] + #[case::an_observer_reads_it(vec![Arc::new(Bodies) as Arc], b"{\"model\":\"m\"}")] + #[tokio::test] + async fn the_raw_body_is_released_after_the_hooks( + #[case] observers: Vec>, + #[case] kept: &[u8], + ) { + let buffer = Bytes::from_static(b"{\"model\":\"m\"} and the rest of the read buffer"); + let raw = buffer.slice(..13); + let request = Arc::new(RequestFacts::new( + "r".into(), + Route::Messages, + None, + None, + "m".into(), + Arc::new(HeaderMap::new()), + false, + None, + raw.clone(), + )); + assert_eq!(request.body().as_ptr(), raw.as_ptr()); + let pipeline = Pipeline::new(Vec::new(), Observers::start(observers, 8), Arc::default()); + let admission = pipeline.admit(request).await.unwrap(); + + let body = admission.request().body(); + assert_eq!(&body[..], kept); + assert!(body.is_empty() || body.as_ptr() != raw.as_ptr()); + assert_eq!(admission.request().input_tokens(), 4); + } + #[tokio::test] async fn a_refusal_settles_the_hooks_before_it() { let log = Log::default(); diff --git a/litellm-rust/crates/gateway-hooks/src/usage.rs b/litellm-rust/crates/gateway-hooks/src/usage.rs index a3aa04a9fe0..a110fdffc10 100644 --- a/litellm-rust/crates/gateway-hooks/src/usage.rs +++ b/litellm-rust/crates/gateway-hooks/src/usage.rs @@ -206,6 +206,15 @@ enum Line { Skipping, } +/// What the stream's `event:` line named for the event in progress. Anthropic streams name every +/// event, which decides a data line whatever order its JSON keys come in. +#[derive(Clone, Copy, Debug, PartialEq)] +enum Event { + Unnamed, + Usage, + Other, +} + /// Reads usage out of a server-sent-event stream as its chunks pass through. Only the line in /// progress is buffered, and only while it may carry usage. #[derive(Debug)] @@ -213,6 +222,7 @@ pub struct StreamUsage { format: StreamFormat, line: Vec, state: Line, + event: Event, usage: Option, model: Option, } @@ -223,6 +233,7 @@ impl StreamUsage { format, line: Vec::new(), state: Line::Reading, + event: Event::Unnamed, usage: None, model: None, } @@ -239,8 +250,10 @@ impl StreamUsage { self.take(part); } if end.is_some() { - if let Line::Reading = self.state { - self.read_line(); + match self.state { + Line::Reading => self.read_line(), + // A skipped line is content, never the blank line that ends an event. + Line::Skipping => {} } self.line.clear(); self.state = Line::Reading; @@ -294,23 +307,43 @@ impl StreamUsage { return false; } match self.format { - StreamFormat::Anthropic if line.len() >= DECIDE_BYTES => { - let head = &line[..DECIDE_BYTES]; - contains(head, b"\"message_start\"") || contains(head, b"\"message_delta\"") - } + StreamFormat::Anthropic if line.len() >= DECIDE_BYTES => match self.event { + Event::Usage => true, + Event::Other => false, + Event::Unnamed => { + let head = &line[..DECIDE_BYTES]; + contains(head, b"\"message_start\"") || contains(head, b"\"message_delta\"") + } + }, StreamFormat::Anthropic | StreamFormat::OpenAi => true, } } fn read_line(&mut self) { let line = self.line.strip_suffix(b"\r").unwrap_or(&self.line); + if line.is_empty() { + self.event = Event::Unnamed; + return; + } + if let Some(name) = line.strip_prefix(b"event:") { + let name = name.strip_prefix(b" ").unwrap_or(name); + self.event = match name { + b"message_start" | b"message_delta" => Event::Usage, + _ => Event::Other, + }; + return; + } let Some(data) = line.strip_prefix(b"data:") else { return; }; let data = data.strip_prefix(b" ").unwrap_or(data); match self.format { StreamFormat::Anthropic => { - if !(contains(data, b"\"message_start\"") || contains(data, b"\"message_delta\"")) { + let named = self.event == Event::Usage; + if !named + && !(contains(data, b"\"message_start\"") + || contains(data, b"\"message_delta\"")) + { return; } let Ok(event) = serde_json::from_slice::(data) else { @@ -417,6 +450,32 @@ mod tests { assert_eq!(meter.usage(), None); } + /// Key order is not part of JSON: an upstream may put `type` after the message object, past + /// the decision window, and the event line still marks the usage events. + #[rstest] + #[case::whole(usize::MAX)] + #[case::bytes(1)] + #[case::small(7)] + fn anthropic_usage_events_are_found_by_event_name(#[case] size: usize) { + let stream = concat!( + "event: message_start\n", + "data: {\"message\":{\"content\":[],\"id\":\"msg_0123456789abcdef0123456789abcdef\",\"model\":\"claude-x\",\"role\":\"assistant\",\"stop_reason\":null,\"type\":\"message\",\"usage\":{\"input_tokens\":1234,\"output_tokens\":1}},\"type\":\"message_start\"}\n", + "\n", + "event: content_block_delta\n", + "data: {\"delta\":{\"text\":\"text that mentions \\\"message_delta\\\" and is longer than the decision window\",\"type\":\"text_delta\"},\"index\":0,\"type\":\"content_block_delta\",\"usage\":{\"output_tokens\":999}}\n", + "\n", + "event: message_delta\n", + "data: {\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null},\"type\":\"message_delta\",\"usage\":{\"output_tokens\":42}}\n", + "\n", + ); + let mut meter = StreamUsage::new(StreamFormat::Anthropic); + for chunk in stream.as_bytes().chunks(size.min(stream.len())) { + meter.feed(chunk); + } + let usage = meter.usage().expect("usage"); + assert_eq!((usage.input_tokens, usage.output_tokens), (1234, 42)); + } + #[rstest] fn crlf_lines_are_read() { let mut meter = StreamUsage::new(StreamFormat::Anthropic); diff --git a/litellm-rust/crates/gateway-inference/src/chat_completions.rs b/litellm-rust/crates/gateway-inference/src/chat_completions.rs index 6f4442dd053..41a3ac0fbc9 100644 --- a/litellm-rust/crates/gateway-inference/src/chat_completions.rs +++ b/litellm-rust/crates/gateway-inference/src/chat_completions.rs @@ -21,6 +21,8 @@ use litellm_gateway_hooks::Route; use crate::{Deployment, Error, Gateway, JsonBody, request}; +const ENV_REFERENCE: &str = "os.environ/"; + pub(crate) async fn create( State(gateway): State>, identity: AuthenticatedRequest, @@ -135,17 +137,21 @@ async fn handle( ) { return openai(gateway, deployment, body, extra_headers.clone()).await; } - let (messages, optional_params) = split(body); + let (messages, mut optional_params) = split(body); let stream = ChatCompletionsRoute::is_stream(&optional_params); + let (api_key, api_base) = + connection(gateway, deployment, &mut optional_params).await?; let call = ChatCompletionsCall { model: deployment.model.clone(), messages, optional_params, - api_key: deployment.api_key.clone(), - api_base: deployment.api_base.clone(), + api_key, + api_base, custom_llm_provider: deployment.custom_llm_provider.clone(), extra_headers: extra_headers.clone(), timeout: deployment.timeout, + drop_params: Some(drop_params(gateway, deployment)), + model_info: gateway.model_info(), }; if stream { return translated_stream(route, call, name).await; @@ -179,6 +185,60 @@ fn split(mut body: Map) -> (Value, Map) { (messages, body) } +/// A deployment's connection for a provider chat translates. Python's router reads each +/// `os.environ/` setting from the environment when it loads the deployment, and `completion` +/// hands the Vertex AI settings to the call, where a request body cannot name them. +async fn connection( + gateway: &Gateway, + deployment: &Deployment, + optional_params: &mut Map, +) -> Result<(Option, Option), Error> { + let vertex = match serde_json::to_value(&deployment.litellm_params.vertex) { + Ok(Value::Object(fields)) => fields, + _ => Map::new(), + }; + let references: Vec<&str> = [ + deployment.api_key.as_deref(), + deployment.api_base.as_deref(), + ] + .into_iter() + .flatten() + .chain(vertex.values().filter_map(Value::as_str)) + .filter_map(|value| value.strip_prefix(ENV_REFERENCE)) + .collect(); + let secrets = match references.is_empty() { + true => None, + false => Some( + gateway + .secrets + .resolve(&references) + .await + .map_err(|error| Error::Internal(error.to_string()))?, + ), + }; + // A reference to an unset variable counts as unset. + let resolve = |value: &str| match value.strip_prefix(ENV_REFERENCE) { + Some(name) => secrets.as_ref().and_then(|secrets| secrets.get(name)), + None => Some(value.to_owned()), + }; + for (name, value) in vertex { + let value = match value { + Value::String(text) => resolve(&text).map(Value::String), + Value::Null => None, + other => Some(other), + }; + if let Some(value) = value + && !optional_params.contains_key(&name) + { + optional_params.insert(name, value); + } + } + Ok(( + deployment.api_key.as_deref().and_then(resolve), + deployment.api_base.as_deref().and_then(resolve), + )) +} + /// The deployment's `litellm_params` a provider reads beyond its connection: its /// `api_version`, `base_model` and every key the config adds. fn provider_params(deployment: &Deployment) -> Map { @@ -197,12 +257,24 @@ fn provider_params(deployment: &Deployment) -> Map { provider_params } +/// `litellm.drop_params` or the deployment's own: either one set drops what the provider +/// cannot take, as Python's `get_optional_params` reads them. +fn drop_params(gateway: &Gateway, deployment: &Deployment) -> bool { + gateway.drop_params() + || deployment + .litellm_params + .drop_params + .as_ref() + .is_some_and(|flag| flag.is_set()) +} + async fn openai( gateway: &Gateway, deployment: &Deployment, body: Map, extra_headers: Option>, ) -> Result { + let usage = ReportedUsage::default(); let machine = gateway.chat_completions.clone().openai_machine( OpenAiChatCall { model: deployment.model.clone(), @@ -213,13 +285,9 @@ async fn openai( extra_headers, timeout: deployment.timeout, litellm_params: provider_params(deployment), - drop_params: gateway.drop_params() - || deployment - .litellm_params - .drop_params - .as_ref() - .is_some_and(|flag| flag.is_set()), + drop_params: drop_params(gateway, deployment), model_info: gateway.model_info(), + usage: usage.clone(), }, None, ); @@ -227,7 +295,9 @@ async fn openai( |body: Bytes| ([(CONTENT_TYPE, "application/json")], body), |error| Bytes::from(Error::from(error).openai_sse_frame()), ); - Ok(litellm_host_http::serve(machine, (), (), encoder, None).await?) + let mut response = litellm_host_http::serve(machine, (), (), encoder, None).await?; + response.extensions_mut().insert(usage); + Ok(response) } /// A stream from a provider chat translates (Anthropic, Bedrock), sent as OpenAI chunks. The diff --git a/litellm-rust/crates/gateway-inference/src/hooks.rs b/litellm-rust/crates/gateway-inference/src/hooks.rs index 8ecaa26880f..d03cc2b83c4 100644 --- a/litellm-rust/crates/gateway-inference/src/hooks.rs +++ b/litellm-rust/crates/gateway-inference/src/hooks.rs @@ -10,7 +10,7 @@ use std::{ use axum::{ body::{Body, BodyDataStream, Bytes, HttpBody}, - http::{HeaderMap, header::CONTENT_TYPE}, + http::{HeaderMap, HeaderName, HeaderValue, header::CONTENT_TYPE}, response::Response, }; use futures_util::Stream; @@ -81,7 +81,7 @@ pub(crate) async fn admit( .then(|| key_facts(hooks, identity)) .flatten(); let client_headers = match pipeline.reads(Facts::CLIENT_HEADERS) { - true => Arc::new(headers.clone()), + true => Arc::new(owned_headers(headers)), false => Arc::default(), }; let request = RequestFacts::new( @@ -107,6 +107,21 @@ pub(crate) async fn admit( })) } +/// A copy of the client's headers that owns its bytes. Hyper parses header values, and names it +/// does not know, as slices of the connection's read buffer, so a plain clone kept in the facts +/// would keep that whole buffer for as long as the response streams. +fn owned_headers(headers: &HeaderMap) -> HeaderMap { + let mut owned = HeaderMap::with_capacity(headers.len()); + for (name, value) in headers { + let name = + HeaderName::from_bytes(name.as_str().as_bytes()).unwrap_or_else(|_| name.clone()); + let mut copy = HeaderValue::from_bytes(value.as_bytes()).unwrap_or_else(|_| value.clone()); + copy.set_sensitive(value.is_sensitive()); + owned.append(name, copy); + } + owned +} + fn key_facts(hooks: &Hooks, identity: &AuthenticatedRequest) -> Option> { let caller = identity.caller(); let authentication = caller.authentication(); @@ -277,7 +292,7 @@ enum Meter { Off, Stream(StreamUsage), /// A JSON body, read for its `usage` once complete. - Body(Vec), + Body(Vec), } /// Records the request when the response body ends or is dropped. @@ -291,7 +306,10 @@ struct Finish { meter: Meter, /// The usage a translated chat stream ended with, sent to the client or not. reported: Option, - body: Option>, + /// The response text for observers that read it. Copied, not kept as chunks: each chunk is a + /// slice of the upstream connection's read buffer, and holding it would hold that buffer for + /// the rest of the stream. + body: Option>, started: Instant, first_byte: Option, ended: bool, @@ -304,10 +322,10 @@ impl Finish { match &mut self.meter { Meter::Off => {} Meter::Stream(meter) => meter.feed(chunk), - Meter::Body(chunks) => chunks.push(chunk.clone()), + Meter::Body(body) => body.extend_from_slice(chunk), } if let Some(body) = &mut self.body { - body.push(chunk.clone()); + body.extend_from_slice(chunk); } } @@ -322,10 +340,9 @@ impl Finish { Some(usage) => (Some(usage), None), None => (meter.usage(), meter.model().map(str::to_owned)), }, - Meter::Body(chunks) => { - let body = chunks.concat(); - let model = by_model.then(|| body_model(&body)).flatten(); - (body_usage(self.route, &body), model) + Meter::Body(body) => { + let model = by_model.then(|| body_model(body)).flatten(); + (body_usage(self.route, body), model) } }; let cost = by_model @@ -356,7 +373,7 @@ impl Drop for Finish { ended_at: SystemTime::now(), interrupted: !self.ended, }; - let body = self.body.take().map(|chunks| Bytes::from(chunks.concat())); + let body = self.body.take().map(Bytes::from); self.pipeline.finish(admission, outcome, body); } } diff --git a/litellm-rust/crates/gateway-inference/src/lib.rs b/litellm-rust/crates/gateway-inference/src/lib.rs index 65aa2d72779..010e92cfbe1 100644 --- a/litellm-rust/crates/gateway-inference/src/lib.rs +++ b/litellm-rust/crates/gateway-inference/src/lib.rs @@ -18,7 +18,10 @@ mod responses; use std::sync::{Arc, OnceLock}; use axum::{Router, routing::post}; -use litellm_http::{ClientVariant, HttpClientConfig, media::UrlPolicy}; +use litellm_http::{ + ClientVariant, HttpClientConfig, + media::{MediaFetcher, UrlPolicy}, +}; use litellm_inference::resources::CoreResources; use litellm_inference_chat::ChatCompletionsRoute; use litellm_inference_messages::MessagesRoute; @@ -138,7 +141,12 @@ impl Gateway { provider.clone(), auth.clone(), secrets.clone(), - ), + ) + .with_media(MediaFetcher::new( + &resources.pool, + &http, + UrlPolicy::default(), + )?), messages: MessagesRoute::new(provider.clone(), auth.clone(), secrets.clone()), responses: ResponsesRoute::new(provider, auth.clone(), secrets.clone()), ocr: OcrRoute::new(OcrClient::new( diff --git a/litellm-rust/crates/gateway-inference/tests/routes.rs b/litellm-rust/crates/gateway-inference/tests/routes.rs index 73925399e38..7452306338e 100644 --- a/litellm-rust/crates/gateway-inference/tests/routes.rs +++ b/litellm-rust/crates/gateway-inference/tests/routes.rs @@ -105,6 +105,8 @@ async fn chat_errors_come_from_core( custom_llm_provider: None, extra_headers: None, timeout: None, + drop_params: Some(false), + model_info: None, }, &(), None, diff --git a/litellm-rust/crates/gateway/src/hooks.rs b/litellm-rust/crates/gateway/src/hooks.rs index 6c393816cbf..e5a3c739577 100644 --- a/litellm-rust/crates/gateway/src/hooks.rs +++ b/litellm-rust/crates/gateway/src/hooks.rs @@ -1,7 +1,7 @@ //! Builds the hooks and observers inference requests run through, and the per-deployment prices //! observers that read cost need. -use std::{collections::HashSet, sync::Arc}; +use std::{borrow::Cow, collections::HashSet, sync::Arc}; use litellm_config::{Config, Model}; use litellm_gateway_hooks::{ @@ -30,19 +30,19 @@ pub async fn install( hooks: Vec>, observers: Vec>, ) -> Result<(), Error> { - let serves_azure = config.model_list.iter().any(is_azure); - if hooks.is_empty() && observers.is_empty() && !serves_azure { + let reads_model_info = config.model_list.iter().any(reads_model_info); + if hooks.is_empty() && observers.is_empty() && !reads_model_info { return Ok(()); } let needs_cost = observers .iter() .any(|observer| observer.reads().contains(Facts::COST)) || hooks.iter().any(|hook| hook.reads().contains(Facts::COST)); - let catalog = match needs_cost || serves_azure { + let catalog = match needs_cost || reads_model_info { true => load_catalog(inference).await.map(Arc::new), false => None, }; - if let Some(catalog) = catalog.as_ref().filter(|_| serves_azure) { + if let Some(catalog) = catalog.as_ref().filter(|_| reads_model_info) { inference.install_model_info(Arc::new(CostMap::new(catalog.clone()))); } if hooks.is_empty() && observers.is_empty() { @@ -204,7 +204,39 @@ fn is_azure(model: &Model) -> bool { } } -/// The model cost map as the capabilities Azure's param rules read. +/// Whether the deployment's chat param rules read model capabilities from the cost map: +/// Azure, the Anthropic, Bedrock, Gemini and Claude on Vertex AI chat translations, and the +/// OpenAI-compatible providers. +fn reads_model_info(model: &Model) -> bool { + is_azure(model) + || litellm_inference::provider::resolve_llm_provider( + &model.litellm_params.model, + model.litellm_params.custom_llm_provider.as_deref(), + "chat completions", + ) + .is_ok_and(|resolved| { + matches!( + <&str>::from(resolved.provider), + "anthropic" + | "bedrock" + | "gemini" + | "vertex_ai" + | "cerebras" + | "deepseek" + | "fireworks_ai" + | "groq" + | "hosted_vllm" + | "mistral" + | "nvidia_nim" + | "openrouter" + | "perplexity" + | "together_ai" + | "xai" + ) + }) +} + +/// The model cost map as the capabilities the chat param rules read. struct CostMap { catalog: Arc, keys: HashSet, @@ -222,8 +254,10 @@ impl CostMap { } impl ModelInfo for CostMap { - fn entry(&self, model: &str) -> Option<&Map> { - self.catalog.lookup(model).map(|found| found.entry.fields()) + fn entry(&self, model: &str) -> Option>> { + self.catalog + .lookup(model) + .map(|found| Cow::Borrowed(found.entry.fields())) } fn declares(&self, key: &str) -> bool { diff --git a/litellm-rust/crates/gateway/src/rate_limits.rs b/litellm-rust/crates/gateway/src/rate_limits.rs index 2948e3bc842..cdf41a0e231 100644 --- a/litellm-rust/crates/gateway/src/rate_limits.rs +++ b/litellm-rust/crates/gateway/src/rate_limits.rs @@ -1029,7 +1029,7 @@ impl Hook for RateLimiter { let mut tokens = None; if let Some(min_tpm) = limited.iter().map(|(_, limit)| *limit).min() { let body: Map = - serde_json::from_slice(request.body()).unwrap_or_default(); + serde_json::from_slice(&request.body()).unwrap_or_default(); let estimate = estimate( &body, request.key.as_deref(), diff --git a/litellm-rust/crates/gateway/src/s3_log/payload.rs b/litellm-rust/crates/gateway/src/s3_log/payload.rs index 8b67803dfb5..b1e49cb6fc5 100644 --- a/litellm-rust/crates/gateway/src/s3_log/payload.rs +++ b/litellm-rust/crates/gateway/src/s3_log/payload.rs @@ -82,7 +82,7 @@ pub(super) fn standard_logging_payload( let outcome = &record.outcome; let deployment = outcome.deployment.as_ref(); let succeeded = record.succeeded(); - let body: Map = serde_json::from_slice(request.body()).unwrap_or_default(); + let body: Map = serde_json::from_slice(&request.body()).unwrap_or_default(); let response = match succeeded { true => response(record), false => Value::Object(Map::new()), diff --git a/litellm-rust/crates/gateway/tests/chat_translation.rs b/litellm-rust/crates/gateway/tests/chat_translation.rs new file mode 100644 index 00000000000..aa0c1da8f6f --- /dev/null +++ b/litellm-rust/crates/gateway/tests/chat_translation.rs @@ -0,0 +1,722 @@ +//! Chat completions for deployments the gateway translates (Anthropic, Bedrock Converse): the +//! OpenAI request widened to tools, images and the provider's own limits, the provider body +//! the upstream sees, and the OpenAI answer the client gets back, streamed and not. + +use std::{ + io::Write, + sync::{Arc, Mutex, Once}, +}; + +use axum::{ + Router, + body::{Body, Bytes, to_bytes}, + http::{Request, header::CONTENT_TYPE}, + response::IntoResponse, +}; +use litellm_config::Config; +use litellm_framer::aws_event_stream::{AwsEventStreamCodec, Header, HeaderValue, Message}; +use rstest::rstest; +use serde_json::{Value, json}; +use tokio::net::TcpListener; +use tokio_util::codec::Encoder; +use tower::ServiceExt; + +const ANTHROPIC_MODEL: &str = "claude-x"; +const BEDROCK_MODEL: &str = "anthropic.claude-x-v1:0"; +/// A one-pixel PNG. +const PNG: &str = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="; + +/// The gateway reads its cost map from this variable; every test sets it before building a +/// gateway, so no test reads the environment while it changes. +fn use_local_cost_map() { + static ONCE: Once = Once::new(); + ONCE.call_once(|| { + let capabilities = json!({ + "mode": "chat", + "max_tokens": 8192, + "max_output_tokens": 8192, + "supports_tool_choice": true, + "supports_function_calling": true, + "supports_vision": true, + "supports_pdf_input": true, + "supports_prompt_caching": true, + "supports_response_schema": true, + }); + let entry = |provider: &str| { + let mut entry = capabilities.clone(); + entry["litellm_provider"] = json!(provider); + entry + }; + let map = json!({ + ANTHROPIC_MODEL: entry("anthropic"), + BEDROCK_MODEL: entry("bedrock_converse"), + }); + let mut file = tempfile::NamedTempFile::new().unwrap(); + file.write_all(map.to_string().as_bytes()).unwrap(); + let (_, path) = file.keep().unwrap(); + // SAFETY: set once, before any test builds a gateway or spawns a thread that reads the + // environment. + unsafe { std::env::set_var("LITELLM_MODEL_COST_MAP_PATH", path) }; + }); +} + +fn anthropic_events(events: &[Value]) -> String { + events + .iter() + .map(|event| { + format!( + "event: {}\ndata: {event}\n\n", + event["type"].as_str().unwrap() + ) + }) + .collect() +} + +fn anthropic_tool_message() -> Value { + json!({ + "id": "msg_1", "type": "message", "role": "assistant", "model": ANTHROPIC_MODEL, + "content": [ + {"type": "text", "text": "Checking."}, + {"type": "tool_use", "id": "toolu_1", "name": "get_weather", "input": {"city": "Paris"}} + ], + "stop_reason": "tool_use", "stop_sequence": null, + "usage": {"input_tokens": 10, "output_tokens": 5} + }) +} + +/// The answer to a `response_format` request, given as the tool it became. +fn anthropic_json_message() -> Value { + json!({ + "id": "msg_3", "type": "message", "role": "assistant", "model": ANTHROPIC_MODEL, + "content": [{"type": "tool_use", "id": "toolu_2", "name": "json_tool_call", "input": {"city": "Paris"}}], + "stop_reason": "tool_use", "stop_sequence": null, + "usage": {"input_tokens": 10, "output_tokens": 5} + }) +} + +fn anthropic_text_message() -> Value { + json!({ + "id": "msg_2", "type": "message", "role": "assistant", "model": ANTHROPIC_MODEL, + "content": [{"type": "text", "text": "Sunny."}], + "stop_reason": "end_turn", "stop_sequence": null, + "usage": {"input_tokens": 10, "output_tokens": 2} + }) +} + +fn anthropic_tool_stream() -> String { + anthropic_events(&[ + json!({"type": "message_start", "message": {"id": "msg_1", "type": "message", "role": "assistant", "model": ANTHROPIC_MODEL, "content": [], "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 10, "output_tokens": 1}}}), + json!({"type": "content_block_start", "index": 0, "content_block": {"type": "tool_use", "id": "toolu_1", "name": "get_weather", "input": {}}}), + json!({"type": "content_block_delta", "index": 0, "delta": {"type": "input_json_delta", "partial_json": "{\"city\": "}}), + json!({"type": "content_block_delta", "index": 0, "delta": {"type": "input_json_delta", "partial_json": "\"Paris\"}"}}), + json!({"type": "content_block_stop", "index": 0}), + json!({"type": "message_delta", "delta": {"stop_reason": "tool_use", "stop_sequence": null}, "usage": {"output_tokens": 5}}), + json!({"type": "message_stop"}), + ]) +} + +fn converse_tool_response() -> Value { + json!({ + "output": {"message": {"role": "assistant", "content": [ + {"text": "Checking."}, + {"toolUse": {"toolUseId": "tooluse_1", "name": "get_weather", "input": {"city": "Paris"}}} + ]}}, + "stopReason": "tool_use", + "usage": {"inputTokens": 10, "outputTokens": 5, "totalTokens": 15} + }) +} + +fn converse_text_response() -> Value { + json!({ + "output": {"message": {"role": "assistant", "content": [{"text": "Sunny."}]}}, + "stopReason": "end_turn", + "usage": {"inputTokens": 10, "outputTokens": 2, "totalTokens": 12} + }) +} + +fn bedrock_frame(name: &str, payload: Value, wire: &mut bytes::BytesMut) { + let message = Message::new(Bytes::from(payload.to_string())) + .add_header(Header::new( + ":message-type", + HeaderValue::String("event".to_owned().into()), + )) + .add_header(Header::new( + ":event-type", + HeaderValue::String(name.to_owned().into()), + )); + AwsEventStreamCodec.encode(message, wire).unwrap(); +} + +fn converse_tool_stream() -> Bytes { + let mut wire = bytes::BytesMut::new(); + for (name, payload) in [ + ("messageStart", json!({"role": "assistant"})), + ( + "contentBlockStart", + json!({"contentBlockIndex": 0, "start": {"toolUse": {"toolUseId": "tooluse_1", "name": "get_weather"}}}), + ), + ( + "contentBlockDelta", + json!({"contentBlockIndex": 0, "delta": {"toolUse": {"input": "{\"city\": "}}}), + ), + ( + "contentBlockDelta", + json!({"contentBlockIndex": 0, "delta": {"toolUse": {"input": "\"Paris\"}"}}}), + ), + ("contentBlockStop", json!({"contentBlockIndex": 0})), + ("messageStop", json!({"stopReason": "tool_use"})), + ( + "metadata", + json!({"usage": {"inputTokens": 10, "outputTokens": 5, "totalTokens": 15}, "metrics": {"latencyMs": 9}}), + ), + ] { + bedrock_frame(name, payload, &mut wire); + } + wire.freeze() +} + +#[derive(Clone, Debug)] +struct Sent { + path: String, + headers: Vec<(String, String)>, + body: Value, +} + +impl Sent { + fn header(&self, name: &str) -> Option<&str> { + self.headers + .iter() + .find(|(key, _)| key == name) + .map(|(_, value)| value.as_str()) + } +} + +type Calls = Arc>>; + +/// Answers each provider by path: a tool call while the conversation has no tool result yet, +/// the final answer once it does. +async fn upstream(calls: Calls) -> String { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let app = Router::new().fallback(move |request: Request| { + let calls = calls.clone(); + async move { + let (parts, body) = request.into_parts(); + let body: Value = + serde_json::from_slice(&to_bytes(body, 1 << 22).await.unwrap()).unwrap(); + let path = parts.uri.path().to_owned(); + calls.lock().unwrap().push(Sent { + path: path.clone(), + headers: parts + .headers + .iter() + .map(|(name, value)| { + ( + name.to_string(), + value.to_str().unwrap_or_default().to_owned(), + ) + }) + .collect(), + body: body.clone(), + }); + let answered = + body.to_string().contains("tool_result") || body.to_string().contains("toolResult"); + let json = [(CONTENT_TYPE, "application/json")]; + if path.ends_with("/converse-stream") { + return ( + [(CONTENT_TYPE, "application/vnd.amazon.eventstream")], + converse_tool_stream(), + ) + .into_response(); + } + if path.ends_with("/converse") { + let answer = match answered { + true => converse_text_response(), + false => converse_tool_response(), + }; + return (json, answer.to_string()).into_response(); + } + if body["stream"] == json!(true) { + return ( + [(CONTENT_TYPE, "text/event-stream")], + anthropic_tool_stream(), + ) + .into_response(); + } + let answer = match answered { + _ if body["tools"][0]["name"] == "json_tool_call" => anthropic_json_message(), + true => anthropic_text_message(), + false => anthropic_tool_message(), + }; + (json, answer.to_string()).into_response() + } + }); + tokio::spawn(async move { axum::serve(listener, app).await }); + format!("http://{address}") +} + +struct Harness { + app: Router, + calls: Calls, +} + +async fn harness() -> Harness { + use_local_cost_map(); + let calls = Calls::default(); + let upstream = upstream(calls.clone()).await; + let config = Config::from_yaml(&format!( + "model_list: + - model_name: claude + litellm_params: + model: anthropic/{ANTHROPIC_MODEL} + api_base: {upstream} + api_key: sk-anthropic + - model_name: bedrock + litellm_params: + model: bedrock/{BEDROCK_MODEL} + api_base: {upstream} + api_key: bedrock-token + - model_name: claude-dropping + litellm_params: + model: anthropic/{ANTHROPIC_MODEL} + api_base: {upstream} + api_key: sk-anthropic + drop_params: true +router_settings: + num_retries: 0 +general_settings: + master_key: sk-master +" + )) + .unwrap(); + let inference = litellm_gateway::build_inference(&config).unwrap(); + litellm_gateway::hooks::install(&inference, &config, Vec::new(), Vec::new()) + .await + .unwrap(); + let app = litellm_gateway::router(inference, &config, None, None); + Harness { app, calls } +} + +impl Harness { + async fn send(&self, body: Value) -> (u16, String) { + let request = Request::post("/v1/chat/completions") + .header("content-type", "application/json") + .header("authorization", "Bearer sk-master") + .body(Body::from(body.to_string())) + .unwrap(); + let response = self.app.clone().oneshot(request).await.unwrap(); + let status = response.status().as_u16(); + let body = to_bytes(response.into_body(), 1 << 22).await.unwrap(); + (status, String::from_utf8(body.to_vec()).unwrap()) + } + + async fn json(&self, body: Value) -> Value { + let (status, text) = self.send(body).await; + assert_eq!(status, 200, "{text}"); + serde_json::from_str(&text).unwrap() + } + + fn sent(&self) -> Vec { + self.calls.lock().unwrap().clone() + } +} + +fn weather_tool() -> Value { + json!({"type": "function", "function": { + "name": "get_weather", + "description": "The weather in a city", + "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]} + }}) +} + +fn question() -> Value { + json!({"role": "user", "content": "Weather in Paris?"}) +} + +/// The conversation after the model asked for the tool and the tool answered. +fn answered(call_id: &str) -> Value { + json!([ + question(), + {"role": "assistant", "content": null, "tool_calls": [ + {"id": call_id, "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}} + ]}, + {"role": "tool", "tool_call_id": call_id, "content": "sunny, 21C"} + ]) +} + +/// The streamed chunks' JSON and whether the stream ended with `[DONE]`. +fn frames(body: &str) -> (Vec, bool) { + let mut done = false; + let frames = body + .split("\n\n") + .filter_map(|frame| frame.strip_prefix("data: ")) + .filter(|data| { + let last = *data == "[DONE]"; + done |= last; + !last + }) + .map(|data| serde_json::from_str(data).unwrap()) + .collect(); + (frames, done) +} + +/// The tool calls the chunks stream, joined: (id, name, arguments) per index. +fn streamed_tool_calls(chunks: &[Value]) -> Vec<(String, String, String)> { + let mut calls: Vec<(String, String, String)> = Vec::new(); + for chunk in chunks { + let Some(deltas) = chunk["choices"][0]["delta"]["tool_calls"].as_array() else { + continue; + }; + for delta in deltas { + let index = delta["index"].as_u64().unwrap() as usize; + if calls.len() <= index { + calls.resize(index + 1, Default::default()); + } + let call = &mut calls[index]; + if let Some(id) = delta["id"].as_str() { + call.0 = id.to_owned(); + } + if let Some(name) = delta["function"]["name"].as_str() { + call.1 = name.to_owned(); + } + call.2 + .push_str(delta["function"]["arguments"].as_str().unwrap_or_default()); + } + } + calls +} + +fn finish_reason(chunks: &[Value]) -> Option { + chunks + .iter() + .find_map(|chunk| chunk["choices"][0]["finish_reason"].as_str()) + .map(str::to_owned) +} + +#[tokio::test] +async fn anthropic_tool_round_trip() { + let harness = harness().await; + + let asked = harness + .json(json!({ + "model": "claude", + "messages": [question()], + "tools": [weather_tool()], + "tool_choice": "required", + "max_tokens": 64, + })) + .await; + let message = &asked["choices"][0]["message"]; + assert_eq!(asked["choices"][0]["finish_reason"], "tool_calls"); + assert_eq!(message["content"], "Checking."); + assert_eq!( + message["tool_calls"], + json!([{"id": "toolu_1", "type": "function", "index": 1, + "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}}]) + ); + + let done = harness + .json(json!({ + "model": "claude", + "messages": answered("toolu_1"), + "tools": [weather_tool()], + "max_tokens": 64, + })) + .await; + assert_eq!(done["choices"][0]["message"]["content"], "Sunny."); + assert_eq!(done["choices"][0]["finish_reason"], "stop"); + + let sent = harness.sent(); + assert_eq!(sent[0].path, "/v1/messages"); + assert_eq!( + sent[0].body, + json!({ + "model": ANTHROPIC_MODEL, + "messages": [{"role": "user", "content": [{"type": "text", "text": "Weather in Paris?"}]}], + "max_tokens": 64, + "tools": [{"name": "get_weather", "type": "custom", "description": "The weather in a city", + "input_schema": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}], + "tool_choice": {"type": "any"} + }) + ); + assert_eq!( + sent[1].body["messages"], + json!([ + {"role": "user", "content": [{"type": "text", "text": "Weather in Paris?"}]}, + {"role": "assistant", "content": [{"type": "tool_use", "id": "toolu_1", "name": "get_weather", "input": {"city": "Paris"}}]}, + {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "toolu_1", "content": "sunny, 21C"}]} + ]) + ); +} + +#[tokio::test] +async fn bedrock_tool_round_trip() { + let harness = harness().await; + + let asked = harness + .json(json!({ + "model": "bedrock", + "messages": [question()], + "tools": [weather_tool()], + "tool_choice": {"type": "function", "function": {"name": "get_weather"}}, + "max_tokens": 64, + })) + .await; + assert_eq!(asked["choices"][0]["finish_reason"], "tool_calls"); + assert_eq!( + asked["choices"][0]["message"]["tool_calls"], + json!([{"id": "tooluse_1", "type": "function", "index": 1, + "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}}]) + ); + + let done = harness + .json(json!({ + "model": "bedrock", + "messages": answered("tooluse_1"), + "tools": [weather_tool()], + "max_tokens": 64, + })) + .await; + assert_eq!(done["choices"][0]["message"]["content"], "Sunny."); + assert_eq!(done["choices"][0]["finish_reason"], "stop"); + + let sent = harness.sent(); + assert_eq!(sent[0].path, format!("/model/{BEDROCK_MODEL}/converse")); + assert_eq!( + sent[0].body["toolConfig"], + json!({ + "tools": [{"toolSpec": {"name": "get_weather", "description": "The weather in a city", + "inputSchema": {"json": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}}], + "toolChoice": {"tool": {"name": "get_weather"}} + }) + ); + assert_eq!( + sent[1].body["messages"], + json!([ + {"role": "user", "content": [{"text": "Weather in Paris?"}]}, + {"role": "assistant", "content": [{"toolUse": {"toolUseId": "tooluse_1", "name": "get_weather", "input": {"city": "Paris"}}}]}, + {"role": "user", "content": [{"toolResult": {"toolUseId": "tooluse_1", "content": [{"text": "sunny, 21C"}]}}]} + ]) + ); +} + +#[rstest] +#[case::anthropic("claude", "toolu_1")] +#[case::bedrock("bedrock", "tooluse_1")] +#[tokio::test] +async fn a_streamed_tool_call_arrives_as_openai_tool_call_chunks( + #[case] model: &str, + #[case] id: &str, +) { + let harness = harness().await; + + let (status, body) = harness + .send(json!({ + "model": model, + "messages": [question()], + "tools": [weather_tool()], + "max_tokens": 64, + "stream": true, + })) + .await; + + assert_eq!(status, 200, "{body}"); + let (chunks, done) = frames(&body); + assert!(done, "{body}"); + assert_eq!( + streamed_tool_calls(&chunks), + [( + id.to_owned(), + "get_weather".to_owned(), + "{\"city\": \"Paris\"}".to_owned() + )] + ); + assert_eq!(finish_reason(&chunks).as_deref(), Some("tool_calls")); + let sent = &harness.sent()[0]; + let tools = match model { + "claude" => sent.body["tools"][0]["name"].clone(), + _ => sent.body["toolConfig"]["tools"][0]["toolSpec"]["name"].clone(), + }; + assert_eq!(tools, "get_weather"); +} + +#[rstest] +#[case::max_tokens("max_tokens")] +#[case::max_completion_tokens("max_completion_tokens")] +#[tokio::test] +async fn bedrock_takes_the_token_limit_in_its_inference_config(#[case] param: &str) { + let harness = harness().await; + + harness + .json(json!({ + "model": "bedrock", + "messages": [{"role": "system", "content": "Be brief."}, question()], + param: 32, + "temperature": 0.5, + "stop": ["END"], + })) + .await; + + let sent = &harness.sent()[0]; + assert_eq!( + sent.body["inferenceConfig"], + json!({"maxTokens": 32, "temperature": 0.5, "stopSequences": ["END"]}) + ); + assert_eq!(sent.body["system"], json!([{"text": "Be brief."}])); +} + +#[tokio::test] +async fn anthropic_gets_the_model_default_max_tokens_when_the_caller_names_none() { + let harness = harness().await; + + harness + .json(json!({"model": "claude", "messages": [question()]})) + .await; + + assert_eq!(harness.sent()[0].body["max_tokens"], 8192); +} + +#[tokio::test] +async fn an_image_data_url_reaches_each_provider_inline() { + let harness = harness().await; + let messages = json!([{"role": "user", "content": [ + {"type": "text", "text": "What is this?"}, + {"type": "image_url", "image_url": {"url": format!("data:image/png;base64,{PNG}")}} + ]}]); + + harness + .json(json!({"model": "claude", "messages": messages, "max_tokens": 16})) + .await; + harness + .json(json!({"model": "bedrock", "messages": messages, "max_tokens": 16})) + .await; + + let sent = harness.sent(); + assert_eq!( + sent[0].body["messages"][0]["content"][1], + json!({"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": PNG}}) + ); + assert_eq!( + sent[1].body["messages"][0]["content"][1], + json!({"image": {"format": "png", "source": {"bytes": PNG}}}) + ); +} + +#[tokio::test] +async fn anthropic_response_format_comes_back_as_the_answer() { + let harness = harness().await; + let schema = + json!({"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}); + + let answer = harness + .json(json!({ + "model": "claude", + "messages": [question()], + "max_tokens": 64, + "response_format": {"type": "json_schema", "json_schema": {"name": "place", "schema": schema}}, + "user": "end-user-1", + })) + .await; + + assert_eq!( + answer["choices"][0]["message"]["content"], + "{\"city\": \"Paris\"}" + ); + assert_eq!(answer["choices"][0]["message"]["tool_calls"], Value::Null); + assert_eq!(answer["choices"][0]["finish_reason"], "stop"); + let sent = &harness.sent()[0]; + assert_eq!(sent.body["metadata"], json!({"user_id": "end-user-1"})); + assert_eq!(sent.body["tools"][0]["name"], "json_tool_call"); + assert_eq!(sent.body["tools"][0]["input_schema"], schema); + assert_eq!( + sent.body["tool_choice"], + json!({"name": "json_tool_call", "type": "tool"}) + ); +} + +#[rstest] +#[case::anthropic("claude", json!({"logprobs": true}))] +#[case::bedrock("bedrock", json!({"tool_choice": "none", "tools": [weather_tool()]}))] +#[case::n_choices("claude", json!({"n": 2}))] +#[tokio::test] +async fn an_unsupported_param_is_refused_and_dropped_with_drop_params( + #[case] model: &str, + #[case] extra: Value, +) { + let harness = harness().await; + let mut body = json!({"model": model, "messages": [question()], "max_tokens": 16}); + body.as_object_mut() + .unwrap() + .extend(extra.as_object().unwrap().clone()); + + let (status, refused) = harness.send(body.clone()).await; + assert_eq!(status, 400, "{refused}"); + assert!( + harness.sent().is_empty(), + "a refused request never goes out" + ); + + body["drop_params"] = json!(true); + harness.json(body).await; + let sent = &harness.sent()[0]; + for name in extra.as_object().unwrap().keys() { + if name == "tools" { + continue; + } + assert!( + sent.body.get(name).is_none(), + "{name} reached {}", + sent.body + ); + } + assert!( + sent.body + .get("toolConfig") + .and_then(|config| config.get("toolChoice")) + .is_none() + ); + assert!(sent.body.get("drop_params").is_none()); +} + +#[tokio::test] +async fn a_deployment_that_drops_params_drops_them_without_the_request_asking() { + let harness = harness().await; + + harness + .json(json!({ + "model": "claude-dropping", + "messages": [question()], + "max_tokens": 16, + "logprobs": true, + })) + .await; + + assert!(harness.sent()[0].body.get("logprobs").is_none()); +} + +#[tokio::test] +async fn a_cache_control_part_reaches_anthropic_and_bedrock() { + let harness = harness().await; + let messages = json!([ + {"role": "system", "content": [{"type": "text", "text": "Long instructions.", "cache_control": {"type": "ephemeral"}}]}, + question() + ]); + + harness + .json(json!({"model": "claude", "messages": messages, "max_tokens": 16})) + .await; + harness + .json(json!({"model": "bedrock", "messages": messages, "max_tokens": 16})) + .await; + + let sent = harness.sent(); + assert_eq!( + sent[0].body["system"], + json!([{"type": "text", "text": "Long instructions.", "cache_control": {"type": "ephemeral"}}]) + ); + assert_eq!( + sent[1].body["system"], + json!([{"text": "Long instructions."}, {"cachePoint": {"type": "default"}}]) + ); + assert!(sent[0].header("x-api-key").is_some()); +} diff --git a/litellm-rust/crates/gateway/tests/gemini_chat.rs b/litellm-rust/crates/gateway/tests/gemini_chat.rs new file mode 100644 index 00000000000..7eda09bc0ea --- /dev/null +++ b/litellm-rust/crates/gateway/tests/gemini_chat.rs @@ -0,0 +1,559 @@ +//! Gemini chat completions through the gateway, on Google AI Studio (`gemini/`, an API +//! key in `x-goog-api-key`) and on Vertex AI (`vertex_ai/`, a service account traded for +//! an access token): the request Gemini gets, the OpenAI answer the client gets, and the usage +//! and cost the request is recorded with. + +use std::{ + io::Write, + sync::{Arc, Mutex, Once}, +}; + +use axum::{ + Router, + body::{Body, to_bytes}, + http::{Request, StatusCode, header::CONTENT_TYPE}, + response::IntoResponse, +}; +use futures_util::future::BoxFuture; +use litellm_config::Config; +use litellm_gateway_hooks::{Facts, Observer, Record}; +use rstest::rstest; +use serde_json::{Value, json}; +use tokio::net::TcpListener; +use tower::ServiceExt; + +/// Input and output prices of every deployment. +const RATES: (f64, f64) = (1.0, 2.0); + +const STUDIO_KEY: &str = "studio-key"; +const STUDIO_KEY_ENV: &str = "GEMINI_CHAT_TEST_STUDIO_KEY"; +const ACCESS_TOKEN: &str = "ya29.test-token"; +const PROJECT: &str = "test-project"; +const LOCATION: &str = "europe-west4"; + +/// A throwaway RSA key the fake service account signs its token request with. +const TEST_PRIVATE_KEY: &str = "-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC95ihJQlfJuaQe +ZCwF+EFy/Y5vk7Lx6Nm74He9Oes+6ZbgA2LVdcoFJrNEYHd8S0CCv7cqEp4dpIoR +B4HAmQAQF92ZC/S2lcgLAezWQU21TSjDLngdVbn+Jr4syzOwVw/NUigydtQ66Iqv +1FpXs/4gQXMgs8FW5XgOhgfKhyMPf71A2qiRaA0vwsrfsmKOY0QVVooXPRfpLIvZ +VEOV89ZMQErnYan+m8OADoZ5NJ/Ij6QKb2HkQJNhQC2Zq2T1Y9VLyxY6pXURdVqc +XJgFT6YPGjLABLX/oId4g+Fs2YI5mM+g72pYEntnIaE5yicGKBdjyqeVfKZ6/7jg +g8evHi6vAgMBAAECggEAGdJjlP6b8Fa5bdaCM/ebcrbuuNZVJVbb0JPHxGfNSLs7 +pE9hj5QaOdQW2Uviw3h6F61ZCzQH4xD+Iy2po5ZKb2XHYKnDB1bboj+LRGER337T +9aJqe9at2VTMVEv3Rdm40NsEk0QcPLxlK16NQFK90gYEUSSQPDAswJDSG2R/zHn+ +vADI907mW/goEJHeLn8PWGlNlSiR6x+5JJtq+GXCzUzVvJYQSCLGxCSl2x2H+0g7 +NhFI0zPpdzNmO/h+yhzaFb6Rp5U8+ZsnZ3qYjQ/03gw1myTDKJt1YaO9JvArnNYX +hcJQQ8Rt0bHhcrZA16bBOpqZlo5pKCicwI/netgN8QKBgQDcFz7AzdJ26sMSV32V +rwrMgIoggt8qDjO1ARwqW35A1TIge0FoW4M4KpsXQGGfT341uU1esXEcyZ/1L/5X +3ql2gX4DbOYLZLWYzZGR2hq33oi8HkhN98QrEwL9emSH8NqYX3Xxja3PrmCrSYJe +Zbnd9TIm2XkxyMoyXJu6M/QvnwKBgQDc4dzqTbxoGEGa5MuJoGmMwPnqgdG9UM5J +eExVnh7osxc2sOdsiPeRjjQTxs9v2kJwctC359OJoo9yGaaJeSghU4LEWJo1sqnA +fzSCLammYvtVAtniyNv5Mxk/6Uimi4NNDKaAKB+m4K2uSn3U9AmY7KPYMGaSbS9W +XSnobjxm8QKBgC8bPpAvvWs8ZhIn7bY659nLbUT2HeO3dHO6UBf0yzn/J6JyHxbB +93zvCZDZc8uQTRgcmCW7XtVlhjoJUqvl+Wlm39zF0xr/LCsPXKfWAb/2/lcdOCaP +8Emz4QD10EyUTYUtcWYJB/mafhBLRH8F0Nlj4J8WDu2L51MOJTqeYhZLAoGAWffN +icocAbJPlo22sdoa4+/+W5yBF8GAJMDRJtZ+9H1t6SLpQHYRkMIBSETkXUTjZvX9 +Ocs9iIQkNW9pO/mTdO+VBfCo71JUfknR02xR+6m5gYjlws/ZeYlssXGN2/hbhNiw +QOcW7Vv6olFJK6Iy/oz0t6wPO3kpnN3Zogi0paECgYEAwo44M1DdYCtV0snhmYM9 +5u0mPfYt5P2SVLXyUbr+vFTfrTL/WKnXIJgbsnj3Gvf+GIZv9tKcXhSNmEHQCYX4 +X3w9iTPddCHuvZ1fpufi2TyArJh0OkoNtLXJHTKrHjf2N+61AQzFiv5WieJrdE+H +qr32PTUuVGPyO9LyTY4/RL0= +-----END PRIVATE KEY----- +"; + +/// The gateway reads its cost map from this variable; every test sets it before building a +/// gateway, so no test reads the environment while it changes. +fn use_local_cost_map() { + static ONCE: Once = Once::new(); + ONCE.call_once(|| { + let entry = |provider: &str| { + json!({ + "mode": "chat", + "litellm_provider": provider, + "max_tokens": 8192, + "supports_system_messages": true, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_reasoning": true, + }) + }; + let map = json!({ + "gemini/gemini-2.5-flash": entry("gemini"), + "gemini-2.5-flash": entry("vertex_ai-language-models"), + }); + let mut file = tempfile::NamedTempFile::new().unwrap(); + file.write_all(map.to_string().as_bytes()).unwrap(); + let (_, path) = file.keep().unwrap(); + // SAFETY: set once, before any test builds a gateway or spawns a thread that reads the + // environment. + unsafe { + std::env::set_var("LITELLM_MODEL_COST_MAP_PATH", path); + std::env::set_var(STUDIO_KEY_ENV, STUDIO_KEY); + }; + }); +} + +fn usage_metadata() -> Value { + json!({"promptTokenCount": 10, "candidatesTokenCount": 5, "thoughtsTokenCount": 3, "totalTokenCount": 18}) +} + +fn text_answer(text: &str) -> Value { + json!({ + "candidates": [{"content": {"role": "model", "parts": [{"text": text}]}, "finishReason": "STOP", "index": 0}], + "usageMetadata": usage_metadata(), + "modelVersion": "gemini-2.5-flash", + "responseId": "resp-1", + }) +} + +fn tool_answer() -> Value { + json!({ + "candidates": [{"content": {"role": "model", "parts": [{"functionCall": {"name": "get_weather", "args": {"city": "Paris"}}}]}, "finishReason": "STOP", "index": 0}], + "usageMetadata": usage_metadata(), + "modelVersion": "gemini-2.5-flash", + }) +} + +fn sse_answer() -> String { + [ + json!({"candidates": [{"content": {"role": "model", "parts": [{"text": "Hello"}]}}], "modelVersion": "gemini-2.5-flash", "responseId": "resp-2"}), + json!({"candidates": [{"content": {"role": "model", "parts": [{"text": " world"}]}, "finishReason": "STOP"}], "usageMetadata": usage_metadata(), "modelVersion": "gemini-2.5-flash", "responseId": "resp-2"}), + ] + .iter() + .map(|event| format!("data: {event}\r\n\r\n")) + .collect() +} + +#[derive(Clone, Debug)] +struct Sent { + path: String, + query: Option, + api_key: Option, + authorization: Option, + body: Value, +} + +type Calls = Arc>>; + +fn has_function_response(body: &Value) -> bool { + body["contents"].as_array().is_some_and(|contents| { + contents.iter().any(|content| { + content["parts"].as_array().is_some_and(|parts| { + parts + .iter() + .any(|part| part.get("function_response").is_some()) + }) + }) + }) +} + +/// Google's token endpoint and Gemini's `generateContent`, answering by what the body asks. +async fn upstream(calls: Calls) -> String { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let app = Router::new().fallback(move |request: Request| { + let calls = calls.clone(); + async move { + let (parts, body) = request.into_parts(); + let header = |name: &str| { + parts + .headers + .get(name) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) + }; + let bytes = to_bytes(body, 1 << 20).await.unwrap(); + let path = parts.uri.path().to_owned(); + if path == "/token" { + return ( + [(CONTENT_TYPE, "application/json")], + json!({"access_token": ACCESS_TOKEN, "expires_in": 3600, "token_type": "Bearer"}) + .to_string(), + ) + .into_response(); + } + let body: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null); + calls.lock().unwrap().push(Sent { + path: path.clone(), + query: parts.uri.query().map(str::to_owned), + api_key: header("x-goog-api-key"), + authorization: header("authorization"), + body: body.clone(), + }); + match path.ends_with(":streamGenerateContent") { + true => ([(CONTENT_TYPE, "text/event-stream")], sse_answer()).into_response(), + false => { + let answer = match (has_function_response(&body), body.get("tools")) { + (true, _) => text_answer("It is sunny in Paris"), + (false, Some(_)) => tool_answer(), + (false, None) => text_answer("Hello world"), + }; + ( + StatusCode::OK, + [(CONTENT_TYPE, "application/json")], + answer.to_string(), + ) + .into_response() + } + } + } + }); + tokio::spawn(async move { axum::serve(listener, app).await }); + format!("http://{address}") +} + +#[derive(Debug)] +struct Seen { + deployment: Option, + usage: Option, + cost: Option, +} + +struct Collect(Mutex>); + +impl Observer for Collect { + fn name(&self) -> &'static str { + "collect" + } + + fn reads(&self) -> Facts { + Facts::DEPLOYMENT | Facts::USAGE | Facts::COST + } + + fn observe<'a>(&'a self, record: &'a Record) -> BoxFuture<'a, ()> { + Box::pin(async move { + self.0.lock().unwrap().push(Seen { + deployment: record + .outcome + .deployment + .as_ref() + .map(|deployment| deployment.id.to_string()), + usage: record.outcome.usage, + cost: record.outcome.cost, + }); + }) + } +} + +struct Harness { + app: Router, + inference: Arc, + calls: Calls, + seen: Arc, +} + +fn service_account(upstream: &str) -> String { + json!({ + "type": "service_account", + "project_id": "credentials-project", + "private_key_id": "test-key-id", + "private_key": TEST_PRIVATE_KEY, + "client_email": "gateway-test@credentials-project.iam.gserviceaccount.com", + "client_id": "1", + "token_uri": format!("{upstream}/token"), + }) + .to_string() +} + +async fn harness() -> Harness { + use_local_cost_map(); + let calls = Calls::default(); + let upstream = upstream(calls.clone()).await; + let (input, output) = RATES; + let prices = + format!(" input_cost_per_token: {input}\n output_cost_per_token: {output}\n"); + let credentials = service_account(&upstream); + let config = Config::from_yaml(&format!( + "model_list: + - model_name: studio + litellm_params: + model: gemini/gemini-2.5-flash + api_base: {upstream}/v1beta + api_key: {STUDIO_KEY} +{prices} model_info: + id: studio-1 + - model_name: studio-env + litellm_params: + model: gemini/gemini-2.5-flash + api_base: {upstream}/v1beta + api_key: os.environ/{STUDIO_KEY_ENV} +{prices} model_info: + id: studio-env-1 + - model_name: vertex + litellm_params: + model: vertex_ai/gemini-2.5-flash + api_base: {upstream} + vertex_project: {PROJECT} + vertex_location: {LOCATION} + vertex_credentials: '{credentials}' +{prices} model_info: + id: vertex-1 +general_settings: + master_key: sk-master +" + )) + .unwrap(); + let inference = litellm_gateway::build_inference(&config).unwrap(); + let seen = Arc::new(Collect(Mutex::new(Vec::new()))); + litellm_gateway::hooks::install(&inference, &config, Vec::new(), vec![seen.clone()]) + .await + .unwrap(); + let app = litellm_gateway::router(inference.clone(), &config, None, None); + Harness { + app, + inference, + calls, + seen, + } +} + +async fn send(app: &Router, body: Value) -> (u16, String) { + let request = Request::post("/v1/chat/completions") + .header("content-type", "application/json") + .header("authorization", "Bearer sk-master") + .body(Body::from(body.to_string())) + .unwrap(); + let response = app.clone().oneshot(request).await.unwrap(); + let status = response.status().as_u16(); + let body = to_bytes(response.into_body(), 1 << 20).await.unwrap(); + (status, String::from_utf8(body.to_vec()).unwrap()) +} + +async fn seen(harness: &Harness) -> Vec { + litellm_gateway::hooks::close(&harness.inference).await; + std::mem::take(&mut *harness.seen.0.lock().unwrap()) +} + +fn expected_cost() -> f64 { + let (input, output) = RATES; + 10.0 * input + 8.0 * output +} + +fn hi(model: &str) -> Value { + json!({ + "model": model, + "messages": [ + {"role": "system", "content": "Be brief"}, + {"role": "user", "content": "hi"} + ], + "temperature": 0.2, + "max_tokens": 64, + }) +} + +fn sent_body() -> Value { + json!({ + "contents": [{"role": "user", "parts": [{"text": "hi"}]}], + "system_instruction": {"parts": [{"text": "Be brief"}]}, + "generationConfig": {"temperature": 0.2, "max_output_tokens": 64}, + }) +} + +#[rstest] +#[case::studio( + "studio", + "studio-1", + "/v1beta/models/gemini-2.5-flash:generateContent", + Some(STUDIO_KEY), + None +)] +#[case::studio_key_from_env( + "studio-env", + "studio-env-1", + "/v1beta/models/gemini-2.5-flash:generateContent", + Some(STUDIO_KEY), + None +)] +#[case::vertex( + "vertex", + "vertex-1", + "/v1/projects/test-project/locations/europe-west4/publishers/google/models/gemini-2.5-flash:generateContent", + None, + Some("Bearer ya29.test-token") +)] +#[tokio::test] +async fn a_chat_completion_calls_generate_content_and_records_its_cost( + #[case] model: &str, + #[case] id: &str, + #[case] path: &str, + #[case] api_key: Option<&str>, + #[case] authorization: Option<&str>, +) { + let harness = harness().await; + let (status, body) = send(&harness.app, hi(model)).await; + assert_eq!(status, 200, "{body}"); + let answer: Value = serde_json::from_str(&body).unwrap(); + assert_eq!(answer["choices"][0]["message"]["content"], "Hello world"); + assert_eq!(answer["choices"][0]["finish_reason"], "stop"); + assert_eq!(answer["usage"]["prompt_tokens"], 10); + assert_eq!(answer["usage"]["completion_tokens"], 8); + assert_eq!( + answer["usage"]["completion_tokens_details"]["reasoning_tokens"], + 3 + ); + + let sent = harness.calls.lock().unwrap()[0].clone(); + assert_eq!(sent.path, path); + assert_eq!(sent.query, None); + assert_eq!(sent.api_key.as_deref(), api_key); + assert_eq!(sent.authorization.as_deref(), authorization); + assert_eq!(sent.body, sent_body()); + + let seen = seen(&harness).await; + assert_eq!(seen.len(), 1, "{seen:?}"); + assert_eq!(seen[0].deployment.as_deref(), Some(id)); + let usage = seen[0].usage.unwrap(); + assert_eq!((usage.prompt_tokens(), usage.output_tokens), (10, 8)); + assert_eq!(seen[0].cost, Some(expected_cost())); +} + +/// The JSON of each `data:` frame, and whether the stream closed with `[DONE]`. +fn frames(body: &str) -> (Vec, bool) { + let mut done = false; + let frames = body + .split("\n\n") + .filter(|frame| !frame.is_empty()) + .filter_map(|frame| { + let data = frame.strip_prefix("data: ").expect("bare data frames"); + match data { + "[DONE]" => { + done = true; + None + } + data => Some(serde_json::from_str(data).unwrap()), + } + }) + .collect(); + (frames, done) +} + +#[rstest] +#[case::studio( + "studio", + "studio-1", + "/v1beta/models/gemini-2.5-flash:streamGenerateContent" +)] +#[case::vertex( + "vertex", + "vertex-1", + "/v1/projects/test-project/locations/europe-west4/publishers/google/models/gemini-2.5-flash:streamGenerateContent" +)] +#[tokio::test] +async fn a_stream_calls_stream_generate_content_and_records_its_usage( + #[case] model: &str, + #[case] id: &str, + #[case] path: &str, +) { + let harness = harness().await; + let mut request = hi(model); + request["stream"] = json!(true); + request["stream_options"] = json!({"include_usage": true}); + let (status, body) = send(&harness.app, request).await; + assert_eq!(status, 200, "{body}"); + + let (frames, done) = frames(&body); + assert!(done, "{body}"); + assert!( + frames + .iter() + .all(|frame| frame["object"] == "chat.completion.chunk") + ); + let content: String = frames + .iter() + .filter_map(|frame| frame["choices"][0]["delta"]["content"].as_str()) + .collect(); + assert_eq!(content, "Hello world", "{body}"); + let finishes: Vec<&Value> = frames + .iter() + .filter_map(|frame| frame["choices"][0].get("finish_reason")) + .filter(|reason| !reason.is_null()) + .collect(); + assert_eq!(finishes, [&json!("stop")], "{body}"); + let usages: Vec<&Value> = frames + .iter() + .filter_map(|frame| frame.get("usage")) + .filter(|usage| !usage.is_null()) + .collect(); + assert_eq!(usages.len(), 1, "{body}"); + assert_eq!(usages[0]["prompt_tokens"], 10); + assert_eq!(usages[0]["completion_tokens"], 8); + + let sent = harness.calls.lock().unwrap()[0].clone(); + assert_eq!(sent.path, path); + assert_eq!(sent.query.as_deref(), Some("alt=sse")); + assert_eq!(sent.body, sent_body()); + + let seen = seen(&harness).await; + assert_eq!(seen.len(), 1, "{seen:?}"); + assert_eq!(seen[0].deployment.as_deref(), Some(id)); + assert_eq!(seen[0].cost, Some(expected_cost())); +} + +#[tokio::test] +async fn a_tool_call_round_trips_through_function_call_and_response() { + let harness = harness().await; + let tools = json!([{"type": "function", "function": { + "name": "get_weather", + "description": "Weather for a city", + "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], "additionalProperties": false} + }}]); + let question = json!({"role": "user", "content": "Weather in Paris?"}); + let (status, body) = send( + &harness.app, + json!({"model": "studio", "messages": [question], "tools": tools, "tool_choice": "auto"}), + ) + .await; + assert_eq!(status, 200, "{body}"); + let answer: Value = serde_json::from_str(&body).unwrap(); + let message = &answer["choices"][0]["message"]; + assert_eq!(answer["choices"][0]["finish_reason"], "tool_calls"); + let call = &message["tool_calls"][0]; + assert_eq!(call["type"], "function"); + assert_eq!(call["function"]["name"], "get_weather"); + assert_eq!(call["function"]["arguments"], "{\"city\": \"Paris\"}"); + + let first = harness.calls.lock().unwrap()[0].body.clone(); + assert_eq!( + first["tools"], + json!([{"function_declarations": [{ + "name": "get_weather", + "description": "Weather for a city", + "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]} + }]}]) + ); + assert_eq!( + first["toolConfig"], + json!({"functionCallingConfig": {"mode": "AUTO"}}) + ); + + let (status, body) = send( + &harness.app, + json!({"model": "studio", "tools": tools, "messages": [ + question, + {"role": "assistant", "content": null, "tool_calls": [call]}, + {"role": "tool", "tool_call_id": call["id"], "content": "sunny"} + ]}), + ) + .await; + assert_eq!(status, 200, "{body}"); + let answer: Value = serde_json::from_str(&body).unwrap(); + assert_eq!( + answer["choices"][0]["message"]["content"], + "It is sunny in Paris" + ); + let second = harness.calls.lock().unwrap()[1].body.clone(); + assert_eq!( + second["contents"], + json!([ + {"role": "user", "parts": [{"text": "Weather in Paris?"}]}, + {"role": "model", "parts": [{"function_call": {"name": "get_weather", "args": {"city": "Paris"}}}]}, + {"role": "user", "parts": [{"function_response": {"name": "get_weather", "response": {"content": "sunny"}}}]} + ]) + ); + + let seen = seen(&harness).await; + assert_eq!(seen.len(), 2, "{seen:?}"); + assert!(seen.iter().all(|seen| seen.cost == Some(expected_cost()))); +} diff --git a/litellm-rust/crates/gateway/tests/openai_compat_chat.rs b/litellm-rust/crates/gateway/tests/openai_compat_chat.rs new file mode 100644 index 00000000000..a8bb0c2bc88 --- /dev/null +++ b/litellm-rust/crates/gateway/tests/openai_compat_chat.rs @@ -0,0 +1,442 @@ +//! Chat completions for the OpenAI-compatible providers through the gateway: the URL, auth +//! and exact body each provider's upstream sees, and the usage and cost a stream is recorded +//! with whether or not the client asked to see the usage. + +use std::{ + collections::BTreeMap, + io::Write, + sync::{Arc, Mutex, Once}, +}; + +use axum::{ + Router, + body::{Body, to_bytes}, + http::{Request, header::CONTENT_TYPE}, + response::IntoResponse, +}; +use futures_util::future::BoxFuture; +use litellm_config::Config; +use litellm_gateway_hooks::{Facts, Observer, Record}; +use rstest::rstest; +use serde_json::{Value, json}; +use tokio::net::TcpListener; +use tower::ServiceExt; + +const COMPLETION: &str = r#"{"id":"chatcmpl-1","object":"chat.completion","created":1,"model":"m","choices":[{"index":0,"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}],"usage":{"prompt_tokens":5,"completion_tokens":6,"total_tokens":11}}"#; +const CONTENT_CHUNK: &str = "data: {\"id\":\"c\",\"object\":\"chat.completion.chunk\",\"model\":\"m\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"hi\"}}]}\n\n"; +const USAGE_CHUNK: &str = "data: {\"id\":\"c\",\"object\":\"chat.completion.chunk\",\"model\":\"m\",\"choices\":[],\"usage\":{\"prompt_tokens\":7,\"completion_tokens\":3,\"total_tokens\":10}}\n\n"; +/// xAI counts reasoning apart from `completion_tokens` but inside `total_tokens`. +const XAI_USAGE_CHUNK: &str = "data: {\"id\":\"c\",\"object\":\"chat.completion.chunk\",\"model\":\"m\",\"choices\":[],\"usage\":{\"prompt_tokens\":7,\"completion_tokens\":3,\"total_tokens\":14,\"completion_tokens_details\":{\"reasoning_tokens\":4}}}\n\n"; +/// Mistral ends its stream with the usage on the last content chunk, unasked. +const MISTRAL_LAST_CHUNK: &str = "data: {\"id\":\"c\",\"object\":\"chat.completion.chunk\",\"model\":\"m\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":7,\"completion_tokens\":3,\"total_tokens\":10}}\n\n"; +const DONE: &str = "data: [DONE]\n\n"; + +const RATES: (f64, f64) = (1.0, 2.0); + +fn use_local_cost_map() { + static ONCE: Once = Once::new(); + ONCE.call_once(|| { + let price = json!({ + "mode": "chat", + "input_cost_per_token": RATES.0, + "output_cost_per_token": RATES.1, + }); + let map: serde_json::Map = [ + ("groq/llama-3.1-8b-instant", "groq"), + ("mistral/mistral-small-latest", "mistral"), + ("xai/grok-3", "xai"), + ( + "hosted_vllm/meta-llama/Llama-3.1-8B-Instruct", + "hosted_vllm", + ), + ] + .into_iter() + .map(|(model, provider)| { + let mut entry = price.clone(); + entry["litellm_provider"] = json!(provider); + (model.to_owned(), entry) + }) + .collect(); + let mut file = tempfile::NamedTempFile::new().unwrap(); + file.write_all(Value::Object(map).to_string().as_bytes()) + .unwrap(); + let (_, path) = file.keep().unwrap(); + // SAFETY: set once, before any test builds a gateway or spawns a thread that reads the + // environment. + unsafe { std::env::set_var("LITELLM_MODEL_COST_MAP_PATH", path) }; + }); +} + +#[derive(Clone, Debug)] +struct Sent { + path: String, + headers: BTreeMap, + body: Value, +} + +type Calls = Arc>>; + +fn stream_answer(path: &str) -> String { + match path.split('/').nth(1) { + Some("xai") => [CONTENT_CHUNK, XAI_USAGE_CHUNK, DONE].concat(), + Some("mistral") => [CONTENT_CHUNK, MISTRAL_LAST_CHUNK, DONE].concat(), + Some("vllm") => [ + "data: {\"id\":\"c\",\"choices\":[{\"index\":0,\"delta\":{\"reasoning\":\"hmm\"}}]}\n\n", + CONTENT_CHUNK, + USAGE_CHUNK, + DONE, + ] + .concat(), + _ => [CONTENT_CHUNK, USAGE_CHUNK, DONE].concat(), + } +} + +async fn upstream(calls: Calls) -> String { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let app = Router::new().fallback(move |request: Request| { + let calls = calls.clone(); + async move { + let (parts, body) = request.into_parts(); + let headers = parts + .headers + .iter() + .filter_map(|(name, value)| { + Some((name.as_str().to_owned(), value.to_str().ok()?.to_owned())) + }) + .collect(); + let path = parts.uri.path().to_owned(); + let body: Value = + serde_json::from_slice(&to_bytes(body, 1 << 20).await.unwrap()).unwrap(); + let stream = body["stream"].as_bool().unwrap_or(false); + calls.lock().unwrap().push(Sent { + path: path.clone(), + headers, + body, + }); + match stream { + true => { + ([(CONTENT_TYPE, "text/event-stream")], stream_answer(&path)).into_response() + } + false => ([(CONTENT_TYPE, "application/json")], COMPLETION).into_response(), + } + } + }); + tokio::spawn(async move { axum::serve(listener, app).await }); + format!("http://{address}") +} + +#[derive(Debug)] +struct Seen { + deployment: Option, + usage: Option<(u64, u64)>, + cost: Option, +} + +struct Collect(Mutex>); + +impl Observer for Collect { + fn name(&self) -> &'static str { + "collect" + } + + fn reads(&self) -> Facts { + Facts::DEPLOYMENT | Facts::USAGE | Facts::COST + } + + fn observe<'a>(&'a self, record: &'a Record) -> BoxFuture<'a, ()> { + Box::pin(async move { + self.0.lock().unwrap().push(Seen { + deployment: record + .outcome + .deployment + .as_ref() + .map(|deployment| deployment.id.to_string()), + usage: record + .outcome + .usage + .map(|usage| (usage.prompt_tokens(), usage.output_tokens)), + cost: record.outcome.cost, + }); + }) + } +} + +struct Harness { + app: Router, + inference: Arc, + calls: Calls, + seen: Arc, +} + +/// One deployment per provider, each at its own path on the mock upstream. +const DEPLOYMENTS: &[(&str, &str, &str, Option<&str>)] = &[ + ( + "vllm", + "hosted_vllm/meta-llama/Llama-3.1-8B-Instruct", + "/vllm/v1", + None, + ), + ( + "nim", + "nvidia_nim/meta/llama3-70b-instruct", + "/nim/v1", + Some("sk-nim"), + ), + ( + "groq", + "groq/llama-3.1-8b-instant", + "/groq/openai/v1", + Some("sk-groq"), + ), + ( + "together", + "together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo", + "/together/v1", + Some("sk-together"), + ), + ( + "fireworks", + "fireworks_ai/llama-v3p1-8b-instruct", + "/fireworks/inference/v1", + Some("sk-fireworks"), + ), + ("xai", "xai/grok-3", "/xai/v1", Some("sk-xai")), + ( + "mistral", + "mistral/mistral-small-latest", + "/mistral", + Some("sk-mistral"), + ), + ( + "deepseek", + "deepseek/deepseek-chat", + "/deepseek/beta", + Some("sk-deepseek"), + ), + ( + "openrouter", + "openrouter/openai/gpt-4o", + "/openrouter/api/v1", + Some("sk-openrouter"), + ), + ( + "cerebras", + "cerebras/llama3.1-8b", + "/cerebras/v1", + Some("sk-cerebras"), + ), + ( + "perplexity", + "perplexity/sonar", + "/perplexity", + Some("sk-perplexity"), + ), + ( + "custom-openai", + "openai/my-served-model", + "/custom/v1", + Some("sk-custom"), + ), +]; + +async fn harness() -> Harness { + use_local_cost_map(); + let calls = Calls::default(); + let upstream = upstream(calls.clone()).await; + let models: String = DEPLOYMENTS + .iter() + .map(|(name, model, base, key)| { + let key = key.map_or_else(String::new, |key| format!("\n api_key: {key}")); + format!( + " - model_name: {name} + litellm_params: + model: {model} + api_base: {upstream}{base}{key} + model_info: + id: {name}-1 +" + ) + }) + .collect(); + let config = Config::from_yaml(&format!( + "model_list: +{models}router_settings: + num_retries: 0 +general_settings: + master_key: sk-master +" + )) + .unwrap(); + let inference = litellm_gateway::build_inference(&config).unwrap(); + let seen = Arc::new(Collect(Mutex::new(Vec::new()))); + litellm_gateway::hooks::install(&inference, &config, Vec::new(), vec![seen.clone()]) + .await + .unwrap(); + let app = litellm_gateway::router(inference.clone(), &config, None, None); + Harness { + app, + inference, + calls, + seen, + } +} + +async fn send(app: &Router, body: Value) -> (u16, String) { + let request = Request::post("/v1/chat/completions") + .header("content-type", "application/json") + .header("authorization", "Bearer sk-master") + .body(Body::from(body.to_string())) + .unwrap(); + let response = app.clone().oneshot(request).await.unwrap(); + let status = response.status().as_u16(); + let body = to_bytes(response.into_body(), 1 << 20).await.unwrap(); + (status, String::from_utf8(body.to_vec()).unwrap()) +} + +async fn seen(harness: &Harness) -> Vec { + litellm_gateway::hooks::close(&harness.inference).await; + std::mem::take(&mut *harness.seen.0.lock().unwrap()) +} + +fn hi(model: &str) -> Value { + json!({ + "model": model, + "messages": [{"role": "user", "content": "hi"}], + "max_completion_tokens": 8, + "temperature": 0.5, + "metadata": {"trace": "t"}, + }) +} + +fn messages() -> Value { + json!([{"role": "user", "content": "hi"}]) +} + +#[rstest] +#[case::hosted_vllm("vllm", "/vllm/v1/chat/completions", "Bearer fake-api-key", None, + json!({"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": messages(), "max_completion_tokens": 8, "temperature": 0.5}))] +#[case::nvidia_nim("nim", "/nim/v1/chat/completions", "Bearer sk-nim", None, + json!({"model": "meta/llama3-70b-instruct", "messages": messages(), "max_tokens": 8, "temperature": 0.5}))] +#[case::groq("groq", "/groq/openai/v1/chat/completions", "Bearer sk-groq", None, + json!({"model": "llama-3.1-8b-instant", "messages": messages(), "max_tokens": 8, "temperature": 0.5}))] +#[case::together_ai("together", "/together/v1/chat/completions", "Bearer sk-together", None, + json!({"model": "meta-llama/Llama-3.3-70B-Instruct-Turbo", "messages": messages(), "max_completion_tokens": 8, "temperature": 0.5}))] +#[case::fireworks_ai("fireworks", "/fireworks/inference/v1/chat/completions", "Bearer sk-fireworks", None, + json!({"model": "accounts/fireworks/models/llama-v3p1-8b-instruct", "messages": messages(), "max_tokens": 8, "temperature": 0.5}))] +#[case::xai("xai", "/xai/v1/chat/completions", "Bearer sk-xai", None, + json!({"model": "grok-3", "messages": messages(), "max_tokens": 8, "temperature": 0.5}))] +#[case::mistral("mistral", "/mistral/v1/chat/completions", "Bearer sk-mistral", None, + json!({"model": "mistral-small-latest", "messages": messages(), "max_tokens": 8, "temperature": 0.5}))] +#[case::deepseek("deepseek", "/deepseek/beta/chat/completions", "Bearer sk-deepseek", None, + json!({"model": "deepseek-chat", "messages": messages(), "max_completion_tokens": 8, "temperature": 0.5}))] +#[case::openrouter("openrouter", "/openrouter/api/v1/chat/completions", "Bearer sk-openrouter", + Some(("x-title", "liteLLM")), + json!({"model": "openai/gpt-4o", "messages": messages(), "max_completion_tokens": 8, "temperature": 0.5, "usage": {"include": true}}))] +#[case::cerebras("cerebras", "/cerebras/v1/chat/completions", "Bearer sk-cerebras", None, + json!({"model": "llama3.1-8b", "messages": messages(), "max_tokens": 8, "temperature": 0.5}))] +#[case::perplexity("perplexity", "/perplexity/chat/completions", "Bearer sk-perplexity", + Some(("x-pplx-integration", "litellm")), + json!({"model": "sonar", "messages": messages(), "max_completion_tokens": 8, "temperature": 0.5}))] +#[case::openai_custom_base("custom-openai", "/custom/v1/chat/completions", "Bearer sk-custom", None, + json!({"model": "my-served-model", "messages": messages(), "max_completion_tokens": 8, "temperature": 0.5}))] +#[tokio::test] +async fn each_provider_gets_its_url_auth_and_body( + #[case] model: &str, + #[case] path: &str, + #[case] authorization: &str, + #[case] header: Option<(&str, &str)>, + #[case] body: Value, +) { + let harness = harness().await; + let (status, answer) = send(&harness.app, hi(model)).await; + assert_eq!(status, 200, "{answer}"); + assert_eq!( + serde_json::from_str::(&answer).unwrap()["choices"][0]["message"]["content"], + "hi" + ); + + let sent = harness.calls.lock().unwrap()[0].clone(); + assert_eq!(sent.path, path); + assert_eq!( + sent.headers.get("authorization").map(String::as_str), + Some(authorization) + ); + if let Some((name, value)) = header { + assert_eq!(sent.headers.get(name).map(String::as_str), Some(value)); + } + assert_eq!(sent.body, body); + + let seen = seen(&harness).await; + assert_eq!(seen.len(), 1, "{seen:?}"); + assert_eq!(seen[0].deployment, Some(format!("{model}-1"))); + assert_eq!(seen[0].usage, Some((5, 6))); +} + +#[rstest] +#[case::groq_asks_on_the_clients_behalf("groq", false, Some(json!({"include_usage": true})), (7, 3))] +#[case::groq_client_asked("groq", true, Some(json!({"include_usage": true})), (7, 3))] +#[case::mistral_reports_unasked("mistral", false, None, (7, 3))] +#[case::xai_folds_reasoning_into_completion("xai", false, Some(json!({"include_usage": true})), (7, 7))] +#[case::hosted_vllm("vllm", false, Some(json!({"include_usage": true})), (7, 3))] +#[tokio::test] +async fn a_stream_is_metered_and_priced( + #[case] model: &str, + #[case] client_usage: bool, + #[case] stream_options: Option, + #[case] usage: (u64, u64), +) { + let harness = harness().await; + let mut request = hi(model); + request["stream"] = json!(true); + if client_usage { + request["stream_options"] = json!({"include_usage": true}); + } + let (status, body) = send(&harness.app, request).await; + assert_eq!(status, 200, "{body}"); + assert!(body.contains("\"content\":\"hi\""), "{body}"); + assert!(body.ends_with(DONE), "{body}"); + assert_eq!(body.contains("\"usage\":{"), client_usage, "{body}"); + + let sent = harness.calls.lock().unwrap()[0].clone(); + assert_eq!(sent.body.get("stream_options").cloned(), stream_options); + + let seen = seen(&harness).await; + assert_eq!(seen.len(), 1, "{seen:?}"); + assert_eq!(seen[0].usage, Some(usage)); + let expected = usage.0 as f64 * RATES.0 + usage.1 as f64 * RATES.1; + assert!((seen[0].cost.unwrap() - expected).abs() < 1e-9, "{seen:?}"); +} + +#[rstest] +#[tokio::test] +async fn a_vllm_stream_names_its_reasoning_as_litellm_does() { + let harness = harness().await; + let mut request = hi("vllm"); + request["stream"] = json!(true); + let (status, body) = send(&harness.app, request).await; + assert_eq!(status, 200, "{body}"); + assert!(body.contains("\"reasoning_content\":\"hmm\""), "{body}"); + assert!(!body.contains("\"reasoning\":"), "{body}"); +} + +#[rstest] +#[case::groq_streamed_response_format("groq", json!({"stream": true, "response_format": {"type": "json_object"}}), 501)] +#[case::param_the_model_lacks("cerebras", json!({"frequency_penalty": 1}), 400)] +#[case::fireworks_file_part("fireworks", json!({"messages": [{"role": "user", "content": [{"type": "file", "file": {"file_id": "f"}}]}]}), 400)] +#[tokio::test] +async fn requests_a_provider_cannot_take_are_refused_before_the_call( + #[case] model: &str, + #[case] extra: Value, + #[case] status: u16, +) { + let harness = harness().await; + let mut request = hi(model); + let (Value::Object(request_fields), Value::Object(extra)) = (&mut request, extra) else { + unreachable!() + }; + request_fields.extend(extra); + let (sent_status, body) = send(&harness.app, request).await; + assert_eq!(sent_status, status, "{body}"); + assert!(harness.calls.lock().unwrap().is_empty()); +} diff --git a/litellm-rust/crates/gateway/tests/vertex_claude_chat.rs b/litellm-rust/crates/gateway/tests/vertex_claude_chat.rs new file mode 100644 index 00000000000..d2e8c711a02 --- /dev/null +++ b/litellm-rust/crates/gateway/tests/vertex_claude_chat.rs @@ -0,0 +1,624 @@ +//! Claude chat completions on Vertex AI through the gateway (`vertex_ai/claude-*`, a service +//! account traded for an access token): the `rawPredict` request Vertex gets, the OpenAI answer +//! the client gets, and the usage and cost the request is recorded with. + +use std::{ + io::Write, + sync::{Arc, Mutex, Once}, +}; + +use axum::{ + Router, + body::{Body, to_bytes}, + http::{Request, StatusCode, header::CONTENT_TYPE}, + response::IntoResponse, +}; +use futures_util::future::BoxFuture; +use litellm_config::Config; +use litellm_gateway_hooks::{Facts, Observer, Record}; +use rstest::rstest; +use serde_json::{Value, json}; +use tokio::net::TcpListener; +use tower::ServiceExt; + +/// Input and output prices of every deployment. +const RATES: (f64, f64) = (1.0, 2.0); + +const ACCESS_TOKEN: &str = "ya29.test-token"; +const PROJECT: &str = "test-project"; +const LOCATION: &str = "us-east5"; +const MODEL: &str = "claude-sonnet-4-5@20250929"; + +/// A throwaway RSA key the fake service account signs its token request with. +const TEST_PRIVATE_KEY: &str = "-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC95ihJQlfJuaQe +ZCwF+EFy/Y5vk7Lx6Nm74He9Oes+6ZbgA2LVdcoFJrNEYHd8S0CCv7cqEp4dpIoR +B4HAmQAQF92ZC/S2lcgLAezWQU21TSjDLngdVbn+Jr4syzOwVw/NUigydtQ66Iqv +1FpXs/4gQXMgs8FW5XgOhgfKhyMPf71A2qiRaA0vwsrfsmKOY0QVVooXPRfpLIvZ +VEOV89ZMQErnYan+m8OADoZ5NJ/Ij6QKb2HkQJNhQC2Zq2T1Y9VLyxY6pXURdVqc +XJgFT6YPGjLABLX/oId4g+Fs2YI5mM+g72pYEntnIaE5yicGKBdjyqeVfKZ6/7jg +g8evHi6vAgMBAAECggEAGdJjlP6b8Fa5bdaCM/ebcrbuuNZVJVbb0JPHxGfNSLs7 +pE9hj5QaOdQW2Uviw3h6F61ZCzQH4xD+Iy2po5ZKb2XHYKnDB1bboj+LRGER337T +9aJqe9at2VTMVEv3Rdm40NsEk0QcPLxlK16NQFK90gYEUSSQPDAswJDSG2R/zHn+ +vADI907mW/goEJHeLn8PWGlNlSiR6x+5JJtq+GXCzUzVvJYQSCLGxCSl2x2H+0g7 +NhFI0zPpdzNmO/h+yhzaFb6Rp5U8+ZsnZ3qYjQ/03gw1myTDKJt1YaO9JvArnNYX +hcJQQ8Rt0bHhcrZA16bBOpqZlo5pKCicwI/netgN8QKBgQDcFz7AzdJ26sMSV32V +rwrMgIoggt8qDjO1ARwqW35A1TIge0FoW4M4KpsXQGGfT341uU1esXEcyZ/1L/5X +3ql2gX4DbOYLZLWYzZGR2hq33oi8HkhN98QrEwL9emSH8NqYX3Xxja3PrmCrSYJe +Zbnd9TIm2XkxyMoyXJu6M/QvnwKBgQDc4dzqTbxoGEGa5MuJoGmMwPnqgdG9UM5J +eExVnh7osxc2sOdsiPeRjjQTxs9v2kJwctC359OJoo9yGaaJeSghU4LEWJo1sqnA +fzSCLammYvtVAtniyNv5Mxk/6Uimi4NNDKaAKB+m4K2uSn3U9AmY7KPYMGaSbS9W +XSnobjxm8QKBgC8bPpAvvWs8ZhIn7bY659nLbUT2HeO3dHO6UBf0yzn/J6JyHxbB +93zvCZDZc8uQTRgcmCW7XtVlhjoJUqvl+Wlm39zF0xr/LCsPXKfWAb/2/lcdOCaP +8Emz4QD10EyUTYUtcWYJB/mafhBLRH8F0Nlj4J8WDu2L51MOJTqeYhZLAoGAWffN +icocAbJPlo22sdoa4+/+W5yBF8GAJMDRJtZ+9H1t6SLpQHYRkMIBSETkXUTjZvX9 +Ocs9iIQkNW9pO/mTdO+VBfCo71JUfknR02xR+6m5gYjlws/ZeYlssXGN2/hbhNiw +QOcW7Vv6olFJK6Iy/oz0t6wPO3kpnN3Zogi0paECgYEAwo44M1DdYCtV0snhmYM9 +5u0mPfYt5P2SVLXyUbr+vFTfrTL/WKnXIJgbsnj3Gvf+GIZv9tKcXhSNmEHQCYX4 +X3w9iTPddCHuvZ1fpufi2TyArJh0OkoNtLXJHTKrHjf2N+61AQzFiv5WieJrdE+H +qr32PTUuVGPyO9LyTY4/RL0= +-----END PRIVATE KEY----- +"; + +/// The gateway reads its cost map from this variable; every test sets it before building a +/// gateway, so no test reads the environment while it changes. +fn use_local_cost_map() { + static ONCE: Once = Once::new(); + ONCE.call_once(|| { + let map = json!({ + format!("vertex_ai/{MODEL}"): { + "mode": "chat", + "litellm_provider": "vertex_ai-anthropic_models", + "max_output_tokens": 64000, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_reasoning": true, + }, + }); + let mut file = tempfile::NamedTempFile::new().unwrap(); + file.write_all(map.to_string().as_bytes()).unwrap(); + let (_, path) = file.keep().unwrap(); + // SAFETY: set once, before any test builds a gateway or spawns a thread that reads the + // environment. + unsafe { std::env::set_var("LITELLM_MODEL_COST_MAP_PATH", path) }; + }); +} + +fn message(content: Value, stop_reason: &str) -> Value { + json!({ + "id": "msg_1", + "type": "message", + "role": "assistant", + "model": "claude-sonnet-4-5", + "content": content, + "stop_reason": stop_reason, + "stop_sequence": null, + "usage": {"input_tokens": 10, "output_tokens": 4}, + }) +} + +fn text_answer(text: &str) -> Value { + message(json!([{"type": "text", "text": text}]), "end_turn") +} + +fn tool_answer() -> Value { + message( + json!([{"type": "tool_use", "id": "toolu_1", "name": "get_weather", "input": {"city": "Paris"}}]), + "tool_use", + ) +} + +fn thinking_answer() -> Value { + message( + json!([ + {"type": "thinking", "thinking": "The user says hi.", "signature": "sig-1"}, + {"type": "text", "text": "Hello"} + ]), + "end_turn", + ) +} + +fn sse_answer() -> String { + [ + json!({"type": "message_start", "message": {"id": "msg_2", "type": "message", "role": "assistant", "model": "claude-sonnet-4-5", "content": [], "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 10, "output_tokens": 1}}}), + json!({"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}), + json!({"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "Hello"}}), + json!({"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": " world"}}), + json!({"type": "content_block_stop", "index": 0}), + json!({"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}, "usage": {"output_tokens": 12}}), + json!({"type": "message_stop"}), + ] + .iter() + .map(|event| format!("event: {}\ndata: {event}\n\n", event["type"].as_str().unwrap())) + .collect() +} + +#[derive(Clone, Debug)] +struct Sent { + path: String, + query: Option, + authorization: Option, + api_key: Option, + anthropic_version: Option, + anthropic_beta: Option, + body: Value, +} + +type Calls = Arc>>; + +fn has_tool_result(body: &Value) -> bool { + body["messages"].as_array().is_some_and(|messages| { + messages.iter().any(|message| { + message["content"] + .as_array() + .is_some_and(|blocks| blocks.iter().any(|block| block["type"] == "tool_result")) + }) + }) +} + +/// Google's token endpoint and Vertex's `rawPredict` and `streamRawPredict`, answering by what +/// the body asks. +async fn upstream(calls: Calls) -> String { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let app = Router::new().fallback(move |request: Request| { + let calls = calls.clone(); + async move { + let (parts, body) = request.into_parts(); + let header = |name: &str| { + parts + .headers + .get(name) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned) + }; + let bytes = to_bytes(body, 1 << 20).await.unwrap(); + let path = parts.uri.path().to_owned(); + if path == "/token" { + return ( + [(CONTENT_TYPE, "application/json")], + json!({"access_token": ACCESS_TOKEN, "expires_in": 3600, "token_type": "Bearer"}) + .to_string(), + ) + .into_response(); + } + let body: Value = serde_json::from_slice(&bytes).unwrap_or(Value::Null); + calls.lock().unwrap().push(Sent { + path: path.clone(), + query: parts.uri.query().map(str::to_owned), + authorization: header("authorization"), + api_key: header("x-api-key"), + anthropic_version: header("anthropic-version"), + anthropic_beta: header("anthropic-beta"), + body: body.clone(), + }); + match path.ends_with(":streamRawPredict") { + true => ([(CONTENT_TYPE, "text/event-stream")], sse_answer()).into_response(), + false => { + let answer = match (has_tool_result(&body), body.get("tools"), body.get("thinking")) { + (true, _, _) => text_answer("It is sunny in Paris"), + (false, Some(_), _) => tool_answer(), + (false, None, Some(_)) => thinking_answer(), + (false, None, None) => text_answer("Hello world"), + }; + ( + StatusCode::OK, + [(CONTENT_TYPE, "application/json")], + answer.to_string(), + ) + .into_response() + } + } + } + }); + tokio::spawn(async move { axum::serve(listener, app).await }); + format!("http://{address}") +} + +#[derive(Debug)] +struct Seen { + deployment: Option, + usage: Option, + cost: Option, +} + +struct Collect(Mutex>); + +impl Observer for Collect { + fn name(&self) -> &'static str { + "collect" + } + + fn reads(&self) -> Facts { + Facts::DEPLOYMENT | Facts::USAGE | Facts::COST + } + + fn observe<'a>(&'a self, record: &'a Record) -> BoxFuture<'a, ()> { + Box::pin(async move { + self.0.lock().unwrap().push(Seen { + deployment: record + .outcome + .deployment + .as_ref() + .map(|deployment| deployment.id.to_string()), + usage: record.outcome.usage, + cost: record.outcome.cost, + }); + }) + } +} + +struct Harness { + app: Router, + inference: Arc, + calls: Calls, + seen: Arc, +} + +fn service_account(upstream: &str) -> String { + json!({ + "type": "service_account", + "project_id": "credentials-project", + "private_key_id": "test-key-id", + "private_key": TEST_PRIVATE_KEY, + "client_email": "gateway-test@credentials-project.iam.gserviceaccount.com", + "client_id": "1", + "token_uri": format!("{upstream}/token"), + }) + .to_string() +} + +async fn harness() -> Harness { + use_local_cost_map(); + let calls = Calls::default(); + let upstream = upstream(calls.clone()).await; + let (input, output) = RATES; + let prices = + format!(" input_cost_per_token: {input}\n output_cost_per_token: {output}\n"); + let credentials = service_account(&upstream); + let config = Config::from_yaml(&format!( + "model_list: + - model_name: claude + litellm_params: + model: vertex_ai/{MODEL} + api_base: {upstream} + vertex_project: {PROJECT} + vertex_location: {LOCATION} + vertex_credentials: '{credentials}' +{prices} model_info: + id: claude-1 + - model_name: claude-global + litellm_params: + model: vertex_ai/{MODEL} + api_base: {upstream} + vertex_project: {PROJECT} + vertex_location: global + vertex_credentials: '{credentials}' +{prices} model_info: + id: claude-global-1 +general_settings: + master_key: sk-master +" + )) + .unwrap(); + let inference = litellm_gateway::build_inference(&config).unwrap(); + let seen = Arc::new(Collect(Mutex::new(Vec::new()))); + litellm_gateway::hooks::install(&inference, &config, Vec::new(), vec![seen.clone()]) + .await + .unwrap(); + let app = litellm_gateway::router(inference.clone(), &config, None, None); + Harness { + app, + inference, + calls, + seen, + } +} + +async fn send(app: &Router, body: Value) -> (u16, String) { + let request = Request::post("/v1/chat/completions") + .header("content-type", "application/json") + .header("authorization", "Bearer sk-master") + .body(Body::from(body.to_string())) + .unwrap(); + let response = app.clone().oneshot(request).await.unwrap(); + let status = response.status().as_u16(); + let body = to_bytes(response.into_body(), 1 << 20).await.unwrap(); + (status, String::from_utf8(body.to_vec()).unwrap()) +} + +async fn seen(harness: &Harness) -> Vec { + litellm_gateway::hooks::close(&harness.inference).await; + std::mem::take(&mut *harness.seen.0.lock().unwrap()) +} + +fn cost(prompt: u64, completion: u64) -> f64 { + let (input, output) = RATES; + prompt as f64 * input + completion as f64 * output +} + +fn hi(model: &str) -> Value { + json!({ + "model": model, + "messages": [ + {"role": "system", "content": "Be brief"}, + {"role": "user", "content": "hi"} + ], + "temperature": 0.2, + "max_tokens": 64, + }) +} + +fn sent_body() -> Value { + json!({ + "messages": [{"role": "user", "content": [{"type": "text", "text": "hi"}]}], + "temperature": 0.2, + "max_tokens": 64, + "anthropic_version": "vertex-2023-10-16", + "system": [{"type": "text", "text": "Be brief"}], + }) +} + +fn raw_predict_path(location: &str, method: &str) -> String { + format!( + "/v1/projects/{PROJECT}/locations/{location}/publishers/anthropic/models/{MODEL}:{method}" + ) +} + +#[rstest] +#[case::regional("claude", "claude-1", LOCATION)] +#[case::global("claude-global", "claude-global-1", "global")] +#[tokio::test] +async fn a_chat_completion_calls_raw_predict_and_records_its_cost( + #[case] model: &str, + #[case] id: &str, + #[case] location: &str, +) { + let harness = harness().await; + let (status, body) = send(&harness.app, hi(model)).await; + assert_eq!(status, 200, "{body}"); + let answer: Value = serde_json::from_str(&body).unwrap(); + // `VertexAIAnthropicConfig.transform_response` names the requested model. + assert_eq!(answer["model"], MODEL); + assert_eq!(answer["choices"][0]["message"]["role"], "assistant"); + assert_eq!(answer["choices"][0]["message"]["content"], "Hello world"); + assert_eq!(answer["choices"][0]["finish_reason"], "stop"); + assert_eq!(answer["usage"]["prompt_tokens"], 10); + assert_eq!(answer["usage"]["completion_tokens"], 4); + assert_eq!(answer["usage"]["total_tokens"], 14); + + let sent = harness.calls.lock().unwrap()[0].clone(); + assert_eq!(sent.path, raw_predict_path(location, "rawPredict")); + assert_eq!(sent.query, None); + assert_eq!( + sent.authorization.as_deref(), + Some("Bearer ya29.test-token") + ); + assert_eq!(sent.api_key, None); + assert_eq!(sent.anthropic_version.as_deref(), Some("2023-06-01")); + assert_eq!(sent.anthropic_beta, None); + assert_eq!(sent.body, sent_body()); + + let seen = seen(&harness).await; + assert_eq!(seen.len(), 1, "{seen:?}"); + assert_eq!(seen[0].deployment.as_deref(), Some(id)); + let usage = seen[0].usage.unwrap(); + assert_eq!((usage.prompt_tokens(), usage.output_tokens), (10, 4)); + assert_eq!(seen[0].cost, Some(cost(10, 4))); +} + +/// The JSON of each `data:` frame, and whether the stream closed with `[DONE]`. +fn frames(body: &str) -> (Vec, bool) { + let mut done = false; + let frames = body + .split("\n\n") + .filter(|frame| !frame.is_empty()) + .filter_map(|frame| { + let data = frame.strip_prefix("data: ").expect("bare data frames"); + match data { + "[DONE]" => { + done = true; + None + } + data => Some(serde_json::from_str(data).unwrap()), + } + }) + .collect(); + (frames, done) +} + +#[rstest] +#[case::regional("claude", "claude-1", LOCATION)] +#[case::global("claude-global", "claude-global-1", "global")] +#[tokio::test] +async fn a_stream_calls_stream_raw_predict_and_records_its_usage( + #[case] model: &str, + #[case] id: &str, + #[case] location: &str, +) { + let harness = harness().await; + let mut request = hi(model); + request["stream"] = json!(true); + request["stream_options"] = json!({"include_usage": true}); + let (status, body) = send(&harness.app, request).await; + assert_eq!(status, 200, "{body}"); + + let (frames, done) = frames(&body); + assert!(done, "{body}"); + assert!( + frames + .iter() + .all(|frame| frame["object"] == "chat.completion.chunk" && frame["model"] == model), + "{body}" + ); + let content: String = frames + .iter() + .filter_map(|frame| frame["choices"][0]["delta"]["content"].as_str()) + .collect(); + assert_eq!(content, "Hello world", "{body}"); + let finishes: Vec<&Value> = frames + .iter() + .filter_map(|frame| frame["choices"][0].get("finish_reason")) + .filter(|reason| !reason.is_null()) + .collect(); + assert_eq!(finishes, [&json!("stop")], "{body}"); + let usages: Vec<&Value> = frames + .iter() + .filter_map(|frame| frame.get("usage")) + .filter(|usage| !usage.is_null()) + .collect(); + assert_eq!(usages.len(), 1, "{body}"); + assert_eq!(usages[0]["prompt_tokens"], 10); + assert_eq!(usages[0]["completion_tokens"], 12); + + let sent = harness.calls.lock().unwrap()[0].clone(); + assert_eq!(sent.path, raw_predict_path(location, "streamRawPredict")); + // A custom `api_base` asks for SSE, as `_check_custom_proxy` does. + assert_eq!(sent.query.as_deref(), Some("alt=sse")); + assert_eq!( + sent.authorization.as_deref(), + Some("Bearer ya29.test-token") + ); + let mut expected = sent_body(); + expected["stream"] = json!(true); + assert_eq!(sent.body, expected); + + let seen = seen(&harness).await; + assert_eq!(seen.len(), 1, "{seen:?}"); + assert_eq!(seen[0].deployment.as_deref(), Some(id)); + let usage = seen[0].usage.unwrap(); + assert_eq!((usage.prompt_tokens(), usage.output_tokens), (10, 12)); + assert_eq!(seen[0].cost, Some(cost(10, 12))); +} + +#[tokio::test] +async fn a_stream_without_include_usage_still_reports_usage_to_the_hooks() { + let harness = harness().await; + let mut request = hi("claude"); + request["stream"] = json!(true); + let (status, body) = send(&harness.app, request).await; + assert_eq!(status, 200, "{body}"); + let (frames, done) = frames(&body); + assert!(done, "{body}"); + assert!( + frames + .iter() + .all(|frame| frame.get("usage").is_none_or(Value::is_null)), + "{body}" + ); + let seen = seen(&harness).await; + assert_eq!(seen.len(), 1, "{seen:?}"); + assert_eq!(seen[0].cost, Some(cost(10, 12))); +} + +#[tokio::test] +async fn a_tool_call_round_trips_through_tool_use_and_tool_result() { + let harness = harness().await; + let tools = json!([{"type": "function", "function": { + "name": "get_weather", + "description": "Weather for a city", + "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]} + }}]); + let question = json!({"role": "user", "content": "Weather in Paris?"}); + let (status, body) = send( + &harness.app, + json!({"model": "claude", "messages": [question], "tools": tools, "tool_choice": "auto", "max_tokens": 64}), + ) + .await; + assert_eq!(status, 200, "{body}"); + let answer: Value = serde_json::from_str(&body).unwrap(); + assert_eq!(answer["choices"][0]["finish_reason"], "tool_calls"); + let call = &answer["choices"][0]["message"]["tool_calls"][0]; + assert_eq!(call["id"], "toolu_1"); + assert_eq!(call["type"], "function"); + assert_eq!(call["function"]["name"], "get_weather"); + assert_eq!(call["function"]["arguments"], "{\"city\": \"Paris\"}"); + + let first = harness.calls.lock().unwrap()[0].body.clone(); + assert_eq!( + first, + json!({ + "messages": [{"role": "user", "content": [{"type": "text", "text": "Weather in Paris?"}]}], + "tools": [{ + "name": "get_weather", + "input_schema": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}, + "type": "custom", + "description": "Weather for a city" + }], + "tool_choice": {"type": "auto"}, + "max_tokens": 64, + "anthropic_version": "vertex-2023-10-16", + }) + ); + + let (status, body) = send( + &harness.app, + json!({"model": "claude", "tools": tools, "max_tokens": 64, "messages": [ + question, + {"role": "assistant", "content": null, "tool_calls": [call]}, + {"role": "tool", "tool_call_id": "toolu_1", "content": "sunny"} + ]}), + ) + .await; + assert_eq!(status, 200, "{body}"); + let answer: Value = serde_json::from_str(&body).unwrap(); + assert_eq!( + answer["choices"][0]["message"]["content"], + "It is sunny in Paris" + ); + let second = harness.calls.lock().unwrap()[1].body.clone(); + assert_eq!( + second["messages"], + json!([ + {"role": "user", "content": [{"type": "text", "text": "Weather in Paris?"}]}, + {"role": "assistant", "content": [{"type": "tool_use", "id": "toolu_1", "name": "get_weather", "input": {"city": "Paris"}}]}, + {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "toolu_1", "content": "sunny"}]} + ]) + ); + + let seen = seen(&harness).await; + assert_eq!(seen.len(), 2, "{seen:?}"); + assert!(seen.iter().all(|seen| seen.cost == Some(cost(10, 4)))); +} + +#[tokio::test] +async fn reasoning_effort_becomes_a_thinking_budget_and_comes_back_as_reasoning() { + let harness = harness().await; + let (status, body) = send( + &harness.app, + json!({"model": "claude", "messages": [{"role": "user", "content": "hi"}], "reasoning_effort": "low"}), + ) + .await; + assert_eq!(status, 200, "{body}"); + let answer: Value = serde_json::from_str(&body).unwrap(); + let message = &answer["choices"][0]["message"]; + assert_eq!(message["content"], "Hello"); + assert_eq!(message["reasoning_content"], "The user says hi."); + assert_eq!( + message["thinking_blocks"], + json!([{"type": "thinking", "thinking": "The user says hi.", "signature": "sig-1"}]) + ); + + let sent = harness.calls.lock().unwrap()[0].body.clone(); + assert_eq!( + sent, + json!({ + "messages": [{"role": "user", "content": [{"type": "text", "text": "hi"}]}], + "thinking": {"type": "enabled", "budget_tokens": 1024}, + "max_tokens": 1024 + 4096, + "anthropic_version": "vertex-2023-10-16", + }) + ); + let seen = seen(&harness).await; + assert_eq!(seen[0].cost, Some(cost(10, 4))); +} + +#[tokio::test] +async fn an_anthropic_api_only_param_is_refused() { + let harness = harness().await; + let mut request = hi("claude"); + request["service_tier"] = json!("auto"); + let (status, body) = send(&harness.app, request).await; + assert_eq!(status, 400, "{body}"); + assert!(harness.calls.lock().unwrap().is_empty()); +} diff --git a/litellm-rust/crates/inference-chat/Cargo.toml b/litellm-rust/crates/inference-chat/Cargo.toml index 4cc32bb7656..2b9413c2467 100644 --- a/litellm-rust/crates/inference-chat/Cargo.toml +++ b/litellm-rust/crates/inference-chat/Cargo.toml @@ -6,11 +6,13 @@ license.workspace = true repository.workspace = true [dependencies] +base64.workspace = true bytes.workspace = true futures-util.workspace = true litellm-auth = { workspace = true, features = ["aws", "azure", "gcp"] } litellm-cache-response.workspace = true litellm-core-utils.workspace = true +litellm-framer.workspace = true litellm-host.workspace = true litellm-http.workspace = true litellm-inference.workspace = true @@ -20,17 +22,16 @@ litellm-secrets.workspace = true reqwest.workspace = true serde_json = { workspace = true, features = ["preserve_order"] } tokio = { workspace = true, features = ["time"] } +tokio-util = { version = "0.7", features = ["codec"] } tracing.workspace = true uuid.workspace = true [dev-dependencies] litellm-cache-memory.workspace = true -litellm-framer.workspace = true litellm-host-native.workspace = true litellm-http = { workspace = true, features = ["test-support"] } litellm-inference-testing.workspace = true litellm-tracing.workspace = true rstest.workspace = true -tokio-util = { version = "0.7", features = ["codec"] } tokio = { workspace = true, features = ["io-util", "sync"] } wiremock.workspace = true diff --git a/litellm-rust/crates/inference-chat/src/azure.rs b/litellm-rust/crates/inference-chat/src/azure.rs index 2f2a59c39b3..3085fbb78a5 100644 --- a/litellm-rust/crates/inference-chat/src/azure.rs +++ b/litellm-rust/crates/inference-chat/src/azure.rs @@ -129,6 +129,7 @@ impl ChatCompletionsRoute { headers: authenticated.headers, body, api_key, + adapter: None, }) } diff --git a/litellm-rust/crates/inference-chat/src/common_utils.rs b/litellm-rust/crates/inference-chat/src/common_utils.rs index 90b75dcddaa..89834504fcd 100644 --- a/litellm-rust/crates/inference-chat/src/common_utils.rs +++ b/litellm-rust/crates/inference-chat/src/common_utils.rs @@ -4,7 +4,16 @@ use litellm_llms::{ anthropic::chat::transformation::ANTHROPIC_CHAT_COMPLETIONS_CONFIG, base_llm::chat::transformation::BaseConfig, bedrock::chat::converse_transformation::BEDROCK_CHAT_COMPLETIONS_CONFIG, + gemini::chat::transformation::GOOGLE_AI_STUDIO_GEMINI_CHAT_COMPLETIONS_CONFIG, openai_like::chat::transformation::OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG, + vertex_ai::{ + gemini::vertex_and_google_ai_studio_gemini::{ + VERTEX_GEMINI_CHAT_COMPLETIONS_CONFIG, is_vertex_gemini_model, + }, + vertex_ai_partner_models::anthropic::transformation::{ + VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG, is_vertex_claude_model, + }, + }, }; use serde_json::{Map, Value}; @@ -15,7 +24,10 @@ const HEADER_CONTEXT: &str = "chat completions"; pub(super) enum ChatProvider { Anthropic, Bedrock, + Gemini, OpenaiLike, + VertexAnthropic, + VertexGemini, } impl ChatProvider { @@ -23,16 +35,28 @@ impl ChatProvider { match self { Self::Anthropic => &ANTHROPIC_CHAT_COMPLETIONS_CONFIG, Self::Bedrock => &BEDROCK_CHAT_COMPLETIONS_CONFIG, + Self::Gemini => &GOOGLE_AI_STUDIO_GEMINI_CHAT_COMPLETIONS_CONFIG, Self::OpenaiLike => &OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG, + Self::VertexAnthropic => &VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG, + Self::VertexGemini => &VERTEX_GEMINI_CHAT_COMPLETIONS_CONFIG, } } } -pub(super) fn chat_completions_provider(provider: LlmProviders) -> Option { +/// Vertex AI serves its Gemini and Claude models here; its other partner models stay refused. +pub(super) fn chat_completions_provider( + provider: LlmProviders, + model: &str, +) -> Option { match provider { LlmProviders::Anthropic => Some(ChatProvider::Anthropic), LlmProviders::Bedrock => Some(ChatProvider::Bedrock), + LlmProviders::Gemini => Some(ChatProvider::Gemini), LlmProviders::OpenaiLike => Some(ChatProvider::OpenaiLike), + LlmProviders::VertexAi if is_vertex_gemini_model(model) => Some(ChatProvider::VertexGemini), + LlmProviders::VertexAi if is_vertex_claude_model(model) => { + Some(ChatProvider::VertexAnthropic) + } _ => None, } } @@ -42,3 +66,35 @@ pub(super) fn string_headers( ) -> Result, Error> { shared_string_headers(HEADER_CONTEXT, extra_headers).map_err(Error::from) } + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + #[rstest] + #[case::studio(LlmProviders::Gemini, "gemini-2.5-flash", Some("gemini"))] + #[case::vertex_gemini(LlmProviders::VertexAi, "gemini-2.5-pro", Some("vertex_gemini"))] + #[case::vertex_claude( + LlmProviders::VertexAi, + "claude-sonnet-4@20250514", + Some("vertex_claude") + )] + #[case::vertex_partner(LlmProviders::VertexAi, "meta/llama-3.1-405b-instruct-maas", None)] + #[case::vertex_mistral(LlmProviders::VertexAi, "mistral-large@2407", None)] + #[case::vertex_gemma(LlmProviders::VertexAi, "gemma/gemma-3-27b-it", None)] + fn vertex_routes_its_gemini_and_claude_models( + #[case] provider: LlmProviders, + #[case] model: &str, + #[case] expected: Option<&str>, + ) { + let routed = chat_completions_provider(provider, model).map(|provider| match provider { + ChatProvider::Gemini => "gemini", + ChatProvider::VertexGemini => "vertex_gemini", + ChatProvider::VertexAnthropic => "vertex_claude", + ChatProvider::Anthropic | ChatProvider::Bedrock | ChatProvider::OpenaiLike => "other", + }); + assert_eq!(routed, expected); + } +} diff --git a/litellm-rust/crates/inference-chat/src/handler.rs b/litellm-rust/crates/inference-chat/src/handler.rs index 5f4d00e1d7c..95764ccda42 100644 --- a/litellm-rust/crates/inference-chat/src/handler.rs +++ b/litellm-rust/crates/inference-chat/src/handler.rs @@ -29,7 +29,7 @@ pub(super) async fn execute( config, url, body, - stream_shape: _, + stream_shape, optional_params, environment, secrets, @@ -122,7 +122,7 @@ pub(super) async fn execute( )) })?; config - .transform_response(&model, ProviderChatResponseData { body }) + .transform_response(&model, ProviderChatResponseData { body }, &stream_shape) .map_err(Error::from) .map_err(as_response_error) }, @@ -229,6 +229,8 @@ mod tests { custom_llm_provider: None, extra_headers: None, timeout: None, + drop_params: Some(false), + model_info: None, }) .unwrap(), std::sync::Arc::new(|_: &str| None), diff --git a/litellm-rust/crates/inference-chat/src/lib.rs b/litellm-rust/crates/inference-chat/src/lib.rs index db4f28f4d2a..7b43105b251 100644 --- a/litellm-rust/crates/inference-chat/src/lib.rs +++ b/litellm-rust/crates/inference-chat/src/lib.rs @@ -8,7 +8,9 @@ pub use litellm_inference::RouteError as Error; mod common_utils; pub mod constants; pub(crate) mod handler; +mod media; pub mod openai; +pub mod openai_compatible; mod prepare; pub mod stream; use litellm_llms_types::formats::chat_completions::ChatCompletionsResponse; @@ -25,6 +27,7 @@ pub struct ChatCompletionsRoute { auth: Arc, secrets: Arc, cache: Option, + media: Option, } impl ChatCompletionsRoute { @@ -38,6 +41,16 @@ impl ChatCompletionsRoute { auth, secrets, cache: None, + media: None, + } + } + + /// Downloads the media URLs a provider cannot take by reference. Without one such a + /// request is refused as unsupported. + pub fn with_media(self, media: litellm_http::media::MediaFetcher) -> Self { + Self { + media: Some(media), + ..self } } @@ -77,7 +90,8 @@ impl ChatCompletionsRoute { interceptors: &impl litellm_host::interceptors::Interceptors, observers: Option<&ObservationSender>, ) -> Result { - let resolved = resolve_request(request)?; + let mut resolved = resolve_request(request)?; + self.inline_remote_media(&mut resolved).await?; let snapshot = self .secrets .resolve(&resolved.config.secret_names()) diff --git a/litellm-rust/crates/inference-chat/src/media.rs b/litellm-rust/crates/inference-chat/src/media.rs new file mode 100644 index 00000000000..8287b496eba --- /dev/null +++ b/litellm-rust/crates/inference-chat/src/media.rs @@ -0,0 +1,158 @@ +//! The media a translated request needs downloaded first: an image or file URL the provider +//! cannot take by reference, which Python fetches while it builds the request. + +use std::{collections::HashMap, time::Duration}; + +use base64::Engine; +use futures_util::future::try_join_all; +use litellm_http::media::DownloadPolicy; +use litellm_llms::base_llm::chat::media::{ + MAX_MEDIA_DOWNLOAD_BYTES, downloaded_media_type, inline_media, +}; + +use crate::{ChatCompletionsRoute, Error, types::ResolvedChatCompletionsRequest}; + +const MEDIA_DOWNLOAD: DownloadPolicy = DownloadPolicy { + timeout: Duration::from_secs(60), + max_bytes: MAX_MEDIA_DOWNLOAD_BYTES, + max_redirects: 10, +}; + +impl ChatCompletionsRoute { + /// Downloads every URL the config names and inlines each as a data URL. A route with no + /// fetcher leaves such a request to the host; a URL that cannot be fetched is the + /// caller's error, as Python's image fetch failure is. + pub(crate) async fn inline_remote_media( + &self, + request: &mut ResolvedChatCompletionsRequest<'_>, + ) -> Result<(), Error> { + let urls = request.config.remote_media(&request.messages); + if urls.is_empty() { + return Ok(()); + } + let fetcher = self + .media + .as_ref() + .ok_or(Error::Unsupported("message media URL"))?; + let fetch = |url: String| async move { + let unable = |reason: String| { + Error::InvalidRequest( + format!("Unable to fetch media from url {url}: {reason}").into(), + ) + }; + let parsed = reqwest::Url::parse(&url).map_err(|err| unable(err.to_string()))?; + let media = fetcher + .fetch(parsed, MEDIA_DOWNLOAD) + .await + .map_err(|err| unable(err.to_string()))?; + let data = format!( + "data:{};base64,{}", + downloaded_media_type(&url, &media.content_type), + base64::engine::general_purpose::STANDARD.encode(media.bytes) + ); + Ok::<_, Error>((url, data)) + }; + let downloaded: HashMap = try_join_all(urls.into_iter().map(fetch)) + .await? + .into_iter() + .collect(); + inline_media(&mut request.messages, &downloaded); + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use litellm_http::{Client, media::MediaFetcher}; + + use rstest::rstest; + use serde_json::{Value, json}; + use wiremock::{ + Mock, MockServer, ResponseTemplate, + matchers::{method, path}, + }; + + use crate::{ + ChatCompletionsRoute, Error, prepare::resolve_request, types::ChatCompletionsRequest, + }; + + fn route(media: bool) -> ChatCompletionsRoute { + let route = crate::test_support::route(); + match media { + true => route.with_media(MediaFetcher::for_test(Client::plain_for_test())), + false => route, + } + } + + fn request(url: &str) -> ChatCompletionsRequest<'static> { + ChatCompletionsRequest { + model: "anthropic/claude-sonnet-4-5", + messages: json!([{"role": "user", "content": [ + {"type": "image_url", "image_url": {"url": url}} + ]}]), + optional_params: serde_json::Map::new(), + api_key: Some("sk-test"), + api_base: None, + custom_llm_provider: None, + extra_headers: None, + timeout: None, + drop_params: Some(false), + model_info: None, + } + } + + fn image(request: &crate::types::ResolvedChatCompletionsRequest<'_>) -> Value { + serde_json::to_value(&request.messages[0].content).unwrap()[0]["image_url"]["url"].clone() + } + + #[rstest] + #[tokio::test] + async fn a_remote_image_is_downloaded_and_inlined() { + let server = MockServer::start().await; + Mock::given(method("GET")) + .and(path("/cat.png")) + .respond_with(ResponseTemplate::new(200).set_body_raw(b"png".to_vec(), "image/png")) + .mount(&server) + .await; + let url = format!("{}/cat.png", server.uri()); + let url: &'static str = Box::leak(url.into_boxed_str()); + let mut resolved = resolve_request(request(url)).unwrap(); + + route(true) + .inline_remote_media(&mut resolved) + .await + .unwrap(); + + assert_eq!(image(&resolved), "data:image/png;base64,cG5n"); + } + + #[rstest] + #[tokio::test] + async fn a_media_url_that_cannot_be_fetched_is_the_callers_error() { + let server = MockServer::start().await; + let url = format!("{}/missing.png", server.uri()); + let url: &'static str = Box::leak(url.into_boxed_str()); + let mut resolved = resolve_request(request(url)).unwrap(); + + let error = route(true) + .inline_remote_media(&mut resolved) + .await + .unwrap_err(); + + assert!(matches!(error, Error::InvalidRequest(_)), "{error:?}"); + } + + #[rstest] + #[tokio::test] + async fn without_a_fetcher_a_media_url_is_unsupported_and_a_data_url_is_left_alone() { + let mut remote = resolve_request(request("http://example.com/cat.png")).unwrap(); + assert!(matches!( + route(false).inline_remote_media(&mut remote).await, + Err(Error::Unsupported(_)) + )); + + let mut inline = resolve_request(request("data:image/png;base64,cG5n")).unwrap(); + route(false).inline_remote_media(&mut inline).await.unwrap(); + assert_eq!(image(&inline), "data:image/png;base64,cG5n"); + } +} diff --git a/litellm-rust/crates/inference-chat/src/openai.rs b/litellm-rust/crates/inference-chat/src/openai.rs index bfc82c4b09a..7e97767ccb8 100644 --- a/litellm-rust/crates/inference-chat/src/openai.rs +++ b/litellm-rust/crates/inference-chat/src/openai.rs @@ -24,8 +24,11 @@ use litellm_llms::{ use serde_json::{Map, Value}; use crate::{ - ChatCompletionsRoute, Error, common_utils::string_headers, + ChatCompletionsRoute, Error, + common_utils::string_headers, constants::CHAT_COMPLETIONS_TIMEOUT_SECS, + openai_compatible::{ResponseAdapter, compatible_chat_config}, + stream::ReportedUsage, }; pub struct OpenAiChatCompletions; @@ -61,6 +64,9 @@ pub struct OpenAiChatCall { pub drop_params: bool, /// The model cost map, which Azure's param rules read. pub model_info: Option>, + /// Where a stream's usage is recorded for the host when the client did not ask to see + /// it. + pub usage: ReportedUsage, } /// A provider request ready to go out: where, with which headers and body, and who serves it. @@ -71,17 +77,18 @@ pub(crate) struct PreparedCall { pub(crate) headers: Headers, pub(crate) body: Map, pub(crate) api_key: Option, + pub(crate) adapter: Option, } impl ChatCompletionsRoute { - /// Whether the deployment is served by the OpenAI wire passthrough: OpenAI itself, or - /// Azure OpenAI. + /// Whether the deployment is served by the OpenAI wire passthrough: OpenAI itself, Azure + /// OpenAI, or an OpenAI-compatible provider. pub fn serves_openai(model: &str, custom_llm_provider: Option<&str>) -> bool { resolve_llm_provider(model, custom_llm_provider, "chat completions").is_ok_and(|resolved| { matches!( resolved.provider, LlmProviders::Openai | LlmProviders::Azure - ) + ) || compatible_chat_config(resolved.provider).is_some() }) } @@ -149,10 +156,14 @@ impl ChatCompletionsRoute { "chat completions", )?; let model = resolved.model.to_string(); - let prepared = match resolved.provider { - LlmProviders::Openai => self.prepare_openai(model, call).await?, - LlmProviders::Azure => self.prepare_azure(model, call).await?, - other => return Err(Error::InvalidProvider(<&str>::from(other).to_string())), + let usage = call.usage.clone(); + let prepared = match (resolved.provider, compatible_chat_config(resolved.provider)) { + (LlmProviders::Openai, _) => self.prepare_openai(model, call).await?, + (LlmProviders::Azure, _) => self.prepare_azure(model, call).await?, + (_, Some(config)) => self.prepare_openai_compatible(config, model, call).await?, + (other, None) => { + return Err(Error::InvalidProvider(<&str>::from(other).to_string())); + } }; let PreparedCall { model, @@ -161,6 +172,7 @@ impl ChatCompletionsRoute { headers, body, api_key, + adapter, } = prepared; litellm_inference::diagnostic::provider(&model, provider); let request_context = RequestContext { @@ -192,12 +204,20 @@ impl ChatCompletionsRoute { .unwrap_or(false); let response = send(&self.http, wire, stream, timeout).await?; let output = if stream { + let chunks = chunks(response, timeout); CallOutput::Stream { head: (), - chunks: chunks(response, timeout), + chunks: match adapter { + Some(adapter) => adapter.stream(chunks, usage), + None => chunks, + }, } } else { let body = response.bytes().await.map_err(network)?; + let body = match &adapter { + Some(adapter) => adapter.complete(body), + None => body, + }; context .response_received(&String::from_utf8_lossy(&body)) .await?; @@ -246,6 +266,7 @@ impl ChatCompletionsRoute { headers: authenticated.headers, body, api_key: call.api_key, + adapter: None, }) } } @@ -354,6 +375,9 @@ mod tests { #[case::azure("azure/gpt-4o", None, true)] #[case::azure_ai("azure_ai/gpt-4o", None, false)] #[case::openai_like("openai_like/gpt-6-astra", None, false)] + #[case::groq("groq/llama-3.1-8b-instant", None, true)] + #[case::hosted_vllm("meta-llama/Llama-3.1-8B", Some("hosted_vllm"), true)] + #[case::nvidia_nim("nvidia_nim/meta/llama3-70b-instruct", None, true)] #[case::anthropic("anthropic/claude", None, false)] #[case::unprefixed("gpt-6-astra", None, false)] fn only_openai_deployments_take_the_passthrough( diff --git a/litellm-rust/crates/inference-chat/src/openai_compatible.rs b/litellm-rust/crates/inference-chat/src/openai_compatible.rs new file mode 100644 index 00000000000..19677ba2154 --- /dev/null +++ b/litellm-rust/crates/inference-chat/src/openai_compatible.rs @@ -0,0 +1,416 @@ +//! The OpenAI-compatible providers (`litellm/llms//chat/transformation.py`) through +//! the OpenAI passthrough: each provider's config shapes the request, and the response goes +//! back as the provider sent it but for the fixups its config makes. A stream is relayed +//! event by event, with the usage the call asked for on the client's behalf recorded for the +//! host and kept from the client. + +use bytes::{Bytes, BytesMut}; +use futures_util::{StreamExt, stream::BoxStream}; +use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use litellm_framer::sse::{SseBlock, SseBlockCodec}; +use litellm_llms::{ + ErrorDetail, + base_llm::{ + auth::{AuthScheme, ValidatedEnvironment, resolve_auth}, + sse::DONE_FRAME, + }, + cerebras::chat::transformation::CEREBRAS_CHAT_CONFIG, + deepseek::chat::transformation::DEEPSEEK_CHAT_CONFIG, + fireworks_ai::chat::transformation::FIREWORKS_AI_CHAT_CONFIG, + groq::chat::transformation::GROQ_CHAT_CONFIG, + hosted_vllm::chat::transformation::HOSTED_VLLM_CHAT_CONFIG, + mistral::chat::transformation::MISTRAL_CHAT_CONFIG, + nvidia_nim::chat::transformation::NVIDIA_NIM_CHAT_CONFIG, + openai::chat::compatible::{ + CompatibleChatBody, CompatibleContext, OpenAiCompatibleChatConfig, ResponseShape, + fix_response, transform_request, + }, + openrouter::chat::transformation::OPENROUTER_CHAT_CONFIG, + perplexity::chat::transformation::PERPLEXITY_CHAT_CONFIG, + together_ai::chat::transformation::TOGETHER_AI_CHAT_CONFIG, + xai::chat::transformation::XAI_CHAT_CONFIG, +}; +use litellm_llms_types::formats::chat_completions::{ + ChatCompletionsUsage, CompletionTokensDetails, PromptTokensDetails, +}; +use serde_json::Value; +use tokio_util::codec::Decoder; + +use crate::{ + ChatCompletionsRoute, Error, + common_utils::string_headers, + openai::{OpenAiChatCall, PreparedCall}, + stream::ReportedUsage, +}; + +const ENV_REFERENCE: &str = "os.environ/"; + +/// The config of a provider served through the OpenAI passthrough with its own transforms. +pub fn compatible_chat_config( + provider: LlmProviders, +) -> Option<&'static dyn OpenAiCompatibleChatConfig> { + Some(match provider { + LlmProviders::Cerebras => &CEREBRAS_CHAT_CONFIG, + LlmProviders::Deepseek => &DEEPSEEK_CHAT_CONFIG, + LlmProviders::FireworksAi => &FIREWORKS_AI_CHAT_CONFIG, + LlmProviders::Groq => &GROQ_CHAT_CONFIG, + LlmProviders::HostedVllm => &HOSTED_VLLM_CHAT_CONFIG, + LlmProviders::Mistral => &MISTRAL_CHAT_CONFIG, + LlmProviders::NvidiaNim => &NVIDIA_NIM_CHAT_CONFIG, + LlmProviders::Openrouter => &OPENROUTER_CHAT_CONFIG, + LlmProviders::Perplexity => &PERPLEXITY_CHAT_CONFIG, + LlmProviders::TogetherAi => &TOGETHER_AI_CHAT_CONFIG, + LlmProviders::Xai => &XAI_CHAT_CONFIG, + _ => return None, + }) +} + +/// What the response of a call to an OpenAI-compatible provider goes through on its way back. +#[derive(Clone)] +pub(crate) struct ResponseAdapter { + config: &'static dyn OpenAiCompatibleChatConfig, + shape: ResponseShape, +} + +impl ChatCompletionsRoute { + pub(crate) async fn prepare_openai_compatible( + &self, + config: &'static dyn OpenAiCompatibleChatConfig, + model: String, + call: OpenAiChatCall, + ) -> Result { + let OpenAiChatCall { + body, + api_key, + api_base, + extra_headers, + drop_params, + model_info, + .. + } = call; + let references: Vec<&str> = [api_key.as_deref(), api_base.as_deref()] + .into_iter() + .flatten() + .filter_map(|value| value.strip_prefix(ENV_REFERENCE)) + .collect(); + let names: Vec<&str> = config + .secret_names() + .into_iter() + .chain(references) + .collect(); + let secrets = self.secrets.resolve(&names).await?; + let lookup = |name: &str| secrets.get(name); + // Python's router reads every `os.environ/` setting from the environment when it + // loads the deployment. A reference to an unset variable counts as unset. + let resolve = |value: &str| match value.strip_prefix(ENV_REFERENCE) { + Some(name) => lookup(name), + None => Some(value.to_owned()), + }; + let api_key = api_key.as_deref().and_then(resolve); + let api_base = api_base.as_deref().and_then(resolve); + let url = config.complete_url(&config.api_base(api_base.as_deref(), &lookup)?); + let key = config.api_key(api_key.as_deref(), &lookup)?; + let provider_headers = config.headers(&body, &lookup); + let context = CompatibleContext { + model: &model, + drop_params, + model_info: model_info.as_deref(), + }; + let CompatibleChatBody { body, shape } = transform_request(config, &context, body)?; + let defaults: Vec<(&str, &str)> = provider_headers + .iter() + .map(|(name, value)| (name.as_str(), value.as_str())) + .chain([("content-type", "application/json")]) + .collect(); + let environment = ValidatedEnvironment { + headers: litellm_http::request::with_default_headers( + string_headers(extra_headers)?, + &defaults, + ), + auth: AuthScheme::Credential { + placement: CredentialPlacement::Bearer, + secret: SecretValue::new(key), + }, + }; + let authenticated = resolve_auth(&self.auth, environment, &lookup).await?; + Ok(PreparedCall { + model, + provider: config.name(), + url, + headers: authenticated.headers, + body, + api_key, + adapter: Some(ResponseAdapter { config, shape }), + }) + } +} + +impl ResponseAdapter { + /// A complete response, re-encoded only when a fixup changed it. + pub(crate) fn complete(&self, body: Bytes) -> Bytes { + let Ok(Value::Object(mut response)) = serde_json::from_slice::(&body) else { + return body; + }; + match fix_response(self.config, &mut response, &self.shape) { + true => serde_json::to_vec(&response).map_or(body, Bytes::from), + false => body, + } + } + + pub(crate) fn stream( + self, + upstream: BoxStream<'static, Result>, + usage: ReportedUsage, + ) -> BoxStream<'static, Result> { + let relay = Relay { + upstream, + buffer: BytesMut::new(), + codec: SseBlockCodec::default(), + adapter: self, + usage, + ended: false, + }; + futures_util::stream::try_unfold(relay, |mut relay| async move { + relay + .next_frame() + .await + .map(|frame| frame.map(|frame| (frame, relay))) + }) + .boxed() + } + + /// One event's frame for the client, or `None` when the client is not to see it. + fn frame(&self, block: SseBlock, usage: &ReportedUsage) -> Option { + if block.event.data.trim() == "[DONE]" { + return Some(Bytes::from_static(DONE_FRAME)); + } + let Ok(Value::Object(mut chunk)) = serde_json::from_str::(&block.event.data) else { + return Some(block.bytes); + }; + let mut changed = self.config.fix_chunk(&mut chunk, &self.shape); + if let Some(reported) = chunk.get("usage").and_then(usage_from_json) { + usage.set(reported); + } + if !self.shape.client_usage { + if chunk.get("usage").is_some_and(|usage| !usage.is_null()) { + chunk.remove("usage"); + changed = true; + } + if chunk + .get("choices") + .and_then(Value::as_array) + .is_some_and(Vec::is_empty) + { + return None; + } + } + if !changed { + return Some(block.bytes); + } + let mut frame = b"data: ".to_vec(); + serde_json::to_writer(&mut frame, &chunk).ok()?; + frame.extend_from_slice(b"\n\n"); + Some(Bytes::from(frame)) + } +} + +struct Relay { + upstream: BoxStream<'static, Result>, + buffer: BytesMut, + codec: SseBlockCodec, + adapter: ResponseAdapter, + usage: ReportedUsage, + ended: bool, +} + +impl Relay { + async fn next_frame(&mut self) -> Result, Error> { + loop { + let block = match self.ended { + false => self.codec.decode(&mut self.buffer), + true => self.codec.decode_eof(&mut self.buffer), + } + .map_err(|error| { + Error::InvalidResponse(ErrorDetail::from(format!( + "the provider sent a malformed stream: {error}" + ))) + })?; + match block { + Some(block) => { + if let Some(frame) = self.adapter.frame(block, &self.usage) { + return Ok(Some(frame)); + } + } + None if self.ended => return Ok(None), + None => match self.upstream.next().await { + Some(bytes) => self.buffer.extend_from_slice(&bytes?), + None => self.ended = true, + }, + } + } + } +} + +/// The usage a chunk or response reports, read as leniently as Python's `Usage` reads it. +fn usage_from_json(usage: &Value) -> Option { + let usage = usage.as_object()?; + let count = |value: Option<&Value>| value.and_then(Value::as_u64); + let prompt = usage.get("prompt_tokens_details"); + let completion = usage.get("completion_tokens_details"); + Some(ChatCompletionsUsage { + prompt_tokens: count(usage.get("prompt_tokens")).unwrap_or(0), + completion_tokens: count(usage.get("completion_tokens")).unwrap_or(0), + total_tokens: count(usage.get("total_tokens")).unwrap_or(0), + prompt_tokens_details: PromptTokensDetails { + cached_tokens: count(prompt.and_then(|d| d.get("cached_tokens"))).unwrap_or(0), + cache_creation_tokens: count(prompt.and_then(|d| d.get("cache_creation_tokens"))) + .unwrap_or(0), + text_tokens: count(prompt.and_then(|d| d.get("text_tokens"))).unwrap_or(0), + cache_write_tokens: None, + cache_creation_token_details: prompt + .and_then(|d| d.get("cache_creation_token_details")) + .filter(|details| details.is_object()) + .cloned(), + }, + completion_tokens_details: completion.filter(|details| details.is_object()).map( + |details| CompletionTokensDetails { + reasoning_tokens: count(details.get("reasoning_tokens")), + text_tokens: count(details.get("text_tokens")), + }, + ), + server_tool_use: usage + .get("server_tool_use") + .filter(|tools| tools.is_object()) + .cloned(), + cache_creation_input_tokens: None, + cache_read_input_tokens: None, + }) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + + fn adapter( + config: &'static dyn OpenAiCompatibleChatConfig, + client_usage: bool, + ) -> ResponseAdapter { + ResponseAdapter { + config, + shape: ResponseShape { + stream: true, + client_usage, + ..ResponseShape::default() + }, + } + } + + async fn relay(adapter: ResponseAdapter, parts: &[&'static str]) -> (String, ReportedUsage) { + let usage = ReportedUsage::default(); + let upstream = futures_util::stream::iter( + parts + .iter() + .map(|part| Ok::<_, Error>(Bytes::from_static(part.as_bytes()))) + .collect::>(), + ) + .boxed(); + let frames: Vec = adapter + .stream(upstream, usage.clone()) + .map(Result::unwrap) + .collect() + .await; + (String::from_utf8(frames.concat()).unwrap(), usage) + } + + const CONTENT: &str = + "data: {\"id\":\"c\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"hi\"}}]}\n\n"; + const USAGE: &str = "data: {\"id\":\"c\",\"choices\":[],\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":2,\"total_tokens\":5}}\n\n"; + const DONE: &str = "data: [DONE]\n\n"; + + #[rstest] + #[case::kept_from_the_client(false, format!("{CONTENT}{DONE}"))] + #[case::shown_when_asked(true, format!("{CONTENT}{USAGE}{DONE}"))] + #[tokio::test] + async fn usage_is_recorded_and_shown_only_when_asked( + #[case] client_usage: bool, + #[case] expected: String, + ) { + let (sent, usage) = relay( + adapter(&GROQ_CHAT_CONFIG, client_usage), + &[ + "data: {\"id\":\"c\",\"choices\":[{\"index\":0,", + "\"delta\":{\"content\":\"hi\"}}]}\n\n", + USAGE, + DONE, + ], + ) + .await; + assert_eq!(sent, expected); + let usage = usage.get().unwrap(); + assert_eq!( + ( + usage.prompt_tokens, + usage.completion_tokens, + usage.total_tokens + ), + (3, 2, 5) + ); + } + + #[rstest] + #[tokio::test] + async fn chunk_fixups_reencode_only_the_chunks_they_change() { + let (sent, _) = relay( + adapter(&HOSTED_VLLM_CHAT_CONFIG, true), + &[ + CONTENT, + "data: {\"choices\":[{\"index\":0,\"delta\":{\"reasoning\":\"r\"}}]}\n\n", + DONE, + ], + ) + .await; + assert_eq!( + sent, + format!( + "{CONTENT}data: {{\"choices\":[{{\"index\":0,\"delta\":{{\"reasoning_content\":\"r\"}}}}]}}\n\n{DONE}" + ) + ); + } + + #[rstest] + #[tokio::test] + async fn a_usage_chunk_with_choices_loses_only_its_usage() { + let (sent, usage) = relay( + adapter(&MISTRAL_CHAT_CONFIG, false), + &["data: {\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":1,\"completion_tokens\":1,\"total_tokens\":2}}\n\n"], + ) + .await; + assert_eq!( + sent, + "data: {\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n" + ); + assert_eq!(usage.get().unwrap().total_tokens, 2); + } + + #[rstest] + fn a_complete_response_is_reencoded_only_when_fixed() { + let raw = Bytes::from_static( + br#"{"choices": [{"message": {"role": "assistant", "content": "x"}}]}"#, + ); + assert_eq!( + adapter(&CEREBRAS_CHAT_CONFIG, false).complete(raw.clone()), + raw + ); + let fixed = adapter(&HOSTED_VLLM_CHAT_CONFIG, false).complete(Bytes::from_static( + br#"{"choices":[{"message":{"role":"assistant","content":"x","reasoning":"r"}}]}"#, + )); + assert_eq!( + serde_json::from_slice::(&fixed).unwrap(), + json!({"choices": [{"message": {"role": "assistant", "content": "x", "reasoning_content": "r"}}]}) + ); + } +} diff --git a/litellm-rust/crates/inference-chat/src/prepare.rs b/litellm-rust/crates/inference-chat/src/prepare.rs index 058ca245e79..4e1382eef28 100644 --- a/litellm-rust/crates/inference-chat/src/prepare.rs +++ b/litellm-rust/crates/inference-chat/src/prepare.rs @@ -1,7 +1,10 @@ use litellm_auth::SecretValue; use litellm_core_utils::settings::Lookup; use litellm_http::request::with_default_headers; -use litellm_llms::base_llm::{auth::ValidatedEnvironment, chat::transformation::BaseConfig}; +use litellm_llms::base_llm::{ + auth::{Headers, ValidatedEnvironment}, + chat::{params::ChatRequestContext, transformation::BaseConfig}, +}; use litellm_llms_types::formats::chat_completions::ChatMessage; use litellm_secrets::source::Secrets; use serde_json::Value; @@ -26,7 +29,7 @@ pub(super) fn resolve_provider_config<'a>( custom_llm_provider: Option<&'a str>, ) -> Result { let provider_info = resolve_llm_provider(model, custom_llm_provider, "chat completions")?; - let config = chat_completions_provider(provider_info.provider) + let config = chat_completions_provider(provider_info.provider, provider_info.model) .ok_or_else(|| Error::InvalidProvider(<&str>::from(provider_info.provider).to_string()))? .config(); Ok(ResolvedProvider { @@ -72,9 +75,38 @@ pub(super) fn resolve_request( api_base: request.api_base, extra_headers: request.extra_headers, timeout: request.timeout, + drop_params: request.drop_params, + model_info: request.model_info, }) } +/// The headers a transformation asked for, added to the ones already going out. A header +/// already there (a forwarded `anthropic-beta`, say) keeps its values and gains the new ones, +/// comma separated, the way Python unions beta headers. +fn merge_headers(headers: &mut Headers, added: Headers) { + for (name, value) in added { + match headers + .iter_mut() + .find(|(existing, _)| existing.eq_ignore_ascii_case(&name)) + { + Some((_, existing)) => { + let mut values: Vec<&str> = existing + .split(',') + .map(str::trim) + .filter(|value| !value.is_empty()) + .collect(); + for value in value.split(',').map(str::trim) { + if !value.is_empty() && !values.contains(&value) { + values.push(value); + } + } + *existing = values.join(", "); + } + None => headers.push((name, value)), + } + } +} + fn validate_environment( request: &ResolvedChatCompletionsRequest<'_>, model: &str, @@ -101,7 +133,7 @@ pub(super) fn prepare_provider_request( request: ResolvedChatCompletionsRequest<'_>, secrets: Secrets, ) -> Result { - let environment = + let mut environment = validate_environment(&request, &request.model, request.config, secrets.as_ref())?; let model = request.model; let config = request.config; @@ -112,8 +144,17 @@ pub(super) fn prepare_provider_request( &request.optional_params, &env_lookup, )?; - let transformed = - config.transform_request(&model, request.messages, request.optional_params.clone())?; + let context = ChatRequestContext { + drop_params: request.drop_params, + model_info: request.model_info.as_deref(), + }; + let transformed = config.transform_request( + &model, + request.messages, + request.optional_params.clone(), + &context, + )?; + merge_headers(&mut environment.headers, transformed.headers); Ok(ProviderChatCompletionsRequest { model, @@ -136,7 +177,7 @@ mod tests { use litellm_llms::base_llm::auth::{AuthScheme, resolve_auth}; use serde_json::{Map, Value, json}; - use super::{prepare_provider_request, resolve_request}; + use super::{merge_headers, prepare_provider_request, resolve_request}; use crate::{ Error, types::{ChatCompletionsRequest, ProviderChatCompletionsRequest}, @@ -183,6 +224,8 @@ mod tests { custom_llm_provider: provider, extra_headers: None, timeout: None, + drop_params: Some(false), + model_info: None, } } @@ -636,4 +679,27 @@ mod tests { "prepare did not carry the bearer token" ); } + + #[test] + fn transformation_headers_union_with_the_forwarded_ones() { + let mut headers = vec![ + ("Anthropic-Beta".to_owned(), "a,b".to_owned()), + ("x-other".to_owned(), "1".to_owned()), + ]; + merge_headers( + &mut headers, + vec![ + ("anthropic-beta".to_owned(), "b, c".to_owned()), + ("x-new".to_owned(), "2".to_owned()), + ], + ); + assert_eq!( + headers, + [ + ("Anthropic-Beta".to_owned(), "a, b, c".to_owned()), + ("x-other".to_owned(), "1".to_owned()), + ("x-new".to_owned(), "2".to_owned()), + ] + ); + } } diff --git a/litellm-rust/crates/inference-chat/src/route.rs b/litellm-rust/crates/inference-chat/src/route.rs index 50fc412a705..73eb69ddc0c 100644 --- a/litellm-rust/crates/inference-chat/src/route.rs +++ b/litellm-rust/crates/inference-chat/src/route.rs @@ -69,6 +69,8 @@ impl ChatCompletionsRoute { custom_llm_provider: call.custom_llm_provider.as_deref(), extra_headers: call.extra_headers, timeout: call.timeout, + drop_params: call.drop_params, + model_info: call.model_info, }; self.run(request, cache_options, interceptors, observers) .await diff --git a/litellm-rust/crates/inference-chat/src/stream.rs b/litellm-rust/crates/inference-chat/src/stream.rs index 33dd1d8a441..9c71ac516af 100644 --- a/litellm-rust/crates/inference-chat/src/stream.rs +++ b/litellm-rust/crates/inference-chat/src/stream.rs @@ -71,7 +71,7 @@ impl ReportedUsage { self.0.lock().unwrap_or_else(|e| e.into_inner()).clone() } - fn set(&self, usage: ChatCompletionsUsage) { + pub(crate) fn set(&self, usage: ChatCompletionsUsage) { *self.0.lock().unwrap_or_else(|e| e.into_inner()) = Some(usage); } } @@ -140,7 +140,7 @@ impl ChatCompletionsRoute { let timeout = call .timeout .unwrap_or(Duration::from_secs(CHAT_COMPLETIONS_TIMEOUT_SECS)); - let resolved = resolve_request(ChatCompletionsRequest { + let mut resolved = resolve_request(ChatCompletionsRequest { model: &call.model, messages: call.messages, optional_params, @@ -149,7 +149,10 @@ impl ChatCompletionsRoute { custom_llm_provider: call.custom_llm_provider.as_deref(), extra_headers: call.extra_headers, timeout: call.timeout, + drop_params: call.drop_params, + model_info: call.model_info, })?; + self.inline_remote_media(&mut resolved).await?; let snapshot = self .secrets .resolve(&resolved.config.secret_names()) @@ -1019,6 +1022,8 @@ mod tests { custom_llm_provider: None, extra_headers: None, timeout: Some(Duration::from_secs(5)), + drop_params: Some(false), + model_info: None, }, response_model: "public/model".into(), usage: ReportedUsage::default(), diff --git a/litellm-rust/crates/inference-chat/src/types.rs b/litellm-rust/crates/inference-chat/src/types.rs index d0cb7b24d6c..e02b67e5d2b 100644 --- a/litellm-rust/crates/inference-chat/src/types.rs +++ b/litellm-rust/crates/inference-chat/src/types.rs @@ -1,20 +1,20 @@ use litellm_secrets::source::Secrets; -use std::time::Duration; +use std::{sync::Arc, time::Duration}; use litellm_auth::SecretValue; use litellm_llms::base_llm::{ auth::ValidatedEnvironment, chat::{streaming::StreamShape, transformation::BaseConfig}, + model_info::ModelInfo, }; use litellm_llms_types::formats::chat_completions::ChatMessage; use serde_json::{Map, Value}; /// A `/chat/completions` call as it crosses into the core. /// -/// `optional_params` arrives already mapped to the provider's own parameter -/// names by the host, exactly as the messages route receives an already -/// Anthropic-shaped body. The core owns the conversation translation, the -/// provider call, and the response normalization. +/// `optional_params` are the OpenAI request's own params; the provider config maps them, +/// refusing what the provider cannot take unless `drop_params` says to drop it. The core +/// owns the conversation translation, the provider call, and the response normalization. pub struct ChatCompletionsRequest<'a> { pub model: &'a str, pub messages: Value, @@ -24,6 +24,11 @@ pub struct ChatCompletionsRequest<'a> { pub custom_llm_provider: Option<&'a str>, pub extra_headers: Option>, pub timeout: Option, + /// `drop_params` from the deployment or the proxy settings, `None` when neither says. A + /// request's own `drop_params` rides in `optional_params`. + pub drop_params: Option, + /// The model cost map, for the param rules that read model capabilities. + pub model_info: Option>, } pub struct ChatCompletionsCall { @@ -35,6 +40,11 @@ pub struct ChatCompletionsCall { pub custom_llm_provider: Option, pub extra_headers: Option>, pub timeout: Option, + /// `drop_params` from the deployment or the proxy settings, `None` when neither says. A + /// request's own `drop_params` rides in `optional_params`. + pub drop_params: Option, + /// The model cost map, for the param rules that read model capabilities. + pub model_info: Option>, } impl From> for ChatCompletionsCall { @@ -48,6 +58,8 @@ impl From> for ChatCompletionsCall { custom_llm_provider: request.custom_llm_provider.map(str::to_owned), extra_headers: request.extra_headers, timeout: request.timeout, + drop_params: request.drop_params, + model_info: request.model_info, } } } @@ -62,6 +74,11 @@ pub struct ResolvedChatCompletionsRequest<'a> { pub api_base: Option<&'a str>, pub extra_headers: Option>, pub timeout: Option, + /// `drop_params` from the deployment or the proxy settings, `None` when neither says. A + /// request's own `drop_params` rides in `optional_params`. + pub drop_params: Option, + /// The model cost map, for the param rules that read model capabilities. + pub model_info: Option>, } pub struct ProviderChatCompletionsRequest { diff --git a/litellm-rust/crates/inference-chat/tests/caching.rs b/litellm-rust/crates/inference-chat/tests/caching.rs index 3fc432f8b46..69b6ed917be 100644 --- a/litellm-rust/crates/inference-chat/tests/caching.rs +++ b/litellm-rust/crates/inference-chat/tests/caching.rs @@ -84,6 +84,8 @@ async fn the_same_route_entrypoint_reports_facts_with_or_without_caching( custom_llm_provider: None, extra_headers: None, timeout: None, + drop_params: Some(false), + model_info: None, }, &(), Some(observer.clone()), @@ -254,6 +256,8 @@ async fn chat_cache_identity_follows_resolved_configuration_and_request_callback custom_llm_provider: None, extra_headers: None, timeout: None, + drop_params: Some(false), + model_info: None, }, &hooks, None, @@ -319,6 +323,8 @@ async fn signed_requests_bypass_response_caching(cache: Arc ChatCompletionsRequest<'static> { custom_llm_provider: None, extra_headers: None, timeout: Some(Duration::from_secs(10)), + drop_params: Some(false), + model_info: None, } } @@ -163,7 +165,7 @@ async fn bedrock_round_trip_is_signed_and_normalized(request: ChatCompletionsReq #[case::missing_usage( r#"{"model":"m","content":[{"type":"text","text":"hi"}],"stop_reason":"end_turn"}"# )] -#[case::tool_use_block(r#"{"model":"m","content":[{"type":"tool_use","id":"t","name":"f","input":{}}],"stop_reason":"tool_use","usage":{"input_tokens":1,"output_tokens":1}}"#)] +#[case::malformed_token_count(r#"{"model":"m","content":[{"type":"text","text":"hi"}],"stop_reason":"end_turn","usage":{"input_tokens":"1","output_tokens":1}}"#)] #[case::not_json("not json")] #[tokio::test] async fn a_response_it_cannot_normalize_is_reported_as_already_sent( @@ -294,6 +296,8 @@ async fn direct_and_hosted_calls_share_hooks_and_lifecycle( custom_llm_provider: call.custom_llm_provider.as_deref(), extra_headers: call.extra_headers, timeout: call.timeout, + drop_params: Some(false), + model_info: None, }, &host, Some(host.events.0.sender.clone()), diff --git a/litellm-rust/crates/llms-types/src/formats/chat_completions/response.rs b/litellm-rust/crates/llms-types/src/formats/chat_completions/response.rs index 44d66d5ebe6..730ae7c6129 100644 --- a/litellm-rust/crates/llms-types/src/formats/chat_completions/response.rs +++ b/litellm-rust/crates/llms-types/src/formats/chat_completions/response.rs @@ -17,6 +17,9 @@ pub struct ChatCompletionToolCallChunk { pub tool_type: String, pub function: ChatCompletionToolCallFunctionChunk, pub index: i64, + /// Provider data that rides on the call, such as Gemini's `thought_signature`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub provider_specific_fields: Option>, } #[macro_rules_attribute::apply(crate::wire_type)] @@ -87,6 +90,7 @@ pub struct ChatCompletionsUsage { } #[macro_rules_attribute::apply(crate::wire_type)] +#[derive(Default)] pub struct ChatCompletionsChoiceMessage { pub role: String, // Whether an empty turn is `None` or `""` is the provider's choice, not a @@ -94,6 +98,17 @@ pub struct ChatCompletionsChoiceMessage { // while Converse assigns the joined string unconditionally. Each config // mirrors its own, so keep this optional and serialize it even when None. pub content: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tool_calls: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub reasoning_content: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub thinking_blocks: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub provider_specific_fields: Option>, + /// OpenAI `url_citation` annotations, which Gemini's grounding sources become. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub annotations: Option>, } #[macro_rules_attribute::apply(crate::wire_type)] @@ -101,6 +116,8 @@ pub struct ChatCompletionsChoice { pub index: u64, pub message: ChatCompletionsChoiceMessage, pub finish_reason: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub logprobs: Option, } /// The normalized response handed back to the host. @@ -114,4 +131,8 @@ pub struct ChatCompletionsResponse { pub model: String, pub choices: Vec, pub usage: ChatCompletionsUsage, + /// Top-level fields a provider adds beside the OpenAI ones, such as Vertex AI's + /// `vertex_ai_grounding_metadata`. + #[serde(flatten)] + pub extra: Map, } diff --git a/litellm-rust/crates/llms-types/src/providers/vertex_ai/AGENTS.md b/litellm-rust/crates/llms-types/src/providers/vertex_ai/AGENTS.md index 03b04d87af1..fc335a8c7b7 100644 --- a/litellm-rust/crates/llms-types/src/providers/vertex_ai/AGENTS.md +++ b/litellm-rust/crates/llms-types/src/providers/vertex_ai/AGENTS.md @@ -1,6 +1,7 @@ # rules - Shared across Vertex AI's Messages and OCR adapters: the global host, the global and default locations, the `rawPredict` methods and the Anthropic API version Vertex expects in the body +- Shared by the Gemini chat adapters on Vertex AI and Google AI Studio: the `generateContent` methods, the Gemini API host, its Files API prefix and its key header - Location validation and host selection are in `llms/src/vertex_ai/common_utils.rs` # references diff --git a/litellm-rust/crates/llms-types/src/providers/vertex_ai/constants.rs b/litellm-rust/crates/llms-types/src/providers/vertex_ai/constants.rs index fccf7cdefe8..2ce78fe29cb 100644 --- a/litellm-rust/crates/llms-types/src/providers/vertex_ai/constants.rs +++ b/litellm-rust/crates/llms-types/src/providers/vertex_ai/constants.rs @@ -5,3 +5,13 @@ pub const DEFAULT_LOCATION: &str = "us-central1"; pub const RAW_PREDICT: &str = "rawPredict"; pub const STREAM_RAW_PREDICT: &str = "streamRawPredict"; pub const ANTHROPIC_VERSION: &str = "vertex-2023-10-16"; + +pub const GENERATE_CONTENT: &str = "generateContent"; +pub const STREAM_GENERATE_CONTENT: &str = "streamGenerateContent"; +/// The Gemini API host Google AI Studio keys address. +pub const GEMINI_API_BASE: &str = "https://generativelanguage.googleapis.com"; +/// Files uploaded through the Gemini Files API, passed by reference and never fetched. +pub const GEMINI_FILES_API_URI_PREFIX: &str = + "https://generativelanguage.googleapis.com/v1beta/files/"; +/// The header a Gemini API key travels in, never the `key=` query parameter. +pub const GEMINI_API_KEY_HEADER: &str = "x-goog-api-key"; diff --git a/litellm-rust/crates/llms/Cargo.toml b/litellm-rust/crates/llms/Cargo.toml index 2d3ea85d1fc..629b6ee75e1 100644 --- a/litellm-rust/crates/llms/Cargo.toml +++ b/litellm-rust/crates/llms/Cargo.toml @@ -22,6 +22,8 @@ litellm-secrets.workspace = true litellm-python-compat.workspace = true litellm-router-types.workspace = true base64.workspace = true +sha2.workspace = true +uuid.workspace = true bytes.workspace = true data-url = "0.3.2" futures-util.workspace = true diff --git a/litellm-rust/crates/llms/src/anthropic/chat/AGENTS.md b/litellm-rust/crates/llms/src/anthropic/chat/AGENTS.md index dab8abcc5ad..1b0af4c5c8c 100644 --- a/litellm-rust/crates/llms/src/anthropic/chat/AGENTS.md +++ b/litellm-rust/crates/llms/src/anthropic/chat/AGENTS.md @@ -2,3 +2,4 @@ - Chat Completions translated onto Anthropic Messages. Both format specs live under `llms-types/src/formats/` - `top_k` is not forwarded, because Python gates it per model inside `transform_request` +- `build_messages_request` serves every host of the translation; `AnthropicHost` picks the host's cost-map reads, `service_tier`/`speed` support and whether the Anthropic API betas are added (Vertex AI sets its own in `vertex_ai/vertex_ai_partner_models/anthropic`) diff --git a/litellm-rust/crates/llms/src/anthropic/chat/handler.rs b/litellm-rust/crates/llms/src/anthropic/chat/handler.rs index a373e22873a..594e2798c0a 100644 --- a/litellm-rust/crates/llms/src/anthropic/chat/handler.rs +++ b/litellm-rust/crates/llms/src/anthropic/chat/handler.rs @@ -4,8 +4,9 @@ //! Each event becomes one provider chunk, empty ones included; the host's stream wrapper //! (Python's `CustomStreamWrapper`) decides which of them the client sees. //! -//! DIVERGENCE: not built here, since the chat route never asks for what produces them: -//! `json_mode` / `response_format` tool conversion, `web_search_calls` and +//! DIVERGENCE: a `response_format` tool's answer is held until its block ends and sent as +//! one piece of content, where Python streams it as it decides the envelope; an answer +//! spread over several blocks is sent per block. Not built: `web_search_calls` and //! `code_interpreter_results` summaries, and a tool call's `caller`. The reasoning token //! estimate Python makes from streamed thinking text (with its tokenizer) is not made, so //! such a stream reports `completion_tokens_details` without counts unless Anthropic sends @@ -28,7 +29,14 @@ use serde_json::{Map, Value, json}; use crate::{ Error, - base_llm::{base_model_iterator::StreamTransformer, chat::streaming::StreamShape}, + base_llm::{ + base_model_iterator::StreamTransformer, + chat::{ + params::{RESPONSE_FORMAT_TOOL_NAME, python_dumps}, + response_format::{envelope_contents, json_tool_answer}, + streaming::StreamShape, + }, + }, }; /// Anthropic sends no status with an `error` event; Python raises it as a 500. @@ -51,6 +59,10 @@ pub struct ModelResponseIterator { web_search_results: Vec, tool_results: Vec, compaction_blocks: Vec, + /// Whether the tool block in progress is the `response_format` tool. + is_response_format_tool: bool, + converted_response_format_tool: bool, + response_format_buffer: String, } /// What one event contributes to its chunk. @@ -79,9 +91,58 @@ impl ModelResponseIterator { web_search_results: Vec::new(), tool_results: Vec::new(), compaction_blocks: Vec::new(), + is_response_format_tool: false, + converted_response_format_tool: false, + response_format_buffer: String::new(), } } + /// `_handle_json_mode_chunk`: the `response_format` tool's arguments become content. + fn handle_json_mode(&mut self, parsed: &mut Parsed) { + if !self.shape.json_mode { + return; + } + let Some(tool_use) = parsed.tool_use.take() else { + return; + }; + if tool_use.id.is_some() { + self.is_response_format_tool = + tool_use.function.name.as_deref() == Some(RESPONSE_FORMAT_TOOL_NAME); + if self.is_response_format_tool { + self.tool_index -= 1; + } + } + if !self.is_response_format_tool { + parsed.tool_use = Some(tool_use); + return; + } + self.converted_response_format_tool = true; + if self.shape.response_format_schema.is_some() { + match tool_use.id { + Some(_) => self.response_format_buffer.clear(), + None => self + .response_format_buffer + .push_str(&tool_use.function.arguments), + } + parsed.text = String::new(); + return; + } + parsed.text = json_tool_answer(&[tool_use.function.arguments], None); + } + + /// `_held_answer`: the held arguments, or what their envelope wraps. + fn flush_response_format_buffer(&mut self) -> String { + let held = std::mem::take(&mut self.response_format_buffer); + let Some(schema) = &self.shape.response_format_schema else { + return String::new(); + }; + serde_json::from_str::(&held) + .ok() + .and_then(|payload| envelope_contents(&payload, schema, RESPONSE_FORMAT_TOOL_NAME)) + .map(|contents| python_dumps(&contents)) + .unwrap_or(held) + } + /// Whether the tool block so far streamed only empty argument fragments. pub fn check_empty_tool_call_args(&self) -> bool { match self.content_blocks.first() { @@ -125,6 +186,7 @@ impl ModelResponseIterator { provider_specific_fields: None, }, index: self.tool_index, + provider_specific_fields: None, } } @@ -273,6 +335,10 @@ impl ModelResponseIterator { if in_tool && self.check_empty_tool_call_args() { parsed.tool_use = Some(self.tool_call(None, None, "{}".into())); } + if self.is_response_format_tool { + parsed.text = self.flush_response_format_buffer(); + } + self.is_response_format_tool = false; self.current_content_block_type = None; parsed } @@ -282,10 +348,16 @@ impl ModelResponseIterator { delta: MessagesDelta, usage: Option, ) -> Parsed { + let provider_finish_reason = + finish_reason_for(delta.stop_reason.as_deref().unwrap_or("stop")).to_string(); + let called_only_the_response_format_tool = provider_finish_reason == "tool_calls" + && self.converted_response_format_tool + && self.tool_index < 0; let mut parsed = Parsed { - finish_reason: Some( - finish_reason_for(delta.stop_reason.as_deref().unwrap_or("stop")).to_string(), - ), + finish_reason: Some(match called_only_the_response_format_tool { + true => "stop".to_owned(), + false => provider_finish_reason, + }), usage: usage.map(|usage| self.handle_usage(&usage)), ..Parsed::default() }; @@ -314,7 +386,7 @@ impl ModelResponseIterator { &mut self, event: MessagesStreamEvent, ) -> Result { - let parsed = match event { + let mut parsed = match event { MessagesStreamEvent::ContentBlockDelta { delta, .. } => self.content_block_delta(delta), MessagesStreamEvent::ContentBlockStart { content_block, .. } => { self.content_block_start(content_block) @@ -341,6 +413,7 @@ impl ModelResponseIterator { }); } }; + self.handle_json_mode(&mut parsed); Ok(self.chunk(parsed)) } diff --git a/litellm-rust/crates/llms/src/anthropic/chat/messages.rs b/litellm-rust/crates/llms/src/anthropic/chat/messages.rs new file mode 100644 index 00000000000..8f4559f1e73 --- /dev/null +++ b/litellm-rust/crates/llms/src/anthropic/chat/messages.rs @@ -0,0 +1,777 @@ +//! Chat messages as the Messages API takes them: Python's `translate_system_message`, +//! `place_mid_conversation_system` and `anthropic_messages_pt`. + +use std::collections::HashSet; + +use base64::Engine; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, ErrorDetail, + anthropic::common_utils::ENCRYPTED_REASONING_SIGNATURE_PREFIX, + base_llm::chat::{ + media::{data_url, image_url}, + params::{parse_tool_arguments, truthy}, + }, +}; + +pub const EMPTY_TEXT_PLACEHOLDER: &str = + "[System: Empty message content sanitised to satisfy protocol]"; +pub const CONVERTED_SYSTEM_NOTE: &str = "Operator note (not from the user): the following was originally a mid-conversation system-role reminder."; +const TOOL_USE_ID_FALLBACK: &str = "tool_use_id"; + +fn bad_request(message: impl Into) -> Error { + Error::InvalidRequest(ErrorDetail::Message(message.into())) +} + +pub fn role(message: &Value) -> &str { + match message.get("role").and_then(Value::as_str) { + Some("developer") => "system", + Some(role) => role, + None => "", + } +} + +/// Python's truthiness of a message's `content`. +fn content_is_empty(message: &Value) -> bool { + match message.get("content") { + None | Some(Value::Null) => true, + Some(Value::String(text)) => text.is_empty(), + Some(Value::Array(items)) => items.is_empty(), + Some(_) => false, + } +} + +/// `[^a-zA-Z0-9_-]` -> `_`, an empty id becoming `tool_use_id`. +pub fn sanitize_tool_use_id(id: &str) -> String { + let sanitized: String = id + .chars() + .map( + |c| match c.is_ascii_alphanumeric() || c == '_' || c == '-' { + true => c, + false => '_', + }, + ) + .collect(); + match sanitized.is_empty() { + true => TOOL_USE_ID_FALLBACK.to_owned(), + false => sanitized, + } +} + +/// `add_cache_control_to_content`: a dict `cache_control` on the source is copied over. +fn with_cache_control(mut block: Value, source: &Value) -> Value { + if let (Some(cache_control @ Value::Object(_)), Value::Object(fields)) = + (source.get("cache_control"), &mut block) + { + fields.insert("cache_control".into(), cache_control.clone()); + } + block +} + +/// `_cache_control` of the mid-conversation module: only an ephemeral one, rebuilt. +fn normalized_cache_control(source: &Value) -> Option { + let cache_control = source.get("cache_control")?.as_object()?; + if cache_control.get("type").and_then(Value::as_str) != Some("ephemeral") { + return None; + } + Some(match cache_control.get("ttl").and_then(Value::as_str) { + Some(ttl @ ("1h" | "5m")) => json!({"type": "ephemeral", "ttl": ttl}), + _ => json!({"type": "ephemeral"}), + }) +} + +/// The leading run of system messages, and the rest. +pub fn split_leading_system_run(messages: Vec) -> (Vec, Vec) { + let leading = messages + .iter() + .position(|message| role(message) != "system") + .unwrap_or(messages.len()); + let mut messages = messages; + let rest = messages.split_off(leading); + (messages, rest) +} + +/// `translate_system_message` for the leading run. +pub fn system_blocks(run: &[Value]) -> Vec { + let mut blocks = Vec::new(); + for message in run { + match message.get("content") { + Some(Value::String(text)) => { + if text.is_empty() { + continue; + } + let mut block = Map::from_iter([ + ("type".to_owned(), json!("text")), + ("text".to_owned(), json!(text)), + ]); + if let Some(cache_control) = message.get("cache_control") { + block.insert("cache_control".into(), cache_control.clone()); + } + blocks.push(Value::Object(block)); + } + Some(Value::Array(parts)) => { + for part in parts { + let text = part.get("text").cloned().unwrap_or(Value::Null); + let kind = part.get("type").cloned().unwrap_or(Value::Null); + let empty = match &text { + Value::String(text) => text.is_empty(), + Value::Null => true, + _ => false, + }; + if kind.as_str() == Some("text") && empty { + continue; + } + let mut block = + Map::from_iter([("type".to_owned(), kind), ("text".to_owned(), text)]); + if let Some(cache_control) = part.get("cache_control") { + block.insert("cache_control".into(), cache_control.clone()); + } + blocks.push(Value::Object(block)); + } + } + _ => {} + } + } + blocks +} + +/// The non-empty text parts of a system message, each with its normalized `cache_control`. +fn system_text_parts(message: &Value) -> Vec { + let block = |text: &str, cache_control: Option| { + let mut block = Map::from_iter([ + ("type".to_owned(), json!("text")), + ("text".to_owned(), json!(text)), + ]); + if let Some(cache_control) = cache_control { + block.insert("cache_control".into(), cache_control); + } + Value::Object(block) + }; + match message.get("content") { + Some(Value::String(text)) if !text.is_empty() => { + vec![block(text, normalized_cache_control(message))] + } + Some(Value::Array(parts)) => parts + .iter() + .filter(|part| part.get("type").and_then(Value::as_str) == Some("text")) + .filter_map(|part| { + let text = part.get("text")?.as_str().filter(|text| !text.is_empty())?; + Some(block(text, normalized_cache_control(part))) + }) + .collect(), + _ => Vec::new(), + } +} + +/// `system_message_as_user` for each message of a run that carries text. +pub fn converted_user_turns(run: &[Value]) -> Vec { + run.iter() + .filter_map(|message| { + let parts = system_text_parts(message); + if parts.is_empty() { + return None; + } + let mut content = vec![json!({"type": "text", "text": CONVERTED_SYSTEM_NOTE})]; + content.extend(parts); + Some(json!({"role": "user", "content": content})) + }) + .collect() +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Kind { + System, + Tool, + User, + Other, +} + +fn kind(message: &Value) -> Kind { + match role(message) { + "system" => Kind::System, + "tool" | "function" => Kind::Tool, + "user" => Kind::User, + _ => Kind::Other, + } +} + +/// `_converted_for_unflagged_model`: every later system message becomes a user turn in +/// place, a run caught before a tool result moving behind it. +pub fn place_mid_conversation_system(messages: Vec) -> Vec { + if !messages.iter().any(|message| kind(message) == Kind::System) { + return messages; + } + let mut runs: Vec<(Kind, Vec)> = Vec::new(); + for message in messages { + let message_kind = kind(&message); + match runs.last_mut() { + Some((last, run)) if *last == message_kind => run.push(message), + _ => runs.push((message_kind, vec![message])), + } + } + let mut placed = Vec::new(); + for index in 0..runs.len() { + let (run_kind, run) = &runs[index]; + let follower = runs.get(index + 1).map(|(kind, _)| *kind); + match run_kind { + Kind::System => { + if follower != Some(Kind::Tool) { + placed.extend(converted_user_turns(run)); + } + } + Kind::Tool if index > 0 && runs[index - 1].0 == Kind::System => { + placed.extend(run.iter().cloned()); + placed.extend(converted_user_turns(&runs[index - 1].1)); + } + _ => placed.extend(run.iter().cloned()), + } + } + placed +} + +/// `_sanitize_empty_text_content`. +fn sanitize_empty_text(mut message: Value) -> Value { + if !matches!(role(&message), "user" | "assistant") { + return message; + } + match message.get_mut("content") { + Some(Value::String(text)) if text.trim().is_empty() => { + *text = EMPTY_TEXT_PLACEHOLDER.to_owned(); + } + Some(Value::Array(parts)) => { + for part in parts { + if part.get("type").and_then(Value::as_str) != Some("text") { + continue; + } + let blank = part + .get("text") + .and_then(Value::as_str) + .is_none_or(|text| text.trim().is_empty()); + if blank && let Value::Object(fields) = part { + fields.insert("text".into(), json!(EMPTY_TEXT_PLACEHOLDER)); + } + } + } + _ => {} + } + message +} + +/// `convert_to_anthropic_image_obj`: a data URL as its media type and payload. +fn image_source(url: &str, format: Option<&str>) -> Result<(String, String), Error> { + let (media_type, data) = data_url(url).ok_or_else(|| { + bad_request(format!( + "Image url not in expected format. Example Expected input - \"image_url\": \"data:image/jpeg;base64,{{base64_image}}\". Supported formats - ['image/jpeg', 'image/png', 'image/gif', 'image/webp']. Error: {url:.40}" + )) + })?; + let media_type = match format { + Some(format) => format.to_owned(), + None => media_type.replace("\\/", "/"), + }; + Ok((media_type, data.to_owned())) +} + +/// `create_anthropic_image_param`. +fn image_block(part: &Value) -> Result { + let url = image_url(part).unwrap_or_default(); + let format = part + .get("image_url") + .and_then(|image| image.get("format")) + .and_then(Value::as_str); + if url.starts_with("https://") { + return Ok(json!({"type": "image", "source": {"type": "url", "url": url}})); + } + if url.starts_with("http") { + return Err(Error::Unsupported( + "an http:// image the host did not download", + )); + } + let (media_type, data) = image_source(url, format)?; + Ok( + json!({"type": "image", "source": {"type": "base64", "media_type": media_type, "data": data}}), + ) +} + +/// `anthropic_process_openai_file_message`. +fn file_block(part: &Value) -> Result { + let file = part.get("file").ok_or_else(|| { + bad_request("Content block has type='file' but is missing the required 'file' field") + })?; + let format = file.get("format").and_then(Value::as_str); + if let Some(data) = file + .get("file_data") + .and_then(Value::as_str) + .filter(|data| !data.is_empty()) + { + if data.starts_with("http") { + return Err(Error::Unsupported( + "a file_data URL the host did not download", + )); + } + let (media_type, data) = image_source(data, format)?; + return Ok( + json!({"type": "document", "source": {"type": "base64", "media_type": media_type, "data": data}}), + ); + } + match file + .get("file_id") + .and_then(Value::as_str) + .filter(|id| !id.is_empty()) + { + Some(id) if format.is_none() && id.starts_with("http") => { + Ok(json!({"type": "document", "source": {"type": "url", "url": id}})) + } + // Uploaded files need the Files API betas and, for a container upload, the code + // execution tool. + Some(_) => Err(Error::Unsupported("an Anthropic Files API file_id")), + None => Err(bad_request(format!( + "Either file_data or file_id must be present in the file message: {part}" + ))), + } +} + +/// `_is_anthropic_document_data_uri`. +fn is_document_data_uri(url: &str) -> bool { + url.strip_prefix("data:") + .map(|rest| rest.split([';', ',']).next().unwrap_or_default()) + .is_some_and(|media_type| matches!(media_type, "application/pdf" | "text/plain")) +} + +/// `convert_to_anthropic_tool_result`. +fn tool_result(message: &Value) -> Result { + let content = match message.get("content") { + Some(Value::String(text)) => json!(text), + Some(Value::Array(parts)) => { + let mut blocks = Vec::new(); + for part in parts { + match part.get("type").and_then(Value::as_str) { + Some("text") => { + let mut block = Map::from_iter([ + ("type".to_owned(), json!("text")), + ( + "text".to_owned(), + part.get("text").cloned().unwrap_or(Value::Null), + ), + ]); + if let Some(cache_control) = + part.get("cache_control").filter(|value| !value.is_null()) + { + block.insert("cache_control".into(), cache_control.clone()); + } + blocks.push(Value::Object(block)); + } + Some("image_url") => { + let url = image_url(part).unwrap_or_default(); + let block = match is_document_data_uri(url) { + true => { + file_block(&json!({"type": "file", "file": {"file_data": url}}))? + } + false => image_block(part)?, + }; + blocks.push(with_cache_control(block, part)); + } + Some("tool_reference") => blocks.push(json!({ + "type": "tool_reference", + "tool_name": part.get("tool_name").cloned().unwrap_or(Value::Null), + })), + Some("file") => blocks.push(with_cache_control(file_block(part)?, part)), + _ => {} + } + } + Value::Array(blocks) + } + _ => json!(""), + }; + let id = match role(message) { + "tool" => message + .get("tool_call_id") + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| bad_request("A tool message needs a tool_call_id."))?, + _ => message + .get("tool_call_id") + .and_then(Value::as_str) + .filter(|id| !id.is_empty()) + .map(str::to_owned) + .unwrap_or_else(|| uuid::Uuid::new_v4().to_string()), + }; + let mut result = Map::from_iter([ + ("type".to_owned(), json!("tool_result")), + ("tool_use_id".to_owned(), json!(sanitize_tool_use_id(&id))), + ("content".to_owned(), content), + ]); + if let Some(cache_control) = message + .get("cache_control") + .filter(|value| !value.is_null()) + { + result.insert("cache_control".into(), cache_control.clone()); + } + Ok(Value::Object(result)) +} + +pub fn is_encrypted_reasoning(block: &Value) -> bool { + let field = match block.get("type").and_then(Value::as_str) { + Some("thinking") => block.get("signature"), + Some("redacted_thinking") => block.get("data"), + _ => None, + }; + field + .and_then(Value::as_str) + .is_some_and(|value| value.starts_with(ENCRYPTED_REASONING_SIGNATURE_PREFIX)) +} + +/// `is_unsignable_thinking_block`. +pub fn is_unsignable_thinking_block(block: &Value) -> bool { + if is_encrypted_reasoning(block) { + return true; + } + block.get("type").and_then(Value::as_str) == Some("thinking") + && block + .get("signature") + .and_then(Value::as_str) + .is_none_or(str::is_empty) +} + +/// `native_redacted_thinking_data`: the base64 wrapper Converse adds, removed. +pub fn native_redacted_thinking_data(data: &str) -> String { + let engine = base64::engine::general_purpose::STANDARD; + engine + .decode(data) + .ok() + .and_then(|bytes| String::from_utf8(bytes).ok()) + .filter(|payload| payload.is_ascii() && engine.decode(payload).is_ok()) + .unwrap_or_else(|| data.to_owned()) +} + +/// `_replayable_thinking_blocks`. +fn replayable_thinking_blocks(blocks: &[Value]) -> Vec { + blocks + .iter() + .filter(|block| !is_unsignable_thinking_block(block)) + .map(|block| { + let mut block = block.clone(); + if let Value::Object(fields) = &mut block { + if fields.get("type").and_then(Value::as_str) == Some("redacted_thinking") + && let Some(Value::String(data)) = fields.get("data") + { + let native = native_redacted_thinking_data(data); + fields.insert("data".into(), json!(native)); + } + fields.remove("index"); + } + block + }) + .collect() +} + +/// The tool calls of an assistant message, as `tool_use` blocks. +fn tool_uses(tool_calls: &[Value], seen: &mut HashSet) -> Result, Error> { + let mut blocks = Vec::new(); + for call in tool_calls { + if call.get("type").and_then(Value::as_str) != Some("function") { + continue; + } + let id = call.get("id").and_then(Value::as_str).unwrap_or_default(); + if id.starts_with("srvtoolu_") { + return Err(Error::Unsupported( + "replaying an Anthropic server tool call", + )); + } + let function = call.get("function"); + let name = function + .and_then(|function| function.get("name")) + .cloned() + .unwrap_or(Value::Null); + let input = parse_tool_arguments(function.and_then(|function| function.get("arguments")))?; + let id = sanitize_tool_use_id(id); + let block = with_cache_control( + json!({"type": "tool_use", "id": id, "name": name, "input": input}), + call, + ); + if !seen.insert(id) { + continue; + } + blocks.push(block); + } + Ok(blocks) +} + +/// `anthropic_messages_pt` over messages whose later system messages are already placed. +pub fn anthropic_messages_pt(messages: Vec) -> Result, Error> { + let messages: Vec = messages.into_iter().map(sanitize_empty_text).collect(); + if messages.is_empty() { + return Err(bad_request( + "Anthropic requires at least one non-system message. Either provide one, or set `litellm.modify_params = True` // `litellm_settings::modify_params: True` to add the dummy user message - {'role': 'user', 'content': 'Please continue.'}.", + )); + } + let user_types = |message: &Value| matches!(role(message), "user" | "tool" | "function"); + // `_tool_result_later_in_turn`: whether a client tool result follows in the same turn. + let mut tool_result_later = vec![false; messages.len()]; + let mut later = false; + for index in (0..messages.len()).rev() { + tool_result_later[index] = later; + let message = &messages[index]; + later = user_types(message) && (later || matches!(role(message), "tool" | "function")); + } + + let mut out: Vec = Vec::new(); + let mut index = 0; + while index < messages.len() { + let start = index; + let mut user_content: Vec = Vec::new(); + while index < messages.len() && user_types(&messages[index]) { + let message = &messages[index]; + if role(message) == "user" { + match message.get("content") { + _ if content_is_empty(message) => { + if !tool_result_later[index] + && message.get("cache_control").is_some_and(truthy) + { + user_content.push(with_cache_control( + json!({"type": "text", "text": EMPTY_TEXT_PLACEHOLDER}), + message, + )); + } + } + Some(Value::Array(parts)) => { + for part in parts { + match part.get("type").and_then(Value::as_str) { + Some("image_url") => { + user_content.push(with_cache_control(image_block(part)?, part)) + } + Some("text") => user_content.push(with_cache_control( + json!({"type": "text", "text": part.get("text").cloned().unwrap_or(Value::Null)}), + part, + )), + Some("document") => { + user_content.push(with_cache_control(part.clone(), part)) + } + Some("file") => { + user_content.push(with_cache_control(file_block(part)?, part)) + } + _ => {} + } + } + } + Some(Value::String(text)) => user_content.push(with_cache_control( + json!({"type": "text", "text": text}), + message, + )), + _ => {} + } + } else { + user_content.push(tool_result(message)?); + } + index += 1; + } + if !user_content.is_empty() { + out.push(json!({"role": "user", "content": user_content})); + } else if let Some(empty) = messages[start..index] + .iter() + .rev() + .find(|message| role(message) == "user" && content_is_empty(message)) + { + out.push(json!({"role": "user", "content": [with_cache_control( + json!({"type": "text", "text": EMPTY_TEXT_PLACEHOLDER}), + empty, + )]})); + } + + let mut seen_tool_ids: HashSet = HashSet::new(); + let mut assistant_content: Vec = Vec::new(); + while index < messages.len() && role(&messages[index]) == "assistant" { + let message = &messages[index]; + let provider_fields = message.get("provider_specific_fields"); + if let Some(Value::Array(blocks)) = + provider_fields.and_then(|fields| fields.get("compaction_blocks")) + { + assistant_content.extend(blocks.iter().cloned()); + } + if provider_fields.is_some_and(|fields| { + fields + .get("web_search_results") + .is_some_and(|v| !v.is_null()) + || fields.get("tool_results").is_some_and(|v| !v.is_null()) + }) && message + .get("tool_calls") + .and_then(Value::as_array) + .is_some_and(|calls| !calls.is_empty()) + { + return Err(Error::Unsupported( + "replaying Anthropic server tool results", + )); + } + let thinking_blocks = match message.get("thinking_blocks") { + Some(Value::Array(blocks)) => Some(replayable_thinking_blocks(blocks)), + _ => None, + }; + let parts = message.get("content").and_then(Value::as_array); + let inline_thinking = parts.is_some_and(|parts| { + parts.iter().any(|part| { + matches!( + part.get("type").and_then(Value::as_str), + Some("thinking" | "redacted_thinking") + ) + }) + }); + if let Some(blocks) = thinking_blocks + && !inline_thinking + { + assistant_content.extend(blocks); + } + match message.get("content") { + Some(Value::Array(parts)) => { + for part in parts.iter().filter(|part| part.is_object()) { + let kind = part.get("type").and_then(Value::as_str).unwrap_or_default(); + let text = part.get("text").and_then(Value::as_str).unwrap_or_default(); + if kind == "thinking" && !is_unsignable_thinking_block(part) { + assistant_content.push(part.clone()); + } else if kind == "text" && !text.is_empty() { + assistant_content.push(with_cache_control( + json!({"type": "text", "text": text}), + part, + )); + } else if kind == "server_tool_use" || kind.ends_with("_tool_result") { + assistant_content.push(part.clone()); + } + } + } + Some(Value::String(text)) if !text.is_empty() => { + assistant_content.push(with_cache_control( + json!({"type": "text", "text": text}), + message, + )); + } + _ => {} + } + if let Some(Value::Array(calls)) = message.get("tool_calls") { + assistant_content.extend(tool_uses(calls, &mut seen_tool_ids)?); + } + if let Some(call) = message.get("function_call").filter(|call| !call.is_null()) { + let input = parse_tool_arguments(call.get("arguments"))?; + assistant_content.push(json!({ + "type": "tool_use", + "id": uuid::Uuid::new_v4().to_string(), + "name": call.get("name").filter(|name| !name.is_null()).cloned().unwrap_or_else(|| json!("")), + "input": input, + })); + } + index += 1; + } + if !assistant_content.is_empty() { + out.push(json!({"role": "assistant", "content": assistant_content})); + } + if index == start { + return Err(bad_request(format!( + "Invalid Message passed in {}", + messages[index] + ))); + } + } + if let Some(last) = out.last_mut() + && last["role"] == "assistant" + && let Some(Value::Array(blocks)) = last.get_mut("content") + { + for block in blocks { + if block.get("type").and_then(Value::as_str) == Some("text") + && let Some(Value::String(text)) = block.get_mut("text") + { + *text = text.trim_end().to_owned(); + } + } + } + Ok(out) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + fn list(value: Value) -> Vec { + value.as_array().unwrap().clone() + } + + #[rstest] + fn tool_round_trip_merges_turns() { + let messages = anthropic_messages_pt(list(json!([ + {"role": "user", "content": "weather?"}, + {"role": "assistant", "content": null, "tool_calls": [{"id": "call.1", "type": "function", "function": {"name": "get", "arguments": "{\"c\": \"P\"}"}}]}, + {"role": "tool", "tool_call_id": "call.1", "content": "sunny"}, + {"role": "user", "content": [{"type": "text", "text": "thanks", "cache_control": {"type": "ephemeral"}}]} + ]))) + .unwrap(); + assert_eq!( + messages, + list(json!([ + {"role": "user", "content": [{"type": "text", "text": "weather?"}]}, + {"role": "assistant", "content": [{"type": "tool_use", "id": "call_1", "name": "get", "input": {"c": "P"}}]}, + {"role": "user", "content": [ + {"type": "tool_result", "tool_use_id": "call_1", "content": "sunny"}, + {"type": "text", "text": "thanks", "cache_control": {"type": "ephemeral"}} + ]} + ])) + ); + } + + #[rstest] + fn images_empty_text_and_final_assistant_whitespace() { + let messages = anthropic_messages_pt(list(json!([ + {"role": "user", "content": [ + {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBO"}}, + {"type": "image_url", "image_url": "https://x/y.png"}, + {"type": "text", "text": " "} + ]}, + {"role": "assistant", "content": "Sure "} + ]))) + .unwrap(); + assert_eq!( + messages[0]["content"], + json!([ + {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "iVBO"}}, + {"type": "image", "source": {"type": "url", "url": "https://x/y.png"}}, + {"type": "text", "text": EMPTY_TEXT_PLACEHOLDER} + ]) + ); + assert_eq!( + messages[1]["content"], + json!([{"type": "text", "text": "Sure"}]) + ); + } + + #[rstest] + fn later_system_messages_become_operator_notes() { + let placed = place_mid_conversation_system(list(json!([ + {"role": "user", "content": "a"}, + {"role": "assistant", "content": "b", "tool_calls": [{"id": "t", "type": "function", "function": {"name": "f", "arguments": "{}"}}]}, + {"role": "system", "content": "remember", "cache_control": {"type": "ephemeral", "ttl": "1h", "x": 1}}, + {"role": "tool", "tool_call_id": "t", "content": "r"} + ]))); + assert_eq!(placed[2]["role"], "tool"); + assert_eq!( + placed[3], + json!({"role": "user", "content": [ + {"type": "text", "text": CONVERTED_SYSTEM_NOTE}, + {"type": "text", "text": "remember", "cache_control": {"type": "ephemeral", "ttl": "1h"}} + ]}) + ); + } + + #[rstest] + fn unsigned_thinking_is_dropped_and_signed_thinking_leads() { + let messages = anthropic_messages_pt(list(json!([ + {"role": "user", "content": "q"}, + {"role": "assistant", "content": "a", "thinking_blocks": [ + {"type": "thinking", "thinking": "t", "signature": "s", "index": 0}, + {"type": "thinking", "thinking": "u"} + ]} + ]))) + .unwrap(); + assert_eq!( + messages[1]["content"], + json!([{"type": "thinking", "thinking": "t", "signature": "s"}, {"type": "text", "text": "a"}]) + ); + } +} diff --git a/litellm-rust/crates/llms/src/anthropic/chat/mod.rs b/litellm-rust/crates/llms/src/anthropic/chat/mod.rs index f0050b7dc71..955055f2fbd 100644 --- a/litellm-rust/crates/llms/src/anthropic/chat/mod.rs +++ b/litellm-rust/crates/llms/src/anthropic/chat/mod.rs @@ -1,2 +1,4 @@ pub mod handler; +pub mod messages; +pub mod params; pub mod transformation; diff --git a/litellm-rust/crates/llms/src/anthropic/chat/params.rs b/litellm-rust/crates/llms/src/anthropic/chat/params.rs new file mode 100644 index 00000000000..cd377b3385a --- /dev/null +++ b/litellm-rust/crates/llms/src/anthropic/chat/params.rs @@ -0,0 +1,1415 @@ +//! Python's `AnthropicConfig.map_openai_params` and the model-map reads behind it: the +//! OpenAI params a chat request carries, as the Messages API takes them. + +use serde_json::{Map, Value, json}; + +use crate::{ + Error, ErrorDetail, + base_llm::chat::{ + params::{ + ChatRequestContext, DEFAULT_MAX_TOKENS, DropPolicy, RESPONSE_FORMAT_TOOL_NAME, + capability, ordered, truthy, unsupported_value, + }, + response_format::{ResponseFormatSchema, unpack_defs, unpack_legacy_defs}, + }, + bedrock::chat::converse_params::bedrock_base_model, +}; + +/// `ANTHROPIC_MIN_THINKING_BUDGET_TOKENS`. +pub const MIN_THINKING_BUDGET_TOKENS: i64 = 1024; + +/// The thinking budgets `_map_reasoning_effort` gives a legacy-thinking model. +pub fn reasoning_effort_budget(effort: &str) -> Option { + Some(match effort { + "low" => 1024, + "medium" => 2048, + "high" => 4096, + "xhigh" => 8192, + "max" => 16384, + // `max(DEFAULT_REASONING_EFFORT_MINIMAL_THINKING_BUDGET, ANTHROPIC_MIN_THINKING_BUDGET_TOKENS)` + "minimal" => 128.max(MIN_THINKING_BUDGET_TOKENS), + _ => return None, + }) +} + +/// `REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT`. +pub fn output_config_effort(effort: &str) -> Option<&'static str> { + Some(match effort { + "low" | "minimal" => "low", + "medium" => "medium", + "high" => "high", + "xhigh" => "xhigh", + "max" => "max", + _ => return None, + }) +} + +pub fn unmapped_reasoning_effort(effort: &str) -> Error { + Error::InvalidRequest(ErrorDetail::InvalidChoice { + field: "reasoning effort", + actual: format!("'{effort}'"), + choices: vec!["minimal", "low", "medium", "high", "xhigh", "max", "none"], + }) +} + +/// Strips a trailing `-v` or `-v:`. +fn strip_bedrock_version(model: &str) -> String { + let bytes = model.as_bytes(); + let mut end = bytes.len(); + // Optional `:`. + let mut cut = end; + while cut > 0 && bytes[cut - 1].is_ascii_digit() { + cut -= 1; + } + if cut < end && cut > 0 && bytes[cut - 1] == b':' { + end = cut - 1; + } + let mut digits = end; + while digits > 0 && bytes[digits - 1].is_ascii_digit() { + digits -= 1; + } + if digits < end && digits >= 2 && &bytes[digits - 2..digits] == b"-v" { + return model[..digits - 2].to_owned(); + } + // The `:` alone is not a version suffix, so only the full shape is removed. + model.to_owned() +} + +/// Strips a trailing `:`. +fn strip_profile_minor(model: &str) -> String { + let trimmed = model.trim_end_matches(|c: char| c.is_ascii_digit()); + match trimmed.strip_suffix(':') { + Some(head) if trimmed.len() < model.len() => head.to_owned(), + _ => model.to_owned(), + } +} + +/// Strips a trailing `-<8 digits>`. +fn strip_dated_release(model: &str) -> String { + let bytes = model.as_bytes(); + if bytes.len() >= 9 { + let tail = &bytes[bytes.len() - 9..]; + if tail[0] == b'-' && tail[1..].iter().all(u8::is_ascii_digit) { + return model[..model.len() - 9].to_owned(); + } + } + model.to_owned() +} + +/// `(\d)\.(\d)` -> `\1-\2`, non-overlapping like `re.sub`. +fn hyphenate_dotted_versions(model: &str) -> String { + let chars: Vec = model.chars().collect(); + let mut out = String::with_capacity(model.len()); + let mut index = 0; + while index < chars.len() { + if index + 2 < chars.len() + && chars[index].is_ascii_digit() + && chars[index + 1] == '.' + && chars[index + 2].is_ascii_digit() + { + out.push(chars[index]); + out.push('-'); + out.push(chars[index + 2]); + index += 3; + } else { + out.push(chars[index]); + index += 1; + } + } + out +} + +fn strip_at_version(model: &str) -> String { + match model.rfind('@') { + Some(at) if at > 0 => model[..at].to_owned(), + _ => model.to_owned(), + } +} + +/// `AnthropicModelInfo._model_map_lookup_candidates`. +pub fn model_map_candidates(model: &str) -> Vec { + let mut primary = vec![model.to_owned()]; + for prefix in [ + "bedrock/converse/", + "bedrock/invoke/", + "bedrock/", + "vertex_ai/", + ] { + if let Some(rest) = model.strip_prefix(prefix) { + primary.push(rest.to_owned()); + } + } + let base = bedrock_base_model(model); + if !base.is_empty() { + primary.push(base.clone()); + primary.push(format!("bedrock/{base}")); + } + let mut candidates: Vec = Vec::new(); + let mut push = |candidate: String| { + if !candidates.contains(&candidate) { + candidates.push(candidate); + } + }; + for candidate in &primary { + push(candidate.clone()); + } + for candidate in &primary { + push(strip_bedrock_version(candidate)); + push(strip_profile_minor(candidate)); + push(strip_dated_release(candidate)); + push(hyphenate_dotted_versions(candidate)); + push(strip_dated_release(&strip_bedrock_version( + &hyphenate_dotted_versions(candidate), + ))); + push(strip_at_version(candidate)); + } + candidates +} + +/// The host an Anthropic Messages request built from a chat call goes to. The request is the +/// same; the cost-map reads and a few params depend on the host, as Python's +/// `custom_llm_provider` on the config decides them. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum AnthropicHost { + /// The Anthropic API, `AnthropicConfig`. + #[default] + Anthropic, + /// Claude on Vertex AI, `VertexAIAnthropicConfig`. + VertexAi, +} + +impl AnthropicHost { + /// The provider an unsupported-params error names. + pub fn provider(self) -> &'static str { + match self { + Self::Anthropic => "anthropic", + Self::VertexAi => "vertex_ai", + } + } +} + +/// `_strip_stable_vertex_version`: a trailing `-` removed. +fn strip_stable_vertex_version(model: &str) -> String { + let trimmed = model.trim_end_matches(|c: char| c.is_ascii_digit()); + match trimmed.strip_suffix('-') { + Some(head) if trimmed.len() < model.len() => head.to_owned(), + _ => model.to_owned(), + } +} + +/// The keys `get_model_info_helper` tries for a Vertex AI model, most specific first: the +/// `vertex_ai/` entry, the bare id, then both with a stable version suffix removed. +pub fn vertex_model_info_candidates(model: &str) -> Vec { + let stripped = strip_stable_vertex_version(model); + let mut candidates: Vec = Vec::new(); + for candidate in [ + format!("vertex_ai/{model}"), + model.to_owned(), + format!("vertex_ai/{stripped}"), + stripped, + ] { + if !candidates.contains(&candidate) { + candidates.push(candidate); + } + } + candidates +} + +/// The cost-map reads a model's request rules make, through its lookup candidates. +pub struct ModelCapabilities<'a> { + pub model: String, + pub context: ChatRequestContext<'a>, + pub candidates: Vec, + pub host: AnthropicHost, + /// The keys a provider-aware lookup (`_get_provider_resolved_capability`) resolves the + /// model through; the first entry found answers. Empty when the host has none. + pub resolved: Vec, +} + +impl<'a> ModelCapabilities<'a> { + pub fn new(model: &str, context: &ChatRequestContext<'a>) -> Self { + Self::on(AnthropicHost::Anthropic, model, context) + } + + pub fn on(host: AnthropicHost, model: &str, context: &ChatRequestContext<'a>) -> Self { + Self { + model: model.to_owned(), + context: *context, + candidates: model_map_candidates(model), + host, + resolved: match host { + AnthropicHost::Anthropic => Vec::new(), + AnthropicHost::VertexAi => vertex_model_info_candidates(model), + }, + } + } + + /// `_get_model_capability`: the flag the first candidate the map knows sets. + pub fn walk_flag(&self, key: &str) -> Result, Error> { + capability(&self.context, &self.candidates, key) + } + + /// `_get_provider_resolved_capability`: the flag of the entry the host's own lookup + /// resolves the model to, `None` when that entry does not set it. + pub fn resolved_flag(&self, key: &str) -> Result, Error> { + if self.resolved.is_empty() { + return Ok(None); + } + let Some(info) = self.context.model_info else { + return match self.context.drop_params { + None => Err(Error::Unsupported( + "a request whose translation reads the model cost map", + )), + Some(_) => Ok(None), + }; + }; + Ok(self + .resolved + .iter() + .find_map(|candidate| info.entry(candidate)) + .and_then(|entry| entry.get(key).and_then(Value::as_bool))) + } + + /// `_supports_model_capability`'s answer: the host's resolved flag when it sets one, else + /// the model-map walk. + pub fn flag(&self, key: &str) -> Result, Error> { + match self.resolved_flag(key)? { + Some(flag) => Ok(Some(flag)), + None => self.walk_flag(key), + } + } + + /// `_supports_model_capability`: the flag is set to `true`. + pub fn supports(&self, key: &str) -> Result { + Ok(self.flag(key)? == Some(true)) + } + + /// `uses_native_structured_output`: Vertex AI reads only its own entry for it. + pub fn native_structured_output(&self) -> Result { + match self.host { + AnthropicHost::Anthropic => self.supports("supports_native_structured_output"), + AnthropicHost::VertexAi => { + Ok(self.resolved_flag("supports_native_structured_output")? == Some(true)) + } + } + } + + /// `_model_supports_effort_param`: `output_config` or any reasoning effort level. + pub fn supports_effort(&self) -> Result { + if self.supports("supports_output_config")? { + return Ok(true); + } + for level in ["low", "minimal", "medium", "high", "xhigh", "max"] { + if self.supports(&format!("supports_{level}_reasoning_effort"))? { + return Ok(true); + } + } + Ok(false) + } + + /// `_supports_sampling_params`. + pub fn supports_sampling(&self) -> Result { + if let Some(flag) = self.walk_flag("supports_sampling_params")? { + return Ok(flag); + } + let lower = self.model.to_lowercase(); + Ok(![ + "fable", "opus-4-7", "opus_4_7", "opus-4.7", "opus_4.7", "opus-4-8", "opus_4_8", + "opus-4.8", "opus_4.8", + ] + .iter() + .any(|name| lower.contains(name))) + } + + pub fn forced_tool_use_unsupported(&self) -> Result { + Ok(self.walk_flag("supports_forced_tool_use")? == Some(false)) + } + + pub fn is_adaptive(&self) -> Result { + self.supports("supports_adaptive_thinking") + } +} + +/// Python's `value` in an error message: its `str()`. +pub fn python_str(value: &Value) -> String { + match value { + Value::String(text) => text.clone(), + Value::Bool(true) => "True".into(), + Value::Bool(false) => "False".into(), + Value::Null => "None".into(), + other => other.to_string(), + } +} + +/// `_apply_sampling_param`: the param, unless the model removed sampling params. +pub fn apply_sampling_param( + out: &mut Map, + caps: &ModelCapabilities<'_>, + drop: DropPolicy, + param: &str, + value: &Value, + output_key: &str, +) -> Result<(), Error> { + if caps.supports_sampling()? || (param == "temperature" && value.as_f64() == Some(1.0)) { + out.insert(output_key.to_owned(), value.clone()); + return Ok(()); + } + drop.unsupported(|| { + Error::InvalidRequest(ErrorDetail::UnsupportedParameter { + model: caps.model.clone(), + param: param.to_owned(), + value: python_str(value), + hint: match param { + "temperature" => "Only temperature=1 is supported. ".into(), + _ => String::new(), + }, + }) + }) +} + +/// `_valid_user_id`: Anthropic rejects an email or a phone number as `metadata.user_id`. +pub fn valid_user_id(user_id: &str) -> bool { + !(looks_like_email(user_id) || looks_like_phone(user_id)) +} + +fn looks_like_email(text: &str) -> bool { + let Some((local, domain)) = text.split_once('@') else { + return false; + }; + let local_ok = !local.is_empty() + && local + .chars() + .all(|c| c.is_ascii_alphanumeric() || "._%+-".contains(c)); + let domain_ok = domain + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-') + && domain.rsplit_once('.').is_some_and(|(head, tld)| { + !head.is_empty() && tld.len() >= 2 && tld.chars().all(|c| c.is_ascii_alphabetic()) + }); + local_ok && domain_ok +} + +fn looks_like_phone(text: &str) -> bool { + let body = text.strip_prefix('+').unwrap_or(text); + body.chars().count() >= 7 + && body + .chars() + .all(|c| c.is_numeric() || c.is_whitespace() || "()-".contains(c)) +} + +/// `_map_stop_sequences`: whitespace-only entries are dropped only under `drop_params`. +pub fn map_stop_sequences(stop: &Value, drop: DropPolicy) -> Option { + let keep = |text: &str| !(is_python_space(text) && drop.drops()); + match stop { + Value::String(text) => keep(text).then(|| json!([text])), + Value::Array(items) => { + let kept: Vec = items + .iter() + .filter(|item| item.as_str().is_none_or(keep)) + .cloned() + .collect(); + (!kept.is_empty()).then_some(Value::Array(kept)) + } + _ => None, + } +} + +/// `str.isspace()`: non-empty and all whitespace. +pub fn is_python_space(text: &str) -> bool { + !text.is_empty() && text.chars().all(char::is_whitespace) +} + +/// `sanitize_input_schema_for_anthropic`. +pub fn sanitize_input_schema(schema: Option<&Value>) -> Value { + let mut normalized = match schema { + Some(Value::Object(object)) => object.clone(), + _ => Map::new(), + }; + if normalized.get("type").and_then(Value::as_str) != Some("object") { + normalized.insert("type".into(), json!("object")); + } + normalized + .entry("properties") + .or_insert_with(|| Value::Object(Map::new())); + unpack_legacy_defs(&mut normalized); + Value::Object( + normalized + .into_iter() + .filter(|(key, _)| { + matches!( + key.as_str(), + "type" + | "properties" + | "additionalProperties" + | "required" + | "$defs" + | "strict" + ) + }) + .collect(), + ) +} + +fn invalid_tool(message: &str) -> Error { + unsupported_value(message.to_owned()) +} + +/// `_map_tool_helper` for the tool shapes chat translates: function and custom tools. +fn map_function_tool(tool: &Map) -> Result { + let function = tool.get("function").and_then(Value::as_object); + let Some(function) = function else { + return Err(invalid_tool("A function tool needs a `function` object.")); + }; + let name = function + .get("name") + .cloned() + .ok_or_else(|| invalid_tool("A function tool needs a name."))?; + let default_schema = json!({"type": "object", "properties": {}}); + let schema = sanitize_input_schema(Some(function.get("parameters").unwrap_or(&default_schema))); + let mut mapped = Map::new(); + mapped.insert("name".into(), name); + mapped.insert("input_schema".into(), schema); + mapped.insert("type".into(), json!("custom")); + let eager = match tool.get("eager_input_streaming") { + Some(Value::Bool(flag)) => Some(*flag), + Some(Value::Null) | None => match function.get("eager_input_streaming") { + Some(Value::Bool(flag)) => Some(*flag), + Some(Value::Null) | None => None, + Some(_) => return Err(invalid_tool("eager_input_streaming must be a boolean")), + }, + Some(_) => return Err(invalid_tool("eager_input_streaming must be a boolean")), + }; + if let Some(eager) = eager { + mapped.insert("eager_input_streaming".into(), json!(eager)); + } + if let Some(description) = function.get("description").filter(|d| !d.is_null()) { + mapped.insert("description".into(), description.clone()); + } + match (tool.get("cache_control"), function.get("cache_control")) { + (Some(cache_control), _) if !cache_control.is_null() => { + mapped.insert("cache_control".into(), cache_control.clone()); + } + (_, Some(Value::Object(cache_control))) => { + mapped.insert("cache_control".into(), Value::Object(cache_control.clone())); + } + _ => {} + } + match (tool.get("defer_loading"), function.get("defer_loading")) { + (Some(Value::Bool(flag)), _) => { + mapped.insert("defer_loading".into(), json!(flag)); + } + (Some(other), _) if !other.is_null() => { + return Err(invalid_tool("defer_loading must be a boolean")); + } + (_, Some(Value::Bool(flag))) => { + mapped.insert("defer_loading".into(), json!(flag)); + } + (_, Some(other)) if !other.is_null() => { + return Err(invalid_tool("defer_loading must be a boolean")); + } + _ => {} + } + let string_list = |value: &Value| { + value + .as_array() + .is_some_and(|items| items.iter().all(Value::is_string)) + }; + for key in ["allowed_callers", "input_examples"] { + let value = tool + .get(key) + .filter(|v| !v.is_null()) + .or_else(|| function.get(key).filter(|v| !v.is_null())); + if let Some(value) = value { + if key == "allowed_callers" && !string_list(value) { + return Err(invalid_tool("allowed_callers must be a list of strings")); + } + if key == "input_examples" && !value.is_array() { + return Err(invalid_tool("input_examples must be a list")); + } + mapped.insert(key.into(), value.clone()); + } + } + Ok(Value::Object(mapped)) +} + +/// `_map_tools` for function and custom tools and tools already in Anthropic's shape. +pub fn map_tools(tools: &Value) -> Result, Error> { + let Some(tools) = tools.as_array() else { + return Err(invalid_tool("tools must be a list")); + }; + let mut mapped = Vec::new(); + for tool in tools { + let Some(object) = tool.as_object() else { + return Err(invalid_tool("each tool must be an object")); + }; + if object.contains_key("input_schema") { + mapped.push(tool.clone()); + continue; + } + match object.get("type").and_then(Value::as_str) { + Some("function" | "custom") => mapped.push(map_function_tool(object)?), + Some("namespace") => { + for nested in object + .get("tools") + .and_then(Value::as_array) + .into_iter() + .flatten() + { + let Some(nested_object) = nested.as_object() else { + continue; + }; + if nested_object.contains_key("input_schema") { + mapped.push(nested.clone()); + } else if !nested_object.contains_key("function") + && nested_object.contains_key("name") + { + let kind = nested_object + .get("type") + .cloned() + .unwrap_or_else(|| json!("function")); + let function: Map = nested_object + .iter() + .filter(|(key, _)| key.as_str() != "type") + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); + let wrapped = Map::from_iter([ + ("type".to_owned(), kind.clone()), + ("function".to_owned(), Value::Object(function)), + ]); + match kind.as_str() { + Some("function" | "custom") => { + mapped.push(map_function_tool(&wrapped)?) + } + _ => { + return Err(Error::Unsupported("a hosted tool inside a namespace")); + } + } + } else if nested_object.contains_key("function") { + match nested_object.get("type").and_then(Value::as_str) { + Some("function" | "custom") => { + mapped.push(map_function_tool(nested_object)?) + } + _ => { + return Err(Error::Unsupported("a hosted tool inside a namespace")); + } + } + } + } + } + // Computer use, hosted tools, MCP servers and tool search each have their own + // shape and betas; they stay on Python's path. + _ => { + return Err(Error::Unsupported( + "a chat tool other than a function or custom tool", + )); + } + } + } + Ok(mapped) +} + +/// `_map_tool_choice`. +pub fn map_tool_choice(tool_choice: Option<&Value>, parallel: Option<&Value>) -> Option { + let mut choice: Option> = match tool_choice { + Some(Value::String(choice)) => match choice.as_str() { + "auto" => Some(json!({"type": "auto"})), + "required" => Some(json!({"type": "any"})), + "none" => Some(json!({"type": "none"})), + _ => None, + }, + Some(Value::Object(choice)) + if choice.contains_key("type") && !choice.contains_key("function") => + { + match choice.get("type").and_then(Value::as_str) { + Some("auto") => Some(json!({"type": "auto"})), + Some("required" | "any") => Some(json!({"type": "any"})), + Some("none") => Some(json!({"type": "none"})), + _ => None, + } + } + Some(Value::Object(choice)) => choice + .get("function") + .and_then(|function| function.get("name")) + .filter(|name| !name.is_null()) + .map(|name| json!({"type": "tool", "name": name})), + _ => None, + } + .and_then(|value| value.as_object().cloned()); + if let Some(parallel) = parallel.filter(|parallel| !parallel.is_null()) { + let disable = json!(!truthy(parallel)); + let choice_is_none = tool_choice.and_then(Value::as_str) == Some("none"); + match &mut choice { + _ if choice_is_none => {} + Some(choice) => { + choice.insert("disable_parallel_tool_use".into(), disable); + } + None => { + choice = Some(Map::from_iter([ + ("type".to_owned(), json!("auto")), + ("disable_parallel_tool_use".to_owned(), disable), + ])); + } + } + } + choice.map(Value::Object) +} + +/// `_apply_forced_tool_choice`: `any` / `tool` on a model that refuses forced tool use. +pub fn apply_forced_tool_choice( + choice: Value, + caps: &ModelCapabilities<'_>, + drop: DropPolicy, +) -> Result { + let kind = choice.get("type").and_then(Value::as_str); + if !matches!(kind, Some("any" | "tool")) || !caps.forced_tool_use_unsupported()? { + return Ok(choice); + } + drop.unsupported(|| { + unsupported_value(format!( + "{} does not support forced tool use (tool_choice='required' or a named tool). Use tool_choice='auto' and tell the model in the prompt when to call the tool, or set `litellm.drop_params = True` to downgrade to 'auto' automatically.", + caps.model + )) + })?; + let mut auto = Map::from_iter([("type".to_owned(), json!("auto"))]); + if let Some(disable) = choice.get("disable_parallel_tool_use") { + auto.insert("disable_parallel_tool_use".into(), disable.clone()); + } + Ok(Value::Object(auto)) +} + +fn enum_conflicts_with_declared_type(schema: &Map) -> bool { + let (Some(Value::Array(values)), Some(declared)) = (schema.get("enum"), schema.get("type")) + else { + return false; + }; + if declared.is_array() { + return true; + } + let check: fn(&Value) -> bool = match declared.as_str() { + Some("null") => Value::is_null, + Some("boolean") => Value::is_boolean, + Some("integer") => |v| v.is_i64() || v.is_u64(), + Some("number") => Value::is_number, + Some("string") => Value::is_string, + Some("array") => Value::is_array, + Some("object") => Value::is_object, + _ => return false, + }; + !values.iter().all(check) +} + +const CONSTRAINT_LABELS: [(&str, &str); 25] = [ + ("minItems", "minimum number of items: {}"), + ("maxItems", "maximum number of items: {}"), + ("uniqueItems", "all array items must be unique"), + ("contains", "array must contain an item matching: {}"), + ("minContains", "minimum number of matching items: {}"), + ("maxContains", "maximum number of matching items: {}"), + ("prefixItems", "leading items must match, in order: {}"), + ("minimum", "minimum value: {}"), + ("maximum", "maximum value: {}"), + ("exclusiveMinimum", "exclusive minimum value: {}"), + ("exclusiveMaximum", "exclusive maximum value: {}"), + ("multipleOf", "must be a multiple of {}"), + ("minLength", "minimum length: {}"), + ("maxLength", "maximum length: {}"), + ("minProperties", "minimum number of properties: {}"), + ("maxProperties", "maximum number of properties: {}"), + ( + "patternProperties", + "properties whose names match each pattern must satisfy: {}", + ), + ("propertyNames", "property names must satisfy: {}"), + ("dependentRequired", "dependent required properties: {}"), + ("dependentSchemas", "dependent schemas: {}"), + ( + "unevaluatedProperties", + "unevaluated properties must satisfy: {}", + ), + ("if", "conditional (if): {}"), + ("then", "conditional (then): {}"), + ("else", "conditional (else): {}"), + ("not", "must not match: {}"), +]; + +/// `filter_anthropic_output_schema`: the constraints structured output cannot enforce, +/// removed and described instead. +pub fn filter_output_schema(schema: &Value) -> Value { + let Some(object) = schema.as_object() else { + return schema.clone(); + }; + let notes: Vec = CONSTRAINT_LABELS + .iter() + .filter_map(|(field, label)| { + let value = object.get(*field)?; + if value == &Value::Bool(false) { + return None; + } + let rendered = match value { + Value::Object(_) | Value::Array(_) => { + crate::base_llm::chat::params::python_dumps(value) + } + other => python_str(other), + }; + Some(label.replace("{}", &rendered)) + }) + .collect(); + let mut result = Map::new(); + if !notes.is_empty() { + let note = format!("Note: {}.", notes.join(", ")); + let existing = object + .get("description") + .and_then(Value::as_str) + .unwrap_or_default(); + let description = match existing.is_empty() { + true => note, + false => format!("{existing} {note}"), + }; + result.insert("description".into(), json!(description)); + } + let drops_type = enum_conflicts_with_declared_type(object); + for (key, value) in object { + if CONSTRAINT_LABELS.iter().any(|(field, _)| field == key) + || (key == "type" && drops_type) + || (key == "description" && result.contains_key("description")) + { + continue; + } + match (key.as_str(), value) { + ("properties" | "$defs", Value::Object(nested)) => { + result.insert( + key.clone(), + Value::Object( + nested + .iter() + .map(|(k, v)| (k.clone(), filter_output_schema(v))) + .collect(), + ), + ); + } + ("items", Value::Object(_)) => { + result.insert(key.clone(), filter_output_schema(value)); + } + ("anyOf" | "oneOf", Value::Array(items)) => { + let mut any_of = match result.remove("anyOf") { + Some(Value::Array(existing)) => existing, + _ => Vec::new(), + }; + any_of.extend(items.iter().map(filter_output_schema)); + result.insert("anyOf".into(), Value::Array(any_of)); + } + ("allOf", Value::Array(items)) => { + result.insert( + key.clone(), + Value::Array(items.iter().map(filter_output_schema).collect()), + ); + } + _ => { + result.insert(key.clone(), value.clone()); + } + } + } + if result.get("type").and_then(Value::as_str) == Some("object") { + result.insert("additionalProperties".into(), json!(false)); + } + Value::Object(result) +} + +/// The schema a `response_format` names, if any. A `json_schema` without `schema` names none. +fn response_format_schema(value: &Map) -> Option { + if let Some(schema) = value.get("response_schema") { + return Some(schema.clone()).filter(|schema| !schema.is_null()); + } + value + .get("json_schema") + .and_then(|json_schema| json_schema.get("schema")) + .filter(|schema| !schema.is_null()) + .cloned() +} + +/// `map_response_format_to_anthropic_output_format`. +fn output_format(value: &Map) -> Option { + let mut schema = response_format_schema(value)?; + let mut defs = Map::new(); + if let Value::Object(object) = &mut schema { + let definitions = object.remove("definitions"); + let dollar_defs = object.remove("$defs"); + if let Some(Value::Object(found)) = dollar_defs.or(definitions) { + defs = found; + } + } + if !defs.is_empty() { + unpack_defs(&mut schema, &defs); + } + Some(json!({"type": "json_schema", "schema": filter_output_schema(&schema)})) +} + +/// `_create_json_tool_call_for_response_format`. +fn response_format_tool(schema: &Value) -> Value { + let mut input_schema = Map::from_iter([("type".to_owned(), json!("object"))]); + if let Value::Object(schema) = schema { + input_schema.extend(schema.iter().map(|(k, v)| (k.clone(), v.clone()))); + } + json!({"name": RESPONSE_FORMAT_TOOL_NAME, "input_schema": input_schema}) +} + +/// `cap_thinking_budget_to_max_tokens`. +pub fn cap_thinking_budget(thinking: Value, max_tokens: Option<&Value>) -> Option { + let budget = thinking.get("budget_tokens").and_then(Value::as_i64); + let (Some(max_tokens), Some(budget)) = (max_tokens.and_then(Value::as_f64), budget) else { + return Some(thinking); + }; + if max_tokens <= MIN_THINKING_BUDGET_TOKENS as f64 { + return None; + } + if (budget as f64) < max_tokens { + return Some(thinking); + } + let kind = thinking + .get("type") + .cloned() + .unwrap_or_else(|| json!("enabled")); + Some(json!({"type": kind, "budget_tokens": (max_tokens - 1.0) as i64})) +} + +/// `_map_reasoning_effort`: `None` for `none`. +pub fn map_reasoning_effort( + effort: &str, + caps: &ModelCapabilities<'_>, +) -> Result, Error> { + if effort == "none" { + return Ok(None); + } + if caps.is_adaptive()? { + return Ok(Some(json!({"type": "adaptive", "display": "summarized"}))); + } + let budget = + reasoning_effort_budget(effort).ok_or_else(|| unmapped_reasoning_effort(effort))?; + Ok(Some(json!({"type": "enabled", "budget_tokens": budget}))) +} + +/// `_legacy_budget_to_effort`. +fn legacy_budget_to_effort( + budget: i64, + caps: &ModelCapabilities<'_>, +) -> Result<&'static str, Error> { + Ok( + if budget >= 8192 && caps.supports("supports_xhigh_reasoning_effort")? { + "xhigh" + } else if budget >= 4096 { + "high" + } else if budget >= 2048 { + "medium" + } else { + "low" + }, + ) +} + +/// `translate_legacy_thinking_for_adaptive_model`. +pub fn translate_legacy_thinking( + out: &mut Map, + caps: &ModelCapabilities<'_>, + drop: DropPolicy, +) -> Result<(), Error> { + if !caps.is_adaptive()? || caps.supports("supports_legacy_thinking")? { + return Ok(()); + } + let Some(Value::Object(thinking)) = out.get("thinking").cloned() else { + return Ok(()); + }; + if thinking.get("type").and_then(Value::as_str) != Some("enabled") { + return Ok(()); + } + let budget = match thinking.get("budget_tokens") { + Some(Value::Number(number)) => number.as_f64().unwrap_or(0.0) as i64, + Some(Value::String(text)) => text.trim().parse().unwrap_or(0), + _ => 0, + }; + let effort = legacy_budget_to_effort(budget, caps)?; + let mut rebuilt: Map = thinking + .iter() + .filter(|(key, _)| { + !matches!(key.as_str(), "type" | "budget_tokens" | "display") + && (key.as_str() == "block_binding" || !drop.drops()) + }) + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); + rebuilt.insert("type".into(), json!("adaptive")); + if let Some(display) = thinking + .get("display") + .filter(|display| matches!(display.as_str(), Some("summarized" | "omitted"))) + { + rebuilt.insert("display".into(), display.clone()); + } + let mut output_config = Map::from_iter([("effort".to_owned(), json!(effort))]); + if let Some(Value::Object(existing)) = out.get("output_config") { + output_config.extend(existing.iter().map(|(k, v)| (k.clone(), v.clone()))); + } + out.insert("thinking".into(), Value::Object(rebuilt)); + out.insert("output_config".into(), Value::Object(output_config)); + Ok(()) +} + +/// `web_search_options` as Anthropic's hosted web search tool. +pub fn web_search_tool(options: &Map) -> Value { + let mut tool = Map::from_iter([ + ("type".to_owned(), json!("web_search_20250305")), + ("name".to_owned(), json!("web_search")), + ]); + if let Some(location) = options.get("user_location").filter(|v| !v.is_null()) { + let mut user_location = Map::from_iter([("type".to_owned(), json!("approximate"))]); + if let Some(Value::Object(approximate)) = location.get("approximate") { + for (key, value) in approximate { + if matches!(key.as_str(), "city" | "region" | "country" | "timezone") { + user_location.insert(key.clone(), value.clone()); + } + } + } + tool.insert("user_location".into(), Value::Object(user_location)); + } + if let Some(size) = options.get("search_context_size").and_then(Value::as_str) { + let max_uses = match size { + "low" => Some(1), + "medium" => Some(5), + "high" => Some(10), + _ => None, + }; + if let Some(max_uses) = max_uses { + tool.insert("max_uses".into(), json!(max_uses)); + } + } + Value::Object(tool) +} + +/// `map_openai_context_management_to_anthropic`. +pub fn map_context_management(value: &Value) -> Option { + match value { + Value::Object(object) if object.contains_key("edits") => Some(value.clone()), + Value::Array(entries) => { + let edits: Vec = entries + .iter() + .filter_map(Value::as_object) + .filter(|entry| entry.get("type").and_then(Value::as_str) == Some("compaction")) + .map(|entry| { + let mut edit = Map::from_iter([("type".to_owned(), json!("compact_20260112"))]); + if let Some(threshold) = entry.get("compact_threshold").and_then(Value::as_f64) + { + edit.insert( + "trigger".into(), + json!({"type": "input_tokens", "value": threshold as i64}), + ); + } + for (key, value) in entry { + if !matches!(key.as_str(), "type" | "compact_threshold") { + edit.insert(key.clone(), value.clone()); + } + } + Value::Object(edit) + }) + .collect(); + (!edits.is_empty()).then(|| json!({"edits": edits})) + } + _ => None, + } +} + +/// `max_tokens` as Python keeps it: an int as sent, any other number rounded to at least 1. +pub fn max_tokens_value(value: &Value) -> Result { + match value { + Value::Number(number) if number.is_i64() || number.is_u64() => Ok(value.clone()), + Value::Number(number) => { + let rounded = number.as_f64().unwrap_or(1.0).round_ties_even(); + Ok(json!((rounded as i64).max(1))) + } + Value::Bool(flag) => Ok(json!(i64::from(*flag))), + other => Err(Error::InvalidRequest(ErrorDetail::InvalidValue { + field: "max_tokens", + expected: "a number", + actual: other.clone(), + })), + } +} + +/// What `map_openai_params` hands to `transform_request`. +#[derive(Debug, Default)] +pub struct MappedParams { + pub params: Map, + pub json_mode: bool, +} + +/// `get_supported_openai_params`. +pub fn supported_params(caps: &ModelCapabilities<'_>) -> Result, Error> { + let mut params = vec![ + "stream", + "stop", + "temperature", + "top_p", + "max_tokens", + "max_completion_tokens", + "tools", + "tool_choice", + "extra_headers", + "parallel_tool_calls", + "response_format", + "user", + "web_search_options", + "speed", + "context_management", + "cache_control", + ]; + let supports_reasoning = match caps.host { + AnthropicHost::Anthropic => { + let mut reasoning_keys = caps.candidates.clone(); + reasoning_keys.insert(1, format!("anthropic/{}", caps.model)); + capability(&caps.context, &reasoning_keys, "supports_reasoning")? == Some(true) + } + // `supports_reasoning(model, "vertex_ai")` reads the entry Vertex resolves to. + AnthropicHost::VertexAi => caps.resolved_flag("supports_reasoning")? == Some(true), + }; + if caps.model.contains("claude-3-7-sonnet") || caps.is_adaptive()? || supports_reasoning { + params.extend(["thinking", "reasoning_effort"]); + } + // `SERVICE_TIER_PROVIDERS`: Vertex AI rejects the field. + if caps.host == AnthropicHost::Anthropic { + params.push("service_tier"); + } + Ok(params) +} + +/// `AnthropicConfig.map_openai_params`. +pub fn map_openai_params( + openai: &Map, + caps: &ModelCapabilities<'_>, + drop: DropPolicy, +) -> Result { + let thinking_enabled = crate::base_llm::chat::params::is_thinking_enabled(openai); + let mut out = Map::new(); + let mut json_mode = false; + let add_tools = |out: &mut Map, tools: Vec| match out.get_mut("tools") { + Some(Value::Array(existing)) => existing.extend(tools), + _ => { + out.insert("tools".into(), Value::Array(tools)); + } + }; + for (name, value) in ordered(openai) { + match name { + "max_tokens" | "max_completion_tokens" => { + out.insert("max_tokens".into(), max_tokens_value(value)?); + } + "tools" => add_tools(&mut out, map_tools(value)?), + "tool_choice" | "parallel_tool_calls" => { + if let Some(choice) = + map_tool_choice(openai.get("tool_choice"), openai.get("parallel_tool_calls")) + { + out.insert( + "tool_choice".into(), + apply_forced_tool_choice(choice, caps, drop)?, + ); + } + } + "stream" if value == &Value::Bool(true) => { + out.insert("stream".into(), json!(true)); + } + "stop" if value.is_string() || value.is_array() => { + if let Some(stop) = map_stop_sequences(value, drop) { + out.insert("stop_sequences".into(), stop); + } + } + "temperature" | "top_p" => { + apply_sampling_param(&mut out, caps, drop, name, value, name)? + } + "response_format" => { + let Some(format) = value.as_object() else { + continue; + }; + if caps.native_structured_output()? { + if let Some(output_format) = output_format(format) { + out.insert("output_format".into(), output_format); + } + } else { + if format.get("type").and_then(Value::as_str) == Some("text") { + continue; + } + let Some(schema) = response_format_schema(format) else { + continue; + }; + if !thinking_enabled && !caps.forced_tool_use_unsupported()? { + out.insert( + "tool_choice".into(), + json!({"name": RESPONSE_FORMAT_TOOL_NAME, "type": "tool"}), + ); + } + add_tools(&mut out, vec![response_format_tool(&schema)]); + } + json_mode = true; + } + "user" => { + if let Some(user) = value.as_str().filter(|user| valid_user_id(user)) { + out.insert("metadata".into(), json!({"user_id": user})); + } + } + "thinking" => { + let adaptive_request = + value.get("type").and_then(Value::as_str) == Some("adaptive"); + if value.is_object() && adaptive_request && !caps.is_adaptive()? { + let max_tokens = openai + .get("max_completion_tokens") + .filter(|v| truthy(v)) + .or_else(|| openai.get("max_tokens")) + .filter(|v| !v.is_null()); + let legacy = map_reasoning_effort("medium", caps)?; + match legacy.and_then(|legacy| cap_thinking_budget(legacy, max_tokens)) { + Some(capped) => { + out.insert("thinking".into(), capped); + } + None => { + out.remove("thinking"); + } + } + } else { + out.insert("thinking".into(), value.clone()); + translate_legacy_thinking(&mut out, caps, drop)?; + } + } + "reasoning_effort" => { + let effort = match value { + Value::Object(effort) => effort.get("effort"), + other => Some(other), + }; + let Some(effort) = effort.and_then(Value::as_str) else { + continue; + }; + match map_reasoning_effort(effort, caps)? { + None => { + out.remove("thinking"); + out.remove("output_config"); + } + Some(thinking) => { + out.insert("thinking".into(), thinking); + if caps.is_adaptive()? { + let mapped = output_config_effort(effort) + .ok_or_else(|| unmapped_reasoning_effort(effort))?; + out.insert("output_config".into(), json!({"effort": mapped})); + } + } + } + } + "web_search_options" => { + if let Some(options) = value.as_object() { + add_tools(&mut out, vec![web_search_tool(options)]); + } + } + "context_management" => { + if let Some(mapped) = map_context_management(value) { + out.insert("context_management".into(), mapped); + } + } + "service_tier" => { + let Some(tier) = value.as_str() else { continue }; + match tier { + "auto" => { + out.insert("service_tier".into(), json!("auto")); + } + "default" | "standard_only" => { + out.insert("service_tier".into(), json!("standard_only")); + } + other => drop.unsupported(|| { + unsupported_value(format!( + "{} does not support service_tier='{other}'. Anthropic accepts 'auto', 'standard_only', or OpenAI's 'default' (sent as 'standard_only'). To drop unsupported params, set `litellm.drop_params = True`.", + caps.model + )) + })?, + } + } + _ => {} + } + } + // `update_optional_params_with_thinking_tokens`. + let thinking_on = out + .get("thinking") + .and_then(|thinking| thinking.get("type")) + .and_then(Value::as_str) + == Some("enabled"); + if thinking_on + && !openai.contains_key("max_tokens") + && !openai.contains_key("max_completion_tokens") + && let Some(budget) = out + .get("thinking") + .and_then(|thinking| thinking.get("budget_tokens")) + .and_then(Value::as_i64) + { + out.insert( + "max_tokens".into(), + json!(budget + DEFAULT_MAX_TOKENS as i64), + ); + } + Ok(MappedParams { + params: out, + json_mode, + }) +} + +/// The schema of the request's `json_tool_call` tool (`JsonModeRequest.tool_schema`). +pub fn json_tool_schema(tools: Option<&Value>) -> Option { + tools? + .as_array()? + .iter() + .find(|tool| tool.get("name").and_then(Value::as_str) == Some(RESPONSE_FORMAT_TOOL_NAME)) + .and_then(|tool| tool.get("input_schema")) + .and_then(ResponseFormatSchema::from_schema) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + fn object(value: Value) -> Map { + value.as_object().unwrap().clone() + } + + fn caps(model: &str) -> ModelCapabilities<'static> { + ModelCapabilities::new( + model, + &ChatRequestContext { + drop_params: Some(false), + model_info: None, + }, + ) + } + + fn map(params: Value) -> MappedParams { + map_openai_params( + &object(params), + &caps("claude-sonnet-4-5"), + DropPolicy::new(Some(false)), + ) + .unwrap() + } + + #[rstest] + fn candidates_cover_bedrock_ids_and_dotted_versions() { + let candidates = model_map_candidates("us.anthropic.claude-sonnet-4.6-20251101-v1:0"); + assert!(candidates.contains(&"anthropic.claude-sonnet-4.6-20251101-v1:0".to_string())); + assert!(candidates.contains(&"us.anthropic.claude-sonnet-4-6".to_string())); + assert!( + candidates.contains(&"bedrock/anthropic.claude-sonnet-4.6-20251101-v1:0".to_string()) + ); + assert!(model_map_candidates("claude-x@20250101").contains(&"claude-x".to_string())); + } + + #[rstest] + fn tools_and_choices_map_like_python() { + let mapped = map(json!({ + "tools": [{"type": "function", "function": {"name": "f", "description": "d", "parameters": {"type": "object", "properties": {"a": {}}, "title": "x"}}, "cache_control": {"type": "ephemeral"}}], + "tool_choice": "required", + "parallel_tool_calls": false + })); + assert_eq!( + mapped.params, + object(json!({ + "tools": [{"name": "f", "input_schema": {"type": "object", "properties": {"a": {}}}, "type": "custom", "description": "d", "cache_control": {"type": "ephemeral"}}], + "tool_choice": {"type": "any", "disable_parallel_tool_use": true} + })) + ); + assert_eq!( + map(json!({"tool_choice": {"type": "function", "function": {"name": "f"}}})).params["tool_choice"], + json!({"type": "tool", "name": "f"}) + ); + assert_eq!( + map(json!({"parallel_tool_calls": true})).params["tool_choice"], + json!({"type": "auto", "disable_parallel_tool_use": false}) + ); + } + + #[rstest] + fn max_tokens_stop_user_and_sampling() { + let mapped = map( + json!({"max_completion_tokens": 10.6, "stop": "END", "user": "u-1", "temperature": 0.2, "top_p": 0.5}), + ); + assert_eq!( + mapped.params, + object( + json!({"temperature": 0.2, "top_p": 0.5, "stop_sequences": ["END"], "max_tokens": 11, "metadata": {"user_id": "u-1"}}) + ) + ); + assert!( + !map(json!({"user": "a@b.co"})) + .params + .contains_key("metadata") + ); + assert!( + !map(json!({"user": "+1 (555) 123-4567"})) + .params + .contains_key("metadata") + ); + } + + #[rstest] + fn removed_sampling_params_are_refused_or_dropped() { + let refused = map_openai_params( + &object(json!({"temperature": 0.3})), + &caps("claude-opus-4-7"), + DropPolicy::new(Some(false)), + ) + .unwrap_err(); + assert_eq!( + refused.to_string(), + "invalid request: claude-opus-4-7 does not support temperature=0.3. Only temperature=1 is supported. To drop unsupported params, set `litellm.drop_params = True`." + ); + let dropped = map_openai_params( + &object(json!({"temperature": 0.3, "top_p": 0.1})), + &caps("claude-opus-4-7"), + DropPolicy::new(Some(true)), + ) + .unwrap(); + assert!(dropped.params.is_empty()); + } + + #[rstest] + fn response_format_becomes_a_forced_tool() { + let mapped = map( + json!({"response_format": {"type": "json_schema", "json_schema": {"name": "x", "schema": {"type": "object", "properties": {"a": {"type": "string"}}}}}}), + ); + assert!(mapped.json_mode); + assert_eq!( + mapped.params, + object(json!({ + "tool_choice": {"name": "json_tool_call", "type": "tool"}, + "tools": [{"name": "json_tool_call", "input_schema": {"type": "object", "properties": {"a": {"type": "string"}}}}] + })) + ); + assert!(!map(json!({"response_format": {"type": "text"}})).json_mode); + } + + #[rstest] + fn reasoning_effort_sets_a_budget_and_max_tokens() { + let mapped = map(json!({"reasoning_effort": "low"})); + assert_eq!( + mapped.params, + object( + json!({"thinking": {"type": "enabled", "budget_tokens": 1024}, "max_tokens": 5120}) + ) + ); + assert!( + map_openai_params( + &object(json!({"reasoning_effort": "huge"})), + &caps("claude-sonnet-4-5"), + DropPolicy::new(Some(false)) + ) + .is_err() + ); + } + + #[rstest] + fn output_schemas_lose_unenforceable_constraints() { + assert_eq!( + filter_output_schema( + &json!({"type": "object", "properties": {"n": {"type": "integer", "minimum": 1, "description": "count"}, "o": {"oneOf": [{"type": "string"}]}}}) + ), + json!({"type": "object", "properties": {"n": {"description": "count Note: minimum value: 1.", "type": "integer"}, "o": {"anyOf": [{"type": "string"}]}}, "additionalProperties": false}) + ); + } +} diff --git a/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs b/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs index 49a81adc32e..e281a07bcc4 100644 --- a/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs +++ b/litellm-rust/crates/llms/src/anthropic/chat/transformation.rs @@ -1,11 +1,15 @@ +//! Python's `AnthropicConfig` for `/chat/completions`: `get_optional_params` and +//! `transform_request` on the way in, `transform_parsed_response` on the way out. + +use std::collections::{BTreeSet, HashMap, HashSet}; + use litellm_auth::SecretValue; -use litellm_core_utils::{ - core_helpers::{finish_reason_for, unix_now, usage_from_parts}, - prompt_templates::factory::{Conversation, build_conversation}, -}; +use litellm_core_utils::core_helpers::{finish_reason_for, unix_now, usage_from_parts}; use litellm_llms_types::{ formats::chat_completions::{ - ChatCompletionsChoice, ChatCompletionsChoiceMessage, ChatCompletionsResponse, ChatMessage, + ChatCompletionThinkingBlock, ChatCompletionToolCallChunk, + ChatCompletionToolCallFunctionChunk, ChatCompletionsChoice, ChatCompletionsChoiceMessage, + ChatCompletionsResponse, ChatMessage, }, providers::anthropic::DEFAULT_HEADERS, }; @@ -13,9 +17,19 @@ use serde::Deserialize; use serde_json::{Map, Value, json}; use crate::{ - Error, + Error, ErrorDetail, anthropic::{ - chat::handler::ModelResponseIterator, + chat::{ + handler::ModelResponseIterator, + messages::{ + anthropic_messages_pt, place_mid_conversation_system, role, + split_leading_system_run, system_blocks, + }, + params::{ + AnthropicHost, MappedParams, ModelCapabilities, apply_sampling_param, + json_tool_schema, map_openai_params, map_tools, supported_params, valid_user_id, + }, + }, common_utils::{ API_KEY_PLACEMENT, complete_anthropic_url, forwarded_oauth_bearer, resolve_anthropic_api_key, @@ -24,50 +38,39 @@ use crate::{ base_llm::{ auth::AuthScheme, chat::{ + media::{data_url, remote_urls}, + params::{ + ChatRequestContext, DEFAULT_MAX_TOKENS, RESPONSE_FORMAT_TOOL_NAME, capability, + python_dumps, sort_params, + }, + response_format::json_tool_answer, streaming::{ChatStream, StreamShape}, transformation::{ BaseConfig, Headers, ProviderChatRequestData, ProviderChatResponseData, - STREAM_PARAM, Unsupported, ValidatedEnvironment, unsupported_message, - unsupported_param, + STREAM_PARAM, Unsupported, ValidatedEnvironment, }, }, messages::streaming::event_stream as messages_event_stream, }, }; -/// Anthropic parameter names, post `map_openai_params`, that the Rust path can -/// place verbatim in the Messages body. -/// -/// `top_k` is deliberately absent even though the Messages API takes it. -/// `temperature` and `top_p` reach this gate already resolved, because -/// `map_openai_params` runs first and applies `_apply_sampling_param` to them. -/// `top_k` bypasses `map_openai_params` entirely, so Python applies that same -/// per-model gate inside `transform_request`, the function this route replaces. -/// Forwarding it would send `top_k` to a model that removed sampling params and -/// take a 400 after the call, where Python drops it and succeeds. -const SUPPORTED_PARAMS: &[(&str, &str)] = &[ - ("max_tokens", "max_tokens"), - ("temperature", "temperature"), - ("top_p", "top_p"), - ("stop", "stop_sequences"), -]; +pub const BETA_HEADER: &str = "anthropic-beta"; +const TOOL_NAME_MAX_LEN: usize = 128; +const VALID_EFFORTS: [&str; 5] = ["high", "medium", "low", "xhigh", "max"]; #[derive(Deserialize)] -struct TextResponseProjection { +struct ResponseProjection { model: String, - content: Vec, + content: Vec, usage: ResponseUsageProjection, + #[serde(default)] stop_reason: Option, -} - -#[derive(Deserialize)] -#[serde(tag = "type", rename_all = "snake_case")] -enum TextResponseBlock { - Text { - text: String, - }, - #[serde(other)] - Other, + #[serde(default)] + stop_details: Option, + #[serde(default)] + context_management: Option, + #[serde(default)] + container: Option, } #[derive(Deserialize)] @@ -84,6 +87,719 @@ pub struct AnthropicConfig; pub const ANTHROPIC_CHAT_COMPLETIONS_CONFIG: AnthropicConfig = AnthropicConfig; +/// `add_dummy_tool`, the tool Anthropic needs when the history holds tool calls and the +/// request names no tools. +fn dummy_tool() -> Value { + json!([{ + "type": "function", + "function": { + "name": "dummy_tool", + "description": "This is a dummy tool call", + "parameters": {"type": "object", "properties": {}}, + }, + }]) +} + +/// `get_prefix_prompt`: the text of a trailing `{"role": "assistant", "prefix": true}`. +fn prefix_prompt(messages: &[Value]) -> Option { + let last = messages.last()?; + let is_prefix = last + .get("prefix") + .is_some_and(crate::base_llm::chat::params::truthy); + (last.get("role").and_then(Value::as_str) == Some("assistant") && is_prefix) + .then(|| { + last.get("content") + .and_then(Value::as_str) + .map(str::to_owned) + }) + .flatten() +} + +/// `_basic_sanitize_anthropic_tool_name`. +fn basic_sanitize_tool_name(name: &str) -> String { + name.chars() + .map( + |c| match c.is_ascii_alphanumeric() || c == '_' || c == '-' { + true => c, + false => '_', + }, + ) + .take(TOOL_NAME_MAX_LEN) + .collect() +} + +/// `_build_anthropic_tool_name_maps`: `(forward, reverse)` for the names that change. +fn tool_name_maps(originals: &[String]) -> (HashMap, HashMap) { + let mut forward: HashMap = HashMap::new(); + let mut used: HashSet = originals + .iter() + .filter(|original| basic_sanitize_tool_name(original) == **original) + .cloned() + .collect(); + for original in originals { + let candidate = basic_sanitize_tool_name(original); + if candidate == *original || forward.contains_key(original) { + continue; + } + let mut unique = candidate.clone(); + let mut n = 1; + while used.contains(&unique) { + n += 1; + let suffix = format!("_{n}"); + let head: String = candidate + .chars() + .take(TOOL_NAME_MAX_LEN - suffix.len()) + .collect(); + unique = format!("{head}{suffix}"); + } + used.insert(unique.clone()); + forward.insert(original.clone(), unique); + } + let reverse = forward + .iter() + .map(|(original, sanitized)| (sanitized.clone(), original.clone())) + .collect(); + (forward, reverse) +} + +/// `_sanitize_tool_names_in_request` and `_rewrite_tool_names_in_messages`. +fn sanitize_tool_names( + params: &mut Map, + messages: &mut [Value], +) -> HashMap { + let Some(Value::Array(tools)) = params.get_mut("tools") else { + return HashMap::new(); + }; + let originals: Vec = tools + .iter() + .filter(|tool| tool.get("type").and_then(Value::as_str) == Some("custom")) + .filter_map(|tool| tool.get("name").and_then(Value::as_str)) + .filter(|name| !name.is_empty()) + .map(str::to_owned) + .collect(); + let (forward, reverse) = tool_name_maps(&originals); + if forward.is_empty() { + return reverse; + } + let rename = |value: Option<&mut Value>| { + if let Some(Value::String(name)) = value + && let Some(sanitized) = forward.get(name.as_str()) + { + *name = sanitized.clone(); + } + }; + for tool in tools.iter_mut() { + if tool.get("type").and_then(Value::as_str) == Some("custom") { + rename(tool.get_mut("name")); + } + } + if let Some(choice) = params.get_mut("tool_choice") + && choice.get("type").and_then(Value::as_str) == Some("tool") + { + rename(choice.get_mut("name")); + } + for message in messages.iter_mut() { + if let Some(Value::Array(calls)) = message.get_mut("tool_calls") { + for call in calls { + rename(call.get_mut("function").and_then(|f| f.get_mut("name"))); + } + } + rename( + message + .get_mut("function_call") + .and_then(|f| f.get_mut("name")), + ); + } + reverse +} + +/// The `anthropic-beta` values the request's features need: the ones Python's +/// `validate_environment` collects and those `update_headers_with_optional_anthropic_beta` +/// adds. Python joins the first lot from a set, so their order carries no meaning. +fn request_betas( + params: &Map, + original_messages: &[Value], + anthropic_messages: &[Value], + caps: &ModelCapabilities<'_>, +) -> Result, Error> { + let tools: Vec<&Value> = params + .get("tools") + .and_then(Value::as_array) + .into_iter() + .flatten() + .collect(); + let mut environment: BTreeSet<&'static str> = BTreeSet::new(); + let tool_field = |tool: &Value, key: &str| -> Option { + tool.get(key) + .or_else(|| tool.get("function").and_then(|f| f.get(key))) + .cloned() + }; + if params + .get("mcp_servers") + .is_some_and(crate::base_llm::chat::params::truthy) + { + environment.insert("mcp-client-2025-04-04"); + } + if tools.iter().any(|tool| { + tool_field(tool, "allowed_callers") + .and_then(|callers| callers.as_array().cloned()) + .is_some_and(|callers| callers.iter().any(|c| c == "code_execution_20250825")) + || tool_field(tool, "input_examples") + .and_then(|examples| examples.as_array().map(|e| !e.is_empty())) + .unwrap_or(false) + }) { + environment.insert("advanced-tool-use-2025-11-20"); + } + let output_config = params.get("output_config"); + if output_config + .and_then(|config| config.get("effort")) + .and_then(Value::as_str) + .is_some_and(|effort| !effort.is_empty()) + && !caps.is_adaptive()? + { + environment.insert("effort-2025-11-24"); + } + if original_messages + .iter() + .any(|message| message.get("output_config").is_some()) + { + environment.insert("mid-conversation-output-config-2026-07-01"); + } + let thinking = params.get("thinking"); + if thinking + .and_then(|t| t.get("type")) + .and_then(Value::as_str) + .is_some_and(|kind| matches!(kind, "adaptive" | "enabled")) + && thinking + .and_then(|t| t.get("display")) + .and_then(Value::as_str) + == Some("updates") + { + environment.insert("thinking-display-updates-2026-08-18"); + } + if original_messages.iter().any(|message| { + role(message) == "system" + && message + .get("content") + .and_then(Value::as_array) + .into_iter() + .flatten() + .any(|block| { + matches!( + block.get("type").and_then(Value::as_str), + Some("tool_addition" | "tool_removal") + ) && block + .get("tool") + .and_then(|t| t.get("type")) + .and_then(Value::as_str) + == Some("tool_reference") + }) + }) { + environment.insert("mid-conversation-tool-changes-2026-07-01"); + } + + let mut betas: Vec = environment.into_iter().map(str::to_owned).collect(); + let mut add = |beta: &str| { + if !betas.iter().any(|existing| existing == beta) { + betas.push(beta.to_owned()); + } + }; + let signed_compaction = anthropic_messages.iter().any(|message| { + message + .get("content") + .and_then(Value::as_array) + .into_iter() + .flatten() + .any(|block| { + block.get("type").and_then(Value::as_str) == Some("compaction") + && block + .get("signature") + .and_then(Value::as_str) + .is_some_and(|s| !s.is_empty()) + }) + }); + if params.get("compaction").is_some_and(|v| !v.is_null()) || signed_compaction { + add("compact-2026-09-04"); + } + if let Some(context_management) = params.get("context_management").filter(|v| !v.is_null()) { + let edits = match context_management { + Value::Object(object) if object.contains_key("edits") => { + object.get("edits").and_then(Value::as_array).cloned() + } + Value::Array(entries) => Some(entries.clone()), + _ => None, + }; + if let Some(edits) = edits { + let compact = |edit: &Value| { + matches!( + edit.get("type").and_then(Value::as_str), + Some("compact_20260112" | "compaction") + ) + }; + if edits.iter().any(compact) { + add("compact-2026-01-12"); + } + if edits.iter().any(|edit| !compact(edit)) { + add("context-management-2025-06-27"); + } + } + } + if params.get("output_format").is_some_and(|v| !v.is_null()) + || output_config + .and_then(|config| config.get("format")) + .is_some_and(|v| !v.is_null()) + { + add("structured-outputs-2025-11-13"); + } + if params.get("speed").and_then(Value::as_str) == Some("fast") { + add("fast-mode-2026-02-01"); + } + Ok(betas) +} + +/// `_apply_output_config`: the effort check Python makes before sending `output_config`. +fn apply_output_config( + body: &mut Map, + caps: &ModelCapabilities<'_>, + drops: bool, +) -> Result<(), Error> { + let Some(Value::Object(output_config)) = body.get("output_config").cloned() else { + return Ok(()); + }; + if output_config.is_empty() { + return Ok(()); + } + if drops && output_config.keys().any(|key| key != "format") && !caps.supports_effort()? { + match output_config.get("format") { + None | Some(Value::Null) => { + body.remove("output_config"); + } + Some(format) => { + body.insert("output_config".into(), json!({"format": format})); + } + } + return Ok(()); + } + let effort = output_config + .get("effort") + .filter(|effort| !effort.is_null()); + if let Some(effort) = effort { + let text = effort.as_str().filter(|text| VALID_EFFORTS.contains(text)); + if text.is_none() { + let shown = match effort { + Value::String(text) => format!("'{text}'"), + other => python_dumps(other), + }; + return Err(Error::InvalidRequest(ErrorDetail::Message(format!( + "Invalid effort value: {shown}. Must be one of: 'high', 'medium', 'low', 'xhigh', 'max'" + )))); + } + let value = match text { + Some("max") + if !(caps.is_adaptive()? || caps.supports("supports_max_reasoning_effort")?) => + { + Some("max") + } + Some("xhigh") if !caps.supports("supports_xhigh_reasoning_effort")? => Some("xhigh"), + _ => None, + }; + if let Some(value) = value { + return Err(Error::InvalidRequest(ErrorDetail::UnsupportedValue { + field: "effort", + value, + model: caps.model.clone(), + })); + } + } + Ok(()) +} + +/// `get_max_tokens_for_model`: the exact cost-map entry's limit, else 4096. +fn default_max_tokens(model: &str, context: &ChatRequestContext<'_>) -> Result { + let Some(info) = context.model_info else { + return match context.drop_params { + None => Err(Error::Unsupported( + "a request whose default max_tokens comes from the model cost map", + )), + Some(_) => Ok(json!(DEFAULT_MAX_TOKENS)), + }; + }; + let limit = info + .entry(model) + .and_then(|entry| match entry.get("max_output_tokens") { + Some(value) => Some(value.clone()), + None => entry.get("max_tokens").cloned(), + }); + Ok(match limit { + Some(value) if value.is_number() => value, + _ => json!(DEFAULT_MAX_TOKENS), + }) +} + +/// `AnthropicConfig.transform_request` after `map_openai_params`, for the host `caps` names: +/// the Messages body, `model` included, and the `anthropic-beta` values the Anthropic API +/// needs. A host with its own beta rules (Vertex AI) gets no header here and sets its own. +pub fn build_messages_request( + model: &str, + messages: Vec, + optional_params: Map, + context: &ChatRequestContext<'_>, + host: AnthropicHost, + config_params: &[&str], +) -> Result { + let caps = ModelCapabilities::on(host, model, context); + let request_metadata = optional_params.get("metadata").cloned(); + let sorted = sort_params( + optional_params, + config_params, + context, + |_| supported_params(&caps), + host.provider(), + model, + )?; + let drop = sorted.drop; + let MappedParams { + mut params, + json_mode, + } = map_openai_params(&sorted.openai, &caps, drop)?; + for name in &sorted.allowed { + if let Some(value) = sorted.openai.get(name) + && !params.contains_key(name) + { + params.insert(name.clone(), value.clone()); + } + } + params.extend(sorted.provider); + + let mut messages: Vec = messages + .iter() + .map(serde_json::to_value) + .collect::>() + .map_err(|error| Error::InvalidRequest(ErrorDetail::invalid("messages", error)))?; + let original_messages = messages.clone(); + let prefix_prompt = prefix_prompt(&messages); + + if !params.contains_key("tools") + && messages.iter().any(|message| { + message + .get("tool_calls") + .is_some_and(|calls| !calls.is_null()) + }) + { + params.insert("tools".into(), Value::Array(map_tools(&dummy_tool())?)); + } + + // `_maybe_drop_speed_param`: fast mode is the Anthropic API's own, per exact entry. + if let Some(speed) = params.get("speed").cloned() + && (host != AnthropicHost::Anthropic + || capability(context, &[model.to_owned()], "supports_speed")? != Some(true)) + { + drop.unsupported(|| { + let shown = match &speed { + Value::String(text) => format!("'{text}'"), + other => python_dumps(other), + }; + Error::InvalidRequest(ErrorDetail::Message(format!( + "{model} does not support speed={shown}. To drop unsupported params, set `litellm.drop_params = True`." + ))) + })?; + params.remove("speed"); + } + + // `maybe_drop_disabled_thinking`. + if params + .get("thinking") + .and_then(|thinking| thinking.get("type")) + .and_then(Value::as_str) + == Some("disabled") + && caps.supports("thinking_always_on")? + { + params.remove("thinking"); + } + + let tool_name_reverse_map = sanitize_tool_names(&mut params, &mut messages); + + let (leading, later) = split_leading_system_run(messages); + let system = system_blocks(&leading); + if !system.is_empty() { + params.insert("system".into(), Value::Array(system)); + } + let anthropic_messages = anthropic_messages_pt(place_mid_conversation_system(later))?; + + // `update_headers_with_optional_anthropic_beta` skips a Vertex request, and Vertex's + // `get_anthropic_headers` sends none of these. + let betas = match host { + AnthropicHost::Anthropic => { + request_betas(¶ms, &original_messages, &anthropic_messages, &caps)? + } + AnthropicHost::VertexAi => Vec::new(), + }; + + if !params.contains_key("max_tokens") { + params.insert("max_tokens".into(), default_max_tokens(model, context)?); + } + + if let Some(user_id) = request_metadata + .as_ref() + .and_then(|metadata| metadata.get("user_id")) + .and_then(Value::as_str) + .filter(|user_id| valid_user_id(user_id)) + { + params.insert("metadata".into(), json!({"user_id": user_id})); + } + if let Some(Value::Object(metadata)) = params.get("metadata") { + match metadata.get("user_id").filter(|user_id| !user_id.is_null()) { + Some(user_id) => { + let reduced = json!({"user_id": user_id}); + params.insert("metadata".into(), reduced); + } + None => { + params.remove("metadata"); + } + } + } + params.remove("is_vertex_request"); + params.remove("client_metadata"); + params.remove("json_mode"); + + if let Some(top_k) = params.remove("top_k").filter(|top_k| !top_k.is_null()) { + apply_sampling_param(&mut params, &caps, drop, "top_k", &top_k, "top_k")?; + } + + let speed = params + .get("speed") + .and_then(Value::as_str) + .map(str::to_owned); + let response_format_schema = json_mode + .then(|| json_tool_schema(params.get("tools"))) + .flatten(); + + let mut body = Map::from_iter([ + ("model".to_owned(), json!(model)), + ("messages".to_owned(), Value::Array(anthropic_messages)), + ]); + body.extend(params); + apply_output_config(&mut body, &caps, drop.drops())?; + + Ok(ProviderChatRequestData { + body: Value::Object(body), + stream_shape: StreamShape { + json_mode, + speed, + tool_name_reverse_map, + response_format_schema, + prefix_prompt, + }, + headers: match betas.is_empty() { + true => Vec::new(), + false => vec![(BETA_HEADER.to_owned(), betas.join(", "))], + }, + }) +} + +/// `transform_parsed_response`: a Messages API answer as a chat completion. +pub fn transform_messages_response( + body: Value, + shape: &StreamShape, +) -> Result { + let body: ResponseProjection = serde_json::from_value(body).map_err(|error| { + Error::InvalidResponse(ErrorDetail::invalid("messages response", error)) + })?; + anthropic_response(body, shape) +} + +/// `convert_tool_use_to_openai_format`, the name put back the way the caller spelled it. +fn tool_call(block: &Value, index: usize, shape: &StreamShape) -> ChatCompletionToolCallChunk { + let name = block.get("name").and_then(Value::as_str).map(|name| { + shape + .tool_name_reverse_map + .get(name) + .cloned() + .unwrap_or_else(|| name.to_owned()) + }); + ChatCompletionToolCallChunk { + id: block.get("id").and_then(Value::as_str).map(str::to_owned), + tool_type: "function".into(), + function: ChatCompletionToolCallFunctionChunk { + name, + arguments: python_dumps(block.get("input").unwrap_or(&Value::Null)), + provider_specific_fields: None, + }, + index: index as i64, + provider_specific_fields: None, + } +} + +/// Python's `refusal_provider_specific_fields`. +fn refusal_fields(stop_reason: Option<&str>, stop_details: Option<&Value>) -> Map { + let mut fields = Map::new(); + match stop_details.filter(|details| !details.is_null()) { + Some(details) => { + fields.insert("stop_reason".into(), json!(stop_reason)); + fields.insert("stop_details".into(), details.clone()); + } + None if stop_reason == Some("refusal") => { + fields.insert("stop_reason".into(), json!("refusal")); + } + None => {} + } + fields +} + +fn anthropic_response( + body: ResponseProjection, + shape: &StreamShape, +) -> Result { + let mut text = String::new(); + let mut citations: Option> = None; + let mut thinking_blocks: Option> = None; + let mut tool_calls: Vec = Vec::new(); + let mut web_search_results: Option> = None; + let mut tool_results: Option> = None; + let mut compaction_blocks: Option> = None; + for (index, block) in body.content.iter().enumerate() { + let kind = block + .get("type") + .and_then(Value::as_str) + .unwrap_or_default(); + match kind { + "text" => text.push_str( + block + .get("text") + .and_then(Value::as_str) + .unwrap_or_default(), + ), + "tool_use" | "server_tool_use" => tool_calls.push(tool_call(block, index, shape)), + result if result.ends_with("_tool_result") => { + let list = match result { + "web_search_tool_result" | "web_fetch_tool_result" => &mut web_search_results, + _ => &mut tool_results, + }; + list.get_or_insert_with(Vec::new).push(block.clone()); + } + "thinking" | "redacted_thinking" => thinking_blocks + .get_or_insert_with(Vec::new) + .push(block.clone()), + "compaction" => compaction_blocks + .get_or_insert_with(Vec::new) + .push(block.clone()), + _ => {} + } + if let Some(Value::Array(block_citations)) = block.get("citations") { + let supported_text = block.get("text").cloned().unwrap_or_else(|| json!("")); + citations.get_or_insert_with(Vec::new).push(Value::Array( + block_citations + .iter() + .map(|citation| { + let mut citation = citation.as_object().cloned().unwrap_or_default(); + citation.insert("supported_text".into(), supported_text.clone()); + Value::Object(citation) + }) + .collect(), + )); + } + } + let reasoning_content = thinking_blocks.as_ref().map(|blocks| { + blocks + .iter() + .filter_map(|block| block.get("thinking").and_then(Value::as_str)) + .collect::() + }); + if let Some(prefix) = &shape.prefix_prompt + && !text.starts_with(prefix.as_str()) + { + text = format!("{prefix}{text}"); + } + + // `_build_provider_specific_fields`, unset fields left out. + let mut provider_fields = Map::new(); + let mut put = |key: &str, value: Option| { + if let Some(value) = value.filter(|value| !value.is_null()) { + provider_fields.insert(key.to_owned(), value); + } + }; + put("citations", citations.map(Value::Array)); + put("thinking_blocks", thinking_blocks.clone().map(Value::Array)); + put("context_management", body.context_management.clone()); + put("web_search_results", web_search_results.map(Value::Array)); + put("tool_results", tool_results.map(Value::Array)); + put("container", body.container.clone()); + put("compaction_blocks", compaction_blocks.map(Value::Array)); + let refusal = refusal_fields(body.stop_reason.as_deref(), body.stop_details.as_ref()); + provider_fields.extend(refusal.clone()); + + let thinking_blocks: Option> = thinking_blocks + .map(|blocks| serde_json::from_value(Value::Array(blocks))) + .transpose() + .map_err(|error| Error::InvalidResponse(ErrorDetail::invalid("thinking block", error)))?; + + let mut stop_reason = body.stop_reason.clone().unwrap_or_default(); + let json_indices: Vec = match shape.json_mode { + true => tool_calls + .iter() + .enumerate() + .filter(|(_, call)| call.function.name.as_deref() == Some(RESPONSE_FORMAT_TOOL_NAME)) + .map(|(index, _)| index) + .collect(), + false => Vec::new(), + }; + let schema = shape.response_format_schema.as_ref(); + let message = if !json_indices.is_empty() && json_indices.len() == tool_calls.len() { + stop_reason = "stop".into(); + let arguments: Vec = tool_calls + .iter() + .map(|call| call.function.arguments.clone()) + .collect(); + ChatCompletionsChoiceMessage { + role: "assistant".into(), + content: Some(json_tool_answer(&arguments, schema)), + provider_specific_fields: (!refusal.is_empty()).then_some(refusal), + ..ChatCompletionsChoiceMessage::default() + } + } else { + if !json_indices.is_empty() { + let arguments: Vec = json_indices + .iter() + .map(|index| tool_calls[*index].function.arguments.clone()) + .collect(); + text.push_str(&json_tool_answer(&arguments, schema)); + let mut position = 0; + tool_calls.retain(|_| { + let keep = !json_indices.contains(&position); + position += 1; + keep + }); + } + ChatCompletionsChoiceMessage { + role: "assistant".into(), + content: (!text.is_empty()).then_some(text), + tool_calls: (!tool_calls.is_empty()).then_some(tool_calls), + reasoning_content, + thinking_blocks, + provider_specific_fields: (!provider_fields.is_empty()).then_some(provider_fields), + annotations: None, + } + }; + + Ok(ChatCompletionsResponse { + created: unix_now(), + model: body.model, + choices: vec![ChatCompletionsChoice { + index: 0, + message, + finish_reason: finish_reason_for(&stop_reason).to_string(), + logprobs: None, + }], + usage: usage_from_parts( + body.usage.input_tokens, + body.usage.output_tokens, + body.usage.cache_read_input_tokens.unwrap_or(0), + body.usage.cache_creation_input_tokens.unwrap_or(0), + ), + extra: Map::new(), + }) +} + impl BaseConfig for AnthropicConfig { fn secret_names(&self) -> Vec<&'static str> { use crate::anthropic::common_utils::{ @@ -96,10 +812,6 @@ impl BaseConfig for AnthropicConfig { ] } - fn supported_openai_param_mappings(&self) -> &'static [(&'static str, &'static str)] { - SUPPORTED_PARAMS - } - fn get_complete_url( &self, api_base: Option<&str>, @@ -115,59 +827,31 @@ impl BaseConfig for AnthropicConfig { model: &str, messages: Vec, optional_params: Map, + context: &ChatRequestContext<'_>, ) -> Result { - Ok(ProviderChatRequestData { - body: anthropic_body(model, &build_conversation(&messages), optional_params), - stream_shape: StreamShape::default(), - }) + build_messages_request( + model, + messages, + optional_params, + context, + AnthropicHost::Anthropic, + &[], + ) } fn transform_response( &self, _model: &str, response: ProviderChatResponseData, + shape: &StreamShape, ) -> Result { - let body: TextResponseProjection = - serde_json::from_value(response.body).map_err(|error| { - Error::InvalidResponse(crate::ErrorDetail::invalid("messages response", error)) - })?; - // The route declines tool and thinking requests, so a non-text block - // means the response carries something this path never asked for. - // Decline rather than silently dropping it; the host falls back. - if body - .content - .iter() - .any(|block| matches!(block, TextResponseBlock::Other)) - { - return Err(Error::Unsupported("non-text response content block")); - } - let text: String = body - .content - .into_iter() - .map(|block| match block { - TextResponseBlock::Text { text } => text, - TextResponseBlock::Other => String::new(), - }) - .collect(); + transform_messages_response(response.body, shape) + } - Ok(ChatCompletionsResponse { - created: unix_now(), - model: body.model, - choices: vec![ChatCompletionsChoice { - index: 0, - message: ChatCompletionsChoiceMessage { - role: "assistant".to_string(), - content: (!text.is_empty()).then_some(text), - }, - finish_reason: finish_reason_for(body.stop_reason.as_deref().unwrap_or("")) - .to_string(), - }], - usage: usage_from_parts( - body.usage.input_tokens, - body.usage.output_tokens, - body.usage.cache_read_input_tokens.unwrap_or(0), - body.usage.cache_creation_input_tokens.unwrap_or(0), - ), + /// `inlines_remote_media`: Anthropic fetches an `https` image itself, not an `http` one. + fn remote_media(&self, messages: &[ChatMessage]) -> Vec { + remote_urls(messages, &["user"], false, |url| { + url.starts_with("http://") && data_url(url).is_none() }) } @@ -214,107 +898,105 @@ impl BaseConfig for AnthropicConfig { DEFAULT_HEADERS } - /// An OAuth bearer is the whole credential: Python's `validate_environment` - /// authenticates with it and drops `x-api-key` rather than resolving one, so - /// the resolved key must not be applied over the top. Any other forwarded - /// `authorization` is unrelated to this header and does not defer, which is - /// also what Python does: it sends the deployment's `x-api-key` alongside. + /// Every request is translated; a param or message Python refuses is refused here too. fn unsupported_reason( &self, - messages: &[ChatMessage], - optional_params: &Map, + _messages: &[ChatMessage], + _optional_params: &Map, ) -> Option { - unsupported_param(self.supported_openai_param_mappings(), &[], optional_params) - .or_else(|| messages.iter().find_map(unsupported_message)) - // Anthropic rejects a request whose first turn is not a user turn. - // Python only repairs that under `litellm.modify_params`, which the - // core cannot observe, so decline instead of guessing. - .or_else(|| { - (!build_conversation(messages).opens_on_user_turn()) - .then_some(Unsupported("conversation does not open on a user turn")) - }) + None } } -fn text_block(text: &str) -> Value { - json!({"type": "text", "text": text}) -} - -fn anthropic_body( - model: &str, - conversation: &Conversation, - optional_params: Map, -) -> Value { - let messages: Vec = conversation - .turns - .iter() - .map(|turn| { - json!({ - "role": <&'static str>::from(turn.role), - "content": turn.texts.iter().map(|text| text_block(text)).collect::>(), - }) - }) - .collect(); - - let system: Vec = conversation.system.iter().map(|s| text_block(s)).collect(); - - let body = Map::from_iter( - [ - ("model".to_string(), json!(model)), - ("messages".to_string(), json!(messages)), - ] - .into_iter() - // Python builds `{"model", "messages", **optional_params}` with - // `system` already folded into optional_params, so a caller-supplied - // key of the same name wins here too. - .chain((!system.is_empty()).then(|| ("system".to_string(), json!(system)))) - .chain(optional_params), - ); - Value::Object(body) -} - #[cfg(test)] mod tests { - use crate::{ - Error, - anthropic::chat::transformation::ANTHROPIC_CHAT_COMPLETIONS_CONFIG, - base_llm::{ - auth::AuthScheme, - chat::transformation::{BaseConfig, ProviderChatResponseData, Unsupported}, - }, - }; + use std::collections::HashMap; + use litellm_llms_types::formats::chat_completions::{ChatCompletionsResponse, ChatMessage}; use rstest::rstest; use serde_json::{Map, Value, json}; + use super::*; + use crate::base_llm::model_info::ModelInfo; + + struct Map1(HashMap>); + + impl ModelInfo for Map1 { + fn entry(&self, model: &str) -> Option>> { + self.0.get(model).map(std::borrow::Cow::Borrowed) + } + fn declares(&self, key: &str) -> bool { + self.0.values().any(|entry| entry.contains_key(key)) + } + } + + fn cost_map() -> Map1 { + let entry = |value: Value| value.as_object().cloned().unwrap(); + Map1(HashMap::from([ + ( + "claude-sonnet-4-5".to_owned(), + entry(json!({"max_output_tokens": 64000, "supports_reasoning": true})), + ), + ( + "claude-opus-4-8".to_owned(), + entry( + json!({"max_output_tokens": 128000, "supports_adaptive_thinking": true, + "supports_speed": true, "supports_sampling_params": false, + "supports_native_structured_output": true}), + ), + ), + ])) + } + fn messages(value: Value) -> Vec { serde_json::from_value(value).expect("valid messages") } fn params(value: Value) -> Map { - match value { - Value::Object(map) => map, - other => panic!("params must be an object, got {other}"), - } + value.as_object().cloned().expect("params object") + } + + fn request( + model: &str, + msgs: Value, + opts: Value, + drop: bool, + ) -> Result { + let info = cost_map(); + let context = ChatRequestContext { + drop_params: Some(drop), + model_info: Some(&info), + }; + ANTHROPIC_CHAT_COMPLETIONS_CONFIG.transform_request( + model, + messages(msgs), + params(opts), + &context, + ) } fn transform(model: &str, msgs: Value, opts: Value) -> Value { - ANTHROPIC_CHAT_COMPLETIONS_CONFIG - .transform_request(model, messages(msgs), params(opts)) + request(model, msgs, opts, false) .expect("request transforms") .body } fn transform_response(body: Value) -> Result { - ANTHROPIC_CHAT_COMPLETIONS_CONFIG - .transform_response("claude-sonnet-4-5", ProviderChatResponseData { body }) + transform_response_with(body, &StreamShape::default()) } - fn reason(msgs: Value, opts: Value) -> Option { - ANTHROPIC_CHAT_COMPLETIONS_CONFIG.unsupported_reason(&messages(msgs), ¶ms(opts)) + fn transform_response_with( + body: Value, + shape: &StreamShape, + ) -> Result { + ANTHROPIC_CHAT_COMPLETIONS_CONFIG.transform_response( + "claude-sonnet-4-5", + ProviderChatResponseData { body }, + shape, + ) } - #[test] + #[rstest] fn builds_the_messages_body_python_builds() { let body = transform( "claude-sonnet-4-5", @@ -322,7 +1004,7 @@ mod tests { {"role": "system", "content": "be terse"}, {"role": "user", "content": "hi"} ]), - json!({"max_tokens": 128, "temperature": 0.2}), + json!({"max_tokens": 128, "temperature": 0.2, "stop": "END", "user": "u-1"}), ); assert_eq!( body, @@ -331,354 +1013,466 @@ mod tests { "messages": [ {"role": "user", "content": [{"type": "text", "text": "hi"}]} ], - "system": [{"type": "text", "text": "be terse"}], + "temperature": 0.2, + "stop_sequences": ["END"], "max_tokens": 128, - "temperature": 0.2 + "metadata": {"user_id": "u-1"}, + "system": [{"type": "text", "text": "be terse"}] }) ); } - #[test] - fn omits_system_when_no_system_message_is_present() { + #[rstest] + #[case::from_the_cost_map("claude-sonnet-4-5", 64000)] + #[case::unknown_model("claude-unknown", 4096)] + fn max_tokens_defaults_like_python(#[case] model: &str, #[case] expected: u64) { + let body = transform(model, json!([{"role": "user", "content": "hi"}]), json!({})); + assert_eq!(body["max_tokens"], json!(expected)); let body = transform( - "claude-sonnet-4-5", + model, json!([{"role": "user", "content": "hi"}]), - json!({"max_tokens": 16}), + json!({"max_completion_tokens": 7}), ); - assert!(body.get("system").is_none()); + assert_eq!(body["max_tokens"], json!(7)); } - #[test] - fn merges_consecutive_turns_and_wraps_every_text_in_a_block() { + #[rstest] + fn tools_tool_choice_and_history_round_trip() { let body = transform( "claude-sonnet-4-5", json!([ - {"role": "user", "content": "one"}, - {"role": "user", "content": [{"type": "text", "text": "two"}]}, - {"role": "assistant", "content": "ack"} - ]), - json!({"max_tokens": 16}), - ); - assert_eq!( - body["messages"], - json!([ - {"role": "user", "content": [ - {"type": "text", "text": "one"}, - {"type": "text", "text": "two"} + {"role": "user", "content": "weather?"}, + {"role": "assistant", "content": null, "tool_calls": [ + {"id": "call_1", "type": "function", + "function": {"name": "get.weather", "arguments": "{\"city\":\"Paris\"}"}} ]}, - {"role": "assistant", "content": [{"type": "text", "text": "ack"}]} - ]) - ); - } - - #[test] - fn right_strips_a_trailing_assistant_prefill_like_python() { - let body = transform( - "claude-sonnet-4-5", - json!([ - {"role": "user", "content": "hi"}, - {"role": "assistant", "content": "Argentina "} + {"role": "tool", "tool_call_id": "call_1", "content": "sunny"} ]), - json!({"max_tokens": 16}), - ); - assert_eq!( - body["messages"][1]["content"][0]["text"], - json!("Argentina") - ); - } - - #[test] - fn passes_every_supported_param_through_untouched() { - let body = transform( - "claude-sonnet-4-5", - json!([{"role": "user", "content": "hi"}]), json!({ - "max_tokens": 64, - "temperature": 0.1, - "top_p": 0.9, - "stop_sequences": ["STOP"] + "max_tokens": 10, + "tools": [{"type": "function", "function": { + "name": "get.weather", "description": "d", + "parameters": {"type": "object", "properties": {"city": {"type": "string"}}} + }}], + "tool_choice": {"type": "function", "function": {"name": "get.weather"}}, + "parallel_tool_calls": false }), ); - assert_eq!(body["max_tokens"], json!(64)); - assert_eq!(body["temperature"], json!(0.1)); - assert_eq!(body["top_p"], json!(0.9)); - assert_eq!(body["stop_sequences"], json!(["STOP"])); - } - - #[test] - fn declines_top_k_because_python_gates_it_by_model_below_this_point() { - // `temperature` and `top_p` arrive already resolved, because - // `map_openai_params` applies `_apply_sampling_param` to them before the - // gate runs. `top_k` bypasses that and is gated inside `transform_request`, - // the function this route replaces, so forwarding it would send `top_k` to - // a model that removed sampling params and take a 400 after the call, where - // Python drops it and succeeds. assert_eq!( - reason( - json!([{"role": "user", "content": "hi"}]), - json!({"top_k": 40}) - ), - Some(Unsupported("unrecognized request parameter")) + body["tools"], + json!([{"name": "get_weather", "input_schema": {"type": "object", "properties": {"city": {"type": "string"}}}, "type": "custom", "description": "d"}]) ); - } - - #[test] - fn declines_streaming_before_anything_else() { assert_eq!( - reason( - json!([{"role": "user", "content": "hi"}]), - json!({"stream": true, "max_tokens": 16}) - ), - Some(Unsupported("streaming")) + body["tool_choice"], + json!({"type": "tool", "name": "get_weather", "disable_parallel_tool_use": true}) ); - } - - #[test] - fn accepts_an_explicit_stream_false() { assert_eq!( - reason( - json!([{"role": "user", "content": "hi"}]), - json!({"stream": false, "max_tokens": 16}) - ), - None + body["messages"][1], + json!({"role": "assistant", "content": [ + {"type": "tool_use", "id": "call_1", "name": "get_weather", "input": {"city": "Paris"}} + ]}) + ); + assert_eq!( + body["messages"][2], + json!({"role": "user", "content": [ + {"type": "tool_result", "tool_use_id": "call_1", "content": "sunny"} + ]}) ); } #[rstest] - #[case::tools(json!({"tools": []}))] - #[case::tool_choice(json!({"tool_choice": {"type": "auto"}}))] - #[case::thinking(json!({"thinking": {"type": "enabled"}}))] - #[case::system(json!({"system": "injected"}))] - #[case::metadata(json!({"metadata": {"user_id": "u1"}}))] - #[case::output_config(json!({"output_config": {"effort": "high"}}))] - fn declines_any_param_outside_the_allowlist(#[case] param: Value) { + fn tool_history_without_tools_gets_the_dummy_tool() { + let body = transform( + "claude-sonnet-4-5", + json!([ + {"role": "user", "content": "hi"}, + {"role": "assistant", "tool_calls": [ + {"id": "c", "type": "function", "function": {"name": "f", "arguments": ""}} + ]}, + {"role": "tool", "tool_call_id": "c", "content": "ok"} + ]), + json!({"max_tokens": 5}), + ); + assert_eq!(body["tools"][0]["name"], "dummy_tool"); + assert_eq!(body["tools"][0]["description"], "This is a dummy tool call"); + } + + #[rstest] + fn response_format_becomes_a_forced_json_tool() { + let data = request( + "claude-sonnet-4-5", + json!([{"role": "user", "content": "hi"}]), + json!({"max_tokens": 5, "response_format": {"type": "json_schema", "json_schema": { + "name": "x", "schema": {"type": "object", "properties": {"a": {"type": "string"}}} + }}}), + false, + ) + .unwrap(); assert_eq!( - reason(json!([{"role": "user", "content": "hi"}]), param.clone()), - Some(Unsupported("unrecognized request parameter")), - "expected {param} to decline" + data.body["tool_choice"], + json!({"name": "json_tool_call", "type": "tool"}) + ); + assert_eq!(data.body["tools"][0]["name"], "json_tool_call"); + assert!(data.body.get("json_mode").is_none()); + assert!(data.stream_shape.json_mode); + assert!(data.stream_shape.response_format_schema.is_some()); + } + + #[rstest] + fn native_structured_output_needs_the_beta() { + let data = request( + "claude-opus-4-8", + json!([{"role": "user", "content": "hi"}]), + json!({"max_tokens": 5, "response_format": {"type": "json_schema", "json_schema": { + "name": "x", "schema": {"type": "object", "properties": {"a": {"type": "string"}}} + }}}), + false, + ) + .unwrap(); + assert!(data.body.get("output_format").is_some()); + assert_eq!( + data.headers, + vec![( + "anthropic-beta".to_owned(), + "structured-outputs-2025-11-13".to_owned() + )] ); } - #[test] - fn declines_tool_calls_tool_results_and_multimodal_content() { + #[rstest] + fn unsupported_params_are_refused_or_dropped() { + let msgs = json!([{"role": "user", "content": "hi"}]); + let error = request("claude-sonnet-4-5", msgs.clone(), json!({"seed": 1}), false) + .map(|_| ()) + .unwrap_err(); + assert!(matches!( + error, + Error::InvalidRequest(ErrorDetail::UnsupportedParams { ref params, .. }) if params == &["seed"] + )); + let body = request("claude-sonnet-4-5", msgs.clone(), json!({"seed": 1}), true) + .unwrap() + .body; + assert!(body.get("seed").is_none()); + let body = request( + "claude-sonnet-4-5", + msgs.clone(), + json!({"seed": 1, "drop_params": true}), + false, + ) + .unwrap() + .body; + assert!(body.get("seed").is_none()); + let body = request( + "claude-sonnet-4-5", + msgs.clone(), + json!({"seed": 1, "allowed_openai_params": ["seed"]}), + false, + ) + .unwrap() + .body; + assert_eq!(body["seed"], 1); + let body = request( + "claude-sonnet-4-5", + msgs, + json!({"temperature": 0.5, "additional_drop_params": ["temperature"]}), + false, + ) + .unwrap() + .body; + assert!(body.get("temperature").is_none()); + } + + #[rstest] + fn n_above_one_is_refused() { + let error = request( + "claude-sonnet-4-5", + json!([{"role": "user", "content": "hi"}]), + json!({"n": 2}), + false, + ) + .map(|_| ()) + .unwrap_err(); + assert!(matches!( + error, + Error::InvalidRequest(ErrorDetail::UnsupportedParams { .. }) + )); + } + + #[rstest] + fn sampling_params_follow_the_model() { + let msgs = json!([{"role": "user", "content": "hi"}]); + let error = request("claude-opus-4-8", msgs.clone(), json!({"top_k": 3}), false) + .map(|_| ()) + .unwrap_err(); assert_eq!( - reason( - json!([ - {"role": "user", "content": "hi"}, - {"role": "assistant", "content": null, "tool_calls": [ - {"id": "c1", "type": "function", - "function": {"name": "f", "arguments": "{}"}} - ]} - ]), - json!({}) - ), - Some(Unsupported("unrecognized message field")) + error.to_string(), + "invalid request: claude-opus-4-8 does not support top_k=3. To drop unsupported params, set `litellm.drop_params = True`." ); + let body = request("claude-opus-4-8", msgs.clone(), json!({"top_k": 3}), true) + .unwrap() + .body; + assert!(body.get("top_k").is_none()); + let body = transform("claude-sonnet-4-5", msgs, json!({"top_k": 3})); + assert_eq!(body["top_k"], 3); + } + + #[rstest] + fn speed_is_checked_against_the_exact_entry() { + let msgs = json!([{"role": "user", "content": "hi"}]); + let data = request( + "claude-opus-4-8", + msgs.clone(), + json!({"speed": "fast"}), + false, + ) + .unwrap(); + assert_eq!(data.body["speed"], "fast"); + assert_eq!(data.stream_shape.speed.as_deref(), Some("fast")); + assert_eq!(data.headers[0].1, "fast-mode-2026-02-01"); + let error = request("claude-sonnet-4-5", msgs, json!({"speed": "fast"}), false) + .map(|_| ()) + .unwrap_err(); assert_eq!( - reason( - json!([ - {"role": "user", "content": "hi"}, - {"role": "tool", "tool_call_id": "c1", "content": "ok"} - ]), - json!({}) - ), - Some(Unsupported("unrecognized message field")) - ); - assert_eq!( - reason( - json!([ - {"role": "user", "content": [ - {"type": "image_url", "image_url": {"url": "https://x/y.png"}} - ]}]), - json!({}) - ), - Some(Unsupported("non-text message content")) - ); - assert_eq!( - reason( - json!([ - {"role": "user", "content": [ - {"type": "text", "text": "hi", "cache_control": {"type": "ephemeral"}} - ]}]), - json!({}) - ), - Some(Unsupported("non-text message content")) + error.to_string(), + "invalid request: claude-sonnet-4-5 does not support speed='fast'. To drop unsupported params, set `litellm.drop_params = True`." ); } - #[test] - fn declines_a_message_whose_content_list_is_empty() { - // An empty list passes every per-part check, so without this it would reach - // the provider as an empty `content` array and fail after the call rather - // than declining to Python before it. - assert_eq!( - reason(json!([{"role": "user", "content": []}]), json!({})), - Some(Unsupported("message without content")) + #[rstest] + fn reasoning_effort_maps_to_thinking() { + let body = transform( + "claude-sonnet-4-5", + json!([{"role": "user", "content": "hi"}]), + json!({"reasoning_effort": "low"}), ); assert_eq!( - reason( - json!([{"role": "user", "content": [{"type": "text", "text": "hi"}]}]), - json!({}) - ), - None + body["thinking"], + json!({"type": "enabled", "budget_tokens": 1024}) + ); + assert_eq!(body["max_tokens"], json!(1024 + 4096)); + } + + #[rstest] + fn output_config_effort_is_validated() { + let error = request( + "claude-sonnet-4-5", + json!([{"role": "user", "content": "hi"}]), + json!({"output_config": {"effort": "max"}}), + false, + ) + .map(|_| ()) + .unwrap_err(); + assert_eq!( + error.to_string(), + "invalid request: effort='max' is not supported by this model. Got model: claude-sonnet-4-5" + ); + let error = request( + "claude-sonnet-4-5", + json!([{"role": "user", "content": "hi"}]), + json!({"output_config": {"effort": "huge"}}), + false, + ) + .map(|_| ()) + .unwrap_err(); + assert!(error.to_string().contains("Invalid effort value: 'huge'")); + } + + #[rstest] + fn metadata_keeps_only_a_user_id() { + let msgs = json!([{"role": "user", "content": "hi"}]); + let body = transform( + "claude-sonnet-4-5", + msgs.clone(), + json!({"metadata": {"user_id": "abc", "tags": ["x"]}}), + ); + assert_eq!(body["metadata"], json!({"user_id": "abc"})); + let body = transform("claude-sonnet-4-5", msgs, json!({"user": "a@b.com"})); + assert!(body.get("metadata").is_none()); + } + + #[rstest] + fn without_a_known_drop_policy_refusals_go_back_to_python() { + let context = ChatRequestContext::default(); + let error = ANTHROPIC_CHAT_COMPLETIONS_CONFIG + .transform_request( + "claude-sonnet-4-5", + messages(json!([{"role": "user", "content": "hi"}])), + params(json!({"max_tokens": 3})), + &context, + ) + .map(|_| ()) + .unwrap_err(); + assert!(matches!(error, Error::Unsupported(_))); + } + + #[rstest] + fn only_http_images_are_downloaded() { + let urls = ANTHROPIC_CHAT_COMPLETIONS_CONFIG.remote_media(&messages(json!([ + {"role": "user", "content": [ + {"type": "image_url", "image_url": {"url": "http://x/a.png"}}, + {"type": "image_url", "image_url": {"url": "https://x/b.png"}} + ]} + ]))); + assert_eq!(urls, vec!["http://x/a.png"]); + } + + #[rstest] + fn a_data_url_image_becomes_a_base64_block() { + let body = transform( + "claude-sonnet-4-5", + json!([{"role": "user", "content": [ + {"type": "text", "text": "what is this"}, + {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo="}} + ]}]), + json!({"max_tokens": 5}), + ); + assert_eq!( + body["messages"][0]["content"][1], + json!({"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "iVBORw0KGgo="}}) ); } - #[test] - fn declines_a_conversation_that_does_not_open_on_a_user_turn() { - assert_eq!( - reason( - json!([ - {"role": "system", "content": "be terse"}, - {"role": "assistant", "content": "prefill"} - ]), - json!({}) - ), - Some(Unsupported("conversation does not open on a user turn")) - ); - } - - #[test] - fn accepts_a_plain_text_conversation() { - assert_eq!( - reason( - json!([ - {"role": "system", "content": "be terse"}, - {"role": "user", "content": "hi"}, - {"role": "assistant", "content": "hello"}, - {"role": "user", "content": [{"type": "text", "text": "again"}]} - ]), - json!({"max_tokens": 16, "temperature": 0.5}) - ), - None - ); - } - - #[test] + #[rstest] fn normalizes_a_text_response_into_openai_shape() { let response = transform_response(json!({ "id": "msg_123", - "type": "message", - "role": "assistant", "model": "claude-sonnet-4-5-20260101", "content": [{"type": "text", "text": "hello"}, {"type": "text", "text": " there"}], "stop_reason": "end_turn", - "stop_sequence": null, "usage": {"input_tokens": 11, "output_tokens": 4} })) .expect("response transforms"); - assert_eq!(response.model, "claude-sonnet-4-5-20260101"); - assert_eq!(response.choices.len(), 1); - assert_eq!(response.choices[0].index, 0); - assert_eq!(response.choices[0].message.role, "assistant"); assert_eq!( response.choices[0].message.content.as_deref(), Some("hello there") ); assert_eq!(response.choices[0].finish_reason, "stop"); - assert_eq!(response.usage.prompt_tokens, 11); - assert_eq!(response.usage.completion_tokens, 4); assert_eq!(response.usage.total_tokens, 15); + let value = serde_json::to_value(response).unwrap(); + assert!(value.get("id").is_none()); + assert!(value["choices"][0]["message"].get("tool_calls").is_none()); } - #[test] + #[rstest] fn folds_cache_tokens_into_prompt_tokens_like_python() { let response = transform_response(json!({ "model": "claude-sonnet-4-5", "content": [{"type": "text", "text": "hi"}], "stop_reason": "end_turn", - "usage": { - "input_tokens": 10, - "output_tokens": 2, - "cache_read_input_tokens": 5, - "cache_creation_input_tokens": 3 - } + "usage": {"input_tokens": 10, "output_tokens": 2, + "cache_read_input_tokens": 5, "cache_creation_input_tokens": 3} })) .expect("response transforms"); assert_eq!(response.usage.prompt_tokens, 18); - assert_eq!(response.usage.total_tokens, 20); assert_eq!(response.usage.prompt_tokens_details.cached_tokens, 5); assert_eq!( response.usage.prompt_tokens_details.cache_creation_tokens, 3 ); - assert_eq!(response.usage.prompt_tokens_details.text_tokens, 10); } - #[test] - fn maps_max_tokens_stop_reason_to_length() { + #[rstest] + #[case::refusal("refusal", "content_filter")] + #[case::max_tokens("max_tokens", "length")] + #[case::tool_use("tool_use", "tool_calls")] + fn maps_stop_reasons(#[case] stop_reason: &str, #[case] expected: &str) { let response = transform_response(json!({ - "model": "claude-sonnet-4-5", - "content": [{"type": "text", "text": "hi"}], - "stop_reason": "max_tokens", + "model": "m", "content": [], "stop_reason": stop_reason, "usage": {"input_tokens": 1, "output_tokens": 1} })) - .expect("response transforms"); - assert_eq!(response.choices[0].finish_reason, "length"); - } - - #[test] - fn a_refusal_returns_the_completion_python_returns() { - // `refusal` is a stop_reason, not a content block type, so the content is - // ordinary text and this normalizes rather than declining. Python maps it - // to content_filter in _FINISH_REASON_MAP and returns the completion. - let response = transform_response(json!({ - "model": "claude-sonnet-4-5", - "content": [{"type": "text", "text": "I can't help with that."}], - "stop_reason": "refusal", - "usage": {"input_tokens": 9, "output_tokens": 6} - })) - .expect("a refusal still transforms"); - assert_eq!(response.choices[0].finish_reason, "content_filter"); - assert_eq!( - response.choices[0].message.content.as_deref(), - Some("I can't help with that.") - ); - } - - #[test] - fn reports_no_content_rather_than_an_empty_string() { - let response = transform_response(json!({ - "model": "claude-sonnet-4-5", - "content": [], - "stop_reason": "end_turn", - "usage": {"input_tokens": 1, "output_tokens": 0} - })) - .expect("response transforms"); + .unwrap(); + assert_eq!(response.choices[0].finish_reason, expected); assert_eq!(response.choices[0].message.content, None); } - #[test] - fn response_carries_no_id_so_python_keeps_its_chatcmpl_id() { - let response = transform_response(json!({ - "id": "msg_should_not_leak", - "model": "claude-sonnet-4-5", - "content": [{"type": "text", "text": "hi"}], - "stop_reason": "end_turn", - "usage": {"input_tokens": 1, "output_tokens": 1} - })) - .expect("response transforms"); - let value = serde_json::to_value(response).expect("serializable"); - assert!( - value.get("id").is_none(), - "the rust response must not carry an id, got {value}" + #[rstest] + fn tool_use_and_thinking_come_back_as_openai_fields() { + let shape = StreamShape { + tool_name_reverse_map: HashMap::from([("get_weather".into(), "get.weather".into())]), + ..StreamShape::default() + }; + let response = transform_response_with( + json!({ + "model": "m", + "content": [ + {"type": "thinking", "thinking": "hmm", "signature": "sig"}, + {"type": "text", "text": "checking"}, + {"type": "tool_use", "id": "toolu_1", "name": "get_weather", "input": {"city": "Paris"}} + ], + "stop_reason": "tool_use", + "usage": {"input_tokens": 1, "output_tokens": 1} + }), + &shape, + ) + .unwrap(); + let message = serde_json::to_value(&response.choices[0].message).unwrap(); + assert_eq!( + message["tool_calls"], + json!([{"id": "toolu_1", "type": "function", "function": {"name": "get.weather", "arguments": "{\"city\": \"Paris\"}"}, "index": 2}]) + ); + assert_eq!(message["reasoning_content"], "hmm"); + assert_eq!( + message["thinking_blocks"], + json!([{"type": "thinking", "thinking": "hmm", "signature": "sig"}]) + ); + assert_eq!(message["content"], "checking"); + assert_eq!(response.choices[0].finish_reason, "tool_calls"); + } + + #[rstest] + fn a_json_tool_answer_becomes_the_content() { + let shape = StreamShape { + json_mode: true, + ..StreamShape::default() + }; + let response = transform_response_with( + json!({ + "model": "m", + "content": [{"type": "tool_use", "id": "t", "name": "json_tool_call", "input": {"a": "b"}}], + "stop_reason": "tool_use", + "usage": {"input_tokens": 1, "output_tokens": 1} + }), + &shape, + ) + .unwrap(); + assert_eq!( + response.choices[0].message.content.as_deref(), + Some("{\"a\": \"b\"}") + ); + assert_eq!(response.choices[0].message.tool_calls, None); + assert_eq!(response.choices[0].finish_reason, "stop"); + } + + #[rstest] + fn a_prefix_prompt_is_put_back() { + let data = request( + "claude-sonnet-4-5", + json!([ + {"role": "user", "content": "hi"}, + {"role": "assistant", "content": "Argentina", "prefix": true} + ]), + json!({"max_tokens": 5}), + false, + ) + .unwrap(); + assert_eq!( + data.stream_shape.prefix_prompt.as_deref(), + Some("Argentina") + ); + let response = transform_response_with( + json!({"model": "m", "content": [{"type": "text", "text": " is"}], + "stop_reason": "end_turn", "usage": {"input_tokens": 1, "output_tokens": 1}}), + &data.stream_shape, + ) + .unwrap(); + assert_eq!( + response.choices[0].message.content.as_deref(), + Some("Argentina is") ); } - #[test] - fn declines_a_response_carrying_a_non_text_block() { - let err = transform_response(json!({ - "model": "claude-sonnet-4-5", - "content": [{"type": "tool_use", "id": "t1", "name": "f", "input": {}}], - "stop_reason": "tool_use", - "usage": {"input_tokens": 1, "output_tokens": 1} - })) - .expect_err("non-text block"); - assert_eq!(err, Error::Unsupported("non-text response content block")); - } - - #[rstest::rstest] + #[rstest] #[case::not_an_object(json!("nope"))] #[case::missing_content(json!({"model": "test-model", "usage": {"input_tokens": 1, "output_tokens": 1}}))] #[case::missing_usage(json!({"model": "test-model", "content": []}))] @@ -697,7 +1491,7 @@ mod tests { ); } - #[test] + #[rstest] fn resolves_the_messages_url_and_x_api_key_auth() { let config = &ANTHROPIC_CHAT_COMPLETIONS_CONFIG; assert_eq!( @@ -730,4 +1524,14 @@ mod tests { ] ); } + + #[rstest] + fn tool_names_are_sanitized_without_collisions() { + let (forward, reverse) = + tool_name_maps(&["a.b".to_owned(), "a_b".to_owned(), "c d".to_owned()]); + assert_eq!(forward["a.b"], "a_b_2"); + assert_eq!(forward["c d"], "c_d"); + assert!(!forward.contains_key("a_b")); + assert_eq!(reverse["a_b_2"], "a.b"); + } } diff --git a/litellm-rust/crates/llms/src/azure/chat/transformation.rs b/litellm-rust/crates/llms/src/azure/chat/transformation.rs index 7c6c1056f10..38df96b20cf 100644 --- a/litellm-rust/crates/llms/src/azure/chat/transformation.rs +++ b/litellm-rust/crates/llms/src/azure/chat/transformation.rs @@ -8,56 +8,15 @@ use serde_json::{Map, Value}; use crate::{ Error, ErrorDetail, - base_llm::model_info::{self, ModelInfo}, + base_llm::{ + chat::params::{ + OPENAI_PARAMS, UNVALIDATED_PARAMS, ordered, request_drop_params, string_list, truthy, + }, + model_info::{self, ModelInfo}, + }, openai::chat::transformation::{is_litellm_owned, take_extra_body}, }; -/// The keys of `DEFAULT_CHAT_COMPLETION_PARAM_VALUES` a request carries as OpenAI params, in -/// that table's order. Every other key goes to Azure through `extra_body`. -const OPENAI_PARAMS: &[&str] = &[ - "functions", - "function_call", - "temperature", - "top_p", - "n", - "stream", - "stream_options", - "stop", - "max_tokens", - "max_completion_tokens", - "modalities", - "prediction", - "audio", - "presence_penalty", - "frequency_penalty", - "logit_bias", - "user", - "response_format", - "seed", - "tools", - "tool_choice", - "max_retries", - "logprobs", - "top_logprobs", - "extra_headers", - "parallel_tool_calls", - "reasoning_effort", - "verbosity", - "thinking", - "web_search_options", - "include_server_side_tool_invocations", - "service_tier", - "safety_identifier", - "prompt_cache_key", - "prompt_cache_retention", - "store", - "metadata", - "context_management", -]; - -/// `PROVIDER_UNVALIDATED_PARAMS`: never refused, only left out when unsupported. -const UNVALIDATED_PARAMS: &[&str] = &["user", "stream_options", "stream", "max_retries"]; - /// `AzureOpenAIConfig.get_supported_openai_params`. const AZURE_GPT_PARAMS: &[&str] = &[ "temperature", @@ -410,6 +369,7 @@ impl AzureChatRequest<'_> { let model = self.deployment; if model.starts_with(RESPONSES_ROUTE) || model_info::field(self.info(), &format!("azure/{model}"), "mode") + .as_ref() .and_then(Value::as_str) == Some("responses") { @@ -830,7 +790,9 @@ fn openai_gpt_params(model: &str, info: Option<&dyn ModelInfo>) -> Vec<&'static .split_once(RESPONSES_ROUTE) .map_or(model, |(_, rest)| rest); if matches!( - model_info::field(info, catalog_name, "litellm_provider").and_then(Value::as_str), + model_info::field(info, catalog_name, "litellm_provider") + .as_ref() + .and_then(Value::as_str), Some("openai" | "text-completion-openai") ) { params.push("user"); @@ -913,14 +875,6 @@ fn azure_messages(messages: Value) -> Value { Value::Array(messages) } -/// The request's OpenAI params in `DEFAULT_CHAT_COMPLETION_PARAM_VALUES` order, the order -/// Python maps them in. -fn ordered(params: &Map) -> impl Iterator { - OPENAI_PARAMS - .iter() - .filter_map(|name| params.get(*name).map(|value| (*name, value))) -} - /// `_map_openai_params`: every supported param still unmapped goes as sent. fn copy_supported( non_default: &mut Map, @@ -936,40 +890,6 @@ fn copy_supported( } } -/// Python's truthiness for a JSON value. -fn truthy(value: &Value) -> bool { - match value { - Value::Null => false, - Value::Bool(flag) => *flag, - Value::Number(number) => number.as_f64() != Some(0.0), - Value::String(text) => !text.is_empty(), - Value::Array(items) => !items.is_empty(), - Value::Object(object) => !object.is_empty(), - } -} - -/// `normalize_drop_params`: a boolean, or a config's `"true"`. -fn request_drop_params(value: Option) -> bool { - match value { - Some(Value::Bool(flag)) => flag, - Some(Value::String(text)) => text.trim().eq_ignore_ascii_case("true"), - _ => false, - } -} - -fn string_list(value: Option) -> Vec { - match value { - Some(Value::Array(items)) => items - .into_iter() - .filter_map(|item| match item { - Value::String(text) => Some(text), - _ => None, - }) - .collect(), - _ => Vec::new(), - } -} - #[cfg(test)] mod tests { use std::collections::HashMap; @@ -997,8 +917,8 @@ mod tests { } impl ModelInfo for Entries { - fn entry(&self, model: &str) -> Option<&Map> { - self.0.get(model) + fn entry(&self, model: &str) -> Option>> { + self.0.get(model).map(std::borrow::Cow::Borrowed) } fn declares(&self, key: &str) -> bool { diff --git a/litellm-rust/crates/llms/src/base_llm/chat/media.rs b/litellm-rust/crates/llms/src/base_llm/chat/media.rs new file mode 100644 index 00000000000..8f631b5bb89 --- /dev/null +++ b/litellm-rust/crates/llms/src/base_llm/chat/media.rs @@ -0,0 +1,209 @@ +//! Media a chat message points at. Python downloads an image or file URL a provider cannot +//! take by reference while it builds the request; here the host downloads them first, from +//! the list a config names, and hands the messages back with each one inlined as a data URL. + +use std::collections::HashMap; + +use litellm_llms_types::formats::chat_completions::{ChatMessage, ChatMessageContent}; +use serde_json::Value; + +/// Python's `MAX_IMAGE_URL_DOWNLOAD_SIZE_MB`. +pub const MAX_MEDIA_DOWNLOAD_BYTES: u64 = 50 * 1024 * 1024; + +/// A part's `image_url`, given as a string or as `{"url": ...}`. +pub fn image_url(part: &Value) -> Option<&str> { + let image_url = part.get("image_url")?; + image_url + .as_str() + .or_else(|| image_url.get("url").and_then(Value::as_str)) +} + +/// `data:;base64,` split into its media type and payload, the way Python's +/// `split("data:")[1].split(";base64,")` reads it. +pub fn data_url(url: &str) -> Option<(&str, &str)> { + let (_, rest) = url.split_once("data:")?; + let (media_type, data) = rest.split_once(";base64,")?; + (!data.contains(";base64,")).then_some((media_type, data)) +} + +fn parts(message: &ChatMessage) -> &[Value] { + match &message.content { + Some(ChatMessageContent::Parts(parts)) => parts, + _ => &[], + } +} + +/// The URLs of `image_url` parts (and, with `files`, of `file.file_id` / `file.file_data`) +/// in the messages `roles` names that `remote` says must be downloaded, in order, each once. +pub fn remote_urls( + messages: &[ChatMessage], + roles: &[&str], + files: bool, + remote: impl Fn(&str) -> bool, +) -> Vec { + let mut urls: Vec = Vec::new(); + let mut add = |url: &str| { + if remote(url) && !urls.iter().any(|seen| seen == url) { + urls.push(url.to_owned()); + } + }; + for message in messages + .iter() + .filter(|message| roles.contains(&message.role.as_str())) + { + for part in parts(message) { + match part.get("type").and_then(Value::as_str) { + Some("image_url") => { + if let Some(url) = image_url(part) { + add(url); + } + } + Some("file") if files => { + let file = part.get("file"); + let source = ["file_id", "file_data"].into_iter().find_map(|key| { + file.and_then(|file| file.get(key)).and_then(Value::as_str) + }); + if let Some(url) = source { + add(url); + } + } + _ => {} + } + } + } + urls +} + +/// Every reference to a downloaded URL replaced by its data URL. +pub fn inline_media(messages: &mut [ChatMessage], downloaded: &HashMap) { + if downloaded.is_empty() { + return; + } + for message in messages { + let Some(ChatMessageContent::Parts(parts)) = &mut message.content else { + continue; + }; + for part in parts { + match part.get("type").and_then(Value::as_str) { + Some("image_url") => match part.get_mut("image_url") { + Some(Value::String(url)) => { + if let Some(data) = downloaded.get(url.as_str()) { + *url = data.clone(); + } + } + Some(Value::Object(image)) => { + if let Some(Value::String(url)) = image.get_mut("url") + && let Some(data) = downloaded.get(url.as_str()) + { + *url = data.clone(); + } + } + _ => {} + }, + Some("file") => { + if let Some(Value::Object(file)) = part.get_mut("file") { + for key in ["file_id", "file_data"] { + if let Some(Value::String(url)) = file.get_mut(key) + && let Some(data) = downloaded.get(url.as_str()) + { + *url = data.clone(); + } + } + } + } + _ => {} + } + } + } +} + +/// The media type a download is inlined with: the response's own, or one guessed from the +/// URL's extension the way Python's `infer_content_type_from_url_and_content` falls back. +pub fn downloaded_media_type(url: &str, content_type: &str) -> String { + let declared = content_type + .split(';') + .next() + .unwrap_or_default() + .trim() + .to_ascii_lowercase(); + if !declared.is_empty() && declared != "application/octet-stream" && declared.contains('/') { + return declared; + } + let path = url + .split(['?', '#']) + .next() + .unwrap_or(url) + .to_ascii_lowercase(); + let extension = path.rsplit('.').next().unwrap_or_default(); + match extension { + "png" => "image/png", + "jpg" | "jpeg" => "image/jpeg", + "gif" => "image/gif", + "webp" => "image/webp", + "pdf" => "application/pdf", + "txt" => "text/plain", + "csv" => "text/csv", + "html" | "htm" => "text/html", + "md" => "text/markdown", + "mp4" => "video/mp4", + _ => "image/jpeg", + } + .to_owned() +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + + fn messages(value: Value) -> Vec { + serde_json::from_value(value).unwrap() + } + + #[rstest] + fn remote_urls_are_listed_once_and_inlined() { + let mut list = messages(json!([ + {"role": "user", "content": [ + {"type": "image_url", "image_url": "http://a/x.png"}, + {"type": "image_url", "image_url": {"url": "http://a/x.png"}}, + {"type": "image_url", "image_url": {"url": "data:image/png;base64,AA=="}}, + {"type": "file", "file": {"file_id": "https://b/doc.pdf"}} + ]}, + {"role": "assistant", "content": [{"type": "image_url", "image_url": "http://c"}]} + ])); + let urls = remote_urls(&list, &["user", "tool"], true, |url| { + url.starts_with("http") + }); + assert_eq!(urls, vec!["http://a/x.png", "https://b/doc.pdf"]); + let downloaded = HashMap::from([( + "http://a/x.png".to_owned(), + "data:image/png;base64,Zm8=".to_owned(), + )]); + inline_media(&mut list, &downloaded); + let content = serde_json::to_value(&list[0].content).unwrap(); + assert_eq!(content[0]["image_url"], "data:image/png;base64,Zm8="); + assert_eq!(content[1]["image_url"]["url"], "data:image/png;base64,Zm8="); + assert_eq!(content[3]["file"]["file_id"], "https://b/doc.pdf"); + } + + #[rstest] + #[case("data:image/png;base64,AAAA", Some(("image/png", "AAAA")))] + #[case("https://x", None)] + fn data_urls_split(#[case] url: &str, #[case] expected: Option<(&str, &str)>) { + assert_eq!(data_url(url), expected); + } + + #[rstest] + #[case("https://x/a.PNG?x=1", "", "image/png")] + #[case("https://x/a", "image/webp; charset=x", "image/webp")] + #[case("https://x/a.pdf", "application/octet-stream", "application/pdf")] + fn media_types_fall_back_to_the_extension( + #[case] url: &str, + #[case] content_type: &str, + #[case] expected: &str, + ) { + assert_eq!(downloaded_media_type(url, content_type), expected); + } +} diff --git a/litellm-rust/crates/llms/src/base_llm/chat/mod.rs b/litellm-rust/crates/llms/src/base_llm/chat/mod.rs index fa7df180f50..4c43dde0757 100644 --- a/litellm-rust/crates/llms/src/base_llm/chat/mod.rs +++ b/litellm-rust/crates/llms/src/base_llm/chat/mod.rs @@ -1,2 +1,5 @@ +pub mod media; +pub mod params; +pub mod response_format; pub mod streaming; pub mod transformation; diff --git a/litellm-rust/crates/llms/src/base_llm/chat/params.rs b/litellm-rust/crates/llms/src/base_llm/chat/params.rs new file mode 100644 index 00000000000..9ffacff1611 --- /dev/null +++ b/litellm-rust/crates/llms/src/base_llm/chat/params.rs @@ -0,0 +1,434 @@ +//! Python's `get_optional_params` for the providers chat translates: the request's keys are +//! sorted into OpenAI params, keys LiteLLM consumes itself, and provider-specific keys that +//! pass through; an OpenAI param the provider does not take is dropped under `drop_params` +//! and refused otherwise. + +use serde_json::{Map, Value}; + +use crate::{ + Error, ErrorDetail, base_llm::model_info::ModelInfo, + openai::chat::transformation::is_litellm_owned, +}; + +/// The keys of `DEFAULT_CHAT_COMPLETION_PARAM_VALUES` a request carries as OpenAI params, in +/// that table's order, which is the order Python maps them in. +pub const OPENAI_PARAMS: &[&str] = &[ + "functions", + "function_call", + "temperature", + "top_p", + "n", + "stream", + "stream_options", + "stop", + "max_tokens", + "max_completion_tokens", + "modalities", + "prediction", + "audio", + "presence_penalty", + "frequency_penalty", + "logit_bias", + "user", + "response_format", + "seed", + "tools", + "tool_choice", + "max_retries", + "logprobs", + "top_logprobs", + "extra_headers", + "parallel_tool_calls", + "reasoning_effort", + "verbosity", + "thinking", + "web_search_options", + "include_server_side_tool_invocations", + "service_tier", + "safety_identifier", + "prompt_cache_key", + "prompt_cache_retention", + "store", + "metadata", + "context_management", +]; + +/// `PROVIDER_UNVALIDATED_PARAMS`: never refused, only left out when unsupported. +pub const UNVALIDATED_PARAMS: &[&str] = &["user", "stream_options", "stream", "max_retries"]; + +/// The tool Python adds when `response_format` asks for JSON and the model has no native +/// structured output. +pub const RESPONSE_FORMAT_TOOL_NAME: &str = "json_tool_call"; + +/// Python's `DEFAULT_MAX_TOKENS`, added to a thinking budget when the caller set no limit. +pub const DEFAULT_MAX_TOKENS: u64 = 4096; + +/// The request's OpenAI params in `DEFAULT_CHAT_COMPLETION_PARAM_VALUES` order. +pub fn ordered(params: &Map) -> impl Iterator { + OPENAI_PARAMS + .iter() + .filter_map(|name| params.get(*name).map(|value| (*name, value))) +} + +/// Python's truthiness for a JSON value. +pub fn truthy(value: &Value) -> bool { + match value { + Value::Null => false, + Value::Bool(flag) => *flag, + Value::Number(number) => number.as_f64() != Some(0.0), + Value::String(text) => !text.is_empty(), + Value::Array(items) => !items.is_empty(), + Value::Object(object) => !object.is_empty(), + } +} + +/// `normalize_drop_params`: a boolean, or a config's `"true"`. +pub fn request_drop_params(value: Option) -> bool { + match value { + Some(Value::Bool(flag)) => flag, + Some(Value::String(text)) => text.trim().eq_ignore_ascii_case("true"), + _ => false, + } +} + +pub fn string_list(value: Option) -> Vec { + match value { + Some(Value::Array(items)) => items + .into_iter() + .filter_map(|item| match item { + Value::String(text) => Some(text), + _ => None, + }) + .collect(), + _ => Vec::new(), + } +} + +/// What the host tells a config about the request beyond its params. +#[derive(Clone, Copy, Default)] +pub struct ChatRequestContext<'a> { + /// `drop_params` from the proxy settings or the deployment. `None` when the host cannot + /// say, as when Python hosts the call: a request Python would drop or refuse a param of + /// then goes back to Python instead. + pub drop_params: Option, + /// The model cost map, which several param rules read. + pub model_info: Option<&'a dyn ModelInfo>, +} + +/// Whether a param the provider does not take is dropped, refused, or left to Python. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct DropPolicy(Option); + +impl DropPolicy { + pub fn new(drop_params: Option) -> Self { + Self(drop_params) + } + + /// Whether unsupported params are dropped. + pub fn drops(self) -> bool { + self.0 == Some(true) + } + + /// Whether the host said what `drop_params` is. + pub fn known(self) -> bool { + self.0.is_some() + } + + /// A param Python drops under `drop_params` and refuses otherwise: `Ok` means drop it. + pub fn unsupported(self, refusal: impl FnOnce() -> Error) -> Result<(), Error> { + match self.0 { + Some(true) => Ok(()), + Some(false) => Err(refusal()), + None => Err(Error::Unsupported( + "a param Python drops or refuses by drop_params", + )), + } + } +} + +/// Python's `UnsupportedParamsError` for one param with its own message. +pub fn unsupported_value(message: impl Into) -> Error { + Error::InvalidRequest(ErrorDetail::Message(message.into())) +} + +/// A request's params as `get_optional_params` sorts them. +#[derive(Debug, Default)] +pub struct SortedParams { + /// The OpenAI params, refused or dropped ones removed. + pub openai: Map, + /// Every other key, sent to the provider as given. + pub provider: Map, + /// Keys the config consumes as call configuration. + pub config: Map, + /// `allowed_openai_params`: sent as given even when the provider does not list them. + pub allowed: Vec, + pub drop: DropPolicy, +} + +/// Sorts `params` and refuses or drops the OpenAI params `supported` leaves out. +pub fn sort_params( + params: Map, + config_params: &[&str], + context: &ChatRequestContext<'_>, + supported: impl FnOnce(&Map) -> Result, Error>, + provider: &'static str, + model: &str, +) -> Result { + let mut params = params; + let request_drop = request_drop_params(params.remove("drop_params")); + let drop = DropPolicy::new(match context.drop_params { + Some(configured) => Some(configured || request_drop), + None if request_drop => Some(true), + None => None, + }); + let allowed = string_list(params.remove("allowed_openai_params")); + let additional_drop = string_list(params.remove("additional_drop_params")); + let mut sorted = SortedParams { + allowed, + drop, + ..SortedParams::default() + }; + for (name, value) in params { + if config_params.contains(&name.as_str()) { + sorted.config.insert(name, value); + continue; + } + if value.is_null() || is_litellm_owned(&name) || additional_drop.contains(&name) { + continue; + } + match OPENAI_PARAMS.contains(&name.as_str()) { + true => sorted.openai.insert(name, value), + false => sorted.provider.insert(name, value), + }; + } + let supported = supported(&sorted.openai)?; + let unsupported: Vec = sorted + .openai + .iter() + .filter(|(name, _)| { + !supported.contains(&name.as_str()) && !sorted.allowed.iter().any(|a| a == *name) + }) + .map(|(name, _)| name.clone()) + .collect(); + let refused: Vec = unsupported + .iter() + .filter(|name| { + !UNVALIDATED_PARAMS.contains(&name.as_str()) + && !(name.as_str() == "n" + && sorted.openai.get(name.as_str()).and_then(Value::as_f64) == Some(1.0)) + }) + .cloned() + .collect(); + if !refused.is_empty() { + drop.unsupported(|| { + Error::InvalidRequest(ErrorDetail::UnsupportedParams { + provider, + params: refused.clone(), + model: model.to_string(), + }) + })?; + } + for name in unsupported { + sorted.openai.remove(&name); + } + Ok(sorted) +} + +/// A capability flag the cost map sets for the first of `candidates` it knows. +/// +/// Without a cost map Python's own lookups have nothing to go on either, so the answer is +/// `None`, except when the host could still fall back to Python, which does have one. +pub fn capability( + context: &ChatRequestContext<'_>, + candidates: &[String], + key: &str, +) -> Result, Error> { + let Some(info) = context.model_info else { + return match context.drop_params { + None => Err(Error::Unsupported( + "a request whose translation reads the model cost map", + )), + Some(_) => Ok(None), + }; + }; + Ok(candidates + .iter() + .filter_map(|candidate| info.entry(candidate)) + .find_map(|entry| entry.get(key).and_then(Value::as_bool))) +} + +/// A numeric field of the first of `candidates` the cost map knows. +pub fn model_field( + context: &ChatRequestContext<'_>, + candidates: &[String], + key: &str, +) -> Option { + let info = context.model_info?; + candidates + .iter() + .filter_map(|candidate| info.entry(candidate)) + .find_map(|entry| entry.get(key).filter(|value| !value.is_null()).cloned()) +} + +/// Whether the cost map has an entry for any of `candidates`. +pub fn model_known(context: &ChatRequestContext<'_>, candidates: &[String]) -> bool { + context + .model_info + .is_some_and(|info| candidates.iter().any(|c| info.entry(c).is_some())) +} + +/// `json.dumps(value)` with Python's default separators and ASCII escaping. +pub fn python_dumps(value: &Value) -> String { + litellm_python_compat::json::dumps(&litellm_python_compat::json::from_json(value.clone())) + .unwrap_or_else(|_| value.to_string()) +} + +/// `json.loads` of tool call arguments, as `parse_tool_call_arguments` reads them: empty +/// arguments are `{}`. +pub fn parse_tool_arguments(arguments: Option<&Value>) -> Result { + match arguments { + None | Some(Value::Null) => Ok(Value::Object(Map::new())), + Some(Value::String(text)) if text.trim().is_empty() => Ok(Value::Object(Map::new())), + Some(Value::String(text)) => serde_json::from_str(text).map_err(|error| { + Error::InvalidRequest(ErrorDetail::Message(format!( + "tool call arguments are not valid JSON: {error}" + ))) + }), + Some(other) => Ok(other.clone()), + } +} + +/// `is_thinking_enabled` over a params map. +pub fn is_thinking_enabled(params: &Map) -> bool { + let thinking = params.get("thinking"); + thinking == Some(&Value::Bool(true)) + || thinking + .and_then(|thinking| thinking.get("type")) + .and_then(Value::as_str) + == Some("enabled") + || params + .get("reasoning_effort") + .is_some_and(|effort| !effort.is_null()) +} + +/// The JSON schema a `response_format` names: `response_schema`, or `json_schema.schema`. +pub fn response_format_schema(value: &Value) -> Option { + if let Some(schema) = value.get("response_schema") { + return Some(schema.clone()); + } + value + .get("json_schema") + .and_then(|json_schema| json_schema.get("schema")) + .cloned() +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + + fn object(value: Value) -> Map { + match value { + Value::Object(map) => map, + other => panic!("expected an object, got {other}"), + } + } + + fn sort(params: Value, drop_params: Option) -> Result { + sort_params( + object(params), + &["aws_region_name"], + &ChatRequestContext { + drop_params, + model_info: None, + }, + |_| Ok(vec!["temperature", "max_tokens"]), + "anthropic", + "claude", + ) + } + + #[rstest] + fn params_sort_into_openai_provider_config_and_owned() { + let sorted = sort( + json!({"temperature": 0.5, "top_k": 3, "aws_region_name": "us-east-1", + "metadata": {"x": 1}, "mock_response": "hi", "_litellm_x": 1, "user": "u", "seed": null}), + Some(false), + ) + .unwrap(); + assert_eq!(sorted.openai, object(json!({"temperature": 0.5}))); + assert_eq!(sorted.provider, object(json!({"top_k": 3}))); + assert_eq!( + sorted.config, + object(json!({"aws_region_name": "us-east-1"})) + ); + } + + #[rstest] + #[case::refused(Some(false), false)] + #[case::dropped(Some(true), true)] + #[case::request_drop(None, true)] + fn an_unsupported_param_is_refused_or_dropped( + #[case] drop_params: Option, + #[case] dropped: bool, + ) { + let mut body = json!({"temperature": 0.5, "seed": 3}); + if drop_params.is_none() { + body["drop_params"] = json!(true); + } + let result = sort(body, drop_params); + match dropped { + true => assert_eq!(result.unwrap().openai, object(json!({"temperature": 0.5}))), + false => assert!(matches!( + result, + Err(Error::InvalidRequest(ErrorDetail::UnsupportedParams { ref params, .. })) + if params == &vec!["seed".to_string()] + )), + } + } + + #[rstest] + fn an_unknown_drop_policy_falls_back() { + assert!(matches!( + sort(json!({"seed": 3}), None), + Err(Error::Unsupported(_)) + )); + } + + #[rstest] + fn allowed_and_additional_drop_params_are_honoured() { + let sorted = sort( + json!({"seed": 3, "top_k": 2, "allowed_openai_params": ["seed"], "additional_drop_params": ["top_k"]}), + Some(false), + ) + .unwrap(); + assert_eq!(sorted.openai, object(json!({"seed": 3}))); + assert!(sorted.provider.is_empty()); + assert_eq!(sorted.allowed, vec!["seed".to_string()]); + } + + #[rstest] + #[case::n_of_one(json!({"n": 1}), true)] + #[case::n_of_two(json!({"n": 2}), false)] + #[case::user(json!({"user": "u"}), true)] + fn unvalidated_params_and_one_choice_are_never_refused( + #[case] body: Value, + #[case] accepted: bool, + ) { + let result = sort(body, Some(false)); + assert_eq!(result.is_ok(), accepted, "{result:?}"); + if let Ok(sorted) = result { + assert!(sorted.openai.is_empty()); + } + } + + #[rstest] + fn python_dumps_matches_json_dumps() { + assert_eq!( + python_dumps(&json!({"b": "é", "a": [1, 2.5]})), + r#"{"b": "\u00e9", "a": [1, 2.5]}"# + ); + } +} diff --git a/litellm-rust/crates/llms/src/base_llm/chat/response_format.rs b/litellm-rust/crates/llms/src/base_llm/chat/response_format.rs new file mode 100644 index 00000000000..a7f139c123a --- /dev/null +++ b/litellm-rust/crates/llms/src/base_llm/chat/response_format.rs @@ -0,0 +1,445 @@ +//! `response_format` answers delivered through the `json_tool_call` tool, as Python's +//! `base_utils.py` reads them back: the schema's shape, the envelopes a model wraps the +//! answer in, and an answer split across several tool calls. + +use std::collections::BTreeSet; + +use serde_json::{Map, Value}; + +use crate::base_llm::chat::params::python_dumps; + +/// `ResponseFormatSchema`: what the requested schema declares, enough to tell an answer +/// from an envelope around it. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct ResponseFormatSchema { + pub property_names: BTreeSet, + pub required_names: BTreeSet, + pub allows_undeclared_keys: bool, +} + +impl ResponseFormatSchema { + /// `response_format_schema`: `None` unless `value` is an object whose `properties`, if + /// present, is one too. + pub fn from_schema(value: &Value) -> Option { + let document = value.as_object()?; + let properties = match document.get("properties") { + None => Map::new(), + Some(Value::Object(properties)) => properties.clone(), + Some(_) => return None, + }; + let required_names = match document.get("required") { + Some(Value::Array(items)) if items.iter().all(Value::is_string) => items + .iter() + .filter_map(Value::as_str) + .map(str::to_owned) + .collect(), + _ => BTreeSet::new(), + }; + let additional = document.get("additionalProperties"); + let allows_undeclared_keys = additional == Some(&Value::Bool(true)) + || additional.is_some_and(Value::is_object) + || document + .get("patternProperties") + .and_then(Value::as_object) + .is_some_and(|patterns| !patterns.is_empty()); + Some(Self { + property_names: properties.keys().cloned().collect(), + required_names, + allows_undeclared_keys, + }) + } + + fn accepts_a_lone_undeclared_key(&self) -> bool { + self.allows_undeclared_keys && self.required_names.is_empty() + } +} + +const ENVELOPE_NAME_KEYS: [&str; 2] = ["name", "tool_name"]; + +/// `_json_object_or_encoded_object`. +fn object_or_encoded_object(value: &Value) -> Option> { + match value { + Value::String(text) => match serde_json::from_str(text) { + Ok(Value::Object(object)) => Some(object), + _ => None, + }, + Value::Object(object) => Some(object.clone()), + _ => None, + } +} + +struct EnvelopeLayer { + inner: Map, + names_the_tool: bool, +} + +fn envelope_layer( + payload: &Map, + schema: &ResponseFormatSchema, + tool_name: &str, +) -> Option { + let name_keys: BTreeSet<&str> = ENVELOPE_NAME_KEYS + .into_iter() + .filter(|key| payload.get(*key).and_then(Value::as_str) == Some(tool_name)) + .collect(); + let label_keys: BTreeSet<&str> = match (name_keys.is_empty(), schema.required_names.is_empty()) + { + (false, _) => name_keys.clone(), + (true, true) => BTreeSet::new(), + (true, false) => payload + .iter() + .filter(|(key, value)| { + value.is_string() + && !schema.property_names.contains(*key) + && !schema.required_names.contains(*key) + }) + .map(|(key, _)| key.as_str()) + .collect(), + }; + let wrapper_keys: Vec<&String> = payload + .keys() + .filter(|key| !label_keys.contains(key.as_str())) + .collect(); + let [wrapper] = wrapper_keys.as_slice() else { + return None; + }; + if schema.property_names.contains(*wrapper) { + return None; + } + let wrapped = &payload[wrapper.as_str()]; + let inner = match name_keys.is_empty() { + false => object_or_encoded_object(wrapped), + true => wrapped.as_object().cloned(), + }?; + Some(EnvelopeLayer { + inner, + names_the_tool: !name_keys.is_empty(), + }) +} + +fn holds_the_schema_object(layer: &EnvelopeLayer, schema: &ResponseFormatSchema) -> bool { + layer.names_the_tool + || (!schema.accepts_a_lone_undeclared_key() + && layer + .inner + .keys() + .any(|key| schema.property_names.contains(key))) +} + +fn schema_object_inside_envelope( + payload: &Map, + schema: &ResponseFormatSchema, + tool_name: &str, +) -> Option> { + let mut layer = envelope_layer(payload, schema, tool_name); + let mut first = true; + while let Some(current) = layer { + if !first && schema.property_names.is_empty() { + return None; + } + first = false; + if holds_the_schema_object(¤t, schema) { + return Some(current.inner); + } + layer = envelope_layer(¤t.inner, schema, tool_name); + } + None +} + +/// `response_format_envelope_contents`. +pub fn envelope_contents( + payload: &Value, + schema: &ResponseFormatSchema, + tool_name: &str, +) -> Option { + let object = payload.as_object()?; + if let Some(inner) = schema_object_inside_envelope(object, schema, tool_name) { + return Some(Value::Object(inner)); + } + if schema.property_names.contains("values") { + return None; + } + object + .get("values") + .filter(|values| !values.is_null()) + .cloned() +} + +/// `unwrap_response_format_envelope`. +pub fn unwrap_envelope(payload: Value, schema: &ResponseFormatSchema, tool_name: &str) -> Value { + envelope_contents(&payload, schema, tool_name).unwrap_or(payload) +} + +fn field_names( + blocks: &[Map], + key: &str, + schema: &ResponseFormatSchema, +) -> Option> { + let names: Vec = blocks + .iter() + .filter_map(|block| block[key].as_str()) + .filter(|name| schema.property_names.contains(*name)) + .map(str::to_owned) + .collect(); + let distinct: BTreeSet<&String> = names.iter().collect(); + (distinct.len() == blocks.len()).then_some(names) +} + +fn is_a_label_every_block_repeats(blocks: &[Map], key: &str) -> bool { + blocks[0][key].is_string() && blocks.iter().all(|block| block[key] == blocks[0][key]) +} + +fn value_key(blocks: &[Map], keys: &[&String]) -> Option { + if let [only] = keys { + return Some((*only).clone()); + } + let varying: Vec<&&String> = keys + .iter() + .filter(|key| !is_a_label_every_block_repeats(blocks, key)) + .collect(); + match varying.as_slice() { + [only] => Some((**only).clone()), + _ => None, + } +} + +fn split_answer_fields( + blocks: &[Map], + schema: &ResponseFormatSchema, +) -> Option> { + let keys: BTreeSet<&String> = blocks[0].keys().collect(); + if keys.iter().any(|key| schema.property_names.contains(*key)) + || blocks + .iter() + .any(|block| block.keys().collect::>() != keys) + { + return None; + } + // Python iterates a frozenset here; the outcome only depends on there being one match. + let named: Vec<(&String, Vec)> = keys + .iter() + .filter_map(|key| Some((*key, field_names(blocks, key, schema)?))) + .collect(); + let [(name_key, names)] = named.as_slice() else { + return None; + }; + let rest: Vec<&String> = keys.iter().copied().filter(|key| key != name_key).collect(); + let value_key = value_key(blocks, &rest)?; + Some( + names + .iter() + .cloned() + .zip(blocks.iter().map(|block| block[value_key.as_str()].clone())) + .collect(), + ) +} + +fn merged_partial_answers( + blocks: &[Map], + schema: &ResponseFormatSchema, +) -> Option> { + let keys: BTreeSet<&String> = blocks.iter().flat_map(Map::keys).collect(); + if !keys.iter().all(|key| schema.property_names.contains(*key)) + || keys.len() != blocks.iter().map(Map::len).sum::() + { + return None; + } + Some( + blocks + .iter() + .flat_map(|block| block.iter().map(|(k, v)| (k.clone(), v.clone()))) + .collect(), + ) +} + +/// `split_response_format_answer`: one answer spread over several tool calls. +pub fn split_answer( + encoded_blocks: &[String], + schema: &ResponseFormatSchema, +) -> Option> { + let blocks: Vec> = encoded_blocks + .iter() + .map(|encoded| object_or_encoded_object(&Value::String(encoded.clone()))) + .collect::>()?; + if blocks.len() < 2 { + return None; + } + split_answer_fields(&blocks, schema).or_else(|| merged_partial_answers(&blocks, schema)) +} + +/// Anthropic's `_convert_tool_response_to_message`: the message content the `json_tool_call` +/// calls' arguments become. +pub fn json_tool_answer(arguments: &[String], schema: Option<&ResponseFormatSchema>) -> String { + let first = arguments.first().cloned().unwrap_or_default(); + if let Some(split) = schema.and_then(|schema| split_answer(arguments, schema)) { + return python_dumps(&Value::Object(split)); + } + let Ok(payload) = serde_json::from_str::(&first) else { + return first; + }; + match schema { + Some(schema) => python_dumps(&unwrap_envelope( + payload, + schema, + crate::base_llm::chat::params::RESPONSE_FORMAT_TOOL_NAME, + )), + None => { + let values = payload + .as_object() + .and_then(|object| object.get("values")) + .filter(|values| !values.is_null()) + .cloned(); + python_dumps(values.as_ref().unwrap_or(&payload)) + } + } +} + +/// `unpack_defs`: every `$ref` into `$defs` / `definitions` replaced by a copy of its +/// target, nested definitions shadowing outer ones, a circular reference left as it is. +pub fn unpack_defs(schema: &mut Value, defs: &Map) { + fn scoped(node: &Map, outer: &Map) -> Map { + let mut defs = outer.clone(); + for key in ["$defs", "definitions"] { + if let Some(Value::Object(local)) = node.get(key) { + defs.extend(local.iter().map(|(k, v)| (k.clone(), v.clone()))); + } + } + defs + } + fn walk(node: &mut Value, defs: &Map, chain: &BTreeSet) { + match node { + Value::Object(object) => { + if let Some(reference) = object.get("$ref").and_then(Value::as_str) { + let name = reference.rsplit('/').next().unwrap_or(reference).to_owned(); + if chain.contains(&name) { + return; + } + let Some(target) = defs.get(&name).cloned() else { + return; + }; + let child_defs = match &target { + Value::Object(target) => scoped(target, defs), + _ => defs.clone(), + }; + *node = target; + let mut chain = chain.clone(); + chain.insert(name); + walk(node, &child_defs, &chain); + return; + } + if object.contains_key("$ref") { + return; + } + let current = scoped(object, defs); + for value in object.values_mut() { + walk(value, ¤t, chain); + } + } + Value::Array(items) => { + for item in items { + walk(item, defs, chain); + } + } + _ => {} + } + } + let root = match schema { + Value::Object(object) => scoped(object, defs), + _ => defs.clone(), + }; + walk(schema, &root, &BTreeSet::new()); +} + +/// `unpack_legacy_defs`: refs into `definitions` or OpenAPI `components.schemas` inlined, +/// those blocks removed. `$defs` is left alone. +pub fn unpack_legacy_defs(schema: &mut Map) { + let has_components = schema + .get("components") + .and_then(|components| components.get("schemas")) + .is_some(); + if !schema.contains_key("definitions") && !has_components { + return; + } + let mut defs = match schema.remove("components") { + Some(Value::Object(mut components)) => match components.remove("schemas") { + Some(Value::Object(schemas)) => schemas, + _ => Map::new(), + }, + _ => Map::new(), + }; + if let Some(Value::Object(definitions)) = schema.remove("definitions") { + defs.extend(definitions); + } + let mut value = Value::Object(std::mem::take(schema)); + unpack_defs(&mut value, &defs); + if let Value::Object(object) = value { + *schema = object; + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + + fn schema(value: Value) -> ResponseFormatSchema { + ResponseFormatSchema::from_schema(&value).unwrap() + } + + #[rstest] + #[case::bare(json!({"name": "a"}), json!({"name": "a"}))] + #[case::named_envelope(json!({"name": "json_tool_call", "arguments": {"name": "a"}}), json!({"name": "a"}))] + #[case::encoded_envelope(json!({"tool_name": "json_tool_call", "parameters": "{\"name\": \"a\"}"}), json!({"name": "a"}))] + #[case::wrapper(json!({"answer": {"name": "a"}}), json!({"name": "a"}))] + #[case::values(json!({"values": {"x": 1}}), json!({"x": 1}))] + fn envelopes_are_unwrapped(#[case] payload: Value, #[case] expected: Value) { + let schema = schema( + json!({"type": "object", "properties": {"name": {"type": "string"}}, "required": ["name"]}), + ); + assert_eq!( + unwrap_envelope(payload, &schema, "json_tool_call"), + expected + ); + } + + #[rstest] + fn an_answer_split_over_named_fields_is_joined() { + let schema = schema(json!({"properties": {"a": {}, "b": {}}})); + let blocks = vec![ + r#"{"field": "a", "value": 1}"#.to_owned(), + r#"{"field": "b", "value": 2}"#.to_owned(), + ]; + assert_eq!( + split_answer(&blocks, &schema), + Some(json!({"a": 1, "b": 2}).as_object().unwrap().clone()) + ); + let partial = vec![r#"{"a": 1}"#.to_owned(), r#"{"b": 2}"#.to_owned()]; + assert_eq!( + json_tool_answer(&partial, Some(&schema)), + r#"{"a": 1, "b": 2}"# + ); + } + + #[rstest] + fn an_undecodable_answer_is_returned_as_sent() { + assert_eq!(json_tool_answer(&["{bad".to_owned()], None), "{bad"); + assert_eq!( + json_tool_answer(&[r#"{"values": [1]}"#.to_owned()], None), + "[1]" + ); + } + + #[rstest] + fn refs_are_inlined_and_cycles_left_alone() { + let mut value = + json!({"properties": {"a": {"$ref": "#/$defs/A"}, "n": {"$ref": "#/$defs/Node"}}}); + let defs = json!({"A": {"type": "string"}, "Node": {"properties": {"next": {"$ref": "#/$defs/Node"}}}}); + unpack_defs(&mut value, defs.as_object().unwrap()); + assert_eq!( + value, + json!({"properties": {"a": {"type": "string"}, "n": {"properties": {"next": {"$ref": "#/$defs/Node"}}}}}) + ); + } +} diff --git a/litellm-rust/crates/llms/src/base_llm/chat/streaming.rs b/litellm-rust/crates/llms/src/base_llm/chat/streaming.rs index 2faa5699751..1c938b33a1e 100644 --- a/litellm-rust/crates/llms/src/base_llm/chat/streaming.rs +++ b/litellm-rust/crates/llms/src/base_llm/chat/streaming.rs @@ -1,5 +1,7 @@ use std::collections::HashMap; +use crate::base_llm::chat::response_format::ResponseFormatSchema; + use bytes::Bytes; use futures_util::{StreamExt, stream::BoxStream}; use litellm_llms_types::formats::chat_completions::ChatCompletionChunk; @@ -35,6 +37,12 @@ pub struct StreamShape { pub json_mode: bool, pub speed: Option, pub tool_name_reverse_map: HashMap, + /// The schema of the `json_tool_call` tool a `response_format` added, which tells its + /// answer apart from an envelope around it. + pub response_format_schema: Option, + /// The text of a trailing assistant message sent with `prefix: true`, which Anthropic + /// continues and Python puts back in front of the answer. + pub prefix_prompt: Option, } /// A wire decoder paired with the iterator that turns its events into chat chunks. diff --git a/litellm-rust/crates/llms/src/base_llm/chat/transformation.rs b/litellm-rust/crates/llms/src/base_llm/chat/transformation.rs index e70001bf0be..77c89e0e85a 100644 --- a/litellm-rust/crates/llms/src/base_llm/chat/transformation.rs +++ b/litellm-rust/crates/llms/src/base_llm/chat/transformation.rs @@ -5,7 +5,10 @@ use serde_json::{Map, Value}; use crate::{ Error, - base_llm::chat::streaming::{ChatStream, StreamShape}, + base_llm::chat::{ + params::ChatRequestContext, + streaming::{ChatStream, StreamShape}, + }, }; /// The provider-shaped request body a config produces. Named rather than a bare @@ -14,6 +17,8 @@ use crate::{ pub struct ProviderChatRequestData { pub body: Value, pub stream_shape: StreamShape, + /// Headers the request needs beyond the config's defaults, such as `anthropic-beta`. + pub headers: Vec<(String, String)>, } /// The raw provider response body handed back to a config for normalization. @@ -42,8 +47,11 @@ pub struct Unsupported(pub &'static str); pub trait BaseConfig: Sync { fn secret_names(&self) -> Vec<&'static str>; - /// Supported OpenAI parameter names paired with their provider names. - fn supported_openai_param_mappings(&self) -> &'static [(&'static str, &'static str)]; + /// Supported OpenAI parameter names paired with their provider names, for a config + /// whose requests the host checks against an allowlist. + fn supported_openai_param_mappings(&self) -> &'static [(&'static str, &'static str)] { + &[] + } fn get_complete_url( &self, @@ -58,14 +66,23 @@ pub trait BaseConfig: Sync { model: &str, messages: Vec, optional_params: Map, + context: &ChatRequestContext<'_>, ) -> Result; + /// `shape` is what the request decided about its answer, the same one a stream reads. fn transform_response( &self, model: &str, response: ProviderChatResponseData, + shape: &StreamShape, ) -> Result; + /// The media URLs the messages point at that Python downloads while building this + /// provider's request. The host fetches them and inlines each as a data URL first. + fn remote_media(&self, _messages: &[ChatMessage]) -> Vec { + Vec::new() + } + /// `None` means this config has no streaming path yet, so the host keeps the request. fn model_response_iterator(&self, _shape: StreamShape) -> Option { None diff --git a/litellm-rust/crates/llms/src/base_llm/model_info.rs b/litellm-rust/crates/llms/src/base_llm/model_info.rs index a2861d138c7..3260c270306 100644 --- a/litellm-rust/crates/llms/src/base_llm/model_info.rs +++ b/litellm-rust/crates/llms/src/base_llm/model_info.rs @@ -1,11 +1,14 @@ //! What the model cost map says about a model, for configs whose request rules depend on it //! the way Python's `supports_*` helpers read `litellm.model_cost`. +use std::borrow::Cow; + use serde_json::{Map, Value}; pub trait ModelInfo: Send + Sync { - /// The cost map entry `model` names, if there is one. - fn entry(&self, model: &str) -> Option<&Map>; + /// The cost map entry `model` names, if there is one. A map held in memory lends it; one + /// read from elsewhere, such as Python's `litellm.model_cost`, hands over a copy. + fn entry(&self, model: &str) -> Option>>; /// Whether any entry of the map sets `key`. fn declares(&self, key: &str) -> bool; @@ -14,21 +17,22 @@ pub trait ModelInfo: Send + Sync { /// Python's `_supports_factory`: the entry sets `key` to `true`. An unknown model supports /// nothing. pub fn supports(info: Option<&dyn ModelInfo>, model: &str, key: &str) -> bool { - field(info, model, key).and_then(Value::as_bool) == Some(true) + field(info, model, key).and_then(|value| value.as_bool()) == Some(true) } /// Python's `is_explicitly_disabled_factory`: the entry sets `key` to `false`. An unknown /// model or an unset key is not disabled. pub fn explicitly_disabled(info: Option<&dyn ModelInfo>, model: &str, key: &str) -> bool { - field(info, model, key).and_then(Value::as_bool) == Some(false) + field(info, model, key).and_then(|value| value.as_bool()) == Some(false) } /// The value the entry for `model` sets for `key`. -pub fn field<'a>(info: Option<&'a dyn ModelInfo>, model: &str, key: &str) -> Option<&'a Value> { +pub fn field(info: Option<&dyn ModelInfo>, model: &str, key: &str) -> Option { info? .entry(model)? .get(key) .filter(|value| !value.is_null()) + .cloned() } /// Whether the map has an entry for `model`. diff --git a/litellm-rust/crates/llms/src/bedrock/chat/converse_messages.rs b/litellm-rust/crates/llms/src/bedrock/chat/converse_messages.rs new file mode 100644 index 00000000000..67cbd088a7a --- /dev/null +++ b/litellm-rust/crates/llms/src/bedrock/chat/converse_messages.rs @@ -0,0 +1,1343 @@ +//! Chat messages and tools as Converse takes them: Python's `_transform_system_message`, +//! `_neutralize_orphaned_tool_blocks`, `_bedrock_converse_messages_pt` and `_bedrock_tools_pt`. + +use std::collections::HashSet; + +use serde_json::{Map, Value, json}; +use sha2::{Digest, Sha256}; + +use crate::{ + Error, ErrorDetail, + anthropic::chat::messages::{CONVERTED_SYSTEM_NOTE, native_redacted_thinking_data}, + base_llm::chat::{params::truthy, response_format::unpack_defs}, + bedrock::chat::converse_params::{BedrockModel, make_valid_bedrock_tool_name}, +}; + +const SUPPORTED_IMAGE_FORMATS: [&str; 4] = ["png", "jpeg", "gif", "webp"]; +const SUPPORTED_DOCUMENT_FORMATS: [&str; 9] = [ + "pdf", "csv", "doc", "docx", "xls", "xlsx", "html", "txt", "md", +]; +const SUPPORTED_VIDEO_FORMATS: [&str; 9] = [ + "mp4", "mov", "mkv", "webm", "flv", "mpeg", "mpg", "wmv", "3gp", +]; +const DOCUMENT_PLACEHOLDER_TEXT: &str = "."; +const TOOL_RESULT_IMAGE_PLACEHOLDER: &str = + "[Tool returned an image - see the following user message]"; +const TOOL_USE_ID_MAX_LEN: usize = 64; +const TOOL_USE_ID_HASH_LEN: usize = 8; +const HASH_SAMPLE_BYTES: usize = 64 * 1024; +const BAD_MESSAGE: &str = "Invalid Message "; + +fn bad_request(message: impl Into) -> Error { + Error::InvalidRequest(ErrorDetail::Message(message.into())) +} + +fn role(message: &Value) -> &str { + message + .get("role") + .and_then(Value::as_str) + .unwrap_or_default() +} + +fn is_system(message: &Value) -> bool { + role(message) == "system" +} + +/// `CachePointBlock` for a `cache_control`, or nothing for a model that refuses them. +pub fn cache_point(control: Option<&Value>, model: &BedrockModel<'_>) -> Option { + control.filter(|control| !control.is_null())?; + if !model.accepts_cache_points() { + return None; + } + Some(json!({"cachePoint": cache_point_block(control, model)})) +} + +fn cache_point_block(control: Option<&Value>, model: &BedrockModel<'_>) -> Value { + let mut block = Map::from_iter([("type".to_owned(), json!("default"))]); + if let Some(ttl) = control + .and_then(|control| control.get("ttl")) + .and_then(Value::as_str) + .filter(|ttl| matches!(*ttl, "5m" | "1h")) + && model.cache_ttl_supported() + { + block.insert("ttl".into(), json!(ttl)); + } + Value::Object(block) +} + +/// The leading run of system messages as Converse `system` blocks. +fn system_blocks(leading: &[Value], model: &BedrockModel<'_>) -> Vec { + let mut blocks = Vec::new(); + for message in leading { + match message.get("content") { + Some(Value::String(text)) if !text.is_empty() => { + blocks.push(json!({"text": text})); + blocks.extend(cache_point(message.get("cache_control"), model)); + } + Some(Value::Array(parts)) => { + for part in parts { + let text = part.get("text").and_then(Value::as_str).unwrap_or_default(); + if part.get("type").and_then(Value::as_str) == Some("text") && !text.is_empty() + { + blocks.push(json!({"text": text})); + blocks.extend(cache_point(part.get("cache_control"), model)); + } + } + } + _ => {} + } + } + blocks +} + +fn assistant_has_tool_calls(message: &Value) -> bool { + role(message) == "assistant" && message.get("tool_calls").is_some_and(truthy) +} + +/// `_opens_with_tool_result`. +fn opens_with_tool_result(message: &Value) -> bool { + match role(message) { + "tool" => true, + "user" => { + let first = match message.get("content") { + Some(Value::Array(parts)) => parts.first(), + _ => None, + }; + first + .and_then(|part| part.get("type")) + .and_then(Value::as_str) + == Some("tool_result") + } + _ => false, + } +} + +/// `_reordered_around_tool_results`. +fn reordered_around_tool_results(messages: &[Value], index: usize) -> Vec { + let message = &messages[index]; + let system_run_start = |end: usize| { + (0..end) + .rev() + .find(|j| !is_system(&messages[*j])) + .map_or(0, |j| j + 1) + }; + if opens_with_tool_result(message) { + if index + 1 < messages.len() && opens_with_tool_result(&messages[index + 1]) { + return vec![message.clone()]; + } + let tool_run_start = (0..=index) + .rev() + .find(|j| !opens_with_tool_result(&messages[*j])) + .map_or(0, |j| j + 1); + let run_start = system_run_start(tool_run_start); + let run = &messages[run_start..tool_run_start]; + if !run.is_empty() && run_start >= 1 && assistant_has_tool_calls(&messages[run_start - 1]) { + let mut kept = vec![message.clone()]; + kept.extend(run.iter().cloned()); + return kept; + } + return vec![message.clone()]; + } + if !is_system(message) { + return vec![message.clone()]; + } + let run_start = system_run_start(index); + let run_end = (index..messages.len()) + .find(|j| !is_system(&messages[*j])) + .unwrap_or(messages.len()); + if run_end < messages.len() + && opens_with_tool_result(&messages[run_end]) + && run_start > 0 + && assistant_has_tool_calls(&messages[run_start - 1]) + { + return Vec::new(); + } + vec![message.clone()] +} + +/// `_system_role_message_as_user`: a mid-conversation system message as a user turn. +fn system_as_user(message: &Value) -> Option { + let block = |text: &str, cache_control: Option<&Value>| { + let mut block = Map::from_iter([ + ("type".to_owned(), json!("text")), + ("text".to_owned(), json!(text)), + ]); + if let Some(cache_control) = cache_control.filter(|c| !c.is_null()) { + block.insert("cache_control".into(), cache_control.clone()); + } + Value::Object(block) + }; + let texts: Vec = match message.get("content") { + Some(Value::String(text)) if !text.is_empty() => { + vec![block(text, message.get("cache_control"))] + } + Some(Value::Array(parts)) => parts + .iter() + .filter(|part| part.get("type").and_then(Value::as_str) == Some("text")) + .filter_map(|part| { + let text = part + .get("text") + .and_then(Value::as_str) + .filter(|t| !t.is_empty())?; + Some(block(text, part.get("cache_control"))) + }) + .collect(), + _ => Vec::new(), + }; + if texts.is_empty() { + return None; + } + let mut content = vec![json!({"type": "text", "text": CONVERTED_SYSTEM_NOTE})]; + content.extend(texts); + Some(json!({"role": "user", "content": content})) +} + +/// `_transform_system_message`: the conversation after the leading system run, and that run +/// as `system` blocks. +pub fn transform_system_message( + messages: Vec, + model: &BedrockModel<'_>, +) -> (Vec, Vec) { + let leading = messages + .iter() + .position(|message| !is_system(message)) + .unwrap_or(messages.len()); + let system = system_blocks(&messages[..leading], model); + let remaining = &messages[leading..]; + let kept = (0..remaining.len()) + .flat_map(|index| reordered_around_tool_results(remaining, index)) + .filter_map(|message| match is_system(&message) { + true => system_as_user(&message), + false => Some(message), + }) + .collect(); + (kept, system) +} + +/// `convert_content_list_to_str`. +fn content_text(message: &Value) -> String { + match message.get("content") { + Some(Value::String(text)) => text.clone(), + Some(Value::Array(parts)) => parts + .iter() + .filter_map(|part| part.get("text").and_then(Value::as_str)) + .collect(), + _ => String::new(), + } +} + +/// `_neutralize_orphaned_tool_blocks`: tool calls and results without `tools=` as text. +pub fn neutralize_orphaned_tool_blocks( + messages: Vec, + has_tools: bool, + guardrail_active: bool, +) -> Vec { + let orphaned = messages.iter().any(|message| { + assistant_has_tool_calls(message) || matches!(role(message), "tool" | "function") + }); + if has_tools || !orphaned { + return messages; + } + messages + .into_iter() + .map(|message| { + if assistant_has_tool_calls(&message) { + let base = content_text(&message); + let calls = message + .get("tool_calls") + .and_then(Value::as_array) + .into_iter() + .flatten() + .map(|call| { + let function = call.get("function"); + let name = function + .and_then(|f| f.get("name")) + .and_then(Value::as_str) + .filter(|name| !name.is_empty()) + .unwrap_or("unknown_tool"); + let arguments = function + .and_then(|f| f.get("arguments")) + .and_then(Value::as_str) + .unwrap_or_default(); + let label = match call + .get("id") + .and_then(Value::as_str) + .filter(|id| !id.is_empty()) + { + Some(id) => format!("tool call {id}"), + None => "tool call".to_owned(), + }; + format!("[{label}: {name}({arguments})]") + }); + let text: Vec = std::iter::once(base) + .chain(calls) + .filter(|part| !part.is_empty()) + .collect(); + return json!({"role": "assistant", "content": text.join("\n")}); + } + if matches!(role(&message), "tool" | "function") { + let target = ["tool_call_id", "name"] + .iter() + .find_map(|key| { + message + .get(*key) + .and_then(Value::as_str) + .filter(|v| !v.is_empty()) + }) + .unwrap_or("unknown"); + let rendered = content_text(&message).trim().to_owned(); + let rendered = match rendered.is_empty() { + true => "".to_owned(), + false => rendered, + }; + let text = format!("[tool result for {target}: {rendered}]"); + return match guardrail_active { + true => { + json!({"role": "user", "content": [{"type": "guarded_text", "text": text}]}) + } + false => json!({"role": "user", "content": text}), + }; + } + message + }) + .collect() +} + +/// `_convert_consecutive_user_messages_to_guarded_text`. +pub fn guard_trailing_user_messages(mut messages: Vec) -> Vec { + for message in messages.iter_mut().rev() { + if role(message) != "user" { + break; + } + match message.get_mut("content") { + Some(Value::Array(parts)) => { + if parts + .iter() + .any(|part| part.get("type").and_then(Value::as_str) == Some("guarded_text")) + { + continue; + } + for part in parts.iter_mut() { + if part.get("type").and_then(Value::as_str) == Some("text") { + let text = part.get("text").cloned().unwrap_or(Value::Null); + *part = json!({"type": "guarded_text", "text": text}); + } + } + } + Some(Value::String(text)) => { + let text = text.clone(); + message["content"] = json!([{"type": "guarded_text", "text": text}]); + } + _ => {} + } + } + messages +} + +/// `_sanitize_bedrock_tool_use_id`. +pub fn sanitize_tool_use_id(id: &str) -> String { + let sanitized: String = id + .chars() + .map( + |c| match c.is_ascii_alphanumeric() || matches!(c, '_' | '.' | ':' | '-') { + true => c, + false => '_', + }, + ) + .collect(); + if sanitized == id && sanitized.len() <= TOOL_USE_ID_MAX_LEN { + return sanitized; + } + let digest = hex(&Sha256::digest(id.as_bytes())); + let head: String = sanitized + .chars() + .take(TOOL_USE_ID_MAX_LEN - TOOL_USE_ID_HASH_LEN - 1) + .collect(); + format!("{head}_{}", &digest[..TOOL_USE_ID_HASH_LEN]) +} + +fn hex(bytes: &[u8]) -> String { + bytes.iter().map(|byte| format!("{byte:02x}")).collect() +} + +/// `mimetypes.guess_all_extensions` for the document types Converse takes. +fn document_format(mime_type: &str) -> Result { + let format = match mime_type { + "application/pdf" => "pdf", + "text/csv" => "csv", + "application/msword" => "doc", + "application/vnd.openxmlformats-officedocument.wordprocessingml.document" => "docx", + "application/vnd.ms-excel" => "xls", + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" => "xlsx", + "text/html" => "html", + "text/plain" => "txt", + "text/markdown" | "text/x-markdown" => "md", + _ => { + return Err(bad_request(format!( + "No supported extensions for MIME type: {mime_type}. Supported formats: {}", + python_list(&SUPPORTED_DOCUMENT_FORMATS) + ))); + } + }; + Ok(format.to_owned()) +} + +fn python_list(items: &[&str]) -> String { + format!( + "[{}]", + items + .iter() + .map(|item| format!("'{item}'")) + .collect::>() + .join(", ") + ) +} + +fn is_document_mime(mime_type: &str) -> bool { + mime_type.starts_with("application") || mime_type.starts_with("text") +} + +/// `BedrockImageProcessor.validate_format`. +fn validate_format(mime_type: &str, image_format: &str) -> Result { + if is_document_mime(mime_type) { + return document_format(mime_type); + } + let supported: Vec<&str> = SUPPORTED_VIDEO_FORMATS + .iter() + .chain(SUPPORTED_IMAGE_FORMATS.iter()) + .copied() + .collect(); + if !supported.contains(&image_format) { + return Err(bad_request(format!( + "Unsupported image format: {image_format}. Supported formats: {}", + python_list(&supported) + ))); + } + Ok(image_format.to_owned()) +} + +/// The name Python derives from a document's content: a hash of its first 64 KB and length. +fn document_hash(data: &str) -> String { + let normalized: String = data.split_whitespace().collect(); + let bytes = normalized.as_bytes(); + let mut hasher = Sha256::new(); + hasher.update(&bytes[..bytes.len().min(HASH_SAMPLE_BYTES)]); + hasher.update(bytes.len().to_string().as_bytes()); + hex(&hasher.finalize())[..16].to_owned() +} + +/// `BedrockImageProcessor.process_image_sync` over a URL the host already inlined. +fn media_block(url: &str, format: Option<&str>) -> Result { + if !url.contains("base64") { + return Err(match url.contains("http://") || url.contains("https://") { + true => Error::Unsupported("a media URL the host did not download"), + false => bad_request( + "Unsupported image type. Expected either image url or base64 encoded string", + ), + }); + } + let (metadata, data) = url.split_once(',').unwrap_or((url, "")); + let (mut mime_type, mut image_format) = match metadata + .strip_prefix("data:") + .and_then(|rest| rest.split(";base64").next()) + { + Some(mime) => { + let mime = mime.split(';').next().unwrap_or(mime).to_owned(); + let format = mime.split('/').nth(1).unwrap_or_default().to_owned(); + (mime, format) + } + None => ("image/jpeg".to_owned(), "jpeg".to_owned()), + }; + if let Some(format) = format.filter(|format| !format.is_empty()) { + mime_type = format.to_owned(); + image_format = mime_type.split('/').nth(1).unwrap_or_default().to_owned(); + } + let image_format = validate_format(&mime_type, &image_format)?; + let source = json!({"bytes": data}); + if is_document_mime(&mime_type) { + let name = format!("DocumentPDFmessages_{}_{image_format}", document_hash(data)); + return Ok(json!({"document": {"source": source, "format": image_format, "name": name}})); + } + if SUPPORTED_VIDEO_FORMATS + .iter() + .any(|video| image_format.starts_with(video)) + { + return Ok(json!({"video": {"source": source, "format": image_format}})); + } + Ok(json!({"image": {"source": source, "format": image_format}})) +} + +fn image_url_block(part: &Value) -> Result { + let image_url = part.get("image_url"); + let (url, format) = match image_url { + Some(Value::Object(image)) => ( + image.get("url").and_then(Value::as_str).unwrap_or_default(), + image.get("format").and_then(Value::as_str), + ), + Some(Value::String(url)) => (url.as_str(), None), + _ => ("", None), + }; + media_block(url, format) +} + +/// `process_file_message`. +fn file_block(part: &Value) -> Result { + let Some(file) = part.get("file").filter(|file| !file.is_null()) else { + return Err(bad_request( + "Content block has type='file' but is missing the required 'file' field", + )); + }; + let source = ["file_id", "file_data"].iter().find_map(|key| { + file.get(*key) + .and_then(Value::as_str) + .filter(|v| !v.is_empty()) + }); + let Some(source) = source else { + return Err(bad_request(format!( + "file_data and file_id cannot both be None. Got={part}" + ))); + }; + media_block(source, file.get("format").and_then(Value::as_str)) +} + +/// `_process_document_message`. +fn document_block(part: &Value) -> Result { + let source = part.get("source").cloned().unwrap_or(Value::Null); + let source_type = source + .get("type") + .and_then(Value::as_str) + .unwrap_or_default(); + if source_type != "base64" { + return Err(bad_request(format!( + "Bedrock Converse only supports base64-encoded document sources, got '{source_type}'. Please convert the document to base64 before sending to Bedrock." + ))); + } + let media_type = source + .get("media_type") + .and_then(Value::as_str) + .unwrap_or_default(); + let data = source + .get("data") + .and_then(Value::as_str) + .unwrap_or_default(); + let format = validate_format(media_type, media_type.split('/').nth(1).unwrap_or_default())?; + let name = format!("Document_{}_{format}", document_hash(data)); + Ok(json!({"document": {"source": {"bytes": data}, "format": format, "name": name}})) +} + +/// `skip_empty_text_blocks` for a list: whitespace-only text parts go. +fn non_empty_parts(parts: &[Value]) -> Vec { + parts + .iter() + .filter(|part| { + !(part.get("type").and_then(Value::as_str) == Some("text") + && part + .get("text") + .and_then(Value::as_str) + .unwrap_or_default() + .trim() + .is_empty()) + }) + .cloned() + .collect() +} + +fn user_blocks(message: &Value, model: &BedrockModel<'_>) -> Result, Error> { + let mut blocks = Vec::new(); + match message.get("content") { + Some(Value::Array(parts)) => { + for part in non_empty_parts(parts) { + if !part.is_object() { + continue; + } + let text = || part.get("text").cloned().unwrap_or(Value::Null); + match part.get("type").and_then(Value::as_str) { + Some("text" | "grounding_source" | "query") => { + blocks.push(json!({"text": text()})) + } + Some("guarded_text") => { + blocks.push(json!({"guardContent": {"text": {"text": text()}}})) + } + Some("image_url") => blocks.push(image_url_block(&part)?), + Some("file") => blocks.push(file_block(&part)?), + Some("document") => blocks.push(document_block(&part)?), + _ => {} + } + blocks.extend(cache_point(part.get("cache_control"), model)); + } + } + Some(Value::String(text)) if !text.is_empty() => { + blocks.push(json!({"text": text})); + blocks.extend(cache_point(message.get("cache_control"), model)); + } + _ => {} + } + Ok(blocks) +} + +/// `_parse_bedrock_tool_result_content_list` item by item. +fn tool_result_parts(content: &[Value]) -> Result, Error> { + let mut parts = Vec::new(); + for item in content { + match item.get("type").and_then(Value::as_str) { + Some("text") => { + parts.push(json!({"text": item.get("text").cloned().unwrap_or(Value::Null)})) + } + Some(kind @ ("image_url" | "file")) => { + let block = match kind { + "image_url" => image_url_block(item)?, + _ => file_block(item)?, + }; + for key in ["image", "document"] { + if let Some(media) = block.get(key) { + parts.push(json!({key: media})); + } + } + } + Some("tool_reference") => { + if let Some(name) = item.get("tool_name").and_then(Value::as_str) { + parts.push(json!({"text": name})); + } + } + _ => {} + } + } + Ok(parts) +} + +/// `_convert_to_bedrock_tool_call_result`. +fn tool_result(message: &Value) -> Result { + let search_results: Vec = message + .get("search_results") + .and_then(Value::as_array) + .into_iter() + .flatten() + .filter(|result| result.is_object()) + .map(|result| json!({"searchResult": result})) + .collect(); + let used_search_results = !search_results.is_empty(); + let content = match used_search_results { + true => search_results, + false => match message.get("content") { + Some(Value::String(text)) => vec![json!({"text": text})], + Some(Value::Array(items)) => tool_result_parts(items)?, + _ => Vec::new(), + }, + }; + let id = match message.get("tool_call_id") { + Some(Value::String(id)) => id.clone(), + Some(Value::Null) => "None".to_owned(), + Some(other) => other.to_string(), + None => uuid::Uuid::new_v4().to_string(), + }; + let mut result = Map::from_iter([ + ("content".to_owned(), Value::Array(content)), + ("toolUseId".to_owned(), json!(sanitize_tool_use_id(&id))), + ]); + if used_search_results { + result.insert("status".into(), json!("success")); + } + Ok(json!({"toolResult": result})) +} + +/// `split_concatenated_json_objects`. +fn concatenated_objects(raw: &str) -> Vec { + serde_json::Deserializer::from_str(raw.trim()) + .into_iter::() + .map_while(Result::ok) + .map(|value| match value { + Value::Object(_) => value, + _ => json!({}), + }) + .collect() +} + +/// `_convert_to_bedrock_tool_call_invoke`. +fn tool_uses(calls: &[Value], model: &BedrockModel<'_>) -> Result, Error> { + let mut blocks = Vec::new(); + for call in calls.iter().filter(|call| call.get("function").is_some()) { + let id = call.get("id").and_then(Value::as_str).unwrap_or_default(); + let function = &call["function"]; + let name = make_valid_bedrock_tool_name( + function + .get("name") + .and_then(Value::as_str) + .unwrap_or_default(), + ); + let arguments = function + .get("arguments") + .and_then(Value::as_str) + .unwrap_or_default(); + let cache = cache_point(call.get("cache_control"), model); + if arguments.trim().is_empty() { + blocks.push(json!({"toolUse": {"input": {}, "name": name, "toolUseId": sanitize_tool_use_id(id)}})); + } else { + match serde_json::from_str::(arguments) { + Ok(parsed) => { + let input = match parsed { + Value::Object(_) => parsed, + _ => json!({}), + }; + blocks.push(json!({"toolUse": {"input": input, "name": name, "toolUseId": sanitize_tool_use_id(id)}})); + } + Err(_) => { + let objects = concatenated_objects(arguments); + if objects.is_empty() { + blocks.push(json!({"toolUse": {"input": {}, "name": name, "toolUseId": sanitize_tool_use_id(id)}})); + } else { + for (index, object) in objects.into_iter().enumerate() { + let block_id = match index { + 0 => id.to_owned(), + _ => format!("{id}_{index}"), + }; + blocks.push(json!({"toolUse": {"input": object, "name": name, "toolUseId": sanitize_tool_use_id(&block_id)}})); + } + } + } + } + } + blocks.extend(cache); + } + Ok(blocks) +} + +/// `translate_thinking_blocks_to_reasoning_content_blocks` and +/// `add_thinking_blocks_to_assistant_content`. +fn add_thinking_blocks(blocks: &[Value], to_claude: bool, out: &mut Vec) { + let mut reasoning = Vec::new(); + for block in blocks { + if to_claude && crate::anthropic::chat::messages::is_encrypted_reasoning(block) { + continue; + } + let thinking = block + .get("thinking") + .and_then(Value::as_str) + .unwrap_or_default(); + let data = block + .get("data") + .and_then(Value::as_str) + .filter(|d| !d.trim().is_empty()); + if block.get("type").and_then(Value::as_str) == Some("redacted_thinking") + && to_claude + && let Some(data) = data + { + use base64::Engine; + let content = base64::engine::general_purpose::STANDARD + .encode(native_redacted_thinking_data(data).as_bytes()); + reasoning.push(json!({"reasoningContent": {"redactedContent": content}})); + continue; + } + match block.get("signature").filter(|s| !s.is_null()) { + Some(signature) if !signature.as_str().is_some_and(str::is_empty) => reasoning.push( + json!({"reasoningContent": {"reasoningText": {"text": thinking, "signature": signature}}}), + ), + _ => { + if !thinking.trim().is_empty() { + out.push(json!({"text": thinking})); + } + } + } + } + out.extend(reasoning); +} + +fn assistant_blocks(message: &Value, model: &BedrockModel<'_>) -> Result, Error> { + let mut blocks = Vec::new(); + let to_claude = model.accepts_redacted_thinking(); + if let Some(Value::Array(thinking)) = message.get("thinking_blocks") { + add_thinking_blocks(thinking, to_claude, &mut blocks); + } + match message.get("content") { + Some(Value::Array(parts)) => { + let parts = non_empty_parts(parts); + let mut converted = Vec::new(); + for part in parts.iter().filter(|part| part.is_object()) { + match part.get("type").and_then(Value::as_str) { + Some("thinking" | "redacted_thinking") => { + add_thinking_blocks(std::slice::from_ref(part), to_claude, &mut converted) + } + Some("text") => { + let text = part.get("text").and_then(Value::as_str).unwrap_or_default(); + if !text.trim().is_empty() { + converted.push(json!({"text": text})); + } + } + Some("image_url") => converted.push(image_url_block(part)?), + _ => {} + } + converted.extend(cache_point(part.get("cache_control"), model)); + } + blocks.extend(converted); + } + Some(Value::String(text)) => { + let non_content = message.get("tool_calls").is_some_and(truthy) + || message.get("function_call").is_some_and(truthy); + if !text.trim().is_empty() { + blocks.push(json!({"text": text})); + } + // An all-whitespace string becomes `None` when the message carries tool calls, + // and a `None` content adds no cache point. + if !(text.trim().is_empty() && non_content) { + blocks.extend(cache_point(message.get("cache_control"), model)); + } + } + _ => {} + } + if let Some(Value::Array(calls)) = message.get("tool_calls") + && !calls.is_empty() + { + blocks.extend(tool_uses(calls, model)?); + } + Ok(blocks) +} + +/// `_deduplicate_bedrock_content_blocks`. +fn dedupe(blocks: Vec, key: &str) -> Vec { + let mut seen: HashSet = HashSet::new(); + blocks + .into_iter() + .filter(|block| { + match block + .get(key) + .and_then(|keyed| keyed.get("toolUseId")) + .and_then(Value::as_str) + .filter(|id| !id.is_empty()) + { + Some(id) => seen.insert(id.to_owned()), + None => true, + } + }) + .collect() +} + +/// `_sort_bedrock_assistant_content_blocks`. +fn sorted_assistant_blocks(mut blocks: Vec) -> Vec { + blocks.sort_by_key(|block| { + if block.get("reasoningContent").is_some() { + 0 + } else if block.get("toolUse").is_some() || block.get("cachePoint").is_some() { + 2 + } else { + 1 + } + }); + blocks +} + +fn push_user_turn(contents: &mut Vec, blocks: Vec) { + if let Some(last) = contents.last_mut() + && last["role"] == "user" + && let Some(Value::Array(existing)) = last.get_mut("content") + { + existing.extend(blocks); + return; + } + contents.push(json!({"role": "user", "content": blocks})); +} + +/// `_bedrock_converse_messages_pt`. +pub fn converse_messages( + messages: &[Value], + model: &BedrockModel<'_>, +) -> Result, Error> { + if messages.is_empty() { + return Err(bad_request(format!( + "{BAD_MESSAGE}bedrock requires at least one non-system message" + ))); + } + let mut contents: Vec = Vec::new(); + let mut index = 0; + while index < messages.len() { + let start = index; + let mut user_content = Vec::new(); + while index < messages.len() && role(&messages[index]) == "user" { + user_content.extend(user_blocks(&messages[index], model)?); + index += 1; + } + if !user_content.is_empty() { + push_user_turn(&mut contents, user_content); + } + let mut tool_content = Vec::new(); + while index < messages.len() && role(&messages[index]) == "tool" { + let message = &messages[index]; + tool_content.push(tool_result(message)?); + let control = message + .get("cache_control") + .filter(|c| !c.is_null()) + .or_else(|| { + message + .get("content") + .and_then(Value::as_array) + .into_iter() + .flatten() + .find_map(|part| part.get("cache_control").filter(|c| !c.is_null())) + }); + tool_content.extend(cache_point(control, model)); + index += 1; + } + let tool_content = dedupe(tool_content, "toolResult"); + if !tool_content.is_empty() { + push_user_turn(&mut contents, tool_content); + } + let mut assistant_content = Vec::new(); + while index < messages.len() && role(&messages[index]) == "assistant" { + assistant_content.extend(assistant_blocks(&messages[index], model)?); + index += 1; + } + let assistant_content = sorted_assistant_blocks(dedupe(assistant_content, "toolUse")); + if !assistant_content.is_empty() { + contents.push(json!({"role": "assistant", "content": assistant_content})); + } + if index == start { + return Err(bad_request(format!( + "{BAD_MESSAGE}passed in {}", + crate::base_llm::chat::params::python_dumps(&messages[index]) + ))); + } + } + rename_duplicate_documents(&mut contents); + let mut contents: Vec = contents + .into_iter() + .map(with_text_when_document_only) + .collect(); + if model.accepts_redacted_thinking() + && let Some(last) = contents.last_mut() + && last["role"] == "assistant" + && let Some(Value::Array(blocks)) = last.get_mut("content") + && let Some(position) = blocks.iter().rposition(|block| block.get("text").is_some()) + { + let text = blocks[position]["text"] + .as_str() + .unwrap_or_default() + .trim_end() + .to_owned(); + blocks[position] = json!({"text": text}); + } + if !model.supports_tool_result_images() { + contents = contents + .into_iter() + .map(images_beside_tool_results) + .collect(); + } + Ok(contents) +} + +/// `_rename_duplicate_bedrock_document_names`. +fn rename_duplicate_documents(contents: &mut [Value]) { + let names = |contents: &[Value]| -> Vec { + contents + .iter() + .flat_map(|message| message["content"].as_array().cloned().unwrap_or_default()) + .filter_map(|block| { + block + .get("document")? + .get("name")? + .as_str() + .map(str::to_owned) + }) + .filter(|name| !name.is_empty()) + .collect() + }; + let mut used: HashSet = names(contents).into_iter().collect(); + let mut counts: std::collections::HashMap = std::collections::HashMap::new(); + for message in contents.iter_mut() { + let Some(Value::Array(blocks)) = message.get_mut("content") else { + continue; + }; + for block in blocks { + let Some(Value::Object(document)) = block.get_mut("document") else { + continue; + }; + let Some(name) = document + .get("name") + .and_then(Value::as_str) + .map(str::to_owned) + else { + continue; + }; + if name.is_empty() { + continue; + } + let count = counts.entry(name.clone()).or_insert(0); + *count += 1; + if *count > 1 { + let mut suffix = *count; + let mut renamed = format!("{name}_{suffix}"); + while used.contains(&renamed) { + suffix += 1; + renamed = format!("{name}_{suffix}"); + } + used.insert(renamed.clone()); + document.insert("name".into(), json!(renamed)); + } + } + } +} + +/// `_with_text_when_document_only`. +fn with_text_when_document_only(message: Value) -> Value { + let blocks = message["content"].as_array().cloned().unwrap_or_default(); + let needs_text = message["role"] == "user" + && blocks.iter().any(|block| block.get("document").is_some()) + && blocks.iter().all(|block| block.get("text").is_none()); + if !needs_text { + return message; + } + let cut = match blocks + .last() + .is_some_and(|block| block.get("cachePoint").is_some()) + { + true => blocks.len() - 1, + false => blocks.len(), + }; + let mut content = blocks[..cut].to_vec(); + content.push(json!({"text": DOCUMENT_PLACEHOLDER_TEXT})); + content.extend(blocks[cut..].iter().cloned()); + json!({"role": "user", "content": content}) +} + +/// `place_tool_result_images` for a model that refuses an image inside a tool result. +fn images_beside_tool_results(message: Value) -> Value { + let Some(blocks) = message["content"].as_array() else { + return message; + }; + let mut rewritten = Vec::new(); + let mut changed = false; + for block in blocks { + let Some(result) = block.get("toolResult") else { + rewritten.push(block.clone()); + continue; + }; + let parts = result["content"].as_array().cloned().unwrap_or_default(); + let images: Vec = parts + .iter() + .filter_map(|part| part.get("image").cloned()) + .collect(); + if images.is_empty() { + rewritten.push(block.clone()); + continue; + } + changed = true; + let mut kept: Vec = parts + .into_iter() + .filter(|part| part.get("image").is_none()) + .collect(); + if kept.is_empty() { + kept.push(json!({"text": TOOL_RESULT_IMAGE_PLACEHOLDER})); + } + let mut result = result.clone(); + result["content"] = Value::Array(kept); + rewritten.push(json!({"toolResult": result})); + rewritten.extend(images.into_iter().map(|image| json!({"image": image}))); + } + match changed { + true => json!({"role": message["role"].clone(), "content": rewritten}), + false => message, + } +} + +/// `normalize_json_schema_custom_types_to_object`. +fn normalize_custom_types(schema: &mut Value) { + let Value::Object(node) = schema else { return }; + if node.get("type").and_then(Value::as_str) == Some("custom") { + node.insert("type".into(), json!("object")); + } + for key in ["items", "additionalProperties"] { + if let Some(child @ Value::Object(_)) = node.get_mut(key) { + normalize_custom_types(child); + } + } + for key in ["properties", "$defs", "definitions"] { + if let Some(Value::Object(children)) = node.get_mut(key) { + children.values_mut().for_each(normalize_custom_types); + } + } + for key in ["anyOf", "allOf", "oneOf"] { + if let Some(Value::Array(children)) = node.get_mut(key) { + children.iter_mut().for_each(normalize_custom_types); + } + } +} + +/// `_bedrock_tools_pt`, with the names it rewrote mapped back for the response. +pub fn converse_tools( + tools: &[Value], + model: &BedrockModel<'_>, +) -> Result<(Vec, std::collections::HashMap), Error> { + let strict_supported = model.supports_strict_tools(); + let mut blocks = Vec::new(); + let mut renamed = std::collections::HashMap::new(); + for (index, tool) in tools.iter().enumerate() { + let Some(object) = tool.as_object() else { + continue; + }; + if ["systemTool", "toolSpec", "cachePoint"] + .iter() + .any(|key| object.contains_key(*key)) + { + blocks.push(tool.clone()); + continue; + } + let kind = object + .get("type") + .and_then(Value::as_str) + .unwrap_or_default(); + if kind.starts_with("computer_") { + return Err(Error::Unsupported( + "a computer use tool on Bedrock Converse", + )); + } + if matches!( + kind, + "tool_search_tool_regex_20251119" | "tool_search_tool_bm25_20251119" + ) { + continue; + } + if !object.contains_key("function") && !object.contains_key("input_schema") { + continue; + } + let default_schema = json!({"type": "object", "properties": {}}); + let (mut parameters, raw_name, description) = + if object.contains_key("input_schema") && !object.contains_key("function") { + ( + object + .get("input_schema") + .filter(|v| truthy(v)) + .cloned() + .unwrap_or(default_schema), + object + .get("name") + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + object.get("description").cloned(), + ) + } else { + let function = object.get("function").cloned().unwrap_or(Value::Null); + ( + function + .get("parameters") + .cloned() + .unwrap_or(default_schema), + function + .get("name") + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + function.get("description").cloned(), + ) + }; + let raw_name = match raw_name.trim().is_empty() { + true => format!("litellm_unnamed_tool_{index}"), + false => raw_name, + }; + let name = make_valid_bedrock_tool_name(&raw_name); + if name != raw_name { + renamed.insert(name.clone(), raw_name.clone()); + } + let description = match description { + Some(Value::String(text)) if !text.is_empty() => json!(text), + Some(other) if truthy(&other) => other, + _ => json!(name), + }; + let defs = match parameters.as_object_mut().and_then(|p| p.remove("$defs")) { + Some(Value::Object(defs)) => defs, + _ => Map::new(), + }; + unpack_defs(&mut parameters, &defs); + normalize_custom_types(&mut parameters); + let valid_roots = [ + "array", "boolean", "integer", "null", "number", "object", "string", + ]; + if !parameters + .get("type") + .and_then(Value::as_str) + .is_some_and(|kind| valid_roots.contains(&kind)) + && let Value::Object(object) = &mut parameters + { + object.insert("type".into(), json!("object")); + } + let mut json_schema = Map::from_iter([ + ( + "type".to_owned(), + parameters.get("type").cloned().unwrap_or(Value::Null), + ), + ( + "properties".to_owned(), + parameters + .get("properties") + .cloned() + .unwrap_or_else(|| json!({})), + ), + ( + "required".to_owned(), + parameters + .get("required") + .cloned() + .unwrap_or_else(|| json!([])), + ), + ]); + if strict_supported + && let Some(additional) = parameters + .get("additionalProperties") + .filter(|v| !v.is_null()) + { + json_schema.insert("additionalProperties".into(), additional.clone()); + } + let mut spec = Map::from_iter([ + ("inputSchema".to_owned(), json!({"json": json_schema})), + ("name".to_owned(), json!(name)), + ("description".to_owned(), description), + ]); + let strict = object.get("function").and_then(|f| f.get("strict")); + if strict_supported && strict.is_some_and(truthy) { + spec.insert("strict".into(), strict.cloned().unwrap_or(Value::Null)); + } + blocks.push(json!({"toolSpec": spec})); + if let Some(control) = object.get("cache_control").filter(|c| !c.is_null()) + && model.accepts_cache_points() + && control + .get("type") + .and_then(Value::as_str) + .unwrap_or("ephemeral") + == "ephemeral" + { + blocks.push(json!({"cachePoint": cache_point_block(Some(control), model)})); + } + } + Ok((blocks, renamed)) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::base_llm::chat::params::ChatRequestContext; + + fn model() -> BedrockModel<'static> { + BedrockModel::new( + "anthropic.claude-sonnet-4-5-v1:0", + &ChatRequestContext { + drop_params: Some(false), + model_info: None, + }, + ) + } + + #[rstest] + #[case("call_1", "call_1")] + #[case("call 1", "call_1_")] + fn tool_use_ids_are_sanitized(#[case] id: &str, #[case] prefix: &str) { + let sanitized = sanitize_tool_use_id(id); + assert!(sanitized.starts_with(prefix)); + if id != prefix { + assert_eq!(sanitized.len(), prefix.len() + TOOL_USE_ID_HASH_LEN); + } + } + + #[rstest] + fn a_tool_round_trip_becomes_tool_use_and_tool_result() { + let messages = vec![ + json!({"role": "user", "content": "weather?"}), + json!({"role": "assistant", "content": "checking", "tool_calls": [ + {"id": "c1", "type": "function", "function": {"name": "get.weather", "arguments": "{\"city\":\"Paris\"}"}} + ]}), + json!({"role": "tool", "tool_call_id": "c1", "content": "sunny"}), + ]; + let converted = converse_messages(&messages, &model()).unwrap(); + assert_eq!( + converted, + vec![ + json!({"role": "user", "content": [{"text": "weather?"}]}), + json!({"role": "assistant", "content": [ + {"text": "checking"}, + {"toolUse": {"input": {"city": "Paris"}, "name": "get_weather", "toolUseId": "c1"}} + ]}), + json!({"role": "user", "content": [{"toolResult": {"content": [{"text": "sunny"}], "toolUseId": "c1"}}]}), + ] + ); + } + + #[rstest] + fn images_and_documents_become_blocks() { + let messages = vec![json!({"role": "user", "content": [ + {"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}}, + {"type": "file", "file": {"file_data": "data:application/pdf;base64,JVBE"}, "cache_control": {"type": "ephemeral"}} + ]})]; + let converted = converse_messages(&messages, &model()).unwrap(); + let content = converted[0]["content"].as_array().unwrap(); + assert_eq!( + content[0], + json!({"image": {"source": {"bytes": "AAAA"}, "format": "png"}}) + ); + assert_eq!(content[1]["document"]["format"], "pdf"); + assert!( + content[1]["document"]["name"] + .as_str() + .unwrap() + .starts_with("DocumentPDFmessages_") + ); + assert_eq!(content[2], json!({"text": "."})); + assert_eq!(content[3], json!({"cachePoint": {"type": "default"}})); + } + + #[rstest] + fn system_messages_become_system_blocks_and_operator_notes() { + let (messages, system) = transform_system_message( + vec![ + json!({"role": "system", "content": "be terse", "cache_control": {"type": "ephemeral"}}), + json!({"role": "user", "content": "hi"}), + json!({"role": "system", "content": "later"}), + ], + &model(), + ); + assert_eq!( + system, + vec![ + json!({"text": "be terse"}), + json!({"cachePoint": {"type": "default"}}) + ] + ); + assert_eq!(messages[1]["content"][0]["text"], CONVERTED_SYSTEM_NOTE); + assert_eq!(messages[1]["content"][1]["text"], "later"); + } + + #[rstest] + fn orphaned_tool_blocks_become_text() { + let messages = neutralize_orphaned_tool_blocks( + vec![ + json!({"role": "assistant", "tool_calls": [{"id": "c", "function": {"name": "f", "arguments": "{}"}}]}), + json!({"role": "tool", "tool_call_id": "c", "content": "ok"}), + ], + false, + false, + ); + assert_eq!( + messages[0], + json!({"role": "assistant", "content": "[tool call c: f({})]"}) + ); + assert_eq!( + messages[1], + json!({"role": "user", "content": "[tool result for c: ok]"}) + ); + } + + #[rstest] + fn tools_become_tool_specs() { + let (tools, renamed) = converse_tools( + &[json!({"type": "function", "function": {"name": "get.weather", "parameters": { + "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] + }}})], + &model(), + ) + .unwrap(); + assert_eq!( + tools, + vec![ + json!({"toolSpec": {"inputSchema": {"json": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}, "name": "get_weather", "description": "get_weather"}}) + ] + ); + assert_eq!(renamed["get_weather"], "get.weather"); + } +} diff --git a/litellm-rust/crates/llms/src/bedrock/chat/converse_params.rs b/litellm-rust/crates/llms/src/bedrock/chat/converse_params.rs new file mode 100644 index 00000000000..04c44d8f16c --- /dev/null +++ b/litellm-rust/crates/llms/src/bedrock/chat/converse_params.rs @@ -0,0 +1,1039 @@ +//! Python's `AmazonConverseConfig.get_supported_openai_params` and `map_openai_params`, and +//! the Bedrock model-id reads behind them. + +use serde_json::{Map, Value, json}; + +use crate::{ + Error, ErrorDetail, + anthropic::chat::params::{ + ModelCapabilities, apply_sampling_param, cap_thinking_budget, map_context_management, + map_reasoning_effort, output_config_effort, python_str, translate_legacy_thinking, + unmapped_reasoning_effort, + }, + base_llm::chat::{ + params::{ + ChatRequestContext, DEFAULT_MAX_TOKENS, DropPolicy, RESPONSE_FORMAT_TOOL_NAME, + capability, is_thinking_enabled, model_field, ordered, python_dumps, unsupported_value, + }, + response_format::ResponseFormatSchema, + }, +}; + +/// `get_bedrock_cross_region_inference_regions`. +const CROSS_REGION_PREFIXES: [&str; 7] = ["global", "us", "eu", "apac", "jp", "au", "us-gov"]; + +/// `BEDROCK_MIN_THINKING_BUDGET_TOKENS`. +const MIN_THINKING_BUDGET_TOKENS: i64 = 1024; + +/// `BEDROCK_OPENAI_COMPAT_MIN_MAX_TOKENS`. +const OPENAI_COMPAT_MIN_MAX_TOKENS: i64 = 16; + +/// `_BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER`. +const OUTPUT_CONFIG_EFFORT_ORDER: [&str; 5] = ["low", "medium", "high", "xhigh", "max"]; + +/// `strip_bedrock_routing_prefix`: each prefix in turn, as Python applies them. +pub fn strip_bedrock_routing_prefix(model: &str) -> String { + let mut model = model.to_owned(); + for prefix in [ + "bedrock/", + "chat_completions/", + "converse/", + "invoke/", + "openai/", + "mantle/", + "nova-2/", + "nova/", + ] { + if model.starts_with(prefix) { + model = model + .split_once('/') + .map(|(_, rest)| rest.to_owned()) + .unwrap_or_default(); + } + } + model +} + +/// `strip_bedrock_throughput_suffix`: `::k` loses its throughput and `[...]` is cut. +fn strip_throughput_suffix(model: &str) -> String { + let mut model = model.to_owned(); + if let Some(head) = model.strip_suffix('k') + && let Some((rest, digits)) = head.rsplit_once(':') + && !digits.is_empty() + && digits.chars().all(|c| c.is_ascii_digit()) + && let Some((_, minor)) = rest.rsplit_once(':') + && !minor.is_empty() + && minor.chars().all(|c| c.is_ascii_digit()) + { + model = rest.to_owned(); + } + if model.ends_with(']') + && let Some(open) = model.rfind('[') + && model[open + 1..model.len() - 1] + .chars() + .all(|c| c.is_alphanumeric() || c == '_') + && open + 1 < model.len() - 1 + { + model.truncate(open); + } + model +} + +/// Whether `segment` reads as an AWS region, such as `us-east-1` or `us-gov-west-1`. +fn is_aws_region(segment: &str) -> bool { + let parts: Vec<&str> = segment.split('-').collect(); + parts.len() >= 3 + && parts[0].len() == 2 + && parts[0].chars().all(|c| c.is_ascii_lowercase()) + && parts + .last() + .is_some_and(|last| !last.is_empty() && last.chars().all(|c| c.is_ascii_digit())) + && parts[1..parts.len() - 1] + .iter() + .all(|part| !part.is_empty() && part.chars().all(|c| c.is_ascii_lowercase())) +} + +/// `get_bedrock_base_model`. +pub fn bedrock_base_model(model: &str) -> String { + let mut stripped = model; + for prefix in ["bedrock/converse/", "bedrock/", "converse/"] { + if let Some(rest) = stripped.strip_prefix(prefix) { + stripped = rest; + break; + } + } + if stripped.starts_with("nova-2/") { + return "amazon.nova-2-custom".into(); + } + if stripped.starts_with("nova/") { + return "amazon.nova-custom".into(); + } + let mut model = strip_bedrock_routing_prefix(model); + if model.to_lowercase().contains("arn") { + model = model.rsplit('/').next().unwrap_or_default().to_owned(); + } + let model = strip_throughput_suffix(&model); + if let Some((region, rest)) = model.split_once('.') + && CROSS_REGION_PREFIXES.contains(®ion) + { + return rest.to_owned(); + } + if let Some((region, rest)) = model.split_once('/') + && is_aws_region(region) + { + return rest.to_owned(); + } + model +} + +/// `_is_nova_2_model`. +fn is_nova_2_model(model: &str) -> bool { + let mut stripped = model; + for prefix in ["bedrock/converse/", "bedrock/", "converse/"] { + if let Some(rest) = stripped.strip_prefix(prefix) { + stripped = rest; + break; + } + } + for prefix in ["us.", "eu.", "apac."] { + if let Some(rest) = stripped.strip_prefix(prefix) { + stripped = rest; + break; + } + } + stripped.starts_with("amazon.nova-2-") || stripped.starts_with("nova-2/") +} + +/// `openai.gpt-` anywhere in the id. +fn is_openai_gpt_reasoning_model(model: &str) -> bool { + model.match_indices("openai.gpt-").any(|(at, _)| { + model[at + "openai.gpt-".len()..] + .chars() + .next() + .is_some_and(|c| c.is_ascii_digit()) + }) +} + +/// `xai.grok-` after a start, `.` or `/`. +fn is_xai_grok_model(model: &str) -> bool { + model + .match_indices("xai.grok-") + .any(|(at, _)| at == 0 || matches!(model.as_bytes()[at - 1], b'.' | b'/')) +} + +/// `openai.gpt-[.]` after a start, `.` or `/`. +fn openai_gpt_version(model: &str) -> Option<(u32, u32)> { + model.match_indices("openai.").find_map(|(at, _)| { + if at != 0 && !matches!(model.as_bytes()[at - 1], b'.' | b'/') { + return None; + } + let rest = model[at..].strip_prefix("openai.gpt-")?; + let major: String = rest.chars().take_while(char::is_ascii_digit).collect(); + if major.is_empty() || major.len() > 3 { + return None; + } + let after = &rest[major.len()..]; + let minor = after + .strip_prefix('.') + .map(|tail| { + tail.chars() + .take_while(char::is_ascii_digit) + .collect::() + }) + .filter(|minor| !minor.is_empty() && minor.len() <= 3) + .unwrap_or_default(); + Some((major.parse().ok()?, minor.parse().unwrap_or(0))) + }) +} + +/// `bedrock_rejects_stop_sequences`. +fn rejects_stop_sequences(model: &str) -> bool { + is_xai_grok_model(model) || openai_gpt_version(model).is_some_and(|version| version >= (5, 6)) +} + +/// The model a Converse request is for, and the cost-map reads its rules make. +pub struct BedrockModel<'a> { + pub model: String, + pub base: String, + pub context: ChatRequestContext<'a>, + pub caps: ModelCapabilities<'a>, +} + +impl<'a> BedrockModel<'a> { + pub fn new(model: &str, context: &ChatRequestContext<'a>) -> Self { + Self { + model: model.to_owned(), + base: bedrock_base_model(model), + context: *context, + caps: ModelCapabilities::new(model, context), + } + } + + pub fn is_anthropic(&self) -> bool { + self.base.starts_with("anthropic") + } + + /// The keys Python's provider-aware lookups resolve `model` through. + pub fn candidates(&self) -> Vec { + let stripped = strip_bedrock_routing_prefix(&self.model); + let mut candidates = vec![ + self.model.clone(), + format!("bedrock/{}", self.model), + format!("bedrock_converse/{}", self.model), + stripped.clone(), + self.base.clone(), + format!("bedrock/{}", self.base), + ]; + candidates.dedup(); + candidates + } + + pub fn flag(&self, key: &str) -> Result, Error> { + capability(&self.context, &self.candidates(), key) + } + + pub fn supports(&self, key: &str) -> Result { + Ok(self.flag(key)? == Some(true)) + } + + /// `_supports_sampling_params`. + pub fn supports_sampling(&self) -> Result { + if self.is_anthropic() { + return Ok(true); + } + let mut keys = vec![self.model.clone()]; + keys.extend(["global.", "us.", "eu."].map(|prefix| format!("{prefix}{}", self.base))); + Ok(capability(&self.context, &keys, "supports_sampling_params")?.unwrap_or(true)) + } + + fn is_deepseek(&self) -> bool { + self.model.contains("deepseek") || self.base.contains("deepseek") + } + + fn is_deepseek_r1(&self) -> bool { + self.model.contains("deepseek.r1") || self.base.contains("deepseek.r1") + } + + /// `_model_accepts_anthropic_thinking_param`. + fn accepts_anthropic_thinking(&self) -> Result { + if self.is_deepseek() { + return Ok(false); + } + Ok(self.model.contains("claude-3-7") + || self.model.contains("claude-sonnet-4") + || self.model.contains("claude-opus-4") + || self.supports("supports_reasoning")?) + } + + /// `bedrock_converse_supports_parallel_tool_use_config`. + pub fn supports_parallel_tool_use_config(&self) -> Result { + capability( + &self.context, + &[self.model.clone(), self.base.clone()], + "supports_parallel_tool_use_config", + ) + .map(|flag| flag == Some(true)) + } + + /// `bedrock_model_accepts_cache_points`. + pub fn accepts_cache_points(&self) -> bool { + let openai_family = self + .model + .match_indices("openai.") + .any(|(at, _)| at == 0 || matches!(self.model.as_bytes()[at - 1], b'.' | b'/')); + if openai_family { + return false; + } + let Some(info) = self.context.model_info else { + return true; + }; + let map_keys = [self.model.clone(), self.base.clone()]; + let registered = ["", "converse/", "converse_like/"] + .map(|route| format!("bedrock/{route}{}", self.model)); + let explicit = registered + .iter() + .chain(map_keys.iter()) + .filter_map(|key| info.entry(key)) + .find_map(|entry| { + entry + .get("supports_prompt_cache_breakpoint") + .filter(|flag| !flag.is_null()) + .map(|flag| flag == &Value::Bool(true)) + }); + if let Some(explicit) = explicit { + return explicit; + } + let entries: Vec<_> = map_keys.iter().filter_map(|key| info.entry(key)).collect(); + entries.is_empty() + || entries + .iter() + .any(|entry| entry.get("supports_prompt_caching") == Some(&Value::Bool(true))) + } + + /// `supports_bedrock_prompt_cache_ttl`. + pub fn cache_ttl_supported(&self) -> bool { + model_field( + &self.context, + &[self.model.clone(), self.base.clone()], + "cache_creation_input_token_cost_above_1hr", + ) + .is_some() + } + + /// `bedrock_converse_supports_strict_tools`. + pub fn supports_strict_tools(&self) -> bool { + if !self.is_anthropic() { + return false; + } + self.context + .model_info + .and_then(|info| { + [self.base.clone(), format!("bedrock/{}", self.base)] + .iter() + .filter_map(|key| info.entry(key)) + .find_map(|entry| { + entry + .get("bedrock_converse_supports_strict_tools") + .and_then(Value::as_bool) + }) + }) + .unwrap_or(true) + } + + /// `bedrock_converse_supports_tool_result_images`. + pub fn supports_tool_result_images(&self) -> bool { + let Some(info) = self.context.model_info else { + return true; + }; + let keys = [ + self.model.clone(), + strip_bedrock_routing_prefix(&self.model), + ]; + keys.iter().filter_map(|key| info.entry(key)).all(|entry| { + entry.get("supports_bedrock_converse_tool_result_images") != Some(&Value::Bool(false)) + }) + } + + /// `bedrock_converse_accepts_redacted_thinking` for a model id that names its model. + pub fn accepts_redacted_thinking(&self) -> bool { + self.base.starts_with("anthropic") || self.base.starts_with("claude") + } + + /// `normalize_bedrock_opus_output_config_effort`: an effort above the model's ceiling + /// comes down to it. + pub fn normalize_output_config_effort(&self, output_config: &mut Map) { + let Some(effort) = output_config.get("effort").and_then(Value::as_str) else { + return; + }; + let Some(rank) = OUTPUT_CONFIG_EFFORT_ORDER.iter().position(|e| *e == effort) else { + return; + }; + let ceiling = model_field( + &self.context, + &self.candidates(), + "bedrock_output_config_effort_ceiling", + ); + let Some(ceiling) = ceiling.as_ref().and_then(Value::as_str) else { + return; + }; + if let Some(ceiling_rank) = OUTPUT_CONFIG_EFFORT_ORDER + .iter() + .position(|e| *e == ceiling) + && rank > ceiling_rank + { + output_config.insert("effort".into(), json!(ceiling)); + } + } + + /// `_validate_anthropic_adaptive_effort`. + pub fn validate_adaptive_effort(&self, effort: &Value) -> Result<(), Error> { + let text = effort + .as_str() + .filter(|effort| OUTPUT_CONFIG_EFFORT_ORDER.contains(effort)); + let Some(text) = text else { + let shown = match effort { + Value::String(text) => format!("'{text}'"), + other => python_dumps(other), + }; + return Err(unsupported_value(format!( + "Invalid reasoning_effort/output_config.effort value: {shown}. Must be one of: 'low', 'medium', 'high', 'xhigh', or 'max'." + ))); + }; + let refused = match text { + "max" + if !(self.caps.is_adaptive()? + || self.caps.supports("supports_max_reasoning_effort")?) => + { + Some("max") + } + "xhigh" if !self.caps.supports("supports_xhigh_reasoning_effort")? => Some("xhigh"), + _ => None, + }; + match refused { + Some(value) => Err(Error::InvalidRequest(ErrorDetail::UnsupportedValue { + field: "effort", + value, + model: self.model.clone(), + })), + None => Ok(()), + } + } + + /// `AnthropicConfig.model_supports_effort_param` under `bedrock`. + pub fn supports_effort_param(&self) -> Result { + if self.caps.supports("supports_output_config")? { + return Ok(true); + } + for level in ["low", "minimal", "medium", "high", "xhigh", "max"] { + if self + .caps + .supports(&format!("supports_{level}_reasoning_effort"))? + { + return Ok(true); + } + } + Ok(false) + } +} + +/// `get_supported_openai_params`. +pub fn supported_params(model: &BedrockModel<'_>) -> Result, Error> { + let mut params = vec![ + "max_tokens", + "max_completion_tokens", + "stream", + "stream_options", + "stop", + ]; + if model.supports_sampling()? { + params.extend(["temperature", "top_p"]); + } + params.extend([ + "extra_headers", + "response_format", + "requestMetadata", + "service_tier", + "parallel_tool_calls", + ]); + if model.model.contains("arn") { + params.extend([ + "tools", + "tool_choice", + "thinking", + "reasoning_effort", + "output_config", + ]); + if model.model.to_lowercase().contains("nova") { + params.push("web_search_options"); + } + return Ok(params); + } + let base = model.base.as_str(); + if [ + "anthropic", + "mistral", + "cohere", + "meta.llama3-1", + "meta.llama3-2", + "meta.llama3-3", + "meta.llama4", + "amazon.nova", + ] + .iter() + .any(|family| base.starts_with(family)) + || model.supports("supports_function_calling")? + { + params.push("tools"); + } + if base.starts_with("amazon.nova") { + params.push("web_search_options"); + } + if model.supports("supports_tool_choice")? { + params.push("tool_choice"); + } + if model.model.contains("gpt-oss") + || is_openai_gpt_reasoning_model(&model.model) + || is_openai_gpt_reasoning_model(base) + { + params.push("reasoning_effort"); + } else if model.is_deepseek() { + if !model.is_deepseek_r1() { + params.push("reasoning_effort"); + } + } else if is_nova_2_model(&model.model) { + params.push("reasoning_effort"); + } else if model.accepts_anthropic_thinking()? { + params.extend(["thinking", "reasoning_effort", "output_config"]); + } + if model.is_anthropic() { + params.push("context_management"); + } + Ok(params) +} + +/// `map_tool_choice_values`. +pub fn map_tool_choice_values( + model: &BedrockModel<'_>, + tool_choice: &Value, + drop: DropPolicy, +) -> Result, Error> { + // `forced_tool_use_downgraded`: refused without `drop_params`, `auto` with it. + let downgraded = || -> Result { + if !model.caps.forced_tool_use_unsupported()? { + return Ok(false); + } + drop.unsupported(|| forced_tool_use_refusal(&model.model))?; + Ok(true) + }; + match tool_choice { + Value::String(choice) if choice == "none" => { + drop.unsupported(|| { + unsupported_value( + "Bedrock doesn't support tool_choice=none. To drop it from the call, set `litellm.drop_params = True.", + ) + })?; + Ok(None) + } + Value::String(choice) if choice == "required" => Ok(Some(match downgraded()? { + true => json!({"auto": {}}), + false => json!({"any": {}}), + })), + Value::String(choice) if choice == "auto" => Ok(Some(json!({"auto": {}}))), + Value::Object(choice) => { + if downgraded()? { + return Ok(Some(json!({"auto": {}}))); + } + let name = choice + .get("function") + .and_then(|function| function.get("name")) + .and_then(Value::as_str) + .unwrap_or_default(); + Ok(Some( + json!({"tool": {"name": make_valid_bedrock_tool_name(name)}}), + )) + } + other => Err(unsupported_value(format!( + "Bedrock doesn't support tool_choice={}. Supported tool_choice values=['auto', 'required', json object]. To drop it from the call, set `litellm.drop_params = True.", + python_str(other) + ))), + } +} + +fn forced_tool_use_refusal(model: &str) -> Error { + unsupported_value(format!( + "{model} does not support forced tool use (tool_choice='required' or a named tool). Use tool_choice='auto' and tell the model in the prompt when to call the tool, or set `litellm.drop_params = True` to downgrade to 'auto' automatically." + )) +} + +/// `make_valid_bedrock_tool_name`: `[a-zA-Z][a-zA-Z0-9_-]*`. +pub fn make_valid_bedrock_tool_name(name: &str) -> String { + let Some(first) = name.chars().next() else { + return String::new(); + }; + let prefixed = match first.is_alphabetic() { + true => name.to_owned(), + false => format!("a{name}"), + }; + prefixed + .chars() + .map(|c| match c.is_alphanumeric() || c == '_' || c == '-' { + true => c, + false => '_', + }) + .collect() +} + +/// `_add_additional_properties_to_schema`. +fn with_closed_objects(schema: &Value) -> Value { + let Value::Object(object) = schema else { + return schema.clone(); + }; + let mut result = object.clone(); + if result.get("type").and_then(Value::as_str) == Some("object") + && !result.contains_key("additionalProperties") + { + result.insert("additionalProperties".into(), json!(false)); + } + for key in ["properties", "$defs", "definitions"] { + if let Some(Value::Object(children)) = result.get(key).cloned() { + let closed: Map = children + .iter() + .map(|(name, child)| (name.clone(), with_closed_objects(child))) + .collect(); + result.insert(key.into(), Value::Object(closed)); + } + } + if let Some(items @ Value::Object(_)) = result.get("items").cloned() { + result.insert("items".into(), with_closed_objects(&items)); + } + for key in ["anyOf", "allOf", "oneOf"] { + if let Some(Value::Array(items)) = result.get(key).cloned() { + result.insert( + key.into(), + Value::Array(items.iter().map(with_closed_objects).collect()), + ); + } + } + Value::Object(result) +} + +/// `_create_output_config_for_response_format`. +pub fn output_config_for_response_format( + schema: Option<&Value>, + name: Option<&Value>, + description: Option<&Value>, +) -> Value { + let schema_text = match schema { + Some(schema) => python_dumps(&with_closed_objects(schema)), + None => "{}".to_owned(), + }; + let mut json_schema = Map::from_iter([("schema".to_owned(), json!(schema_text))]); + if let Some(name) = name.filter(|v| !v.is_null()) { + json_schema.insert("name".into(), name.clone()); + } + if let Some(description) = description.filter(|v| !v.is_null()) { + json_schema.insert("description".into(), description.clone()); + } + json!({"textFormat": {"type": "json_schema", "structure": {"jsonSchema": json_schema}}}) +} + +/// What `map_openai_params` hands to the request build. +#[derive(Debug, Default)] +pub struct MappedParams { + pub params: Map, + pub json_mode: bool, + /// The schema of the `json_tool_call` tool a `response_format` added. + pub response_format_schema: Option, +} + +fn add_tools(out: &mut Map, tools: Vec) { + match out.get_mut("tools") { + Some(Value::Array(existing)) => existing.extend(tools), + _ => { + out.insert("tools".into(), Value::Array(tools)); + } + } +} + +/// `_handle_reasoning_effort_parameter`. +fn map_reasoning_effort_param( + out: &mut Map, + model: &BedrockModel<'_>, + effort: &str, +) -> Result<(), Error> { + if model.model.contains("gpt-oss") || model.model.contains("deepseek") { + out.insert("reasoning_effort".into(), json!(effort)); + } else if is_openai_gpt_reasoning_model(&model.model) { + out.insert("reasoning".into(), json!({"effort": effort})); + } else if is_nova_2_model(&model.model) { + if !matches!(effort, "low" | "medium" | "high") { + return Err(unsupported_value(format!( + "Invalid reasoning_effort value '{effort}' for Nova 2 models. Supported values: ['low', 'medium', 'high']" + ))); + } + out.insert( + "reasoningConfig".into(), + json!({"type": "enabled", "maxReasoningEffort": effort}), + ); + } else { + match map_reasoning_effort(effort, &model.caps)? { + None => { + out.remove("thinking"); + out.remove("output_config"); + } + Some(thinking) => { + out.insert("thinking".into(), thinking); + if model.caps.is_adaptive()? { + let mapped = output_config_effort(effort) + .ok_or_else(|| unmapped_reasoning_effort(effort))?; + let mut output_config = match out.get("output_config") { + Some(Value::Object(existing)) => existing.clone(), + _ => Map::new(), + }; + output_config + .entry("effort") + .or_insert_with(|| json!(mapped)); + model.normalize_output_config_effort(&mut output_config); + if let Some(effort) = output_config.get("effort") { + model.validate_adaptive_effort(effort)?; + } + out.insert("output_config".into(), Value::Object(output_config)); + out.insert("_output_config_normalized".into(), json!(true)); + } + } + } + } + Ok(()) +} + +/// `AmazonConverseConfig.map_openai_params`. +pub fn map_openai_params( + openai: &Map, + model: &BedrockModel<'_>, + drop: DropPolicy, +) -> Result { + let thinking_requested = is_thinking_enabled(openai); + let mut out = Map::new(); + let mut json_mode = false; + let mut response_format_schema = None; + for (name, value) in ordered(openai) { + match name { + "response_format" => { + let Some(format) = value.as_object() else { + continue; + }; + if format.get("type").and_then(Value::as_str) == Some("text") { + continue; + } + let (schema, schema_name, description) = match format.get("response_schema") { + Some(schema) => (Some(schema.clone()), None, None), + None => match format.get("json_schema") { + Some(json_schema) => ( + json_schema.get("schema").cloned(), + json_schema.get("name").cloned(), + json_schema.get("description").cloned(), + ), + None => (None, None, None), + }, + }; + let schema = schema.filter(|schema| !schema.is_null()); + if schema.is_some() && model.supports("supports_native_structured_output")? { + out.insert( + "outputConfig".into(), + output_config_for_response_format( + schema.as_ref(), + schema_name.as_ref(), + description.as_ref(), + ), + ); + } else if let Some(schema) = schema { + let mut function = Map::from_iter([ + ("name".to_owned(), json!(RESPONSE_FORMAT_TOOL_NAME)), + ("parameters".to_owned(), schema.clone()), + ]); + if let Some(description) = + description.filter(|d| d.as_str().is_some_and(|d| !d.is_empty())) + { + function.insert("description".into(), description); + } + add_tools( + &mut out, + vec![json!({"type": "function", "function": function})], + ); + response_format_schema = ResponseFormatSchema::from_schema(&schema); + if model.supports("supports_tool_choice")? + && !thinking_requested + && !model.caps.forced_tool_use_unsupported()? + { + out.insert( + "tool_choice".into(), + json!({"tool": {"name": RESPONSE_FORMAT_TOOL_NAME}}), + ); + } + } + json_mode = true; + } + "max_tokens" | "max_completion_tokens" => { + let requires_min = is_openai_gpt_reasoning_model(&model.model) + || model.model.contains("xai.grok-"); + let value = match value.as_i64() { + Some(tokens) if requires_min && value.is_i64() => { + json!(tokens.max(OPENAI_COMPAT_MIN_MAX_TOKENS)) + } + _ => value.clone(), + }; + out.insert("maxTokens".into(), value); + } + "stream" => { + out.insert("stream".into(), value.clone()); + } + "stop" if !rejects_stop_sequences(&model.model) => { + let stop = match value { + Value::String(text) if text.is_empty() => continue, + Value::String(text) => json!([text]), + other => other.clone(), + }; + out.insert("stopSequences".into(), stop); + } + "temperature" | "top_p" => { + let output_key = match name { + "top_p" => "topP", + _ => name, + }; + if model.is_anthropic() { + apply_sampling_param(&mut out, &model.caps, drop, name, value, output_key)?; + } else if !model.supports_sampling()? { + drop.unsupported(|| { + unsupported_value(format!( + "{} does not support {name}={}. To drop unsupported params, set `litellm.drop_params = True`.", + model.model, + python_str(value) + )) + })?; + } else { + out.insert(output_key.into(), value.clone()); + } + } + "tools" => { + let Some(tools) = value.as_array() else { + continue; + }; + if tools + .iter() + .any(|tool| tool.get("type").and_then(Value::as_str) == Some("tool_search")) + { + return Err(Error::Unsupported("the OpenAI tool_search tool on Bedrock")); + } + add_tools(&mut out, tools.clone()); + } + "tool_choice" => { + if let Some(choice) = map_tool_choice_values(model, value, drop)? { + out.insert("tool_choice".into(), choice); + } + } + "parallel_tool_calls" => { + out.insert( + "_parallel_tool_use_config".into(), + json!({"tool_choice": {"type": "auto", "disable_parallel_tool_use": !crate::base_llm::chat::params::truthy(value)}}), + ); + } + "thinking" if model.is_deepseek() => {} + "thinking" if !is_openai_gpt_reasoning_model(&model.model) => { + let adaptive_request = + value.get("type").and_then(Value::as_str) == Some("adaptive"); + if adaptive_request && !model.caps.is_adaptive()? { + let max_tokens = openai + .get("max_completion_tokens") + .filter(|v| crate::base_llm::chat::params::truthy(v)) + .or_else(|| openai.get("max_tokens")) + .filter(|v| !v.is_null()); + let legacy = map_reasoning_effort("medium", &model.caps)?; + if let Some(capped) = + legacy.and_then(|legacy| cap_thinking_budget(legacy, max_tokens)) + { + out.insert("thinking".into(), capped); + } + } else { + out.insert("thinking".into(), value.clone()); + translate_legacy_thinking(&mut out, &model.caps, drop)?; + } + } + "reasoning_effort" => { + let Some(effort) = value.as_str() else { + continue; + }; + if model.is_deepseek_r1() { + continue; + } + if is_openai_gpt_reasoning_model(&model.model) { + return Err(Error::Unsupported( + "reasoning_effort for an OpenAI GPT model on Bedrock Converse", + )); + } + map_reasoning_effort_param(&mut out, model, effort)?; + } + "context_management" => { + if let Some(mapped) = match value { + Value::Array(_) => map_context_management(value), + Value::Object(_) => Some(value.clone()), + _ => None, + } { + out.insert("context_management".into(), mapped); + } + } + "service_tier" => { + let Some(tier) = value.as_str() else { continue }; + let tier = match tier { + "auto" => "default", + other => other, + }; + if matches!(tier, "default" | "flex" | "priority") { + out.insert("serviceTier".into(), json!({"type": tier})); + } + } + "web_search_options" + if value.is_object() && model.model.to_lowercase().contains("nova") => + { + add_tools( + &mut out, + vec![json!({"systemTool": {"name": "nova_grounding"}})], + ); + } + _ => {} + } + } + if !model.model.contains("gpt-oss") && !is_nova_2_model(&model.model) { + if let Some(Value::Object(thinking)) = out.get_mut("thinking") + && let Some(budget) = thinking.get("budget_tokens").and_then(Value::as_i64) + && thinking.get("budget_tokens").is_some_and(Value::is_i64) + && budget < MIN_THINKING_BUDGET_TOKENS + { + thinking.insert("budget_tokens".into(), json!(MIN_THINKING_BUDGET_TOKENS)); + } + let max_tokens_in_request = + openai.contains_key("max_tokens") || openai.contains_key("max_completion_tokens"); + if is_thinking_enabled(&out) + && !max_tokens_in_request + && let Some(budget) = out + .get("thinking") + .and_then(|thinking| thinking.get("budget_tokens")) + .and_then(Value::as_i64) + { + out.insert( + "maxTokens".into(), + json!(budget + DEFAULT_MAX_TOKENS as i64), + ); + } + } + if is_thinking_enabled(&out) + && out + .get("tool_choice") + .and_then(Value::as_object) + .is_some_and(|choice| choice.contains_key("any") || choice.contains_key("tool")) + { + out.insert("tool_choice".into(), json!({"auto": {}})); + } + Ok(MappedParams { + params: out, + json_mode, + response_format_schema, + }) +} + +/// `_validate_request_metadata`. +pub fn validate_request_metadata(metadata: &Value) -> Result<(), Error> { + let Some(metadata) = metadata.as_object() else { + return Err(unsupported_value("requestMetadata must be a dictionary")); + }; + if metadata.len() > 16 { + return Err(unsupported_value( + "requestMetadata can contain a maximum of 16 items", + )); + } + let allowed = + |c: char| c.is_ascii_alphanumeric() || c.is_whitespace() || ":_@$#=/+,.-".contains(c); + for (key, value) in metadata { + let Some(value) = value.as_str() else { + return Err(unsupported_value("requestMetadata values must be strings")); + }; + if key.is_empty() || key.chars().count() > 256 { + return Err(unsupported_value( + "requestMetadata key length must be 1-256 characters", + )); + } + if value.chars().count() > 256 { + return Err(unsupported_value( + "requestMetadata value length must be 0-256 characters", + )); + } + if !key.chars().all(allowed) { + return Err(unsupported_value(format!( + "requestMetadata key '{key}' contains invalid characters. Allowed: [a-zA-Z0-9\\s:_@$#=/+,.-]" + ))); + } + if !value.chars().all(allowed) { + return Err(unsupported_value(format!( + "requestMetadata value '{value}' contains invalid characters. Allowed: [a-zA-Z0-9\\s:_@$#=/+,.-]" + ))); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + #[rstest] + #[case( + "us.meta.llama3-2-11b-instruct-v1:0", + "meta.llama3-2-11b-instruct-v1:0" + )] + #[case( + "bedrock/converse/anthropic.claude-3-5-sonnet-20240620-v1:0", + "anthropic.claude-3-5-sonnet-20240620-v1:0" + )] + #[case( + "anthropic.claude-3-5-sonnet-20241022-v2:0:51k", + "anthropic.claude-3-5-sonnet-20241022-v2:0" + )] + #[case("bedrock/nova-2/arn:aws:bedrock:x", "amazon.nova-2-custom")] + #[case( + "arn:aws:bedrock:us-east-1:1:inference-profile/us.anthropic.claude-x", + "anthropic.claude-x" + )] + #[case("us-west-2/mistral.mistral-large", "mistral.mistral-large")] + #[case("global.anthropic.claude-sonnet-4-5", "anthropic.claude-sonnet-4-5")] + fn base_models_match_python(#[case] model: &str, #[case] expected: &str) { + assert_eq!(bedrock_base_model(model), expected); + } + + #[rstest] + #[case("get weather", "get_weather")] + #[case("1tool", "a1tool")] + #[case("", "")] + #[case("ok-name_1", "ok-name_1")] + fn tool_names_are_made_valid(#[case] name: &str, #[case] expected: &str) { + assert_eq!(make_valid_bedrock_tool_name(name), expected); + } + + #[rstest] + fn gpt_versions_parse() { + assert_eq!(openai_gpt_version("openai.gpt-5.6"), Some((5, 6))); + assert!(rejects_stop_sequences("us.xai.grok-4")); + assert!(!rejects_stop_sequences("openai.gpt-oss-20b-1:0")); + } + + #[rstest] + fn request_metadata_is_validated() { + assert!(validate_request_metadata(&json!({"a": "b"})).is_ok()); + assert_eq!( + validate_request_metadata(&json!({"a": 1})) + .unwrap_err() + .to_string(), + "invalid request: requestMetadata values must be strings" + ); + } +} diff --git a/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs b/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs index a27a71b2802..70425293611 100644 --- a/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs +++ b/litellm-rust/crates/llms/src/bedrock/chat/converse_transformation.rs @@ -1,3 +1,6 @@ +//! Python's `AmazonConverseConfig` for `/chat/completions`: `get_optional_params` and +//! `_transform_request` on the way in, `_transform_response` on the way out. + use litellm_auth::AwsParams; use litellm_auth::{CredentialPlacement, SecretValue}; use litellm_auth_aws::{ @@ -5,51 +8,51 @@ use litellm_auth_aws::{ constants::{AWS_BEARER_TOKEN_BEDROCK, BEDROCK_RUNTIME_ENDPOINT_TEMPLATE, BEDROCK_SERVICE}, resolve_bedrock_region, }; -use litellm_core_utils::{ - core_helpers::{finish_reason_for, unix_now, usage_from_parts}, - prompt_templates::factory::{Conversation, TurnRole, build_conversation}, -}; +use litellm_core_utils::core_helpers::{finish_reason_for, unix_now, usage_from_parts}; use litellm_llms_types::{ formats::chat_completions::{ - ChatCompletionsChoice, ChatCompletionsChoiceMessage, ChatCompletionsResponse, - ChatCompletionsUsage, ChatMessage, ChatMessageContent, + ChatCompletionThinkingBlock, ChatCompletionToolCallChunk, + ChatCompletionToolCallFunctionChunk, ChatCompletionsChoice, ChatCompletionsChoiceMessage, + ChatCompletionsResponse, ChatCompletionsUsage, ChatMessage, }, - providers::bedrock::{CONVERSE_PATH, ConverseResponse}, + providers::bedrock::{CONVERSE_PATH, ConverseUsage}, }; +use serde::Deserialize; use serde_json::{Map, Value, json}; use crate::{ - Error, + Error, ErrorDetail, + anthropic::chat::{messages::native_redacted_thinking_data, params::apply_sampling_param}, base_llm::{ auth::AuthScheme, chat::{ + media::{data_url, remote_urls}, + params::{ + ChatRequestContext, RESPONSE_FORMAT_TOOL_NAME, python_dumps, sort_params, truthy, + }, streaming::{ChatStream, StreamShape}, transformation::{ BaseConfig, Headers, ProviderChatRequestData, ProviderChatResponseData, - Unsupported, ValidatedEnvironment, unsupported_message, unsupported_param, + Unsupported, ValidatedEnvironment, }, }, }, - bedrock::chat::invoke_handler::converse_chat_stream, + bedrock::chat::{ + converse_messages::{ + converse_messages, converse_tools, guard_trailing_user_messages, + neutralize_orphaned_tool_blocks, transform_system_message, + }, + converse_params::{ + BedrockModel, MappedParams, map_openai_params, output_config_for_response_format, + supported_params, validate_request_metadata, + }, + invoke_handler::converse_chat_stream, + }, }; /// The action a streaming Converse call goes to, in place of [`CONVERSE_PATH`]. const CONVERSE_STREAM_PATH: &str = "/converse-stream"; -/// Converse parameter names, post `map_openai_params`, that the Rust path can -/// place verbatim in `inferenceConfig`. -/// -/// `topK` is deliberately absent: Python routes it to -/// `additionalModelRequestFields` for Anthropic base models and to -/// `inferenceConfig` otherwise, and that branch reads the model catalog the -/// core cannot see. -const SUPPORTED_PARAMS: &[(&str, &str)] = &[ - ("max_tokens", "maxTokens"), - ("temperature", "temperature"), - ("top_p", "topP"), - ("stop", "stopSequences"), -]; - const AWS_BEDROCK_RUNTIME_ENDPOINT: &str = "aws_bedrock_runtime_endpoint"; /// AWS call configuration a host passes down: consumed for signing and endpoint @@ -68,6 +71,30 @@ const CONFIG_PARAMS: &[&str] = &[ AWS_BEDROCK_RUNTIME_ENDPOINT, ]; +/// `AmazonConverseConfig.__annotations__` plus `top_k`, the tool params and the config +/// blocks: what stays out of `additionalModelRequestFields`. +const INFERENCE_PARAMS: [&str; 6] = [ + "maxTokens", + "stopSequences", + "temperature", + "topP", + "topK", + "top_k", +]; +const TOOL_PARAMS: [&str; 2] = ["tools", "tool_choice"]; +const CONFIG_BLOCKS: [&str; 3] = ["guardrailConfig", "performanceConfig", "serviceTier"]; +/// `filter_internal_params`. +const INTERNAL_PARAMS: [&str; 3] = [ + "skip_mcp_handler", + "mcp_handler_context", + "_skip_mcp_handler", +]; + +const FINE_GRAINED_TOOL_STREAMING_BETA: &str = "fine-grained-tool-streaming-2025-05-14"; +const EFFORT_BETA: &str = "effort-2025-11-24"; +const COMPACT_BETA: &str = "compact-2026-01-12"; +const COMPACT_EDIT_TYPE: &str = "compact_20260112"; + enum ConverseStopReason { Value(String), Unknown(String), @@ -92,10 +119,556 @@ impl ConverseStopReason { } } +#[derive(Deserialize)] +struct ResponseProjection { + output: OutputProjection, + #[serde(default)] + usage: ConverseUsage, + #[serde(rename = "stopReason")] + stop_reason: Option, +} + +#[derive(Deserialize)] +struct OutputProjection { + message: Option, +} + +#[derive(Deserialize)] +struct MessageProjection { + content: Vec, +} + +#[derive(Deserialize)] +struct ContentProjection { + #[serde(default)] + text: Option, + #[serde(default, rename = "toolUse")] + tool_use: Option, + #[serde(default, rename = "reasoningContent")] + reasoning_content: Option, + #[serde(default, rename = "citationsContent")] + citations_content: Option, +} + pub struct AmazonConverseConfig; pub const BEDROCK_CHAT_COMPLETIONS_CONFIG: AmazonConverseConfig = AmazonConverseConfig; +/// `parse_content_for_reasoning`: a leading `` block split from the answer. +fn parse_content_for_reasoning(text: &str) -> (Option, String) { + for tag in ["think", "thinking", "budget:thinking"] { + let Some(rest) = text.strip_prefix(&format!("<{tag}>")) else { + continue; + }; + let closes = ["think", "thinking", "budget:thinking"] + .iter() + .filter_map(|close| { + let marker = format!(""); + rest.find(&marker).map(|at| (at, marker.len())) + }) + .min_by_key(|(at, _)| *at); + if let Some((at, len)) = closes { + return (Some(rest[..at].to_owned()), rest[at + len..].to_owned()); + } + } + (None, text.to_owned()) +} + +/// `_unwrap_bedrock_properties`: a lone `properties` key gives up its value. +pub fn unwrap_bedrock_properties(text: &str) -> String { + match serde_json::from_str::(text) { + Ok(Value::Object(object)) if object.len() == 1 && object.contains_key("properties") => { + python_dumps(&object["properties"]) + } + _ => text.to_owned(), + } +} + +/// `eager_input_streaming_flag` over a chat tool. +fn eager_input_streaming(tool: &Value) -> bool { + tool.get("eager_input_streaming") == Some(&Value::Bool(true)) + || tool + .get("function") + .and_then(|function| function.get("eager_input_streaming")) + == Some(&Value::Bool(true)) +} + +/// `_merge_parallel_tool_use_config`. +fn merge_parallel_tool_use_config(additional: &mut Map, config: Map) { + for (key, value) in config { + let merged = match (additional.get(&key), &value) { + (Some(Value::Object(existing)), Value::Object(incoming)) => { + let mut merged = incoming.clone(); + merged.extend(existing.iter().map(|(k, v)| (k.clone(), v.clone()))); + merged.extend( + incoming + .iter() + .filter(|(k, _)| k.as_str() != "type") + .map(|(k, v)| (k.clone(), v.clone())), + ); + Value::Object(merged) + } + _ => value, + }; + additional.insert(key, merged); + } +} + +impl AmazonConverseConfig { + fn build_request( + &self, + model: &str, + messages: Vec, + optional_params: Map, + context: &ChatRequestContext<'_>, + ) -> Result { + let bedrock = BedrockModel::new(model, context); + let sorted = sort_params( + optional_params, + CONFIG_PARAMS, + context, + |_| supported_params(&bedrock), + "bedrock", + model, + )?; + let drop = sorted.drop; + let MappedParams { + mut params, + json_mode, + response_format_schema, + } = map_openai_params(&sorted.openai, &bedrock, drop)?; + for name in &sorted.allowed { + if let Some(value) = sorted.openai.get(name) + && !params.contains_key(name) + { + params.insert(name.clone(), value.clone()); + } + } + params.extend(sorted.provider); + + let messages: Vec = messages + .iter() + .map(|message| { + serde_json::to_value(message).map(|mut value| { + if value.get("role").and_then(Value::as_str) == Some("developer") { + value["role"] = json!("system"); + } + value + }) + }) + .collect::>() + .map_err(|error| Error::InvalidRequest(ErrorDetail::invalid("messages", error)))?; + let (messages, system) = transform_system_message(messages, &bedrock); + let guardrail = params.contains_key("guardrailConfig"); + let has_tools = params.get("tools").is_some_and(truthy); + let mut messages = neutralize_orphaned_tool_blocks(messages, has_tools, guardrail); + if guardrail { + messages = guard_trailing_user_messages(messages); + } + + // `maybe_drop_disabled_thinking`. + if params + .get("thinking") + .and_then(|thinking| thinking.get("type")) + .and_then(Value::as_str) + == Some("disabled") + && bedrock.caps.supports("thinking_always_on")? + { + params.remove("thinking"); + } + + // `_prepare_request_params`. + let normalized = params + .remove("_output_config_normalized") + .is_some_and(|flag| truthy(&flag)); + for internal in ["json_mode", "stream", "fake_stream"] { + params.remove(internal); + } + let mut anthropic_output_config = match params.remove("output_config") { + Some(Value::Object(config)) => Some(config), + _ => None, + }; + let mut output_config_format = None; + if let Some(config) = &mut anthropic_output_config { + if let Some(Value::Object(format)) = config.remove("format") { + output_config_format = Some(format); + } + if config.is_empty() { + anthropic_output_config = None; + } + } + let request_metadata = params.remove("requestMetadata").filter(|v| !v.is_null()); + if let Some(metadata) = &request_metadata { + validate_request_metadata(metadata)?; + } + let mut output_config = params.remove("outputConfig").filter(|v| !v.is_null()); + if output_config.is_none() + && let Some(format) = &output_config_format + && format.get("type").and_then(Value::as_str) == Some("json_schema") + && bedrock.is_anthropic() + && bedrock.supports("supports_native_structured_output")? + { + output_config = Some(output_config_for_response_format( + format.get("schema"), + format.get("name"), + format.get("description"), + )); + } + let (mut inference, mut additional): (Map, Map) = + params.into_iter().partition(|(key, _)| { + INFERENCE_PARAMS.contains(&key.as_str()) + || TOOL_PARAMS.contains(&key.as_str()) + || CONFIG_BLOCKS.contains(&key.as_str()) + }); + if let Some(Value::Object(config)) = additional.remove("_parallel_tool_use_config") + && bedrock.supports_parallel_tool_use_config()? + { + merge_parallel_tool_use_config(&mut additional, config); + } + additional.remove("parallel_tool_calls"); + additional.remove("client_metadata"); + let top_k = inference + .remove("topK") + .or_else(|| inference.remove("top_k")) + .filter(|top_k| !top_k.is_null()); + if let Some(top_k) = top_k { + if bedrock.is_anthropic() { + apply_sampling_param( + &mut additional, + &bedrock.caps, + drop, + "top_k", + &top_k, + "top_k", + )?; + } else if bedrock.base.starts_with("amazon.nova") { + additional.insert("inferenceConfig".into(), json!({"topK": top_k})); + } + } + for internal in INTERNAL_PARAMS { + additional.remove(internal); + } + if let Some(mut config) = anthropic_output_config { + if model.contains(":application-inference-profile/") { + additional.insert("output_config".into(), Value::Object(config)); + } else if bedrock.is_anthropic() + && !(drop.drops() && !bedrock.supports_effort_param()?) + { + if !normalized { + bedrock.normalize_output_config_effort(&mut config); + } + if let Some(effort) = config.get("effort").filter(|effort| !effort.is_null()) { + bedrock.validate_adaptive_effort(effort)?; + } + additional.insert("output_config".into(), Value::Object(config)); + } + } + + // `_process_tools_and_beta`. + let tools = match inference.remove("tools") { + Some(Value::Array(tools)) => tools, + _ => Vec::new(), + }; + let (tool_blocks, tool_name_reverse_map) = converse_tools(&tools, &bedrock)?; + let mut betas: Vec<&str> = Vec::new(); + if bedrock.is_anthropic() { + if tools.iter().any(eager_input_streaming) { + betas.push(FINE_GRAINED_TOOL_STREAMING_BETA); + } + if additional + .get("output_config") + .and_then(|config| config.get("effort")) + .is_some_and(|effort| !effort.is_null()) + && !bedrock.caps.is_adaptive()? + { + betas.push(EFFORT_BETA); + } + let compact_edits: Option<(Map, Vec)> = match additional + .get("context_management") + { + Some(Value::Object(management)) => match management.get("edits") { + Some(Value::Array(edits)) => Some(( + management.clone(), + edits + .iter() + .filter(|edit| { + edit.get("type").and_then(Value::as_str) == Some(COMPACT_EDIT_TYPE) + }) + .cloned() + .collect(), + )), + _ => None, + }, + _ => None, + }; + match compact_edits { + Some((mut management, edits)) if !edits.is_empty() => { + betas.push(COMPACT_BETA); + management.insert("edits".into(), Value::Array(edits)); + additional.insert("context_management".into(), Value::Object(management)); + } + _ => { + additional.remove("context_management"); + } + } + if !betas.is_empty() { + additional.insert("anthropic_beta".into(), json!(betas)); + } + } + additional.remove("cache_control_injection_points"); + + let mut tool_config = None; + if !tool_blocks.is_empty() { + let mut config = Map::from_iter([("tools".to_owned(), Value::Array(tool_blocks))]); + if let Some(choice) = inference.remove("tool_choice") { + config.insert("toolChoice".into(), choice); + if let Some(Value::Object(choice)) = additional.get_mut("tool_choice") { + choice.remove("type"); + } + } + tool_config = Some(Value::Object(config)); + } + let config_blocks: Vec<(&str, Option)> = CONFIG_BLOCKS + .iter() + .map(|name| (*name, inference.remove(*name))) + .collect(); + + let mut body = Map::from_iter([( + "messages".to_owned(), + Value::Array(converse_messages(&messages, &bedrock)?), + )]); + body.insert("inferenceConfig".into(), Value::Object(inference)); + if !additional.is_empty() { + let thinking = additional.contains_key("thinking"); + body.insert( + "additionalModelRequestFields".into(), + Value::Object(additional), + ); + if thinking { + body.insert( + "additionalModelResponseFieldPaths".into(), + json!(["/usage/output_tokens_details"]), + ); + } + } + if !system.is_empty() { + body.insert("system".into(), Value::Array(system)); + } + for (name, value) in config_blocks { + if let Some(value) = value.filter(|value| !value.is_null()) { + body.insert(name.into(), value); + } + } + if let Some(tool_config) = tool_config { + body.insert("toolConfig".into(), tool_config); + } + if let Some(metadata) = request_metadata { + body.insert("requestMetadata".into(), metadata); + } + if let Some(output_config) = output_config { + body.insert("outputConfig".into(), output_config); + } + Ok(ProviderChatRequestData { + body: Value::Object(body), + stream_shape: StreamShape { + json_mode, + tool_name_reverse_map, + response_format_schema, + ..StreamShape::default() + }, + headers: Vec::new(), + }) + } +} + +/// `_transform_thinking_blocks`. +fn thinking_blocks(blocks: &[Value]) -> Vec { + blocks + .iter() + .filter_map(|block| { + if let Some(text) = block.get("reasoningText") { + return Some(ChatCompletionThinkingBlock::Thinking { + thinking: text.get("text").and_then(Value::as_str).map(str::to_owned), + signature: text + .get("signature") + .and_then(Value::as_str) + .map(str::to_owned), + cache_control: None, + index: None, + }); + } + block + .get("redactedContent") + .and_then(Value::as_str) + .map(|data| ChatCompletionThinkingBlock::RedactedThinking { + data: Some(native_redacted_thinking_data(data)), + cache_control: None, + index: None, + }) + }) + .collect() +} + +fn converse_response( + model: &str, + response: ResponseProjection, + shape: &StreamShape, +) -> ChatCompletionsResponse { + let mut content = String::new(); + let mut tool_calls: Vec = Vec::new(); + let mut reasoning_blocks: Option> = None; + let mut citations: Option> = None; + for (index, block) in response + .output + .message + .map(|message| message.content) + .unwrap_or_default() + .into_iter() + .enumerate() + { + let mut extracted_reasoning = None; + if let Some(text) = &block.text { + let (reasoning, answer) = parse_content_for_reasoning(text); + extracted_reasoning = reasoning; + content.push_str(&answer); + } + if let Some(tool) = &block.tool_use { + let name = tool.get("name").and_then(Value::as_str).unwrap_or_default(); + tool_calls.push(ChatCompletionToolCallChunk { + id: tool + .get("toolUseId") + .and_then(Value::as_str) + .map(str::to_owned), + tool_type: "function".into(), + function: ChatCompletionToolCallFunctionChunk { + name: Some( + shape + .tool_name_reverse_map + .get(name) + .cloned() + .unwrap_or_else(|| name.to_owned()), + ), + arguments: python_dumps(tool.get("input").unwrap_or(&Value::Null)), + provider_specific_fields: None, + }, + index: index as i64, + provider_specific_fields: None, + }); + } + if let Some(reasoning) = extracted_reasoning { + reasoning_blocks + .get_or_insert_with(Vec::new) + .push(json!({"reasoningText": {"text": reasoning}})); + } + if let Some(reasoning) = block.reasoning_content { + reasoning_blocks + .get_or_insert_with(Vec::new) + .push(reasoning); + } + if let Some(citation) = block.citations_content { + citations.get_or_insert_with(Vec::new).push(citation); + } + } + let mut provider_fields = Map::new(); + if let Some(blocks) = &reasoning_blocks { + provider_fields.insert( + "reasoningContentBlocks".into(), + Value::Array(blocks.clone()), + ); + } + if let Some(citations) = citations { + provider_fields.insert("citationsContent".into(), Value::Array(citations)); + } + let reasoning_content = reasoning_blocks.as_ref().map(|blocks| { + blocks + .iter() + .filter_map(|block| block.get("reasoningText")?.get("text")?.as_str()) + .collect::() + }); + let thinking = reasoning_blocks.as_deref().map(thinking_blocks); + + // `_filter_json_mode_tools`. + let had_tools = !tool_calls.is_empty(); + let json_indices: Vec = match shape.json_mode { + true => tool_calls + .iter() + .enumerate() + .filter(|(_, call)| call.function.name.as_deref() == Some(RESPONSE_FORMAT_TOOL_NAME)) + .map(|(index, _)| index) + .collect(), + false => Vec::new(), + }; + let filtered: Vec = if json_indices.is_empty() { + tool_calls + } else { + let answer = unwrap_bedrock_properties(&tool_calls[json_indices[0]].function.arguments); + if json_indices.len() == tool_calls.len() { + content = answer; + Vec::new() + } else { + content = match content.is_empty() { + true => answer, + false => format!("{content}{answer}"), + }; + tool_calls + .into_iter() + .enumerate() + .filter(|(index, _)| !json_indices.contains(index)) + .map(|(_, call)| call) + .collect() + } + }; + let mapped = finish_reason_for( + ConverseStopReason::parse(response.stop_reason) + .as_ref() + .map(ConverseStopReason::as_str) + .unwrap_or(""), + ) + .to_string(); + let finish_reason = + match shape.json_mode && filtered.is_empty() && had_tools && mapped == "tool_calls" { + true => "stop".to_owned(), + false => mapped, + }; + + let computed = usage_from_parts( + response.usage.input_tokens, + response.usage.output_tokens, + response.usage.cache_read_input_tokens, + response.usage.cache_write_input_tokens, + ); + // Converse reports `totalTokens` and Python passes it straight through. Python does a + // bare `usage["totalTokens"]` lookup, so a body without the key raises there; the + // computed total stands in rather than failing the call. + let usage = ChatCompletionsUsage { + total_tokens: response.usage.total_tokens.unwrap_or(computed.total_tokens), + ..computed + }; + ChatCompletionsResponse { + created: unix_now(), + // Converse echoes no model id, so Python reports the requested one. + model: model.to_string(), + choices: vec![ChatCompletionsChoice { + index: 0, + message: ChatCompletionsChoiceMessage { + role: "assistant".to_string(), + // Converse assigns the joined string unconditionally, so an empty answer + // is `""` here and not `None` as it is on Anthropic. + content: Some(content), + tool_calls: (!filtered.is_empty()).then_some(filtered), + reasoning_content, + thinking_blocks: thinking, + provider_specific_fields: (!provider_fields.is_empty()).then_some(provider_fields), + annotations: None, + }, + finish_reason, + logprobs: None, + }], + usage, + extra: Map::new(), + } +} + impl BaseConfig for AmazonConverseConfig { fn secret_names(&self) -> Vec<&'static str> { litellm_auth::AwsParams::secret_names() @@ -105,10 +678,6 @@ impl BaseConfig for AmazonConverseConfig { .collect() } - fn supported_openai_param_mappings(&self) -> &'static [(&'static str, &'static str)] { - SUPPORTED_PARAMS - } - fn get_complete_url( &self, api_base: Option<&str>, @@ -142,20 +711,19 @@ impl BaseConfig for AmazonConverseConfig { fn transform_request( &self, - _model: &str, + model: &str, messages: Vec, optional_params: Map, + context: &ChatRequestContext<'_>, ) -> Result { - Ok(ProviderChatRequestData { - body: converse_body(&build_conversation(&messages), &optional_params), - stream_shape: StreamShape::default(), - }) + self.build_request(model, messages, optional_params, context) } fn transform_response( &self, model: &str, response: ProviderChatResponseData, + shape: &StreamShape, ) -> Result { let body = response.body; if !body.is_object() { @@ -170,55 +738,18 @@ impl BaseConfig for AmazonConverseConfig { )); } } - let response: ConverseResponse = serde_json::from_value(body).map_err(|error| { + let response: ResponseProjection = serde_json::from_value(body).map_err(|error| { Error::InvalidResponse(format!("invalid Converse response: {error}").into()) })?; - // The route declines tool requests, so anything other than a text block - // is something this path never asked for. Decline; the host falls back. - if response.message_content_is_non_text() { - return Err(Error::Unsupported("non-text response content block")); - } - let text = response.content_text(); - let computed = usage_from_parts( - response.usage.input_tokens, - response.usage.output_tokens, - response.usage.cache_read_input_tokens, - response.usage.cache_write_input_tokens, - ); - // Converse reports `totalTokens` and Python passes it straight through, - // where Anthropic has no such field and Python adds the two counts - // instead, so only this provider overrides the computed total. Python - // does a bare `usage["totalTokens"]` lookup, so a body without the key - // raises there rather than reporting a zero; fall back to the computed - // total, which is the closest thing to that without failing the call. - let usage = ChatCompletionsUsage { - total_tokens: response.usage.total_tokens.unwrap_or(computed.total_tokens), - ..computed - }; + Ok(converse_response(model, response, shape)) + } - Ok(ChatCompletionsResponse { - created: unix_now(), - // Converse echoes no model id, so Python reports the requested one. - model: model.to_string(), - choices: vec![ChatCompletionsChoice { - index: 0, - message: ChatCompletionsChoiceMessage { - role: "assistant".to_string(), - // Converse assigns the joined string unconditionally, so an - // empty response is `""` here and not `None` as it is on - // Anthropic. A caller calling `.strip()` on it would break - // on this path alone. - content: Some(text), - }, - finish_reason: finish_reason_for( - ConverseStopReason::parse(response.stop_reason) - .as_ref() - .map(ConverseStopReason::as_str) - .unwrap_or(""), - ) - .to_string(), - }], - usage, + /// Python's `BedrockImageProcessor` downloads every image or file URL a message names. + fn remote_media(&self, messages: &[ChatMessage]) -> Vec { + remote_urls(messages, &["user", "assistant", "tool"], true, |url| { + data_url(url).is_none() + && !url.contains("base64") + && (url.contains("http://") || url.contains("https://")) }) } @@ -282,100 +813,30 @@ impl BaseConfig for AmazonConverseConfig { CONFIG_PARAMS } + /// Every request is translated; a param or message Python refuses is refused here too. fn unsupported_reason( &self, - messages: &[ChatMessage], - optional_params: &Map, + _messages: &[ChatMessage], + _optional_params: &Map, ) -> Option { - unsupported_param( - self.supported_openai_param_mappings(), - CONFIG_PARAMS, - optional_params, - ) - .or_else(|| messages.iter().find_map(unsupported_message)) - // Python's Converse translation drops blank text blocks instead of - // substituting the placeholder the shared conversation builder - // applies, so decline blank text rather than diverge. - .or_else(|| { - messages - .iter() - .any(has_blank_text) - .then_some(Unsupported("blank message text")) - }) - // Converse has no assistant prefill: Python inserts a continue turn - // when a conversation opens or closes on an assistant message, and - // only under `litellm.modify_params`, which the core cannot see. - // Declining both ends also keeps the shared builder's final - // assistant right-strip (an Anthropic rule) unreachable here. - .or_else(|| { - let conversation = build_conversation(messages); - let ends_on_assistant = conversation - .turns - .last() - .is_some_and(|turn| turn.role == TurnRole::Assistant); - (!conversation.opens_on_user_turn() || ends_on_assistant).then_some(Unsupported( - "conversation does not run user turn to user turn", - )) - }) - } -} - -fn converse_body(conversation: &Conversation, optional_params: &Map) -> Value { - let messages: Vec = conversation - .turns - .iter() - .map(|turn| { - json!({ - "role": <&'static str>::from(turn.role), - "content": turn.texts.iter().map(|text| json!({"text": text})).collect::>(), - }) - }) - .collect(); - - let inference_config = Map::from_iter(SUPPORTED_PARAMS.iter().filter_map(|(_, name)| { - optional_params - .get(*name) - .map(|value| ((*name).to_string(), value.clone())) - })); - - let system: Vec = conversation - .system - .iter() - .map(|text| json!({"text": text})) - .collect(); - - Value::Object(Map::from_iter( - [ - ( - "inferenceConfig".to_string(), - Value::Object(inference_config), - ), - ("messages".to_string(), json!(messages)), - ] - .into_iter() - .chain((!system.is_empty()).then(|| ("system".to_string(), json!(system)))), - )) -} - -fn has_blank_text(message: &ChatMessage) -> bool { - match &message.content { - None => false, - Some(ChatMessageContent::Text(text)) => text.trim().is_empty(), - Some(ChatMessageContent::Parts(parts)) => parts.iter().any(|part| { - part.get("text") - .and_then(Value::as_str) - .is_none_or(|text| text.trim().is_empty()) - }), + None } } #[cfg(test)] mod tests { + use std::collections::HashMap as StdHashMap; + use crate::{ Error, base_llm::{ auth::AuthScheme, - chat::transformation::{BaseConfig, ProviderChatResponseData, Unsupported}, + chat::{ + params::ChatRequestContext, + streaming::StreamShape, + transformation::{BaseConfig, ProviderChatResponseData}, + }, + model_info::ModelInfo, }, bedrock::chat::converse_transformation::BEDROCK_CHAT_COMPLETIONS_CONFIG, }; @@ -384,6 +845,27 @@ mod tests { use rstest::rstest; use serde_json::{Map, Value, json}; + struct CostMap(StdHashMap>); + + impl ModelInfo for CostMap { + fn entry(&self, model: &str) -> Option>> { + self.0.get(model).map(std::borrow::Cow::Borrowed) + } + fn declares(&self, key: &str) -> bool { + self.0.values().any(|entry| entry.contains_key(key)) + } + } + + fn cost_map() -> CostMap { + CostMap(StdHashMap::from([( + "anthropic.claude-sonnet-4-5-v1:0".to_owned(), + json!({"supports_tool_choice": true, "supports_reasoning": true, "supports_prompt_caching": true}) + .as_object() + .cloned() + .unwrap(), + )])) + } + fn messages(value: Value) -> Vec { serde_json::from_value(value).expect("valid messages") } @@ -395,207 +877,204 @@ mod tests { } } - fn transform(msgs: Value, opts: Value) -> Value { - BEDROCK_CHAT_COMPLETIONS_CONFIG - .transform_request( - "anthropic.claude-sonnet-4-5-v1:0", - messages(msgs), - params(opts), - ) - .expect("request transforms") - .body - } - - fn transform_response(body: Value) -> Result { - BEDROCK_CHAT_COMPLETIONS_CONFIG.transform_response( + fn request( + msgs: Value, + opts: Value, + drop: bool, + ) -> Result { + let info = cost_map(); + BEDROCK_CHAT_COMPLETIONS_CONFIG.transform_request( "anthropic.claude-sonnet-4-5-v1:0", - ProviderChatResponseData { body }, + messages(msgs), + params(opts), + &ChatRequestContext { + drop_params: Some(drop), + model_info: Some(&info), + }, ) } - fn reason(msgs: Value, opts: Value) -> Option { - BEDROCK_CHAT_COMPLETIONS_CONFIG.unsupported_reason(&messages(msgs), ¶ms(opts)) + fn transform(msgs: Value, opts: Value) -> Value { + request(msgs, opts, false).expect("request transforms").body } - #[test] + fn transform_response(body: Value) -> Result { + transform_response_with(body, &StreamShape::default()) + } + + fn transform_response_with( + body: Value, + shape: &StreamShape, + ) -> Result { + BEDROCK_CHAT_COMPLETIONS_CONFIG.transform_response( + "anthropic.claude-sonnet-4-5-v1:0", + ProviderChatResponseData { body }, + shape, + ) + } + + #[rstest] fn builds_the_converse_body_python_builds() { let body = transform( json!([ {"role": "system", "content": "be terse"}, {"role": "user", "content": "hi"} ]), - json!({"maxTokens": 128, "temperature": 0.2}), + json!({"max_tokens": 128, "temperature": 0.2, "top_p": 0.9, "stop": "END"}), ); assert_eq!( body, json!({ - "inferenceConfig": {"maxTokens": 128, "temperature": 0.2}, "messages": [{"role": "user", "content": [{"text": "hi"}]}], + "inferenceConfig": {"maxTokens": 128, "temperature": 0.2, "topP": 0.9, "stopSequences": ["END"]}, "system": [{"text": "be terse"}] }) ); } - #[test] - fn always_emits_inference_config_even_when_empty() { - let body = transform(json!([{"role": "user", "content": "hi"}]), json!({})); - assert_eq!(body["inferenceConfig"], json!({})); - assert!(body.get("system").is_none()); - } - - #[test] - fn places_only_inference_params_in_inference_config() { + #[rstest] + fn tools_and_tool_choice_become_a_tool_config() { let body = transform( - json!([{"role": "user", "content": "hi"}]), + json!([{"role": "user", "content": "weather?"}]), json!({ - "maxTokens": 64, - "temperature": 0.1, - "topP": 0.9, - "stopSequences": ["STOP"] + "max_tokens": 10, + "tools": [{"type": "function", "function": {"name": "get_weather", "description": "d", + "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}], + "tool_choice": "required" }), ); assert_eq!( - body["inferenceConfig"], - json!({"maxTokens": 64, "temperature": 0.1, "topP": 0.9, "stopSequences": ["STOP"]}) + body["toolConfig"], + json!({"tools": [{"toolSpec": {"inputSchema": {"json": {"type": "object", "properties": {"city": {"type": "string"}}, "required": []}}, "name": "get_weather", "description": "d"}}], + "toolChoice": {"any": {}}}) ); - assert!(body.get("additionalModelRequestFields").is_none()); } - #[test] - fn merges_consecutive_user_turns_into_one_message() { + #[rstest] + fn tool_choice_none_is_refused_or_dropped() { + let msgs = json!([{"role": "user", "content": "hi"}]); + let error = request(msgs.clone(), json!({"tool_choice": "none"}), false) + .map(|_| ()) + .unwrap_err(); + assert_eq!( + error.to_string(), + "invalid request: Bedrock doesn't support tool_choice=none. To drop it from the call, set `litellm.drop_params = True." + ); + let body = request(msgs, json!({"tool_choice": "none"}), true) + .unwrap() + .body; + assert!(body.get("toolConfig").is_none()); + } + + #[rstest] + fn unsupported_params_are_refused_or_dropped() { + let msgs = json!([{"role": "user", "content": "hi"}]); + assert!(request(msgs.clone(), json!({"seed": 3}), false).is_err()); + let body = request(msgs, json!({"seed": 3}), true).unwrap().body; + assert!(body.get("additionalModelRequestFields").is_none()); + assert_eq!(body["inferenceConfig"], json!({})); + } + + #[rstest] + fn reasoning_effort_becomes_thinking_in_additional_fields() { + let body = transform( + json!([{"role": "user", "content": "hi"}]), + json!({"reasoning_effort": "low"}), + ); + assert_eq!( + body["additionalModelRequestFields"], + json!({"thinking": {"type": "enabled", "budget_tokens": 1024}}) + ); + assert_eq!( + body["additionalModelResponseFieldPaths"], + json!(["/usage/output_tokens_details"]) + ); + assert_eq!(body["inferenceConfig"]["maxTokens"], json!(1024 + 4096)); + } + + #[rstest] + fn response_format_becomes_a_json_tool() { + let data = request( + json!([{"role": "user", "content": "hi"}]), + json!({"response_format": {"type": "json_schema", "json_schema": {"name": "x", + "schema": {"type": "object", "properties": {"a": {"type": "string"}}}}}}), + false, + ) + .unwrap(); + assert_eq!( + data.body["toolConfig"]["toolChoice"], + json!({"tool": {"name": "json_tool_call"}}) + ); + assert!(data.stream_shape.json_mode); + let response = transform_response_with( + json!({"output": {"message": {"content": [ + {"toolUse": {"toolUseId": "t", "name": "json_tool_call", "input": {"properties": {"a": "b"}}}} + ]}}, "stopReason": "tool_use", "usage": {"inputTokens": 1, "outputTokens": 1, "totalTokens": 2}}), + &data.stream_shape, + ) + .unwrap(); + assert_eq!( + response.choices[0].message.content.as_deref(), + Some("{\"a\": \"b\"}") + ); + assert_eq!(response.choices[0].finish_reason, "stop"); + } + + #[rstest] + fn orphaned_tool_calls_are_sent_as_text() { let body = transform( json!([ - {"role": "user", "content": "one"}, - {"role": "user", "content": [{"type": "text", "text": "two"}]}, - {"role": "assistant", "content": "ack"}, - {"role": "user", "content": "three"} + {"role": "user", "content": "hi"}, + {"role": "assistant", "tool_calls": [{"id": "c", "type": "function", "function": {"name": "f", "arguments": "{}"}}]}, + {"role": "tool", "tool_call_id": "c", "content": "ok"} ]), json!({}), ); assert_eq!( - body["messages"], - json!([ - {"role": "user", "content": [{"text": "one"}, {"text": "two"}]}, - {"role": "assistant", "content": [{"text": "ack"}]}, - {"role": "user", "content": [{"text": "three"}]} - ]) - ); - } - - #[test] - fn declines_streaming() { - assert_eq!( - reason( - json!([{"role": "user", "content": "hi"}]), - json!({"stream": true}) - ), - Some(Unsupported("streaming")) - ); - } - - #[test] - fn declines_top_k_because_python_routes_it_by_base_model() { - assert_eq!( - reason( - json!([{"role": "user", "content": "hi"}]), - json!({"topK": 40}) - ), - Some(Unsupported("unrecognized request parameter")) + body["messages"][1]["content"][0]["text"], + "[tool call c: f({})]" ); } #[rstest] - #[case::tools(json!({"tools": []}))] - #[case::tool_choice(json!({"tool_choice": {"auto": {}}}))] - #[case::thinking(json!({"thinking": {"type": "enabled"}}))] - #[case::request_metadata(json!({"requestMetadata": {"k": "v"}}))] - #[case::output_config(json!({"outputConfig": {}}))] - #[case::parallel_tool_use_config(json!({"_parallel_tool_use_config": {}}))] - fn declines_tools_and_other_params_outside_the_allowlist(#[case] param: Value) { + fn tool_use_and_reasoning_come_back_as_openai_fields() { + let shape = StreamShape { + tool_name_reverse_map: StdHashMap::from([("get_weather".into(), "get.weather".into())]), + ..StreamShape::default() + }; + let response = transform_response_with( + json!({"output": {"message": {"content": [ + {"reasoningContent": {"reasoningText": {"text": "hmm", "signature": "sig"}}}, + {"text": "checking"}, + {"toolUse": {"toolUseId": "t1", "name": "get_weather", "input": {"city": "Paris"}}} + ]}}, "stopReason": "tool_use", "usage": {"inputTokens": 1, "outputTokens": 1}}), + &shape, + ) + .unwrap(); + let message = serde_json::to_value(&response.choices[0].message).unwrap(); assert_eq!( - reason(json!([{"role": "user", "content": "hi"}]), param.clone()), - Some(Unsupported("unrecognized request parameter")), - "expected {param} to decline" + message["tool_calls"], + json!([{"id": "t1", "type": "function", "function": {"name": "get.weather", "arguments": "{\"city\": \"Paris\"}"}, "index": 2}]) ); + assert_eq!(message["reasoning_content"], "hmm"); + assert_eq!( + message["thinking_blocks"], + json!([{"type": "thinking", "thinking": "hmm", "signature": "sig"}]) + ); + assert_eq!(response.choices[0].finish_reason, "tool_calls"); } #[rstest] - #[case::empty_string(json!(""))] - #[case::whitespace_string(json!(" "))] - #[case::whitespace_text_block(json!([{"type": "text", "text": " "}]))] - fn declines_blank_text_rather_than_substituting_the_anthropic_placeholder( - #[case] content: Value, + #[case("ab", Some("a"), "b")] + #[case("plain", None, "plain")] + fn leading_think_tags_are_reasoning( + #[case] text: &str, + #[case] reasoning: Option<&str>, + #[case] answer: &str, ) { - assert_eq!( - reason( - json!([{"role": "user", "content": content}, {"role": "user", "content": "hi"}]), - json!({}) - ), - Some(Unsupported("blank message text")), - "expected blank content {content} to decline" - ); - } - - #[test] - fn declines_a_message_whose_content_list_is_empty() { - // The blank-text check scans parts, so an empty list clears it; Converse - // rejects an empty `content` array, which is a decline the core owes the - // host before the call rather than an error after it. - assert_eq!( - reason(json!([{"role": "user", "content": []}]), json!({})), - Some(Unsupported("message without content")) - ); - assert_eq!( - reason( - json!([{"role": "user", "content": [{"type": "text", "text": "hi"}]}]), - json!({}) - ), - None - ); - } - - #[test] - fn declines_a_conversation_that_opens_or_closes_on_an_assistant_turn() { - assert_eq!( - reason( - json!([ - {"role": "assistant", "content": "prefill"}, - {"role": "user", "content": "hi"} - ]), - json!({}) - ), - Some(Unsupported( - "conversation does not run user turn to user turn" - )) - ); - assert_eq!( - reason( - json!([ - {"role": "user", "content": "hi"}, - {"role": "assistant", "content": "prefill"} - ]), - json!({}) - ), - Some(Unsupported( - "conversation does not run user turn to user turn" - )) - ); - } - - #[test] - fn accepts_a_user_to_user_text_conversation() { - assert_eq!( - reason( - json!([ - {"role": "system", "content": "be terse"}, - {"role": "user", "content": "hi"}, - {"role": "assistant", "content": "hello"}, - {"role": "user", "content": "again"} - ]), - json!({"maxTokens": 16}) - ), - None - ); + let (found, rest) = super::parse_content_for_reasoning(text); + assert_eq!(found.as_deref(), reasoning); + assert_eq!(rest, answer); } #[test] @@ -822,24 +1301,6 @@ mod tests { assert_eq!(response.usage.total_tokens, 14); } - #[test] - fn declines_a_cache_control_message_so_widening_the_gate_is_a_red_test() { - // Converse only reports cache token counts when the request carries a - // cachePoint block, which is why the provider total and the computed one - // cannot disagree today. This is the tripwire: whoever widens the gate to - // admit prompt caching has to come back and re-check the usage mapping - // rather than discovering a silent number change in production. - assert_eq!( - reason( - json!([{"role": "user", "content": [ - {"type": "text", "text": "hi", "cache_control": {"type": "ephemeral"}} - ]}]), - json!({}) - ), - Some(Unsupported("non-text message content")) - ); - } - #[test] fn folds_converse_cache_tokens_into_prompt_tokens() { let response = transform_response(json!({ @@ -862,19 +1323,6 @@ mod tests { assert_eq!(response.usage.prompt_tokens_details.text_tokens, 10); } - #[test] - fn declines_a_response_carrying_a_tool_use_block() { - let err = transform_response(json!({ - "output": {"message": {"content": [ - {"toolUse": {"toolUseId": "t1", "name": "f", "input": {}}} - ]}}, - "stopReason": "tool_use", - "usage": {"inputTokens": 1, "outputTokens": 1} - })) - .expect_err("tool use block"); - assert_eq!(err, Error::Unsupported("non-text response content block")); - } - #[test] fn errors_on_a_response_missing_required_fields() { assert_eq!( @@ -908,7 +1356,7 @@ mod tests { #[test] fn accepts_aws_call_configuration_without_serializing_it() { let call_config = json!({ - "maxTokens": 16, + "max_tokens": 16, "aws_access_key_id": "AKIA", "aws_secret_access_key": "secret", "aws_session_token": "token", @@ -921,19 +1369,12 @@ mod tests { "aws_external_id": "ext", "aws_bedrock_runtime_endpoint": "https://vpce.internal" }); - assert_eq!( - reason( - json!([{"role": "user", "content": "hi"}]), - call_config.clone() - ), - None - ); let body = transform(json!([{"role": "user", "content": "hi"}]), call_config); assert_eq!( body, json!({ - "inferenceConfig": {"maxTokens": 16}, - "messages": [{"role": "user", "content": [{"text": "hi"}]}] + "messages": [{"role": "user", "content": [{"text": "hi"}]}], + "inferenceConfig": {"maxTokens": 16} }), "aws call configuration must not reach the Converse body" ); diff --git a/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs b/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs index d5ae8d6a323..14ab3448864 100644 --- a/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs +++ b/litellm-rust/crates/llms/src/bedrock/chat/invoke_handler.rs @@ -23,9 +23,13 @@ use crate::{ anthropic::chat::handler::ModelResponseIterator, base_llm::{ base_model_iterator::StreamTransformer, - chat::streaming::{ChatStream, StreamShape}, + chat::{ + params::RESPONSE_FORMAT_TOOL_NAME, + streaming::{ChatStream, StreamShape}, + }, messages::streaming::{ByteStream, EventStream}, }, + bedrock::chat::converse_transformation::unwrap_bedrock_properties, }; #[derive(Deserialize)] @@ -200,7 +204,13 @@ pub fn converse_event_stream(bytes: ByteStream) -> BoxStream<'static, Result, + /// The block in progress is the `json_tool_call` tool a `response_format` became. + in_response_format_tool: bool, + /// Some `json_tool_call` block has been streamed back as text. + converted_response_format_tool: bool, } #[derive(Default)] @@ -233,6 +247,8 @@ impl ConverseStreamIterator { thinking_ran: false, thinking_block_index: -1, thinking_content_block_index: None, + in_response_format_tool: false, + converted_response_format_tool: false, } } @@ -254,6 +270,7 @@ impl ConverseStreamIterator { index: self .tool_calls_index .unwrap_or_else(|| i64::try_from(index).unwrap_or(i64::MAX)), + provider_specific_fields: None, } } @@ -295,7 +312,15 @@ impl ConverseStreamIterator { fn start(&mut self, start: &Value, index: u64) -> ConverseParsed { let mut parsed = ConverseParsed::default(); self.content_blocks.clear(); + self.in_response_format_tool = false; if let Some(tool) = start.get("toolUse").filter(|tool| !tool.is_null()) { + if self.shape.json_mode + && tool.get("name").and_then(Value::as_str) == Some(RESPONSE_FORMAT_TOOL_NAME) + { + self.in_response_format_tool = true; + self.converted_response_format_tool = true; + return parsed; + } self.tool_calls_index = Some(self.tool_calls_index.map_or(0, |index| index + 1)); let name = tool.get("name").and_then(Value::as_str).map(|name| { self.shape @@ -329,6 +354,9 @@ impl ConverseStreamIterator { .unwrap_or_default() .to_owned(); self.content_blocks.push(input.clone()); + if self.in_response_format_tool { + return parsed; + } parsed.tool_use = Some(self.tool_call(None, None, input, index)); } else if let Some(reasoning) = delta.get("reasoningContent") { parsed @@ -359,6 +387,13 @@ impl ConverseStreamIterator { self.start(start, index) } else if let Some(delta) = data.get("delta") { self.delta(delta, index) + } else if data.get("contentBlockIndex").is_some() && self.in_response_format_tool { + self.in_response_format_tool = false; + let answer = std::mem::take(&mut self.content_blocks).concat(); + ConverseParsed { + text: unwrap_bedrock_properties(if answer.is_empty() { "{}" } else { &answer }), + ..ConverseParsed::default() + } } else if data.get("contentBlockIndex").is_some() { // A block's stop: a tool that streamed only empty fragments closes with `{}`. let empty = @@ -368,10 +403,15 @@ impl ConverseStreamIterator { ..ConverseParsed::default() } } else if let Some(reason) = data.get("stopReason") { + let mut finish = finish_reason_for(reason.as_str().unwrap_or("stop")); + if finish == "tool_calls" + && self.converted_response_format_tool + && self.tool_calls_index.is_none() + { + finish = "stop"; + } ConverseParsed { - finish_reason: Some( - finish_reason_for(reason.as_str().unwrap_or("stop")).to_owned(), - ), + finish_reason: Some(finish.to_owned()), ..ConverseParsed::default() } } else if let Some(usage) = data.get("usage") { @@ -735,6 +775,39 @@ mod tests { assert_eq!(chunks[7]["choices"][0]["finish_reason"], "tool_calls"); } + #[test] + fn a_response_format_tool_streams_back_as_text() { + let mut iterator = ConverseStreamIterator::new(StreamShape { + json_mode: true, + ..StreamShape::default() + }); + let chunks: Vec = serde_json::json!([ + {"contentBlockIndex": 0, "start": {"toolUse": {"toolUseId": "t1", "name": "json_tool_call"}}}, + {"contentBlockIndex": 0, "delta": {"toolUse": {"input": "{\"properties\": {\"a\""}}}, + {"contentBlockIndex": 0, "delta": {"toolUse": {"input": ": 1}}"}}}, + {"contentBlockIndex": 0}, + {"stopReason": "tool_use"} + ]) + .as_array() + .unwrap() + .iter() + .flat_map(|event| iterator.transform(event.clone()).unwrap()) + .map(|chunk| serde_json::to_value(chunk).unwrap()) + .collect(); + for chunk in &chunks { + assert_eq!( + chunk["choices"][0]["delta"]["tool_calls"], + serde_json::Value::Null + ); + } + let content: String = chunks + .iter() + .filter_map(|chunk| chunk["choices"][0]["delta"]["content"].as_str()) + .collect(); + assert_eq!(content, "{\"a\": 1}"); + assert_eq!(chunks[4]["choices"][0]["finish_reason"], "stop"); + } + #[test] fn converse_reasoning_streams_as_thinking_blocks() { let chunks = converse_chunks(serde_json::json!([ diff --git a/litellm-rust/crates/llms/src/bedrock/chat/mod.rs b/litellm-rust/crates/llms/src/bedrock/chat/mod.rs index a46514aa697..e35bd6d88e7 100644 --- a/litellm-rust/crates/llms/src/bedrock/chat/mod.rs +++ b/litellm-rust/crates/llms/src/bedrock/chat/mod.rs @@ -1,2 +1,4 @@ +pub mod converse_messages; +pub mod converse_params; pub mod converse_transformation; pub mod invoke_handler; diff --git a/litellm-rust/crates/llms/src/cerebras/chat/AGENTS.md b/litellm-rust/crates/llms/src/cerebras/chat/AGENTS.md new file mode 100644 index 00000000000..1d09b3ad06c --- /dev/null +++ b/litellm-rust/crates/llms/src/cerebras/chat/AGENTS.md @@ -0,0 +1,10 @@ +# rules + +- Cerebras through the OpenAI SDK URL rule at `https://api.cerebras.ai/v1` +- `reasoning_effort` only for models the cost map marks as reasoning; `max_completion_tokens` goes as `max_tokens` +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://inference-docs.cerebras.ai/api-reference/chat-completions +- litellm/llms/cerebras/chat.py diff --git a/litellm-rust/crates/llms/src/cerebras/chat/mod.rs b/litellm-rust/crates/llms/src/cerebras/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/cerebras/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/cerebras/chat/transformation.rs b/litellm-rust/crates/llms/src/cerebras/chat/transformation.rs new file mode 100644 index 00000000000..393203d8ecd --- /dev/null +++ b/litellm-rust/crates/llms/src/cerebras/chat/transformation.rs @@ -0,0 +1,163 @@ +//! `litellm/llms/cerebras/chat.py`: Cerebras through the OpenAI SDK at +//! `https://api.cerebras.ai/v1`, with `reasoning_effort` for the models the cost map marks as +//! reasoning ones and `max_completion_tokens` sent as `max_tokens`. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value}; + +use crate::{ + Error, + openai::chat::compatible::{CompatibleContext, OpenAiCompatibleChatConfig, sdk_complete_url}, +}; + +const SUPPORTED_PARAMS: &[&str] = &[ + "max_tokens", + "max_completion_tokens", + "response_format", + "seed", + "stop", + "stream", + "temperature", + "top_p", + "tool_choice", + "tools", + "user", + "max_retries", + "extra_headers", +]; + +pub struct CerebrasChatConfig; + +pub const CEREBRAS_CHAT_CONFIG: CerebrasChatConfig = CerebrasChatConfig; + +impl OpenAiCompatibleChatConfig for CerebrasChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::Cerebras + } + + fn label(&self) -> &'static str { + "Cerebras" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["CEREBRAS_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://api.cerebras.ai/v1") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &["CEREBRAS_API_KEY"] + } + + fn complete_url(&self, api_base: &str) -> String { + sdk_complete_url(api_base) + } + + fn supported_params(&self, context: &CompatibleContext<'_>) -> Vec<&'static str> { + let mut params = SUPPORTED_PARAMS.to_vec(); + if context.supports(self.name(), "supports_reasoning") { + params.push("reasoning_effort"); + } + params + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + let supported = self.supported_params(context); + for (name, value) in non_default { + if name == "max_completion_tokens" { + optional.insert("max_tokens".into(), value.clone()); + } else if supported.contains(&name.as_str()) { + optional.insert(name.clone(), value.clone()); + } + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use std::borrow::Cow; + + use rstest::rstest; + use serde_json::json; + + use super::*; + use crate::{ + ErrorDetail, base_llm::model_info::ModelInfo, openai::chat::compatible::transform_request, + }; + + struct Reasoning; + + impl ModelInfo for Reasoning { + fn entry(&self, model: &str) -> Option>> { + (model == "cerebras/gpt-oss-120b").then(|| { + Cow::Owned(Map::from_iter([( + "supports_reasoning".to_owned(), + json!(true), + )])) + }) + } + + fn declares(&self, _key: &str) -> bool { + true + } + } + + fn send(model: &str, params: Value) -> Result, Error> { + let Value::Object(body) = params else { + panic!("an object") + }; + transform_request( + &CEREBRAS_CHAT_CONFIG, + &CompatibleContext { + model, + drop_params: false, + model_info: Some(&Reasoning), + }, + body, + ) + .map(|sent| sent.body) + } + + #[rstest] + fn reasoning_effort_goes_to_reasoning_models() { + let sent = send( + "gpt-oss-120b", + json!({"messages": [], "reasoning_effort": "low", "max_completion_tokens": 5}), + ) + .unwrap(); + assert_eq!( + Value::Object(sent), + json!({"model": "gpt-oss-120b", "messages": [], "reasoning_effort": "low", "max_tokens": 5}) + ); + } + + #[rstest] + fn reasoning_effort_is_refused_elsewhere() { + assert!(matches!( + send( + "llama3.1-8b", + json!({"messages": [], "reasoning_effort": "low"}) + ), + Err(Error::InvalidRequest(ErrorDetail::UnsupportedParams { .. })) + )); + } + + #[rstest] + fn frequency_penalty_is_refused() { + assert!( + send( + "llama3.1-8b", + json!({"messages": [], "frequency_penalty": 1}) + ) + .is_err() + ); + } +} diff --git a/litellm-rust/crates/llms/src/cerebras/mod.rs b/litellm-rust/crates/llms/src/cerebras/mod.rs new file mode 100644 index 00000000000..30a62fc508f --- /dev/null +++ b/litellm-rust/crates/llms/src/cerebras/mod.rs @@ -0,0 +1 @@ +pub mod chat; diff --git a/litellm-rust/crates/llms/src/deepseek/chat/AGENTS.md b/litellm-rust/crates/llms/src/deepseek/chat/AGENTS.md new file mode 100644 index 00000000000..81ab7aa56be --- /dev/null +++ b/litellm-rust/crates/llms/src/deepseek/chat/AGENTS.md @@ -0,0 +1,11 @@ +# rules + +- DeepSeek at `https://api.deepseek.com/beta`; `/chat/completions` is appended unless the base ends with it +- `thinking` and `reasoning_effort` become the `thinking` toggle, only function tools are sent +- A thinking turn replays `reasoning_content` on every assistant message +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://api-docs.deepseek.com/api/create-chat-completion +- litellm/llms/deepseek/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/deepseek/chat/mod.rs b/litellm-rust/crates/llms/src/deepseek/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/deepseek/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/deepseek/chat/transformation.rs b/litellm-rust/crates/llms/src/deepseek/chat/transformation.rs new file mode 100644 index 00000000000..a01272f0b78 --- /dev/null +++ b/litellm-rust/crates/llms/src/deepseek/chat/transformation.rs @@ -0,0 +1,352 @@ +//! `litellm/llms/deepseek/chat/transformation.py`: DeepSeek at `https://api.deepseek.com/beta`. +//! `thinking` and `reasoning_effort` become DeepSeek's thinking toggle, only function tools go, +//! content lists collapse to strings unless a vision model can take them, and a thinking turn +//! replays `reasoning_content` on every assistant message. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, + base_llm::chat::params::truthy, + openai::chat::{ + compatible::{ + CompatibleContext, GPT_BASE_PARAMS, OpenAiCompatibleChatConfig, content_list_to_str, + copy_supported, gpt_messages, role, search_results_text, + }, + transformation::OPENAI_CHAT_COMPLETIONS_PATH, + }, +}; + +pub struct DeepSeekChatConfig; + +pub const DEEPSEEK_CHAT_CONFIG: DeepSeekChatConfig = DeepSeekChatConfig; + +impl OpenAiCompatibleChatConfig for DeepSeekChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::Deepseek + } + + fn label(&self) -> &'static str { + "DeepSeek" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["DEEPSEEK_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://api.deepseek.com/beta") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &["DEEPSEEK_API_KEY"] + } + + fn complete_url(&self, api_base: &str) -> String { + let base = api_base.trim_end_matches('/'); + match base.ends_with(OPENAI_CHAT_COMPLETIONS_PATH) { + true => base.to_owned(), + false => format!("{base}{OPENAI_CHAT_COMPLETIONS_PATH}"), + } + } + + fn supported_params(&self, _context: &CompatibleContext<'_>) -> Vec<&'static str> { + GPT_BASE_PARAMS + .iter() + .copied() + .chain(["response_format", "thinking", "reasoning_effort"]) + .collect() + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + copy_supported(non_default, optional, &self.supported_params(context)); + let thinking = optional.remove("thinking"); + let effort = optional.remove("reasoning_effort"); + let thinking_type = thinking + .as_ref() + .and_then(|thinking| thinking.get("type")) + .and_then(Value::as_str) + .filter(|kind| ["enabled", "disabled"].contains(kind)); + if let Some(kind) = thinking_type { + optional.insert("thinking".into(), json!({"type": kind})); + } else if let Some(effort) = effort.filter(|effort| !effort.is_null()) { + let kind = match effort.as_str() { + Some("none") => "disabled", + _ => "enabled", + }; + optional.insert("thinking".into(), json!({"type": kind})); + } + drop_unsupported_tools(optional); + Ok(()) + } + + fn transform_messages( + &self, + messages: Vec, + optional: &Map, + context: &CompatibleContext<'_>, + ) -> Result, Error> { + let thinking = context.supports(self.name(), "supports_reasoning") + && optional + .get("thinking") + .and_then(|thinking| thinking.get("type")) + == Some(&json!("enabled")); + let messages: Vec = match thinking { + true => messages.into_iter().map(fill_reasoning_content).collect(), + false => messages, + }; + let forward_images = messages + .iter() + .any(|message| message.get("content").is_some_and(Value::is_array)) + && context.supports(self.name(), "supports_vision"); + gpt_messages( + messages + .into_iter() + .map(|message| forward_or_collapse_content(message, forward_images)) + .collect(), + ) + } +} + +/// `_drop_unsupported_tools`: DeepSeek takes only function tools. +fn drop_unsupported_tools(optional: &mut Map) { + let Some(Value::Array(tools)) = optional.get("tools") else { + return; + }; + let is_function = |tool: &Value| tool.get("type").and_then(Value::as_str) == Some("function"); + if tools.is_empty() || tools.iter().all(is_function) { + return; + } + let functions: Vec = tools + .iter() + .filter(|tool| is_function(tool)) + .cloned() + .collect(); + if functions.is_empty() { + for key in ["tools", "tool_choice", "parallel_tool_calls"] { + optional.remove(key); + } + return; + } + let names: Vec<&str> = functions + .iter() + .filter_map(|tool| tool.get("function")?.get("name")?.as_str()) + .collect(); + let choice_matches = match optional.get("tool_choice") { + Some(Value::Object(choice)) => { + choice.get("type").and_then(Value::as_str) == Some("function") + && choice + .get("function") + .and_then(|function| function.get("name")) + .and_then(Value::as_str) + .is_some_and(|name| names.contains(&name)) + } + _ => true, + }; + if !choice_matches { + optional.remove("tool_choice"); + } + optional.insert("tools".into(), Value::Array(functions)); +} + +/// `_fill_reasoning_content`. +fn fill_reasoning_content(mut message: Value) -> Value { + if role(&message) != Some("assistant") || message.get("reasoning_content").is_some_and(truthy) { + return message; + } + let Some(object) = message.as_object_mut() else { + return message; + }; + let stored = object + .get("provider_specific_fields") + .and_then(|fields| fields.get("reasoning_content")) + .filter(|stored| truthy(stored)) + .cloned(); + match stored { + Some(stored) => { + object.insert("reasoning_content".into(), stored); + if let Some(Value::Object(fields)) = object.get_mut("provider_specific_fields") { + fields.remove("reasoning_content"); + } + } + None => { + object.insert("reasoning_content".into(), json!(" ")); + } + } + message +} + +fn forward_or_collapse_content(message: Value, forward_images: bool) -> Value { + if forward_images + && let Some(Value::Array(content)) = message.get("content") + && is_vision_forwardable(&message, content) + { + return with_search_results_text_block(message); + } + let collapsed = content_list_to_str(&message); + if collapsed.is_empty() || message.get("content") == Some(&Value::String(collapsed.clone())) { + return message; + } + let mut message = message; + if let Some(object) = message.as_object_mut() { + object.insert("content".into(), Value::String(collapsed)); + } + message +} + +fn is_vision_forwardable(message: &Value, content: &[Value]) -> bool { + let forwardable = |block: &Value| match block.get("type").and_then(Value::as_str) { + Some("image_url") => match block.get("image_url") { + Some(Value::String(url)) => !url.is_empty(), + Some(Value::Object(image)) => image + .get("url") + .and_then(Value::as_str) + .is_some_and(|url| !url.is_empty()), + _ => false, + }, + Some("text") => block.get("text").is_some_and(Value::is_string), + _ => false, + }; + role(message) == Some("user") + && content.iter().all(forwardable) + && content + .iter() + .any(|block| block.get("type").and_then(Value::as_str) == Some("image_url")) +} + +fn with_search_results_text_block(mut message: Value) -> Value { + let text = search_results_text(message.get("search_results")); + if text.is_empty() { + return message; + } + if let Some(object) = message.as_object_mut() { + object.remove("search_results"); + if let Some(Value::Array(content)) = object.get_mut("content") { + content.push(json!({"type": "text", "text": text})); + } + } + message +} + +#[cfg(test)] +mod tests { + use std::borrow::Cow; + + use rstest::rstest; + + use super::*; + use crate::{base_llm::model_info::ModelInfo, openai::chat::compatible::transform_request}; + + struct Capable; + + impl ModelInfo for Capable { + fn entry(&self, model: &str) -> Option>> { + let entry = match model { + "deepseek/deepseek-reasoner" => json!({"supports_reasoning": true}), + "deepseek/deepseek-vl" => json!({"supports_vision": true}), + _ => return None, + }; + let Value::Object(entry) = entry else { + unreachable!() + }; + Some(Cow::Owned(entry)) + } + + fn declares(&self, _key: &str) -> bool { + true + } + } + + fn send(model: &str, params: Value) -> Map { + let Value::Object(body) = params else { + panic!("an object") + }; + transform_request( + &DEEPSEEK_CHAT_CONFIG, + &CompatibleContext { + model, + drop_params: false, + model_info: Some(&Capable), + }, + body, + ) + .unwrap() + .body + } + + #[rstest] + #[case::base( + "https://api.deepseek.com/beta", + "https://api.deepseek.com/beta/chat/completions" + )] + #[case::named("https://h/chat/completions/", "https://h/chat/completions")] + fn the_url_ends_in_chat_completions_once(#[case] base: &str, #[case] expected: &str) { + assert_eq!(DEEPSEEK_CHAT_CONFIG.complete_url(base), expected); + } + + #[rstest] + #[case::thinking_wins(json!({"thinking": {"type": "disabled", "budget_tokens": 9}, "reasoning_effort": "high"}), json!({"type": "disabled"}))] + #[case::effort_none(json!({"reasoning_effort": "none"}), json!({"type": "disabled"}))] + #[case::effort_on(json!({"reasoning_effort": "low"}), json!({"type": "enabled"}))] + fn reasoning_becomes_the_thinking_toggle(#[case] params: Value, #[case] thinking: Value) { + let Value::Object(mut params) = params else { + unreachable!() + }; + params.insert("messages".into(), json!([])); + let sent = send("deepseek-chat", Value::Object(params)); + assert_eq!(sent["thinking"], thinking); + assert!(!sent.contains_key("reasoning_effort")); + } + + #[rstest] + fn only_function_tools_go_and_a_choice_naming_a_dropped_tool_goes_too() { + let sent = send( + "deepseek-chat", + json!({"messages": [], + "tools": [{"type": "function", "function": {"name": "f"}}, {"type": "namespace", "name": "n"}], + "tool_choice": {"type": "function", "function": {"name": "n"}}}), + ); + assert_eq!( + sent["tools"], + json!([{"type": "function", "function": {"name": "f"}}]) + ); + assert!(!sent.contains_key("tool_choice")); + } + + #[rstest] + fn a_thinking_turn_replays_reasoning_on_assistant_messages() { + let sent = send( + "deepseek-reasoner", + json!({"thinking": {"type": "enabled"}, "messages": [ + {"role": "assistant", "content": "a", "provider_specific_fields": {"reasoning_content": "r"}}, + {"role": "assistant", "content": "b"}]}), + ); + assert_eq!( + sent["messages"], + json!([ + {"role": "assistant", "content": "a", "provider_specific_fields": {}, "reasoning_content": "r"}, + {"role": "assistant", "content": "b", "reasoning_content": " "}]) + ); + } + + #[rstest] + #[case::collapsed("deepseek-chat", json!("look"))] + #[case::forwarded("deepseek-vl", json!([{"type": "text", "text": "look"}, {"type": "image_url", "image_url": {"url": "https://i"}}]))] + fn content_lists_collapse_unless_vision_takes_them( + #[case] model: &str, + #[case] content: Value, + ) { + let sent = send( + model, + json!({"messages": [{"role": "user", "content": [ + {"type": "text", "text": "look"}, {"type": "image_url", "image_url": "https://i"}]}]}), + ); + assert_eq!(sent["messages"][0]["content"], content); + } +} diff --git a/litellm-rust/crates/llms/src/deepseek/mod.rs b/litellm-rust/crates/llms/src/deepseek/mod.rs index ba63992f3cb..bca4ce419f2 100644 --- a/litellm-rust/crates/llms/src/deepseek/mod.rs +++ b/litellm-rust/crates/llms/src/deepseek/mod.rs @@ -1 +1,2 @@ +pub mod chat; pub mod messages; diff --git a/litellm-rust/crates/llms/src/fireworks_ai/chat/AGENTS.md b/litellm-rust/crates/llms/src/fireworks_ai/chat/AGENTS.md new file mode 100644 index 00000000000..5f9c81a56bc --- /dev/null +++ b/litellm-rust/crates/llms/src/fireworks_ai/chat/AGENTS.md @@ -0,0 +1,12 @@ +# rules + +- Fireworks AI at `https://api.fireworks.ai/inference/v1`, the model named by its full resource (`resolve_fireworks_resource_name`) +- Tool, tool choice and reasoning params are gated by the cost map, file parts and images on non-vision models are rejected +- vLLM/NIM `extra_body` keys are translated (`truncate_prompt_tokens`, `chat_template_kwargs`, `guided_*`) or dropped +- Tool calls Fireworks writes into the content go back as `tool_calls`; the answer's model gets the `fireworks_ai/` prefix +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://docs.fireworks.ai/api-reference/post-chatcompletions +- litellm/llms/fireworks_ai/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/fireworks_ai/chat/mod.rs b/litellm-rust/crates/llms/src/fireworks_ai/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/fireworks_ai/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/fireworks_ai/chat/transformation.rs b/litellm-rust/crates/llms/src/fireworks_ai/chat/transformation.rs new file mode 100644 index 00000000000..e95d5942d0a --- /dev/null +++ b/litellm-rust/crates/llms/src/fireworks_ai/chat/transformation.rs @@ -0,0 +1,680 @@ +//! `litellm/llms/fireworks_ai/chat/transformation.py`: Fireworks AI at +//! `https://api.fireworks.ai/inference/v1`, with models named by their Fireworks resource, +//! capability gated tool and reasoning params, vLLM style `extra_body` keys translated or +//! dropped, and tool calls Fireworks writes into the content moved back to `tool_calls`. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, ErrorDetail, + base_llm::chat::{ + params::{python_dumps, truthy}, + response_format::unpack_legacy_defs, + }, + openai::chat::compatible::{ + CompatibleContext, OpenAiCompatibleChatConfig, RESPONSE_FORMAT_TOOL_NAME, ResponseShape, + gpt_messages, messages_mut, role, strip_response_only_fields, + }, +}; + +const SUPPORTED_PARAMS: &[&str] = &[ + "stream", + "max_completion_tokens", + "max_tokens", + "temperature", + "top_p", + "top_k", + "frequency_penalty", + "presence_penalty", + "n", + "stop", + "response_format", + "user", + "logprobs", + "prompt_truncate_len", + "context_length_exceeded_behavior", + "seed", + "top_logprobs", + "min_p", + "typical_p", + "repetition_penalty", + "mirostat_target", + "mirostat_lr", + "logit_bias", + "echo", + "echo_last", + "ignore_eos", + "prompt_cache_key", + "prompt_cache_isolation_key", + "raw_output", + "perf_metrics_in_response", + "return_token_ids", + "safe_tokenization", + "service_tier", + "speculation", + "prediction", + "stream_options", + "sampling_mask", +]; +const NIM_VLLM_STRIP_PARAMS: &[&str] = &[ + "stop_token_ids", + "include_stop_str_in_output", + "skip_special_tokens", + "spaces_between_special_tokens", + "best_of", + "use_beam_search", + "guided_decoding_backend", + "guided_regex", + "add_generation_prompt", + "continue_final_message", + "add_special_tokens", + "detokenize", + "allowed_token_ids", + "bad_words", + "include_reasoning", + "nvext", +]; +const EXTRA_BODY_CONSUMED_PARAMS: &[&str] = &[ + "truncate_prompt_tokens", + "chat_template_kwargs", + "guided_json", + "guided_grammar", + "guided_choice", +]; +const FIREROUTER: &str = "firerouter"; +const ROUTER_SHORT_NAMES: &[&str] = &[FIREROUTER, "auto", "auto-instant"]; +const SESSION_AFFINITY_HEADER: &str = "x-session-affinity"; + +pub struct FireworksAiChatConfig; + +pub const FIREWORKS_AI_CHAT_CONFIG: FireworksAiChatConfig = FireworksAiChatConfig; + +impl OpenAiCompatibleChatConfig for FireworksAiChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::FireworksAi + } + + fn label(&self) -> &'static str { + "Fireworks AI" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["FIREWORKS_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://api.fireworks.ai/inference/v1") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &[ + "FIREWORKS_API_KEY", + "FIREWORKS_AI_API_KEY", + "FIREWORKSAI_API_KEY", + "FIREWORKS_AI_TOKEN", + ] + } + + /// `with_fireworks_session_affinity`: the caller's session pins its calls to one node so + /// prompt caching hits. + fn headers( + &self, + body: &Map, + _lookup: &dyn Fn(&str) -> Option, + ) -> Vec<(String, String)> { + let session = body + .get("litellm_session_id") + .filter(|id| truthy(id)) + .or_else(|| { + body.get("metadata") + .and_then(|metadata| metadata.get("session_id")) + .filter(|id| truthy(id)) + }); + session + .map(|id| match id { + Value::String(id) => id.clone(), + other => other.to_string(), + }) + .map(|id| vec![(SESSION_AFFINITY_HEADER.to_owned(), id)]) + .unwrap_or_default() + } + + fn wire_model(&self, model: &str) -> String { + resolve_fireworks_resource_name(model) + } + + fn supported_params(&self, context: &CompatibleContext<'_>) -> Vec<&'static str> { + let mut params = SUPPORTED_PARAMS.to_vec(); + if exact_capability(context, "supports_function_calling") != Some(false) { + params.extend(["tools", "parallel_tool_calls"]); + } + if capability(context, "supports_tool_choice") { + params.push("tool_choice"); + } + if capability(context, "supports_reasoning") { + params.extend(["reasoning_effort", "reasoning_history", "thinking"]); + } + params + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + if non_default.get("thinking").is_some_and(|v| !v.is_null()) + && non_default + .get("reasoning_effort") + .is_some_and(|v| !v.is_null()) + { + return Err(Error::InvalidRequest(ErrorDetail::Message( + "Fireworks AI chat completions does not support specifying both `thinking` and \ + `reasoning_effort` in the same request." + .into(), + ))); + } + let supported = self.supported_params(context); + let tools_set = non_default + .get("tools") + .is_some_and(|tools| !tools.is_null()); + let mut response_format_tool = None; + for (name, value) in non_default { + match name.as_str() { + "tool_choice" => { + let choice = match value.as_str() { + Some("required") => json!("any"), + _ => value.clone(), + }; + optional.insert("tool_choice".into(), choice); + } + "response_format" if tools_set => { + if let Some(schema) = response_schema(value) { + response_format_tool = Some(json!({"type": "function", + "function": {"name": RESPONSE_FORMAT_TOOL_NAME, "parameters": schema}})); + optional.insert("json_mode".into(), Value::Bool(true)); + } + } + "max_completion_tokens" => { + optional.insert("max_tokens".into(), value.clone()); + } + "reasoning_effort" => { + if let Some(effort) = map_reasoning_effort(value) { + optional.insert("reasoning_effort".into(), effort); + } + } + name if supported.contains(&name) && !value.is_null() => { + optional.insert(name.to_owned(), value.clone()); + } + _ => {} + } + } + if let Some(tool) = response_format_tool + && let Some(Value::Array(tools)) = optional.get_mut("tools") + { + tools.push(tool); + } + Ok(()) + } + + /// `map_extra_body_params`. + fn map_extra_body( + &self, + extra_body: &mut Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + let mut promoted = Map::new(); + if let Some(tokens) = extra_body + .get("truncate_prompt_tokens") + .filter(|v| !v.is_null()) + && !extra_body.contains_key("prompt_truncate_len") + && !optional.contains_key("prompt_truncate_len") + { + promoted.insert("prompt_truncate_len".into(), tokens.clone()); + } + let explicit_reasoning = ["reasoning_effort", "thinking"] + .iter() + .any(|key| optional.contains_key(*key) || extra_body.contains_key(*key)); + if let Some(Value::Object(kwargs)) = extra_body.get("chat_template_kwargs") + && !explicit_reasoning + && let Some(effort) = effort_from_chat_template_kwargs(kwargs) + && context.supports("fireworks_ai", "supports_reasoning") + { + promoted.insert("reasoning_effort".into(), effort); + } + let has_response_format = + optional.contains_key("response_format") || extra_body.contains_key("response_format"); + if !has_response_format && let Some(format) = guided_response_format(extra_body) { + promoted.insert("response_format".into(), format); + } + extra_body.retain(|key, _| { + !NIM_VLLM_STRIP_PARAMS.contains(&key.as_str()) + && !EXTRA_BODY_CONSUMED_PARAMS.contains(&key.as_str()) + && (key != "response_format" || !optional.contains_key("response_format")) + }); + optional.extend(promoted); + Ok(()) + } + + /// `_transform_messages_helper`, then the base transform. + fn transform_messages( + &self, + messages: Vec, + _optional: &Map, + context: &CompatibleContext<'_>, + ) -> Result, Error> { + let vision = exact_capability(context, "supports_vision"); + for message in messages + .iter() + .filter(|message| role(message) == Some("user")) + { + for kind in message + .get("content") + .and_then(Value::as_array) + .into_iter() + .flatten() + .filter_map(|part| part.get("type").and_then(Value::as_str)) + { + if kind == "file" { + return Err(Error::InvalidRequest(ErrorDetail::Message( + "Fireworks AI chat completions does not support file content blocks. For \ + PDFs, convert pages to images and send image_url blocks to a Fireworks \ + vision model, or extract text before calling a text-only model." + .into(), + ))); + } + if kind == "image_url" && vision == Some(false) { + return Err(Error::InvalidRequest(ErrorDetail::Message(format!( + "Fireworks AI model {} does not support image inputs. Use a Fireworks \ + vision model or remove image_url content blocks.", + resolve_fireworks_resource_name(context.model) + )))); + } + } + } + gpt_messages( + messages + .into_iter() + .map(|mut message| { + strip_response_only_fields(&mut message); + message + }) + .collect(), + ) + } + + fn finish_request( + &self, + body: &mut Map, + _context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + if let Some(Value::Array(tools)) = body.get_mut("tools") { + for function in tools + .iter_mut() + .filter(|tool| tool.get("type").and_then(Value::as_str) == Some("function")) + .filter_map(|tool| tool.get_mut("function")?.as_object_mut()) + { + function.remove("strict"); + if let Some(Value::Object(parameters)) = function.get_mut("parameters") { + unpack_legacy_defs(parameters); + } + } + } + if body.get("stream").is_some_and(truthy) { + match body.get_mut("stream_options") { + Some(Value::Object(options)) => { + if options.get("include_usage") != Some(&Value::Bool(false)) { + options.insert("include_usage".into(), Value::Bool(true)); + } + } + _ => { + body.insert("stream_options".into(), json!({"include_usage": true})); + } + } + } + Ok(()) + } + + fn fix_response(&self, body: &mut Map, shape: &ResponseShape) -> bool { + let mut changed = false; + if let Some(Value::String(model)) = body.get_mut("model") { + *model = format!("fireworks_ai/{model}"); + changed = true; + } + if !shape.tools_sent { + return changed; + } + for message in messages_mut(body) { + changed |= content_tool_call_to_tool_calls(message, &shape.tool_names); + } + changed + } +} + +/// `resolve_fireworks_resource_name`: the full Fireworks resource a short model name means. +pub fn resolve_fireworks_resource_name(model: &str) -> String { + let stripped = model.strip_prefix("fireworks_ai/").unwrap_or(model); + if stripped.starts_with("accounts/") || stripped.starts_with("FW-") || stripped.contains('#') { + return stripped.to_owned(); + } + if stripped.starts_with("routers/") || stripped.starts_with("models/") { + return format!("accounts/fireworks/{stripped}"); + } + if stripped.ends_with("-fast") + || ROUTER_SHORT_NAMES.contains(&stripped) + || stripped.starts_with(&format!("{FIREROUTER}/")) + { + return format!("accounts/fireworks/routers/{stripped}"); + } + format!("accounts/fireworks/models/{stripped}") +} + +/// `_get_model_cost_capability_exact`. +fn exact_capability(context: &CompatibleContext<'_>, key: &str) -> Option { + let short = context + .model + .strip_prefix("fireworks_ai/") + .unwrap_or(context.model); + let short = short + .strip_prefix("accounts/fireworks/models/") + .unwrap_or(short); + let firerouter = format!("accounts/fireworks/routers/{FIREROUTER}"); + let mut candidates = vec![ + context.model.to_owned(), + format!("fireworks_ai/{short}"), + format!("fireworks_ai/accounts/fireworks/models/{short}"), + ]; + if resolve_fireworks_resource_name(context.model).starts_with(&format!("{firerouter}/")) { + candidates.push(format!("fireworks_ai/{firerouter}")); + } + context.capability_of(&candidates, key) +} + +fn capability(context: &CompatibleContext<'_>, key: &str) -> bool { + exact_capability(context, key) == Some(true) || context.supports("fireworks_ai", key) +} + +/// `_map_reasoning_effort`: booleans as levels, `auto` left to Fireworks. +fn map_reasoning_effort(value: &Value) -> Option { + let effort = match value { + Value::Object(object) => object.get("effort").cloned().unwrap_or(Value::Null), + other => other.clone(), + }; + match effort { + Value::Bool(true) => Some(json!("medium")), + Value::Bool(false) => Some(json!("none")), + Value::String(effort) if effort == "auto" => None, + Value::Null => None, + other => Some(other), + } +} + +fn response_schema(value: &Value) -> Option { + let schema = match value.get("response_schema") { + Some(schema) => schema.clone(), + None => value.get("json_schema")?.get("schema")?.clone(), + }; + truthy(&schema).then_some(schema) +} + +/// `effort_from_chat_template_kwargs`. +fn effort_from_chat_template_kwargs(kwargs: &Map) -> Option { + let flag = |keys: &[&str]| keys.iter().find_map(|key| kwargs.get(*key)?.as_bool()); + if flag(&["enable_thinking", "thinking"]) == Some(false) { + return Some(json!("none")); + } + if let Some(budget) = kwargs.get("reasoning_budget").and_then(Value::as_f64) + && budget > 0.0 + { + return Some(json!(budget as i64)); + } + (flag(&["low_effort"]) == Some(true)).then(|| json!("low")) +} + +/// `translate_guided_params`. +fn guided_response_format(extra_body: &Map) -> Option { + let present = |key: &str| extra_body.get(key).filter(|value| !value.is_null()); + let schema = |schema: &Value, name: &str| json!({"type": "json_schema", "json_schema": {"name": name, "schema": schema}}); + if let Some(guided) = present("guided_json") { + return Some(schema(guided, "response")); + } + if let Some(grammar) = present("guided_grammar") { + return Some(json!({"type": "grammar", "grammar": grammar})); + } + present("guided_choice") + .map(|choices| schema(&json!({"type": "string", "enum": choices}), "choice")) +} + +/// `_handle_message_content_with_tool_calls`: a call to a sent tool, written as the content. +fn content_tool_call_to_tool_calls( + message: &mut Map, + tool_names: &[String], +) -> bool { + if message + .get("tool_calls") + .is_some_and(|calls| !calls.is_null()) + { + return false; + } + let Some(Value::Object(call)) = message + .get("content") + .and_then(Value::as_str) + .and_then(|content| serde_json::from_str::(content).ok()) + else { + return false; + }; + let Some(name) = call.get("name").and_then(Value::as_str) else { + return false; + }; + if name == RESPONSE_FORMAT_TOOL_NAME || !tool_names.iter().any(|tool| tool == name) { + return false; + } + let arguments = match call.get("arguments") { + Some(Value::String(arguments)) => arguments.clone(), + Some(Value::Object(arguments)) => python_dumps(&Value::Object(arguments.clone())), + Some(Value::Null) | None => match call.get("parameters") { + Some(parameters @ Value::Object(_)) => python_dumps(parameters), + _ => String::new(), + }, + Some(_) => return false, + }; + message.insert( + "tool_calls".into(), + json!([{"id": uuid::Uuid::new_v4().to_string(), "type": "function", + "function": {"name": name, "arguments": arguments}}]), + ); + message.insert("content".into(), Value::Null); + true +} + +#[cfg(test)] +mod tests { + use std::borrow::Cow; + + use rstest::rstest; + + use super::*; + use crate::{ + base_llm::model_info::ModelInfo, + openai::chat::compatible::{CompatibleChatBody, fix_response, transform_request}, + }; + + struct Registry; + + impl ModelInfo for Registry { + fn entry(&self, model: &str) -> Option>> { + let entry = match model { + "fireworks_ai/accounts/fireworks/models/thinker" => { + json!({"supports_reasoning": true, "supports_tool_choice": true}) + } + "fireworks_ai/text-only" => { + json!({"supports_vision": false, "supports_function_calling": false}) + } + _ => return None, + }; + let Value::Object(entry) = entry else { + unreachable!() + }; + Some(Cow::Owned(entry)) + } + + fn declares(&self, _key: &str) -> bool { + true + } + } + + fn send(model: &str, params: Value) -> Result { + let Value::Object(body) = params else { + panic!("an object") + }; + transform_request( + &FIREWORKS_AI_CHAT_CONFIG, + &CompatibleContext { + model, + drop_params: false, + model_info: Some(&Registry), + }, + body, + ) + } + + #[rstest] + #[case::short( + "llama-v3p1-8b-instruct", + "accounts/fireworks/models/llama-v3p1-8b-instruct" + )] + #[case::router("glm-4p6-fast", "accounts/fireworks/routers/glm-4p6-fast")] + #[case::models_prefix("models/x", "accounts/fireworks/models/x")] + #[case::full("accounts/me/models/x", "accounts/me/models/x")] + #[case::deployment("x#accounts/me/deployments/d", "x#accounts/me/deployments/d")] + fn models_are_named_by_their_resource(#[case] model: &str, #[case] resource: &str) { + assert_eq!(resolve_fireworks_resource_name(model), resource); + } + + #[rstest] + fn reasoning_models_take_tool_choice_and_mapped_efforts() { + let sent = send("thinker", json!({"messages": [], "tool_choice": "required", + "tools": [{"type": "function", "function": {"name": "f", "strict": true, + "parameters": {"type": "object", "properties": {"a": {"$ref": "#/definitions/A"}}, "definitions": {"A": {"type": "string"}}}}}], + "reasoning_effort": true, "max_completion_tokens": 8})) + .unwrap(); + assert_eq!( + Value::Object(sent.body), + json!({"model": "accounts/fireworks/models/thinker", "messages": [], + "tool_choice": "any", + "tools": [{"type": "function", "function": {"name": "f", + "parameters": {"type": "object", "properties": {"a": {"type": "string"}}}}}], + "reasoning_effort": "medium", "max_tokens": 8}) + ); + } + + #[rstest] + fn thinking_and_effort_together_are_rejected() { + assert!(matches!( + send( + "thinker", + json!({"messages": [], "thinking": {"type": "enabled"}, "reasoning_effort": "low"}) + ), + Err(Error::InvalidRequest(ErrorDetail::Message(_))) + )); + } + + #[rstest] + #[case::file(json!({"type": "file", "file": {"file_id": "f"}}))] + #[case::image_on_text_model(json!({"type": "image_url", "image_url": {"url": "https://i"}}))] + fn content_fireworks_cannot_read_is_rejected(#[case] part: Value) { + assert!(matches!( + send( + "text-only", + json!({"messages": [{"role": "user", "content": [part]}]}) + ), + Err(Error::InvalidRequest(ErrorDetail::Message(_))) + )); + } + + #[rstest] + fn tools_are_refused_where_the_registry_rules_out_function_calling() { + assert!(matches!( + send( + "text-only", + json!({"messages": [], "tools": [{"type": "function", "function": {"name": "f"}}]}) + ), + Err(Error::InvalidRequest(ErrorDetail::UnsupportedParams { .. })) + )); + } + + #[rstest] + fn a_response_format_beside_tools_becomes_one_more_tool() { + let sent = send("unknown", json!({"messages": [], + "tools": [{"type": "function", "function": {"name": "f"}}], + "response_format": {"type": "json_schema", "json_schema": {"schema": {"type": "object"}}}})) + .unwrap(); + assert_eq!( + sent.body["tools"], + json!([{"type": "function", "function": {"name": "f"}}, + {"type": "function", "function": {"name": "json_tool_call", "parameters": {"type": "object"}}}]) + ); + assert!(!sent.body.contains_key("response_format")); + } + + #[rstest] + fn vllm_extra_body_keys_are_translated_or_dropped() { + let sent = send( + "unknown", + json!({"messages": [], "truncate_prompt_tokens": 100, + "guided_choice": ["a", "b"], "stop_token_ids": [1], "custom": 1}), + ) + .unwrap(); + assert_eq!( + Value::Object(sent.body), + json!({"model": "accounts/fireworks/models/unknown", "messages": [], + "prompt_truncate_len": 100, + "response_format": {"type": "json_schema", "json_schema": {"name": "choice", "schema": {"type": "string", "enum": ["a", "b"]}}}, + "custom": 1}) + ); + } + + #[rstest] + fn a_session_pins_the_call() { + let body = Map::from_iter([("metadata".to_owned(), json!({"session_id": "s1"}))]); + assert_eq!( + FIREWORKS_AI_CHAT_CONFIG.headers(&body, &|_| None), + vec![("x-session-affinity".to_owned(), "s1".to_owned())] + ); + } + + #[rstest] + #[case::sent_tool("f", true)] + #[case::json_tool("json_tool_call", false)] + #[case::unknown_tool("g", false)] + fn tool_calls_in_the_content_move_to_tool_calls(#[case] name: &str, #[case] moved: bool) { + let content = json!({"name": name, "arguments": {"a": 1}}).to_string(); + let mut body = Map::from_iter([ + ("model".to_owned(), json!("accounts/fireworks/models/x")), + ( + "choices".to_owned(), + json!([{"message": {"role": "assistant", "content": content}}]), + ), + ]); + let shape = ResponseShape { + tools_sent: true, + tool_names: vec!["f".into(), "json_tool_call".into()], + ..ResponseShape::default() + }; + fix_response(&FIREWORKS_AI_CHAT_CONFIG, &mut body, &shape); + assert_eq!( + body["model"], + json!("fireworks_ai/accounts/fireworks/models/x") + ); + let message = &body["choices"][0]["message"]; + assert_eq!(message["content"].is_null(), moved); + if moved { + assert_eq!( + message["tool_calls"][0]["function"], + json!({"name": "f", "arguments": "{\"a\": 1}"}) + ); + } + } +} diff --git a/litellm-rust/crates/llms/src/fireworks_ai/mod.rs b/litellm-rust/crates/llms/src/fireworks_ai/mod.rs new file mode 100644 index 00000000000..30a62fc508f --- /dev/null +++ b/litellm-rust/crates/llms/src/fireworks_ai/mod.rs @@ -0,0 +1 @@ +pub mod chat; diff --git a/litellm-rust/crates/llms/src/gemini/chat/AGENTS.md b/litellm-rust/crates/llms/src/gemini/chat/AGENTS.md new file mode 100644 index 00000000000..8635b0d1403 --- /dev/null +++ b/litellm-rust/crates/llms/src/gemini/chat/AGENTS.md @@ -0,0 +1,11 @@ +# rules + +- Gemini on Google AI Studio: reuses `vertex_ai/gemini` for params, the body, the response and the stream, with `GeminiSurface::AiStudio` +- The API key goes in the `x-goog-api-key` header, never in the URL +- Gemini 3 and newer call `v1alpha`, older models `v1beta`; a custom `api_base` is called at `{api_base}/models/{model}:generateContent` + +# references + +- https://ai.google.dev/api/generate-content +- litellm/llms/gemini/chat/transformation.py +- litellm/llms/vertex_ai/vertex_llm_base.py diff --git a/litellm-rust/crates/llms/src/gemini/chat/mod.rs b/litellm-rust/crates/llms/src/gemini/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/gemini/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/gemini/chat/transformation.rs b/litellm-rust/crates/llms/src/gemini/chat/transformation.rs new file mode 100644 index 00000000000..acf8d4d6a48 --- /dev/null +++ b/litellm-rust/crates/llms/src/gemini/chat/transformation.rs @@ -0,0 +1,252 @@ +//! Python's `GoogleAIStudioGeminiConfig` for `/chat/completions`, with the key and URL +//! `_get_token_and_url` and `_check_custom_proxy` build for `custom_llm_provider="gemini"`. + +use litellm_auth::{CredentialPlacement, SecretValue}; +use litellm_llms_types::{ + formats::chat_completions::{ChatCompletionsResponse, ChatMessage}, + providers::vertex_ai::{GEMINI_API_BASE, GEMINI_API_KEY_HEADER, GENERATE_CONTENT}, +}; +use serde_json::{Map, Value}; + +use crate::{ + Error, + base_llm::{ + auth::{AuthScheme, Headers, ValidatedEnvironment}, + chat::{ + params::ChatRequestContext, + streaming::{ChatStream, StreamShape}, + transformation::{ + BaseConfig, ProviderChatRequestData, ProviderChatResponseData, Unsupported, + }, + }, + }, + vertex_ai::gemini::{ + handler::gemini_chat_stream, + transformation::remote_media, + vertex_and_google_ai_studio_gemini::{ + CONFIG_PARAMS, GeminiSurface, build_request, is_gemini_3_or_newer, streaming_url, + transform_generate_content_response, + }, + }, +}; + +pub const GEMINI_API_BASE_ENV: &str = "GEMINI_API_BASE"; +pub const GEMINI_API_KEY_ENV: &str = "GEMINI_API_KEY"; + +/// `get_api_key_from_env`, then the older PaLM key. +const API_KEY_ENVS: &[&str] = &["GOOGLE_API_KEY", GEMINI_API_KEY_ENV, "PALM_API_KEY"]; + +pub struct GoogleAiStudioGeminiConfig; + +pub const GOOGLE_AI_STUDIO_GEMINI_CHAT_COMPLETIONS_CONFIG: GoogleAiStudioGeminiConfig = + GoogleAiStudioGeminiConfig; + +/// The id AI Studio is called with: Python strips the provider prefix. +fn studio_model(model: &str) -> &str { + model.strip_prefix("gemini/").unwrap_or(model) +} + +fn api_key( + api_key: Option<&str>, + env_lookup: &dyn Fn(&str) -> Option, +) -> Result { + api_key + .filter(|key| !key.is_empty()) + .map(str::to_owned) + .or_else(|| { + API_KEY_ENVS + .iter() + .find_map(|name| env_lookup(name).filter(|key| !key.is_empty())) + }) + .ok_or(Error::Auth(litellm_auth::Error::MissingApiKey { + provider: "Gemini", + environment_variable: GEMINI_API_KEY_ENV, + })) +} + +impl BaseConfig for GoogleAiStudioGeminiConfig { + fn secret_names(&self) -> Vec<&'static str> { + API_KEY_ENVS + .iter() + .copied() + .chain([GEMINI_API_BASE_ENV]) + .collect() + } + + /// `get_gemini_url`, or `{api_base}/models/{model}:generateContent` for a custom base. + fn get_complete_url( + &self, + api_base: Option<&str>, + model: &str, + _optional_params: &Map, + env_lookup: &dyn Fn(&str) -> Option, + ) -> Result { + let model = studio_model(model); + let custom = api_base + .map(str::to_owned) + .or_else(|| env_lookup(GEMINI_API_BASE_ENV)) + .filter(|base| !base.is_empty()); + Ok(match custom { + Some(base) => format!("{base}/models/{model}:{GENERATE_CONTENT}"), + None => { + let version = match is_gemini_3_or_newer(model) { + true => "v1alpha", + false => "v1beta", + }; + format!("{GEMINI_API_BASE}/{version}/models/{model}:{GENERATE_CONTENT}") + } + }) + } + + fn transform_request( + &self, + model: &str, + messages: Vec, + optional_params: Map, + context: &ChatRequestContext<'_>, + ) -> Result { + build_request( + GeminiSurface::AiStudio, + model, + messages, + optional_params, + context, + ) + } + + fn transform_response( + &self, + model: &str, + response: ProviderChatResponseData, + _shape: &StreamShape, + ) -> Result { + transform_generate_content_response(model, response.body) + } + + fn remote_media(&self, messages: &[ChatMessage]) -> Vec { + remote_media(messages, GeminiSurface::AiStudio) + } + + fn model_response_iterator(&self, _shape: StreamShape) -> Option { + Some(gemini_chat_stream()) + } + + fn streaming_request(&self, url: String, body: Value) -> Result<(String, Value), Error> { + Ok((streaming_url(&url), body)) + } + + /// The key always wins over a forwarded header, as Python writes `x-goog-api-key` last. + fn validate_environment( + &self, + headers: Headers, + api_key_param: Option<&str>, + _model: &str, + _optional_params: &Map, + env_lookup: &dyn Fn(&str) -> Option, + ) -> Result { + Ok(ValidatedEnvironment { + headers, + auth: AuthScheme::Credential { + placement: CredentialPlacement::Header(GEMINI_API_KEY_HEADER), + secret: SecretValue::new(api_key(api_key_param, env_lookup)?), + }, + }) + } + + fn config_params(&self) -> &'static [&'static str] { + CONFIG_PARAMS + } + + /// Every request is translated; a param or message Python refuses is refused here too. + fn unsupported_reason( + &self, + _messages: &[ChatMessage], + _optional_params: &Map, + ) -> Option { + None + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + fn no_env(_: &str) -> Option { + None + } + + fn base_env(name: &str) -> Option { + (name == GEMINI_API_BASE_ENV).then(|| "https://proxy.example/gemini".to_owned()) + } + + fn key_env(name: &str) -> Option { + match name { + "GEMINI_API_KEY" => Some("gemini-key".into()), + "PALM_API_KEY" => Some("palm-key".into()), + _ => None, + } + } + + #[rstest] + #[case::v1beta(None, "gemini/gemini-2.5-flash", no_env as fn(&str) -> Option, "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent")] + #[case::v1alpha(None, "gemini-3-pro-preview", no_env as fn(&str) -> Option, "https://generativelanguage.googleapis.com/v1alpha/models/gemini-3-pro-preview:generateContent")] + #[case::custom(Some("https://gw.example/v1beta"), "gemini-2.5-flash", no_env as fn(&str) -> Option, "https://gw.example/v1beta/models/gemini-2.5-flash:generateContent")] + #[case::env_base(None, "gemini-2.5-flash", base_env as fn(&str) -> Option, "https://proxy.example/gemini/models/gemini-2.5-flash:generateContent")] + fn urls_follow_pythons_gemini_url( + #[case] api_base: Option<&str>, + #[case] model: &str, + #[case] env: fn(&str) -> Option, + #[case] expected: &str, + ) { + let url = GOOGLE_AI_STUDIO_GEMINI_CHAT_COMPLETIONS_CONFIG + .get_complete_url(api_base, model, &Map::new(), &env) + .unwrap(); + assert_eq!(url, expected); + } + + #[rstest] + #[case::explicit(Some("explicit"), "explicit")] + #[case::from_env(None, "gemini-key")] + fn the_key_goes_in_the_goog_header(#[case] key: Option<&str>, #[case] expected: &str) { + let validated = GOOGLE_AI_STUDIO_GEMINI_CHAT_COMPLETIONS_CONFIG + .validate_environment(Vec::new(), key, "gemini-2.5-flash", &Map::new(), &key_env) + .unwrap(); + assert!(matches!( + validated.auth, + AuthScheme::Credential { + placement: CredentialPlacement::Header("x-goog-api-key"), + ref secret + } if secret.expose() == expected + )); + } + + #[test] + fn a_missing_key_is_an_auth_error() { + let error = GOOGLE_AI_STUDIO_GEMINI_CHAT_COMPLETIONS_CONFIG + .validate_environment(Vec::new(), None, "gemini-2.5-flash", &Map::new(), &no_env) + .unwrap_err(); + assert!(matches!( + error, + Error::Auth(litellm_auth::Error::MissingApiKey { + provider: "Gemini", + .. + }) + )); + } + + #[test] + fn streams_call_stream_generate_content_with_sse() { + let (url, _) = GOOGLE_AI_STUDIO_GEMINI_CHAT_COMPLETIONS_CONFIG + .streaming_request( + "https://generativelanguage.googleapis.com/v1beta/models/m:generateContent".into(), + Value::Null, + ) + .unwrap(); + assert_eq!( + url, + "https://generativelanguage.googleapis.com/v1beta/models/m:streamGenerateContent?alt=sse" + ); + } +} diff --git a/litellm-rust/crates/llms/src/gemini/mod.rs b/litellm-rust/crates/llms/src/gemini/mod.rs new file mode 100644 index 00000000000..30a62fc508f --- /dev/null +++ b/litellm-rust/crates/llms/src/gemini/mod.rs @@ -0,0 +1 @@ +pub mod chat; diff --git a/litellm-rust/crates/llms/src/groq/chat/AGENTS.md b/litellm-rust/crates/llms/src/groq/chat/AGENTS.md new file mode 100644 index 00000000000..3a9f0278dc4 --- /dev/null +++ b/litellm-rust/crates/llms/src/groq/chat/AGENTS.md @@ -0,0 +1,12 @@ +# rules + +- Groq at `https://api.groq.com/openai/v1` +- A JSON schema on a model without native structured outputs goes as a forced `json_tool_call` tool, refused beside caller tools +- A streamed `response_format` is refused: Python fakes that stream from a complete call +- `web_search_options` adds the `browser_search` tool; `service_tier` in the answer is held to OpenAI's values +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://console.groq.com/docs/api-reference#chat +- litellm/llms/groq/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/groq/chat/mod.rs b/litellm-rust/crates/llms/src/groq/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/groq/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/groq/chat/transformation.rs b/litellm-rust/crates/llms/src/groq/chat/transformation.rs new file mode 100644 index 00000000000..5b1a15637f4 --- /dev/null +++ b/litellm-rust/crates/llms/src/groq/chat/transformation.rs @@ -0,0 +1,368 @@ +//! `litellm/llms/groq/chat/transformation.py`: Groq at `https://api.groq.com/openai/v1`. A +//! JSON schema a model cannot take natively goes as a forced `json_tool_call` tool whose call +//! comes back as content, `web_search_options` turns on the `browser_search` tool, and the +//! response's `service_tier` is held to the values OpenAI defines. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, ErrorDetail, + openai::chat::compatible::{ + CompatibleContext, GPT_BASE_PARAMS, OpenAiCompatibleChatConfig, RESPONSE_FORMAT_TOOL_NAME, + ResponseShape, copy_supported, gpt_messages, json_mode_tool_call_to_content, messages_mut, + role, sanitize_usage, strip_response_only_fields, + }, +}; + +const COMPOUND_MODELS: &[&str] = &["compound", "compound-mini"]; +const SERVICE_TIERS: &[&str] = &["auto", "default", "flex"]; + +pub struct GroqChatConfig; + +pub const GROQ_CHAT_CONFIG: GroqChatConfig = GroqChatConfig; + +impl OpenAiCompatibleChatConfig for GroqChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::Groq + } + + fn label(&self) -> &'static str { + "Groq" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["GROQ_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://api.groq.com/openai/v1") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &["GROQ_API_KEY"] + } + + fn supported_params(&self, context: &CompatibleContext<'_>) -> Vec<&'static str> { + let web_search = + is_compound(context.model) || context.supports(self.name(), "supports_web_search"); + let mut params: Vec<&'static str> = GPT_BASE_PARAMS + .iter() + .copied() + .filter(|param| *param != "max_retries") + .filter(|param| web_search || *param != "web_search_options") + .chain(["response_format"]) + .collect(); + if context.supports(self.name(), "supports_reasoning") { + params.push("reasoning_effort"); + } + params + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + let mut params = non_default.clone(); + if params.contains_key("response_format") + && params.get("stream") == Some(&Value::Bool(true)) + { + return Err(Error::Unsupported( + "streaming a groq response_format, which Python fakes from a complete call", + )); + } + if let Some(schema) = response_schema(params.get("response_format")) + && context.capability(self.name(), "supports_response_schema") != Some(true) + { + if params.contains_key("tools") { + return Err(Error::InvalidRequest(ErrorDetail::Message(format!( + "Groq model '{}' does not support native structured outputs. LiteLLM uses a \ + tool-calling workaround for structured outputs on this model, which is \ + incompatible with user-provided tools. Either use a model that supports \ + native structured outputs (e.g., gpt-oss-120b, llama-4, kimi-k2), or remove \ + the tools parameter. See: \ + https://console.groq.com/docs/structured-outputs#supported-models", + context.model + )))); + } + optional.insert( + "tools".into(), + json!([{"type": "function", "function": {"name": RESPONSE_FORMAT_TOOL_NAME, "parameters": schema}}]), + ); + optional.insert( + "tool_choice".into(), + json!({"type": "function", "function": {"name": RESPONSE_FORMAT_TOOL_NAME}}), + ); + optional.insert("json_mode".into(), Value::Bool(true)); + params.remove("response_format"); + } + let web_search = params.remove("web_search_options"); + copy_supported(¶ms, optional, &self.supported_params(context)); + if let Some(max_tokens) = optional.remove("max_completion_tokens") { + optional.insert("max_tokens".into(), max_tokens); + } + if web_search.is_none() || is_compound(context.model) { + return Ok(()); + } + let tools = optional + .entry("tools") + .or_insert_with(|| Value::Array(Vec::new())); + if tools.is_null() { + *tools = Value::Array(Vec::new()); + } + if let Value::Array(tools) = tools + && !tools + .iter() + .any(|tool| tool.get("type").and_then(Value::as_str) == Some("browser_search")) + { + tools.push(json!({"type": "browser_search"})); + } + Ok(()) + } + + fn transform_messages( + &self, + messages: Vec, + _optional: &Map, + _context: &CompatibleContext<'_>, + ) -> Result, Error> { + gpt_messages( + messages + .into_iter() + .map(|mut message| { + if role(&message) == Some("assistant") + && let Value::Object(object) = &mut message + { + object.retain(|_, value| !value.is_null()); + strip_response_only_fields(&mut message); + } + message + }) + .collect(), + ) + } + + fn fix_response(&self, body: &mut Map, shape: &ResponseShape) -> bool { + let mut changed = sanitize_usage(body); + if shape.json_mode { + changed |= json_mode_tool_call_to_content(body); + } + let tier = body.get("service_tier").and_then(Value::as_str); + if !tier.is_some_and(|tier| SERVICE_TIERS.contains(&tier)) { + body.insert("service_tier".into(), json!("auto")); + changed = true; + } + changed | add_web_search_usage(body) + } +} + +fn is_compound(model: &str) -> bool { + COMPOUND_MODELS.contains(&model.strip_prefix("groq/").unwrap_or(model)) +} + +fn response_schema(response_format: Option<&Value>) -> Option { + let format = response_format?.as_object()?; + match format.get("response_schema") { + Some(schema) => Some(schema.clone()), + None => format.get("json_schema")?.get("schema").cloned(), + } + .filter(|schema| !schema.is_null()) +} + +/// `_add_web_search_usage`: the searches and page opens Groq ran, from each message's +/// `executed_tools`. +fn add_web_search_usage(body: &mut Map) -> bool { + if !body.get("usage").is_some_and(Value::is_object) { + return false; + } + let actions: Vec = messages_mut(body) + .filter_map(|message| message.get("executed_tools")?.as_array()) + .flatten() + .filter_map(|tool| { + tool.get("name") + .and_then(Value::as_str) + .filter(|name| !name.is_empty()) + .or_else(|| tool.get("type").and_then(Value::as_str)) + .map(str::to_owned) + }) + .collect(); + let count = |names: &[&str]| { + actions + .iter() + .filter(|a| names.contains(&a.as_str())) + .count() + }; + let searches = count(&["browser.search", "browser_search"]); + let opens = count(&["browser.open"]); + if searches == 0 && opens == 0 { + return false; + } + if let Some(Value::Object(usage)) = body.get_mut("usage") { + usage.insert( + "server_tool_use".into(), + json!({"web_search_requests": searches, "browser_open_requests": opens}), + ); + } + true +} + +#[cfg(test)] +mod tests { + use std::borrow::Cow; + + use rstest::rstest; + + use super::*; + use crate::{ + base_llm::model_info::ModelInfo, + openai::chat::compatible::{fix_response, transform_request}, + }; + + fn object(value: Value) -> Map { + let Value::Object(map) = value else { + panic!("an object") + }; + map + } + + fn send(model: &str, params: Value) -> Result, Error> { + transform_request( + &GROQ_CHAT_CONFIG, + &CompatibleContext { + model, + drop_params: false, + model_info: None, + }, + object(params), + ) + .map(|sent| sent.body) + } + + #[rstest] + fn a_schema_without_native_support_goes_as_a_forced_tool() { + let sent = transform_request( + &GROQ_CHAT_CONFIG, + &CompatibleContext { + model: "llama-3.1-8b-instant", + drop_params: false, + model_info: None, + }, + object(json!({"messages": [], "max_completion_tokens": 3, + "response_format": {"type": "json_schema", "json_schema": {"name": "x", "schema": {"type": "object"}}}})), + ) + .unwrap(); + assert_eq!( + Value::Object(sent.body), + json!({"model": "llama-3.1-8b-instant", "messages": [], + "tools": [{"type": "function", "function": {"name": "json_tool_call", "parameters": {"type": "object"}}}], + "tool_choice": {"type": "function", "function": {"name": "json_tool_call"}}, + "max_tokens": 3}) + ); + assert!(sent.shape.json_mode); + } + + #[rstest] + fn a_schema_workaround_with_user_tools_is_rejected() { + assert!(matches!( + send( + "llama-3.1-8b-instant", + json!({"messages": [], "tools": [{"type": "function", "function": {"name": "f"}}], + "response_format": {"type": "json_schema", "json_schema": {"schema": {}}}}) + ), + Err(Error::InvalidRequest(ErrorDetail::Message(_))) + )); + } + + #[rstest] + fn a_streamed_response_format_is_refused() { + assert!(matches!( + send( + "llama-3.1-8b-instant", + json!({"messages": [], "stream": true, "response_format": {"type": "json_object"}}) + ), + Err(Error::Unsupported(_)) + )); + } + + struct WebSearch; + + impl ModelInfo for WebSearch { + fn entry(&self, model: &str) -> Option>> { + (model == "groq/openai/gpt-oss-20b").then(|| { + Cow::Owned(Map::from_iter([( + "supports_web_search".to_owned(), + json!(true), + )])) + }) + } + + fn declares(&self, _key: &str) -> bool { + true + } + } + + #[rstest] + #[case::adds_browser_search("openai/gpt-oss-20b", json!([{"type": "browser_search"}]))] + #[case::compound_searches_itself("compound", Value::Null)] + fn web_search_turns_on_browser_search(#[case] model: &str, #[case] tools: Value) { + let sent = transform_request( + &GROQ_CHAT_CONFIG, + &CompatibleContext { + model, + drop_params: false, + model_info: Some(&WebSearch), + }, + object(json!({"messages": [], "web_search_options": {"search_context_size": "low"}})), + ) + .unwrap() + .body; + assert_eq!(sent.get("tools").cloned().unwrap_or(Value::Null), tools); + assert!(!sent.contains_key("web_search_options")); + } + + #[rstest] + fn assistant_turns_lose_nulls_and_response_only_fields() { + let sent = send( + "llama-3.1-8b-instant", + json!({"messages": [ + {"role": "assistant", "content": "a", "function_call": null, "thinking_blocks": []}]}), + ) + .unwrap(); + assert_eq!( + sent["messages"], + json!([{"role": "assistant", "content": "a"}]) + ); + } + + #[rstest] + #[case::unknown(json!("on_demand"), json!("auto"))] + #[case::missing(Value::Null, json!("auto"))] + #[case::kept(json!("flex"), json!("flex"))] + fn service_tier_is_held_to_openai_values(#[case] tier: Value, #[case] expected: Value) { + let mut body = + object(json!({"choices": [], "service_tier": tier, "usage": {"prompt_tokens": null}})); + fix_response(&GROQ_CHAT_CONFIG, &mut body, &ResponseShape::default()); + assert_eq!(body["service_tier"], expected); + assert_eq!(body["usage"]["prompt_tokens"], json!(0)); + } + + #[rstest] + fn executed_browser_tools_reach_the_usage() { + let mut body = object( + json!({"service_tier": "auto", "usage": {"prompt_tokens": 1}, + "choices": [{"message": {"role": "assistant", "content": "x", + "executed_tools": [{"type": "browser_search"}, {"name": "browser.open"}, {"name": "browser.search"}]}}]}), + ); + assert!(fix_response( + &GROQ_CHAT_CONFIG, + &mut body, + &ResponseShape::default() + )); + assert_eq!( + body["usage"]["server_tool_use"], + json!({"web_search_requests": 2, "browser_open_requests": 1}) + ); + } +} diff --git a/litellm-rust/crates/llms/src/groq/mod.rs b/litellm-rust/crates/llms/src/groq/mod.rs new file mode 100644 index 00000000000..30a62fc508f --- /dev/null +++ b/litellm-rust/crates/llms/src/groq/mod.rs @@ -0,0 +1 @@ +pub mod chat; diff --git a/litellm-rust/crates/llms/src/hosted_vllm/chat/AGENTS.md b/litellm-rust/crates/llms/src/hosted_vllm/chat/AGENTS.md new file mode 100644 index 00000000000..67abee530f8 --- /dev/null +++ b/litellm-rust/crates/llms/src/hosted_vllm/chat/AGENTS.md @@ -0,0 +1,11 @@ +# rules + +- vLLM's OpenAI-compatible server at the deployment's `api_base`; there is no default host, and an unset key sends `fake-api-key` +- Tools lose `additionalProperties: false` and `strict`, custom tools go as function tools, `thinking` becomes `reasoning_effort` +- Replayed assistant turns lose `thinking_blocks` and non-string `reasoning_content`, content lists flatten to text +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html +- litellm/llms/hosted_vllm/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/hosted_vllm/chat/mod.rs b/litellm-rust/crates/llms/src/hosted_vllm/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/hosted_vllm/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/hosted_vllm/chat/transformation.rs b/litellm-rust/crates/llms/src/hosted_vllm/chat/transformation.rs new file mode 100644 index 00000000000..54fa0878d89 --- /dev/null +++ b/litellm-rust/crates/llms/src/hosted_vllm/chat/transformation.rs @@ -0,0 +1,385 @@ +//! `litellm/llms/hosted_vllm/chat/transformation.py`: a vLLM server at the deployment's +//! `api_base`. Tools lose `additionalProperties: false` and `strict` and custom tools become +//! function tools, `thinking` becomes a `reasoning_effort`, replayed assistant turns lose the +//! fields vLLM rejects with their content lists flattened, and video files go as `video_url`. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, + base_llm::chat::params::python_dumps, + openai::chat::compatible::{ + CompatibleContext, GPT_BASE_PARAMS, OpenAiCompatibleChatConfig, copy_supported, + gpt_messages, reasoning_effort_from_thinking_budget, role, + }, + vertex_ai::gemini::schema::{remove_additional_properties, remove_strict_from_schema}, +}; + +pub struct HostedVllmChatConfig; + +pub const HOSTED_VLLM_CHAT_CONFIG: HostedVllmChatConfig = HostedVllmChatConfig; + +impl OpenAiCompatibleChatConfig for HostedVllmChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::HostedVllm + } + + fn label(&self) -> &'static str { + "hosted vLLM" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["HOSTED_VLLM_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + None + } + + fn api_key_env(&self) -> &'static [&'static str] { + &["HOSTED_VLLM_API_KEY"] + } + + /// vLLM needs no key, and Python sends this one when none is set. + fn default_api_key(&self) -> Option<&'static str> { + Some("fake-api-key") + } + + fn supported_params(&self, _context: &CompatibleContext<'_>) -> Vec<&'static str> { + GPT_BASE_PARAMS + .iter() + .copied() + .chain(["response_format", "reasoning_effort", "thinking"]) + .collect() + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + let mut params = non_default.clone(); + if let Some(mut tools) = params.remove("tools").filter(|tools| !tools.is_null()) { + remove_additional_properties(&mut tools); + remove_strict_from_schema(&mut tools); + if let Value::Array(items) = tools { + tools = Value::Array( + items + .into_iter() + .enumerate() + .map(|(index, tool)| custom_tool_as_function(index, tool)) + .collect(), + ); + } + params.insert("tools".into(), tools); + } + if let Some(Value::Object(thinking)) = params.remove("thinking") + && thinking.get("type").and_then(Value::as_str) == Some("enabled") + && !params.contains_key("reasoning_effort") + { + let budget = thinking + .get("budget_tokens") + .and_then(Value::as_i64) + .unwrap_or(0); + params.insert( + "reasoning_effort".into(), + json!(reasoning_effort_from_thinking_budget(budget)), + ); + } + copy_supported(¶ms, optional, &self.supported_params(context)); + Ok(()) + } + + fn transform_messages( + &self, + messages: Vec, + _optional: &Map, + _context: &CompatibleContext<'_>, + ) -> Result, Error> { + let messages = messages + .into_iter() + .map(|mut message| { + match role(&message) { + Some("assistant") => assistant_for_vllm(&mut message), + Some("user") => video_files_as_video_urls(&mut message), + _ => {} + } + message + }) + .collect(); + gpt_messages(messages) + } +} + +/// `_convert_custom_tools_to_function_tools` for one tool. +fn custom_tool_as_function(index: usize, tool: Value) -> Value { + if tool.get("type").and_then(Value::as_str) != Some("custom") { + return tool; + } + let custom = tool + .get("custom") + .filter(|custom| custom.is_object()) + .cloned() + .unwrap_or_else(|| json!({})); + let pick = |key: &str| { + custom + .get(key) + .filter(|value| crate::base_llm::chat::params::truthy(value)) + .or_else(|| tool.get(key).filter(|value| !value.is_null())) + .cloned() + }; + let name = match pick("name") { + Some(Value::String(name)) => name, + Some(other) => python_str(&other), + None => format!("custom_tool_{index}"), + }; + let parameters = match pick("input_schema") { + Some(schema @ Value::Object(_)) => schema, + _ => json!({ + "type": "object", + "properties": {"input": {"type": "string", "description": "Raw tool input payload."}}, + "required": ["input"], + }), + }; + let mut function = Map::from_iter([ + ("name".to_owned(), Value::String(name)), + ("parameters".to_owned(), parameters), + ]); + if let Some(Value::String(description)) = pick("description") { + function.insert("description".into(), Value::String(description)); + } + json!({"type": "function", "function": function}) +} + +fn python_str(value: &Value) -> String { + match value { + Value::String(text) => text.clone(), + other => python_dumps(other), + } +} + +/// The assistant half of `HostedVLLMChatConfig._transform_messages`. +fn assistant_for_vllm(message: &mut Value) { + let Some(object) = message.as_object_mut() else { + return; + }; + object.remove("thinking_blocks"); + if !object + .get("reasoning_content") + .is_some_and(Value::is_string) + { + object.remove("reasoning_content"); + } + let Some(Value::Array(parts)) = object.get("content").cloned() else { + return; + }; + let mut texts = Vec::new(); + let mut tool_calls = Vec::new(); + let mut blocks = Vec::new(); + let mut structured = false; + for part in parts { + match part.get("type").and_then(Value::as_str) { + Some("text") if part.is_object() => { + texts.push( + part.get("text") + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + ); + blocks.push(part); + } + Some("tool_use") if part.is_object() => { + let input = part.get("input").cloned().unwrap_or_else(|| json!({})); + let arguments = match input { + Value::String(text) => text, + other => python_dumps(&other), + }; + tool_calls.push(json!({ + "id": part.get("id").cloned().unwrap_or(Value::Null), + "type": "function", + "function": {"name": part.get("name").cloned().unwrap_or(Value::Null), "arguments": arguments}, + })); + } + _ => { + blocks.push(part); + structured = true; + } + } + } + if !tool_calls.is_empty() { + match object.get_mut("tool_calls") { + Some(Value::Array(existing)) => { + let known: Vec = existing + .iter() + .filter_map(|call| call.get("id").filter(|id| !id.is_null()).cloned()) + .collect(); + existing.extend( + tool_calls + .into_iter() + .filter(|call| !known.contains(&call["id"])), + ); + } + _ => { + object.insert("tool_calls".into(), Value::Array(tool_calls)); + } + } + } + let content = match structured { + true => Value::Array(blocks), + false => Value::String(texts.join("\n")), + }; + object.insert("content".into(), content); +} + +/// `_is_video_file`: a `file` part whose format, data URL or file id names a video. +fn is_video_file(part: &Value) -> bool { + if part.get("type").and_then(Value::as_str) != Some("file") { + return false; + } + let file = part.get("file"); + let field = |key: &str| { + file.and_then(|file| file.get(key)) + .and_then(Value::as_str) + .filter(|value| !value.is_empty()) + }; + if field("format").is_some_and(|format| format.starts_with("video/")) { + return true; + } + if let Some(data) = field("file_data") { + return data_url_mime(data).is_some_and(|mime| mime.starts_with("video/")); + } + if let Some(id) = field("file_id") { + return url_video_mime(id); + } + false +} + +/// `parse_mime_type`: the media type of a `data:` URL. +fn data_url_mime(data: &str) -> Option<&str> { + let rest = data.strip_prefix("data:")?; + let end = rest.find([';', ','])?; + Some(&rest[..end]) +} + +/// `get_image_mime_type_from_url` for the video extensions it knows. +fn url_video_mime(url: &str) -> bool { + let path = url.split(['?', '#']).next().unwrap_or(url).to_lowercase(); + [ + ".mp4", ".mov", ".avi", ".flv", ".mpeg", ".mpg", ".webm", ".wmv", ".3gpp", + ] + .iter() + .any(|extension| path.ends_with(extension)) +} + +fn video_files_as_video_urls(message: &mut Value) { + let Some(Value::Array(parts)) = message.get_mut("content") else { + return; + }; + for part in parts.iter_mut() { + if !is_video_file(part) { + continue; + } + let file = part.get("file").cloned().unwrap_or_default(); + let url = ["file_id", "file_data"] + .iter() + .find_map(|key| { + file.get(*key) + .filter(|value| crate::base_llm::chat::params::truthy(value)) + }) + .cloned(); + if let Some(url) = url { + *part = json!({"type": "video_url", "video_url": {"url": url}}); + } + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::openai::chat::compatible::transform_request; + + fn send(params: Value) -> Map { + let Value::Object(body) = params else { + panic!("an object") + }; + transform_request( + &HOSTED_VLLM_CHAT_CONFIG, + &CompatibleContext { + model: "Qwen/Qwen3-8B", + drop_params: false, + model_info: None, + }, + body, + ) + .unwrap() + .body + } + + #[rstest] + fn tools_lose_strict_and_closed_objects_and_custom_tools_become_functions() { + let sent = send(json!({"messages": [], "tools": [ + {"type": "function", "function": {"name": "f", "strict": true, + "parameters": {"type": "object", "additionalProperties": false, "properties": {}}}}, + {"type": "custom", "custom": {"name": "grep", "description": "search"}}, + ]})); + assert_eq!( + sent["tools"], + json!([ + {"type": "function", "function": {"name": "f", "parameters": {"type": "object", "properties": {}}}}, + {"type": "function", "function": {"name": "grep", "description": "search", "parameters": { + "type": "object", + "properties": {"input": {"type": "string", "description": "Raw tool input payload."}}, + "required": ["input"]}}}, + ]) + ); + } + + #[rstest] + #[case::budget(json!({"thinking": {"type": "enabled", "budget_tokens": 3000}}), Some("medium"))] + #[case::effort_wins(json!({"thinking": {"type": "enabled", "budget_tokens": 3000}, "reasoning_effort": "low"}), Some("low"))] + #[case::disabled(json!({"thinking": {"type": "disabled"}}), None)] + fn thinking_becomes_a_reasoning_effort(#[case] params: Value, #[case] effort: Option<&str>) { + let mut body = json!({"messages": []}); + body.as_object_mut() + .unwrap() + .extend(params.as_object().unwrap().clone()); + let sent = send(body); + assert!(!sent.contains_key("thinking")); + assert_eq!(sent.get("reasoning_effort").and_then(Value::as_str), effort); + } + + #[rstest] + fn assistant_turns_are_flattened_for_vllm() { + let sent = send(json!({"messages": [ + {"role": "assistant", "thinking_blocks": [{"type": "thinking"}], "reasoning_content": {"x": 1}, + "content": [{"type": "text", "text": "a"}, {"type": "text", "text": "b"}, + {"type": "tool_use", "id": "t1", "name": "f", "input": {"q": 1}}]}, + {"role": "user", "content": [{"type": "file", "file": {"file_data": "data:video/mp4;base64,AA"}}]}, + ]})); + assert_eq!( + sent["messages"], + json!([ + {"role": "assistant", "content": "a\nb", "tool_calls": [ + {"id": "t1", "type": "function", "function": {"name": "f", "arguments": "{\"q\": 1}"}}]}, + {"role": "user", "content": [{"type": "video_url", "video_url": {"url": "data:video/mp4;base64,AA"}}]}, + ]) + ); + } + + #[rstest] + fn the_key_defaults_for_a_server_that_takes_none() { + assert_eq!( + HOSTED_VLLM_CHAT_CONFIG.api_key(None, &|_| None).unwrap(), + "fake-api-key" + ); + } + + #[rstest] + fn an_api_base_is_required() { + assert!(HOSTED_VLLM_CHAT_CONFIG.api_base(None, &|_| None).is_err()); + } +} diff --git a/litellm-rust/crates/llms/src/hosted_vllm/mod.rs b/litellm-rust/crates/llms/src/hosted_vllm/mod.rs new file mode 100644 index 00000000000..30a62fc508f --- /dev/null +++ b/litellm-rust/crates/llms/src/hosted_vllm/mod.rs @@ -0,0 +1 @@ +pub mod chat; diff --git a/litellm-rust/crates/llms/src/lib.rs b/litellm-rust/crates/llms/src/lib.rs index 23ed649ac8e..bb6f00350b9 100644 --- a/litellm-rust/crates/llms/src/lib.rs +++ b/litellm-rust/crates/llms/src/lib.rs @@ -5,17 +5,26 @@ pub mod azure_ai; pub mod base_llm; pub mod bedrock; pub mod bedrock_mantle; +pub mod cerebras; pub mod cohere; pub mod deepseek; pub mod edenai; mod error; +pub mod fireworks_ai; +pub mod gemini; +pub mod groq; +pub mod hosted_vllm; pub mod minimax; pub mod mistral; +pub mod nvidia_nim; pub mod openai; pub mod openai_like; pub mod openrouter; +pub mod perplexity; pub mod reducto; pub mod tencent; +pub mod together_ai; pub mod vertex_ai; +pub mod xai; pub use error::{Error, ErrorDetail, ErrorSource}; diff --git a/litellm-rust/crates/llms/src/mistral/chat/AGENTS.md b/litellm-rust/crates/llms/src/mistral/chat/AGENTS.md new file mode 100644 index 00000000000..8b8f1a7bc0f --- /dev/null +++ b/litellm-rust/crates/llms/src/mistral/chat/AGENTS.md @@ -0,0 +1,11 @@ +# rules + +- Mistral at `https://api.mistral.ai/v1`; a base without `/v1` gets it appended +- `tool_choice` narrows to `auto`/`any`/`none`, `seed` goes as `random_seed`, magistral reasoning adds Mistral's system prompt +- Content blocks in answers and chunks become text plus `reasoning_content` +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://docs.mistral.ai/api/#tag/chat +- litellm/llms/mistral/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/mistral/chat/mod.rs b/litellm-rust/crates/llms/src/mistral/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/mistral/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/mistral/chat/transformation.rs b/litellm-rust/crates/llms/src/mistral/chat/transformation.rs new file mode 100644 index 00000000000..780f603c343 --- /dev/null +++ b/litellm-rust/crates/llms/src/mistral/chat/transformation.rs @@ -0,0 +1,605 @@ +//! `litellm/llms/mistral/chat/transformation.py`: Mistral at `https://api.mistral.ai/v1`. +//! Tool choices narrow to Mistral's `auto`/`any`/`none`, `seed` goes as `random_seed`, +//! magistral models get Mistral's reasoning system prompt, replayed messages lose what +//! Mistral's schema forbids, and content lists in the answer become text plus +//! `reasoning_content`. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, + openai::chat::compatible::{ + CompatibleContext, OpenAiCompatibleChatConfig, ResponseShape, content_list_to_str, + gpt_messages, messages_mut, nearest_declared_reasoning_effort, role, strip_internal_fields, + }, +}; + +const REASONING_PROMPT_FLAG: &str = "_add_reasoning_prompt"; +/// `_get_mistral_reasoning_system_prompt`, verbatim with its indentation. +const REASONING_SYSTEM_PROMPT: &str = r" + [SYSTEM_PROMPT]system_prompt + A user will ask you to solve a task. You should first draft your thinking process (inner monologue) until you have derived the final answer. Afterwards, write a self-contained summary of your thoughts (i.e. your summary should be succinct but contain all the critical steps you needed to reach the conclusion). You should use Markdown to format your response. Write both your thoughts and summary in the same language as the task posed by the user. NEVER use \boxed{} in your response. + + Your thinking process must follow the template below: + + Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct answer. + + + Here, provide a concise summary that reflects your reasoning and presents a clear final answer to the user. Don't mention that this is a summary. + + Problem: + + [/SYSTEM_PROMPT][INST]user_message[/INST] + reasoning_traces + + assistant_response[INST]user_message[/INST] + "; +/// The keys of `ChatCompletionAssistantMessage` besides `role`. +const ASSISTANT_MESSAGE_KEYS: &[&str] = &[ + "content", + "name", + "tool_calls", + "function_call", + "reasoning_content", + "cache_control", + "thinking_blocks", + "reasoning_items", +]; + +pub struct MistralChatConfig; + +pub const MISTRAL_CHAT_CONFIG: MistralChatConfig = MistralChatConfig; + +impl OpenAiCompatibleChatConfig for MistralChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::Mistral + } + + fn label(&self) -> &'static str { + "Mistral" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["MISTRAL_AZURE_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://api.mistral.ai/v1") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &["MISTRAL_AZURE_API_KEY", "MISTRAL_API_KEY"] + } + + /// Mistral always needs `/v1` at the end of the base. + fn api_base( + &self, + configured: Option<&str>, + lookup: &dyn Fn(&str) -> Option, + ) -> Result { + let base = configured + .filter(|base| !base.is_empty()) + .map(str::to_owned) + .or_else(|| lookup("MISTRAL_AZURE_API_BASE")) + .unwrap_or_else(|| "https://api.mistral.ai/v1".to_owned()); + Ok(match base.ends_with("/v1") { + true => base, + false => format!("{base}/v1"), + }) + } + + fn supported_params(&self, context: &CompatibleContext<'_>) -> Vec<&'static str> { + let magistral = is_magistral(context.model); + let mut params = vec![ + "stream", + "temperature", + "top_p", + "max_tokens", + "max_completion_tokens", + "tools", + "tool_choice", + "seed", + "stop", + "response_format", + "parallel_tool_calls", + ]; + if magistral { + params.push("thinking"); + } + if magistral || context.supports(self.name(), "supports_reasoning") { + params.push("reasoning_effort"); + } + params + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + let magistral = is_magistral(context.model); + for (name, value) in non_default { + match name.as_str() { + "max_tokens" if !non_default.contains_key("max_completion_tokens") => { + optional.insert("max_tokens".into(), value.clone()); + } + "max_completion_tokens" => { + optional.insert("max_tokens".into(), value.clone()); + } + "tools" => { + let mut tools = value.clone(); + remove_json_schema_refs(&mut tools, 10); + optional.insert("tools".into(), tools); + } + "stream" if value == &Value::Bool(true) => { + optional.insert("stream".into(), value.clone()); + } + "temperature" | "top_p" | "stop" | "response_format" | "parallel_tool_calls" => { + optional.insert(name.clone(), value.clone()); + } + "tool_choice" => { + if let Some(choice) = value.as_str() { + let choice = match choice { + "auto" | "none" => choice, + _ => "any", + }; + optional.insert("tool_choice".into(), json!(choice)); + } + } + "seed" => { + optional.insert("extra_body".into(), json!({"random_seed": value})); + } + "reasoning_effort" | "thinking" if magistral => { + optional.insert(REASONING_PROMPT_FLAG.into(), Value::Bool(true)); + } + "reasoning_effort" => { + let effort = match ( + value.as_str(), + context.declared_reasoning_efforts(self.name()), + ) { + (Some(requested), Some(declared)) => { + json!(nearest_declared_reasoning_effort(requested, &declared)) + } + _ => value.clone(), + }; + optional.insert("reasoning_effort".into(), effort); + } + _ => {} + } + } + Ok(()) + } + + fn map_extra_body( + &self, + extra_body: &mut Map, + _optional: &mut Map, + _context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + extra_body.remove("client_metadata"); + Ok(()) + } + + fn transform_messages( + &self, + messages: Vec, + optional: &Map, + context: &CompatibleContext<'_>, + ) -> Result, Error> { + let mut messages = messages; + if is_magistral(context.model) && optional.contains_key(REASONING_PROMPT_FLAG) { + add_reasoning_system_prompt(&mut messages); + } + for message in messages.iter_mut() { + if role(message) == Some("assistant") { + strip_internal_fields(message); + } + } + let has_media = messages.iter().any(|message| { + message + .get("content") + .and_then(Value::as_array) + .is_some_and(|parts| { + parts.iter().any(|part| { + matches!( + part.get("type").and_then(Value::as_str), + Some("image_url" | "file") + ) + }) + }) + }); + if has_media { + let mut messages = gpt_messages(messages)?; + messages.iter_mut().for_each(file_parts_to_file_ids); + return Ok(messages); + } + let messages = messages + .into_iter() + .map(|mut message| { + let text = content_list_to_str(&message); + if let Some(object) = message.as_object_mut() { + if !text.is_empty() { + object.insert("content".into(), Value::String(text)); + } + keep_name_on_tool_messages(object); + rebuild_tool_calls(object); + } + message + }) + .filter(|message| !is_empty_assistant_message(message)) + .map(|mut message| { + if let Some(object) = message.as_object_mut() { + object.retain(|_, value| !value.is_null()); + } + message + }) + .collect(); + gpt_messages(messages) + } + + fn finish_request( + &self, + body: &mut Map, + _context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + body.remove(REASONING_PROMPT_FLAG); + body.remove("client_metadata"); + Ok(()) + } + + fn fix_response(&self, body: &mut Map, _shape: &ResponseShape) -> bool { + let mut changed = false; + for message in messages_mut(body) { + match message.get("content") { + Some(Value::String(content)) if content.is_empty() => { + message.insert("content".into(), Value::Null); + changed = true; + } + Some(Value::Array(blocks)) if !blocks.is_empty() => { + let (text, thinking) = split_content_blocks(blocks); + message.insert("content".into(), Value::String(text)); + if !thinking.is_empty() { + message.insert("reasoning_content".into(), Value::String(thinking)); + } + changed = true; + } + _ => {} + } + } + changed + } + + fn fix_chunk(&self, chunk: &mut Map, _shape: &ResponseShape) -> bool { + let mut changed = false; + let Some(Value::Array(choices)) = chunk.get_mut("choices") else { + return false; + }; + for delta in choices + .iter_mut() + .filter_map(|choice| choice.get_mut("delta")?.as_object_mut()) + { + let Some(Value::Array(blocks)) = delta.get("content") else { + continue; + }; + let (text, thinking_blocks, reasoning) = normalize_stream_blocks(blocks); + delta.insert("content".into(), text.map_or(Value::Null, Value::String)); + match reasoning { + Some(reasoning) if !thinking_blocks.is_empty() => { + delta.insert("thinking_blocks".into(), Value::Array(thinking_blocks)); + delta.insert("reasoning_content".into(), Value::String(reasoning)); + } + _ => { + delta.remove("thinking_blocks"); + delta.remove("reasoning_content"); + } + } + changed = true; + } + changed + } +} + +fn is_magistral(model: &str) -> bool { + model.to_lowercase().contains("magistral") +} + +/// `remove_json_schema_refs`. +fn remove_json_schema_refs(value: &mut Value, depth: usize) { + if depth == 0 { + return; + } + match value { + Value::Object(object) => { + object.remove("$id"); + object.remove("$schema"); + object + .values_mut() + .for_each(|nested| remove_json_schema_refs(nested, depth - 1)); + } + Value::Array(items) => items + .iter_mut() + .for_each(|item| remove_json_schema_refs(item, depth - 1)), + _ => {} + } +} + +/// `_add_reasoning_system_prompt_if_needed`. +fn add_reasoning_system_prompt(messages: &mut Vec) { + let Some(system) = messages + .iter_mut() + .find(|message| role(message) == Some("system")) + .and_then(Value::as_object_mut) + else { + messages.insert( + 0, + json!({"role": "system", "content": REASONING_SYSTEM_PROMPT}), + ); + return; + }; + let content = match system.remove("content") { + Some(Value::String(existing)) => json!(format!("{REASONING_SYSTEM_PROMPT}\n\n{existing}")), + Some(Value::Array(existing)) => Value::Array( + std::iter::once( + json!({"type": "text", "text": format!("{REASONING_SYSTEM_PROMPT}\n\n")}), + ) + .chain(existing) + .collect(), + ), + Some(Value::Null) => json!(format!("{REASONING_SYSTEM_PROMPT}\n\nNone")), + Some(other) => json!(format!("{REASONING_SYSTEM_PROMPT}\n\n{other}")), + None => json!(format!("{REASONING_SYSTEM_PROMPT}\n\n")), + }; + system.insert("content".into(), content); +} + +/// `_handle_message_with_file`: Mistral takes a file part by its `file_id`. +fn file_parts_to_file_ids(message: &mut Value) { + let Some(Value::Array(parts)) = message.get_mut("content") else { + return; + }; + for part in parts + .iter_mut() + .filter_map(Value::as_object_mut) + .filter(|part| part.get("type").and_then(Value::as_str) == Some("file")) + { + let file_id = part + .get("file") + .and_then(|file| file.get("file_id")) + .filter(|id| crate::base_llm::chat::params::truthy(id)) + .cloned(); + if let Some(file_id) = file_id { + part.insert("file_id".into(), file_id); + part.remove("file"); + } + } +} + +/// `_handle_name_in_message`: Mistral takes `name` only on tool messages. +fn keep_name_on_tool_messages(message: &mut Map) { + let Some(name) = message.get("name").filter(|name| !name.is_null()) else { + return; + }; + let blank = name.as_str().is_some_and(|name| name.trim().is_empty()); + if message.get("role").and_then(Value::as_str) != Some("tool") || blank { + message.remove("name"); + } +} + +/// `_handle_tool_call_message`: each call as `{id, type: "function", function}`. +fn rebuild_tool_calls(message: &mut Map) { + let Some(Value::Array(calls)) = message.get_mut("tool_calls") else { + return; + }; + for call in calls.iter_mut() { + *call = json!({ + "id": call.get("id").cloned().unwrap_or(Value::Null), + "type": "function", + "function": call.get("function").cloned().unwrap_or(Value::Null), + }); + } +} + +/// `_is_empty_assistant_message`. +fn is_empty_assistant_message(message: &Value) -> bool { + role(message) == Some("assistant") + && ASSISTANT_MESSAGE_KEYS + .iter() + .all(|key| match message.get(*key) { + None | Some(Value::Null) => true, + Some(Value::String(content)) => *key == "content" && content.is_empty(), + Some(_) => false, + }) +} + +/// `_handle_content_list_to_str_conversion` for one message: the last text block, and the +/// last thinking block's text parts joined by newlines. +fn split_content_blocks(blocks: &[Value]) -> (String, String) { + let mut text = String::new(); + let mut thinking = String::new(); + for block in blocks { + match block.get("type").and_then(Value::as_str) { + Some("thinking") => thinking = thinking_texts(block).join("\n"), + Some("text") => { + text = block + .get("text") + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(); + } + _ => {} + } + } + (text, thinking) +} + +fn thinking_texts(block: &Value) -> Vec<&str> { + block + .get("thinking") + .and_then(Value::as_array) + .into_iter() + .flatten() + .filter(|part| part.get("type").and_then(Value::as_str) == Some("text")) + .map(|part| part.get("text").and_then(Value::as_str).unwrap_or_default()) + .collect() +} + +/// `MistralChatResponseIterator._normalize_content_blocks`. +fn normalize_stream_blocks(blocks: &[Value]) -> (Option, Vec, Option) { + let mut text: Option = None; + let mut thinking_blocks = Vec::new(); + let mut reasoning = Vec::new(); + for block in blocks { + match block.get("type").and_then(Value::as_str) { + Some("thinking") => { + let thought = thinking_texts(block).concat(); + if !thought.is_empty() { + thinking_blocks.push( + json!({"type": "thinking", "thinking": thought, "signature": "mistral"}), + ); + reasoning.push(thought); + } + } + Some("text") => text.get_or_insert_with(String::new).push_str( + block + .get("text") + .and_then(Value::as_str) + .unwrap_or_default(), + ), + _ => {} + } + } + let reasoning = (!reasoning.is_empty()).then(|| reasoning.join("\n")); + (text, thinking_blocks, reasoning) +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::openai::chat::compatible::{fix_response, transform_request}; + + fn object(value: Value) -> Map { + let Value::Object(map) = value else { + panic!("an object") + }; + map + } + + fn send(model: &str, params: Value) -> Map { + transform_request( + &MISTRAL_CHAT_CONFIG, + &CompatibleContext { + model, + drop_params: false, + model_info: None, + }, + object(params), + ) + .unwrap() + .body + } + + #[rstest] + #[case::bare("https://example.ai", "https://example.ai/v1")] + #[case::versioned("https://example.ai/v1", "https://example.ai/v1")] + fn the_base_always_ends_in_v1(#[case] configured: &str, #[case] expected: &str) { + assert_eq!( + MISTRAL_CHAT_CONFIG + .api_base(Some(configured), &|_| None) + .unwrap(), + expected + ); + } + + #[rstest] + fn params_take_mistral_names() { + let sent = send( + "mistral-large-latest", + json!({"messages": [], "max_tokens": 1, + "max_completion_tokens": 9, "seed": 3, "stream": false, "tool_choice": "required", + "tools": [{"type": "function", "function": {"name": "f", "parameters": {"$schema": "s", "$id": "i", "type": "object"}}}]}), + ); + assert_eq!( + Value::Object(sent), + json!({"model": "mistral-large-latest", "messages": [], "max_tokens": 9, + "tools": [{"type": "function", "function": {"name": "f", "parameters": {"type": "object"}}}], + "tool_choice": "any", "random_seed": 3}) + ); + } + + #[rstest] + fn magistral_reasoning_adds_the_system_prompt_instead_of_the_param() { + let sent = send( + "magistral-medium-latest", + json!({"reasoning_effort": "high", + "messages": [{"role": "system", "content": "be brief"}]}), + ); + assert_eq!( + sent["messages"], + json!([{"role": "system", "content": format!("{REASONING_SYSTEM_PROMPT}\n\nbe brief")}]) + ); + assert!(!sent.contains_key("reasoning_effort")); + assert!(!sent.contains_key(REASONING_PROMPT_FLAG)); + } + + #[rstest] + fn replayed_messages_lose_what_mistral_forbids() { + let sent = send( + "mistral-small", + json!({"messages": [ + {"role": "user", "name": "u", "content": [{"type": "text", "text": "hi"}]}, + {"role": "assistant", "content": "", "reasoning_content": "r", "thinking_blocks": []}, + {"role": "assistant", "content": null, "tool_calls": [{"index": 0, "id": "c", "type": "function", "function": {"name": "f", "arguments": "{}"}}]}, + {"role": "tool", "name": "f", "tool_call_id": "c", "content": "ok"}]}), + ); + assert_eq!( + sent["messages"], + json!([ + {"role": "user", "content": "hi"}, + {"role": "assistant", "tool_calls": [{"id": "c", "type": "function", "function": {"name": "f", "arguments": "{}"}}]}, + {"role": "tool", "name": "f", "tool_call_id": "c", "content": "ok"}]) + ); + } + + #[rstest] + fn file_parts_go_by_file_id() { + let sent = send( + "mistral-small", + json!({"messages": [{"role": "user", "content": [ + {"type": "text", "text": "read"}, {"type": "file", "file": {"file_id": "abc"}}]}]}), + ); + assert_eq!( + sent["messages"][0]["content"], + json!([{"type": "text", "text": "read"}, {"type": "file", "file_id": "abc"}]) + ); + } + + #[rstest] + #[case::empty(json!(""), json!({"role": "assistant", "content": null}))] + #[case::blocks(json!([{"type": "thinking", "thinking": [{"type": "text", "text": "a"}, {"type": "text", "text": "b"}]}, {"type": "text", "text": "done"}]), + json!({"role": "assistant", "content": "done", "reasoning_content": "a\nb"}))] + fn answers_become_text_and_reasoning(#[case] content: Value, #[case] message: Value) { + let mut body = + object(json!({"choices": [{"message": {"role": "assistant", "content": content}}]})); + assert!(fix_response( + &MISTRAL_CHAT_CONFIG, + &mut body, + &ResponseShape::default() + )); + assert_eq!(body["choices"][0]["message"], message); + } + + #[rstest] + fn streamed_blocks_become_text_and_thinking() { + let mut chunk = object(json!({"choices": [{"index": 0, "delta": {"content": [ + {"type": "thinking", "thinking": [{"type": "text", "text": "hm"}]}, {"type": "text", "text": "ok"}]}}]})); + assert!(MISTRAL_CHAT_CONFIG.fix_chunk(&mut chunk, &ResponseShape::default())); + assert_eq!( + chunk["choices"][0]["delta"], + json!({"content": "ok", "thinking_blocks": [{"type": "thinking", "thinking": "hm", "signature": "mistral"}], "reasoning_content": "hm"}) + ); + } +} diff --git a/litellm-rust/crates/llms/src/mistral/mod.rs b/litellm-rust/crates/llms/src/mistral/mod.rs index 3621ff6a2fd..5a7774dbe88 100644 --- a/litellm-rust/crates/llms/src/mistral/mod.rs +++ b/litellm-rust/crates/llms/src/mistral/mod.rs @@ -1 +1,2 @@ +pub mod chat; pub mod ocr; diff --git a/litellm-rust/crates/llms/src/nvidia_nim/chat/AGENTS.md b/litellm-rust/crates/llms/src/nvidia_nim/chat/AGENTS.md new file mode 100644 index 00000000000..54690606483 --- /dev/null +++ b/litellm-rust/crates/llms/src/nvidia_nim/chat/AGENTS.md @@ -0,0 +1,11 @@ +# rules + +- NIM's hosted API through the OpenAI SDK URL rule: `{api_base}/chat/completions` +- The supported params depend on the model family; `max_completion_tokens` goes as `max_tokens` +- No `stream_options`, so stream usage is only what NIM sends unasked +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://docs.api.nvidia.com/nim/reference/llm-apis +- litellm/llms/nvidia_nim/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/nvidia_nim/chat/mod.rs b/litellm-rust/crates/llms/src/nvidia_nim/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/nvidia_nim/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/nvidia_nim/chat/transformation.rs b/litellm-rust/crates/llms/src/nvidia_nim/chat/transformation.rs new file mode 100644 index 00000000000..bd3dd84c7fe --- /dev/null +++ b/litellm-rust/crates/llms/src/nvidia_nim/chat/transformation.rs @@ -0,0 +1,196 @@ +//! `litellm/llms/nvidia_nim/chat/transformation.py`: NVIDIA NIM through the OpenAI SDK at +//! `https://integrate.api.nvidia.com/v1`, with the params each model family takes and +//! `max_completion_tokens` sent as `max_tokens`. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value}; + +use crate::{ + Error, + openai::chat::compatible::{CompatibleContext, OpenAiCompatibleChatConfig, sdk_complete_url}, +}; + +const GEMMA_MODELS: &[&str] = &[ + "google/recurrentgemma-2b", + "google/gemma-2-27b-it", + "google/gemma-2-9b-it", + "gemma-2-9b-it", +]; + +pub struct NvidiaNimChatConfig; + +pub const NVIDIA_NIM_CHAT_CONFIG: NvidiaNimChatConfig = NvidiaNimChatConfig; + +impl OpenAiCompatibleChatConfig for NvidiaNimChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::NvidiaNim + } + + fn label(&self) -> &'static str { + "NVIDIA NIM" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["NVIDIA_NIM_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://integrate.api.nvidia.com/v1") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &["NVIDIA_NIM_API_KEY"] + } + + fn complete_url(&self, api_base: &str) -> String { + sdk_complete_url(api_base) + } + + fn supported_params(&self, context: &CompatibleContext<'_>) -> Vec<&'static str> { + let model = context.model; + if GEMMA_MODELS.contains(&model) { + return vec![ + "stream", + "temperature", + "top_p", + "max_tokens", + "stop", + "seed", + ]; + } + match model { + "nvidia/nemotron-4-340b-instruct" => vec![ + "stream", + "temperature", + "top_p", + "max_tokens", + "max_completion_tokens", + ], + "nvidia/nemotron-4-340b-reward" => vec!["stream"], + "google/codegemma-1.1-7b" => vec![ + "stream", + "temperature", + "top_p", + "frequency_penalty", + "presence_penalty", + "max_tokens", + "max_completion_tokens", + "stop", + ], + _ => vec![ + "stream", + "temperature", + "top_p", + "frequency_penalty", + "presence_penalty", + "max_tokens", + "max_completion_tokens", + "stop", + "seed", + "tools", + "tool_choice", + "parallel_tool_calls", + "response_format", + ], + } + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + let supported = self.supported_params(context); + for (name, value) in non_default { + if name == "max_completion_tokens" { + optional.insert("max_tokens".into(), value.clone()); + } else if supported.contains(&name.as_str()) { + optional.insert(name.clone(), value.clone()); + } + } + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + use crate::{ErrorDetail, openai::chat::compatible::transform_request}; + + fn context(model: &str, drop_params: bool) -> CompatibleContext<'_> { + CompatibleContext { + model, + drop_params, + model_info: None, + } + } + + fn body(value: Value) -> Map { + let Value::Object(map) = value else { + panic!("an object") + }; + map + } + + #[rstest] + fn max_completion_tokens_goes_as_max_tokens() { + let sent = transform_request( + &NVIDIA_NIM_CHAT_CONFIG, + &context("meta/llama3-70b-instruct", false), + body( + json!({"model": "x", "messages": [{"role": "user", "content": "hi"}], + "max_completion_tokens": 9, "seed": 1, "top_k": 4}), + ), + ) + .unwrap(); + assert_eq!( + Value::Object(sent.body), + json!({"model": "meta/llama3-70b-instruct", "messages": [{"role": "user", "content": "hi"}], + "max_tokens": 9, "seed": 1, "top_k": 4}) + ); + } + + #[rstest] + #[case::refused(false, true)] + #[case::dropped(true, false)] + fn a_param_the_model_family_lacks_is_refused_or_dropped( + #[case] drop_params: bool, + #[case] refused: bool, + ) { + let result = transform_request( + &NVIDIA_NIM_CHAT_CONFIG, + &context("google/gemma-2-9b-it", drop_params), + body( + json!({"messages": [], "tools": [{"type": "function", "function": {"name": "f"}}]}), + ), + ); + match refused { + true => assert!(matches!( + result, + Err(Error::InvalidRequest(ErrorDetail::UnsupportedParams { + provider: "nvidia_nim", + .. + })) + )), + false => assert!(!result.unwrap().body.contains_key("tools")), + } + } + + #[rstest] + fn stream_options_are_not_sent_since_nim_does_not_list_them() { + let sent = transform_request( + &NVIDIA_NIM_CHAT_CONFIG, + &context("meta/llama3-70b-instruct", false), + body( + json!({"messages": [], "stream": true, "stream_options": {"include_usage": true}}), + ), + ) + .unwrap(); + assert!(!sent.body.contains_key("stream_options")); + assert!(sent.shape.client_usage); + } +} diff --git a/litellm-rust/crates/llms/src/nvidia_nim/mod.rs b/litellm-rust/crates/llms/src/nvidia_nim/mod.rs new file mode 100644 index 00000000000..30a62fc508f --- /dev/null +++ b/litellm-rust/crates/llms/src/nvidia_nim/mod.rs @@ -0,0 +1 @@ +pub mod chat; diff --git a/litellm-rust/crates/llms/src/openai/chat/AGENTS.md b/litellm-rust/crates/llms/src/openai/chat/AGENTS.md index 356742bc872..f38c3e29313 100644 --- a/litellm-rust/crates/llms/src/openai/chat/AGENTS.md +++ b/litellm-rust/crates/llms/src/openai/chat/AGENTS.md @@ -4,3 +4,4 @@ - Passthrough, not a `BaseConfig`: that contract parses messages and normalizes the response into `ChatCompletionsResponse`, which drops tool calls, ids and every field it does not type. OpenAI's own body and SSE bytes reach the caller as sent - The request changes only by the deployment model name, the `extra_body` merge the OpenAI SDK does, and dropping the LiteLLM-owned keys Python consumes before the call - Endpoint and credential fallbacks are shared with Responses in `../common_utils.rs` +- `compatible.rs` is the request flow and response fixups the OpenAI-compatible providers share. Each provider's config lives in `src//chat/`, and its body and SSE bytes change only where its config's fixups change them diff --git a/litellm-rust/crates/llms/src/openai/chat/compatible.rs b/litellm-rust/crates/llms/src/openai/chat/compatible.rs new file mode 100644 index 00000000000..083170b2797 --- /dev/null +++ b/litellm-rust/crates/llms/src/openai/chat/compatible.rs @@ -0,0 +1,985 @@ +//! The chat flow every OpenAI-compatible provider shares, `litellm/llms/openai/chat/gpt_transformation.py` +//! (`OpenAIGPTConfig`) as the base each provider config overrides, run over the caller's +//! OpenAI-shaped body: Python's `get_optional_params` sorts the request's OpenAI params from the +//! rest, refuses or drops what the provider does not take, maps the rest by the provider's +//! rules, and passes every other key through as `extra_body` does. The messages go through the +//! provider's `_transform_messages`, and the response reaches the caller as the provider sent +//! it, changed only by the fixups each provider's `transform_response` and stream iterator make. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, ErrorDetail, + base_llm::{ + chat::params::{ChatRequestContext, sort_params, truthy}, + model_info::ModelInfo, + }, + openai::chat::transformation::{OPENAI_CHAT_COMPLETIONS_PATH, is_stream, take_extra_body}, +}; + +/// The `base_params` of `OpenAIGPTConfig.get_supported_openai_params`. +pub const GPT_BASE_PARAMS: &[&str] = &[ + "frequency_penalty", + "logit_bias", + "logprobs", + "top_logprobs", + "max_tokens", + "max_completion_tokens", + "modalities", + "prediction", + "n", + "presence_penalty", + "seed", + "stop", + "stream", + "stream_options", + "temperature", + "top_p", + "tools", + "tool_choice", + "function_call", + "functions", + "max_retries", + "extra_headers", + "parallel_tool_calls", + "audio", + "web_search_options", + "service_tier", + "safety_identifier", + "prompt_cache_key", + "prompt_cache_retention", + "store", +]; + +/// `LITELLM_RESPONSE_ONLY_MESSAGE_FIELDS`: what LiteLLM adds to a returned message that no +/// OpenAI-compatible endpoint takes back. +const RESPONSE_ONLY_MESSAGE_FIELDS: &[&str] = &[ + "thinking_blocks", + "reasoning_items", + "provider_specific_fields", +]; +/// `LITELLM_INTERNAL_MESSAGE_FIELDS`: the response-only fields and `reasoning_content`. +const INTERNAL_MESSAGE_FIELDS: &[&str] = &[ + "thinking_blocks", + "reasoning_items", + "provider_specific_fields", + "reasoning_content", +]; +/// `DEFAULT_MAX_RECURSE_DEPTH`. +const MAX_RECURSE_DEPTH: usize = 100; +const STREAM_OPTIONS: &str = "stream_options"; +const INCLUDE_USAGE: &str = "include_usage"; +/// `RESPONSE_FORMAT_TOOL_NAME`. +pub const RESPONSE_FORMAT_TOOL_NAME: &str = "json_tool_call"; +const POTENTIAL_PDF_URL_STARTS: &[&str] = &["https://", "http://", "www."]; +/// The content part types `_content_parts_contain_image` looks for. +const IMAGE_PART_TYPES: &[&str] = &["image_url", "input_image", "image"]; +/// `DEFAULT_REASONING_EFFORT_{LOW,MEDIUM,HIGH}_THINKING_BUDGET` at their defaults. +const THINKING_BUDGETS: (u64, u64, u64) = (1024, 2048, 4096); +/// `REASONING_EFFORT_STRENGTH_ORDER`. +const EFFORT_STRENGTH_ORDER: &[&str] = &["minimal", "low", "medium", "high", "xhigh", "max"]; + +/// What a provider's param rules read about the call. +#[derive(Clone, Copy)] +pub struct CompatibleContext<'a> { + /// The model as the provider names it, without the route prefix. + pub model: &'a str, + /// `drop_params` from the deployment, the proxy settings or the request. + pub drop_params: bool, + pub model_info: Option<&'a dyn ModelInfo>, +} + +impl CompatibleContext<'_> { + /// `supports_factory`: the `/` entry's flag, else the bare model's. + pub fn capability(&self, provider: &str, key: &str) -> Option { + self.capability_of( + &[format!("{provider}/{}", self.model), self.model.to_string()], + key, + ) + } + + /// The first of `candidates` whose cost map entry sets `key`. + pub fn capability_of(&self, candidates: &[String], key: &str) -> Option { + let info = self.model_info?; + candidates + .iter() + .filter_map(|candidate| info.entry(candidate)) + .find_map(|entry| entry.get(key).and_then(Value::as_bool)) + } + + pub fn supports(&self, provider: &str, key: &str) -> bool { + self.capability(provider, key) == Some(true) + } + + /// `declared_reasoning_efforts_for_model`: the `reasoning_effort_levels` the entry for the + /// model declares, read through its bare twin. `None` when the map says nothing. + pub fn declared_reasoning_efforts(&self, provider: &str) -> Option> { + let info = self.model_info?; + let prefixed = format!("{provider}/{}", self.model); + let (entry, prefixed_hit) = match info.entry(&prefixed) { + Some(entry) => (entry, true), + None => (info.entry(self.model)?, false), + }; + let levels = match entry.get("reasoning_effort_levels") { + Some(levels) if !levels.is_null() => levels.clone(), + _ if prefixed_hit => info + .entry(self.model) + .and_then(|bare| bare.get("reasoning_effort_levels").cloned()) + .unwrap_or(Value::Null), + _ => Value::Null, + }; + let Value::Array(levels) = levels else { + return None; + }; + Some( + levels + .into_iter() + .filter_map(|level| level.as_str().map(str::to_owned)) + .collect(), + ) + } +} + +/// `nearest_declared_reasoning_effort`: the weakest declared level at least as strong as the +/// request, else the strongest declared one. `none` and unknown levels go as asked. +pub fn nearest_declared_reasoning_effort(requested: &str, declared: &[String]) -> String { + let rank = |effort: &str| EFFORT_STRENGTH_ORDER.iter().position(|e| *e == effort); + let mut ranked: Vec<&str> = declared + .iter() + .map(String::as_str) + .filter(|effort| rank(effort).is_some()) + .collect(); + ranked.sort_by_key(|effort| rank(effort)); + let Some(requested_rank) = rank(requested) else { + return requested.to_owned(); + }; + if ranked.is_empty() || ranked.contains(&requested) { + return requested.to_owned(); + } + ranked + .iter() + .find(|effort| rank(effort) >= Some(requested_rank)) + .or(ranked.last()) + .map_or_else(|| requested.to_owned(), |effort| (*effort).to_owned()) +} + +/// `reasoning_effort_from_thinking_budget` at the default budget thresholds. +pub fn reasoning_effort_from_thinking_budget(budget_tokens: i64) -> &'static str { + let (low, medium, high) = THINKING_BUDGETS; + match budget_tokens { + tokens if tokens >= high as i64 => "high", + tokens if tokens >= medium as i64 => "medium", + tokens if tokens >= low as i64 => "low", + _ => "minimal", + } +} + +/// What a request decided about its answer, which the response and stream fixups read. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct ResponseShape { + pub stream: bool, + /// The client asked to see the usage on a stream. When it did not, the usage the call asked + /// for on its behalf is recorded and kept from the client. + pub client_usage: bool, + /// The provider answers a `response_format` with a `json_tool_call` tool call, which goes + /// back as content. + pub json_mode: bool, + /// The names of the function tools the request sent. + pub tool_names: Vec, + /// Whether the request sent `tools` at all. + pub tools_sent: bool, +} + +/// A request ready for an OpenAI-compatible provider. +#[derive(Debug, PartialEq)] +pub struct CompatibleChatBody { + pub body: Map, + pub shape: ResponseShape, +} + +/// One OpenAI-compatible provider's config: its endpoint and credential fallbacks, and the +/// overrides it makes to `OpenAIGPTConfig`. +pub trait OpenAiCompatibleChatConfig: Sync { + fn provider(&self) -> LlmProviders; + + /// The provider as a missing key error names it. + fn label(&self) -> &'static str; + + /// The variables an unset `api_base` falls back to, in order. + fn api_base_env(&self) -> &'static [&'static str]; + + fn default_api_base(&self) -> Option<&'static str>; + + /// The variables an unset key falls back to, in order. + fn api_key_env(&self) -> &'static [&'static str]; + + /// The key sent when neither the deployment nor the environment has one; `None` means the + /// call needs one. + fn default_api_key(&self) -> Option<&'static str> { + None + } + + /// Other variables the provider's headers read. + fn header_env(&self) -> &'static [&'static str] { + &[] + } + + fn name(&self) -> &'static str { + self.provider().into() + } + + /// Every variable a call may read. + fn secret_names(&self) -> Vec<&'static str> { + self.api_base_env() + .iter() + .chain(self.api_key_env()) + .chain(self.header_env()) + .copied() + .collect() + } + + /// The deployment's `api_base`, else the environment's, else the provider's own. + fn api_base( + &self, + configured: Option<&str>, + lookup: &dyn Fn(&str) -> Option, + ) -> Result { + configured + .filter(|base| !base.is_empty()) + .map(str::to_owned) + .or_else(|| self.api_base_env().iter().find_map(|name| lookup(name))) + .or_else(|| self.default_api_base().map(str::to_owned)) + .ok_or_else(|| { + Error::InvalidRequest(ErrorDetail::Message(format!( + "Missing API Base - set api_base on the {} deployment or {}", + self.name(), + self.api_base_env().join(" or ") + ))) + }) + } + + /// `OpenAIGPTConfig.get_complete_url`: `/chat/completions` appended unless the base already + /// names it. + fn complete_url(&self, api_base: &str) -> String { + gpt_complete_url(api_base) + } + + /// The deployment's key, else the environment's, else the provider's default. + fn api_key( + &self, + configured: Option<&str>, + lookup: &dyn Fn(&str) -> Option, + ) -> Result { + configured + .filter(|key| !key.is_empty()) + .map(str::to_owned) + .or_else(|| { + self.api_key_env() + .iter() + .find_map(|name| lookup(name).filter(|key| !key.is_empty())) + }) + .or_else(|| self.default_api_key().map(str::to_owned)) + .ok_or(Error::Auth(litellm_auth::Error::MissingApiKey { + provider: self.label(), + environment_variable: self.api_key_env().first().copied().unwrap_or_default(), + })) + } + + /// Headers the provider adds to every call, read off the caller's body before it is + /// sorted. + fn headers( + &self, + _body: &Map, + _lookup: &dyn Fn(&str) -> Option, + ) -> Vec<(String, String)> { + Vec::new() + } + + /// The model the body names. + fn wire_model(&self, model: &str) -> String { + model.to_owned() + } + + /// `get_supported_openai_params`. + fn supported_params(&self, context: &CompatibleContext<'_>) -> Vec<&'static str>; + + /// `map_openai_params`: `non_default` holds only supported params. + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + copy_supported(non_default, optional, &self.supported_params(context)); + Ok(()) + } + + /// The keys sent as `extra_body`, which a provider may rewrite. + fn map_extra_body( + &self, + _extra_body: &mut Map, + _optional: &mut Map, + _context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + Ok(()) + } + + /// `_transform_messages`. + fn transform_messages( + &self, + messages: Vec, + _optional: &Map, + _context: &CompatibleContext<'_>, + ) -> Result, Error> { + gpt_messages(messages) + } + + /// Whether `cache_control` marks reach the provider; `OpenAIGPTConfig` strips them. + fn keeps_cache_control(&self, _context: &CompatibleContext<'_>) -> bool { + false + } + + /// The provider's `transform_request` changes over the assembled body, before the + /// `extra_body` keys are merged over it. + fn finish_request( + &self, + _body: &mut Map, + _context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + Ok(()) + } + + /// `transform_response`'s fixups to a complete response. Returns whether it changed. + fn fix_response(&self, _body: &mut Map, _shape: &ResponseShape) -> bool { + false + } + + /// The stream iterator's fixups to one chunk. Returns whether it changed. + fn fix_chunk(&self, chunk: &mut Map, _shape: &ResponseShape) -> bool { + rename_reasoning_in_deltas(chunk) + } +} + +/// `OpenAIGPTConfig.get_complete_url`. +pub fn gpt_complete_url(api_base: &str) -> String { + let base = api_base.trim_end_matches('/'); + match base.contains("chat/completions") { + true => base.to_owned(), + false => format!("{base}{OPENAI_CHAT_COMPLETIONS_PATH}"), + } +} + +/// The URL the OpenAI SDK calls for a `base_url`. +pub fn sdk_complete_url(api_base: &str) -> String { + format!( + "{}{OPENAI_CHAT_COMPLETIONS_PATH}", + api_base.trim_end_matches('/') + ) +} + +/// `OpenAIGPTConfig._map_openai_params`: every supported param as sent. +pub fn copy_supported( + non_default: &Map, + optional: &mut Map, + supported: &[&str], +) { + for (name, value) in non_default { + if supported.contains(&name.as_str()) { + optional.insert(name.clone(), value.clone()); + } + } +} + +/// The request for `config`, built from the caller's body. +pub fn transform_request( + config: &dyn OpenAiCompatibleChatConfig, + context: &CompatibleContext<'_>, + body: Map, +) -> Result { + let mut body = body; + body.remove("model"); + let messages = match body.remove("messages") { + None | Some(Value::Null) => Vec::new(), + Some(Value::Array(messages)) => messages, + Some(other) => { + return Err(Error::InvalidRequest(ErrorDetail::InvalidValue { + field: "messages", + expected: "an array", + actual: other, + })); + } + }; + let mut user_extra_body = take_extra_body(&mut body)?; + user_extra_body.remove("model"); + let stream = is_stream(&body)?; + let client_usage = stream + && body + .get(STREAM_OPTIONS) + .and_then(|options| options.get(INCLUDE_USAGE)) + == Some(&Value::Bool(true)); + let supported = config.supported_params(context); + let sorted = sort_params( + body, + &[], + &ChatRequestContext { + drop_params: Some(context.drop_params), + model_info: context.model_info, + }, + |_| Ok(supported.clone()), + config.name(), + context.model, + )?; + let mut non_default = sorted.openai; + // The proxy's stream usage tracking: a stream the provider can report usage on asks for + // it, so the call is accounted for, and the client sees it only if it asked. + if stream && !client_usage && supported.contains(&STREAM_OPTIONS) { + let mut options = match non_default.remove(STREAM_OPTIONS) { + Some(Value::Object(options)) => options, + _ => Map::new(), + }; + options.insert(INCLUDE_USAGE.into(), Value::Bool(true)); + non_default.insert(STREAM_OPTIONS.into(), Value::Object(options)); + } + + let mut optional = Map::new(); + config.map_params(&non_default, &mut optional, context)?; + // `apply_openai_param_overrides`: what the caller allowed goes as sent. + for name in &sorted.allowed { + if !optional.contains_key(name) + && let Some(value) = non_default.get(name) + { + optional.insert(name.clone(), value.clone()); + } + } + let json_mode = optional + .remove("json_mode") + .is_some_and(|value| value == Value::Bool(true)); + let mut extra_body = match optional.remove("extra_body") { + Some(Value::Object(extra)) => extra, + _ => Map::new(), + }; + extra_body.extend(user_extra_body); + extra_body.extend(sorted.provider); + config.map_extra_body(&mut extra_body, &mut optional, context)?; + + let mut messages = config.transform_messages(messages, &optional, context)?; + if !config.keeps_cache_control(context) { + for message in &mut messages { + filter_value(message, "cache_control", 0); + } + if let Some(Value::Array(tools)) = optional.get_mut("tools") { + for tool in tools { + filter_value(tool, "cache_control", 0); + } + } + } + if context.drop_params { + for message in &mut messages { + without_prompt_cache_breakpoints(message); + } + if let Some(Value::Array(tools)) = optional.get_mut("tools") { + for tool in tools { + without_prompt_cache_breakpoints(tool); + } + } + } + optional.remove("max_retries"); + if !optional.get("tools").is_some_and(truthy) && !optional.get("functions").is_some_and(truthy) + { + optional.remove("tool_choice"); + } + + let mut sent = Map::new(); + sent.insert( + "model".into(), + Value::String(config.wire_model(context.model)), + ); + sent.insert("messages".into(), Value::Array(messages)); + sent.extend(optional); + config.finish_request(&mut sent, context)?; + extra_body.remove("model"); + sent.extend(extra_body); + + let tools = sent.get("tools").and_then(Value::as_array); + let shape = ResponseShape { + stream, + client_usage, + json_mode, + tool_names: tools + .into_iter() + .flatten() + .filter_map(|tool| tool.get("function")?.get("name")?.as_str()) + .map(str::to_owned) + .collect(), + tools_sent: sent.get("tools").is_some_and(|tools| !tools.is_null()), + }; + Ok(CompatibleChatBody { body: sent, shape }) +} + +/// A complete response as the caller receives it. Returns whether it changed. +pub fn fix_response( + config: &dyn OpenAiCompatibleChatConfig, + body: &mut Map, + shape: &ResponseShape, +) -> bool { + let renamed = rename_reasoning_in_messages(body); + config.fix_response(body, shape) || renamed +} + +/// `OpenAIGPTConfig._transform_messages`: `tool_reference` parts dropped from tool messages, +/// and each user content part normalized. +pub fn gpt_messages(messages: Vec) -> Result, Error> { + if messages.iter().any(tool_message_carries_image) { + // `hoist_images_from_tool_messages` moves them into an inserted user message. + return Err(Error::Unsupported( + "images in tool messages, which Python moves into a user message", + )); + } + messages + .into_iter() + .map(drop_tool_reference_parts) + .map(|mut message| { + if role(&message) == Some("user") + && let Some(Value::Array(parts)) = message.get_mut("content") + { + for part in parts.iter_mut() { + transform_content_item(part)?; + } + } + Ok(message) + }) + .collect() +} + +pub fn role(message: &Value) -> Option<&str> { + message.get("role").and_then(Value::as_str) +} + +fn is_tool_reference(part: &Value) -> bool { + part.get("type").and_then(Value::as_str) == Some("tool_reference") +} + +/// `drop_tool_reference_parts_from_tool_messages` for one message. +fn drop_tool_reference_parts(mut message: Value) -> Value { + if role(&message) != Some("tool") { + return message; + } + if let Some(content) = message.get_mut("content") + && let Value::Array(parts) = content + && parts.iter().any(is_tool_reference) + { + parts.retain(|part| !is_tool_reference(part)); + if parts.is_empty() { + *content = Value::String(String::new()); + } + } + message +} + +fn tool_message_carries_image(message: &Value) -> bool { + fn has_image(parts: &[Value], depth: usize) -> bool { + parts.iter().any(|part| { + part.get("type") + .and_then(Value::as_str) + .is_some_and(|kind| IMAGE_PART_TYPES.contains(&kind)) + || (depth < 4 + && part + .get("content") + .and_then(Value::as_array) + .is_some_and(|nested| has_image(nested, depth + 1))) + }) + } + role(message) == Some("tool") + && message + .get("content") + .and_then(Value::as_array) + .is_some_and(|parts| has_image(parts, 1)) +} + +/// `OpenAIGPTConfig._transform_content_item`. +fn transform_content_item(part: &mut Value) -> Result<(), Error> { + let Some(object) = part.as_object_mut() else { + return Ok(()); + }; + match object.get("type").and_then(Value::as_str) { + Some("image_url") => match object.get_mut("image_url") { + Some(Value::String(url)) => { + let url = std::mem::take(url); + object.insert("image_url".into(), json!({"url": url})); + } + Some(Value::Object(image)) => { + image.remove("format"); + } + _ => {} + }, + Some("file") => { + let Some(Value::Object(file)) = object.get_mut("file") else { + return Err(Error::InvalidRequest(ErrorDetail::Message( + "Content block has type='file' but is missing the required 'file' field".into(), + ))); + }; + file.remove("format"); + if file + .get("file_id") + .and_then(Value::as_str) + .is_some_and(|id| POTENTIAL_PDF_URL_STARTS.iter().any(|s| id.starts_with(s))) + { + // `_handle_pdf_url` downloads it into `file_data`. + return Err(Error::Unsupported( + "a file_id URL Python downloads into file_data", + )); + } + if file.get("file_data").is_some_and(|data| !data.is_null()) + && file.get("filename").is_none_or(Value::is_null) + { + file.insert("filename".into(), Value::String("my_file.pdf".into())); + } + } + _ => {} + } + Ok(()) +} + +/// `filter_value_from_dict`: `key` removed from every object, at any depth. +pub fn filter_value(value: &mut Value, key: &str, depth: usize) { + if depth > MAX_RECURSE_DEPTH { + return; + } + if let Value::Object(object) = value { + object.remove(key); + for nested in object.values_mut() { + match nested { + Value::Object(_) => filter_value(nested, key, depth + 1), + Value::Array(items) => items + .iter_mut() + .filter(|item| item.is_object()) + .for_each(|item| filter_value(item, key, depth + 1)), + _ => {} + } + } + } +} + +/// `without_prompt_cache_breakpoints`: the marker removed from a message or tool, its content +/// parts, its tool calls and its `function`. +fn without_prompt_cache_breakpoints(block: &mut Value) { + const MARKER: &str = "prompt_cache_breakpoint"; + let Some(object) = block.as_object_mut() else { + return; + }; + object.remove(MARKER); + for key in ["content", "tool_calls"] { + if let Some(Value::Array(items)) = object.get_mut(key) { + for item in items.iter_mut().filter_map(Value::as_object_mut) { + item.remove(MARKER); + } + } + } + if let Some(Value::Object(function)) = object.get_mut("function") { + function.remove(MARKER); + } +} + +fn without_fields(message: &mut Value, fields: &[&str]) { + let Some(object) = message.as_object_mut() else { + return; + }; + for field in fields { + object.remove(*field); + } + if let Some(Value::Array(calls)) = object.get_mut("tool_calls") { + for call in calls.iter_mut().filter_map(Value::as_object_mut) { + call.remove("provider_specific_fields"); + if let Some(Value::Object(function)) = call.get_mut("function") { + function.remove("provider_specific_fields"); + } + } + } +} + +/// `strip_litellm_response_only_fields`. +pub fn strip_response_only_fields(message: &mut Value) { + without_fields(message, RESPONSE_ONLY_MESSAGE_FIELDS); +} + +/// `strip_litellm_internal_message_fields`. +pub fn strip_internal_fields(message: &mut Value) { + without_fields(message, INTERNAL_MESSAGE_FIELDS); +} + +/// `extract_search_results_text`. +pub fn search_results_text(search_results: Option<&Value>) -> String { + let Some(Value::Array(results)) = search_results else { + return String::new(); + }; + let mut text = String::new(); + for result in results.iter().filter_map(Value::as_object) { + for key in ["source", "title"] { + if let Some(value) = result.get(key).and_then(Value::as_str) { + text.push_str(value); + } + } + if let Some(Value::Array(blocks)) = result.get("content") { + for block in blocks { + if let Some(value) = block.get("text").and_then(Value::as_str) { + text.push_str(value); + } + } + } + if let Some(citations) = result.get("citations").filter(|c| !c.is_null()) { + text.push_str(&serde_json::to_string(citations).unwrap_or_default()); + } + } + text +} + +/// `convert_content_list_to_str`: the text parts joined, then any `search_results` text. +pub fn content_list_to_str(message: &Value) -> String { + let mut text = String::new(); + match message.get("content") { + Some(Value::Array(parts)) => { + for part in parts { + if let Some(value) = part.get("text").and_then(Value::as_str) { + text.push_str(value); + } + } + } + Some(Value::String(content)) => text.push_str(content), + _ => {} + } + text.push_str(&search_results_text(message.get("search_results"))); + text +} + +/// The `reasoning` some OpenAI-compatible servers send, under LiteLLM's `reasoning_content`. +pub fn rename_reasoning(message: &mut Map) -> bool { + if message.contains_key("reasoning_content") || !message.contains_key("reasoning") { + return false; + } + if let Some(reasoning) = message.remove("reasoning") { + message.insert("reasoning_content".into(), reasoning); + } + true +} + +/// `_map_reasoning_to_reasoning_content` over a chunk's deltas. +pub fn rename_reasoning_in_deltas(chunk: &mut Map) -> bool { + let mut changed = false; + if let Some(Value::Array(choices)) = chunk.get_mut("choices") { + for delta in choices + .iter_mut() + .filter_map(|choice| choice.get_mut("delta")?.as_object_mut()) + { + changed |= rename_reasoning(delta); + } + } + changed +} + +/// `extract_reasoning_content` over a complete response's messages. +pub fn rename_reasoning_in_messages(body: &mut Map) -> bool { + let mut changed = false; + for message in messages_mut(body) { + changed |= rename_reasoning(message); + } + changed +} + +/// Each choice's message in a complete response. +pub fn messages_mut( + body: &mut Map, +) -> impl Iterator> { + body.get_mut("choices") + .and_then(Value::as_array_mut) + .into_iter() + .flatten() + .filter_map(|choice| choice.get_mut("message")?.as_object_mut()) +} + +/// `OpenAILikeChatConfig._sanitize_usage_obj`: a null `*_tokens` usage count becomes 0. +pub fn sanitize_usage(body: &mut Map) -> bool { + let mut changed = false; + if let Some(Value::Object(usage)) = body.get_mut("usage") { + for (key, value) in usage.iter_mut() { + if key.ends_with("_tokens") && value.is_null() { + *value = json!(0); + changed = true; + } + } + } + changed +} + +/// `_json_mode_convert_tool_response_to_message`: a lone tool call's arguments become the +/// content. +pub fn json_mode_tool_call_to_content(body: &mut Map) -> bool { + let mut changed = false; + for message in messages_mut(body) { + let arguments = match message.get("tool_calls") { + Some(Value::Array(calls)) if calls.len() == 1 => calls[0] + .get("function") + .and_then(|function| function.get("arguments")) + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + _ => continue, + }; + message.insert("content".into(), Value::String(arguments)); + message.insert("tool_calls".into(), Value::Null); + changed = true; + } + changed +} + +/// Python's `str` indices of `[n]` citation markers in `text`: `(number, start, end)` in code +/// points, as `re.finditer(r"\[(\d+)\]")` reports them. +pub fn citation_markers(text: &str) -> Vec<(usize, usize, usize)> { + let chars: Vec = text.chars().collect(); + let mut markers = Vec::new(); + let mut index = 0; + while index < chars.len() { + if chars[index] == '[' { + let digits = chars[index + 1..] + .iter() + .take_while(|c| c.is_ascii_digit()) + .count(); + let close = index + 1 + digits; + if digits > 0 && chars.get(close) == Some(&']') { + let number: String = chars[index + 1..close].iter().collect(); + if let Ok(number) = number.parse() { + markers.push((number, index, close + 1)); + } + index = close + 1; + continue; + } + } + index += 1; + } + markers +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + fn object(value: Value) -> Map { + match value { + Value::Object(map) => map, + other => panic!("expected an object, got {other}"), + } + } + + #[rstest] + #[case::bare("https://h/v1", "https://h/v1/chat/completions")] + #[case::slash("https://h/v1/", "https://h/v1/chat/completions")] + #[case::named("https://h/v1/chat/completions", "https://h/v1/chat/completions")] + fn the_gpt_url_appends_the_path_once(#[case] base: &str, #[case] expected: &str) { + assert_eq!(gpt_complete_url(base), expected); + } + + #[rstest] + fn user_content_parts_are_normalized() { + let messages = gpt_messages(vec![json!({"role": "user", "content": [ + {"type": "image_url", "image_url": "https://img"}, + {"type": "image_url", "image_url": {"url": "u", "format": "image/png", "detail": "low"}}, + {"type": "file", "file": {"file_data": "data:application/pdf;base64,AA", "format": "application/pdf"}}, + ]})]) + .unwrap(); + assert_eq!( + messages[0]["content"], + json!([ + {"type": "image_url", "image_url": {"url": "https://img"}}, + {"type": "image_url", "image_url": {"url": "u", "detail": "low"}}, + {"type": "file", "file": {"file_data": "data:application/pdf;base64,AA", "filename": "my_file.pdf"}}, + ]) + ); + } + + #[rstest] + #[case::pdf_url(json!({"role": "user", "content": [{"type": "file", "file": {"file_id": "https://x/a.pdf"}}]}))] + #[case::tool_image(json!({"role": "tool", "tool_call_id": "c", "content": [{"type": "image_url", "image_url": {"url": "u"}}]}))] + fn shapes_python_rewrites_with_io_are_refused(#[case] message: Value) { + assert!(matches!( + gpt_messages(vec![message]), + Err(Error::Unsupported(_)) + )); + } + + #[rstest] + fn a_file_part_without_its_file_is_rejected() { + assert!(matches!( + gpt_messages(vec![json!({"role": "user", "content": [{"type": "file"}]})]), + Err(Error::InvalidRequest(_)) + )); + } + + #[rstest] + fn tool_reference_parts_leave_tool_messages() { + let messages = gpt_messages(vec![json!({"role": "tool", "tool_call_id": "c", + "content": [{"type": "tool_reference", "tool_name": "f"}]})]) + .unwrap(); + assert_eq!(messages[0]["content"], json!("")); + } + + #[rstest] + fn cache_control_is_filtered_at_any_depth() { + let mut message = json!({"role": "user", "cache_control": {"type": "ephemeral"}, + "content": [{"type": "text", "text": "hi", "cache_control": {"type": "ephemeral"}}]}); + filter_value(&mut message, "cache_control", 0); + assert_eq!( + message, + json!({"role": "user", "content": [{"type": "text", "text": "hi"}]}) + ); + } + + #[rstest] + #[case::exact(1024, "low")] + #[case::medium(3000, "medium")] + #[case::high(5000, "high")] + #[case::minimal(10, "minimal")] + fn thinking_budgets_bucket_into_efforts(#[case] budget: i64, #[case] effort: &str) { + assert_eq!(reasoning_effort_from_thinking_budget(budget), effort); + } + + #[rstest] + #[case::declared("high", &["low", "high"], "high")] + #[case::rounds_up("medium", &["low", "high"], "high")] + #[case::rounds_down("max", &["low", "high"], "high")] + #[case::none("none", &["low"], "none")] + fn efforts_round_to_a_declared_level( + #[case] requested: &str, + #[case] declared: &[&str], + #[case] expected: &str, + ) { + let declared: Vec = declared.iter().map(|d| (*d).to_owned()).collect(); + assert_eq!( + nearest_declared_reasoning_effort(requested, &declared), + expected + ); + } + + #[rstest] + fn citation_markers_count_code_points() { + assert_eq!( + citation_markers("é [1] and [22][x]"), + vec![(1, 2, 5), (22, 10, 14)] + ); + } + + #[rstest] + fn json_mode_moves_a_lone_tool_call_into_the_content() { + let mut body = object( + json!({"choices": [{"message": {"role": "assistant", "content": null, + "tool_calls": [{"id": "c", "type": "function", "function": {"name": "json_tool_call", "arguments": "{\"a\":1}"}}]}, + "finish_reason": "tool_calls"}]}), + ); + assert!(json_mode_tool_call_to_content(&mut body)); + assert_eq!( + body["choices"][0]["message"], + json!({"role": "assistant", "content": "{\"a\":1}", "tool_calls": null}) + ); + } +} diff --git a/litellm-rust/crates/llms/src/openai/chat/mod.rs b/litellm-rust/crates/llms/src/openai/chat/mod.rs index f239b6921fa..40fd00f7449 100644 --- a/litellm-rust/crates/llms/src/openai/chat/mod.rs +++ b/litellm-rust/crates/llms/src/openai/chat/mod.rs @@ -1 +1,2 @@ +pub mod compatible; pub mod transformation; diff --git a/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs b/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs index 5ca084cd517..7924888a016 100644 --- a/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs +++ b/litellm-rust/crates/llms/src/openai_like/chat/transformation.rs @@ -16,9 +16,13 @@ use crate::{ Error, base_llm::{ auth::AuthScheme, - chat::transformation::{ - BaseConfig, Headers, ProviderChatRequestData, ProviderChatResponseData, - ValidatedEnvironment, + chat::{ + params::ChatRequestContext, + streaming::StreamShape, + transformation::{ + BaseConfig, Headers, ProviderChatRequestData, ProviderChatResponseData, + ValidatedEnvironment, + }, }, }, openai_like::common_utils::{complete_openai_like_url, openai_compatible_provider_info}, @@ -90,6 +94,7 @@ impl BaseConfig for OpenAILikeChatConfig { model: &str, messages: Vec, optional_params: Map, + _context: &ChatRequestContext<'_>, ) -> Result { let mut params = Map::from_iter( optional_params @@ -113,6 +118,7 @@ impl BaseConfig for OpenAILikeChatConfig { Ok(ProviderChatRequestData { body: Value::Object(body), stream_shape: Default::default(), + headers: Vec::new(), }) } @@ -120,6 +126,7 @@ impl BaseConfig for OpenAILikeChatConfig { &self, model: &str, response: ProviderChatResponseData, + _shape: &StreamShape, ) -> Result { let mut body = response.body; sanitize_usage(&mut body); @@ -177,6 +184,7 @@ impl BaseConfig for OpenAILikeChatConfig { }, ..ChatCompletionsUsage::default() }, + extra: Map::new(), }) } @@ -266,12 +274,14 @@ fn normalize_choice(position: usize, choice: &Value) -> Result Value { OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG - .transform_request(model, messages(msgs), params(opts)) + .transform_request(model, messages(msgs), params(opts), &Default::default()) .expect("request transforms") .body } fn transform_response(body: Value) -> Result { - OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG - .transform_response("some-model", ProviderChatResponseData { body }) + OPENAI_LIKE_CHAT_COMPLETIONS_CONFIG.transform_response( + "some-model", + ProviderChatResponseData { body }, + &Default::default(), + ) } fn reason(msgs: Value, opts: Value) -> Option { diff --git a/litellm-rust/crates/llms/src/openrouter/chat/AGENTS.md b/litellm-rust/crates/llms/src/openrouter/chat/AGENTS.md new file mode 100644 index 00000000000..8ba2bf4016d --- /dev/null +++ b/litellm-rust/crates/llms/src/openrouter/chat/AGENTS.md @@ -0,0 +1,11 @@ +# rules + +- OpenRouter at `https://openrouter.ai/api/v1` with the `HTTP-Referer` and `X-Title` attribution headers +- `cache_control` is kept for caching model families and moved into the last content block +- `usage: {include: true}` is always asked for unless the caller set `usage` +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://openrouter.ai/docs/api-reference/chat-completion +- litellm/llms/openrouter/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/openrouter/chat/mod.rs b/litellm-rust/crates/llms/src/openrouter/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/openrouter/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/openrouter/chat/transformation.rs b/litellm-rust/crates/llms/src/openrouter/chat/transformation.rs new file mode 100644 index 00000000000..43c4f5b7848 --- /dev/null +++ b/litellm-rust/crates/llms/src/openrouter/chat/transformation.rs @@ -0,0 +1,231 @@ +//! `litellm/llms/openrouter/chat/transformation.py`: OpenRouter at +//! `https://openrouter.ai/api/v1`, with LiteLLM's attribution headers, `cache_control` kept +//! for the model families that cache, moved into the last content block, and the usage +//! accounting OpenRouter reports cost in always asked for. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, + openai::chat::compatible::{ + CompatibleContext, GPT_BASE_PARAMS, OpenAiCompatibleChatConfig, copy_supported, + gpt_messages, + }, +}; + +const CACHE_CONTROL_MODEL_FAMILIES: &[&str] = &["claude", "gemini", "minimax", "glm", "z-ai"]; + +pub struct OpenRouterChatConfig; + +pub const OPENROUTER_CHAT_CONFIG: OpenRouterChatConfig = OpenRouterChatConfig; + +impl OpenAiCompatibleChatConfig for OpenRouterChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::Openrouter + } + + fn label(&self) -> &'static str { + "OpenRouter" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["OPENROUTER_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://openrouter.ai/api/v1") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &["OPENROUTER_API_KEY", "OR_API_KEY"] + } + + fn header_env(&self) -> &'static [&'static str] { + &["OR_SITE_URL", "OR_APP_NAME"] + } + + fn headers( + &self, + _body: &Map, + lookup: &dyn Fn(&str) -> Option, + ) -> Vec<(String, String)> { + let setting = |name: &str, default: &str| { + lookup(name) + .filter(|value| !value.is_empty()) + .unwrap_or_else(|| default.to_owned()) + }; + vec![ + ( + "HTTP-Referer".into(), + setting("OR_SITE_URL", "https://litellm.ai"), + ), + ("X-Title".into(), setting("OR_APP_NAME", "liteLLM")), + ] + } + + fn supported_params(&self, _context: &CompatibleContext<'_>) -> Vec<&'static str> { + GPT_BASE_PARAMS + .iter() + .copied() + .chain(["response_format", "reasoning_effort", "thinking"]) + .collect() + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + copy_supported(non_default, optional, &self.supported_params(context)); + if optional.get("reasoning_effort") == Some(&json!("max")) { + optional.insert("reasoning_effort".into(), json!("xhigh")); + } + Ok(()) + } + + fn keeps_cache_control(&self, context: &CompatibleContext<'_>) -> bool { + caches(context.model) + } + + fn transform_messages( + &self, + messages: Vec, + _optional: &Map, + context: &CompatibleContext<'_>, + ) -> Result, Error> { + match caches(context.model) { + true => gpt_messages( + messages + .into_iter() + .map(move_cache_control_to_content) + .collect(), + ), + false => gpt_messages(messages), + } + } + + fn finish_request( + &self, + body: &mut Map, + _context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + body.entry("usage") + .or_insert_with(|| json!({"include": true})); + Ok(()) + } +} + +fn caches(model: &str) -> bool { + let model = model.to_lowercase(); + CACHE_CONTROL_MODEL_FAMILIES + .iter() + .any(|family| model.contains(family)) +} + +/// `_move_cache_control_to_content`: only the last block carries it, to stay within +/// Anthropic's four cache breakpoints. +fn move_cache_control_to_content(mut message: Value) -> Value { + let Some(object) = message.as_object_mut() else { + return message; + }; + let Some(cache_control) = object.remove("cache_control").filter(|c| !c.is_null()) else { + return message; + }; + match object.get_mut("content") { + Some(Value::Array(blocks)) => { + if let Some(Value::Object(last)) = blocks.last_mut() { + last.insert("cache_control".into(), cache_control); + } + } + content => { + let text = content.map(Value::take).unwrap_or(Value::Null); + object.insert( + "content".into(), + json!([{"type": "text", "text": text, "cache_control": cache_control}]), + ); + } + } + message +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::openai::chat::compatible::transform_request; + + fn send(model: &str, params: Value) -> Map { + let Value::Object(body) = params else { + panic!("an object") + }; + transform_request( + &OPENROUTER_CHAT_CONFIG, + &CompatibleContext { + model, + drop_params: false, + model_info: None, + }, + body, + ) + .unwrap() + .body + } + + #[rstest] + fn openrouter_params_and_usage_accounting_go_along() { + let sent = send( + "openai/gpt-4o", + json!({"messages": [], "reasoning_effort": "max", + "transforms": ["middle-out"], "route": "fallback"}), + ); + assert_eq!( + Value::Object(sent), + json!({"model": "openai/gpt-4o", "messages": [], "reasoning_effort": "xhigh", + "usage": {"include": true}, "transforms": ["middle-out"], "route": "fallback"}) + ); + } + + #[rstest] + fn a_callers_usage_setting_wins() { + let sent = send( + "openai/gpt-4o", + json!({"messages": [], "usage": {"include": false}}), + ); + assert_eq!(sent["usage"], json!({"include": false})); + } + + #[rstest] + #[case::caching_family("anthropic/claude-sonnet-4", json!([ + {"role": "system", "content": [{"type": "text", "text": "s", "cache_control": {"type": "ephemeral"}}]}, + {"role": "user", "content": [{"type": "text", "text": "a"}, {"type": "text", "text": "b", "cache_control": {"type": "ephemeral"}}]}]))] + #[case::other_family("openai/gpt-4o", json!([ + {"role": "system", "content": "s"}, + {"role": "user", "content": [{"type": "text", "text": "a"}, {"type": "text", "text": "b"}]}]))] + fn cache_control_reaches_only_families_that_cache( + #[case] model: &str, + #[case] messages: Value, + ) { + let sent = send( + model, + json!({"messages": [ + {"role": "system", "content": "s", "cache_control": {"type": "ephemeral"}}, + {"role": "user", "cache_control": {"type": "ephemeral"}, "content": [{"type": "text", "text": "a"}, {"type": "text", "text": "b"}]}]}), + ); + assert_eq!(sent["messages"], messages); + } + + #[rstest] + fn attribution_headers_default_to_litellm() { + assert_eq!( + OPENROUTER_CHAT_CONFIG.headers(&Map::new(), &|name| (name == "OR_APP_NAME") + .then(|| "app".to_owned())), + vec![ + ("HTTP-Referer".to_owned(), "https://litellm.ai".to_owned()), + ("X-Title".to_owned(), "app".to_owned()), + ] + ); + } +} diff --git a/litellm-rust/crates/llms/src/openrouter/mod.rs b/litellm-rust/crates/llms/src/openrouter/mod.rs index e4838133a04..f58a06aeb58 100644 --- a/litellm-rust/crates/llms/src/openrouter/mod.rs +++ b/litellm-rust/crates/llms/src/openrouter/mod.rs @@ -1,2 +1,3 @@ +pub mod chat; pub mod messages; pub mod responses; diff --git a/litellm-rust/crates/llms/src/perplexity/chat/AGENTS.md b/litellm-rust/crates/llms/src/perplexity/chat/AGENTS.md new file mode 100644 index 00000000000..5cba3ad7adc --- /dev/null +++ b/litellm-rust/crates/llms/src/perplexity/chat/AGENTS.md @@ -0,0 +1,10 @@ +# rules + +- Perplexity through the OpenAI SDK URL rule at `https://api.perplexity.ai`, with `X-Pplx-Integration: litellm` +- Citations become `url_citation` annotations on the first choice, search counts reach `prompt_tokens_details` +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://docs.perplexity.ai/api-reference/chat-completions-post +- litellm/llms/perplexity/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/perplexity/chat/mod.rs b/litellm-rust/crates/llms/src/perplexity/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/perplexity/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/perplexity/chat/transformation.rs b/litellm-rust/crates/llms/src/perplexity/chat/transformation.rs new file mode 100644 index 00000000000..b5f47bfb6f7 --- /dev/null +++ b/litellm-rust/crates/llms/src/perplexity/chat/transformation.rs @@ -0,0 +1,283 @@ +//! `litellm/llms/perplexity/chat/transformation.py`: Perplexity through the OpenAI SDK at +//! `https://api.perplexity.ai`, with the subset of OpenAI params it takes, LiteLLM's +//! `X-Pplx-Integration` attribution header, and the citations of an answer added to its +//! message as `url_citation` annotations with the search counts in the usage. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::openai::chat::compatible::{ + CompatibleContext, OpenAiCompatibleChatConfig, ResponseShape, citation_markers, + sdk_complete_url, +}; + +const SUPPORTED_PARAMS: &[&str] = &[ + "frequency_penalty", + "max_tokens", + "max_completion_tokens", + "presence_penalty", + "response_format", + "stream", + "temperature", + "top_p", + "max_retries", + "extra_headers", +]; +const ATTRIBUTION_HEADER: (&str, &str) = ("X-Pplx-Integration", "litellm"); + +pub struct PerplexityChatConfig; + +pub const PERPLEXITY_CHAT_CONFIG: PerplexityChatConfig = PerplexityChatConfig; + +impl OpenAiCompatibleChatConfig for PerplexityChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::Perplexity + } + + fn label(&self) -> &'static str { + "Perplexity" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["PERPLEXITY_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://api.perplexity.ai") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &["PERPLEXITYAI_API_KEY", "PERPLEXITY_API_KEY"] + } + + fn complete_url(&self, api_base: &str) -> String { + sdk_complete_url(api_base) + } + + fn headers( + &self, + _body: &Map, + _lookup: &dyn Fn(&str) -> Option, + ) -> Vec<(String, String)> { + vec![(ATTRIBUTION_HEADER.0.into(), ATTRIBUTION_HEADER.1.into())] + } + + fn supported_params(&self, context: &CompatibleContext<'_>) -> Vec<&'static str> { + let mut params = SUPPORTED_PARAMS.to_vec(); + if context.supports(self.name(), "supports_reasoning") { + params.push("reasoning_effort"); + } + if context.supports(self.name(), "supports_web_search") { + params.push("web_search_options"); + } + params + } + + /// `_enhance_usage_with_perplexity_fields` and `_add_citations_as_annotations`. + fn fix_response(&self, body: &mut Map, _shape: &ResponseShape) -> bool { + let usage_changed = enhance_usage(body); + add_citations_as_annotations(body) || usage_changed + } +} + +/// The search count Perplexity reports, wherever it put it. +fn search_queries(body: &Map) -> Option { + let usage = body.get("usage"); + let field = |value: Option<&Value>, key: &str| value.and_then(|v| v.get(key)).cloned(); + [ + field(usage, "num_search_queries"), + body.get("num_search_queries").cloned(), + field(usage, "search_queries"), + body.get("search_queries").cloned(), + ] + .into_iter() + .flatten() + .find(|value| !value.is_null()) + .and_then(|value| value.as_u64()) +} + +fn enhance_usage(body: &mut Map) -> bool { + let citation_chars: usize = match body.get("citations") { + Some(Value::Array(citations)) => citations + .iter() + .filter(|citation| crate::base_llm::chat::params::truthy(citation)) + .map(|citation| match citation { + Value::String(text) => text.chars().count(), + other => other.to_string().chars().count(), + }) + .sum(), + _ => 0, + }; + let citation_tokens = match citation_chars { + 0 => 0, + chars => (chars / 4).max(1), + }; + let searches = search_queries(body).filter(|count| *count > 0); + if citation_tokens == 0 && searches.is_none() { + return false; + } + let usage = body + .entry("usage") + .or_insert_with(|| json!({"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0})); + if usage.is_null() { + *usage = json!({"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}); + } + let Some(usage) = usage.as_object_mut() else { + return false; + }; + if citation_tokens > 0 { + usage.insert("citation_tokens".into(), json!(citation_tokens)); + } + if let Some(searches) = searches { + let details = usage + .entry("prompt_tokens_details") + .or_insert_with(|| json!({})); + if !details.is_object() { + *details = json!({}); + } + if let Some(details) = details.as_object_mut() { + details.insert("web_search_requests".into(), json!(searches)); + } + } + true +} + +fn add_citations_as_annotations(body: &mut Map) -> bool { + let citations: Vec> = match body.get("citations") { + Some(Value::Array(citations)) => citations + .iter() + .map(|citation| citation.as_str().map(str::to_owned)) + .collect(), + _ => Vec::new(), + }; + let titles: Vec<(String, String)> = match body.get("search_results") { + Some(Value::Array(results)) => results + .iter() + .filter_map(|result| { + Some(( + result.get("url")?.as_str()?.to_owned(), + result.get("title")?.as_str()?.to_owned(), + )) + }) + .collect(), + _ => Vec::new(), + }; + let Some(message) = body + .get_mut("choices") + .and_then(Value::as_array_mut) + .and_then(|choices| choices.first_mut()) + .and_then(|choice| choice.get_mut("message")) + .and_then(Value::as_object_mut) + else { + return false; + }; + let Some(content) = message + .get("content") + .and_then(Value::as_str) + .filter(|content| !content.is_empty()) + else { + return false; + }; + let annotations: Vec = citation_markers(content) + .into_iter() + .filter_map(|(number, start, end)| { + let url = citations.get(number.checked_sub(1)?)?.as_ref()?; + let title = titles + .iter() + .rev() + .find(|(known, _)| known == url) + .map_or("", |(_, title)| title.as_str()); + Some(json!({"type": "url_citation", "url_citation": { + "url": url, "title": title, "start_index": start, "end_index": end, + }})) + }) + .collect(); + if annotations.is_empty() { + return false; + } + match message.get_mut("annotations") { + Some(Value::Array(existing)) => existing.extend(annotations), + _ => { + message.insert("annotations".into(), Value::Array(annotations)); + } + } + true +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::{ + Error, ErrorDetail, + openai::chat::compatible::{fix_response, transform_request}, + }; + + fn object(value: Value) -> Map { + let Value::Object(map) = value else { + panic!("an object") + }; + map + } + + fn send(params: Value) -> Result, Error> { + transform_request( + &PERPLEXITY_CHAT_CONFIG, + &CompatibleContext { + model: "sonar", + drop_params: false, + model_info: None, + }, + object(params), + ) + .map(|sent| sent.body) + } + + #[rstest] + fn max_completion_tokens_goes_as_sent() { + let sent = send( + json!({"messages": [], "max_completion_tokens": 7, "search_domain_filter": ["a.com"]}), + ) + .unwrap(); + assert_eq!( + Value::Object(sent), + json!({"model": "sonar", "messages": [], "max_completion_tokens": 7, "search_domain_filter": ["a.com"]}) + ); + } + + #[rstest] + #[case::tools(json!({"messages": [], "tools": [{"type": "function", "function": {"name": "f"}}]}))] + #[case::seed(json!({"messages": [], "seed": 1}))] + fn params_perplexity_lacks_are_refused(#[case] params: Value) { + assert!(matches!( + send(params), + Err(Error::InvalidRequest(ErrorDetail::UnsupportedParams { .. })) + )); + } + + #[rstest] + fn citations_become_annotations_and_searches_reach_the_usage() { + let mut body = object(json!({ + "model": "sonar", + "citations": ["https://a.example", "https://b.example"], + "search_results": [{"url": "https://b.example", "title": "B"}], + "choices": [{"index": 0, "message": {"role": "assistant", "content": "é see [2] and [3]"}, "finish_reason": "stop"}], + "usage": {"prompt_tokens": 3, "completion_tokens": 4, "total_tokens": 7, "num_search_queries": 2}, + })); + assert!(fix_response( + &PERPLEXITY_CHAT_CONFIG, + &mut body, + &ResponseShape::default() + )); + assert_eq!( + body["choices"][0]["message"]["annotations"], + json!([{"type": "url_citation", "url_citation": {"url": "https://b.example", "title": "B", "start_index": 6, "end_index": 9}}]) + ); + assert_eq!(body["usage"]["citation_tokens"], json!(8)); + assert_eq!( + body["usage"]["prompt_tokens_details"], + json!({"web_search_requests": 2}) + ); + } +} diff --git a/litellm-rust/crates/llms/src/perplexity/mod.rs b/litellm-rust/crates/llms/src/perplexity/mod.rs new file mode 100644 index 00000000000..30a62fc508f --- /dev/null +++ b/litellm-rust/crates/llms/src/perplexity/mod.rs @@ -0,0 +1 @@ +pub mod chat; diff --git a/litellm-rust/crates/llms/src/together_ai/chat/AGENTS.md b/litellm-rust/crates/llms/src/together_ai/chat/AGENTS.md new file mode 100644 index 00000000000..8e19632d24e --- /dev/null +++ b/litellm-rust/crates/llms/src/together_ai/chat/AGENTS.md @@ -0,0 +1,11 @@ +# rules + +- Together AI at `https://api.together.ai/v1` +- Tool and `response_format` params are refused or dropped only when the `together_ai/` cost map entry rules them out +- `reasoning_effort` is translated per model family, `none` turns reasoning off with `reasoning.enabled: false` +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://docs.together.ai/docs/chat-overview +- litellm/llms/together_ai/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/together_ai/chat/mod.rs b/litellm-rust/crates/llms/src/together_ai/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/together_ai/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/together_ai/chat/transformation.rs b/litellm-rust/crates/llms/src/together_ai/chat/transformation.rs new file mode 100644 index 00000000000..3835653bb4a --- /dev/null +++ b/litellm-rust/crates/llms/src/together_ai/chat/transformation.rs @@ -0,0 +1,324 @@ +//! `litellm/llms/together_ai/chat/transformation.py`: Together AI at +//! `https://api.together.ai/v1`. Tool and structured output params the cost map rules out are +//! refused or dropped, and `reasoning_effort` is translated to the levels each reasoning model +//! family takes. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, ErrorDetail, + openai::chat::compatible::{ + CompatibleContext, GPT_BASE_PARAMS, OpenAiCompatibleChatConfig, copy_supported, + gpt_messages, role, strip_response_only_fields, + }, +}; + +const TOOL_CALLING_PARAMS: &[&str] = &["tools", "tool_choice", "function_call"]; +const ADJUSTABLE_EFFORT_REASONING_MODELS: &[&str] = &["openai/gpt-oss-120b", "openai/gpt-oss-20b"]; +const HYBRID_REASONING_MODELS: &[&str] = &[ + "MiniMaxAI/MiniMax-M3", + "Qwen/Qwen3.5-9B", + "Qwen/Qwen3.6-Plus", + "deepseek-ai/DeepSeek-V4-Pro", + "moonshotai/Kimi-K3", + "nvidia/nemotron-3-ultra-550b-a55b", + "zai-org/GLM-5.2", +]; +const HIGH_MAX_EFFORT_MODEL_PREFIX: &str = "deepseek-ai/DeepSeek-V4-Pro"; + +pub struct TogetherAiChatConfig; + +pub const TOGETHER_AI_CHAT_CONFIG: TogetherAiChatConfig = TogetherAiChatConfig; + +impl OpenAiCompatibleChatConfig for TogetherAiChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::TogetherAi + } + + fn label(&self) -> &'static str { + "Together AI" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["TOGETHER_AI_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://api.together.ai/v1") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &[ + "TOGETHER_API_KEY", + "TOGETHER_AI_API_KEY", + "TOGETHERAI_API_KEY", + "TOGETHER_AI_TOKEN", + ] + } + + fn supported_params(&self, context: &CompatibleContext<'_>) -> Vec<&'static str> { + let mut params: Vec<&'static str> = GPT_BASE_PARAMS + .iter() + .copied() + .chain(["response_format"]) + .collect(); + if supports_reasoning(context) { + params.push("reasoning_effort"); + } + params + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + copy_supported(non_default, optional, &self.supported_params(context)); + let passed_tools: Vec<&str> = TOOL_CALLING_PARAMS + .iter() + .copied() + .filter(|param| optional.contains_key(*param)) + .collect(); + if !passed_tools.is_empty() + && registry_rules_out(context, "supports_function_calling", &passed_tools)? + { + for param in passed_tools { + optional.remove(param); + } + } + if optional.contains_key("response_format") + && registry_rules_out(context, "supports_response_schema", &["response_format"])? + { + optional.remove("response_format"); + } + let Some(Value::String(effort)) = optional.get("reasoning_effort").cloned() else { + return Ok(()); + }; + optional.remove("reasoning_effort"); + for (key, value) in reasoning_effort_payload(&effort, context) { + optional.entry(key).or_insert(value); + } + Ok(()) + } + + fn transform_messages( + &self, + messages: Vec, + _optional: &Map, + _context: &CompatibleContext<'_>, + ) -> Result, Error> { + gpt_messages( + messages + .into_iter() + .map(|mut message| { + if role(&message) == Some("assistant") { + strip_response_only_fields(&mut message); + } + message + }) + .collect(), + ) + } +} + +fn supports_reasoning(context: &CompatibleContext<'_>) -> bool { + let model = context.model; + ADJUSTABLE_EFFORT_REASONING_MODELS.contains(&model) + || HYBRID_REASONING_MODELS.contains(&model) + || model.starts_with(HIGH_MAX_EFFORT_MODEL_PREFIX) + || context.supports("together_ai", "supports_reasoning") +} + +/// `_registry_verdict`: true when the `together_ai/` entry sets `flag` to false and the +/// params are to be dropped. Without `drop_params` that is a refusal. +fn registry_rules_out( + context: &CompatibleContext<'_>, + flag: &str, + params: &[&str], +) -> Result { + if context.supports("together_ai", flag) + || context.capability_of(&[format!("together_ai/{}", context.model)], flag) != Some(false) + { + return Ok(false); + } + if context.drop_params { + return Ok(true); + } + Err(Error::InvalidRequest(ErrorDetail::UnsupportedParams { + provider: "together_ai", + params: params.iter().map(|param| (*param).to_owned()).collect(), + model: context.model.to_owned(), + })) +} + +fn reasoning_effort_payload(effort: &str, context: &CompatibleContext<'_>) -> Vec<(String, Value)> { + let model = context.model; + let effort_only = |effort: &str| vec![("reasoning_effort".to_owned(), json!(effort))]; + if effort == "default" { + return Vec::new(); + } + if ADJUSTABLE_EFFORT_REASONING_MODELS.contains(&model) { + return effort_only(match effort { + "none" => "low", + other => translate(other), + }); + } + if effort == "none" { + return vec![("reasoning".to_owned(), json!({"enabled": false}))]; + } + if context + .declared_reasoning_efforts("together_ai") + .is_some_and(|declared| declared.iter().any(|level| level == effort)) + { + return effort_only(effort); + } + if model.starts_with(HIGH_MAX_EFFORT_MODEL_PREFIX) { + return effort_only(match effort { + "minimal" | "low" | "medium" => "high", + "xhigh" => "max", + other => other, + }); + } + effort_only(translate(effort)) +} + +fn translate(effort: &str) -> &str { + match effort { + "minimal" => "low", + "xhigh" | "max" => "high", + other => other, + } +} + +#[cfg(test)] +mod tests { + use std::borrow::Cow; + + use rstest::rstest; + + use super::*; + use crate::{base_llm::model_info::ModelInfo, openai::chat::compatible::transform_request}; + + struct Registry; + + impl ModelInfo for Registry { + fn entry(&self, model: &str) -> Option>> { + let entry = match model { + "together_ai/no-tools" => { + json!({"supports_function_calling": false, "supports_response_schema": false}) + } + "together_ai/declared" => { + json!({"supports_reasoning": true, "reasoning_effort_levels": ["low", "xhigh"]}) + } + _ => return None, + }; + let Value::Object(entry) = entry else { + unreachable!() + }; + Some(Cow::Owned(entry)) + } + + fn declares(&self, _key: &str) -> bool { + true + } + } + + fn send(model: &str, drop_params: bool, params: Value) -> Result, Error> { + let Value::Object(body) = params else { + panic!("an object") + }; + transform_request( + &TOGETHER_AI_CHAT_CONFIG, + &CompatibleContext { + model, + drop_params, + model_info: Some(&Registry), + }, + body, + ) + .map(|sent| sent.body) + } + + #[rstest] + #[case::adjustable_none("openai/gpt-oss-20b", "none", json!({"reasoning_effort": "low"}))] + #[case::adjustable_max("openai/gpt-oss-20b", "max", json!({"reasoning_effort": "high"}))] + #[case::hybrid_off("zai-org/GLM-5.2", "none", json!({"reasoning": {"enabled": false}}))] + #[case::declared("declared", "xhigh", json!({"reasoning_effort": "xhigh"}))] + #[case::high_max_floor("deepseek-ai/DeepSeek-V4-Pro", "low", json!({"reasoning_effort": "high"}))] + #[case::high_max_top("deepseek-ai/DeepSeek-V4-Pro", "xhigh", json!({"reasoning_effort": "max"}))] + #[case::hybrid_minimal("zai-org/GLM-5.2", "minimal", json!({"reasoning_effort": "low"}))] + #[case::default("zai-org/GLM-5.2", "default", json!({}))] + fn reasoning_effort_fits_the_model_family( + #[case] model: &str, + #[case] effort: &str, + #[case] expected: Value, + ) { + let mut sent = send( + model, + false, + json!({"messages": [], "reasoning_effort": effort}), + ) + .unwrap(); + sent.remove("model"); + sent.remove("messages"); + assert_eq!(Value::Object(sent), expected); + } + + #[rstest] + #[case::refused(false)] + #[case::dropped(true)] + fn params_the_registry_rules_out_are_refused_or_dropped(#[case] drop_params: bool) { + let result = send( + "no-tools", + drop_params, + json!({"messages": [], + "tools": [{"type": "function", "function": {"name": "f"}}], "tool_choice": "auto", + "response_format": {"type": "json_object"}}), + ); + match drop_params { + false => assert!(matches!( + result, + Err(Error::InvalidRequest(ErrorDetail::UnsupportedParams { + provider: "together_ai", + .. + })) + )), + true => assert_eq!( + Value::Object(result.unwrap()), + json!({"model": "no-tools", "messages": []}) + ), + } + } + + #[rstest] + fn unknown_models_pass_tools_through() { + let sent = send( + "unknown", + false, + json!({"messages": [], + "tools": [{"type": "function", "function": {"name": "f"}}]}), + ) + .unwrap(); + assert_eq!( + sent["tools"], + json!([{"type": "function", "function": {"name": "f"}}]) + ); + } + + #[rstest] + fn replayed_reasoning_stays_while_response_only_fields_go() { + let sent = send( + "unknown", + false, + json!({"messages": [{"role": "assistant", "content": "a", + "reasoning_content": "r", "thinking_blocks": [], "provider_specific_fields": {}}]}), + ) + .unwrap(); + assert_eq!( + sent["messages"], + json!([{"role": "assistant", "content": "a", "reasoning_content": "r"}]) + ); + } +} diff --git a/litellm-rust/crates/llms/src/together_ai/mod.rs b/litellm-rust/crates/llms/src/together_ai/mod.rs new file mode 100644 index 00000000000..30a62fc508f --- /dev/null +++ b/litellm-rust/crates/llms/src/together_ai/mod.rs @@ -0,0 +1 @@ +pub mod chat; diff --git a/litellm-rust/crates/llms/src/vertex_ai/common_utils.rs b/litellm-rust/crates/llms/src/vertex_ai/common_utils.rs index 43b0f3f0de3..883ec6f64a9 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/common_utils.rs +++ b/litellm-rust/crates/llms/src/vertex_ai/common_utils.rs @@ -44,6 +44,52 @@ pub fn get_vertex_base_url(location: &str) -> Result { Ok(format!("https://{location}-aiplatform.googleapis.com")) } +/// Python's `_check_custom_proxy` for a Vertex URL: a custom base with no path of its own, or +/// only an API version, takes the default URL's path; any other base is called as given with +/// the method appended. +pub fn graft_custom_vertex_base(api_base: Option<&str>, default_url: &str, method: &str) -> String { + let Some(custom) = api_base.filter(|base| !base.is_empty()) else { + return default_url.to_owned(); + }; + let path_of = |url: &str| -> String { + let rest = url.split_once("://").map_or(url, |(_, rest)| rest); + let rest = rest.split(['?', '#']).next().unwrap_or(rest); + rest.find('/') + .map_or(String::new(), |at| rest[at..].to_owned()) + }; + let default_path = path_of(default_url); + let base_path = path_of(custom); + match base_path.trim_end_matches('/') { + "" => format!("{}{default_path}", custom.trim_end_matches('/')), + version @ ("/v1" | "/v1beta1") if default_path.contains("/projects/") => { + let tail = default_path + .split_once("/projects/") + .map_or("", |(_, tail)| tail); + format!( + "{}{version}/projects/{tail}", + custom.trim_end_matches('/').trim_end_matches(version) + ) + } + _ => format!("{custom}:{method}"), + } +} + +/// `url` with `pair` added to its query, before any fragment. +pub fn with_query(url: &str, pair: &str) -> String { + let (head, fragment) = url + .split_once('#') + .map_or((url, None), |(head, fragment)| (head, Some(fragment))); + let joined = match head.split_once('?') { + Some((_, query)) if !query.is_empty() => format!("{head}&{pair}"), + Some((path, _)) => format!("{path}?{pair}"), + None => format!("{head}?{pair}"), + }; + match fragment { + Some(fragment) => format!("{joined}#{fragment}"), + None => joined, + } +} + #[cfg(test)] mod tests { use rstest::rstest; @@ -67,4 +113,38 @@ mod tests { fn a_location_that_is_not_a_token_is_rejected(#[case] location: &str) { assert!(get_vertex_base_url(location).is_err()); } + + const DEFAULT: &str = "https://us-central1-aiplatform.googleapis.com/v1/projects/p/locations/us-central1/publishers/google/models/gemini-2.5-flash:generateContent"; + + #[rstest] + #[case::none(None, DEFAULT)] + #[case::bare_host( + Some("https://proxy.example/"), + "https://proxy.example/v1/projects/p/locations/us-central1/publishers/google/models/gemini-2.5-flash:generateContent" + )] + #[case::version_path( + Some("https://proxy.example/v1beta1"), + "https://proxy.example/v1beta1/projects/p/locations/us-central1/publishers/google/models/gemini-2.5-flash:generateContent" + )] + #[case::own_path( + Some("https://gw.example/route/x"), + "https://gw.example/route/x:generateContent" + )] + fn a_custom_base_grafts_or_appends_the_method( + #[case] api_base: Option<&str>, + #[case] expected: &str, + ) { + assert_eq!( + graft_custom_vertex_base(api_base, DEFAULT, "generateContent"), + expected + ); + } + + #[rstest] + #[case::no_query("https://h/x:m", "https://h/x:m?alt=sse")] + #[case::query("https://h/x?k=1", "https://h/x?k=1&alt=sse")] + #[case::empty_query("https://h/x?", "https://h/x?alt=sse")] + fn a_query_pair_joins_the_query(#[case] url: &str, #[case] expected: &str) { + assert_eq!(with_query(url, "alt=sse"), expected); + } } diff --git a/litellm-rust/crates/llms/src/vertex_ai/gemini/AGENTS.md b/litellm-rust/crates/llms/src/vertex_ai/gemini/AGENTS.md new file mode 100644 index 00000000000..b2d330697c3 --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/gemini/AGENTS.md @@ -0,0 +1,13 @@ +# rules + +- Gemini `generateContent` for Chat Completions, shared by Vertex AI and Google AI Studio (`gemini/chat`) through `GeminiSurface` +- `transformation.rs` turns OpenAI messages into `contents` and `systemInstruction`, `schema.rs` turns JSON schemas into Gemini's subset, `vertex_and_google_ai_studio_gemini.rs` maps params, builds URLs and turns responses back, `handler.rs` turns `alt=sse` stream events into chunks +- Only Gemini models route here; Claude and partner models on Vertex keep their own configs +- Vertex authenticates with a Google access token, or forwards a caller's `Authorization` header + +# references + +- https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference +- https://ai.google.dev/api/generate-content +- litellm/llms/vertex_ai/gemini/transformation.py +- litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py diff --git a/litellm-rust/crates/llms/src/vertex_ai/gemini/handler.rs b/litellm-rust/crates/llms/src/vertex_ai/gemini/handler.rs new file mode 100644 index 00000000000..b24fc13ba36 --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/gemini/handler.rs @@ -0,0 +1,266 @@ +//! Gemini `streamGenerateContent?alt=sse` events to Chat Completions chunks, as Python's +//! `ModelResponseIterator.chunk_parser` in `vertex_and_google_ai_studio_gemini.py` builds them. +//! +//! DIVERGENCE: grounding, URL context, safety and citation metadata on stream chunks are not +//! forwarded as hidden params, and inline images and audio in a chunk are dropped. + +use litellm_core_utils::core_helpers::{finish_reason_for, unix_now}; +use litellm_llms_types::formats::chat_completions::{ + ChatCompletionChunk, ChatCompletionDelta, ChatCompletionStreamingChoice, +}; +use serde_json::{Map, Value}; + +use crate::{ + Error, + base_llm::{base_model_iterator::StreamTransformer, chat::streaming::ChatStream, sse}, + vertex_ai::gemini::vertex_and_google_ai_studio_gemini::{ + calculate_usage, candidate_message, prompt_blocked, served_model_name, + }, +}; + +/// Gemini's `error` events carry their own HTTP code; Python falls back to a 500. +const STREAM_ERROR_STATUS: u16 = 500; + +pub fn gemini_chat_stream() -> ChatStream { + ChatStream::new( + |bytes| sse::event_stream::(bytes, sse::is_done), + GeminiStreamIterator::default(), + ) +} + +#[derive(Default)] +pub struct GeminiStreamIterator { + tool_index: i64, + has_seen_tool_calls: bool, +} + +fn stream_error(error: &Value) -> Error { + let status = match error.get("code") { + Some(Value::Number(code)) => code.as_u64().and_then(|code| u16::try_from(code).ok()), + Some(Value::String(code)) => code.parse().ok(), + _ => None, + } + .unwrap_or(STREAM_ERROR_STATUS); + let message = error + .get("message") + .and_then(Value::as_str) + .unwrap_or_default(); + Error::Provider { + status, + body: format!("{status} - {message}"), + } +} + +impl GeminiStreamIterator { + fn choice(&mut self, position: usize, candidate: &Value) -> ChatCompletionStreamingChoice { + let message = candidate_message(candidate, &mut self.tool_index); + let has_tool_calls = !message.tool_calls.is_empty(); + self.has_seen_tool_calls |= has_tool_calls; + let finish_reason = match (&message.finish_reason, has_tool_calls) { + (_, true) => Some("tool_calls".to_owned()), + (Some(reason), false) => Some( + match (finish_reason_for(reason), self.has_seen_tool_calls) { + ("stop", true) => "tool_calls", + (mapped, _) => mapped, + } + .to_owned(), + ), + (None, false) => None, + }; + let mut extra = Map::new(); + if let Some(annotations) = message.annotations { + extra.insert("annotations".into(), Value::Array(annotations)); + } + ChatCompletionStreamingChoice { + index: candidate + .get("index") + .and_then(Value::as_u64) + .unwrap_or(position as u64), + delta: ChatCompletionDelta { + content: message.content, + role: None, + tool_calls: has_tool_calls.then_some(message.tool_calls), + reasoning_content: message.reasoning_content, + thinking_blocks: message.thinking_blocks.filter(|blocks| !blocks.is_empty()), + provider_specific_fields: message.provider_specific_fields, + extra, + }, + finish_reason, + logprobs: message.logprobs, + } + } +} + +impl StreamTransformer for GeminiStreamIterator { + type Input = Value; + type Output = ChatCompletionChunk; + type Error = Error; + + fn transform(&mut self, event: Value) -> Result, Error> { + if let Some(error) = event.get("error") { + return Err(stream_error(error)); + } + let choices = match prompt_blocked(&event) { + true => vec![ChatCompletionStreamingChoice { + index: 0, + delta: ChatCompletionDelta { + role: Some("assistant".into()), + ..ChatCompletionDelta::default() + }, + finish_reason: Some("content_filter".into()), + logprobs: None, + }], + false => event + .get("candidates") + .and_then(Value::as_array) + .map(|candidates| { + candidates + .iter() + .enumerate() + .map(|(position, candidate)| self.choice(position, candidate)) + .collect() + }) + .unwrap_or_default(), + }; + let usage = match event.get("usageMetadata") { + Some(_) => Some(calculate_usage(&event)?), + None => None, + }; + Ok(vec![ChatCompletionChunk { + id: event + .get("responseId") + .and_then(Value::as_str) + .map(str::to_owned), + created: Some(unix_now()), + model: served_model_name(&event), + object: Some("chat.completion.chunk".into()), + choices, + usage, + provider_specific_fields: None, + extra: Map::new(), + }]) + } + + fn finish(&mut self) -> Result, Error> { + Ok(Vec::new()) + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + + fn first_choice(chunks: &[ChatCompletionChunk]) -> &ChatCompletionStreamingChoice { + &chunks[0].choices[0] + } + + #[test] + fn text_streams_without_a_finish_until_gemini_sends_one() { + let mut iterator = GeminiStreamIterator::default(); + let chunks = iterator + .transform(json!({ + "candidates": [{"content": {"role": "model", "parts": [{"text": "Hel"}]}}], + "modelVersion": "gemini-2.5-flash", + "responseId": "r1" + })) + .unwrap(); + let choice = first_choice(&chunks); + assert_eq!(choice.delta.content.as_deref(), Some("Hel")); + assert_eq!(choice.finish_reason, None); + assert_eq!(chunks[0].id.as_deref(), Some("r1")); + assert_eq!(chunks[0].model.as_deref(), Some("gemini-2.5-flash")); + assert!(chunks[0].usage.is_none()); + } + + #[test] + fn the_last_event_carries_the_finish_and_usage() { + let mut iterator = GeminiStreamIterator::default(); + let chunks = iterator + .transform(json!({ + "candidates": [{"content": {"role": "model", "parts": [{"text": "lo"}]}, "finishReason": "MAX_TOKENS"}], + "usageMetadata": {"promptTokenCount": 4, "candidatesTokenCount": 2, "totalTokenCount": 6} + })) + .unwrap(); + assert_eq!( + first_choice(&chunks).finish_reason.as_deref(), + Some("length") + ); + let usage = chunks[0].usage.as_ref().unwrap(); + assert_eq!((usage.prompt_tokens, usage.completion_tokens), (4, 2)); + } + + #[test] + fn tool_calls_count_across_events_and_turn_stop_into_tool_calls() { + let mut iterator = GeminiStreamIterator::default(); + let call = |name: &str| json!({"candidates": [{"content": {"parts": [{"functionCall": {"name": name, "args": {"a": 1}}}]}}]}); + let first = iterator.transform(call("one")).unwrap(); + let second = iterator.transform(call("two")).unwrap(); + let last = iterator + .transform(json!({"candidates": [{"content": {"parts": [{"text": ""}]}, "finishReason": "STOP"}]})) + .unwrap(); + let tool = |chunks: &[ChatCompletionChunk]| { + first_choice(chunks).delta.tool_calls.as_ref().unwrap()[0].clone() + }; + assert_eq!(tool(&first).index, 0); + assert_eq!(tool(&second).index, 1); + assert_eq!(tool(&second).function.arguments, "{\"a\": 1}"); + assert_eq!( + first_choice(&first).finish_reason.as_deref(), + Some("tool_calls") + ); + assert_eq!( + first_choice(&last).finish_reason.as_deref(), + Some("tool_calls") + ); + } + + #[rstest] + #[case::numeric(json!({"error": {"code": 429, "message": "slow down", "status": "RESOURCE_EXHAUSTED"}}), 429, "429 - slow down")] + #[case::text(json!({"error": {"code": "503", "message": "busy"}}), 503, "503 - busy")] + #[case::bare(json!({"error": {"message": "oops"}}), 500, "500 - oops")] + fn an_error_event_ends_the_stream_with_its_status( + #[case] event: Value, + #[case] status: u16, + #[case] body: &str, + ) { + let error = GeminiStreamIterator::default() + .transform(event) + .unwrap_err(); + assert_eq!( + error, + Error::Provider { + status, + body: body.into() + } + ); + } + + #[test] + fn a_blocked_prompt_is_a_content_filter() { + let chunks = GeminiStreamIterator::default() + .transform(json!({"promptFeedback": {"blockReason": "SAFETY"}})) + .unwrap(); + let choice = first_choice(&chunks); + assert_eq!(choice.finish_reason.as_deref(), Some("content_filter")); + assert_eq!(choice.delta.role.as_deref(), Some("assistant")); + } + + #[test] + fn reasoning_and_signatures_stream_on_the_delta() { + let chunks = GeminiStreamIterator::default() + .transform(json!({"candidates": [{"content": {"parts": [ + {"text": "think", "thought": true, "thoughtSignature": "sig"} + ]}}]})) + .unwrap(); + let delta = &first_choice(&chunks).delta; + assert_eq!(delta.reasoning_content.as_deref(), Some("think")); + assert_eq!(delta.content, None); + assert_eq!( + delta.provider_specific_fields.as_ref().unwrap()["thought_signatures"], + json!(["sig"]) + ); + } +} diff --git a/litellm-rust/crates/llms/src/vertex_ai/gemini/mod.rs b/litellm-rust/crates/llms/src/vertex_ai/gemini/mod.rs new file mode 100644 index 00000000000..8bff54a9d99 --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/gemini/mod.rs @@ -0,0 +1,4 @@ +pub mod handler; +pub mod schema; +pub mod transformation; +pub mod vertex_and_google_ai_studio_gemini; diff --git a/litellm-rust/crates/llms/src/vertex_ai/gemini/schema.rs b/litellm-rust/crates/llms/src/vertex_ai/gemini/schema.rs new file mode 100644 index 00000000000..ea71952ce9a --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/gemini/schema.rs @@ -0,0 +1,461 @@ +//! Python's schema helpers in `litellm/llms/vertex_ai/common_utils.py` and `litellm/utils.py`: +//! a JSON schema rewritten into the OpenAPI subset Gemini's `Schema` accepts. + +use serde_json::{Map, Value, json}; + +use crate::base_llm::chat::response_format::unpack_defs; + +/// The fields of Python's `Schema` TypedDict, all a `responseSchema` or tool parameter keeps. +const SCHEMA_FIELDS: &[&str] = &[ + "type", + "format", + "title", + "description", + "nullable", + "default", + "items", + "minItems", + "maxItems", + "enum", + "properties", + "propertyOrdering", + "required", + "minProperties", + "maxProperties", + "minimum", + "maximum", + "minLength", + "maxLength", + "pattern", + "example", + "anyOf", +]; + +/// The fields `_convert_schema_types` moves into the object or array branch of a type list. +const TYPE_SPECIFIC_FIELDS: &[&str] = &[ + "properties", + "required", + "additionalProperties", + "items", + "minItems", + "maxItems", + "minProperties", + "maxProperties", +]; + +/// `remove_additional_properties`: every `additionalProperties: false` removed. +pub fn remove_additional_properties(schema: &mut Value) { + match schema { + Value::Object(object) => { + if object.get("additionalProperties") == Some(&Value::Bool(false)) { + object.remove("additionalProperties"); + } + object.values_mut().for_each(remove_additional_properties); + } + Value::Array(items) => items.iter_mut().for_each(remove_additional_properties), + _ => {} + } +} + +/// `remove_strict_from_schema`: every `strict` key removed, at any depth. +pub fn remove_strict_from_schema(schema: &mut Value) { + match schema { + Value::Object(object) => { + object.remove("strict"); + object.values_mut().for_each(remove_strict_from_schema); + } + Value::Array(items) => items.iter_mut().for_each(remove_strict_from_schema), + _ => {} + } +} + +/// `build_vertex_schema`: `$defs` unpacked, type lists turned into `anyOf`, null branches +/// turned into `nullable`, enums kept only on strings, arrays given `items`, objects typed, +/// every field `Schema` does not know dropped, and `propertyOrdering` added when asked. +pub fn build_vertex_schema(mut parameters: Value, add_property_ordering: bool) -> Value { + let defs = match parameters + .as_object_mut() + .and_then(|object| object.remove("$defs")) + { + Some(Value::Object(defs)) => defs, + _ => Map::new(), + }; + unpack_defs(&mut parameters, &defs); + convert_schema_types(&mut parameters); + convert_anyof_null_to_nullable(&mut parameters); + fix_enum_empty_strings(&mut parameters); + fix_enum_types(&mut parameters); + process_items(&mut parameters); + add_object_type(&mut parameters); + let mut parameters = filter_schema_fields(parameters); + if add_property_ordering { + set_schema_property_ordering(&mut parameters); + } + parameters +} + +fn is_type(value: Option<&Value>, name: &str) -> bool { + value + .and_then(Value::as_str) + .is_some_and(|kind| kind.eq_ignore_ascii_case(name)) +} + +/// `_convert_schema_types`: a type list of several entries becomes `anyOf`, a one-entry list +/// its only entry. +fn convert_schema_types(schema: &mut Value) { + let Value::Object(object) = schema else { + return; + }; + match object.get("type") { + Some(Value::Array(types)) if types.len() > 1 => { + let types: Vec = types + .iter() + .filter_map(Value::as_str) + .map(str::to_owned) + .collect(); + let any_of: Vec = types + .iter() + .map(|kind| match kind.as_str() { + "object" | "array" => { + let mut branch = Map::from_iter([("type".to_owned(), json!(kind))]); + for field in TYPE_SPECIFIC_FIELDS { + if let Some(value) = object.get(*field) { + branch.insert((*field).to_owned(), value.clone()); + } + } + Value::Object(branch) + } + _ => json!({"type": kind}), + }) + .collect(); + if types.iter().any(|kind| kind == "object" || kind == "array") { + for field in TYPE_SPECIFIC_FIELDS { + object.remove(*field); + } + } + object.insert("anyOf".into(), Value::Array(any_of)); + object.remove("type"); + } + Some(Value::Array(types)) if types.len() == 1 => { + let only = types[0].clone(); + object.insert("type".into(), only); + } + _ => {} + } + if let Some(Value::Object(properties)) = object.get_mut("properties") { + properties.values_mut().for_each(convert_schema_types); + } + if let Some(items) = object.get_mut("items") { + convert_schema_types(items); + } + if let Some(Value::Array(any_of)) = object.get_mut("anyOf") { + any_of.iter_mut().for_each(convert_schema_types); + } +} + +/// `convert_anyof_null_to_nullable`: a `null` branch of `anyOf` removed and every other +/// branch marked `nullable`. +fn convert_anyof_null_to_nullable(schema: &mut Value) { + let Value::Object(object) = schema else { + return; + }; + if let Some(Value::Array(any_of)) = object.get_mut("anyOf") { + let is_null = |branch: &Value| branch.get("type").and_then(Value::as_str) == Some("null"); + let contains_null = any_of.iter().any(is_null); + let non_null: Vec = any_of + .iter() + .filter(|branch| !is_null(branch)) + .cloned() + .collect(); + if non_null.is_empty() { + any_of.truncate(1); + } else { + *any_of = non_null; + } + for branch in any_of.iter_mut() { + let Value::Object(fields) = branch else { + continue; + }; + if fields.is_empty() { + fields.insert("type".into(), json!("object")); + } + if contains_null { + fields.insert("nullable".into(), json!(true)); + } + convert_anyof_null_to_nullable(branch); + } + } + if let Some(Value::Object(properties)) = object.get_mut("properties") { + properties + .values_mut() + .for_each(convert_anyof_null_to_nullable); + } + if let Some(items) = object.get_mut("items") { + convert_anyof_null_to_nullable(items); + } +} + +/// `_fix_enum_empty_strings`: Gemini rejects `""` in an enum, so it becomes `null`. +fn fix_enum_empty_strings(schema: &mut Value) { + let Value::Object(object) = schema else { + return; + }; + if let Some(Value::Array(values)) = object.get_mut("enum") { + for value in values.iter_mut() { + if value.as_str() == Some("") { + *value = Value::Null; + } + } + } + if let Some(Value::Object(properties)) = object.get_mut("properties") { + properties.values_mut().for_each(fix_enum_empty_strings); + } + if let Some(items) = object.get_mut("items") { + fix_enum_empty_strings(items); + } +} + +/// `_fix_enum_types`: an enum survives only on a string type, or an `anyOf` with one. +fn fix_enum_types(schema: &mut Value) { + let Value::Object(object) = schema else { + return; + }; + if matches!(object.get("enum"), Some(Value::Array(_))) { + let keep = is_type(object.get("type"), "string") + || matches!(object.get("anyOf"), Some(Value::Array(branches)) + if branches.iter().any(|branch| is_type(branch.get("type"), "string"))); + if !keep { + object.remove("enum"); + } + } + if let Some(Value::Object(properties)) = object.get_mut("properties") { + properties.values_mut().for_each(fix_enum_types); + } + if let Some(items) = object.get_mut("items") { + fix_enum_types(items); + } + if let Some(Value::Array(any_of)) = object.get_mut("anyOf") { + any_of.iter_mut().for_each(fix_enum_types); + } +} + +/// `process_items`: every array, at any depth, gets `items`, `{"type": "object"}` when it has +/// none or an empty one. +fn process_items(schema: &mut Value) { + let Value::Object(object) = schema else { + return; + }; + let empty_items = match object.get("items") { + None => true, + Some(Value::Object(items)) => items.is_empty(), + Some(_) => false, + }; + if is_type(object.get("type"), "array") && empty_items { + object.insert("items".into(), json!({"type": "object"})); + } + for value in object.values_mut() { + match value { + Value::Object(_) => process_items(value), + Value::Array(items) => items.iter_mut().for_each(process_items), + _ => {} + } + } +} + +/// `add_object_type`: an untyped schema is an object, as is anything with properties; empty +/// properties are dropped. +fn add_object_type(schema: &mut Value) { + let Value::Object(object) = schema else { + return; + }; + if !["type", "anyOf", "oneOf", "allOf"] + .iter() + .any(|key| object.contains_key(*key)) + { + object.insert("type".into(), json!("object")); + } + if let Some(properties) = object.get("properties") { + let empty = match properties { + Value::Object(properties) => properties.is_empty(), + Value::Array(items) => items.is_empty(), + Value::Null => true, + Value::Bool(flag) => !flag, + Value::String(text) => text.is_empty(), + Value::Number(_) => false, + }; + if object.get("required") == Some(&Value::Null) { + object.remove("required"); + } + if empty { + object.remove("properties"); + object.remove("required"); + } + object.insert("type".into(), json!("object")); + if let Some(Value::Object(properties)) = object.get_mut("properties") { + properties.values_mut().for_each(add_object_type); + } + } + if let Some(items) = object.get_mut("items") { + add_object_type(items); + } + for key in ["anyOf", "oneOf", "allOf"] { + if let Some(Value::Array(branches)) = object.get_mut(key) { + branches.iter_mut().for_each(add_object_type); + } + } +} + +/// `filter_schema_fields` with `_filter_anyof_fields`: only `Schema`'s fields survive, `format` +/// only as `enum` or `date-time`, and a schema with `anyOf` keeps nothing else, its title and +/// description pushed into the branches. +fn filter_schema_fields(schema: Value) -> Value { + let Value::Object(mut object) = schema else { + return schema; + }; + let any_of_set = object + .get("anyOf") + .is_some_and(crate::base_llm::chat::params::truthy); + if any_of_set { + let label = |key: &str| { + object + .get(key) + .filter(|value| crate::base_llm::chat::params::truthy(value)) + .cloned() + }; + let (title, description) = (label("title"), label("description")); + let mut any_of = object.remove("anyOf").unwrap_or_default(); + if let Value::Array(branches) = &mut any_of + && (title.is_some() || description.is_some()) + && branches.iter().all(Value::is_object) + { + for branch in branches.iter_mut() { + if let Some(title) = &title { + branch["title"] = title.clone(); + } + if let Some(description) = &description { + branch["description"] = description.clone(); + } + } + } + object = Map::from_iter([("anyOf".to_owned(), any_of)]); + } + let filtered = object + .into_iter() + .filter(|(key, _)| SCHEMA_FIELDS.contains(&key.as_str())) + .filter_map(|(key, value)| { + let value = match (key.as_str(), value) { + ("properties", Value::Object(properties)) => Value::Object( + properties + .into_iter() + .map(|(name, property)| (name, filter_schema_fields(property))) + .collect(), + ), + ("format", format) => match format.as_str() { + Some("enum" | "date-time") => format, + _ => return None, + }, + ("items", items @ Value::Object(_)) => filter_schema_fields(items), + ("anyOf", Value::Array(branches)) => { + Value::Array(branches.into_iter().map(filter_schema_fields).collect()) + } + (_, value) => value, + }; + Some((key, value)) + }) + .collect(); + Value::Object(filtered) +} + +/// `set_schema_property_ordering`: each object lists its properties in the order given, +/// unless it already says. +fn set_schema_property_ordering(schema: &mut Value) { + let Value::Object(object) = schema else { + return; + }; + if let Some(Value::Object(properties)) = object.get("properties") { + let names: Vec = properties.keys().cloned().map(Value::String).collect(); + if !object.contains_key("propertyOrdering") { + object.insert("propertyOrdering".into(), Value::Array(names)); + } + if let Some(Value::Object(properties)) = object.get_mut("properties") { + properties + .values_mut() + .for_each(set_schema_property_ordering); + } + } + if let Some(items @ Value::Object(_)) = object.get_mut("items") { + set_schema_property_ordering(items); + } +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + use serde_json::json; + + use super::*; + + #[rstest] + #[case::plain_object( + json!({"type": "object", "properties": {"city": {"type": "string", "description": "c"}}, "required": ["city"], "additionalProperties": false}), + json!({"type": "object", "properties": {"city": {"type": "string", "description": "c"}}, "required": ["city"]}) + )] + #[case::type_list_with_null( + json!({"type": "object", "properties": {"a": {"type": ["string", "null"]}}}), + json!({"type": "object", "properties": {"a": {"anyOf": [{"type": "string", "nullable": true}]}}}) + )] + #[case::anyof_null_keeps_only_anyof( + json!({"type": "object", "properties": {"a": {"anyOf": [{"type": "integer"}, {"type": "null"}], "title": "A", "default": 1}}}), + json!({"type": "object", "properties": {"a": {"anyOf": [{"type": "integer", "nullable": true, "title": "A"}]}}}) + )] + #[case::refs_unpacked( + json!({"type": "object", "properties": {"p": {"$ref": "#/$defs/P"}}, "$defs": {"P": {"type": "object", "properties": {"n": {"type": "string"}}}}}), + json!({"type": "object", "properties": {"p": {"type": "object", "properties": {"n": {"type": "string"}}}}}) + )] + #[case::enum_on_integer_dropped( + json!({"type": "object", "properties": {"n": {"type": "integer", "enum": [1, 2]}, "s": {"type": "string", "enum": ["a", ""]}}}), + json!({"type": "object", "properties": {"n": {"type": "integer"}, "s": {"type": "string", "enum": ["a", null]}}}) + )] + #[case::array_without_items( + json!({"type": "object", "properties": {"l": {"type": "array", "items": {}}}}), + json!({"type": "object", "properties": {"l": {"type": "array", "items": {"type": "object"}}}}) + )] + #[case::empty_properties( + json!({"type": "object", "properties": {}, "required": []}), + json!({"type": "object"}) + )] + #[case::untyped(json!({}), json!({"type": "object"}))] + #[case::format_filtered( + json!({"type": "object", "properties": {"d": {"type": "string", "format": "date-time"}, "u": {"type": "string", "format": "uri"}}}), + json!({"type": "object", "properties": {"d": {"type": "string", "format": "date-time"}, "u": {"type": "string"}}}) + )] + fn tool_schemas_become_gemini_schemas(#[case] schema: Value, #[case] expected: Value) { + assert_eq!(build_vertex_schema(schema, false), expected); + } + + #[rstest] + fn response_schemas_get_property_ordering() { + assert_eq!( + build_vertex_schema( + json!({"type": "object", "properties": {"b": {"type": "string"}, "a": {"type": "object", "properties": {"y": {"type": "number"}, "x": {"type": "number"}}}}}), + true + ), + json!({"type": "object", "properties": {"b": {"type": "string"}, "a": {"type": "object", "properties": {"y": {"type": "number"}, "x": {"type": "number"}}, "propertyOrdering": ["y", "x"]}}, "propertyOrdering": ["b", "a"]}) + ); + } + + #[rstest] + #[case::strict(json!({"strict": true, "a": [{"strict": 1}]}), json!({"a": [{}]}))] + fn strict_is_removed_everywhere(#[case] mut schema: Value, #[case] expected: Value) { + remove_strict_from_schema(&mut schema); + assert_eq!(schema, expected); + } + + #[rstest] + #[case::false_removed(json!({"additionalProperties": false, "p": {"additionalProperties": false}}), json!({"p": {}}))] + #[case::true_kept(json!({"additionalProperties": true}), json!({"additionalProperties": true}))] + fn additional_properties_false_is_removed(#[case] mut schema: Value, #[case] expected: Value) { + remove_additional_properties(&mut schema); + assert_eq!(schema, expected); + } +} diff --git a/litellm-rust/crates/llms/src/vertex_ai/gemini/transformation.rs b/litellm-rust/crates/llms/src/vertex_ai/gemini/transformation.rs new file mode 100644 index 00000000000..42dd4bdd1eb --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/gemini/transformation.rs @@ -0,0 +1,1538 @@ +//! Python's `vertex_ai/gemini/transformation.py`: OpenAI messages become Gemini `contents`, +//! the media they point at becomes `inline_data` or `file_data` parts, and the mapped params +//! become the `generateContent` body around them. + +use litellm_llms_types::{ + formats::chat_completions::{ChatMessage, ChatMessageContent}, + providers::vertex_ai::GEMINI_FILES_API_URI_PREFIX, +}; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, ErrorDetail, + base_llm::chat::{ + media::{data_url, image_url}, + params::truthy, + }, + vertex_ai::gemini::vertex_and_google_ai_studio_gemini::{GeminiSurface, is_gemini_3_or_newer}, +}; + +/// `THOUGHT_SIGNATURE_SEPARATOR`: a tool call id carries its thought signature after it. +pub const THOUGHT_SIGNATURE_SEPARATOR: &str = "__thought__"; + +/// `_get_dummy_thought_signature`: base64 of `skip_thought_signature_validator`. +const DUMMY_THOUGHT_SIGNATURE: &str = "c2tpcF90aG91Z2h0X3NpZ25hdHVyZV92YWxpZGF0b3I="; + +/// `GenerationConfig.__annotations__`: the params that travel in `generationConfig`. +const GENERATION_CONFIG_FIELDS: &[&str] = &[ + "temperature", + "top_p", + "top_k", + "candidate_count", + "max_output_tokens", + "stop_sequences", + "presence_penalty", + "frequency_penalty", + "response_mime_type", + "response_schema", + "response_json_schema", + "responseFormat", + "seed", + "responseLogprobs", + "logprobs", + "responseModalities", + "imageConfig", + "thinkingConfig", + "mediaResolution", + "speechConfig", +]; + +/// `_LITELLM_INTERNAL_EXTRA_BODY_KEYS`. +const INTERNAL_EXTRA_BODY_KEYS: &[&str] = &["cache", "tags"]; + +/// `FILE_EXTENSIONS` with each type's name and, for the types Gemini accepts +/// (`GEMINI_1_5_ACCEPTED_FILE_TYPES`), its `FILE_MIME_TYPES` entry. +const FILE_TYPES: &[(&str, &str, Option<&str>)] = &[ + ("aac", "AAC", Some("audio/aac")), + ("csv", "CSV", None), + ("doc", "DOC", None), + ("docx", "DOCX", None), + ("flac", "FLAC", Some("audio/flac")), + ("flv", "FLV", Some("video/x-flv")), + ("gif", "GIF", None), + ("gdoc", "GOOGLE_DOC", None), + ("gdraw", "GOOGLE_DRAWINGS", None), + ("gsheet", "GOOGLE_SHEETS", None), + ("gslides", "GOOGLE_SLIDES", None), + ("heic", "HEIC", None), + ("heif", "HEIF", None), + ("html", "HTML", None), + ("htm", "HTML", None), + ("jpeg", "JPEG", Some("image/jpeg")), + ("jpg", "JPEG", Some("image/jpeg")), + ("json", "JSON", None), + ("m4a", "M4A", None), + ("m4v", "M4V", None), + ("mov", "MOV", Some("video/quicktime")), + ("mp3", "MP3", Some("audio/mpeg")), + ("mp4", "MP4", Some("video/mp4")), + ("mpeg", "MPEG", Some("video/mpeg")), + ("mpegps", "MPEGPS", Some("video/mpegps")), + ("mpg", "MPG", Some("video/mpg")), + ("mpa", "MPA", Some("audio/m4a")), + ("mpga", "MPGA", Some("audio/mpga")), + ("ogg", "OGG", None), + ("opus", "OPUS", Some("audio/opus")), + ("pdf", "PDF", Some("application/pdf")), + ("pcm", "PCM", Some("audio/pcm")), + ("png", "PNG", Some("image/png")), + ("ppt", "PPT", None), + ("pptx", "PPTX", None), + ("rtf", "RTF", None), + ("3gpp", "THREE_GPP", Some("video/3gpp")), + ("txt", "TXT", Some("text/plain")), + ("wav", "WAV", Some("audio/wav")), + ("webm", "WEBM", Some("video/webm")), + ("webp", "WEBP", Some("image/webp")), + ("wmv", "WMV", Some("video/wmv")), + ("xls", "XLS", None), + ("xlsx", "XLSX", None), +]; + +/// `get_image_mime_type_from_url`, in its table's order. +const URL_MIME_TYPES: &[(&[&str], &str)] = &[ + (&[".jpg", ".jpeg"], "image/jpeg"), + (&[".png"], "image/png"), + (&[".webp"], "image/webp"), + (&[".mp4"], "video/mp4"), + (&[".mov"], "video/mov"), + (&[".mpeg", ".mpg"], "video/mpeg"), + (&[".avi"], "video/avi"), + (&[".wmv"], "video/wmv"), + (&[".mpegps"], "video/mpegps"), + (&[".flv"], "video/flv"), + (&[".mp3"], "audio/mp3"), + (&[".wav"], "audio/wav"), + (&[".ogg"], "audio/ogg"), + (&[".pdf"], "application/pdf"), + (&[".txt"], "text/plain"), +]; + +/// `GEMINI_VIDEO_METADATA_KEYS`. +const VIDEO_METADATA_KEYS: &[(&str, &str)] = &[ + ("fps", "fps"), + ("start_offset", "startOffset"), + ("end_offset", "endOffset"), +]; + +fn bad_request(message: impl Into) -> Error { + Error::InvalidRequest(ErrorDetail::Message(message.into())) +} + +/// `get_image_mime_type_from_url`: the media type a URL's path extension names. +pub fn mime_type_from_url(url: &str) -> Option<&'static str> { + let lowered = url.to_lowercase(); + let rest = lowered + .split_once("://") + .map_or(lowered.as_str(), |(_, rest)| rest); + let path = rest.find('/').map_or("", |at| &rest[at..]); + let path = path.split(['?', '#']).next().unwrap_or_default(); + URL_MIME_TYPES + .iter() + .find(|(extensions, _)| extensions.iter().any(|ext| path.ends_with(ext))) + .map(|(_, mime)| *mime) +} + +/// `_explicit_mime_type`: `format`, `mime_type` or `content_type`, the first one set. +fn explicit_mime_type(fields: Option<&Map>) -> Option<&str> { + let fields = fields?; + ["format", "mime_type", "content_type"] + .iter() + .find_map(|key| fields.get(*key).filter(|value| truthy(value))) + .and_then(Value::as_str) +} + +/// `_apply_gemini_mime_type_aliases`. +fn mime_type_alias(mime_type: &str) -> String { + let normalized = mime_type + .split(';') + .next() + .unwrap_or_default() + .trim() + .to_lowercase(); + match normalized.as_str() { + "image/jpg" => "image/jpeg".to_owned(), + _ => normalized, + } +} + +/// `os.path.splitext(url)[1][1:]`. +fn extension(url: &str) -> &str { + let base = url.rsplit('/').next().unwrap_or(url); + let base = base.trim_start_matches('.'); + base.rfind('.').map_or("", |at| &base[at + 1..]) +} + +/// The media type Gemini takes a `gs://` object of this extension as. +fn gcs_mime_type(extension: &str) -> Result<&'static str, Error> { + let lowered = extension.to_lowercase(); + let (_, name, mime) = FILE_TYPES + .iter() + .find(|(ext, _, _)| *ext == lowered) + .ok_or_else(|| bad_request(format!("Unknown file type for extension: {extension}")))?; + mime.ok_or_else(|| { + bad_request(format!( + "File type not supported by gemini - FileType.{name}" + )) + }) +} + +/// `_convert_detail_to_media_resolution_enum`. +fn media_resolution(detail: Option<&Value>) -> Option<&'static str> { + match detail?.as_str()? { + "low" => Some("MEDIA_RESOLUTION_LOW"), + "medium" => Some("MEDIA_RESOLUTION_MEDIUM"), + "high" => Some("MEDIA_RESOLUTION_HIGH"), + "ultra_high" => Some("MEDIA_RESOLUTION_ULTRA_HIGH"), + _ => None, + } +} + +fn resolution_priority(detail: &str) -> u8 { + match detail { + "ultra_high" => 4, + "high" => 3, + "medium" => 2, + "low" => 1, + _ => 0, + } +} + +/// A `data:` URL as `{data, mime_type}`, the way `convert_to_anthropic_image_obj` reads it. A +/// remote URL here was not downloaded first, which only the host can do. +fn inline_blob(url: &str, format: Option<&str>) -> Result { + if url.starts_with("http") { + return Err(Error::Unsupported("media the host did not download")); + } + let (media_type, data) = data_url(url).ok_or_else(|| { + bad_request(format!( + "Image url not in expected format. Example Expected input - \"image_url\": \"data:image/jpeg;base64,{{base64_image}}\". Received: {}", + url.chars().take(64).collect::() + )) + })?; + let media_type = match format { + Some(format) => format.to_owned(), + None => media_type.replace("\\/", "/"), + }; + Ok(json!({"data": data, "mime_type": media_type})) +} + +/// `_process_gemini_media` with `_apply_gemini_metadata`. +fn media_part( + url: &str, + format: Option<&str>, + resolution: Option<&'static str>, + video_metadata: Option<&Value>, + model: &str, +) -> Result { + let mut part = if url.contains("gs://") { + let mime_type = match format { + Some(format) => mime_type_alias(format), + None => match extension(url) { + "" => { + return Err(Error::Unsupported( + "a gs:// URI with no file extension and no format", + )); + } + extension => gcs_mime_type(extension)?.to_owned(), + }, + }; + json!({"file_data": {"mime_type": mime_type, "file_uri": url}}) + } else if url.starts_with(GEMINI_FILES_API_URI_PREFIX) { + match format { + Some(format) => json!({"file_data": {"mime_type": format, "file_uri": url}}), + None => json!({"file_data": {"file_uri": url}}), + } + } else if let Some(mime_type) = url + .contains("https://") + .then(|| format.or_else(|| mime_type_from_url(url))) + .flatten() + { + json!({"file_data": {"mime_type": mime_type, "file_uri": url}}) + } else if url.contains("http://") || url.contains("https://") || url.contains("base64") { + json!({"inline_data": inline_blob(url, format)?}) + } else { + return Err(bad_request(format!("Invalid image received - {url}"))); + }; + if let Some(level) = resolution + && is_gemini_3_or_newer(model) + { + part["media_resolution"] = json!({"level": level}); + } + if let Some(Value::Object(metadata)) = video_metadata { + let mapped: Map = VIDEO_METADATA_KEYS + .iter() + .filter_map(|(openai, gemini)| { + metadata + .get(*openai) + .map(|value| ((*gemini).to_owned(), value.clone())) + }) + .collect(); + if !mapped.is_empty() { + part["video_metadata"] = Value::Object(mapped); + } + } + Ok(part) +} + +/// The `image_url` element of a user message. +fn image_url_part(element: &Value, model: &str) -> Result { + let (url, format, resolution) = match element.get("image_url") { + None | Some(Value::Null) => { + return Err(bad_request( + "Invalid message content: element type is 'image_url' but 'image_url' field is missing ", + )); + } + Some(Value::Object(image)) => { + let url = image.get("url").and_then(Value::as_str).ok_or_else(|| { + bad_request( + "Invalid message content: element type is 'image_url' but 'url' field is missing inside 'image_url' ", + ) + })?; + ( + url, + explicit_mime_type(Some(image)), + media_resolution(image.get("detail")), + ) + } + Some(Value::String(url)) => (url.as_str(), None, None), + Some(other) => { + return Err(bad_request(format!( + "Invalid message content: image_url must be a string or an object, got {other}" + ))); + } + }; + media_part(url, format, resolution, None, model) +} + +/// The `file` element of a user message. +fn file_part(element: &Value, model: &str) -> Result { + let Some(Value::Object(file)) = element.get("file") else { + return Err(bad_request( + "Content block has type='file' but is missing the required 'file' field", + )); + }; + let source = ["file_id", "file_data"] + .iter() + .find_map(|key| file.get(*key).filter(|value| truthy(value))) + .and_then(Value::as_str) + .ok_or_else(|| bad_request("Unknown file type. Please pass in a file_id or file_data"))?; + media_part( + source, + explicit_mime_type(Some(file)), + media_resolution(file.get("detail")), + file.get("video_metadata"), + model, + ) +} + +/// The `input_audio` element of a user message, which Gemini takes inline. +fn audio_part(element: &Value, model: &str) -> Result, Error> { + let audio = element.get("input_audio"); + let data = audio + .and_then(|audio| audio.get("data")) + .and_then(Value::as_str); + let format = audio + .and_then(|audio| audio.get("format")) + .and_then(Value::as_str); + let (Some(data), Some(format)) = (data, format) else { + return Ok(None); + }; + let format = match format.starts_with("audio/") { + true => format.to_owned(), + false => format!("audio/{format}"), + }; + media_part( + &format!("data:{format};base64,{data}"), + Some(&format), + None, + None, + model, + ) + .map(Some) +} + +/// The parts of one user or (merged) system message. +fn user_parts(message: &Value, model: &str) -> Result, Error> { + match message.get("content") { + Some(Value::String(text)) => Ok(vec![json!({"text": text})]), + Some(Value::Array(elements)) => { + let mut parts = Vec::new(); + for element in elements { + match element.get("type").and_then(Value::as_str) { + Some("text") => { + if let Some(text) = element.get("text").and_then(Value::as_str) + && !text.is_empty() + { + parts.push(json!({"text": text})); + } + } + Some("image_url") => parts.push(image_url_part(element, model)?), + Some("input_audio") => parts.extend(audio_part(element, model)?), + Some("file") => parts.push(file_part(element, model)?), + _ => {} + } + } + Ok(parts) + } + _ => Ok(Vec::new()), + } +} + +fn string_field<'a>(value: &'a Value, key: &str) -> Option<&'a str> { + value + .get(key) + .and_then(Value::as_str) + .filter(|text| !text.is_empty()) +} + +fn provider_signature(holder: &Value) -> Option<&str> { + holder + .get("provider_specific_fields") + .and_then(|fields| string_field(fields, "thought_signature")) +} + +/// `get_thought_signature_from_tool`: the tool's, its function's, or the one after the id's +/// separator. +pub fn thought_signature_from_tool(tool: &Value) -> Option<&str> { + provider_signature(tool) + .or_else(|| tool.get("function").and_then(provider_signature)) + .or_else(|| { + tool.get("id") + .and_then(Value::as_str) + .and_then(|id| id.split_once(THOUGHT_SIGNATURE_SEPARATOR)) + .map(|(_, signature)| signature) + .filter(|signature| !signature.is_empty()) + }) +} + +/// `_collect_tool_call_thought_signatures`. +fn tool_call_signatures(message: &Value) -> Vec<&str> { + let mut signatures: Vec<&str> = message + .get("tool_calls") + .and_then(Value::as_array) + .into_iter() + .flatten() + .filter_map(thought_signature_from_tool) + .collect(); + if let Some(function_call) = message.get("function_call").filter(|f| f.is_object()) { + signatures.extend(provider_signature(function_call)); + } + let invocations = message + .get("provider_specific_fields") + .and_then(|fields| fields.get("server_side_tool_invocations")) + .and_then(Value::as_array); + for invocation in invocations.into_iter().flatten() { + for key in ["thought_signature", "response_thought_signature"] { + signatures.extend(string_field(invocation, key)); + } + } + signatures +} + +/// `_gemini_tool_call_invoke_helper`. +fn function_call(function: &Value, id: Option<&str>) -> Result { + let name = function + .get("name") + .and_then(Value::as_str) + .unwrap_or_default(); + let args = match function.get("arguments") { + None => json!({"type": "object"}), + Some(Value::String(arguments)) if arguments.is_empty() => json!({"type": "object"}), + Some(Value::String(arguments)) => serde_json::from_str(arguments).map_err(|error| { + bad_request(format!( + "Unable to convert openai tool calls to gemini tool calls. Received error={error}" + )) + })?, + Some(other) => { + return Err(bad_request(format!( + "Unable to convert openai tool calls to gemini tool calls. Received arguments={other}" + ))); + } + }; + let mut call = json!({"name": name, "args": args}); + if let Some(id) = id + .map(|id| id.split(THOUGHT_SIGNATURE_SEPARATOR).next().unwrap_or(id)) + .filter(|id| !id.is_empty()) + { + call["id"] = json!(id); + } + Ok(call) +} + +/// `convert_to_gemini_tool_call_invoke`. Gemini 3 and newer sign the first call of a turn, so +/// history from an older model gets Google's placeholder signature there. +fn tool_call_parts(message: &Value, gemini_3: bool) -> Result, Error> { + let mut parts: Vec = Vec::new(); + if let Some(Value::Array(calls)) = message.get("tool_calls") { + for tool in calls { + let Some(function) = tool.get("function") else { + continue; + }; + let id = gemini_3 + .then(|| tool.get("id").and_then(Value::as_str)) + .flatten(); + let mut part = json!({"function_call": function_call(function, id)?}); + let signature = thought_signature_from_tool(tool) + .or((parts.is_empty() && gemini_3).then_some(DUMMY_THOUGHT_SIGNATURE)); + if let Some(signature) = signature { + part["thoughtSignature"] = json!(signature); + } + parts.push(part); + } + } else if let Some(function) = message.get("function_call").filter(|f| !f.is_null()) { + let mut part = json!({"function_call": function_call(function, None)?}); + let signature = + provider_signature(function).or(gemini_3.then_some(DUMMY_THOUGHT_SIGNATURE)); + if let Some(signature) = signature { + part["thoughtSignature"] = json!(signature); + } + parts.push(part); + } + Ok(parts) +} + +/// `snake_to_camel`. +pub fn snake_to_camel(name: &str) -> String { + let mut pieces = name.split('_'); + let mut camel = pieces.next().unwrap_or_default().to_owned(); + for piece in pieces { + let mut chars = piece.chars(); + if let Some(first) = chars.next() { + camel.extend(first.to_uppercase()); + camel.push_str(&chars.as_str().to_lowercase()); + } + } + camel +} + +/// `camel_to_snake`. +pub fn camel_to_snake(name: &str) -> String { + let mut snake = String::with_capacity(name.len() + 4); + for (at, ch) in name.chars().enumerate() { + if at > 0 && ch.is_uppercase() { + snake.push('_'); + } + snake.extend(ch.to_lowercase()); + } + snake +} + +/// `_get_equivalent_key`. +fn equivalent_key<'a>(key: &str, keys: impl Fn(&str) -> Option<&'a str>) -> Option<&'a str> { + keys(key) + .or_else(|| keys(&snake_to_camel(key))) + .or_else(|| keys(&camel_to_snake(key))) +} + +/// `check_if_part_exists_in_parts`, with `thoughtSignature` left out of the comparison. +fn part_exists(parts: &[Value], part: &Value) -> bool { + let Some(part) = part.as_object() else { + return false; + }; + parts.iter().filter_map(Value::as_object).any(|existing| { + part.iter() + .filter(|(key, _)| key.as_str() != "thoughtSignature") + .all(|(key, value)| { + equivalent_key(key, |name| { + existing.get_key_value(name).map(|(key, _)| key.as_str()) + }) + .is_some_and(|found| existing.get(found) == Some(value)) + }) + }) +} + +/// The `toolCall` and `toolResponse` parts Gemini ran server side, put back for the next turn. +fn server_side_parts(message: &Value) -> Vec { + let invocations = message + .get("provider_specific_fields") + .and_then(|fields| fields.get("server_side_tool_invocations")) + .and_then(Value::as_array); + let mut parts = Vec::new(); + for invocation in invocations.into_iter().flatten().filter(|i| i.is_object()) { + let field = |key: &str| invocation.get(key).cloned().unwrap_or(Value::Null); + let mut call = json!({"toolCall": {"toolType": field("tool_type"), "id": field("id"), "args": field("args")}}); + if let Some(signature) = invocation.get("thought_signature") { + call["thoughtSignature"] = signature.clone(); + } + parts.push(call); + if let Some(response) = invocation.get("response") { + let mut tool_response = json!({"id": field("id"), "response": response}); + if let Some(tool_type) = invocation.get("tool_type").filter(|t| truthy(t)) { + tool_response["toolType"] = tool_type.clone(); + } + let mut part = json!({"toolResponse": tool_response}); + if let Some(signature) = invocation.get("response_thought_signature") { + part["thoughtSignature"] = signature.clone(); + } + parts.push(part); + } + } + parts +} + +/// The parts of one assistant message. +fn assistant_parts( + message: &Value, + parts: &mut Vec, + model: &str, + gemini_3: bool, +) -> Result<(), Error> { + if let Some(reasoning) = message.get("reasoning_content").filter(|r| !r.is_null()) { + parts.push(json!({"thought": true, "text": reasoning})); + } + match message.get("content") { + Some(Value::Array(elements)) => parts.extend( + elements + .iter() + .filter(|element| element.get("type").and_then(Value::as_str) == Some("text")) + .map( + |element| json!({"text": element.get("text").cloned().unwrap_or(Value::Null)}), + ), + ), + Some(Value::String(text)) => { + let signature = message + .get("provider_specific_fields") + .and_then(|fields| fields.get("thought_signatures")) + .and_then(Value::as_array) + .and_then(|signatures| signatures.first()) + .filter(|first| { + first + .as_str() + .is_none_or(|first| !tool_call_signatures(message).contains(&first)) + }); + parts.push(match signature { + Some(signature) => json!({"text": text, "thoughtSignature": signature}), + None => json!({"text": text}), + }); + } + _ => {} + } + for image in message + .get("images") + .and_then(Value::as_array) + .into_iter() + .flatten() + { + let Some(Value::Object(image_url)) = image.get("image_url") else { + continue; + }; + if let Some(url) = image_url + .get("url") + .and_then(Value::as_str) + .filter(|url| !url.is_empty()) + { + parts.push(media_part( + url, + explicit_mime_type(Some(image_url)), + media_resolution(image_url.get("detail")), + None, + model, + )?); + } + } + for part in tool_call_parts(message, gemini_3)? { + if !part_exists(parts, &part) { + parts.push(part); + } + } + parts.extend(server_side_parts(message)); + Ok(()) +} + +/// `_contains_part_reference`: Gemini reads `{"$ref": ""}` in a function response +/// as a pointer to one of its parts, except under a list nested directly in a list. +fn contains_part_reference(node: &Value) -> bool { + match node { + Value::Object(object) => { + object + .get("$ref") + .and_then(Value::as_str) + .is_some_and(|reference| !reference.is_empty()) + || object.values().any(contains_part_reference) + } + Value::Array(items) => items + .iter() + .filter(|item| !item.is_array()) + .any(contains_part_reference), + _ => false, + } +} + +/// `_may_spell_ref_key`. +fn may_spell_ref_key(raw: &str) -> bool { + raw.contains("\"$ref\"") + || ["\\u0024", "\\u0072", "\\u0065", "\\u0066"] + .iter() + .any(|escape| raw.contains(escape)) +} + +/// The media of a tool result, inline the way `convert_to_gemini_tool_call_result` puts it. +fn tool_result_blob(content: &Value) -> Result, Error> { + match content.get("type").and_then(Value::as_str) { + Some("image") => { + let source = content.get("source"); + if source.and_then(|s| s.get("type")).and_then(Value::as_str) != Some("base64") { + return Ok(None); + } + let field = |key: &str, default: &str| { + source + .and_then(|s| s.get(key)) + .cloned() + .unwrap_or_else(|| json!(default)) + }; + Ok(Some( + json!({"data": field("data", ""), "mime_type": field("media_type", "image/jpeg")}), + )) + } + Some("input_image" | "image_url") => { + let url = match content.get("image_url") { + Some(Value::Object(image)) => image.get("url").and_then(Value::as_str), + Some(Value::String(url)) => Some(url.as_str()), + _ => None, + }; + match url.filter(|url| !url.is_empty()) { + Some(url) => tool_result_inline(url), + None => Ok(None), + } + } + Some("file" | "input_file") => { + let data = content + .get("file_data") + .filter(|data| truthy(data)) + .or_else(|| match content.get("file") { + Some(Value::Object(file)) => file.get("file_data"), + Some(file @ Value::String(_)) => Some(file), + _ => None, + }) + .and_then(Value::as_str) + .filter(|data| !data.is_empty()); + match data { + Some(data) => tool_result_inline(data), + None => Ok(None), + } + } + _ => Ok(None), + } +} + +/// Python fetches a remote tool result image while it converts the message and skips one it +/// cannot read; only the host can fetch, so a remote one it left is refused. +fn tool_result_inline(url: &str) -> Result, Error> { + if url.starts_with("http") { + return Err(Error::Unsupported("remote media in a tool result")); + } + Ok(inline_blob(url, None).ok()) +} + +/// `convert_to_gemini_tool_call_result`. +fn tool_result_part( + message: &Value, + last_tool_calls: Option<&Value>, + gemini_3: bool, +) -> Result { + let mut text = String::new(); + let mut blobs: Vec = Vec::new(); + match message.get("content") { + Some(Value::String(content)) => { + text = content.clone(); + if content.len() >= 5 + && content.is_char_boundary(5) + && content[..5].eq_ignore_ascii_case("data:") + && let Some((mime, data)) = content[5..].split_once(";base64,") + && mime.starts_with("image/") + { + let mime = mime.split(';').next().unwrap_or(mime).trim(); + blobs.push(json!({"data": data, "mime_type": mime})); + text.clear(); + } + } + Some(Value::Array(contents)) => { + for content in contents { + if content.get("type").and_then(Value::as_str) == Some("text") { + text.push_str( + content + .get("text") + .and_then(Value::as_str) + .unwrap_or_default(), + ); + } else { + blobs.extend(tool_result_blob(content)?); + } + } + } + _ => {} + } + let mut name = message + .get("name") + .and_then(Value::as_str) + .unwrap_or_default(); + let tool_call_id = message + .get("tool_call_id") + .and_then(Value::as_str) + .filter(|id| !id.is_empty()); + if let Some(calls) = last_tool_calls + .and_then(|last| last.get("tool_calls")) + .and_then(Value::as_array) + { + for call in calls { + if let Some(id) = tool_call_id + && call.get("id").and_then(Value::as_str) == Some(id) + { + name = call + .get("function") + .and_then(|f| f.get("name")) + .and_then(Value::as_str) + .unwrap_or_default(); + } + } + } + if name.is_empty() { + return Err(bad_request(format!( + "Missing corresponding tool call for tool response message. Received - message={message}" + ))); + } + let trimmed = text.trim(); + let response = match trimmed.starts_with('{') || trimmed.starts_with('[') { + true => match serde_json::from_str::(&text) { + Ok(parsed @ Value::Object(_)) + if !(may_spell_ref_key(&text) && contains_part_reference(&parsed)) => + { + parsed + } + _ => json!({"content": text}), + }, + false => json!({"content": text}), + }; + let mut function_response = json!({"name": name, "response": response}); + if gemini_3 + && let Some(id) = tool_call_id + .and_then(|id| id.split(THOUGHT_SIGNATURE_SEPARATOR).next()) + .filter(|id| !id.is_empty()) + { + function_response["id"] = json!(id); + } + if !blobs.is_empty() { + function_response["parts"] = Value::Array( + blobs + .into_iter() + .map(|blob| json!({"inline_data": blob})) + .collect(), + ); + } + Ok(json!({"function_response": function_response})) +} + +fn role(message: &Value) -> &str { + message + .get("role") + .and_then(Value::as_str) + .unwrap_or_default() +} + +/// `gemini_convert_messages_with_history`: consecutive user (and unextracted system) messages +/// merge into one `user` turn, consecutive assistant messages into one `model` turn, and +/// consecutive tool results into one `user` turn of function responses. +pub fn convert_messages(messages: &[Value], model: &str) -> Result, Error> { + let gemini_3 = is_gemini_3_or_newer(model); + let mut contents: Vec = Vec::new(); + let mut last_tool_calls: Option<&Value> = None; + let mut tool_responses: Vec = Vec::new(); + let mut at = 0; + while at < messages.len() { + let start = at; + let mut user: Vec = Vec::new(); + while at < messages.len() && matches!(role(&messages[at]), "user" | "system") { + user.extend(user_parts(&messages[at], model)?); + at += 1; + } + if !user.is_empty() { + if !user + .iter() + .any(|part| part.get("text").is_some_and(|text| !text.is_null())) + { + user.push(json!({"text": " "})); + } + contents.push(json!({"role": "user", "parts": user})); + } + let mut assistant: Vec = Vec::new(); + while at < messages.len() && role(&messages[at]) == "assistant" { + let message = &messages[at]; + assistant_parts(message, &mut assistant, model, gemini_3)?; + if message.get("tool_calls").is_some_and(truthy) + || message.get("function_call").is_some_and(|f| !f.is_null()) + { + last_tool_calls = Some(message); + } + at += 1; + } + if !assistant.is_empty() { + contents.push(json!({"role": "model", "parts": assistant})); + } + let is_tool_result = |message: &Value| matches!(role(message), "tool" | "function"); + if at < messages.len() && is_tool_result(&messages[at]) { + tool_responses.push(tool_result_part(&messages[at], last_tool_calls, gemini_3)?); + at += 1; + } + if at < messages.len() && !is_tool_result(&messages[at]) && !tool_responses.is_empty() { + contents.push(json!({"role": "user", "parts": std::mem::take(&mut tool_responses)})); + } + if at == start { + return Err(bad_request(format!( + "Invalid Message passed in - {}", + messages[at] + ))); + } + } + if !tool_responses.is_empty() { + contents.push(json!({"role": "user", "parts": tool_responses})); + } + if contents.is_empty() { + contents.push(json!({"role": "user", "parts": [{"text": " "}]})); + } + Ok(contents) +} + +/// `transform_system_message`: when the model takes a system instruction, the system messages +/// leave the conversation for it, and a conversation left empty gets a `.` user turn. +pub fn transform_system_message( + supports_system_message: bool, + messages: Vec, +) -> (Option, Vec) { + if !supports_system_message { + return (None, messages); + } + let mut blocks: Vec = Vec::new(); + let mut rest: Vec = Vec::new(); + for message in messages { + let block = match (role(&message), message.get("content")) { + ("system", Some(Value::String(text))) => Some(json!({"text": text})), + ("system", Some(Value::Array(items))) => Some(json!({"text": items + .iter() + .filter_map(|item| item.get("text").and_then(Value::as_str)) + .collect::()})), + _ => None, + }; + match block { + Some(block) => blocks.push(block), + None => rest.push(message), + } + } + if blocks.is_empty() { + return (None, rest); + } + if rest.is_empty() { + rest.push(json!({"role": "user", "content": "."})); + } + (Some(json!({"parts": blocks})), rest) +} + +/// `is_cached_message`: an ephemeral `cache_control` on the message or a text part, which +/// Python turns into a Gemini context cache it creates first. +pub fn is_cached_message(message: &Value) -> bool { + let ephemeral = |holder: &Value| { + holder + .get("cache_control") + .and_then(|control| control.get("type")) + .and_then(Value::as_str) + == Some("ephemeral") + }; + ephemeral(message) + || message + .get("content") + .and_then(Value::as_array) + .is_some_and(|parts| { + parts.iter().any(|part| { + part.get("type").and_then(Value::as_str) == Some("text") && ephemeral(part) + }) + }) +} + +/// `_extract_max_media_resolution_from_messages`. +fn max_media_resolution(messages: &[Value]) -> Option<&'static str> { + let mut best: Option<&str> = None; + for part in messages + .iter() + .filter_map(|message| message.get("content").and_then(Value::as_array)) + .flatten() + { + let holder = match part.get("type").and_then(Value::as_str) { + Some("image_url") => part.get("image_url"), + Some("file") => part.get("file"), + _ => None, + }; + let detail = holder + .filter(|holder| holder.is_object()) + .and_then(|holder| holder.get("detail")) + .and_then(Value::as_str) + .filter(|detail| !detail.is_empty()); + if let Some(detail) = detail + && resolution_priority(detail) > best.map_or(0, resolution_priority) + { + best = Some(detail); + } + } + media_resolution(best.map(|detail| json!(detail)).as_ref()) +} + +/// `_rewrite_google_maps_response_format`: Google Maps grounding rejects +/// `response_mime_type: application/json`, so the newer `responseFormat` carries it. +fn rewrite_google_maps_response_format(body: &mut Map) { + let has_maps = body + .get("tools") + .and_then(Value::as_array) + .is_some_and(|tools| tools.iter().any(|tool| tool.get("googleMaps").is_some())); + let Some(Value::Object(config)) = body.get_mut("generationConfig") else { + return; + }; + if !has_maps + || config.get("response_mime_type").and_then(Value::as_str) != Some("application/json") + { + return; + } + let schema = config + .remove("response_json_schema") + .or_else(|| config.remove("response_schema")); + config.remove("response_mime_type"); + let mut text = json!({"mimeType": "APPLICATION_JSON"}); + if let Some(schema) = schema { + text["schema"] = schema; + } + config.insert("responseFormat".into(), json!({"text": text})); +} + +/// `transform_request_body`: the `generateContent` body for messages already stripped of the +/// system instruction and params already mapped. +pub fn transform_request_body( + messages: &[Value], + system_instruction: Option, + model: &str, + mut params: Map, + surface: GeminiSurface, +) -> Result { + let contents = convert_messages(messages, model)?; + let tools = params.remove("tools"); + let tool_choice = params.remove("tool_choice"); + let include_server_side_tool_invocations = params + .remove("include_server_side_tool_invocations") + .is_some_and(|flag| truthy(&flag)); + let safety_settings = params.remove("safety_settings"); + params.remove("output_config"); + let labels: Option> = match params.remove("labels") { + Some(Value::Object(labels)) => Some( + labels + .into_iter() + .filter(|(_, value)| value.is_string()) + .collect(), + ), + _ => None, + }; + let cached_content = params.remove("cached_content"); + let service_tier = params.remove("service_tier").filter(truthy); + let continuation_token = params.remove("continuation_token").filter(truthy); + let extra_body = params.remove("extra_body"); + + let mut generation_config: Map = params + .into_iter() + .filter(|(key, _)| { + equivalent_key(key, |name| { + GENERATION_CONFIG_FIELDS + .iter() + .find(|field| **field == name) + .copied() + }) + .is_some() + }) + .collect(); + if model.contains("gemini-2") + && let Some(level) = max_media_resolution(messages) + { + generation_config.insert("mediaResolution".into(), json!(level)); + } + + let mut body = Map::new(); + body.insert("contents".into(), Value::Array(contents)); + if let Some(system_instruction) = system_instruction { + body.insert("system_instruction".into(), system_instruction); + } + if let Some(tools) = tools.filter(|tools| !tools.is_null()) { + body.insert("tools".into(), tools); + } + if let Some(tool_choice) = tool_choice.filter(|choice| !choice.is_null()) { + body.insert("toolConfig".into(), tool_choice); + } + if include_server_side_tool_invocations { + let config = body + .entry("toolConfig") + .or_insert_with(|| Value::Object(Map::new())); + config["includeServerSideToolInvocations"] = json!(true); + } + if let Some(safety_settings) = safety_settings.filter(|s| !s.is_null()) { + body.insert("safetySettings".into(), safety_settings); + } + if !generation_config.is_empty() { + body.insert("generationConfig".into(), Value::Object(generation_config)); + } + if let Some(cached_content) = cached_content.filter(|c| !c.is_null()) { + body.insert("cachedContent".into(), cached_content); + } + if let Some(service_tier) = service_tier { + let tier = match service_tier.as_str().map(str::to_lowercase) { + Some(tier) if tier == "default" => json!("standard"), + Some(tier) => json!(tier), + None => service_tier, + }; + body.insert("serviceTier".into(), tier); + } + if let Some(token) = continuation_token { + body.insert("continuationToken".into(), token); + } + if let Some(labels) = labels.filter(|labels| !labels.is_empty()) + && surface == GeminiSurface::Vertex + { + body.insert("labels".into(), Value::Object(labels)); + } + if let Some(Value::Object(extra)) = extra_body { + for (key, value) in extra { + if INTERNAL_EXTRA_BODY_KEYS.contains(&key.as_str()) { + continue; + } + match (body.get_mut(&key), value) { + (Some(Value::Object(existing)), Value::Object(update)) => existing.extend(update), + (_, value) => { + body.insert(key, value); + } + } + } + } + rewrite_google_maps_response_format(&mut body); + Ok(Value::Object(body)) +} + +/// The media Python downloads while it builds the request (`should_inline_remote_media`, and +/// the images a tool result names, which `convert_to_gemini_tool_call_result` fetches). Google +/// AI Studio cannot read a URL; Vertex AI reads an `https` one whose media type is known. +pub fn remote_media(messages: &[ChatMessage], surface: GeminiSurface) -> Vec { + let mut urls: Vec = Vec::new(); + for message in messages { + let Some(ChatMessageContent::Parts(parts)) = &message.content else { + continue; + }; + for part in parts { + let kind = part.get("type").and_then(Value::as_str); + let (url, fields) = match kind { + Some("image_url") => (image_url(part), part.get("image_url")), + Some("file") => ( + part.get("file") + .and_then(|file| file.get("file_id")) + .and_then(Value::as_str), + part.get("file"), + ), + _ => continue, + }; + let Some(url) = + url.filter(|url| url.starts_with("http://") || url.starts_with("https://")) + else { + continue; + }; + let by_reference = url.starts_with(GEMINI_FILES_API_URI_PREFIX); + let inlined = !by_reference + && match surface { + GeminiSurface::AiStudio => true, + GeminiSurface::Vertex => { + url.starts_with("http://") + || (explicit_mime_type(fields.and_then(Value::as_object)).is_none() + && mime_type_from_url(url).is_none()) + } + }; + let tool_image = message.role == "tool" && kind == Some("image_url"); + if (inlined || tool_image) && !urls.iter().any(|seen| seen == url) { + urls.push(url.to_owned()); + } + } + } + urls +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + fn messages(value: Value) -> Vec { + let Value::Array(messages) = value else { + panic!("messages are a list"); + }; + messages + } + + #[rstest] + #[case::user_and_system_merge( + json!([{"role": "system", "content": "be brief"}, {"role": "user", "content": "hi"}]), + json!([{"role": "user", "parts": [{"text": "be brief"}, {"text": "hi"}]}]) + )] + #[case::consecutive_assistants_merge( + json!([{"role": "user", "content": "a"}, {"role": "assistant", "content": "b"}, {"role": "assistant", "content": "c"}]), + json!([{"role": "user", "parts": [{"text": "a"}]}, {"role": "model", "parts": [{"text": "b"}, {"text": "c"}]}]) + )] + #[case::empty_conversation(json!([]), json!([{"role": "user", "parts": [{"text": " "}]}]))] + #[case::image_only_gets_blank_text( + json!([{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}}]}]), + json!([{"role": "user", "parts": [{"inline_data": {"data": "AAAA", "mime_type": "image/png"}}, {"text": " "}]}]) + )] + #[case::reasoning_becomes_a_thought( + json!([{"role": "user", "content": "q"}, {"role": "assistant", "content": "a", "reasoning_content": "think"}]), + json!([{"role": "user", "parts": [{"text": "q"}]}, {"role": "model", "parts": [{"thought": true, "text": "think"}, {"text": "a"}]}]) + )] + #[case::text_signature_replayed( + json!([{"role": "user", "content": "q"}, {"role": "assistant", "content": "a", "provider_specific_fields": {"thought_signatures": ["sig"]}}]), + json!([{"role": "user", "parts": [{"text": "q"}]}, {"role": "model", "parts": [{"text": "a", "thoughtSignature": "sig"}]}]) + )] + fn converts_messages_like_python(#[case] input: Value, #[case] expected: Value) { + assert_eq!( + Value::Array(convert_messages(&messages(input), "gemini-2.5-flash").unwrap()), + expected + ); + } + + fn tool_conversation() -> Value { + json!([ + {"role": "user", "content": "weather?"}, + {"role": "assistant", "content": null, "tool_calls": [ + {"id": "call_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}}, + {"id": "call_2", "type": "function", "function": {"name": "get_time", "arguments": ""}} + ]}, + {"role": "tool", "tool_call_id": "call_1", "content": "{\"temp\": 21}"}, + {"role": "tool", "tool_call_id": "call_2", "content": "noon"}, + {"role": "user", "content": "thanks"} + ]) + } + + #[rstest] + #[case::older_model("gemini-2.5-flash", json!([ + {"role": "user", "parts": [{"text": "weather?"}]}, + {"role": "model", "parts": [ + {"function_call": {"name": "get_weather", "args": {"city": "Paris"}}}, + {"function_call": {"name": "get_time", "args": {"type": "object"}}} + ]}, + {"role": "user", "parts": [ + {"function_response": {"name": "get_weather", "response": {"temp": 21}}}, + {"function_response": {"name": "get_time", "response": {"content": "noon"}}} + ]}, + {"role": "user", "parts": [{"text": "thanks"}]} + ]))] + #[case::gemini_3_forwards_ids_and_signs_the_first_call("gemini-3-pro-preview", json!([ + {"role": "user", "parts": [{"text": "weather?"}]}, + {"role": "model", "parts": [ + {"function_call": {"name": "get_weather", "args": {"city": "Paris"}, "id": "call_1"}, "thoughtSignature": DUMMY_THOUGHT_SIGNATURE}, + {"function_call": {"name": "get_time", "args": {"type": "object"}, "id": "call_2"}} + ]}, + {"role": "user", "parts": [ + {"function_response": {"name": "get_weather", "response": {"temp": 21}, "id": "call_1"}}, + {"function_response": {"name": "get_time", "response": {"content": "noon"}, "id": "call_2"}} + ]}, + {"role": "user", "parts": [{"text": "thanks"}]} + ]))] + fn tool_calls_and_results_round_trip(#[case] model: &str, #[case] expected: Value) { + assert_eq!( + Value::Array(convert_messages(&messages(tool_conversation()), model).unwrap()), + expected + ); + } + + #[rstest] + #[case::from_provider_fields(json!({"id": "call_1", "provider_specific_fields": {"thought_signature": "a"}, "function": {}}), Some("a"))] + #[case::from_function_fields(json!({"id": "call_1", "function": {"provider_specific_fields": {"thought_signature": "b"}}}), Some("b"))] + #[case::from_id(json!({"id": "call_1__thought__c", "function": {}}), Some("c"))] + #[case::none(json!({"id": "call_1", "function": {}}), None)] + fn thought_signatures_come_from_where_python_looks( + #[case] tool: Value, + #[case] expected: Option<&str>, + ) { + assert_eq!(thought_signature_from_tool(&tool), expected); + } + + #[rstest] + fn a_signed_call_id_is_stripped_and_its_signature_replayed() { + let conversation = json!([ + {"role": "user", "content": "q"}, + {"role": "assistant", "content": "calling", "provider_specific_fields": {"thought_signatures": ["s1"]}, "tool_calls": [ + {"id": "call_1__thought__s1", "type": "function", "function": {"name": "f", "arguments": "{}"}} + ]}, + {"role": "tool", "tool_call_id": "call_1__thought__s1", "content": "ok"} + ]); + let contents = convert_messages(&messages(conversation), "gemini-3-flash-preview").unwrap(); + assert_eq!( + contents[1]["parts"], + json!([ + {"text": "calling"}, + {"function_call": {"name": "f", "args": {}, "id": "call_1"}, "thoughtSignature": "s1"} + ]) + ); + assert_eq!(contents[2]["parts"][0]["function_response"]["id"], "call_1"); + } + + #[rstest] + #[case::no_matching_call(json!([{"role": "user", "content": "q"}, {"role": "tool", "tool_call_id": "x", "content": "r"}]))] + #[case::unknown_role(json!([{"role": "narrator", "content": "q"}]))] + fn refuses_what_python_refuses(#[case] input: Value) { + assert!(matches!( + convert_messages(&messages(input), "gemini-2.5-flash"), + Err(Error::InvalidRequest(_)) + )); + } + + #[rstest] + #[case::object(r#"{"a": 1}"#, json!({"a": 1}))] + #[case::list(r#"[1, 2]"#, json!({"content": "[1, 2]"}))] + #[case::part_reference(r##"{"x": {"$ref": "#/p"}}"##, json!({"content": r##"{"x": {"$ref": "#/p"}}"##}))] + #[case::reference_under_nested_list(r#"{"x": [[{"$ref": "p"}]]}"#, json!({"x": [[{"$ref": "p"}]]}))] + #[case::text("plain", json!({"content": "plain"}))] + fn tool_result_responses_parse_like_python(#[case] content: &str, #[case] expected: Value) { + let conversation = json!([ + {"role": "user", "content": "q"}, + {"role": "assistant", "tool_calls": [{"id": "c", "type": "function", "function": {"name": "f", "arguments": "{}"}}]}, + {"role": "tool", "tool_call_id": "c", "content": content} + ]); + let contents = convert_messages(&messages(conversation), "gemini-2.5-flash").unwrap(); + assert_eq!( + contents[2]["parts"][0]["function_response"]["response"], + expected + ); + } + + #[rstest] + fn a_tool_result_image_nests_inside_the_function_response() { + let conversation = json!([ + {"role": "user", "content": "q"}, + {"role": "assistant", "tool_calls": [{"id": "c", "type": "function", "function": {"name": "shot", "arguments": "{}"}}]}, + {"role": "tool", "tool_call_id": "c", "content": [ + {"type": "text", "text": "here"}, + {"type": "image_url", "image_url": {"url": "data:image/png;base64,QUJD"}} + ]} + ]); + let contents = convert_messages(&messages(conversation), "gemini-2.5-flash").unwrap(); + assert_eq!( + contents[2]["parts"], + json!([{"function_response": {"name": "shot", "response": {"content": "here"}, + "parts": [{"inline_data": {"data": "QUJD", "mime_type": "image/png"}}]}}]) + ); + } + + #[rstest] + #[case::gcs_extension("gs://b/x.png", None, Ok(json!({"file_data": {"mime_type": "image/png", "file_uri": "gs://b/x.png"}})))] + #[case::gcs_format("gs://b/x", Some("image/JPG"), Ok(json!({"file_data": {"mime_type": "image/jpeg", "file_uri": "gs://b/x"}})))] + #[case::gcs_rejected_type("gs://b/x.gif", None, Err(()))] + #[case::https_known_type("https://h/a.pdf?sig=1", None, Ok(json!({"file_data": {"mime_type": "application/pdf", "file_uri": "https://h/a.pdf?sig=1"}})))] + #[case::files_api("https://generativelanguage.googleapis.com/v1beta/files/abc", None, Ok(json!({"file_data": {"file_uri": "https://generativelanguage.googleapis.com/v1beta/files/abc"}})))] + #[case::data_url("data:audio/wav;base64,UklG", None, Ok(json!({"inline_data": {"data": "UklG", "mime_type": "audio/wav"}})))] + #[case::not_media("ftp://x", None, Err(()))] + fn media_becomes_file_data_or_inline_data( + #[case] url: &str, + #[case] format: Option<&str>, + #[case] expected: Result, + ) { + assert_eq!( + media_part(url, format, None, None, "gemini-2.5-flash").map_err(|_| ()), + expected + ); + } + + #[rstest] + fn an_extensionless_gcs_uri_without_a_format_is_left_to_python() { + assert_eq!( + media_part("gs://b/object", None, None, None, "gemini-2.5-flash"), + Err(Error::Unsupported( + "a gs:// URI with no file extension and no format" + )) + ); + } + + #[rstest] + #[case::gemini_3("gemini-3-pro-preview", true)] + #[case::gemini_2("gemini-2.5-pro", false)] + fn media_resolution_rides_on_the_part_only_for_gemini_3( + #[case] model: &str, + #[case] on_part: bool, + ) { + let part = image_url_part( + &json!({"type": "image_url", "image_url": {"url": "https://h/a.png", "detail": "high"}}), + model, + ) + .unwrap(); + assert_eq!(part.get("media_resolution").is_some(), on_part); + } + + #[rstest] + fn video_metadata_is_camel_cased() { + let part = file_part( + &json!({"type": "file", "file": {"file_id": "gs://b/v.mp4", "video_metadata": {"fps": 2, "start_offset": "1s"}}}), + "gemini-2.5-flash", + ) + .unwrap(); + assert_eq!( + part["video_metadata"], + json!({"fps": 2, "startOffset": "1s"}) + ); + } + + #[rstest] + #[case::strings(json!([{"role": "system", "content": "a"}, {"role": "user", "content": "b"}]), Some(json!({"parts": [{"text": "a"}]})), 1)] + #[case::only_system(json!([{"role": "system", "content": [{"type": "text", "text": "a"}, {"type": "text", "text": "b"}]}]), Some(json!({"parts": [{"text": "ab"}]})), 1)] + #[case::none(json!([{"role": "user", "content": "b"}]), None, 1)] + fn system_messages_become_the_system_instruction( + #[case] input: Value, + #[case] instruction: Option, + #[case] left: usize, + ) { + let (system, rest) = transform_system_message(true, messages(input)); + assert_eq!(system, instruction); + assert_eq!(rest.len(), left); + } + + #[rstest] + #[case::message_level(json!({"role": "user", "content": "x", "cache_control": {"type": "ephemeral"}}), true)] + #[case::part_level(json!({"role": "user", "content": [{"type": "text", "text": "x", "cache_control": {"type": "ephemeral"}}]}), true)] + #[case::plain(json!({"role": "user", "content": "x"}), false)] + fn cached_messages_are_recognized(#[case] message: Value, #[case] cached: bool) { + assert_eq!(is_cached_message(&message), cached); + } + + #[rstest] + #[case::snake("max_output_tokens", "maxOutputTokens")] + #[case::single("seed", "seed")] + fn snake_case_becomes_camel_case(#[case] snake: &str, #[case] camel: &str) { + assert_eq!(snake_to_camel(snake), camel); + assert_eq!(camel_to_snake(camel), snake); + } + + fn params(value: Value) -> Map { + let Value::Object(map) = value else { + panic!("params are an object"); + }; + map + } + + #[rstest] + fn the_body_sorts_params_into_their_blocks() { + let body = transform_request_body( + &messages(json!([{"role": "user", "content": "hi"}])), + Some(json!({"parts": [{"text": "sys"}]})), + "gemini-2.5-flash", + params(json!({ + "temperature": 0.2, + "topK": 3, + "max_output_tokens": 10, + "tools": [{"function_declarations": [{"name": "f"}]}], + "tool_choice": {"functionCallingConfig": {"mode": "AUTO"}}, + "safety_settings": [{"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_NONE"}], + "labels": {"team": "a", "n": 1}, + "service_tier": "default", + "parallel_tool_calls": true, + "extra_body": {"generationConfig": {"seed": 4}, "cache": {"no-cache": true}} + })), + GeminiSurface::Vertex, + ) + .unwrap(); + assert_eq!( + body, + json!({ + "contents": [{"role": "user", "parts": [{"text": "hi"}]}], + "system_instruction": {"parts": [{"text": "sys"}]}, + "tools": [{"function_declarations": [{"name": "f"}]}], + "toolConfig": {"functionCallingConfig": {"mode": "AUTO"}}, + "safetySettings": [{"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_NONE"}], + "generationConfig": {"temperature": 0.2, "topK": 3, "max_output_tokens": 10, "seed": 4}, + "serviceTier": "standard", + "labels": {"team": "a"} + }) + ); + } + + #[rstest] + #[case::ai_studio(GeminiSurface::AiStudio, false)] + #[case::vertex(GeminiSurface::Vertex, true)] + fn labels_reach_vertex_only(#[case] surface: GeminiSurface, #[case] sent: bool) { + let body = transform_request_body( + &messages(json!([{"role": "user", "content": "hi"}])), + None, + "gemini-2.5-flash", + params(json!({"labels": {"team": "a"}})), + surface, + ) + .unwrap(); + assert_eq!(body.get("labels").is_some(), sent); + } + + #[rstest] + fn google_maps_with_json_output_uses_response_format() { + let body = transform_request_body( + &messages(json!([{"role": "user", "content": "hi"}])), + None, + "gemini-2.5-flash", + params(json!({ + "tools": [{"googleMaps": {}}], + "response_mime_type": "application/json", + "response_json_schema": {"type": "object"} + })), + GeminiSurface::AiStudio, + ) + .unwrap(); + assert_eq!( + body["generationConfig"], + json!({"responseFormat": {"text": {"mimeType": "APPLICATION_JSON", "schema": {"type": "object"}}}}) + ); + } + + #[rstest] + fn gemini_2_takes_the_highest_image_detail_globally() { + let body = transform_request_body( + &messages(json!([{"role": "user", "content": [ + {"type": "image_url", "image_url": {"url": "https://h/a.png", "detail": "low"}}, + {"type": "image_url", "image_url": {"url": "https://h/b.png", "detail": "high"}} + ]}])), + None, + "gemini-2.5-flash", + Map::new(), + GeminiSurface::Vertex, + ) + .unwrap(); + assert_eq!( + body["generationConfig"], + json!({"mediaResolution": "MEDIA_RESOLUTION_HIGH"}) + ); + } + + fn chat_messages(value: Value) -> Vec { + serde_json::from_value(value).unwrap() + } + + #[rstest] + #[case::ai_studio_fetches_every_url(GeminiSurface::AiStudio, vec!["https://h/a.png", "http://h/b", "https://h/c", "https://h/d"])] + #[case::vertex_fetches_http_and_unknown_types(GeminiSurface::Vertex, vec!["http://h/b", "https://h/c"])] + fn remote_media_follows_the_surface( + #[case] surface: GeminiSurface, + #[case] expected: Vec<&str>, + ) { + let list = chat_messages(json!([{"role": "user", "content": [ + {"type": "image_url", "image_url": {"url": "https://h/a.png"}}, + {"type": "image_url", "image_url": "http://h/b"}, + {"type": "file", "file": {"file_id": "https://h/c"}}, + {"type": "file", "file": {"file_id": "https://h/d", "format": "application/pdf"}}, + {"type": "file", "file": {"file_id": "https://generativelanguage.googleapis.com/v1beta/files/x"}}, + {"type": "image_url", "image_url": {"url": "gs://b/x.png"}} + ]}])); + assert_eq!(remote_media(&list, surface), expected); + } +} diff --git a/litellm-rust/crates/llms/src/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.rs b/litellm-rust/crates/llms/src/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.rs new file mode 100644 index 00000000000..891fff963bf --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.rs @@ -0,0 +1,2174 @@ +//! Python's `VertexGeminiConfig` for `/chat/completions`, shared with Google AI Studio's +//! `GoogleAIStudioGeminiConfig`: `get_supported_openai_params` and `map_openai_params` on the +//! way in, `transform_response` and `_calculate_usage` on the way out. + +use std::collections::BTreeMap; + +use litellm_auth::VertexParams; +use litellm_auth_gcp::{ + VertexConfig, get_vertex_ai_location, get_vertex_ai_project, + get_vertex_ai_project_from_credentials, +}; +use litellm_core_utils::core_helpers::{finish_reason_for, unix_now}; +use litellm_http::request::has_header; +use litellm_llms_types::{ + formats::chat_completions::{ + ChatCompletionThinkingBlock, ChatCompletionToolCallChunk, + ChatCompletionToolCallFunctionChunk, ChatCompletionsChoice, ChatCompletionsChoiceMessage, + ChatCompletionsResponse, ChatCompletionsUsage, ChatMessage, CompletionTokensDetails, + PromptTokensDetails, + }, + providers::vertex_ai::{DEFAULT_LOCATION, GENERATE_CONTENT, STREAM_GENERATE_CONTENT}, +}; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, + base_llm::{ + auth::{AuthScheme, Headers, ValidatedEnvironment}, + chat::{ + params::{ + ChatRequestContext, capability, model_field, ordered, sort_params, truthy, + unsupported_value, + }, + streaming::{ChatStream, StreamShape}, + transformation::{ + BaseConfig, ProviderChatRequestData, ProviderChatResponseData, Unsupported, + }, + }, + }, + vertex_ai::{ + common_utils::{get_vertex_base_url, graft_custom_vertex_base, with_query}, + gemini::{ + handler::gemini_chat_stream, + schema::{ + build_vertex_schema, remove_additional_properties, remove_strict_from_schema, + }, + transformation::{ + THOUGHT_SIGNATURE_SEPARATOR, is_cached_message, remote_media, + transform_request_body, transform_system_message, + }, + }, + }, +}; + +/// Vertex AI call configuration a host passes down: consumed for auth and the URL, never +/// serialized into the body. +pub const CONFIG_PARAMS: &[&str] = &[ + "vertex_credentials", + "vertex_project", + "vertex_location", + "vertex_ai_credentials", + "vertex_ai_project", + "vertex_ai_location", +]; + +/// The env var a Vertex AI base URL falls back to. +pub const VERTEXAI_API_BASE_ENV: &str = "VERTEXAI_API_BASE"; + +/// `SUPPORTED_REASONING_EFFORTS`. +const REASONING_EFFORTS: &[&str] = &["minimal", "low", "medium", "high", "none", "disable"]; + +/// `OPENAI_TO_GEMINI_TTS_VOICES`. +const TTS_VOICES: &[(&str, &str)] = &[ + ("alloy", "Kore"), + ("ash", "Iapetus"), + ("ballad", "Algieba"), + ("cedar", "Achird"), + ("coral", "Sulafat"), + ("echo", "Charon"), + ("fable", "Umbriel"), + ("marin", "Despina"), + ("onyx", "Orus"), + ("sage", "Vindemiatrix"), + ("shimmer", "Achernar"), + ("verse", "Puck"), +]; + +/// `get_flagged_finish_reasons`: a first candidate stopped for one of these answers nothing. +const FLAGGED_FINISH_REASONS: &[&str] = &[ + "SAFETY", + "RECITATION", + "BLOCKLIST", + "PROHIBITED_CONTENT", + "SPII", + "IMAGE_SAFETY", + "IMAGE_PROHIBITED_CONTENT", +]; + +/// `_search_tool_keys`. +const SEARCH_TOOL_KEYS: &[&str] = &[ + "googleSearch", + "googleSearchRetrieval", + "enterpriseWebSearch", + "url_context", + "google_search", + "google_search_retrieval", + "enterprise_web_search", + "urlContext", +]; + +/// The models whose penalties Gemini rejects although they predate Gemini 3. +const NO_PENALTY_MODELS: &[&str] = &["gemini-2.5-pro-preview-06-05"]; + +/// `VertexAIPartnerModels.is_vertex_partner_model` prefixes. +const PARTNER_MODEL_PREFIXES: &[&str] = &[ + "meta/", + "deepseek-ai", + "mistral", + "codestral", + "jamba", + "claude", + "qwen", + "openai/gpt-oss-", + "minimaxai/", + "moonshotai/", + "zai-org/", + "google/gemma-", +]; + +/// Which Google API serves the Gemini model. +#[derive(Clone, Copy, Debug, PartialEq, Eq, strum::IntoStaticStr)] +pub enum GeminiSurface { + /// Google AI Studio, `gemini/`, authenticated with an API key. + #[strum(serialize = "gemini")] + AiStudio, + /// Vertex AI, `vertex_ai/`, authenticated with a Google access token. + #[strum(serialize = "vertex_ai")] + Vertex, +} + +impl GeminiSurface { + pub fn provider(self) -> &'static str { + self.into() + } + + /// The cost map keys Python's `supports_*` helpers try for the model. + pub fn candidates(self, model: &str) -> Vec { + match self { + Self::AiStudio => vec![format!("gemini/{model}"), model.to_owned()], + Self::Vertex => vec![model.to_owned(), format!("vertex_ai/{model}")], + } + } +} + +/// `_is_gemini_3_or_newer`: anything named after Gemini that is not a 1.x, 2.x, `exp` or +/// unversioned `pro`/`flash` model, a fine-tuned endpoint or a Gemma model. +pub fn is_gemini_3_or_newer(model: &str) -> bool { + let name = model.rsplit('/').next().unwrap_or(model).to_lowercase(); + let fine_tuned = name.chars().all(|c| c.is_ascii_digit()) + || (model.starts_with("gemini/") && !name.starts_with("gemini-")); + if name.is_empty() || fine_tuned || name.starts_with("gemma-") { + return false; + } + !is_pre_gemini_3(&name) +} + +/// `^gemini-(?:[12](?:\.\d+)?|exp|(?:pro|flash)(?!-(?:lite-)?latest$))(?:-|$)`. +fn is_pre_gemini_3(name: &str) -> bool { + let Some(rest) = name.strip_prefix("gemini-") else { + return false; + }; + let boundary = |tail: &str| tail.is_empty() || tail.starts_with('-'); + if let Some(after) = rest.strip_prefix(['1', '2']) { + if boundary(after) { + return true; + } + if let Some(fraction) = after.strip_prefix('.') { + let digits = fraction.chars().take_while(char::is_ascii_digit).count(); + if digits > 0 && boundary(&fraction[digits..]) { + return true; + } + } + } + if rest.strip_prefix("exp").is_some_and(boundary) { + return true; + } + ["pro", "flash"].iter().any(|word| { + rest.strip_prefix(word) + .is_some_and(|after| boundary(after) && after != "-latest" && after != "-lite-latest") + }) +} + +/// `supports_response_json_schema`: Gemini 2.0 and newer take a JSON Schema as is. +pub fn supports_response_json_schema(model: &str) -> bool { + let lowered = model.to_lowercase(); + lowered.match_indices("gemini-").any(|(at, _)| { + let tail = &lowered[at + "gemini-".len()..]; + let digits = tail.chars().take_while(char::is_ascii_digit).count(); + let version = &tail[..digits]; + let separated = tail[digits..].starts_with(['.', '-']); + separated + && match version.len() { + 0 => false, + 1 => version != "0" && version != "1", + _ => !version.starts_with('0'), + } + }) +} + +/// `get_vertex_ai_model_route(model) == GEMINI`: the models Vertex AI serves through +/// `generateContent`, as opposed to its partner, Gemma and Model Garden routes. +pub fn is_vertex_gemini_model(model: &str) -> bool { + !model.contains("agent_engine/") + && !PARTNER_MODEL_PREFIXES + .iter() + .any(|prefix| model.starts_with(prefix)) + && !model.to_lowercase().contains("bge") + && !model.contains("gemma/") + && !model.contains("openai") + && !model.starts_with("xai/") + && model.contains("gemini") +} + +/// `_supports_penalty_parameters`. +fn supports_penalties(model: &str) -> bool { + !is_gemini_3_or_newer(model) && !NO_PENALTY_MODELS.contains(&model) +} + +/// `get_supported_openai_params` of either config. The reasoning params depend on the cost +/// map, which is only read when the request carries one. +pub fn supported_params( + surface: GeminiSurface, + model: &str, + context: &ChatRequestContext<'_>, + openai: &Map, +) -> Result, Error> { + let mut params: Vec<&'static str> = match surface { + GeminiSurface::Vertex => vec![ + "temperature", + "top_p", + "top_k", + "max_tokens", + "max_completion_tokens", + "stream", + "tools", + "functions", + "tool_choice", + "response_format", + "n", + "stop", + "extra_headers", + "seed", + "logprobs", + "top_logprobs", + "modalities", + "audio", + "parallel_tool_calls", + "web_search_options", + "include_server_side_tool_invocations", + "service_tier", + ], + GeminiSurface::AiStudio => vec![ + "temperature", + "top_p", + "max_tokens", + "max_completion_tokens", + "stream", + "tools", + "tool_choice", + "functions", + "response_format", + "n", + "stop", + "logprobs", + "frequency_penalty", + "presence_penalty", + "seed", + "modalities", + "parallel_tool_calls", + "web_search_options", + "include_server_side_tool_invocations", + "service_tier", + ], + }; + if surface == GeminiSurface::Vertex && supports_penalties(model) { + params.extend(["frequency_penalty", "presence_penalty"]); + } + let asks_reasoning = openai.contains_key("reasoning_effort") || openai.contains_key("thinking"); + if asks_reasoning + && (is_gemini_3_or_newer(model) + || capability(context, &surface.candidates(model), "supports_reasoning")? == Some(true)) + { + params.extend(["reasoning_effort", "thinking"]); + } + if surface == GeminiSurface::AiStudio && model.contains("tts") { + params.push("audio"); + } + Ok(params) +} + +fn add_tools(params: &mut Map, tools: Vec) { + match params.get_mut("tools") { + Some(Value::Array(existing)) => existing.extend(tools), + _ => { + params.insert("tools".into(), Value::Array(tools)); + } + } +} + +/// `get_tool_value`: the tool's config under its camelCase name or its snake_case one. +fn tool_value(tool: &Map, name: &str) -> Option { + let snake: String = name + .chars() + .flat_map(|c| match c.is_uppercase() { + true => vec!['_', c.to_ascii_lowercase()], + false => vec![c], + }) + .collect::() + .trim_start_matches('_') + .to_owned(); + tool.get(name) + .filter(|value| !value.is_null()) + .or_else(|| tool.get(&snake).filter(|value| !value.is_null())) + .cloned() +} + +/// `_transform_computer_use_config`. +fn computer_use_config(config: &Value) -> Value { + let mut mapped = Map::new(); + match config.get("environment").and_then(Value::as_str) { + Some("browser") => { + mapped.insert("environment".into(), json!("ENVIRONMENT_BROWSER")); + } + Some("unspecified") => { + mapped.insert("environment".into(), json!("ENVIRONMENT_UNSPECIFIED")); + } + Some(value @ ("ENVIRONMENT_BROWSER" | "ENVIRONMENT_UNSPECIFIED")) => { + mapped.insert("environment".into(), json!(value)); + } + _ => {} + } + if let Some(excluded) = config + .get("excluded_predefined_functions") + .or_else(|| config.get("excludedPredefinedFunctions")) + { + mapped.insert("excludedPredefinedFunctions".into(), excluded.clone()); + } + Value::Object(mapped) +} + +/// `_map_function`: function declarations grouped in one tool, each built-in tool in its own. +fn map_function( + tools: &Value, + include_server_side_tool_invocations: bool, +) -> Result, Error> { + let mut tools = tools.clone(); + remove_additional_properties(&mut tools); + remove_strict_from_schema(&mut tools); + let mut declarations: Vec = Vec::new(); + let mut google_search = None; + let mut google_search_retrieval = None; + let mut enterprise_web_search = None; + let mut url_context = None; + let mut code_execution = None; + let mut google_maps = None; + let mut computer_use = None; + for tool in tools.as_array().into_iter().flatten() { + let Some(original) = tool.as_object() else { + continue; + }; + let function = match original.get("function") { + Some(Value::Object(function)) => { + let mut function = function.clone(); + if let Some(parameters @ Value::Object(_)) = function.remove("parameters") { + function.insert("parameters".into(), build_vertex_schema(parameters, false)); + } else if let Some(parameters) = original + .get("function") + .and_then(|function| function.get("parameters")) + { + function.insert("parameters".into(), parameters.clone()); + } + Some(function) + } + _ if original.contains_key("name") => Some(original.clone()), + _ => None, + }; + let tool: Map = match original.get("type").and_then(Value::as_str) { + Some("computer_use") => { + let config: Map = original + .iter() + .filter(|(key, _)| key.as_str() != "type") + .map(|(key, value)| (key.clone(), value.clone())) + .collect(); + Map::from_iter([("computerUse".to_owned(), Value::Object(config))]) + } + Some("web_search" | "web_search_preview") => { + Map::from_iter([("googleSearch".to_owned(), json!({}))]) + } + _ => original + .iter() + .filter(|(key, _)| key.as_str() != "type") + .map(|(key, value)| (key.clone(), value.clone())) + .collect(), + }; + let name = (tool.len() == 1) + .then(|| tool.keys().next().map(String::as_str)) + .flatten(); + match name { + Some("codeExecution" | "code_execution") => { + code_execution = tool_value(&tool, "codeExecution"); + } + Some(name @ ("googleSearch" | "google_search")) => { + google_search = tool_value(&tool, name); + } + Some(name @ ("googleSearchRetrieval" | "google_search_retrieval")) => { + google_search_retrieval = tool_value(&tool, name); + } + Some(name @ ("enterpriseWebSearch" | "enterprise_web_search")) => { + enterprise_web_search = tool_value(&tool, name); + } + Some(name @ ("url_context" | "urlContext")) => { + url_context = tool_value(&tool, name); + } + Some("googleMaps" | "google_maps") => { + if let Some(Value::Object(config)) = tool_value(&tool, "googleMaps") { + google_maps = Some(Value::Object( + config + .into_iter() + .filter(|(key, _)| { + !matches!(key.as_str(), "latitude" | "longitude" | "languageCode") + }) + .collect(), + )); + } + } + Some("computerUse" | "computer_use") => { + computer_use = Some(match tool_value(&tool, "computerUse") { + Some(config) => computer_use_config(&config), + None => json!({}), + }); + } + _ => { + if let Some(function) = function { + let name = function.get("name").cloned().ok_or_else(|| { + unsupported_value("Invalid tool: a function declaration needs a name") + })?; + let mut declaration = json!({"name": name}); + if let Some(description) = function.get("description").filter(|d| !d.is_null()) + { + declaration["description"] = description.clone(); + } + match function.get("parameters") { + Some(Value::String(text)) if text.is_empty() => { + declaration["parameters"] = json!({"type": "object"}); + } + Some(Value::Null) | None => {} + Some(parameters) => declaration["parameters"] = parameters.clone(), + } + declarations.push(declaration); + } + } + } + } + let has_search = [ + &google_search, + &google_search_retrieval, + &enterprise_web_search, + &url_context, + ] + .iter() + .any(|tool| tool.is_some()); + if !declarations.is_empty() && has_search && !include_server_side_tool_invocations { + google_search = None; + google_search_retrieval = None; + enterprise_web_search = None; + url_context = None; + } + let mut mapped = Vec::new(); + if !declarations.is_empty() { + mapped.push(json!({"function_declarations": declarations})); + } + for (key, value) in [ + ("googleSearch", google_search), + ("googleSearchRetrieval", google_search_retrieval), + ("enterpriseWebSearch", enterprise_web_search), + ("code_execution", code_execution), + ("url_context", url_context), + ("googleMaps", google_maps), + ("computerUse", computer_use), + ] { + if let Some(value) = value { + mapped.push(json!({key: value})); + } + } + Ok(mapped) +} + +/// `map_tool_choice_values`. +fn map_tool_choice(value: &Value) -> Result { + let mode = |mode: &str| json!({"functionCallingConfig": {"mode": mode}}); + match value { + Value::String(choice) if choice == "none" => Ok(mode("NONE")), + Value::String(choice) if choice == "required" => Ok(mode("ANY")), + Value::String(choice) if choice == "auto" => Ok(mode("AUTO")), + Value::Object(choice) => { + let name = choice + .get("function") + .and_then(|function| function.get("name")) + .cloned() + .unwrap_or_else(|| json!("")); + Ok(json!({"functionCallingConfig": {"mode": "ANY", "allowed_function_names": [name]}})) + } + other => Err(unsupported_value(format!( + "VertexAI doesn't support tool_choice={}. Supported tool_choice values=['auto', 'required', json object]. To drop it from the call, set `litellm.drop_params = True.", + other + .as_str() + .map_or_else(|| other.to_string(), str::to_owned) + ))), + } +} + +/// `apply_response_schema_transformation`. +fn map_response_format(value: &Value, model: &str, params: &mut Map) { + let mut value = value.clone(); + remove_strict_from_schema(&mut value); + let json_schema = supports_response_json_schema(model); + if !json_schema { + remove_additional_properties(&mut value); + } + match value.get("type").and_then(Value::as_str) { + Some("json_object") => { + params.insert("response_mime_type".into(), json!("application/json")); + } + Some("text") => { + params.insert("response_mime_type".into(), json!("text/plain")); + } + _ => {} + } + let schema = if let Some(schema) = value.get("response_schema") { + params.insert("response_mime_type".into(), json!("application/json")); + Some(schema.clone()) + } else if value.get("type").and_then(Value::as_str) == Some("json_schema") + && let Some(schema) = value.get("json_schema").and_then(|s| s.get("schema")) + { + params.insert("response_mime_type".into(), json!("application/json")); + Some(schema.clone()) + } else { + None + }; + if let Some(schema) = schema.filter(|schema| schema.as_object().is_some_and(|s| !s.is_empty())) + { + match json_schema { + true => params.insert("response_json_schema".into(), schema), + false => params.insert("response_schema".into(), build_vertex_schema(schema, true)), + }; + } +} + +/// `_map_audio_params`. +fn map_audio(value: &Value) -> Result { + if let Some(format) = value.get("format").filter(|format| !format.is_null()) + && format.as_str() != Some("pcm16") + { + return Err(unsupported_value(format!( + "Unsupported audio format for Gemini TTS models: {}. Gemini TTS models only support 'pcm16' format as they return audio data in L16 PCM format. Please set audio format to 'pcm16'.", + format + .as_str() + .map_or_else(|| format.to_string(), str::to_owned) + ))); + } + let mut speech = Map::new(); + if let Some(voice) = value.get("voice") { + let name = match voice.as_str() { + Some(voice) => { + let lowered = voice.to_lowercase(); + json!( + TTS_VOICES + .iter() + .find(|(openai, _)| *openai == lowered) + .map_or(voice, |(_, gemini)| *gemini) + ) + } + None => voice.clone(), + }; + speech.insert( + "voiceConfig".into(), + json!({"prebuiltVoiceConfig": {"voiceName": name}}), + ); + } + if let Some(language) = value.get("language_code") { + speech.insert("languageCode".into(), language.clone()); + } + Ok(Value::Object(speech)) +} + +/// `map_response_modalities`. +fn map_modalities(value: &[Value]) -> Value { + Value::Array( + value + .iter() + .map(|modality| { + json!(match modality.as_str() { + Some("text") => "TEXT", + Some("image") => "IMAGE", + Some("audio") => "AUDIO", + Some("video") => "VIDEO", + _ => "MODALITY_UNSPECIFIED", + }) + }) + .collect(), + ) +} + +struct ReasoningModel<'a> { + model: &'a str, + surface: GeminiSurface, + context: &'a ChatRequestContext<'a>, +} + +impl ReasoningModel<'_> { + /// `_supports_minimal_thinking_level`. + fn supports_minimal(&self) -> bool { + is_gemini_3_or_newer(self.model) + && self.model.to_lowercase().contains("flash") + && model_field( + self.context, + &self.surface.candidates(self.model), + "supports_minimal_reasoning_effort", + ) + .and_then(|value| value.as_bool()) + != Some(false) + } + + /// `_map_reasoning_effort`: a thinking level on Gemini 3 and newer, a token budget before. + fn reasoning_effort(&self, value: &Value) -> Result, Error> { + let effort = match value { + Value::String(effort) => effort.as_str(), + Value::Object(object) => match object.get("effort").and_then(Value::as_str) { + Some(effort) => effort, + None => return Ok(None), + }, + _ => return Ok(None), + }; + if !REASONING_EFFORTS.contains(&effort) { + return Err(unsupported_value(format!( + "Invalid `reasoning_effort`: '{effort}'. Must be one of: {}. To drop this param, set `litellm.drop_params = True` or pass in `(.., drop_params=True)` in the request - https://docs.litellm.ai/docs/completion/drop_params", + REASONING_EFFORTS + .iter() + .map(|effort| format!("'{effort}'")) + .collect::>() + .join(", ") + ))); + } + let thoughts = true; + if is_gemini_3_or_newer(self.model) { + let floor = match self.supports_minimal() { + true => "minimal", + false => "low", + }; + return Ok(Some(match effort { + "minimal" => json!({"thinkingLevel": floor, "includeThoughts": thoughts}), + "disable" | "none" => json!({"thinkingLevel": floor, "includeThoughts": false}), + level => json!({"thinkingLevel": level, "includeThoughts": thoughts}), + })); + } + let lowered = self.model.to_lowercase(); + Ok(Some(match effort { + "minimal" => { + let budget = if lowered.contains("gemini-2.5-flash-lite") { + 512 + } else if lowered.contains("gemini-2.5-pro") { + 128 + } else if lowered.contains("gemini-2.5-flash") { + 1 + } else { + 128 + }; + json!({"thinkingBudget": budget, "includeThoughts": thoughts}) + } + "low" => json!({"thinkingBudget": 1024, "includeThoughts": thoughts}), + "medium" => json!({"thinkingBudget": 2048, "includeThoughts": thoughts}), + "high" => json!({"thinkingBudget": 4096, "includeThoughts": thoughts}), + _ => json!({"thinkingBudget": 0, "includeThoughts": false}), + })) + } + + /// `_map_thinking_param`. + fn thinking(&self, value: &Value) -> Value { + let enabled = value.get("type").and_then(Value::as_str) == Some("enabled"); + let budget = value + .get("budget_tokens") + .filter(|budget| !budget.is_null()); + let zero = budget.and_then(Value::as_f64) == Some(0.0); + let mut config = Map::new(); + if is_gemini_3_or_newer(self.model) { + config.insert("includeThoughts".into(), json!(enabled && !zero)); + } else { + if enabled && !zero { + config.insert("includeThoughts".into(), json!(true)); + } + if let Some(budget) = budget.filter(|budget| budget.is_i64() || budget.is_u64()) { + config.insert("thinkingBudget".into(), budget.clone()); + } + } + Value::Object(config) + } +} + +/// `_validate_thinking_config_conflicts`. +fn check_thinking_conflict(params: &Map, param: &str) -> Result<(), Error> { + if params + .get("thinkingConfig") + .is_some_and(|config| config.get("thinkingLevel").is_some()) + { + return Err(unsupported_value(format!( + "Cannot specify both `{param}` (which maps to `thinking_budget`) and `thinking_level` in the same request. For Gemini 3 models, use `thinking_level` instead." + ))); + } + Ok(()) +} + +/// `map_openai_params`, over the OpenAI params in Python's mapping order. +pub fn map_openai_params( + openai: &Map, + surface: GeminiSurface, + model: &str, + context: &ChatRequestContext<'_>, +) -> Result, Error> { + let reasoning = ReasoningModel { + model, + surface, + context, + }; + let mut params = Map::new(); + let server_side = openai.get("include_server_side_tool_invocations") == Some(&json!(true)); + if server_side { + params.insert("include_server_side_tool_invocations".into(), json!(true)); + } + for (name, value) in ordered(openai) { + match (name, value) { + ("temperature" | "top_p" | "seed", value) => { + params.insert(name.into(), value.clone()); + } + ("n", value) => { + params.insert("candidate_count".into(), value.clone()); + } + ("audio", value @ Value::Object(_)) => { + params.insert("speechConfig".into(), map_audio(value)?); + } + ("stop", Value::String(stop)) => { + params.insert("stop_sequences".into(), json!([stop])); + } + ("stop", stop @ Value::Array(_)) => { + params.insert("stop_sequences".into(), stop.clone()); + } + ("max_tokens" | "max_completion_tokens", value) => { + params.insert("max_output_tokens".into(), value.clone()); + } + ("response_format", value @ Value::Object(_)) => { + map_response_format(value, model, &mut params); + } + ("frequency_penalty" | "presence_penalty", value) => { + if supports_penalties(model) { + params.insert(name.into(), value.clone()); + } + } + ("logprobs", value) => { + params.insert("responseLogprobs".into(), value.clone()); + } + ("top_logprobs", value) => { + params.insert("logprobs".into(), value.clone()); + } + ("tools" | "functions", tools @ Value::Array(list)) if !list.is_empty() => { + let mapped = map_function(tools, server_side)?; + add_tools(&mut params, mapped); + } + ("tool_choice", choice @ (Value::String(_) | Value::Object(_))) => { + params.insert("tool_choice".into(), map_tool_choice(choice)?); + } + ("parallel_tool_calls", value) => { + let declared = openai + .get("tools") + .or_else(|| openai.get("functions")) + .and_then(Value::as_array) + .map_or(0, Vec::len); + if !(value == &json!(false) && declared > 1) { + params.insert(name.into(), value.clone()); + } + } + ("reasoning_effort", value) => { + if let Some(config) = reasoning.reasoning_effort(value)? { + check_thinking_conflict(¶ms, "reasoning_effort")?; + params.insert("thinkingConfig".into(), config); + } + } + ("thinking", value) => { + check_thinking_conflict(¶ms, "thinking")?; + params.insert("thinkingConfig".into(), reasoning.thinking(value)); + } + ("modalities", Value::Array(modalities)) => { + params.insert("responseModalities".into(), map_modalities(modalities)); + } + ("web_search_options", Value::Object(_)) => { + add_tools(&mut params, vec![json!({"googleSearch": {}})]); + } + ("service_tier", Value::String(tier)) => { + let tier = match tier.to_lowercase() { + tier if tier == "auto" => "priority".to_owned(), + tier => tier, + }; + params.insert("service_tier".into(), json!(tier)); + } + _ => {} + } + } + if params.contains_key("speechConfig") { + match params.get_mut("responseModalities") { + Some(Value::Array(modalities)) => { + if !modalities.contains(&json!("AUDIO")) { + modalities.push(json!("AUDIO")); + } + } + _ => { + params.insert("responseModalities".into(), json!(["AUDIO"])); + } + } + } + drop_search_tools_mixed_with_functions(&mut params); + Ok(params) +} + +/// `_drop_search_tools_mixed_with_functions`: Vertex AI rejects search tools beside function +/// declarations unless the server side tool invocations are circulated. +fn drop_search_tools_mixed_with_functions(params: &mut Map) { + if params + .get("include_server_side_tool_invocations") + .is_some_and(truthy) + { + return; + } + let Some(Value::Array(tools)) = params.get_mut("tools") else { + return; + }; + let is_search = |tool: &Value| SEARCH_TOOL_KEYS.iter().any(|key| tool.get(*key).is_some()); + let has_functions = tools + .iter() + .any(|tool| tool.get("function_declarations").is_some_and(truthy)); + if has_functions && tools.iter().any(is_search) { + tools.retain(|tool| !is_search(tool)); + } +} + +/// Builds the `generateContent` body for either surface. +pub fn build_request( + surface: GeminiSurface, + model: &str, + messages: Vec, + optional_params: Map, + context: &ChatRequestContext<'_>, +) -> Result { + let candidates = surface.candidates(model); + let sorted = sort_params( + optional_params, + CONFIG_PARAMS, + context, + |openai| supported_params(surface, model, context, openai), + surface.provider(), + model, + )?; + if sorted.openai.contains_key("functions") { + return Err(Error::Unsupported( + "legacy functions, answered as function_call", + )); + } + let mut params = map_openai_params(&sorted.openai, surface, model, context)?; + for name in &sorted.allowed { + if let Some(value) = sorted.openai.get(name) + && !params.contains_key(name) + { + params.insert(name.clone(), value.clone()); + } + } + params.extend(sorted.provider); + + let messages: Vec = messages + .iter() + .map(|message| { + serde_json::to_value(message).map(|mut value| { + if value.get("role").and_then(Value::as_str) == Some("developer") { + value["role"] = json!("system"); + } + value + }) + }) + .collect::>() + .map_err(|error| Error::InvalidRequest(crate::ErrorDetail::invalid("messages", error)))?; + if messages.iter().any(is_cached_message) { + return Err(Error::Unsupported("Gemini context caching")); + } + let supports_system = model.contains("gemini/") + || capability(context, &candidates, "supports_system_messages")? == Some(true); + let (system_instruction, messages) = transform_system_message(supports_system, messages); + if params.contains_key("response_schema") + && capability(context, &candidates, "supports_response_schema")? != Some(true) + { + return Err(Error::Unsupported( + "response_schema for a model without native response schema support", + )); + } + let body = transform_request_body(&messages, system_instruction, model, params, surface)?; + Ok(ProviderChatRequestData { + body, + stream_shape: StreamShape::default(), + headers: Vec::new(), + }) +} + +/// `_served_model_name`. +pub fn served_model_name(body: &Value) -> Option { + body.get("modelVersion") + .and_then(Value::as_str) + .filter(|version| !version.is_empty()) + .map(|version| version.split('@').next().unwrap_or(version).to_owned()) +} + +/// `_extract_candidate_metadata`: the grounding, URL context, safety and citation metadata. +pub fn candidate_metadata(candidate: &Value) -> [Vec; 4] { + let listed = |key: &str| match candidate.get(key) { + Some(Value::Array(items)) => items.clone(), + Some(Value::Null) | None => Vec::new(), + Some(value) => vec![value.clone()], + }; + let safety = candidate + .get("safetyRatings") + .map(|ratings| vec![ratings.clone()]) + .unwrap_or_default(); + let citations = candidate + .get("citationMetadata") + .map(|citation| vec![citation.clone()]) + .unwrap_or_default(); + let url_context = candidate + .get("urlContextMetadata") + .map(|context| vec![context.clone()]) + .unwrap_or_default(); + [listed("groundingMetadata"), url_context, safety, citations] +} + +/// `_convert_grounding_metadata_to_annotations`. +fn grounding_annotations(grounding: &[Value]) -> Vec { + let mut annotations = Vec::new(); + for metadata in grounding { + let chunks = metadata + .get("groundingChunks") + .and_then(Value::as_array) + .cloned() + .unwrap_or_default(); + for support in metadata + .get("groundingSupports") + .and_then(Value::as_array) + .into_iter() + .flatten() + { + let segment = support.get("segment"); + let start = segment + .and_then(|s| s.get("startIndex")) + .filter(|v| !v.is_null()); + let end = segment + .and_then(|s| s.get("endIndex")) + .filter(|v| !v.is_null()); + let first = support + .get("groundingChunkIndices") + .and_then(Value::as_array) + .and_then(|indices| indices.first()) + .and_then(Value::as_u64); + let (Some(start), Some(end), Some(first)) = (start, end, first) else { + continue; + }; + let Some(web) = chunks + .get(first as usize) + .and_then(|chunk| chunk.get("web")) + else { + continue; + }; + let field = |key: &str| web.get(key).cloned().unwrap_or_else(|| json!("")); + annotations.push(json!({"type": "url_citation", "url_citation": { + "start_index": start, "end_index": end, "url": field("uri"), "title": field("title") + }})); + } + } + annotations +} + +/// `_extract_server_side_tool_invocations`: each server side `toolCall` with its +/// `toolResponse`. +fn server_side_tool_invocations(parts: &[Value]) -> Option> { + let mut calls: Vec<(String, Map)> = Vec::new(); + let mut responses: Vec<(String, Map)> = Vec::new(); + let upsert = |entries: &mut Vec<(String, Map)>, id: String, entry| match entries + .iter_mut() + .find(|(seen, _)| *seen == id) + { + Some((_, existing)) => *existing = entry, + None => entries.push((id, entry)), + }; + for part in parts { + let field = |holder: &Value, key: &str| holder.get(key).cloned().unwrap_or(Value::Null); + let id_of = |holder: &Value| { + holder + .get("id") + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned() + }; + if let Some(call) = part.get("toolCall") { + let mut entry = Map::new(); + entry.insert("tool_type".into(), field(call, "toolType")); + entry.insert("id".into(), field(call, "id")); + entry.insert("args".into(), field(call, "args")); + if let Some(signature) = part.get("thoughtSignature").filter(|s| !s.is_null()) { + entry.insert("thought_signature".into(), signature.clone()); + } + upsert(&mut calls, id_of(call), entry); + } else if let Some(response) = part.get("toolResponse") { + let mut entry = Map::new(); + entry.insert("id".into(), field(response, "id")); + entry.insert("tool_type".into(), field(response, "toolType")); + entry.insert("response".into(), field(response, "response")); + if let Some(signature) = part.get("thoughtSignature").filter(|s| !s.is_null()) { + entry.insert("thought_signature".into(), signature.clone()); + } + upsert(&mut responses, id_of(response), entry); + } + } + let mut invocations = Vec::new(); + for (id, mut call) in calls { + if let Some(at) = responses.iter().position(|(seen, _)| *seen == id) { + let (_, response) = responses.remove(at); + call.insert( + "response".into(), + response.get("response").cloned().unwrap_or(Value::Null), + ); + if let Some(signature) = response.get("thought_signature") { + call.insert("response_thought_signature".into(), signature.clone()); + } + } + invocations.push(Value::Object(call)); + } + for (_, mut response) in responses { + if let Some(signature) = response.get("thought_signature").cloned() { + response.insert("response_thought_signature".into(), signature); + } + invocations.push(Value::Object(response)); + } + (!invocations.is_empty()).then_some(invocations) +} + +/// `_transform_logprobs`. +fn transform_logprobs(result: &Value) -> Option { + let chosen = result.get("chosenCandidates")?.as_array()?; + let top = result.get("topCandidates").and_then(Value::as_array); + let field = |holder: &Value, key: &str| holder.get(key).cloned().unwrap_or(Value::Null); + let content: Vec = chosen + .iter() + .enumerate() + .map(|(index, candidate)| { + let top_logprobs: Vec = top + .and_then(|top| top.get(index)) + .and_then(|top| top.get("candidates")) + .and_then(Value::as_array) + .into_iter() + .flatten() + .map(|option| json!({"token": field(option, "token"), "logprob": field(option, "logProbability")})) + .collect(); + json!({"token": field(candidate, "token"), "logprob": field(candidate, "logProbability"), "top_logprobs": top_logprobs}) + }) + .collect(); + Some(json!({"content": content})) +} + +/// What one candidate says, as `_process_candidates` reads it. +#[derive(Debug, Default)] +pub struct CandidateMessage { + pub content: Option, + pub reasoning_content: Option, + pub thinking_blocks: Option>, + pub tool_calls: Vec, + pub provider_specific_fields: Option>, + pub annotations: Option>, + pub logprobs: Option, + /// Gemini's own `finishReason`. + pub finish_reason: Option, +} + +impl CandidateMessage { + /// `_check_finish_reason`. + pub fn openai_finish_reason(&self) -> String { + match (&self.finish_reason, self.tool_calls.is_empty()) { + (_, false) => "tool_calls".to_owned(), + (Some(reason), true) => finish_reason_for(reason).to_owned(), + (None, true) => "stop".to_owned(), + } + } +} + +fn is_audio_data_url(text: &str) -> bool { + text.starts_with("data:audio") + && text + .split_once("data:") + .and_then(|(_, rest)| rest.split_once(";base64,")) + .is_some_and(|(media_type, _)| media_type.starts_with("audio/")) +} + +/// One candidate's message. `next_tool_index` counts tool calls across a stream's chunks. +pub fn candidate_message(candidate: &Value, next_tool_index: &mut i64) -> CandidateMessage { + let mut message = CandidateMessage { + finish_reason: candidate + .get("finishReason") + .and_then(Value::as_str) + .map(str::to_owned), + logprobs: candidate.get("logprobsResult").and_then(transform_logprobs), + ..CandidateMessage::default() + }; + let parts = candidate + .get("content") + .filter(|content| truthy(content)) + .and_then(|content| content.get("parts")) + .and_then(Value::as_array); + let Some(parts) = parts else { + return message; + }; + let mut content: Option = None; + let mut reasoning: Option = None; + let mut thinking_blocks = Vec::new(); + let mut signatures = Vec::new(); + for part in parts { + let text = match (part.get("text"), part.get("inlineData")) { + (Some(Value::String(text)), _) if is_audio_data_url(text) => String::new(), + (Some(Value::String(text)), _) => text.clone(), + (Some(_), _) => String::new(), + (None, Some(inline)) => { + let mime = inline + .get("mimeType") + .and_then(Value::as_str) + .unwrap_or_default(); + let data = inline + .get("data") + .and_then(Value::as_str) + .unwrap_or_default(); + match mime.starts_with("audio/") || mime.starts_with("image/") { + true => String::new(), + false => format!("data:{mime};base64,{data}"), + } + } + (None, None) => String::new(), + }; + let thought = part.get("thought") == Some(&json!(true)); + if !text.is_empty() { + let target = match thought { + true => &mut reasoning, + false => &mut content, + }; + target.get_or_insert_with(String::new).push_str(&text); + } + let signature = part.get("thoughtSignature").filter(|s| !s.is_null()); + if thought { + thinking_blocks.push(ChatCompletionThinkingBlock::Thinking { + thinking: Some( + part.get("text") + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + ), + signature: signature.and_then(Value::as_str).map(str::to_owned), + cache_control: None, + index: None, + }); + } + if let Some(signature) = signature { + signatures.push(signature.clone()); + } + if let Some(call) = part.get("functionCall") { + let arguments = litellm_python_compat::json::dumps_unicode( + &litellm_python_compat::json::from_json( + call.get("args").cloned().unwrap_or(Value::Null), + ), + ) + .unwrap_or_default(); + let mut id = call + .get("id") + .and_then(Value::as_str) + .filter(|id| !id.is_empty()) + .map_or_else( + || format!("call_{}", &uuid::Uuid::new_v4().simple().to_string()[..28]), + str::to_owned, + ); + let signature = signature.and_then(Value::as_str); + if let Some(signature) = signature { + id = format!("{id}{THOUGHT_SIGNATURE_SEPARATOR}{signature}"); + } + message.tool_calls.push(ChatCompletionToolCallChunk { + id: Some(id), + tool_type: "function".into(), + function: ChatCompletionToolCallFunctionChunk { + name: call.get("name").and_then(Value::as_str).map(str::to_owned), + arguments, + provider_specific_fields: None, + }, + index: *next_tool_index, + provider_specific_fields: signature.map(|signature| { + Map::from_iter([("thought_signature".to_owned(), json!(signature))]) + }), + }); + *next_tool_index += 1; + } + } + message.content = content; + message.reasoning_content = reasoning; + message.thinking_blocks = Some(thinking_blocks); + let grounding = &candidate_metadata(candidate)[0]; + if !grounding.is_empty() { + let annotations = grounding_annotations(grounding); + message.annotations = (!annotations.is_empty()).then_some(annotations); + } + let mut fields = Map::new(); + if !signatures.is_empty() { + fields.insert("thought_signatures".into(), Value::Array(signatures)); + } + if let Some(invocations) = server_side_tool_invocations(parts) { + fields.insert( + "server_side_tool_invocations".into(), + Value::Array(invocations), + ); + } + message.provider_specific_fields = (!fields.is_empty()).then_some(fields); + message +} + +fn usage_count(usage: &Value, key: &str) -> u64 { + usage.get(key).and_then(Value::as_u64).unwrap_or(0) +} + +fn modality_counts(usage: &Value, key: &str) -> Option> { + let details = usage.get(key)?.as_array()?; + let mut counts = BTreeMap::new(); + for detail in details { + let modality = detail + .get("modality") + .and_then(Value::as_str) + .unwrap_or_default() + .to_uppercase(); + let tokens = detail + .get("tokenCount") + .or_else(|| detail.get("token_count")) + .and_then(Value::as_u64) + .unwrap_or(0); + let slot = match modality.as_str() { + "TEXT" | "DOCUMENT" => "text", + "AUDIO" => "audio", + "IMAGE" => "image", + "VIDEO" => "video", + _ => continue, + }; + *counts.entry(slot).or_insert(0) += tokens; + } + Some(counts) +} + +/// `calculate_grounding_requests(...).has_billable_grounding()` over every candidate. +fn has_search_grounding(body: &Value) -> bool { + let items: Vec = body + .get("candidates") + .and_then(Value::as_array) + .into_iter() + .flatten() + .flat_map(|candidate| candidate_metadata(candidate)[0].clone()) + .collect(); + items.iter().any(|item| { + let queries = item + .get("webSearchQueries") + .and_then(Value::as_array) + .is_some_and(|queries| { + queries + .iter() + .any(|query| query.as_str().is_some_and(|q| !q.is_empty())) + }); + let maps = item + .get("groundingChunks") + .and_then(Value::as_array) + .is_some_and(|chunks| chunks.iter().any(|chunk| chunk.get("maps").is_some())) + || item.get("googleMapsWidgetContextToken").is_some_and(truthy); + queries || maps + }) +} + +/// `_calculate_usage`: cached tokens are part of `promptTokenCount`, thinking tokens are added +/// to the candidates when Gemini counted them apart, and tool use prompt tokens are billed +/// unless a search grounding (billed per query) produced them. +pub fn calculate_usage(body: &Value) -> Result { + let usage = body.get("usageMetadata").ok_or_else(|| { + Error::InvalidResponse( + format!("usageMetadata not found in completion_response. Got={body}").into(), + ) + })?; + let cached = usage.get("cachedContentTokenCount").and_then(Value::as_u64); + let candidates = usage_count(usage, "candidatesTokenCount"); + let completion_details = modality_counts(usage, "candidatesTokensDetails"); + let explicit_text = completion_details + .as_ref() + .and_then(|counts| counts.get("text").copied()); + let other_completion: u64 = completion_details + .as_ref() + .map(|counts| { + ["image", "audio", "video"] + .iter() + .filter_map(|slot| counts.get(slot)) + .sum() + }) + .unwrap_or(0); + let completion_text = explicit_text + .or_else(|| (candidates > 0).then(|| candidates.saturating_sub(other_completion))); + let reasoning = usage.get("thoughtsTokenCount").and_then(Value::as_u64); + + let prompt_details = modality_counts(usage, "promptTokensDetails"); + let cache_details = modality_counts(usage, "cacheTokensDetails"); + let mut prompt_text = prompt_details + .as_ref() + .and_then(|counts| counts.get("text").copied()); + match ( + cache_details + .as_ref() + .and_then(|counts| counts.get("text").copied()), + cached, + ) { + (Some(cached_text), _) => prompt_text = prompt_text.map(|t| t.saturating_sub(cached_text)), + (None, Some(cached)) if cache_details.is_none() => { + prompt_text = prompt_text.map(|t| t.saturating_sub(cached)); + } + _ => {} + } + let tool_use = usage_count(usage, "toolUsePromptTokenCount"); + let billable_tool_use = match has_search_grounding(body) { + true => 0, + false => tool_use, + }; + let prompt = usage_count(usage, "promptTokenCount"); + let total = usage_count(usage, "totalTokenCount"); + let inclusive = prompt + candidates + tool_use == total; + let completion = match (inclusive, reasoning) { + (false, Some(reasoning)) if reasoning > 0 => candidates + reasoning, + _ => candidates, + }; + let completion_tokens_details = + (completion_text.is_some() || reasoning.is_some()).then_some(CompletionTokensDetails { + reasoning_tokens: reasoning, + text_tokens: completion_text, + }); + Ok(ChatCompletionsUsage { + prompt_tokens: prompt + billable_tool_use, + completion_tokens: completion, + total_tokens: total, + prompt_tokens_details: PromptTokensDetails { + cached_tokens: cached.unwrap_or(0), + text_tokens: prompt_text.map_or(0, |text| text + billable_tool_use), + ..PromptTokensDetails::default() + }, + completion_tokens_details, + cache_read_input_tokens: cached, + ..ChatCompletionsUsage::default() + }) +} + +/// The answer Python gives when the prompt or the first candidate was blocked. +fn blocked_response(model: String, body: &Value) -> ChatCompletionsResponse { + let usage = body.get("usageMetadata").cloned().unwrap_or(Value::Null); + let prompt = usage_count(&usage, "promptTokenCount"); + let completion = usage_count(&usage, "candidatesTokenCount"); + ChatCompletionsResponse { + created: unix_now(), + model, + choices: vec![ChatCompletionsChoice { + index: 0, + message: ChatCompletionsChoiceMessage { + role: "assistant".into(), + ..ChatCompletionsChoiceMessage::default() + }, + finish_reason: "content_filter".into(), + logprobs: None, + }], + usage: ChatCompletionsUsage { + prompt_tokens: prompt, + completion_tokens: completion, + total_tokens: usage_count(&usage, "totalTokenCount"), + ..ChatCompletionsUsage::default() + }, + extra: Map::new(), + } +} + +/// Whether the prompt itself was blocked. +pub fn prompt_blocked(body: &Value) -> bool { + body.get("promptFeedback") + .is_some_and(|feedback| feedback.get("blockReason").is_some()) +} + +/// `_transform_google_generate_content_to_openai_model_response`. +pub fn transform_generate_content_response( + model: &str, + body: Value, +) -> Result { + if !body.is_object() { + return Err(Error::InvalidResponse( + "Gemini response is not an object".into(), + )); + } + let model = served_model_name(&body).unwrap_or_else(|| model.to_owned()); + if prompt_blocked(&body) { + return Ok(blocked_response(model, &body)); + } + let candidates = body + .get("candidates") + .and_then(Value::as_array) + .cloned() + .unwrap_or_default(); + if candidates + .first() + .and_then(|first| first.get("finishReason")) + .and_then(Value::as_str) + .is_some_and(|reason| FLAGGED_FINISH_REASONS.contains(&reason)) + { + return Ok(blocked_response(model, &body)); + } + let mut metadata: [Vec; 4] = Default::default(); + let mut choices = Vec::new(); + let mut tool_index = 0; + for (position, candidate) in candidates.iter().enumerate() { + for (all, found) in metadata.iter_mut().zip(candidate_metadata(candidate)) { + all.extend(found); + } + if candidate.get("content").is_none() && candidate.get("finishReason").is_none() { + continue; + } + let message = candidate_message(candidate, &mut tool_index); + let finish_reason = message.openai_finish_reason(); + choices.push(ChatCompletionsChoice { + index: candidate + .get("index") + .and_then(Value::as_u64) + .unwrap_or(position as u64), + message: ChatCompletionsChoiceMessage { + role: "assistant".into(), + content: message.content, + tool_calls: (!message.tool_calls.is_empty()).then_some(message.tool_calls), + reasoning_content: message.reasoning_content, + thinking_blocks: message.thinking_blocks, + provider_specific_fields: message.provider_specific_fields, + annotations: message.annotations, + }, + finish_reason, + logprobs: message.logprobs, + }); + } + let usage = calculate_usage(&body)?; + let [grounding, url_context, safety, citations] = metadata; + let extra = Map::from_iter([ + ( + "vertex_ai_grounding_metadata".to_owned(), + Value::Array(grounding), + ), + ( + "vertex_ai_url_context_metadata".to_owned(), + Value::Array(url_context), + ), + ("vertex_ai_safety_results".to_owned(), Value::Array(safety)), + ( + "vertex_ai_citation_metadata".to_owned(), + Value::Array(citations), + ), + ]); + Ok(ChatCompletionsResponse { + created: unix_now(), + model, + choices, + usage, + extra, + }) +} + +/// The same call as a stream: `streamGenerateContent` with `alt=sse`. +pub fn streaming_url(url: &str) -> String { + let method = format!(":{GENERATE_CONTENT}"); + let url = match url.rfind(&method) { + Some(at) => format!( + "{}:{STREAM_GENERATE_CONTENT}{}", + &url[..at], + &url[at + method.len()..] + ), + None => url.to_owned(), + }; + with_query(&url, "alt=sse") +} + +/// `VertexGeminiConfig`: Gemini on Vertex AI, addressed to a project and location and +/// authenticated with a Google access token. +pub struct VertexGeminiConfig; + +pub const VERTEX_GEMINI_CHAT_COMPLETIONS_CONFIG: VertexGeminiConfig = VertexGeminiConfig; + +/// `get_model_for_vertex_ai_url`: a `gemini/` model calls the id. +fn model_for_vertex_url(model: &str) -> &str { + match model.contains("gemini/") { + true => model.rsplit('/').next().unwrap_or(model), + false => model, + } +} + +fn vertex_config(optional_params: &Map) -> Result { + Ok(VertexConfig::from_sourced_optional_params( + optional_params, + &BTreeMap::new(), + )?) +} + +impl BaseConfig for VertexGeminiConfig { + fn secret_names(&self) -> Vec<&'static str> { + litellm_auth_gcp::secret_names() + .iter() + .copied() + .chain([VERTEXAI_API_BASE_ENV]) + .collect() + } + + /// `get_vertex_url` with `_check_custom_proxy`. + fn get_complete_url( + &self, + api_base: Option<&str>, + model: &str, + optional_params: &Map, + env_lookup: &dyn Fn(&str) -> Option, + ) -> Result { + let config = vertex_config(optional_params)?; + let project = get_vertex_ai_project(&config, env_lookup) + .or_else(|| get_vertex_ai_project_from_credentials(&config, env_lookup)) + .ok_or_else(|| Error::Auth(VertexParams::PROJECT.missing("Vertex AI")))?; + let location = get_vertex_ai_location(&config, env_lookup) + .unwrap_or_else(|| DEFAULT_LOCATION.to_owned()); + let model = model_for_vertex_url(model); + let target = match model.chars().all(|c| c.is_ascii_digit()) { + true => format!("endpoints/{model}"), + false => format!("publishers/google/models/{model}"), + }; + let default = format!( + "{}/v1/projects/{project}/locations/{location}/{target}:{GENERATE_CONTENT}", + get_vertex_base_url(&location)? + ); + let api_base = api_base + .map(str::to_owned) + .or_else(|| env_lookup(VERTEXAI_API_BASE_ENV)) + .filter(|base| !base.trim().is_empty()); + Ok(graft_custom_vertex_base( + api_base.as_deref(), + &default, + GENERATE_CONTENT, + )) + } + + fn transform_request( + &self, + model: &str, + messages: Vec, + optional_params: Map, + context: &ChatRequestContext<'_>, + ) -> Result { + build_request( + GeminiSurface::Vertex, + model, + messages, + optional_params, + context, + ) + } + + fn transform_response( + &self, + model: &str, + response: ProviderChatResponseData, + _shape: &StreamShape, + ) -> Result { + transform_generate_content_response(model, response.body) + } + + fn remote_media(&self, messages: &[ChatMessage]) -> Vec { + remote_media(messages, GeminiSurface::Vertex) + } + + fn model_response_iterator(&self, _shape: StreamShape) -> Option { + Some(gemini_chat_stream()) + } + + fn streaming_request(&self, url: String, body: Value) -> Result<(String, Value), Error> { + Ok((streaming_url(&url), body)) + } + + /// A forwarded bearer is sent as is; otherwise a Google access token for the configured + /// credentials is acquired when the request is sent. Vertex AI ignores `api_key`. + fn validate_environment( + &self, + headers: Headers, + _api_key: Option<&str>, + _model: &str, + optional_params: &Map, + _env_lookup: &dyn Fn(&str) -> Option, + ) -> Result { + if has_header(&headers, "authorization") { + return Ok(ValidatedEnvironment { + headers, + auth: AuthScheme::Forwarded, + }); + } + Ok(ValidatedEnvironment { + headers, + auth: AuthScheme::GcpAccessToken { + config: Box::new(vertex_config(optional_params)?), + }, + }) + } + + fn config_params(&self) -> &'static [&'static str] { + CONFIG_PARAMS + } + + /// Every request is translated; a param or message Python refuses is refused here too. + fn unsupported_reason( + &self, + _messages: &[ChatMessage], + _optional_params: &Map, + ) -> Option { + None + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use rstest::rstest; + + use super::*; + use crate::base_llm::model_info::ModelInfo; + + struct CostMap(HashMap>); + + impl ModelInfo for CostMap { + fn entry(&self, model: &str) -> Option>> { + self.0.get(model).map(std::borrow::Cow::Borrowed) + } + + fn declares(&self, key: &str) -> bool { + self.0.values().any(|entry| entry.contains_key(key)) + } + } + + fn cost_map() -> CostMap { + let entry = json!({ + "supports_system_messages": true, + "supports_reasoning": true, + "supports_response_schema": true + }); + let Value::Object(entry) = entry else { + unreachable!() + }; + CostMap(HashMap::from([ + ("gemini-2.5-flash".to_owned(), entry.clone()), + ("gemini/gemini-2.5-flash".to_owned(), entry), + ])) + } + + fn object(value: Value) -> Map { + let Value::Object(map) = value else { + panic!("expected an object"); + }; + map + } + + fn request( + surface: GeminiSurface, + model: &str, + messages: Value, + params: Value, + drop_params: bool, + ) -> Result { + let map = cost_map(); + let context = ChatRequestContext { + drop_params: Some(drop_params), + model_info: Some(&map), + }; + build_request( + surface, + model, + serde_json::from_value(messages).unwrap(), + object(params), + &context, + ) + .map(|request| request.body) + } + + fn hi() -> Value { + json!([{"role": "system", "content": "be brief"}, {"role": "user", "content": "hi"}]) + } + + #[rstest] + #[case::gemini_1("gemini-1.5-pro", false)] + #[case::gemini_2("gemini-2.0-flash-001", false)] + #[case::gemini_25("vertex_ai/gemini-2.5-flash", false)] + #[case::exp("gemini-exp-1206", false)] + #[case::pro("gemini-pro", false)] + #[case::flash_latest("gemini-flash-latest", true)] + #[case::flash_lite_latest("gemini-flash-lite-latest", true)] + #[case::gemini_3("gemini-3-pro-preview", true)] + #[case::gemini_31("gemini-3.1-flash", true)] + #[case::gemini_10("gemini-10-ultra", true)] + #[case::fine_tuned("1234567890", false)] + #[case::ai_studio_tuned("gemini/tunedModels/abc", false)] + #[case::gemma("gemma-3-27b-it", false)] + fn gemini_3_or_newer_follows_pythons_pattern(#[case] model: &str, #[case] expected: bool) { + assert_eq!(is_gemini_3_or_newer(model), expected); + } + + #[rstest] + #[case::gemini_15("gemini-1.5-pro", false)] + #[case::gemini_20("gemini-2.0-flash", true)] + #[case::gemini_3("gemini-3-pro", true)] + #[case::gemini_10("gemini-10.0-x", true)] + #[case::pro("gemini-pro", false)] + fn json_schema_support_starts_at_gemini_2(#[case] model: &str, #[case] expected: bool) { + assert_eq!(supports_response_json_schema(model), expected); + } + + #[rstest] + #[case::gemini("gemini-2.5-flash", true)] + #[case::tuned("gemini/123", true)] + #[case::claude("claude-sonnet-4", false)] + #[case::gemma("gemma/gemma-3", false)] + #[case::openai("openai/gemini-like", false)] + #[case::partner_gemma("google/gemma-3-gemini", false)] + #[case::legacy("chat-bison", false)] + fn vertex_routes_only_gemini_models_here(#[case] model: &str, #[case] expected: bool) { + assert_eq!(is_vertex_gemini_model(model), expected); + } + + #[rstest] + fn builds_the_body_python_builds() { + let body = request( + GeminiSurface::Vertex, + "gemini-2.5-flash", + hi(), + json!({"temperature": 0.3, "max_tokens": 50, "stop": "END", "n": 2, "seed": 7, + "top_k": 4, "frequency_penalty": 0.1, "logprobs": true, "top_logprobs": 2}), + false, + ) + .unwrap(); + assert_eq!( + body, + json!({ + "contents": [{"role": "user", "parts": [{"text": "hi"}]}], + "system_instruction": {"parts": [{"text": "be brief"}]}, + "generationConfig": {"temperature": 0.3, "candidate_count": 2, "stop_sequences": ["END"], + "max_output_tokens": 50, "frequency_penalty": 0.1, "seed": 7, "responseLogprobs": true, + "logprobs": 2, "top_k": 4} + }) + ); + } + + #[rstest] + #[case::refused(false, true)] + #[case::dropped(true, false)] + fn gemini_3_takes_no_penalties_on_vertex(#[case] drop_params: bool, #[case] refused: bool) { + let result = request( + GeminiSurface::Vertex, + "gemini-3-pro-preview", + hi(), + json!({"presence_penalty": 0.5}), + drop_params, + ); + assert_eq!(result.is_err(), refused); + } + + #[rstest] + #[case::none(json!("none"), json!({"functionCallingConfig": {"mode": "NONE"}}))] + #[case::required(json!("required"), json!({"functionCallingConfig": {"mode": "ANY"}}))] + #[case::auto(json!("auto"), json!({"functionCallingConfig": {"mode": "AUTO"}}))] + #[case::named(json!({"type": "function", "function": {"name": "f"}}), json!({"functionCallingConfig": {"mode": "ANY", "allowed_function_names": ["f"]}}))] + fn tool_choice_becomes_a_tool_config(#[case] choice: Value, #[case] expected: Value) { + let body = request( + GeminiSurface::AiStudio, + "gemini-2.5-flash", + hi(), + json!({"tools": [{"type": "function", "function": {"name": "f", "parameters": {"type": "object", "properties": {"x": {"type": "string"}}, "additionalProperties": false}, "strict": true}}], "tool_choice": choice}), + false, + ) + .unwrap(); + assert_eq!(body["toolConfig"], expected); + assert_eq!( + body["tools"], + json!([{"function_declarations": [{"name": "f", "parameters": {"type": "object", "properties": {"x": {"type": "string"}}}}]}]) + ); + } + + #[rstest] + fn an_unknown_tool_choice_is_refused_like_python() { + assert!(matches!( + request( + GeminiSurface::Vertex, + "gemini-2.5-flash", + hi(), + json!({"tools": [{"type": "function", "function": {"name": "f"}}], "tool_choice": "any"}), + true, + ), + Err(Error::InvalidRequest(_)) + )); + } + + #[rstest] + #[case::functions_drop_search(json!([{"type": "function", "function": {"name": "f"}}, {"googleSearch": {}}]), false, json!([{"function_declarations": [{"name": "f"}]}]))] + #[case::circulated_keeps_search(json!([{"type": "function", "function": {"name": "f"}}, {"googleSearch": {}}]), true, json!([{"function_declarations": [{"name": "f"}]}, {"googleSearch": {}}]))] + #[case::built_ins(json!([{"code_execution": {}}, {"urlContext": {}}, {"type": "web_search"}]), false, json!([{"googleSearch": {}}, {"code_execution": {}}, {"url_context": {}}]))] + #[case::computer_use(json!([{"type": "computer_use", "environment": "browser"}]), false, json!([{"computerUse": {"environment": "ENVIRONMENT_BROWSER"}}]))] + fn built_in_tools_map_like_python( + #[case] tools: Value, + #[case] circulate: bool, + #[case] expected: Value, + ) { + let mut params = json!({"tools": tools}); + if circulate { + params["include_server_side_tool_invocations"] = json!(true); + } + let body = request( + GeminiSurface::Vertex, + "gemini-3-pro-preview", + hi(), + params, + false, + ) + .unwrap(); + assert_eq!(body["tools"], expected); + } + + #[rstest] + fn web_search_options_add_google_search() { + let body = request( + GeminiSurface::Vertex, + "gemini-2.5-flash", + hi(), + json!({"web_search_options": {}}), + false, + ) + .unwrap(); + assert_eq!(body["tools"], json!([{"googleSearch": {}}])); + } + + #[rstest] + #[case::budget_low("gemini-2.5-flash", json!("low"), json!({"thinkingBudget": 1024, "includeThoughts": true}))] + #[case::budget_minimal_flash("gemini-2.5-flash", json!("minimal"), json!({"thinkingBudget": 1, "includeThoughts": true}))] + #[case::budget_disable("gemini-2.5-flash", json!("disable"), json!({"thinkingBudget": 0, "includeThoughts": false}))] + #[case::level_high("gemini-3-pro-preview", json!("high"), json!({"thinkingLevel": "high", "includeThoughts": true}))] + #[case::level_minimal_pro("gemini-3-pro-preview", json!("minimal"), json!({"thinkingLevel": "low", "includeThoughts": true}))] + #[case::level_minimal_flash("gemini-3-flash-preview", json!("minimal"), json!({"thinkingLevel": "minimal", "includeThoughts": true}))] + #[case::level_none("gemini-3-flash-preview", json!("none"), json!({"thinkingLevel": "minimal", "includeThoughts": false}))] + #[case::summary_dict("gemini-2.5-flash", json!({"effort": "high", "summary": "auto"}), json!({"thinkingBudget": 4096, "includeThoughts": true}))] + fn reasoning_effort_becomes_a_thinking_config( + #[case] model: &str, + #[case] effort: Value, + #[case] expected: Value, + ) { + let body = request( + GeminiSurface::Vertex, + model, + hi(), + json!({"reasoning_effort": effort}), + false, + ) + .unwrap(); + assert_eq!(body["generationConfig"]["thinkingConfig"], expected); + } + + #[rstest] + fn an_invalid_reasoning_effort_is_refused() { + assert!(matches!( + request( + GeminiSurface::Vertex, + "gemini-2.5-flash", + hi(), + json!({"reasoning_effort": "extreme"}), + true + ), + Err(Error::InvalidRequest(_)) + )); + } + + #[rstest] + #[case::budget("gemini-2.5-flash", json!({"type": "enabled", "budget_tokens": 2000}), json!({"includeThoughts": true, "thinkingBudget": 2000}))] + #[case::disabled_budget_zero("gemini-2.5-flash", json!({"type": "enabled", "budget_tokens": 0}), json!({"thinkingBudget": 0}))] + #[case::gemini_3("gemini-3-pro-preview", json!({"type": "enabled", "budget_tokens": 2000}), json!({"includeThoughts": true}))] + #[case::gemini_3_disabled("gemini-3-pro-preview", json!({"type": "disabled"}), json!({"includeThoughts": false}))] + fn thinking_becomes_a_thinking_config( + #[case] model: &str, + #[case] thinking: Value, + #[case] expected: Value, + ) { + let body = request( + GeminiSurface::Vertex, + model, + hi(), + json!({"thinking": thinking}), + false, + ) + .unwrap(); + assert_eq!(body["generationConfig"]["thinkingConfig"], expected); + } + + #[rstest] + #[case::json_schema_on_gemini_2("gemini-2.5-flash", + json!({"type": "json_schema", "json_schema": {"name": "x", "strict": true, "schema": {"type": "object", "properties": {"a": {"type": "string"}}, "additionalProperties": false}}}), + json!({"response_mime_type": "application/json", "response_json_schema": {"type": "object", "properties": {"a": {"type": "string"}}, "additionalProperties": false}}))] + #[case::json_object("gemini-2.5-flash", json!({"type": "json_object"}), json!({"response_mime_type": "application/json"}))] + #[case::text("gemini-2.5-flash", json!({"type": "text"}), json!({"response_mime_type": "text/plain"}))] + fn response_format_sets_the_output_type( + #[case] model: &str, + #[case] format: Value, + #[case] expected: Value, + ) { + let body = request( + GeminiSurface::Vertex, + model, + hi(), + json!({"response_format": format}), + false, + ) + .unwrap(); + assert_eq!(body["generationConfig"], expected); + } + + #[rstest] + fn audio_maps_to_a_speech_config() { + let body = request( + GeminiSurface::Vertex, + "gemini-2.5-flash-preview-tts", + hi(), + json!({"audio": {"voice": "alloy", "format": "pcm16"}, "modalities": ["text"]}), + false, + ) + .unwrap(); + assert_eq!( + body["generationConfig"], + json!({"speechConfig": {"voiceConfig": {"prebuiltVoiceConfig": {"voiceName": "Kore"}}}, "responseModalities": ["TEXT", "AUDIO"]}) + ); + } + + #[rstest] + #[case::cache_control(json!([{"role": "user", "content": [{"type": "text", "text": "x", "cache_control": {"type": "ephemeral"}}]}]), json!({}))] + #[case::functions(json!([{"role": "user", "content": "x"}]), json!({"functions": [{"name": "f"}]}))] + fn what_python_answers_differently_is_left_to_it( + #[case] messages: Value, + #[case] params: Value, + ) { + assert!(matches!( + request( + GeminiSurface::Vertex, + "gemini-2.5-flash", + messages, + params, + false + ), + Err(Error::Unsupported(_)) + )); + } + + #[rstest] + #[case::unknown_model_merges_system("gemini-unknown", json!([{"role": "user", "parts": [{"text": "be brief"}, {"text": "hi"}]}]))] + fn a_model_without_system_support_gets_the_system_text_as_user( + #[case] model: &str, + #[case] contents: Value, + ) { + let body = request(GeminiSurface::Vertex, model, hi(), json!({}), false).unwrap(); + assert_eq!(body["contents"], contents); + assert!(body.get("system_instruction").is_none()); + } + + fn env(_: &str) -> Option { + None + } + + #[rstest] + #[case::regional( + None, + "gemini-2.5-flash", + "https://europe-west4-aiplatform.googleapis.com/v1/projects/p/locations/europe-west4/publishers/google/models/gemini-2.5-flash:generateContent" + )] + #[case::tuned( + None, + "gemini/987", + "https://europe-west4-aiplatform.googleapis.com/v1/projects/p/locations/europe-west4/endpoints/987:generateContent" + )] + #[case::custom_host( + Some("http://127.0.0.1:9"), + "gemini-2.5-flash", + "http://127.0.0.1:9/v1/projects/p/locations/europe-west4/publishers/google/models/gemini-2.5-flash:generateContent" + )] + fn vertex_urls_name_the_project_and_location( + #[case] api_base: Option<&str>, + #[case] model: &str, + #[case] expected: &str, + ) { + let params = object(json!({"vertex_project": "p", "vertex_location": "europe-west4"})); + assert_eq!( + VERTEX_GEMINI_CHAT_COMPLETIONS_CONFIG + .get_complete_url(api_base, model, ¶ms, &env) + .unwrap(), + expected + ); + } + + #[rstest] + #[case::default( + "https://h/v1/projects/p/x:generateContent", + "https://h/v1/projects/p/x:streamGenerateContent?alt=sse" + )] + #[case::custom_query( + "https://h/x:generateContent?k=1", + "https://h/x:streamGenerateContent?k=1&alt=sse" + )] + fn streaming_swaps_the_method_and_asks_for_sse(#[case] url: &str, #[case] expected: &str) { + assert_eq!(streaming_url(url), expected); + } + + #[rstest] + #[case::forwarded(vec![("Authorization".to_owned(), "Bearer t".to_owned())], true)] + #[case::token(Vec::new(), false)] + fn vertex_takes_a_forwarded_bearer_or_a_google_token( + #[case] headers: Headers, + #[case] forwarded: bool, + ) { + let validated = VERTEX_GEMINI_CHAT_COMPLETIONS_CONFIG + .validate_environment( + headers, + Some("ignored"), + "gemini-2.5-flash", + &Map::new(), + &env, + ) + .unwrap(); + assert_eq!(matches!(validated.auth, AuthScheme::Forwarded), forwarded); + assert_eq!( + matches!(validated.auth, AuthScheme::GcpAccessToken { .. }), + !forwarded + ); + } + + fn response(body: Value) -> Value { + serde_json::to_value(transform_generate_content_response("gemini-2.5-flash", body).unwrap()) + .unwrap() + } + + #[rstest] + fn text_reasoning_and_usage_come_back_in_openai_shape() { + let answer = response(json!({ + "candidates": [{"content": {"role": "model", "parts": [ + {"text": "thinking", "thought": true, "thoughtSignature": "sig"}, + {"text": "Hello"} + ]}, "finishReason": "STOP"}], + "usageMetadata": {"promptTokenCount": 10, "candidatesTokenCount": 5, "thoughtsTokenCount": 3, + "totalTokenCount": 18, "cachedContentTokenCount": 4, + "promptTokensDetails": [{"modality": "TEXT", "tokenCount": 10}]}, + "modelVersion": "gemini-2.5-flash@001" + })); + assert_eq!(answer["model"], "gemini-2.5-flash"); + assert_eq!( + answer["choices"], + json!([{"index": 0, "finish_reason": "stop", "message": { + "role": "assistant", "content": "Hello", "reasoning_content": "thinking", + "thinking_blocks": [{"type": "thinking", "thinking": "thinking", "signature": "sig"}], + "provider_specific_fields": {"thought_signatures": ["sig"]} + }}]) + ); + assert_eq!( + answer["usage"], + json!({"prompt_tokens": 10, "completion_tokens": 8, "total_tokens": 18, + "prompt_tokens_details": {"cached_tokens": 4, "cache_creation_tokens": 0, "text_tokens": 6}, + "completion_tokens_details": {"reasoning_tokens": 3, "text_tokens": 5}, + "cache_read_input_tokens": 4}) + ); + } + + #[rstest] + fn function_calls_become_tool_calls_with_their_signature() { + let answer = response(json!({ + "candidates": [{"content": {"parts": [ + {"functionCall": {"name": "get_weather", "args": {"city": "Zürich"}, "id": "fc_1"}, "thoughtSignature": "s"} + ]}, "finishReason": "STOP"}], + "usageMetadata": {"promptTokenCount": 1, "candidatesTokenCount": 1, "totalTokenCount": 2} + })); + assert_eq!(answer["choices"][0]["finish_reason"], "tool_calls"); + assert_eq!( + answer["choices"][0]["message"]["tool_calls"], + json!([{"id": "fc_1__thought__s", "type": "function", "index": 0, + "function": {"name": "get_weather", "arguments": "{\"city\": \"Zürich\"}"}, + "provider_specific_fields": {"thought_signature": "s"}}]) + ); + } + + #[rstest] + #[case::prompt_blocked(json!({"promptFeedback": {"blockReason": "SAFETY"}, "usageMetadata": {"promptTokenCount": 3, "totalTokenCount": 3}}))] + #[case::candidate_flagged(json!({"candidates": [{"finishReason": "SAFETY"}], "usageMetadata": {"promptTokenCount": 3, "totalTokenCount": 3}}))] + fn a_blocked_answer_is_a_content_filter(#[case] body: Value) { + let answer = response(body); + assert_eq!(answer["choices"][0]["finish_reason"], "content_filter"); + assert_eq!(answer["choices"][0]["message"]["content"], Value::Null); + assert_eq!(answer["usage"]["prompt_tokens"], 3); + } + + #[rstest] + #[case::max_tokens("MAX_TOKENS", "length")] + #[case::malformed("MALFORMED_FUNCTION_CALL", "stop")] + #[case::language("LANGUAGE", "content_filter")] + fn finish_reasons_map_like_python(#[case] reason: &str, #[case] expected: &str) { + let answer = response(json!({ + "candidates": [{"content": {"parts": [{"text": "x"}]}, "finishReason": reason}], + "usageMetadata": {"promptTokenCount": 1, "candidatesTokenCount": 1, "totalTokenCount": 2} + })); + assert_eq!(answer["choices"][0]["finish_reason"], expected); + } + + #[rstest] + fn grounding_becomes_annotations_and_metadata() { + let grounding = json!({ + "webSearchQueries": ["weather paris"], + "groundingChunks": [{"web": {"uri": "https://w", "title": "W"}}], + "groundingSupports": [{"segment": {"startIndex": 0, "endIndex": 5}, "groundingChunkIndices": [0]}] + }); + let answer = response(json!({ + "candidates": [{"content": {"parts": [{"text": "Sunny"}]}, "finishReason": "STOP", "groundingMetadata": grounding}], + "usageMetadata": {"promptTokenCount": 4, "candidatesTokenCount": 1, "toolUsePromptTokenCount": 7, "totalTokenCount": 12} + })); + assert_eq!( + answer["choices"][0]["message"]["annotations"], + json!([{"type": "url_citation", "url_citation": {"start_index": 0, "end_index": 5, "url": "https://w", "title": "W"}}]) + ); + assert_eq!(answer["vertex_ai_grounding_metadata"], json!([grounding])); + assert_eq!(answer["usage"]["prompt_tokens"], 4); + } + + #[rstest] + #[case::billed_tool_use(json!({"promptTokenCount": 4, "candidatesTokenCount": 1, "toolUsePromptTokenCount": 7, "totalTokenCount": 12, + "promptTokensDetails": [{"modality": "TEXT", "tokenCount": 4}]}), 11, 11)] + #[case::explicit_cache(json!({"promptTokenCount": 10, "candidatesTokenCount": 1, "totalTokenCount": 11, "cachedContentTokenCount": 6, + "promptTokensDetails": [{"modality": "TEXT", "tokenCount": 8}, {"modality": "IMAGE", "tokenCount": 2}], + "cacheTokensDetails": [{"modality": "TEXT", "tokenCount": 5}]}), 10, 3)] + fn usage_counts_prompt_tokens_like_python( + #[case] usage: Value, + #[case] prompt: u64, + #[case] text: u64, + ) { + let usage = calculate_usage(&json!({"usageMetadata": usage})).unwrap(); + assert_eq!(usage.prompt_tokens, prompt); + assert_eq!(usage.prompt_tokens_details.text_tokens, text); + } + + #[rstest] + fn a_response_without_usage_is_invalid() { + assert!(matches!( + transform_generate_content_response("m", json!({"candidates": []})), + Err(Error::InvalidResponse(_)) + )); + } +} diff --git a/litellm-rust/crates/llms/src/vertex_ai/mod.rs b/litellm-rust/crates/llms/src/vertex_ai/mod.rs index 138b893e4ca..561fd3bcc79 100644 --- a/litellm-rust/crates/llms/src/vertex_ai/mod.rs +++ b/litellm-rust/crates/llms/src/vertex_ai/mod.rs @@ -1,3 +1,5 @@ pub mod common_utils; +pub mod gemini; pub mod messages; pub mod ocr; +pub mod vertex_ai_partner_models; diff --git a/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/AGENTS.md b/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/AGENTS.md new file mode 100644 index 00000000000..ce3acb08b20 --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/AGENTS.md @@ -0,0 +1,12 @@ +# rules + +- Claude on Vertex AI for Chat Completions: the `anthropic/chat` translation built with `AnthropicHost::VertexAi`, addressed to the publisher `rawPredict` URL and authenticated with a Google access token +- The body drops `model`, carries `anthropic_version: vertex-2023-10-16`, sanitizes `output_config` and names its betas in `anthropic_beta`, filtered to the ones Vertex accepts +- Capability reads resolve the `vertex_ai/` cost map entry first; fast mode and `service_tier` are the Anthropic API's own + +# references + +- https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude/use-claude +- litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/transformation.py +- litellm/llms/vertex_ai/vertex_ai_partner_models/main.py +- litellm/llms/anthropic/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/mod.rs b/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/transformation.rs b/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/transformation.rs new file mode 100644 index 00000000000..0fbc2a9203b --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/anthropic/transformation.rs @@ -0,0 +1,847 @@ +//! Python's `VertexAIAnthropicConfig` for `/chat/completions`: Claude on Vertex AI, reached +//! through `VertexAIPartnerModels.completion`. The request is `AnthropicConfig`'s, built for the +//! `vertex_ai` host, then addressed to the publisher URL: no `model` in the body, Vertex's +//! `anthropic_version`, a sanitized `output_config`, and the betas in `anthropic_beta`. + +use std::collections::BTreeMap; + +use litellm_auth::VertexParams; +use litellm_auth_gcp::{ + VertexConfig, get_vertex_ai_location, get_vertex_ai_project, + get_vertex_ai_project_from_credentials, +}; +use litellm_llms_types::{ + formats::chat_completions::{ChatCompletionsResponse, ChatMessage}, + providers::{ + anthropic::{AnthropicBeta, BETA_HEADER, BetaProvider, BetaSet, DEFAULT_HEADERS}, + vertex_ai::{ + ANTHROPIC_VERSION, DEFAULT_LOCATION, GLOBAL_API_BASE, RAW_PREDICT, STREAM_RAW_PREDICT, + }, + }, +}; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, + anthropic::{ + beta_headers::BetaPolicy, + chat::{ + handler::ModelResponseIterator, + params::{AnthropicHost, ModelCapabilities}, + transformation::{build_messages_request, transform_messages_response}, + }, + }, + base_llm::{ + auth::{AuthScheme, Headers, ValidatedEnvironment}, + chat::{ + media::{data_url, remote_urls}, + params::ChatRequestContext, + streaming::{ChatStream, StreamShape}, + transformation::{ + BaseConfig, ProviderChatRequestData, ProviderChatResponseData, STREAM_PARAM, + Unsupported, + }, + }, + messages::streaming::event_stream as messages_event_stream, + }, + vertex_ai::{ + common_utils::with_query, + gemini::vertex_and_google_ai_studio_gemini::{CONFIG_PARAMS, VERTEXAI_API_BASE_ENV}, + messages::transformation::complete_vertex_anthropic_url, + }, +}; + +/// `PartnerModelPrefixes`: the ids `is_vertex_partner_model` sends to the partner handler. +const PARTNER_MODEL_PREFIXES: &[&str] = &[ + "meta/", + "deepseek-ai", + "mistral", + "codestral", + "jamba", + "claude", + "qwen", + "openai/gpt-oss-", + "minimaxai/", + "moonshotai/", + "zai-org/", + "google/gemma-", +]; + +/// `should_use_openai_handler`: the partners served through Vertex's OpenAI-compatible API. +const OPENAI_LIKE_PARTNERS: &[&str] = &[ + "llama", + "deepseek-ai", + "qwen", + "openai/gpt-oss-", + "minimaxai/", + "moonshotai/", + "zai-org/", + "google/gemma-", +]; + +/// The tool type prefix `is_web_search_tool_used` looks for. +const WEB_SEARCH_TOOL_PREFIX: &str = "web_search"; + +/// `get_vertex_ai_model_route(model) == PARTNER_MODELS` and the partner handler picking its +/// Claude branch: a partner model that is neither OpenAI-compatible, Mistral, nor AI21. +pub fn is_vertex_claude_model(model: &str) -> bool { + !model.contains("agent_engine/") + && PARTNER_MODEL_PREFIXES + .iter() + .any(|prefix| model.starts_with(prefix)) + && !OPENAI_LIKE_PARTNERS + .iter() + .any(|partner| model.contains(partner)) + && !model.contains("mistral") + && !model.contains("codestral") + && !model.contains("jamba") + && model.contains("claude") +} + +/// `VertexAIAnthropicConfig`: Claude on Vertex AI, addressed to a project and location and +/// authenticated with a Google access token. +pub struct VertexAiAnthropicConfig; + +pub const VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG: VertexAiAnthropicConfig = + VertexAiAnthropicConfig; + +fn vertex_config(optional_params: &Map) -> Result { + Ok(VertexConfig::from_sourced_optional_params( + optional_params, + &BTreeMap::new(), + )?) +} + +/// Whether `url` is on a host `get_vertex_base_url` builds. Any other host is a custom +/// `api_base`, which `_check_custom_proxy` asks for `alt=sse` when streaming. +fn is_vertex_host(url: &str) -> bool { + let rest = url.split_once("://").map_or(url, |(_, rest)| rest); + let host = rest.split(['/', '?', '#']).next().unwrap_or(rest); + let global = GLOBAL_API_BASE + .split_once("://") + .map_or(GLOBAL_API_BASE, |(_, host)| host); + host == global + || host.ends_with("-aiplatform.googleapis.com") + || (host.starts_with("aiplatform.") && host.ends_with(".rep.googleapis.com")) +} + +/// The same call made as a stream: `create_vertex_url` with `stream=True`, which names +/// `streamRawPredict`, plus `alt=sse` on a custom `api_base`. +pub fn streaming_url(url: &str) -> String { + let method = format!(":{RAW_PREDICT}"); + let (head, query) = url + .split_once('?') + .map_or((url, None), |(head, query)| (head, Some(query))); + let head = match head.strip_suffix(&method) { + Some(stem) => format!("{stem}:{STREAM_RAW_PREDICT}"), + None => head.to_owned(), + }; + let swapped = match query { + Some(query) => format!("{head}?{query}"), + None => head, + }; + match is_vertex_host(&swapped) { + true => swapped, + false => with_query(&swapped, "alt=sse"), + } +} + +/// The betas a caller named in the request's `extra_headers`, as a comma separated +/// `anthropic-beta` string or a list. +fn extra_header_betas(optional_params: &Map) -> Vec { + let value = optional_params + .get("extra_headers") + .and_then(|headers| headers.get(BETA_HEADER)); + match value { + Some(Value::String(text)) => text + .split(',') + .map(str::trim) + .filter(|beta| !beta.is_empty()) + .map(str::to_owned) + .collect(), + Some(Value::Array(items)) => items + .iter() + .filter_map(Value::as_str) + .map(str::to_owned) + .collect(), + _ => Vec::new(), + } +} + +/// `_add_context_management_beta_headers`: `compact_20260112` edits need the compaction beta, +/// any other edit the context management one. +fn context_management_betas(context_management: Option<&Value>) -> Vec { + let Some(Value::Object(context_management)) = context_management else { + return Vec::new(); + }; + let edits = context_management + .get("edits") + .and_then(Value::as_array) + .map(Vec::as_slice) + .unwrap_or_default(); + let compact = + |edit: &Value| edit.get("type").and_then(Value::as_str) == Some("compact_20260112"); + [ + edits + .iter() + .any(compact) + .then_some(AnthropicBeta::Compact20260112), + edits + .iter() + .any(|edit| !compact(edit)) + .then_some(AnthropicBeta::ContextManagement20250627), + ] + .into_iter() + .flatten() + .collect() +} + +/// `filter_and_transform_beta_headers` for Vertex AI: the betas it accepts, under its names. +fn accepted_on_vertex(betas: &BetaSet) -> BetaSet { + betas + .iter() + .filter_map(|beta| beta.on(BetaProvider::VertexAi)) + .collect() +} + +/// `sanitize_vertex_anthropic_output_params`: `output_format` folded into +/// `output_config.format`, `effort` kept only for models that accept it, and an emptied or +/// malformed `output_config` left out. +fn sanitize_output_params( + body: &mut Map, + caps: &ModelCapabilities<'_>, +) -> Result<(), Error> { + let output_format = body + .remove("output_format") + .filter(|value| !value.is_null()); + let output_config = match body.get("output_config") { + Some(Value::Object(config)) => Some(config.clone()), + _ => None, + }; + let output_config = match (output_config, output_format) { + (config, Some(format)) => { + let mut config = config.unwrap_or_default(); + config.insert("format".into(), format); + Some(config) + } + (config, None) => config, + }; + let Some(mut output_config) = output_config else { + body.remove("output_config"); + return Ok(()); + }; + if output_config.contains_key("effort") && !caps.supports_effort()? { + output_config.remove("effort"); + } + match output_config.is_empty() { + true => { + body.remove("output_config"); + } + false => { + body.insert("output_config".into(), Value::Object(output_config)); + } + } + Ok(()) +} + +impl VertexAiAnthropicConfig { + fn build_request( + &self, + model: &str, + messages: Vec, + mut optional_params: Map, + context: &ChatRequestContext<'_>, + ) -> Result { + let caller_betas = extra_header_betas(&optional_params); + // `VertexAIPartnerModels.completion` adds the version after the mapped params. + optional_params.insert("anthropic_version".into(), json!(ANTHROPIC_VERSION)); + let built = build_messages_request( + model, + messages, + optional_params, + context, + AnthropicHost::VertexAi, + CONFIG_PARAMS, + )?; + let Value::Object(mut body) = built.body else { + return Err(Error::InvalidRequest( + "the Messages request is not an object".into(), + )); + }; + body.remove("model"); + body.remove("extra_headers"); + let caps = ModelCapabilities::on(AnthropicHost::VertexAi, model, context); + sanitize_output_params(&mut body, &caps)?; + + let betas: BetaSet = context_management_betas(body.get("context_management")) + .into_iter() + .chain( + caller_betas + .iter() + .map(|beta| beta.parse().unwrap_or_else(|never| match never {})), + ) + .collect(); + let web_search = body + .get("tools") + .and_then(Value::as_array) + .into_iter() + .flatten() + .any(|tool| { + tool.get("type") + .and_then(Value::as_str) + .is_some_and(|kind| kind.starts_with(WEB_SEARCH_TOOL_PREFIX)) + }); + // `get_anthropic_headers` sends Vertex only the web search beta; the request's own + // betas replace it when there are any. + let header_betas = match (betas.is_empty(), web_search) { + (false, _) => accepted_on_vertex(&betas), + (true, true) => { + accepted_on_vertex(&[AnthropicBeta::WebSearch20250305].into_iter().collect()) + } + (true, false) => BetaSet::default(), + }; + let body_betas = accepted_on_vertex(&betas); + if !body_betas.is_empty() { + body.insert( + "anthropic_beta".into(), + Value::Array(body_betas.iter().map(|beta| json!(beta.as_str())).collect()), + ); + } + Ok(ProviderChatRequestData { + body: Value::Object(body), + stream_shape: built.stream_shape, + headers: match header_betas.is_empty() { + true => Vec::new(), + false => vec![(BETA_HEADER.to_owned(), header_betas.to_string())], + }, + }) + } +} + +impl BaseConfig for VertexAiAnthropicConfig { + fn secret_names(&self) -> Vec<&'static str> { + litellm_auth_gcp::secret_names() + .iter() + .copied() + .chain([VERTEXAI_API_BASE_ENV]) + .collect() + } + + /// `get_complete_vertex_url` for the Claude partner, with `VERTEXAI_API_BASE` as the + /// fallback base `completion` reads. + fn get_complete_url( + &self, + api_base: Option<&str>, + model: &str, + optional_params: &Map, + env_lookup: &dyn Fn(&str) -> Option, + ) -> Result { + let config = vertex_config(optional_params)?; + let project = get_vertex_ai_project(&config, env_lookup) + .or_else(|| get_vertex_ai_project_from_credentials(&config, env_lookup)) + .ok_or_else(|| Error::Auth(VertexParams::PROJECT.missing("Vertex AI")))?; + let location = get_vertex_ai_location(&config, env_lookup) + .unwrap_or_else(|| DEFAULT_LOCATION.to_owned()); + let api_base = api_base + .map(str::to_owned) + .or_else(|| env_lookup(VERTEXAI_API_BASE_ENV)) + .filter(|base| !base.trim().is_empty()); + complete_vertex_anthropic_url(api_base.as_deref(), &project, &location, model, false) + } + + fn transform_request( + &self, + model: &str, + messages: Vec, + optional_params: Map, + context: &ChatRequestContext<'_>, + ) -> Result { + self.build_request(model, messages, optional_params, context) + } + + /// `transform_response` names the requested model, not the one Vertex reports. + fn transform_response( + &self, + model: &str, + response: ProviderChatResponseData, + shape: &StreamShape, + ) -> Result { + let mut body = response.body; + if let Value::Object(fields) = &mut body { + fields.insert("model".into(), json!(model)); + } + transform_messages_response(body, shape) + } + + /// `inline_remote_image_urls`: Vertex takes no image by URL, so every remote image is + /// downloaded. + fn remote_media(&self, messages: &[ChatMessage]) -> Vec { + remote_urls(messages, &["user"], false, |url| { + (url.starts_with("http://") || url.starts_with("https://")) && data_url(url).is_none() + }) + } + + /// Python's partner handler sets `Authorization: Bearer ` over whatever was + /// forwarded, so the Google token is always the credential. A forwarded `anthropic-beta` + /// keeps only what Vertex accepts. + fn validate_environment( + &self, + headers: Headers, + _api_key: Option<&str>, + _model: &str, + optional_params: &Map, + _env_lookup: &dyn Fn(&str) -> Option, + ) -> Result { + Ok(ValidatedEnvironment { + headers: BetaPolicy::Filter(BetaProvider::VertexAi).apply(headers), + auth: AuthScheme::GcpAccessToken { + config: Box::new(vertex_config(optional_params)?), + }, + }) + } + + fn model_response_iterator(&self, shape: StreamShape) -> Option { + Some(ChatStream::new( + messages_event_stream, + ModelResponseIterator::new(shape), + )) + } + + /// `streamRawPredict` with `stream: true` in the body, after the other params. + fn streaming_request(&self, url: String, mut body: Value) -> Result<(String, Value), Error> { + if let Value::Object(fields) = &mut body { + fields.insert(STREAM_PARAM.into(), Value::Bool(true)); + } + Ok((streaming_url(&url), body)) + } + + fn default_headers(&self) -> &'static [(&'static str, &'static str)] { + DEFAULT_HEADERS + } + + fn config_params(&self) -> &'static [&'static str] { + CONFIG_PARAMS + } + + /// Every request is translated; a param or message Python refuses is refused here too. + fn unsupported_reason( + &self, + _messages: &[ChatMessage], + _optional_params: &Map, + ) -> Option { + None + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use rstest::rstest; + + use super::*; + use crate::base_llm::model_info::ModelInfo; + + struct CostMap(HashMap>); + + impl ModelInfo for CostMap { + fn entry(&self, model: &str) -> Option>> { + self.0.get(model).map(std::borrow::Cow::Borrowed) + } + + fn declares(&self, key: &str) -> bool { + self.0.values().any(|entry| entry.contains_key(key)) + } + } + + const SONNET: &str = "claude-sonnet-4-5@20250929"; + const OPUS: &str = "claude-opus-4-7"; + + fn cost_map() -> CostMap { + let entry = |value: Value| value.as_object().cloned().unwrap(); + CostMap(HashMap::from([ + ( + format!("vertex_ai/{SONNET}"), + entry(json!({"max_output_tokens": 64000, "supports_reasoning": true})), + ), + ( + "claude-sonnet-4-5".to_owned(), + entry( + json!({"max_output_tokens": 64000, "supports_native_structured_output": true}), + ), + ), + ( + format!("vertex_ai/{OPUS}"), + entry(json!({"supports_reasoning": true, "supports_adaptive_thinking": true})), + ), + ( + OPUS.to_owned(), + entry(json!({"supports_output_config": true, "supports_speed": true})), + ), + ( + "vertex_ai/claude-haiku-4-5@20251001".to_owned(), + entry(json!({"supports_reasoning": true})), + ), + ])) + } + + fn object(value: Value) -> Map { + value.as_object().cloned().expect("an object") + } + + fn request_with( + model: &str, + messages: Value, + params: Value, + drop_params: bool, + ) -> Result { + let info = cost_map(); + let context = ChatRequestContext { + drop_params: Some(drop_params), + model_info: Some(&info), + }; + VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG.transform_request( + model, + serde_json::from_value(messages).unwrap(), + object(params), + &context, + ) + } + + fn body(model: &str, params: Value) -> Value { + request_with(model, hi(), params, false) + .expect("request transforms") + .body + } + + fn hi() -> Value { + json!([{"role": "system", "content": "be terse"}, {"role": "user", "content": "hi"}]) + } + + fn no_env(_: &str) -> Option { + None + } + + #[rstest] + #[case::versioned("claude-sonnet-4-5@20250929", true)] + #[case::bare("claude-opus-4-7", true)] + #[case::gemini("gemini-2.5-pro", false)] + #[case::llama("meta/llama-3.1-405b-instruct-maas", false)] + #[case::mistral("mistral-large@2407", false)] + #[case::not_a_partner_prefix("my-claude", false)] + fn only_claude_partner_models_route_here(#[case] model: &str, #[case] expected: bool) { + assert_eq!(is_vertex_claude_model(model), expected); + } + + #[rstest] + fn builds_the_body_python_builds() { + let body = body( + SONNET, + json!({ + "max_tokens": 128, + "temperature": 0.2, + "stop": "END", + "vertex_project": "p", + "vertex_location": "us-east5", + "vertex_credentials": "{}" + }), + ); + assert_eq!( + body, + json!({ + "messages": [{"role": "user", "content": [{"type": "text", "text": "hi"}]}], + "max_tokens": 128, + "temperature": 0.2, + "stop_sequences": ["END"], + "anthropic_version": "vertex-2023-10-16", + "system": [{"type": "text", "text": "be terse"}], + }) + ); + } + + #[rstest] + fn max_tokens_defaults_from_the_bare_entry_like_python() { + // `get_max_tokens` reads the bare id, which the cost map has no entry for. + assert_eq!(body(SONNET, json!({}))["max_tokens"], json!(4096)); + } + + #[rstest] + fn reasoning_effort_becomes_a_thinking_budget() { + let body = body(SONNET, json!({"reasoning_effort": "low"})); + assert_eq!( + body["thinking"], + json!({"type": "enabled", "budget_tokens": 1024}) + ); + assert_eq!(body["max_tokens"], json!(1024 + 4096)); + } + + #[rstest] + fn an_adaptive_model_gets_adaptive_thinking_and_its_effort() { + let body = body(OPUS, json!({"reasoning_effort": "high", "max_tokens": 64})); + assert_eq!( + body["thinking"], + json!({"type": "adaptive", "display": "summarized"}) + ); + assert_eq!(body["output_config"], json!({"effort": "high"})); + } + + #[rstest] + fn effort_is_dropped_for_a_model_that_takes_none() { + let body = body( + "claude-haiku-4-5@20251001", + json!({"output_config": {"effort": "low"}, "max_tokens": 64}), + ); + assert!(body.get("output_config").is_none(), "{body}"); + } + + #[rstest] + fn native_structured_output_is_read_from_the_vertex_entry_only() { + // The bare Anthropic entry supports it; the Vertex one does not say, so Python uses + // the forced JSON tool. + let request = request_with( + SONNET, + hi(), + json!({"response_format": {"type": "json_schema", "json_schema": {"name": "x", "schema": {"type": "object", "properties": {"a": {"type": "string"}}}}}, "max_tokens": 64}), + false, + ) + .unwrap(); + assert!(request.body.get("output_config").is_none()); + assert!(request.body.get("output_format").is_none()); + assert_eq!( + request.body["tool_choice"], + json!({"name": "json_tool_call", "type": "tool"}) + ); + assert!(request.stream_shape.json_mode); + } + + #[rstest] + #[case::service_tier(json!({"service_tier": "auto"}))] + #[case::speed(json!({"speed": "fast"}))] + fn anthropic_api_only_params_are_refused_or_dropped(#[case] params: Value) { + assert!(matches!( + request_with(OPUS, hi(), params.clone(), false), + Err(Error::InvalidRequest(_)) + )); + let body = request_with(OPUS, hi(), params, true).unwrap().body; + assert!(body.get("service_tier").is_none() && body.get("speed").is_none()); + } + + #[rstest] + fn context_management_names_its_betas_in_the_body_and_the_header() { + let request = request_with( + SONNET, + hi(), + json!({ + "context_management": {"edits": [{"type": "clear_tool_uses_20250919"}, {"type": "compact_20260112"}]}, + "extra_headers": {"anthropic-beta": "advanced-tool-use-2025-11-20, oauth-2025-04-20"}, + "max_tokens": 64 + }), + false, + ) + .unwrap(); + assert_eq!( + request.body["anthropic_beta"], + json!([ + "compact-2026-01-12", + "context-management-2025-06-27", + "tool-search-tool-2025-10-19" + ]) + ); + assert_eq!( + request.headers, + [( + "anthropic-beta".to_owned(), + "compact-2026-01-12,context-management-2025-06-27,tool-search-tool-2025-10-19" + .to_owned() + )] + ); + assert!(request.body.get("extra_headers").is_none()); + } + + #[rstest] + fn web_search_alone_sends_only_its_header() { + let request = request_with( + SONNET, + hi(), + json!({"web_search_options": {}, "max_tokens": 64}), + false, + ) + .unwrap(); + assert!(request.body.get("anthropic_beta").is_none()); + assert_eq!( + request.headers, + [( + "anthropic-beta".to_owned(), + "web-search-2025-03-05".to_owned() + )] + ); + } + + #[rstest] + fn without_betas_no_header_or_body_field() { + let request = request_with(SONNET, hi(), json!({"max_tokens": 8}), false).unwrap(); + assert!(request.headers.is_empty()); + assert!(request.body.get("anthropic_beta").is_none()); + } + + #[rstest] + fn tools_round_trip_into_anthropic_shape() { + let body = request_with( + SONNET, + json!([ + {"role": "user", "content": "weather?"}, + {"role": "assistant", "content": null, "tool_calls": [{"id": "toolu_1", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}}]}, + {"role": "tool", "tool_call_id": "toolu_1", "content": "sunny"} + ]), + json!({"tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}], "max_tokens": 64}), + false, + ) + .unwrap() + .body; + assert_eq!( + body["tools"], + json!([{"name": "get_weather", "input_schema": {"type": "object", "properties": {"city": {"type": "string"}}}, "type": "custom"}]) + ); + assert_eq!( + body["messages"][1]["content"][0], + json!({"type": "tool_use", "id": "toolu_1", "name": "get_weather", "input": {"city": "Paris"}}) + ); + assert_eq!(body["messages"][2]["content"][0]["type"], "tool_result"); + } + + fn url(api_base: Option<&str>, location: &str) -> String { + VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG + .get_complete_url( + api_base, + SONNET, + &object(json!({"vertex_project": "proj", "vertex_location": location})), + &no_env, + ) + .unwrap() + } + + const PATH: &str = "/v1/projects/proj/locations/{location}/publishers/anthropic/models/claude-sonnet-4-5@20250929:rawPredict"; + + #[rstest] + #[case::regional(None, "us-east5", "https://us-east5-aiplatform.googleapis.com")] + #[case::global(None, "global", "https://aiplatform.googleapis.com")] + #[case::custom_host(Some("http://127.0.0.1:9/"), "us-east5", "http://127.0.0.1:9")] + fn the_url_names_the_project_location_and_model( + #[case] api_base: Option<&str>, + #[case] location: &str, + #[case] host: &str, + ) { + assert_eq!( + url(api_base, location), + format!("{host}{}", PATH.replace("{location}", location)) + ); + } + + #[rstest] + fn the_location_defaults_to_us_central1() { + let url = VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG + .get_complete_url( + None, + SONNET, + &object(json!({"vertex_project": "proj"})), + &no_env, + ) + .unwrap(); + assert!( + url.starts_with("https://us-central1-aiplatform.googleapis.com/v1/projects/proj/locations/us-central1/"), + "{url}" + ); + } + + #[rstest] + #[case::regional( + "https://us-east5-aiplatform.googleapis.com/v1/x/models/m:rawPredict", + "https://us-east5-aiplatform.googleapis.com/v1/x/models/m:streamRawPredict" + )] + #[case::global( + "https://aiplatform.googleapis.com/v1/x/models/m:rawPredict", + "https://aiplatform.googleapis.com/v1/x/models/m:streamRawPredict" + )] + #[case::custom_base( + "http://127.0.0.1:9/v1/x/models/m:rawPredict", + "http://127.0.0.1:9/v1/x/models/m:streamRawPredict?alt=sse" + )] + #[case::custom_path( + "https://proxy.example/route:rawPredict", + "https://proxy.example/route:streamRawPredict?alt=sse" + )] + fn a_stream_calls_stream_raw_predict(#[case] url: &str, #[case] expected: &str) { + let (streamed, body) = VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG + .streaming_request(url.to_owned(), json!({"max_tokens": 1})) + .unwrap(); + assert_eq!(streamed, expected); + assert_eq!(body, json!({"max_tokens": 1, "stream": true})); + } + + #[rstest] + fn the_google_token_replaces_a_forwarded_bearer_and_betas_are_filtered() { + let validated = VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG + .validate_environment( + vec![ + ("Authorization".to_owned(), "Bearer caller".to_owned()), + ( + "anthropic-beta".to_owned(), + "oauth-2025-04-20,context-management-2025-06-27".to_owned(), + ), + ], + None, + SONNET, + &Map::new(), + &no_env, + ) + .unwrap(); + assert!(matches!(validated.auth, AuthScheme::GcpAccessToken { .. })); + assert!(validated.headers.contains(&( + "anthropic-beta".to_owned(), + "context-management-2025-06-27".to_owned() + ))); + } + + #[rstest] + fn the_response_names_the_requested_model() { + let response = VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG + .transform_response( + SONNET, + ProviderChatResponseData { + body: json!({ + "id": "msg_1", "type": "message", "role": "assistant", + "content": [ + {"type": "thinking", "thinking": "hmm", "signature": "sig"}, + {"type": "text", "text": "hello"} + ], + "stop_reason": "end_turn", + "usage": {"input_tokens": 11, "output_tokens": 4, "cache_read_input_tokens": 2} + }), + }, + &StreamShape::default(), + ) + .unwrap(); + assert_eq!(response.model, SONNET); + let message = &response.choices[0].message; + assert_eq!(message.content.as_deref(), Some("hello")); + assert_eq!(message.reasoning_content.as_deref(), Some("hmm")); + assert_eq!(response.choices[0].finish_reason, "stop"); + let usage = &response.usage; + assert_eq!((usage.prompt_tokens, usage.completion_tokens), (13, 4)); + } + + #[rstest] + fn every_remote_image_is_downloaded() { + let messages: Vec = + serde_json::from_value(json!([{"role": "user", "content": [ + {"type": "image_url", "image_url": {"url": "https://example.com/a.png"}}, + {"type": "image_url", "image_url": {"url": "http://example.com/b.png"}}, + {"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}} + ]}])) + .unwrap(); + assert_eq!( + VERTEX_ANTHROPIC_CHAT_COMPLETIONS_CONFIG.remote_media(&messages), + ["https://example.com/a.png", "http://example.com/b.png"] + ); + } +} diff --git a/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/mod.rs b/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/mod.rs new file mode 100644 index 00000000000..e529997219e --- /dev/null +++ b/litellm-rust/crates/llms/src/vertex_ai/vertex_ai_partner_models/mod.rs @@ -0,0 +1 @@ +pub mod anthropic; diff --git a/litellm-rust/crates/llms/src/xai/chat/AGENTS.md b/litellm-rust/crates/llms/src/xai/chat/AGENTS.md new file mode 100644 index 00000000000..9290e053e74 --- /dev/null +++ b/litellm-rust/crates/llms/src/xai/chat/AGENTS.md @@ -0,0 +1,11 @@ +# rules + +- xAI at `https://api.x.ai/v1` +- `web_search_options` and non-user `name`s are dropped, tools lose `strict` +- Reasoning tokens xAI counts apart are folded into `completion_tokens`, in answers and stream chunks +- Shared flow and response handling: `../../openai/chat/compatible.rs` + +# references + +- https://docs.x.ai/docs/api-reference#chat-completions +- litellm/llms/xai/chat/transformation.py diff --git a/litellm-rust/crates/llms/src/xai/chat/mod.rs b/litellm-rust/crates/llms/src/xai/chat/mod.rs new file mode 100644 index 00000000000..f239b6921fa --- /dev/null +++ b/litellm-rust/crates/llms/src/xai/chat/mod.rs @@ -0,0 +1 @@ +pub mod transformation; diff --git a/litellm-rust/crates/llms/src/xai/chat/transformation.rs b/litellm-rust/crates/llms/src/xai/chat/transformation.rs new file mode 100644 index 00000000000..b346e5d30f4 --- /dev/null +++ b/litellm-rust/crates/llms/src/xai/chat/transformation.rs @@ -0,0 +1,272 @@ +//! `litellm/llms/xai/chat/transformation.py`: xAI at `https://api.x.ai/v1`. Tools lose +//! `strict`, `web_search_options` and non-user `name`s are dropped, and the usage folds the +//! separately counted reasoning tokens into `completion_tokens` as OpenAI counts them. + +use litellm_core_utils::get_llm_provider_logic::LlmProviders; +use serde_json::{Map, Value, json}; + +use crate::{ + Error, + openai::chat::compatible::{ + CompatibleContext, OpenAiCompatibleChatConfig, ResponseShape, filter_value, role, + }, +}; + +const SUPPORTED_PARAMS: &[&str] = &[ + "logit_bias", + "logprobs", + "max_completion_tokens", + "max_tokens", + "n", + "parallel_tool_calls", + "presence_penalty", + "response_format", + "seed", + "stream", + "stream_options", + "temperature", + "tool_choice", + "tools", + "top_logprobs", + "top_p", + "user", + "web_search_options", +]; + +pub struct XaiChatConfig; + +pub const XAI_CHAT_CONFIG: XaiChatConfig = XaiChatConfig; + +impl OpenAiCompatibleChatConfig for XaiChatConfig { + fn provider(&self) -> LlmProviders { + LlmProviders::Xai + } + + fn label(&self) -> &'static str { + "xAI" + } + + fn api_base_env(&self) -> &'static [&'static str] { + &["XAI_API_BASE"] + } + + fn default_api_base(&self) -> Option<&'static str> { + Some("https://api.x.ai/v1") + } + + fn api_key_env(&self) -> &'static [&'static str] { + &["XAI_API_KEY"] + } + + fn supported_params(&self, context: &CompatibleContext<'_>) -> Vec<&'static str> { + let model = context.model; + let mut params = SUPPORTED_PARAMS.to_vec(); + if !["grok-3-mini", "grok-4", "grok-code-fast"] + .iter() + .any(|family| model.contains(family)) + { + params.push("stop"); + } + if !["grok-4", "grok-code-fast"] + .iter() + .any(|family| model.contains(family)) + { + params.push("frequency_penalty"); + } + if context.supports(self.name(), "supports_reasoning") { + params.push("reasoning_effort"); + } + params + } + + fn map_params( + &self, + non_default: &Map, + optional: &mut Map, + context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + let supported = self.supported_params(context); + for (name, value) in non_default { + match name.as_str() { + "max_completion_tokens" => { + optional.insert("max_tokens".into(), value.clone()); + } + "tools" => { + let Value::Array(tools) = value else { continue }; + let tools: Vec = tools + .iter() + .cloned() + .map(|mut tool| { + filter_value(&mut tool, "strict", 0); + tool + }) + .collect(); + if !tools.is_empty() { + optional.insert("tools".into(), Value::Array(tools)); + } + } + name if supported.contains(&name) && !value.is_null() => { + optional.insert(name.to_owned(), value.clone()); + } + _ => {} + } + } + Ok(()) + } + + fn finish_request( + &self, + body: &mut Map, + _context: &CompatibleContext<'_>, + ) -> Result<(), Error> { + body.remove("web_search_options"); + if let Some(Value::Array(messages)) = body.get_mut("messages") { + for message in messages.iter_mut() { + if role(message) != Some("user") + && let Some(object) = message.as_object_mut() + { + object.remove("name"); + } + } + } + Ok(()) + } + + fn fix_response(&self, body: &mut Map, _shape: &ResponseShape) -> bool { + let mut changed = false; + if let Some(Value::Array(choices)) = body.get_mut("choices") { + for choice in choices.iter_mut().filter_map(Value::as_object_mut) { + let has_tool_calls = choice + .get("message") + .and_then(|message| message.get("tool_calls")) + .and_then(Value::as_array) + .is_some_and(|calls| !calls.is_empty()); + if has_tool_calls && choice.get("finish_reason") == Some(&json!("")) { + choice.insert("finish_reason".into(), json!("tool_calls")); + changed = true; + } + } + } + changed | fix_usage(body) + } + + fn fix_chunk(&self, chunk: &mut Map, _shape: &ResponseShape) -> bool { + fix_usage(chunk) + } +} + +fn count(usage: &Map, key: &str) -> i64 { + usage.get(key).and_then(Value::as_i64).unwrap_or(0) +} + +/// `fold_reasoning_tokens_into_completion` then `_normalize_openai_compatible_usage_totals`. +fn fix_usage(body: &mut Map) -> bool { + let Some(Value::Object(usage)) = body.get_mut("usage") else { + return false; + }; + let mut changed = false; + let reasoning = usage + .get("completion_tokens_details") + .and_then(|details| details.get("reasoning_tokens")) + .and_then(Value::as_i64) + .unwrap_or(0); + let (prompt, completion, total) = ( + count(usage, "prompt_tokens"), + count(usage, "completion_tokens"), + count(usage, "total_tokens"), + ); + if reasoning > 0 && total != prompt + completion && total == prompt + completion + reasoning { + usage.insert("completion_tokens".into(), json!(completion + reasoning)); + changed = true; + } + let expected = prompt + count(usage, "completion_tokens"); + if count(usage, "total_tokens") < expected { + usage.insert("total_tokens".into(), json!(expected)); + changed = true; + } + changed +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + use crate::openai::chat::compatible::{fix_response, transform_request}; + + fn object(value: Value) -> Map { + let Value::Object(map) = value else { + panic!("an object") + }; + map + } + + fn send(model: &str, params: Value) -> Result, Error> { + transform_request( + &XAI_CHAT_CONFIG, + &CompatibleContext { + model, + drop_params: false, + model_info: None, + }, + object(params), + ) + .map(|sent| sent.body) + } + + #[rstest] + fn tools_lose_strict_and_web_search_and_names_are_dropped() { + let sent = send("grok-3", json!({"messages": [ + {"role": "user", "name": "u", "content": "hi"}, + {"role": "assistant", "name": "a", "content": "yo"}], + "tools": [{"type": "function", "function": {"name": "f", "strict": true, "parameters": {"type": "object"}}}], + "web_search_options": {}, "max_completion_tokens": 4, "stop": ["x"]})) + .unwrap(); + assert_eq!( + Value::Object(sent), + json!({"model": "grok-3", "messages": [ + {"role": "user", "name": "u", "content": "hi"}, + {"role": "assistant", "content": "yo"}], + "tools": [{"type": "function", "function": {"name": "f", "parameters": {"type": "object"}}}], + "max_tokens": 4, "stop": ["x"]}) + ); + } + + #[rstest] + #[case::stop("grok-4", json!({"messages": [], "stop": ["x"]}))] + #[case::frequency_penalty("grok-code-fast-1", json!({"messages": [], "frequency_penalty": 1}))] + fn params_some_grok_models_reject_are_refused(#[case] model: &str, #[case] params: Value) { + assert!(send(model, params).is_err()); + } + + #[rstest] + #[case::folds(json!({"prompt_tokens": 2, "completion_tokens": 3, "total_tokens": 9, + "completion_tokens_details": {"reasoning_tokens": 4}}), json!(7), json!(9))] + #[case::already_folded(json!({"prompt_tokens": 2, "completion_tokens": 7, "total_tokens": 9, + "completion_tokens_details": {"reasoning_tokens": 4}}), json!(7), json!(9))] + #[case::raises_total(json!({"prompt_tokens": 2, "completion_tokens": 3, "total_tokens": 1}), json!(3), json!(5))] + fn usage_counts_reasoning_as_completion( + #[case] usage: Value, + #[case] completion: Value, + #[case] total: Value, + ) { + let mut body = object(json!({"choices": [], "usage": usage})); + XAI_CHAT_CONFIG.fix_chunk(&mut body, &ResponseShape::default()); + assert_eq!(body["usage"]["completion_tokens"], completion); + assert_eq!(body["usage"]["total_tokens"], total); + } + + #[rstest] + fn an_empty_finish_reason_with_tool_calls_becomes_tool_calls() { + let mut body = object( + json!({"choices": [{"finish_reason": "", "message": {"role": "assistant", + "tool_calls": [{"id": "c", "type": "function", "function": {"name": "f", "arguments": "{}"}}]}}]}), + ); + assert!(fix_response( + &XAI_CHAT_CONFIG, + &mut body, + &ResponseShape::default() + )); + assert_eq!(body["choices"][0]["finish_reason"], json!("tool_calls")); + } +} diff --git a/litellm-rust/crates/llms/src/xai/mod.rs b/litellm-rust/crates/llms/src/xai/mod.rs new file mode 100644 index 00000000000..30a62fc508f --- /dev/null +++ b/litellm-rust/crates/llms/src/xai/mod.rs @@ -0,0 +1 @@ +pub mod chat; diff --git a/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs b/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs index 5ad9f8df8b8..dd2c079082c 100644 --- a/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs +++ b/litellm-rust/crates/python-bridge/src/routes/chat_completions/host.rs @@ -1,5 +1,6 @@ -use std::convert::Infallible; +use std::{convert::Infallible, sync::Arc}; +use super::model_cost::PythonModelCost; use crate::routes::inference::InferenceHost; use litellm_host_python::{InvokeError, PythonBinding, PythonHostCalls, PythonOwned}; use litellm_inference_chat::{Error, route::ChatCompletions, types::ChatCompletionsCall}; @@ -26,6 +27,10 @@ pub(super) fn project( custom_llm_provider: call.options.custom_llm_provider, extra_headers: call.options.extra_headers, timeout: call.options.timeout, + // Python's own settings decide what a drop would do, so a param that needs one goes + // back to Python. Capabilities come from the cost map Python itself reads. + drop_params: None, + model_info: Some(Arc::new(PythonModelCost::current(py)?)), }) } diff --git a/litellm-rust/crates/python-bridge/src/routes/chat_completions/mod.rs b/litellm-rust/crates/python-bridge/src/routes/chat_completions/mod.rs index d78d141f0f2..ce319189a60 100644 --- a/litellm-rust/crates/python-bridge/src/routes/chat_completions/mod.rs +++ b/litellm-rust/crates/python-bridge/src/routes/chat_completions/mod.rs @@ -1,4 +1,5 @@ mod host; +mod model_cost; use std::sync::Arc; diff --git a/litellm-rust/crates/python-bridge/src/routes/chat_completions/model_cost.rs b/litellm-rust/crates/python-bridge/src/routes/chat_completions/model_cost.rs new file mode 100644 index 00000000000..09a7e64f101 --- /dev/null +++ b/litellm-rust/crates/python-bridge/src/routes/chat_completions/model_cost.rs @@ -0,0 +1,88 @@ +//! Python's `litellm.model_cost` as the cost map chat translation reads, so a capability +//! lookup sees the map Python itself would, models registered at runtime included. + +use std::{borrow::Cow, collections::HashSet, sync::OnceLock}; + +use litellm_host_python::from_py; +use litellm_llms::base_llm::model_info::ModelInfo; +use pyo3::{prelude::*, types::PyDict}; +use serde_json::{Map, Value}; + +pub(super) struct PythonModelCost { + map: Py, + declared: OnceLock>, +} + +impl PythonModelCost { + /// The dict `litellm.model_cost` holds now. A map Python replaces later is not followed, + /// as a call already under way would not follow it either. + pub(super) fn current(py: Python<'_>) -> PyResult { + let map = py + .import("litellm")? + .getattr("model_cost")? + .cast_into::()? + .unbind(); + Ok(Self { + map, + declared: OnceLock::new(), + }) + } +} + +impl ModelInfo for PythonModelCost { + fn entry(&self, model: &str) -> Option>> { + Python::attach(|py| { + let entry = self.map.bind(py).get_item(model).ok()??; + from_py::>(&entry).ok().map(Cow::Owned) + }) + } + + fn declares(&self, key: &str) -> bool { + self.declared + .get_or_init(|| { + Python::attach(|py| { + self.map + .bind(py) + .values() + .iter() + .filter_map(|entry| entry.cast_into::().ok()) + .flat_map(|entry| { + entry + .keys() + .iter() + .filter_map(|key| key.extract::().ok()) + .collect::>() + }) + .collect() + }) + }) + .contains(key) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn reads_the_entry_python_holds_and_the_keys_any_entry_sets() { + Python::initialize(); + Python::attach(|py| { + let map = PyDict::new(py); + let entry = PyDict::new(py); + entry.set_item("supports_reasoning", true).unwrap(); + entry.set_item("max_output_tokens", 64000).unwrap(); + map.set_item("claude-x", entry).unwrap(); + let cost = PythonModelCost { + map: map.unbind(), + declared: OnceLock::new(), + }; + let found = cost.entry("claude-x").unwrap(); + assert_eq!(found.get("supports_reasoning"), Some(&Value::Bool(true))); + assert_eq!(found.get("max_output_tokens"), Some(&Value::from(64000))); + assert!(cost.entry("unknown").is_none()); + assert!(cost.declares("max_output_tokens")); + assert!(!cost.declares("default_reasoning_effort")); + }); + } +} diff --git a/litellm-rust/crates/python-compat/src/json.rs b/litellm-rust/crates/python-compat/src/json.rs index d3436519ac4..47c6a1de8d8 100644 --- a/litellm-rust/crates/python-compat/src/json.rs +++ b/litellm-rust/crates/python-compat/src/json.rs @@ -13,7 +13,14 @@ use crate::{Error, Value, repr::float_repr}; /// `json.dumps(value)`. pub fn dumps(value: &Value) -> Result { let mut out = String::new(); - write_value(&mut out, value)?; + write_value(&mut out, value, true)?; + Ok(out) +} + +/// `json.dumps(value, ensure_ascii=False)`: text outside ASCII is written as is. +pub fn dumps_unicode(value: &Value) -> Result { + let mut out = String::new(); + write_value(&mut out, value, false)?; Ok(out) } @@ -79,7 +86,7 @@ pub fn to_json(value: &Value) -> Result { }) } -fn write_value(out: &mut String, value: &Value) -> Result<(), Error> { +fn write_value(out: &mut String, value: &Value, ascii: bool) -> Result<(), Error> { match value { Value::None => out.push_str("null"), Value::Bool(true) => out.push_str("true"), @@ -88,14 +95,14 @@ fn write_value(out: &mut String, value: &Value) -> Result<(), Error> { let _ = write!(out, "{value}"); } Value::Float(value) => out.push_str(&float_text(*value)), - Value::Str(text) => write_string(out, text), + Value::Str(text) => write_string(out, text, ascii), Value::List(values) | Value::Tuple(values) => { out.push('['); for (index, value) in values.iter().enumerate() { if index > 0 { out.push_str(", "); } - write_value(out, value)?; + write_value(out, value, ascii)?; } out.push(']'); } @@ -105,9 +112,9 @@ fn write_value(out: &mut String, value: &Value) -> Result<(), Error> { if index > 0 { out.push_str(", "); } - write_string(out, &json_key(key)?); + write_string(out, &json_key(key)?, ascii); out.push_str(": "); - write_value(out, value)?; + write_value(out, value, ascii)?; } out.push('}'); } @@ -143,8 +150,9 @@ fn json_key(key: &Value) -> Result { } /// `py_encode_basestring_ascii`: escape `"`, `\`, control characters, and everything outside -/// printable ASCII as `\uXXXX`, with surrogate pairs above the BMP. -fn write_string(out: &mut String, text: &str) { +/// printable ASCII as `\uXXXX`, with surrogate pairs above the BMP. Without `ascii`, +/// `py_encode_basestring`: only `"`, `\` and control characters are escaped. +fn write_string(out: &mut String, text: &str, ascii: bool) { out.push('"'); for ch in text.chars() { match ch { @@ -156,6 +164,7 @@ fn write_string(out: &mut String, text: &str) { '\u{8}' => out.push_str("\\b"), '\u{c}' => out.push_str("\\f"), ' '..='~' => out.push(ch), + ch if !ascii && ch >= ' ' => out.push(ch), ch => { let mut units = [0u16; 2]; for unit in ch.encode_utf16(&mut units) { @@ -166,3 +175,22 @@ fn write_string(out: &mut String, text: &str) { } out.push('"'); } + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + #[rstest] + #[case::ascii(true, r#"{"city": "M\u00fcnchen\n"}"#)] + #[case::unicode(false, "{\"city\": \"München\\n\"}")] + fn non_ascii_text_is_escaped_only_with_ensure_ascii(#[case] ascii: bool, #[case] text: &str) { + let value = from_json(serde_json::json!({"city": "München\n"})); + let dumped = match ascii { + true => dumps(&value), + false => dumps_unicode(&value), + }; + assert_eq!(dumped.unwrap(), text); + } +} diff --git a/litellm-rust/docs/customer-proxy/gemini-chat.svg b/litellm-rust/docs/customer-proxy/gemini-chat.svg new file mode 100644 index 00000000000..85515dbb171 --- /dev/null +++ b/litellm-rust/docs/customer-proxy/gemini-chat.svg @@ -0,0 +1,95 @@ + + + + + + + + POST /v1/chat/completions to a gemini/ or vertex_ai/ Gemini deployment + + + + Client + OpenAI SDK body + + + + gateway-inference chat_completions + auth, hooks admit, route deployment + os.environ/ refs resolved via the secret source, + deployment vertex_* params added to the call + + + + inference-chat chat_completions_provider + gemini: GoogleAiStudioGeminiConfig + vertex_ai + Gemini model: VertexGeminiConfig + Claude, partner, Gemma on Vertex: still refused + + + + + llms vertex_ai/gemini build_request (both surfaces) + 1. sort_params: supported OpenAI params per surface and model, + unsupported refused (400) or dropped with drop_params + 2. map_openai_params: generationConfig, tools, toolConfig, + thinkingConfig, responseSchema, speechConfig, safety + 3. messages to contents: system_instruction (cost map), + images, files, audio, function_call / function_response, + thought signatures back on calls + 4. Gemini 3: a dummy signature on a call that lost its own + refused (501): cache_control, legacy functions, response_schema + without support, remote media in a tool result + + + + + URL and credential + AI Studio + {v1beta|v1alpha}/models/{m}:generateContent + api_base: {api_base}/models/{m}:generateContent + x-goog-api-key: api_key, GOOGLE_API_KEY, + GEMINI_API_KEY, PALM_API_KEY + Vertex AI + {region}-aiplatform/v1/projects/{p}/locations/ + {l}/publishers/google/models/{m}:generateContent + Bearer: forwarded, or a Google access token + from vertex_credentials, env or ADC + + + + + Gemini + generateContent: application/json + stream: :streamGenerateContent?alt=sse + data: frames, no [DONE] + + + + + response and stream (transform_generate_content_response, handler) + text, thought parts to reasoning_content and thinking_blocks + functionCall to tool_calls, signature kept in the id + groundingMetadata to annotations and vertex_ai_* metadata + finishReason mapped, tool calls make it tool_calls + blocked prompt or flagged candidate: content_filter + stream error event: its code and message end the stream + + + + + usage and cost (gateway-inference Finish) + prompt_tokens: promptTokenCount (+ tool use) + completion_tokens: candidates + thoughts when + the total does not already include them + cached_tokens: cachedContentTokenCount + config prices win, then the cost map entry + + + gateway startup (hooks::install) + a gemini or vertex_ai deployment loads the cost map + and installs it as ModelInfo, which answers + supports_system_messages, supports_reasoning, + supports_response_schema for gemini/{m} or {m} + + diff --git a/litellm-rust/docs/customer-proxy/openai-compat-chat.svg b/litellm-rust/docs/customer-proxy/openai-compat-chat.svg new file mode 100644 index 00000000000..164429964f2 --- /dev/null +++ b/litellm-rust/docs/customer-proxy/openai-compat-chat.svg @@ -0,0 +1,98 @@ + + + + + + + + POST /v1/chat/completions to an OpenAI-compatible deployment + + + + Client + OpenAI SDK body + + + + gateway-inference chat_completions + auth, hooks admit, route deployment + serves_openai: OpenAI passthrough machine, + a ReportedUsage rides on the response + + + + inference-chat compatible_chat_config + hosted_vllm, nvidia_nim, groq, together_ai, + fireworks_ai, xai, mistral, deepseek, + openrouter, cerebras, perplexity + + + + + llms openai/chat/compatible transform_request + 1. sort_params: the provider's supported params for the model, + unsupported refused (400) or dropped with drop_params + 2. a stream asks for stream_options.include_usage when the + provider takes it and the client did not + 3. map_params: max_tokens names, tool_choice, reasoning, + json_tool_call workaround, web search tools + 4. extra_body keys merged, the provider's own rewritten + 5. messages: response-only fields, content lists, names, + cache_control kept only where the provider caches + refused (501): groq streamed response_format, images in + tool messages, file_id URLs Python downloads + + + + + URL and credential + api_base: deployment, then the provider's + env var, then its default host + (hosted_vllm has no default host) + {api_base}/chat/completions, once + mistral adds /v1 to the base first + Bearer: deployment key, then env vars + hosted_vllm falls back to fake-api-key + headers: openrouter HTTP-Referer, X-Title, + perplexity X-Pplx-Integration, + fireworks x-session-affinity + os.environ/ refs resolved via the secret source + + + + + Provider + OpenAI chat.completion JSON + stream: chat.completion.chunk SSE + data: frames closed by [DONE] + + + + + response and stream (ResponseAdapter) + bytes pass through, re-encoded only when a fixup changes them + reasoning to reasoning_content, json_tool_call to content + groq service_tier, xai reasoning in completion_tokens, + perplexity citations, mistral content blocks, fireworks + tool calls in content + stream: usage recorded, hidden unless the client asked + + + + + usage and cost (gateway-inference Finish) + complete: the usage object of the body + stream: the ReportedUsage the relay recorded, + even from a chunk the client never saw + cached_tokens, reasoning, server_tool_use kept + config prices win, then the cost map entry + + + gateway startup (hooks::install) + an OpenAI-compatible deployment loads the cost map + and installs it as ModelInfo, which answers + supports_reasoning, supports_web_search, vision, + function calling, reasoning_effort_levels + for {provider}/{m} or {m} + + diff --git a/litellm-rust/docs/customer-proxy/vertex-claude-chat.svg b/litellm-rust/docs/customer-proxy/vertex-claude-chat.svg new file mode 100644 index 00000000000..df44cc0062a --- /dev/null +++ b/litellm-rust/docs/customer-proxy/vertex-claude-chat.svg @@ -0,0 +1,98 @@ + + + + + + + + POST /v1/chat/completions to a vertex_ai/ Claude deployment + + + + Client + OpenAI SDK body + + + + gateway-inference chat_completions + auth, hooks admit, route deployment + os.environ/ refs resolved via the secret source, + deployment vertex_* params added to the call + + + + inference-chat chat_completions_provider + vertex_ai + Gemini model: VertexGeminiConfig + vertex_ai + Claude partner model: VertexAiAnthropicConfig + Llama, Mistral, AI21, Gemma on Vertex: still refused + + + + + llms anthropic/chat build_messages_request (AnthropicHost::VertexAi) + 1. anthropic_version: vertex-2023-10-16 added after the params + 2. sort_params + map_openai_params: the Anthropic port; cost map + read as vertex_ai/{m} first, then the model-map walk + 3. service_tier and speed are refused (400) or dropped + with drop_params: Anthropic API only + 4. messages, system, tools, thinking as for anthropic/ + VertexAIAnthropicConfig.transform_request + model popped; output_format folded into output_config, + effort kept only where the model takes it + context_management + extra_headers betas, filtered for + Vertex, in body anthropic_beta and the anthropic-beta header + + + + + URL and credential + https://{l}-aiplatform.googleapis.com + global: https://aiplatform.googleapis.com + /v1/projects/{p}/locations/{l}/publishers/ + anthropic/models/{m@version}:rawPredict + api_base or VERTEXAI_API_BASE: path grafted, + or {api_base}:rawPredict when it has its own + location default us-central1 + Bearer: a Google access token from + vertex_credentials, env or ADC; it replaces + a forwarded Authorization, as Python does + anthropic-version: 2023-06-01 header + + + + + Claude on Vertex AI + rawPredict: Messages API JSON + stream: :streamRawPredict, "stream": true + custom api_base also gets ?alt=sse + + + + + response and stream (anthropic/chat, shared) + model: the requested {m@version}, as Python sets it + text, tool_use to tool_calls with names put back, + thinking to reasoning_content and thinking_blocks + json_tool_call answer becomes the content + stream: Anthropic SSE through ModelResponseIterator and + StreamWrapper, usage handed to ReportedUsage + + + + + usage and cost (gateway-inference Finish) + prompt_tokens: input + cache read + cache write + completion_tokens: output_tokens + streams report it whether or not the client + asked for include_usage + config prices win, then the vertex_ai/{m} entry + + + gateway startup (hooks::install) + a vertex_ai deployment loads the cost map and + installs it as ModelInfo: supports_reasoning, + supports_adaptive_thinking, supports_output_config, + supports_native_structured_output (vertex entry only), + default max_tokens: the bare {m} entry, else 4096 + + diff --git a/litellm-rust/loadtest/Dockerfile b/litellm-rust/loadtest/Dockerfile index ea8f3c7c9a0..ba42fd6be3f 100644 --- a/litellm-rust/loadtest/Dockerfile +++ b/litellm-rust/loadtest/Dockerfile @@ -1,4 +1,4 @@ -# One image with every load-test tool: mockllm, loadgen, genconfig, summarize, procmon. +# One image with every load-test tool: mockllm, loadgen, genconfig, summarize, procmon, verify, s3stub. # docker build -t litellm-loadtest litellm-rust/loadtest # docker run litellm-loadtest mockllm -addr :9000 FROM golang:1.24.7-bookworm@sha256:b8bae5bd9ba9b1f89b635c91c24cc75cea335a16fb5076310f38566fc674b1ec AS build diff --git a/litellm-rust/loadtest/README.md b/litellm-rust/loadtest/README.md index e1d98e20b47..7a03f1c3fb2 100644 --- a/litellm-rust/loadtest/README.md +++ b/litellm-rust/loadtest/README.md @@ -8,9 +8,11 @@ machines. | --- | --- | | `mockllm` | Fake OpenAI chat completions, Anthropic messages and Bedrock invoke upstream. Streams or not, and takes its timing from a directive in the prompt (`[[mock ttft_ms=800 dur_ms=300000 out=2000]]`). Counts hits per deployment (`/d//` path prefix) at `GET /stats`; `-faults "chat-0-3=429:1.0"` makes chosen deployments fail. | | `loadgen` | Open-loop Poisson arrivals from a workload file: weighted request classes with their own endpoint, stream share, context size (10k to 1M tokens), time to first token and total time. Writes one JSON line per request. Shard across machines with `-shard i -shards n`. | -| `genconfig` | Gateway `config.yaml` with many model groups and deployments pointing at the mocks, plus `keys.txt` and `keys.sql` (rows for `LiteLLM_VerificationToken`). `-key-seed` makes every machine generate the same keys. | +| `genconfig` | Gateway `config.yaml` with many model groups and deployments pointing at the mocks, plus `keys.txt` and `keys.sql` (rows for `LiteLLM_VerificationToken`). `-key-seed` makes every machine generate the same keys. The `bifrost` key set is written as a Bifrost config; see [Bifrost](#bifrost). | | `summarize` | Merges result files and prints proxy overhead (observed minus planned time) by stage, class and context size. | | `procmon` | Samples the gateway process's CPU, RSS, fds and TCP connections from `/proc`. | +| `verify` | Checks what each gateway recorded for a run against what loadgen sent: spend logs, key/team/user/member spend, daily tables, Prometheus counters, S3 request logs and the headers the provider received. See [Full-feature runs](#full-feature-runs). | +| `s3stub` | In-memory path-style S3 (PUT, GET, ListObjectsV2) for the `s3_v2` logs of a full-feature run. | Workloads live in `workloads/`: `smoke`, `realistic-ramp`, `long-streams`, `large-context`, `key-churn` and `soak`. @@ -107,6 +109,74 @@ gateway samples and mock hit counts, and prints the summary. `LITELLM_UNREADY_DELAY_SECONDS` (default 0), then open requests and streams get up to `LITELLM_DRAIN_SECONDS` (default 900, the NLB deregistration delay) to finish +## Full-feature runs + +`genconfig -features full -key-sets python,rust` turns on everything both gateways do per request, +with limits no load-test rate reaches, so every check runs and nothing is refused: + +- every key has a user and a team (with a membership budget), and keys, users, teams and members + carry rpm, tpm and budgets; keys carry `max_parallel_requests`; deployments carry rpm and tpm +- callbacks `dynamic_rate_limiter_v3`, `prometheus` and `s3_v2` (S3 settings from `LOADTEST_S3_*`), + plus the Rust gateway's four request hooks under `litellm_settings.hooks` and their Python + stand-ins in `custom_callbacks.py`, written beside the config +- `headers-.txt`: the request headers the hooks act on, one loadgen `-header` per line + +Each key set (one per gateway) has its own keys, users and teams, so both gateways can share one +Postgres and one mock and still be told apart. Every request names its input token count to the +mock (`in=`), so both sides must record the same tokens. Set `DEFAULT_S3_FLUSH_INTERVAL_SECONDS` on +both gateways to control the S3 batch interval. After the run and the spend flush: + +```sh +verify expect -out col/expected-rust.json res/rust-*.jsonl +verify sql -set rust | psql "$DATABASE_URL" -At > col/db-rust.json # same for python +verify check -dir col -s3 http://s3stub:9100/logs -mock http://mockllm:9000 \ + -side python=python-gateway:4000,python -side rust=rust-gateway:4000,rust -report report.md +``` + +`check` scrapes `/metrics` on every address the host name resolves to (one per replica) and +exits 1 when any check fails. Across sides it compares each deployment's fitted input and output +price rather than total spend, since routing spreads requests over differently priced +deployments, and it lists the metrics, labels and S3 fields one side records and the other lacks. +mockllm needs `-count-headers x-request-origin,x-trace-id,x-caller-team,x-cost-center,x-litellm-spend-logs-metadata`. + +## Bifrost + +A key set named `bifrost` (`-key-sets python,rust,bifrost`) is written as a config for +[Bifrost](https://github.com/maximhq/bifrost)'s open-source build instead of key rows: +`bifrost-config.json` (load it as the gateway's `config.json`), `bifrost-pricing.json` (mount it at +`-bifrost-pricing-file`, default `/config/bifrost-pricing.json`), `keys-bifrost.txt` and, with +`-features full`, `headers-bifrost.txt`. Each deployment is a custom provider pointing at the mock's +`/d/`, each model group a routing rule with equal-weight targets, and each key a virtual key. +Bifrost reads its config and log store from `BIFROST_PG_{HOST,PORT,USER,PASSWORD,DB}` and, with full +features, S3 from `LOADTEST_S3_*` (objects under `bifrost/`). + +Full features map onto what open-source Bifrost has: key, team and customer budgets and rate limits +(the customer stands in for LiteLLM's user; users and team members are enterprise-only), model +configs for deployment rpm and tpm, the telemetry plugin's `/metrics`, the logs store with S3 +payloads, `header_filter_config` for forward_headers and strip_headers, and `logging_headers` for +spend_metadata. provider_metadata (adding the key's team as a header) has no counterpart. +Bifrost keeps rate limits, budgets and counters in each replica's memory, with no shared store, so +it has no cross-replica state at all. + +Bifrost's Bedrock provider uses Converse over HTTPS only, so run mockllm with `-tls-addr :9443` +(a self-signed listener; genconfig points Bedrock at `-bifrost-bedrock-tls` and skips verification). +The mock echoes the request's model in responses, since Bifrost prices by the response's model. +Bifrost serves Anthropic messages at `/anthropic/v1/messages`: + +```sh +loadgen -workload workloads/smoke.json -target http://bifrost:8080 -keys out/keys-bifrost.txt \ + -messages-path /anthropic/v1/messages -messages-auth x-api-key \ + -header '' ... +sleep 15 # Bifrost writes budget usage to Postgres every 10 s +verify sql -set bifrost | psql "$BIFROST_DATABASE_URL" -At > col/db-bifrost.json +verify check ... -side bifrost=bifrost:8080,bifrost +``` + +`verify` reads a side named `bifrost` in Bifrost's schema (the `logs` and `governance_budgets` +tables, `bifrost_*` metrics, its S3 log objects) and reports what Bifrost has no counterpart for as +n/a. With three sides it compares each pair's fitted deployment prices; metric and S3 field lists +are only compared between the LiteLLM sides, since Bifrost's schema differs throughout. + ## Bedrock deployments `genconfig` serves half of each messages group through Bedrock invoke by default diff --git a/litellm-rust/loadtest/cmd/genconfig/bifrost.go b/litellm-rust/loadtest/cmd/genconfig/bifrost.go new file mode 100644 index 00000000000..5082522cc91 --- /dev/null +++ b/litellm-rust/loadtest/cmd/genconfig/bifrost.go @@ -0,0 +1,261 @@ +package main + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "log" + "path/filepath" + "strings" +) + +// bifrostSet is the key set genconfig writes as a Bifrost config instead of LiteLLM key rows. +// Bifrost (github.com/maximhq/bifrost, open-source build) boots its providers, routing and +// governance from config.json into its config store, so the seed Job loads nothing for it. +const bifrostSet = "bifrost" + +// bifrostDeployment is one deployment of the LiteLLM config as Bifrost sees it: a custom provider +// named after the mock's /d/ path, so log rows and metrics name the deployment. +type bifrostDeployment struct { + id, group, base, model string + kind string // openai, anthropic or bedrock +} + +// bifrostPrices are the cost map's prices (model_prices_and_context_window.json) for the models +// genconfig deploys. Bifrost reads LiteLLM's field names; the lane's verify compares each +// deployment's fitted price across sides, which catches drift. +var bifrostPrices = map[string]map[string]any{ + "gpt-6-astra": { + "mode": "chat", + "input_cost_per_token": 1e-05, + "output_cost_per_token": 5e-05, + "input_cost_per_token_above_272k_tokens": 2e-05, + "output_cost_per_token_above_272k_tokens": 7.5e-05, + "cache_read_input_token_cost": 1e-06, + "cache_creation_input_token_cost": 1.25e-05, + }, + "claude-opus-5-5": { + "mode": "chat", + "input_cost_per_token": 4e-06, + "output_cost_per_token": 2e-05, + "cache_read_input_token_cost": 2e-07, + "cache_creation_input_token_cost": 5e-06, + }, + "us.anthropic.claude-opus-5-5": { + "mode": "chat", + "input_cost_per_token": 4.4e-06, + "output_cost_per_token": 2.2e-05, + "cache_read_input_token_cost": 2.2e-07, + "cache_creation_input_token_cost": 5.5e-06, + }, +} + +// bifrostOptions are the knobs a fair comparison pins; deploy prints them. +type bifrostOptions struct { + full bool + keys int + keySeed string + concurrency int + bufferSize int + pricingFile string + bedrockTLS string // the mock's TLS listener, host:port + outDir string +} + +// writeBifrost writes bifrost-config.json and bifrost-pricing.json beside config.yaml, plus +// keys-bifrost.txt and, with full features, headers-bifrost.txt. Connection settings come from +// environment variables (Bifrost's env. references): BIFROST_PG_{HOST,PORT,USER,PASSWORD,DB} for +// the config and logs stores and LOADTEST_S3_* for the request log payloads, which go under the +// bifrost/ prefix. The pricing file is read from o.pricingFile, where the gateway mounts it. +func writeBifrost(deps []bifrostDeployment, o bifrostOptions) { + cfg := map[string]any{"$schema": "https://www.getbifrost.ai/schema"} + client := map[string]any{ + "initial_pool_size": o.bufferSize, + "drop_excess_requests": false, + "enable_logging": o.full, + "disable_content_logging": false, + "enforce_auth_on_inference": true, + // Like the LiteLLM configs' require_auth_for_metrics_endpoint: false, so verify can scrape. + "whitelisted_routes": []string{"/metrics"}, + } + pg := map[string]any{ + "host": "env.BIFROST_PG_HOST", "port": "env.BIFROST_PG_PORT", "user": "env.BIFROST_PG_USER", + "password": "env.BIFROST_PG_PASSWORD", "db_name": "env.BIFROST_PG_DB", "ssl_mode": "disable", + } + cfg["config_store"] = map[string]any{"enabled": true, "type": "postgres", "config": pg} + // Prices from the file genconfig writes, not getbifrost.ai; no background catalog fetches. + cfg["framework"] = map[string]any{"pricing": map[string]any{ + "pricing_url": "file://" + o.pricingFile, + "model_parameters_url": "file://" + o.pricingFile, + "pricing_sync_interval": 86400, + "mcp_library_sync_interval": 0, + "live_models_sync_interval": 0, + }} + + providers := map[string]any{} + for _, d := range deps { + var key map[string]any + network := map[string]any{ + "default_request_timeout_in_seconds": 1800, + "stream_idle_timeout_in_seconds": 600, + // LiteLLM's router_settings.num_retries; no cooldown equivalent. + "max_retries": 2, + "allow_private_network": true, + } + if d.kind == "bedrock" { + // Static keys, so every request is SigV4-signed as on the other sides. Bifrost's + // Bedrock provider takes hosts, not a base URL, and always uses HTTPS: the runtime and + // control-plane hosts are the mock's TLS listener, and the deployment travels in the + // inference profile ARN, which Bifrost puts in the path (/model/d///...). + host := strings.TrimPrefix(strings.TrimPrefix(o.bedrockTLS, "https://"), "http://") + key = map[string]any{"name": d.id, "models": []string{"*"}, "weight": 1.0, + "bedrock_key_config": map[string]any{"access_key": "mock-loadtest-access-key", + "secret_key": "mock-loadtest-secret", "region": "us-east-1", "arn": "d/" + d.id, + "endpoints": map[string]any{"runtime": host, "control_plane": host}}} + network["insecure_skip_verify"] = true + } else { + network["base_url"] = d.base + key = map[string]any{"name": d.id, "value": "sk-mock", "models": []string{"*"}, "weight": 1.0} + } + providers[d.id] = map[string]any{ + "keys": []any{key}, + "network_config": network, + "concurrency_and_buffer_size": map[string]any{"concurrency": o.concurrency, "buffer_size": o.bufferSize}, + "custom_provider_config": map[string]any{"base_provider_type": d.kind}, + } + } + cfg["providers"] = providers + + gov := map[string]any{} + // Each model group is a routing rule with its deployments as equal-weight targets, Bifrost's + // counterpart of simple-shuffle. The rule's name is the group, so log rows name it. + var rules []any + var groups []string + targets := map[string][]any{} + for _, d := range deps { + if _, ok := targets[d.group]; !ok { + groups = append(groups, d.group) + } + targets[d.group] = append(targets[d.group], map[string]any{"provider": d.id, "model": d.model}) + } + for i, g := range groups { + ts := targets[g] + for _, t := range ts { + t.(map[string]any)["weight"] = 1.0 / float64(len(ts)) + } + rules = append(rules, map[string]any{ + "id": "rule-" + g, "name": g, "enabled": true, "scope": "global", "priority": i, + "cel_expression": fmt.Sprintf("model == %q", g), "targets": ts, + }) + } + gov["routing_rules"] = rules + + prefix := setPrefix(bifrostSet) + keys := bifrostKeys(o.keys, o.keySeed) + if o.full { + var limits, teams, vks, models []any + limit := func(id string) string { + limits = append(limits, map[string]any{"id": id, + "request_max_limit": fullRPM, "request_reset_duration": "1m", + "token_max_limit": fullTPM, "token_reset_duration": "1m"}) + return id + } + // Budgets are declared inline on their owner, which is how Bifrost links them. + budget := func(id string) []any { + return []any{map[string]any{"id": id, "max_limit": fullBudget, "reset_duration": "1M"}} + } + // Bifrost's open-source hierarchy is virtual key -> team -> customer; users and team + // members are enterprise-only, so the customer stands in for the user level. + customer := prefix + "-customer" + gov["customers"] = []any{map[string]any{"id": customer, "name": customer, + "budgets": budget("b-" + customer), "rate_limit_id": limit("rl-" + customer)}} + for t := range fullTeams { + id := fmt.Sprintf("%s-team-%d", prefix, t) + teams = append(teams, map[string]any{"id": id, "name": id, "customer_id": customer, + "budgets": budget("b-" + id), "rate_limit_id": limit("rl-" + id)}) + } + for i, k := range keys { + id := fmt.Sprintf("%s-%d", prefix, i) + vks = append(vks, map[string]any{"id": id, "name": id, "value": k, "is_active": true, + "allow_all_providers": true, "team_id": fmt.Sprintf("%s-team-%d", prefix, (i%fullUsers)%fullTeams), + "budgets": budget("b-" + id), "rate_limit_id": limit("rl-" + id)}) + } + // Model capacity: each deployment's rpm and tpm, as model configs scoped to its provider. + for _, d := range deps { + models = append(models, map[string]any{"id": "mc-" + d.id, "model_name": d.model, "provider": d.id, + "rate_limit_id": limit("rl-" + d.id)}) + } + gov["rate_limits"], gov["teams"], gov["virtual_keys"], gov["model_configs"] = limits, teams, vks, models + + // forward_headers forwards these directly; strip_headers' names never leave (only the + // allowlist is forwarded) and are denied anyway. spend_metadata's nearest equivalent copies + // the raw header values into the log row's metadata. provider_metadata has none. + client["header_filter_config"] = map[string]any{ + "allowlist": []string{"x-request-origin", "x-trace-*"}, + "denylist": []string{"x-litellm-spend-logs-metadata", "x-cost-center"}, + } + client["logging_headers"] = []string{"x-litellm-spend-logs-metadata", "x-cost-center"} + cfg["logs_store"] = map[string]any{"enabled": true, "type": "postgres", "config": pg, + "object_storage": map[string]any{ + "type": "s3", "bucket": "env.LOADTEST_S3_BUCKET", "region": "us-east-1", + "endpoint": "env.LOADTEST_S3_ENDPOINT", "access_key_id": "env.LOADTEST_S3_ACCESS_KEY_ID", + "secret_access_key": "env.LOADTEST_S3_SECRET_ACCESS_KEY", "prefix": bifrostSet, + "force_path_style": true, + }} + cfg["plugins"] = []any{ + map[string]any{"enabled": true, "name": "telemetry", "config": map[string]any{}}, + } + } else { + var vks []any + for i, k := range keys { + id := fmt.Sprintf("%s-%d", prefix, i) + vks = append(vks, map[string]any{"id": id, "name": id, "value": k, "is_active": true, "allow_all_providers": true}) + } + gov["virtual_keys"] = vks + cfg["logs_store"] = map[string]any{"enabled": false} + } + cfg["governance"] = gov + cfg["client"] = client + + writeJSONFile(filepath.Join(o.outDir, "bifrost-config.json"), cfg) + // Bifrost prices a request by its provider's name, and a custom provider is not priced as its + // base provider, so bifrost-pricing.json has one row per deployment ("/"). + pricing := map[string]map[string]any{} + for _, d := range deps { + row := map[string]any{"provider": d.id} + for k, v := range bifrostPrices[d.model] { + row[k] = v + } + pricing[d.id+"/"+d.model] = row + } + writeJSONFile(filepath.Join(o.outDir, "bifrost-pricing.json"), pricing) + write(filepath.Join(o.outDir, "keys-"+bifrostSet+".txt"), strings.Join(keys, "\n")+"\n") + if o.full { + write(filepath.Join(o.outDir, "headers-"+bifrostSet+".txt"), strings.Join(fullFeatureHeaders(prefix), "\n")+"\n") + } +} + +// bifrostKeys derives the set's keys like writeKeys does, with Bifrost's sk-bf- prefix, which is +// how it tells a virtual key from a provider key. +func bifrostKeys(n int, keySeed string) []string { + keys := make([]string, 0, n) + for i := range n { + var raw [16]byte + if keySeed == "" { + keySeed = "loadtest" + } + derived := sha256.Sum256(fmt.Appendf(nil, "%s/%s/%d", keySeed, bifrostSet, i)) + copy(raw[:], derived[:]) + keys = append(keys, "sk-bf-lt-"+hex.EncodeToString(raw[:])) + } + return keys +} + +func writeJSONFile(path string, v any) { + b, err := json.MarshalIndent(v, "", " ") + if err != nil { + log.Fatal(err) + } + write(path, string(b)+"\n") +} diff --git a/litellm-rust/loadtest/cmd/genconfig/custom_callbacks.py b/litellm-rust/loadtest/cmd/genconfig/custom_callbacks.py new file mode 100644 index 00000000000..592d766f0a7 --- /dev/null +++ b/litellm-rust/loadtest/cmd/genconfig/custom_callbacks.py @@ -0,0 +1,123 @@ +"""Python proxy stand-ins for the Rust gateway's four request hooks, so a load test can run both +gateways with the same work per request. genconfig -features full lists them as +custom_callbacks. and writes this file beside the config.yaml it generates. + +Each mirrors the Rust hook configured in that config (litellm-rust/crates/gateway/src/request_hooks.rs): + + forward_headers copies x-request-origin and x-trace-* to the provider request + spend_metadata adds x-litellm-spend-logs-metadata (JSON) and x-cost-center to the + request's spend_logs_metadata; the key's cost_center wins + strip_headers drops x-litellm-spend-logs-metadata and x-cost-center from the request + provider_metadata sets metadata.user_id in the provider body and x-caller-team in its headers +""" + +import json +from typing import Any + +from litellm.integrations.custom_logger import CustomLogger +from litellm.proxy._types import UserAPIKeyAuth + +FORWARDED = ("x-request-origin",) +FORWARDED_PREFIXES = ("x-trace-",) +STRIPPED = ("x-litellm-spend-logs-metadata", "x-cost-center") + + +def _client_headers(data: dict) -> dict: + request = data.get("proxy_server_request") or {} + headers = request.get("headers") + return headers if isinstance(headers, dict) else {} + + +def _litellm_metadata(data: dict) -> dict: + # /v1/messages and other newer routes keep proxy metadata under litellm_metadata. + name = "litellm_metadata" if "litellm_metadata" in data else "metadata" + metadata = data.get(name) + if not isinstance(metadata, dict): + metadata = {} + data[name] = metadata + return metadata + + +def _extra(data: dict, field: str) -> dict: + value = data.get(field) + if not isinstance(value, dict): + value = {} + data[field] = value + return value + + +class ForwardHeaders(CustomLogger): + async def async_pre_call_hook( + self, user_api_key_dict: UserAPIKeyAuth, cache: Any, data: dict, call_type: str + ) -> dict | None: + headers = _client_headers(data) + forwarded = { + name: value + for name, value in headers.items() + if name.lower() in FORWARDED or name.lower().startswith(FORWARDED_PREFIXES) + } + if forwarded: + _extra(data, "extra_headers").update(forwarded) + return data + + +class SpendMetadata(CustomLogger): + async def async_pre_call_hook( + self, user_api_key_dict: UserAPIKeyAuth, cache: Any, data: dict, call_type: str + ) -> dict | None: + headers = {name.lower(): value for name, value in _client_headers(data).items()} + added: dict = {} + raw = headers.get("x-litellm-spend-logs-metadata") + if raw: + try: + parsed = json.loads(raw) + except ValueError: + parsed = None + if isinstance(parsed, dict): + added.update(parsed) + if headers.get("x-cost-center"): + added["cost_center"] = headers["x-cost-center"] + key_metadata = user_api_key_dict.metadata or {} + if "cost_center" in key_metadata: + added["cost_center"] = key_metadata["cost_center"] + if added: + metadata = _litellm_metadata(data) + spend = metadata.get("spend_logs_metadata") + spend = dict(spend) if isinstance(spend, dict) else {} + spend.update(added) + metadata["spend_logs_metadata"] = spend + return data + + +class StripHeaders(CustomLogger): + async def async_pre_call_hook( + self, user_api_key_dict: UserAPIKeyAuth, cache: Any, data: dict, call_type: str + ) -> dict | None: + headers = _client_headers(data) + for name in [name for name in headers if name.lower() in STRIPPED]: + headers.pop(name) + extra = data.get("extra_headers") + if isinstance(extra, dict): + for name in [name for name in extra if name.lower() in STRIPPED]: + extra.pop(name) + return data + + +class ProviderMetadata(CustomLogger): + async def async_pre_call_hook( + self, user_api_key_dict: UserAPIKeyAuth, cache: Any, data: dict, call_type: str + ) -> dict | None: + user_id = user_api_key_dict.end_user_id or user_api_key_dict.user_id + if user_id: + body_metadata = _extra(data, "extra_body").setdefault("metadata", {}) + if isinstance(body_metadata, dict): + body_metadata["user_id"] = user_id + if user_api_key_dict.team_alias: + _extra(data, "extra_headers")["x-caller-team"] = user_api_key_dict.team_alias + return data + + +forward_headers = ForwardHeaders() +spend_metadata = SpendMetadata() +strip_headers = StripHeaders() +provider_metadata = ProviderMetadata() diff --git a/litellm-rust/loadtest/cmd/genconfig/full.go b/litellm-rust/loadtest/cmd/genconfig/full.go new file mode 100644 index 00000000000..8ab2ccd665c --- /dev/null +++ b/litellm-rust/loadtest/cmd/genconfig/full.go @@ -0,0 +1,174 @@ +package main + +import ( + "bufio" + "crypto/rand" + "crypto/sha256" + _ "embed" + "encoding/hex" + "fmt" + "log" +) + +// customCallbacks holds the Python proxy's stand-ins for the Rust gateway's request hooks. +// +//go:embed custom_callbacks.py +var customCallbacks string + +// Owners per key set in full mode: key i belongs to user i%fullUsers, and user u to team +// u%fullTeams, so every user and team holds several keys. +const ( + fullUsers = 100 + fullTeams = 10 + // Limits no load-test rate reaches, so every request runs every check and none is refused. + fullRPM = 1_000_000 + fullTPM = 1_000_000_000_000 + fullBudget = 1_000_000 + fullParallel = 1000 +) + +// fullFeatureHeaders are what a client of a key set sends with -features full so each request +// hook has work: the Rust gateway's forward_headers, spend_metadata, strip_headers and +// provider_metadata, and the Python stand-ins in custom_callbacks.py, read the +// same names. Values carry the set's prefix, so mockllm -count-headers can tell sides apart. +// genconfig writes them to headers.txt (headers-.txt), one loadgen -header per line. +func fullFeatureHeaders(prefix string) []string { + return []string{ + "x-request-origin: " + prefix, + "x-trace-id: " + prefix + "-trace", + "x-cost-center: " + prefix, + `x-litellm-spend-logs-metadata: {"run": "` + prefix + `"}`, + } +} + +const deploymentCapacity = " rpm: 1000000\n tpm: 1000000000000\n" + +// The callbacks are Python's names, which the Rust gateway reads too; custom_callbacks.* are the +// Python stand-ins for the Rust request hooks, which the Rust gateway skips. +const fullFeatureSettings = `litellm_settings: + require_auth_for_metrics_endpoint: false + callbacks: + - dynamic_rate_limiter_v3 + - prometheus + - s3_v2 + - custom_callbacks.forward_headers + - custom_callbacks.spend_metadata + - custom_callbacks.strip_headers + - custom_callbacks.provider_metadata + s3_callback_params: + s3_bucket_name: os.environ/LOADTEST_S3_BUCKET + s3_region_name: us-east-1 + s3_endpoint_url: os.environ/LOADTEST_S3_ENDPOINT + s3_path: os.environ/LOADTEST_S3_PATH + s3_aws_access_key_id: os.environ/LOADTEST_S3_ACCESS_KEY_ID + s3_aws_secret_access_key: os.environ/LOADTEST_S3_SECRET_ACCESS_KEY + hooks: + - forward_headers: + headers: ["x-request-origin", "x-trace-*"] + - spend_metadata: + from_headers: + x-litellm-spend-logs-metadata: json + x-cost-center: cost_center + from_key_metadata: [cost_center] + - strip_headers: + headers: ["x-litellm-spend-logs-metadata", "x-cost-center"] + - provider_metadata: + body: + metadata.user_id: "{end_user|key.user_id}" + headers: + x-caller-team: "{key.team_alias}" +` + +// setPrefix names a key set's keys, users, teams and budgets, so sets never collide in one +// database. +func setPrefix(set string) string { + if set == "" { + return "loadtest" + } + return "loadtest-" + set +} + +// writeOwners writes a key set's member budget, teams, users and team memberships. +func writeOwners(sw *bufio.Writer, prefix string) { + fmt.Fprintf(sw, `INSERT INTO "LiteLLM_BudgetTable" (budget_id, max_budget, rpm_limit, tpm_limit, created_by, updated_by) VALUES + ('%s-member', %d, %d, %d, 'loadtest', 'loadtest') +ON CONFLICT (budget_id) DO NOTHING; +`, prefix, fullBudget, fullRPM, fullTPM) + sw.WriteString(`INSERT INTO "LiteLLM_TeamTable" (team_id, team_alias, max_budget, rpm_limit, tpm_limit, metadata) VALUES` + "\n") + for t := range fullTeams { + if t > 0 { + sw.WriteString(",\n") + } + fmt.Fprintf(sw, ` ('%s-team-%d', '%s-team-%d', %d, %d, %d, '{"cost_center": "team-%d"}')`, + prefix, t, prefix, t, fullBudget, fullRPM, fullTPM, t) + } + sw.WriteString("\nON CONFLICT (team_id) DO NOTHING;\n") + sw.WriteString(`INSERT INTO "LiteLLM_UserTable" (user_id, user_alias, max_budget, rpm_limit, tpm_limit, teams) VALUES` + "\n") + for u := range fullUsers { + if u > 0 { + sw.WriteString(",\n") + } + fmt.Fprintf(sw, ` ('%s-user-%d', '%s-user-%d', %d, %d, %d, ARRAY['%s-team-%d'])`, + prefix, u, prefix, u, fullBudget, fullRPM, fullTPM, prefix, u%fullTeams) + } + sw.WriteString("\nON CONFLICT (user_id) DO NOTHING;\n") + sw.WriteString(`INSERT INTO "LiteLLM_TeamMembership" (user_id, team_id, budget_id) VALUES` + "\n") + for u := range fullUsers { + if u > 0 { + sw.WriteString(",\n") + } + fmt.Fprintf(sw, ` ('%s-user-%d', '%s-team-%d', '%s-member')`, prefix, u, prefix, u%fullTeams, prefix) + } + sw.WriteString("\nON CONFLICT (user_id, team_id) DO NOTHING;\n") +} + +// writeKeys writes one key set: raw keys to kw, hashed rows to sw, with owners and limits in +// full mode. +func writeKeys(kw, sw *bufio.Writer, set string, keys int, keySeed string, full bool) { + prefix := setPrefix(set) + if full && keys > 0 { + writeOwners(sw, prefix) + } + const batch = 1000 + for i := range keys { + raw := make([]byte, 16) + if keySeed == "" { + if _, err := rand.Read(raw); err != nil { + log.Fatal(err) + } + } else { + // Predictable by design, for load tests only. The default set keeps its old keys. + seed := keySeed + if set != "" { + seed += "/" + set + } + derived := sha256.Sum256(fmt.Appendf(nil, "%s/%d", seed, i)) + copy(raw, derived[:]) + } + key := "sk-lt-" + hex.EncodeToString(raw) + sum := sha256.Sum256([]byte(key)) + fmt.Fprintln(kw, key) + if i%batch == 0 { + if i > 0 { + sw.WriteString("\nON CONFLICT (token) DO NOTHING;\n") + } + if full { + sw.WriteString(`INSERT INTO "LiteLLM_VerificationToken" (token, key_alias, models, user_id, team_id, rpm_limit, tpm_limit, max_parallel_requests, max_budget, metadata) VALUES` + "\n") + } else { + sw.WriteString(`INSERT INTO "LiteLLM_VerificationToken" (token, key_alias, models) VALUES` + "\n") + } + } else { + sw.WriteString(",\n") + } + if full { + u := i % fullUsers + fmt.Fprintf(sw, ` ('%s', '%s-%d', ARRAY[]::text[], '%s-user-%d', '%s-team-%d', %d, %d, %d, %d, '{"cost_center": "key-%d"}')`, + hex.EncodeToString(sum[:]), prefix, i, prefix, u, prefix, u%fullTeams, fullRPM, fullTPM, fullParallel, fullBudget, i) + } else { + fmt.Fprintf(sw, " ('%s', '%s-%d', ARRAY[]::text[])", hex.EncodeToString(sum[:]), prefix, i) + } + } + if keys > 0 { + sw.WriteString("\nON CONFLICT (token) DO NOTHING;\n") + } +} diff --git a/litellm-rust/loadtest/cmd/genconfig/main.go b/litellm-rust/loadtest/cmd/genconfig/main.go index 69285470ca8..ad92b7ca252 100644 --- a/litellm-rust/loadtest/cmd/genconfig/main.go +++ b/litellm-rust/loadtest/cmd/genconfig/main.go @@ -2,17 +2,25 @@ // plus a set of virtual keys: // // config.yaml model_list + router_settings + master key -// keys.txt one virtual key per line, for loadgen +// keys.txt one virtual key per line, for loadgen (keys-.txt with -key-sets) // keys.sql the same keys, hashed, as rows of LiteLLM_VerificationToken // // Deployment api_bases carry /d/ so mockllm can count hits and inject faults per deployment. +// +// -features full turns on everything the gateways enforce and record per request, set so nothing +// is refused at load-test rates: every key belongs to a user and a team (with a membership budget) +// and every owner carries rpm, tpm and budget limits; deployments carry rpm and tpm for model +// capacity; and the config lists the dynamic_rate_limiter_v3, prometheus and s3_v2 callbacks, the +// Rust gateway's request hooks and their Python stand-ins, written beside the config as +// custom_callbacks.py. headers.txt (headers-.txt) holds the request headers those hooks act +// on. S3 settings come from LOADTEST_S3_* environment variables. +// +// -key-sets python,rust writes one key set, with its own users and teams, per name, so each +// gateway's spend can be told apart in a shared database. package main import ( "bufio" - "crypto/rand" - "crypto/sha256" - "encoding/hex" "flag" "fmt" "log" @@ -36,7 +44,21 @@ func main() { keySeed := flag.String("key-seed", "", "derive keys from this seed so every machine generates the same set (default: random keys)") masterKey := flag.String("master-key", "sk-loadtest-master", "gateway master key") outDir := flag.String("out", ".", "output directory") + features := flag.String("features", "basic", "basic (virtual keys only) or full (limits, budgets, callbacks and hooks)") + keySets := flag.String("key-sets", "", "comma-separated key set names, each written to keys-.txt with its own users and teams (default: one set in keys.txt); the set named bifrost becomes bifrost-config.json") + bifrostConcurrency := flag.Int("bifrost-concurrency", 500, "Bifrost workers per deployment (each deployment is a provider with its own worker pool)") + bifrostBuffer := flag.Int("bifrost-buffer", 5000, "Bifrost queue length per deployment, and its client pool size") + bifrostBedrock := flag.String("bifrost-bedrock-tls", "127.0.0.1:9443", "host:port of mockllm -tls-addr, where Bifrost's Bedrock deployments go") + bifrostPricing := flag.String("bifrost-pricing-file", "/config/bifrost-pricing.json", "where the Bifrost gateway finds bifrost-pricing.json") flag.Parse() + var full bool + switch *features { + case "basic": + case "full": + full = true + default: + log.Fatalf("-features must be basic or full, got %q", *features) + } if err := os.MkdirAll(*outDir, 0o755); err != nil { log.Fatal(err) @@ -45,17 +67,29 @@ func main() { var cfg strings.Builder cfg.WriteString("# Generated by loadtest/cmd/genconfig. Every deployment points at mockllm.\nmodel_list:\n") n := 0 + var bifrostDeps []bifrostDeployment deployment := func(group, id, model, apiBase string) { + kind, upstream, _ := strings.Cut(model, "/") + bifrostDeps = append(bifrostDeps, bifrostDeployment{id: id, group: group, kind: kind, model: upstream, + base: strings.TrimSuffix(apiBase, "/v1")}) fmt.Fprintf(&cfg, " - model_name: %s\n litellm_params:\n model: %s\n api_base: %s\n api_key: sk-mock\n", group, model, apiBase) + if full { + cfg.WriteString(deploymentCapacity) + } n++ } // Bedrock deployments carry static AWS keys instead of an api_key (which Bedrock reads as a // bearer token), so every request goes through SigV4 signing as it does with IAM credentials. bedrockDeployment := func(group, id, model, apiBase string) { + bifrostDeps = append(bifrostDeps, bifrostDeployment{id: id, group: group, kind: "bedrock", + model: strings.TrimPrefix(model, "bedrock/"), base: apiBase}) fmt.Fprintf(&cfg, " - model_name: %s\n litellm_params:\n model: %s\n api_base: %s\n"+ " aws_access_key_id: mock-loadtest-access-key\n aws_secret_access_key: mock-loadtest-secret\n aws_region_name: us-east-1\n", group, model, apiBase) + if full { + cfg.WriteString(deploymentCapacity) + } n++ } for g := range *chatGroups { @@ -87,51 +121,52 @@ func main() { general_settings: master_key: %s `, *masterKey) + if full { + cfg.WriteString(fullFeatureSettings) + write(filepath.Join(*outDir, "custom_callbacks.py"), customCallbacks) + } write(filepath.Join(*outDir, "config.yaml"), cfg.String()) - kf, err := os.Create(filepath.Join(*outDir, "keys.txt")) - if err != nil { - log.Fatal(err) + sets := []string{""} + if *keySets != "" { + sets = strings.Split(*keySets, ",") } sf, err := os.Create(filepath.Join(*outDir, "keys.sql")) if err != nil { log.Fatal(err) } - kw, sw := bufio.NewWriter(kf), bufio.NewWriter(sf) + sw := bufio.NewWriter(sf) sw.WriteString("-- Load-test virtual keys. Hash = sha256 hex of the key, as the proxy stores it.\n") - const batch = 1000 - for i := range *keys { - raw := make([]byte, 16) - if *keySeed == "" { - if _, err := rand.Read(raw); err != nil { - log.Fatal(err) - } - } else { - // Predictable by design, for load tests only. - derived := sha256.Sum256(fmt.Appendf(nil, "%s/%d", *keySeed, i)) - copy(raw, derived[:]) + for _, set := range sets { + if set == bifrostSet { + writeBifrost(bifrostDeps, bifrostOptions{full: full, keys: *keys, keySeed: *keySeed, + concurrency: *bifrostConcurrency, bufferSize: *bifrostBuffer, pricingFile: *bifrostPricing, bedrockTLS: *bifrostBedrock, outDir: *outDir}) + continue } - key := "sk-lt-" + hex.EncodeToString(raw) - sum := sha256.Sum256([]byte(key)) - fmt.Fprintln(kw, key) - if i%batch == 0 { - if i > 0 { - sw.WriteString("\nON CONFLICT (token) DO NOTHING;\n") - } - sw.WriteString(`INSERT INTO "LiteLLM_VerificationToken" (token, key_alias, models) VALUES` + "\n") - } else { - sw.WriteString(",\n") + name := "keys.txt" + if set != "" { + name = "keys-" + set + ".txt" } - fmt.Fprintf(sw, " ('%s', 'loadtest-%d', ARRAY[]::text[])", hex.EncodeToString(sum[:]), i) + kf, err := os.Create(filepath.Join(*outDir, name)) + if err != nil { + log.Fatal(err) + } + kw := bufio.NewWriter(kf) + writeKeys(kw, sw, set, *keys, *keySeed, full) + if full { + headers := "headers.txt" + if set != "" { + headers = "headers-" + set + ".txt" + } + write(filepath.Join(*outDir, headers), strings.Join(fullFeatureHeaders(setPrefix(set)), "\n")+"\n") + } + kw.Flush() + kf.Close() } - if *keys > 0 { - sw.WriteString("\nON CONFLICT (token) DO NOTHING;\n") - } - kw.Flush() sw.Flush() - kf.Close() sf.Close() - log.Printf("wrote %d deployments in %d groups and %d keys to %s", n, *chatGroups+*messagesGroups, *keys, *outDir) + log.Printf("wrote %d deployments in %d groups, %d keys in each of %d sets (%s features) to %s", + n, *chatGroups+*messagesGroups, *keys, len(sets), *features, *outDir) } func write(path, content string) { diff --git a/litellm-rust/loadtest/cmd/loadgen/body.go b/litellm-rust/loadtest/cmd/loadgen/body.go index d8d24e39a43..0f4959f8012 100644 --- a/litellm-rust/loadtest/cmd/loadgen/body.go +++ b/litellm-rust/loadtest/cmd/loadgen/body.go @@ -79,7 +79,7 @@ func buildBody(pool textPool, r *rand.Rand, res result, outTokens int, nonce str remaining -= n if remaining <= 0 && role == "user" { // The directive goes last so mockllm finds it by scanning only the tail. - fmt.Fprintf(&b, " [[mock ttft_ms=%d dur_ms=%d out=%d]]", res.TTFTPlanMs, res.DurPlanMs, outTokens) + fmt.Fprintf(&b, " [[mock ttft_ms=%d dur_ms=%d out=%d in=%d]]", res.TTFTPlanMs, res.DurPlanMs, outTokens, res.CtxTokens) b.WriteString(`"}`) break } diff --git a/litellm-rust/loadtest/cmd/loadgen/main.go b/litellm-rust/loadtest/cmd/loadgen/main.go index e503741db9c..8b2322deb90 100644 --- a/litellm-rust/loadtest/cmd/loadgen/main.go +++ b/litellm-rust/loadtest/cmd/loadgen/main.go @@ -67,6 +67,11 @@ func main() { key := flag.String("key", "", "use this one key for every request instead of the keys file") messagesAuth := flag.String("messages-auth", "bearer", "how /v1/messages sends the key: bearer or x-api-key") seed := flag.Uint64("seed", 1, "random seed (combined with the shard index)") + keysFile := flag.String("keys", "", "keys file to use instead of the workload's keys.file") + chatPath := flag.String("chat-path", "/v1/chat/completions", "path of the chat endpoint under -target") + messagesPath := flag.String("messages-path", "/v1/messages", "path of the messages endpoint under -target (Bifrost serves it at /anthropic/v1/messages)") + var headers headerFlags + flag.Var(&headers, "header", "extra request header as 'Name: value' (repeatable)") flag.Parse() w, err := loadWorkload(*workloadPath) @@ -75,7 +80,11 @@ func main() { } keys := []string{*key} if *key == "" { - keys, err = readLines(w.Keys.File) + file := w.Keys.File + if *keysFile != "" { + file = *keysFile + } + keys, err = readLines(file) if err != nil { log.Fatalf("keys: %v (or pass -key)", err) } @@ -221,7 +230,7 @@ schedule: go func() { defer wg.Done() defer c.inflight.Add(-1) - do(reqCtx, transport, *target, keys[keyIdx], *messagesAuth, body, &res) + do(reqCtx, transport, *target, paths{*chatPath, *messagesPath}, keys[keyIdx], *messagesAuth, headers, body, &res) if res.OK { c.ok.Add(1) } else { @@ -252,17 +261,37 @@ schedule: func seconds(s float64) time.Duration { return time.Duration(s * float64(time.Second)) } -func do(ctx context.Context, transport *http.Transport, target, key, messagesAuth string, body []byte, res *result) { +// headerFlags collects repeated -header 'Name: value' flags. +type headerFlags [][2]string + +func (h *headerFlags) String() string { return fmt.Sprint(*h) } + +func (h *headerFlags) Set(value string) error { + name, v, ok := strings.Cut(value, ":") + if !ok || strings.TrimSpace(name) == "" { + return fmt.Errorf("want 'Name: value', got %q", value) + } + *h = append(*h, [2]string{strings.TrimSpace(name), strings.TrimSpace(v)}) + return nil +} + +// paths are the chat and messages endpoints under the target. +type paths struct{ chat, messages string } + +func do(ctx context.Context, transport *http.Transport, target string, p paths, key, messagesAuth string, headers headerFlags, body []byte, res *result) { limit := time.Duration(res.DurPlanMs)*time.Millisecond + 3*time.Minute ctx, cancel := context.WithTimeout(ctx, limit) defer cancel() - path := "/v1/chat/completions" + path := p.chat if res.Endpoint == "messages" { - path = "/v1/messages" + path = p.messages } req, _ := http.NewRequestWithContext(ctx, http.MethodPost, target+path, bytes.NewReader(body)) req.Header.Set("content-type", "application/json") + for _, h := range headers { + req.Header.Add(h[0], h[1]) + } if res.Endpoint == "messages" { req.Header.Set("anthropic-version", "2023-06-01") if messagesAuth == "x-api-key" { diff --git a/litellm-rust/loadtest/cmd/mockllm/main.go b/litellm-rust/loadtest/cmd/mockllm/main.go index 8dc30473868..da9f6ae7e2f 100644 --- a/litellm-rust/loadtest/cmd/mockllm/main.go +++ b/litellm-rust/loadtest/cmd/mockllm/main.go @@ -1,20 +1,36 @@ // mockllm is a fake LLM upstream for load tests. It speaks OpenAI chat completions, Anthropic -// messages and Bedrock's Anthropic invoke API, streaming and non-streaming, and takes its timing from a directive +// messages, Bedrock's Anthropic invoke API and Bedrock Converse (which Bifrost calls instead of +// invoke), streaming and non-streaming, and takes its timing from a directive // the load generator embeds in the prompt, so the proxy in between forwards it untouched: // -// [[mock ttft_ms=800 dur_ms=300000 out=2000 status=200]] +// [[mock ttft_ms=800 dur_ms=300000 out=2000 in=10000 status=200]] // // ttft_ms is the delay before the first byte, dur_ms the total response time, out the -// number of output tokens, status an HTTP status to fail with. Without a directive the +// number of output tokens, in the input tokens to report (a quarter of the body's bytes when +// absent, which varies with how each proxy serializes the request), status an HTTP status to +// fail with. Without a directive the // mock answers at once. // // Paths may carry a deployment prefix, /d//..., which is how one mock process stands in // for many deployments. Hits are counted per deployment at GET /stats, and -faults makes -// chosen deployments fail, e.g. -faults "chat-0-3=429:1.0,msg-1-0=500:0.2". +// chosen deployments fail, e.g. -faults "chat-0-3=429:1.0,msg-1-0=500:0.2". -count-headers +// counts the requests carrying each named header, by value, so a test can check what a proxy +// forwarded, added or stripped. +// +// Bifrost's Bedrock provider takes a host rather than a base URL and always speaks HTTPS, so +// -tls-addr also serves everything over TLS with a throwaway self-signed certificate, and a +// Bedrock path of the form /model/d///... (the deployment carried in the inference +// profile ARN) counts as deployment . package main import ( "bytes" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" "encoding/base64" "encoding/binary" "encoding/json" @@ -23,7 +39,8 @@ import ( "hash/crc32" "io" "log" - "math/rand/v2" + "math/big" + mrand "math/rand/v2" "net/http" "strconv" "strings" @@ -43,6 +60,8 @@ type stats struct { faults map[string]int64 inflight atomic.Int64 bytesIn atomic.Int64 + // headers counts requests by header name, then value, for the -count-headers names. + headers map[string]map[string]int64 } func (s *stats) hit(dep string, faulted bool) { @@ -58,11 +77,18 @@ type directive struct { ttft time.Duration dur time.Duration out int + in int status int } var directiveMarker = []byte("[[mock ") +// responseSeq numbers responses; responseRun keeps ids unique across mock restarts. +var ( + responseSeq atomic.Uint64 + responseRun = mrand.Uint32() +) + // findDirective looks in the tail first, where the load generator puts it, and only scans the // whole body when a proxy has reordered fields. func findDirective(body []byte) directive { @@ -100,6 +126,8 @@ func findDirective(body []byte) directive { d.dur = time.Duration(n) * time.Millisecond case "out": d.out = max(n, 1) + case "in": + d.in = max(n, 0) case "status": d.status = n } @@ -115,6 +143,10 @@ func deploymentOf(path string) (string, string) { id, tail, _ := strings.Cut(rest, "/") return id, "/" + tail } + if rest, ok := strings.CutPrefix(path, "/model/d/"); ok { + id, tail, _ := strings.Cut(rest, "/") + return id, "/model/" + tail + } return "default", path } @@ -139,6 +171,8 @@ func main() { addr := flag.String("addr", ":9000", "listen address") faultSpec := flag.String("faults", "", "deployment=status:rate,... failures to inject") tokensPerChunk := flag.Int("tokens-per-chunk", 4, "output tokens per streamed chunk") + countHeaders := flag.String("count-headers", "", "comma-separated request headers to count by value at /stats") + tlsAddr := flag.String("tls-addr", "", "also serve over TLS here, with a self-signed certificate (for Bifrost's Bedrock provider)") flag.Parse() faults := map[string]fault{} @@ -156,14 +190,21 @@ func main() { faults[dep] = fault{status: status, rate: r} } - st := &stats{hits: map[string]int64{}, faults: map[string]int64{}} + st := &stats{hits: map[string]int64{}, faults: map[string]int64{}, headers: map[string]map[string]int64{}} + var counted []string + for _, name := range strings.Split(*countHeaders, ",") { + if name = strings.ToLower(strings.TrimSpace(name)); name != "" { + counted = append(counted, name) + st.headers[name] = map[string]int64{} + } + } mux := http.NewServeMux() mux.HandleFunc("GET /stats", func(w http.ResponseWriter, r *http.Request) { st.mu.Lock() defer st.mu.Unlock() w.Header().Set("content-type", "application/json") json.NewEncoder(w).Encode(map[string]any{ - "hits": st.hits, "faults": st.faults, + "hits": st.hits, "faults": st.faults, "headers": st.headers, "inflight": st.inflight.Load(), "bytes_in": st.bytesIn.Load(), }) }) @@ -183,11 +224,22 @@ func main() { body := buf.Bytes() d := findDirective(body) f, faulty := faults[dep] - injected := faulty && rand.Float64() < f.rate + injected := faulty && mrand.Float64() < f.rate st.hit(dep, injected) + if len(counted) > 0 { + st.mu.Lock() + for _, name := range counted { + for _, value := range r.Header.Values(name) { + st.headers[name][value]++ + } + } + st.mu.Unlock() + } - bedrockStream := strings.HasSuffix(path, "/invoke-with-response-stream") - bedrock := bedrockStream || strings.HasSuffix(path, "/invoke") + converseStream := strings.HasSuffix(path, "/converse-stream") + converse := converseStream || strings.HasSuffix(path, "/converse") + bedrockStream := converseStream || strings.HasSuffix(path, "/invoke-with-response-stream") + bedrock := bedrockStream || converse || strings.HasSuffix(path, "/invoke") anthropic := bedrock || strings.HasSuffix(path, "/messages") if !anthropic && !strings.HasSuffix(path, "/chat/completions") { http.Error(w, "unknown path "+path, http.StatusNotFound) @@ -206,6 +258,14 @@ func main() { stream := bedrockStream || bytes.Contains(body, []byte(`"stream":true`)) || bytes.Contains(body, []byte(`"stream": true`)) usage := bytes.Contains(body, []byte(`"include_usage":true`)) || bytes.Contains(body, []byte(`"include_usage": true`)) inputTokens := len(body) / 4 + // Unique per response, as providers' are: the Python proxy keys spend logs on it. + seq := responseSeq.Add(1) + msgID, chatID := fmt.Sprintf("msg_mock%08x%d", responseRun, seq), fmt.Sprintf("chatcmpl-mock%08x%d", responseRun, seq) + if d.in > 0 { + inputTokens = d.in + } + // Echo the model like a provider does: Bifrost prices a response by the model it names. + model := requestModel(body) if !stream { if !sleep(r, d.dur) { @@ -213,16 +273,23 @@ func main() { } w.Header().Set("content-type", "application/json") text := strings.Repeat(word, d.out) - if anthropic { + if converse { json.NewEncoder(w).Encode(map[string]any{ - "id": "msg_mock", "type": "message", "role": "assistant", "model": "mock", + "output": map[string]any{"message": map[string]any{"role": "assistant", "content": []any{map[string]any{"text": text}}}}, + "stopReason": "end_turn", + "usage": map[string]any{"inputTokens": inputTokens, "outputTokens": d.out, "totalTokens": inputTokens + d.out}, + "metrics": map[string]any{"latencyMs": d.dur.Milliseconds()}, + }) + } else if anthropic { + json.NewEncoder(w).Encode(map[string]any{ + "id": msgID, "type": "message", "role": "assistant", "model": model, "content": []any{map[string]any{"type": "text", "text": text}}, "stop_reason": "end_turn", "stop_sequence": nil, "usage": map[string]any{"input_tokens": inputTokens, "output_tokens": d.out}, }) } else { json.NewEncoder(w).Encode(map[string]any{ - "id": "chatcmpl-mock", "object": "chat.completion", "created": time.Now().Unix(), "model": "mock", + "id": chatID, "object": "chat.completion", "created": time.Now().Unix(), "model": model, "choices": []any{map[string]any{"index": 0, "finish_reason": "stop", "message": map[string]any{"role": "assistant", "content": text}}}, "usage": map[string]any{"prompt_tokens": inputTokens, "completion_tokens": d.out, "total_tokens": inputTokens + d.out}, @@ -248,11 +315,18 @@ func main() { start := time.Now() var event func(name string, data any) - if bedrock { + if converse { + // Converse events are typed by the frame's :event-type, with the JSON as the payload. + event = func(name string, data any) { + b, _ := json.Marshal(data) + w.Write(eventStreamFrame(name, b)) + flusher.Flush() + } + } else if bedrock { event = func(_ string, data any) { b, _ := json.Marshal(data) payload, _ := json.Marshal(map[string]string{"bytes": base64.StdEncoding.EncodeToString(b)}) - w.Write(eventStreamFrame(payload)) + w.Write(eventStreamFrame("chunk", payload)) flusher.Flush() } } else if anthropic { @@ -268,16 +342,18 @@ func main() { flusher.Flush() } } - if anthropic { + if converse { + event("messageStart", map[string]any{"role": "assistant"}) + } else if anthropic { event("message_start", map[string]any{"type": "message_start", "message": map[string]any{ - "id": "msg_mock", "type": "message", "role": "assistant", "model": "mock", "content": []any{}, + "id": msgID, "type": "message", "role": "assistant", "model": model, "content": []any{}, "stop_reason": nil, "stop_sequence": nil, "usage": map[string]any{"input_tokens": inputTokens, "output_tokens": 1}}}) event("content_block_start", map[string]any{"type": "content_block_start", "index": 0, "content_block": map[string]any{"type": "text", "text": ""}}) } created := time.Now().Unix() openaiChunk := func(delta map[string]any, finish any) map[string]any { - return map[string]any{"id": "chatcmpl-mock", "object": "chat.completion.chunk", "created": created, "model": "mock", + return map[string]any{"id": chatID, "object": "chat.completion.chunk", "created": created, "model": model, "choices": []any{map[string]any{"index": 0, "delta": delta, "finish_reason": finish}}} } for i := range chunks { @@ -285,7 +361,9 @@ func main() { if !sleep(r, time.Until(start.Add(interval*time.Duration(i)))) { return } - if anthropic { + if converse { + event("contentBlockDelta", map[string]any{"contentBlockIndex": 0, "delta": map[string]any{"text": piece}}) + } else if anthropic { event("content_block_delta", map[string]any{"type": "content_block_delta", "index": 0, "delta": map[string]any{"type": "text_delta", "text": piece}}) } else { @@ -295,6 +373,14 @@ func main() { if !sleep(r, time.Until(start.Add(d.dur-d.ttft))) { return } + if converse { + event("contentBlockStop", map[string]any{"contentBlockIndex": 0}) + event("messageStop", map[string]any{"stopReason": "end_turn"}) + event("metadata", map[string]any{ + "usage": map[string]any{"inputTokens": inputTokens, "outputTokens": d.out, "totalTokens": inputTokens + d.out}, + "metrics": map[string]any{"latencyMs": d.dur.Milliseconds()}}) + return + } if anthropic { event("content_block_stop", map[string]any{"type": "content_block_stop", "index": 0}) event("message_delta", map[string]any{"type": "message_delta", @@ -305,7 +391,7 @@ func main() { } event("", openaiChunk(map[string]any{}, "stop")) if usage { - event("", map[string]any{"id": "chatcmpl-mock", "object": "chat.completion.chunk", "created": created, "model": "mock", + event("", map[string]any{"id": chatID, "object": "chat.completion.chunk", "created": created, "model": model, "choices": []any{}, "usage": map[string]any{"prompt_tokens": inputTokens, "completion_tokens": d.out, "total_tokens": inputTokens + d.out}}) } io.WriteString(w, "data: [DONE]\n\n") @@ -313,6 +399,12 @@ func main() { }) server := &http.Server{Addr: *addr, Handler: mux, IdleTimeout: 15 * time.Minute, ReadHeaderTimeout: 30 * time.Second} + if *tlsAddr != "" { + tlsServer := &http.Server{Addr: *tlsAddr, Handler: mux, IdleTimeout: 15 * time.Minute, ReadHeaderTimeout: 30 * time.Second, + TLSConfig: &tls.Config{Certificates: []tls.Certificate{selfSigned()}}} + go func() { log.Fatal(tlsServer.ListenAndServeTLS("", "")) }() + log.Printf("mockllm listening on %s (TLS)", *tlsAddr) + } log.Printf("mockllm listening on %s (faults: %v)", *addr, faults) log.Fatal(server.ListenAndServe()) } @@ -335,10 +427,11 @@ func writeError(w http.ResponseWriter, anthropic bool, status int) { } // eventStreamFrame encodes one AWS event-stream message, the framing of Bedrock's -// invoke-with-response-stream: prelude (lengths + CRC), headers, payload, message CRC. -func eventStreamFrame(payload []byte) []byte { +// invoke-with-response-stream and converse-stream: prelude (lengths + CRC), headers, payload, +// message CRC. +func eventStreamFrame(eventType string, payload []byte) []byte { var headers bytes.Buffer - for _, h := range [][2]string{{":event-type", "chunk"}, {":content-type", "application/json"}, {":message-type", "event"}} { + for _, h := range [][2]string{{":event-type", eventType}, {":content-type", "application/json"}, {":message-type", "event"}} { headers.WriteByte(byte(len(h[0]))) headers.WriteString(h[0]) headers.WriteByte(7) @@ -354,3 +447,38 @@ func eventStreamFrame(payload []byte) []byte { frame = append(frame, payload...) return binary.BigEndian.AppendUint32(frame, crc32.ChecksumIEEE(frame)) } + +// selfSigned makes a throwaway certificate; clients of -tls-addr skip verification. +func selfSigned() tls.Certificate { + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + log.Fatal(err) + } + tmpl := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "mockllm"}, + DNSNames: []string{"*"}, NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(365 * 24 * time.Hour)} + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + if err != nil { + log.Fatal(err) + } + return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: key} +} + +// requestModel is the request body's top-level "model" string, or "mock" when it has none +// (Bedrock carries the model in the path). A cheap scan: model is a short field near the start. +func requestModel(body []byte) string { + i := bytes.Index(body, []byte(`"model"`)) + if i < 0 { + return "mock" + } + rest, ok := bytes.CutPrefix(bytes.TrimLeft(body[i+len(`"model"`):], " \t\r\n"), []byte(":")) + if !ok { + return "mock" + } + if end := bytes.IndexAny(rest, ",}"); end > 0 { + var model string + if json.Unmarshal(bytes.TrimSpace(rest[:end]), &model) == nil && model != "" { + return model + } + } + return "mock" +} diff --git a/litellm-rust/loadtest/cmd/mockllm/main_test.go b/litellm-rust/loadtest/cmd/mockllm/main_test.go new file mode 100644 index 00000000000..a503e8b3866 --- /dev/null +++ b/litellm-rust/loadtest/cmd/mockllm/main_test.go @@ -0,0 +1,27 @@ +package main + +import "testing" + +func TestRequestModelReadsTheTopLevelModel(t *testing.T) { + for body, want := range map[string]string{ + `{"model": "gpt-6-astra", "messages":[{"content":"say \"model\": \"x\""}]}`: "gpt-6-astra", + `{"messages":[],"model":"claude-opus-5-5"}`: "claude-opus-5-5", + `{"messages":[]}`: "mock", + } { + if got := requestModel([]byte(body)); got != want { + t.Fatalf("requestModel(%s) = %q, want %q", body, got, want) + } + } +} + +func TestDeploymentOfReadsBedrockModelPaths(t *testing.T) { + for path, want := range map[string][2]string{ + "/d/chat-0-1/v1/chat/completions": {"chat-0-1", "/v1/chat/completions"}, + "/model/d/bedrock-0-0/us.anthropic.claude/converse-stream": {"bedrock-0-0", "/model/us.anthropic.claude/converse-stream"}, + "/v1/messages": {"default", "/v1/messages"}, + } { + if id, rest := deploymentOf(path); id != want[0] || rest != want[1] { + t.Fatalf("deploymentOf(%s) = %s, %s", path, id, rest) + } + } +} diff --git a/litellm-rust/loadtest/cmd/s3stub/main.go b/litellm-rust/loadtest/cmd/s3stub/main.go new file mode 100644 index 00000000000..679cb94f4f6 --- /dev/null +++ b/litellm-rust/loadtest/cmd/s3stub/main.go @@ -0,0 +1,164 @@ +// s3stub is an in-memory, path-style S3 endpoint for load tests: PUT stores an object, GET reads +// it, and GET /?list-type=2 lists keys (ListObjectsV2, 1,000 per page). It checks no +// credentials, so gateways can sign their uploads as they would for AWS and verify can read them +// back without any. Objects live until the process exits. +// +// -max-string N shortens every JSON string longer than N bytes in a stored object to its first N +// bytes, so request logs keep their fields and numbers but not whole prompts, and a long run's logs +// fit in memory. Objects that are not JSON are stored as sent. +// +// s3stub -addr :9100 -max-string 256 +package main + +import ( + "bytes" + "encoding/json" + "encoding/xml" + "flag" + "io" + "log" + "net/http" + "sort" + "strconv" + "strings" + "sync" +) + +type store struct { + mu sync.RWMutex + objects map[string][]byte // "/" + maxString int +} + +type listResult struct { + XMLName xml.Name `xml:"ListBucketResult"` + Name string `xml:"Name"` + Prefix string `xml:"Prefix"` + KeyCount int `xml:"KeyCount"` + IsTruncated bool `xml:"IsTruncated"` + NextContinuationToken string `xml:"NextContinuationToken,omitempty"` + Contents []entry `xml:"Contents"` +} + +type entry struct { + Key string `xml:"Key"` + Size int `xml:"Size"` +} + +const page = 1000 + +func main() { + addr := flag.String("addr", ":9100", "listen address") + maxString := flag.Int("max-string", 0, "shorten JSON strings longer than this many bytes (0 keeps objects as sent)") + flag.Parse() + s := &store{objects: map[string][]byte{}, maxString: *maxString} + http.HandleFunc("GET /health", func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "ok") }) + http.HandleFunc("/", s.serve) + log.Printf("s3stub listening on %s", *addr) + log.Fatal(http.ListenAndServe(*addr, nil)) +} + +func (s *store) serve(w http.ResponseWriter, r *http.Request) { + path := strings.TrimPrefix(r.URL.Path, "/") + bucket, key, _ := strings.Cut(path, "/") + if bucket == "" { + http.Error(w, "no bucket", http.StatusBadRequest) + return + } + switch { + case r.Method == http.MethodPut && key != "": + body, err := io.ReadAll(r.Body) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + if s.maxString > 0 { + body = shorten(body, s.maxString) + } + s.mu.Lock() + s.objects[bucket+"/"+key] = body + s.mu.Unlock() + w.Header().Set("ETag", `"stub"`) + case r.Method == http.MethodGet && key == "": + s.list(w, r, bucket) + case r.Method == http.MethodHead && key == "": + // HeadBucket: every bucket exists. Bifrost checks its log bucket at startup. + case (r.Method == http.MethodGet || r.Method == http.MethodHead) && key != "": + s.mu.RLock() + body, ok := s.objects[bucket+"/"+key] + s.mu.RUnlock() + if !ok { + http.Error(w, "NoSuchKey", http.StatusNotFound) + return + } + w.Header().Set("content-length", strconv.Itoa(len(body))) + if r.Method == http.MethodGet { + w.Write(body) + } + default: + http.Error(w, "unsupported", http.StatusMethodNotAllowed) + } +} + +func (s *store) list(w http.ResponseWriter, r *http.Request, bucket string) { + prefix := r.URL.Query().Get("prefix") + after := r.URL.Query().Get("continuation-token") + s.mu.RLock() + var keys []string + sizes := map[string]int{} + for full, body := range s.objects { + b, key, _ := strings.Cut(full, "/") + if b == bucket && strings.HasPrefix(key, prefix) && key > after { + keys = append(keys, key) + sizes[key] = len(body) + } + } + s.mu.RUnlock() + sort.Strings(keys) + res := listResult{Name: bucket, Prefix: prefix} + if len(keys) > page { + res.IsTruncated = true + res.NextContinuationToken = keys[page-1] + keys = keys[:page] + } + for _, k := range keys { + res.Contents = append(res.Contents, entry{Key: k, Size: sizes[k]}) + } + res.KeyCount = len(res.Contents) + w.Header().Set("content-type", "application/xml") + io.WriteString(w, xml.Header) + xml.NewEncoder(w).Encode(res) +} + +// shorten re-encodes a JSON object with every string cut to max bytes; numbers keep their text. +func shorten(body []byte, max int) []byte { + dec := json.NewDecoder(bytes.NewReader(body)) + dec.UseNumber() + var v any + if err := dec.Decode(&v); err != nil || dec.More() { + return body + } + out, err := json.Marshal(cut(v, max)) + if err != nil { + return body + } + return out +} + +func cut(v any, max int) any { + switch v := v.(type) { + case string: + if len(v) > max { + return v[:max] + } + case []any: + for i := range v { + v[i] = cut(v[i], max) + } + case map[string]any: + for k := range v { + v[k] = cut(v[k], max) + } + } + return v +} diff --git a/litellm-rust/loadtest/cmd/s3stub/main_test.go b/litellm-rust/loadtest/cmd/s3stub/main_test.go new file mode 100644 index 00000000000..0f1644b3009 --- /dev/null +++ b/litellm-rust/loadtest/cmd/s3stub/main_test.go @@ -0,0 +1,15 @@ +package main + +import "testing" + +func TestShortenKeepsFieldsAndNumbers(t *testing.T) { + in := []byte(`{"messages":[{"content":"abcdefgh"}],"response_cost":0.10262000000000001,"n":3,"ok":true}`) + got := string(shorten(in, 3)) + want := `{"messages":[{"content":"abc"}],"n":3,"ok":true,"response_cost":0.10262000000000001}` + if got != want { + t.Fatalf("got %s", got) + } + if got := string(shorten([]byte("not json"), 3)); got != "not json" { + t.Fatalf("non-JSON changed: %s", got) + } +} diff --git a/litellm-rust/loadtest/cmd/verify/bifrost.go b/litellm-rust/loadtest/cmd/verify/bifrost.go new file mode 100644 index 00000000000..9372329aebe --- /dev/null +++ b/litellm-rust/loadtest/cmd/verify/bifrost.go @@ -0,0 +1,190 @@ +package main + +import ( + "encoding/json" + "fmt" + "regexp" + "strconv" + "strings" +) + +// bifrostSide is the side and key set name genconfig writes as a Bifrost config. Bifrost records +// requests in its own schema (the logs and governance_budgets tables, bifrost_* metrics, its own +// S3 log objects), so its records are mapped onto the same checks; the levels and records it has +// no counterpart for are reported as not applicable rather than failed. +const bifrostSide = "bifrost" + +func isBifrost(side string) bool { return side == bifrostSide } + +// bifrostQuery selects the Bifrost key set's records as the same JSON row query() selects for a +// LiteLLM side. It runs against Bifrost's logs database. Log rows are found by virtual key id, +// budgets by the ids genconfig gives them; Bifrost's customer stands in for LiteLLM's user, and +// the open-source build has no users, team members or daily tables. +func bifrostQuery(set string) string { + prefix := strings.ReplaceAll(setPrefix(set), "'", "''") + return fmt.Sprintf(`WITH s AS (SELECT * FROM logs WHERE virtual_key_id LIKE '%[1]s-%%'), +g AS (SELECT coalesce(routing_rule_name, '') AS model_group, count(*) AS rows, sum(prompt_tokens) AS p, + sum(completion_tokens) AS c, sum(cost)::float8 AS sp FROM s GROUP BY 1), +-- Each deployment is its own custom provider, named like the mock's /d/ path. +d AS (SELECT provider AS name, count(*) AS rows, + sum(prompt_tokens::float8 * prompt_tokens) AS pp, sum(prompt_tokens::float8 * completion_tokens) AS pc, + sum(completion_tokens::float8 * completion_tokens) AS cc, sum(cost::float8 * prompt_tokens) AS sp, + sum(cost::float8 * completion_tokens) AS sc + FROM s GROUP BY 1), +b AS (SELECT id, current_usage::float8 AS usage FROM governance_budgets WHERE id LIKE 'b-%[1]s-%%') +SELECT json_build_object( + 'dialect', 'bifrost', + 'rows', (SELECT count(*) FROM s), + 'success_rows', (SELECT count(*) FROM s WHERE status = 'success'), + 'prompt_tokens', (SELECT coalesce(sum(prompt_tokens), 0) FROM s), + 'completion_tokens', (SELECT coalesce(sum(completion_tokens), 0) FROM s), + 'spend', (SELECT coalesce(sum(cost), 0)::float8 FROM s), + 'zero_spend_rows', (SELECT count(*) FROM s WHERE coalesce(cost, 0) <= 0), + 'rows_with_team', (SELECT count(*) FROM s WHERE team_id LIKE '%[1]s-team-%%'), + 'rows_with_user', (SELECT count(*) FROM s WHERE customer_id = '%[1]s-customer'), + 'rows_with_model_id', (SELECT count(*) FROM s WHERE coalesce(provider, '') <> ''), + -- logging_headers copies the raw header values into metadata. + 'rows_with_spend_metadata', (SELECT count(*) FROM s + WHERE metadata::jsonb->>'x-cost-center' = '%[1]s' + AND (metadata::jsonb->>'x-litellm-spend-logs-metadata')::jsonb->>'run' = '%[1]s'), + 'groups', (SELECT coalesce(json_object_agg(model_group, json_build_object('rows', rows, + 'prompt_tokens', p, 'completion_tokens', c, 'spend', sp)), '{}') FROM g), + 'deployments', (SELECT coalesce(json_object_agg(name, json_build_object('rows', rows, 'pp', pp, 'pc', pc, + 'cc', cc, 'sp', sp, 'sc', sc)), '{}') FROM d), + 'keys_spend', (SELECT coalesce(sum(usage), 0) FROM b WHERE id ~ '^b-%[1]s-[0-9]+$'), + 'teams_spend', (SELECT coalesce(sum(usage), 0) FROM b WHERE id LIKE 'b-%[1]s-team-%%'), + 'users_spend', (SELECT coalesce(sum(usage), 0) FROM b WHERE id = 'b-%[1]s-customer') +); +`, prefix) +} + +func notApplicable(r *report, name, side, why string) { + r.add(check{Name: name, Side: side, Want: "-", Got: "n/a", OK: true, Note: why}) +} + +func checkBifrostDB(r *report, side string, e expected, db dbTotals) { + ok := e.Total.Requests + r.add(info("cross-replica state", side, "none: open-source Bifrost keeps rate limits, budgets and "+ + "counters in each replica's memory (written to Postgres every 10 s), with no Redis or shared store")) + exact(r, "log rows", side, ok, db.Rows) + exact(r, "log rows marked success", side, ok, db.Success) + exact(r, "log prompt tokens", side, e.Total.Prompt, db.Prompt) + exact(r, "log completion tokens", side, e.Total.Completion, db.Completion) + exact(r, "log rows priced at $0", side, 0, db.ZeroSpend) + exact(r, "log rows naming the key's team", side, ok, db.WithTeam) + exact(r, "log rows naming the key's customer", side, ok, db.WithUser) + notApplicable(r, "log rows naming the key's user", side, "users are enterprise-only") + exact(r, "log rows naming the deployment", side, ok, db.WithModel) + exact(r, "log rows with request spend metadata", side, ok, db.Metadata) + notApplicable(r, "log rows with key spend metadata", side, "virtual keys carry no metadata") + for _, name := range sortedKeys(e.Groups) { + want := e.Groups[name] + got := db.Groups[name] + if got == nil { + got = &groupTotals{} + } + r.add(check{Name: "rows / prompt / completion for " + name, Side: side, + Want: fmt.Sprintf("%d / %d / %d", want.Requests, want.Prompt, want.Completion), + Got: fmt.Sprintf("%d / %d / %d", got.Rows, got.Prompt, got.Completion), + OK: want.Requests == got.Rows && want.Prompt == got.Prompt && want.Completion == got.Completion}) + } + dollars(r, "key budget usage = logs", side, db.Spend, db.KeySpend) + dollars(r, "team budget usage = logs", side, db.Spend, db.TeamSpend) + dollars(r, "customer budget usage = logs", side, db.Spend, db.UserSpend) + notApplicable(r, "user spend = spend logs", side, "users are enterprise-only") + notApplicable(r, "team member spend = spend logs", side, "team members are enterprise-only") + notApplicable(r, "daily user and team tables", side, "Bifrost keeps no daily rollups") +} + +// checkBifrostMetrics maps the LiteLLM counters onto Bifrost's telemetry plugin. Bifrost counts +// requests by outcome and upstream attempts, which retries can only add to. +func checkBifrostMetrics(r *report, side string, e expected, db dbTotals, m map[string]*family) { + value := func(name string) (float64, bool) { return sample(m, name) } + success, okS := value("bifrost_success_requests_total") + failed, _ := value("bifrost_error_requests_total") + r.add(check{Name: "metric bifrost_success_requests_total + bifrost_error_requests_total", Side: side, + Want: fmt.Sprint(e.Sent), Got: present(okS, fmt.Sprintf("%.0f", success+failed)), OK: okS && success+failed == float64(e.Sent)}) + counts := []struct { + name string + want int64 + }{ + {"bifrost_success_requests_total", e.Total.Requests}, + {"bifrost_input_tokens_total", e.Total.Prompt}, + {"bifrost_output_tokens_total", e.Total.Completion}, + } + for _, c := range counts { + got, ok := value(c.name) + r.add(check{Name: "metric " + c.name, Side: side, Want: fmt.Sprint(c.want), + Got: present(ok, fmt.Sprintf("%.0f", got)), OK: ok && got == float64(c.want)}) + } + for _, name := range []string{"bifrost_upstream_requests_total", "bifrost_upstream_latency_seconds_count"} { + got, ok := value(name) + r.add(check{Name: "metric " + name, Side: side, Want: fmt.Sprintf(">= %d", e.Total.Requests), + Got: present(ok, fmt.Sprintf("%.0f", got)), OK: ok && got >= float64(e.Total.Requests)}) + } + got, ok := value("bifrost_cost_total") + r.add(check{Name: "metric bifrost_cost_total = logs", Side: side, + Want: fmt.Sprintf("$%.6f", db.Spend), Got: present(ok, fmt.Sprintf("$%.6f", got)), OK: ok && near(db.Spend, got)}) +} + +// Bifrost's S3 objects hold the log row with its JSON columns as strings, which s3stub +// -max-string may cut short, so token counts are read from the start of token_usage. +var ( + bifrostPrompt = regexp.MustCompile(`"prompt_tokens":(\d+)`) + bifrostCompletion = regexp.MustCompile(`"completion_tokens":(\d+)`) +) + +func (t *s3Totals) addBifrost(p map[string]any) { + text := func(k string) string { s, _ := p[k].(string); return s } + filled := func(keys ...string) bool { + for _, k := range keys { + if s := text(k); s != "" && s != "[]" && s != "null" { + return true + } + } + return false + } + if text("status") == "success" { + t.Success++ + } + usage := text("token_usage") + for _, f := range []struct { + re *regexp.Regexp + sum *int64 + }{{bifrostPrompt, &t.Prompt}, {bifrostCompletion, &t.Completion}} { + if m := f.re.FindStringSubmatch(usage); m != nil { + n, _ := strconv.ParseInt(m[1], 10, 64) + *f.sum += n + } + } + cost, _ := strconv.ParseFloat(text("cost"), 64) + t.Cost += cost + if filled("input_history", "responses_input_history") && filled("output_message", "responses_output") { + t.WithBodies++ + } + if strings.HasPrefix(text("team_id"), setPrefix(bifrostSide)+"-team-") { + t.WithTeam++ + } + var meta map[string]any + if json.Unmarshal([]byte(text("metadata")), &meta) == nil { + for k := range meta { + t.Metadata[k] = true + } + } +} + +func checkBifrostLogs(r *report, side string, e expected, db dbTotals, t *s3Totals) { + ok := e.Total.Requests + exact(r, "S3 request logs", side, ok, t.Objects) + if t.Read < t.Objects { + r.add(check{Name: "S3 logs read", Side: side, Want: fmt.Sprint(t.Objects), Got: fmt.Sprint(t.Read), + OK: true, Note: fmt.Sprintf("totals below cover the first %d", t.Read)}) + return + } + exact(r, "S3 logs with status success", side, ok, t.Success) + exact(r, "S3 logs with input and output", side, ok, t.WithBodies) + exact(r, "S3 logs naming the key's team", side, ok, t.WithTeam) + exact(r, "S3 log prompt tokens", side, e.Total.Prompt, t.Prompt) + exact(r, "S3 log completion tokens", side, e.Total.Completion, t.Completion) + dollars(r, "S3 log cost = logs", side, db.Spend, t.Cost) +} diff --git a/litellm-rust/loadtest/cmd/verify/compare.go b/litellm-rust/loadtest/cmd/verify/compare.go new file mode 100644 index 00000000000..9df41210178 --- /dev/null +++ b/litellm-rust/loadtest/cmd/verify/compare.go @@ -0,0 +1,83 @@ +package main + +import ( + "fmt" + "math" + "strings" +) + +// compareSides checks that two sides, given identical requests, recorded identical spend, and +// lists the telemetry one side has and the other lacks. Missing telemetry is reported, not +// failed: some of it is deliberately left out of one gateway. +func compareSides(r *report, a, b string, da, db *dbTotals, ma, mb map[string]*family, la, lb *s3Totals) { + both := a + " vs " + b + if da != nil && db != nil { + // Routing picks deployments at random and they are priced differently, so the sides' + // spend agrees per deployment price, not in total. + for _, name := range sortedKeys(da.Deployments) { + pb := db.Deployments[name] + if pb == nil { + continue + } + inA, outA, okA := da.Deployments[name].prices() + inB, outB, okB := pb.prices() + if !okA || !okB { + continue + } + price := func(in, out float64) string { + return fmt.Sprintf("$%.4f / $%.4f per 1M", in*1e6, out*1e6) + } + r.add(check{Name: "input / output price for " + name, Side: both, Want: price(inA, outA), + Got: price(inB, outB), OK: closeTo(inA, inB) && closeTo(outA, outB)}) + } + r.add(info("total spend ("+a+" / "+b+")", both, fmt.Sprintf("$%.6f / $%.6f", da.Spend, db.Spend))) + } + // Bifrost's metrics and log objects have their own schema; listing their differences from + // LiteLLM's says nothing, so only LiteLLM sides compare telemetry. + if isBifrost(a) || isBifrost(b) { + return + } + if ma != nil && mb != nil { + fa, fb := litellmFamilies(ma), litellmFamilies(mb) + r.add(info("metrics only on "+a, both, missing(fa, fb))) + r.add(info("metrics only on "+b, both, missing(fb, fa))) + var labels []string + for _, name := range sortedKeys(fa) { + if f, ok := fb[name]; ok { + if only := missing(fa[name].labels, f.labels); only != "" { + labels = append(labels, name+": "+only) + } + } + } + r.add(info("labels only on "+a+" (shared metrics)", both, strings.Join(labels, "; "))) + } + if la != nil && lb != nil { + r.add(info("S3 log fields only on "+a, both, missing(la.Fields, lb.Fields))) + r.add(info("S3 log fields only on "+b, both, missing(lb.Fields, la.Fields))) + r.add(info("S3 log metadata only on "+a, both, missing(la.Metadata, lb.Metadata))) + r.add(info("S3 log metadata only on "+b, both, missing(lb.Metadata, la.Metadata))) + } +} + +func info(name, side, got string) check { + if got == "" { + got = "none" + } + return check{Name: name, Side: side, Want: "-", Got: got, OK: true, Note: "reported"} +} + +// missing lists the keys of a that b lacks. +func missing[V any, W any](a map[string]V, b map[string]W) string { + var out []string + for _, k := range sortedKeys(a) { + if _, ok := b[k]; !ok { + out = append(out, k) + } + } + return strings.Join(out, ", ") +} + +// closeTo allows for the float error of fitting prices from summed spend. +func closeTo(a, b float64) bool { + return math.Abs(a-b) <= 1e-4*math.Max(math.Abs(a), math.Abs(b)) +} diff --git a/litellm-rust/loadtest/cmd/verify/db.go b/litellm-rust/loadtest/cmd/verify/db.go new file mode 100644 index 00000000000..8fbf169ed81 --- /dev/null +++ b/litellm-rust/loadtest/cmd/verify/db.go @@ -0,0 +1,183 @@ +package main + +import ( + "flag" + "fmt" + "math" + "os" + "strings" +) + +// groupTotals is one model group's spend logs. +type groupTotals struct { + Rows int64 `json:"rows"` + Prompt int64 `json:"prompt_tokens"` + Completion int64 `json:"completion_tokens"` + Spend float64 `json:"spend"` +} + +// deploymentTotals are one deployment's sums over its spend logs, enough to fit its per-token +// prices by least squares: spend = in * prompt + out * completion on every row. +type deploymentTotals struct { + Rows int64 `json:"rows"` + PP float64 `json:"pp"` + PC float64 `json:"pc"` + CC float64 `json:"cc"` + SP float64 `json:"sp"` + SC float64 `json:"sc"` +} + +// prices solves the normal equations; ok is false when the rows cannot tell the two prices apart. +func (d deploymentTotals) prices() (in, out float64, ok bool) { + det := d.PP*d.CC - d.PC*d.PC + if d.Rows < 2 || det <= 1e-9*d.PP*d.CC { + return 0, 0, false + } + return (d.SP*d.CC - d.SC*d.PC) / det, (d.PP*d.SC - d.PC*d.SP) / det, true +} + +// dbTotals is the one JSON row `verify sql` selects for a key set. +type dbTotals struct { + Dialect string `json:"dialect"` // "bifrost" for bifrostQuery's row + Rows int64 `json:"rows"` + Success int64 `json:"success_rows"` + Prompt int64 `json:"prompt_tokens"` + Completion int64 `json:"completion_tokens"` + Spend float64 `json:"spend"` + ZeroSpend int64 `json:"zero_spend_rows"` + WithTeam int64 `json:"rows_with_team"` + WithUser int64 `json:"rows_with_user"` + WithModel int64 `json:"rows_with_model_id"` + Metadata int64 `json:"rows_with_spend_metadata"` + Groups map[string]*groupTotals `json:"groups"` + Deployments map[string]*deploymentTotals `json:"deployments"` + KeySpend float64 `json:"keys_spend"` + TeamSpend float64 `json:"teams_spend"` + UserSpend float64 `json:"users_spend"` + MemberSpend float64 `json:"members_spend"` + DailyUser daily `json:"daily_user"` + DailyTeam daily `json:"daily_team"` +} + +type daily struct { + Spend float64 `json:"spend"` + Requests int64 `json:"requests"` + Successful int64 `json:"successful"` + Prompt int64 `json:"prompt_tokens"` + Completion int64 `json:"completion_tokens"` +} + +func sqlCmd(args []string) { + fs := flag.NewFlagSet("sql", flag.ExitOnError) + set := fs.String("set", "", "key set name, as given to genconfig -key-sets") + fs.Parse(args) + if isBifrost(*set) { + fmt.Fprint(os.Stdout, bifrostQuery(*set)) + return + } + fmt.Fprint(os.Stdout, query(*set)) +} + +// query selects a key set's records as one JSON value. Owners are found by the set's id prefix, +// spend logs by the set's keys. +func query(set string) string { + prefix := setPrefix(set) + p := strings.ReplaceAll(prefix, "'", "''") + "-" + return fmt.Sprintf(`WITH k AS (SELECT token, key_alias, spend FROM "LiteLLM_VerificationToken" WHERE key_alias LIKE '%[1]s%%'), +s AS (SELECT l.* FROM "LiteLLM_SpendLogs" l JOIN k ON k.token = l.api_key), +g AS (SELECT coalesce(model_group, '') AS model_group, count(*) AS rows, sum(prompt_tokens) AS p, + sum(completion_tokens) AS c, sum(spend) AS sp FROM s GROUP BY 1), +du AS (SELECT * FROM "LiteLLM_DailyUserSpend" WHERE user_id LIKE '%[1]s%%'), +dt AS (SELECT * FROM "LiteLLM_DailyTeamSpend" WHERE team_id LIKE '%[1]s%%'), +-- Deployments by the mock's /d/ path: the gateways give deployments different ids. +d AS (SELECT substring(api_base FROM '/d/([^/]+)') AS name, count(*) AS rows, + sum(prompt_tokens::float8 * prompt_tokens) AS pp, sum(prompt_tokens::float8 * completion_tokens) AS pc, + sum(completion_tokens::float8 * completion_tokens) AS cc, sum(spend * prompt_tokens) AS sp, + sum(spend * completion_tokens) AS sc + FROM s WHERE api_base LIKE '%%/d/%%' GROUP BY 1) +SELECT json_build_object( + 'rows', (SELECT count(*) FROM s), + 'success_rows', (SELECT count(*) FROM s WHERE coalesce(status, 'success') = 'success'), + 'prompt_tokens', (SELECT coalesce(sum(prompt_tokens), 0) FROM s), + 'completion_tokens', (SELECT coalesce(sum(completion_tokens), 0) FROM s), + 'spend', (SELECT coalesce(sum(spend), 0) FROM s), + 'zero_spend_rows', (SELECT count(*) FROM s WHERE spend <= 0), + 'rows_with_team', (SELECT count(*) FROM s WHERE team_id LIKE '%[1]s%%'), + 'rows_with_user', (SELECT count(*) FROM s WHERE "user" LIKE '%[1]s%%'), + 'rows_with_model_id', (SELECT count(*) FROM s WHERE coalesce(model_id, '') <> ''), + 'rows_with_spend_metadata', (SELECT count(*) FROM s + WHERE metadata->'spend_logs_metadata'->>'run' = '%[2]s' + AND metadata->'spend_logs_metadata'->>'cost_center' LIKE 'key-%%'), + 'groups', (SELECT coalesce(json_object_agg(model_group, json_build_object('rows', rows, + 'prompt_tokens', p, 'completion_tokens', c, 'spend', sp)), '{}') FROM g), + 'deployments', (SELECT coalesce(json_object_agg(name, json_build_object('rows', rows, 'pp', pp, 'pc', pc, + 'cc', cc, 'sp', sp, 'sc', sc)), '{}') FROM d), + 'keys_spend', (SELECT coalesce(sum(spend), 0) FROM k), + 'teams_spend', (SELECT coalesce(sum(spend), 0) FROM "LiteLLM_TeamTable" WHERE team_id LIKE '%[1]s%%'), + 'users_spend', (SELECT coalesce(sum(spend), 0) FROM "LiteLLM_UserTable" WHERE user_id LIKE '%[1]s%%'), + 'members_spend', (SELECT coalesce(sum(spend), 0) FROM "LiteLLM_TeamMembership" WHERE team_id LIKE '%[1]s%%'), + 'daily_user', (SELECT json_build_object('spend', coalesce(sum(spend), 0), 'requests', coalesce(sum(api_requests), 0), + 'successful', coalesce(sum(successful_requests), 0), 'prompt_tokens', coalesce(sum(prompt_tokens), 0), + 'completion_tokens', coalesce(sum(completion_tokens), 0)) FROM du), + 'daily_team', (SELECT json_build_object('spend', coalesce(sum(spend), 0), 'requests', coalesce(sum(api_requests), 0), + 'successful', coalesce(sum(successful_requests), 0), 'prompt_tokens', coalesce(sum(prompt_tokens), 0), + 'completion_tokens', coalesce(sum(completion_tokens), 0)) FROM dt) +); +`, p, strings.ReplaceAll(prefix, "'", "''")) +} + +// close reports whether two dollar amounts agree to a millionth (summing in another order, or a +// counter's float accumulation, changes the last digits). +func near(a, b float64) bool { + return math.Abs(a-b) <= 1e-6*math.Max(1, math.Max(math.Abs(a), math.Abs(b))) +} + +func exact(r *report, name, side string, want, got int64) { + r.add(check{Name: name, Side: side, Want: fmt.Sprint(want), Got: fmt.Sprint(got), OK: want == got}) +} + +func dollars(r *report, name, side string, want, got float64) { + r.add(check{Name: name, Side: side, Want: fmt.Sprintf("$%.6f", want), Got: fmt.Sprintf("$%.6f", got), OK: near(want, got)}) +} + +func checkDB(r *report, side string, e expected, db dbTotals) { + if isBifrost(side) { + checkBifrostDB(r, side, e, db) + return + } + ok := e.Total.Requests + exact(r, "spend log rows", side, ok, db.Rows) + exact(r, "spend log rows marked success", side, ok, db.Success) + exact(r, "spend log prompt tokens", side, e.Total.Prompt, db.Prompt) + exact(r, "spend log completion tokens", side, e.Total.Completion, db.Completion) + exact(r, "spend log rows priced at $0", side, 0, db.ZeroSpend) + exact(r, "spend log rows naming the key's team", side, ok, db.WithTeam) + exact(r, "spend log rows naming the key's user", side, ok, db.WithUser) + exact(r, "spend log rows naming the deployment", side, ok, db.WithModel) + exact(r, "spend log rows with request + key spend metadata", side, ok, db.Metadata) + for _, name := range sortedKeys(e.Groups) { + want := e.Groups[name] + got := db.Groups[name] + if got == nil { + got = &groupTotals{} + } + r.add(check{Name: "rows / prompt / completion for " + name, Side: side, + Want: fmt.Sprintf("%d / %d / %d", want.Requests, want.Prompt, want.Completion), + Got: fmt.Sprintf("%d / %d / %d", got.Rows, got.Prompt, got.Completion), + OK: want.Requests == got.Rows && want.Prompt == got.Prompt && want.Completion == got.Completion}) + } + dollars(r, "key spend = spend logs", side, db.Spend, db.KeySpend) + dollars(r, "team spend = spend logs", side, db.Spend, db.TeamSpend) + dollars(r, "user spend = spend logs", side, db.Spend, db.UserSpend) + dollars(r, "team member spend = spend logs", side, db.Spend, db.MemberSpend) + for _, d := range []struct { + name string + daily daily + }{{"daily user", db.DailyUser}, {"daily team", db.DailyTeam}} { + dollars(r, d.name+" spend = spend logs", side, db.Spend, d.daily.Spend) + exact(r, d.name+" requests", side, ok, d.daily.Requests) + exact(r, d.name+" successful requests", side, ok, d.daily.Successful) + exact(r, d.name+" prompt tokens", side, e.Total.Prompt, d.daily.Prompt) + exact(r, d.name+" completion tokens", side, e.Total.Completion, d.daily.Completion) + } +} diff --git a/litellm-rust/loadtest/cmd/verify/main.go b/litellm-rust/loadtest/cmd/verify/main.go new file mode 100644 index 00000000000..952224d8dc6 --- /dev/null +++ b/litellm-rust/loadtest/cmd/verify/main.go @@ -0,0 +1,262 @@ +// verify checks that a load test's requests were recorded correctly by each gateway: spend logs, +// the key, team, user and membership spend and the daily tables in Postgres, the Prometheus +// counters, and the S3 request logs. Run with genconfig -features full and one key set per side, +// so every side's records can be told apart in the shared database. A side named bifrost is read +// in Bifrost's schema (see bifrost.go); with three sides each pair is compared. +// +// verify expect -out expected-python.json results/python-*.jsonl probes-python.jsonl +// verify sql -set python # the query whose one JSON row is db-python.json +// verify check -dir collected -side python=, -side rust=... \ +// -s3 http://s3stub:9100/bucket -mock http://mockllm:9000 -report report.md +// +// Every side gets the same requests (same loadgen seed, the mock's in= directive fixing input +// tokens), so the sides' totals must also agree with each other, not just with what was sent. +package main + +import ( + "bufio" + "encoding/json" + "flag" + "fmt" + "log" + "os" + "path/filepath" + "sort" + "strings" +) + +func main() { + log.SetFlags(0) + if len(os.Args) < 2 { + log.Fatal("usage: verify expect|sql|check ...") + } + switch os.Args[1] { + case "expect": + expectCmd(os.Args[2:]) + case "sql": + sqlCmd(os.Args[2:]) + case "check": + checkCmd(os.Args[2:]) + default: + log.Fatalf("unknown subcommand %q", os.Args[1]) + } +} + +// result is the part of a loadgen record verify reads. +type result struct { + Model string `json:"model"` + Key int `json:"key"` + CtxTokens int64 `json:"ctx"` + OutTokens int64 `json:"out"` + OK bool `json:"ok"` +} + +type tokens struct { + Requests int64 `json:"requests"` + Prompt int64 `json:"prompt_tokens"` + Completion int64 `json:"completion_tokens"` +} + +func (t *tokens) add(r result) { + t.Requests++ + t.Prompt += r.CtxTokens + t.Completion += r.OutTokens +} + +// expected is what the gateway was asked to serve successfully. +type expected struct { + Sent int64 `json:"sent"` + Failed int64 `json:"failed"` + Total tokens `json:"total"` + Groups map[string]*tokens `json:"groups"` +} + +func expectCmd(args []string) { + fs := flag.NewFlagSet("expect", flag.ExitOnError) + out := fs.String("out", "expected.json", "where to write the totals") + fs.Parse(args) + e := expected{Groups: map[string]*tokens{}} + for _, path := range fs.Args() { + f, err := os.Open(path) + if err != nil { + log.Fatal(err) + } + sc := bufio.NewScanner(f) + sc.Buffer(make([]byte, 1<<20), 1<<24) + for sc.Scan() { + var r result + if err := json.Unmarshal(sc.Bytes(), &r); err != nil { + log.Fatalf("%s: %v", path, err) + } + e.Sent++ + if !r.OK { + e.Failed++ + continue + } + e.Total.add(r) + g := e.Groups[r.Model] + if g == nil { + g = &tokens{} + e.Groups[r.Model] = g + } + g.add(r) + } + f.Close() + } + writeJSON(*out, e) +} + +func writeJSON(path string, v any) { + b, err := json.MarshalIndent(v, "", " ") + if err != nil { + log.Fatal(err) + } + if err := os.WriteFile(path, append(b, '\n'), 0o644); err != nil { + log.Fatal(err) + } +} + +func readJSON(path string, v any) error { + b, err := os.ReadFile(path) + if err != nil { + return err + } + return json.Unmarshal(b, v) +} + +// check is one line of the report. +type check struct { + Name, Side, Want, Got string + OK bool + Note string +} + +type report struct{ checks []check } + +func (r *report) add(c check) { r.checks = append(r.checks, c) } + +func (r *report) failed() int { + n := 0 + for _, c := range r.checks { + if !c.OK { + n++ + } + } + return n +} + +func (r *report) markdown() string { + var b strings.Builder + fmt.Fprintf(&b, "### Recorded data checks: %d of %d passed\n\n", len(r.checks)-r.failed(), len(r.checks)) + b.WriteString("| check | side | expected | recorded | result |\n|---|---|---|---|---|\n") + for _, c := range r.checks { + mark := "ok" + if !c.OK { + mark = "**FAIL**" + } + if c.Note != "" { + mark += " (" + c.Note + ")" + } + fmt.Fprintf(&b, "| %s | %s | %s | %s | %s |\n", c.Name, c.Side, c.Want, c.Got, mark) + } + return b.String() +} + +// side is one gateway under test. +type side struct { + name, metricsHost, s3Prefix string +} + +type sideFlags []side + +func (s *sideFlags) String() string { return fmt.Sprint(*s) } + +func (s *sideFlags) Set(v string) error { + name, rest, ok := strings.Cut(v, "=") + host, prefix, ok2 := strings.Cut(rest, ",") + if !ok || !ok2 || name == "" { + return fmt.Errorf("want name=,, got %q", v) + } + *s = append(*s, side{name: name, metricsHost: host, s3Prefix: strings.Trim(prefix, "/")}) + return nil +} + +func checkCmd(args []string) { + fs := flag.NewFlagSet("check", flag.ExitOnError) + dir := fs.String("dir", ".", "directory with expected-.json and db-.json") + s3 := fs.String("s3", "", "S3 bucket URL readable without credentials (empty skips S3)") + mock := fs.String("mock", "", "mock base URL whose /stats counts request headers (empty skips)") + reportPath := fs.String("report", "report.md", "markdown report") + var sides sideFlags + fs.Var(&sides, "side", "name=, (repeatable)") + fs.Parse(args) + + var r report + dbs := map[string]*dbTotals{} + metrics := map[string]map[string]*family{} + logs := map[string]*s3Totals{} + var headers map[string]map[string]int64 + if *mock != "" { + h, err := mockHeaders(*mock) + if err != nil { + r.add(check{Name: "mock header counts read", Side: "mock", Want: "/stats", Got: err.Error()}) + } + headers = h + } + for _, s := range sides { + var e expected + if err := readJSON(filepath.Join(*dir, "expected-"+s.name+".json"), &e); err != nil { + log.Fatal(err) + } + var db dbTotals + if err := readJSON(filepath.Join(*dir, "db-"+s.name+".json"), &db); err != nil { + log.Fatal(err) + } + dbs[s.name] = &db + checkDB(&r, s.name, e, db) + if headers != nil { + checkHeaders(&r, s.name, e, headers) + } + if s.metricsHost != "" { + m, err := scrapeAll(s.metricsHost) + if err != nil { + r.add(check{Name: "metrics scraped", Side: s.name, Want: "every replica", Got: err.Error()}) + } else { + metrics[s.name] = m + checkMetrics(&r, s.name, e, db, m) + } + } + if *s3 != "" { + t, err := listLogs(*s3, s.s3Prefix, isBifrost(s.name)) + if err != nil { + r.add(check{Name: "S3 logs listed", Side: s.name, Want: "a listing", Got: err.Error()}) + } else { + logs[s.name] = t + checkLogs(&r, s.name, e, db, t) + } + } + } + for i := range sides { + for _, other := range sides[i+1:] { + a, b := sides[i].name, other.name + compareSides(&r, a, b, dbs[a], dbs[b], metrics[a], metrics[b], logs[a], logs[b]) + } + } + md := r.markdown() + if err := os.WriteFile(*reportPath, []byte(md), 0o644); err != nil { + log.Fatal(err) + } + fmt.Print(md) + if r.failed() > 0 { + os.Exit(1) + } +} + +func sortedKeys[V any](m map[string]V) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} diff --git a/litellm-rust/loadtest/cmd/verify/metrics.go b/litellm-rust/loadtest/cmd/verify/metrics.go new file mode 100644 index 00000000000..35d54588e2e --- /dev/null +++ b/litellm-rust/loadtest/cmd/verify/metrics.go @@ -0,0 +1,171 @@ +package main + +import ( + "bufio" + "fmt" + "io" + "net" + "net/http" + "sort" + "strconv" + "strings" + "time" +) + +// family is one metric name summed over every series and replica. +type family struct { + sum float64 + labels map[string]bool +} + +// scrapeAll reads /metrics from every address host resolves to (a headless Service names each +// replica) and sums each sample name across them. +func scrapeAll(hostPort string) (map[string]*family, error) { + host, port, err := net.SplitHostPort(hostPort) + if err != nil { + return nil, err + } + addrs, err := net.LookupHost(host) + if err != nil { + return nil, err + } + sort.Strings(addrs) + client := &http.Client{Timeout: 30 * time.Second} + out := map[string]*family{} + for _, addr := range addrs { + resp, err := client.Get("http://" + net.JoinHostPort(addr, port) + "/metrics") + if err != nil { + return nil, err + } + if resp.StatusCode != http.StatusOK { + resp.Body.Close() + return nil, fmt.Errorf("%s: /metrics answered %d", addr, resp.StatusCode) + } + err = parseMetrics(resp.Body, out) + resp.Body.Close() + if err != nil { + return nil, err + } + } + return out, nil +} + +// parseMetrics adds each sample of a Prometheus text exposition to out, by sample name. +func parseMetrics(r io.Reader, out map[string]*family) error { + sc := bufio.NewScanner(r) + sc.Buffer(make([]byte, 1<<20), 1<<24) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + name, labels, rest := line, "", "" + if i := strings.IndexByte(line, '{'); i >= 0 { + j := strings.LastIndexByte(line, '}') + if j < i { + return fmt.Errorf("malformed sample %q", line) + } + name, labels, rest = line[:i], line[i+1:j], strings.TrimSpace(line[j+1:]) + } else if i := strings.IndexByte(line, ' '); i >= 0 { + name, rest = line[:i], strings.TrimSpace(line[i+1:]) + } + fields := strings.Fields(rest) + if len(fields) == 0 { + return fmt.Errorf("sample without a value %q", line) + } + value, err := strconv.ParseFloat(fields[0], 64) + if err != nil { + return fmt.Errorf("sample value in %q: %v", line, err) + } + f := out[name] + if f == nil { + f = &family{labels: map[string]bool{}} + out[name] = f + } + f.sum += value + for _, pair := range splitLabels(labels) { + if k, _, ok := strings.Cut(pair, "="); ok { + f.labels[strings.TrimSpace(k)] = true + } + } + } + return sc.Err() +} + +// splitLabels splits `a="x",b="y,z"` on the commas outside quotes. +func splitLabels(s string) []string { + var parts []string + quoted, escaped, start := false, false, 0 + for i, c := range s { + switch { + case escaped: + escaped = false + case c == '\\': + escaped = true + case c == '"': + quoted = !quoted + case c == ',' && !quoted: + parts = append(parts, s[start:i]) + start = i + 1 + } + } + if start < len(s) { + parts = append(parts, s[start:]) + } + return parts +} + +func sample(m map[string]*family, name string) (float64, bool) { + f, ok := m[name] + if !ok { + return 0, false + } + return f.sum, true +} + +func checkMetrics(r *report, side string, e expected, db dbTotals, m map[string]*family) { + if isBifrost(side) { + checkBifrostMetrics(r, side, e, db, m) + return + } + counts := []struct { + name string + want int64 + }{ + {"litellm_proxy_total_requests_metric_total", e.Sent}, + {"litellm_request_total_latency_metric_count", e.Total.Requests}, + {"litellm_llm_api_latency_metric_count", e.Total.Requests}, + {"litellm_deployment_success_responses_total", e.Total.Requests}, + {"litellm_input_tokens_metric_total", e.Total.Prompt}, + {"litellm_output_tokens_metric_total", e.Total.Completion}, + {"litellm_total_tokens_metric_total", e.Total.Prompt + e.Total.Completion}, + } + for _, c := range counts { + got, ok := sample(m, c.name) + r.add(check{Name: "metric " + c.name, Side: side, Want: fmt.Sprint(c.want), + Got: present(ok, fmt.Sprintf("%.0f", got)), OK: ok && got == float64(c.want)}) + } + got, ok := sample(m, "litellm_spend_metric_total") + r.add(check{Name: "metric litellm_spend_metric_total = spend logs", Side: side, + Want: fmt.Sprintf("$%.6f", db.Spend), Got: present(ok, fmt.Sprintf("$%.6f", got)), OK: ok && near(db.Spend, got)}) +} + +func present(ok bool, value string) string { + if !ok { + return "missing" + } + return value +} + +// litellmFamilies lists the sample names a gateway serves for its requests, so the sides can be +// compared without process and runtime metrics. +func litellmFamilies(m map[string]*family) map[string]*family { + out := map[string]*family{} + for name, f := range m { + // prometheus_client's _created series only timestamp the counters beside them. + if strings.HasPrefix(name, "litellm_") && !strings.HasSuffix(name, "_created") { + out[name] = f + } + } + return out +} diff --git a/litellm-rust/loadtest/cmd/verify/mock.go b/litellm-rust/loadtest/cmd/verify/mock.go new file mode 100644 index 00000000000..e7e992fd19f --- /dev/null +++ b/litellm-rust/loadtest/cmd/verify/mock.go @@ -0,0 +1,69 @@ +package main + +import ( + "encoding/json" + "fmt" + "net/http" + "strings" +) + +// mockHeaders reads the mock's per-value request header counts (mockllm -count-headers). +func mockHeaders(base string) (map[string]map[string]int64, error) { + resp, err := http.Get(strings.TrimSuffix(base, "/") + "/stats") + if err != nil { + return nil, err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("/stats answered %d", resp.StatusCode) + } + var stats struct { + Headers map[string]map[string]int64 `json:"headers"` + } + if err := json.NewDecoder(resp.Body).Decode(&stats); err != nil { + return nil, err + } + return stats.Headers, nil +} + +// checkHeaders checks what the request hooks did to the side's requests, as the provider saw +// them: forwarded headers arrive, stripped ones do not, and the caller's team is added. Upstream +// retries can only add requests, so arrivals are at least the successful requests. +func checkHeaders(r *report, side string, e expected, headers map[string]map[string]int64) { + prefix := setPrefix(side) + count := func(name string, match func(string) bool) int64 { + var n int64 + for value, c := range headers[name] { + if match(value) { + n += c + } + } + return n + } + ok := e.Sent - e.Failed + arrived := count("x-request-origin", func(v string) bool { return v == prefix }) + atLeast := func(name string, got int64) { + r.add(check{Name: name, Side: side, Want: fmt.Sprintf(">= %d", ok), Got: fmt.Sprint(got), OK: got >= ok}) + } + atLeast("upstream requests with x-request-origin forwarded", arrived) + exact(r, "upstream requests with x-trace-* forwarded", side, arrived, + count("x-trace-id", func(v string) bool { return v == prefix+"-trace" })) + if isBifrost(side) { + notApplicable(r, "upstream requests with x-caller-team added", side, "Bifrost cannot add headers from the key's team") + } else { + exact(r, "upstream requests with x-caller-team added", side, arrived, + count("x-caller-team", func(v string) bool { return strings.HasPrefix(v, prefix+"-team-") })) + } + for _, name := range []string{"x-cost-center", "x-litellm-spend-logs-metadata"} { + exact(r, "upstream requests still carrying "+name, side, 0, + count(name, func(v string) bool { return strings.Contains(v, prefix) })) + } +} + +// setPrefix is genconfig's id prefix for a key set. +func setPrefix(set string) string { + if set == "" { + return "loadtest" + } + return "loadtest-" + set +} diff --git a/litellm-rust/loadtest/cmd/verify/s3.go b/litellm-rust/loadtest/cmd/verify/s3.go new file mode 100644 index 00000000000..82cce7e3b6e --- /dev/null +++ b/litellm-rust/loadtest/cmd/verify/s3.go @@ -0,0 +1,161 @@ +package main + +import ( + "encoding/json" + "encoding/xml" + "fmt" + "io" + "net/http" + "net/url" + "sort" + "strings" + "time" +) + +// maxLogs bounds how many request logs one side's check downloads. +const maxLogs = 20_000 + +// s3Totals sums one side's request logs: one StandardLoggingPayload object per request, or one +// Bifrost log row. +type s3Totals struct { + bifrost bool + Objects int64 + Read int64 + Success int64 + Prompt int64 + Completion int64 + Cost float64 + WithBodies int64 + WithTeam int64 + Fields map[string]bool + Metadata map[string]bool +} + +type listing struct { + Contents []struct { + Key string `xml:"Key"` + } `xml:"Contents"` + IsTruncated bool `xml:"IsTruncated"` + NextContinuationToken string `xml:"NextContinuationToken"` +} + +// listLogs lists bucket objects under prefix (ListObjectsV2, no credentials: the bucket allows +// anonymous reads) and reads up to maxLogs of them, as Bifrost log rows when bifrost is set. +func listLogs(bucket, prefix string, bifrost bool) (*s3Totals, error) { + client := &http.Client{Timeout: 60 * time.Second} + bucket = strings.TrimRight(bucket, "/") + var keys []string + token := "" + for { + q := url.Values{"list-type": {"2"}, "prefix": {prefix + "/"}} + if token != "" { + q.Set("continuation-token", token) + } + resp, err := client.Get(bucket + "?" + q.Encode()) + if err != nil { + return nil, err + } + body, err := io.ReadAll(resp.Body) + resp.Body.Close() + if err != nil { + return nil, err + } + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("list answered %d: %.200s", resp.StatusCode, body) + } + var l listing + if err := xml.Unmarshal(body, &l); err != nil { + return nil, err + } + for _, c := range l.Contents { + keys = append(keys, c.Key) + } + if !l.IsTruncated || l.NextContinuationToken == "" { + break + } + token = l.NextContinuationToken + } + sort.Strings(keys) + t := &s3Totals{bifrost: bifrost, Objects: int64(len(keys)), Fields: map[string]bool{}, Metadata: map[string]bool{}} + for i, key := range keys { + if i >= maxLogs { + break + } + resp, err := client.Get(bucket + "/" + (&url.URL{Path: key}).EscapedPath()) + if err != nil { + return nil, err + } + body, err := io.ReadAll(resp.Body) + resp.Body.Close() + if err != nil { + return nil, err + } + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("%s answered %d", key, resp.StatusCode) + } + // One payload per object, or one per line with batched uploads. + for _, line := range strings.Split(strings.TrimSpace(string(body)), "\n") { + if err := t.add([]byte(line)); err != nil { + return nil, fmt.Errorf("%s: %v", key, err) + } + } + } + return t, nil +} + +func (t *s3Totals) add(raw []byte) error { + var p map[string]any + if err := json.Unmarshal(raw, &p); err != nil { + return err + } + t.Read++ + for k := range p { + t.Fields[k] = true + } + if t.bifrost { + t.addBifrost(p) + return nil + } + number := func(v any) float64 { + f, _ := v.(float64) + return f + } + if p["status"] == "success" { + t.Success++ + } + t.Prompt += int64(number(p["prompt_tokens"])) + t.Completion += int64(number(p["completion_tokens"])) + t.Cost += number(p["response_cost"]) + if p["messages"] != nil && p["response"] != nil { + t.WithBodies++ + } + if m, ok := p["metadata"].(map[string]any); ok { + for k := range m { + t.Metadata[k] = true + } + if team, _ := m["user_api_key_team_id"].(string); team != "" { + t.WithTeam++ + } + } + return nil +} + +func checkLogs(r *report, side string, e expected, db dbTotals, t *s3Totals) { + if isBifrost(side) { + checkBifrostLogs(r, side, e, db, t) + return + } + ok := e.Total.Requests + exact(r, "S3 request logs", side, ok, t.Objects) + if t.Read < t.Objects { + r.add(check{Name: "S3 logs read", Side: side, Want: fmt.Sprint(t.Objects), Got: fmt.Sprint(t.Read), + OK: true, Note: fmt.Sprintf("totals below cover the first %d", t.Read)}) + return + } + exact(r, "S3 logs with status success", side, ok, t.Success) + exact(r, "S3 logs with messages and response", side, ok, t.WithBodies) + exact(r, "S3 logs naming the key's team", side, ok, t.WithTeam) + exact(r, "S3 log prompt tokens", side, e.Total.Prompt, t.Prompt) + exact(r, "S3 log completion tokens", side, e.Total.Completion, t.Completion) + dollars(r, "S3 log response_cost = spend logs", side, db.Spend, t.Cost) +} diff --git a/litellm-rust/loadtest/cmd/verify/verify_test.go b/litellm-rust/loadtest/cmd/verify/verify_test.go new file mode 100644 index 00000000000..8a92df7e381 --- /dev/null +++ b/litellm-rust/loadtest/cmd/verify/verify_test.go @@ -0,0 +1,162 @@ +package main + +import ( + "strings" + "testing" +) + +func TestParseMetricsSumsSeriesAndKeepsLabelNames(t *testing.T) { + text := `# HELP litellm_spend_metric_total Spend +# TYPE litellm_spend_metric_total counter +litellm_spend_metric_total{model="a",team="x,y"} 1.5 +litellm_spend_metric_total{model="b",end_user="q\"z"} 2.25 1700000000 +litellm_requests 3 +` + out := map[string]*family{} + if err := parseMetrics(strings.NewReader(text), out); err != nil { + t.Fatal(err) + } + if got := out["litellm_spend_metric_total"].sum; got != 3.75 { + t.Fatalf("sum = %v", got) + } + labels := out["litellm_spend_metric_total"].labels + for _, want := range []string{"model", "team", "end_user"} { + if !labels[want] { + t.Fatalf("label %s missing from %v", want, labels) + } + } + if out["litellm_requests"].sum != 3 { + t.Fatalf("unlabelled sample not read") + } +} + +func TestChecksPassOnlyWhenRecordsMatch(t *testing.T) { + e := expected{Sent: 2, Total: tokens{Requests: 2, Prompt: 20, Completion: 10}, + Groups: map[string]*tokens{"g": {Requests: 2, Prompt: 20, Completion: 10}}} + day := daily{Spend: 0.5, Requests: 2, Successful: 2, Prompt: 20, Completion: 10} + db := dbTotals{Rows: 2, Success: 2, Prompt: 20, Completion: 10, Spend: 0.5, WithTeam: 2, WithUser: 2, + WithModel: 2, Metadata: 2, Groups: map[string]*groupTotals{"g": {Rows: 2, Prompt: 20, Completion: 10, Spend: 0.5}}, + KeySpend: 0.5, TeamSpend: 0.5, UserSpend: 0.5, MemberSpend: 0.5, DailyUser: day, DailyTeam: day} + var r report + checkDB(&r, "rust", e, db) + if n := r.failed(); n != 0 { + t.Fatalf("%d checks failed:\n%s", n, r.markdown()) + } + db.TeamSpend = 0.25 + db.Groups["g"].Completion = 9 + r = report{} + checkDB(&r, "rust", e, db) + if n := r.failed(); n != 2 { + t.Fatalf("want 2 failures, got %d:\n%s", n, r.markdown()) + } +} + +func TestQueryScopesToTheKeySet(t *testing.T) { + q := query("rust") + for _, want := range []string{"key_alias LIKE 'loadtest-rust-%'", `'run' = 'loadtest-rust'`, `"LiteLLM_DailyTeamSpend"`} { + if !strings.Contains(q, want) { + t.Fatalf("query lacks %q", want) + } + } +} + +func TestPricesAreFittedFromRows(t *testing.T) { + // Rows (prompt, completion) priced at $3 and $15 per million tokens. + rows := [][2]float64{{10000, 50}, {30000, 10}, {20000, 400}} + var d deploymentTotals + for _, row := range rows { + spend := row[0]*3e-6 + row[1]*15e-6 + d.Rows++ + d.PP += row[0] * row[0] + d.PC += row[0] * row[1] + d.CC += row[1] * row[1] + d.SP += spend * row[0] + d.SC += spend * row[1] + } + in, out, ok := d.prices() + if !ok || !closeTo(in, 3e-6) || !closeTo(out, 15e-6) { + t.Fatalf("prices = %v, %v, %v", in, out, ok) + } + if _, _, ok := (deploymentTotals{Rows: 1, PP: 1, CC: 1}).prices(); ok { + t.Fatal("one row fitted two prices") + } +} + +func TestHeaderChecksSeeForwardedAndStrippedHeaders(t *testing.T) { + e := expected{Sent: 3} + headers := map[string]map[string]int64{ + "x-request-origin": {"loadtest-rust": 3, "loadtest-python": 3}, + "x-trace-id": {"loadtest-rust-trace": 3}, + "x-caller-team": {"loadtest-rust-team-0": 2, "loadtest-rust-team-1": 1}, + "x-cost-center": {}, + "x-litellm-spend-logs-metadata": {`{"run": "loadtest-python"}`: 3}, + } + var r report + checkHeaders(&r, "rust", e, headers) + if n := r.failed(); n != 0 { + t.Fatalf("%d checks failed:\n%s", n, r.markdown()) + } + r = report{} + checkHeaders(&r, "python", e, headers) + // No trace or team headers for python, and its spend metadata header was not stripped. + if n := r.failed(); n != 3 { + t.Fatalf("want 3 failures, got %d:\n%s", n, r.markdown()) + } +} + +func TestBifrostQueryScopesToTheKeySet(t *testing.T) { + q := bifrostQuery("bifrost") + for _, want := range []string{"virtual_key_id LIKE 'loadtest-bifrost-%'", "id ~ '^b-loadtest-bifrost-[0-9]+$'", + "id = 'b-loadtest-bifrost-customer'", "'dialect', 'bifrost'"} { + if !strings.Contains(q, want) { + t.Fatalf("query lacks %q", want) + } + } +} + +func TestBifrostChecksReportMissingLevelsAsNotApplicable(t *testing.T) { + e := expected{Sent: 2, Total: tokens{Requests: 2, Prompt: 20, Completion: 10}, + Groups: map[string]*tokens{"g": {Requests: 2, Prompt: 20, Completion: 10}}} + db := dbTotals{Dialect: "bifrost", Rows: 2, Success: 2, Prompt: 20, Completion: 10, Spend: 0.5, WithTeam: 2, + WithUser: 2, WithModel: 2, Metadata: 2, Groups: map[string]*groupTotals{"g": {Rows: 2, Prompt: 20, Completion: 10}}, + KeySpend: 0.5, TeamSpend: 0.5, UserSpend: 0.5} + var r report + checkDB(&r, bifrostSide, e, db) + if n := r.failed(); n != 0 { + t.Fatalf("%d checks failed:\n%s", n, r.markdown()) + } + if md := r.markdown(); !strings.Contains(md, "| team member spend = spend logs | bifrost | - | n/a |") { + t.Fatalf("member spend not reported as n/a:\n%s", md) + } + db.UserSpend = 0.25 + r = report{} + checkDB(&r, bifrostSide, e, db) + if n := r.failed(); n != 1 { + t.Fatalf("want 1 failure, got %d:\n%s", n, r.markdown()) + } +} + +func TestBifrostLogObjectsReadCutTokenUsage(t *testing.T) { + tot := &s3Totals{bifrost: true, Fields: map[string]bool{}, Metadata: map[string]bool{}} + obj := `{"status":"success","cost":"0.00055","team_id":"loadtest-bifrost-team-3",` + + `"token_usage":"{\"prompt_tokens\":100,\"completion_tokens\":5,\"total_tokens\":105,\"cost\":{\"inp",` + + `"input_history":"[]","responses_input_history":"[{\"type\":\"message\"}]","output_message":"",` + + `"responses_output":"[{\"id\":\"x\"}]","metadata":"{\"x-cost-center\":\"loadtest-bifrost\"}"}` + if err := tot.add([]byte(obj)); err != nil { + t.Fatal(err) + } + if tot.Prompt != 100 || tot.Completion != 5 || !near(tot.Cost, 0.00055) || tot.WithBodies != 1 || tot.WithTeam != 1 || + tot.Success != 1 || !tot.Metadata["x-cost-center"] { + t.Fatalf("totals %+v", tot) + } +} + +func TestComparisonsWithBifrostSkipTelemetrySchemas(t *testing.T) { + d := &dbTotals{Deployments: map[string]*deploymentTotals{}} + m := map[string]*family{"litellm_spend_metric_total": {labels: map[string]bool{}}} + var r report + compareSides(&r, "rust", bifrostSide, d, d, m, map[string]*family{}, nil, nil) + if md := r.markdown(); strings.Contains(md, "metrics only on") { + t.Fatalf("compared metric names with Bifrost:\n%s", md) + } +} diff --git a/tests/test_litellm_rust/chat_completions/test_callbacks.py b/tests/test_litellm_rust/chat_completions/test_callbacks.py index 6743f9adbf0..ba5461bd24f 100644 --- a/tests/test_litellm_rust/chat_completions/test_callbacks.py +++ b/tests/test_litellm_rust/chat_completions/test_callbacks.py @@ -22,13 +22,13 @@ _OBJECT: Final = TypeAdapter(dict[str, object]) async def test_returns_a_model_response_and_logs_success_once(server: RecordingServer, asynchronous: bool) -> None: recorder: Final = RecordingLogger() - result: Final = await call(server, asynchronous=asynchronous, callbacks=[recorder], temperature=0.25) + result: Final = await call(server, asynchronous=asynchronous, callbacks=[recorder], temperature=1) assert_served_natively(server) sent: Final = server.requests[0] body: Final = _OBJECT.validate_python(sent.body) assert sent.path == "/v1/messages" - assert body["temperature"] == 0.25 + assert body["temperature"] == 1 assert request_body(_OBJECT.validate_python(recorder.wait_for("log_pre_api_call")[0].kwargs)) == body assert isinstance(result, ModelResponse) assert result.choices[0].message.content == MESSAGES_TEXT diff --git a/tests/test_litellm_rust/chat_completions/test_requests.py b/tests/test_litellm_rust/chat_completions/test_requests.py index c8800501cd4..41aa9b33932 100644 --- a/tests/test_litellm_rust/chat_completions/test_requests.py +++ b/tests/test_litellm_rust/chat_completions/test_requests.py @@ -18,7 +18,8 @@ _OBJECT: Final = TypeAdapter(dict[str, object]) @pytest.mark.asyncio @pytest.mark.parametrize("asynchronous", (False, True)) async def test_positional_parameters_reach_the_wire(server: RecordingServer, asynchronous: bool) -> None: - temperature: Final = 0.35 + # The model takes no sampling params but temperature=1, as Python itself rules. + temperature: Final = 1.0 arguments: Final = ( (MESSAGES_MODEL, list(MESSAGES), None, None, 12.0, temperature) if asynchronous diff --git a/uv.lock b/uv.lock index d473444177d..5d2ac88996d 100644 --- a/uv.lock +++ b/uv.lock @@ -85,7 +85,7 @@ wheels = [ [[package]] name = "aiohttp" -version = "3.14.3" +version = "3.14.5" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "aiohappyeyeballs" }, @@ -98,126 +98,127 @@ dependencies = [ { name = "typing-extensions", marker = "python_full_version < '3.13'" }, { name = "yarl" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/58/d9/22ce5786ac0c1653ae8b6c23bded02c1686d11f0dbb45b31ce128e0df985/aiohttp-3.14.3.tar.gz", hash = "sha256:9491196535a88924a60afd5b5f434b5b203b6cc616250878dbdb223a8f7844bc", size = 7971213, upload-time = "2026-07-23T01:57:27.037Z" } +sdist = { url = "https://files.pythonhosted.org/packages/6c/4c/bdccd81e9ee225b69c60e7766c9a5b05364f118f4d383713b89a682d772d/aiohttp-3.14.5.tar.gz", hash = "sha256:5558a7f5a05af9ecf744af91e5baefc436f93c9333e656c27ec253f9a6bbe178", size = 8078052, upload-time = "2026-10-11T01:05:12.408Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/2d/4d/4a99fb425c5e0cad715eea7bd190aff46f38b959a0a2dadb993705d34b26/aiohttp-3.14.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:eb0495d778817619273c108784292be161a924b9f5ae5cbbc70a2caa6838250b", size = 765848, upload-time = "2026-07-23T01:52:08.217Z" }, - { url = "https://files.pythonhosted.org/packages/74/e8/43b85dc55b8e950dc644babe762add781319ea881b57b33d2cce12017d12/aiohttp-3.14.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:c3c200cf9757edd785051dc699c7ecbec22110dbfcb3fefc7a9f9695eda8ea7a", size = 517476, upload-time = "2026-07-23T01:52:10.846Z" }, - { url = "https://files.pythonhosted.org/packages/7f/9e/73b582c4dbbc3c12ef4473822475effaabf1f934b56f14f5b03fe5d3a2af/aiohttp-3.14.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:fd51ebf9d3a00c074df4ede271023f4d2dba289bcc740b88191872716014e3c5", size = 515334, upload-time = "2026-07-23T01:52:12.636Z" }, - { url = "https://files.pythonhosted.org/packages/79/03/e98c3c9e05a5bdf97defe5ff9169baba4f0ec9a901f2d60e0f060c2f051e/aiohttp-3.14.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:134ac5ddcf61c6fad984b9a5727d83492ada43d63471db20fb73042c13fca62f", size = 1708830, upload-time = "2026-07-23T01:52:14.538Z" }, - { url = "https://files.pythonhosted.org/packages/d7/2c/26e60b694844dfd2176c57f913a22d0cd6a16f9ff202cbda7580d0328b98/aiohttp-3.14.3-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:70c987b27534f9ae1a723f47ae921571d616da21d3208282bf4c52af5164ac43", size = 1674012, upload-time = "2026-07-23T01:52:16.486Z" }, - { url = "https://files.pythonhosted.org/packages/38/65/672df92e3172cd876aacfa97a952ac560877eb169384b2991ac5b273de4c/aiohttp-3.14.3-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1b59533861b70a2185c8f4f350f791f39d64358ef6944ce71c5240c9ec0982c9", size = 1767015, upload-time = "2026-07-23T01:52:18.28Z" }, - { url = "https://files.pythonhosted.org/packages/9e/c5/228dec7bfec1c373cc2217cdeb47d6456dcd7a13a4c55144930a75ae3851/aiohttp-3.14.3-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1c5281acc88b92396f88c7e1e2748f8466689df22b80170e4f51efa712fb47a8", size = 1858700, upload-time = "2026-07-23T01:52:20.08Z" }, - { url = "https://files.pythonhosted.org/packages/bd/ff/cb36724e8c8d17f90ada567a9ff3efe1d6e9b549fba697a242aece180f21/aiohttp-3.14.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:48d67b87db6279c044760787eb01f6413032c2e6f3ba1cafaa492b1c8e578479", size = 1714075, upload-time = "2026-07-23T01:52:22.071Z" }, - { url = "https://files.pythonhosted.org/packages/9f/3a/296a4135c6366376263aeef54b15caca1f07676c2ae0c525d7832f2f808a/aiohttp-3.14.3-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f53bcd52f585e1ac3e590d61434eb61f9a88c38df041b4ea126d97144344a77b", size = 1588234, upload-time = "2026-07-23T01:52:23.757Z" }, - { url = "https://files.pythonhosted.org/packages/7d/81/9d5d853ef892dc066d1eb6db0e87a47348b920c1c879aa554612fdbd9d79/aiohttp-3.14.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:0fdea2281997af69da84c77ffa6f5938a0285f21fb3887c249d67419ca865b3d", size = 1677300, upload-time = "2026-07-23T01:52:25.861Z" }, - { url = "https://files.pythonhosted.org/packages/68/96/021d386ae32d9b26d4b88df2e794546232ff56bb6be952bf6be227c0bbc7/aiohttp-3.14.3-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:cda5fd5c95ad7a125a2e8464acc78b98b94c475a3780d6aa0aa157c93f470f4d", size = 1691501, upload-time = "2026-07-23T01:52:28Z" }, - { url = "https://files.pythonhosted.org/packages/29/9f/af66adce26a14af135c003cbd0f44ccaa68cebd30ff8ac99ca47fb4958f7/aiohttp-3.14.3-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:6debfa7312ff9d4c124dc71d72e9a0a4b9e0879e48ba6fcb42bef5c3300289e2", size = 1735113, upload-time = "2026-07-23T01:52:29.995Z" }, - { url = "https://files.pythonhosted.org/packages/2f/90/28c390d4c9851effe52ac25b5a2e1d92246acd00728b4fc7975dafb67484/aiohttp-3.14.3-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:f4e05329faa0ea1a404b37de4f034fd2c2defcca06a68dc6745e4e56c88e8a48", size = 1577486, upload-time = "2026-07-23T01:52:31.937Z" }, - { url = "https://files.pythonhosted.org/packages/db/c2/00e23a1bf2abb70dd353f6987db7e7f2491d0261f7363997738c71c98f95/aiohttp-3.14.3-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:a3a8296e7ab5c295f53f1041487cb088e1480775aafbf7fe545d93b770a0f96f", size = 1751353, upload-time = "2026-07-23T01:52:33.688Z" }, - { url = "https://files.pythonhosted.org/packages/6e/7d/d51a706a8cbfa57f0611127daf61ab3ae02ab8420b0407412079227d1c65/aiohttp-3.14.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:5373dc80ad1aa2fb9ad95c83f24eef418bbda3a61375f128e5b0192e4f3f9b32", size = 1698681, upload-time = "2026-07-23T01:52:38.167Z" }, - { url = "https://files.pythonhosted.org/packages/ec/b0/90bd5cd9fdd9787cb4211d284d1fb8401339a933cb0227a15b71e789232f/aiohttp-3.14.3-cp310-cp310-win32.whl", hash = "sha256:a3e22975f905b89a55a488c2a08f2fdb2186175349e917d48985cc468a3d4c6e", size = 456733, upload-time = "2026-07-23T01:52:41.823Z" }, - { url = "https://files.pythonhosted.org/packages/d8/15/fe5b8f6a71ae112bc677163d0b0701bda5dc15005249582258ede0eb88c7/aiohttp-3.14.3-cp310-cp310-win_amd64.whl", hash = "sha256:bdd0e2834dce1a26c1bbe26464861e16bbe217042cbff619247c11594472518c", size = 480460, upload-time = "2026-07-23T01:52:43.905Z" }, - { url = "https://files.pythonhosted.org/packages/54/00/45e98b6645cd7f00a4b78b749ebd309094b0eaeb2d2e96157eadbc0d0050/aiohttp-3.14.3-cp310-cp310-win_arm64.whl", hash = "sha256:eac645b09bcfdf73df7536331f0678c1086ea250981118ddb5199e17ccef72bb", size = 453479, upload-time = "2026-07-23T01:52:46.075Z" }, - { url = "https://files.pythonhosted.org/packages/f8/5c/b3e4ff8ad43a8afef9602c5e90285936da1beaea8b029016b793891f03c3/aiohttp-3.14.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:e568e14940c09955aa51f4e645b6daa18a581c5dcfcd73744dcc86a856e3ced3", size = 764250, upload-time = "2026-07-23T01:52:48.525Z" }, - { url = "https://files.pythonhosted.org/packages/0e/da/f1b384465e51449d844056b75070461da03a9a23e6c1747003695bf4172a/aiohttp-3.14.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:54cfcdee2770dac994417cbb0ee1f3eb0e7cb6b30c79bf44f2c02ff79ec5124a", size = 516281, upload-time = "2026-07-23T01:52:51.047Z" }, - { url = "https://files.pythonhosted.org/packages/b9/3f/01264f820ee2e3712a827892b1cd6ff80f3300c1fcbffbb45714a915d47a/aiohttp-3.14.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:21c016079415ed3fd676963e9793700a566d85dbbd6bfc564b9b2d209147dcc8", size = 514742, upload-time = "2026-07-23T01:52:53.779Z" }, - { url = "https://files.pythonhosted.org/packages/9e/8d/a71c6f2db52ac1ed142b133f7feddaa6b70539c3f4de24d7e226c95b794c/aiohttp-3.14.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d6088ec9894113802bddb3c09e974929aed2c7b3a8c456219b8aab4481f1a239", size = 1780613, upload-time = "2026-07-23T01:52:56.948Z" }, - { url = "https://files.pythonhosted.org/packages/a5/11/3dd9b3fb3a170f6ec9011b5291d876a6fab4086714c9e158600edf01b4fd/aiohttp-3.14.3-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:16ea7e24c309fb7c0bbd505d149abe4fe4dccfb8db911db7dbec0921bc889a6f", size = 1737688, upload-time = "2026-07-23T01:52:59.294Z" }, - { url = "https://files.pythonhosted.org/packages/6d/3e/834c26918be7d88068822b40e0db30fca50b5f4fe79104aa16a93f1d74e6/aiohttp-3.14.3-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:56f355e79f71aef2a85c80305cc915f894b170dba76de5fe84f6351939b83c06", size = 1845742, upload-time = "2026-07-23T01:53:01.641Z" }, - { url = "https://files.pythonhosted.org/packages/cc/c9/49ab8572df7d66bc13d11e31f781292badb04180dd87ba98733066c6aed7/aiohttp-3.14.3-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:18c441d0a8fca6de8d1f546849b9f0ab20d435993e2c5b59562b2fae6be2f929", size = 1928412, upload-time = "2026-07-23T01:53:04.018Z" }, - { url = "https://files.pythonhosted.org/packages/a5/b9/2b8f0c0ce09c87a1daf80fd483431b56b1435d3f62789bc86f572e1245de/aiohttp-3.14.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:53e7b4ce82b54a8bcc71b3b67a5cbd177ca1d7f592cbc92cd38b7349f73482db", size = 1786220, upload-time = "2026-07-23T01:53:06.481Z" }, - { url = "https://files.pythonhosted.org/packages/85/00/9c45f81de11710460edfa1dc81317b6e882703b160926c879a9d20da9fcc/aiohttp-3.14.3-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f55119f7bf25f49ed210f6096090715da24f2943c62102448915fde3c62877ce", size = 1637231, upload-time = "2026-07-23T01:53:10.258Z" }, - { url = "https://files.pythonhosted.org/packages/19/ce/967d628e910756f3539c6107cb7844a1b69440dcb3029a5ee7871b09ab63/aiohttp-3.14.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:9aa6e61fdf20105c4144e755bd586008ff450791d67b1c8146fdc15959c4d51c", size = 1753161, upload-time = "2026-07-23T01:53:13.817Z" }, - { url = "https://files.pythonhosted.org/packages/11/b2/0c3d4114f0aee4f580f5b3b4eb71b24d7a23b834ea506a4dfebe76513f35/aiohttp-3.14.3-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:ccd4893707b3e2a13e39c90d43cf80edf2e4d0457935bcc103bf2346214c3f15", size = 1756356, upload-time = "2026-07-23T01:53:16.211Z" }, - { url = "https://files.pythonhosted.org/packages/63/5d/99e7d91c82f1399d1ae2a854e080bd1493fbc31e5e959dbc4ec33dac3bec/aiohttp-3.14.3-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:b2466434105a4e03113c36ec775cc2ebe6676b62eae326fa670bb607ef788c1c", size = 1819846, upload-time = "2026-07-23T01:53:18.289Z" }, - { url = "https://files.pythonhosted.org/packages/ad/05/d5e1cb6480eeffd3f901d40a2c5e2d1e7effdc797837da3b490272699f13/aiohttp-3.14.3-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:ba59d59aba08ac02fc03b0c8983ccd5ee39a199d0552ce9e6d2b4845b34d59ae", size = 1628531, upload-time = "2026-07-23T01:53:23.86Z" }, - { url = "https://files.pythonhosted.org/packages/c9/90/b934682bcaefae18a9e04f3dff5b68522ba810906358ae5029b68110ea3b/aiohttp-3.14.3-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:ed099d105449c4f9e84f24af203cd131349d4761d8813fa7e02c32e7128cd910", size = 1832712, upload-time = "2026-07-23T01:53:27.551Z" }, - { url = "https://files.pythonhosted.org/packages/21/df/6061679faaf81fac746e7307c7adb71e858071a5d34c27583afefc64f543/aiohttp-3.14.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:152516815ef926786a0b6ae2b8f1fd2e0c71582dee0b435636865316fd4891b7", size = 1775014, upload-time = "2026-07-23T01:53:30.223Z" }, - { url = "https://files.pythonhosted.org/packages/8a/1d/f854878bbc69b88faefe924b619a34a6f59ec05fd387c77690667eaa75eb/aiohttp-3.14.3-cp311-cp311-win32.whl", hash = "sha256:a4af35c443e0b1a1bd6a8af3f3485d7fda15c142751a00f3ff8090f0b93346fa", size = 456006, upload-time = "2026-07-23T01:53:34.97Z" }, - { url = "https://files.pythonhosted.org/packages/73/0c/2af9d1674baccd1dbd47282a93d660a22e57ef6167c856deb24b4214fbab/aiohttp-3.14.3-cp311-cp311-win_amd64.whl", hash = "sha256:e1e74298bab6ee0d6e749ed4fd1901c7e604bdda32c03d787a2cc71c46d0433d", size = 481069, upload-time = "2026-07-23T01:53:39.673Z" }, - { url = "https://files.pythonhosted.org/packages/8e/76/88401ff3fc95e85c5fc38d588f36f55e61ecb64343b2bc8d69326f453cc0/aiohttp-3.14.3-cp311-cp311-win_arm64.whl", hash = "sha256:03cd2bde3d7f085b64e549c985f4bb928cad7e8ecf5323bfca320db548d81b39", size = 453021, upload-time = "2026-07-23T01:53:43.749Z" }, - { url = "https://files.pythonhosted.org/packages/18/d4/eb96299230e20acf2efae207cb8d69051f1f68e357e5ea5e479bf6fb097a/aiohttp-3.14.3-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:39aded8c7f3b935b54aab1d8d73c70ec0ee2d3ec3b943e0e86611bc150ba47f5", size = 754690, upload-time = "2026-07-23T01:53:47.332Z" }, - { url = "https://files.pythonhosted.org/packages/88/11/e7a70a209eb9a067c0d3212b518a0134e3484f5178c7533878b6b514d469/aiohttp-3.14.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:5bcb6ff3fdab1258a192679ff1a05d44f59626430aa05cd1a9d2447423599228", size = 509484, upload-time = "2026-07-23T01:53:51.159Z" }, - { url = "https://files.pythonhosted.org/packages/30/07/4bbc222cc8dbe31d4c3e8a5baad2286e4d42026ac0c570027b89afce6344/aiohttp-3.14.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:617105e2c3018ee38d0c8ce5ee3c84f621a6d8b9f723202aacaff28449ca91ee", size = 511949, upload-time = "2026-07-23T01:53:55.083Z" }, - { url = "https://files.pythonhosted.org/packages/54/b9/42e74c46b7b7c794b995bbc1f573fb48950c38b19d8600c62a6804ee2d67/aiohttp-3.14.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f631fe87a6f30df5fbe6d79640b25e4cffb38c31c7fb6f10871517b84b0f8c1a", size = 1765282, upload-time = "2026-07-23T01:53:59.662Z" }, - { url = "https://files.pythonhosted.org/packages/6b/ed/62bc4d74363ad346d518e0720363a949f63e2e23439a79eb5813d4d29bb3/aiohttp-3.14.3-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:a94dbaae5ae27bd849c93570669bff91e0510f33a80805738e3de72a7be0447b", size = 1741511, upload-time = "2026-07-23T01:54:04.063Z" }, - { url = "https://files.pythonhosted.org/packages/d0/9f/181e8a8bc79e47d13c7fc4540bd7a3b729d9505609c61f392a8dd2fbfe55/aiohttp-3.14.3-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8f2f1c4c032c7cedd7d8da6f54c97b70266c6570c3108d3fdffee7188bb70529", size = 1810680, upload-time = "2026-07-23T01:54:09.882Z" }, - { url = "https://files.pythonhosted.org/packages/5c/9a/dec94d6ad694552fe3424e3f1928d7a606a5d9d9433a04e7ecdd9d38ae7f/aiohttp-3.14.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ea05e1f97ceea523942d9b2a7d7c0359d781d683d6b043f5943a602b14da4787", size = 1905646, upload-time = "2026-07-23T01:54:13.475Z" }, - { url = "https://files.pythonhosted.org/packages/52/b7/7cd31f29d6055bd711ae6e669367fba6f5ae9de463910a793e30556a8db7/aiohttp-3.14.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:543906c127fb1d929b95076db19b83fa2d46751006ff1e23b093aa5ac4d8db42", size = 1792122, upload-time = "2026-07-23T01:54:15.752Z" }, - { url = "https://files.pythonhosted.org/packages/66/73/10b1ef93afa61f4963c746257b70ced619cf31a4798671de5fdb2608501d/aiohttp-3.14.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0a5ff2dfbb9ce645fa5b8ef3e02c6c0b9cc3f6030ff863d0c51fffc50cb5541b", size = 1591127, upload-time = "2026-07-23T01:54:19.489Z" }, - { url = "https://files.pythonhosted.org/packages/49/ed/3b203fa6de1b338c14acdc06bf6ca9b043b7944f005966958c2ced932cde/aiohttp-3.14.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:041badb8f84396357c4d3ad26de6afd7a32b112f43d3c63045c0c8278cfd2043", size = 1725210, upload-time = "2026-07-23T01:54:24.129Z" }, - { url = "https://files.pythonhosted.org/packages/28/b7/1c2aab8c706436dcc28598452488ac9cd7c409da815237c28c27d58993e6/aiohttp-3.14.3-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:530125ee1163c4219af35dc3aa1206e541e7b31b6efc1a3f93b70a136f65d427", size = 1764848, upload-time = "2026-07-23T01:54:27.973Z" }, - { url = "https://files.pythonhosted.org/packages/54/50/94c28f08b131c4bf10984ea2c7a536c9920608bb2d6e7f95642c30cc87b7/aiohttp-3.14.3-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:c8653fd547c93a61aadc612007790f5555cdd18946fa48cf45e26d8ea4ea473d", size = 1777102, upload-time = "2026-07-23T01:54:31.775Z" }, - { url = "https://files.pythonhosted.org/packages/13/d4/e7d09ba7d345fb2d74440fd2fa033c5e079fac05552927705986f41a364f/aiohttp-3.14.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:89176250f686cb9853c0fb7ead90e639e915b84a6f43eedc2a4e7ec21f1037f0", size = 1580205, upload-time = "2026-07-23T01:54:34.518Z" }, - { url = "https://files.pythonhosted.org/packages/a3/84/072a91d68e1e1eb587985b54baab94221277f877e8ef274fc213a0ceae28/aiohttp-3.14.3-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3a26434dafe408229ff3403458ca58de24fb51936504decac49ce6755f77e59d", size = 1797219, upload-time = "2026-07-23T01:54:36.995Z" }, - { url = "https://files.pythonhosted.org/packages/e0/eb/aad34e897e668424d6e995da5dff8a4a09af93363d3392488772957a63aa/aiohttp-3.14.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:d1558173930a5a8d3069cee5c92fc91c87c4dbcb099debbb3622053717145a19", size = 1768629, upload-time = "2026-07-23T01:54:40.103Z" }, - { url = "https://files.pythonhosted.org/packages/b6/2b/6bb88ddba0fecd9122aa3ebcad25996cf6c083a4a7040dbb3a4f97972af6/aiohttp-3.14.3-cp312-cp312-win32.whl", hash = "sha256:16100ad3ab8d649fdfbee87602d9d2dcdca9df0b9eda8a1b5fdc0d41f96da559", size = 451481, upload-time = "2026-07-23T01:54:42.547Z" }, - { url = "https://files.pythonhosted.org/packages/76/9b/f2f8f108da17ecef2cc3efc424e8b7ad3782b1a8360f7b8eae8ced84f6ea/aiohttp-3.14.3-cp312-cp312-win_amd64.whl", hash = "sha256:33a2d7c28d33797a2e99923dffa63f83d908a19b6bf26cfe80fa790aa5e1a75a", size = 476845, upload-time = "2026-07-23T01:54:44.853Z" }, - { url = "https://files.pythonhosted.org/packages/3e/44/28dac80a8941b604f4da10ce21097614ca1bf905ce93dca28d8d7de9c1e7/aiohttp-3.14.3-cp312-cp312-win_arm64.whl", hash = "sha256:362a3fd481769cac1a824514bcd86fda51c65e8fe6e051099e008fddde6db17c", size = 448050, upload-time = "2026-07-23T01:54:47.087Z" }, - { url = "https://files.pythonhosted.org/packages/57/be/5afd201cc0ab139029aadb75392efe85a293403d9dd3a3226161c21ce00c/aiohttp-3.14.3-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:2e9878ae68e4a5f1c0abe4dd497dbc3d51946f5837b56759e2a02e78fa90ef86", size = 506269, upload-time = "2026-07-23T01:54:49.075Z" }, - { url = "https://files.pythonhosted.org/packages/22/09/dec8189d62b45ade009f6792a2264b942a90cb88aeaf181239933cd72c3c/aiohttp-3.14.3-cp313-cp313-android_21_x86_64.whl", hash = "sha256:f3d2669fe7dec7fc359ecdb5984b29b50d85d5d00f8c1cb61de4f4a24ee42627", size = 515166, upload-time = "2026-07-23T01:54:51.894Z" }, - { url = "https://files.pythonhosted.org/packages/28/24/2854869d29ed8a8b19d74f9ec6629515f7e04d02dd329d9d179201e58e47/aiohttp-3.14.3-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:cc7cb243a68167172f48c1fd43cee91ec4b1d40cefd190edd43369d1a6bc9c82", size = 486263, upload-time = "2026-07-23T01:54:54.223Z" }, - { url = "https://files.pythonhosted.org/packages/d4/dd/57187c8be2a35aea65eaee3bd2c3dcbbcf0204f5106c89637e3610380cd1/aiohttp-3.14.3-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:78253b573e6ffab5028924fc98bc281aae05445969982a10864bc360dea2016c", size = 492299, upload-time = "2026-07-23T01:54:56.236Z" }, - { url = "https://files.pythonhosted.org/packages/b9/11/06ae6ed8f0d414edf4068861e233d8fe23ee699bfd4b3ceb8663db948a62/aiohttp-3.14.3-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:7041d52c3a7fa20c9e8c182b534704abb19502c8bdcbde7ab23bfda6f642394f", size = 502235, upload-time = "2026-07-23T01:54:58.377Z" }, - { url = "https://files.pythonhosted.org/packages/7e/a3/559639c34a345d2cf7c52dff6838119f2eaf29eb508227b5b83f573af813/aiohttp-3.14.3-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ac74facc01463f138b0da5580329cfcc82818dea5656e83ddcd11268fc12ff80", size = 750883, upload-time = "2026-07-23T01:55:00.65Z" }, - { url = "https://files.pythonhosted.org/packages/91/cd/41e131f13afd1e7b0172a9d9eda085ef90eb8439f41f0d279db81ed3ae60/aiohttp-3.14.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d6218d92e450824e9b4881f44e8c09f1853b490f9a64130801024a4793b1b3b0", size = 508473, upload-time = "2026-07-23T01:55:02.945Z" }, - { url = "https://files.pythonhosted.org/packages/bc/6b/e7f13410d391c6e55b4c007a8de024355389d7d459e3d64c42b2d33617e5/aiohttp-3.14.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:11fb37ef075669eee52ab1928fbf6e1741fada40409fa309ebde9607a962aebf", size = 509190, upload-time = "2026-07-23T01:55:05.173Z" }, - { url = "https://files.pythonhosted.org/packages/97/21/6464573e53d69672cc1eada3e5c5cb2d2efa82701e8305a0f2047a576967/aiohttp-3.14.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:55bdcc472aafe2de4a253045cc128007a64f1e0264fb675791e132ea5edaa3bd", size = 1761478, upload-time = "2026-07-23T01:55:07.383Z" }, - { url = "https://files.pythonhosted.org/packages/1a/81/d217043a4c17fbce360905e3b2bdd20139ebc9a2de836d035d179c4da006/aiohttp-3.14.3-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c39846c3aad97a8530c89d7a3869a8f8e9e3762c6ac0504481e5c80948f7e807", size = 1735092, upload-time = "2026-07-23T01:55:09.803Z" }, - { url = "https://files.pythonhosted.org/packages/a1/66/e13a02d0eeb1a9a502402a977abb4e4abff9fe4051c26f80558c57a7c975/aiohttp-3.14.3-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5895ef58c4620afe02fa16044f023dc4dafec08158f9d08874a46a7dbc0341b8", size = 1800546, upload-time = "2026-07-23T01:55:12.012Z" }, - { url = "https://files.pythonhosted.org/packages/26/5e/57d42fca1d18cb5acc1cad945d017fabc5d6ae71d8a08ad66be8dc3ee544/aiohttp-3.14.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fa9467a8113aa69d3d7c55a70ef0b7c636010a40993f3df9d9d0d73b3eb7ef24", size = 1895250, upload-time = "2026-07-23T01:55:14.357Z" }, - { url = "https://files.pythonhosted.org/packages/ca/1c/7da8d08e74d56f00070822f9638ff3f1c563f8ad87d1efa996c87bfc8644/aiohttp-3.14.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d7d2deec16eeedf55f2c7cf75b521ea3856a5177e123844f8fd0f114ce252cb5", size = 1789289, upload-time = "2026-07-23T01:55:16.668Z" }, - { url = "https://files.pythonhosted.org/packages/cd/0f/cf16bcf56896981c1a0319f5d5db9337994b5165730c48a8fa07e9b34be6/aiohttp-3.14.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:dd54d0e8717de95939766febac482ac0474d8ac3b048115f9f2b1d23a16e7db4", size = 1586706, upload-time = "2026-07-23T01:55:18.913Z" }, - { url = "https://files.pythonhosted.org/packages/fe/6f/76eac12a7f2480e1e304f842efdb07db33256b0d9165b866b6ef0806c202/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:df82f3787c940c94986b34222d59c9e38843fba85139f36e85255a82ad5355a9", size = 1724652, upload-time = "2026-07-23T01:55:21.296Z" }, - { url = "https://files.pythonhosted.org/packages/39/b6/19c8c592baeeb94b75f966547d40c02ac7590902306ec5863d5c027cf506/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:42a67efc36300d052fb4508a53e8b6901b9284b599ae63945c377569c5fcc1e1", size = 1756239, upload-time = "2026-07-23T01:55:23.705Z" }, - { url = "https://files.pythonhosted.org/packages/dc/c9/4e9383150296f97f873b680c4de8fb2cd88608fb9f48c79edcb111611abc/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:7a75aa63cbf9b21cfaf60dc2657e19df2c2867d91707d653fee171ffeedd1371", size = 1769161, upload-time = "2026-07-23T01:55:26.082Z" }, - { url = "https://files.pythonhosted.org/packages/aa/1e/147bdc6cc5de5f3ab011be8bf5d6e786633249f22c20bae06f85e45f5387/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:e92eb8acc45eb6a9f4935071a77edf5b85cc6f8dfad5cd99e97653c26593cdde", size = 1578759, upload-time = "2026-07-23T01:55:28.846Z" }, - { url = "https://files.pythonhosted.org/packages/fd/31/78388a9d6040ece2e11df62ea229a822cf5e52d238374b220ae9975b2623/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:b014a6ed7cf912e787149fdc529166d3ceabac23f26efeea3158c9aba2354e7e", size = 1792025, upload-time = "2026-07-23T01:55:31.457Z" }, - { url = "https://files.pythonhosted.org/packages/03/51/a3d29fdf2c25d796746af8ad6fe56a45d6256c38b0a8a2ed752e1160b3a2/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:3d4f72af88ac2474bb5bca640030320e3d38a0163a1d7533500e87be458eef71", size = 1768477, upload-time = "2026-07-23T01:55:33.87Z" }, - { url = "https://files.pythonhosted.org/packages/29/a6/442e18b5afeade534d877a2dc3c3e392aff8d49787890b0cf84790410267/aiohttp-3.14.3-cp313-cp313-win32.whl", hash = "sha256:5f08ec777f35ee70720233b8b9811d3bb5d728137f30ac91b7457709c3261ac0", size = 451069, upload-time = "2026-07-23T01:55:36.121Z" }, - { url = "https://files.pythonhosted.org/packages/9d/69/3d876ac02659f271cf7f6769f14a8e3de5b6e888ed8b5a7e998086a4cec8/aiohttp-3.14.3-cp313-cp313-win_amd64.whl", hash = "sha256:dff9461ec275f22135650d5ba4b4931a11f3958df7dfbb8db630000d4dee0883", size = 476518, upload-time = "2026-07-23T01:55:38.303Z" }, - { url = "https://files.pythonhosted.org/packages/b2/0e/50d6e6471cd31edce8b282bdec59375a3a69124d8a989a0b1313355cae52/aiohttp-3.14.3-cp313-cp313-win_arm64.whl", hash = "sha256:ddcac3c6b382e81f1dd0499199d4136b877beb4cb5ef770bbbfba56c4b8f55d2", size = 447676, upload-time = "2026-07-23T01:55:40.451Z" }, - { url = "https://files.pythonhosted.org/packages/c8/20/887fdcf832326571b370ffc347b3e70abe101096f3720126aac161b1d872/aiohttp-3.14.3-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:49f7325beb0f85ef4aef5f48f490269575f83e6e2acad00a1d80b807eb027062", size = 509067, upload-time = "2026-07-23T01:55:42.618Z" }, - { url = "https://files.pythonhosted.org/packages/ad/a3/92cec936f78cc4bf0fa5554ebe593b73459d94e3c62303e1902a4cccb6f7/aiohttp-3.14.3-cp314-cp314-android_24_x86_64.whl", hash = "sha256:e3be98a7c30b8c25d573dafba7171d66dfb05ee6a9070fc46535464ff97700a6", size = 514774, upload-time = "2026-07-23T01:55:44.937Z" }, - { url = "https://files.pythonhosted.org/packages/29/ba/2a0c38df3fc557620b6a5acd98364af050053b6285b4dc7ee74100c63c18/aiohttp-3.14.3-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:614c61d478b83953e261d02bb2df750f17227cd33ef8002945bf5aebbde21919", size = 488134, upload-time = "2026-07-23T01:55:47.135Z" }, - { url = "https://files.pythonhosted.org/packages/48/d6/d51b7d4bf309af3693940d8ffd2b9ed0b682434ef85959b7c9c137f60cf8/aiohttp-3.14.3-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:1caa7b0d05f3e3a36f87788c59e970a7ee1cefcfcbb924a9f138c4a6551c9cb7", size = 494201, upload-time = "2026-07-23T01:55:49.451Z" }, - { url = "https://files.pythonhosted.org/packages/3f/5a/8f624384e5f1efabb5229b94157eb966b021e97bdb188c62860c2ae243c2/aiohttp-3.14.3-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:dfa68deb2a443bdaa3ea5297b0699c1464f08aef3812b486d1348eee61b07dc0", size = 502766, upload-time = "2026-07-23T01:55:51.656Z" }, - { url = "https://files.pythonhosted.org/packages/a6/26/4ff0164370deec18fb19254ee4ab10b7a73304ac0c860b13f5f84663759b/aiohttp-3.14.3-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:e72ee89e28d907a18f46959b4eb0bb06701cc7f8cf4366e00029e2ccfaaf5924", size = 756557, upload-time = "2026-07-23T01:55:53.964Z" }, - { url = "https://files.pythonhosted.org/packages/97/a3/7056b86dc0d9ec709ea9777eae3b0161428f943372f8b98c01c11593b682/aiohttp-3.14.3-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:ad4c8b7488d745d2ca4838ebd8ae5ba9b56341d30b1da43640e4ce87f9f49646", size = 510168, upload-time = "2026-07-23T01:55:56.22Z" }, - { url = "https://files.pythonhosted.org/packages/85/ed/0357a015892fd68058bf2d39d3fd1958e459b997a7db30aaa6aaa434ae96/aiohttp-3.14.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:db332af25642007330fca8be5c4d194caf2bea7a7fc84415aff3497af5dfee6b", size = 512957, upload-time = "2026-07-23T01:55:58.437Z" }, - { url = "https://files.pythonhosted.org/packages/47/d1/8aba53f15ccb2238405f5e9d30e2a8ca44f93878c26e7165ade00d374b1c/aiohttp-3.14.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:25bd2708db6bdf6a6630dd37bdcdfcb47c4434d22ac69c64665b802910140b30", size = 1750149, upload-time = "2026-07-23T01:56:00.856Z" }, - { url = "https://files.pythonhosted.org/packages/49/bd/40c3fee327529284375c6701cbb0fa4600cc2e8432af1378f897e2ef7d3a/aiohttp-3.14.3-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:cef89a58e628c4efcac3275c2d68083f82426dcdc89c1492a6f654f9f7ea6ab9", size = 1707685, upload-time = "2026-07-23T01:56:03.371Z" }, - { url = "https://files.pythonhosted.org/packages/2a/a3/ca0cc6724cca8114b05694abd916060758c79894c3aa5b012cdadc1bc28e/aiohttp-3.14.3-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c23ec8ee9d5ab2f5421f9c7fffce208435607af27fd46d4a44e031954352838f", size = 1803911, upload-time = "2026-07-23T01:56:05.817Z" }, - { url = "https://files.pythonhosted.org/packages/95/b5/85b099c299c3ffd38ad9b3e43694c8a346934e4a30c88c4fd5a841234f77/aiohttp-3.14.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e2667f0bbe7eb6c74eae5e9691441ad186e5845ca3cff63230fc09c4e7514f5d", size = 1876929, upload-time = "2026-07-23T01:56:08.413Z" }, - { url = "https://files.pythonhosted.org/packages/d5/b7/1da684a04175473fa4cddbf9a2f572e79514c3fd27a74597f43057d4f3da/aiohttp-3.14.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:18cb43369747b2ae007bd2655fb8e63a099c2ff1d207962943636dac989b3147", size = 1761112, upload-time = "2026-07-23T01:56:10.918Z" }, - { url = "https://files.pythonhosted.org/packages/d1/16/bc4b55e3e5cb175fd69c53c90d60d2f47797cb343da5106e23863dc4dba4/aiohttp-3.14.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d77640cc618c1d99fc4f8589c0f24a730adfa54eb1e57ef7bf0c8dfb78da898c", size = 1583500, upload-time = "2026-07-23T01:56:13.613Z" }, - { url = "https://files.pythonhosted.org/packages/2a/e8/13a9d957a1ee40837f46aa30f0f4c657e673ad86a2e6362a9f9be20d26d9/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:53e5179d8abb5710f8e83ba207c41c8d1261fcffd4616500e15ca2b7a33be10a", size = 1713940, upload-time = "2026-07-23T01:56:15.969Z" }, - { url = "https://files.pythonhosted.org/packages/38/05/d33c680c1bcf1c7e130f9cbfc1fc02fe8bb0c4af2a94a53dd5fb56131e5c/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:cd817772b2fcf2b8c0905795318485f9ec16eae60b29feb7f4c77085311637f0", size = 1724413, upload-time = "2026-07-23T01:56:18.591Z" }, - { url = "https://files.pythonhosted.org/packages/85/1d/af798d306f7a74b6a632dbcabcf62a4c91391b7582d2a8c6d7712e2cc54e/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:4e3ac92d90e92773b2362d506068e9a948192bd553e743c5b2429e28527c8661", size = 1770748, upload-time = "2026-07-23T01:56:21.074Z" }, - { url = "https://files.pythonhosted.org/packages/a8/92/ad720d472556a995049206867765e9410969684f86ee09423ff9969044c1/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:3f42e9b78301f11c8f861746175d8b9c1ccef713fcad9eab396e2f6db8ed4a22", size = 1577564, upload-time = "2026-07-23T01:56:23.475Z" }, - { url = "https://files.pythonhosted.org/packages/60/ad/0ed7586cbef7a884e23a752fa2bb987a122e6a5dd50dab109258d0a95193/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:9d9edccfe496b476db5f398d97b865e9a6752bcf8aec4eef8390ce20fb64bb41", size = 1782080, upload-time = "2026-07-23T01:56:25.994Z" }, - { url = "https://files.pythonhosted.org/packages/97/ea/dbaed0d73e8a69aad653b045dab451c67c2454bb731a37b45a86593e9422/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:1c5ec8fb1bcc31a8466f74aaf26c345d5c386fa4bd08a3f0eb9c7a4a3fe8b5bf", size = 1745813, upload-time = "2026-07-23T01:56:28.604Z" }, - { url = "https://files.pythonhosted.org/packages/81/1b/6893d4bc57e434fc93a6c9217c637d967a0b651d989f6e3265179375754a/aiohttp-3.14.3-cp314-cp314-win32.whl", hash = "sha256:38901a84da3ce22249f6e860bf8f90d141bcab7da090cc398f8bb58c0e44b7da", size = 455872, upload-time = "2026-07-23T01:56:31.031Z" }, - { url = "https://files.pythonhosted.org/packages/f5/8b/c7baa1ba1eda4db6989baefe5de6d99834921b84ebd7918624febcb9f290/aiohttp-3.14.3-cp314-cp314-win_amd64.whl", hash = "sha256:8b3b60de05f3dcb6f6a00f818bb2ec781cee4de0645f59ccaf99b1d1823b6100", size = 481030, upload-time = "2026-07-23T01:56:33.365Z" }, - { url = "https://files.pythonhosted.org/packages/22/8c/c29d067df825a2df88ca432db848aa2fe8199598359cc06c12b09320cac9/aiohttp-3.14.3-cp314-cp314-win_arm64.whl", hash = "sha256:1576145bdceeb92382d899751e12743a3a5b8e460a841e3e50543859e54864dc", size = 453669, upload-time = "2026-07-23T01:56:35.731Z" }, - { url = "https://files.pythonhosted.org/packages/6a/a4/9c033beb355d39b6147980597ec9645e4729243f686ee4dc73945de72030/aiohttp-3.14.3-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:8800c996b01c2772a783e3e46f3e1abd5823029adca0df54231960de9bfefa5b", size = 791403, upload-time = "2026-07-23T01:56:37.972Z" }, - { url = "https://files.pythonhosted.org/packages/80/ca/87c32a0a7704583cfc49660bd817889bae5b830bf53b5dcb4e92145ac2da/aiohttp-3.14.3-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:ebe8e504f058fe91223351cecd2d9d6946c9d241bb0250d898ffbdf584cc72b0", size = 526413, upload-time = "2026-07-23T01:56:40.523Z" }, - { url = "https://files.pythonhosted.org/packages/9e/d8/8ec0e471248c500acdce2be3f46db8fb62b5eb60efef072529cc85ee1d26/aiohttp-3.14.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:30402d03a7c0ff52bce290b57e564e9079fd9d0cb545c8aba73f86a103162d2e", size = 532135, upload-time = "2026-07-23T01:56:42.876Z" }, - { url = "https://files.pythonhosted.org/packages/fe/45/f8919fd936e8b79fcd9bda7b6d8e62613462a713f4f17987fd7c34399142/aiohttp-3.14.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9fc7b5bfec6573f3ae844f457fdde5adeb713f8b8e4a81ad64fc207b49383716", size = 1922742, upload-time = "2026-07-23T01:56:45.528Z" }, - { url = "https://files.pythonhosted.org/packages/f6/ec/9ca76b28a27525b0cc53e20842e0228b022f301ce1f436b7d814b4aaf2df/aiohttp-3.14.3-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:8a5fd34f7f7410d1730d5c2ba873cacb2eed3fede366feb268a70ba22581ed8f", size = 1787371, upload-time = "2026-07-23T01:56:48.045Z" }, - { url = "https://files.pythonhosted.org/packages/b1/04/6acdbf17315f7b55f1937e3387acb89a3cddeb4995689553d064af8e92ab/aiohttp-3.14.3-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:270d3dace9ca2f10f0da5d8ebe519b7a310fc6112ed916e32df5866df0888553", size = 1912623, upload-time = "2026-07-23T01:56:50.605Z" }, - { url = "https://files.pythonhosted.org/packages/86/e6/438b0c79ca6f45eb9fd9817dd4c01a91919a38c0de5ee9e05e2b4dc0ece7/aiohttp-3.14.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3ae5b3a59436d089b5395d910121a390feed4d00578eb95a0fd1a329fe963100", size = 2005515, upload-time = "2026-07-23T01:56:53.153Z" }, - { url = "https://files.pythonhosted.org/packages/bb/6b/62cbd6577758699525f5c712d1ddef57d9875fbab0ae8d5f5a202fd598f8/aiohttp-3.14.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2498f0fe69ead802f9675beca44a7c21c62fdaa4ec5145ea1c3ad6edbee29f85", size = 1879906, upload-time = "2026-07-23T01:56:55.818Z" }, - { url = "https://files.pythonhosted.org/packages/00/95/18bcbf830a21dc3aae24d8f6b6feaf3db1d2090242d00a7868db2ffb0b67/aiohttp-3.14.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a0dc483c00da8b673abbb367eb6f8d8f4bcec30eb58529ea13cb42e7fd2dfa33", size = 1675849, upload-time = "2026-07-23T01:56:58.861Z" }, - { url = "https://files.pythonhosted.org/packages/a9/19/47f4968659c5e23606c3790c80fc624e691c153d036148449ee84d31b287/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c7d3a97c678d34fc5b59da671ee9cd630096ddc643e7b5a30d54a2a6f3574d3f", size = 1843496, upload-time = "2026-07-23T01:57:01.591Z" }, - { url = "https://files.pythonhosted.org/packages/64/af/38c33c4dd82fddcb4e56c4653b6f1072a8edbc6b7fa15809f14932c41e2d/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:f8fb78a83c9e5f741ca3a68cfb455c1f5bb83b4e7249a3848b3cd78d0a8563b0", size = 1827746, upload-time = "2026-07-23T01:57:05.131Z" }, - { url = "https://files.pythonhosted.org/packages/a1/9d/0537cda4885ac8f5b7053d164dd06312f4c483a4edcb8ee5b8aaf2a989bf/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:74ab5b6a9fb13e873e5a90946588baecaf488745e1db1a4a5c433f971f035098", size = 1853810, upload-time = "2026-07-23T01:57:08.043Z" }, - { url = "https://files.pythonhosted.org/packages/19/fe/26f9c5e6458385aa86497836b0dea6fb2f027827d63f37c7856cce9286ee/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:bd52f811e65f6fb634b1047159657c98f52b407f8efec907bcfc09da9a4c0a25", size = 1668895, upload-time = "2026-07-23T01:57:10.837Z" }, - { url = "https://files.pythonhosted.org/packages/ec/4c/618b1db9b9ba079b8875d2cdf78e7c4a3bf72903bd5850fee7dd9544600a/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:f0f177d1b195b9e06376cfd7d308d8a1b920909a609d03ac82a8c73bbb16d3b9", size = 1883833, upload-time = "2026-07-23T01:57:13.672Z" }, - { url = "https://files.pythonhosted.org/packages/94/c6/bd959bd1e4771f9fd944e9e436224c48c77b018b73b519b5aad346335bcc/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:498c6c623134f8e09a3c4e60bcd607a0b4590dd7dbf08dd40851b27cbb520ccb", size = 1844251, upload-time = "2026-07-23T01:57:16.593Z" }, - { url = "https://files.pythonhosted.org/packages/5e/19/08d41839658bdd44a0ed2480f3891705ecb487ce28c0dde62c9040c997e0/aiohttp-3.14.3-cp314-cp314t-win32.whl", hash = "sha256:b304db572b4368edd8dda8a2274f73156fe15558fca4a917cb8a09fc47af5963", size = 474180, upload-time = "2026-07-23T01:57:19.306Z" }, - { url = "https://files.pythonhosted.org/packages/99/5d/3cd6ef0a2b2851f7ab913b5b079334781bd50ff56a323e4454063377a080/aiohttp-3.14.3-cp314-cp314t-win_amd64.whl", hash = "sha256:b20032766aedf6261c7a566585a40867d092ac03a0d81592d5370ef9b054f99b", size = 500528, upload-time = "2026-07-23T01:57:21.762Z" }, - { url = "https://files.pythonhosted.org/packages/a4/37/cfd1ed540a4d318da025590d96b728e63713c09e9377950fc655dadeb856/aiohttp-3.14.3-cp314-cp314t-win_arm64.whl", hash = "sha256:2e1161602f45a54de2ce0905243a95f58cb42dcd378402f3697f5e0b21e9d2e7", size = 469280, upload-time = "2026-07-23T01:57:24.241Z" }, + { url = "https://files.pythonhosted.org/packages/ca/82/f7f0652e3a845e095d8505a2807a0fe5782abb71f99b2f516c93ff0be492/aiohttp-3.14.5-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:ef692a24087a699c0a4a26af45e746e0c1eae2116f6d8a5ff91d8aae2b867b45", size = 834351, upload-time = "2026-10-11T00:59:07.668Z" }, + { url = "https://files.pythonhosted.org/packages/95/c5/10cb0a5195bc3ec05a74ee494f8431a15e559800f2e869f3526ad4a16f04/aiohttp-3.14.5-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:1220353657ad49493551f089ce02f1a348fd57ffd585bfec77f2f3c4fe3a7346", size = 560663, upload-time = "2026-10-11T00:59:09.853Z" }, + { url = "https://files.pythonhosted.org/packages/32/dc/b7fefdd7dd64d080da2d69d609344191cc94b0e2eba70200c371014c9b36/aiohttp-3.14.5-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:330900acd0dc4cb8b27f9c127fbaad770964845338493e7906ae3822e82dbf8d", size = 553263, upload-time = "2026-10-11T00:59:11.5Z" }, + { url = "https://files.pythonhosted.org/packages/61/5f/3271cad1b34347e8bf27abee20d501ebdc56cec97e8ae59cd3fabb23d917/aiohttp-3.14.5-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0684952aeae1f5dbfe02d46039338513b94009baecd15d8e4098a357c4c4a2a6", size = 1843813, upload-time = "2026-10-11T00:59:13.143Z" }, + { url = "https://files.pythonhosted.org/packages/67/af/2011b30fe61889af112ad380e6fd44a93cca4b51296f612cac666c177882/aiohttp-3.14.5-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:d94e44be379e569758fee8a9a58431cfc3c2598c708b92b1cfe96c66b4c94aef", size = 1811243, upload-time = "2026-10-11T00:59:15.091Z" }, + { url = "https://files.pythonhosted.org/packages/e3/ba/72f3ac523ed48d38cdfb169f1b2a2e93ce06d9fcf4f46d0f5692e9539bcc/aiohttp-3.14.5-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5af42135fdfebdadbc2bcd9c0842a48ccf0d62794c36a260b21dc4b94d1e0119", size = 1898815, upload-time = "2026-10-11T00:59:17.224Z" }, + { url = "https://files.pythonhosted.org/packages/e6/69/bbf24f3b555ea1befa7db1e2510c728c8b0e80e61ce360beb7d94bc562b7/aiohttp-3.14.5-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4f07fe3ac408d8b3f768be471dc3f56d43843c47d97c66120534467a15ead197", size = 2022390, upload-time = "2026-10-11T00:59:19.244Z" }, + { url = "https://files.pythonhosted.org/packages/5d/38/01f5fc732d5373d6c099db0343279cd4e6607287d336bab6ff6ae9085a0d/aiohttp-3.14.5-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f375db73a39f5cf83696d500e21a67f418dc9a988955756f254be8f03b7b3651", size = 1856329, upload-time = "2026-10-11T00:59:21.19Z" }, + { url = "https://files.pythonhosted.org/packages/15/85/1f86fbd8fe34ad447c63e679404a1d5be435fe256a9d281b791fc1ed0d35/aiohttp-3.14.5-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8df7d481654ac96fe1ba9a02a9f67770fdd367823e0d5ef01b922725c4bd2cfa", size = 1712763, upload-time = "2026-10-11T00:59:22.982Z" }, + { url = "https://files.pythonhosted.org/packages/39/1e/ced51cf47e427e7aa6142c3bf6f33022472703bc9061289329532d3c40c0/aiohttp-3.14.5-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:e95c8def4b81c5d68d5cf1f54c07acd7c0d2577af244e5b6da802120825737c6", size = 1812125, upload-time = "2026-10-11T00:59:25.032Z" }, + { url = "https://files.pythonhosted.org/packages/6a/f1/9610b4e263b554ca4191501ba72b952efe06556b647da74887abc5d41049/aiohttp-3.14.5-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:f2ebb54b3f932210503072f09974b4fb574d823e497a944adfdcd140a6a00255", size = 1829550, upload-time = "2026-10-11T00:59:27.078Z" }, + { url = "https://files.pythonhosted.org/packages/ac/8a/1d1fb27cf9fa99b13133de33044c4d10d7705be0d4173d6d56abad1b67f9/aiohttp-3.14.5-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:038c2c7e8caa26b6c8423779b5eaf1893904048a512c19b32fe841ffa5592b50", size = 1868179, upload-time = "2026-10-11T00:59:29.315Z" }, + { url = "https://files.pythonhosted.org/packages/24/9f/3cfafa86ff025623dcca0d30335db7b8a6f3fa348148441208147e827e04/aiohttp-3.14.5-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:b7806e804889231b0e06469fd4a5c06313d1c0a3377322b6d9237fa5e0fe4167", size = 1697721, upload-time = "2026-10-11T00:59:31.161Z" }, + { url = "https://files.pythonhosted.org/packages/62/1d/86439a23bf32296e715d8c75535f0cd701821d36719324baef65bcd6af9e/aiohttp-3.14.5-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:9bab2045550c4fe0f7baf89574db1b455c195750702ba96fef1f16972b146617", size = 1887000, upload-time = "2026-10-11T00:59:33.329Z" }, + { url = "https://files.pythonhosted.org/packages/1d/0f/6044f2d02b3874966d84cd119f9b8eda46aa2991230078aea4dd17750cc9/aiohttp-3.14.5-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:96a2e584f0b9ed8f1fa33211397dcf67bb7069866402cb405d191c2f0defb9a3", size = 1842020, upload-time = "2026-10-11T00:59:35.384Z" }, + { url = "https://files.pythonhosted.org/packages/18/64/b49d2a34428ff937631b07dde74d5879a7a2c8559aab39331d40769e8cb6/aiohttp-3.14.5-cp310-cp310-win32.whl", hash = "sha256:602c1e9b718a3275c580149f947e7fac65044c0a20e599553fb12e9700da9eca", size = 489282, upload-time = "2026-10-11T00:59:37.061Z" }, + { url = "https://files.pythonhosted.org/packages/a6/7c/1114e723f1c4a65597820b605a0ddcd7f3b28fdfde3ac8d520d517ca582c/aiohttp-3.14.5-cp310-cp310-win_amd64.whl", hash = "sha256:bea559ad70218d230663e4210875735076a9bfea5994cef34a55a25faeaf2544", size = 512021, upload-time = "2026-10-11T00:59:38.7Z" }, + { url = "https://files.pythonhosted.org/packages/48/97/e08c646f4bad63d77876b24591d013d7f33f02fda7c931eafe3224f38467/aiohttp-3.14.5-cp310-cp310-win_arm64.whl", hash = "sha256:dca3fa8d8a0a26679862eccb0b1a9151b2b9f1cd2c212e7a6335778faaff5833", size = 493016, upload-time = "2026-10-11T00:59:40.525Z" }, + { url = "https://files.pythonhosted.org/packages/d8/f3/8997f18890a92c79f77fcfbb4f78ca17c2d4ab109e9eb6d31b4e9de194a0/aiohttp-3.14.5-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:d51db97c96384fbfcaf8f4c65922183a68b94f891c3c10c862ef5f6df2adbb1f", size = 832358, upload-time = "2026-10-11T00:59:42.195Z" }, + { url = "https://files.pythonhosted.org/packages/8e/42/084651e9efb5cadb99265f786df60f2a7b353cead7bfd2c87003cb4867ad/aiohttp-3.14.5-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ae53924aa853a7a2ca20ed4142c7c6b56338e4d4cd999e2980075b9efc2e257a", size = 559636, upload-time = "2026-10-11T00:59:43.677Z" }, + { url = "https://files.pythonhosted.org/packages/3d/fe/92838944e601f0fe195fd3f3ada37e29fbee3d19bfd323fe15937ec79d36/aiohttp-3.14.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:a2c473a355f9239efcb72c92d5abfd8fcdb0cc78c8e9af607e72ca12dbb36593", size = 552588, upload-time = "2026-10-11T00:59:45.592Z" }, + { url = "https://files.pythonhosted.org/packages/8f/b8/dd9b95c5c20ceae1b47738e656bd79e9883e3117a9cc8b143901eb604e9a/aiohttp-3.14.5-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:32e8fa6644e541fcd7e02430588c7fc93b602c1778ea0bc345505db76b61cfb4", size = 1928224, upload-time = "2026-10-11T00:59:47.294Z" }, + { url = "https://files.pythonhosted.org/packages/c6/47/70010cd2ba8746968d53d433ff328c33f06a66c99da5f3de759c64a6b2eb/aiohttp-3.14.5-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:579f97d5120f2971876d2ddca2968135f6944d00c44c3a6590ad7d86ca9b403f", size = 1877674, upload-time = "2026-10-11T00:59:49.472Z" }, + { url = "https://files.pythonhosted.org/packages/c5/a1/b026f071dbd87f0bdba86b92d47e46f3899e3ad7143b6e0fe7e7887090b4/aiohttp-3.14.5-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:24409db442e2fb6e766bc7f3943851a8381dec3098140e43bb2e843b79e31b12", size = 1985850, upload-time = "2026-10-11T00:59:51.296Z" }, + { url = "https://files.pythonhosted.org/packages/36/d1/f7b6f6f8c3cb5a71b53baeddc2e70c224c28b12a23a55c42994c504019b7/aiohttp-3.14.5-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5e8f97c0488ffda3082766ac0f2c8150a9a58c4d05788330e479cfd449b37939", size = 2105471, upload-time = "2026-10-11T00:59:53.191Z" }, + { url = "https://files.pythonhosted.org/packages/42/74/2a2b22953de15c6c8a80debf26db3047e4e8a5a5db6dd7c2c6c01eec0605/aiohttp-3.14.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:50a195903119008fe9cc68710535eb37f556ffffd6a7759afe70a2c145587045", size = 1929915, upload-time = "2026-10-11T00:59:55.609Z" }, + { url = "https://files.pythonhosted.org/packages/33/ad/f80e8d33933d0eacd0217efa3b7fb32c9f48ed480ed0db53cf9948bf474f/aiohttp-3.14.5-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0133c3c3b54a0bf1e71fa5c1ad95c93f07fd54e24ef1fe182f5122e1573d2bf1", size = 1773832, upload-time = "2026-10-11T00:59:57.611Z" }, + { url = "https://files.pythonhosted.org/packages/04/a4/0273d239f3e3bb69438f209c64c82e1f98dc60d5db264795e41f7dd05dcb/aiohttp-3.14.5-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c172db893e516e1358e65a95ee20b7ce7173963eefe318b6ab2a2220688b999e", size = 1901001, upload-time = "2026-10-11T01:00:00.605Z" }, + { url = "https://files.pythonhosted.org/packages/16/ed/7dd439d26c654645bc34ca3c9a822fc11c1c51801fed1cefdfe1cc41c1f6/aiohttp-3.14.5-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:f2a7966bda23dd85051f1661ce0ace38d6890e05ec6c357ecae9d2479cba377e", size = 1907218, upload-time = "2026-10-11T01:00:02.699Z" }, + { url = "https://files.pythonhosted.org/packages/32/dd/86249c3b8248562a17fd3e22ee0c164378d65766f01aceef62cf52783710/aiohttp-3.14.5-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:4887d130a7bbfed3a85493bb5a25e5b5b558d40c1d986dd16970d2bb26d63793", size = 1958650, upload-time = "2026-10-11T01:00:04.851Z" }, + { url = "https://files.pythonhosted.org/packages/bc/0b/ca4d53d68f683ce195807fd0aeb063bf6d1c6b39343fff234c9524e8d5a9/aiohttp-3.14.5-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:225c579c23b68b343cccea27a7e06e3bd8ec23a09c30b427eb3f1e4ca6239b20", size = 1760057, upload-time = "2026-10-11T01:00:06.978Z" }, + { url = "https://files.pythonhosted.org/packages/6b/42/005437ffd7fa56c2ce3347add40404b654a32754ce91f2e2a14b9e5ab2bc/aiohttp-3.14.5-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:ab52d8f1fc1b64821c1fbad64a647ed6203627004059a6d1ed4f0858a1499703", size = 1978081, upload-time = "2026-10-11T01:00:08.868Z" }, + { url = "https://files.pythonhosted.org/packages/03/71/3a5b66fe1b7b23d3c6524350817ed55de5feb7fbb4cccba7cbe044d118dd/aiohttp-3.14.5-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:cb131d775a1573c1aee66656bd78b023577bbdb6cb8349a07773bd4f73e68a6e", size = 1922954, upload-time = "2026-10-11T01:00:11.035Z" }, + { url = "https://files.pythonhosted.org/packages/69/84/d08a554f281d42fd7c6b2b6dd7738f3e3a7a9f657ddd87e11ca2091cdfe3/aiohttp-3.14.5-cp311-cp311-win32.whl", hash = "sha256:e87046c8ff77a8decdb6a41d8ab25824b47531b2da933aeab0c1e21c7acff329", size = 488101, upload-time = "2026-10-11T01:00:12.933Z" }, + { url = "https://files.pythonhosted.org/packages/0f/15/52b5e65f02b33686ae49f1518548ae4f5a7adaf31d9d3c54fa12a143137b/aiohttp-3.14.5-cp311-cp311-win_amd64.whl", hash = "sha256:6f275c11d1aa6d4c458e05a68be084efe3c55a113d99e3f46a318098e52948fc", size = 512575, upload-time = "2026-10-11T01:00:14.743Z" }, + { url = "https://files.pythonhosted.org/packages/b0/b9/bb75012635f3defb0f7cbb7c4ae391e36bb9cb3fbdca3b9944e2ebebf743/aiohttp-3.14.5-cp311-cp311-win_arm64.whl", hash = "sha256:b032a0023eb41d768ce77d83210ab2a3c389bc0b09313273c7e1eca48c10a755", size = 493179, upload-time = "2026-10-11T01:00:16.961Z" }, + { url = "https://files.pythonhosted.org/packages/d5/94/6ba86efddcb616c811b40e6a0dfdd862738f647e4e3860a961075d5e9ed8/aiohttp-3.14.5-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:df37b620684e19b5e25724412518ccafc3b1a49cdac706fdbd2f983fad943450", size = 817499, upload-time = "2026-10-11T01:00:18.901Z" }, + { url = "https://files.pythonhosted.org/packages/5e/e1/7bca6d84dabd228aa8eb4b7f9feac2586aaa9be5505d7d65bf287a615c43/aiohttp-3.14.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ef60869969180ec2464f1349aff07138ae35ca2200f0946cb3552e49e8f301a8", size = 549467, upload-time = "2026-10-11T01:00:20.854Z" }, + { url = "https://files.pythonhosted.org/packages/64/91/11b89f45ca486252dd67dd5f3231fec04bf5518da39a95cb3997619f17fb/aiohttp-3.14.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d079c0a0135c36e7beb6f1c88087c8f108dc5891cdd0b5eafa778421bda70ed2", size = 548067, upload-time = "2026-10-11T01:00:22.659Z" }, + { url = "https://files.pythonhosted.org/packages/22/ff/c6615806c14aab34f82b9424ccde8ce6e417315fd57ce1c5b4d4747888e1/aiohttp-3.14.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:abfda5cb094a829f7bc25216a32f7db2e85cc65bd59910f8e7b40b3d9b224764", size = 1896245, upload-time = "2026-10-11T01:00:24.638Z" }, + { url = "https://files.pythonhosted.org/packages/39/b2/25a8c971ae6a92c8394d77e42422d7f38989e05cf41a5ceb92d73d67ab7e/aiohttp-3.14.5-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:9cc882cf8619109583c906b4d4a85d6a111a98afa34b7a450d1e08118d016820", size = 1872180, upload-time = "2026-10-11T01:00:26.838Z" }, + { url = "https://files.pythonhosted.org/packages/2f/d5/99f93ea36cc5205e47c1e5a803e087f2ad21b5430b5db2e942cb6e988a37/aiohttp-3.14.5-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7457580535e019e1247ea35d6a02bf081ad30c26d0cbc210c93f6c3ab67a0835", size = 1937261, upload-time = "2026-10-11T01:00:28.74Z" }, + { url = "https://files.pythonhosted.org/packages/40/a6/9ac9c9e6695040bd73d2584a1b59a9388f6433c76d5294a7bf591e21ffe5/aiohttp-3.14.5-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c5ed596aedb9c42afd3fe0aae3117725378ac73d2cc5ddc735056fbdb96c5d02", size = 2064994, upload-time = "2026-10-11T01:00:30.623Z" }, + { url = "https://files.pythonhosted.org/packages/da/e4/aa172eb534b7f02f1f8ff1c3213347eaf3cf218db91a727c6863c22f1035/aiohttp-3.14.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:20f085697d7e911f1f73c43ed03fafbed1e7121797e2eb5428efa80398060584", size = 1919630, upload-time = "2026-10-11T01:00:32.548Z" }, + { url = "https://files.pythonhosted.org/packages/06/7d/4eedafc5bababa8932636c141e346806966eade12c0b7e5946d43bf8218b/aiohttp-3.14.5-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:74b0a9c8270f9b0a11410e124ff8d4f18bfc1f1837440ec84da5ae7b50927b5d", size = 1710016, upload-time = "2026-10-11T01:00:34.471Z" }, + { url = "https://files.pythonhosted.org/packages/b2/94/eee018537ba19da0ceb2ac79cab83faed4ac49568e08376e2799043f2538/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:19e2ba471507c34f8252402ab50f5ab512398b9ea8c8f1cb26beb3f75793ba30", size = 1858112, upload-time = "2026-10-11T01:00:36.581Z" }, + { url = "https://files.pythonhosted.org/packages/68/76/354653a306547238f3427283905972d796ba7c292ba9977abec9f2b6f260/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:d418ce2af40c6bb685b3f663e9e8de27cb0a22431d8e88a167348d7f01878073", size = 1896371, upload-time = "2026-10-11T01:00:38.478Z" }, + { url = "https://files.pythonhosted.org/packages/f2/ec/63e8c7136b570e356345ad3174e3820fdc973ea10712cb6c649bf875755a/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:70cb4008ac2ed1e0ca9e824deb4b53d3aa0d939109698ebf1e723a84337bd794", size = 1905836, upload-time = "2026-10-11T01:00:40.527Z" }, + { url = "https://files.pythonhosted.org/packages/57/d8/11365bda144b127928cd42533d0eff78a55613c9e28f81941bd6630ea887/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:a23fe35d776bc03cb495938b9594450d047e3bc08c5255315a82323e9cb7d2dd", size = 1696534, upload-time = "2026-10-11T01:00:42.686Z" }, + { url = "https://files.pythonhosted.org/packages/2f/3d/82df0461b18e00b2998f205c03e0d3010222478c43640aceb8e03dcd7e8f/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3e0eb43bed3c6801a6cee315195377789e90b2a72c2277a475b578535312488d", size = 1933970, upload-time = "2026-10-11T01:00:44.71Z" }, + { url = "https://files.pythonhosted.org/packages/03/ad/6ddfe0aacd931c17b53533336d97e9d11a98b96d6ae815a9da0b19f82ccf/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3be7dd397d64ca3e1869626fa9318aaebb54b7bf93bc72d7a205448d83e4f748", size = 1899501, upload-time = "2026-10-11T01:00:46.629Z" }, + { url = "https://files.pythonhosted.org/packages/9e/8e/189bdd9ae4793059bb09f6dc880f211a6c6c7859cebcbf33912dbfab7dd7/aiohttp-3.14.5-cp312-cp312-win32.whl", hash = "sha256:eb324e2009fb54db30a071dad7caf6998ee2879c4704007efb244514dad1fec1", size = 480370, upload-time = "2026-10-11T01:00:48.468Z" }, + { url = "https://files.pythonhosted.org/packages/ae/ce/1f08114679d49655b30a6e0a29858375c94b82c1de1a0bd0a20c2fee8b02/aiohttp-3.14.5-cp312-cp312-win_amd64.whl", hash = "sha256:2cc38a4f2b516bef1714e690df87a0e043faf1a7693c82d860091684453d5111", size = 506141, upload-time = "2026-10-11T01:00:50.272Z" }, + { url = "https://files.pythonhosted.org/packages/79/d4/c7b4f60b16a1b7e43249fa9031ae05e7e8c341ba4b7d1866f914dafeaa0e/aiohttp-3.14.5-cp312-cp312-win_arm64.whl", hash = "sha256:a63afd1f757de949028387e65a7127b61ad0f775432dbb0e62816ae619fe69ac", size = 488108, upload-time = "2026-10-11T01:00:52.321Z" }, + { url = "https://files.pythonhosted.org/packages/d3/e1/2841e020ebb7aefae5513586193e011e06313d9a6bdbd296622afbbce204/aiohttp-3.14.5-cp313-cp313-android_24_arm64_v8a.whl", hash = "sha256:9ad7e6aa38c20da1be697874349c4c273c8a03b7887169665081706398d0439a", size = 538932, upload-time = "2026-10-11T01:00:54.3Z" }, + { url = "https://files.pythonhosted.org/packages/f7/a6/7fb8ea8fe96bcc7b7c7a36d10f021d99d01a8dc8a4b3f0ddacecfad9a80e/aiohttp-3.14.5-cp313-cp313-android_24_x86_64.whl", hash = "sha256:f59c7673465908cbe506117176156c127f29f917677afceada34957179221d91", size = 553329, upload-time = "2026-10-11T01:00:56.442Z" }, + { url = "https://files.pythonhosted.org/packages/de/64/d056e3c27647dc25af1a592cf356245382ea7c808171b9dac7677afedfc8/aiohttp-3.14.5-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:1b5416552740edf07234cc9437d0706f2acb67b93c198670b1a68e1b2b587dec", size = 521367, upload-time = "2026-10-11T01:00:58.345Z" }, + { url = "https://files.pythonhosted.org/packages/3e/e4/95226147e11d4db916fd1d495dcf85af8e3816e38333e42718241196e848/aiohttp-3.14.5-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:43351bdb5e4c3cb7d1772368e988534e869a74db7778079a83782c11c69535c7", size = 528322, upload-time = "2026-10-11T01:01:00.211Z" }, + { url = "https://files.pythonhosted.org/packages/17/cd/1d3c9192cafdb51cad62b2d3ded96cff9cc8af51893210aadd448a325389/aiohttp-3.14.5-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:c8c4478bef6d57fcfda15dae461ea3c9f06aa7b257c58df3f2300174ccbb185a", size = 539874, upload-time = "2026-10-11T01:01:02.06Z" }, + { url = "https://files.pythonhosted.org/packages/f6/0c/dfa33aecc7d4d1dc75e05248f5eac5a0edf4d09e7b44d93ab62529b0c1db/aiohttp-3.14.5-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:c2c30484dd1417ef98b51021ffa2cc0d7f3c78918adaaaab7e70817335ab3e02", size = 813836, upload-time = "2026-10-11T01:01:04.01Z" }, + { url = "https://files.pythonhosted.org/packages/33/17/4a63738052d20567d55529d6daa1b9480d906fd52930fbcf6d3fbed618f0/aiohttp-3.14.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:dab9ac5a67c8d1f070c00fa8fccb7cbd1b8dcc1a8d6b42f37540df9b3d4cc603", size = 548070, upload-time = "2026-10-11T01:01:06.035Z" }, + { url = "https://files.pythonhosted.org/packages/15/e5/b57e58695a757fd4c02497c033fced96a69c631b13866c43d336530c9670/aiohttp-3.14.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e1cc2bfaee8c214f06080a7c7d5772419b8a1108e8e5349236189811823fb02a", size = 545786, upload-time = "2026-10-11T01:01:07.817Z" }, + { url = "https://files.pythonhosted.org/packages/9a/68/8c2c67a3aedf46e00f3c42f04fbc6983de80d4ed5786151e33681ba45883/aiohttp-3.14.5-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:74efb69332b85675b1eabd760a8cfc2e2cf42c60607c66f88014c1bdfb40942d", size = 1886492, upload-time = "2026-10-11T01:01:09.834Z" }, + { url = "https://files.pythonhosted.org/packages/ab/4b/74aab5e8d28c62e8f795b4fe8f38cf5586fd264a9a27bd2141ef6490333d/aiohttp-3.14.5-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:42b5e616946dbaf505e2bff18c9af2cd4ef9e7ef300ee58a6e951a5b7cf147ae", size = 1870057, upload-time = "2026-10-11T01:01:12.045Z" }, + { url = "https://files.pythonhosted.org/packages/a8/f7/eafc3b1988302b1815d9fd4a21071be5c360d616c0a430d02fd92dc97688/aiohttp-3.14.5-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f9033b43f511f27547c557dcaba0177649e10a3725336ccd2cce0fdc1dc4850d", size = 1925119, upload-time = "2026-10-11T01:01:14.09Z" }, + { url = "https://files.pythonhosted.org/packages/a1/04/78d8f294f74dd570f3898ff20402349fce524176045df98ba727d6846a68/aiohttp-3.14.5-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:15a310d3c71398e3d7bfc93a1a73fbe664315cd9e9016b8efc1cff85eeab7155", size = 2053137, upload-time = "2026-10-11T01:01:16.344Z" }, + { url = "https://files.pythonhosted.org/packages/32/51/395d225ef36f5a50d8e548dcd3141bfdbcd31fb6eed859022c573d2c4d66/aiohttp-3.14.5-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1ffa3a523a36d8628f98c06492ae16a31a23d14c0b4ec721757b477319f656d6", size = 1911038, upload-time = "2026-10-11T01:01:18.653Z" }, + { url = "https://files.pythonhosted.org/packages/ff/a4/2aec1aa06d82e8a244843b5dae31d78061e5e76744270a86dd0ee051c889/aiohttp-3.14.5-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5fb6a6e919bfb703227bc1ce6579281b84b1a2ba57deb9794dfdbec7dcd1e40c", size = 1702205, upload-time = "2026-10-11T01:01:20.904Z" }, + { url = "https://files.pythonhosted.org/packages/16/27/6051bfde7b6f418f70edd60d655fa426abb3355fa0981764739d87ecf160/aiohttp-3.14.5-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:43e1b7994a8b038125f722bff07492ef501110722c2727c408995d9fb864c421", size = 1848588, upload-time = "2026-10-11T01:01:22.918Z" }, + { url = "https://files.pythonhosted.org/packages/9e/44/55efc06fc26c4e6e1c095f231b4c222bf2d86d64a8eebbc24b2bb5958ea8/aiohttp-3.14.5-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:5f3e96071686755d9cd3600c3880183eb94b012178e92746d68101800f0ed8a3", size = 1887540, upload-time = "2026-10-11T01:01:25.278Z" }, + { url = "https://files.pythonhosted.org/packages/48/dc/1502bfdc2a65760d386ac6a00090b0addaa8a3c9c60b3f8127fad3a9afb2/aiohttp-3.14.5-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:cd88b01f3d37b7a2a34f91d98f14720206f1ea3d540843fab2d649dd5fb91fec", size = 1895030, upload-time = "2026-10-11T01:01:27.316Z" }, + { url = "https://files.pythonhosted.org/packages/93/7e/44174bb6288264418c9eec07a5e35180969c0d5a796c7af544db3cb8a33a/aiohttp-3.14.5-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f2ed8b64dc0c651c0f5a9c926777719770021251b8f336d97c4b80b660836ce1", size = 1693244, upload-time = "2026-10-11T01:01:29.39Z" }, + { url = "https://files.pythonhosted.org/packages/47/dd/b507d64e50db23888582fff08eda13998b12f9dea70c072edaac19218380/aiohttp-3.14.5-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:a9918e58faf62ba2c7147927d06057aec78f42475aff5048047ec47e7265a600", size = 1926308, upload-time = "2026-10-11T01:01:31.634Z" }, + { url = "https://files.pythonhosted.org/packages/98/4b/5b51b4f63e3f2793151f4aea49c48fe1e00baeb7cec9c7a206de499f8de0/aiohttp-3.14.5-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:42f320d4a5b00b9af0bddcfec5407dc6f2d9816f006b2f79ebbaa31f16895df3", size = 1888406, upload-time = "2026-10-11T01:01:33.715Z" }, + { url = "https://files.pythonhosted.org/packages/ff/13/d5e818a5eaba9f822016727299f41c0e1075799f82a6433ec508a93ab867/aiohttp-3.14.5-cp313-cp313-win32.whl", hash = "sha256:3ae800a20947e2c2e53088047d021e6bf7d51560cc49f6a0737a1f79d2e3a13c", size = 479970, upload-time = "2026-10-11T01:01:35.677Z" }, + { url = "https://files.pythonhosted.org/packages/8d/d0/8eca2c65aa467320990d78fb2005f38ed3588944280c39ff9deb6423fef1/aiohttp-3.14.5-cp313-cp313-win_amd64.whl", hash = "sha256:d05e94cdfe0d15d0206f970722d2554780ce562787b21b218b275447f8751319", size = 505985, upload-time = "2026-10-11T01:01:37.574Z" }, + { url = "https://files.pythonhosted.org/packages/7a/f8/4cdd65305d2fca14b886bea9ed2abb1fe726287872524e56e3d26692b47d/aiohttp-3.14.5-cp313-cp313-win_arm64.whl", hash = "sha256:f001b571ead90ca1770f1e616db255351a1703317f20374c361ef22f12c06d09", size = 487629, upload-time = "2026-10-11T01:01:39.477Z" }, + { url = "https://files.pythonhosted.org/packages/43/be/3184a1d34a8be665569eadb7e9e764b4629e4f3413e241cb2e4d6fecf3b3/aiohttp-3.14.5-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:939042d5cda21d41a6f512e7cc8b8e33a2aebff863352251da495fbd91b673b5", size = 540673, upload-time = "2026-10-11T01:01:41.354Z" }, + { url = "https://files.pythonhosted.org/packages/60/2a/d35f3ba4cf157b072e3b674bf9983047ca5ea5173c995d32d877e1191d36/aiohttp-3.14.5-cp314-cp314-android_24_x86_64.whl", hash = "sha256:6da32b5ff3fd78d244e37300463434c7145162bfd2b6e9e915ab164da37f7343", size = 554087, upload-time = "2026-10-11T01:01:43.719Z" }, + { url = "https://files.pythonhosted.org/packages/fc/d2/61a33880ca4eaca95a9c60ca3f6beed15555af1028652dfaac601627787b/aiohttp-3.14.5-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:1b438b73c38111818d0c9d6a5c2bfed8584c8e503a49ef085d70e874ec846738", size = 522613, upload-time = "2026-10-11T01:01:46.154Z" }, + { url = "https://files.pythonhosted.org/packages/31/1d/de579b299d2225dc2c6fd99d579d91f16c02a913fb5af9a3cf2fe9bd88ba/aiohttp-3.14.5-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:755933b107ea7a6a9ac916f635a70595a5b1a32fac10a8ff0b9f2ab88555550c", size = 529682, upload-time = "2026-10-11T01:01:48.477Z" }, + { url = "https://files.pythonhosted.org/packages/f4/4a/ddb923564e15e053b6e060b0036e1694dcadcb13aa476c5a87dcad20e336/aiohttp-3.14.5-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:b3cc509327c7b27f6f4727a8830f4004f6df7766e179f2f4b8e54e65c0bec5d3", size = 540120, upload-time = "2026-10-11T01:01:50.474Z" }, + { url = "https://files.pythonhosted.org/packages/3d/36/a640fbecaa53727a5900b892bdbe17b5f3e8cc88903e22864fe41b654def/aiohttp-3.14.5-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:7bd8ac754ebd6733a3e2a0dd1674c4d8ab086196803fd8dcd776f07b4e2607d9", size = 818795, upload-time = "2026-10-11T01:01:52.665Z" }, + { url = "https://files.pythonhosted.org/packages/ef/b6/d52ca608859e271b5fa7944074802dc45f60e52c318a4ddc34edbf73586e/aiohttp-3.14.5-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:e724a7b6091f0b1ac064f9d1b15ff9ec52e6033a86cdae649e5f086e32a3c0db", size = 549667, upload-time = "2026-10-11T01:01:54.65Z" }, + { url = "https://files.pythonhosted.org/packages/ce/b5/05b8ac39a76ff4bca89f42a4c2471560c71f894ff6e4bc16158c951874e3/aiohttp-3.14.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c32e26310cc10e547f53cd13d39a369034f69dcb7d749d5cb0e5f67bc196b6ba", size = 548922, upload-time = "2026-10-11T01:01:56.547Z" }, + { url = "https://files.pythonhosted.org/packages/19/b0/5aa186d56ce2334dabe29b70bd99dc8ae926ee44184c0de64a53d415a4f3/aiohttp-3.14.5-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1eb8167961ec4dfcc8cb9dd50bd0ee72519f7ef496be95203e49e27b01618382", size = 1882422, upload-time = "2026-10-11T01:01:59.258Z" }, + { url = "https://files.pythonhosted.org/packages/db/f7/7d5c91bb9620db300c8ddb05337a9014301acb626223faff3abcb8ea47d7/aiohttp-3.14.5-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c1d60eafd9c7e8e74abd03a5b00df44e7febfe6d9b89b559c0a6551eef0699d4", size = 1843621, upload-time = "2026-10-11T01:02:01.417Z" }, + { url = "https://files.pythonhosted.org/packages/30/0a/b208953b96d8f24b75f6da704f508e6c5cf3022f52b60c61933df082e89c/aiohttp-3.14.5-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:137351bf20bbed9a65e839f4a4452ac377389bdb2f2857d2acffef38f5e9f2d1", size = 1927915, upload-time = "2026-10-11T01:02:03.697Z" }, + { url = "https://files.pythonhosted.org/packages/f6/79/90ebcccb55e2d1e11a1fed581d83bb966e38fb35fb4b2577fdc980f8707a/aiohttp-3.14.5-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fba47bc2c3d7303c3d027c6cf4d07626c37b1314ac81f5820c31032e0ca1f677", size = 2036339, upload-time = "2026-10-11T01:02:06.046Z" }, + { url = "https://files.pythonhosted.org/packages/a6/66/55a8904b3a129fafdf94f9cc0a2e4ca09a3c914650be52355db7ad0bbdb6/aiohttp-3.14.5-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:94684b879ac1d71e4238850c99b62dc1b28d9086b156a2555f082010b85a865c", size = 1888041, upload-time = "2026-10-11T01:02:08.384Z" }, + { url = "https://files.pythonhosted.org/packages/0b/b8/96b25da7329a52e42c812b1e8b076386039ec4fc312afa043d173d8147fc/aiohttp-3.14.5-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:56572c42e3ecd636de8d2c3dd54cf5fc939cb5c32eb56297f176a0d366fac622", size = 1703849, upload-time = "2026-10-11T01:02:10.903Z" }, + { url = "https://files.pythonhosted.org/packages/1b/43/fbf976e3ae4c038d6f5c84945ab2150298c2d71201674d4e53d158063e75/aiohttp-3.14.5-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a95529a92a446db351675f4aab518feaf5e99842f63f5dd17160c2b74f382db3", size = 1849703, upload-time = "2026-10-11T01:02:13.15Z" }, + { url = "https://files.pythonhosted.org/packages/8b/7d/218e912f4c1d89bde7ac551409be57ad2f6121638e56942d395a7cb1fa58/aiohttp-3.14.5-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:3edbece0379b8b4aaa67619b8aa2399bb66fce372cd5911098a434ea77220aa0", size = 1852695, upload-time = "2026-10-11T01:02:15.295Z" }, + { url = "https://files.pythonhosted.org/packages/5a/42/252a1b9287e3b6e393a1c3bd1776f36af30f5f25f071bbf2b0cb7eba9116/aiohttp-3.14.5-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:56d9828f204331a5ca8850fcfe2bcce95a149f1f223f60cc7216e5524978e480", size = 1893493, upload-time = "2026-10-11T01:02:17.93Z" }, + { url = "https://files.pythonhosted.org/packages/78/97/71cae83d5100556fad1521684f7cd1e3e578432644f850245ed3bd969310/aiohttp-3.14.5-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:20064a177a070d789ee64a50b01a9161d3468e989baacfc6c714aa685c4b332f", size = 1693468, upload-time = "2026-10-11T01:02:20.666Z" }, + { url = "https://files.pythonhosted.org/packages/1f/69/73d88e97a8b5f0ca7a946d0011c0de99fb188b1687c7948ecd0553dc5bf0/aiohttp-3.14.5-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:81c2b3dfd56c62bee6108e4852d5970b4cf9086390b6983f52b666e878c1f115", size = 1915398, upload-time = "2026-10-11T01:02:23.011Z" }, + { url = "https://files.pythonhosted.org/packages/05/f0/881644bcb15d4b258daea9b720a0af9dc4330496cc8d6ade9090cdd0cffc/aiohttp-3.14.5-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:09ec102b4b8c9a920275733bbc11fdbb615efe6f9231a06007c0218d336fb77a", size = 1871470, upload-time = "2026-10-11T01:02:25.278Z" }, + { url = "https://files.pythonhosted.org/packages/7f/de/19d9ebbcce5aedaa3242d8a99ff8816a60bdb7629fb0084bf4a45bfd1f62/aiohttp-3.14.5-cp314-cp314-win32.whl", hash = "sha256:9c428eb2bd8817588d16a0ab898aa4eb5d141f896aa2b394cc79a4cf61d9a8e2", size = 485517, upload-time = "2026-10-11T01:02:27.579Z" }, + { url = "https://files.pythonhosted.org/packages/a9/74/8cdaf0e58c2588371670d5a9a8215bbb971d36940b6e5d051967dd05c07d/aiohttp-3.14.5-cp314-cp314-win_amd64.whl", hash = "sha256:6f967dde489ca6a8c02d093ab245d2cbf50ccb5c36adf0188b17b0ca39d24b67", size = 511755, upload-time = "2026-10-11T01:02:29.685Z" }, + { url = "https://files.pythonhosted.org/packages/1a/6b/e0100e25502430a531c7cf1482a378d0b65bf728ab60c01ee270e56bc469/aiohttp-3.14.5-cp314-cp314-win_arm64.whl", hash = "sha256:1d2d981b53dd09a319e3570ef8cc3bbc3ef86f5a7abef0f6b2bff3867db3a9e7", size = 495151, upload-time = "2026-10-11T01:02:32.163Z" }, + { url = "https://files.pythonhosted.org/packages/9d/c2/ca2ead7b655688c53c03aeeb6e96e6851c9ff08d6be7b13802f53a6ae8fd/aiohttp-3.14.5-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:9ce66feae6ac65327379460380549bf1b8df8e17c4e25df2a2bcf168272e3bed", size = 855396, upload-time = "2026-10-11T01:02:34.443Z" }, + { url = "https://files.pythonhosted.org/packages/a7/70/22206fea409255a240c926ce11de48de354ae2bb90ca44f709c05497585d/aiohttp-3.14.5-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:27c2322e03f66101acb09869ce1cf1efc04994ee95e1735b69827bf8c8b9d781", size = 567734, upload-time = "2026-10-11T01:02:36.644Z" }, + { url = "https://files.pythonhosted.org/packages/ce/e5/79a36c118308b56f8667d67e05d2fb6dc638ab45985704cdb199631bedaa/aiohttp-3.14.5-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ff75a7537413a86e7cafe98e0e1d6e3dc4b15c6349896e7d5c6b881bfdb6d550", size = 567379, upload-time = "2026-10-11T01:02:38.757Z" }, + { url = "https://files.pythonhosted.org/packages/63/a3/2ebec7dece3b1f02c30d2e484647f6f7b13952b1bb40a4cb285b849e8432/aiohttp-3.14.5-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9c061aa954daaf57d2a4b8374f9fca621ef0e1b603584431c220c22458c59b6d", size = 2050686, upload-time = "2026-10-11T01:02:41.169Z" }, + { url = "https://files.pythonhosted.org/packages/3a/d2/7e4d093db2f4450482652e7ef19a9e19919028f5135aa52bc4078c3beb80/aiohttp-3.14.5-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:1612fa5857b37bf32e5c1eaeefb96e3b01e9c70679eec81f0934e8a600080863", size = 1915077, upload-time = "2026-10-11T01:02:43.563Z" }, + { url = "https://files.pythonhosted.org/packages/a6/88/bd40d09958442a0a1df67da81de496361d6e2afc04f9db2950d835da750b/aiohttp-3.14.5-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:adbeee7d6fd4cf5fe0aece2fb3edc4243615d3180430ba8149d01a90670cac99", size = 2027204, upload-time = "2026-10-11T01:02:46.185Z" }, + { url = "https://files.pythonhosted.org/packages/63/eb/3a601c1f8d3103c1a60ea981f20924855da9f575d2007fc38f8898b792fb/aiohttp-3.14.5-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b2966998927d7bed9db12c0a4647b0c7b179755878fc9c357fe1ffd3e3b0c1a5", size = 2149520, upload-time = "2026-10-11T01:02:48.812Z" }, + { url = "https://files.pythonhosted.org/packages/22/d0/4e41bfe1b1ce1cb6f6d2e59fa7a88ef5cf92c402b2d07e9018778ba9edbc/aiohttp-3.14.5-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8e317e0fb6b16212c881d2205a7d87414c29acd69320b3aa6dce9d9c7b86fe4f", size = 2002039, upload-time = "2026-10-11T01:02:51.293Z" }, + { url = "https://files.pythonhosted.org/packages/6f/5e/72067019545c502b881b031153c437752ecef48d7d213bc0218ccebb4bfb/aiohttp-3.14.5-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:50343c1757b4b6f6708eeaf24534b32f19dfb99fb1b762c00420867a62fc81e0", size = 1786164, upload-time = "2026-10-11T01:02:53.533Z" }, + { url = "https://files.pythonhosted.org/packages/d9/fe/7741efd6119bfb7a00827fe6f7b84b4409de58d888ae21adc5a9a6824992/aiohttp-3.14.5-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:083673c7a94c3ea035caaa5ca04288bdb44887abfe1f5ba23294e6a4b03efd2d", size = 1970689, upload-time = "2026-10-11T01:02:55.888Z" }, + { url = "https://files.pythonhosted.org/packages/43/e7/342a13bf67f34d269bf2f7e870ecd72b99c832cc6f0a271a9210c0ebfb84/aiohttp-3.14.5-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:2528cb4c6b92008c76ac9ac6298624069bb2db91ff4929905512d1d84485f658", size = 1951967, upload-time = "2026-10-11T01:02:58.467Z" }, + { url = "https://files.pythonhosted.org/packages/e7/d4/fdb3b27340617e5e64df18a70fa89097778652ea5c7c7e2f79def76999c3/aiohttp-3.14.5-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:b1b8ece1e71132d2afba4dbc0c3d62c766e25165990b25db1196c04969eb3d84", size = 1968511, upload-time = "2026-10-11T01:03:00.883Z" }, + { url = "https://files.pythonhosted.org/packages/83/b2/e8f88298de78d1a951f36f9f966d38ec6ed1d4721303ba02064a545e8aa6/aiohttp-3.14.5-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:fce9523df31cea6284f3e2c479876750d7687cf671d7b25d32b19effc0e86441", size = 1778497, upload-time = "2026-10-11T01:03:03.206Z" }, + { url = "https://files.pythonhosted.org/packages/22/68/9ccdb93d664345c546be7f34480b921774d8c0d98f47e70d7e03b115d475/aiohttp-3.14.5-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:09e0eb18c7e0c8777e2f9149de63799195b9b3ca1b5c81ba6f32f2c6b8628210", size = 1999676, upload-time = "2026-10-11T01:03:05.829Z" }, + { url = "https://files.pythonhosted.org/packages/57/4a/a33cfa6dcb00e94194ae4fe710432ca4ed111e016356b4ba4d2f4c3a124c/aiohttp-3.14.5-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:6774814fd5c338e72ee0da5cbb9432816df450e69c019f72b5d29bdec2a1792d", size = 1963437, upload-time = "2026-10-11T01:03:08.196Z" }, + { url = "https://files.pythonhosted.org/packages/f4/20/eacbecfea3b5c3dcbfc9b023e3a5460f43e5dda16d06a2877ebe7184c3f3/aiohttp-3.14.5-cp314-cp314t-win32.whl", hash = "sha256:33f706574e32c6e694f352a856e05caf18f7f2c871b3e87b41c55ea452b409ab", size = 503973, upload-time = "2026-10-11T01:03:10.481Z" }, + { url = "https://files.pythonhosted.org/packages/ff/78/18eec294f6c8c5dc845dcf6d730a0147d8d0f17e86138a7bdb85e43a30fa/aiohttp-3.14.5-cp314-cp314t-win_amd64.whl", hash = "sha256:5ba14a839fbe87cf7c12a6b5661c05f324a296eb8363141edb3944ba63d4c9d3", size = 531190, upload-time = "2026-10-11T01:03:12.716Z" }, + { url = "https://files.pythonhosted.org/packages/9d/39/e53f8169acc85271ebd12b5b32ad7f1541b35639ccbe0f49034c64785d10/aiohttp-3.14.5-cp314-cp314t-win_arm64.whl", hash = "sha256:1061b364556e8172e8d46b0b183adeeb73e8c42d30ebc745591e1bd89acad52e", size = 510360, upload-time = "2026-10-11T01:03:15.08Z" }, + { url = "https://files.pythonhosted.org/packages/68/30/173960c42b05a6c59f7558e4b12a4b0d9ba376cf6aa9bde7f9e08a30ca8d/aiohttp-3.14.5-py3-none-any.whl", hash = "sha256:efc21a454892828368b11c2c780de0ff8bc991f73f6b99c6b66e56205470929b", size = 279517, upload-time = "2026-10-11T01:05:08.523Z" }, ] [[package]] @@ -10792,142 +10793,118 @@ wheels = [ [[package]] name = "yarl" -version = "1.23.0" +version = "1.25.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "idna" }, { name = "multidict" }, { name = "propcache" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/23/6e/beb1beec874a72f23815c1434518bfc4ed2175065173fb138c3705f658d4/yarl-1.23.0.tar.gz", hash = "sha256:53b1ea6ca88ebd4420379c330aea57e258408dd0df9af0992e5de2078dc9f5d5", size = 194676, upload-time = "2026-03-01T22:07:53.373Z" } +sdist = { url = "https://files.pythonhosted.org/packages/75/16/e8be8e2fb175bbf41a0680381a319f1199fae256588241a2ac8677eafb49/yarl-1.25.1.tar.gz", hash = "sha256:03dd38de09bc213e9a8b29761eec33ee1d5318dac0e49d8af36e4d27830e23a7", size = 246245, upload-time = "2026-09-15T19:35:02.264Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/8b/0d/9cc638702f6fc3c7a3685bcc8cf2a9ed7d6206e932a49f5242658047ef51/yarl-1.23.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:cff6d44cb13d39db2663a22b22305d10855efa0fa8015ddeacc40bc59b9d8107", size = 123764, upload-time = "2026-03-01T22:04:09.7Z" }, - { url = "https://files.pythonhosted.org/packages/7a/35/5a553687c5793df5429cd1db45909d4f3af7eee90014888c208d086a44f0/yarl-1.23.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e4c53f8347cd4200f0d70a48ad059cabaf24f5adc6ba08622a23423bc7efa10d", size = 86282, upload-time = "2026-03-01T22:04:11.892Z" }, - { url = "https://files.pythonhosted.org/packages/68/2e/c5a2234238f8ce37a8312b52801ee74117f576b1539eec8404a480434acc/yarl-1.23.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2a6940a074fb3c48356ed0158a3ca5699c955ee4185b4d7d619be3c327143e05", size = 86053, upload-time = "2026-03-01T22:04:13.292Z" }, - { url = "https://files.pythonhosted.org/packages/74/3f/bbd8ff36fb038622797ffbaf7db314918bb4d76f1cc8a4f9ca7a55fe5195/yarl-1.23.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ed5f69ce7be7902e5c70ea19eb72d20abf7d725ab5d49777d696e32d4fc1811d", size = 99395, upload-time = "2026-03-01T22:04:15.133Z" }, - { url = "https://files.pythonhosted.org/packages/77/04/9516bc4e269d2a3ec9c6779fcdeac51ce5b3a9b0156f06ac7152e5bba864/yarl-1.23.0-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:389871e65468400d6283c0308e791a640b5ab5c83bcee02a2f51295f95e09748", size = 92143, upload-time = "2026-03-01T22:04:16.829Z" }, - { url = "https://files.pythonhosted.org/packages/c7/63/88802d1f6b1cb1fc67d67a58cd0cf8a1790de4ce7946e434240f1d60ab4a/yarl-1.23.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:dda608c88cf709b1d406bdfcd84d8d63cff7c9e577a403c6108ce8ce9dcc8764", size = 107643, upload-time = "2026-03-01T22:04:18.519Z" }, - { url = "https://files.pythonhosted.org/packages/8e/db/4f9b838f4d8bdd6f0f385aed8bbf21c71ed11a0b9983305c302cbd557815/yarl-1.23.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8c4fe09e0780c6c3bf2b7d4af02ee2394439d11a523bbcf095cf4747c2932007", size = 108700, upload-time = "2026-03-01T22:04:20.373Z" }, - { url = "https://files.pythonhosted.org/packages/50/12/95a1d33f04a79c402664070d43b8b9f72dc18914e135b345b611b0b1f8cc/yarl-1.23.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:31c9921eb8bd12633b41ad27686bbb0b1a2a9b8452bfdf221e34f311e9942ed4", size = 102769, upload-time = "2026-03-01T22:04:23.055Z" }, - { url = "https://files.pythonhosted.org/packages/86/65/91a0285f51321369fd1a8308aa19207520c5f0587772cfc2e03fc2467e90/yarl-1.23.0-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5f10fd85e4b75967468af655228fbfd212bdf66db1c0d135065ce288982eda26", size = 101114, upload-time = "2026-03-01T22:04:25.031Z" }, - { url = "https://files.pythonhosted.org/packages/58/80/c7c8244fc3e5bc483dc71a09560f43b619fab29301a0f0a8f936e42865c7/yarl-1.23.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:dbf507e9ef5688bada447a24d68b4b58dd389ba93b7afc065a2ba892bea54769", size = 98883, upload-time = "2026-03-01T22:04:27.281Z" }, - { url = "https://files.pythonhosted.org/packages/86/e7/71ca9cc9ca79c0b7d491216177d1aed559d632947b8ffb0ee60f7d8b23e3/yarl-1.23.0-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:85e9beda1f591bc73e77ea1c51965c68e98dafd0fec72cdd745f77d727466716", size = 94172, upload-time = "2026-03-01T22:04:28.554Z" }, - { url = "https://files.pythonhosted.org/packages/6a/3f/6c6c8a0fe29c26fb2db2e8d32195bb84ec1bfb8f1d32e7f73b787fcf349b/yarl-1.23.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:0e1fdaa14ef51366d7757b45bde294e95f6c8c049194e793eedb8387c86d5993", size = 107010, upload-time = "2026-03-01T22:04:30.385Z" }, - { url = "https://files.pythonhosted.org/packages/56/38/12730c05e5ad40a76374d440ed8b0899729a96c250516d91c620a6e38fc2/yarl-1.23.0-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:75e3026ab649bf48f9a10c0134512638725b521340293f202a69b567518d94e0", size = 100285, upload-time = "2026-03-01T22:04:31.752Z" }, - { url = "https://files.pythonhosted.org/packages/34/92/6a7be9239f2347234e027284e7a5f74b1140cc86575e7b469d13fba1ebfe/yarl-1.23.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:80e6d33a3d42a7549b409f199857b4fb54e2103fc44fb87605b6663b7a7ff750", size = 108230, upload-time = "2026-03-01T22:04:33.844Z" }, - { url = "https://files.pythonhosted.org/packages/5e/81/4aebccfa9376bd98b9d8bfad20621a57d3e8cfc5b8631c1fa5f62cdd03f4/yarl-1.23.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:5ec2f42d41ccbd5df0270d7df31618a8ee267bfa50997f5d720ddba86c4a83a6", size = 103008, upload-time = "2026-03-01T22:04:35.856Z" }, - { url = "https://files.pythonhosted.org/packages/38/0f/0b4e3edcec794a86b853b0c6396c0a888d72dfce19b2d88c02ac289fb6c1/yarl-1.23.0-cp310-cp310-win32.whl", hash = "sha256:debe9c4f41c32990771be5c22b56f810659f9ddf3d63f67abfdcaa2c6c9c5c1d", size = 83073, upload-time = "2026-03-01T22:04:38.268Z" }, - { url = "https://files.pythonhosted.org/packages/a0/71/ad95c33da18897e4c636528bbc24a1dd23fe16797de8bc4ec667b8db0ba4/yarl-1.23.0-cp310-cp310-win_amd64.whl", hash = "sha256:ab5f043cb8a2d71c981c09c510da013bc79fd661f5c60139f00dd3c3cc4f2ffb", size = 87328, upload-time = "2026-03-01T22:04:39.558Z" }, - { url = "https://files.pythonhosted.org/packages/e2/14/dfa369523c79bccf9c9c746b0a63eb31f65db9418ac01275f7950962e504/yarl-1.23.0-cp310-cp310-win_arm64.whl", hash = "sha256:263cd4f47159c09b8b685890af949195b51d1aa82ba451c5847ca9bc6413c220", size = 82463, upload-time = "2026-03-01T22:04:41.454Z" }, - { url = "https://files.pythonhosted.org/packages/a2/aa/60da938b8f0997ba3a911263c40d82b6f645a67902a490b46f3355e10fae/yarl-1.23.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:b35d13d549077713e4414f927cdc388d62e543987c572baee613bf82f11a4b99", size = 123641, upload-time = "2026-03-01T22:04:42.841Z" }, - { url = "https://files.pythonhosted.org/packages/24/84/e237607faf4e099dbb8a4f511cfd5efcb5f75918baad200ff7380635631b/yarl-1.23.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:cbb0fef01f0c6b38cb0f39b1f78fc90b807e0e3c86a7ff3ce74ad77ce5c7880c", size = 86248, upload-time = "2026-03-01T22:04:44.757Z" }, - { url = "https://files.pythonhosted.org/packages/b2/0d/71ceabc14c146ba8ee3804ca7b3d42b1664c8440439de5214d366fec7d3a/yarl-1.23.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:dc52310451fc7c629e13c4e061cbe2dd01684d91f2f8ee2821b083c58bd72432", size = 85988, upload-time = "2026-03-01T22:04:46.365Z" }, - { url = "https://files.pythonhosted.org/packages/8c/6c/4a90d59c572e46b270ca132aca66954f1175abd691f74c1ef4c6711828e2/yarl-1.23.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b2c6b50c7b0464165472b56b42d4c76a7b864597007d9c085e8b63e185cf4a7a", size = 100566, upload-time = "2026-03-01T22:04:47.639Z" }, - { url = "https://files.pythonhosted.org/packages/49/fb/c438fb5108047e629f6282a371e6e91cf3f97ee087c4fb748a1f32ceef55/yarl-1.23.0-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:aafe5dcfda86c8af00386d7781d4c2181b5011b7be3f2add5e99899ea925df05", size = 92079, upload-time = "2026-03-01T22:04:48.925Z" }, - { url = "https://files.pythonhosted.org/packages/d9/13/d269aa1aed3e4f50a5a103f96327210cc5fa5dd2d50882778f13c7a14606/yarl-1.23.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9ee33b875f0b390564c1fb7bc528abf18c8ee6073b201c6ae8524aca778e2d83", size = 108741, upload-time = "2026-03-01T22:04:50.838Z" }, - { url = "https://files.pythonhosted.org/packages/85/fb/115b16f22c37ea4437d323e472945bea97301c8ec6089868fa560abab590/yarl-1.23.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4c41e021bc6d7affb3364dc1e1e5fa9582b470f283748784bd6ea0558f87f42c", size = 108099, upload-time = "2026-03-01T22:04:52.499Z" }, - { url = "https://files.pythonhosted.org/packages/9a/64/c53487d9f4968045b8afa51aed7ca44f58b2589e772f32745f3744476c82/yarl-1.23.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:99c8a9ed30f4164bc4c14b37a90208836cbf50d4ce2a57c71d0f52c7fb4f7598", size = 102678, upload-time = "2026-03-01T22:04:55.176Z" }, - { url = "https://files.pythonhosted.org/packages/85/59/cd98e556fbb2bf8fab29c1a722f67ad45c5f3447cac798ab85620d1e70af/yarl-1.23.0-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f2af5c81a1f124609d5f33507082fc3f739959d4719b56877ab1ee7e7b3d602b", size = 100803, upload-time = "2026-03-01T22:04:56.588Z" }, - { url = "https://files.pythonhosted.org/packages/9e/c0/b39770b56d4a9f0bb5f77e2f1763cd2d75cc2f6c0131e3b4c360348fcd65/yarl-1.23.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6b41389c19b07c760c7e427a3462e8ab83c4bb087d127f0e854c706ce1b9215c", size = 100163, upload-time = "2026-03-01T22:04:58.492Z" }, - { url = "https://files.pythonhosted.org/packages/e7/64/6980f99ab00e1f0ff67cb84766c93d595b067eed07439cfccfc8fb28c1a6/yarl-1.23.0-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:1dc702e42d0684f42d6519c8d581e49c96cefaaab16691f03566d30658ee8788", size = 93859, upload-time = "2026-03-01T22:05:00.268Z" }, - { url = "https://files.pythonhosted.org/packages/38/69/912e6c5e146793e5d4b5fe39ff5b00f4d22463dfd5a162bec565ac757673/yarl-1.23.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:0e40111274f340d32ebcc0a5668d54d2b552a6cca84c9475859d364b380e3222", size = 108202, upload-time = "2026-03-01T22:05:02.273Z" }, - { url = "https://files.pythonhosted.org/packages/59/97/35ca6767524687ad64e5f5c31ad54bc76d585585a9fcb40f649e7e82ffed/yarl-1.23.0-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:4764a6a7588561a9aef92f65bda2c4fb58fe7c675c0883862e6df97559de0bfb", size = 99866, upload-time = "2026-03-01T22:05:03.597Z" }, - { url = "https://files.pythonhosted.org/packages/d3/1c/1a3387ee6d73589f6f2a220ae06f2984f6c20b40c734989b0a44f5987308/yarl-1.23.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:03214408cfa590df47728b84c679ae4ef00be2428e11630277be0727eba2d7cc", size = 107852, upload-time = "2026-03-01T22:05:04.986Z" }, - { url = "https://files.pythonhosted.org/packages/a4/b8/35c0750fcd5a3f781058bfd954515dd4b1eab45e218cbb85cf11132215f1/yarl-1.23.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:170e26584b060879e29fac213e4228ef063f39128723807a312e5c7fec28eff2", size = 102919, upload-time = "2026-03-01T22:05:06.397Z" }, - { url = "https://files.pythonhosted.org/packages/e5/1c/9a1979aec4a81896d597bcb2177827f2dbee3f5b7cc48b2d0dadb644b41d/yarl-1.23.0-cp311-cp311-win32.whl", hash = "sha256:51430653db848d258336cfa0244427b17d12db63d42603a55f0d4546f50f25b5", size = 82602, upload-time = "2026-03-01T22:05:08.444Z" }, - { url = "https://files.pythonhosted.org/packages/93/22/b85eca6fa2ad9491af48c973e4c8cf6b103a73dbb271fe3346949449fca0/yarl-1.23.0-cp311-cp311-win_amd64.whl", hash = "sha256:bf49a3ae946a87083ef3a34c8f677ae4243f5b824bfc4c69672e72b3d6719d46", size = 87461, upload-time = "2026-03-01T22:05:10.145Z" }, - { url = "https://files.pythonhosted.org/packages/93/95/07e3553fe6f113e6864a20bdc53a78113cda3b9ced8784ee52a52c9f80d8/yarl-1.23.0-cp311-cp311-win_arm64.whl", hash = "sha256:b39cb32a6582750b6cc77bfb3c49c0f8760dc18dc96ec9fb55fbb0f04e08b928", size = 82336, upload-time = "2026-03-01T22:05:11.554Z" }, - { url = "https://files.pythonhosted.org/packages/88/8a/94615bc31022f711add374097ad4144d569e95ff3c38d39215d07ac153a0/yarl-1.23.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:1932b6b8bba8d0160a9d1078aae5838a66039e8832d41d2992daa9a3a08f7860", size = 124737, upload-time = "2026-03-01T22:05:12.897Z" }, - { url = "https://files.pythonhosted.org/packages/e3/6f/c6554045d59d64052698add01226bc867b52fe4a12373415d7991fdca95d/yarl-1.23.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:411225bae281f114067578891bc75534cfb3d92a3b4dfef7a6ca78ba354e6069", size = 87029, upload-time = "2026-03-01T22:05:14.376Z" }, - { url = "https://files.pythonhosted.org/packages/19/2a/725ecc166d53438bc88f76822ed4b1e3b10756e790bafd7b523fe97c322d/yarl-1.23.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:13a563739ae600a631c36ce096615fe307f131344588b0bc0daec108cdb47b25", size = 86310, upload-time = "2026-03-01T22:05:15.71Z" }, - { url = "https://files.pythonhosted.org/packages/99/30/58260ed98e6ff7f90ba84442c1ddd758c9170d70327394a6227b310cd60f/yarl-1.23.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9cbf44c5cb4a7633d078788e1b56387e3d3cf2b8139a3be38040b22d6c3221c8", size = 97587, upload-time = "2026-03-01T22:05:17.384Z" }, - { url = "https://files.pythonhosted.org/packages/76/0a/8b08aac08b50682e65759f7f8dde98ae8168f72487e7357a5d684c581ef9/yarl-1.23.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:53ad387048f6f09a8969631e4de3f1bf70c50e93545d64af4f751b2498755072", size = 92528, upload-time = "2026-03-01T22:05:18.804Z" }, - { url = "https://files.pythonhosted.org/packages/52/07/0b7179101fe5f8385ec6c6bb5d0cb9f76bd9fb4a769591ab6fb5cdbfc69a/yarl-1.23.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4a59ba56f340334766f3a4442e0efd0af895fae9e2b204741ef885c446b3a1a8", size = 105339, upload-time = "2026-03-01T22:05:20.235Z" }, - { url = "https://files.pythonhosted.org/packages/d3/8a/36d82869ab5ec829ca8574dfcb92b51286fcfb1e9c7a73659616362dc880/yarl-1.23.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:803a3c3ce4acc62eaf01eaca1208dcf0783025ef27572c3336502b9c232005e7", size = 105061, upload-time = "2026-03-01T22:05:22.268Z" }, - { url = "https://files.pythonhosted.org/packages/66/3e/868e5c3364b6cee19ff3e1a122194fa4ce51def02c61023970442162859e/yarl-1.23.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a3d2bff8f37f8d0f96c7ec554d16945050d54462d6e95414babaa18bfafc7f51", size = 100132, upload-time = "2026-03-01T22:05:23.638Z" }, - { url = "https://files.pythonhosted.org/packages/cf/26/9c89acf82f08a52cb52d6d39454f8d18af15f9d386a23795389d1d423823/yarl-1.23.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c75eb09e8d55bceb4367e83496ff8ef2bc7ea6960efb38e978e8073ea59ecb67", size = 99289, upload-time = "2026-03-01T22:05:25.749Z" }, - { url = "https://files.pythonhosted.org/packages/6f/54/5b0db00d2cb056922356104468019c0a132e89c8d3ab67d8ede9f4483d2a/yarl-1.23.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:877b0738624280e34c55680d6054a307aa94f7d52fa0e3034a9cc6e790871da7", size = 96950, upload-time = "2026-03-01T22:05:27.318Z" }, - { url = "https://files.pythonhosted.org/packages/f6/40/10fa93811fd439341fad7e0718a86aca0de9548023bbb403668d6555acab/yarl-1.23.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b5405bb8f0e783a988172993cfc627e4d9d00432d6bbac65a923041edacf997d", size = 93960, upload-time = "2026-03-01T22:05:28.738Z" }, - { url = "https://files.pythonhosted.org/packages/bc/d2/8ae2e6cd77d0805f4526e30ec43b6f9a3dfc542d401ac4990d178e4bf0cf/yarl-1.23.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1c3a3598a832590c5a3ce56ab5576361b5688c12cb1d39429cf5dba30b510760", size = 104703, upload-time = "2026-03-01T22:05:30.438Z" }, - { url = "https://files.pythonhosted.org/packages/2f/0c/b3ceacf82c3fe21183ce35fa2acf5320af003d52bc1fcf5915077681142e/yarl-1.23.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:8419ebd326430d1cbb7efb5292330a2cf39114e82df5cc3d83c9a0d5ebeaf2f2", size = 98325, upload-time = "2026-03-01T22:05:31.835Z" }, - { url = "https://files.pythonhosted.org/packages/9d/e0/12900edd28bdab91a69bd2554b85ad7b151f64e8b521fe16f9ad2f56477a/yarl-1.23.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:be61f6fff406ca40e3b1d84716fde398fc08bc63dd96d15f3a14230a0973ed86", size = 105067, upload-time = "2026-03-01T22:05:33.358Z" }, - { url = "https://files.pythonhosted.org/packages/15/61/74bb1182cf79c9bbe4eb6b1f14a57a22d7a0be5e9cedf8e2d5c2086474c3/yarl-1.23.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3ceb13c5c858d01321b5d9bb65e4cf37a92169ea470b70fec6f236b2c9dd7e34", size = 100285, upload-time = "2026-03-01T22:05:35.4Z" }, - { url = "https://files.pythonhosted.org/packages/69/7f/cd5ef733f2550de6241bd8bd8c3febc78158b9d75f197d9c7baa113436af/yarl-1.23.0-cp312-cp312-win32.whl", hash = "sha256:fffc45637bcd6538de8b85f51e3df3223e4ad89bccbfca0481c08c7fc8b7ed7d", size = 82359, upload-time = "2026-03-01T22:05:36.811Z" }, - { url = "https://files.pythonhosted.org/packages/f5/be/25216a49daeeb7af2bec0db22d5e7df08ed1d7c9f65d78b14f3b74fd72fc/yarl-1.23.0-cp312-cp312-win_amd64.whl", hash = "sha256:f69f57305656a4852f2a7203efc661d8c042e6cc67f7acd97d8667fb448a426e", size = 87674, upload-time = "2026-03-01T22:05:38.171Z" }, - { url = "https://files.pythonhosted.org/packages/d2/35/aeab955d6c425b227d5b7247eafb24f2653fedc32f95373a001af5dfeb9e/yarl-1.23.0-cp312-cp312-win_arm64.whl", hash = "sha256:6e87a6e8735b44816e7db0b2fbc9686932df473c826b0d9743148432e10bb9b9", size = 81879, upload-time = "2026-03-01T22:05:40.006Z" }, - { url = "https://files.pythonhosted.org/packages/9a/4b/a0a6e5d0ee8a2f3a373ddef8a4097d74ac901ac363eea1440464ccbe0898/yarl-1.23.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:16c6994ac35c3e74fb0ae93323bf8b9c2a9088d55946109489667c510a7d010e", size = 123796, upload-time = "2026-03-01T22:05:41.412Z" }, - { url = "https://files.pythonhosted.org/packages/67/b6/8925d68af039b835ae876db5838e82e76ec87b9782ecc97e192b809c4831/yarl-1.23.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4a42e651629dafb64fd5b0286a3580613702b5809ad3f24934ea87595804f2c5", size = 86547, upload-time = "2026-03-01T22:05:42.841Z" }, - { url = "https://files.pythonhosted.org/packages/ae/50/06d511cc4b8e0360d3c94af051a768e84b755c5eb031b12adaaab6dec6e5/yarl-1.23.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7c6b9461a2a8b47c65eef63bb1c76a4f1c119618ffa99ea79bc5bb1e46c5821b", size = 85854, upload-time = "2026-03-01T22:05:44.85Z" }, - { url = "https://files.pythonhosted.org/packages/c4/f4/4e30b250927ffdab4db70da08b9b8d2194d7c7b400167b8fbeca1e4701ca/yarl-1.23.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2569b67d616eab450d262ca7cb9f9e19d2f718c70a8b88712859359d0ab17035", size = 98351, upload-time = "2026-03-01T22:05:46.836Z" }, - { url = "https://files.pythonhosted.org/packages/86/fc/4118c5671ea948208bdb1492d8b76bdf1453d3e73df051f939f563e7dcc5/yarl-1.23.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e9d9a4d06d3481eab79803beb4d9bd6f6a8e781ec078ac70d7ef2dcc29d1bea5", size = 92711, upload-time = "2026-03-01T22:05:48.316Z" }, - { url = "https://files.pythonhosted.org/packages/56/11/1ed91d42bd9e73c13dc9e7eb0dd92298d75e7ac4dd7f046ad0c472e231cd/yarl-1.23.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f514f6474e04179d3d33175ed3f3e31434d3130d42ec153540d5b157deefd735", size = 106014, upload-time = "2026-03-01T22:05:50.028Z" }, - { url = "https://files.pythonhosted.org/packages/ce/c9/74e44e056a23fbc33aca71779ef450ca648a5bc472bdad7a82339918f818/yarl-1.23.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fda207c815b253e34f7e1909840fd14299567b1c0eb4908f8c2ce01a41265401", size = 105557, upload-time = "2026-03-01T22:05:51.416Z" }, - { url = "https://files.pythonhosted.org/packages/66/fe/b1e10b08d287f518994f1e2ff9b6d26f0adeecd8dd7d533b01bab29a3eda/yarl-1.23.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:34b6cf500e61c90f305094911f9acc9c86da1a05a7a3f5be9f68817043f486e4", size = 101559, upload-time = "2026-03-01T22:05:52.872Z" }, - { url = "https://files.pythonhosted.org/packages/72/59/c5b8d94b14e3d3c2a9c20cb100119fd534ab5a14b93673ab4cc4a4141ea5/yarl-1.23.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d7504f2b476d21653e4d143f44a175f7f751cd41233525312696c76aa3dbb23f", size = 100502, upload-time = "2026-03-01T22:05:54.954Z" }, - { url = "https://files.pythonhosted.org/packages/77/4f/96976cb54cbfc5c9fd73ed4c51804f92f209481d1fb190981c0f8a07a1d7/yarl-1.23.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:578110dd426f0d209d1509244e6d4a3f1a3e9077655d98c5f22583d63252a08a", size = 98027, upload-time = "2026-03-01T22:05:56.409Z" }, - { url = "https://files.pythonhosted.org/packages/63/6e/904c4f476471afdbad6b7e5b70362fb5810e35cd7466529a97322b6f5556/yarl-1.23.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:609d3614d78d74ebe35f54953c5bbd2ac647a7ddb9c30a5d877580f5e86b22f2", size = 95369, upload-time = "2026-03-01T22:05:58.141Z" }, - { url = "https://files.pythonhosted.org/packages/9d/40/acfcdb3b5f9d68ef499e39e04d25e141fe90661f9d54114556cf83be8353/yarl-1.23.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4966242ec68afc74c122f8459abd597afd7d8a60dc93d695c1334c5fd25f762f", size = 105565, upload-time = "2026-03-01T22:06:00.286Z" }, - { url = "https://files.pythonhosted.org/packages/5e/c6/31e28f3a6ba2869c43d124f37ea5260cac9c9281df803c354b31f4dd1f3c/yarl-1.23.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:e0fd068364a6759bc794459f0a735ab151d11304346332489c7972bacbe9e72b", size = 99813, upload-time = "2026-03-01T22:06:01.712Z" }, - { url = "https://files.pythonhosted.org/packages/08/1f/6f65f59e72d54aa467119b63fc0b0b1762eff0232db1f4720cd89e2f4a17/yarl-1.23.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:39004f0ad156da43e86aa71f44e033de68a44e5a31fc53507b36dd253970054a", size = 105632, upload-time = "2026-03-01T22:06:03.188Z" }, - { url = "https://files.pythonhosted.org/packages/a3/c4/18b178a69935f9e7a338127d5b77d868fdc0f0e49becd286d51b3a18c61d/yarl-1.23.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e5723c01a56c5028c807c701aa66722916d2747ad737a046853f6c46f4875543", size = 101895, upload-time = "2026-03-01T22:06:04.651Z" }, - { url = "https://files.pythonhosted.org/packages/8f/54/f5b870b5505663911dba950a8e4776a0dbd51c9c54c0ae88e823e4b874a0/yarl-1.23.0-cp313-cp313-win32.whl", hash = "sha256:1b6b572edd95b4fa8df75de10b04bc81acc87c1c7d16bcdd2035b09d30acc957", size = 82356, upload-time = "2026-03-01T22:06:06.04Z" }, - { url = "https://files.pythonhosted.org/packages/7a/84/266e8da36879c6edcd37b02b547e2d9ecdfea776be49598e75696e3316e1/yarl-1.23.0-cp313-cp313-win_amd64.whl", hash = "sha256:baaf55442359053c7d62f6f8413a62adba3205119bcb6f49594894d8be47e5e3", size = 87515, upload-time = "2026-03-01T22:06:08.107Z" }, - { url = "https://files.pythonhosted.org/packages/00/fd/7e1c66efad35e1649114fa13f17485f62881ad58edeeb7f49f8c5e748bf9/yarl-1.23.0-cp313-cp313-win_arm64.whl", hash = "sha256:fb4948814a2a98e3912505f09c9e7493b1506226afb1f881825368d6fb776ee3", size = 81785, upload-time = "2026-03-01T22:06:10.181Z" }, - { url = "https://files.pythonhosted.org/packages/9c/fc/119dd07004f17ea43bb91e3ece6587759edd7519d6b086d16bfbd3319982/yarl-1.23.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:aecfed0b41aa72b7881712c65cf764e39ce2ec352324f5e0837c7048d9e6daaa", size = 130719, upload-time = "2026-03-01T22:06:11.708Z" }, - { url = "https://files.pythonhosted.org/packages/e6/0d/9f2348502fbb3af409e8f47730282cd6bc80dec6630c1e06374d882d6eb2/yarl-1.23.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:a41bcf68efd19073376eb8cf948b8d9be0af26256403e512bb18f3966f1f9120", size = 89690, upload-time = "2026-03-01T22:06:13.429Z" }, - { url = "https://files.pythonhosted.org/packages/50/93/e88f3c80971b42cfc83f50a51b9d165a1dbf154b97005f2994a79f212a07/yarl-1.23.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:cde9a2ecd91668bcb7f077c4966d8ceddb60af01b52e6e3e2680e4cf00ad1a59", size = 89851, upload-time = "2026-03-01T22:06:15.53Z" }, - { url = "https://files.pythonhosted.org/packages/1c/07/61c9dd8ba8f86473263b4036f70fb594c09e99c0d9737a799dfd8bc85651/yarl-1.23.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5023346c4ee7992febc0068e7593de5fa2bf611848c08404b35ebbb76b1b0512", size = 95874, upload-time = "2026-03-01T22:06:17.553Z" }, - { url = "https://files.pythonhosted.org/packages/9e/e9/f9ff8ceefba599eac6abddcfb0b3bee9b9e636e96dbf54342a8577252379/yarl-1.23.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:d1009abedb49ae95b136a8904a3f71b342f849ffeced2d3747bf29caeda218c4", size = 88710, upload-time = "2026-03-01T22:06:19.004Z" }, - { url = "https://files.pythonhosted.org/packages/eb/78/0231bfcc5d4c8eec220bc2f9ef82cb4566192ea867a7c5b4148f44f6cbcd/yarl-1.23.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a8d00f29b42f534cc8aa3931cfe773b13b23e561e10d2b26f27a8d309b0e82a1", size = 101033, upload-time = "2026-03-01T22:06:21.203Z" }, - { url = "https://files.pythonhosted.org/packages/cd/9b/30ea5239a61786f18fd25797151a17fbb3be176977187a48d541b5447dd4/yarl-1.23.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:95451e6ce06c3e104556d73b559f5da6c34a069b6b62946d3ad66afcd51642ea", size = 100817, upload-time = "2026-03-01T22:06:22.738Z" }, - { url = "https://files.pythonhosted.org/packages/62/e2/a4980481071791bc83bce2b7a1a1f7adcabfa366007518b4b845e92eeee3/yarl-1.23.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:531ef597132086b6cf96faa7c6c1dcd0361dd5f1694e5cc30375907b9b7d3ea9", size = 97482, upload-time = "2026-03-01T22:06:24.21Z" }, - { url = "https://files.pythonhosted.org/packages/e5/1e/304a00cf5f6100414c4b5a01fc7ff9ee724b62158a08df2f8170dfc72a2d/yarl-1.23.0-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:88f9fb0116fbfcefcab70f85cf4b74a2b6ce5d199c41345296f49d974ddb4123", size = 95949, upload-time = "2026-03-01T22:06:25.697Z" }, - { url = "https://files.pythonhosted.org/packages/68/03/093f4055ed4cae649ac53bca3d180bd37102e9e11d048588e9ab0c0108d0/yarl-1.23.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:e7b0460976dc75cb87ad9cc1f9899a4b97751e7d4e77ab840fc9b6d377b8fd24", size = 95839, upload-time = "2026-03-01T22:06:27.309Z" }, - { url = "https://files.pythonhosted.org/packages/b9/28/4c75ebb108f322aa8f917ae10a8ffa4f07cae10a8a627b64e578617df6a0/yarl-1.23.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:115136c4a426f9da976187d238e84139ff6b51a20839aa6e3720cd1026d768de", size = 90696, upload-time = "2026-03-01T22:06:29.048Z" }, - { url = "https://files.pythonhosted.org/packages/23/9c/42c2e2dd91c1a570402f51bdf066bfdb1241c2240ba001967bad778e77b7/yarl-1.23.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:ead11956716a940c1abc816b7df3fa2b84d06eaed8832ca32f5c5e058c65506b", size = 100865, upload-time = "2026-03-01T22:06:30.525Z" }, - { url = "https://files.pythonhosted.org/packages/74/05/1bcd60a8a0a914d462c305137246b6f9d167628d73568505fce3f1cb2e65/yarl-1.23.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:fe8f8f5e70e6dbdfca9882cd9deaac058729bcf323cf7a58660901e55c9c94f6", size = 96234, upload-time = "2026-03-01T22:06:32.692Z" }, - { url = "https://files.pythonhosted.org/packages/90/b2/f52381aac396d6778ce516b7bc149c79e65bfc068b5de2857ab69eeea3b7/yarl-1.23.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:a0e317df055958a0c1e79e5d2aa5a5eaa4a6d05a20d4b0c9c3f48918139c9fc6", size = 100295, upload-time = "2026-03-01T22:06:34.268Z" }, - { url = "https://files.pythonhosted.org/packages/e5/e8/638bae5bbf1113a659b2435d8895474598afe38b4a837103764f603aba56/yarl-1.23.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6f0fd84de0c957b2d280143522c4f91a73aada1923caee763e24a2b3fda9f8a5", size = 97784, upload-time = "2026-03-01T22:06:35.864Z" }, - { url = "https://files.pythonhosted.org/packages/80/25/a3892b46182c586c202629fc2159aa13975d3741d52ebd7347fd501d48d5/yarl-1.23.0-cp313-cp313t-win32.whl", hash = "sha256:93a784271881035ab4406a172edb0faecb6e7d00f4b53dc2f55919d6c9688595", size = 88313, upload-time = "2026-03-01T22:06:37.39Z" }, - { url = "https://files.pythonhosted.org/packages/43/68/8c5b36aa5178900b37387937bc2c2fe0e9505537f713495472dcf6f6fccc/yarl-1.23.0-cp313-cp313t-win_amd64.whl", hash = "sha256:dd00607bffbf30250fe108065f07453ec124dbf223420f57f5e749b04295e090", size = 94932, upload-time = "2026-03-01T22:06:39.579Z" }, - { url = "https://files.pythonhosted.org/packages/c6/cc/d79ba8292f51f81f4dc533a8ccfb9fc6992cabf0998ed3245de7589dc07c/yarl-1.23.0-cp313-cp313t-win_arm64.whl", hash = "sha256:ac09d42f48f80c9ee1635b2fcaa819496a44502737660d3c0f2ade7526d29144", size = 84786, upload-time = "2026-03-01T22:06:41.988Z" }, - { url = "https://files.pythonhosted.org/packages/90/98/b85a038d65d1b92c3903ab89444f48d3cee490a883477b716d7a24b1a78c/yarl-1.23.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:21d1b7305a71a15b4794b5ff22e8eef96ff4a6d7f9657155e5aa419444b28912", size = 124455, upload-time = "2026-03-01T22:06:43.615Z" }, - { url = "https://files.pythonhosted.org/packages/39/54/bc2b45559f86543d163b6e294417a107bb87557609007c007ad889afec18/yarl-1.23.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:85610b4f27f69984932a7abbe52703688de3724d9f72bceb1cca667deff27474", size = 86752, upload-time = "2026-03-01T22:06:45.425Z" }, - { url = "https://files.pythonhosted.org/packages/24/f9/e8242b68362bffe6fb536c8db5076861466fc780f0f1b479fc4ffbebb128/yarl-1.23.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:23f371bd662cf44a7630d4d113101eafc0cfa7518a2760d20760b26021454719", size = 86291, upload-time = "2026-03-01T22:06:46.974Z" }, - { url = "https://files.pythonhosted.org/packages/ea/d8/d1cb2378c81dd729e98c716582b1ccb08357e8488e4c24714658cc6630e8/yarl-1.23.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c4a80f77dc1acaaa61f0934176fccca7096d9b1ff08c8ba9cddf5ae034a24319", size = 99026, upload-time = "2026-03-01T22:06:48.459Z" }, - { url = "https://files.pythonhosted.org/packages/0a/ff/7196790538f31debe3341283b5b0707e7feb947620fc5e8236ef28d44f72/yarl-1.23.0-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:bd654fad46d8d9e823afbb4f87c79160b5a374ed1ff5bde24e542e6ba8f41434", size = 92355, upload-time = "2026-03-01T22:06:50.306Z" }, - { url = "https://files.pythonhosted.org/packages/c1/56/25d58c3eddde825890a5fe6aa1866228377354a3c39262235234ab5f616b/yarl-1.23.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:682bae25f0a0dd23a056739f23a134db9f52a63e2afd6bfb37ddc76292bbd723", size = 106417, upload-time = "2026-03-01T22:06:52.1Z" }, - { url = "https://files.pythonhosted.org/packages/51/8a/882c0e7bc8277eb895b31bce0138f51a1ba551fc2e1ec6753ffc1e7c1377/yarl-1.23.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a82836cab5f197a0514235aaf7ffccdc886ccdaa2324bc0aafdd4ae898103039", size = 106422, upload-time = "2026-03-01T22:06:54.424Z" }, - { url = "https://files.pythonhosted.org/packages/42/2b/fef67d616931055bf3d6764885990a3ac647d68734a2d6a9e1d13de437a2/yarl-1.23.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1c57676bdedc94cd3bc37724cf6f8cd2779f02f6aba48de45feca073e714fe52", size = 101915, upload-time = "2026-03-01T22:06:55.895Z" }, - { url = "https://files.pythonhosted.org/packages/18/6a/530e16aebce27c5937920f3431c628a29a4b6b430fab3fd1c117b26ff3f6/yarl-1.23.0-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c7f8dc16c498ff06497c015642333219871effba93e4a2e8604a06264aca5c5c", size = 100690, upload-time = "2026-03-01T22:06:58.21Z" }, - { url = "https://files.pythonhosted.org/packages/88/08/93749219179a45e27b036e03260fda05190b911de8e18225c294ac95bbc9/yarl-1.23.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:5ee586fb17ff8f90c91cf73c6108a434b02d69925f44f5f8e0d7f2f260607eae", size = 98750, upload-time = "2026-03-01T22:06:59.794Z" }, - { url = "https://files.pythonhosted.org/packages/d9/cf/ea424a004969f5d81a362110a6ac1496d79efdc6d50c2c4b2e3ea0fc2519/yarl-1.23.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:17235362f580149742739cc3828b80e24029d08cbb9c4bda0242c7b5bc610a8e", size = 94685, upload-time = "2026-03-01T22:07:01.375Z" }, - { url = "https://files.pythonhosted.org/packages/e2/b7/14341481fe568e2b0408bcf1484c652accafe06a0ade9387b5d3fd9df446/yarl-1.23.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:0793e2bd0cf14234983bbb371591e6bea9e876ddf6896cdcc93450996b0b5c85", size = 106009, upload-time = "2026-03-01T22:07:03.151Z" }, - { url = "https://files.pythonhosted.org/packages/0a/e6/5c744a9b54f4e8007ad35bce96fbc9218338e84812d36f3390cea616881a/yarl-1.23.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:3650dc2480f94f7116c364096bc84b1d602f44224ef7d5c7208425915c0475dd", size = 100033, upload-time = "2026-03-01T22:07:04.701Z" }, - { url = "https://files.pythonhosted.org/packages/0c/23/e3bfc188d0b400f025bc49d99793d02c9abe15752138dcc27e4eaf0c4a9e/yarl-1.23.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:f40e782d49630ad384db66d4d8b73ff4f1b8955dc12e26b09a3e3af064b3b9d6", size = 106483, upload-time = "2026-03-01T22:07:06.231Z" }, - { url = "https://files.pythonhosted.org/packages/72/42/f0505f949a90b3f8b7a363d6cbdf398f6e6c58946d85c6d3a3bc70595b26/yarl-1.23.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:94f8575fbdf81749008d980c17796097e645574a3b8c28ee313931068dad14fe", size = 102175, upload-time = "2026-03-01T22:07:08.4Z" }, - { url = "https://files.pythonhosted.org/packages/aa/65/b39290f1d892a9dd671d1c722014ca062a9c35d60885d57e5375db0404b5/yarl-1.23.0-cp314-cp314-win32.whl", hash = "sha256:c8aa34a5c864db1087d911a0b902d60d203ea3607d91f615acd3f3108ac32169", size = 83871, upload-time = "2026-03-01T22:07:09.968Z" }, - { url = "https://files.pythonhosted.org/packages/a9/5b/9b92f54c784c26e2a422e55a8d2607ab15b7ea3349e28359282f84f01d43/yarl-1.23.0-cp314-cp314-win_amd64.whl", hash = "sha256:63e92247f383c85ab00dd0091e8c3fa331a96e865459f5ee80353c70a4a42d70", size = 89093, upload-time = "2026-03-01T22:07:11.501Z" }, - { url = "https://files.pythonhosted.org/packages/e0/7d/8a84dc9381fd4412d5e7ff04926f9865f6372b4c2fd91e10092e65d29eb8/yarl-1.23.0-cp314-cp314-win_arm64.whl", hash = "sha256:70efd20be968c76ece7baa8dafe04c5be06abc57f754d6f36f3741f7aa7a208e", size = 83384, upload-time = "2026-03-01T22:07:13.069Z" }, - { url = "https://files.pythonhosted.org/packages/dd/8d/d2fad34b1c08aa161b74394183daa7d800141aaaee207317e82c790b418d/yarl-1.23.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:9a18d6f9359e45722c064c97464ec883eb0e0366d33eda61cb19a244bf222679", size = 131019, upload-time = "2026-03-01T22:07:14.903Z" }, - { url = "https://files.pythonhosted.org/packages/19/ff/33009a39d3ccf4b94d7d7880dfe17fb5816c5a4fe0096d9b56abceea9ac7/yarl-1.23.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:2803ed8b21ca47a43da80a6fd1ed3019d30061f7061daa35ac54f63933409412", size = 89894, upload-time = "2026-03-01T22:07:17.372Z" }, - { url = "https://files.pythonhosted.org/packages/0c/f1/dab7ac5e7306fb79c0190766a3c00b4cb8d09a1f390ded68c85a5934faf5/yarl-1.23.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:394906945aa8b19fc14a61cf69743a868bb8c465efe85eee687109cc540b98f4", size = 89979, upload-time = "2026-03-01T22:07:19.361Z" }, - { url = "https://files.pythonhosted.org/packages/aa/b1/08e95f3caee1fad6e65017b9f26c1d79877b502622d60e517de01e72f95d/yarl-1.23.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:71d006bee8397a4a89f469b8deb22469fe7508132d3c17fa6ed871e79832691c", size = 95943, upload-time = "2026-03-01T22:07:21.266Z" }, - { url = "https://files.pythonhosted.org/packages/c0/cc/6409f9018864a6aa186c61175b977131f373f1988e198e031236916e87e4/yarl-1.23.0-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:62694e275c93d54f7ccedcfef57d42761b2aad5234b6be1f3e3026cae4001cd4", size = 88786, upload-time = "2026-03-01T22:07:23.129Z" }, - { url = "https://files.pythonhosted.org/packages/76/40/cc22d1d7714b717fde2006fad2ced5efe5580606cb059ae42117542122f3/yarl-1.23.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a31de1613658308efdb21ada98cbc86a97c181aa050ba22a808120bb5be3ab94", size = 101307, upload-time = "2026-03-01T22:07:24.689Z" }, - { url = "https://files.pythonhosted.org/packages/8f/0d/476c38e85ddb4c6ec6b20b815bdd779aa386a013f3d8b85516feee55c8dc/yarl-1.23.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fb1e8b8d66c278b21d13b0a7ca22c41dd757a7c209c6b12c313e445c31dd3b28", size = 100904, upload-time = "2026-03-01T22:07:26.287Z" }, - { url = "https://files.pythonhosted.org/packages/72/32/0abe4a76d59adf2081dcb0397168553ece4616ada1c54d1c49d8936c74f8/yarl-1.23.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:50f9d8d531dfb767c565f348f33dd5139a6c43f5cbdf3f67da40d54241df93f6", size = 97728, upload-time = "2026-03-01T22:07:27.906Z" }, - { url = "https://files.pythonhosted.org/packages/b7/35/7b30f4810fba112f60f5a43237545867504e15b1c7647a785fbaf588fac2/yarl-1.23.0-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:575aa4405a656e61a540f4a80eaa5260f2a38fff7bfdc4b5f611840d76e9e277", size = 95964, upload-time = "2026-03-01T22:07:30.198Z" }, - { url = "https://files.pythonhosted.org/packages/2d/86/ed7a73ab85ef00e8bb70b0cb5421d8a2a625b81a333941a469a6f4022828/yarl-1.23.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:041b1a4cefacf65840b4e295c6985f334ba83c30607441ae3cf206a0eed1a2e4", size = 95882, upload-time = "2026-03-01T22:07:32.132Z" }, - { url = "https://files.pythonhosted.org/packages/19/90/d56967f61a29d8498efb7afb651e0b2b422a1e9b47b0ab5f4e40a19b699b/yarl-1.23.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:d38c1e8231722c4ce40d7593f28d92b5fc72f3e9774fe73d7e800ec32299f63a", size = 90797, upload-time = "2026-03-01T22:07:34.404Z" }, - { url = "https://files.pythonhosted.org/packages/72/00/8b8f76909259f56647adb1011d7ed8b321bcf97e464515c65016a47ecdf0/yarl-1.23.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:d53834e23c015ee83a99377db6e5e37d8484f333edb03bd15b4bc312cc7254fb", size = 101023, upload-time = "2026-03-01T22:07:35.953Z" }, - { url = "https://files.pythonhosted.org/packages/ac/e2/cab11b126fb7d440281b7df8e9ddbe4851e70a4dde47a202b6642586b8d9/yarl-1.23.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:2e27c8841126e017dd2a054a95771569e6070b9ee1b133366d8b31beb5018a41", size = 96227, upload-time = "2026-03-01T22:07:37.594Z" }, - { url = "https://files.pythonhosted.org/packages/c2/9b/2c893e16bfc50e6b2edf76c1a9eb6cb0c744346197e74c65e99ad8d634d0/yarl-1.23.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:76855800ac56f878847a09ce6dba727c93ca2d89c9e9d63002d26b916810b0a2", size = 100302, upload-time = "2026-03-01T22:07:39.334Z" }, - { url = "https://files.pythonhosted.org/packages/28/ec/5498c4e3a6d5f1003beb23405671c2eb9cdbf3067d1c80f15eeafe301010/yarl-1.23.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e09fd068c2e169a7070d83d3bde728a4d48de0549f975290be3c108c02e499b4", size = 98202, upload-time = "2026-03-01T22:07:41.717Z" }, - { url = "https://files.pythonhosted.org/packages/fe/c3/cd737e2d45e70717907f83e146f6949f20cc23cd4bf7b2688727763aa458/yarl-1.23.0-cp314-cp314t-win32.whl", hash = "sha256:73309162a6a571d4cbd3b6a1dcc703c7311843ae0d1578df6f09be4e98df38d4", size = 90558, upload-time = "2026-03-01T22:07:43.433Z" }, - { url = "https://files.pythonhosted.org/packages/e1/19/3774d162f6732d1cfb0b47b4140a942a35ca82bb19b6db1f80e9e7bdc8f8/yarl-1.23.0-cp314-cp314t-win_amd64.whl", hash = "sha256:4503053d296bc6e4cbd1fad61cf3b6e33b939886c4f249ba7c78b602214fabe2", size = 97610, upload-time = "2026-03-01T22:07:45.773Z" }, - { url = "https://files.pythonhosted.org/packages/51/47/3fa2286c3cb162c71cdb34c4224d5745a1ceceb391b2bd9b19b668a8d724/yarl-1.23.0-cp314-cp314t-win_arm64.whl", hash = "sha256:44bb7bef4ea409384e3f8bc36c063d77ea1b8d4a5b2706956c0d6695f07dcc25", size = 86041, upload-time = "2026-03-01T22:07:49.026Z" }, - { url = "https://files.pythonhosted.org/packages/69/68/c8739671f5699c7dc470580a4f821ef37c32c4cb0b047ce223a7f115757f/yarl-1.23.0-py3-none-any.whl", hash = "sha256:a2df6afe50dea8ae15fa34c9f824a3ee958d785fd5d089063d960bae1daa0a3f", size = 48288, upload-time = "2026-03-01T22:07:51.388Z" }, + { url = "https://files.pythonhosted.org/packages/85/92/55fa9ee84cb8ec9930a910b5753936a926f918d8cc8965bdac571479c095/yarl-1.25.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:142c06c4d6a35ee3ec5da08499805e879cb3ca7c1fbfbecb0140fe72403818d6", size = 144695, upload-time = "2026-09-15T19:29:53.114Z" }, + { url = "https://files.pythonhosted.org/packages/16/b4/9edc8e605b16b2eb5bd0c2ccc73e2b15aab583862460767fb43f91f11839/yarl-1.25.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:24ce942011a61953e7d313438038f4d32ff21387b775f58a957f7a07dd55ef95", size = 104332, upload-time = "2026-09-15T19:29:55.443Z" }, + { url = "https://files.pythonhosted.org/packages/ae/de/4204e6646e278b1cd4a9cf73ebe8b7cfcf16693f1249b324eb273d67602a/yarl-1.25.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9e23c82b63cd7652fc24d33ed6cc17099d607aa3b4fc4ddc75e95062f3d82df4", size = 104449, upload-time = "2026-09-15T19:29:57.239Z" }, + { url = "https://files.pythonhosted.org/packages/1d/fc/21d74789198ad68a23a209ab0d73fe5a52d39ebbd9945041e72827acdd77/yarl-1.25.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8ee202350cf57abf0e9502a41601841019c25d3db7ff52d980aaf31446254059", size = 116828, upload-time = "2026-09-15T19:29:59.031Z" }, + { url = "https://files.pythonhosted.org/packages/8f/31/90891ddb61848c067ebd1fb505ab5902a6e7d4707c2f3823f67ff37883d3/yarl-1.25.1-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:5df89f769cc8ff94c3d7e7603386fba309d25ce5240132d26c15baa8d0e96c4c", size = 107150, upload-time = "2026-09-15T19:30:00.868Z" }, + { url = "https://files.pythonhosted.org/packages/a1/15/f3d18743d3688b5f01aa83034b6c00665692ab20e1d0c89a29000d4979ea/yarl-1.25.1-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:83e9f4a25085bd4b7214701a0794ff1f50fc633ffb8bdfebf07abdd81c2db126", size = 124704, upload-time = "2026-09-15T19:30:03.128Z" }, + { url = "https://files.pythonhosted.org/packages/99/37/718789d8004775d6a2db86b2afc72b1d156e0590b88a1b8634ddd4bdb8ee/yarl-1.25.1-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e636b64d24fd9c38053c5e389a1174c66361fa49dcfd220f4dd35b4abde7cb89", size = 129246, upload-time = "2026-09-15T19:30:05.014Z" }, + { url = "https://files.pythonhosted.org/packages/28/b9/7818ac6dec7fbcd16be2d19495807c01e2c38834e57d41c63356a75e786b/yarl-1.25.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e5637ca8d0bd7fb72648a6c7934af4baaccb697657f7438c9d264fc2abb8b0b1", size = 118071, upload-time = "2026-09-15T19:30:06.977Z" }, + { url = "https://files.pythonhosted.org/packages/b2/e3/8d098cafb30a64df7283b5a5dd0d32d22a71b18a1b9ec071d0776ffe5c2a/yarl-1.25.1-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:683e362b8ba453080f7489c66f4ea794e751c35b72e7eab3575ef784c2fbc7fb", size = 116180, upload-time = "2026-09-15T19:30:08.836Z" }, + { url = "https://files.pythonhosted.org/packages/20/9c/fbe9432478d87e00eef80242e15179639df162cb3d5d8d978fc56dcb6c51/yarl-1.25.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:df23df54b5114a17c2d0ef192433e2e5a9f0c5178c32375e90b7cfc965f349d0", size = 116578, upload-time = "2026-09-15T19:30:10.782Z" }, + { url = "https://files.pythonhosted.org/packages/b1/8e/b3dfd03732236b86b0bcf06a72a37de412ff6f82e719a947c3c651d4dece/yarl-1.25.1-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:f53dcd26694f148f738edc052b5a69234833e739f10f4c3287bdfd8ec0f7b326", size = 108886, upload-time = "2026-09-15T19:30:12.951Z" }, + { url = "https://files.pythonhosted.org/packages/83/e1/94dacb650d4963d5f844bc3df7ba70288cb7b68ed5a4d070da71acb19b14/yarl-1.25.1-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:b8075fe90bc08e40b8b8a1874fab42ee4c7b56af05c5886e9cc841397f916908", size = 124098, upload-time = "2026-09-15T19:30:14.963Z" }, + { url = "https://files.pythonhosted.org/packages/d6/3f/c93f76a218258c7bfc60dd27fd56e9102bdbc625517f4a6ee673295fe4e4/yarl-1.25.1-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:a8c2b841478068440d8b733005d13a5ef535b9928cbc05f17182d410f32ba449", size = 115768, upload-time = "2026-09-15T19:30:16.736Z" }, + { url = "https://files.pythonhosted.org/packages/a8/79/4d93f13c3b05cda3c962805dec28cbc255c50239b3457808abc5633a00c2/yarl-1.25.1-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:ca32926d7d77bcc8838425c4c95e040a3ace1cb7dfdae599013458dcda2607ca", size = 122461, upload-time = "2026-09-15T19:30:18.901Z" }, + { url = "https://files.pythonhosted.org/packages/c5/8e/e2ba83b3a9bfc1d3b882ad35fa8abc04a7af4d9356f2e329a23d0c8d889d/yarl-1.25.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:192a866877a49993949ef1975864ad8728bea28ee810f6abe1a0729c2b500426", size = 118295, upload-time = "2026-09-15T19:30:21.07Z" }, + { url = "https://files.pythonhosted.org/packages/42/67/cb5ea1baa0c0ac60bda44ce59f601133154af2a2e67722d3517f8793855d/yarl-1.25.1-cp310-cp310-win_amd64.whl", hash = "sha256:3f4d48a6112712973e676bd792121fee470e432d749177162d9949d5c9460a1b", size = 102929, upload-time = "2026-09-15T19:30:23.122Z" }, + { url = "https://files.pythonhosted.org/packages/2f/85/9a1e98de10fc0e738d517efd2a9984c3e7db6cf35fa72b2b373988a9e9b7/yarl-1.25.1-cp310-cp310-win_arm64.whl", hash = "sha256:48796ea00a303961507dc6c8437c4b325a6fc3f95f7c36c71b91ea9a8150963c", size = 98854, upload-time = "2026-09-15T19:30:25.102Z" }, + { url = "https://files.pythonhosted.org/packages/83/b3/2cea721d495ca57f8f414aa4867ee263f486b274e07463158cf52f02ba7f/yarl-1.25.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:9d693bf4bf534e9ba3ae2780cfd577f5135629f7b5ac653490859d0b77864865", size = 143794, upload-time = "2026-09-15T19:30:26.946Z" }, + { url = "https://files.pythonhosted.org/packages/55/e6/cd145cff8e5cf60b8b3c41fbecfa2a45028a8dec3fbc52bec03595ff3d3b/yarl-1.25.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ab2054c5531af2a9ba7b69b8ec91e4f884420e83a8c5e579b013084cb57e5e5d", size = 103993, upload-time = "2026-09-15T19:30:28.661Z" }, + { url = "https://files.pythonhosted.org/packages/ce/7c/fbb40fe2d53747c40aa36a9e2bd2178a202f942bda0b670f3306d4aefbce/yarl-1.25.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:564fdc7085d2245ab84f88882fdb1d6ac0723124bff6ded35bfb1c00f812630d", size = 104010, upload-time = "2026-09-15T19:30:31.069Z" }, + { url = "https://files.pythonhosted.org/packages/f0/0b/5a516f70641092283f57cf3670bdb75e7327bcc0dcb5038697e4dfbfd569/yarl-1.25.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:acae6b45d1ace09b6ba3876da43b88366ef368f73b988c7f57e14231753d4420", size = 116444, upload-time = "2026-09-15T19:30:32.894Z" }, + { url = "https://files.pythonhosted.org/packages/b6/8a/d1a627f827b0a404ae0f5647cab0534959081cde13b0756a783469fb3b5e/yarl-1.25.1-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:1fb2a01ba8cd9c5d2c5dc1ec35e0fc951d04b4f037541d4ac090c993ce58b3d7", size = 107565, upload-time = "2026-09-15T19:30:35.188Z" }, + { url = "https://files.pythonhosted.org/packages/aa/cf/c8e0aaec886840a6c4480fe44eaec7cd4319f79a563b79321473be79c56f/yarl-1.25.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e92b6bcc741b86d67606c40d3cb9c7cc8e6c737f81e31f4a94efc204456c92e3", size = 125006, upload-time = "2026-09-15T19:30:37.1Z" }, + { url = "https://files.pythonhosted.org/packages/50/26/0cce366d54a93cdc8342965dc7663385e4db161625cc1e8b18e786753d24/yarl-1.25.1-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:72c34ac7ad4314c19362d5ce27626dcc8429bd30bbf8c179f4234078851f9492", size = 128717, upload-time = "2026-09-15T19:30:38.904Z" }, + { url = "https://files.pythonhosted.org/packages/95/0c/a71501bbc1a674ff72c4d6c2b75f4d9a5af819f5244c3a7558080a8802c5/yarl-1.25.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d5add7b4ca7afeea91d52e4d4e4db3b1fe9885b71f07054560d8c4296b7441a2", size = 117728, upload-time = "2026-09-15T19:30:40.809Z" }, + { url = "https://files.pythonhosted.org/packages/e7/6f/c3267ca01defeed9ed9c4ff9b17bd54915432c405945233265b707475d1c/yarl-1.25.1-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:def538065f9e4d4cf1ae164bd59aba00dfa84f03923e0de4c3788f252d6bcd17", size = 116224, upload-time = "2026-09-15T19:30:42.818Z" }, + { url = "https://files.pythonhosted.org/packages/8f/69/fad57ee52d648431718ee0f1f68966a99c1352c3924688ffbdcc9d3fe51a/yarl-1.25.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:0a191bfdb30a79b98e5d175d75285f9fcb78bf0e46ba5efda042e1c72071a0de", size = 116299, upload-time = "2026-09-15T19:30:44.966Z" }, + { url = "https://files.pythonhosted.org/packages/2a/d4/6a8c1e29f33338687ca278cba0a8fbf6525a322c2c02a9a500ccbe041152/yarl-1.25.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:71f42c5b9a948c113bbdebfa544598321431d064ff959d32e99b1feb61d68345", size = 108625, upload-time = "2026-09-15T19:30:46.874Z" }, + { url = "https://files.pythonhosted.org/packages/e8/d2/3a35ae791c9cb6522c106923ff25c3230d999091e5e65511bca23bbd9914/yarl-1.25.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:72849d892954be4d09e569b8b831ac39ce58417fedc767d4308a0fe542018a40", size = 124515, upload-time = "2026-09-15T19:30:49.082Z" }, + { url = "https://files.pythonhosted.org/packages/be/fd/2b022109a6b4af0f7dc371cf7500af380b0d4f034010243e1b0ce218dc93/yarl-1.25.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:efb01a106f971cb3752856bca2318bbdf7f01bd8823779c461586cbe5ffd5258", size = 115711, upload-time = "2026-09-15T19:30:51.208Z" }, + { url = "https://files.pythonhosted.org/packages/18/59/f7586271136c3ddb0126bbfe661844699369b76b555fe38c4efe86870b2e/yarl-1.25.1-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:a1daf47cd95a7c3a63456336bc5aaa8c86dd3a47d07ed3d0e76132ae4666a5a1", size = 122751, upload-time = "2026-09-15T19:30:53.535Z" }, + { url = "https://files.pythonhosted.org/packages/a0/0b/07f7a2d881f7e16c385b47fc1753382600847cad305dcc7fa0c25828acf8/yarl-1.25.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:9489e6abf47ba37f332075a91444c7cfedb03e6ce99fbb2f116bfe1ce810da3b", size = 117983, upload-time = "2026-09-15T19:30:55.277Z" }, + { url = "https://files.pythonhosted.org/packages/aa/9d/8cdceec66a9b940700cb45931741403f045b162afd79bcb93c41cadd0972/yarl-1.25.1-cp311-cp311-win_amd64.whl", hash = "sha256:d7306dee25b8a0e737363f347362b875094b4dc4e367311470656ae420fdbf8e", size = 102894, upload-time = "2026-09-15T19:30:57.606Z" }, + { url = "https://files.pythonhosted.org/packages/17/f1/7ec357db1d3ad2863542d71e8fe64a126bbec17b134cd7d30951196809a5/yarl-1.25.1-cp311-cp311-win_arm64.whl", hash = "sha256:abb1384477f5901d436b5d2e5465954de46ea6098f59163d243660b5c4461d35", size = 98609, upload-time = "2026-09-15T19:30:59.705Z" }, + { url = "https://files.pythonhosted.org/packages/75/b3/cd32ac66ae622b854c2df0ac52106dda220d361b65a64fde7d5b3684aa3f/yarl-1.25.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:94d7aa6debf92a1dd14cb5280b083a764169a13cfb23a452111160274ed989f4", size = 144798, upload-time = "2026-09-15T19:31:01.821Z" }, + { url = "https://files.pythonhosted.org/packages/61/fb/a2c52a8007c2051ba74662afb112ecf3d00346af4c25e33df9d80fd14fb8/yarl-1.25.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:83d4a37e4b95da4d8bda930d6d35b75b4cdadbacbb4980cae290ea3100b5d51d", size = 104583, upload-time = "2026-09-15T19:31:04.05Z" }, + { url = "https://files.pythonhosted.org/packages/be/dd/ee38aec8e09fdf957e50d4085453fbe202f56c6c3b4cf07b81cdb4f09ee9/yarl-1.25.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e029648f9c951db30e98a7d7ec90835db88ec4b32820efe2a9bdc2287e032eb6", size = 104325, upload-time = "2026-09-15T19:31:06.338Z" }, + { url = "https://files.pythonhosted.org/packages/1e/b3/058dbfb1857b484c9cf9cc135659f50b85ce66e03c99e44dc2f7b6161f55/yarl-1.25.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4d781294bb815ecb5ea57ff6bbf8038e0a31a95fdf3e1788f66e0dc100d64b58", size = 115358, upload-time = "2026-09-15T19:31:08.593Z" }, + { url = "https://files.pythonhosted.org/packages/db/39/29693446cf0cf6b15a0e2f75a5d40f93c56819b05b0622196f45e95b5cc0/yarl-1.25.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e12c538e00e7c1b286a07061046b90e8124e6a9793efae2c70db6a4aad07faad", size = 107658, upload-time = "2026-09-15T19:31:10.802Z" }, + { url = "https://files.pythonhosted.org/packages/86/b3/3c4dd7e1af43b931fba95e0a722737f2ea94a6d199c802585282831d7abd/yarl-1.25.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7e4de3ac4adbad3d0bc7c6f4360a7dbff5de2f15e3b723be3198074e17fd9c40", size = 122660, upload-time = "2026-09-15T19:31:12.84Z" }, + { url = "https://files.pythonhosted.org/packages/bd/b5/1b60dbc3cfc9c5712b15148c206748f2bc93953ffdbe25ea75b63dfc89c9/yarl-1.25.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:419f392a1da624877975709e3864dfe833af6cc7671b39318086d456e288380c", size = 126506, upload-time = "2026-09-15T19:31:15.088Z" }, + { url = "https://files.pythonhosted.org/packages/bc/7b/ca212cbe170ac8b96e45317ecbcf9c3c3ecf0cdec98d5b088a9c4088929b/yarl-1.25.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c6f117789d22dce188e5754e8bc65b7e6ebf8cb73963b9fa761f672a5883769d", size = 117050, upload-time = "2026-09-15T19:31:17.241Z" }, + { url = "https://files.pythonhosted.org/packages/cb/c3/72b4938cdbe619ad71ac156182faef4908846b84dc3ca4dbb4c4e6f84014/yarl-1.25.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:80e47012e730da131c9f059c80936783f9659aae22dc31c03c0595590d11ed54", size = 114174, upload-time = "2026-09-15T19:31:19.294Z" }, + { url = "https://files.pythonhosted.org/packages/e8/43/268717870f9ba0cc9701a95181587f6dc8c5f387aab4aeecc83158f38a79/yarl-1.25.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e80f557716fd765439577131e526b8942ffc2c07bdbc5e39fa62f660ba1e963f", size = 114944, upload-time = "2026-09-15T19:31:21.414Z" }, + { url = "https://files.pythonhosted.org/packages/da/84/baa5bf504d51fe062c4bcaf62936da97fffb43285978d0b39984824231fd/yarl-1.25.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:f61964f235a43738bfac50da46fc4254943a7eea3051aeb0b6fc7c992c29fadc", size = 108263, upload-time = "2026-09-15T19:31:23.388Z" }, + { url = "https://files.pythonhosted.org/packages/a4/28/779a2ed9e0152a601a27039bed9aead3f0b79797a67e2c44bfa444622dd8/yarl-1.25.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:e546fe1d4a93ebc2910f0d768baff19faa09843ab3f2036a67ed6e69fae4419d", size = 122184, upload-time = "2026-09-15T19:31:25.343Z" }, + { url = "https://files.pythonhosted.org/packages/f8/1f/118e9e5b8f07694d63fd3222e801d7782270003f1a222aa798df3f8d5933/yarl-1.25.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:cce0727fd5ac04d372fa9bbfde9febc2bcf209aadfcf0468e45dec72719895d1", size = 114001, upload-time = "2026-09-15T19:31:27.465Z" }, + { url = "https://files.pythonhosted.org/packages/0f/ae/a4cf1cf372313734b17996d4007f9f73596e7a178b9485802e5494ecf484/yarl-1.25.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:af4ea5b37403ef4e30f3927eaed540db942bde01d8d3ff083527c0704d1c9c68", size = 120565, upload-time = "2026-09-15T19:31:29.47Z" }, + { url = "https://files.pythonhosted.org/packages/05/79/ad94f93ca731bc9e44d321833ab96b82a4f9f5f63cf773f81a4aeea5ecc1/yarl-1.25.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:68782fdb4027b8d1eee25ec35e9a6db05e863b899eb0310b3a33b6c3fef55707", size = 117060, upload-time = "2026-09-15T19:31:31.367Z" }, + { url = "https://files.pythonhosted.org/packages/bb/cc/51a7b4abf4ac593b8e7eb3794b28e5a35ae26eed8bc04787628d215af82f/yarl-1.25.1-cp312-cp312-win_amd64.whl", hash = "sha256:7d575b54cb3863ef9bc290ea4b009999d55dc237326131e4853cf33e888fee03", size = 102593, upload-time = "2026-09-15T19:31:33.329Z" }, + { url = "https://files.pythonhosted.org/packages/9d/21/0941a6b93a58b59a1ec75e5333bf06929b671309c43c0cd201c172d9c39f/yarl-1.25.1-cp312-cp312-win_arm64.whl", hash = "sha256:bc3ac7bf569f6b64dad04dd7808c7872dae8a97df657856eac05e9b7e3614a85", size = 97697, upload-time = "2026-09-15T19:31:35.855Z" }, + { url = "https://files.pythonhosted.org/packages/7b/ed/2f3129bbcc9a5c8ba12cc2b29d8060a3bab9c8043c456cfd4b5ca3188890/yarl-1.25.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:25868beca8b6765f8f7d0e11fe6dd7c66dd4b0793b9500286d20cc92352126a5", size = 143623, upload-time = "2026-09-15T19:31:37.966Z" }, + { url = "https://files.pythonhosted.org/packages/17/e1/f1bc3390fdca352826676b531d0712736f156919090206700421d46b2c37/yarl-1.25.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:10b2fd95332f0d716d5eee3c9fb2ce8eada19082de7fee83d32e37992fd75c26", size = 104011, upload-time = "2026-09-15T19:31:40.25Z" }, + { url = "https://files.pythonhosted.org/packages/a8/aa/50acc5c3e5da04172ae3c281c75405af4d2ca911e16120ab0563f4dffb66/yarl-1.25.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0f12afda4eea8c8994a76d4df1875c765194f5fbe8a9d197929ea303caee29ec", size = 103677, upload-time = "2026-09-15T19:31:42.46Z" }, + { url = "https://files.pythonhosted.org/packages/30/d2/7d1e0ab9f8390e1fbcede5a6dbf70d23c96ad09b8c5567f3a514d1ddb0e2/yarl-1.25.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:14b79a30a93a3ce2e8832603fd0ab780ada281b0ba5110b519a634f2d7d7d1fc", size = 115392, upload-time = "2026-09-15T19:31:44.371Z" }, + { url = "https://files.pythonhosted.org/packages/71/e1/5ba1e3a2a22139213655e760919038e8ed7e2d4a99826d0bbddb3beb96e5/yarl-1.25.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:4bd6340d20ae2c7ca719b87b426e808e90743b676d05d4c26c4fb5ca71f41184", size = 107493, upload-time = "2026-09-15T19:31:46.273Z" }, + { url = "https://files.pythonhosted.org/packages/f5/53/780653d5e0f73831f467cf13548912e5eec97f21dc49fc8daf21da027df4/yarl-1.25.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:126a2533570c554719ca40a1288fdee1700b6bc82e7131aa69fa85252d92e651", size = 122537, upload-time = "2026-09-15T19:31:48.654Z" }, + { url = "https://files.pythonhosted.org/packages/03/92/d54fa70236c6036271c9c9c09fd978df5cbe3ef49ef6c46e9b833476d215/yarl-1.25.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a3faadac7d812ddac258feb57b9846b60c1b437c4f4b9ad42595c6f6fe4390df", size = 126170, upload-time = "2026-09-15T19:31:50.872Z" }, + { url = "https://files.pythonhosted.org/packages/0e/b7/a82a49bf88340b837ef6972b508a1604ae377b9e6904b46b10cf5f1cf925/yarl-1.25.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:be80550d9bfe83d9b62398a37081a90434e6df2d978ec345c3d2820de6beddab", size = 117012, upload-time = "2026-09-15T19:31:53.189Z" }, + { url = "https://files.pythonhosted.org/packages/ef/78/5d684b411e3f3602464ee9b538db48205038f8605872985f61efb809ced0/yarl-1.25.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e07595c7d6f4db270ceede356a1bd1c07a34f1c26f958d1ed0cd7b48e0d2bba3", size = 114950, upload-time = "2026-09-15T19:31:55.694Z" }, + { url = "https://files.pythonhosted.org/packages/2f/11/51d82b852c64f7fad0fc7a7ff3031517204887e874c722bbca839c0b23ac/yarl-1.25.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:eb96ed1ae6c7d072d60840c0434aef07a2df611812810807fbc54263a6053e9a", size = 115428, upload-time = "2026-09-15T19:31:57.966Z" }, + { url = "https://files.pythonhosted.org/packages/e4/49/9d1978049bf646b9ea918313926453c6901b71c92f097467777d47d36a88/yarl-1.25.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:3feb99222553a8cbedfa52c2f59dd84c3f50d5b582c728d522caf8d72769a54b", size = 108428, upload-time = "2026-09-15T19:32:00.048Z" }, + { url = "https://files.pythonhosted.org/packages/43/35/7b8f1ebb45d7ec3dda7d1909bf44f458de41ef91e2937f107733582a5166/yarl-1.25.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:a2ed0ba415ccdf08f14bf544cb78346d0f76086707ffee24921a2c84dbf1305a", size = 121961, upload-time = "2026-09-15T19:32:02.436Z" }, + { url = "https://files.pythonhosted.org/packages/63/d6/d8b689ab7ca26edeb85f6ff28812aac7a25376eefc1780e303a7bfbaceff/yarl-1.25.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:2b49375d22299b0a834c2bca72f39aaecc270d96fb24c30424899676f487b22a", size = 114961, upload-time = "2026-09-15T19:32:04.456Z" }, + { url = "https://files.pythonhosted.org/packages/cf/d5/1a1798ea4dc6b7ee3260010a27907ebc697c95dae99817d817ed446d24aa/yarl-1.25.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:ef74070ac553c59eb4f04258722066d6c6135b7baa03b2e9f2da65c096e96d98", size = 120036, upload-time = "2026-09-15T19:32:06.5Z" }, + { url = "https://files.pythonhosted.org/packages/91/8d/b1b35ed7903da6669b1d367cb2c09436acd4ff508029b4f39a0c0c2058fc/yarl-1.25.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:0a66db89ea473abeac4b70523cafd94db3772380e565f9d28af7a179b7af71fa", size = 117276, upload-time = "2026-09-15T19:32:09.401Z" }, + { url = "https://files.pythonhosted.org/packages/3a/8f/4db01cef62caff0d7a4593ed694fb8a41a27a11158cab80d290221f13e57/yarl-1.25.1-cp313-cp313-win_amd64.whl", hash = "sha256:1f51020b2eb8a003c84925638ec63c21a750a4bddd3a22ec8eac6a742dadf1b9", size = 101945, upload-time = "2026-09-15T19:32:11.545Z" }, + { url = "https://files.pythonhosted.org/packages/c0/5e/3ce00497c5c0babb74d4130c10c3828ccd215b4819d12020c42429f991ac/yarl-1.25.1-cp313-cp313-win_arm64.whl", hash = "sha256:b10dd0557ba422715b5206b3743192135a6022acca8baec51aa127d0a75db8fe", size = 97270, upload-time = "2026-09-15T19:32:14.127Z" }, + { url = "https://files.pythonhosted.org/packages/80/cf/54023edfab7aa773b860503db0c56e962ccab0922803ee97988c176ea090/yarl-1.25.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:a9ca696eb02e5c02a8afd872ada510eba9b7fe6e68b9572c2e9a9b1941e31e2e", size = 143975, upload-time = "2026-09-15T19:32:16.416Z" }, + { url = "https://files.pythonhosted.org/packages/d7/a8/e6c1be0e6761d0f2d10bbf33a3e1e02b99dc83874d92945d7b461a72481e/yarl-1.25.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a5877f2255aab518ebe528289037699201d5dc5f045f2396cb30aa02db22f57f", size = 104018, upload-time = "2026-09-15T19:32:18.364Z" }, + { url = "https://files.pythonhosted.org/packages/6e/bb/dda344765ffd3430afe1a1c66c866a57fae67786537d4f14607df6505ac1/yarl-1.25.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:7a5c3115595995779ee21f2567035793911c3802a43c74f3fbb0314929ec67ac", size = 104156, upload-time = "2026-09-15T19:32:20.459Z" }, + { url = "https://files.pythonhosted.org/packages/e5/5f/ed1538bcd06009fe990d6d283dd7667f639e62a81e35c6d8c6ef6c08fb3c/yarl-1.25.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:77e5099b99b37f3cf79c246998ca9f7313a78054cd1809ec46bc1afad47e1c4c", size = 116025, upload-time = "2026-09-15T19:32:22.766Z" }, + { url = "https://files.pythonhosted.org/packages/a2/af/2185daf56b99830d3356ecfada46faaa49945de6626e842b7728088d4980/yarl-1.25.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:6efaf45df6a849cef613a03a94c845647456662f85438c886bb67a9c027c8c2c", size = 106985, upload-time = "2026-09-15T19:32:24.749Z" }, + { url = "https://files.pythonhosted.org/packages/c1/65/bc1ae564fb4b04a30b6a8f250e787772581c57e4c3d5cf07ac3359de3103/yarl-1.25.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d5f90e44653c4e0f78501ed9bb7d3fce835a8d62b7c6ed0cb16557534087e743", size = 123030, upload-time = "2026-09-15T19:32:27.084Z" }, + { url = "https://files.pythonhosted.org/packages/6a/3e/e2afcde10d74e53b3fa889960991efb3019beda2b1682a01de720a302056/yarl-1.25.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:632da579b2d879f6bad20f2cfa35ded1efe2f4f77f8abb26a6234a5b236acd2f", size = 126765, upload-time = "2026-09-15T19:32:29.332Z" }, + { url = "https://files.pythonhosted.org/packages/a2/be/415b00c0fe5a0615b062a456b26623d7ec91c2bee20faea1a14045aa0469/yarl-1.25.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:30eec96e8a91bd588ce897c9543f6d5d8d34b28fbcba28a4dedf20ebeae9fe57", size = 117199, upload-time = "2026-09-15T19:32:31.49Z" }, + { url = "https://files.pythonhosted.org/packages/97/27/3d8c63ddd3e8bcfd033748ab93876678ce59bacd66e4cb1ed851c9c5b37e/yarl-1.25.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:12b6bc4906e11f5e1a1cdcb12296e7afbd366c783cc8073403cd2fb74334e453", size = 115187, upload-time = "2026-09-15T19:32:34.137Z" }, + { url = "https://files.pythonhosted.org/packages/39/b7/7a81d0be1a502a26a0d4326c6f2ecb736c824f570ea1c6529f2b0b227b50/yarl-1.25.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9d6ed3d17bccce4c05343e1ca8da13bc5c02c812a4e7282ddd05e8769322d3fc", size = 116085, upload-time = "2026-09-15T19:32:36.438Z" }, + { url = "https://files.pythonhosted.org/packages/f0/69/39fff459916aa0fab42215dc47b759586fd80f94aa56dfc4a7c15ba6e0dc/yarl-1.25.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:f38a70074041d3b7e138e452799f5174198bae5bd5ab2000917badf403908c5f", size = 107996, upload-time = "2026-09-15T19:32:38.959Z" }, + { url = "https://files.pythonhosted.org/packages/c0/39/80b9a55a3335590451d9ecf3eb593a8c635351f4c905ef056d7e8a8fd9e7/yarl-1.25.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:4ca89e4e21854ed27ec753297dde84b16c9f8e53b14a4866fb44457d643c19f8", size = 122549, upload-time = "2026-09-15T19:32:41.151Z" }, + { url = "https://files.pythonhosted.org/packages/42/7d/a179c6757818bb59372a4adafd09f7f26a3b4a0f04c3ae404b544c0b0c82/yarl-1.25.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:1ab7618921a93767387a4b83776f751588f5b5ae9bb5bc96620e2e2e00bca868", size = 115107, upload-time = "2026-09-15T19:32:43.072Z" }, + { url = "https://files.pythonhosted.org/packages/32/2b/a773ac867e4ab53a98ed98e5cefe3bae31e6f550252ca9d1de266f1a40c5/yarl-1.25.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:0ae12ff2b805fa02c4dab838005caef735e39986322698c48588d3beacb65c62", size = 120666, upload-time = "2026-09-15T19:32:45.061Z" }, + { url = "https://files.pythonhosted.org/packages/bc/41/52be6505e85b0f76b4f85b01b5de7e06a0512201abc2c95e14e099549174/yarl-1.25.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:90c30ed53546da833c700115c0064c22120d1b1560f474699fd31f22dd668233", size = 117505, upload-time = "2026-09-15T19:32:47.177Z" }, + { url = "https://files.pythonhosted.org/packages/f5/01/349c0386caedbbe488d519f252df54efac8a1459282d466c474bdd84a620/yarl-1.25.1-cp314-cp314-win_amd64.whl", hash = "sha256:acfa7e22aa6c6e7a5996a41d275bfa01efa7ea56ab890590280e9063e2cf5c1b", size = 103446, upload-time = "2026-09-15T19:32:49.615Z" }, + { url = "https://files.pythonhosted.org/packages/5c/f0/8ec63180f77912f0dc4e5a42760cb8c08d20da1d5ace3578a01b84d1f3d8/yarl-1.25.1-cp314-cp314-win_arm64.whl", hash = "sha256:8e7d98cdbb6d71e726f7d525952867096053d1f290dd4e3c50d7d313a136f414", size = 99159, upload-time = "2026-09-15T19:32:51.686Z" }, + { url = "https://files.pythonhosted.org/packages/47/7d/92d2220d6886b70ab1ed8579533ac2af2dfac716d5d929001daff7986df9/yarl-1.25.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:d21f0fa80a02d05299207eeaafef345d812ace96d5306e4ef265e1d419a615fa", size = 150071, upload-time = "2026-09-15T19:32:53.911Z" }, + { url = "https://files.pythonhosted.org/packages/64/fc/b245e448124bcda9340df38e3553fa222b50260fca027a84095e9bd8642d/yarl-1.25.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:17c9877a89fb6e2bca6f9087eb24cd7fb434653946ef5075e470d23d49b52287", size = 106780, upload-time = "2026-09-15T19:32:56.443Z" }, + { url = "https://files.pythonhosted.org/packages/51/e2/9a6ce2e334ebf218a30335ae76fb1696459430d42f733b8cb0d7d65b84d3/yarl-1.25.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:29273edf1530e397bd07cb784db1fbe0d2590b77569f2e24679a9c0a2d763b94", size = 107361, upload-time = "2026-09-15T19:32:58.827Z" }, + { url = "https://files.pythonhosted.org/packages/ed/70/66e8c76b569b450d16e190f15071c916c3df70b0e33927e415ac497cf0c2/yarl-1.25.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b7abffdf37af1cec6a2ad69b827aa84320db5894791bc8ed932dc93fb274b7e9", size = 114396, upload-time = "2026-09-15T19:33:02.24Z" }, + { url = "https://files.pythonhosted.org/packages/73/23/0d82838a05c57fdc05bc8b66e8c92dcc0df15e27463a5f163142d521c682/yarl-1.25.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:2239a02249d9326655419e0168a28ca9008938eaab31dc29fc875c217927a6c0", size = 104882, upload-time = "2026-09-15T19:33:04.494Z" }, + { url = "https://files.pythonhosted.org/packages/86/d4/ea08615c4edaa6049a13a2f1128944d068d1893abda7d708d4d7ea01599a/yarl-1.25.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:664ec6a520b74a1df2810666eb67695fcb77fa663e6ea0a25aaf2e529cb24dfa", size = 119485, upload-time = "2026-09-15T19:33:06.583Z" }, + { url = "https://files.pythonhosted.org/packages/1a/82/0898bdce9b1ae403b308b9c733d0d24af4a3464270c2c081f457b16c3e0d/yarl-1.25.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4f1c91f5a5980a937ff8e238e98e6897e1ad74a4b1e2c0d68c73b5ffbb3f5c0b", size = 122490, upload-time = "2026-09-15T19:33:08.653Z" }, + { url = "https://files.pythonhosted.org/packages/d1/38/97d79b81c342b78246cfedb74809e68841f3198d21653e10d3232bd9c622/yarl-1.25.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7c88edaec8c349ad4c5ad4c486a3defcc4b80ceb2f074436ffa0a87caf5e76a6", size = 115336, upload-time = "2026-09-15T19:33:11.056Z" }, + { url = "https://files.pythonhosted.org/packages/8e/9d/2577896554cd310dc470adb6da0b7dd0b435cb63e2565204a7ac240e504c/yarl-1.25.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:35dcbea443fafb3eece757ad4e514560ddeb6c34cfae1582c620d7b293d7feee", size = 111825, upload-time = "2026-09-15T19:33:13.204Z" }, + { url = "https://files.pythonhosted.org/packages/29/6b/7ac49d8ba84a5c4bd73415a4c949d22c749cb3762579b3d50e48019a78aa/yarl-1.25.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:882569ff613758cac762a457a5d72d6e211b28d4bcfea89d1d71ea942b02eac0", size = 114655, upload-time = "2026-09-15T19:33:15.553Z" }, + { url = "https://files.pythonhosted.org/packages/e5/18/e5942a16723f5b72f9b1297fd5a85a54f6300cd15c0dcb5005b90cd89156/yarl-1.25.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:d0f1489233a254bb3643d2f05de7d59019254d81daeca6b9162fe9edef57e0c7", size = 106395, upload-time = "2026-09-15T19:33:17.599Z" }, + { url = "https://files.pythonhosted.org/packages/7b/2d/549fa46240781513ebc47ae7eb418df428a163a2a3d644cc9cbb3ecb7846/yarl-1.25.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:f41753a76f4f63927d03a0d8ba8f5ce0f2083bec29a8cfaccc55371b1564b96b", size = 119277, upload-time = "2026-09-15T19:33:19.973Z" }, + { url = "https://files.pythonhosted.org/packages/76/16/4763f78dcdc0b3b9fb3842b04afe72b9320857c6a69300c62a0eab03d119/yarl-1.25.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:8fb0eb4955adf0579001581f2f71a126e8781ba61bcd120f127b0401163c6c2d", size = 112504, upload-time = "2026-09-15T19:33:22.464Z" }, + { url = "https://files.pythonhosted.org/packages/ae/b4/974e3edfe0d188393ce1cb9de400111c63fe61f4eb3b772a500d84c970d1/yarl-1.25.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:a1e32763e641a1566507d90a8d3b19bfc3cc04a9d4e5ae3e32189874ed4b58a3", size = 116243, upload-time = "2026-09-15T19:33:24.788Z" }, + { url = "https://files.pythonhosted.org/packages/b0/aa/157b940428da80c104ca09666a740e51c94963df65d5b112e06b52e4d7a8/yarl-1.25.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:65b5b2066651b7432d389e9799d979c703bcc6ef44266bb8153ef54e91e4aab3", size = 115822, upload-time = "2026-09-15T19:33:26.886Z" }, + { url = "https://files.pythonhosted.org/packages/7e/af/19fbdce41412e1b96825544cc52cd7029d3724655d0988237972f078bd29/yarl-1.25.1-cp314-cp314t-win_amd64.whl", hash = "sha256:734f6e5400352ac4254456003d462866c684703570929cff7a7bde015d0cb371", size = 107386, upload-time = "2026-09-15T19:33:29.009Z" }, + { url = "https://files.pythonhosted.org/packages/2a/99/f6431c8968e89be608d74b28ae2d024521b2953f27dd44e0dece5e04f67a/yarl-1.25.1-cp314-cp314t-win_arm64.whl", hash = "sha256:287e99ff5aa4dc1c7630bfc683ded6f106d756c99dec432a2d7f197a784f51c6", size = 102094, upload-time = "2026-09-15T19:33:31.151Z" }, + { url = "https://files.pythonhosted.org/packages/54/22/318c7980066769c6bcd9221ed2248294f5698811da099013098c670565ed/yarl-1.25.1-py3-none-any.whl", hash = "sha256:681c758b0490f9e96b78e5fa8e8dc6e648e9185bb6eaebe73183c33ea0c445f3", size = 63617, upload-time = "2026-09-15T19:34:59.616Z" }, ] [[package]]