* feat(proxy): log one line per token count call
Each count call on /v1/messages/count_tokens, /utils/token_counter, /v1/responses/input_tokens and the Gemini countTokens routes now logs one INFO line with the litellm call id, the requested model, the model the count ran against, the count, the count source and the status code, so it joins the inference call that sends the same x-litellm-call-id
* test(google_endpoints): give the count test's fake request a headers list
* fix(proxy): keep token count log lines one per call and trusted in JSON logs, and log Bedrock native counts
Quote log values that carry whitespace, quotes, equals, or control characters so a model name can't split or forge a line. Let JsonFormatter record extras win over a dict parsed from the message. Log the Bedrock native count-tokens route through the same line.
* fix(proxy): escape braces in token count log values, log the Bedrock model AWS counts, and skip the line when there is nothing to count
* fix(proxy): log the deployment id the count routed to on every token count line
* fix(proxy): log count requests rejected before counting with the status they answer
A count route that rejects a request before the shared count function runs now logs the same token count line with its 400. An unsupported content block on the Messages count route becomes its 400 inside the logged wrapper, so its line says 400 instead of 500. A count that ends in a BaseException other than cancellation logs a 500.
* fix(proxy): compute the Bedrock counted model id only after a successful count
* revert(logging): keep the JSON log formatter's field order as on main
The count line already escapes braces in its values, so the formatter never parses a dict from it. Changing the order for every logger is a separate decision.
mistral/mistral-large-4 and mistral/mistral-large-4-0 still carried the launch sale price (input 6.8e-07, cache read 6.8e-08, output 2.09e-06). Mistral's model card and pricing page now list only the list price (input 1.36e-06, cached input 1.4e-07, output 4.18e-06), so every call was billed at about half its real cost. The test derives expected cost from the row's own prices instead of pinning Mistral's numbers.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(chatgpt): derive Responses session id from prompt_cache_key
ChatGPT partitions the Codex prompt cache by the Responses session-id
header (the Codex CLI sends a stable per-conversation key derived from
prompt_cache_key). The provider only consulted explicit session ids
and otherwise stamped a per-request uuid4, so no turn ever reused the
prompt cache -- 0% cache hits on repeat turns, an order-of-magnitude
quota/cost multiplier on subscription accounts.
Resolution order after this change:
explicit session config (session_id / litellm_session_id /
metadata.session_id) > prompt_cache_key (sanitized with
_safe_header_value) > litellm-internal ids > uuid4.
Two guards, mirroring get_fireworks_session_id:
- proxy-generated session ids (missing_session_id: "generate") are
per-request and are skipped, checked in both "metadata" and
"litellm_metadata" (LITELLM_METADATA_ROUTES, responses included)
- internal per-request ids (litellm_trace_id / litellm_call_id) lose
to a stable conversation key
Supersedes the body-key forwarding of #37630: a random session header
overrides the body key on the provider side, so the header must carry
the caller's key. Related: #29978, #29993.
Verified against a real ChatGPT Codex account: stable session id ->
97-100% cached on repeat turns; random per-request -> 0%.
* style: ruff format
* fix(chatgpt): address review - hash unsafe cache keys, move tests, drop comments
- hash prompt_cache_keys containing header-unsafe chars instead of
mapping them through _safe_header_value, whose replacement char could
collapse distinct keys onto one cache shard and serialize unrelated
requests (greptile review)
- strip the explanatory comment blocks per AGENTS.md comment policy
- move the tests into the existing mapped responses transformation test
file per AGENTS.md test-placement rule
* fix(chatgpt): keep session typing and test diff scoped
* fix(chatgpt): hash surrogate cache keys, type session params, flatten tests
* test(chatgpt): record upstream requests with a mock instead of appending to a list
* fix(chatgpt): keep proxy-mapped session headers ahead of body session_id and ignore non-string cache keys
* fix(chatgpt): hash cache keys with leading or trailing whitespace so the session header stays valid
* test(chatgpt): type the session id test parameters
* fix(chatgpt): keep trace ids copied into litellm_session_id from outranking prompt_cache_key
* fix(chatgpt): mark session ids copied from trace_id so explicit ones keep precedence
* fix(chatgpt): keep the trace-id session marker out of provider params
The marker now uses the _litellm_ internal prefix, so bridges that pass litellm_params back into completion() drop it instead of sending it as a provider param
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(images): cover SDK retry budgets and router ownership
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(images): forward max_retries to the OpenAI image generation client
Per-call, deployment and default_litellm_params max_retries were filtered
out before reaching the OpenAI-compatible image handler, which then always
used 2 SDK retries. Forward the caller's value, and pin it to 0 for routed
calls so the Router stays the only retry owner, matching completion()
Co-authored-by: Ericz <yonglzh@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(images): read image max_retries through a typed kwargs view
Co-authored-by: Ericz <yonglzh@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(images): assert the public invalid retry error mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(images): preserve the invalid retry status code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(images): group the proxy test imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(images): discard proxy retry test routers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ericz <yonglzh@gmail.com>
* test(huggingface): cover TEI rerank truncate passthrough and server default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(huggingface): stop forcing truncate=false on TEI rerank and pass truncate params through
Co-authored-by: Luke Schaefer <225965040+luke-e-schaefer@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(huggingface): keep TEI truncate options provider-specific
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(huggingface): use MockTransport for rerank truncation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Luke Schaefer <225965040+luke-e-schaefer@users.noreply.github.com>
* feat(rust): one typed decline from core up to each host
Rust core says it cannot serve a request with one Decline that travels
unchanged through llms and inference up to the host. The Python bridge
raises RustBridgeDeclined for it in route_failure before the public
exception mapping, so dispatch falls back to legacy Python instead of
failing the call, and the gateway answers 501. Every route collapses
errors after the send with Error::after_send, which keeps a provider
answer and a response fault as they are and nests anything else as the
source of an invalid response, so a billed call can never read as a
decline. Response-side cases that used to decline are now
ErrorDetail::Unhandled response faults.
The rust-error skill documents declined() as the first question both
boundaries ask, and the Python suites pin that a native decline reaches
the dispatcher before any provider IO.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust): keep post-provider hook failures terminal
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy/auth): cover default budgets on cached virtual keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy/auth): use in-memory cache in cached budget tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy/auth): apply max_end_user_budget_id to end users without a row on cached keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy/auth): share cached end-user parameter update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(scim): audit #47544
Add integration cells for the SCIM group delete that now runs through the team delete's bulk path: an admin group delete revokes its members' admin key, a delete sent with the IdP's SCIM token revokes member keys and audits them under that token, a delete that times out on a locked member row changes nothing so a retry works, and two racing team deletes write one tombstone and one team audit row
* test(scim): read the racing team delete's audit rows after the proxy drains them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds a real-proxy case where the MCP follow-up call fails on every router retry, checking that the one failure row bills the first call of every attempt on chat completions and /v1/responses
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #47352
* test(proxy): audit #47352 lazy load during a cold docs build
* test(proxy): route the overlapping chat cell to the rig upstream
* test(proxy): keep the root path cell from rewriting the packaged UI
* test(proxy): prove the lazy and chat cells overlap the cold docs build
* test(proxy): warm the chat path before the overlapping chat cell
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(cli): keep the ChatGPT login with lite codex --subscription
`lite codex --subscription` points Codex at a `litellm_subscription` provider with requires_openai_auth on and no env_key, so Codex keeps its own ChatGPT login and the LiteLLM key is sent only in the x-litellm-api-key header from LITELLM_PROXY_API_KEY. OPENAI_API_KEY is left exactly as the user had it, and the pre-launch key check sends the key the same way
`codex_subscription` is a new `lite config` key that turns the flag on by default, like `claude_subscription` does for `lite claude`. The Claude `--settings` overlay now applies to Claude Code only
* fix(cli): leave the Codex Stop hook out of subscription launches and load tomli on Python 3.10
* fix(cli): leave OPENAI_BASE_URL as the user had it with lite codex --subscription
* fix(cli): give lite codex --subscription its own Codex provider id
* test(cli): type the Codex config override helpers
* fix(cli): leave the Codex model list to Codex in subscription mode
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): per-provider and per-model rpm/tpm requirements for enforce_project_model_quota
* test(proxy): cover per-provider and per-model enforce_project_model_quota policies
* fix(proxy): keep enforce_project_model_quota reads typed
* fix(proxy): apply the quota policy to model add, follow router wildcard rules, keep old setting values
* fix(proxy): keep team model names off quota model entries, read the provider behind api_base, match patterns like the router
A team admin picks a team model's public name, so a name matching an exempt model entry no longer skips its provider's limits. A mapping with no policy key keeps requiring both limits as before, the project create permission check runs before the setting is read, and an unprefixed model behind a known OpenAI-compatible api_base resolves to that provider
* fix(proxy): re-check team projects on team model writes and validate quota provider ids
* fix(proxy): check added limits on model updates and match team models by the model they call
* fix(proxy): re-check team projects by every name a team model serves and check renamed names
* fix(proxy): follow the router's order for tied team patterns and annotate quota tests
* fix(proxy): match a team model's called model through JSON provider prefixes too
* fix(proxy): keep model updates that keep their route as on main
A PATCH or POST model update now checks only the limits its new route adds over the stored one, so removing or zeroing a limit on an unchanged route answers 200 again, under true and under a policy
* test(nvidia_riva): audit #41021
Adds an owned Riva ASR gRPC server and provider integration cells for the riva_offline request-body ban, the offline and streaming paths, the per-deployment caller opt-in, and the 413 and 408 error mappings
* test(nvidia_riva): check word timing requests and audio samples in the Riva double
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator
* test(streaming): cover reasoning text and error frames that mention [DONE]
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(fallback-generalizations): remove the legacy extends and mixed-rule shim
Main's remote cost map, the bundled backup, and the root map all ship the split
routing/capability rule format, so the shim for the old format is dead code. A rule
mixing litellm_provider with other model_info keys is now warned about and skipped.
* test(fallback-generalizations): fail when a shipped cost map rule would be skipped at install
* test(fallback-generalizations): cover ignored extends and annotate the shipped-rules test
* fix(fallback-generalizations): warn about old-format rules with a pointer to a current cost map and skip extends rules
* fix(fallback-generalizations): tell local cost map users to replace the bundled backup in the old-format warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(router): cover silent_model shadow fan-out and streaming end to end
Proxy cells check that each silent_model target gets one request in the
caller's stream mode, that the caller sees only the primary, and that every
shadow lands its own spend row under its own model group. SDK cells check the
sync Router.completion thread path streams each shadow and fires its success
callback.
* test(router): flatten the spend row lookup with itertools.chain
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #47358
Add integration cells for pass-through bodies and settings-endpoint mappings across workers
* test(proxy): wait for every worker to hold the saved mapping before probing
* test(proxy): compare passthrough bodies to the client body and probe each worker on its own connection
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Pin the queue route contracts #47498 changed: the priority header on streamed answers, the 400 for a request with no priority to queue at, a provider bad request keeping its own status, and a deployment-id request served ahead of the queue
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): avoid assistant prefill in Responses fallbacks
* fix(router): keep prefill for Claude fallbacks that take it and skip constrained continuations
A Responses fallback to a Claude model that takes assistant prefill now gets
the partial response as prefill again, so it continues mid-word without the
prompt-shape drift. Every other fallback keeps the request to continue.
A stream cut after text no longer continues when the request constrains its
output (a json_schema or json_object text format, or a forced tool_choice),
since a continued JSON document or tool call would not parse. The client gets
the primary's error, as on the chat path.
The fallback's re-sent leading space at the seam is dropped, and the prefill
shaping runs per deployment after the encrypted reasoning strip, so stripped
reasoning is never replayed.
* fix(router): guard SDK text_format, narrow forced tool_choice, read Claude base_model, and cap Responses continuation output
Cut Responses streams that asked for structured output through the SDK's text_format now surface the primary's
error instead of stitching two models' JSON. Only a required or named tool_choice counts as forced, so allowed_tools
in auto mode and hosted tool choices still continue. A Claude deployment named only through base_model keeps its
prefill continuation, and the continuation's max_output_tokens is lowered by what the cut stream already emitted.
* fix(router): quote the partial's ending in the continue request and surface the primary's error after a cut tool call
* fix(router): stop Responses fallbacks from continuing content-filter cuts and client-run tool calls
A content filter that cuts a /v1/responses stream mid-answer now returns its error, as on chat, instead of letting a content_policy_fallbacks backup finish the filtered text. Shell, apply_patch, computer, local_shell, and MCP approval items count as a started tool call. Chat continuation goes back to treating any tool_choice object as constraining, as on main
* fix(responses): count a stream of tool arguments as a started tool call
* fix(router): let Responses fallbacks continue under allowed_tools auto and after server-run tool calls
A Responses tool_choice of allowed_tools in auto mode no longer blocks the continuation, since the model may still answer in text. Hosted MCP calls and the bridge's server-run tool search no longer count as a started client tool call.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Covers the json_schema envelope edge cases #47478 fixed on the Anthropic json tool path: a key listed only in
`required`, a tool-named wrapper on a stream, non-ASCII and lone-surrogate answers, a stream cut before the tool
block closes (proxy and sync SDK, Anthropic and Bedrock Invoke), and a string `output_format` with and without
`drop_params` on Anthropic and Vertex chat and /v1/messages. The scripted Anthropic peer now refuses a non-object
`output_format` the way Anthropic does
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): let drop_params drop speed on claude-opus-4-7 after Anthropic removed its fast mode
* test(anthropic): cite the Claude Code fast-mode page for the Opus 4.7 removal date
* test(anthropic): cover the drop_params speed drop for claude-opus-4-7
* test(anthropic): type the model parameter of the speed drop test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(anthropic, bedrock): audit #47489 thinking replay across Claude routes and Bedrock Converse
* test(bedrock): compare the whole replayed Converse turn in the merged-message and cache point cells
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(files): audit #40621
Pin the raw cloud storage file id gate on /v1/files content and delete against a running proxy: twice-encoded gs:// ids stay blocked for non-admin keys, proxy admin keys read s3:// objects, and ordinary provider ids keep working
* test(files): check the full SigV4 signature on the admin raw s3 read
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): retry without strict when Bedrock rejects strict tool schemas under drop_params
* test(bedrock): cover the strict tools 400 retry under drop_params
* fix(bedrock): anchor the strict tools rejection match and strip toolSpec strict
* test(bedrock): return captured requests from the strict tools test helper
* fix(bedrock): retry without strict only when a tool sent strict to Bedrock
A strict-limit 400 on a request whose tools carry strict: false, or on a model litellm never sends strict to, was sent again with an identical body. The fallback now takes the model and returns nothing unless a tool puts strict on the wire. Also types the fallback test parameter and records the mock transport's requests on the recorder instead of a mutated list
* fix(bedrock): decide the strict retry on the profile's base model and retry grammar timeouts
An application inference profile with base_model sends strict on the wire, so the retry check now resolves the
capability model the way the request transformation does. Bedrock's "Grammar compilation timed out" 400 is a
strict-grammar rejection too, so it takes the same single retry under drop_params.
* test(bedrock): annotate the strict-tools retry test helper's values as Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #46815
Add integration tests for every count route #46815 changed, and count a Claude
subscription caller's tokens with their forwarded token when the deployment has
no configured key, which #46815 turned into a 401
* test(proxy): accept the forwarded subscription token in the count endpoint mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A DeepSeek V4 stream that ends without a usage block now has a spend-row cell proving its prompt_tokens are counted with DeepSeek's tokenizer, the one path #47353's own cells did not reach
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): apply weighted-failover exclusion before deployment-affinity callback
* fix(router): let affinity pins yield to deployments that already failed this request
Moving the weighted failover exclusion ahead of the affinity check also changed what tag, order, plugin and managed-file filters saw. Restore the original filter order and have the affinity check ignore session and API key pins that point at a deployment already excluded or retry-skipped in this request. previous_response_id pins still win.
* fix(router): release affinity pins on deployments that failed this request
Delete a session or API key pin whose deployment already failed the request, so
the deployment that serves it claims the pin. Prompt caching pins skip failed
deployments too, and the proxy strips the internal failed-id fields from client
bodies.
* fix(router): keep affinity pins in place and skip failed origins in encrypted content affinity
A pin now only yields for the request whose pinned deployment failed. Deleting it moved pins after one 429 or a request-caused 400 and added a Redis call on the routing path. Encrypted content affinity also ignores deployments that failed this request, so a failed origin takes the strip-and-dispatch path.
* fix(router): keep encrypted reasoning when a retry can only go back to its origin
The encrypted content check now drops failed deployments the way the router does: a weighted failover exclusion always holds, and a retry skip that would leave nothing falls back to the whole group. The proxy also drops client-sent metadata._failover_excluded_ids, and the encrypted content tests use a real Router
* fix(router): keep encrypted reasoning when previous_response_id binds the failed origin
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #47418
* test(proxy): hold the stream open past its first chunk and require cost headers in the audit cells
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): count an explicit allowed_fails equal to the default as set
Router(allowed_fails=3) stored the same value as an unset Router, so
_is_allowed_fails_set_on_router treated it as "not set" and kept the
rate-based cooldown logic. Keep the passed value in explicit_allowed_fails,
like _explicit_timeout, and count it as set.
* chore(router): drop the comment on explicit_allowed_fails
* fix(router): keep the fallback deployment out of cooldown when no deployment can be picked
When every deployment of the requested model is in cooldown, the router raises
"No deployments available" and schedules the failure callbacks on the request's
logging object. By the time they run, the fallback has already written its own
deployment into that object, so the 429 counted as a failure of the healthy
fallback deployment and cooled it down too, leaving the model group with nothing
to serve.
Tag the selection error and skip cooldown accounting for it, the same way
advisor orchestration failures are skipped.
* test(router): annotate the new cooldown test variables with Final
* test(router): pin that a saved allowed_fails of 3 stays unset and type the fallback helper
* test(router): use a fixed start time in the fallback helper
* test(router): cover a fallback group staying up after the primary's selection fails
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add aiand provider
* feat: expose aiand provider in admin UI
* feat(aiand): add ai& model pricing to the cost map
* feat(aiand): list ai& models for aiand/* wildcards and mark /v1/messages supported
* feat(aiand): send Anthropic /v1/messages to ai& natively
* fix(aiand): drop the retired max_tokens field and regenerate endpoint support
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds an integration cell proving a failed /v1/responses request counts its input from the newest compaction item on its failure spend row
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
gpt-5-nano and gpt-5-nano-2025-08-07 carried gpt-5's priority input price, 50x the nano standard rate. OpenAI sells no priority tier for nano and answers a nano priority request with service_tier default, so proxy spend on real calls was already billed at the standard rate. The wrong price showed in /model/info, in the tier pricing a custom-priced nano deployment inherited, and in SDK cost estimates like litellm.cost_per_token with service_tier priority. The key is removed, and a new test keeps every priority rate between 1x and 5x of its standard rate in both cost map files
* test(proxy): cover the max_parallel_requests slot a streamed /v1/messages MCP auto-execute keeps
A streamed /v1/messages request that auto-executes MCP tools replays its answer
as a stream after its model rounds already logged success. The stream path frees
the request's max_parallel_requests slot only on a client disconnect, so after
message_stop the slot stays taken and the key's next request answers 429
The test streams such a replay to the end and expects the slot to be free. It
fails until the fix lands
* fix(proxy): free the max_parallel_requests slot when a replayed /v1/messages MCP stream completes
A streamed /v1/messages request that auto-executes MCP tools runs its model
rounds without streaming and replays the final answer as a stream. The first
round claims the request's only success log, and its limiter callback keeps the
slot because the tool call reused it afterwards. The stream path released the
slot only on a client disconnect, so after message_stop the key stayed at its
max_parallel_requests limit and its next request answered 429
Stream cleanup now releases the stashed slot when the stream completed and its
logging object already claimed the success log, which only happens when the
stream replays a response that was logged before it started. A real stream
still releases through its own success callback
* test(mcp): free the key's only parallel slot after a streamed auto-execute ends
Run a streamed MCP auto-execute on a key with max_parallel_requests 1 on
every LLM surface, then send the next request on that key and expect 200.
On main both /v1/messages surfaces answer 429 here, while /v1/responses and
/v1/chat/completions already pass
* test(proxy): keep only the MCP stream regression tests, since #48628 already frees the slot at stream end
* test(proxy): require the MCP stream regression test's response to stream and reach its end event
* test(gemini): audit #47542
Proxy integration cells for the Gemini and Vertex default output budget: the env budget reaches Google on chat and responses, turns on long decode, yields to the caller's own limit, caps per deployment, skips resumes, and a camelCase two-candidate request is never continued
* test(gemini): annotate the budget wire module's pytestmark as Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Checks that a SageMaker stream failing after its first chunks reaches /v1/chat/completions, /v1/messages and /v1/responses callers as an error with no success end event, and that each writes a failed spend log row
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
