* fix(proxy): answer x-litellm-call-id on every token count response
The token count log line names a litellm_call_id, but count responses
never carried it, so a client that sent no id of its own could not find
its count's log line. Every count route (Anthropic Messages, Responses
input_tokens, Gemini countTokens, Bedrock count-tokens, and
/utils/token_counter) now answers the id on success and on errors, and
a crash answers the JSON 500 the Anthropic and Gemini routes already used.
* fix(proxy): keep the gateway's crash answers on count routes and stamp Responses opt-in bodies
A crash on /utils/token_counter or /v1/responses/input_tokens goes back to the
gateway's generic 500, or 503 when the database is down, now with the
x-litellm-call-id header. Responses count rejections carry error.litellm_call_id
when include_call_id_in_error_body is on
* test(proxy): count through count_request_tokens in the token counter loop tests
The token_counter route now also takes the response it sets x-litellm-call-id on, so these tests call the shared count function they exercise
* fix(proxy): keep a count crash's own exception and log pre-count Responses failures
The call id now reaches the crash handler through request.state instead of a wrapper exception, so logs and Sentry see the error the count raised. A Responses count that fails before counting writes its count line for any error, so its header names a line
* fix(proxy): keep the token counter's injected args out of its API docs
Tests the opt-in error body id on an Anthropic count the proxy rejects before counting
* fix(proxy): keep a count route's own HTTP error when adding the call id
The wrapper raised a new HTTPException with the same status, detail and headers, so a subclass lost its type and error tracking saw a new exception. It now adds the header to the original and raises it again
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): reset a deleted customer's admission spend counter so a re-created one is not refused
/customer/delete only dropped the committed twin, so spend:end_user:<id> kept the old spend and a
re-created customer with the same id stayed over budget. Zero the counter and its DB floor marker on
every worker after the delete commits, the same way a key spend reset does.
* fix(proxy): drop a deleted customer's spend counter before zeroing it
A Redis that rejects writes but still serves reads, such as one at its memory limit, kept the old
counter when the zeroing SET failed, so the re-created customer stayed refused. Delete the counter
first, which Redis still allows in that state, then write zero and the floor.
* test(proxy): type the customer delete spend counter test parameters
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* refactor(typing): clear 340 basedpyright Any errors (wave 1)
* refactor(typing): clear 497 basedpyright Any errors (wave 2)
* refactor(typing): clear 74 basedpyright Any errors (wave 3)
* refactor(typing): clear 143 basedpyright Any errors (wave 4)
* refactor(typing): clear 106 basedpyright Any errors (wave 5)
* fix(typing): keep router fence visibility and py310-safe Self import
Bind the Protocol-narrowed routers in pass_through_endpoints and rag
ingestion to router-named locals so router_surface_check still sees the
aadapter_completion and aocr uses, and import Self from typing_extensions
in prisma_client for check_py310_typing_imports.
* perf(core_helpers): fast-path dicts and lists in the checked JSON helpers
as_str_mapping now tries the concrete dict check before the Mapping ABC
check, and iter_items returns lists before the runtime-checkable Protocol
check. Both return the same object as before, and roughly halve the
overhead CodSpeed flagged on the A2A text-extraction benchmarks.
* refactor(realtime): keep the base TypeError for a non-mapping translation expires_after
* fix(proxy): restore the require_attributes import the main merge dropped
* test: cover the retyped realtime, vector store, agent, stream, video and provider response paths
* fix(realtime): name the public request types in non-mapping TypeErrors
keyword_arguments wraps expires_after and transcription_session so a non-mapping value raises the same TypeError text as before the typing pass. Tests narrow Greptile's pegasus case with isinstance and freeze its model.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(otel): let a team or key OTel v2 destination add its own request metadata keys
A team or key callback can now set baggage_metadata_keys in its callback_vars,
a comma-separated list of requester_metadata.<key> paths. The tenant fan-out
adds them as litellm.metadata.<key> to that destination's spans only, on top of
the global LITELLM_OTEL_BAGGAGE_METADATA_KEYS; the operator's exporter and other
destinations never see them. Values are bound when each span ends, an attribute
the span already carries wins, and names of proxy identity fields are rejected
when saved, along with lists over 20 keys.
Resolves LIT-9545
* fix(otel): refuse request headers in a destination's baggage_metadata_keys
The proxy copies the request headers into requester_metadata.headers with
credential headers masked, but a header a gateway in front of it added
(an IAP or OIDC assertion, an internal token) stays as sent. A team or key
admin listing requester_metadata.headers.<name> could export it to an
endpoint the team controls. Entries under requester_metadata.headers are
now rejected when saved, so exporting a header stays an operator decision
in LITELLM_OTEL_BAGGAGE_METADATA_KEYS. Also drops an unused import.
* fix(otel): hold destination metadata to the configured span limits
A destination's extra litellm.metadata.* attributes were added to a span
view built from a plain mapping, so they skipped the limits the SDK
applies in set_attribute: a 1 MiB caller value survived an operator's
OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT. Values are now cut to the length
limit, and keys past the attribute-count limit are dropped and counted
instead of evicting an attribute the span already carries.
The per-destination views are built in one Final tuple ahead of the
forwarding loop, since a Final cannot be assigned inside the loop.
* fix(otel): apply destination metadata after the destination view, within the provider's limits
The span limits were checked against the span as it ended, before the
destination view strips prompt content and backend-only attributes, so a
model-call span full of llm.input_messages.* could drop the team's keys on
a no_content destination that had room. The values are now bound when the
span ends and added as the last step before export, counted against what
that destination actually sends, so their dropped count also survives the
view. The limits come from the tracer provider the fan-out is attached to,
read from the live span at start, instead of a separate env-built SpanLimits.
* refactor(otel): consolidate span limits, destination validators and metadata key parsing
Reads a span's limits in one guarded place that the emitter and the fan-out
share, replaces the two parallel lists of per-destination callback-var
validators with one mapping, splits and strips a baggage_metadata_keys list
once, drops a filter the post-view merge already enforces, and folds the
repeated fan-out test setup into one helper. No behavior change.
* ci: retrigger checks stranded by the GitHub Actions outage
* fix(proxy): stream the guardrail block response when the block lands after an SSE keepalive ping
* docs(proxy): keep the keepalive docstring's reasons together
* fix(proxy): stream the /v1/messages block after a keepalive ping and always end a late block stream
A passthrough guardrail block on a streamed /v1/messages request that landed
after the first keepalive ping still ended as an api_error event. The route now
hands its block stream to the keepalive relay, as chat and text completions do.
If a route's block stream itself fails after a ping, the relay sends the usual
late failure frame instead of raising, so the stream always ends.
The passthrough tests now drive the real routes with a real guardrail.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(completion): answer 400 for a chat message without a role
A message with no role was filled with role assistant and sent upstream, so the caller's text reached the
model as its own earlier reply and the request returned 200. Now a missing, null, or empty role answers 400
with OpenAI's wording, naming the message index. A role-less turn that carries tool_calls is still sent as
assistant, since only an assistant writes tool calls.
* fix(completion): treat a role-less message as the caller's error
The 400 now raises MalformedRequestFieldError with OpenAI's param and code and the provider name, so it no longer counts toward the deployment's failure rate, cools it down, or fires an llm_exceptions alert
* fix(completion): keep the assistant fill for a role-less function_call turn
* refactor(completion): type the role-less call turn check and the request kwargs
Read role, tool_calls and function_call off the cleaned message copy, and validate completion's kwargs into a typed mapping before naming the provider on the 400
* fix(completion): keep the alert and failure-count exclusions to the role error
Raise MissingMessageRoleError, a MalformedRequestFieldError subclass, so a malformed tools 400 alerts and counts as on main, and fill assistant whenever a role-less turn has a tool_calls or function_call key
* fix(completion): keep assistant-only role-less turns and skip SDK and PagerDuty alerts
A role-less turn carrying any assistant-only key, null values included, is still
filled as assistant, so a dumped assistant message or a reasoning turn keeps working.
The Router's alerting_config alert and the PagerDuty failure count skip the role error,
like the proxy alert already does.
* fix(utils): drop the string code from the missing-role 400 so the Errors tab counts it as 400
The spend log stored code missing_required_parameter as the error code, which
the Usage page Errors tab cannot parse as a status, so every role-less request
landed under no status. The malformed tools 400 leaves code out the same way.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The team-model and team-key transactions in /team/delete used Prisma's 2s
default max_wait, so a busy writer pool failed the delete with P2028 while
every plain query in the same request waited up to the 60s pool timeout. Both
now get the 30s team-delete budget for max_wait, and the key transaction gets
it for timeout too.
* test(integration): run baseline accounting store tests in a private schema
The accounting workers share their database with two gateway proxies, and every
spend-log tick on those proxies projects any dirty baseline comparison it finds.
When one of them published the test's scope between its last append and its own
project() call, the test saw "unchanged" instead of "published".
The module fixture now copies the twelve tables the store touches into a fresh
schema and points DATABASE_URL at it for the module, so the gateway proxies,
which only read public, can no longer publish these rows.
* test(integration): wait for H9's own prometheus increments before it returns
H9 stopped as soon as both workers' sample files already counted `other`, so
its own chat's increment could still be in flight and land inside the next
cell's window, which then saw two requests instead of one.
H9 now takes a before scrape, counts the chats it sends, and finishes with the
same _overflowed wait every other TestCapped cell uses.
* feat(gemini): file-backed batch jobs on /v1/batches with output download
Adds GeminiBatchesConfig (create + retrieve) behind custom_llm_provider=gemini,
registers it in ProviderConfigManager, and implements file content retrieval on
GoogleAIStudioFilesHandler so GET /v1/files/{output_file_id}/content returns the
batch output. Gemini runs the OpenAI-shaped JSONL the proxy uploads today unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): type the file content request params so no pyright rule grows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(gemini): price completed batch output rows so spend lands on the key
Gemini writes the usage of an OpenAI-shaped output row as camelCase
(promptTokens, completionTokens, totalTokens) with thinking tokens only in
totalTokens. Map it onto Usage with the thinking tokens folded into
completion_tokens, so the shared batch cost path bills them at the gemini
_batches rates.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): parse nanosecond file timestamps
Gemini stamps files with nine fractional digits, which strptime's %f
rejects, so every retrieve of a batch output file logged a ValueError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): parse file timestamps on Python 3.10 too
fromisoformat on 3.10 takes neither a Z suffix nor more than six fractional digits.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): price native batch output rows, stream output downloads, reject unsupported batch metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): detect native batch rows by key, anchor the streaming file download handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(httpx): make anchor_handler_to public so the file content stream can use it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): supervise the opt-in Rust forwarding server
With LITELLM_RUST on, `litellm` and `python -m gateway.launch` start a
supervisor that runs the Rust gateway on the public port and Python on a
private loopback listener. Python still serves every route and sees each
request as if the client had connected to it directly.
Rust sends the client's TCP peer in `x-litellm-rust-peer`. The supervised
server turns server-level proxy headers off, and a middleware restores
the peer and then applies the operator's own trust, so client IPs,
trusted proxies and every IP-based setting behave the same with Rust on
or off.
The supervisor is transparent to Python. Signals pass through, Python's
exit status is the result, and both children shut down when the
supervisor's end of their stdin closes, so nothing outlives it. Rust
starts only once Python listens. When forwarding cannot start (no
gateway binary, TLS, a Unix socket, Hypercorn or Granian), the entry
point warns and serves from Python alone.
Every forwarding decision lives in litellm/proxy/rust_forward, paired
with the gateway-forward crate. OS differences sit behind a Platform
protocol, with Windows supported for local CLI runs. AGENTS.md files in
the touched folders state the boundary. The `--rust_gateway` flag is
gone in favor of the environment variable.
* refactor(proxy): switch Rust forwarding on with LITELLM_RUST_FIRST
Forwarding gets its own switch, independent of LITELLM_RUST, which keeps
selecting only the Rust paths inside the SDK. Neither turns the other on.
---------
Co-authored-by: narmstrong-berri <narmstrong@berri.ai>
Extract serving_port() and configure_database() as they were inline, turn the
LITELLM_RUST parser into a reusable RustSwitch type, and reformat
tests/unit/proxy/test_proxy_cli.py to the 120 column limit.
run_gunicorn_server takes private_listener, an inherited socket or an assigned
loopback port. With one, gunicorn binds it instead of host:port and trusts no
peer's forwarding headers, since every peer on it is the local process in front.
Both are gunicorn's own settings, bind and forwarded_allow_ips, so the uvicorn
worker is unchanged. The reload options take factory. No caller passes either
yet, so every entry point serves as before.
* feat(proxy): count a rate-limit fallback in x-litellm-attempted-fallbacks and name it in x-litellm-fallback-reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count every fallback candidate a local rate limit rejected in x-litellm-attempted-fallbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count fallback attempts without a mutable list and keep self.data on the served request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tls): exclude RSA key exchange from outbound cipher list in FIPS mode
* fix(tls): apply the FIPS cipher filter to SSL contexts clients already hold
enforce_fips_mode runs after load_config, so guardrail and logger clients
built during config load held an SSL context without the RSA key exchange
exclusion. enforce_fips_mode now updates every cached context in place, and
the SDK no longer imports the proxy's FIPS module.
* fix(tls): refuse to start in FIPS mode when SSL_SECURITY_LEVEL has only RSA key exchange
Excluding RSA key exchange from an SSL_SECURITY_LEVEL that lists only RSA
key-exchange suites leaves no TLS 1.2 suite, so set_ciphers raised on every
client build. enforce_fips_mode now checks the configured list first and
refuses to start with a FipsModeError that names the setting. The cipher
list test no longer assumes the host has RSA key-exchange suites.
* fix(tls): offer only TLS 1.3 instead of refusing to start when no TLS 1.2 suite without RSA key exchange is left
* chore(tls): warn when fips_mode leaves a cipher list with only TLS 1.3
* style(tls): split the TLS 1.3 fallback warning to fit the line limit
* fix(proxy): exclude RSA key exchange before config-load secret reads in FIPS mode
get_config reads secrets (AWS, GCP, HashiCorp, CyberArk secret managers) over the shared
outbound SSL context before enforce_fips_mode runs, so with general_settings.fips_mode true
those first TLS handshakes still offered RSA key exchange. Turn the exclusion on in get_config
from the raw general_settings, right before the secret manager starts, reusing the existing
fips_mode_setting (now public) and exclude_rsa_key_exchange. enforce_fips_mode still owns the
refusal and the final state; fips_mode given as an os.environ/ reference is still handled there.
* test(proxy): skip the config-load RSA key exchange test on an OpenSSL without RSA suites
On an OpenSSL whose default cipher list offers no RSA key exchange suite, fips and standard mode read secrets the same way, so the test has nothing to tell apart and skips, as the http_handler fallback test already does.
* fix(tls): exclude TLS 1.2 ChaCha20 suites from the outbound cipher list in FIPS mode
ChaCha20-Poly1305 is not a FIPS approved algorithm. FIPS providers already refuse it, so on the Chainguard
image this changes nothing, but fips_mode now drops it from TLS 1.2 explicitly next to RSA key exchange.
Python's set_ciphers cannot touch TLS 1.3 suites, which still come from OpenSSL's config.
exclude_rsa_key_exchange is renamed to exclude_rsa_key_exchange_and_chacha20 to match.
* fix(proxy): add tzdata to the proxy extra so a legacy TZ doesn't stop startup
* style(proxy): drop the inline comment on the tzdata requirement
* test(proxy): check tzdata is installed with the proxy extra on every platform
* fix(proxy): set the first spend report run in UTC so a scheduler zone that differs from the system clock can't skip it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): track self-heal for test_with_id_security_off_reads_and_cancels_add_no_rows_and_no_replayed_turns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): preserve response turn rows when id security is disabled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): list ids that route back for a prefixed partial alias over a catch-all deployment
* test(proxy): call the router registry in the catch-all alias listing test
* fix(proxy): keep main's listing for whole-name aliases over a catch-all
Only partial aliases like team1/gpt-* over a bare * run the per-id route check. A whole-name alias like team1/* keeps the string-only listing it had, so it lists the same ids at the same cost. Adds a test that a more specific sibling alias's ids are left out of the partial alias's list.
* fix(proxy): list catch-all aliases fast and skip ids a sibling alias takes
* fix(proxy): check team wildcard aliases without a router call per id and skip suffix ids that name another catalog model
Team keys now read the team's own wildcard patterns the way the router does (any team pattern first, then the provider form when its prefix can be a provider), so ids no rival can take skip the router call. A suffix alias such as team1/*-mini no longer lists team1/gpt-5.4-mini for gpt-5.4.
* fix(proxy): keep team key listings fast when aliases overlap and honor a pinned provider on partial aliases
A team key whose team defines the same alias as the global config, or a team
pattern that can take a global alias's ids, now checks only the team's own
patterns before calling the router, so the listing no longer pays a router
call per catalog id. The router also tries the provider-prefixed form of an id
only when the bare form matches nothing.
A partial alias over `*` with `custom_llm_provider` set now lists only that
provider's catalog, so `team1/gpt-*` pinned to OpenAI no longer lists
`team1/gpt-claude-*` ids that answer 404.
* fix(proxy): ask a pinned catch-all's provider for its models once per listing
A partial alias over a `*` deployment pinned with custom_llm_provider fetched the whole `*` catalog before taking the pinned branch, so with check_provider_endpoint on each listing called the provider's /v1/models one extra time. The pinned branch now runs before that fetch. Tests also cover a provider-less team alias that takes ids under their provider and a wildcard model_group_alias
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
helm/litellm becomes the only chart. Every container runs the one shared image with args only, monolith.enabled renders a single proxy Deployment, the retired chart's KEDA, extraResources, generated master key, helm test, minReadySeconds and rollout strategy settings are ported, and the bundled PostgreSQL and Redis subcharts are removed
Resolves LIT-8880
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: speed up image scans with an opt-in Rust profile
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: separate cached Rust wheels by maturin profile
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp)!: forward the LiteLLM key when the admin configures it
Restores forwarding of the caller's LiteLLM key through explicit channels
(Extra Headers listing X-LiteLLM-API-Key or Authorization, x-mcp-{alias}-authorization
and x-mcp-auth) that #45401 and #46932 started stripping. Implicit Authorization
forwarding stays scrubbed for true_passthrough and unlisted servers, gateway
session tokens and bridge envelopes stay scrubbed, and guardrail and JWT signer
bearer reads now skip the admission key
* test(mcp): serve signer introspection from an in-process transport
* fix(mcp): copy oauth2 headers so a listed key never reaches a later server
* test(mcp): drive the shared-header regression through the real MCP client
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): merge a key's cache controls into the request's cache settings instead of replacing them
* fix(proxy): keep a key with no-cache out of opt-in caching
A key whose cache metadata sets no-cache now drops the caller's use-cache, so on a default_off proxy its responses stay out of the shared cache as before the merge change. default_on proxies ignore use-cache, so they are unchanged
* fix(proxy): keep base handling of non-dict key cache metadata
Key cache metadata that is not a dict, such as false or a string, wipes the request's cache settings as before the merge change, so those keys stay out of opt-in caching
* fix(proxy): keep a key with use-cache false out of opt-in caching
* fix(proxy): only keep request cache settings on an allowed opt-in
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(reasoning): log the GPT-5 reasoning_effort drop warning only when the provider gets no effort
* fix(proxy): count an extra_body effort as sent and pass the drop holder through ChatGPT and Copilot
* fix(proxy): pass the drop holder to get_optional_params under the internal _litellm_ prefix
* fix(openai): keep the reasoning effort drop warning when extra_body sends another level
The held warning now stays quiet only when the request sends the level the gate dropped,
so a top-level xhigh dropped while extra_body sends none still logs. The Copilot test case
writes the access-token file the current authenticator reads.
* fix(openai): keep the reasoning effort drop warning when a request fails before reaching the provider
The drop warning holder now lives for the whole completion call, and
completion's exception handler prints whatever it still holds, since a
request that fails after mapping never reached the provider. Each flush
empties the holder, so a provider error after dispatch does not print the
warning a second time
* fix(openai): warn when extra_body's reasoning_effort carries no effort level on the Responses bridge
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): bill gemini image text, thinking and priority tokens at their own rates
Co-authored-by: kerry <kerry@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(images): bill unitemized output as text when no images return
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): price image calls only by the tier the provider served
---------
Co-authored-by: kenneth <kenneth@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the upgrade notice off the request path and count area-labelled release notes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): regenerate dashboard API types and read the release cache through one typed helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): respond 502 from /get/latest_release_info when no release note entry is countable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): respond 502 when the release fetch failed and announce a release with no countable notes without counts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): re-ask for the latest release while the proxy is still fetching it and keep fetch diagnostics out of the 502 body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep the known release on screen while the proxy refetches it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer 202 while the release is being fetched so the dashboard polls only then
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): name the release query options to stay inside the lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): return the 202 as a response value instead of mutating the injected response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): fetch the release on the background client so its 502 never reaches the error handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): settle a fallback's tokens on the model group that served it
The v3 limiter pinned post-call TPM settlement to the model reserved at
admission, so when the router fell back to another model group the
requested model's counter took the usage and the serving model's counter
stayed empty. When the served model group differs from the admitted one,
charge the served group and refund the abandoned group's per-model
reservation. Same-model retries and alias reloads keep the admission target.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): refund an abandoned fallback reservation only on counters the served model does not settle
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rate-limit): wait for the primary refund and the fallback charge before reading counters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): only treat a limiter-admitted fallback target as the served model at settlement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rate-limit): drop unrelated formatting from the v3 limiter unit test file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rate-limit): remove helper copies duplicated while appending the fallback settlement tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order
A purpose=batch upload on a mixed Azure chat and batch group was order-filtered before batch routing narrowed it, so the file landed only on the order-1 chat deployment and the batch create that followed got Azure's 400 invalid_deployment_type. The upload now runs under the same batch routing as the batch create.
* fix(router): keep a failed batch upload's cooldown and order fallback on the batch deployments
* fix(router): keep a batch upload off chat deployments while its batch deployment cools down
* fix(router): keep a refused batch upload's retry off the chat deployment
* fix(router): keep a refused batch upload's error and skip chat while the batch deployment is paused
* test(router): pin the order fallback for a batch create on a group with no batch deployment
* test(integration): expect an Azure batch upload on the batch deployment alone
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>