The API Reference page was already behind a deprecation banner. This deletes
the /ui/api-reference route and its components, the Developer Tools sidebar
entry and page description, the legacy ?page=api_ref redirects, and the i18n
strings only that page used.
Old ?page=api_ref and ?page=api-reference bookmarks now fall through to the
dashboard like any other unknown page id, since there is no replacement page.
Tests that listed the page are updated: the sidebar and legacy-route vitest
files, the uiHref examples, the e2e sidebar fixtures and Internal Viewer nav
check, and the UI path detection unit test.
* fix(proxy): name the deployment in stream error alerts on every provider and route
The llm_exceptions alert for a cut stream on Anthropic chat, /v1/completions,
native /v1/responses and native /v1/messages had no API Base, model_group or
deployment line. The shared exception mapper now attaches its debug info at its
single exit, mid-stream fallback errors inherit it from the error they wrap,
and the native relays add it to the same error object they re-raise.
* fix(proxy): keep the prompt out of attached alert debug info and name the deployment on more stream exits
* fix(proxy): name the deployment on stream errors that arrive already mapped
exception_type returned errors that were already litellm types before it attached the debug info, so a finish_reason=abort chunk or the max streaming duration timeout still sent an alert with no deployment. It now builds the info first and attaches the prompt-free copy on that early return too, and native Responses streams attach it on their generic error exit
Also adds tests for the dropped `Messages: None` line and the mapping-crash exit
* fix(proxy): read the router's deployment from litellm_metadata on Messages and Responses errors
/v1/messages and /v1/responses keep the router's model_group, deployment and api_base in litellm_metadata, so their errors before the stream named only the model in alerts. The debug info now reads litellm_metadata first, falls back to metadata, and takes API Base from it when the call passes none
get_first_chars_messages returns an empty string again for input that is not a mapping, as it did before this branch
* fix(proxy): merge both metadata buckets when error alerts name the deployment
A pre_call guardrail adds litellm_metadata to a chat request while the router keeps model_group and deployment in metadata, and generic routes such as moderations keep the key alias and team in metadata and the deployment in litellm_metadata. Reading only one bucket dropped lines the merge base showed. The debug info now merges both, and litellm_metadata wins when both carry the same key, so a caller's own metadata cannot replace the router's values
* fix(proxy): keep a reused error's alert details on the request that raised it
An exception instance raised for more than one request, such as an SDK mock_response exception or a handler that raises one shared error, kept the first request's Key Name, model_group and deployment in every later alert
The attached details now record the call id that set them. A later call with a different id replaces them, and a layer that re-maps the error without the request keeps them
* fix(proxy): drop alert details from an error object raised by more than one request
One exception object raised by two concurrent requests (a shared mock_response error, or a
handler that raises one error object for every request) let a request's alert show the other
request's Key Name and deployment, since the later attach replaced the earlier one. An error
object tagged by one call id and attached again under another now drops its details for good,
so no alert names the wrong request. One request re-mapping its own error, with or without a
call id, keeps the lines it already has
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(rate-limit): name the limit that rejected a request in a rate_limited_by header on the 429
* test(rate-limit): expect rate_limited_by in the exact-headers assertion
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer x-litellm-call-id on every token count response
The token count log line names a litellm_call_id, but count responses
never carried it, so a client that sent no id of its own could not find
its count's log line. Every count route (Anthropic Messages, Responses
input_tokens, Gemini countTokens, Bedrock count-tokens, and
/utils/token_counter) now answers the id on success and on errors, and
a crash answers the JSON 500 the Anthropic and Gemini routes already used.
* fix(proxy): keep the gateway's crash answers on count routes and stamp Responses opt-in bodies
A crash on /utils/token_counter or /v1/responses/input_tokens goes back to the
gateway's generic 500, or 503 when the database is down, now with the
x-litellm-call-id header. Responses count rejections carry error.litellm_call_id
when include_call_id_in_error_body is on
* test(proxy): count through count_request_tokens in the token counter loop tests
The token_counter route now also takes the response it sets x-litellm-call-id on, so these tests call the shared count function they exercise
* fix(proxy): keep a count crash's own exception and log pre-count Responses failures
The call id now reaches the crash handler through request.state instead of a wrapper exception, so logs and Sentry see the error the count raised. A Responses count that fails before counting writes its count line for any error, so its header names a line
* fix(proxy): keep the token counter's injected args out of its API docs
Tests the opt-in error body id on an Anthropic count the proxy rejects before counting
* fix(proxy): keep a count route's own HTTP error when adding the call id
The wrapper raised a new HTTPException with the same status, detail and headers, so a subclass lost its type and error tracking saw a new exception. It now adds the header to the original and raises it again
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): reset a deleted customer's admission spend counter so a re-created one is not refused
/customer/delete only dropped the committed twin, so spend:end_user:<id> kept the old spend and a
re-created customer with the same id stayed over budget. Zero the counter and its DB floor marker on
every worker after the delete commits, the same way a key spend reset does.
* fix(proxy): drop a deleted customer's spend counter before zeroing it
A Redis that rejects writes but still serves reads, such as one at its memory limit, kept the old
counter when the zeroing SET failed, so the re-created customer stayed refused. Delete the counter
first, which Redis still allows in that state, then write zero and the floor.
* test(proxy): type the customer delete spend counter test parameters
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* refactor(typing): clear 340 basedpyright Any errors (wave 1)
* refactor(typing): clear 497 basedpyright Any errors (wave 2)
* refactor(typing): clear 74 basedpyright Any errors (wave 3)
* refactor(typing): clear 143 basedpyright Any errors (wave 4)
* refactor(typing): clear 106 basedpyright Any errors (wave 5)
* fix(typing): keep router fence visibility and py310-safe Self import
Bind the Protocol-narrowed routers in pass_through_endpoints and rag
ingestion to router-named locals so router_surface_check still sees the
aadapter_completion and aocr uses, and import Self from typing_extensions
in prisma_client for check_py310_typing_imports.
* perf(core_helpers): fast-path dicts and lists in the checked JSON helpers
as_str_mapping now tries the concrete dict check before the Mapping ABC
check, and iter_items returns lists before the runtime-checkable Protocol
check. Both return the same object as before, and roughly halve the
overhead CodSpeed flagged on the A2A text-extraction benchmarks.
* refactor(realtime): keep the base TypeError for a non-mapping translation expires_after
* fix(proxy): restore the require_attributes import the main merge dropped
* test: cover the retyped realtime, vector store, agent, stream, video and provider response paths
* fix(realtime): name the public request types in non-mapping TypeErrors
keyword_arguments wraps expires_after and transcription_session so a non-mapping value raises the same TypeError text as before the typing pass. Tests narrow Greptile's pegasus case with isinstance and freeze its model.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(otel): let a team or key OTel v2 destination add its own request metadata keys
A team or key callback can now set baggage_metadata_keys in its callback_vars,
a comma-separated list of requester_metadata.<key> paths. The tenant fan-out
adds them as litellm.metadata.<key> to that destination's spans only, on top of
the global LITELLM_OTEL_BAGGAGE_METADATA_KEYS; the operator's exporter and other
destinations never see them. Values are bound when each span ends, an attribute
the span already carries wins, and names of proxy identity fields are rejected
when saved, along with lists over 20 keys.
Resolves LIT-9545
* fix(otel): refuse request headers in a destination's baggage_metadata_keys
The proxy copies the request headers into requester_metadata.headers with
credential headers masked, but a header a gateway in front of it added
(an IAP or OIDC assertion, an internal token) stays as sent. A team or key
admin listing requester_metadata.headers.<name> could export it to an
endpoint the team controls. Entries under requester_metadata.headers are
now rejected when saved, so exporting a header stays an operator decision
in LITELLM_OTEL_BAGGAGE_METADATA_KEYS. Also drops an unused import.
* fix(otel): hold destination metadata to the configured span limits
A destination's extra litellm.metadata.* attributes were added to a span
view built from a plain mapping, so they skipped the limits the SDK
applies in set_attribute: a 1 MiB caller value survived an operator's
OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT. Values are now cut to the length
limit, and keys past the attribute-count limit are dropped and counted
instead of evicting an attribute the span already carries.
The per-destination views are built in one Final tuple ahead of the
forwarding loop, since a Final cannot be assigned inside the loop.
* fix(otel): apply destination metadata after the destination view, within the provider's limits
The span limits were checked against the span as it ended, before the
destination view strips prompt content and backend-only attributes, so a
model-call span full of llm.input_messages.* could drop the team's keys on
a no_content destination that had room. The values are now bound when the
span ends and added as the last step before export, counted against what
that destination actually sends, so their dropped count also survives the
view. The limits come from the tracer provider the fan-out is attached to,
read from the live span at start, instead of a separate env-built SpanLimits.
* refactor(otel): consolidate span limits, destination validators and metadata key parsing
Reads a span's limits in one guarded place that the emitter and the fan-out
share, replaces the two parallel lists of per-destination callback-var
validators with one mapping, splits and strips a baggage_metadata_keys list
once, drops a filter the post-view merge already enforces, and folds the
repeated fan-out test setup into one helper. No behavior change.
* ci: retrigger checks stranded by the GitHub Actions outage
* fix(proxy): stream the guardrail block response when the block lands after an SSE keepalive ping
* docs(proxy): keep the keepalive docstring's reasons together
* fix(proxy): stream the /v1/messages block after a keepalive ping and always end a late block stream
A passthrough guardrail block on a streamed /v1/messages request that landed
after the first keepalive ping still ended as an api_error event. The route now
hands its block stream to the keepalive relay, as chat and text completions do.
If a route's block stream itself fails after a ping, the relay sends the usual
late failure frame instead of raising, so the stream always ends.
The passthrough tests now drive the real routes with a real guardrail.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(completion): answer 400 for a chat message without a role
A message with no role was filled with role assistant and sent upstream, so the caller's text reached the
model as its own earlier reply and the request returned 200. Now a missing, null, or empty role answers 400
with OpenAI's wording, naming the message index. A role-less turn that carries tool_calls is still sent as
assistant, since only an assistant writes tool calls.
* fix(completion): treat a role-less message as the caller's error
The 400 now raises MalformedRequestFieldError with OpenAI's param and code and the provider name, so it no longer counts toward the deployment's failure rate, cools it down, or fires an llm_exceptions alert
* fix(completion): keep the assistant fill for a role-less function_call turn
* refactor(completion): type the role-less call turn check and the request kwargs
Read role, tool_calls and function_call off the cleaned message copy, and validate completion's kwargs into a typed mapping before naming the provider on the 400
* fix(completion): keep the alert and failure-count exclusions to the role error
Raise MissingMessageRoleError, a MalformedRequestFieldError subclass, so a malformed tools 400 alerts and counts as on main, and fill assistant whenever a role-less turn has a tool_calls or function_call key
* fix(completion): keep assistant-only role-less turns and skip SDK and PagerDuty alerts
A role-less turn carrying any assistant-only key, null values included, is still
filled as assistant, so a dumped assistant message or a reasoning turn keeps working.
The Router's alerting_config alert and the PagerDuty failure count skip the role error,
like the proxy alert already does.
* fix(utils): drop the string code from the missing-role 400 so the Errors tab counts it as 400
The spend log stored code missing_required_parameter as the error code, which
the Usage page Errors tab cannot parse as a status, so every role-less request
landed under no status. The malformed tools 400 leaves code out the same way.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(gemini): file-backed batch jobs on /v1/batches with output download
Adds GeminiBatchesConfig (create + retrieve) behind custom_llm_provider=gemini,
registers it in ProviderConfigManager, and implements file content retrieval on
GoogleAIStudioFilesHandler so GET /v1/files/{output_file_id}/content returns the
batch output. Gemini runs the OpenAI-shaped JSONL the proxy uploads today unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): type the file content request params so no pyright rule grows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(gemini): price completed batch output rows so spend lands on the key
Gemini writes the usage of an OpenAI-shaped output row as camelCase
(promptTokens, completionTokens, totalTokens) with thinking tokens only in
totalTokens. Map it onto Usage with the thinking tokens folded into
completion_tokens, so the shared batch cost path bills them at the gemini
_batches rates.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): parse nanosecond file timestamps
Gemini stamps files with nine fractional digits, which strptime's %f
rejects, so every retrieve of a batch output file logged a ValueError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): parse file timestamps on Python 3.10 too
fromisoformat on 3.10 takes neither a Z suffix nor more than six fractional digits.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): price native batch output rows, stream output downloads, reject unsupported batch metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): detect native batch rows by key, anchor the streaming file download handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(httpx): make anchor_handler_to public so the file content stream can use it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): supervise the opt-in Rust forwarding server
With LITELLM_RUST on, `litellm` and `python -m gateway.launch` start a
supervisor that runs the Rust gateway on the public port and Python on a
private loopback listener. Python still serves every route and sees each
request as if the client had connected to it directly.
Rust sends the client's TCP peer in `x-litellm-rust-peer`. The supervised
server turns server-level proxy headers off, and a middleware restores
the peer and then applies the operator's own trust, so client IPs,
trusted proxies and every IP-based setting behave the same with Rust on
or off.
The supervisor is transparent to Python. Signals pass through, Python's
exit status is the result, and both children shut down when the
supervisor's end of their stdin closes, so nothing outlives it. Rust
starts only once Python listens. When forwarding cannot start (no
gateway binary, TLS, a Unix socket, Hypercorn or Granian), the entry
point warns and serves from Python alone.
Every forwarding decision lives in litellm/proxy/rust_forward, paired
with the gateway-forward crate. OS differences sit behind a Platform
protocol, with Windows supported for local CLI runs. AGENTS.md files in
the touched folders state the boundary. The `--rust_gateway` flag is
gone in favor of the environment variable.
* refactor(proxy): switch Rust forwarding on with LITELLM_RUST_FIRST
Forwarding gets its own switch, independent of LITELLM_RUST, which keeps
selecting only the Rust paths inside the SDK. Neither turns the other on.
---------
Co-authored-by: narmstrong-berri <narmstrong@berri.ai>
Extract serving_port() and configure_database() as they were inline, turn the
LITELLM_RUST parser into a reusable RustSwitch type, and reformat
tests/unit/proxy/test_proxy_cli.py to the 120 column limit.
run_gunicorn_server takes private_listener, an inherited socket or an assigned
loopback port. With one, gunicorn binds it instead of host:port and trusts no
peer's forwarding headers, since every peer on it is the local process in front.
Both are gunicorn's own settings, bind and forwarded_allow_ips, so the uvicorn
worker is unchanged. The reload options take factory. No caller passes either
yet, so every entry point serves as before.
* feat(proxy): count a rate-limit fallback in x-litellm-attempted-fallbacks and name it in x-litellm-fallback-reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count every fallback candidate a local rate limit rejected in x-litellm-attempted-fallbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count fallback attempts without a mutable list and keep self.data on the served request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tls): exclude RSA key exchange from outbound cipher list in FIPS mode
* fix(tls): apply the FIPS cipher filter to SSL contexts clients already hold
enforce_fips_mode runs after load_config, so guardrail and logger clients
built during config load held an SSL context without the RSA key exchange
exclusion. enforce_fips_mode now updates every cached context in place, and
the SDK no longer imports the proxy's FIPS module.
* fix(tls): refuse to start in FIPS mode when SSL_SECURITY_LEVEL has only RSA key exchange
Excluding RSA key exchange from an SSL_SECURITY_LEVEL that lists only RSA
key-exchange suites leaves no TLS 1.2 suite, so set_ciphers raised on every
client build. enforce_fips_mode now checks the configured list first and
refuses to start with a FipsModeError that names the setting. The cipher
list test no longer assumes the host has RSA key-exchange suites.
* fix(tls): offer only TLS 1.3 instead of refusing to start when no TLS 1.2 suite without RSA key exchange is left
* chore(tls): warn when fips_mode leaves a cipher list with only TLS 1.3
* style(tls): split the TLS 1.3 fallback warning to fit the line limit
* fix(proxy): exclude RSA key exchange before config-load secret reads in FIPS mode
get_config reads secrets (AWS, GCP, HashiCorp, CyberArk secret managers) over the shared
outbound SSL context before enforce_fips_mode runs, so with general_settings.fips_mode true
those first TLS handshakes still offered RSA key exchange. Turn the exclusion on in get_config
from the raw general_settings, right before the secret manager starts, reusing the existing
fips_mode_setting (now public) and exclude_rsa_key_exchange. enforce_fips_mode still owns the
refusal and the final state; fips_mode given as an os.environ/ reference is still handled there.
* test(proxy): skip the config-load RSA key exchange test on an OpenSSL without RSA suites
On an OpenSSL whose default cipher list offers no RSA key exchange suite, fips and standard mode read secrets the same way, so the test has nothing to tell apart and skips, as the http_handler fallback test already does.
* fix(tls): exclude TLS 1.2 ChaCha20 suites from the outbound cipher list in FIPS mode
ChaCha20-Poly1305 is not a FIPS approved algorithm. FIPS providers already refuse it, so on the Chainguard
image this changes nothing, but fips_mode now drops it from TLS 1.2 explicitly next to RSA key exchange.
Python's set_ciphers cannot touch TLS 1.3 suites, which still come from OpenSSL's config.
exclude_rsa_key_exchange is renamed to exclude_rsa_key_exchange_and_chacha20 to match.
* fix(proxy): add tzdata to the proxy extra so a legacy TZ doesn't stop startup
* style(proxy): drop the inline comment on the tzdata requirement
* test(proxy): check tzdata is installed with the proxy extra on every platform
* fix(proxy): set the first spend report run in UTC so a scheduler zone that differs from the system clock can't skip it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): track self-heal for test_with_id_security_off_reads_and_cancels_add_no_rows_and_no_replayed_turns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): preserve response turn rows when id security is disabled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): list ids that route back for a prefixed partial alias over a catch-all deployment
* test(proxy): call the router registry in the catch-all alias listing test
* fix(proxy): keep main's listing for whole-name aliases over a catch-all
Only partial aliases like team1/gpt-* over a bare * run the per-id route check. A whole-name alias like team1/* keeps the string-only listing it had, so it lists the same ids at the same cost. Adds a test that a more specific sibling alias's ids are left out of the partial alias's list.
* fix(proxy): list catch-all aliases fast and skip ids a sibling alias takes
* fix(proxy): check team wildcard aliases without a router call per id and skip suffix ids that name another catalog model
Team keys now read the team's own wildcard patterns the way the router does (any team pattern first, then the provider form when its prefix can be a provider), so ids no rival can take skip the router call. A suffix alias such as team1/*-mini no longer lists team1/gpt-5.4-mini for gpt-5.4.
* fix(proxy): keep team key listings fast when aliases overlap and honor a pinned provider on partial aliases
A team key whose team defines the same alias as the global config, or a team
pattern that can take a global alias's ids, now checks only the team's own
patterns before calling the router, so the listing no longer pays a router
call per catalog id. The router also tries the provider-prefixed form of an id
only when the bare form matches nothing.
A partial alias over `*` with `custom_llm_provider` set now lists only that
provider's catalog, so `team1/gpt-*` pinned to OpenAI no longer lists
`team1/gpt-claude-*` ids that answer 404.
* fix(proxy): ask a pinned catch-all's provider for its models once per listing
A partial alias over a `*` deployment pinned with custom_llm_provider fetched the whole `*` catalog before taking the pinned branch, so with check_provider_endpoint on each listing called the provider's /v1/models one extra time. The pinned branch now runs before that fetch. Tests also cover a provider-less team alias that takes ids under their provider and a wildcard model_group_alias
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
helm/litellm becomes the only chart. Every container runs the one shared image with args only, monolith.enabled renders a single proxy Deployment, the retired chart's KEDA, extraResources, generated master key, helm test, minReadySeconds and rollout strategy settings are ported, and the bundled PostgreSQL and Redis subcharts are removed
Resolves LIT-8880
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp)!: forward the LiteLLM key when the admin configures it
Restores forwarding of the caller's LiteLLM key through explicit channels
(Extra Headers listing X-LiteLLM-API-Key or Authorization, x-mcp-{alias}-authorization
and x-mcp-auth) that #45401 and #46932 started stripping. Implicit Authorization
forwarding stays scrubbed for true_passthrough and unlisted servers, gateway
session tokens and bridge envelopes stay scrubbed, and guardrail and JWT signer
bearer reads now skip the admission key
* test(mcp): serve signer introspection from an in-process transport
* fix(mcp): copy oauth2 headers so a listed key never reaches a later server
* test(mcp): drive the shared-header regression through the real MCP client
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): merge a key's cache controls into the request's cache settings instead of replacing them
* fix(proxy): keep a key with no-cache out of opt-in caching
A key whose cache metadata sets no-cache now drops the caller's use-cache, so on a default_off proxy its responses stay out of the shared cache as before the merge change. default_on proxies ignore use-cache, so they are unchanged
* fix(proxy): keep base handling of non-dict key cache metadata
Key cache metadata that is not a dict, such as false or a string, wipes the request's cache settings as before the merge change, so those keys stay out of opt-in caching
* fix(proxy): keep a key with use-cache false out of opt-in caching
* fix(proxy): only keep request cache settings on an allowed opt-in
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(reasoning): log the GPT-5 reasoning_effort drop warning only when the provider gets no effort
* fix(proxy): count an extra_body effort as sent and pass the drop holder through ChatGPT and Copilot
* fix(proxy): pass the drop holder to get_optional_params under the internal _litellm_ prefix
* fix(openai): keep the reasoning effort drop warning when extra_body sends another level
The held warning now stays quiet only when the request sends the level the gate dropped,
so a top-level xhigh dropped while extra_body sends none still logs. The Copilot test case
writes the access-token file the current authenticator reads.
* fix(openai): keep the reasoning effort drop warning when a request fails before reaching the provider
The drop warning holder now lives for the whole completion call, and
completion's exception handler prints whatever it still holds, since a
request that fails after mapping never reached the provider. Each flush
empties the holder, so a provider error after dispatch does not print the
warning a second time
* fix(openai): warn when extra_body's reasoning_effort carries no effort level on the Responses bridge
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): bill gemini image text, thinking and priority tokens at their own rates
Co-authored-by: kerry <kerry@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(images): bill unitemized output as text when no images return
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): price image calls only by the tier the provider served
---------
Co-authored-by: kenneth <kenneth@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the upgrade notice off the request path and count area-labelled release notes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): regenerate dashboard API types and read the release cache through one typed helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): respond 502 from /get/latest_release_info when no release note entry is countable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): respond 502 when the release fetch failed and announce a release with no countable notes without counts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): re-ask for the latest release while the proxy is still fetching it and keep fetch diagnostics out of the 502 body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep the known release on screen while the proxy refetches it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer 202 while the release is being fetched so the dashboard polls only then
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): name the release query options to stay inside the lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): return the 202 as a response value instead of mutating the injected response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): fetch the release on the background client so its 502 never reaches the error handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): settle a fallback's tokens on the model group that served it
The v3 limiter pinned post-call TPM settlement to the model reserved at
admission, so when the router fell back to another model group the
requested model's counter took the usage and the serving model's counter
stayed empty. When the served model group differs from the admitted one,
charge the served group and refund the abandoned group's per-model
reservation. Same-model retries and alias reloads keep the admission target.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): refund an abandoned fallback reservation only on counters the served model does not settle
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rate-limit): wait for the primary refund and the fallback charge before reading counters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): only treat a limiter-admitted fallback target as the served model at settlement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rate-limit): drop unrelated formatting from the v3 limiter unit test file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rate-limit): remove helper copies duplicated while appending the fallback settlement tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order
A purpose=batch upload on a mixed Azure chat and batch group was order-filtered before batch routing narrowed it, so the file landed only on the order-1 chat deployment and the batch create that followed got Azure's 400 invalid_deployment_type. The upload now runs under the same batch routing as the batch create.
* fix(router): keep a failed batch upload's cooldown and order fallback on the batch deployments
* fix(router): keep a batch upload off chat deployments while its batch deployment cools down
* fix(router): keep a refused batch upload's retry off the chat deployment
* fix(router): keep a refused batch upload's error and skip chat while the batch deployment is paused
* test(router): pin the order fallback for a batch create on a group with no batch deployment
* test(integration): expect an Azure batch upload on the batch deployment alone
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(anthropic): forward message threads on native /v1/messages
Keep the thread body field on native Anthropic legs, add the
message-threads-2026-08-12 beta for it, reject it with a 400 on legs that
cannot continue a stored thread, and relay error.details of an upstream
Anthropic error so thread_not_found reaches SDK thread helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(anthropic): keep a stale thread 404 from cooling the deployment and type the thread beta ignores
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: sort the cooldown handler imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): dedupe the message-threads beta map rows and annotate test finals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): bind message threads to the key that created them and answer unsupported continues with thread_not_found
Every virtual key on a proxy shares one Anthropic credential, so forwarding thread alone let any key
continue another key's thread by its message id. A new default proxy hook binds each thread reply id
to its virtual key, or to the signed-in identity when there is no key, and answers a continue naming
an id it did not bind with the 404 thread_not_found the SDK replays on.
Legs without message threads now serve a create without the thread and answer a continue with that
404 before any upstream call, instead of answering it with no history.
* fix(proxy): keep a cached message thread reply with the key that got it first
A response cache hit replays the original reply id, and binding it again handed the thread to the second key, so the creator's continue got thread_not_found.
* fix(anthropic): keep message threads bound on every route and stable across turns
Check the thread binding on every route that carries thread except pass-through,
keep bindings in their own cache so they never evict rate-limit counters, add no
tool or message derived betas on thread turns, answer a create on a leg without
threads with a 400 thread_unsupported_request unless drop_params is set, and
forward threads on Claude Platform on AWS
* fix(anthropic): reject message threads on /v1/messages passthrough deployments
A deployment that forwards /v1/messages untranslated to an OpenAI-compatible
server inherited the native config's thread support, so `thread` reached a
server that keeps no thread state. It now answers the same 400
thread_unsupported_request on create and 404 thread_not_found on continue as
a translated leg
* fix(anthropic): resend a thread's create betas on continues and check bound ids on pass-through
A thread create now gets the betas LiteLLM derives from its tools and
messages again, so an advisor tool create that answered 400 on the previous
commit answers 200. Anthropic fingerprints the beta set, so the proxy keeps
the anthropic-beta header the create sent upstream in the thread binding and
adds it to every continue on /v1/messages. It reads that header from the
upstream request, since success logging overwrites the logged request
details before the success hook runs
A continue on the /anthropic pass-through route of a thread the proxy bound
to another key now gets the 404 thread_not_found. Ids the proxy never bound
still pass through, as before
* fix(anthropic): continue unbound threads off /v1/messages and keep every anthropic-beta on chat and Responses
* fix(anthropic): skip non-string anthropic-beta list entries instead of raising
* test(anthropic): keep the header merge regression test clear of main's appended tests
* fix(anthropic): add a thread's create betas to the client's own on continues and keep base header merging on chat and Responses
* fix(anthropic): answer an unbound thread id on chat and Responses with the 404 and drop the stale beta-join tests
* fix(proxy): check a thread continue sent inside extra_body against its owner
Azure AI promotes extra_body into the request body, so another key could continue a thread by putting thread there, past the ownership hook that read only the top-level field
* fix(proxy): write a thread's first owner atomically and bind creates sent inside extra_body
Two workers that both miss a reply's binding now agree on the first key stored in Redis, since the claim uses SET NX and each worker caches the stored winner. A create sent inside extra_body now binds its reply like a top-level create. The redundant Claude Platform supports_message_threads override is removed, since the provider already resolves to the native Anthropic beta list
* fix(proxy): include agent_id in the key-less message thread owner
* fix(proxy): tag message thread ownership Redis calls with their key family
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): judge the zero-cost budget waiver by the model a key or team alias serves
The zero-cost budget waiver judged the name the client sent, but key `aliases` and team
`model_aliases` are applied later during request preparation. An over-budget key got a 422
for an alias of a $0 model, and an alias named like a $0 group that points at a paid model
was waived and served past the budget.
For a string body `model` on chat, completions, responses and messages, the key-path,
centralized and custom-auth waivers now judge the model request preparation serves. That
model comes from the same team, key, global and router alias functions that request
preparation uses, so the two cannot drift. Other routes and the JWT path keep judging the
requested model. common_checks no longer overrides a caller's explicit decision not to skip
budgets, and proxy-made sub-calls are judged by their own model.
Resolves LIT-10283
* fix(proxy): waive only once auth has applied the router aliases, and keep empty team alias targets
The served-model waiver now applies only after auth's router-alias rewrite has run on the
request body. Custom auth without common checks runs its waiver before that rewrite, so it
keeps judging the requested model instead of a key or team alias of a name the router
rewrites elsewhere. The waiver also keeps an empty-string team alias target, matching
request preparation.
Resolves LIT-10283
* fix(proxy): waive by the served model only for a client's own body-model request
The waiver resolved aliases for any request on a chat, completions, responses or messages
path, including the HTTP request the Responses WebSocket builds for first-frame admission.
That session dispatches the model as an endpoint model, which skips the team alias, so a
team alias of a paid model to a free one waived the paid model's budget.
The waiver now resolves aliases only when FastAPI matched the request to one of the body
model routes, which requests built inside the proxy and path model routes never are. The
waiver tests are folded into one table, share a priced router fixture, and cover the
stale team alias flag cache.
Resolves LIT-10283
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(passthrough): bill a background response once however many times it is read
* fix(passthrough): keep the background response flag on spend events sent to the collector
* test(spend): type the background response test helper and its settlement parameter
* fix(proxy): keep a background response's settling read in-process instead of adding a spend event field
An older collector refuses a spend event with a field it does not know and drops the whole charge, so a newer
gateway sending to an older collector lost the background response's cost. The read that settles a background
response now stays in-process, the same way video status polls do, and the spend event keeps its shape
* fix(proxy): keep a batch retrieve in-process when message redaction hides the logged batch
* refactor(proxy): split the is_offloadable_success docstring into one line per reason
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(rust_bridge): route every Rust-to-Python call through host modules
Rust now reaches Python only through litellm/rust_bridge/host/ (plus the
execution driver), and no Python module outside host/ imports it. Library
calls that reproduce Python behavior (httpx and OSError error text,
huggingface_hub downloads, the random generator, os.environ) go through
host functions too. The rule is written into host/AGENTS.md
The live litellm.model_cost table, its version and every Python writer move
to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only
the functions Rust calls. The cost map fetch passes use_http_settings instead
of handing Rust an http_settings() snapshot
* refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust
The Redis bridge asked Python for all of os.environ to read one key. CPython's
os.environ writes through to the process environment, so Rust reads it with
ProcessEnvironment like the rest of the module. Also drops doc comments that
restated their code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci(rust): raise the native extension size cap to 55 MB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(auto_router): report a session's whole spend on /auto_router/session
Claude Code sends its background calls straight to Haiku under the same session id, so those requests never
reached the auto-router session rollup and the status line under-reported the session. Every billed spend
row in a caller-supplied session now increments a key-scoped Redis counter whose one-hour expiry each write
resets, and the endpoint reports it as total_spend, with spend unchanged. The increment runs where the spend
row is accepted, so it counts each row exactly as often as the key's spend counters do. It reads as no total
without a shared Redis
* fix(auto_router): name the unsupported provider when a turn routes off Anthropic
A turn routed to a non-Anthropic tier with no native wire response, such as a Cloudflare DeepSeek model,
failed strict native validation and was logged as observation_unavailable, which reads like a broken
capture. It now reports unsupported_selected_provider. Turns with a native wire response keep their
existing reasons, and no estimate is added, so savings for these turns stay unknown
* docs(auto_router): explain when session savings stay unestimated
saved_spend on /auto_router/session now says a turn inside the longest prompt-cache TTL of the session's
first observed turn, or of its last incomplete observation, is not estimated, so a null there is expected
for that window
* fix(auto_router): say why a system-only cache prefix has no savings estimate
Claude Code puts its cache breakpoint on the system prompt, so the baseline prefix count sends Anthropic a
body with no messages, which it rejects, and every turn read as token_count_unavailable. A count that fails
only on system or tools prefixes now reports uncountable_system_or_tools_prefix. Gateways that accept the
empty body still count it. A live check against claude-opus-5-5 showed a differential count with a
placeholder message is off by -2 tokens for system-only and +68 for tools, so these turns stay unestimated
and the savings gap for Claude Code is not fixed here
* fix(auto_router): declare the session total's Redis key family and test its expiry
The session total read and wrote Redis without a service target, so its calls showed up as bare redis
spans. Both now run under session_spend_totals. The key no longer hashes the api key, which is already a
hashed token, and only digests the session id. A new test drives Redis expiry on a test clock to show an
active session keeps its total and an idle one restarts
* docs(auto_router): note which turns are still estimated inside the history window
Turns that took the identical baseline path, and turns whose whole cached prefix the session had already
written, are estimated even inside the first cache TTL window, so saved_spend now says so
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(router): re-check and reserve per-model rate limits on fallback targets
A provider error fallback handed the caller the fallback model group without
its per-model TPM/RPM limits ever being checked. The router gets a
fallback_rate_limit_check hook next to fallback_budget_check, the proxy wires
it to the v3 limiter, which reserves the target model's per-model descriptors
before each fallback hop and skips targets that are over their limit.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): dedupe a repeated fallback target before the reserving rate limit check
Both walks now consult the attempted set before the rate limit check, so a
chain like ["B", "B"] reserves B once. The deprecated settings parity test
also learns the new setting.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): a bidirectional fallback chain never re-reserves the requested model group
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): match the surfaced failure in the fallback reservation tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): keep the admission stash untouched while building fallback descriptors
Building the fallback target's descriptors could rewrite the request stash's
PTU ceiling, which post-call settlement reads for the model the caller named.
The fallback check now hides the stash while it builds, like the frame check.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The PATCH model update endpoint was missing from the management_routes group, so virtual keys restricted to management_routes got a 403 on it while the legacy POST /model/update worked. Fixes#50918.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(guardrails/rubrik): moderate text-completion prompts and forward rac_agent_id
_moderate_prompt synthesized a user message from texts for /v1/completions
requests but built the payload from structured_messages, so the webhook
received an empty messages list and the prompt went unmoderated. Pass the
resolved messages into _build_prompt_moderation_payload.
Forward metadata.rac_agent_id to the before_prompt and after_completion
webhooks so agent-scoped policies apply. Only that key is forwarded; the
rest of request_data metadata carries non-serializable objects and inbound
headers
* fix(guardrails/rubrik): also read rac_agent_id from litellm_metadata
/v1/messages, /v1/responses and other LITELLM_METADATA_ROUTES keep proxy
metadata under litellm_metadata, so a caller on those routes may set
rac_agent_id there. Read metadata first, then litellm_metadata, forwarding
only a non-empty string
* fix(guardrails/rubrik): take rac_agent_id from key or team metadata only
The Rubrik webhooks use rac_agent_id to choose which agent-scoped
policies apply. Forwarding it from the request body let a caller pick any
agent's policies, including one a policy exempts. Read it from the
caller's admin-configured key metadata, falling back to team metadata,
via CustomGuardrail._get_admin_metadata. The proxy writes those from the
authenticated key and strips client-supplied user_api_key_* fields from
both metadata dicts, so a caller cannot set or override the value
---------
Co-authored-by: yucheng-berri <yucheng@berri.ai>
* perf(spend): narrow user and team spend log reads to the owner's api_keys so the (api_key, startTime) index is used
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): satisfy the type discipline gate for the api_key scope helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): run the api_key lookup under the statement timeout, skip it for search and empty key lists
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
langfuse_otel, weave_otel and langtrace now encode litellm.Message objects, and any value JSON cannot encode, when they write the prompt, so spans keep their token usage and model parameters. Under LITELLM_OTEL_V2 the payload builder dumps Message objects instead of dropping them
