Commit graph

56299 commits

Author SHA1 Message Date
mateo-berri
8a8188664c Merge remote-tracking branch 'origin/main' into HEAD
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
LiteLLM Rust DB / db-queries (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
# Conflicts:
#	tests/unit/test_router/test_router_endpoints.py
2026-11-11 14:48:57 -08:00
tavren27
36c7f10c84
fix(proxy): answer x-litellm-call-id on every token count response (#51037)
* fix(proxy): answer x-litellm-call-id on every token count response

The token count log line names a litellm_call_id, but count responses
never carried it, so a client that sent no id of its own could not find
its count's log line. Every count route (Anthropic Messages, Responses
input_tokens, Gemini countTokens, Bedrock count-tokens, and
/utils/token_counter) now answers the id on success and on errors, and
a crash answers the JSON 500 the Anthropic and Gemini routes already used.

* fix(proxy): keep the gateway's crash answers on count routes and stamp Responses opt-in bodies

A crash on /utils/token_counter or /v1/responses/input_tokens goes back to the
gateway's generic 500, or 503 when the database is down, now with the
x-litellm-call-id header. Responses count rejections carry error.litellm_call_id
when include_call_id_in_error_body is on

* test(proxy): count through count_request_tokens in the token counter loop tests

The token_counter route now also takes the response it sets x-litellm-call-id on, so these tests call the shared count function they exercise

* fix(proxy): keep a count crash's own exception and log pre-count Responses failures

The call id now reaches the crash handler through request.state instead of a wrapper exception, so logs and Sentry see the error the count raised. A Responses count that fails before counting writes its count line for any error, so its header names a line

* fix(proxy): keep the token counter's injected args out of its API docs

Tests the opt-in error body id on an Anthropic count the proxy rejects before counting

* fix(proxy): keep a count route's own HTTP error when adding the call id

The wrapper raised a new HTTPException with the same status, detail and headers, so a subclass lost its type and error tracking saw a new exception. It now adds the header to the original and raises it again

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-11 22:26:47 +00:00
joshua-berri
a45e3b6605
fix(proxy): reset a deleted customer's admission spend counter so a re-created one is not refused (#51224)
* fix(proxy): reset a deleted customer's admission spend counter so a re-created one is not refused

/customer/delete only dropped the committed twin, so spend:end_user:<id> kept the old spend and a
re-created customer with the same id stayed over budget. Zero the counter and its DB floor marker on
every worker after the delete commits, the same way a key spend reset does.

* fix(proxy): drop a deleted customer's spend counter before zeroing it

A Redis that rejects writes but still serves reads, such as one at its memory limit, kept the old
counter when the zeroing SET failed, so the re-created customer stayed refused. Delete the counter
first, which Redis still allows in that state, then write zero and the floor.

* test(proxy): type the customer delete spend counter test parameters

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-11 14:24:26 -08:00
quietjules
89149b4544
refactor(typing): clear 1141 basedpyright Any errors across 182 files (#50711)
* refactor(typing): clear 340 basedpyright Any errors (wave 1)

* refactor(typing): clear 497 basedpyright Any errors (wave 2)

* refactor(typing): clear 74 basedpyright Any errors (wave 3)

* refactor(typing): clear 143 basedpyright Any errors (wave 4)

* refactor(typing): clear 106 basedpyright Any errors (wave 5)

* fix(typing): keep router fence visibility and py310-safe Self import

Bind the Protocol-narrowed routers in pass_through_endpoints and rag
ingestion to router-named locals so router_surface_check still sees the
aadapter_completion and aocr uses, and import Self from typing_extensions
in prisma_client for check_py310_typing_imports.

* perf(core_helpers): fast-path dicts and lists in the checked JSON helpers

as_str_mapping now tries the concrete dict check before the Mapping ABC
check, and iter_items returns lists before the runtime-checkable Protocol
check. Both return the same object as before, and roughly halve the
overhead CodSpeed flagged on the A2A text-extraction benchmarks.

* refactor(realtime): keep the base TypeError for a non-mapping translation expires_after

* fix(proxy): restore the require_attributes import the main merge dropped

* test: cover the retyped realtime, vector store, agent, stream, video and provider response paths

* fix(realtime): name the public request types in non-mapping TypeErrors

keyword_arguments wraps expires_after and transcription_session so a non-mapping value raises the same TypeError text as before the typing pass. Tests narrow Greptile's pegasus case with isinstance and freeze its model.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-11 22:11:47 +00:00
mrinal-berri
5e74515c29
feat(otel): let a team or key OTel v2 destination add its own request metadata keys (#50813)
* feat(otel): let a team or key OTel v2 destination add its own request metadata keys

A team or key callback can now set baggage_metadata_keys in its callback_vars,
a comma-separated list of requester_metadata.<key> paths. The tenant fan-out
adds them as litellm.metadata.<key> to that destination's spans only, on top of
the global LITELLM_OTEL_BAGGAGE_METADATA_KEYS; the operator's exporter and other
destinations never see them. Values are bound when each span ends, an attribute
the span already carries wins, and names of proxy identity fields are rejected
when saved, along with lists over 20 keys.

Resolves LIT-9545

* fix(otel): refuse request headers in a destination's baggage_metadata_keys

The proxy copies the request headers into requester_metadata.headers with
credential headers masked, but a header a gateway in front of it added
(an IAP or OIDC assertion, an internal token) stays as sent. A team or key
admin listing requester_metadata.headers.<name> could export it to an
endpoint the team controls. Entries under requester_metadata.headers are
now rejected when saved, so exporting a header stays an operator decision
in LITELLM_OTEL_BAGGAGE_METADATA_KEYS. Also drops an unused import.

* fix(otel): hold destination metadata to the configured span limits

A destination's extra litellm.metadata.* attributes were added to a span
view built from a plain mapping, so they skipped the limits the SDK
applies in set_attribute: a 1 MiB caller value survived an operator's
OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT. Values are now cut to the length
limit, and keys past the attribute-count limit are dropped and counted
instead of evicting an attribute the span already carries.

The per-destination views are built in one Final tuple ahead of the
forwarding loop, since a Final cannot be assigned inside the loop.

* fix(otel): apply destination metadata after the destination view, within the provider's limits

The span limits were checked against the span as it ended, before the
destination view strips prompt content and backend-only attributes, so a
model-call span full of llm.input_messages.* could drop the team's keys on
a no_content destination that had room. The values are now bound when the
span ends and added as the last step before export, counted against what
that destination actually sends, so their dropped count also survives the
view. The limits come from the tracer provider the fan-out is attached to,
read from the live span at start, instead of a separate env-built SpanLimits.

* refactor(otel): consolidate span limits, destination validators and metadata key parsing

Reads a span's limits in one guarded place that the emitter and the fan-out
share, replaces the two parallel lists of per-destination callback-var
validators with one mapping, splits and strips a baggage_metadata_keys list
once, drops a filter the post-view merge already enforces, and folds the
repeated fan-out test setup into one helper. No behavior change.

* ci: retrigger checks stranded by the GitHub Actions outage
2026-11-11 13:57:41 -08:00
paveln83
b63656a5de
fix(proxy): stream the guardrail block response when the block comes after an SSE keepalive ping (#51114)
* fix(proxy): stream the guardrail block response when the block lands after an SSE keepalive ping

* docs(proxy): keep the keepalive docstring's reasons together

* fix(proxy): stream the /v1/messages block after a keepalive ping and always end a late block stream

A passthrough guardrail block on a streamed /v1/messages request that landed
after the first keepalive ping still ended as an api_error event. The route now
hands its block stream to the keepalive relay, as chat and text completions do.
If a route's block stream itself fails after a ping, the relay sends the usual
late failure frame instead of raising, so the stream always ends.

The passthrough tests now drive the real routes with a real guardrail.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-11 21:39:19 +00:00
zoveki
a1f3dcd01b
fix(completion)!: answer 400 for a chat message without a role (#51124)
* fix(completion): answer 400 for a chat message without a role

A message with no role was filled with role assistant and sent upstream, so the caller's text reached the
model as its own earlier reply and the request returned 200. Now a missing, null, or empty role answers 400
with OpenAI's wording, naming the message index. A role-less turn that carries tool_calls is still sent as
assistant, since only an assistant writes tool calls.

* fix(completion): treat a role-less message as the caller's error

The 400 now raises MalformedRequestFieldError with OpenAI's param and code and the provider name, so it no longer counts toward the deployment's failure rate, cools it down, or fires an llm_exceptions alert

* fix(completion): keep the assistant fill for a role-less function_call turn

* refactor(completion): type the role-less call turn check and the request kwargs

Read role, tool_calls and function_call off the cleaned message copy, and validate completion's kwargs into a typed mapping before naming the provider on the 400

* fix(completion): keep the alert and failure-count exclusions to the role error

Raise MissingMessageRoleError, a MalformedRequestFieldError subclass, so a malformed tools 400 alerts and counts as on main, and fill assistant whenever a role-less turn has a tool_calls or function_call key

* fix(completion): keep assistant-only role-less turns and skip SDK and PagerDuty alerts

A role-less turn carrying any assistant-only key, null values included, is still
filled as assistant, so a dumped assistant message or a reasoning turn keeps working.
The Router's alerting_config alert and the PagerDuty failure count skip the role error,
like the proxy alert already does.

* fix(utils): drop the string code from the missing-role 400 so the Errors tab counts it as 400

The spend log stored code missing_required_parameter as the error code, which
the Usage page Errors tab cannot parse as a status, so every role-less request
landed under no status. The malformed tools 400 leaves code out the same way.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-11 20:59:19 +00:00
yuneng-jiang
f52f26d9c6
fix(proxy): let /team/delete transactions wait for a pooled connection (#51099)
The team-model and team-key transactions in /team/delete used Prisma's 2s
default max_wait, so a busy writer pool failed the delete with P2028 while
every plain query in the same request waited up to the 60s pool timeout. Both
now get the 30s team-delete budget for max_wait, and the key transaction gets
it for timeout too.
2026-11-11 12:37:28 -08:00
yuneng-jiang
f889b2b587
test(integration): fix the baseline accounting and series cap flakes (#51096)
* test(integration): run baseline accounting store tests in a private schema

The accounting workers share their database with two gateway proxies, and every
spend-log tick on those proxies projects any dirty baseline comparison it finds.
When one of them published the test's scope between its last append and its own
project() call, the test saw "unchanged" instead of "published".

The module fixture now copies the twelve tables the store touches into a fresh
schema and points DATABASE_URL at it for the module, so the gateway proxies,
which only read public, can no longer publish these rows.

* test(integration): wait for H9's own prometheus increments before it returns

H9 stopped as soon as both workers' sample files already counted `other`, so
its own chat's increment could still be in flight and land inside the next
cell's window, which then saw two requests instead of one.

H9 now takes a before scrape, counts the chats it sends, and finishes with the
same _overflowed wait every other TestCapped cell uses.
2026-11-11 12:36:02 -08:00
joshua-berri
deb7697a8e
fix(auth): enforce model access on Google and image URL routes (#51226)
* fix(auth): authorize URL-selected Google and image models

* fix(auth): preserve server model selection and budget fallbacks

* test(auth): isolate URL model authorization settings

* fix(auth): type handler model selection boundaries

* fix(auth): validate request model selection inputs

* fix(auth): tolerate absent request path metadata

* fix(auth): support requests without path metadata

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-11 12:22:12 -08:00
joshua-berri
8f42db985d
fix(router): map Gemini stream-opening errors (#51218)
* fix(router): map errors when opening Gemini streams

* fix(streaming): preserve stream-open exception causes

* fix(streaming): limit open-error mapping to provider and transport errors

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-11 12:18:44 -08:00
Kerry Lu
a0f2562088
feat(gemini): file-backed batch jobs on /v1/batches with output download and spend tracking (#51032)
* feat(gemini): file-backed batch jobs on /v1/batches with output download

Adds GeminiBatchesConfig (create + retrieve) behind custom_llm_provider=gemini,
registers it in ProviderConfigManager, and implements file content retrieval on
GoogleAIStudioFilesHandler so GET /v1/files/{output_file_id}/content returns the
batch output. Gemini runs the OpenAI-shaped JSONL the proxy uploads today unchanged.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(gemini): type the file content request params so no pyright rule grows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(gemini): price completed batch output rows so spend lands on the key

Gemini writes the usage of an OpenAI-shaped output row as camelCase
(promptTokens, completionTokens, totalTokens) with thinking tokens only in
totalTokens. Map it onto Usage with the thinking tokens folded into
completion_tokens, so the shared batch cost path bills them at the gemini
_batches rates.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(gemini): parse nanosecond file timestamps

Gemini stamps files with nine fractional digits, which strptime's %f
rejects, so every retrieve of a batch output file logged a ValueError.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(gemini): parse file timestamps on Python 3.10 too

fromisoformat on 3.10 takes neither a Z suffix nor more than six fractional digits.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(gemini): price native batch output rows, stream output downloads, reject unsupported batch metadata

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(gemini): detect native batch rows by key, anchor the streaming file download handler

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(httpx): make anchor_handler_to public so the file content stream can use it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 12:00:32 -08:00
devin-ai-integration[bot]
c9d3c8b52f
fix(streaming): report zero cost for explicitly free calls (#51093)
* fix(streaming): report zero cost for explicitly free calls

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(streaming): require a stored entry for free model pricing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(streaming): reuse declared pricing lookup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(streaming): preserve selected free pricing entries

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(streaming): recognize explicitly free pricing tiers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(streaming): cover invalid pricing tier fallbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kenneth <kenneth@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 12:00:11 -08:00
yujonglee
95ecfbbac6
feat(proxy): supervise the opt-in Rust forwarding server (#49606)
* feat(proxy): supervise the opt-in Rust forwarding server

With LITELLM_RUST on, `litellm` and `python -m gateway.launch` start a
supervisor that runs the Rust gateway on the public port and Python on a
private loopback listener. Python still serves every route and sees each
request as if the client had connected to it directly.

Rust sends the client's TCP peer in `x-litellm-rust-peer`. The supervised
server turns server-level proxy headers off, and a middleware restores
the peer and then applies the operator's own trust, so client IPs,
trusted proxies and every IP-based setting behave the same with Rust on
or off.

The supervisor is transparent to Python. Signals pass through, Python's
exit status is the result, and both children shut down when the
supervisor's end of their stdin closes, so nothing outlives it. Rust
starts only once Python listens. When forwarding cannot start (no
gateway binary, TLS, a Unix socket, Hypercorn or Granian), the entry
point warns and serves from Python alone.

Every forwarding decision lives in litellm/proxy/rust_forward, paired
with the gateway-forward crate. OS differences sit behind a Platform
protocol, with Windows supported for local CLI runs. AGENTS.md files in
the touched folders state the boundary. The `--rust_gateway` flag is
gone in favor of the environment variable.

* refactor(proxy): switch Rust forwarding on with LITELLM_RUST_FIRST

Forwarding gets its own switch, independent of LITELLM_RUST, which keeps
selecting only the Rust paths inside the SDK. Neither turns the other on.

---------

Co-authored-by: narmstrong-berri <narmstrong@berri.ai>
2026-11-11 19:40:00 +00:00
yujonglee
b079bf5a51
refactor(proxy): let the gunicorn server take a private listener (#50871)
Extract serving_port() and configure_database() as they were inline, turn the
LITELLM_RUST parser into a reusable RustSwitch type, and reformat
tests/unit/proxy/test_proxy_cli.py to the 120 column limit.

run_gunicorn_server takes private_listener, an inherited socket or an assigned
loopback port. With one, gunicorn binds it instead of host:port and trusts no
peer's forwarding headers, since every peer on it is the local process in front.
Both are gunicorn's own settings, bind and forwarded_allow_ips, so the uvicorn
worker is unchanged. The reload options take factory. No caller passes either
yet, so every entry point serves as before.
2026-11-11 19:40:00 +00:00
Kerry Lu
fdafffb842
feat(proxy): count a rate-limit fallback in x-litellm-attempted-fallbacks and name it in x-litellm-fallback-reason (#51138)
* feat(proxy): count a rate-limit fallback in x-litellm-attempted-fallbacks and name it in x-litellm-fallback-reason

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): count every fallback candidate a local rate limit rejected in x-litellm-attempted-fallbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): count fallback attempts without a mutable list and keep self.data on the served request

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 11:16:21 -08:00
kenneth-berri
13089b191b
fix(azure): use supported API versions for Responses requests (#51116) 2026-11-11 18:53:54 +00:00
devin-ai-integration[bot]
8fb9f2f090
chore(cost-map): add 24 runwayml models and correct seedance2 and eleven_multilingual_v2 rates (#51216)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 18:52:17 +00:00
devin-ai-integration[bot]
3a91fdfc07
fix(management): keep classifier deployments serving during reloads (#51111)
* fix(management): track self-heal for test_moving_the_workspace_under_a_burst_moves_the_classifier_and_logs_every_request_once

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(management): keep classifier deployments serving during reloads

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 18:48:06 +00:00
devin-ai-integration[bot]
3cd48a6ec7
test(security): expect the async release check on offline mode (#51213)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 18:15:42 +00:00
Mateo Wang
20d53f2951
refactor: remove redundant comments from fresh changes (#51163)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 09:18:18 -08:00
joshua-berri
0a4ccf9554
fix(router): preserve held tiers for OpenCode subagent results (#51176)
* fix(router): preserve held tiers for OpenCode task results

* fix(router): handle OpenCode subagent completion envelopes

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-11 07:44:45 -08:00
devin-ai-integration[bot]
1d97931c84
chore(cost-map): rolling registry PR, fal_ai rows, databricks gemini-4-carbon, deepinfra and bedrock tier prices (#51193) 2026-11-11 07:44:09 -08:00
Oliver Jensen
25b1de6369
docs(security): updated mailinglist and vdp in security.md (#51199)
* tweak wording on ghsa again
2026-11-11 16:23:45 +01:00
devin-ai-integration[bot]
8b8ec4e3a6
fix(vertex): drop floor-only retirement dates, veo 3.1 successor and gemini token limits from the Vertex model pages (#51180)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 10:06:57 +00:00
yucheng-berri
eba880e108
fix(tls)!: exclude RSA key exchange and ChaCha20 from the outbound cipher list in FIPS mode (#50671)
* fix(tls): exclude RSA key exchange from outbound cipher list in FIPS mode

* fix(tls): apply the FIPS cipher filter to SSL contexts clients already hold

enforce_fips_mode runs after load_config, so guardrail and logger clients
built during config load held an SSL context without the RSA key exchange
exclusion. enforce_fips_mode now updates every cached context in place, and
the SDK no longer imports the proxy's FIPS module.

* fix(tls): refuse to start in FIPS mode when SSL_SECURITY_LEVEL has only RSA key exchange

Excluding RSA key exchange from an SSL_SECURITY_LEVEL that lists only RSA
key-exchange suites leaves no TLS 1.2 suite, so set_ciphers raised on every
client build. enforce_fips_mode now checks the configured list first and
refuses to start with a FipsModeError that names the setting. The cipher
list test no longer assumes the host has RSA key-exchange suites.

* fix(tls): offer only TLS 1.3 instead of refusing to start when no TLS 1.2 suite without RSA key exchange is left

* chore(tls): warn when fips_mode leaves a cipher list with only TLS 1.3

* style(tls): split the TLS 1.3 fallback warning to fit the line limit

* fix(proxy): exclude RSA key exchange before config-load secret reads in FIPS mode

get_config reads secrets (AWS, GCP, HashiCorp, CyberArk secret managers) over the shared
outbound SSL context before enforce_fips_mode runs, so with general_settings.fips_mode true
those first TLS handshakes still offered RSA key exchange. Turn the exclusion on in get_config
from the raw general_settings, right before the secret manager starts, reusing the existing
fips_mode_setting (now public) and exclude_rsa_key_exchange. enforce_fips_mode still owns the
refusal and the final state; fips_mode given as an os.environ/ reference is still handled there.

* test(proxy): skip the config-load RSA key exchange test on an OpenSSL without RSA suites

On an OpenSSL whose default cipher list offers no RSA key exchange suite, fips and standard mode read secrets the same way, so the test has nothing to tell apart and skips, as the http_handler fallback test already does.

* fix(tls): exclude TLS 1.2 ChaCha20 suites from the outbound cipher list in FIPS mode

ChaCha20-Poly1305 is not a FIPS approved algorithm. FIPS providers already refuse it, so on the Chainguard
image this changes nothing, but fips_mode now drops it from TLS 1.2 explicitly next to RSA key exchange.
Python's set_ciphers cannot touch TLS 1.3 suites, which still come from OpenSSL's config.

exclude_rsa_key_exchange is renamed to exclude_rsa_key_exchange_and_chacha20 to match.
2026-11-11 00:48:11 -08:00
Satwik Bankapur
3616014548
fix(proxy): add tzdata to the proxy extra so a legacy TZ doesn't stop startup (#49914)
Some checks are pending
unit / key-generation (push) Blocked by required conditions
unit / mcp-oauth (push) Blocked by required conditions
unit / proxy-runtime (push) Blocked by required conditions
unit / proxy-server-core (push) Blocked by required conditions
unit / root (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / jwt-and-keys (push) Blocked by required conditions
unit / logging-misc (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / rust-bridge (push) Waiting to run
unit / assert-shard-coverage (push) Waiting to run
unit / enterprise-managed-files (push) Blocked by required conditions
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(proxy): add tzdata to the proxy extra so a legacy TZ doesn't stop startup

* style(proxy): drop the inline comment on the tzdata requirement

* test(proxy): check tzdata is installed with the proxy extra on every platform

* fix(proxy): set the first spend report run in UTC so a scheduler zone that differs from the system clock can't skip it

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 23:09:43 -08:00
devin-ai-integration[bot]
c2afb07f93
fix(spend): keep free response reads and cancels off the creation spend row (#51110)
* fix(spend): track self-heal for test_with_id_security_off_reads_and_cancels_add_no_rows_and_no_replayed_turns

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend): preserve response turn rows when id security is disabled

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 07:02:00 +00:00
devin-ai-integration[bot]
606490a52d
fix(gemini): batch and flex tier fields the -latest alias rows were missing (#51150)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 06:31:49 +00:00
devin-ai-integration[bot]
525df3f9a1
test(spend): expect token billing for Vertex image edits (#51148)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 06:27:58 +00:00
devin-ai-integration[bot]
97f07af044
test(providers): preserve unspecified Gemini image output modality (#51149)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 06:27:12 +00:00
quietjules
ac6e78cb1f
fix(proxy): list callable ids for partial aliases over a catch-all deployment (#50963)
* fix(proxy): list ids that route back for a prefixed partial alias over a catch-all deployment

* test(proxy): call the router registry in the catch-all alias listing test

* fix(proxy): keep main's listing for whole-name aliases over a catch-all

Only partial aliases like team1/gpt-* over a bare * run the per-id route check. A whole-name alias like team1/* keeps the string-only listing it had, so it lists the same ids at the same cost. Adds a test that a more specific sibling alias's ids are left out of the partial alias's list.

* fix(proxy): list catch-all aliases fast and skip ids a sibling alias takes

* fix(proxy): check team wildcard aliases without a router call per id and skip suffix ids that name another catalog model

Team keys now read the team's own wildcard patterns the way the router does (any team pattern first, then the provider form when its prefix can be a provider), so ids no rival can take skip the router call. A suffix alias such as team1/*-mini no longer lists team1/gpt-5.4-mini for gpt-5.4.

* fix(proxy): keep team key listings fast when aliases overlap and honor a pinned provider on partial aliases

A team key whose team defines the same alias as the global config, or a team
pattern that can take a global alias's ids, now checks only the team's own
patterns before calling the router, so the listing no longer pays a router
call per catalog id. The router also tries the provider-prefixed form of an id
only when the bare form matches nothing.

A partial alias over `*` with `custom_llm_provider` set now lists only that
provider's catalog, so `team1/gpt-*` pinned to OpenAI no longer lists
`team1/gpt-claude-*` ids that answer 404.

* fix(proxy): ask a pinned catch-all's provider for its models once per listing

A partial alias over a `*` deployment pinned with custom_llm_provider fetched the whole `*` catalog before taking the pinned branch, so with check_provider_endpoint on each listing called the provider's /v1/models one extra time. The pinned branch now runs before that fetch. Tests also cover a provider-less team alias that takes ids under their provider and a wildcard model_group_alias

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 22:20:06 -08:00
Mateo Wang
3622831fba
fix(spend): keep empty-key rows in owner-scoped spend log reads (#51127)
* fix(spend): preserve empty-key rows in owner-scoped log reads

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): isolate passthrough tests without reloading shared modules

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 22:06:50 -08:00
devin-ai-integration[bot]
ec8d96d7d4
feat(terraform): run every AWS and GCP workload from the one litellm image (#43358)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 22:03:33 -08:00
devin-ai-integration[bot]
60103ca017
feat(helm): single chart with monolith mode and shared image, remove litellm-helm (#43330)
helm/litellm becomes the only chart. Every container runs the one shared image with args only, monolith.enabled renders a single proxy Deployment, the retired chart's KEDA, extraResources, generated master key, helm test, minReadySeconds and rollout strategy settings are ported, and the bundled PostgreSQL and Redis subcharts are removed

Resolves LIT-8880

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 22:03:33 -08:00
devin-ai-integration[bot]
7ecb9ca704
fix(bedrock): govcloud eol dates from the lifecycle table, gpt-5.6 cyber govcloud mantle row and daybreak blue input limit (#51095)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 05:36:28 +00:00
devin-ai-integration[bot]
407ce3da2f
chore(cost-map): add 3 fal_ai rows priced from the fal pricing API and model page (third fal-ai sync) (#51126)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 05:18:02 +00:00
Mateo Wang
19c48218ae
test: deflake proxy metrics and shutdown drain (#51078)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 21:05:41 -08:00
Mateo Wang
f8a60328b4
ci(image-scan): build Rust with a fast ci cargo profile instead of fat-LTO release (#51075)
* ci: speed up image scans with an opt-in Rust profile

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: separate cached Rust wheels by maturin profile

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 21:05:29 -08:00
devin-ai-integration[bot]
e35eec2bbe
feat(azure): add azure_ai/mistral-large-4-latest from the Azure Models API, priced from Mistral's model page (#51047)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 04:51:57 +00:00
joshua-berri
27e5c3ba75
fix(mcp)!: forward the LiteLLM key when the admin configures it (#50604)
* fix(mcp)!: forward the LiteLLM key when the admin configures it

Restores forwarding of the caller's LiteLLM key through explicit channels
(Extra Headers listing X-LiteLLM-API-Key or Authorization, x-mcp-{alias}-authorization
and x-mcp-auth) that #45401 and #46932 started stripping. Implicit Authorization
forwarding stays scrubbed for true_passthrough and unlisted servers, gateway
session tokens and bridge envelopes stay scrubbed, and guardrail and JWT signer
bearer reads now skip the admission key

* test(mcp): serve signer introspection from an in-process transport

* fix(mcp): copy oauth2 headers so a listed key never reaches a later server

* test(mcp): drive the shared-header regression through the real MCP client

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-10 20:51:34 -08:00
devin-ai-integration[bot]
83c47061e1
feat(replicate): add google/gemini-4-carbon from the Replicate model page (#51125)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 04:33:52 +00:00
quietjules
63d39b41fd
fix(proxy): keep a request's cache opt-in when its key has cache metadata (#50999)
* fix(proxy): merge a key's cache controls into the request's cache settings instead of replacing them

* fix(proxy): keep a key with no-cache out of opt-in caching

A key whose cache metadata sets no-cache now drops the caller's use-cache, so on a default_off proxy its responses stay out of the shared cache as before the merge change. default_on proxies ignore use-cache, so they are unchanged

* fix(proxy): keep base handling of non-dict key cache metadata

Key cache metadata that is not a dict, such as false or a string, wipes the request's cache settings as before the merge change, so those keys stay out of opt-in caching

* fix(proxy): keep a key with use-cache false out of opt-in caching

* fix(proxy): only keep request cache settings on an allowed opt-in

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 20:21:39 -08:00
shrey
478d9bf5d2
test(mcp): read the comma grant through real credential storage (#51065) 2026-11-10 19:50:44 -08:00
paveln83
054ed65cdc
fix(reasoning): log the GPT-5 reasoning_effort drop warning only when the provider gets no effort (#50995)
* fix(reasoning): log the GPT-5 reasoning_effort drop warning only when the provider gets no effort

* fix(proxy): count an extra_body effort as sent and pass the drop holder through ChatGPT and Copilot

* fix(proxy): pass the drop holder to get_optional_params under the internal _litellm_ prefix

* fix(openai): keep the reasoning effort drop warning when extra_body sends another level

The held warning now stays quiet only when the request sends the level the gate dropped,
so a top-level xhigh dropped while extra_body sends none still logs. The Copilot test case
writes the access-token file the current authenticator reads.

* fix(openai): keep the reasoning effort drop warning when a request fails before reaching the provider

The drop warning holder now lives for the whole completion call, and
completion's exception handler prints whatever it still holds, since a
request that fails after mapping never reached the provider. Each flush
empties the holder, so a provider error after dispatch does not print the
warning a second time

* fix(openai): warn when extra_body's reasoning_effort carries no effort level on the Responses bridge

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 19:38:52 -08:00
devin-ai-integration[bot]
cb024047ac
test(spend): expect token billing for Vertex image edits with usage (#51109)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 03:32:19 +00:00
devin-ai-integration[bot]
461b56b14b
fix(vertex_ai): bill gemini image text, thinking and priority tokens at their own rates (#51017)
* fix(vertex_ai): bill gemini image text, thinking and priority tokens at their own rates

Co-authored-by: kerry <kerry@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(images): bill unitemized output as text when no images return

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost_calculator): price image calls only by the tier the provider served

---------

Co-authored-by: kenneth <kenneth@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 02:50:55 +00:00
Kerry Lu
bb89296adf
fix(proxy): keep the upgrade notice off the request path and count area-labelled release notes (#51027)
* fix(proxy): keep the upgrade notice off the request path and count area-labelled release notes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(proxy): regenerate dashboard API types and read the release cache through one typed helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): respond 502 from /get/latest_release_info when no release note entry is countable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): respond 502 when the release fetch failed and announce a release with no countable notes without counts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): re-ask for the latest release while the proxy is still fetching it and keep fetch diagnostics out of the 502 body

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep the known release on screen while the proxy refetches it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): answer 202 while the release is being fetched so the dashboard polls only then

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): name the release query options to stay inside the lint budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): return the 202 as a response value instead of mutating the injected response

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): fetch the release on the background client so its 502 never reaches the error handler

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 18:43:06 -08:00
Kerry Lu
917fd11966
fix(rate-limit): settle a fallback's tokens on the model group that served it (#51082)
* fix(rate-limit): settle a fallback's tokens on the model group that served it

The v3 limiter pinned post-call TPM settlement to the model reserved at
admission, so when the router fell back to another model group the
requested model's counter took the usage and the serving model's counter
stayed empty. When the served model group differs from the admitted one,
charge the served group and refund the abandoned group's per-model
reservation. Same-model retries and alias reloads keep the admission target.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(rate-limit): refund an abandoned fallback reservation only on counters the served model does not settle

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rate-limit): wait for the primary refund and the fallback charge before reading counters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(rate-limit): only treat a limiter-admitted fallback target as the served model at settlement

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rate-limit): drop unrelated formatting from the v3 limiter unit test file

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rate-limit): remove helper copies duplicated while appending the fallback settlement tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 18:36:55 -08:00
mika-lt
373fe799e7
fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order (#50971)
* fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order

A purpose=batch upload on a mixed Azure chat and batch group was order-filtered before batch routing narrowed it, so the file landed only on the order-1 chat deployment and the batch create that followed got Azure's 400 invalid_deployment_type. The upload now runs under the same batch routing as the batch create.

* fix(router): keep a failed batch upload's cooldown and order fallback on the batch deployments

* fix(router): keep a batch upload off chat deployments while its batch deployment cools down

* fix(router): keep a refused batch upload's retry off the chat deployment

* fix(router): keep a refused batch upload's error and skip chat while the batch deployment is paused

* test(router): pin the order fallback for a batch create on a group with no batch deployment

* test(integration): expect an Azure batch upload on the batch deployment alone

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-10 18:35:19 -08:00