Commit graph

43136 commits

Author SHA1 Message Date
Mateo Wang
1a183efaa1
Merge pull request #36159 from daleselaji-dev/codex/bedrock-alias-36156
Some checks are pending
CodSpeed Benchmarks / benchmarks (push) Waiting to run
Publish basedpyright base counts / publish (push) Waiting to run
Code Quality Checks / code-quality (push) Waiting to run
UI Unit Tests / ui-unit-tests (push) Waiting to run
Unit Tests: Core Utilities / core-utils (push) Waiting to run
Unit Tests: Documentation Validation / documentation (push) Waiting to run
Unit Tests: Enterprise, Google GenAI & Routing / enterprise-routing (push) Waiting to run
Unit Tests: Integrations (Callbacks & Logging) / integrations (push) Waiting to run
Unit Tests: LLM Provider Transformations / Vertex AI (push) Waiting to run
Unit Tests: LLM Provider Transformations / All Other Providers (push) Waiting to run
Unit Tests: MCP, Secrets, Containers & Misc / misc (push) Waiting to run
Unit Tests: Proxy Auth & Key Management / proxy-auth (push) Waiting to run
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy API Endpoints / proxy-endpoints (push) Waiting to run
Unit Tests: Proxy API Endpoints / proxy-server (push) Waiting to run
Unit Tests: Proxy Infrastructure / proxy-infra (push) Waiting to run
Unit Tests: Responses, Caching & Types / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
fix(bedrock): resolve aliases in batch file records
2026-08-14 18:20:21 -07:00
Mateo Wang
70d51a196f
Merge pull request #35286 from BerriAI/devin_ai_lit_5013_web_search_cost
fix(cost): track OpenAI/Azure web search tool cost per call
2026-08-14 18:16:59 -07:00
Mateo Wang
cba2beaf42
Merge pull request #36965 from erensh27/feat/per-component-cost-headers
feat(proxy): per-component response cost headers
2026-08-14 18:15:23 -07:00
yuneng-jiang
a4aab2e9f3
Merge pull request #36995 from BerriAI/litellm_/playground-config-select-25ad99
fix(ui): anchor chips-combobox popups to the field instead of the inner input
2026-08-14 18:11:38 -07:00
yuneng-jiang
491e2ffb1b
Merge pull request #36993 from BerriAI/litellm_/scrollable-tabs-layout-6aa9f7
fix(ui): stop the models tab strip from scrolling vertically
2026-08-14 18:04:42 -07:00
Yuneng Jiang
c7084c04c0
test(ui): assert which element the chips-combobox popup anchors to
The previous assertion read data-chips, which is derived from the anchor prop
being truthy, so it stayed true even when the ref never reached the DOM and the
popup was still anchored to the inner input.

Stub distinct widths on the chips container and the input, then read the width
the positioner resolved. Reverting the anchor wiring now reports the input's
width instead of the field's, which is the actual bug.
2026-08-14 18:03:34 -07:00
Yuneng Jiang
6e38e9490d
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/playground-config-select-25ad99 2026-08-14 17:57:27 -07:00
Yuneng Jiang
654fab224a
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/scrollable-tabs-layout-6aa9f7 2026-08-14 17:55:49 -07:00
Yuneng Jiang
a0e5c7e818
fix(ui): anchor chips-combobox popups to the field instead of the inner input
Base UI positions a combobox popup against the Combobox.Input by default. In
chips mode the visible field is the ComboboxChips wrapper and the input is a
smaller box nested inside it, so every chips-combobox in the dashboard opened
its popup 11px right of the field and 17px past its right edge.

shadcn ships the wiring for this and their combobox-multiple example uses it:
useComboboxAnchor on the chips container, passed to ComboboxContent as anchor.
The anchor prop also drives data-chips, which cancels the extra min-width an
ordinary combobox wants. Every chips site in the dashboard omitted it.

The anchor is attached through Base UI's render prop rather than a plain ref,
because React 18 drops refs on function components and ComboboxChips is one.

Adds MultiSelect's first test, covering the anchor wiring plus selection,
chip rendering and custom values.
2026-08-14 17:51:54 -07:00
Mateo Wang
40a418440c
Merge pull request #35969 from milesadkins/fireworks_nim_vllm_compat
feat(fireworks_ai): translate NIM/vLLM extra params to Fireworks-native args
2026-08-14 17:47:47 -07:00
Ahmed N
691c7fd4d6
fix(anthropic_messages): make tool_result images visible to OpenAI-compatible providers (#34462)
Images nested inside an Anthropic `tool_result` block were dropped when the
request was adapted for an OpenAI-compatible provider, because the OpenAI tool
message shape only carried text. Hoist those images out of the tool result and
into a following user message so the model can still see them, and widen the
tool message content type to accept image parts.
2026-08-14 17:47:38 -07:00
Yuneng Jiang
5219658b8f
fix(ui): stop the models tab strip from scrolling vertically
The tab strip carried overflow-x-auto directly on the TabsList. CSS forces
overflow-y from visible to auto once overflow-x is not visible, and the line
variant's active-tab underline is an absolutely positioned ::after that hangs
5px below its trigger, so the strip picked up a pixel of vertical scroll on top
of the horizontal scroll it actually wants.

The scroll container now lives on a wrapper whose bottom padding leaves room for
the underline, offset by a matching negative margin so the row keeps its exact
geometry.
2026-08-14 17:44:00 -07:00
Mateo Wang
118523ede6
Merge pull request #34581 from BerriAI/litellm_anthropic_messages_response_cache
fix(caching): cache anthropic /v1/messages responses, including streaming
2026-08-14 17:34:08 -07:00
Mateo Wang
dc92749c07
Merge pull request #35360 from BerriAI/devin_ai_fix_batch_cost_completed_no_output
fix(batches): mark terminal batch with no output file as processed in CheckBatchCost
2026-08-14 17:33:07 -07:00
mateo-berri
05beb7abb5 fix(proxy): emit uncached input cost so component headers sum to the total 2026-08-14 17:30:50 -07:00
mateo-berri
7e539405ed fix(cost-tracking): price web search on dated search-preview map entries 2026-08-14 17:29:56 -07:00
yuneng-jiang
0f2566e823
Merge pull request #36984 from BerriAI/litellm_/key-info-badge-styling-b293cc
fix(ui): match the MCP servers count badge to its sibling permission badges
2026-08-14 17:26:31 -07:00
Mateo Wang
a5038661c9
Merge pull request #34850 from BerriAI/litellm_lit_4866_anthropic_geo_cache_uplift
fix(anthropic cost): apply regional geo uplift to cached tokens
2026-08-14 17:26:17 -07:00
Mateo Wang
b9d70b5ef5
Merge pull request #34860 from BerriAI/litellm_lit_4868_cache_write_split
fix(anthropic): aggregate 5m/1h cache-write split across iterations path
2026-08-14 17:25:20 -07:00
mateo-berri
7562445273 test(proxy): assert production nesting semantics for component cost headers 2026-08-14 17:24:10 -07:00
Mateo Wang
d74cb6de1b
Merge pull request #36798 from BerriAI/litellm_azure_ai_docs_index_write_grant_rc
fix(azure_ai): recognize real Search doc endpoints so teams can read/write via passthrough
2026-08-14 17:24:06 -07:00
yucheng-berri
f9f5c03884
fix(mcp): drop caller host and configured upstream headers from logged metadata (#36901)
* fix(mcp): drop caller host and configured upstream headers from logged metadata

The synthetic request that carries MCP client headers into
add_litellm_data_to_request forwarded the caller's Host header, and
Request.url is built from it, so a caller chose the proxy_server_request
url and the metadata endpoint that every logging callback records.

_upstream_credential_headers also only knew the configured client side
auth header and the x-mcp- prefix family, so a header name declared in
mcp_servers.<name>.extra_headers reached logging metadata in cleartext.
Those names are admin chosen, so no prefix rule can recognize them; read
them off the server registry instead. The header is still forwarded
upstream, which is what extra_headers is for. authorization is left out
because clean_headers already strips it and claiming it here would move
authenticated_with_header on the oauth passthrough config.

The Responses bridge tests stub the server manager, so their fakes gain
the registry accessor the sanitizer now reads.

* fix(mcp): drop caller host from the sanitized header mapping too

The synthetic request stopped forwarding host, but the parallel sanitizer
did not, so a forged hostname still reached the guardrail payload and the
list_tools spend row. Drop it there as well.

Exempt the configured identity headers from the upstream credential set.
get_user_from_headers resolves end user attribution off the same request
this module reconstructs, and it only fills end_user_id when auth left it
unset, so claiming user_header_name or a user_header_mappings name would
lose attribution on the MCP paths that authenticate upstream.

Drop the isinstance guard on extra_headers entries: the field is typed
list[str], so the check is dead and basedpyright scores it.

* fix(mcp): accept a bare user_header_mappings entry when exempting identity headers

get_internal_user_header_from_mapping and get_customer_user_header_from_mapping
both normalize a single mapping to a one element list, and config_settings.md
documents the key as a dict. Iterating the bare form yields its keys instead,
so the exemption silently matched nothing and an identity header also named in
an MCP server's extra_headers was dropped after all.
2026-08-14 17:21:07 -07:00
mateo-berri
ff547be3e3 fix(cost-tracking): count dict-shaped web_search_call output items 2026-08-14 17:19:37 -07:00
mateo-berri
e8c1fe8b11 fix(bedrock): fall back to the batch deployment model for unmapped record models 2026-08-14 17:11:24 -07:00
Ilan Chemla
f99d0a4b38
feat(search): add Nimble as a search provider (#36347)
* feat(search): add Nimble as a search provider

Adds `NimbleSearchConfig` so `search_provider: nimble` works across the SDK,
the proxy /v1/search endpoint, the Search Tools dashboard, and spend tracking.

Nimble's /v2/search already uses the Perplexity unified spec's parameter names,
so the request transform is close to a pass-through. `search_domain_filter`
splits into include_domains/exclude_domains on the spec's `-` prefix, `country`
is upper-cased to the ISO form Nimble documents, and everything else is
forwarded so focus, search_depth, time_range and the rest stay reachable. On the
response side, snippet prefers `content` and falls back to `description`, and a
malformed body raises an attributed error rather than reporting an empty search.

Also tightens `BaseSearchConfig.get_supported_perplexity_optional_params` to
return `frozenset[str]` instead of a bare mutable `set`, which every caller
already treats as read-only.

* fix(search): surface Nimble error bodies instead of empty results

Greptile flagged that a null or absent `results` degraded to a successful empty
search. A search with no hits comes back as `"results": []`, verified against the
live API, so the field is now required and anything else raises the attributed
schema error the other malformed bodies already take.

Also unwraps Nimble's second error envelope. Collection failures return
`{"success", "task_id", "message"}` rather than the `{"detail"}` shape validation
errors use, and only the latter was being read.

Drops comments that restated the adjacent code.

* docs(search): drop the Nimble param list from the transform docstring

It restated the vendor's API reference, which the module docstring already links,
and would go stale the moment Nimble adds a focus mode.
2026-08-14 17:09:58 -07:00
Shivam Rawat
4110812b52
Merge pull request #33881 from BerriAI/litellm_fix_redis_spend_buffer_requeue_33872
fix(proxy): requeue Redis spend buffer transactions when the DB commit fails
2026-08-14 17:08:05 -07:00
mateo-berri
b3729c50b0 fix(fireworks_ai): move top-level thinking into extra_body on the text completion path 2026-08-14 17:06:14 -07:00
tin-berri
2d3c3e3098
feat(shadow_eval): add reverse-direction shadow eval jobs (#36865)
Shadow eval only answered "should this key adopt this auto-router". Once a key
is on the router it is invisible to the feature, because the sampling gate skips
any request the shadowed router already served, so post-adoption quality
regressions go unmeasured.

Reverse mode inverts the arms: sample the traffic the router did serve and
duplicate it against a fixed baseline_model, judged by the same blind pairwise
judge. Same job table, same attempt rows, same aggregates.

real_* stays the arm the caller was served and shadow_* the duplicated one, so
in reverse real_model is the router's pick and shadow_model is the baseline. The
active-job slot becomes one per (key, direction) so both directions can run at
once, and tier attribution in reverse reads the control request's routing
decision rather than the shadow call's write-back.
2026-08-14 17:05:55 -07:00
mateo-berri
0e0c3ce0d8 Merge remote-tracking branch 'origin/litellm_internal_staging' into devin_ai_lit_5013_web_search_cost 2026-08-14 17:05:08 -07:00
mateo-berri
4193445647 Merge remote-tracking branch 'origin/litellm_internal_staging' into devin_ai_lit_5013_web_search_cost
# Conflicts:
#	litellm/model_prices_and_context_window_backup.json
#	model_prices_and_context_window.json
#	tests/test_litellm/litellm_core_utils/prompt_templates/test_bedrock_converse_strict_tools_opus_47_48.py
2026-08-14 17:05:08 -07:00
tin-berri
d4d6bc2577
fix(proxy): serve aggregate MCP endpoint on bare /mcp instead of 307-redirecting (#34845)
The MCP sub-app is attached with app.mount("/mcp", ...) and a Starlette
mount never matches its bare prefix, so POST /mcp fell through to the
router's redirect_slashes 307. Behind a TLS-terminating ingress whose
peer address is not in uvicorn's forwarded-allow-ips (default: loopback
only) the redirect Location is built from the socket scheme as http://,
and MCP clients strip the Authorization header on the cross-origin
follow, so reconnects fail with ECONNRESET right after a successful
OAuth flow. The redirect also fires before auth, so the bare spelling
never returns the RFC 9728 WWW-Authenticate challenge that OAuth
clients need to start the flow.

Add an explicit /mcp route beside the existing /toolset/{name}/mcp and
/{name}/mcp spellings, forwarding to handle_streamable_http_mcp with
the same scope rewrite those routes already use (path=/mcp,
_original_path preserved for OAuth challenge URL selection). When the
mcp package is unavailable the route 404s, matching what the bare
sub-app serves on /mcp/ in that state. /mcp/, /mcp/{server},
/{server}/mcp and /toolset/{name}/mcp spellings are unchanged; the
exact-match route and the mount have disjoint match sets so
registration order cannot matter.
2026-08-14 17:04:32 -07:00
mateo-berri
9079e4c47b fix(proxy): return cost breakdown header values as a named tuple 2026-08-14 17:04:26 -07:00
mateo-berri
9a1e63c9f0 fix(caching): tolerate SSE chunk splits in anthropic stream cache writer 2026-08-14 17:04:25 -07:00
daleselaji-dev
80c37bfe3a fix(bedrock): resolve aliases in batch file records 2026-08-14 17:04:24 -07:00
mateo-berri
b14c4a8d45 fix(vector_stores): classify write endpoints before reads on substring collisions 2026-08-14 16:53:48 -07:00
Louis Vauterin
f9704497fb
feat(helm): add startupProbe and hpa.behavior to the componentized chart (#36382)
Two small pod-spec passthroughs the componentized chart was missing, both
additive and empty by default so existing renders are unchanged:

- gateway/backend/ui deployments gain a `startupProbe` knob (same
  `{{- with }}` toYaml pattern as liveness/readiness), to gate liveness during
  a slow cold start without a kill loop.
- gateway/backend/ui HPAs gain an `hpa.behavior` passthrough rendered verbatim
  under spec.behavior (scaleUp/scaleDown policies + stabilization windows).

Tests: extend probe_tests.yaml (startupProbe absent by default / renders
verbatim) and add hpa_behavior_tests.yaml. Full chart suite: 76 tests pass.

Signed-off-by: Louis Vauterin <louis.vauterin@doctolib.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-14 16:53:10 -07:00
Yassin Kortam
eb4b847268
fix(proxy): always emit the Anthropic /v1/models token limits, null when unknown (#36961)
Anthropic's Models API declares max_input_tokens and max_tokens as nullable, not
optional, and the live vendor endpoint returns both keys on every entry. The
merged Anthropic-native listing dropped either key whenever LiteLLM could not
resolve a limit, so a client validating against a nullable-but-required schema
saw a malformed entry for any model the cost map does not know.
2026-08-14 16:52:39 -07:00
Yassin Kortam
2959465ea0
fix(openai,azure): return a length-truncated 200 when the output budget fits no token (#36859)
OpenAI and Azure GPT-5.x answer a chat request whose output budget cannot fit a
single visible token with a 400, while the same models return a length-truncated
200 one or two tokens higher. Agents that probe a model with a hardcoded
max_tokens of 1 read that 400 as "model unavailable".

The four chat request helpers now recognise the provider's own sentence and hand
back the length-truncated response the provider gives at a slightly larger
budget: finish_reason "length", empty content, zero completion tokens. Any other
400 still raises. Streaming is covered by the same seam, and the caller's budget
is never raised on their behalf.

The provider bills the prompt it processed but sends no usage object with the
400, so the prompt tokens are estimated with the same token_counter every other
usage-less path uses. Reporting zero would let a caller send an arbitrarily
large prompt with max_tokens 1 and be charged nothing.
2026-08-14 16:51:41 -07:00
mateo-berri
9027ab9485 Merge remote-tracking branch 'origin/litellm_internal_staging' into fireworks_nim_vllm_compat 2026-08-14 16:45:49 -07:00
mateo-berri
e94a97fcfc fix(cost_calculator): mirror the anthropic geo uplift in the token-type cost breakdown 2026-08-14 16:44:22 -07:00
mateo-berri
0ab23f5ce9 fix(anthropic): bill undetailed iteration cache writes at the 5m rate 2026-08-14 16:41:34 -07:00
Yuneng Jiang
94e943144e
refactor(ui): drop the explanatory comment from the badge variant test 2026-08-14 16:40:16 -07:00
Yuneng Jiang
f9f016dc1d
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/key-info-badge-styling-b293cc 2026-08-14 16:36:32 -07:00
Yuneng Jiang
61334ec94a
fix(ui): match the MCP servers count badge to its sibling permission badges
The Object Permissions section rendered the MCP Servers badge with shadcn's
default variant (solid bg-primary), so a plain count showed up as a black pill
next to the light Vector Stores and Agents counts. Counts now use secondary
everywhere, and destructive stays reserved for the blocked state.
2026-08-14 16:32:27 -07:00
Mateo Wang
870a8cf764
Merge pull request #36974 from BerriAI/litellm_vllm_dropdown_labels
fix(ui): distinguish hosted and local vLLM in the provider dropdown
2026-08-14 16:28:57 -07:00
devin-ai-integration[bot]
40d999b693
fix(mcp): keep admin-entered oauth endpoints in management reads (#36888)
* fix(mcp): keep admin-entered oauth endpoints in management reads

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): cover configured oauth endpoints on the config load path

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-14 16:27:37 -07:00
tin-berri
3783bbf2bf
fix(ui): show zeroed auto-router usage stats when a window has no sessions (#36868)
* fix(ui): show zeroed auto-router usage stats when a window has no sessions

* test(ui): assert the muted track on the empty share-of-turns bar
2026-08-14 23:25:58 +00:00
mateo-berri
3b2ed3c018 fix(fireworks_ai): let extra_body thinking/reasoning_effort take precedence over chat_template_kwargs 2026-08-14 16:25:32 -07:00
yuneng-jiang
652f4cb8e4
Merge pull request #36982 from BerriAI/litellm_/revert-36837-ui-regression
Revert "fix(auth): stop the team fallback from widening model access" (#36837)
2026-08-14 16:19:17 -07:00
yucheng-berri
2fc39cde18
fix(langfuse): gate update_trace_keys behind an operator setting (#36862)
update_trace_keys lets a caller name which request metadata entries get copied
onto an existing trace, and the name is unrestricted. Sending
update_trace_keys: ["user_api_key_auth"] with existing_trace_id serializes the
resolved auth object, including the team callback credentials it carries, onto
the trace through Langfuse.trace(**trace_params). TraceBody is Extra.allow, so
an unexpected key ships rather than being dropped.

Any holder of a team key can do this and read the result in the destination the
team already logs to, so the feature is now inert unless an operator turns it on
with langfuse_enable_update_trace_keys.
2026-08-14 16:05:28 -07:00