Commit graph

43876 commits

Author SHA1 Message Date
Mateo Wang
cba2beaf42
Merge pull request #36965 from erensh27/feat/per-component-cost-headers
feat(proxy): per-component response cost headers
2026-08-14 18:15:23 -07:00
mateo
3f64cbe41b fix(ptu): empty a PTU deployment's tiered_pricing instead of dropping it
Dropping it falls back to the public cost map's tier table, whose rates outrank the
zeros written beside them, so a PTU deployment on a tiered model keeps billing its
traffic per token. Stored empty, the tiers no longer apply and the zeros win
2026-08-15 01:14:14 +00:00
yuneng-jiang
a4aab2e9f3
Merge pull request #36995 from BerriAI/litellm_/playground-config-select-25ad99
fix(ui): anchor chips-combobox popups to the field instead of the inner input
2026-08-14 18:11:38 -07:00
mateo
9079844fa2 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_tiered_pricing_cache_creation 2026-08-15 01:05:01 +00:00
yuneng-jiang
491e2ffb1b
Merge pull request #36993 from BerriAI/litellm_/scrollable-tabs-layout-6aa9f7
fix(ui): stop the models tab strip from scrolling vertically
2026-08-14 18:04:42 -07:00
mateo-berri
5970754a85 fix(ptu): clear a PTU deployment's tiered_pricing instead of zeroing it
tiered_pricing is a list, so the 0.0 the flat-rate zeroing stores does not
even validate. Supplying tiers alongside PTU config gets the same 400 as a
flat rate; tiers already stored are dropped from both blobs
2026-08-14 18:04:01 -07:00
mateo-berri
0176e4b3f6 refactor(cost): make the shared token-details parsers public
parse_prompt_tokens_details and parse_completion_tokens_details are imported
by four modules, so the leading underscore made every import a
reportPrivateUsage violation
2026-08-14 18:04:01 -07:00
Yuneng Jiang
c7084c04c0
test(ui): assert which element the chips-combobox popup anchors to
The previous assertion read data-chips, which is derived from the anchor prop
being truthy, so it stayed true even when the ref never reached the DOM and the
popup was still anchored to the inner input.

Stub distinct widths on the chips container and the input, then read the width
the positioner resolved. Reverting the anchor wiring now reports the input's
width instead of the field's, which is the actual bug.
2026-08-14 18:03:34 -07:00
Yuneng Jiang
6e38e9490d
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/playground-config-select-25ad99 2026-08-14 17:57:27 -07:00
Yuneng Jiang
654fab224a
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/scrollable-tabs-layout-6aa9f7 2026-08-14 17:55:49 -07:00
Yuneng Jiang
a0e5c7e818
fix(ui): anchor chips-combobox popups to the field instead of the inner input
Base UI positions a combobox popup against the Combobox.Input by default. In
chips mode the visible field is the ComboboxChips wrapper and the input is a
smaller box nested inside it, so every chips-combobox in the dashboard opened
its popup 11px right of the field and 17px past its right edge.

shadcn ships the wiring for this and their combobox-multiple example uses it:
useComboboxAnchor on the chips container, passed to ComboboxContent as anchor.
The anchor prop also drives data-chips, which cancels the extra min-width an
ordinary combobox wants. Every chips site in the dashboard omitted it.

The anchor is attached through Base UI's render prop rather than a plain ref,
because React 18 drops refs on function components and ComboboxChips is one.

Adds MultiSelect's first test, covering the anchor wiring plus selection,
chip rendering and custom values.
2026-08-14 17:51:54 -07:00
Mateo Wang
40a418440c
Merge pull request #35969 from milesadkins/fireworks_nim_vllm_compat
feat(fireworks_ai): translate NIM/vLLM extra params to Fireworks-native args
2026-08-14 17:47:47 -07:00
Ahmed N
691c7fd4d6
fix(anthropic_messages): make tool_result images visible to OpenAI-compatible providers (#34462)
Images nested inside an Anthropic `tool_result` block were dropped when the
request was adapted for an OpenAI-compatible provider, because the OpenAI tool
message shape only carried text. Hoist those images out of the tool result and
into a following user message so the model can still see them, and widen the
tool message content type to accept image parts.
2026-08-14 17:47:38 -07:00
mateo-berri
2c6409c7e6 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_tiered_pricing_cache_creation 2026-08-14 17:44:39 -07:00
Yuneng Jiang
5219658b8f
fix(ui): stop the models tab strip from scrolling vertically
The tab strip carried overflow-x-auto directly on the TabsList. CSS forces
overflow-y from visible to auto once overflow-x is not visible, and the line
variant's active-tab underline is an absolutely positioned ::after that hangs
5px below its trigger, so the strip picked up a pixel of vertical scroll on top
of the horizontal scroll it actually wants.

The scroll container now lives on a wrapper whose bottom padding leaves room for
the underline, offset by a matching negative margin so the row keeps its exact
geometry.
2026-08-14 17:44:00 -07:00
Yuneng Jiang
b1696b3edf
test(ui): assert cache and retry tags by text instead of class name
Three assertions in LogDetailContent.test.tsx matched a regex against the
rendered class string to prove a tag was green or was not red. That pins styling
rather than behavior, and jsdom does not resolve the utilities anyway, so the
checks only ever proved that a substring survived into the class attribute

The badge re-sync exposed it: base-vega's base string carries aria-invalid
variants of the destructive token, so a "not destructive" regex started matching
every badge regardless of variant

Each one now asserts the tag's text is present, which is what the surrounding
cases already do and what the user actually observes
2026-08-14 17:38:47 -07:00
mateo-berri
de77711cf9 test(vertex_ai): cover duplicated fan-out rows in embeddings batch reassembly
Some checks failed
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Also ruff-formats the batch transformation test file, which the formatter
gate flags once the file is touched.
2026-08-14 17:37:07 -07:00
Mateo Wang
118523ede6
Merge pull request #34581 from BerriAI/litellm_anthropic_messages_response_cache
fix(caching): cache anthropic /v1/messages responses, including streaming
2026-08-14 17:34:08 -07:00
Mateo Wang
dc92749c07
Merge pull request #35360 from BerriAI/devin_ai_fix_batch_cost_completed_no_output
fix(batches): mark terminal batch with no output file as processed in CheckBatchCost
2026-08-14 17:33:07 -07:00
Yuneng Jiang
0da672c8e2
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/blocked-badge-styling-c03b31 2026-08-14 17:31:48 -07:00
Yuneng Jiang
2de3195757
refactor(ui): re-sync badge and skeleton onto the base-vega shadcn style
components.json has declared "style": "base-vega" since cfe9e39e55, but badge
and skeleton were added a few days earlier under new-york and never re-synced,
so both still carried the previous style's classes. Badge's destructive variant
rendered as solid red with white text instead of the tinted wash the rest of the
dashboard uses, which is already the convention for Button

Re-runs npx shadcn add for both and keeps the two local deltas the registry
cannot supply: cva comes from @/lib/cva.config, since class-variance-authority
is not a dependency here, and both stay wrapped in React.forwardRef, which the
tripwire in tests/setupTests.ts requires until the React 19 upgrade

Adds Badge to ref-forwarding.test.tsx. Nothing covered it before, even though
two TooltipTrigger sites compose over it, so the wrapper could have been dropped
by the next re-sync without a single test going red

Retargets one assertion in LogDetailContent.test.tsx. It regex-matched the whole
class string for "destructive" to prove a tag was not alarming red, which the
restored aria-invalid classes now satisfy for every variant; it checks the
variant attribute and red utility classes instead
2026-08-14 17:31:44 -07:00
mateo-berri
05beb7abb5 fix(proxy): emit uncached input cost so component headers sum to the total 2026-08-14 17:30:50 -07:00
mateo-berri
7e539405ed fix(cost-tracking): price web search on dated search-preview map entries 2026-08-14 17:29:56 -07:00
Devin AI
87765fcc76 fix(router): merge model_info without new mutable constructions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-15 00:29:50 +00:00
milan
0d0c712df7 fix(vertex_ai): fail an embeddings batch entry whose fan-out came back incomplete
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-15 00:29:35 +00:00
yuneng-jiang
0f2566e823
Merge pull request #36984 from BerriAI/litellm_/key-info-badge-styling-b293cc
fix(ui): match the MCP servers count badge to its sibling permission badges
2026-08-14 17:26:31 -07:00
mateo-berri
688575e5bf fix(cost): bill reasoning tokens at the selected tier's reasoning rate 2026-08-14 17:26:29 -07:00
Mateo Wang
a5038661c9
Merge pull request #34850 from BerriAI/litellm_lit_4866_anthropic_geo_cache_uplift
fix(anthropic cost): apply regional geo uplift to cached tokens
2026-08-14 17:26:17 -07:00
Mateo Wang
b9d70b5ef5
Merge pull request #34860 from BerriAI/litellm_lit_4868_cache_write_split
fix(anthropic): aggregate 5m/1h cache-write split across iterations path
2026-08-14 17:25:20 -07:00
mateo-berri
7562445273 test(proxy): assert production nesting semantics for component cost headers 2026-08-14 17:24:10 -07:00
Mateo Wang
d74cb6de1b
Merge pull request #36798 from BerriAI/litellm_azure_ai_docs_index_write_grant_rc
fix(azure_ai): recognize real Search doc endpoints so teams can read/write via passthrough
2026-08-14 17:24:06 -07:00
mateo-berri
3217b8edae fix(proxy): count worker heartbeats on the primary so replica lag cannot undercount 2026-08-14 17:23:06 -07:00
mateo-berri
eafddaaa12 feat(scripts): queue heavy gates behind a machine-wide slot lock 2026-08-14 17:22:32 -07:00
mateo-berri
c9685b2a6d fix(router): honor tiered_pricing set in a deployment's litellm_params 2026-08-14 17:21:15 -07:00
yucheng-berri
f9f5c03884
fix(mcp): drop caller host and configured upstream headers from logged metadata (#36901)
* fix(mcp): drop caller host and configured upstream headers from logged metadata

The synthetic request that carries MCP client headers into
add_litellm_data_to_request forwarded the caller's Host header, and
Request.url is built from it, so a caller chose the proxy_server_request
url and the metadata endpoint that every logging callback records.

_upstream_credential_headers also only knew the configured client side
auth header and the x-mcp- prefix family, so a header name declared in
mcp_servers.<name>.extra_headers reached logging metadata in cleartext.
Those names are admin chosen, so no prefix rule can recognize them; read
them off the server registry instead. The header is still forwarded
upstream, which is what extra_headers is for. authorization is left out
because clean_headers already strips it and claiming it here would move
authenticated_with_header on the oauth passthrough config.

The Responses bridge tests stub the server manager, so their fakes gain
the registry accessor the sanitizer now reads.

* fix(mcp): drop caller host from the sanitized header mapping too

The synthetic request stopped forwarding host, but the parallel sanitizer
did not, so a forged hostname still reached the guardrail payload and the
list_tools spend row. Drop it there as well.

Exempt the configured identity headers from the upstream credential set.
get_user_from_headers resolves end user attribution off the same request
this module reconstructs, and it only fills end_user_id when auth left it
unset, so claiming user_header_name or a user_header_mappings name would
lose attribution on the MCP paths that authenticate upstream.

Drop the isinstance guard on extra_headers entries: the field is typed
list[str], so the check is dead and basedpyright scores it.

* fix(mcp): accept a bare user_header_mappings entry when exempting identity headers

get_internal_user_header_from_mapping and get_customer_user_header_from_mapping
both normalize a single mapping to a one element list, and config_settings.md
documents the key as a dict. Iterating the bare form yields its keys instead,
so the exemption silently matched nothing and an identity header also named in
an MCP server's extra_headers was dropped after all.
2026-08-14 17:21:07 -07:00
mateo-berri
ff547be3e3 fix(cost-tracking): count dict-shaped web_search_call output items 2026-08-14 17:19:37 -07:00
mateo-berri
f2a10f6331 merge: litellm_internal_staging into litellm_vertex_batch_embeddings_translation 2026-08-14 17:17:16 -07:00
mateo-berri
e8c1fe8b11 fix(bedrock): fall back to the batch deployment model for unmapped record models 2026-08-14 17:11:24 -07:00
mateo-berri
ed33687422 feat(proxy): auto-suppress the no-Redis banner for confirmed single-worker deployments 2026-08-14 17:11:05 -07:00
Ilan Chemla
f99d0a4b38
feat(search): add Nimble as a search provider (#36347)
* feat(search): add Nimble as a search provider

Adds `NimbleSearchConfig` so `search_provider: nimble` works across the SDK,
the proxy /v1/search endpoint, the Search Tools dashboard, and spend tracking.

Nimble's /v2/search already uses the Perplexity unified spec's parameter names,
so the request transform is close to a pass-through. `search_domain_filter`
splits into include_domains/exclude_domains on the spec's `-` prefix, `country`
is upper-cased to the ISO form Nimble documents, and everything else is
forwarded so focus, search_depth, time_range and the rest stay reachable. On the
response side, snippet prefers `content` and falls back to `description`, and a
malformed body raises an attributed error rather than reporting an empty search.

Also tightens `BaseSearchConfig.get_supported_perplexity_optional_params` to
return `frozenset[str]` instead of a bare mutable `set`, which every caller
already treats as read-only.

* fix(search): surface Nimble error bodies instead of empty results

Greptile flagged that a null or absent `results` degraded to a successful empty
search. A search with no hits comes back as `"results": []`, verified against the
live API, so the field is now required and anything else raises the attributed
schema error the other malformed bodies already take.

Also unwraps Nimble's second error envelope. Collection failures return
`{"success", "task_id", "message"}` rather than the `{"detail"}` shape validation
errors use, and only the latter was being read.

Drops comments that restated the adjacent code.

* docs(search): drop the Nimble param list from the transform docstring

It restated the vendor's API reference, which the module docstring already links,
and would go stale the moment Nimble adds a focus mode.
2026-08-14 17:09:58 -07:00
Shivam Rawat
4110812b52
Merge pull request #33881 from BerriAI/litellm_fix_redis_spend_buffer_requeue_33872
fix(proxy): requeue Redis spend buffer transactions when the DB commit fails
2026-08-14 17:08:05 -07:00
mateo-berri
e06e2e6c12 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_36907_head 2026-08-14 17:06:18 -07:00
mateo-berri
b3729c50b0 fix(fireworks_ai): move top-level thinking into extra_body on the text completion path 2026-08-14 17:06:14 -07:00
tin-berri
2d3c3e3098
feat(shadow_eval): add reverse-direction shadow eval jobs (#36865)
Shadow eval only answered "should this key adopt this auto-router". Once a key
is on the router it is invisible to the feature, because the sampling gate skips
any request the shadowed router already served, so post-adoption quality
regressions go unmeasured.

Reverse mode inverts the arms: sample the traffic the router did serve and
duplicate it against a fixed baseline_model, judged by the same blind pairwise
judge. Same job table, same attempt rows, same aggregates.

real_* stays the arm the caller was served and shadow_* the duplicated one, so
in reverse real_model is the router's pick and shadow_model is the baseline. The
active-job slot becomes one per (key, direction) so both directions can run at
once, and tier attribution in reverse reads the control request's routing
decision rather than the shadow call's write-back.
2026-08-14 17:05:55 -07:00
mateo-berri
0e0c3ce0d8 Merge remote-tracking branch 'origin/litellm_internal_staging' into devin_ai_lit_5013_web_search_cost 2026-08-14 17:05:08 -07:00
mateo-berri
4193445647 Merge remote-tracking branch 'origin/litellm_internal_staging' into devin_ai_lit_5013_web_search_cost
# Conflicts:
#	litellm/model_prices_and_context_window_backup.json
#	model_prices_and_context_window.json
#	tests/test_litellm/litellm_core_utils/prompt_templates/test_bedrock_converse_strict_tools_opus_47_48.py
2026-08-14 17:05:08 -07:00
mateo-berri
2f6f5c4961 fix(cost): reach tiered pricing for models without top-level per-token rates 2026-08-14 17:04:48 -07:00
tin-berri
d4d6bc2577
fix(proxy): serve aggregate MCP endpoint on bare /mcp instead of 307-redirecting (#34845)
The MCP sub-app is attached with app.mount("/mcp", ...) and a Starlette
mount never matches its bare prefix, so POST /mcp fell through to the
router's redirect_slashes 307. Behind a TLS-terminating ingress whose
peer address is not in uvicorn's forwarded-allow-ips (default: loopback
only) the redirect Location is built from the socket scheme as http://,
and MCP clients strip the Authorization header on the cross-origin
follow, so reconnects fail with ECONNRESET right after a successful
OAuth flow. The redirect also fires before auth, so the bare spelling
never returns the RFC 9728 WWW-Authenticate challenge that OAuth
clients need to start the flow.

Add an explicit /mcp route beside the existing /toolset/{name}/mcp and
/{name}/mcp spellings, forwarding to handle_streamable_http_mcp with
the same scope rewrite those routes already use (path=/mcp,
_original_path preserved for OAuth challenge URL selection). When the
mcp package is unavailable the route 404s, matching what the bare
sub-app serves on /mcp/ in that state. /mcp/, /mcp/{server},
/{server}/mcp and /toolset/{name}/mcp spellings are unchanged; the
exact-match route and the mount have disjoint match sets so
registration order cannot matter.
2026-08-14 17:04:32 -07:00
mateo-berri
e4f2ea12bc fix(responses_api): map bridged chat usage on guardrail-blocked replies
Move the blocked-usage mapping for /v1/responses next to
blocked_response_usage in guardrail_translation utils, map bridged chat
prompt/completion tokens to Responses API input/output tokens, and let
raise_passthrough_exception attach the blocked response so post-call
guardrail blocks report real usage
2026-08-14 17:04:27 -07:00
mateo-berri
9079e4c47b fix(proxy): return cost breakdown header values as a named tuple 2026-08-14 17:04:26 -07:00