Commit graph

43197 commits

Author SHA1 Message Date
Anmol Jaiswal
0059b497f4
fix(model_map): flag native structured outputs on Anthropic-direct claude-sonnet-5 and claude-haiku-4-5 (#35930)
* fix(model_map): flag native structured outputs on Anthropic-direct claude-sonnet-5 and claude-haiku-4-5

The Bedrock twins of both models already carry
supports_native_structured_output, but the Anthropic-direct entries do not,
so response_format requests to anthropic/claude-sonnet-5 and
anthropic/claude-haiku-4-5 fall back to the json_tool_call emulation and
inherit its nested-envelope failure modes (#8898) despite the API supporting
output_format natively.

Verified live against the Anthropic API on 2026-08-05: both models accept
output_format (structured outputs beta header) and return exact schema
instances, including a large nested production schema validated with
pydantic. Same two lines applied to the bundled backup map.

* fix(model_map): cover the versioned claude-haiku-4-5-20251001 alias

Exact-match capability lookup of anthropic/claude-haiku-4-5-20251001
resolved the versioned entry, which lacked the flag, so response_format
for that identifier still took the tool-emulation path. Flag it in both
the root and bundled maps, matching its unversioned alias.

* fix(anthropic): bound $defs inlining in output_format with the shared schema-bomb budget

map_response_format_to_anthropic_output_format called unpack_defs with
no max_inlined_bytes, so an authenticated caller could send a compact
schema whose repeated $refs expand without bound before reaching the
provider. Reuse the existing 10MB inlining budget (renamed from
_LEGACY_DEFS_MAX_INLINED_BYTES to DEFS_MAX_INLINED_BYTES now that two
call sites share it); overflow raises ValueError instead of
materialising the expansion.

Regression tests: a compact schema bomb is rejected, a normal $defs
schema still resolves; the bomb test fails when the bound is removed.

* chore: retrigger CI (benchmarks job flaked on a PyPI download timeout)

---------

Co-authored-by: Anmol Jaiswal <anmolg1997@users.noreply.github.com>
2026-08-15 11:51:35 -07:00
devin-ai-integration[bot]
fe9451c6cd
fix(panw_prisma_airs): surface scan_id on allowed requests (#37037)
* fix(panw_prisma_airs): surface scan_id and scan metadata on allowed requests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style: ruff format panw guardrail

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(panw_prisma_airs): expose scan id header only

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(panw_prisma_airs): inject http client instead of patching private api

Adds an http_client seam so the scan-id tests drive the real AIRS request/parse path through a mock transport, plus direct coverage for the scan-id header helper.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): expose guardrail scan id header to browser clients

Keeps the panw optional_fields block untouched to avoid a needless conflict with a sibling PR that deletes it.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-15 11:49:03 -07:00
Mateo Wang
fb3459d78c
Merge pull request #36788 from BerriAI/litellm_model_map_deprecation_refresh
fix(model_prices): add gemini 3.1 flash tts preview and legacy OpenAI shutdown dates
2026-08-15 11:48:39 -07:00
Yassin Kortam
5a50fe0b46
feat(proxy): gate the Global Control Plane worker registry on an enterprise license (#36996)
The Global Control Plane (formerly documented as the HA Control Plane) is
documented as an Enterprise feature, but `worker_registry` carried no premium
check, so any OSS install could run one. Gate it at config load, matching the
`enforced_params` precedent, and fail startup rather than ignoring the registry
silently: a silently dropped registry degrades a control plane into an ordinary
proxy with no signal to the operator.

Also declare `worker_registry` and `general_settings.control_plane_url`, both
load bearing today and neither previously declared, so they appear in the
generated config schema.
2026-08-15 11:40:04 -07:00
yuneng-jiang
3840970613
Merge pull request #37045 from BerriAI/litellm_bump_0815
chore: bump litellm-enterprise 0.1.55 -> 0.1.56, litellm-proxy-extras 0.4.85 -> 0.4.86
2026-08-15 11:25:52 -07:00
Yuneng Jiang
f07eafb686
bump: litellm-enterprise 0.1.55 -> 0.1.56, litellm-proxy-extras 0.4.85 -> 0.4.86 2026-08-15 11:10:10 -07:00
yuneng-jiang
487da4d1e4
Merge pull request #37021 from BerriAI/litellm_models_source_column
fix(ui): rename models table Status column to Source
2026-08-15 10:55:06 -07:00
Mateo Wang
87abb8781e
Merge pull request #36925 from BerriAI/litellm_model_registry_lifecycle_audit_20260814 2026-08-15 09:58:31 -07:00
yuneng-jiang
f1acd6d375
Merge pull request #37022 from BerriAI/litellm_user_teams_walkable
feat(ui): link user detail team names to team pages
2026-08-15 09:44:24 -07:00
yuneng-jiang
d4a52f9fa5
Merge pull request #36991 from BerriAI/litellm_/blocked-badge-styling-c03b31
refactor(ui): re-sync badge and skeleton onto the base-vega shadcn style
2026-08-15 09:24:26 -07:00
Yuneng Jiang
09741740d4
test(ui): follow the models table Status to Source column rename 2026-08-15 09:20:08 -07:00
yuneng-jiang
759cb4f24b
Merge pull request #37033 from BerriAI/litellm_access_group_create_rhf
refactor(ui): migrate access group create modal to RHF + zod + shadcn
2026-08-15 09:15:23 -07:00
ryan-crabbe-berri
8c991076e3 refactor(ui): migrate access group create modal to RHF + zod + shadcn 2026-08-15 08:13:16 -07:00
ryan-crabbe-berri
1b2f05d6b0 feat(ui): link user detail team names to team pages 2026-08-15 03:18:23 -07:00
ryan-crabbe-berri
aacab940f4 fix(ui): rename models table Status column to Source 2026-08-15 03:15:53 -07:00
mateo-berri
32917b07fc fix(model_prices): drop xai/grok-4.6-latest, xAI does not serve that alias
xAI's chat completions API answers grok-4.6-latest with "Model not found"
and its language-models listing shows no alias for grok-4.6
2026-08-15 01:37:23 -07:00
Devin AI
30b14597af fix(model_prices): revert unverified Gemini deprecation dates
Google's Gemini deprecations page lists no shutdown date for the 2.5 GA models and May 14, 2028 for gemini-embedding-001; keep only the DeepSeek V4 max output correction.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-15 08:35:58 +00:00
mateo-berri
16e6aa2e9f fix(model_prices): correct deprecation dates flagged in review
Revert gemini-embedding-001 to its published 2028-05-14 shutdown, move
gpt-4-turbo-preview to the 2026-03-26 shutdown of the gpt-4-0125-preview
snapshot it aliases, and drop the unannounced Gemini 2.5 shutdown dates
2026-08-15 01:33:01 -07:00
Mateo Wang
7a5b98e6a9
Merge pull request #37005 from BerriAI/litellm_claudemd_lit012
docs(claude): require ReadOnly on every TypedDict field (LIT012)
2026-08-15 01:27:50 -07:00
Mateo Wang
3798df392a
chore: make it more concise 2026-08-15 01:13:16 -07:00
Mateo Wang
d70cc14981
Merge pull request #35092 from BerriAI/litellm_vertex_batch_embeddings_translation
fix(vertex_ai): translate /v1/embeddings batch rows to the Gemini embedding shape
2026-08-14 21:52:32 -07:00
Mateo Wang
89f233a15d
Merge pull request #36720 from BerriAI/litellm_tiered_pricing_cache_creation
fix(cost): tiered pricing supports cache creation cost and is all-or-nothing
2026-08-14 21:51:54 -07:00
mateo
1abde19280 docs(claude): require ReadOnly on every TypedDict field (LIT012)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-15 04:24:18 +00:00
mateo-berri
18752c860c fix(cost): honor explicit zero tier rates and skip synthesized backend output rates 2026-08-14 21:05:22 -07:00
devin-ai-integration[bot]
6e7984e537
fix(proxy): requeue spend logs when the DB write fails with a transport error (#36716)
* fix(proxy): requeue spend logs when the DB write fails with a transport error

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): hardcode the spend log queue cap and drop the stale re-export

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): keep the spend log requeue within the type discipline budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): apply the spend log queue cap to producer appends too

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): lower the spend log queue cap to 1k and make it env configurable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): bound the spend log queue by bytes instead of row count

A row cap cannot bound memory: a row carries the whole prompt under store_prompts_in_spend_logs, so a cap that rides out an outage of counter-only rows is an OOM once prompts are stored. Every enqueue and dequeue now goes through one pair that tracks what the queue costs and drops the oldest rows past a 64 MB budget.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): make the spend log queue byte budget env configurable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): use a string default for the spend log queue byte budget env read

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): make the spend log queue byte total a public attribute

The queue it accounts for is already public, and a private name only bought reportPrivateUsage errors at every call site.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
2026-08-14 20:49:45 -07:00
mateo
e46f2ca62f fix(dashscope): honor the model reasoning rate when a tier omits output rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-15 03:42:23 +00:00
mateo-berri
34918d34f9 fix(cost): inherit the backend output rate when a deployment's tiers omit one 2026-08-14 20:37:18 -07:00
Mateo Wang
6c2dcb801b
Merge pull request #36907 from guptaishaan/fix/issue-36880-8933
fix: report real token usage on guardrail-blocked /v1/responses replies
2026-08-14 20:16:25 -07:00
mateo
c3e38a0b52 fix(cost): fall back to the model output rate when a tier omits one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-15 03:05:13 +00:00
Mateo Wang
1a183efaa1
Merge pull request #36159 from daleselaji-dev/codex/bedrock-alias-36156
Some checks are pending
CodSpeed Benchmarks / benchmarks (push) Waiting to run
Publish basedpyright base counts / publish (push) Waiting to run
Code Quality Checks / code-quality (push) Waiting to run
UI Unit Tests / ui-unit-tests (push) Waiting to run
Unit Tests: Core Utilities / core-utils (push) Waiting to run
Unit Tests: Documentation Validation / documentation (push) Waiting to run
Unit Tests: Enterprise, Google GenAI & Routing / enterprise-routing (push) Waiting to run
Unit Tests: Integrations (Callbacks & Logging) / integrations (push) Waiting to run
Unit Tests: LLM Provider Transformations / Vertex AI (push) Waiting to run
Unit Tests: LLM Provider Transformations / All Other Providers (push) Waiting to run
Unit Tests: MCP, Secrets, Containers & Misc / misc (push) Waiting to run
Unit Tests: Proxy Auth & Key Management / proxy-auth (push) Waiting to run
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy API Endpoints / proxy-endpoints (push) Waiting to run
Unit Tests: Proxy API Endpoints / proxy-server (push) Waiting to run
Unit Tests: Proxy Infrastructure / proxy-infra (push) Waiting to run
Unit Tests: Responses, Caching & Types / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
fix(bedrock): resolve aliases in batch file records
2026-08-14 18:20:21 -07:00
Mateo Wang
70d51a196f
Merge pull request #35286 from BerriAI/devin_ai_lit_5013_web_search_cost
fix(cost): track OpenAI/Azure web search tool cost per call
2026-08-14 18:16:59 -07:00
Mateo Wang
cba2beaf42
Merge pull request #36965 from erensh27/feat/per-component-cost-headers
feat(proxy): per-component response cost headers
2026-08-14 18:15:23 -07:00
mateo
3f64cbe41b fix(ptu): empty a PTU deployment's tiered_pricing instead of dropping it
Dropping it falls back to the public cost map's tier table, whose rates outrank the
zeros written beside them, so a PTU deployment on a tiered model keeps billing its
traffic per token. Stored empty, the tiers no longer apply and the zeros win
2026-08-15 01:14:14 +00:00
yuneng-jiang
a4aab2e9f3
Merge pull request #36995 from BerriAI/litellm_/playground-config-select-25ad99
fix(ui): anchor chips-combobox popups to the field instead of the inner input
2026-08-14 18:11:38 -07:00
mateo
9079844fa2 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_tiered_pricing_cache_creation 2026-08-15 01:05:01 +00:00
yuneng-jiang
491e2ffb1b
Merge pull request #36993 from BerriAI/litellm_/scrollable-tabs-layout-6aa9f7
fix(ui): stop the models tab strip from scrolling vertically
2026-08-14 18:04:42 -07:00
mateo-berri
5970754a85 fix(ptu): clear a PTU deployment's tiered_pricing instead of zeroing it
tiered_pricing is a list, so the 0.0 the flat-rate zeroing stores does not
even validate. Supplying tiers alongside PTU config gets the same 400 as a
flat rate; tiers already stored are dropped from both blobs
2026-08-14 18:04:01 -07:00
mateo-berri
0176e4b3f6 refactor(cost): make the shared token-details parsers public
parse_prompt_tokens_details and parse_completion_tokens_details are imported
by four modules, so the leading underscore made every import a
reportPrivateUsage violation
2026-08-14 18:04:01 -07:00
Yuneng Jiang
c7084c04c0
test(ui): assert which element the chips-combobox popup anchors to
The previous assertion read data-chips, which is derived from the anchor prop
being truthy, so it stayed true even when the ref never reached the DOM and the
popup was still anchored to the inner input.

Stub distinct widths on the chips container and the input, then read the width
the positioner resolved. Reverting the anchor wiring now reports the input's
width instead of the field's, which is the actual bug.
2026-08-14 18:03:34 -07:00
Yuneng Jiang
6e38e9490d
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/playground-config-select-25ad99 2026-08-14 17:57:27 -07:00
Yuneng Jiang
654fab224a
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/scrollable-tabs-layout-6aa9f7 2026-08-14 17:55:49 -07:00
Yuneng Jiang
a0e5c7e818
fix(ui): anchor chips-combobox popups to the field instead of the inner input
Base UI positions a combobox popup against the Combobox.Input by default. In
chips mode the visible field is the ComboboxChips wrapper and the input is a
smaller box nested inside it, so every chips-combobox in the dashboard opened
its popup 11px right of the field and 17px past its right edge.

shadcn ships the wiring for this and their combobox-multiple example uses it:
useComboboxAnchor on the chips container, passed to ComboboxContent as anchor.
The anchor prop also drives data-chips, which cancels the extra min-width an
ordinary combobox wants. Every chips site in the dashboard omitted it.

The anchor is attached through Base UI's render prop rather than a plain ref,
because React 18 drops refs on function components and ComboboxChips is one.

Adds MultiSelect's first test, covering the anchor wiring plus selection,
chip rendering and custom values.
2026-08-14 17:51:54 -07:00
Mateo Wang
40a418440c
Merge pull request #35969 from milesadkins/fireworks_nim_vllm_compat
feat(fireworks_ai): translate NIM/vLLM extra params to Fireworks-native args
2026-08-14 17:47:47 -07:00
Ahmed N
691c7fd4d6
fix(anthropic_messages): make tool_result images visible to OpenAI-compatible providers (#34462)
Images nested inside an Anthropic `tool_result` block were dropped when the
request was adapted for an OpenAI-compatible provider, because the OpenAI tool
message shape only carried text. Hoist those images out of the tool result and
into a following user message so the model can still see them, and widen the
tool message content type to accept image parts.
2026-08-14 17:47:38 -07:00
mateo-berri
2c6409c7e6 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_tiered_pricing_cache_creation 2026-08-14 17:44:39 -07:00
Yuneng Jiang
5219658b8f
fix(ui): stop the models tab strip from scrolling vertically
The tab strip carried overflow-x-auto directly on the TabsList. CSS forces
overflow-y from visible to auto once overflow-x is not visible, and the line
variant's active-tab underline is an absolutely positioned ::after that hangs
5px below its trigger, so the strip picked up a pixel of vertical scroll on top
of the horizontal scroll it actually wants.

The scroll container now lives on a wrapper whose bottom padding leaves room for
the underline, offset by a matching negative margin so the row keeps its exact
geometry.
2026-08-14 17:44:00 -07:00
Yuneng Jiang
b1696b3edf
test(ui): assert cache and retry tags by text instead of class name
Three assertions in LogDetailContent.test.tsx matched a regex against the
rendered class string to prove a tag was green or was not red. That pins styling
rather than behavior, and jsdom does not resolve the utilities anyway, so the
checks only ever proved that a substring survived into the class attribute

The badge re-sync exposed it: base-vega's base string carries aria-invalid
variants of the destructive token, so a "not destructive" regex started matching
every badge regardless of variant

Each one now asserts the tag's text is present, which is what the surrounding
cases already do and what the user actually observes
2026-08-14 17:38:47 -07:00
mateo-berri
de77711cf9 test(vertex_ai): cover duplicated fan-out rows in embeddings batch reassembly
Some checks failed
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Also ruff-formats the batch transformation test file, which the formatter
gate flags once the file is touched.
2026-08-14 17:37:07 -07:00
Mateo Wang
118523ede6
Merge pull request #34581 from BerriAI/litellm_anthropic_messages_response_cache
fix(caching): cache anthropic /v1/messages responses, including streaming
2026-08-14 17:34:08 -07:00
Mateo Wang
dc92749c07
Merge pull request #35360 from BerriAI/devin_ai_fix_batch_cost_completed_no_output
fix(batches): mark terminal batch with no output file as processed in CheckBatchCost
2026-08-14 17:33:07 -07:00