Commit graph

48585 commits

Author SHA1 Message Date
devin-ai-integration[bot]
eddfb5fb20
fix(responses): preserve hosted web search calls (#40828)
* fix(responses): preserve hosted web search calls

Co-Authored-By: Claude Code <noreply@anthropic.com>
(cherry picked from commit 09183b3346)

* chore: remove unrelated generated schema documentation changes

(cherry picked from commit ca6a860757)

* fix(responses): preserve hosted search context during replay

(cherry picked from commit 425f1e9b3a)

* chore: regenerate dashboard API types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Tin Chi Lo <tin@berri.ai>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 09:52:19 -07:00
devin-ai-integration[bot]
1c61c2606e
perf(proxy): one MGET and one pipeline for post-call spend counters, no team/user/org refetch on the response path (#40841)
* perf(auth): prefetch user, team, membership, org and project in one MGET, one query and one pipeline

Auth read each object with its own Redis GET and, on a miss, its own DB
query, then the admission spend counters with one GET each. The prefetch
warms every entry the checks read with one MGET, one raw query for the
Redis misses and one pipeline write, and a per-request batch serves the
spend counter reads from one MGET. The per-object getters stay the
readers and the fallback, so enforcement does not depend on the prefetch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(auth): keep prefetch and spend batch collections immutable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(auth): let the cold spend-counter reseed reuse the admission MGET instead of one GET per counter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(auth): prefetch referenced auth objects only after the key's model access check passes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(auth): give the prefetch-ordering test's patches their test-quality reasons

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(auth): move the real-Postgres prefetch join test to the proxy_behavior shard

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): read NULL nested permission and budget lists as [] in the prefetch join

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(proxy): batch post-call spend counter reads and carry budget state through the request

Post-call warm checks, reservation reads and reconcile reads for one request now go through a task-local spend counter batch: one MGET answers every counter, successful increments write their result back into the batch so no second Redis read follows, and invalidation forgets the key. RedisCache.async_increment sends INCRBYFLOAT and its TTL command in one pipeline round trip.

Auth pins frozen team, user and org budget snapshots on UserAPIKeyAuth, the pre-call setup writes them into the request metadata, and Prometheus reads them back instead of calling get_key_object, get_team_object, get_user_object and get_org_object on the response path. The getters stay as the fallback for requests that carried nothing (custom auth, unauthenticated routes, skipped checks).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(proxy): reconcile the budget reservation and the post-call warm checks from one MGET and one pipeline

A scope opened inside an open spend counter batch binds into it instead of starting its own, so the reservation reconcile and the post-call warm checks share the request's single MGET. The reconcile reads every reserved counter concurrently, sends the consistent adjustments in one INCRBYFLOAT+EXPIRE pipeline and settles a flushed or reseeded counter on its own afterwards, keeping the pre-call resize fail-closed. PendingSpendIncrement moves to spend_counter_batch so budget_reservation can build a pipeline without importing a private name

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(proxy): drop the dataclass import left behind by the PendingSpendIncrement move

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(types): import Self from typing_extensions so the proxy imports on Python 3.10

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): use a neutral organization alias in the carried budget state tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): cover recorded and forgotten spend counter values in the request batch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): assert async_set_cache_pipeline_with_ttls keeps per-entry TTLs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): type the reservation entry carried through reconcile adjustments

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): map the model table's aliases column to model_aliases in the prefetch join and read user memberships the way get_user_object does

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 16:05:50 +00:00
devin-ai-integration[bot]
f2e0a5db1e
perf(auth): read user, team, membership, org, project and spend counters in one MGET, one query and one pipeline (#40834)
* perf(auth): prefetch user, team, membership, org and project in one MGET, one query and one pipeline

Auth read each object with its own Redis GET and, on a miss, its own DB
query, then the admission spend counters with one GET each. The prefetch
warms every entry the checks read with one MGET, one raw query for the
Redis misses and one pipeline write, and a per-request batch serves the
spend counter reads from one MGET. The per-object getters stay the
readers and the fallback, so enforcement does not depend on the prefetch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(auth): keep prefetch and spend batch collections immutable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(auth): let the cold spend-counter reseed reuse the admission MGET instead of one GET per counter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(auth): prefetch referenced auth objects only after the key's model access check passes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(auth): give the prefetch-ordering test's patches their test-quality reasons

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(auth): move the real-Postgres prefetch join test to the proxy_behavior shard

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): read NULL nested permission and budget lists as [] in the prefetch join

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): assert async_set_cache_pipeline_with_ttls keeps per-entry TTLs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): map the model table's aliases column to model_aliases in the prefetch join and read user memberships the way get_user_object does

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 08:52:23 -07:00
yuneng-jiang
9071ca503e
Merge pull request #40837 from BerriAI/litellm_user-model-budget-clear
fix(proxy): persist clearing user model budgets
2026-09-11 23:43:25 -07:00
yuneng-jiang
e88ba94da5
Merge pull request #40838 from BerriAI/litellm_fix_ocr_native_vcr
test(ocr): exempt native parity requests from cassette replay
2026-09-11 23:01:13 -07:00
Yuneng Jiang
f44d5ef101
fix(proxy): preserve model budget update compatibility 2026-09-11 22:59:46 -07:00
Yuneng Jiang
aecc961016
test(ocr): exempt native parity requests from cassette replay 2026-09-11 22:50:39 -07:00
Yuneng Jiang
f96af80a0c
fix(proxy): persist clearing user model budgets 2026-09-11 22:49:12 -07:00
Mateo Wang
98d46ee59d
Merge pull request #40765 from BerriAI/litellm_fix_health_scoped_results_and_serialization
fix(proxy): expand access groups in /health scoping and allowlist health display fields
2026-09-11 21:21:44 -07:00
yuneng-jiang
1be89d28b8
Merge pull request #39990 from BerriAI/litellm_e2e_jwt_harness
test(e2e): reusable JWT fixtures and management lifecycle coverage
2026-09-11 21:18:59 -07:00
Mateo Wang
7ad6c628de
Merge pull request #40773 from BerriAI/litellm_e2e_memory_regression_failing_requests
test(e2e): memory regression test for failing requests on the release gate
2026-09-11 20:46:33 -07:00
mateo-berri
5ab6e0858c fix(health): resolve a model name the way a request routes before matching the provider model string 2026-09-11 20:40:00 -07:00
Mateo Wang
99e14fc2e5
Merge pull request #40798 from BerriAI/litellm_lit7523_mantle_reasoning_summary
fix(bedrock_mantle): gate reasoning.summary on the OpenAI Responses path
2026-09-11 20:30:49 -07:00
joshua-berri
70cf348aa5
Merge pull request #40791 from BerriAI/litellm_fix_mcp_root_discovery_6634
fix(mcp): use gateway authentication for root discovery
2026-09-11 20:23:10 -07:00
joshua-berri
e86adf98ac
Merge pull request #40808 from BerriAI/litellm_fix_mcp_oauth_issuer_7078
fix(mcp): match per-server OAuth metadata issuers
2026-09-11 20:23:00 -07:00
joshua-berri
108cacdd2f
Merge pull request #40790 from BerriAI/litellm_fix_mcp_discovery_cache_6585
fix(mcp): cache upstream discovery lists
2026-09-11 20:16:36 -07:00
mateo-berri
eaf3d8ad3e fix(health): target a model name the way a request for it routes
A team's copies published under the name win, then deployments named that way, then a public name only another team's deployment carries (an admin reaches it, routing does too). The endpoint resolver and the live narrowing share one rule.
2026-09-11 20:12:39 -07:00
Mateo Wang
19c8553052
Merge pull request #40820 from BerriAI/litellm_sanitize_unknown_model_error_message
fix(proxy): keep the raw model string out of the unknown-model spend-log error message
2026-09-11 19:54:34 -07:00
mateo-berri
6209f0694b fix(health): keep team public names to the owning team and let model_id win over model
A public name a team publishes its own deployment copy under now targets that copy only for a caller from that team, so an admin or another team probing the shared name gets the global deployment alone

model_id wins when paired with model: a foreign id still gets the 403, and an id no deployment carries gets the 404 of the lone-id path, before any probe runs or a result is stored under it

cache_health_check_results accepts the Mapping sequences perform_health_check returns
2026-09-11 19:49:37 -07:00
mateo-berri
9ba7ec2964 test(e2e): keep the memory regression case in a class and drop the helper docstrings 2026-09-11 19:46:06 -07:00
ryan-crabbe-berri
e65b7ff0b8
Merge pull request #40659 from BerriAI/litellm_team_member_table_search_sort_filter
feat(ui): search, sort and role filter for the team member table
2026-09-11 19:37:29 -07:00
devin-ai-integration[bot]
98f6c14a76
test(pgbouncer): stop the never-listens replacement test flaking under CI load (#40830) 2026-09-11 19:32:56 -07:00
mateo-berri
a8ffc852f2 test(e2e): trim the reliability helper docstrings to the cooldown rationale 2026-09-11 19:32:40 -07:00
mateo-berri
9375719feb Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_e2e_memory_regression_failing_requests
# Conflicts:
#	tests/e2e/CLAUDE.md
#	tests/e2e/models.py
2026-09-11 19:31:11 -07:00
mateo-berri
6a96a47e9f style(proxy): drop the docstring bullet and wrap the sanitizer test call 2026-09-11 19:29:53 -07:00
ryan-crabbe-berri
c1aaaae4e3
Merge pull request #40831 from BerriAI/litellm_dismissible_env_credential_banner 2026-09-11 19:23:23 -07:00
Mateo Wang
55c34951e3
Merge pull request #40609 from csoni-cweave/fix/guardrail-len-mismatch
fix(guardrails): fail closed with a named error when a Responses input rewrite cannot be applied
2026-09-11 19:22:53 -07:00
ryan
2db51046d4 fix(ui): make the env-credential login warning banner dismissible
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 02:20:32 +00:00
mateo-berri
3dafa7aac1 test(proxy): expect the sanitized unknown-model message in the spend-log error test 2026-09-11 19:18:10 -07:00
devin-ai-integration[bot]
b5bf09d22d
fix(db): carry DATABASE_SSLMODE/DATABASE_SSLROOTCERT into the assembled writer and reader URLs (#40815)
* fix(db): carry DATABASE_SSLMODE/DATABASE_SSLROOTCERT into the assembled writer and reader URLs

The componentized gateway supervisor starts the in-container PgBouncer from the
DATABASE_URL assembled out of the discrete DATABASE_* vars before config.yaml is
read, so an IAM URL had no way to request verified TLS: PgBouncer dialed the
server with server_tls_sslmode = prefer (no SNI, no verification) and public
RDS endpoints rejected the handshake. Two new env vars, exposed by the chart as
database.writer.sslMode / sslRootCert, are appended as libpq sslmode/sslrootcert
to every writer and reader URL the settings assemble (never to a pinned URL),
then translated for Prisma as before. Token refresh now also carries Prisma's
sslmode/sslcert/sslaccept over into the re-minted URL

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(db): keep TLS params on the CLI password URL and the initial IAM reader mint

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(db): treat DATABASE_SSLROOTCERT on its own as verify-full and cover collector and migrations TLS env

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(db): type the reader mint TLS test double and drop its mutable capture

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 02:14:39 +00:00
devin-ai-integration[bot]
1fde15c1ec
fix(shadow-eval): skip hosted web search samples (#40827)
(cherry picked from commit a78cd2fe02)

Co-authored-by: Tin Chi Lo <tin@berri.ai>
2026-09-12 02:11:48 +00:00
devin-ai-integration[bot]
7057b2f6c4
fix(fireworks_ai): keep reasoning_content on replayed assistant messages (#40682)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-11 18:47:37 -07:00
mateo-berri
e4030597d8 test(guardrails): prove the structured write-back lands CrowdStrike redactions on Responses instructions and tool items 2026-09-11 18:47:18 -07:00
Mateo Wang
eed11ba59b
Merge pull request #40812 from BerriAI/litellm_fireworks_deepseek_v4p1_flash
feat(model_prices): add DeepSeek V4.1 Flash on Fireworks
2026-09-11 18:43:00 -07:00
mateo-berri
832da2950f fix(proxy): keep the raw model string out of the unknown-model spend-log error message 2026-09-11 18:42:08 -07:00
yuneng-jiang
cf97b757a4
fix(ui): preserve cleared shared select values (#40795)
Preserve explicit null when shared selectors clear and adapt affected forms, validation, and request payloads. Clear stale dependent relationships and retain required-selection checks. Document project detachment, user model-budget clearing, and routing-compression clearing as deferred follow-ups.
2026-09-11 18:37:08 -07:00
Mateo Wang
44ce8bb1ef
Merge pull request #40769 from BerriAI/litellm_azure_realtime_ga_default
fix(realtime): dial Azure's GA realtime upstream for GA clients
2026-09-11 18:34:20 -07:00
devin-ai-integration[bot]
f84f986b4e
fix(guardrails): keep post_call guardrail info on streamed chat completions (#40806)
* fix(guardrails): sync logging_obj guardrail info on every record so post_call entries survive streamed chat completions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(guardrails): hoist regression test imports to module scope

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-11 18:16:33 -07:00
mateo-berri
105dc77109 fix(realtime): probe Azure's GA realtime upstream in health checks when no protocol is pinned 2026-09-11 18:14:21 -07:00
ryan-crabbe-berri
e72d1cfa8d
Merge pull request #40572 from dclarksymmetry/fix/spend-counter-reseed-double-count
fix(proxy): prevent spend counter double counting
2026-09-11 18:14:04 -07:00
Joshua Valluru
9d31de2f20 fix(mcp): bound discovery cache result bytes 2026-09-11 18:08:27 -07:00
ryan-crabbe-berri
50cd26cd9c
Merge pull request #40396 from jon-walton/litellm_user_budget_webhook_alerts
fix(proxy): emit internal user budget webhook alerts
2026-09-11 18:03:35 -07:00
mateo
6c07876dcf test(model_prices): use local cost map for Fireworks cost coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 01:00:11 +00:00
Joshua Valluru
05d2c316f5 refactor(mcp): reuse in-memory discovery storage 2026-09-11 17:53:51 -07:00
mateo
115535c3ad test(model_prices): cover Fireworks DeepSeek V4.1 costs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-12 00:49:26 +00:00
ryan-crabbe-berri
b27d2cce77
feat(ui): link the Organization and Deleted By cells on Deleted Teams (#40751)
* feat(ui): link the Organization and Deleted By cells on Deleted Teams

Both columns rendered as plain text, so tracing a deleted team back to its
org or to whoever removed it meant copying an id into another page's search
box. Route them through IdentityCell with orgDetailHref and userDetailHref.
Team ID stays unlinked because the team itself is gone.

Claude-Session: https://claude.ai/code/session_01NfwfQhamRNnSqgXMUjf3h4

* test(ui): mount a router mock for the Deleted Teams page test

The page test renders the table, and the newly linked cells call useRouter,
which throws without an App Router mounted. Matches how the other 35 test
files in the suite stub next/navigation.

Claude-Session: https://claude.ai/code/session_01NfwfQhamRNnSqgXMUjf3h4
2026-09-11 17:49:07 -07:00
ryan-crabbe-berri
c53f72c764
feat(ui): link the Created By cell on the Prompts page (#40753)
The column was plain muted text, so finding out who owns a prompt meant
copying the id into the Users page search box. Route it through
IdentityCell with userDetailHref, which keeps the proxy admin placeholder
unlinked.

Claude-Session: https://claude.ai/code/session_01NfwfQhamRNnSqgXMUjf3h4
2026-09-11 17:49:01 -07:00
ryan-crabbe-berri
06964e5603
feat(ui): link the User ID, Created By and Deleted By cells on Deleted Keys (#40750)
* feat(ui): link the User ID, Created By and Deleted By cells on Deleted Keys

All three columns rendered as plain text, so auditing a deleted key meant
copying an id into the Users page search box. Route them through
IdentityCell with userDetailHref, which keeps the proxy admin placeholder
unlinked. User Email and Team Alias stay as they are: the deleted key table
has no column for either, so the API never populates them.

Claude-Session: https://claude.ai/code/session_01NfwfQhamRNnSqgXMUjf3h4

* test(ui): mount a router mock for the Deleted Keys page test

The page test renders the table, and the newly linked cells call useRouter,
which throws without an App Router mounted. Matches how the other 35 test
files in the suite stub next/navigation.

Claude-Session: https://claude.ai/code/session_01NfwfQhamRNnSqgXMUjf3h4
2026-09-11 17:48:57 -07:00
ryan-crabbe-berri
1be930664f
feat(ui): link the User ID and Team ID cells on the Memory page (#40752)
Both columns rendered as dead pills, so tracing a memory row back to its
owner meant copying an id into another page's search box. IdCell grows an
href prop that turns the pill into a client-routed link, and the Memory
columns pass the shared entityLinks helpers so the proxy admin and
dashboard sentinels stay unlinked.

Claude-Session: https://claude.ai/code/session_01NfwfQhamRNnSqgXMUjf3h4
2026-09-11 17:48:44 -07:00
kerry-berri
b957c25241
Merge pull request #40811 from BerriAI/litellm_pr_template_assumptions
docs(pr-template): note untested assumptions under Caveats
2026-09-11 17:47:54 -07:00