Commit graph

55202 commits

Author SHA1 Message Date
yucheng-berri
dbd9862940 chore: evidence head-flow-545ae28d7bb.webp 2026-10-30 11:55:57 -07:00
yucheng-berri
bbf4acd720 chore: evidence head-applied-545ae28d7bb.png 2026-10-30 11:55:54 -07:00
yucheng-berri
62e5870ed4 chore: evidence base-flow-79f004516ce.webp 2026-10-30 11:55:51 -07:00
yucheng-berri
cd20c40c47 chore: evidence base-applied-79f004516ce.png 2026-10-30 11:55:49 -07:00
yucheng-berri
8ca1bda154 chore: evidence head-flow-cf863890e59.webp 2026-10-30 02:05:41 -07:00
yucheng-berri
f93a60a41c chore: evidence head-applied-cf863890e59.png 2026-10-30 02:05:39 -07:00
yucheng-berri
09e3261c83 chore: evidence head-flow-dcf0f624006.webp 2026-10-29 23:31:30 -07:00
yucheng-berri
ec2aa1d938 chore: evidence head-applied-dcf0f624006.png 2026-10-29 23:31:28 -07:00
yucheng-berri
6373e00222 chore: evidence head-flow-c9843958792.webp 2026-10-29 08:18:53 -07:00
yucheng-berri
8bd2d6dfcc chore: evidence head-applied-c9843958792.png 2026-10-29 08:18:51 -07:00
yucheng-berri
5c260f9743 chore: evidence head-flow-076e3e374c1.webp 2026-10-28 13:23:20 -07:00
yucheng-berri
8362e0e887 chore: evidence head-applied-076e3e374c1.png 2026-10-28 13:23:18 -07:00
yucheng-berri
bd6da4a222 chore: evidence head-flow-27cb942c58c.webp 2026-10-28 12:51:54 -07:00
yucheng-berri
c0971c6037 chore: evidence head-applied-27cb942c58c.png 2026-10-28 12:51:52 -07:00
yucheng-berri
10a918763b chore: evidence head-flow-5434ce4675f.webp 2026-10-27 17:52:12 -07:00
yucheng-berri
aef9ed6fa7 chore: evidence head-applied-5434ce4675f.png 2026-10-27 17:50:26 -07:00
yucheng-berri
9522bc9de2 chore: evidence head-flow-0676e6a460f.webp 2026-10-27 15:18:55 -07:00
yucheng-berri
97b7d91f26 chore: evidence head-applied-0676e6a460f.png 2026-10-27 15:18:53 -07:00
yucheng-berri
143e33424d chore: evidence base-flow.webp 2026-10-26 15:30:57 -07:00
yucheng-berri
6f7f1e342d chore: evidence head-flow.webp 2026-10-26 15:30:52 -07:00
yucheng-berri
82e0aa3b72 chore: evidence head-applied 2026-10-26 14:51:31 -07:00
yucheng-berri
23ccbd2649 chore: evidence base-applied 2026-10-26 14:51:28 -07:00
tin-berri
ccd783d907
fix(proxy): reject negative RPM and TPM request limits (#48444) 2026-10-26 21:16:15 +00:00
tin-berri
6d93445350
fix(proxy): preserve limiter state across SSE keepalive tasks (#48452) 2026-10-26 14:09:19 -07:00
tin-berri
b7a2226f43
test(proxy): make tokenizer offload check deterministic (#48458) 2026-10-26 20:58:51 +00:00
devin-ai-integration[bot]
c843cee062
feat(pricing): add github_copilot/gpt-6.1-terra (#48450)
* feat(pricing): add github_copilot/gpt-6.1-terra

* fix(pricing): raise github_copilot gpt-6 max_input_tokens to 922000

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 20:55:51 +00:00
moe-berri
3beee3cfe9
fix(elevenlabs): map wav and pcm TTS formats to 24kHz output formats (#48455)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 13:42:01 -07:00
devin-ai-integration[bot]
768960610b
feat(cost-map): add baseten elephant-1 and mammoth-1 decision models (#48453)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 20:41:58 +00:00
shrey-berri
9cb102b721
chore(deps): remove expired boto3 botocore and grpcio overrides (#48447) 2026-10-26 13:41:15 -07:00
tin-berri
f96200bcea
fix(vertex_ai): bound shared credential cache retention (#48442) 2026-10-26 12:56:13 -07:00
devin-ai-integration[bot]
52c11bf383
feat(bedrock): add global. and us. openai.gpt-6.1-terra rows (#48441)
* feat(bedrock): add global. and us. openai.gpt-6.1-terra rows

* chore: rerun ci after flaky event loop timing test

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 19:39:48 +00:00
devin-ai-integration[bot]
c69e82f538
chore(cost-map): add openai batch and ultrafast 12-hour cache write rates from the pricing page (#48433)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 12:18:19 -07:00
devin-ai-integration[bot]
dab39d7402
fix(azure): set azure_ai/Microsoft-Decision-1 max_input_tokens to 32768 (#48434)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 12:09:14 -07:00
mrinal-berri
07a82a287b
fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276)
* fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text

The pass-through guardrail translation json-dumps the request body as the
scan text when no request_fields are configured. The unified guardrail hook
injects litellm_metadata into that body first, so the dump carried the
caller's key metadata. Exclude it alongside metadata and litellm_logging_obj;
vendors still receive it through request_data.

* test(guardrails): type the pass-through scan-text tests per AGENTS.md
2026-10-26 12:07:58 -07:00
tin-berri
85871f791c
fix(ui): render string inputs in general settings (#48431) 2026-10-26 11:50:31 -07:00
tin-berri
1ecc782986
fix(vertex_ai): reuse rerank credential cache (#48430) 2026-10-26 11:49:50 -07:00
yucheng-berri
38f318e736
fix(otel): decide per-team Arize sampling from the trace id so every pod agrees (#47468)
* fix(otel): decide per-team Arize sampling from the trace id so every pod agrees

* test(otel): cover per-team sampling across pods, workers, endpoints and restarts

* fix(otel): remember only a failed tree's sampling verdict so a later failed call answers to the error rate

A successful tree's verdict follows from the trace id alone, so remembering it only made a later failed call of the same trace follow the success rate on the pod that served both, while other pods judged it at the error rate

* test(otel): flush every worker before the restart cell kills it and cover a failed call after a dropped one

* fix(otel): decide team sampling per request, not per trace

Calls sharing a caller's trace id are separate requests, keyed by their root span.
Each is decided by the trace hash at the rate for its own outcome, and only its own
late spans follow its verdict, so no pod applies an earlier call's verdict to a later
call of the trace

* fix(otel): forget a span's request root when it ends, not when newer spans start

The root of each open span was kept in a first-in-first-out table of 16384 entries,
so a span two levels below the root that stayed open while the worker started more
spans than that lost its entry and ended under its parent's id. Its request then
never closed and was held from every sampled destination until shutdown.

Each open span now keeps its entry until it ends, and a span starting under a parent
that has already ended reads the root from the parent span itself

* fix(otel): decide a request as its spans end once it falls out of the open span table

A request whose span was pushed out of the table of open spans, because the worker
held more than 16384 open at once, ended that span under another key and was held
from every sampled destination until shutdown. Pushing a span out now drops its
request's open count too, so the request is decided as each of its spans ends, the
way one pushed out of the open request table already is

* test(otel): annotate the team sampling test locals as Final

* test(otel): build the caller's remote parent in one helper instead of rebinding it per trace

* fix(otel): keep each open span's request until it ends, bounded by the open requests

The span table had its own bound of 16384, so a worker holding more spans open than
that pushed out the oldest, and a failure in such a span no longer reached its
request. Each request now holds the ids of its open spans, and a span's entry in the
lookup goes when the span ends or when its request is pushed out of the open request
table, the same bound main already applies. A failure anywhere in a tracked request
reaches its verdict however many spans are open

---------

Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-26 11:44:30 -07:00
Kerry Lu
1c022d1d06
chore(cost-map): add openai gpt-6.1-terra from the pricing page (#48429)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 11:31:26 -07:00
tin-berri
4c674e8e3f
fix(mcp): report oversized SSE catalog errors (#48424) 2026-10-26 11:18:23 -07:00
tin-berri
be3a7be1db
fix(ui): restore clipboard copying in key and log details (#48425) 2026-10-26 11:12:11 -07:00
Arsh Goyal
96792d0913
test(streaming): pin that a stream stuck on one chunk closes its provider stream and logs one failure (#38753)
* fix(streaming): stop re-raising and re-logging a failed stream on every next()

A stream that raised keeps its provider iterator alive, so a consumer that iterates past the error re-detected, re-raised and re-logged the same failure on every call, flooding failure callbacks for one request. Latch the failure and re-raise it without logging again.

* fix(streaming): drop the retained traceback before re-raising a failed stream

* test(streaming): count failures through a real logging callback and require a clean stop on later reads

* test(streaming): require the stuck stream to close early and release its socket

The provider stream now records a GeneratorExit and caps its chunks, so draining it or detecting the loop late fails. Two wire cases check a sync and an async stuck stream release their TCP session.

* test(streaming): avoid failure recorder name collision

---------

Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
2026-10-26 18:03:37 +00:00
yuneng-jiang
e0982d6248
build(deps): bump webpki-roots and webpki-root-certs to 1.0.11 (#48301)
1.0.11 drops emSign Root CA - C1 and emSign ECC Root CA - C3 from the built-in roots
2026-10-26 10:10:57 -07:00
yujonglee
ebf829c2fa
fix: support Python 3.15 (#48359)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-26 16:00:11 +00:00
yuneng-jiang
dddbaa803d
build(docker): refresh base images and ship only runtime files in component images (#48279)
Some checks failed
unit / router (push) Has been cancelled
unit / rust-bridge-harness (push) Has been cancelled
unit / auth-checks (push) Has been cancelled
unit / budgets (push) Has been cancelled
unit / custom-logging (push) Has been cancelled
unit / db-and-spend (push) Has been cancelled
unit / endpoints-and-responses (push) Has been cancelled
unit / enterprise-routing (push) Has been cancelled
unit / guardrails-hooks (push) Has been cancelled
unit / jwt-and-keys (push) Has been cancelled
unit / key-generation (push) Has been cancelled
unit / logging-misc (push) Has been cancelled
unit / mcp-oauth (push) Has been cancelled
unit / proxy-runtime (push) Has been cancelled
unit / proxy-server-core (push) Has been cancelled
unit / root (push) Has been cancelled
unit / proxy-utils (push) Has been cancelled
unit / core-utils (push) Has been cancelled
unit / redis-py 5.3.1 compatibility (push) Has been cancelled
unit / redis-py 6.4.0 compatibility (push) Has been cancelled
unit / redis-py 7.4.1 compatibility (push) Has been cancelled
unit / redis-py 8.0.1 compatibility (push) Has been cancelled
unit / unit passed (push) Has been cancelled
unit / enterprise-managed-files (push) Has been cancelled
unit / enterprise-repositories-secrets (push) Has been cancelled
unit / proxy-server (push) Has been cancelled
unit / lens-python-310 (push) Has been cancelled
unit / caching-local (push) Has been cancelled
unit / endpoints (push) Has been cancelled
unit / coverage (push) Has been cancelled
* build(docker): move to newer wolfi-base and nginx base images

Bump the pinned wolfi-base digest in every Wolfi Dockerfile to 360896ef, which
ships brotli 1.2.1-r1 and ca-certificates-bundle 20260909-r3. The previous
digest already carried libbrotli 1.2.0-r6 as an OpenSSL dependency, and
`apk add --no-cache` never upgrades a package the base already has

Move the UI image to nginx 1.31.8-alpine3.24, whose CA bundle is the 20261020
Mozilla set, and upgrade brotli-libs in its runtime stage since no nginx
digest ships 1.2.1 yet

* build(docker): copy only runtime files into the gateway, backend and migrations images

The component runtime stages copied the builder's whole /app, which is the
entire repository plus build output: six lockfiles, helm, terraform, docs,
.env.example and the Rust build tree. Measured at runtime with an audit hook
on a booted container, the gateway reads only gateway/, litellm/ and
enterprise/ from /app, the backend only backend/, litellm/ and enterprise/,
and the migrations job only litellm/ and run.py. The pyproject.toml read is
prisma-python's optional [tool.prisma] lookup, which ours does not define

Copy just those, plus .venv, the entrypoint script and schema.prisma for the
gateway and backend. PYTHONPATH=/app and the in-Dockerfile asserts are
unchanged
2026-10-24 22:33:45 -07:00
moe-berri
72f1ac4a27
revert(proxy): revert large-integer request parsing (#48289)
Reverts d3b70c42ed because the JSON scanner causes excessive memory use and blocks concurrent requests on escape-heavy payloads
2026-10-25 03:57:49 +00:00
Jeevan Babu Gotru
d3b70c42ed
fix(proxy): keep JSON integers outside the 64-bit range exact when parsing request bodies (#48213)
Preserve oversized JSON integers in proxy requests and model rewrites
Parse large bodies in a worker thread to avoid blocking request handling

Fixes #47947

Co-authored-by: Jeevan Babu Gotru <114808955+jeevan1098@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
2026-10-24 20:15:46 -07:00
Kerry Lu
de5dc5b238
feat(mistral): add zai-glm-5-3-flash to the cost map (#48270)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 18:20:36 -07:00
Hazelhof
2e52e0fb2a
fix(deepseek): pass reasoning_effort through to the API (#38736)
* fix(deepseek): pass reasoning_effort through to the API

DeepSeekChatConfig.map_openai_params mapped reasoning_effort to a binary
thinking {type: enabled/disabled} toggle and discarded the effort value, so
DeepSeek always ran default high effort. High-effort reasoning can run away
and exhaust max_tokens during reasoning, returning empty content. Preserve
the reasoning_effort value into the request body when thinking is enabled so
callers can bound effort (e.g. reasoning_effort="low"). Backward-compatible.
See https://api-docs.deepseek.com/guides/thinking_mode

* test(deepseek): reasoning_effort passthrough coverage

* fix(deepseek): exclude reasoning_effort=none sentinel from passthrough

"none" is the thinking-OFF sentinel, not a real effort value. Passing it
alongside explicit thinking:enabled would send contradictory instructions to
DeepSeek. Only forward real effort values (low/medium/high/xhigh/max).

Addresses review feedback on the reasoning_effort passthrough change.

* refactor(deepseek): name supported effort values and type mapped params

* fix(deepseek): keep mapping types local to the adapter

* fix(deepseek): remove cast and type mapped return

---------

Co-authored-by: moe-berri <moe@berri.ai>
2026-10-24 18:02:46 -07:00
devin-ai-integration[bot]
24a7bf6af7
fix(proxy): reconcile team member total_spend with current cycle spend (#48260)
* fix(proxy): reconcile team member total_spend with current cycle spend

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): read /team/list as a list and drain the scripted provider

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(repositories): compose the membership reset writer and drop SQL text asserts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: type the cascade repair test and flatten memberships without a stacked comprehension

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: keep the reset job test file's existing typing untouched

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 17:57:09 -07:00
ishaan-j-berri
84754cca0a
fix(groq): allow streamed reasoning replies in history (#47931) (#48268) 2026-10-25 00:52:01 +00:00