Commit graph

51807 commits

Author SHA1 Message Date
Yuneng Jiang
676b682933
build(docker): copy only runtime files into the gateway, backend and migrations images
The component runtime stages copied the builder's whole /app, which is the
entire repository plus build output: six lockfiles, helm, terraform, docs,
.env.example and the Rust build tree. Measured at runtime with an audit hook
on a booted container, the gateway reads only gateway/, litellm/ and
enterprise/ from /app, the backend only backend/, litellm/ and enterprise/,
and the migrations job only litellm/ and run.py. The pyproject.toml read is
prisma-python's optional [tool.prisma] lookup, which ours does not define

Copy just those, plus .venv, the entrypoint script and schema.prisma for the
gateway and backend. PYTHONPATH=/app and the in-Dockerfile asserts are
unchanged
2026-10-24 19:04:52 -07:00
Yuneng Jiang
368a979c7e
build(docker): move to newer wolfi-base and nginx base images
Bump the pinned wolfi-base digest in every Wolfi Dockerfile to 360896ef, which
ships brotli 1.2.1-r1 and ca-certificates-bundle 20260909-r3. The previous
digest already carried libbrotli 1.2.0-r6 as an OpenSSL dependency, and
`apk add --no-cache` never upgrades a package the base already has

Move the UI image to nginx 1.31.8-alpine3.24, whose CA bundle is the 20261020
Mozilla set, and upgrade brotli-libs in its runtime stage since no nginx
digest ships 1.2.1 yet
2026-10-24 19:04:52 -07:00
yuneng-jiang
20d52b018b
Merge pull request #48075 from BerriAI/litellm_backport_47670_47969_stable_1_103_x
Some checks failed
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
chore(release): backport #42431, #47670, #47969, #48055, #48103, #48110 to stable/1.103.x and cut 1.103.8
2026-10-23 16:16:00 -07:00
Yuneng Jiang
a45dc7ae02
chore(deps): bump mlflow, pypdf and setuptools
Redoes #48110 on this line. The lock moves mlflow, mlflow-skinny and mlflow-tracing to 3.17.2, pypdf to 6.21.2 and setuptools to 84.1.0 inside their existing ranges. mlflow 3.17.2 and pypdf 6.21.2 are still inside the 3-day exclude-newer window, so they get their own exclude-newer-package cutoffs, matching main. The lock was regenerated with uv 0.11.7
2026-10-23 15:36:53 -07:00
Yuneng Jiang
b8e5bba722
Rotate the cosign image-signing public key
(cherry picked from commit e82a3c6469)
2026-10-23 15:33:50 -07:00
Yuneng Jiang
6a37aeaecc
chore(deps): bump vscode-extension vite and vitest
Raises the vitest floor to ^4.1.12 to match main and re-resolves the lock inside the 3-day release window. vitest and the @vitest packages go to 4.1.12 and vite to 8.3.4, with rolldown 1.2.14, the same versions main carries

Within those packages' existing ranges postcss lands on 8.5.31, chai on 6.3.0 and tinyrainbow on 3.2.0, one release past main for each, and nothing outside the vite and vitest tree moves
2026-10-23 15:26:11 -07:00
Yuneng Jiang
94909fce81
chore: update Next.js build artifacts (2026-10-23 22:23 UTC, node v24.19.0) 2026-10-23 15:23:45 -07:00
Yuneng Jiang
9d85318726
fix(ui): keep dashboard schemas working on zod 4
zod 4.6 rejects a missing object key unless its schema is optional, even when the schema itself accepts undefined. The key edit form schema is built from z.custom fields that relied on the old behavior, so an empty or partially filled form failed to validate

Build the schema from its shape with partial(), which makes every field optional the way main does with per-field optional(). The other zod importers type-check unchanged and their tests pass on zod 4.6.6
2026-10-23 15:20:14 -07:00
Yuneng Jiang
c5901cef5b
chore(deps): bump dashboard next, zod, vite, vitest and sharp
Moves next to 16.4.1, zod to 4.6.6, vite to 7.3.7, vitest and @vitest/ui to 4.1.12 and raises the sharp override to ^0.35.5, matching main. @vitest/coverage-v8 goes to 4.1.12 with them because it peers on the exact vitest version

A top-level zod override points every nested consumer at the root version, so the lock carries a single zod 4.6.6. This line's dashboard was on zod 3.25.76, so this is the zod 4 major
2026-10-23 15:14:47 -07:00
Yuneng Jiang
99e5c75a07
chore(deps): bump OpenTelemetry, logfire, pypdf, GitPython, graphql-core, Brotli and Pillow
Moves the OpenTelemetry API, SDK and OTLP exporter pins to 1.45.1 and the FastAPI instrumentation to 0.66b1 in every group, logfire to 5.2.0, and splits Pillow so Python 3.11 and newer get 13.0.0 while 3.10 stays on 12.3.0, matching main

traceloop-sdk goes from 0.33.12 to 0.34.0 in the ci group, the version main already uses. Its 0.33.12 instrumentations pin opentelemetry-semantic-conventions below 0.50, which cannot resolve next to the 0.66b1 that OpenTelemetry 1.45.1 and logfire 5.2.0 need

The lock is re-derived on this line with uv 0.11.7. GitPython goes to 3.2.1, graphql-core to 3.2.12, Brotli to 1.2.1 and pypdf to 6.21.1 inside their existing ranges, and the OpenTelemetry 1.45 line pulls in opentelemetry-exporter-http-transport and opentelemetry-exporter-otlp-common. Apart from those, only OpenTelemetry packages and the traceloop instrumentations move

Brotli 1.2.1 is still inside the 3-day release window, so it gets its own exclude-newer-package cutoff
2026-10-23 15:08:56 -07:00
Ishaan
fe7485c837
fix(ci): unblock dependency checks (#48055)
* fix(deps): upgrade Python dependencies

* fix(deps): lock patched Python dependency versions

* fix(otel): preserve JSON export with the updated SDK

* test(otel): cover JSON export with the updated SDK

* test(otel): update SDK log records and worker discovery

* fix(ui): align Lens peer dependency with patched Next.js

* fix(otel): hand the AgentOps exporter a session it owns for the JWT header

OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a
transport, so writing the minted JWT to the exporter's private _session raised,
was swallowed, and every AgentOps batch went out unauthenticated. The exporter
now creates the session, passes it through the public session= argument and
sets the Authorization header on it

The tests now drive a real export, serving the token endpoint with respx and
capturing the OTLP POST with requests_mock, instead of reading SDK internals

* fix(otel): skip FastAPI instrumentation once the app has started

opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead
of calling add_middleware, so instrumenting an app whose middleware stack is
already built no longer raises. The call silently did nothing while we still
registered the global response propagator. Check the built stack up front and
skip both, which is what the existing test expects

* test(otel): assert exported requests and span flags instead of SDK internals

OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C
random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace
and Langfuse tests now export through requests_mock and check the URL and headers
the collector receives, and the traceresponse tests expect the server span's own
trace flags

* fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it

zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A
root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly,
on its own nested copy, so a top-level zod override points every consumer at the
root version and the lock carries a single zod 4.6.6

* test(ui): stub the tag summary fetch the client panel retry refreshes

Retrying the model breakdown refetches /tag/summary, which this integration test
left on the real network. When that request failed before the assertion, the
panel switched to its summary error and unmounted the model row, so the test
passed or failed by timing

* fix(deps): drop the duplicated Lens override from the dashboard package.json

* style(ui): format the client panel fetch stub

---------

Dependency files (pyproject.toml, uv.lock, ui/litellm-dashboard/package.json and package-lock.json) were left out of this pick and are re-derived on the 1.103 line in the following commits
ClientUsagePanel.integration.test.tsx does not exist on this line, so its hunk was dropped
The traceresponse hunks in test_otel_v2_mount.py target tests this line does not have, so that file is unchanged
The Langtrace export test the squash edits in test_litellm_logging.py is not on this line, so that file is unchanged
OTLPJsonSpanExporter on this line never took certificate_file or session, so the fix keeps that signature and adds only the header merge, max_request_size and the export override
The new JSON exporter tests intercept the collector with requests_mock.Mocker instead of injecting a session, and the components test imports Final, requests and requests_mock for them
The TLS export tests and their log record import hunks are not on this line, so those hunks were dropped
test_opentelemetry.py imports Final for the worker thread lookup and keeps its trace import without SpanLimits, LogRecord, Resource or the TLS helpers

Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
(cherry picked from commit 8f9894fbd6)
2026-10-23 15:08:56 -07:00
devin-ai-integration[bot]
3ba758322d
fix(otel): record the GenAI exception event through the Logs API on both OpenTelemetry lines (#42431)
* fix(otel): record the GenAI exception event without the removed Events API

OpenTelemetry removed opentelemetry._events in 1.44.0, so the three imports of
it broke 7 modules under litellm.integrations.otel, including the entry point.
Two things then failed quietly: with LITELLM_OTEL_V2 set the otel callback
resolved to None and nothing was exported, and with it unset the newrelic
callback was dropped as well, because that branch imports the v2 logger
ungated

Build and emit the event through the Logs API, which both lines carry. The
event name keeps riding the event.name attribute: the event_name log record
field that replaces it only exists from 1.44.0, and this package pins 1.28.0,
so the attribute is the only form both can write. It is also what the Events
API wrote, so exported events keep their shape

Emitting a plain record drops the default the Events SDK applied, so the
timestamp now falls back to time_ns() here

* style(otel): trim the event name key and regression test prose

Keep only the constraint a reader cannot infer from the code, that the
event_name record field does not exist on the pinned OpenTelemetry line

* fix(otel): export the GenAI exception event on both OpenTelemetry 1.28 and 1.44 lines

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(otel): drop the record selection comment

Co-authored-by: Pawan-Shahane <shahanepawan511@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(otel): collapse the record selection conditional for ruff format

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(otel): build the record fields with a dict literal

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): set the native event_name on the 1.44 log record

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(otel): spell out the record kwargs so the type gate sees each call

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(otel): suppress the version-gated kwargs for the type gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(otel): drop the version-window prose and correct the event_name suppression reason

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(otel): wrap the compat test docstring to the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

test_otel_v2_components.py on this line has no TLS export tests, so the span context helper and the two new log event tests land at the end of the file instead of ahead of the TLS helpers

Co-authored-by: Pawan-Shahane <shahanepawan511@gmail.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 238f434153)
2026-10-23 15:08:25 -07:00
Yuneng Jiang
bb3a966291
bump: version 1.103.8 2026-10-23 12:04:07 -07:00
shrey-berri
7abddba159
fix(security): update bundled CA roots and preserve Prisma image compatibility (#47969)
* fix(security): bump grpcio and boto3 so their bundled CA files drop the compromised eMudhra roots

grpc ignores SSL_CERT_FILE and verifies every channel against its own bundled roots.pem. In 1.78.0 that file still trusts emSign ECC Root CA - G3 and AAA Certificate Services, so Vertex AI, Google KMS, Chirp STT and OTLP gRPC exporters (Arize) accept all 14 advisory chains. 1.84.1 is the first release whose roots.pem drops both roots.

botocore uses certifi when it is installed and falls back to its bundled cacert.pem otherwise. Through 1.43.116 that file still trusts AAA Certificate Services. Requiring boto3 1.43.117 pulls in botocore 1.43.117, the first release whose cacert.pem rejects all 14 advisory chains, so the fallback stays safe if certifi is ever absent

* fix(docker): provide OpenSSL 3 when generating Prisma engines

(cherry picked from commit 52166b51a1)
2026-10-23 11:58:21 -07:00
ryan-crabbe-berri
1ca58f46ad
fix(security): stop trusting the compromised eMudhra root CA (#47670)
Update certifi, Rust trust roots, and the Docker CA bundle to remove emSign ECC Root CA - G3

Require the fixed certifi release in litellm-core and retain libssl3 in Docker runtimes for Prisma compatibility

Co-authored-by: shin-berri <shin-laptop@berri.ai>
(cherry picked from commit f2118fc767)
2026-10-23 11:58:09 -07:00
yuneng-jiang
30f6910f84
Merge pull request #47732 from BerriAI/litellm_backport_p0_batch_translation_cost_stable_1_103_x
Some checks failed
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
fix(batches)!: backport P0 batch-cost, pricing and translation fixes to stable/1.103.x for v1.103.7
2026-10-21 01:08:12 -07:00
Yuneng Jiang
d7b606ac91
Merge stable/1.103.x (#47743, #47763) into litellm_backport_p0_batch_translation_cost_stable_1_103_x
# Conflicts:
#	litellm/model_prices_and_context_window_backup.json
#	litellm/proxy/hooks/proxy_track_cost_callback.py
#	litellm/responses/litellm_completion_transformation/streaming_iterator.py
#	model_prices_and_context_window.json
#	tests/test_litellm/proxy/hooks/test_proxy_track_cost_callback.py
#	tests/test_litellm/test_utils.py
#	uv.lock
2026-10-21 00:55:49 -07:00
yuneng-jiang
a3c62f9f48
Merge pull request #47763 from BerriAI/litellm_backport_lit145_stable_1_103_x
fix(proxy): backport the LIT-145 spend tracking fixes to stable/1.103.x and bump to 1.103.7
2026-10-21 00:54:25 -07:00
yuneng-jiang
3902cba8a0
Merge pull request #47743 from BerriAI/litellm_backport_45317_46430_stable_1_103_x
fix(count_tokens): backport Bedrock Mantle count_tokens fallback to stable/1.103.x
2026-10-21 00:54:08 -07:00
devin-ai-integration[bot]
a29a29a7c0 bump: version 1.103.7 2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
de59732383 chore(deps): bump aiohttp to 3.14.4 in uv.lock for GHSA-2g87-pp6c-29x5, GHSA-5wr8-gcqf-gqf3 and GHSA-494j-54xm-rwp3 2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
8948ab4d0c fix(logging): spilled over PTU requests without base_model log the served model's rates (#47655)
Backport of #47655 to stable/1.103.x

Cherry-picked from 6b1f4ed46f (main)
2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
bd68d04aae fix(proxy): price a 499 spend row only for the proxy's own disconnect, and never after the call already completed (#47652)
Backport of #47652 to stable/1.103.x

Cherry-picked from ffb7133d32 (main)
2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
db8f45dc87 fix(proxy): price earlier timed-out retry attempts at their own deployment's model rate (#47648)
Backport of #47648 to stable/1.103.x

Cherry-picked from 27efc387f7 (main)
2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
7e630b07d4 fix(proxy): price a billed failure row at the deployment's long-context tier like the success path (#47647)
Backport of #47647 to stable/1.103.x

Cherry-picked from 70ece3501f (main)
2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
8147f8d68f fix(proxy): scope the search free-tier counter to the request's provider (#47646)
Backport of #47646 to stable/1.103.x

Cherry-picked from 640f2d865b (main)
2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
6dd7b489de fix(proxy): strip caller base_model from spend pricing and reject lone surrogates in multipart metadata (#47639)
Backport of #47639 to stable/1.103.x

Cherry-picked from 2a268951c4 (main)
2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
aa838f931b fix(router): price cost-based routing through model_info.base_model (#47617)
Backport of #47617 to stable/1.103.x

Cherry-picked from 0c6731eb5c (main)
2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
be085c9028 fix(proxy): flush the spend-log queue only for Responses turns the bridge reads back (#47614)
Backport of #47614 to stable/1.103.x

Cherry-picked from 423362bab2 (main)
2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
bb2ec9a422 fix(proxy): bill partial Responses API streams when the client disconnects (#47611)
Backport of #47611 to stable/1.103.x

Cherry-picked from f0ae445cc4 (main)
2026-10-21 03:37:25 +00:00
devin-ai-integration[bot]
d7e344440c fix(proxy): requeue spend logs when a Postgres restart tears down the prisma engine mid-write (#47571)
Backport of #47571 to stable/1.103.x

Cherry-picked from ccd0d4ea49 (main)
2026-10-21 01:55:14 +00:00
devin-ai-integration[bot]
6120ddf786 fix(litellm_proxy): forward tool_config cache_control_injection_points to the downstream proxy (#47570)
Backport of #47570 to stable/1.103.x

Cherry-picked from 04e9ed4c52 (main)
2026-10-21 01:55:14 +00:00
devin-ai-integration[bot]
20c0e82370 fix(logging): report the pay-as-you-go entry billing used on Azure PTU spillover (#47567)
Backport of #47567 to stable/1.103.x

Cherry-picked from 4222523261 (main)
2026-10-21 01:55:14 +00:00
devin-ai-integration[bot]
c6ee8e0d54 feat(proxy): write one spend row per billed timed-out retry attempt (#47416)
Backport of #47416 to stable/1.103.x

Cherry-picked from ad8ca533ca (main)
2026-10-21 01:55:14 +00:00
devin-ai-integration[bot]
c23f17dcff feat(proxy): opt-in free-tier allowance for Google Search grounding spend (#47494)
Backport of #47494 to stable/1.103.x

Cherry-picked from e63aec029a (main)
2026-10-21 01:55:14 +00:00
mateo
696e1c3c87 fix(bedrock): add Claude Mantle model helper
Minimal prerequisite from #43646 for stable/1.104.x.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 6a801372ac)
2026-10-21 01:54:50 +00:00
mrinal-berri
de896efc8e fix(count_tokens): fall back to Mantle on Bedrock 404 (Bedrock part of #46430)
(cherry picked from commit 08a24d391a)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-21 01:51:08 +00:00
devin-ai-integration[bot]
71118e893e fix(bedrock): count tokens on bedrock-mantle when bedrock-runtime cannot count a Claude model (#45317)
(cherry picked from commit 577d74c1ce)
2026-10-21 01:50:30 +00:00
Mrinal Chanshetty
d898de4253 bump: version 1.103.7 2026-10-20 18:20:54 -07:00
Mrinal Chanshetty
81b67d70c1 chore(deps): bump aiohttp to 3.14.5 and claude-agent-sdk to 0.2.165
Matches main's resolved aiohttp and its claude-agent-sdk pin (==0.2.165, dev group). Lock moved only those two packages.
2026-10-20 18:20:30 -07:00
Mrinal Chanshetty
f419a596a7 test: retarget cost-map tests to the rows mirrored from main
Test-only. Takes main's version of test_shipped_tool_search_rule_fills_mapped_claude_entries_without_flag, test_cache_control_injection_tool_config_honors_ttl_for_regional_model_lacking_own_pricing and test_get_model_info_gemini; retargets three tests whose rows main changed or removed (gemini-2.0-flash-001 -> gemini-2.5-flash, gpt-5.1-codex-max -> gpt-5.2 for xhigh, gpt-5.1-chat-latest -> gpt-5.1); registers this line's unsupported providers in PROVIDER_LABELS_WITHOUT_A_MODEL_SET; adds extraction mode, main's enum lists and every key main's map uses to test_utils' inline schema, plus main's seedance cost exceptions.
2026-10-20 18:20:07 -07:00
Mrinal Chanshetty
dc314e7f6a chore(cost-map): accept the mirrored map's new keys in the schema
Adds search_free_requests, token_count_model (with litellm_provider enum bedrock/bedrock_mantle), token_count_framing and token_count_tokenizer to ci_cd/generate_model_prices_schema.py using main's definitions, and regenerates model_prices_and_context_window.schema.json. Schema keys are CI-only; the line already fetches main's map at runtime.
2026-10-20 18:02:25 -07:00
Mrinal Chanshetty
105a342e77 chore(cost-map): mirror main's model cost map at 6083cf9776
Both model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json are byte-identical to main at 6083cf9776.
2026-10-20 17:52:13 -07:00
devin-ai-integration[bot]
81b9ab7e51 feat(proxy): lift every earlier billed timed-out retry attempt onto the failed request (#47490)
Backport of #47490 to stable/1.103.x

Cherry-picked from 7a4a255d1d (main)
2026-10-21 00:45:27 +00:00
devin-ai-integration[bot]
e7e1aa8984 fix(vertex_ai): bill per-character text embeddings on Google's billableCharacterCount (#47460)
Backport of #47460 to stable/1.103.x

Cherry-picked from d29200fa05 (main)
2026-10-21 00:45:27 +00:00
devin-ai-integration[bot]
183e1c8a49 fix(logging): do not stamp a provider handoff when pre_call runs for a proxy-only failure (#47456)
Backport of #47456 to stable/1.103.x

Cherry-picked from 3fa1587f43 (main)
2026-10-21 00:45:27 +00:00
devin-ai-integration[bot]
c793111f40 fix(proxy): price a billed failure row at the deployment's own rate like the success path (#47437)
Backport of #47437 to stable/1.103.x

Cherry-picked from fb555b00dc (main)
2026-10-21 00:45:27 +00:00
devin-ai-integration[bot]
6a65634599 fix(proxy): price the estimated input tokens on a client-cancelled (499) request's spend row (#47436)
Backport of #47436 to stable/1.103.x

Cherry-picked from 0e7c4908bb (main)
2026-10-21 00:45:27 +00:00
devin-ai-integration[bot]
543f1899d4 fix(proxy): price the estimated input tokens on a read-timed-out request's spend row (#47435)
Backport of #47435 to stable/1.103.x

Cherry-picked from 7b76b2b15f (main)
2026-10-21 00:45:27 +00:00
devin-ai-integration[bot]
37b7c14830 feat(router): let a deployment set a timeout per service_tier (#47406)
Backport of #47406 to stable/1.103.x

Cherry-picked from 62ce515d15 (main)
2026-10-21 00:45:27 +00:00