aiohttp 3.14.0 and 3.14.1 re-arm the sock_read timer on a keep-alive
connection after it has already been returned to the idle pool. The stray
timer stamps a SocketTimeoutError on the pooled connection without closing
it, so the pool keeps handing it out and the next request to pick it up
fails instantly on an error left behind by an earlier, unrelated request.
Because a single pool is shared across providers, the failures appear
simultaneously across Vertex AI, Bedrock, Anthropic and OpenAI-compatible
deployments as sub-millisecond "Connection timed out" errors.
uv.lock resolved aiohttp 3.14.1 and the published images install via
`uv sync --frozen`, so every image built from that lock shipped the
regression. The wheel's own metadata declared `aiohttp>=3.10,<4.0`, which
also left pip consumers free to resolve into the same broken window, so
both the runtime floor and the uv constraint move to >=3.14.2.
Upstream fixed this in aio-libs/aiohttp#12954, released in aiohttp 3.14.2;
the lock now resolves 3.14.3. Raising the floor rather than capping below
3.14 keeps the advisories that the existing 3.14.1 floor cleared, so no
osv-scanner ignores are needed. litellm requires Python >=3.10 and aiohttp
3.14.2 requires >=3.10, so no supported interpreter loses support.
Both new tests fail on the previous pins and pass on these.
(cherry picked from commit ffd6ac52c5)
A single ddtrace constraint now covers every supported Python version, so this collapses the version split introduced in #33438. Also aligns the build_from_pip image pin and updates the type-only Tracer import to its current module path
(cherry picked from commit edc38eab34)
The #31122 cherry-pick bumps the deps it pins (cryptography 48.0.1, aiohttp 3.14.1,
vcrpy 8.2.1, the langchain/langgraph stack) and relocks, but this line's lock baseline
kept several ranged runtime deps at versions the ranges still allow, so osv-scanner still
flagged them. Pull them to their fixed releases so the scan is clean:
- starlette 1.1.0 -> 1.3.1 (GHSA-82w8-qh3p-5jfq, GHSA-jp82-jpqv-5vv3)
- python-multipart 0.0.27 -> 0.0.32 (GHSA-5rvq-cxj2-64vf and three others)
- pydantic-settings 2.14.1 -> 2.14.2 (GHSA-4xgf-cpjx-pc3j)
- pypdf 6.13.2 -> 6.13.3 (GHSA-jm82-fx9c-mx94)
- pyjwt 2.12.0 -> 2.13.0 (PYSEC-2026-175/177/178/179)
- langsmith 0.8.3 -> 0.8.18 (GHSA-f4xh-w4cj-qxq8)
Dashboard build deps (not in the shipped bundle; lockfile hygiene, no UI rebuild):
- vite 7.3.2 -> 7.3.5 (GHSA-fx2h-pf6j-xcff, GHSA-v6wh-96g9-6wx3)
- esbuild override 0.28.1 (GHSA-g7r4-m6w7-qqqr)
- form-data override 4.0.6 (GHSA-hmw2-7cc7-3qxx)
After this the only osv-scanner finding is diskcache GHSA-w8v5-vhqr-4h9v, which has no
fixed release and is the entry staging's osv-scanner.toml already ignores until a fix lands.
Bumps the 12 packages osv-scanner flags on litellm_internal_staging, taking
the scan from 24 known vulnerabilities to zero. vcrpy goes to 8.2.1 first so
aiohttp can move to 3.14.1 (vcrpy <= 8.1.1 cannot import aiohttp 3.14), then
the two aiohttp ignore entries are dropped from osv-scanner.toml. The
langchain stack moves together since langchain 1.3.9 requires langgraph 1.2.x.
Runtime deps cryptography (48.0.1), starlette (1.3.1), python-multipart
(0.0.32), pydantic-settings (2.14.2) and pypdf (6.13.3) are bumped via relock,
and the dashboard's js-yaml, ws and form-data overrides are bumped too.
Also removes the paths filter on the OSV workflow so it runs on every PR
rather than only when a lockfile changes, which is why it never showed up on
recent code-only PRs
(cherry picked from commit a8a1472428)
Regenerate uv.lock (pypdf 6.13.2, tornado/aiohttp constraints from #30220)
and ui/litellm-dashboard/package-lock.json (vitest 3.2.6). Pin
brace-expansion to 5.0.6 via an explicit package.json override so the
line's lockfile resolves the patched transitive version that #30220
ships on staging
* fix(deps): bump vulnerable proxy dependencies (starlette/fastapi, granian, pyarrow, semantic-router)
Resolve known CVEs flagged by osv-scanner/grype against uv.lock. All bumped
versions verified to resolve, install, and pass the proxy auth/route/middleware
unit suites (717 tests) plus an import smoke on the new stack.
- starlette 0.50.0 -> 1.1.0 (CVE-2026-48710 "BadHost", GHSA-86qp-5c8j-p5mr):
versions <1.0.1 reconstruct request.url from the unvalidated Host header,
poisoning request.url.path. Required raising fastapi 0.124.4 -> 0.136.3,
which dropped fastapi's starlette<0.51.0 cap; an explicit starlette>=1.0.1
floor blocks regression to a vulnerable transitive resolution. The proxy's
own auth already reads scope["path"] via get_request_route, but the locked
starlette still flagged in container scanners and left other request.url
consumers exposed.
- granian 2.5.7 -> 2.7.4 (CVE-2026-42544, unauthenticated DoS via WebSocket
subprotocol header panic; CVE-2026-42545, WSGI response-header-panic DoS).
granian is a selectable proxy server (proxy_cli).
- pyarrow 22.0.0 -> 23.0.1 (CVE-2026-25087 / PYSEC-2026-113).
- semantic-router 0.1.12 -> 0.1.15: 0.1.12 was yanked (CVE-2026-42208 — its
unbounded litellm pin could resolve a credential-exfiltrating litellm==1.82.8
wheel).
Not fixable by bump: diskcache 5.6.3 (CVE-2025-69872, unsafe pickle
deserialization) has no upstream fix and is left pinned; exploiting it requires
write access to the local cache directory.
Relock side effect: sse-starlette 3.4.2 -> 3.4.4.
* deps: relax exact pins in optional extras to compatible ranges
The proxy/optional extras exact-pinned every dependency, which (1) forces
downstream `pip install litellm[proxy]` consumers into version lockstep and
(2) blocks them from pulling transitive security patches without forking — the
structural cause behind needing a litellm release to clear the starlette CVE in
the previous commit.
Convert the ordinary extras deps to `>=current,<next_major` ranges, mirroring
the core [project].dependencies style. Reproducibility for litellm's own
Docker/CI is unaffected: images install via `uv sync --frozen`, and the lock
re-resolves to the identical versions (no locked version changed).
Kept exact-pinned:
- litellm-proxy-extras, litellm-enterprise — litellm's own sub-packages,
versioned in lockstep with the release.
- opentelemetry-api/sdk/exporter-otlp — must resolve to matching versions.
- grpcio — supply-chain-pinned to a vetted, aged release.
Also corrects the stale comment claiming the extras are exact-pinned for Docker
reproducibility (the images use the lock, not these pins).
* fix(ci): resolve license-check lookup version from the floor for ranged deps
check_licenses.py derived the PyPI lookup version with
`next(iter(req.specifier))`, which returns an arbitrary specifier clause. For
a range like `>=0.12.1,<1.0` it picked the upper bound (`1.0`) — a version
that doesn't exist on PyPI — so the license lookup 404'd and the package was
flagged as having an unknown license.
The previous commit's switch from exact pins to ranges exposed this for
soundfile, pyroscope-io, redisvl, diskcache, and mlflow (the ranged deps not
already in liccheck.ini's allowlist). Prefer a lower-bound/exact version (a
real released version) for the lookup.
* fix(proxy): set strict_content_type=False on the FastAPI app
Starlette 1.0 / FastAPI 0.13x flipped the default to strict_content_type=True,
which refuses to parse a JSON request body when the client omits the
Content-Type header. The proxy previously accepted those requests, so the
fastapi/starlette bump in this PR would silently break clients that don't send
a Content-Type. Restore the prior lenient behavior explicitly.
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
* Add granian as a ASGI compliant web server. Provides better stability, 10-20 RPS improvement under standard LT conditions.
TODO: Verify poetry lock details and add locust numbers to PR
* Update granian version in license_cache.json and pyproject.toml to 2.5.7
* Enhance proxy CLI tests by adding SSL initialization checks for Granian server. Remove Python version skip conditions and implement tests to ensure SSL certificate and key are required for server initialization.
* update uv lock to fix granian import error
openai 2.34.0 began rejecting an explicitly-passed empty-string api_key
at client construction (raises OpenAIError before any request), which
broke tests/local_testing/test_exceptions.py::test_exception_with_headers
and related cases after uv.lock floated openai 2.33.0 -> 2.36.0.
Pin back to 2.33.0 (within the existing pyproject >=2.20.0,<3.0.0 range)
as a temporary stopgap; longer-term fix to follow.
Split the monolithic LiteLLM proxy into independently scalable Kubernetes components to allow separate horizontal scaling of the LLM data plane and management API surfaces
- Add DatabaseURLSettings pydantic-settings model that assembles DATABASE_URL (and optional DATABASE_URL_READ_REPLICA) from discrete DATABASE_* env vars before Prisma initializes, supporting both IAM token auth (minting short-lived RDS tokens) and password auth; replaces the CLI-only path that componentized entrypoints bypass
- Add gateway component (port 4000) that trims the proxy route table to the LLM data-plane surface (chat, embeddings, completions, audio, realtime, provider passthroughs, health/metrics) via an allowlist applied inside the lifespan context so plugin-registered routes are captured
- Add backend component (port 4001) that exposes the management/admin surface (keys, users, teams, orgs, spend analytics, model management, SSO, audit logs) with a complementary allowlist
- Add ui component — Next.js static export served by nginx (port 3000) with RSC payload routing, asset prefix aliasing, and SPA fallback for dashboard routes
- Add migrations component with dedicated Dockerfile that runs prisma migrate deploy via a Helm pre-install/pre-upgrade Job, eliminating per-pod schema contention on the Prisma advisory lock
- Add Helm chart (helm/litellm) with separate Deployments, Services, HPAs, and ConfigMap for each component; shared _helpers.tpl emits DATABASE_*, IAM_TOKEN_DB_AUTH, REDIS_*, and DISABLE_SCHEMA_UPDATE env vars from chart values; ingress template routes traffic to the correct component by path prefix
- Add comprehensive tests for DatabaseURLSettings covering IAM auth, password auth, read replica fallbacks, operator-pinned URL preservation, and percent-encoding; add coverage test asserting gateway + backend allowlist union equals the full proxy route set
- Add pydantic-settings>=2.14.1 as a proxy extra dependency and update liccheck allowlist
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Our `uv.lock` already resolves jinja2 to 3.1.6, so Docker / CI installs
get that version. The `pyproject.toml` floor was lagging at 3.1.0,
which means downstream consumers using `--resolution=lowest-direct` or
older constraint files can land on 3.1.0-3.1.5 instead of the version
we actually test against.
Aligns the declared floor with the resolved version so external
installers see the same baseline our test matrix exercises.
`uv lock` diff is metadata-only (no resolved-version drift).
* feat(audio_transcription): add NVIDIA Riva STT provider
Adds nvidia_riva as a new audio transcription provider, supporting both
NVCF-hosted and self-hosted Riva ASR deployments via gRPC streaming.
- Auto-resamples input audio to 16 kHz mono LINEAR_PCM (soundfile + numpy,
audioread fallback) so callers can send any common format.
- Maps OpenAI params: language (en -> en-US), response_format (text/json/
verbose_json), timestamp_granularities=["word"] -> enable_word_time_offsets,
word offsets converted ms -> s for verbose_json.
- Auth: NVCF when nvcf_function_id is set (SSL on by default), self-hosted
otherwise (SSL off by default), with explicit use_ssl override.
- gRPC errors wrapped via NvidiaRivaException -> litellm exception classes.
- Optional deps gated behind [stt-nvidia-riva] extra (nvidia-riva-client,
soundfile, audioread, numpy).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(nvidia_riva): address PR review feedback
- handler: forward call-level `timeout` to streaming_response_generator
(kwarg-detected via inspect for older riva-client compat) so a stalled
Riva server cannot block the caller indefinitely.
- audio_utils: spill bytes to a tempfile before audioread.audio_open;
most audioread backends (FFmpeg, GStreamer) require a real filesystem
path and previously raised TypeError on BytesIO, breaking the mp3/m4a
fallback path.
- audio_utils: prefer soxr / scipy.signal.resample_poly for resampling
(anti-aliased polyphase) when installed, falling back to linear only
as a last resort. Avoids aliasing on 44.1/48 kHz -> 16 kHz downsamples.
- transformation: bare `es` now maps to es-ES (Castilian) instead of
es-US, matching BCP-47 conventions.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore: trigger CI re-run [stabilize loop 1/3]
* Update litellm/llms/nvidia_riva/audio_transcription/transformation.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* chore: trigger CI re-run [stabilize loop 1/3]
* fix code qa
* fix lint
* fix mypy
* fix mypy
* Fix NVIDIA Riva ASR service lookup
* Fix NVIDIA Riva transcription payload logging
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: oss-pr-review-agent-shin[bot] <281797381+oss-pr-review-agent-shin[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Removing the new check-dependency-floors.yml workflow. It only fires when
pyproject.toml changes, which is rare; for those PRs, a maintainer can
run the same check by hand with one command. Documented that command in
a pyproject.toml comment next to the deps.
Also adds the missing upper bound on importlib-metadata (>=8.0.0,<9.0)
for consistency with every other entry in the list.
CI matrix on Python 3.13 caught three floors that predate cp313 prebuilt
wheels and would force users into a Rust/C build:
- tiktoken: 0.7.0 -> 0.8.0 (cp313 wheels start at 0.8)
- tokenizers: 0.20.0 -> 0.21.0 (cp313 wheels start at 0.21; sdist's
pyproject.toml pre-0.21 is also malformed for modern build backends)
- pydantic: 2.5.0 -> 2.10.0 (pydantic-core cp313 wheels start at 2.27,
shipped with pydantic 2.10)
Verified locally on Python 3.10 and 3.13: install at lowest-direct +
import litellm + import every openai-namespace symbol the codebase uses
all pass.
The 12 core `[project.dependencies]` entries in pyproject.toml were exact
`==` pins, a side effect of the Poetry → uv migration. This forces every
downstream package that lists litellm as a dependency to downgrade common
runtime libraries (openai, pydantic, aiohttp, click, jsonschema, ...) to
the exact versions we ship. Customers have flagged this as a coexistence
blocker.
Switch to lower-bounded ranges with upper bounds where the upstream
package is pre-1.0 or has a known breaking-major-version policy.
Reproducibility for our Docker proxy and CI continues to come from
`uv.lock`, which is regenerated here as a metadata-only diff (no
resolved versions or hashes change).
Inspired by #26157 (which got stranded on `litellm_oss_staging_04_21_2026`
when the forward-merge to internal staging in #26216 was closed). Floors
in this PR are tighter than #26157's: they were validated by installing
litellm at `--resolution=lowest-direct` and importing the openai-namespace
symbols the codebase actually uses.
Floor highlights vs #26157:
- openai >= 2.20 (was 2.0) — Responses API symbols + `Omit` need a 2.x mid-range floor
- httpx >= 0.28, < 1.0 (was no upper) — pre-1.0
- importlib-metadata >= 8.0 (was 6.0) — stay in tested major
- tokenizers >= 0.20, < 1.0 (was 0.19, no upper) — pre-1.0
- aiohttp >= 3.10, < 4.0 (was no upper) — bound major
- pydantic >= 2.5, < 3.0 — kept
- All other floors: keep tested major, add upper bound
Adds a `check-dependency-floors.yml` GitHub Actions workflow that
installs litellm at `--resolution=lowest-direct` on Python 3.10 and 3.13
and import-checks every openai symbol the codebase uses, so a future
floor regression fails fast in CI rather than silently in the field.
Per Yuneng's feedback, use a single @pytest.mark.vcr marker so one record
sweep populates cassettes for every marked test across all providers,
instead of forcing each test to bind to a hard-coded cassette path.
Changes vs. the initial scaffolding:
- Add 'pytest-recording==0.13.4' on top of vcrpy. Adopt its layout:
cassettes live at 'cassettes/<test_module>/<test_name>.yaml', resolved
automatically. New tests just decorate with '@pytest.mark.vcr' — no
imports or path bookkeeping.
- Move the shared filter/match config into a 'vcr_config' fixture in
'tests/llm_translation/conftest.py' (consumed by pytest-recording for
every marked test in the dir). Drop the standalone 'vcr_config.py'.
- Bulk record / replay via the standard '--record-mode' CLI flag:
'make test-llm-translation-record' now sweeps every '@pytest.mark.vcr'
test under tests/llm_translation in one shot. Optional 'TARGET=' var
scopes to a single file.
- Move existing cassettes to the per-test paths and update the local
in-process Anthropic regenerator to write to the same paths.
- Refresh README + Makefile target docs to match the sweep workflow.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Live LLM e2e tests have been draining provider billing accounts and going
flaky on outages (LIT-2683). This change introduces vcrpy-backed cassette
replay so CI can exercise the same end-to-end LiteLLM transformation paths
without hitting the live provider:
- Add 'vcrpy==8.1.1' to the dev dependency group.
- New 'tests/llm_translation/vcr_config.py' centralises the VCR config:
filters auth/secret headers and per-request response headers, matches on
method+URI+body, and exposes 'LITELLM_VCR_RECORD_MODE' for re-recording.
- New 'tests/llm_translation/test_anthropic_completion_vcr.py' demonstrates
the pattern with one non-streaming and one streaming Anthropic test that
replay from cassettes shipped under 'cassettes/'.
- New 'tests/llm_translation/cassettes/_record_anthropic_fixtures.py' lets
contributors regenerate the canned Anthropic cassettes against a local
in-process mock (no API key required), and 'cassettes/README.md' documents
the full record/replay/refresh workflow.
- New 'make test-llm-translation-record FILE=...' Makefile target to refresh
cassettes against the live API.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
