Commit graph

839 commits

Author SHA1 Message Date
yujonglee
2c30fe16b0
Merge pull request #38765 from BerriAI/litellm_ocr_sdk_parity_tests
test(harness): add OCR parity with migration strategy runners
2026-09-03 10:16:35 -07:00
devin-ai-integration[bot]
92edcb90db
fix: keep litellm importable on Python 3.10 and guard 3.11-only typing imports in CI (#39448)
* ci: guard against Python 3.10-incompatible typing imports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: address Python 3.10 typing guard review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ci): honor version-guard direction and scan litellm-proxy-extras in py310 typing check

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-02 18:27:19 -07:00
yujonglee
62e318de8e
fix(python-bridge): harden sync and async route boundaries (#39332) 2026-09-02 16:26:35 -07:00
yujonglee
198906495f
refactor(python-bridge): split routes and add shared function tracing (#39031)
* refactor(python-bridge): split non-streaming bridge modules

* refactor(python-bridge): bring shared function tracing into route layer

* feat(dev): list Python route functions and call sites

* feat(dev): list Rust route functions and call sites

* docs(dev): record OCR parity gaps across Python and Rust

* feat(dev): list executed SDK calls with runtime tracing

* feat(dev): report Python vs Rust SDK pipeline steps in one CLI

* feat(dev): side-by-side pipeline step report in compare CLI

* fix(dev): drop invalid Final annotations in compare cell loop

* feat(dev): blue python-only and yellow rust-only steps in compare CLI

* feat(dev): vertical layout with section spacing in compare CLI

* fix(dev): validate SDK trace stages across sync and async routes

* refactor(rust): align SDK route call structure with Python

* refactor(python-bridge): share sync and async route call wrappers

* refactor(dev): split compare CLI into fixtures, runtime, and report modules

* fix(ci): run SDK trace tests and satisfy test lint
2026-09-02 16:26:35 -07:00
Yujong Lee
90eadac409
test(build): keep wheel checks outside package 2026-09-02 12:16:25 -07:00
Yujong Lee
ce0c85ea69
refactor(rust): colocate native wheel contract checks 2026-09-02 12:16:25 -07:00
Yujong Lee
cbb8a1784d
chore(ci): extract setup-uv pin 2026-09-02 12:16:25 -07:00
Yujong Lee
38150dfc2c
fix(ci): pin workflow toolchain dependencies 2026-09-02 12:16:25 -07:00
Yujong Lee
ef9a207ed5
fix(ci): harden release wheel reporting 2026-09-02 12:16:25 -07:00
Yujong Lee
e6a317e079
fix(ci): enforce release wheel metadata contract 2026-09-02 12:16:25 -07:00
Yujong Lee
9dc9cd325c
fix(ci): isolate release wheel reporting permissions 2026-09-02 12:16:25 -07:00
Yujong Lee
2f362cfec2
fix(ci): preserve release wheel contract parity 2026-09-02 12:16:25 -07:00
Yujong Lee
25987cb961
test(build): validate release wheel contracts 2026-09-02 12:16:25 -07:00
devin-ai-integration[bot]
93219a9257
fix(docker): install bedrock-realtime extra in monolith proxy images (#39223)
The Dockerfile, docker/Dockerfile.non_root and docker/Dockerfile.database uv sync stages never passed --extra bedrock-realtime, so aws-sdk-bedrock-runtime was absent from the image venv and Bedrock Nova Sonic /v1/realtime sessions failed with 'Missing aws_sdk_bedrock_runtime'. gateway/Dockerfile already had the extra (PR #34426).

Adds a static check over every uv sync in the proxy Dockerfiles and an image-level import probe that the image-scan workflow runs against the built root, non-root and gateway images.

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-01 17:51:55 -07:00
Kolade Fajimi
d1320404fe
fix(redis): coerce env var string types and fix param discovery through decorator wrappers (#30644)
* fix(redis): coerce env var string types and fix param discovery through decorator wrappers

inspect.getfullargspec doesn't work on redis.Redis/redis.RedisCluster because
their __init__ is wrapped by @deprecated_args, which replaces the explicit
signature with *args/**kwargs internally. getfullargspec returns an empty arg
list, so _get_redis_kwargs and _get_redis_cluster_kwargs silently dropped every
real constructor parameter not in their hand-picked include_args set --
cluster_error_retry_attempts and connection_error_retry_attempts among them, so
an operator's configured retry bound never reached the Redis Cluster client and
it fell back to redis-py's own default instead.

Rebased onto litellm_internal_staging, which had independently added
_init_arg_names (MRO-walking, inspect.unwrap-based) for the same class of bug
in _get_redis_url_kwargs. Reused that pattern (as _unwrapped_init_args, without
the MRO walk: redis.Redis/RedisCluster declare every real parameter directly on
their own __init__, and MRO-walking breaks the tests here that mock the class
with autospec=True, since inspect.getmro needs a real __mro__) rather than
introducing a second, differently-shaped fix for the same problem.
_get_redis_cluster_kwargs now also honors its own client argument instead of
ignoring it, so the async cluster client's own extra constructor kwargs
(cluster_error_retry_attempts, connection_error_retry_attempts,
decode_responses, ...) are no longer filtered out by introspecting the sync
class regardless of which client is actually built.

Also fixes environment variables and Helm --set values always arriving as
strings: redis-py 8.x changed health_check_interval's arithmetic to require a
real number, so a stringified value raised TypeError on every Redis operation
instead of connecting. _coerce_redis_kwargs_types coerces to each parameter's
declared type at the end of _get_redis_client_logic, with an explicit type
table for max_connections/socket_timeout/socket_connect_timeout since redis-py
8.x changed the timeout defaults from None to int 5, which would otherwise
make a fractional value fail int() and get dropped.

Co-authored-by: mangabits <1457532+mangabits@users.noreply.github.com>

* ci: verify redis-py client version compatibility across a version matrix

* test(redis): assert an async-only cluster kwarg every matrix version declares

connection_error_retry_attempts is on the async cluster constructor in redis-py
5.x only; 6.0 removed it in favor of retry. The 6.4.0, 7.4.1 and 8.0.1 legs were
failing on that missing parameter name rather than on the behavior under test,
while the allow-list itself was doing the right thing on all four versions.

decode_responses is async-cluster-only on every version the matrix covers, so it
stands in for the same property: the sync cluster class takes it through **kwargs
and never names it in its signature. Reverting _get_redis_cluster_kwargs to ignore
its client argument still fails both tests on 5.3.1 and 8.0.1.

test_async_cluster_passes_async_only_kwargs now builds the real async cluster
client and reads connection_kwargs off it, so it no longer needs a patched class
factory; the constructor does no I/O. The retry-attempts test keeps its patch,
since redis-py >= 6 stores no cluster_error_retry_attempts attribute on the built
client and the constructor call is the only place the forwarded value shows up.

The _get_redis_cluster_kwargs docstring cited the same two parameters as its
examples of async-only kwargs, which is what made the test look reasonable;
cluster_error_retry_attempts is on both classes and connection_error_retry_attempts
is gone from 6.0 on, so it now names decode_responses instead.

* test(redis): drop internal patches from the kwarg coercion tests

The test-quality gate flagged the new patch() calls on litellm internals these
tests added. Three of them faked litellm._redis.inspect.signature with a MagicMock
to hand _coerce_redis_kwargs_types a synthetic parameter; that function already
takes a client argument, so they pass stub functions instead, matching the
_redis_signature_8x idiom the file uses elsewhere. The fourth patched
_redis_kwargs_from_environment to {} to prove _get_redis_client_logic raises
without a host or url, which clearing the real env keys through
_get_redis_env_kwarg_mapping does without pinning the test to that call.

Both files now sit one TQ008 below the merge base rather than six above it.

* fix(redis): keep the sync client construction inside the basedpyright budget

_get_redis_client_logic now returns dict[str, object] rather than an untyped
dict, which is the honest type for operator-supplied config, but it turns the
33 reportUnknownArgumentType errors at redis.Redis(**redis_kwargs) into 33
reportArgumentType errors plus one reportCallIssue, both over their budget.
No static type fits: redis-py's constructor declares 40-odd differently typed
parameters and the values arrive from config and env, so the allow-list and
coercion above are derived from that same signature and redis-py validates each
value itself at runtime.

The two suppressions name their exact rule and carry that reason. The file ends
up 42 basedpyright errors below the merge base, with reportArgumentType and
reportCallIssue back at the base counts of 3 and 0.

* fix(redis): coerce cluster-only and None-default bool kwargs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mangabits <1457532+mangabits@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-31 20:51:31 -07:00
Yujong Lee
9b774d3dcf
fix(ci): isolate editable Cargo cache namespace 2026-08-31 17:25:28 -07:00
Yujong Lee
849269d52d
build(rust): configure native extension profiles 2026-08-31 17:25:28 -07:00
Mateo Wang
d60e77ae8c
Merge pull request #38819 from BerriAI/litellm_fix_gemini_tts_response_format
fix(speech): stop forwarding response_format as a chat param for Gemini TTS
2026-08-31 15:56:18 -07:00
mateo-berri
b34064fe30 ci: add tests/test_litellm/endpoints to the misc unit-test shard 2026-08-29 15:21:39 -07:00
mateo-berri
539bc8ef92 fix(ci): close only identical-title duplicates, dry-run the sweep, reopen on reply
The merged detector's 0.6 flag threshold had become the close bar, and 6 of the 7 real flagged pairs at 85% or more were not duplicates. The sweep now closes only when an older open issue has the identical normalized title, measures the grace period from the latest bot notice, and leaves the issue open when anyone replies or gives the notice a thumbs down.

A reporter cannot reopen an issue the bot closed, so a reporter comment after the automatic close reopens it, drops the duplicate label, and asks for a human look. Manual dispatch defaults to a dry run and takes a grace_period_days input, the runner supplies the repository, the dead python closer is gone, and the decision core has bun tests on a PR-triggered job.
2026-08-29 17:30:01 -04:00
mubashir1osmani
f4542d9605 fix(ci): pin the Bun runtime instead of tracking latest
The setup step ran `bun-version: latest`, carried over from the upstream layout,
and the step after it holds an issues: write token. A compromised Bun release
would have executed privileged in that job and could rewrite or close issues.

Pinned to 1.4.0, the release the passing runs already resolved to. setup-bun
takes no checksum input, so pinning the action by sha and the runtime by exact
version is as far as this can be hardened without hand-rolling the download.
2026-08-29 17:29:56 -04:00
mubashir1osmani
3ea11b64e6 refactor(ci): run the duplicate sweep as a Bun TypeScript script
Moves the sweep out of inline workflow JavaScript and into scripts/, following
the layout anthropics/claude-code uses for the same job: a checked-out repo, a
sha-pinned setup-bun step, and `bun run scripts/auto-close-duplicates.ts`.

The script mirrors that repo's file shape, keeping the same request helper,
interfaces, per-issue debug logging, and top-level catch, so the two read the
same way side by side.

Two things stay deliberately different. Candidates come from the notice marker's
digits-only field rather than a regex over the comment prose, because titles are
attacker-controlled and are interpolated into that same comment. The label is
also added on its own endpoint instead of alongside the state change, since
sending labels with a PATCH replaces every label already on the issue.
2026-08-29 17:29:56 -04:00
mubashir1osmani
af340c0240 fix(ci): read duplicate candidates from the marker, not the notice prose
The notice interpolates each candidate's title, and the sweep scanned the whole
comment for issue references and took the lowest. Titles are attacker-controlled,
so filing a candidate titled "... see #1" redirected the closure: any later report
matching that candidate would be closed as a duplicate of #1 instead.

The detector now emits the candidate numbers as a digits-only field inside the
marker, built from the API's number field, and the sweep reads only that. Prose is
never parsed, so nothing a reporter can type reaches the target selection.
2026-08-29 17:29:55 -04:00
mubashir1osmani
e2ffb6b01c feat(ci): close duplicate issues after a 3-day grace period
Duplicate detection already labelled and commented on new issues, and then
closed them outright at 0.85 title similarity. That gave the reporter no chance
to push back, and a title-similarity match is not strong enough evidence to
close on its own.

Detection now only flags. A new daily sweep closes a flagged issue three days
later, and only if nobody engaged with the flag. Replying to it, thumbs-downing
it, or applying an opt-out label all keep the issue open. The notice says all of
that up front, so the reporter knows what happens and how to stop it.

The two workflows hand off through an HTML marker in the comment body rather
than its prose, so rewording the notice cannot silently break the sweep. The
sweep lists by label instead of walking the whole backlog: 1663 open issues
against 23 carrying the label meant a comments request each, which would burn
the Actions token's hourly budget for a handful of matches.

Candidates are taken as the lowest issue number, not the first one listed. The
detector orders by score rather than age, so the first candidate can be newer
than the issue being closed, and folding an original report into a later one is
backwards. An issue whose only candidates are newer is skipped.

Closures use state_reason=duplicate rather than not_planned, which reads as
"see the other issue" instead of "we are not doing this".

Also drops {{html_url}} from the notice. The detection action only exposes
number, title and accuracy, so that placeholder had been rendering empty and
every "similar issue" link in the comment pointed nowhere.
2026-08-29 17:29:55 -04:00
devin-ai-integration[bot]
194a3cc202
ci: build the benchmark environment outside the CodSpeed runner (#38426)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-29 13:58:04 -07:00
devin-ai-integration[bot]
f7fb3694f8
feat(terraform): coverage-enforcing CI gate against the latest OpenAPI spec (#38710)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-28 17:11:43 -07:00
mateo-berri
f46f66d101 fix(ci): fall back to github.token when the GH_TOKEN secret is unset in the Together sync workflow 2026-08-28 12:24:05 -07:00
yuneng-jiang
0eb7c3ad05
feat(proxy): add paginated GET /public/v1/model_hub (#38636)
* refactor(proxy): move the shared list framework to a surface-neutral package

The list framework and its RFC 9457 problem machinery sat under
management_endpoints/management_v1/, which was the right home while
/management/v1 was its only consumer. The public surface is about to build
on the same framework, and a control-plane package is the wrong thing for a
public route to import.

Moves list_framework.py in full, plus everything in common.py except
MANAGEMENT_V1_PREFIX, to litellm/proxy/list_api/. Every importer is updated
directly instead of leaving re-export shims, so each symbol keeps exactly
one import path. ManagementProblem keeps its name: renaming it would touch
the app-wide exception handler and every call site for no behavioural gain.

The framework's own tests move alongside the code they cover. The fastapi
removed-name guard in test_common.py now globs both packages, so budgets.py
and spend_logs.py stay covered after leaving the framework's directory.

Pure move, no behaviour change: the 179 tests across both packages pass
unchanged.

* feat(proxy): add paginated GET /public/v1/model_hub

The public Model Hub page loads every public model group in one call.
Measured on a live proxy with 300 published groups, /public/model_hub
answers with 328 KB in a single response and the page renders all 300 rows
into the DOM. At a few thousand models that is multiple megabytes and a
page that stops responding, which is what a customer reported.

Adds GET /public/v1/model_hub, the first resource on the unauthenticated
/public/v1 surface. It is built on the shared list framework, so it gets
the {data, meta, links} envelope, RFC 9457 problems, strict unknown and
duplicate query parameter rejection, and sort validation without
reimplementing any of it. Sorting covers model_group, mode, the token
limits and the per-token costs, `q` searches model_group, and the filters
are the ones the page actually offers: mode and providers. Default sort is
alphabetical, which is what a browse list wants and what these rows can
support: they carry no creation timestamp.

/public/model_hub is untouched. The shipped UI still calls it and its
migration is a separate change, so this is purely additive alongside it.

Model hub rows are computed off the running router rather than read from a
table, so this adds InMemoryListExecutor: the same QueryPlan applied in
Python instead of rendered to SQL. It matches the SQL executors where it
counts, NULLS LAST in both sort directions and NULL satisfying no
comparison, so a filter means the same thing on either. The other three
public hubs have the same shape and can reuse it as is.

The fix itself is ordering. The endpoint being superseded reads every
latest health check and joins it against the whole model list, so paging
the response alone would have changed nothing. Here the health lookup is
an injected dependency the executor calls on the page slice, after the
filter and the sort, so it resolves health for the rows being served and
no others. PrismaClient gains a bounded read for that, next to the
unbounded one it mirrors. The regression test pins the ordering by
asserting which model groups the lookup is asked about, and fails against
an enrich-then-slice implementation.

* fix(proxy): address self-review findings on the public model hub list

Five adversarial review passes over the branch. What they found:

`is_null` was the one predicate in the in-memory executor that read a
repeated field's container instead of its elements, so a field holding only
nulls was indistinguishable from a populated one. It now lifts over elements
like every other predicate does. Not reachable through this endpoint, whose
only repeated field grants `contains` alone, but the executor is written to
be reused by the other three hubs and the inconsistency was a trap for them.

The fastapi removed-name guard globbed the framework packages but not
`public_endpoints/public_v1`, which `proxy_server` also imports unguarded at
module level, so the new package had none of the protection the test claims
to give. It now covers all three.

Regenerates the dashboard's API types, which the OpenAPI sync check requires
whenever the proxy's route surface moves. The diff is the 65 generated lines
for the new operation and nothing else; no dashboard code changes here.

Also trims comments and docstrings that argued for a decision or restated a
signature rather than explaining code, and wraps a docstring line that ran
past 120 characters.

* ci: run the relocated list framework tests in the proxy-endpoints shard

The framework's tests moved from tests/test_litellm/proxy/management_endpoints,
which the proxy-endpoints shard claims, into a new tests/test_litellm/proxy/list_api
that no shard named. Both coverage guards caught it: the semantic shards have no
catch-all bucket, so the directory would have run nowhere.

Claims it alongside management_endpoints, where the same tests ran before.

* docs(proxy): stop restating the list spec in the model hub route docstring

The docstring listed every sortable field, the page-size cap and the filter
set, all of which already live in MODEL_HUB_LIST_SPEC and all of which the
endpoint hands back in the allowed array of a rejected request. Two copies of
one spec is a prose update owed on every change to the real one.

Keeps what a caller cannot derive from the endpoint itself: what the resource
is, that it needs no authentication, and a working example. Regenerates the
dashboard types, which carry the docstring as the operation description.

* fix(proxy): reject a repeated sort field instead of sorting by it twice

sort took any number of comma-separated keys, and the in-memory executor runs
one full sorted() pass per key before slicing. Naming one allowed field N times
therefore bought N passes over every published model group, synchronously on the
event loop, from a route that needs no credentials. Measured on 300 groups:
0.001s for one key, 0.034s for a thousand, 0.166s for five thousand, and it
grows with the catalogue this endpoint exists to make large.

A repeated field cannot change the ordering, so rejecting repeats costs a caller
nothing and bounds the passes at len(sortable), a number the spec author picks
rather than the caller. That beats an arbitrary cap: no magic number, and the
bound holds for every resource built on the framework.

The tiebreaker is appended after parsing, so sorting explicitly by it stays legal.
Budgets renders one ORDER BY in SQL and never had the amplification, but the
check belongs with the rest of the sort validation rather than in one executor.

* fix(proxy): make the search disjunction one level deep by type

Two CI gates, one cause. AnyOf declared its clauses as Predicate, so both
consumers had to recurse to evaluate one: the SQL renderer through
_render/_render_all, and the in-memory executor through _holds. The recursion
detector flags the latter, and its reason is the same one this PR already ran
into once, a caller-controlled cost that shows up as CPU.

Nothing actually builds a nested AnyOf. _search_predicate is its only producer
anywhere in the repo and it emits Compare leaves, in every call site and every
test. Declaring clauses as tuple[Compare, ...] makes that a fact the type
checker keeps rather than a comment, and _holds then evaluates a disjunction of
leaves with no recursion at all.

Also marks the new health read's broad except, which the strict gate counts,
and covers the ordering comparison operators. The endpoint exposes only
eq/in/contains, so gt/gte/lt/lte were live code no test evaluated.

* fix(proxy): keep the new health read inside the type-discipline ceiling

The bounded health query added ten LIT002 violations, which pushed the
codebase total past its budget. The gate counts across the tree and compares
to the merge base, so a file already carrying debt does not absorb new
violations.

Returns an empty tuple rather than an empty list on the two no-result paths:
the signature already promises a Sequence, so that is a free two-violation
reduction and a better type. Builds prisma's order argument from a tuple of
pairs, which turns four literals into one. The three that remain are prisma's
own API shape and each carries its reason.

Both budget gates now pass against the merge base.

* fix(proxy): clear the two basedpyright errors the new route added

The type-check budget is over its ceiling on the base already, so the gate
blames any increase: reportArgumentType 2574/2564 and reportPrivateUsage
1815/1808, one each, both from this file.

fastapi types a route's tags as list[str | Enum], so the tuple was an argument
error; budgets.py has the same one and it is part of what put the rule over.
Passing a list is what the signature asks for, marked because an inline list
is a construction the discipline gate counts.

_get_model_group_info is private by name but is the shared reader the endpoint
this supersedes imports the same way, so the import carries a rule-scoped
ignore with that reason rather than a copy of the function.

basedpyright now reports zero errors across both new modules, and all three
budget gates pass against the merge base.
2026-08-28 10:02:59 -07:00
Mateo Wang
5337c68dd3
Merge pull request #38257 from BerriAI/litellm_together_registry_sync
feat(models): add daily Together AI model registry sync script and workflow
2026-08-27 19:46:37 -07:00
mateo-berri
afe5a240e5 fix(proxy): regenerate lazy OpenAPI snapshot and guard it in CI
The committed snapshot behind /openapi.json for unloaded lazy features had drifted on 30 of 31 fragments and never had one for a2a_registration or gemini_agents, so those routes showed as placeholder GET stubs or old docstrings until traffic loaded them. Regenerate the snapshot and schema.d.ts, make the check-ui-api-types job and make check regenerate the snapshot and fail on drift, and make the generator refuse to write a snapshot when any feature fails to import so a broken import cannot silently drop fragments.
2026-08-26 14:32:04 -07:00
Yuneng Jiang
ac1eb1029a fix(ci): name the file mutmut actually writes partial results to
The step-timeout comment claimed mutmut streams each mutant's result into
mutants/mutmut-stats.json. It does not. That file holds the pre-run test
timings and coverage map (tests_by_mangled_function_name, duration_by_test,
stats_time) written once by save_stats() before mutation starts.

Per-mutant results live in mutants/<source path>.meta. Verified against
mutmut 3.5.0: SourceFileMutationData.register_result() calls save() after
every single result, and export-cicd-stats walks those .meta files to build
mutmut-cicd-stats.json. So the reason the step deadline exists is still
right, an interrupted run keeps the mutants it already scored, but the
comment pointed at the wrong file.

Also upload the .meta files, since they are the partial results the comment
relies on and the artifact could not otherwise show them.
2026-08-25 23:54:48 -07:00
Yuneng Jiang
ae63786cfb
fix(ci): let the mutation workflow find covered lines so it generates mutants
mutmut's gather_coverage() looks each source file's covered lines up by
absolute path, but [tool.coverage.run] sets relative_files = true, so every
lookup misses. With mutate_only_covered_lines = true that leaves no line
eligible for mutation, and the run ends on "Stopping early, because we could
not find any test case for any mutant" after spending 26 minutes collecting
coverage. The last four dispatches all died that way.

Point COVERAGE_RCFILE at a small rc file for mutation runs only, so the
coverage instance mutmut builds stores absolute paths. Scoped to one module
locally this takes the run from 0 mutants to 8 generated and 8 killed.

Also give the mutmut step a deadline inside the job's own. mutmut records
each mutant's verdict to mutants/mutmut-stats.json as it finishes, so a run
that outlasts its budget still scores what it got through, but a cancelled
job skips the report and upload steps and publishes nothing. That is how the
two runs before these four ended.

Ignore mutants/ and .venv-mutmut, which a local run leaves behind untracked.
2026-08-25 23:16:40 -07:00
devin-ai-integration[bot]
bb22742025
fix(rerank): emit latency and cost headers on /rerank (#35419)
* fix(rerank): emit latency and cost headers on /rerank

Thread the logging object into the rerank httpx calls and pass hidden_params through to get_custom_headers, so x-litellm-overhead-duration-ms, x-litellm-response-duration-ms, x-litellm-response-cost, x-litellm-call-id and the LITELLM_DETAILED_TIMING x-litellm-timing-* headers show up on rerank like they do on chat completions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(rerank): keep zero response cost in the /rerank cost header

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: assign the new rerank endpoint tests to the proxy-endpoints shard

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: suppress TQ008 on the rerank header tests with reasons

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
2026-08-25 15:54:25 -07:00
Mateo Wang
296ebc1103
Merge pull request #38252 from BerriAI/litellm_pr_template_caveat_severity
docs(pr-template): split Caveats bullets into severity tiers and call for plain engineering language
2026-08-25 15:04:37 -07:00
mateo-berri
56c2cceeaa fix(ci): raise the open-PR listing limit so the sync-PR guard sees every open PR 2026-08-25 13:42:28 -07:00
mateo-berri
90bc8acd86 feat(models): add daily Together AI model registry sync script and workflow 2026-08-25 13:12:06 -07:00
mateo-berri
8829b3269f docs(pr-template): group caveats under severity subheadings and promote Final Attestation 2026-08-25 12:52:54 -07:00
mateo-berri
d749b186de docs(pr-template): make intent the severe-vs-high discriminator 2026-08-25 12:34:31 -07:00
mateo-berri
2bd2c1393c docs(pr-template): split Caveats bullets into severity tiers and call for plain engineering language 2026-08-25 12:31:43 -07:00
ryan-crabbe-berri
5745313448
Merge pull request #38124 from BerriAI/litellm_codeowners_ui_infra_exemptions
chore(codeowners): unown ui container plumbing and generated files
2026-08-24 20:59:55 -07:00
Mateo Wang
53c9d48bd2
ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163)
Adds a scheduled GitHub Actions lane on top of the merged record/replay
transport. A Saturday cron records the `replayable` e2e tests against the
real providers and publishes the fixture bundle as a private
`e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull
that artifact by its pinned digest, verify the checksum before extracting,
and replay it with provider credentials set to bogus values, so a run that
ever reached a real provider fails instead of passing.

An egress sentinel pins the provider hostnames to a local sink for the whole
replay job and counts every connection that reaches them; the job asserts
that count is zero, so hermeticity is proven by measurement. A red Saturday
publishes no bundle, so the next weekday finds nothing fresh and fails loudly
rather than replaying a week-old recording, and the transport's seven-day
freshness gate hard-fails any bundle that has drifted too far. The lane also
runs on demand from the Actions tab with a record/replay `mode` input.

Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic
test now counts to twenty so its recorded response banks several content
deltas, matching the assertion that the stream arrives incrementally.
2026-08-24 23:49:03 -04:00
Mateo Wang
539ba9f939
Merge pull request #37899 from BerriAI/litellm_ban_data_migrations
ci: ban row-rewriting DML from prisma migrations
2026-08-24 19:44:34 -07:00
tin-berri
f1f6d83d47
fix(ci): give three unit shards a job deadline that outlasts their pytest budget (#38139)
The caching-local, proxy-extras and enterprise-package shards each budget
pytest 20m but cap the whole job at 55m. Setup can consume up to 35m, and
the runner adds 5m of overhead, so the job deadline can preempt pytest
inside its own advertised budget and the shard dies without a test report.

check_workflow_startup_safety enforces that invariant and is currently
failing on litellm_internal_staging, which reds the code-quality job for
every open PR. Raising the three caps to 60m satisfies 20 + 35 + 5.
2026-08-24 13:26:21 -07:00
ryan-crabbe-berri
5ed83942dc chore(codeowners): unown ui container plumbing entirely
The Dockerfile and nginx.conf are deploy infra, so nobody on the dashboard side needs to gate them. Drop the remaining owner instead of making one person the sole required approver.
2026-08-24 12:04:17 -07:00
ryan-crabbe-berri
79d0d7a48e chore(codeowners): drop ryan-crabbe-berri from ui infra and generated files
The /ui/ rule sweeps in the container plumbing (Dockerfile, nginx.conf) and the checked-in tsbuildinfo, none of which are dashboard code. Exempt them so review requests land on the people who actually own that surface.
2026-08-24 11:59:12 -07:00
devin-ai-integration[bot]
a1134755ca
fix(ui): boot the UI image as an arbitrary uid by anchoring nginx writes under /tmp (#37982)
* fix(ui): boot the UI image as an arbitrary uid by anchoring nginx writes under /tmp

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): type the arbitrary-uid image test fixture

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-24 11:57:36 -07:00
yuneng-jiang
d1f3778849
perf(ci): give the two longest unit shards the runner's spare cores (#37804)
proxy-endpoints and proxy-infra are the unit tier's critical path at 358s and
325s of pytest, measured on staging 2026-08-21, and both run two xdist workers
on a four-vCPU runner. proxy-server already runs four. This is the cheaper half
of splitting them: no second job, so no second setup to pay for.
2026-08-22 22:58:26 -07:00
yuneng-jiang
68489f62ff
ci: run the enterprise package suite in GitHub Actions (#37798)
tests/enterprise is 13 files and 244 tests that only CircleCI runs, and CircleCI
gates nothing: it triggers on PR labeled events, none of its jobs are required,
and red runs get merged past. So the suite that covers the enterprise package's
guardrails, auth and management endpoints has had no say in whether a change
lands.

Measured on 2026-08-21 with every credential stripped from the environment: 240
passed, 4 skipped, nothing failed. It needs no provider key, so it can be a
required shard rather than a scheduled lane, unlike the other CircleCI suites in
this group, which each carry a live-API minority.

The CircleCI job is removed in the same commit so the suite runs once, not twice.
2026-08-22 22:57:50 -07:00
yuneng-jiang
ae0e8a20db
fix(ci): run the migration DDL guard, and stop it reading comments as SQL (#37791)
* fix(ci): make the migration DDL guard run, and stop it reading comments as SQL

TestMigrationSQLIdempotency requires guarded DDL across litellm-proxy-extras
and has never run in any job, so the convention eroded quietly. Four of its
assertions fail today, and it was allowlisted rather than wired up because
fixing the migrations is not an option: Prisma checksums an applied migration,
so editing one breaks `migrate deploy` for every existing install.

Two things were wrong with the guard itself. It scanned raw lines, so Prisma's
own `-- CREATE INDEX CONCURRENTLY ...` explanations counted as the statements
they describe, which is two of the reported migrations. And it had no way to
say "these predate the rule", so the only options were editing immutable files
or leaving the whole file unrun.

Comments are now stripped before matching, on the drop-column rule too, and the
migrations that already violate are named once in _PRE_GUARD_MIGRATIONS. The
rules bind everything after them, so a new migration with bare CREATE TABLE,
ADD COLUMN, CREATE INDEX or an unguarded ADD CONSTRAINT now fails a check
instead of landing unnoticed.

That set is 14 migrations, not the 13 previously recorded, measured after
comment-stripping. It can only shrink: a test fails if an entry names no
migration on disk, and another fails if an entry no longer violates anything.

The file now runs as a proxy-extras shard and comes off the coverage allowlist.

* fix(ci): strip block comments in the migration guard too

Prisma opens a destructive migration with a /* Warnings: You are about to
drop the column ... */ header. Nothing in the tree trips a rule on that text
today, but it is prose about a statement rather than the statement, and the
line-comment fix left the class open. Bodies are blanked rather than removed
so the reported line number still points at the real statement.
2026-08-22 22:57:16 -07:00
yuneng-jiang
b31484ed19
ci: run the keyless caching tests that ran in no job (#37790)
The allowlist recorded eight files in tests/local_testing, 118 tests, that
every job globbing that directory then deselects: local_testing_part1 and
part2 carry `-k "... and not caching and not cache"`, and the other three keep
one unrelated keyword each. They counted as covered while running nowhere.

Five of the eight need nothing. Measured with no provider credentials and no
Redis: test_cache_preset_key, test_caching_handler, test_prompt_caching,
test_responses_stream_cache_keys and test_unit_test_caching pass, 45 tests
together, and they now run as a caching-local shard. The other three stay
allowlisted with what they actually need recorded rather than a question:
test_caching wants Redis and a provider key for 37 of its 65, disk-cache wants
OPENAI_API_KEY for 2 of 4, gcs-cache wants GCS credentials for all 4.

Taking them off the allowlist exposed a gap in the slice guard itself: it
reasoned only about CircleCI `-k` expressions, so a file every slice drops read
as unrun even when a workflow names it outright. It now credits workflow
test-paths the way the census already does, and only workflows, so a tree only
CircleCI globs is still reported.
2026-08-22 22:56:43 -07:00