mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
7 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
ryan-crabbe-berri
|
0fcaadf11c
|
test(e2e): move Admin UI Playwright suite to tests/e2e/ui (#34196)
Relocates ui/litellm-dashboard/e2e_tests to tests/e2e/ui so all end to end suites live under tests/e2e. The suite stays in TypeScript and becomes a self-contained npm package with its own package.json, lockfile and tsconfig instead of leaning on the dashboard's toolchain; the dashboard drops its @playwright/test dependency, e2e scripts and knip/vitest/tsconfig carve-outs. CI paths follow the move: both CircleCI jobs (main e2e and the SERVER_ROOT_PATH migration smoke) and the test_server_root_path workflow now install and run Playwright from tests/e2e/ui, with the node cache keyed on both lockfiles. classify_changes.sh treats tests/e2e/ui as client so spec edits keep skipping backend jobs. The suite's mock LLM fixture is excluded from the e2e basedpyright zero-error gate in pyrightconfig.json since it belongs to the TS suite, not the typed Python harness. |
||
|
devin-ai-integration[bot]
|
56f4dbf60a
|
test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548)
* test(claude_code): move the Claude Code compatibility matrix under tests/e2e Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(claude_code): drop the CircleCI compat PR gate; the matrix runs in the scheduled e2e suite instead Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: restore the upload-coverage job dropped by mistake with the compat gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(e2e/claude_code): print rate-limit summary on failed compat runs and fix stale run_daily.sh header comments * test(claude_code): assert fine-grained tool streaming via input_json_delta instead of an event-count floor Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d0c2e87810
|
ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) (#30379)
* ci: enable ruff preview rules under the budgeted strict gate
Turn on ruff preview in the strict-budget lane (ruff-strict.toml) only,
leaving the clean gate (ruff.toml) untouched so make lint-ruff stays at
zero. Enumerate the 118 firing codes explicitly with
explicit-preview-rules so the gate is deterministic and stable across
ruff upgrades rather than depending on preview auto-selecting the broad
catalog.
Grandfather the existing 58438 violations into ruff-strict-budget.json
as per-rule baselines with headroom, so only net-new violations fail CI.
The existing ten rules keep their hand-tuned slack; the new rules get
slack 10 when the baseline is 50 or more and 3 otherwise.
* ci: add ANN return-type rules to the budgeted strict gate
Add ANN201/202/204/205/206 (missing return annotations) to the strict
lane and grandfather the existing counts into ruff-strict-budget.json so
the codebase ratchets toward explicit return types without breaking CI.
* ci: add mypy (disallow_untyped_defs) and basedpyright strict gates with baselines
Add two type-check gates, each grandfathering the current tree so only
net-new violations fail CI, matching the ruff strict-budget ratchet.
mypy gains disallow_untyped_defs in litellm/mypy.ini (the config the CI
invocation actually reads; the root [tool.mypy] is not picked up from the
litellm/ working dir). The 4885 existing missing-annotation errors are
captured in litellm/.mypy-baseline.txt and the run is piped through
mypy-baseline filter so new untyped defs are rejected.
basedpyright runs in strict mode over litellm/, with
enableTypeIgnoreComments disabled so it only honors '# pyright: ignore'
and never polices mypy's '# type: ignore'. The existing strict diagnostics
are grandfathered into .basedpyright/baseline.json.
Both tools are pinned in the dev group and uv.lock; the lint workflow and
Makefile run them filtered through their baselines, with
lint-mypy-baseline-update and lint-basedpyright-baseline-update to ratchet.
* ci: raise lint job timeout to 15m for the basedpyright strict pass
* ci: pin pythonVersion 3.12 and regenerate baselines against merged base
Merge litellm_internal_staging so the baselines cover code the CI merge
includes (e.g. the cisco_ai_defense guardrail), which otherwise tripped
the mypy gate with 3 ungrandfathered no-untyped-def errors. Pin
pythonVersion 3.12 in pyrightconfig so basedpyright's strict analysis is
reproducible across interpreter versions (CI runs 3.12).
* ci: regenerate basedpyright baseline against the frozen lint env
The previous baseline was generated with optional provider deps (azure,
google, anthropic, mcp, numpydoc, google-genai) installed locally, so CI's
dev-only env surfaced ~3500 reportUnknown*/reportMissingTypeStubs errors
not in the baseline. Regenerate after uv sync --frozen so the baseline
reflects the same dependency set the lint job sees.
* ci: regenerate basedpyright baseline on python 3.12 frozen env
The prior baseline still carried proxy-dev packages (e.g. prisma) that the
lint job's dev-only, python 3.12 env lacks, leaving 2 unresolved-import
errors ungrandfathered. Regenerate in a python 3.12 venv synced to the
frozen lock with default groups only, so the baseline matches exactly what
CI sees.
* ci: replace type-check baselines with per-file count budgets
The mypy and basedpyright baselines were position-sensitive (and the
basedpyright one was a 27MB file), so ordinary line shifts churned them.
Replace both with a per-file count gate: scripts/type_check_gate.py reduces
each tool's output to errors-per-file and checks it against a committed
{file: max} budget, ignoring line and column numbers. A file fails only
when it gains more errors than its ceiling; debt can't be shuffled between
files because each file has its own cap and new files default to zero.
Budgets (mypy-file-budget.json 48K, basedpyright-file-budget.json 96K) are
generated in the python 3.12 frozen lint env so they match CI. Drops the
mypy-baseline dependency; basedpyright runs without its native baseline.
ratchet via make lint-mypy-budget-update / lint-basedpyright-budget-update.
* ci: add a small per-file slack to the type-check gate
Allow each file to drift PER_FILE_SLACK (5) errors past its recorded count
before failing, so a basedpyright inference ripple in an unrelated file
doesn't break the build over a couple of errors. Budgets still record exact
counts; the tolerance is applied at check time.
* ci: move type-check slack into the budget json and trim lint timeout
Make slack declarative: the budget is now {"slack": N, "files": {path: count}}
so the tolerance is tuned in JSON without editing the script, mirroring how
ruff-strict-budget.json carries its slack. --update preserves the existing
slack. Also drop the lint job timeout from 15m to 10m; the mypy and
basedpyright passes add ~2m, leaving the job around 4-5m, so 10m is a
comfortable margin.
* ci: collapse fully-adopted ruff categories and drop inert preview flag
ANN (all nine non-removed rules) and BLE (its only rule) were spelled out
code-by-code; replace each with its category selector, which is exactly
equivalent in 0.15.3 (the removed ANN101/ANN102 are skipped by a category
selector and error when named explicitly). explicit-preview-rules was inert:
every selected rule is stable and nothing is selected by category, so the flag
had nothing to gate. Verified the strict-rule counts are identical before and
after (62379 each, zero per-rule drift), so no budget change.
* ci: drop redundant pyright dev dependency
Nothing invokes bare pyright in the Makefile, the linting workflow, or
scripts; the basedpyright gate added on this branch is the only type
checker that runs. basedpyright is a superset fork that reads the same
pyrightconfig.json and honors the same "# pyright: ignore" comments, so
pyright==1.1.408 in the ci group was dead weight. Regenerated uv.lock
under the same exclude-newer cutoff so the only change is removing
pyright and its package stanza
* ci: un-weaken mypy and error on Any in basedpyright
mypy: enable warn_return_any, drop the valid-type silencer, and stop globally ignoring missing first-party imports via [mypy-litellm.*] ignore_missing_imports = False, which surfaced eight real broken litellm.* imports the blanket ignore was hiding; third-party imports stay ignored. The per-file budget moves 4888 -> 5799 (902 no-any-return, 1 valid-type, 8 import-not-found), all grandfathered so only net-new errors fail and the ceilings ratchet down
basedpyright: error on reportExplicitAny and reportAny. The per-file budget moves 117033 -> 148946 (6931 explicit-Any, 24954 Any-typed expressions), grandfathered the same way
* ci: add Any-discipline gate on changed lines under litellm/
Add scripts/check_any_discipline.py, a type-aware gate that fails when a
changed line holds a value typed Any -- including the X | Any unions that
mypy --strict / basedpyright accept (e.g. re.Match.group() -> str | Any,
json.loads() -> Any, bare dict -> dict[Any, Any]).
It reuses the repo's mypyc-compiled mypy 1.19 via a custom generic AST
walker (mypyc precludes subclassing TraverserVisitor), loads litellm/mypy.ini
for parity with lint-mypy, and uses a dedicated incremental cache
(.mypy_cache_any) with mtime+hash invalidation to force re-checks. Scope is
changed-lines-only so editing a legacy file never forces cleaning its
existing Any debt; suppress a genuine typed/untyped boundary with
# any-ok: <reason> (ANY002 requires the reason).
Wire it into the Makefile (lint-any, lint, lint-dev), a parallel
any-discipline CI job with its own actions/cache, .gitignore, and the
CLAUDE.md / CONTRIBUTING.md docs.
* ci: move Any-gate codes into the shared LIT namespace
Renumber the Any-discipline checker into the LIT*** scheme owned by
scripts/check_type_discipline.py (PR #30500) so the two checkers share one
rule namespace and suppression convention:
ANY001 -> LIT002 (Any-typed value; LIT002 was the retired/free slot)
ANY002 -> LIT005 (any-ok without a reason; the shared suppression-reason code)
ANY000 -> LIT000 (setup/build/read error; the shared error code)
Messages and behavior are unchanged; LIT005's text already matches the
"<token> requires a reason" shape used for cast-ok/guard-ok.
* ci: gate mypy and basedpyright per error rule, not per file
Switch the mypy/basedpyright budget gate from per-file error counts to
per-rule-code totals, mirroring the {rule: {baseline, slack}} shape of
ruff-strict-budget.json. A rule fails when its codebase-wide error count
exceeds baseline + slack, so violations are tracked by category rather
than by file location.
scripts/type_check_gate.py now parses mypy from its text output (trailing
[code]) and basedpyright from --outputjson (the JSON `rule` field), since
basedpyright's wrapped text diagnostics mis-attribute the rule on
continuation lines. Replace the *-file-budget.json files with freshly
captured *-code-budget.json baselines and update the Makefile, CI, and
CLAUDE.md accordingly.
* docs: prefer Pydantic validation over any-ok suppression
Point the Any-discipline guidance at validating Any with Pydantic (a model
or TypeAdapter that returns a typed value or raises) and frame
# any-ok as a last resort that should ideally never be used.
* chore: remove extraneous comment
* chore: make the CLAUDE.md more concise
* chore: clean up bloated CONTRIBUTING.md additions
* chore: make Makefile more concise
* ci: add the lint-budget-update target CLAUDE.md references
CLAUDE.md tells contributors to run make lint-budget-update, but the
target was never defined. Add it as an aggregate that re-captures the
ruff, mypy, and basedpyright budgets in one shot.
* ci: recapture mypy and basedpyright budgets in the lint env
The per-rule baselines were captured in a richer dependency env than the
CI lint job's uv sync --frozen, so CI resolved fewer types and reported
more errors than the budgets allowed (no-any-return 902 over cap 900, plus
several basedpyright reportUnknown* rules). Regenerate both in the frozen
env so they grandfather the true CI debt: mypy 5786 -> 5799 (no-any-return
890 -> 902, valid-type 1 restored), basedpyright 146213 -> 148942.
* ci: check out PR head sha in lint and any-discipline jobs
The default pull_request checkout uses refs/pull/N/merge, which folds the
latest base commits into HEAD. The diff-based gates (ruff delta, Any
discipline) then diff against the event's older base.sha and blame base's
own new commits on this branch; staging's otel-v2 and streaming changes
(#30326, #30485) tripped the Any gate on files this branch never touched.
Checking out the PR head sha makes the gates diff the real branch tip
against base, and pins the tree the mypy/basedpyright budgets were captured
against so their counts stay deterministic as the base advances.
* ci(lint): renumber Any-typed-value rule LIT002 -> LIT009
Free up LIT002 for the sibling type-discipline gate (check_type_discipline.py,
#30500), which groups its mutable-collection family at LIT001 (annotation) and
LIT002 (construction). This gate's Any-typed-value rule moves to LIT009 so the
shared LIT namespace stays contiguous with no holes; LIT000 and LIT005 are
unchanged.
* style: rename lint-strict-budget -> lint-ruff-budget
* ci: harden type-check gates against silent passes (greptile review)
type_check_gate.py: refuse to certify a vacuous run. The CI pipe swallows
the tool's exit code ('tool || true'), so a crashed mypy/basedpyright that
emits nothing would parse to zero errors, breach no ceiling, and pass.
is_vacuous_run() now fails when nothing was parsed but the budget expects
errors. Also wrap basedpyright's json.loads in a JSONDecodeError handler
that prints the offending output instead of dumping a raw traceback.
check_any_discipline.py: ALL_LINES was None, which dict.get() also returns
for a path absent from the line map, so a path-normalisation mismatch could
let a violation on an unchanged file pass the scope filter. Make ALL_LINES a
distinct sentinel object so 'whole file' and 'path missing' are unambiguous.
Adds tests for all three.
|
||
|
Krish Dholakia
|
8097fafc05
|
Agents - support agent registration + discovery (A2A spec) (#16615)
* fix: initial commit adding types * refactor: refactor to include agent registry * feat(agents/): endpoints.py working endpoint for agent discovery * feat(agent_endpoints/endpoints.py): add permission management logic to agents endpoint * feat: public endpoint for showing publicly discoverable agents * feat: make /public/agent_hub discoverable * feat(agent_endpoints/endpoints.py): working create agent endpoint adds dynamic agent registration to the proxy * feat: working crud endpoints * feat: working multi-instance create/delete agents * feat(migration.sql): add migration for agents table |
||
|
Krish Dholakia
|
fac3b2ee42
|
Add pyright to ci/cd + Fix remaining type-checking errors (#6082)
* fix: fix type-checking errors * fix: fix additional type-checking errors * fix: additional type-checking error fixes * fix: fix additional type-checking errors * fix: additional type-check fixes * fix: fix all type-checking errors + add pyright to ci/cd * fix: fix incorrect import * ci(config.yml): use mypy on ci/cd * fix: fix type-checking errors in utils.py * fix: fix all type-checking errors on main.py * fix: fix mypy linting errors * fix(anthropic/cost_calculator.py): fix linting errors * fix: fix mypy linting errors * fix: fix linting errors |
||
|
Krish Dholakia
|
98c335acd0
|
LiteLLM Minor Fixes & Improvements (09/17/2024) (#5742)
* fix(proxy_server.py): use default azure credentials to support azure non-client secret kms * fix(langsmith.py): raise error if credentials missing * feat(langsmith.py): support error logging for langsmith + standard logging payload Fixes https://github.com/BerriAI/litellm/issues/5738 * Fix hardcoding of schema in view check (#5749) * fix - deal with case when check view exists returns None (#5740) * Revert "fix - deal with case when check view exists returns None (#5740)" (#5741) This reverts commit |
||
|
Krish Dholakia
|
60709a0753
|
LiteLLM Minor Fixes and Improvements (09/13/2024) (#5689)
* refactor: cleanup unused variables + fix pyright errors * feat(health_check.py): Closes https://github.com/BerriAI/litellm/issues/5686 * fix(o1_reasoning.py): add stricter check for o-1 reasoning model * refactor(mistral/): make it easier to see mistral transformation logic * fix(openai.py): fix openai o-1 model param mapping Fixes https://github.com/BerriAI/litellm/issues/5685 * feat(main.py): infer finetuned gemini model from base model Fixes https://github.com/BerriAI/litellm/issues/5678 * docs(vertex.md): update docs to call finetuned gemini models * feat(proxy_server.py): allow admin to hide proxy model aliases Closes https://github.com/BerriAI/litellm/issues/5692 * docs(load_balancing.md): add docs on hiding alias models from proxy config * fix(base.py): don't raise notimplemented error * fix(user_api_key_auth.py): fix model max budget check * fix(router.py): fix elif * fix(user_api_key_auth.py): don't set team_id to empty str * fix(team_endpoints.py): fix response type * test(test_completion.py): handle predibase error * test(test_proxy_server.py): fix test * fix(o1_transformation.py): fix max_completion_token mapping * test(test_image_generation.py): mark flaky test |