mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
21 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Mateo Wang
|
53c9d48bd2
|
ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163)
Adds a scheduled GitHub Actions lane on top of the merged record/replay transport. A Saturday cron records the `replayable` e2e tests against the real providers and publishes the fixture bundle as a private `e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull that artifact by its pinned digest, verify the checksum before extracting, and replay it with provider credentials set to bogus values, so a run that ever reached a real provider fails instead of passing. An egress sentinel pins the provider hostnames to a local sink for the whole replay job and counts every connection that reaches them; the job asserts that count is zero, so hermeticity is proven by measurement. A red Saturday publishes no bundle, so the next weekday finds nothing fresh and fails loudly rather than replaying a week-old recording, and the transport's seven-day freshness gate hard-fails any bundle that has drifted too far. The lane also runs on demand from the Actions tab with a record/replay `mode` input. Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic test now counts to twenty so its recorded response banks several content deltas, matching the assertion that the stream arrives incrementally. |
||
|
yuneng-jiang
|
b31484ed19
|
ci: run the keyless caching tests that ran in no job (#37790)
The allowlist recorded eight files in tests/local_testing, 118 tests, that every job globbing that directory then deselects: local_testing_part1 and part2 carry `-k "... and not caching and not cache"`, and the other three keep one unrelated keyword each. They counted as covered while running nowhere. Five of the eight need nothing. Measured with no provider credentials and no Redis: test_cache_preset_key, test_caching_handler, test_prompt_caching, test_responses_stream_cache_keys and test_unit_test_caching pass, 45 tests together, and they now run as a caching-local shard. The other three stay allowlisted with what they actually need recorded rather than a question: test_caching wants Redis and a provider key for 37 of its 65, disk-cache wants OPENAI_API_KEY for 2 of 4, gcs-cache wants GCS credentials for all 4. Taking them off the allowlist exposed a gap in the slice guard itself: it reasoned only about CircleCI `-k` expressions, so a file every slice drops read as unrun even when a workflow names it outright. It now credits workflow test-paths the way the census already does, and only workflows, so a tree only CircleCI globs is still reported. |
||
|
yuneng-jiang
|
648c6e7dc5
|
feat(ci): assert .github/workflows holds only workflows, correctly named (#37616)
* feat(ci): assert .github/workflows holds only workflows, correctly named * style(tests): annotate the hygiene test module's names with Final * fix(ci): report a .yaml workflow as a naming finding, not a stray GitHub reads .yml and .yaml alike, so WF001 telling you to move a valid .yaml workflow to .github/scripts/ was wrong advice. WF001 now covers only files that are not workflows at all, and the .yml spelling this directory keeps moves to WF004, which says to rename rather than relocate. WF001 also never looked into subdirectories, since GitHub does not read them either; the message now says so. The directory is injected rather than read off a module constant, so the cases are testable without monkeypatching. |
||
|
yuneng-jiang
|
cde134488c
|
test(ci): reject coverage-allowlist entries that no longer match a file (#37608)
* test(ci): reject coverage-allowlist entries that no longer match a file * fix(ci): match a dockerfile allowlist entry the way the census exempts one |
||
|
yuneng-jiang
|
a48baefc95
|
feat(ci): catch files a -k expression deselects from every job (#37601)
* feat(ci): catch files a -k expression deselects from every job The coverage census asks whether some job names a file. It cannot ask what that job's -k then does with it, and the gap is not hypothetical: tests/local_testing is globbed by five jobs, two of which carry -k "... and not router and not assistants and not langfuse and not caching and not cache" while the other three keep one keyword each. Any file whose path holds an excluded term is dropped by the first two and matched by none of the rest, so it runs nowhere while the census counts it as covered. 118 tests across eight caching files sit in exactly that hole today. The new mode reads the same CircleCI jobs the census already parses and asks whether each globbed file survives its job's selector. Two facts about -k make that decidable without running pytest: it matches an item's own name and its parents', so a term appearing in the module path deselects the whole file; and the names it can match are otherwise the classes and functions in the file, which ast reads. A positive term is therefore satisfied by the path or by a name inside, which is what keeps a langfuse-named test inside test_logging.py from being reported. Where the parser is unsure it stays quiet. An expression with or, parentheses, or a negated group is left unmodelled and its job is treated as claiming everything it globs, so an unparsed selector can never raise a false alarm. Glob translation learned character classes, without which tests/local_testing/**/test_[a-mA-M]*.py matches nothing and the guard would report that whole directory. The census and shard counts are unchanged by it, 2423 files and 327 shard children before and after. Validated against the real thing: collecting tests/local_testing under each job's own selector leaves 175 of 1577 tests unselected, in exactly the ten files this check derives statically, no more and no fewer. Two of the ten are named outright by other jobs, which the check credits, leaving the eight now recorded in the allowlist as a decision rather than an accident. Verified red-first: dropping one of those eight from the allowlist reports it, and adding 'and not embedding' to the two part jobs reports test_embedding.py and test_get_optional_params_embeddings.py. * fix(ci): keep the slice guard from pairing one command's -k with another's glob Two accuracy notes from review, both about the parser's model rather than its current verdicts. A job that runs several pytest commands offers no way to tell which glob a -k belongs to, since both are read out of the same flattened job text. Combining them could pair one command's exclusion with another command's glob and report a file that in fact runs. Such a job is now left unmodelled, which means it claims everything it globs, matching how the parser already treats an expression it cannot read. Only one job in the config has two globs today and it carries no -k at all, so no verdict changes. The second is a deliberate limit, now stated where it lives: an excluded term is only honoured when it sits in the module path, because that is the case that takes the whole file with it. A term matching one function inside drops that test and leaves the file running, and reporting it would be a false alarm. Answering per-test instead would need a baseline of test ids that churns on every rename, for a smaller failure than a file going dark. Both are pinned by tests. |
||
|
yuneng-jiang
|
d7e4b1bdd0
|
feat(ci): guard shard assignment across every sharded test tree (#37593)
tests/proxy_unit_tests had a 30-line YAML parser inlined in its workflow that failed the run when a test file there belonged to no shard. tests/test_litellm is sharded the same way, with no catch-all bucket, and had no such guard: a new directory under it (or under its proxy subtree) is collected by nothing and runs nowhere, and the coverage census cannot see it because a token like tests/test_litellm/test_*.py already answers 'yes, that tree runs'. The two questions differ. The census asks whether a file runs at all, so an ancestor path standing in for everything beneath it is a fine answer. Shard assignment asks which shard owns a child, and there that same ancestor path is precisely the bug. _token_covers keeps the first meaning; _token_names adds the second, and the guard now walks a list of sharded trees rather than one hardcoded directory. Both read the same test-path keys, so there is one workflow parser. A directory needs a shard when it holds a test file, not when it is named test_*. That drops the hardcoded test_configs exception and keeps fixture directories like expected_fine_tuning_api out on their own merits. The job keeps its name and its workflow, since assert-shard-coverage is a required status check on litellm_internal_staging. Verified red-first: a planted directory under tests/test_litellm, a planted directory under tests/test_litellm/proxy, and a planted file under tests/proxy_unit_tests each fail the guard, while a fixture-only directory does not. 327 children across the three trees are assigned today. |
||
|
yuneng-jiang
|
5b1a9563d6
|
chore(ci): close the test-census blind spots and move scripts out of workflows/ (#37586)
The agent job's CircleCI glob collected `tests/agent_tests/**/test_*.py` and then piped it through `grep -v` to drop `local_only_agent_tests/`. `assert_ci_coverage.py` reads the glob but not the pipeline, so those two files looked covered and were invisible to the census. The glob now excludes them structurally and they carry an allowlist entry instead, which is a decision on the record rather than a hidden filter. The collected file set is unchanged: `tests/agent_tests/` holds exactly one CI-runnable test at the top level. `tests/scim_tests/` held a single JSON fixture and no tests, referenced from nowhere. `.github/workflows/` is for workflows. Both stray scripts move to `.github/scripts/` with their callers updated: the price-file updater is invoked by `auto_update_price_and_context_window.yml`, and the translation-report runner by `make test-llm-translation`. The audit listed the latter as orphaned, but Makefile line 317 still runs it, so it moves rather than being deleted. The rollout heads-up workflow was a deliberate one-shot for the agent-shin rollout. That rollout is done, the triage and auto-close workflows have been running daily since June, so the pre-flip warning window is long past. Its script and dedicated test go with it, and the sibling workflow-invariant test drops its entry. |
||
|
yuneng-jiang
|
6811f1d37f
|
fix(ci): run the full dashboard suite when a change reaches outside src/ (#37563)
The UI unit test job narrows a pull request to `vitest related <changed files>`. `related` maps a file to the tests that import it, so a file no test imports maps to nothing, and `--passWithNoTests` turns that empty selection into a green job. package.json, package-lock.json, the Vitest, Tailwind and TypeScript configs and tests/setupTests.ts are all in that category even though each of them can change the behaviour of every test in the suite, so a dashboard dependency bump merged having run no unit tests at all and only got real coverage later, from the full run on the push to litellm_internal_staging. Keep `related` for the common case where a pull request only touches files under src/, and fall back to the full suite as soon as one changed file sits outside it. The decision lives in .github/scripts/select_ui_test_scope.sh so it can be tested on its own, next to the existing classify_changes.sh gate. |
||
|
yuneng-jiang
|
7b6e16cfd3
|
perf(ci): gate the lint, MCP and dashboard jobs on the pull request's file list (#37559)
PR #37550 taught the backend unit-test shards to read the pull request's own file list, but four required jobs were never wired to that gate and ran in full on every pull request regardless of what it touched. A UI-only pull request still paid roughly 17 runner-minutes of Python work it could not have affected, and a backend-only one still installed and built the dashboard. Lint and the MCP suite now take the existing backend decision. The dashboard build and unit tests take a new ui decision, which tracks ui/ rather than reusing client: client deliberately runs whenever the backend changes, because it gates CircleCI's end-to-end jobs that drive a real proxy, while the build and the unit tests cannot see the backend at all. CI config counts as ui-relevant too, so a pull request that rewrites the dashboard workflows still exercises them instead of shipping unvalidated. The gate stays inside the job rather than moving to on.paths or to a job-level condition on the shard callers. A workflow filtered out by on.paths never starts and never reports, so a required check waits forever, and a skipped caller job publishes its own name instead of the nested "<shard> / Run tests" the ruleset requires. Both were measured before settling on this shape. Three setup steps in the shard base and in the documentation job also leaked past the gate, so a skipped shard still spent about twelve seconds installing uv and restoring its cache. They now carry the same condition, and the documentation job stops cloning litellm-docs when it has nothing to validate. |
||
|
yuneng-jiang
|
eecb226762
|
fix(ci): gate backend unit tests on the pull request's own file list (#37550)
detect-backend-changes diffed the event payload's base.sha against the checked-out ref. Those are two different points in time: actions/checkout resolves refs/pull/N/merge, and GitHub recomputes that ref whenever the base branch advances, so the diff picked up whatever landed on staging between the event firing and the job starting. On a recent UI-only pull request three backend commits from staging were attributed to the branch, and every backend shard ran in full Ask the API which files the pull request touches instead. That is the same set the Files changed tab shows, and it is immune to either endpoint moving. The shell body moves into .github/scripts/detect_backend_changes.sh so it can be exercised directly, and the fail-open paths now also cover an API failure, a file list past the API's 3000-entry listing ceiling, and a classifier that prints something unexpected |
||
|
mateo-berri
|
812abcc7f5 | fix(triage): track and credit feature dead-end evidence in the verdict | ||
|
mateo-berri
|
e59da40750 | fix(triage): ask for dead-end evidence in feature request recovery comments | ||
|
mateo
|
38a1bec5c2 |
fix(triage): require evidence of the dead-end in feature requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo
|
7a17735473 |
fix(triage): treat an unfilled bug report scaffold as missing proof
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
7a5d6a0548
|
ci: fail the build when a test file or Dockerfile is invoked by no job (#35991)
The absence of a CI signal is indistinguishable from a passing one, and that shape has now produced several independent holes: whole test directories no job runs, and shipped images no job builds. Nothing was watching for either, so each was found by accident. assert_ci_coverage.py enumerates every test_*.py under tests/ and every Dockerfile in the repo, then credits only what the workflows and the CircleCI config actually invoke. Paths filters and lint steps that merely name a directory do not count as coverage, because crediting a mention is the same mistake one level up. Anything neither invoked nor listed in .github/ci-coverage-allowlist.yml with a written reason fails the job. The guard found 275 uncovered test files and 6 unbuilt Dockerfiles. Fixed here: the root Dockerfile, the primary published image, now gets an image-scan leg that builds it and runs the offline migration check against it, and 8 tests/test_litellm subdirectories join the shards that already enumerate their siblings. Everything else is allowlisted per file, so a new file cannot inherit an exemption, and the remaining decisions are tracked rather than invisible. The job reports but is not in branch protection, so it does not block merges; promoting it is a separate change once the allowlist has survived contact with a few pull requests. |
||
|
ishaan-berri
|
a2d04ccdbb
|
ci: harden cargo fetches during maturin builds (#31348) | ||
|
Mateo Wang
|
669ddc12c7
|
feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle (#30433)
* feat(triage): auto-close stale PRs with Greptile score <4/5
Adds .github/scripts/close_low_quality_prs.py and a daily workflow that
closes PRs which:
- are open for at least 7 days, and
- carry a most-recent greptile-apps review with Confidence Score <4/5,
- and are not drafts or opt-out-labeled ('do not close', 'wip', etc.).
Each closure posts an explanatory comment telling the contributor how to
bring the PR back (rebase, re-request greptile, reopen at 4+/5). The
4/5 bar is already documented in the PR template
(.github/pull_request_template.md), so this just enforces it.
Tested with a dry run against the live BerriAI/litellm backlog of 1000
open PRs: 100 candidates identified, 598 PRs pass the bar (4+/5), 186
are too young, 97 are drafts, 19 lack any Greptile review and are left
alone.
Workflow defaults to closing 25 PRs/run as a safety net and supports
workflow_dispatch with overrides (close=false for a dry run, custom
min_age_days/min_score/limit).
18 unit tests cover score extraction (HTML/markdown/plain text, login
variants, multi-review picks latest) and per-PR evaluation (drafts,
opt-out labels, age, missing/passing/failing scores).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* docs(templates): require expected/actual + QA proof for external contributions
PR template:
- Make the rubric explicit at the top: link an issue, OR provide a clear
problem description + expected vs. actual + visual QA proof.
- Add dedicated sections for each piece so the bot has a deterministic
shape to read.
- Keep the existing 'Linear ticket' section for internal contributors
(they're exempt from the auto-triage rubric).
Bug report template:
- Split 'What happened?' into 'Actual behavior' + 'Expected behavior'.
- Make logs/screenshot a required textarea.
- Warning banner at the top tells external contributors that incomplete
reports will be auto-closed (with re-evaluation on reopen).
Feature request template:
- Require a concrete use case + example in the motivation field, not just
a one-liner pitch.
- Same auto-triage warning banner.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(triage): Agent Shin LLM-as-judge for external PRs and issues
Adds a new triage flow that evaluates external pull requests and issues
against the project's contribution rubric and, when configured to do so,
auto-closes non-conforming ones with an explanatory comment. Contributors
can update + reopen to be re-evaluated.
Scope:
- Internal BerriAI contributors (author_association OWNER/MEMBER/COLLABORATOR)
and bot accounts are skipped entirely.
- 'Fixes #1234' / 'Resolves https://github.com/.../issues/N' in the PR body
short-circuits to PASS without burning LLM tokens.
- LLM judge returns structured JSON (verdict, missing[], explanation);
parser tolerates markdown fences and embedded JSON.
- LLM errors NEVER close PRs/issues — failure surfaces as 'skip-llm-error'.
Safety:
- pull_request_target / issues triggers are FORCED dry-run in the workflow;
only manual workflow_dispatch with close=true (and AGENT_SHIN_ENABLED=true)
takes destructive action.
- Default mode writes verdicts to GITHUB_STEP_SUMMARY only — no public
comments until the team flips the AGENT_SHIN_ENABLED repo variable.
- LLM uses an OpenAI-compatible endpoint (model and base URL configurable
via repo variables; key via OPENAI_API_KEY secret).
Files:
- .github/scripts/triage_with_llm.py - judge orchestrator + CLI
- .github/workflows/triage_pr_with_llm.yml
- .github/workflows/triage_issue_with_llm.yml
- tests/test_litellm/test_github_triage_with_llm.py - 33 unit tests
End-to-end validated against four real PRs (#28117 internal collaborator,
#28108 bot, #28129 'Fixes #28128', #28116 no linked issue) and issue
#28132 with a stubbed LLM judge: each path produces the expected action.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(triage): scope Greptile auto-closer to external contributors + dry-run by default
- close_low_quality_prs.py now filters by GitHub author_association via
the REST API: PRs from OWNER / MEMBER / COLLABORATOR (and bot accounts)
are skipped with a new 'skip-internal' summary bucket.
- close_low_quality_prs.yml now defaults workflow_dispatch close=false,
and ignores 'close=true' unless the new repo variable
AGENT_SHIN_ENABLED is set to 'true'. Scheduled runs are dry-run only
until the team flips that switch.
- Updated unit tests: one new test asserting internal authors are
skipped, and an autouse fixture treats unspecified test PRs as
external so the rest of the suite still exercises the close path.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(workflows): scheduled cron closes PRs; safe --close strip in triage
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(triage): scheduled cron stays dry-run; dedent prompts before interpolation
- close_low_quality_prs.yml: only workflow_dispatch with close=true (and
AGENT_SHIN_ENABLED=true) actually closes PRs. Scheduled runs are always
dry-run, matching the safety invariant documented for triage_pr/issue.
- triage_with_llm.py: textwrap.dedent on an f-string with multi-line
interpolated bodies fails because the body's 2nd+ lines start at column 0,
making the common-indent zero. Dedent the static template first, then
.format() the title/body in.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Fix bugs in auto-close PR triage scripts
- close_low_quality_prs.py: Treat author_association API lookup failures
as internal (fail-safe) so transient errors don't cause internal
contributors' PRs to be auto-closed.
- triage_with_llm.py: Update summary heading from 'Would post comment:'
to 'Posted comment:' since this branch only runs after the comment
has already been posted.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(triage): default Agent Shin to gpt-5.4-mini with reasoning_effort=none
- Bump DEFAULT_MODEL from gpt-4o-mini to gpt-5.4-mini (more modern;
4M total context window per OpenAI catalog, JSON-schema response
format, function calling all supported).
- For gpt-5.x family models, pass reasoning_effort="none" via
extra_body. gpt-5.x rejects temperature != 1 unless reasoning_effort
is explicitly "none"; setting it lets us keep temperature=0 for
deterministic JSON rubric judgments. extra_body works across openai
SDK versions regardless of whether they natively type the kwarg.
- For non-gpt5 overrides (TRIAGE_MODEL=gpt-4o-mini etc.), reasoning_effort
is not sent.
- 4 new unit tests cover: gpt-5.4-mini -> reasoning_effort=none,
capitalized/dated gpt-5 variants -> reasoning_effort=none,
gpt-4o-mini -> no extra_body, base_url passthrough.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(triage): bugbot — drop dead gh_json and fix --optout-label append-with-default
- Removed the unused gh_json helper (bugbot low-severity dead code).
- Replaced argparse `action="append", default=[...]` with default=None
+ DEFAULT_OPTOUT_LABELS fallback. The mutable-default + append combo
silently APPENDS to the canonical defaults instead of replacing them,
so --optout-label could not actually scope the opt-out list.
- Added tests covering both the canonical default and the
flag-replaces-defaults behavior.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(triage): bugbot — tighten linked-issue regex, fail-safe author_association, fix empty TRIAGE_MODEL
Three independent bugbot findings against triage_with_llm.py:
1. LINKED_ISSUE_PATTERN included weak keywords (`see`, `ref`,
`addresses`) so casual mentions like "See #1234 for context" were
short-circuited to pass-linked-issue without ever calling the LLM —
contradicting the prompt's own "a bare issue number without a closing
keyword counts only if it's clearly the related issue (not a passing
mention)" rubric. Limit the regex to GitHub's documented PR-closing
keywords (fixes/fix/fixed/closes/close/closed/resolves/resolve/resolved).
2. is_internal_contributor() treated an empty/missing author_association
as external (eligible for the destructive close path), while the sibling
is_external_pr_author() in close_low_quality_prs.py fail-safes the same
case as internal. Align the two so a partial/unknown GitHub response can
never make a PR eligible for auto-close.
3. argparse `default=os.environ.get("TRIAGE_MODEL", DEFAULT_MODEL)` returns
the empty string when GitHub Actions exposes an unset repo variable as
an empty-string env var (the optional vars.TRIAGE_MODEL case in the
workflow). Use `os.environ.get(...) or DEFAULT_MODEL` so empty -> default,
matching the existing OPENAI_BASE_URL pattern.
Tests:
- Casual mentions now must fall through to the LLM (parametrized);
added an orchestration test ensuring "See #1234" reaches the judge.
- Empty/missing author_association now fails safe (parametrized).
- Empty TRIAGE_MODEL env var falls back to DEFAULT_MODEL; explicit
TRIAGE_MODEL is still honored.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(workflows): bugbot — gate Agent Shin --close on '= true' not '!= false'
The PR and issue Agent Shin workflows gated the destructive --close
flag with [ "${DISPATCH_CLOSE:-false}" != "false" ]. That pattern
treats anything other than the literal string "false" as enabling
closure — "True", "yes", "1", typos, accidental whitespace, etc.
The workflow_dispatch input UI is a 'true'/'false' choice dropdown so
the form is constrained, but the API (`gh workflow run -f close=...`)
accepts any string, and a CI cron / external invoker passing a
non-canonical truthy value would have silently enabled real
contributor PR closures.
Mirror the sibling Greptile closer's [ "${CLOSE_FLAG}" = "true" ]
pattern: only the EXACT string "true" enables --close; every other
value (including the unset/empty default) resolves to dry-run. This is
the fail-safe philosophy applied everywhere else in this PR.
Added tests/test_litellm/test_github_triage_workflows.py with two
parametrized invariants:
1. The destructive gate uses '= "true"' for its env-var
comparison (either bare '${ENV}' or '${ENV:-false}' form
accepted), and never the fail-open '!= "false"' pattern.
2. Every destructive gate is also gated on AGENT_SHIN_ENABLED being
"true" — either by entering the close branch on '=' or by
bailing out early on '!=' — so flipping the repo variable off is
a true kill switch regardless of per-run inputs.
Manually verified the test fails on the buggy '!= "false"' pattern and
passes on the fix, so it would have caught the regression at PR time.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(triage): close any PR (incl. drafts, any age); add @agent-shin reconsider flow
Follow-up to PR #28117. Three behavior changes + one new workflow,
addressing the team's concerns on the original review:
1) Apply auto-close to ALL open PRs, not just those over a week old.
- close_low_quality_prs.py: --min-age-days default flipped from 7 to
0. The flag is preserved as an opt-in safety net for one-off
backfill runs that want to spare very-young PRs, but the daily
scheduled sweep now closes external-author PRs as soon as Greptile
scores them <4/5.
- close_low_quality_prs.yml: workflow_dispatch input default also
flipped to 0; doc comments updated.
2) Apply auto-close to draft PRs too.
- close_low_quality_prs.py: removed the skip-draft branch in
evaluate_pr. Drafts are NOT a free pass — the team's intent is
'open PR count == PRs internal collaborators need to action on',
so a draft Greptile scored 2/5 still belongs in the closed bucket.
Authors who genuinely need a long-lived draft can attach the 'wip'
opt-out label, which is unchanged.
- The 'skip-draft' action is gone; the 'wip' label still skips.
3) Address the 'OSS contributors cannot reopen a bot-closed PR' wrinkle.
GitHub does NOT let an external (non-write-access) contributor
reopen a PR that was closed by a bot or maintainer (long-standing
limitation). The original PR's close-comments told contributors to
'Reopen the PR — I'll re-evaluate automatically', which is broken
for the very audience this triage targets. Two changes:
a) Reword every close-comment (Greptile sweep + Agent Shin PR
close + Agent Shin issue close + PR template) to recommend:
- Open a new PR with the updated branch (primary path).
- Or comment '@agent-shin reconsider' on the closed PR for a
re-evaluation that, on pass, reopens the PR via the bot's
GH_TOKEN write access.
b) Add the @agent-shin reconsider workflow:
- .github/workflows/triage_reconsider.yml: new
'issue_comment'-triggered workflow. Authorizes only the
PR/issue author or an internal collaborator
(OWNER/MEMBER/COLLABORATOR), gated via a step output so
unauthorized commenters never reach the destructive steps.
Globally gated on AGENT_SHIN_ENABLED='true' (positive form,
matching the test_github_triage_workflows guardrail
patterns).
- triage_with_llm.py: --reconsider mode. On a closed PR/issue,
re-runs the LLM judge (or linked-issue regex short-circuit)
and:
- on pass: reopens via reopen_pr/reopen_issue + posts a
'Re-evaluated and reopened' comment.
- on fail: leaves closed and posts a 'still missing X'
comment so the contributor can iterate again.
Reconsider-on-open is a no-op ('skip-not-closed').
Internal-author + bot-account skips still take priority over
reconsider.
4) Greptile-on-closed-PRs question: the team asked whether Greptile can
re-review a closed PR. Greptile's docs don't address this and we
shouldn't promise behavior we can't verify, so the new close-comment
wording does NOT instruct contributors to 're-request greptile on
the closed PR'. Instead it points them at the new-PR path (which
Greptile definitely reviews) or the @agent-shin reconsider trigger
(which re-runs the LiteLLM-side rubric judge, not Greptile).
Tests: 93 passing (was 59).
- test_github_close_low_quality_prs.py: replaced 'skip drafts' test
with 'closes drafts when score is low' + 'closes brand-new PR when
min_age=0' + 'no skip when min_age=0'. The 'skip too young'
assertion is preserved as opt-in.
- test_github_triage_with_llm.py: 6 new TestTriageOrchestration cases
for reconsider mode (skip-not-closed on open, reopen on pass,
still-failing comment on fail, linked-issue short-circuit reopen,
skip internal author in reconsider, reopen-issue on pass) + a new
TestCloseCommentText class that pins the user-facing 'open a new
PR' + '@agent-shin reconsider' wording.
- test_github_triage_workflows.py: added triage_reconsider.yml to
the destructive-gate guardrail table; AGENT_SHIN_ENABLED is its
own destructive gate (no separate per-run flag needed).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(triage): pin safe behavior for curly braces in PR/issue title+body
Adds regression tests covering the bugbot high-severity finding that
str.format() would crash on user-supplied content containing { or }.
Empirically str.format() does NOT re-parse interpolated values — only
the template literal is scanned for replacement fields — so the bug
does not exist in the current code, but pinning the safe behavior
prevents a future templating change from silently reintroducing it.
Also pins the dedented prompt shape (no leading 8-space indentation on
template lines) so a future change to the build_*_prompt functions can't
silently regress the LLM judge prompt format on multi-line bodies.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(triage): bugbot — reconsider dry-run + bot-closed guard + rate limit
Address three Greptile/veria-ai concerns on the @agent-shin reconsider
flow:
1. **Reconsider had no dry-run path.** The previous reconsider mode
ignored `--close` and always posted comments + reopened on a pass.
A local operator running
`python triage_with_llm.py --reconsider --pr N` would silently
take destructive GitHub actions with no way to preview. Reconsider
now honors `close=False` the same way regular triage does and
returns `would-reopen` / `would-reconsider-still-failing` for
step-summary rendering.
2. **Reconsider could reopen maintainer-closed PRs/issues** (Medium
security finding from veria-ai). The workflow only checked that the
commenter was authorized — it did NOT check that the most recent
close was performed by Agent Shin. A contributor could comment
`@agent-shin reconsider` on a PR a maintainer closed for non-rubric
reasons (duplicate, security report, design rejection) and have the
bot reopen it. Add `was_closed_by_agent_shin()` which inspects the
issue events API for the most recent `closed` actor and only
permits reopen when that actor matches the configured bot login
(default `github-actions[bot]`, overridable via env). Fail-closed
on missing events.
3. **No rate-limiting on the reconsider trigger.** Every
`@agent-shin reconsider` comment burns CI minutes + an OpenAI API
call. Add a 10-minute cooldown via
`seconds_since_last_reconsider_verdict()` which greps the issue's
comment list for the bot's own verdict marker
(`<!-- agent-shin:reconsider-verdict -->`). Inside the window the
triage returns `skip-rate-limited` and the LLM never runs.
Workflow update:
- `triage_reconsider.yml` now passes `--close` only when
`AGENT_SHIN_ENABLED=true`, matching the pattern of
`triage_pr_with_llm.yml`. The script runs in both states so the
verdict still appears in the step summary for QA.
Tests:
- Add 5 reconsider safety tests: dry-run for pass / fail / linked-issue
short-circuit, bot-closed-guard refusal on maintainer close,
rate-limit refusal inside the cooldown window, and cooldown-elapsed
acceptance.
- Add unit tests for `was_closed_by_agent_shin` (bot / maintainer /
missing actor / env-override) and
`seconds_since_last_reconsider_verdict` (no marker / multiple
markers / non-bot comment with marker / bot comment without marker).
- Pin the `<!-- agent-shin:reconsider-verdict -->` marker in both
reopen and still-failing comments — dropping it would silently
break the cooldown.
Existing reconsider tests updated to pass `close=True` (the
production path now) + stub the new guards via
`_stub_reconsider_guards`. 112 tests pass (was 93).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(triage): 1-day grace period before close + SwiftWinds immediate-close bypass
- Add a 24-hour grace window between the first low-quality detection
and the actual auto-close. The first detection posts a warning
comment that explicitly says "You have 1 day to address this before
this PR is auto-closed" and points the contributor at:
* `@agent-shin reconsider` to request another look (and re-open)
* `@greptileai` to request a fresh Greptile review — works
even after the PR is closed
- Both `triage_with_llm.py` (LLM judge) and `close_low_quality_prs.py`
(Greptile-score closer) share the same `<!-- agent-shin:grace-warning -->`
HTML marker so a warning posted by either path is recognized by both.
- Add IMMEDIATE_CLOSE_LOGINS = {swiftwinds} to bypass BOTH the grace
period AND the dry-run / AGENT_SHIN_ENABLED gating. SwiftWinds is the
user's personal account (no push permissions to litellm) used to
dogfood the bot; user explicitly asked: "For SwiftWinds, just close
immediately. Faster iteration that way."
- Update the standard close comments to mention that `@greptileai`
works even after the PR is closed.
- Add 23 new tests covering: warn-grace on first detection, skip during
grace window, close after grace expires, SwiftWinds bypass (case
insensitive, with close=False, no random-login false positives), the
grace-warning text invariants, and the SwiftWinds entry in the
IMMEDIATE_CLOSE_LOGINS constant.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: skip grace-period text in close comment for IMMEDIATE_CLOSE_LOGINS
For PRs from IMMEDIATE_CLOSE_LOGINS (e.g. swiftwinds), evaluate_pr
returns 'close' immediately without ever posting a grace warning, so
the close comment should not reference a 1-day grace period.
Make close_pr take a grace_period_elapsed flag, default True, and
pass False from the main loop when the close path was the
immediate-close branch.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(close-low-quality-prs): report actual closes in dry-run summary
IMMEDIATE_CLOSE_LOGINS PRs are closed even when the global --close flag is
not set, but the summary used the global dry-run flag to choose between
'would close' and 'closed'. Split the count so operators can see both
actual closures and dry-run would-be closures.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* chore(triage): vendor Agent Shin (#28117) onto demo branch
Brings the Agent Shin OSS-triage scripts, workflows, issue/PR templates, and
tests from PR #28117 onto this branch so the new review-gate feature and its
end-to-end demo are self-contained and runnable in CI.
https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ
* feat(triage): add "ready for review" label lifecycle to Agent Shin
Adds review_gate(), a state machine that keeps a `ready for review` label in
sync with whether an external PR clears BOTH gates — the LLM rubric and
Greptile's most recent confidence score:
- pass (untagged) -> add label + "ready for review" / "all clear" comment
- pass (already tagged) -> no-op (idempotent across re-runs)
- regress (Greptile < 4/5 or QA proof removed) -> remove label + "what's missing"
comment, PR stays open
- recover after a regression -> "all clear again" comment + re-add the label
- fail & untagged, < 24h old -> one-time "what's missing" notice (grace window)
- fail & untagged, > 24h old -> close + comment (reopen via @agent-shin reconsider)
The label itself is the persisted state, so comments fire only on transitions
(never on every scheduled run). All side effects are gated behind --close, so
the dry-run contract matches the existing triage flow. Lifecycle comments use
hidden HTML markers and deliberately avoid the auto-close marker so they never
trip the reconsider provenance check.
Relocates the shared Greptile helpers (extract_greptile_score, SCORE_PATTERN,
GREPTILE_BOT_LOGINS, parse_iso8601) into triage_with_llm.py so the daily sweep
and the review gate read the score through one implementation, and adds the
review_gate.yml workflow (dry-run unless AGENT_SHIN_ENABLED=true) plus 18 unit
tests covering every branch and a full pass->regress->recover cycle.
https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ
* Port review-gate feature from #28758 onto #28147 triage scripts
Adds the "ready for review" label lifecycle (originally PR #28758) on top
of #28147's refactored triage_with_llm.py. The original commit was
authored against an older snapshot of #28117 and could not be applied
cleanly, so the additions were re-applied surgically:
- New constants: READY_FOR_REVIEW_LABEL, DEFAULT_GRACE_DAYS,
DEFAULT_MIN_GREPTILE_SCORE, READY/REGRESSED/WITHIN_GRACE markers,
GREPTILE_BOT_LOGINS, SCORE_PATTERN, AGENT_SHIN_AUTO_CLOSE_MARKER.
- New helpers: add_label, remove_label, extract_greptile_score,
parse_iso8601 (the latter two mirrored from close_low_quality_prs.py
so the daily sweep and the review gate read the score through the
same logic).
- New comment formatters: format_ready_for_review_comment,
format_all_clear_comment, format_regression_comment,
format_within_grace_comment.
- New entry point: review_gate() implementing the pass/regress/recover
state machine, with the label itself acting as persisted state so
transition comments fire only on actual transitions.
- main() learns --review-gate, --grace-days, --min-greptile-score and
dispatches to review_gate() when the flag is set.
Verified via tests/test_litellm/test_github_review_gate.py (18 tests)
and the existing triage suites (144 more) — all 162 pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* agent_shin: extract shared constants/helpers; cover review_gate.yml in guardrail tests
Bug 1: `triage_with_llm.py` and `close_low_quality_prs.py` each defined
their own copies of `extract_greptile_score`, `parse_iso8601`,
`GREPTILE_BOT_LOGINS`, `SCORE_PATTERN`, `GRACE_COMMENT_MARKER`,
`GRACE_PERIOD_SECONDS`, `IMMEDIATE_CLOSE_LOGINS`, and
`AGENT_SHIN_DEFAULT_BOT_LOGIN`. The comments explicitly said the two
copies had to stay in sync, but nothing enforced it. A future change to
one (e.g. extending `SCORE_PATTERN` for a new Greptile output format)
would silently diverge from the other and the daily sweep and the LLM
judge would disagree on which PRs have low scores.
Extract these to `.github/scripts/agent_shin_shared.py` and re-export
them from each script so the existing test attribute access
(`triage_module.GRACE_COMMENT_MARKER`, etc.) keeps working without
any test changes.
Bug 2: `review_gate.yml` is a destructive workflow (close PRs, add/remove
labels, post comments) with the same gating philosophy as the others
(`AGENT_SHIN_ENABLED = "true"` + a per-run `CLOSE_FLAG = "true"`),
but it was missing from `DESTRUCTIVE_GATE_ENV` in the guardrail tests.
Add it so a future regression (e.g. flipping to `!= "false"`) is
caught by the same parameterized invariants as every other workflow.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* agent_shin: fix bug bundle (gated LLM key, author-filtered marker dedup, dedup gh/grace helpers)
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* agent_shin: fix review_gate close-after-regression and case-insensitive label match
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(triage): add one-shot 7-day heads-up sweep for Agent Shin rollout
Adds a rollout-day workflow that comments on every open external PR/issue
that the new triage bot WOULD auto-close, giving contributors 7 days to
fix their description before any destructive action runs.
Why now: merging this PR enables Agent Shin in dry-run. The follow-up
"enact" PR (next Monday) flips the destructive paths on. Without this
heads-up, contributors would get a close-comment on day 8 with no prior
warning. The heads-up names the cutoff date, lists the rubric, calls out
each PR/issue's specific missing pieces, and explains the recovery paths
(@agent-shin reconsider for PRs, edit + reopen for issues).
Files
- .github/scripts/_agent_shin_actions.py — thin maybe_post_comment /
maybe_close_* / maybe_add_label / etc. wrappers. Each is a single
`if dry_run: log; return; else: call_through()` so a dry-run preview
differs from the real run in exactly one call site per mutation. The
call-through goes via `triage_with_llm.<name>` (module-qualified) so
monkeypatching the underlying function in tests is reflected here.
- .github/scripts/triage_rollout_heads_up.py — the sweep. Iterates every
open PR + issue via `gh pr list` / `gh issue list`, runs the future
rubric (review_gate for PRs, triage(kind="issue") for issues), and
posts the heads-up on any item that would be auto-closed. Idempotent
via a `<!-- agent-shin:rollout-heads-up -->` marker. Defaults to dry-
run; --close opts in to real posts. --close-on overrides the cutoff
date (defaults to today + 7 days).
- .github/workflows/triage_rollout_heads_up.yml — one-shot workflow.
Triggers on push to litellm_internal_staging filtered to the script
path (fires on rollout merge) plus workflow_dispatch with a dry_run
input that defaults to "true" for safe manual re-runs.
- tests/test_litellm/test_triage_rollout_heads_up.py — 28 unit tests
covering: the dry-run wrappers (each maybe_* gates correctly), the
_would_be_closed predicate for PR vs. issue results, the comment
formatter (cutoff/rubric/marker/recovery wording), per-item dispatch
(skip-not-open, skip-internal-author, skip-already-notified,
skip-passing, would-post/posted), and the sweep loop end-to-end.
Local preview (no GitHub mutations):
python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm
Real run (what the workflow does):
python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm --close
TODO: replace the placeholder ROLLOUT_BLOG_URL with the canonical
docs URL once the litellm-docs PR ships.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: gate reconsider workflow OPENAI_API_KEY + remove dead actions wrappers
- Mirror sibling Agent Shin workflows by only exposing OPENAI_API_KEY in
triage_reconsider.yml when vars.AGENT_SHIN_ENABLED == 'true'. Previously
the secret was unconditionally exposed, so any PR/issue author could
trigger paid LLM calls by commenting '@agent-shin reconsider' even while
the bot was supposed to be in dry-run.
- Remove the six unused dry-run wrappers (maybe_close_pr, maybe_close_issue,
maybe_reopen_pr, maybe_reopen_issue, maybe_add_label, maybe_remove_label)
from _agent_shin_actions.py — only maybe_post_comment is used by rollout
scripts. Drop the associated tests that exercised the now-removed
functions.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: address triage script edge cases
- triage_rollout_heads_up.py: replace %-d strftime specifier (GNU-only)
with portable day formatting so the script doesn't crash on Windows.
- close_low_quality_prs.py: skip malformed JSON lines in fetch_pr_comments
instead of letting one bad line abort the daily sweep, matching the
pattern in triage_with_llm._iter_paginated_json.
- triage_with_llm.py: move has_linked_issue short-circuit before
build_pr_prompt to avoid unnecessary prompt construction on PRs that
link an issue.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(scripts): per-PR error isolation and limit grace warnings in close_low_quality_prs
- Wrap per-PR processing in try/except so a transient GitHub API failure
on one PR no longer aborts the entire daily sweep (mirrors the pattern
already used in triage_rollout_heads_up.py).
- Have --limit bound *all* destructive write actions (closures and grace
warnings combined), not just closures. Prevents a backlog of newly
failing PRs from flooding contributors with comments in a single run.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(agent-shin): remove 1000-PR cap on bulk sweeps; sweep entire backlog
Both bulk-sweep scripts hardcoded `gh {pr,issue} list --limit 1000`, and gh
lists newest-first — so the OLDEST ~900 PRs and ~380 issues were silently
dropped. That's exactly the stale backlog the daily closer and one-shot
rollout heads-up exist to catch.
Extract a single `list_open_items(kind, *, repo, fields)` helper into
`agent_shin_shared.py` with `GH_LIST_ALL_LIMIT = 100_000` — a ceiling far
above any realistic open backlog so gh paginates until the queue is
exhausted. `fetch_open_prs` and `_list_open_numbers` both delegate to it,
so the limit lives in exactly one place going forward.
Verified live against BerriAI/litellm:
- `fetch_open_prs` -> 1981 PRs (was 1000)
- `_list_open_numbers(issue)` -> 1382 issues (was 1000)
- `_list_open_numbers(pr)` -> 1981 PRs (was 1000)
Adds 7 regression tests asserting the new limit is passed, the dedicated
`gh {pr,issue} list` command + fields are used per kind, bad kind raises
ValueError, and both callers delegate to the shared helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(agent-shin): require non-mocked end-to-end QA proof for PR pass
The PR rubric previously passed any PR with a linked issue, regardless
of whether it showed the fix actually working. Sample spot-check found
21/25 recent external PRs passing, including ones that linked an issue
but provided zero QA evidence.
Tighten the rubric so a pass now requires BOTH:
(1) CONTEXT — a linked issue OR a clear problem description with
expected-vs-actual behavior.
(2) END-TO-END QA PROOF — at least one of:
(a) screenshot(s) of the fix working,
(b) screen recording / video,
(c) specific commands actually run, paired with their real
output, against the real system.
Mocked unit tests, generic 'I tested it' claims, 'all tests pass'
without output, and the linked issue itself are explicitly excluded
from QA proof.
Also add 'qa_proof_type' to the JSON schema so the per-PR report
surfaces which kind of proof (or 'none') the judge saw.
Re-sample on the same 25 recent external PRs shifts the verdict
distribution from 21 pass / 4 fail to 4 pass / 21 fail, with zero
prior-fails now passing — the stricter rule catches PRs that ship
only with unit-test claims and no real integration evidence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(agent-shin): link blog explainer from every action-required bot comment
Adds "What's this and why am I getting it?" links to docs.litellm.ai/blog/
agent-shin-triage from the four comments contributors actually read when
something went wrong: PR close, PR grace warning, issue close, issue grace
warning. PR comments also link the rubric section directly from the
QA-proof bullet so contributors can self-serve "what counts as proof"
without pinging a maintainer.
Pins the new guarantees in tests: blog link must appear in all four
comments, and the PR close comment must continue to flag mocked-dependency
unit tests as insufficient proof.
The linked blog post is in BerriAI/litellm-docs PR #240; the URL will 404
until that lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review_gate): raise sweep limit from 1000 to 100000 to match GH_LIST_ALL_LIMIT
gh lists newest-first, so capping at 1000 silently drops the oldest open
PRs — exactly the stale ones the daily sweep is meant to reconcile. Use
the same ceiling as agent_shin_shared.GH_LIST_ALL_LIMIT so the workflow
sees the entire backlog.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Fix three Agent Shin triage edge cases
- review_gate: expire the regression-marker short-circuit after grace_days
so PRs that were regressed and then abandoned can eventually be closed.
- review_gate: when the rubric short-circuits to pass via the linked-issue
regex but Greptile drags the PR below the bar, replace the synthetic
'LLM was not called' explanation with the real Greptile shortfall so
regression / close comments are not misleading.
- triage_rollout_heads_up._comments_have_marker: drop the unused 'kind'
parameter and filter by bot author so a contributor quoting the
heads-up via 'Quote reply' cannot trick the idempotency check, matching
the pattern in triage_with_llm._has_marker.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: pass min_greptile_score through to ready-for-review comment text
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(agent-shin): warmer triage comments — bullet-train emoji, 'what you got right' section, softer 'park this for later' framing
User feedback on the auto-triage comments contributors will see:
1. Tone — the previous 'You have 1 day to address this before this PR is
auto-closed' framing reads as an ultimatum. Replace with: 'If the
description isn't updated in the next 1 day, I'll auto-close this PR.
That's not us saying we don't care about the change — we want the
open-PR list to mirror what a maintainer can act on right now, so
contributors don't get lost in a backlog. A closed PR is a soft "park
this for later," not a rejection. Take your time.'
2. Positive feedback — the previous comments only listed what was missing.
Now every close + grace-warning comment opens with a 'What you got
right:' section rendered from the judge's per-field flags. Contributors
see a checkmark for everything they got right (linked issue, problem
description, expected/actual, QA proof for PRs; runnable repro,
screenshot/log, expected/actual, motivation+example for issues) before
the gaps. The block is omitted entirely when nothing is present so
we never render 'What you got right: (nothing).'
3. Reconsider trigger — the previous grace warning told contributors to
comment '@agent-shin reconsider' during the grace window. They don't
need to — the bot re-checks on every sweep. The new copy says 'just
update the description, no need to ping me' for the grace path, and
reserves '@agent-shin reconsider' for the post-close recovery path.
4. Bullet-train emoji — replace 👋 with 🚄 (Shinkansen, the symbol of
Agent Shin) across every action-required comment: PR close, PR grace
warning, issue close, issue grace warning, within-grace, Greptile-
closer grace warning, rollout heads-up. Pinned in tests so a future
refactor can't silently revert.
5. Greptile-post-close — the @greptileai bullet now explicitly says 'a
low Greptile score isn't a blocker either,' since the previous copy
buried the fact that @greptileai works after auto-close.
Comment templates updated: format_pr_close_comment,
format_issue_close_comment, format_grace_warning_pr_comment,
format_grace_warning_issue_comment, format_within_grace_comment
(triage_with_llm.py); format_grace_warning_comment
(close_low_quality_prs.py); format_heads_up_comment header
(triage_rollout_heads_up.py).
New helpers: _format_present_for_pr / _format_present_for_issue /
_format_present_block, driven off the existing per-field flags the
LLM judge already emits — no prompt change needed.
New tests pin: bullet-train emoji in every action-required comment;
'What you got right' appears with ✅ bullets when fields are present;
the block is omitted when no fields are present; 'park this for
later' / 'not a rejection' softer framing; grace warnings tell the
contributor 'no need to ping' during the grace window (reconsider is
the post-close path only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(agent-shin): gate triage on a dogfood allowlist
Add ALLOWLIST_LOGINS to agent_shin_shared so Agent Shin only acts on the
named accounts while the set is non-empty. mateo-berri and SwiftWinds are
allowlisted for the dogfood rollout; everyone else is skipped with
skip-not-allowlisted across all four entrypoints (triage, review gate, the
daily low-quality sweep, and the rollout heads-up).
For an allowlisted author the usual internal/external classification is
bypassed, so a maintainer's own org account still gets triaged during
testing. Emptying the set lifts the restriction and restores full triage
for the public rollout. The gate is dependency-injected via an `allowlist`
parameter defaulting to the constant, so the internal/external-skip paths
stay testable.
* feat(agent-shin): tighten QA-proof and issue rubrics, ack reconsider with reactions
Reorder the end-to-end QA proof options to video, then screenshots, then
exact commands with their real output across the PR template, the LLM judge
prompts, and every contributor-facing comment, and spell out that mocked or
stubbed runs (including pytest on the repo's own unit tests, which mock the
provider, DB, and network) never count as proof. QA proof is now required of
all contributors, not just external ones.
Tighten the issue bug-report rubric to require end-to-end evidence of the bug
(the "before" half: a video, screenshot, or command paired with real output)
plus expected vs. actual behavior, drop the bias toward PASS, and collapse the
separate has_repro/has_proof flags into a single has_repro signal.
Standardize the bullet-train emoji and strip em dashes from the bot's
public-facing messages, and route issue recovery through @agent-shin
reconsider since GitHub doesn't let OSS authors reopen an issue a bot closed.
Acknowledge an @agent-shin reconsider the moment it's accepted with an eyes
reaction and a thumbs-up once the run finishes, both gated on
AGENT_SHIN_ENABLED so dry-run leaves no trace.
* fix(agent-shin): shorten auto-close grace to 2 hours and drop the instant-close bypass
Two dogfooding changes to the Agent Shin grace window. First, the warn-then-close
grace (GRACE_PERIOD_SECONDS) drops from a day to 2 hours so the "fix it before it
closes" loop can be exercised in one sitting; the constant carries a note to bump
it back up for the public rollout.
Second, remove IMMEDIATE_CLOSE_LOGINS entirely. SwiftWinds (the external dogfood
account) used to skip the grace window and close on first detection, which also
meant closing real PRs even during a scheduled dry run because the per-PR
override flipped dry_run off. It now follows the same warn-then-close path as
every other author, so a low-quality PR is warned first and only closed once the
2-hour window elapses. This also closes the Greptile finding that the sweep could
mutate real PRs while AGENT_SHIN_ENABLED was still off.
The review gate's separate age-based grace (DEFAULT_GRACE_DAYS) is left unchanged.
Regression tests pin that SwiftWinds now warns-grace instead of closing instantly,
and that a dry-run sweep over a closeable PR reports "would close" without making
any GitHub mutation.
* fix(agent-shin): gate reconsider reopen on an Agent Shin close marker
was_closed_by_agent_shin only checked that the most recent close actor was
the bot identity. That identity defaults to github-actions[bot], which is
shared by every workflow in the repo (stale/duplicate sweeps included), so a
contributor could @agent-shin reconsider an item another workflow closed and,
if the description passed the rubric, get it reopened even though Agent Shin
was never the closer.
Require a second, Agent-Shin-specific signal alongside the actor check: an
auto-close comment stamped with a hidden AGENT_SHIN_CLOSE_MARKER. Both close
paths (the grace-period close and the review-gate close) flow through
format_pr_close_comment / format_issue_close_comment, so stamping the marker
there covers every real close while leaving the grace warnings unmarked. The
guard stays fail-closed: no marker, no reopen.
This also replaces the unused AGENT_SHIN_AUTO_CLOSE_MARKER constant (a visible
phrase the guard never consulted) with the hidden marker the guard now relies
on.
* fix(agent-shin): stamp close marker on sweep closes and disclose regression deadline
The daily Greptile sweep's close comment advertised `@agent-shin reconsider`
but never stamped AGENT_SHIN_CLOSE_MARKER, so the reconsider reopen guard
(was_closed_by_agent_shin), which now also requires that marker, silently
rejected every sweep-closed PR with `skip-not-bot-closed`. Move the marker into
agent_shin_shared so both close paths share one source of truth, extract
format_close_comment so the sweep close comment is unit-testable, and stamp the
marker there.
Also disclose the grace_days deadline in the review-gate regression comment; it
promised "the PR stays open" without mentioning that a still-failing PR is
auto-closed grace_days after the notice, which would surprise contributors with
a close they were never warned about.
* fix(triage): tighten Agent Shin reconsider reopen guards
The bot-closed guard accepted any historical Agent Shin marker comment
on the thread as proof that Agent Shin owned the latest close, so a
post-reopen close by another workflow under the shared
`github-actions[bot]` identity could still satisfy the gate and let
`@agent-shin reconsider` reopen a PR that Agent Shin did not close
this cycle. `fetch_last_close_event` now also returns the latest
`closed` event timestamp, and `was_closed_by_agent_shin` requires
the most recent Agent Shin marker comment to sit at (or just before)
that timestamp, with a small skew window for clock drift between the
events and comments APIs.
In the same path the LLM verdict check used `decision != "fail"` to
choose the reopen branch, which treated a missing, empty, or typo
verdict as a pass. Reopen is destructive, so the check now requires an
explicit `decision == "pass"` and ambiguous verdicts fall through
to the "still failing" branch instead.
* style(agent-shin): black-format reconsider guard hardening
* docs(agent-shin): scope dry-run wrapper docstring to the single existing helper
The module docstring claimed it wrapped every Agent Shin mutation and
referenced post_comment/close_pr/etc., but only maybe_post_comment exists.
Describe the single helper accurately while keeping the dry-run pattern
guidance for any future wrapper.
* chore(agent-shin): defer issue/PR template changes to the rollout PR
The triage and review-gate automation is gated to the allowlisted authors
(mateo-berri, SwiftWinds) and AGENT_SHIN_ENABLED, so during this rollout it
only acts on internal PRs/issues. The issue and PR templates have no such
gate; they change for every contributor on merge and advertise that an LLM
bot auto-closes external submissions, which won't happen while the allowlist
is the sole author gate. Revert bug_report.yml, feature_request.yml, and
pull_request_template.md to base so the public-facing messaging lands with
the rollout flip instead of ahead of it. The scripts embed their own rubric
and never read these files, so triage behavior is unchanged.
* ci(agent-shin): hash-pin the openai install in privileged triage workflows
The triage workflows install the OpenAI client with `pip install
"openai>=1.40.0"`, a floating lower bound that resolves openai and its
whole transitive tree to whatever PyPI serves at run time. These jobs run
under pull_request_target with a write-scoped GITHUB_TOKEN, and the
install plus the triage run happen on every PR open regardless of the
AGENT_SHIN_ENABLED dry-run gate (that gate only withholds the LLM key and
the destructive --close path), so a compromised release would execute
during install or import while the token is in scope.
Install instead from a new .github/scripts/triage-requirements.txt that
pins openai==2.33.0 and every transitive dependency to an exact version
with sha256 hashes, via pip --require-hashes. The workflows already
sparse-checkout .github/scripts from the base repo (never fork code), so
the pinned file is trusted. Add static guardrails to
test_github_triage_workflows.py that fail if any installer workflow
reverts to a floating openai install or if the requirements file loses
its exact pins or hashes.
* ci(agent-shin): gate rollout heads-up real run behind manual dispatch
The rollout heads-up workflow fired its real `--close` sweep on every push
to litellm_internal_staging that touched the script, and exposed
OPENAI_API_KEY unconditionally, unlike every sibling triage workflow which
only exposes the key on an enabled or dispatched run. That made merging the
script post real heads-up comments (bounded only by the dogfood allowlist),
which contradicts the inert-by-default safety invariant; once the allowlist
is cleared for the public rollout, any later edit to the file would sweep
the whole open backlog with real writes.
The heads-up cannot be gated on AGENT_SHIN_ENABLED: its whole job is to warn
contributors before that flag flips on, so it has to run while the flag is
still off. Instead the automatic push trigger now stays dry-run, and the
real one-shot sweep is a deliberate manual workflow_dispatch with
dry_run=false, the sole path that adds `--close`. OPENAI_API_KEY is exposed
only on that dispatch, matching the sibling workflows.
Add static guardrails that fail if the push path regains a `--close`, if the
dispatch gate stops fail-closing on the exact string "false", or if the key
is exposed unconditionally again.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local>
|
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
Julio Quinteros Pro
|
1ac3c230cb
|
Update .github/scripts/close_duplicate_issues.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Julio Quinteros Pro
|
db3d61f433 |
feat(ci): add duplicate issue detection and auto-close bot
Add a Python script that detects duplicate issues using title similarity (difflib.SequenceMatcher) and closes them via the gh CLI. Two-tier system: - 0.6 threshold: informational comment via existing wow-actions step - 0.85 threshold: auto-close with comment, label, and not_planned reason Includes a workflow_dispatch workflow for one-time batch scans and integrates auto-close into the existing check_duplicate_issues workflow for newly opened issues. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Sameer Kankute
|
74496eefea
|
Implement GitHub Action to auto-label issues with provider keywords (#13537)
* Add auto issue labelling for llm_translation * Add logging * make the formatting better * make the formatting better * make the formatting better * make the formatting better |