Three independent bugbot findings against triage_with_llm.py:
1. LINKED_ISSUE_PATTERN included weak keywords (`see`, `ref`,
`addresses`) so casual mentions like "See #1234 for context" were
short-circuited to pass-linked-issue without ever calling the LLM —
contradicting the prompt's own "a bare issue number without a closing
keyword counts only if it's clearly the related issue (not a passing
mention)" rubric. Limit the regex to GitHub's documented PR-closing
keywords (fixes/fix/fixed/closes/close/closed/resolves/resolve/resolved).
2. is_internal_contributor() treated an empty/missing author_association
as external (eligible for the destructive close path), while the sibling
is_external_pr_author() in close_low_quality_prs.py fail-safes the same
case as internal. Align the two so a partial/unknown GitHub response can
never make a PR eligible for auto-close.
3. argparse `default=os.environ.get("TRIAGE_MODEL", DEFAULT_MODEL)` returns
the empty string when GitHub Actions exposes an unset repo variable as
an empty-string env var (the optional vars.TRIAGE_MODEL case in the
workflow). Use `os.environ.get(...) or DEFAULT_MODEL` so empty -> default,
matching the existing OPENAI_BASE_URL pattern.
Tests:
- Casual mentions now must fall through to the LLM (parametrized);
added an orchestration test ensuring "See #1234" reaches the judge.
- Empty/missing author_association now fails safe (parametrized).
- Empty TRIAGE_MODEL env var falls back to DEFAULT_MODEL; explicit
TRIAGE_MODEL is still honored.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Removed the unused gh_json helper (bugbot low-severity dead code).
- Replaced argparse `action="append", default=[...]` with default=None
+ DEFAULT_OPTOUT_LABELS fallback. The mutable-default + append combo
silently APPENDS to the canonical defaults instead of replacing them,
so --optout-label could not actually scope the opt-out list.
- Added tests covering both the canonical default and the
flag-replaces-defaults behavior.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- Bump DEFAULT_MODEL from gpt-4o-mini to gpt-5.4-mini (more modern;
4M total context window per OpenAI catalog, JSON-schema response
format, function calling all supported).
- For gpt-5.x family models, pass reasoning_effort="none" via
extra_body. gpt-5.x rejects temperature != 1 unless reasoning_effort
is explicitly "none"; setting it lets us keep temperature=0 for
deterministic JSON rubric judgments. extra_body works across openai
SDK versions regardless of whether they natively type the kwarg.
- For non-gpt5 overrides (TRIAGE_MODEL=gpt-4o-mini etc.), reasoning_effort
is not sent.
- 4 new unit tests cover: gpt-5.4-mini -> reasoning_effort=none,
capitalized/dated gpt-5 variants -> reasoning_effort=none,
gpt-4o-mini -> no extra_body, base_url passthrough.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- close_low_quality_prs.py: Treat author_association API lookup failures
as internal (fail-safe) so transient errors don't cause internal
contributors' PRs to be auto-closed.
- triage_with_llm.py: Update summary heading from 'Would post comment:'
to 'Posted comment:' since this branch only runs after the comment
has already been posted.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- close_low_quality_prs.yml: only workflow_dispatch with close=true (and
AGENT_SHIN_ENABLED=true) actually closes PRs. Scheduled runs are always
dry-run, matching the safety invariant documented for triage_pr/issue.
- triage_with_llm.py: textwrap.dedent on an f-string with multi-line
interpolated bodies fails because the body's 2nd+ lines start at column 0,
making the common-indent zero. Dedent the static template first, then
.format() the title/body in.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- close_low_quality_prs.py now filters by GitHub author_association via
the REST API: PRs from OWNER / MEMBER / COLLABORATOR (and bot accounts)
are skipped with a new 'skip-internal' summary bucket.
- close_low_quality_prs.yml now defaults workflow_dispatch close=false,
and ignores 'close=true' unless the new repo variable
AGENT_SHIN_ENABLED is set to 'true'. Scheduled runs are dry-run only
until the team flips that switch.
- Updated unit tests: one new test asserting internal authors are
skipped, and an autouse fixture treats unspecified test PRs as
external so the rest of the suite still exercises the close path.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds a new triage flow that evaluates external pull requests and issues
against the project's contribution rubric and, when configured to do so,
auto-closes non-conforming ones with an explanatory comment. Contributors
can update + reopen to be re-evaluated.
Scope:
- Internal BerriAI contributors (author_association OWNER/MEMBER/COLLABORATOR)
and bot accounts are skipped entirely.
- 'Fixes #1234' / 'Resolves https://github.com/.../issues/N' in the PR body
short-circuits to PASS without burning LLM tokens.
- LLM judge returns structured JSON (verdict, missing[], explanation);
parser tolerates markdown fences and embedded JSON.
- LLM errors NEVER close PRs/issues — failure surfaces as 'skip-llm-error'.
Safety:
- pull_request_target / issues triggers are FORCED dry-run in the workflow;
only manual workflow_dispatch with close=true (and AGENT_SHIN_ENABLED=true)
takes destructive action.
- Default mode writes verdicts to GITHUB_STEP_SUMMARY only — no public
comments until the team flips the AGENT_SHIN_ENABLED repo variable.
- LLM uses an OpenAI-compatible endpoint (model and base URL configurable
via repo variables; key via OPENAI_API_KEY secret).
Files:
- .github/scripts/triage_with_llm.py - judge orchestrator + CLI
- .github/workflows/triage_pr_with_llm.yml
- .github/workflows/triage_issue_with_llm.yml
- tests/test_litellm/test_github_triage_with_llm.py - 33 unit tests
End-to-end validated against four real PRs (#28117 internal collaborator,
#28108 bot, #28129 'Fixes #28128', #28116 no linked issue) and issue
#28132 with a stubbed LLM judge: each path produces the expected action.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds .github/scripts/close_low_quality_prs.py and a daily workflow that
closes PRs which:
- are open for at least 7 days, and
- carry a most-recent greptile-apps review with Confidence Score <4/5,
- and are not drafts or opt-out-labeled ('do not close', 'wip', etc.).
Each closure posts an explanatory comment telling the contributor how to
bring the PR back (rebase, re-request greptile, reopen at 4+/5). The
4/5 bar is already documented in the PR template
(.github/pull_request_template.md), so this just enforces it.
Tested with a dry run against the live BerriAI/litellm backlog of 1000
open PRs: 100 candidates identified, 598 PRs pass the bar (4+/5), 186
are too young, 97 are drafts, 19 lack any Greptile review and are left
alone.
Workflow defaults to closing 25 PRs/run as a safety net and supports
workflow_dispatch with overrides (close=false for a dry run, custom
min_age_days/min_score/limit).
18 unit tests cover score extraction (HTML/markdown/plain text, login
variants, multi-review picks latest) and per-PR evaluation (drafts,
opt-out labels, age, missing/passing/failing scores).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Add a Python script that detects duplicate issues using title similarity
(difflib.SequenceMatcher) and closes them via the gh CLI. Two-tier system:
- 0.6 threshold: informational comment via existing wow-actions step
- 0.85 threshold: auto-close with comment, label, and not_planned reason
Includes a workflow_dispatch workflow for one-time batch scans and
integrates auto-close into the existing check_duplicate_issues workflow
for newly opened issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add auto issue labelling for llm_translation
* Add logging
* make the formatting better
* make the formatting better
* make the formatting better
* make the formatting better
