mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-09 22:33:39 +00:00
5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
dependabot[bot]
|
47f3932c8c
|
chore(deps): bump actions/setup-node from 6.4.0 to 7.0.0
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 6.4.0 to 7.0.0.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](
|
||
|
Gergő Magyar
|
5549403082
|
fix(eval): self-hosted skill-evolution runner + sandbox Python 3 trust fix (#2600)
* fix(eval): move skill-evolution to a self-hosted runner and fix the sandbox's Python 3 trust gap GitHub-hosted runners hard-cap job execution at 6 hours, which is too short once a benchmark session actually invokes Skill/MCP tools for real (the --bare fix in #2584 means sessions no longer no-op). Move the job onto a self-hosted runner (5-day cap instead) and document the activation step in the workflow's own checklist. Validating the self-hosted run surfaced a real bug: gitnexus-plan sessions inside the bwrap sandbox failed with "planning must create or modify exactly one plan artifact; observed 0". Root cause: evidence-provenance.mjs's atomic plan-writer only trusts a Python 3 binary owned by root or by the current process. Inside this --unshare-user sandbox only the calling uid is mapped (root isn't), so the real, root-owned /usr/bin/python3 surfaces as the kernel's overflow uid and gets correctly refused as untrusted. Fix: provision a small, self-owned wrapper script (same pattern already used for shell-prefix) that execs the real interpreter, so the sandbox has a Python 3 candidate the existing trust check can actually accept -- without touching that security-sensitive validation logic at all. Also add visibility so this class of failure isn't quiet next time: report.md now shows why each row failed (error_kinds), not just resolved 0/1, and the benchmark now exits non-zero when an incumbent arm -- the currently-shipped skill -- resolves zero across every task, since that reads as a broken harness rather than a normal candidate miss. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(eval): close the broken_incumbent_arms zero-valid-runs gap; document runner exposure tradeoff Addresses the two MEDIUM findings from the gitnexus-review-agent on this PR (https://github.com/abhigyanpatwari/GitNexus/pull/2600#issuecomment-5033363096). broken_incumbent_arms required valid_runs > 0 before flagging an incumbent, so an incumbent that fails every run with an excluded-but-non-systemic error_kind (e.g. evidence-unverified, which the outage-streak breaker explicitly resets on rather than accumulates) never accumulated a single valid run and sailed through silently -- the exact quiet no-promotion outcome this guard exists to catch, and arguably worse than the some-runs-resolved-zero case since here nothing completed at all. aggregate() never marks an excluded/unverifiable row resolved=True, so dropping the valid_runs requirement and checking resolved == 0 alone correctly covers both cases. Added a test for exactly this all-excluded scenario, which none of the existing three did. Updated the workflow's own activation checklist to reflect what's actually true now (the gitnexus-evolution environment's branch policy and the self-hosted runner are both live, codified in infra/gitnexus-evolution/ in a companion PR) and documented the exposure-window tradeoff the review flagged: the runner is stopped between runs but not destroyed/recreated per run, so it isn't fully ephemeral. Stopping already bounds the exposure window to the job's own runtime on one day out of seven; full per-job ephemeral provisioning is a deliberate non-goal for a job that runs at most weekly, revisit if that changes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(eval): remove public infra/ pointers from the activation checklist PR #2603 (the Terraform codification this checklist pointed to) got closed -- publishing the exact IAM roles, security group rules, and self-hosted runner topology for a real, live AWS account isn't safe to do in a public repo, even with no literal secrets or resource IDs in the diff. The underlying AWS/GitHub setup is unaffected and still documented privately; this just removes the now-dangling references to a directory that won't exist in this repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * ci(actionlint): register the gitnexus-evolution self-hosted runner label actionlint rejected `runs-on: [self-hosted, linux, x64, gitnexus-evolution]` in gitnexus-skill-evolution.yml because it can't discover custom runner labels. Register it in .github/actionlint.yaml so the Workflow Lint check passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Claude
|
eea9ac92dc |
ci: move Node pins to the 22.18 floor
With the supported minimum raised to Node 22.18, retarget every lane and pinned runtime that sat at a lower version so nothing builds or runs the package on an unsupported (EBADENGINE-warning) Node: - ci-tests.yml: node-floor-compat 22.14 -> 22.18.0 (name, comment, pin, version assertion) so the floor gate guards the new minimum; its #2372 registerHooks failure mode cannot recur above 22.15. Containment-canary pin 22.16.0 -> 22.18.0. - gitnexus-review-agent.yml + the pinned review/canary runtime: the reproducible runtime is version-locked in lockstep across .github/{gitnexus-review-runtime,claude-canary-runtime}/package.json and their lockfiles (engines), the workflow's node-version, its two 'node --version = v22.18.0' assertions, the lockfile-engines guard, and NODE_VERSION. Moved all of them 22.16.0 -> 22.18.0. - gitnexus-skill-evolution.yml: pinned runtime 22.16.0 -> 22.18.0. - CONTRIBUTING.md prerequisite floor updated. - review-agent-workflow.test.ts, which enforces the runtime lock, updated to expect 22.18.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Abhigyan Patwari
|
2ea00a2b22
|
fix(ci): install root and shared node_modules for the evolution benchmark (#2575)
* fix(ci): install root and shared node_modules for the evolution benchmark The first real workflow_dispatch of the skill-evolution loop failed at task binding: capture_task_dependency_binding aborted with SandboxError: sandbox_copy path is unavailable: node_modules: No such file or directory The benchmark tasks sandbox-copy node_modules from three locations (tasks.scenarios.yaml) — the monorepo root, gitnexus-shared, and gitnexus — mirroring a full dev checkout. The install step only ran `npm ci` in gitnexus/, so the root and gitnexus-shared node_modules never existed and the loop died before any agent ran. Install all three (root, then build gitnexus-shared, then build gitnexus), matching the per-package install in ci-tests.yml plus the root deps the tasks require. A new contract test pins all three installs so this fails in CI rather than on the next real run — the same guard the workflow's other two P1 fixes got. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * fix(ci): only add the missing root install; the subpackage steps already exist The initial fix redundantly rebuilt gitnexus-shared and gitnexus inside the gitnexus step — but the workflow already builds both in their own dedicated steps. Only the monorepo root node_modules was missing. Add a single "Install monorepo root dependencies" step and leave the two subpackage build steps untouched, so the benchmark's root sandbox_copy resolves without double-building. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ --------- Co-authored-by: Gergo Magyar <gergomagyar@icloud.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Gergő Magyar
|
becac9a5d3
|
feat(eval): run the skill-evolution loop online (#2571)
* feat(eval): run the skill-evolution loop online Add a scheduled + dispatch-gated workflow that runs the offline propose -> benchmark -> gate loop (workflow_bench.evolve) in CI with the pinned Claude canary runtime and bubblewrap containment, uploads the benchmark evidence as an artifact, and on a gate-passed promotion opens a human-reviewed PR via the release App token. The applied overlay is bounded to the canonical skill tree and its shipped mirrors; any escape fails the run instead of reaching a PR. The scheduled lane ships disabled behind GITNEXUS_EVOLUTION_ENABLED and requires the new GITNEXUS_BENCH_AUTH_TOKEN secret (benchmark sessions bill real API usage), mirroring the review agent's staged rollout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * fix(ci): restructure promotion-PR script so no lint suppression is needed Replace the inline single-quoted credential helper with a GIT_ASKPASS file written via a quoted heredoc (the App token still reaches git only through step env at push time), and assemble the PR body from quoted heredocs plus double-quoted printf instead of a backtick-laden single-quoted template. Every run script in the workflow now passes shellcheck with zero findings and zero disables; the body and askpass rendering are smoke-tested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * fix(ci): apply gate-passing overlays in the evolution loop The loop invoked workflow_bench.evolve without --apply, so apply_promoted_overlay (its only working-tree writer, gated by `if args.apply:`) never ran. git status stayed clean, promoted=false was emitted every run, and the App-token/PR-open steps were unreachable dead code — a gate-passing run went green as "No promotion this run". validate_promotion_for_apply already runs before the apply gate, so adding --apply lets a passing candidate reach the tree without weakening the deterministic gate; the boundary check then confirms it stayed in the skill trees. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * fix(ci): provision ~/GitNexus so the benchmark repo resolves on CI Every scenario in tasks.scenarios.yaml addresses the target repo as ~/GitNexus; runner_tasks.py resolves it with expanduser().resolve() then `git -C <repo> rev-parse`, which raises when the path is missing. On a hosted runner the checkout lands in $GITHUB_WORKSPACE and nothing created ~/GitNexus, so the first real run failed at task-binding. Symlink ~/GitNexus -> $GITHUB_WORKSPACE before the loop. The checkout uses fetch-depth: 0 (full history for the parentless clone), and the benchmark only clones the repo copy-on-write and mounts deps read-only, so the checkout is never mutated. GITNEXUS_BENCH_ORACLE_ROOT stays unset — it defaults to the in-repo oracles dir and is staged by the harness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * fix(ci): harden promotion summary output and PR branch recovery Three fixes to the promotion-detection and PR-open steps: - GITHUB_OUTPUT summary used a fixed `PROMOTION_EOF` heredoc delimiter; a value containing that marker on its own line could close the block early and inject output keys. Use a per-run random delimiter, matching the pattern already in tree-sitter-upgrade-readiness.yml. - The summary concatenated every generation's promotion.json (including rejected ones), so the PR body could show a losing generation's decisions. The loop returns on the first promotion, so emit only the highest-numbered gen-N/bench/promotion.json — the decision that fired. - The promotion branch name omitted the run attempt. GITHUB_RUN_ID is stable across re-runs, so a re-run after push-succeeds/PR-create-fails could never push. Include ${GITHUB_RUN_ATTEMPT} (the artifact name already does). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * fix(ci): least-privilege the promotion App token and gate on an Environment The Mint-App-Token step passed only app-id + private-key, so the minted token inherited every permission the Release App installation holds (including Workflows: write) — far more than "push a branch, open a PR". Switch to `client-id` (as publish.yml does) and request only permission-contents: write + permission-pull-requests: write. Bind the job to a protected Environment (gitnexus-evolution) so promotion runs can be gated server-side. workflow_dispatch runs the workflow and in-tree evolve.py from the *dispatched ref*, so a code-side ref guard is removable by the dispatched branch itself; an Environment deployment-branch rule (main only) is the boundary that holds. The admin steps to create it and scope the secrets are documented in the activation checklist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * fix(ci): correct upload-artifact pin comment and add shell strict-mode - The upload-artifact SHA 043fb46d… is v7.0.1 (labeled so in the sibling workflows that pin it); the comment mislabeled it # v6.0.0. Correct the comment; the pin is unchanged. - Add `set -euo pipefail` to the two build steps that lacked it, matching every other run block in the file (GitHub's default shell already sets -eo pipefail; this adds -u and consistency). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * docs(ci): complete the skill-evolution activation checklist - Add RELEASE_APP_ID / RELEASE_APP_PRIVATE_KEY to the required-secrets checklist (the Mint step hard-fails without them on a promotion) and the App-install-scope verification. - Document the protected Environment admin step and why it is the real boundary for the workflow_dispatch ref-secret exposure. - Note that workflow_dispatch runs the billing loop regardless of GITNEXUS_EVOLUTION_ENABLED. - Justify the weekly cron against the README's ~90-day guidance and note the 355-minute timeout ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * fix(eval): redact API tokens from diagnostic fields before artifact upload results.jsonl (runner.py) and proposer-session.json (evolve.py) serialize session records whose error_detail can carry a stderr_tail that echoed the API key. Transcripts are redacted before persistence, but these two sinks were not, and both land in the 14-day evolution artifact. Run each record's serialized JSON through the existing redact_text with the run's auth token before writing. Scoped to these diagnostic sinks only: the promoted overlay and proposal.md are left untouched (the overlay is the applied artifact and must stay byte-identical for apply and the shipped-skills-sync guard). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * test(ci): add a contract test for the skill-evolution workflow No test exercised this workflow's path, which is why both P1 blockers (missing --apply, unresolvable ~/GitNexus task repo) reached production. Parse the workflow YAML and assert the structural contract: --apply is passed, the task repo is provisioned, the promotion branch carries the run attempt, the App token is permission-scoped and the job is Environment- gated, the output summary uses a random delimiter and a single generation, the artifact pin is labelled correctly, and every multi-line shell step sets strict mode. Follows the review-agent-workflow.test.ts precedent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ * feat(ci): run the proposer on its own (stronger) model One `model` input drove both the benchmark arms and the proposer/diagnosis session. Split them: `model` stays the benchmark arms (match the model your skill users run, so a promotion is valid for them and the tasks aren't ceiling-saturated), and a new `proposer_model` input runs the proposer — the harder meta-reasoning task that writes the candidate skill, and only one session per generation, so a stronger model is cheap here. evolve.py already supports --proposer-model; the workflow just didn't expose it. Defaults: arms = claude-sonnet-5, proposer = claude-opus-4-8 (both overridable via workflow_dispatch). The weekly cadence bounds the added spend. Contract test asserts the split stays wired. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |