mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-11 03:38:07 +00:00
* test(release): add fixed-answer native tool accuracy
* fix(release): gate stable artifacts on pinned paired evaluations
Apply review findings #1-4: gate Docker publication, grade in-flight retries, preserve repository context, and execute negative controls in CI.
* fix(eval): verify native context and resolve harness security findings
Inspect Claude's first startup reminder for ordinary repository guidance.
Bound launcher-option matching and escape Markdown backslashes and pipes.
Exclude only the parsed synthetic accuracy corpus from CodeQL; keep the
benchmark harness and evaluator under security analysis.
Validated focused Python and TypeScript regressions plus core typecheck.
* fix(eval): wake the EC2 runner and prove paired evaluator execution
Bootstrap the existing dedicated instance from a protected hosted job, bound
native runner pickup, and stop only instances started by this run. Keep paid
sessions inside the fixed stop window and add a runner-only dispatch.
Require a contained six-cell prepare/session/oracle/report canary in CI,
including a failed repair that remains in the measured denominator.
* fix(ci): reuse existing configuration and disambiguate accuracy cases
Remove the new AWS credential/settings path and release opt-in variable. Bound native runner pickup with the existing GitHub token and retain private EventBridge startup until its existing mechanism can be reused. Reuse current workflow environment names and evolution schedule controls.
Name malformed-observation cases explicitly so the execution audit distinguishes null from missing and actually tests invalid arrays.
* fix(review): isolate release builds and preserve benchmark evidence
* fix(release): gate RC publication on paired quality and manage EC2 lifecycle
* fix(ci): clarify release shell redirects and build script
* fix(eval): remove temporary patch sinks after capture
* fix(bench): retire tool-accuracy allowances repaired on main
Main at 50aa4be3b repaired three #3487 route checks (#3505) and all four
#3499 Python scope checks (#3502, #3504). The ratchet correctly failed the
merged head with stale allowances, so remove them and keep the fixed answers
as release protection: 18/31 pass, 13 recorded gaps remain.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(release): gate RCs on accuracy only and keep paid runs for stable
Follow the cost split proposed in discussion #3493: fixed-answer tool
accuracy on every RC, paired agent runs before stable releases and on a
weekly schedule against the latest RC.
RC publication no longer prepares a bundle, calls release-evaluation.yml
and waits for a paid run of up to 21 hours on the shared EC2 lock. Every
release attaches this run's accuracy report; stable npm publication still
requires exact-commit paired evidence. release-evaluation.yml drops its
workflow_call path and the now-unused candidate bundle module.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(release): address PR #3503 review feedback
- Page through recent release-evaluation runs instead of reading only
the newest 50, so valid stable evidence cannot be pushed off page one.
- Bound each runner-pickup jobs request by the remaining deadline.
- Replace non-finite floats at any depth in failure receipts so the
incomplete report still serializes with allow_nan=False.
- Clarify that the summary.cap case has 501 names over 1001 sites.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(review): harden candidate build and close review gaps
- Mount the candidate checkout's .git read-only inside the build sandbox,
so lifecycle scripts cannot plant git config (e.g. core.fsmonitor) that
host git later runs in that checkout during actions/checkout cleanup.
The real-Bubblewrap canary now also attempts that write.
- Type tool-accuracy observations instead of using explicit any.
- Cover the fail-closed branches reviewers found untested: unsafe or
missing evidence artifacts, mismatched task pins, malformed repo/SHA,
an instance that stops again during bootstrap, and unbalanced GitNexus
guidance markers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(release): describe RC accuracy gate and stable evidence step
- CONTRIBUTING now states that RCs are gated by CI (including the
fixed-answer tool-accuracy check) without a paid run, and that stable
publication needs a passing Release evaluation for the exact commit
within seven days.
- The oracle-control fixture passes tarfile's data filter only where it
exists (3.11.4+), matching the declared Python 3.11 floor.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* refactor(bench): pin tool-accuracy fixture line anchors in one place
run.ts bucketed rename edits and explain findings by line literals that
expectations.ts duplicated. Export FIXTURE_ANCHORS (file, line, pinned text)
from expectations.ts, build the fixed answers and run.ts classifiers from it,
and fail loudly (runner and unit test) when a fixture line no longer holds
the pinned text. Expected answers and observations are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(bench): fail closed when ripgrep is missing from the tool-accuracy run
rename's text-search pass shells out to rg and only logs a degraded warning
when it is absent, so a runner without ripgrep scored a degraded rename
(homonym.ts:1 missing) and still passed. Check rg --version before indexing,
record it in accuracy.json source.externalTools, and install ripgrep in the
ci-tests tool-accuracy job and the docker release-gate (same if: condition).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* refactor(eval): split baseline guidance, share bwrap preamble, drop dead bare option, harden patch cleanup
- Move the baseline (no-GitNexus) guidance scrubbing out of proposer_sandbox.py
into baseline_guidance.py, with its unit tests.
- Expose real_directory, runtime_mount_args and bwrap_base_args publicly so
release_build no longer imports private helpers or re-assembles the shared
bubblewrap preamble; the candidate build command is byte-identical.
- Remove the unused run_claude `bare` parameter and its --bare/--tools branches.
- capture_patch: a failed temporary-directory removal no longer discards the
read patch or replaces an earlier error; it is attached as a note, or warned
on stderr when there was no earlier error.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(ec2-runner): keep stop observing through transient describe failures
- stop() treats a failed or timed-out describe-instances call as unknown
state and keeps observing until the stop deadline; a persistent failure
is still raised at the deadline with the last command error. Unusable
states and identity mismatches still fail at once (new CommandError
subclass separates command failures from semantic ones).
- Readiness CLI has one paid mode: --job-name NAME. --evolve is removed;
skill evolution passes its job name, and tests tie the watchdog command
to the paid job's actual name in both EC2 workflows.
- Jobs-API requests never get a timeout larger than the remaining pickup
budget (previously rounded up to 1s past the deadline).
- Document that the shared concurrency group keeps only one pending run,
so a queued release evaluation can be replaced and must be confirmed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(eval): record verified EC2 lifecycle configuration
The README still described the AWS settings as assumed and unverified.
The OIDC role, environment settings and runner_only lifecycle were
configured and verified live on 2026-10-09.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(release-eval): grade both runtimes against one pinned task toolchain
Candidate and stable each staged task dependencies (node_modules,
gitnexus-shared/dist) from their own runtime checkout, so the same v1.6.12
task source was graded with different TypeScript, Vitest and LadybugDB
versions and a per-task solve delta also measured dependency drift.
- Check out and build the pinned task commit once (tasks-base) and point
every task's repo at it for both runtimes; the runtime under test reaches
sessions only through --gitnexus-root. prepare requires --task-repo at the
single pinned task commit (new task-sha subcommand resolves it).
- Retain sandbox_dependency_content_digest per measured row and make
release_gate fail when any task's cells, across candidate and stable, do not
share one recorded dependency digest.
- Move the task-pin comparison into validate_report(task_pins=...) and use
it from release_report check and release_gate.
- Cover the gate CLI with a missing and a corrupt candidate report.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(ci): warn that a pending release evaluation can be replaced
GitHub keeps one pending run per concurrency group, so a newer queued
EC2 run cancels a pending release evaluation. Operators must confirm the
startup job ran before relying on the evidence.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(eval): prove hidden oracles against the grading toolchain
Release evaluation now grades every task with dependencies built at the
pinned task commit (TypeScript 5.9.3, Vitest 4.1.11), but the oracle
controls still ran against the current checkout's toolchain. The Ubuntu
containment job now builds the pinned task toolchain and points the
controls at it. All seven controls pass locally against that build.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(ci): pin the duplicated EC2 lifecycle jobs together
Both EC2 workflows carry their own copy of the live-verified start,
check, readiness and stop jobs. A reusable workflow would change that
verified job structure, so instead a test requires the copies to stay
identical apart from the job display name and the paid-job dependency.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(release-eval): accept stable evidence only from main's current evaluator
Stable publish validated downloaded agent evidence against the release
commit's eval/ tree and never tied a report to the run that produced it.
Evidence from an older evaluator stayed acceptable for seven days after
graders changed, while task changes on main made every fresh evaluation
mismatch the tag. The first invalid artifact also hid older valid runs.
The composite action now extracts eval/ from main's head and runs the
validator there with locked base dependencies only (no dev extras or
project build). A report counts only when its harness_sha is the head_sha
of its trusted main run and the compare API shows no change under eval/,
the release-evaluation workflow or the pinned agent CLI between that
harness and the evaluator; a missing, diverged or 300-file comparison
fails closed. Runs are tried newest first, rejected runs are skipped
with a bounded reason list, every gh call has a timeout, and task pins
use validate_report's task_pins check.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(eval): break the sandbox import cycle and resolve code-scanning alerts
- baseline_guidance is now pure text handling with its own GuidanceError;
the baseline mount builder lives in proposer_sandbox again and converts
that error to SandboxError, so imports run one way only.
- task_assets uses the public real_directory name; the unused private
alias is gone.
- The oracle-control fixture always extracts with tarfile's data filter
and skips on Pythons that lack it, instead of extracting unfiltered.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(release-eval): run candidate lifecycle scripts offline; reject smuggled report fields
- build_candidate now downloads locked dependencies with --ignore-scripts,
then runs every lifecycle script (dependency installs, prepare, build)
in a fresh network namespace, so candidate code cannot reach host-local
services. Verified locally that the current head and the pinned task
commit build this way; the real-Bubblewrap canary now asserts only
loopback is visible to a lifecycle script.
- validate_report requires the evidence document to equal its
field-whitelisted rebuild, so per-run or top-level fields outside the
published schema are rejected before the publisher copies the file.
- The sandbox test suite also covers a no-GitNexus sandbox refusing
unbalanced guidance markers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* docs(eval): describe the offline candidate lifecycle
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(release-eval): pin downloads to the npm registry; require dependency binding
Address review feedback on PR #3503:
- The candidate's online install phase now refuses lockfile entries that
resolve outside https://registry.npmjs.org/ (other than in-checkout
workspace links) and refuses shipped .npmrc files, so a candidate
cannot steer npm at loopback, private or metadata endpoints.
- Every measured row must carry a 64-hex task-dependency digest, so a
single report cannot be accepted with its grading toolchain unbound.
- Oversized JSON integers mark a measurement untrustworthy instead of
crashing report generation.
- Baseline guidance stripping recognises headings with up to three
leading spaces (CommonMark).
- Remove a stale comment about the old private helper name.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(eval): parse CommonMark ATX headings; compare evidence by exact JSON
- Baseline guidance stripping now parses headings as CommonMark ATX
headings (0-3 leading spaces, optional closing # run, tabs), so a
GitNexus section heading like '## GitNexus rules ##' is removed too.
- Evidence validation compares the serialized rebuild with the report,
so a retyped value (true for 1, 2.0 for 2) no longer passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(eval): drop continuations of every CommonMark list item in baseline guidance
A removed GitNexus instruction left its indented continuation behind when
the item used '+' or an ordered marker (1. / 1)). The scrubber now
recognises every CommonMark list-item marker.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(eval): isolate candidate MCP from agent task writes (#3503)
Run MCP in a nested Bubblewrap boundary with read-only task, graph,
registry and runtime mounts, private state and isolated processes/network.
Keep task writes on the agent's built-in tools and remove mutating MCP grants.
Add startup/tool mutation canaries and preserve explicit unsafe diagnostics.
Validation: focused pytest 138 passed, 16 skipped. Full locked evaluator
996 passed, 29 skipped; real containment canaries require Ubuntu CI.
Note: eight pre-existing comparator-reuse failures reproduce on the
unchanged head due to missing os.supports_dir_fd support. One process-control
timeout failed in the full run and passed alone and on the unchanged head.
* fix(eval): close baseline mount overlap and test symlink evidence (#3503)
Reject supplied no-MCP mounts that cover forbidden GitNexus paths, including ancestor mounts and lexical aliases. Keep ordinary dependency mounts available.
Exercise evidence symlink rejection with a real valid-report target and retain dangling-link coverage.
Validation: 1026 evaluator tests passed, 29 skipped; targeted mount tests 36 passed and evidence tests 34 passed. Removing the symlink guard in memory makes the repaired regression fail. Ruff and diff checks passed.
Note: eight pre-existing comparator-reuse failures remain in this environment because os.supports_dir_fd lacks the required os.lstat support.
---------
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
86 lines
3.1 KiB
Python
86 lines
3.1 KiB
Python
"""Bound this run's native probe or paid job pickup to five minutes."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import time
|
|
|
|
|
|
def wait_for_runner(*, timeout: float = 300, job_name: str | None = None) -> None:
|
|
"""Require probe success, or bound pickup of the named paid job."""
|
|
repository = os.environ.get("GITHUB_REPOSITORY", "")
|
|
run_id = os.environ.get("GITHUB_RUN_ID", "")
|
|
if repository != "abhigyanpatwari/GitNexus" or not re.fullmatch(r"\d+", run_id):
|
|
raise ValueError(
|
|
"runner pickup requires this repository's current workflow run"
|
|
)
|
|
until = time.monotonic() + timeout
|
|
paid = job_name is not None
|
|
job_name = job_name or "Verify the runner service is online"
|
|
# Never start a request with no time left, nor let one outlive the deadline.
|
|
while (remaining := until - time.monotonic()) > 0:
|
|
result = subprocess.run(
|
|
[
|
|
"gh",
|
|
"api",
|
|
f"repos/{repository}/actions/runs/{run_id}/jobs?per_page=100",
|
|
],
|
|
capture_output=True,
|
|
text=True,
|
|
timeout=min(30, remaining),
|
|
)
|
|
if result.returncode:
|
|
raise RuntimeError(
|
|
"cannot verify runner pickup through the current run's jobs API"
|
|
)
|
|
jobs = json.loads(result.stdout).get("jobs", [])
|
|
probes = [
|
|
job for job in jobs if job.get("name", "").split(" / ")[-1] == job_name
|
|
]
|
|
if len(probes) > 1:
|
|
raise RuntimeError("current run has multiple runner pickup probes")
|
|
if paid and probes:
|
|
if probes[0].get("status") == "in_progress":
|
|
return
|
|
if probes[0].get("status") == "completed":
|
|
if probes[0].get("conclusion") in ("skipped", "cancelled"):
|
|
raise RuntimeError("paid job did not run")
|
|
return
|
|
if probes and probes[0].get("status") == "completed":
|
|
if probes[0].get("conclusion") != "success":
|
|
raise RuntimeError("runner pickup probe failed")
|
|
return
|
|
time.sleep(min(10, max(0, until - time.monotonic())))
|
|
raise RuntimeError(
|
|
"runner service did not complete its pickup probe before the deadline"
|
|
)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
try:
|
|
args = sys.argv[1:]
|
|
if len(args) == 2 and args[0] == "--job-name" and args[1].strip():
|
|
wait_for_runner(job_name=args[1])
|
|
elif args == []:
|
|
wait_for_runner()
|
|
else:
|
|
raise ValueError("expected no arguments or --job-name NAME")
|
|
print("The dedicated runner picked up this run's requested job.")
|
|
except (
|
|
ValueError,
|
|
RuntimeError,
|
|
subprocess.TimeoutExpired,
|
|
OSError,
|
|
json.JSONDecodeError,
|
|
) as exc:
|
|
message = (
|
|
str(exc)
|
|
if isinstance(exc, (ValueError, RuntimeError))
|
|
else "runner readiness command failed"
|
|
)
|
|
print(f"::error::{message}")
|
|
raise SystemExit(1) from None
|