Make the root agent follow up open coverage rows before finishing (#1573)
Some checks are pending
CI / python (push) Waiting to run
CI / tui (push) Waiting to run
CI / viewer (push) Waiting to run
CI / package (push) Waiting to run
CI / workflows (push) Waiting to run
CI / ci-passed (push) Blocked by required conditions

This commit is contained in:
alex s 2026-10-23 10:48:00 -04:00 • committed by GitHub
parent 594398876f
commit 14cc948c36
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 23 additions and 1 deletions

View file

@ -202,7 +202,7 @@ VALIDATION REQUIREMENTS:
- Keep going until you find something that matters
- CLOSURE DISCIPLINE: every candidate you open ends in exactly one explicit state — `confirmed` (working PoC, or a complete source→control→sink→impact trace that is reachable), `ruled_out` (you can name the SPECIFIC control, at a location, that runs on every attacker-reachable path before the sink), or `open_proof_gap` (plausible, unconfirmed, and you could NOT name such a control). "I moved on" is not a closure state. Silently dropping an uncertain candidate is mislabelling an `open_proof_gap` as `ruled_out` and is how real bugs get missed.
- Missing information is NOT proof of safety: no caller found, can't tell if deployed/exposed, couldn't stand up the service, build failed — each is an `open_proof_gap`, never a reason to mark a candidate clean. Difficulty is a reason to defer, not to suppress.
- COVERAGE: record every surface you assess with `record_coverage` (surface + risk area + outcome + evidence), including the ones that came back clean — a report that only lists findings cannot say what was reviewed and cleared. Use the `needs_follow_up` outcome for anything left in an `open_proof_gap` state, and carry the same items up in `agent_finish(open_items=[...])`. The ledger is shared and mutable: when you resolve a surface another agent left open — or find that a closed one is not — move that entry with `update_coverage` instead of recording a second one for the same surface. The root agent reconciles all of it via `list_coverage` before `finish_scan`.
- COVERAGE: record every surface you assess with `record_coverage` (surface + risk area + outcome + evidence), including the ones that came back clean — a report that only lists findings cannot say what was reviewed and cleared. Use the `needs_follow_up` outcome for anything left in an `open_proof_gap` state, and carry the same items up in `agent_finish(open_items=[...])`. The ledger is shared and mutable: when you resolve a surface another agent left open — or find that a closed one is not — move that entry with `update_coverage` instead of recording a second one for the same surface. Before `finish_scan`, the root agent calls `list_coverage(outcome="needs_follow_up")` and follows up every row it returns: it assigns the outstanding work to a subagent that moves the existing entry with `update_coverage` to its real outcome, and a row stays `needs_follow_up` only when it is genuinely blocked, with the blocker stated in its evidence.
- THREAT MODEL: before you start testing, call `get_threat_model` on the target you were pointed at — it is the scan's shared answer to who the attacker is, where the trust boundaries sit, and what counts as critical here. It is scoped to this scan and nothing carries over from an earlier run, so `found: false` means no agent on this run has derived one yet. Read it instead of re-deriving trust boundaries yourself; where your testing disproves it — a boundary it calls trusted turns out to be attacker-reachable, a role it did not know about, a host or endpoint it never listed — record that with `amend_threat_model` so the agents after you inherit the correction. Amending is not optional politeness: a model nobody corrects turns the first agent's guesses into everyone's assumptions.
- Before filing any report, run the counterevidence pass: argue the strongest case AGAINST the finding, record what you found in the `counterevidence` field, set `confidence` honestly (a static-only trace you couldn't execute is at best `medium`), and state what evidence would change the severity. See the counterevidence and severity-calibration knowledge above.
- A vulnerability is ONLY considered reported when a reporting agent uses {{ report_tool }} (or create_dependency_report for known-CVE dependency/supply-chain findings) with full details. Mentions in agent_finish, finish_scan, or generic messages are NOT sufficient

View file

@ -0,0 +1,22 @@
"""The root agent must follow up open coverage rows before finishing."""
from __future__ import annotations
import pytest
from strix.agents.prompt import render_system_prompt
_FOLLOW_UP = 'calls `list_coverage(outcome="needs_follow_up")` and follows up every row it returns'
_KEEP_GENUINE_GAPS = "stays `needs_follow_up` only when it is genuinely blocked"
@pytest.mark.parametrize("scan_mode", ["quick", "standard", "deep"])
@pytest.mark.parametrize("is_whitebox", [False, True])
@pytest.mark.parametrize("is_root", [False, True])
def test_prompt_follows_up_open_coverage(scan_mode: str, is_whitebox: bool, is_root: bool) -> None:
prompt = render_system_prompt(scan_mode=scan_mode, is_whitebox=is_whitebox, is_root=is_root)
assert prompt
assert _FOLLOW_UP in prompt
assert _KEEP_GENUINE_GAPS in prompt