diff --git a/strix/agents/prompts/system_prompt.jinja b/strix/agents/prompts/system_prompt.jinja index b8300a1d..dd39ea4f 100644 --- a/strix/agents/prompts/system_prompt.jinja +++ b/strix/agents/prompts/system_prompt.jinja @@ -202,7 +202,7 @@ VALIDATION REQUIREMENTS: - Keep going until you find something that matters - CLOSURE DISCIPLINE: every candidate you open ends in exactly one explicit state — `confirmed` (working PoC, or a complete source→control→sink→impact trace that is reachable), `ruled_out` (you can name the SPECIFIC control, at a location, that runs on every attacker-reachable path before the sink), or `open_proof_gap` (plausible, unconfirmed, and you could NOT name such a control). "I moved on" is not a closure state. Silently dropping an uncertain candidate is mislabelling an `open_proof_gap` as `ruled_out` and is how real bugs get missed. - Missing information is NOT proof of safety: no caller found, can't tell if deployed/exposed, couldn't stand up the service, build failed — each is an `open_proof_gap`, never a reason to mark a candidate clean. Difficulty is a reason to defer, not to suppress. -- COVERAGE: record every surface you assess with `record_coverage` (surface + risk area + outcome + evidence), including the ones that came back clean — a report that only lists findings cannot say what was reviewed and cleared. Use the `needs_follow_up` outcome for anything left in an `open_proof_gap` state, and carry the same items up in `agent_finish(open_items=[...])`. The ledger is shared and mutable: when you resolve a surface another agent left open — or find that a closed one is not — move that entry with `update_coverage` instead of recording a second one for the same surface. The root agent reconciles all of it via `list_coverage` before `finish_scan`. +- COVERAGE: record every surface you assess with `record_coverage` (surface + risk area + outcome + evidence), including the ones that came back clean — a report that only lists findings cannot say what was reviewed and cleared. Use the `needs_follow_up` outcome for anything left in an `open_proof_gap` state, and carry the same items up in `agent_finish(open_items=[...])`. The ledger is shared and mutable: when you resolve a surface another agent left open — or find that a closed one is not — move that entry with `update_coverage` instead of recording a second one for the same surface. Before `finish_scan`, the root agent calls `list_coverage(outcome="needs_follow_up")` and follows up every row it returns: it assigns the outstanding work to a subagent that moves the existing entry with `update_coverage` to its real outcome, and a row stays `needs_follow_up` only when it is genuinely blocked, with the blocker stated in its evidence. - THREAT MODEL: before you start testing, call `get_threat_model` on the target you were pointed at — it is the scan's shared answer to who the attacker is, where the trust boundaries sit, and what counts as critical here. It is scoped to this scan and nothing carries over from an earlier run, so `found: false` means no agent on this run has derived one yet. Read it instead of re-deriving trust boundaries yourself; where your testing disproves it — a boundary it calls trusted turns out to be attacker-reachable, a role it did not know about, a host or endpoint it never listed — record that with `amend_threat_model` so the agents after you inherit the correction. Amending is not optional politeness: a model nobody corrects turns the first agent's guesses into everyone's assumptions. - Before filing any report, run the counterevidence pass: argue the strongest case AGAINST the finding, record what you found in the `counterevidence` field, set `confidence` honestly (a static-only trace you couldn't execute is at best `medium`), and state what evidence would change the severity. See the counterevidence and severity-calibration knowledge above. - A vulnerability is ONLY considered reported when a reporting agent uses {{ report_tool }} (or create_dependency_report for known-CVE dependency/supply-chain findings) with full details. Mentions in agent_finish, finish_scan, or generic messages are NOT sufficient diff --git a/tests/test_coverage_reconciliation_prompt.py b/tests/test_coverage_reconciliation_prompt.py new file mode 100644 index 00000000..e476cc15 --- /dev/null +++ b/tests/test_coverage_reconciliation_prompt.py @@ -0,0 +1,22 @@ +"""The root agent must follow up open coverage rows before finishing.""" + +from __future__ import annotations + +import pytest + +from strix.agents.prompt import render_system_prompt + + +_FOLLOW_UP = 'calls `list_coverage(outcome="needs_follow_up")` and follows up every row it returns' +_KEEP_GENUINE_GAPS = "stays `needs_follow_up` only when it is genuinely blocked" + + +@pytest.mark.parametrize("scan_mode", ["quick", "standard", "deep"]) +@pytest.mark.parametrize("is_whitebox", [False, True]) +@pytest.mark.parametrize("is_root", [False, True]) +def test_prompt_follows_up_open_coverage(scan_mode: str, is_whitebox: bool, is_root: bool) -> None: + prompt = render_system_prompt(scan_mode=scan_mode, is_whitebox=is_whitebox, is_root=is_root) + + assert prompt + assert _FOLLOW_UP in prompt + assert _KEEP_GENUINE_GAPS in prompt