feat(home): support multi-case case-study runs with dedicated LLM and Git sandbox

- Allow home_case_study to investigate up to eight ordered cases per
  invocation with a shared focus, one isolated agent per case, while
  keeping the legacy single-case selector as a deprecated alias
- Configure a dedicated home_case_study LLM component instead of the
  shared judge backend
- Wrap ReMe's OS file sandbox with a Home-specific Git runtime grant
  (developer directory on macOS) and argv-based child environment
- Tighten case-study prompts around evidence certainty, code-tool path
  scoping, citation form, and unresolved-cause reporting
- Extend evidence tracing and expand case-study/sandbox tests and docs
This commit is contained in:
酱牛肉 2026-10-30 17:17:17 +08:00
parent 8e7e6bc91d
commit febc84586e
11 changed files with 824 additions and 85 deletions

View file

@ -674,14 +674,18 @@ index, evaluation history or sampling state.
memory checkpoints without calling a model. Dataset evidence annotations take
precedence over optional inferred session IDs, including an empty annotation.
Returns `query` (question, expected/actual answer, score, error and
`result_missing`), `evidence_source` (`independent_annotation`,
`result_missing` and `expected_kind`), `locations` with JSON Pointers into source
and saved query files, optional `criteria` previews of saved rubric judgments,
`evidence_source` (`independent_annotation`,
`diagnostician_inference`, or `unavailable`), `evidence_is_certain`,
`memory_status`, `sessions` (ID, checkpoint and changed files), and `paths` to
`memory_status`, `sessions` (ID, source JSON Pointer, checkpoint and changed files), and `paths` to
the normalized case, memory repository, query results and query artifacts.
A null checkpoint means no matching `session: <id>` commit was found; missing
results keep scores null. Answer/question previews are limited to 2,000
characters and errors to 600. At most 50 sessions and 50 changed files per
session are returned, with omitted counts when needed. Session text, diffs,
session are returned, with omitted counts when needed. Criterion previews are
limited to 20 entries and 600 characters per requirement/reason; absent or
mismatched rubric judgments are not paired. Session text, diffs,
later history and full answer/judge results remain at the returned paths or
in the inspection/comparison Jobs below. Original transcripts and derived
memories can both be searchable; these coordinates alone do not prove a cause.
@ -726,26 +730,38 @@ form is `evaluations/candidate/abcdef0/1`. For agent tools, host `injected_job_k
`validation_path` and `case_id`; the wrapper hides injected fields and rejects
model overrides. Like the other Home inspection Jobs, these Jobs reject
`_allowed_paths` restrictions rather than bypassing them. Use these tools directly
or through the single-case investigator below.
or through the case-study batch Job below.
## Single-case investigation
## Case-study batches
`home_case_study(validation_path, case_id, focus=...)` runs one fresh diagnostic
agent. It follows dev_meta's case-study workflow: trace judging and answer
failures backward through retrieval, memory construction, source sessions and
`home_case_study(evaluation_path, case_ids, focus=...)` investigates up to eight
unique case IDs in input order, sharing the evaluation and unchanged focus.
Cases run sequentially, each with fresh agent state, sandbox and timer. The
whole selection is validated before model calls; a case execution failure is
returned individually without cancelling later cases. It follows dev_meta's
case-study workflow: trace judging and answer failures backward through retrieval,
memory construction, source sessions and
the evaluated implementation. The text report cites evidence paths/lines or
JSON fields and distinguishes supported causes, hypotheses and missing evidence;
it does not prescribe fixes. There is no batch fan-out or parallel case-study
dispatcher. Each invocation has its own agent state and timer.
it does not prescribe fixes. There is no parallel case-study dispatcher.
The `home_case_study` component uses `home_case_study_agent`, a thin subclass of
the bounded wrapper, defaulting to
the host's existing `as_llm: judge` model, 60 iterations and a 30-minute advisory
budget, matching dev_meta's default budgets. Configure a separate host model by
overriding `components.agent_wrapper.home_case_study.as_llm`. ReMe owns model
the bounded wrapper, defaulting to the dedicated `as_llm: home_case_study`
component with model `qwen3.7-max`, 60 iterations and a 30-minute advisory
budget, matching dev_meta's default budgets. It uses `LLM_BACKEND`, `LLM_API_KEY`
and `LLM_BASE_URL` without changing the evaluation judge's model. Override
`components.as_llm.home_case_study.model` to choose another model, or
`components.agent_wrapper.home_case_study.as_llm` to select another host component. ReMe owns model
startup, token accounting and saved session transcripts. The response answer is
the report; metadata includes the validation, case, evaluated commit, session ID
and `usage`. No case scratch directory or intermediate analysis files are created.
`{"results": [...]}`; each item contains `case_id`, `success`, and either `report`
with its session ID/usage or `error`. Metadata includes `evaluation_path`, the
evaluated commit and total `usage` (null if any case fails or usage is unavailable).
Response success is false if any case fails; completed sibling reports remain
available. No case scratch directory or intermediate analysis files are created.
`validation_path` remains an alias for `evaluation_path`. Legacy `case_id` calls
still return a text report with per-case metadata. Do not supply both path names
or both case selectors. Other inspection Jobs retain their `validation_path` argument.
The toolkit includes validation inspection, compact query-evidence tracing,
memory history/comparison, and the three committed-code navigation Jobs.
@ -785,7 +801,7 @@ jobs:
With the configured service running, call from another terminal:
```bash
reme home_case_study validation_path=evaluations/candidate/abcdef0/1 case_id=case1 'focus=Trace low-scoring queries'
reme home_case_study evaluation_path=evaluations/candidate/abcdef0/1 'case_ids=["case1","case2"]' 'focus=Trace low-scoring queries'
```
Calls with `_allowed_paths` restrictions fail closed, as other Home artifact

View file

@ -1,14 +1,15 @@
"""One case-local investigation using ReMe Jobs and sandboxed command tools."""
"""Sequential case-local investigations using ReMe Jobs and sandboxed command tools."""
import json
import sys
from pathlib import Path
from agentscope.tool import FunctionTool, Toolkit
from pydantic import Field, field_validator
from pydantic import Field, field_validator, model_validator
import yaml
from reme.schema.token_usage import TokenUsage
from reme.steps.base_step import BaseStep, Ref
from reme.utils.file_sandbox import create_file_sandbox_backend
from ..auto_bash import AutoBash
from ..code.repository import resolve_revision, validate_repository
@ -19,9 +20,11 @@ from ..evaluation.inputs import entity_id
from ..evaluation.memory_inspection_step import _repository
from ..preparation.spec import SpecModel
from ..workspace import HomeWorkspace
from .sandbox import create_case_study_sandbox
from .wrapper import BoundedAgentWrapper
PROMPT = Path(__file__).with_suffix(".yaml")
DEFAULT_FOCUS = "Investigate the failed or low-scoring queries and their earliest supported causes."
CASE_TOOLS = (
"home_inspect_validation",
"home_trace_query_evidence",
@ -29,6 +32,33 @@ CASE_TOOLS = (
"home_compare_memory",
)
CODE_TOOLS = ("home_list_code_tree", "home_search_code", "home_read_code_file")
_SANDBOX_PROBE = """\
import json
import pathlib
import subprocess
import sys
scope = json.loads(sys.argv[1])
with open(scope['source_case'], 'rb') as stream:
stream.read(1)
with open(pathlib.Path(scope['validation_path']) / 'manifest.json', 'rb') as stream:
stream.read(1)
subprocess.run([
'/bin/bash', '--noprofile', '--norc', '-c',
'git -C "$1" --no-pager -c core.fsmonitor=false -c log.showSignature=false '
'show --no-ext-diff --no-textconv --no-patch --format=%H "$2"',
'probe', scope['code_repository'], scope['candidate_commit'],
], check=True)
"""
async def probe_sandbox(backend, scope):
"""Verify the same interpreter, shell, Git and evidence access used by the investigation."""
return await backend.exec_shell(
[sys.executable, "-I", "-B", "-c", _SANDBOX_PROBE, json.dumps(scope)],
cwd=scope["case_path"],
timeout=10,
)
class CaseStudyInput(SpecModel):
@ -37,7 +67,7 @@ class CaseStudyInput(SpecModel):
validation_path: str = Field(min_length=1)
case_id: str = Field(min_length=1)
focus: str = Field(
default="Investigate the failed or low-scoring queries and their earliest supported causes.",
default=DEFAULT_FOCUS,
min_length=1,
)
@ -48,6 +78,39 @@ class CaseStudyInput(SpecModel):
return entity_id(value)
class CaseStudyBatchInput(SpecModel):
"""Share one evaluation and focus across an ordered, bounded case selection."""
evaluation_path: str = Field(min_length=1)
case_ids: list[str] = Field(min_length=1, max_length=8)
focus: str = Field(default=DEFAULT_FOCUS, min_length=1)
@model_validator(mode="before")
@classmethod
def normalize_legacy(cls, value):
"""Accept old single-case arguments without accepting ambiguous combinations."""
if not isinstance(value, dict):
return value
value = dict(value)
for old, new in (("validation_path", "evaluation_path"), ("case_id", "case_ids")):
if old in value:
if new in value:
raise ValueError(f"Provide {new} or {old}, not both")
item = value.pop(old)
value[new] = [item] if old == "case_id" else item
return value
@field_validator("case_ids")
@classmethod
def validate_cases(cls, values):
"""Reject unsafe or duplicate selectors before any agent runs."""
for value in values:
entity_id(value)
if len(set(values)) != len(values):
raise ValueError("case_ids must not contain duplicates")
return values
def prepare_scope(home, spec):
"""Resolve evidence and the validation's immutable candidate revision before model calls."""
run, manifest, _memory = _repository(home, spec.validation_path, spec.case_id)
@ -91,6 +154,11 @@ def python_tool(backend, cwd):
) -> dict:
"""Execute code or file, exclusively; print results to stdout. No host path is writable.
Read case JSON, memory files and documentation here, not with home_read_code_file.
For example code="import json; print(json.load(open('/allowed/result.json'))['queries'][0])".
In code mode do not pass args; in file mode pass file and args, without code.
Check returncode and stderr: a denied or failed read is not evidence that content is absent.
Args:
code: Python source, mutually exclusive with file.
timeout: Execution timeout in seconds, greater than zero and at most 600.
@ -135,10 +203,64 @@ def create_toolkit(wrapper, scope, backend):
)
if name == "home_compare_memory":
injected["target_validation_path"] = None
tools.append(wrapper._make_tool(job, injected_job_kwargs=injected)) # pylint: disable=protected-access
tool = wrapper._make_tool( # pylint: disable=protected-access
job,
injected_job_kwargs=injected,
)
if name in CODE_TOOLS:
tool.description += (
" Reads candidate source at the host-pinned evaluated commit, not artifact or documentation files."
" Paths are repository-relative, for example plugins/beam/src/reme_beam/agentic_answer.yaml;"
" omit path to search/list the root, never use '.' or an absolute path."
" Read memory, logs, JSON and home/docs with AutoBash or python_execute instead."
)
if "path" in tool.input_schema.get("properties", {}):
description = (
"Required candidate-repository-relative file path, never absolute."
if name == "home_read_code_file"
else "Literal repository-relative path, never absolute; omit for the whole repository, not '' or '.'."
)
tool.input_schema["properties"]["path"] = {
**tool.input_schema["properties"]["path"],
"description": description,
}
if name == "home_trace_query_evidence":
tool.description += (
" For a selected query, start here. locations are JSON Pointers into paths.source_case/query_result;"
" session source_pointer locates messages in source_case. criteria previews saved rubric judgments"
" when available. Hit counts and memory coordinates do not prove what reached the answerer."
)
if name == "home_inspect_memory_history":
tool.description += " path filters files within the memory repository; omit it to list all checkpoints."
if name == "home_compare_memory":
tool.input_schema["required"] = list(dict.fromkeys([*tool.input_schema.get("required", []), "base_ref"]))
tool.input_schema["properties"]["base_ref"] = {
"type": "string",
"minLength": 1,
"description": (
"Required earlier memory commit from home_inspect_memory_history;"
" compare with a different target_ref."
),
}
tool.description += (
" Same-case comparison only: supply base_ref and target_ref from home_inspect_memory_history."
' Example: {"base_ref": "<earlier commit>", "target_ref": "<later commit>", "path": "daily"}.'
" Identical refs have no changes; omit path for the whole memory tree."
)
tools.append(tool)
bash = AutoBash(cwd=scope["case_path"], backend=backend, is_read_only=False)
bash.description = (
"Read case artifacts, memory, logs and home/docs with shell commands inside the OS file sandbox."
" No command whitelist; host paths are read-only and network is denied."
" Use absolute allowed paths, e.g. cat '<case_path>/queries/<query_id>/answer.log', or"
" git -C '<memory_repository>' show '<commit>:daily/note.md'."
" Prefer pinned code tools for evaluated candidate source. Keep stderr visible; for pipelines use"
" set -o pipefail so a failed read is not hidden by head/grep. A failed read does not prove absence."
" timeout is in milliseconds."
)
tools.extend(
[
AutoBash(cwd=scope["case_path"], backend=backend, is_read_only=False),
bash,
python_tool(backend, scope["case_path"]),
],
)
@ -148,36 +270,30 @@ def create_toolkit(wrapper, scope, backend):
class HomeCaseStudyStep(BaseStep):
"""Run one fresh agent through the application's bounded wrapper and model lifecycle."""
"""Run ordered cases with fresh agents through the normal ReMe lifecycle."""
workspace: HomeWorkspace = Ref(HomeWorkspace, "home.workspace")
agent_wrapper: BoundedAgentWrapper = Ref(BoundedAgentWrapper, "agent_wrapper")
async def execute(self):
"""Prepare scope, require isolation, then return the case report and framework token usage."""
"""Validate the whole batch, then isolate each sequential investigation and failure."""
if self.context.get("_allowed_paths") is not None:
raise ValueError("Home case study requires unrestricted access to its managed artifacts")
if not self.workspace.is_started or not self.agent_wrapper.is_started:
raise RuntimeError("Start the Home workspace and case-study agent wrapper first")
spec = CaseStudyInput.model_validate(
{key: self.context[key] for key in CaseStudyInput.model_fields if key in self.context},
)
scope = await blocking(prepare_scope, self.workspace.home_path, spec)
backend = create_file_sandbox_backend(
cwd=scope["case_path"],
read_only_paths=scope["read_only_paths"],
writable_paths=[],
memory_limit_mib=self.kwargs.get("memory_limit_mib"),
)
# Fail before spending model tokens if the OS sandbox or its runtime cannot start.
probe = await backend.exec_shell(
[sys.executable, "-I", "-B", "-c", "pass"],
cwd=scope["case_path"],
timeout=10,
)
if not probe.ok():
raise RuntimeError("Case-study sandbox unavailable: " + probe.stderr.decode(errors="replace"))
toolkit = create_toolkit(self.agent_wrapper, scope, backend)
fields = (*CaseStudyBatchInput.model_fields, "validation_path", "case_id")
spec = CaseStudyBatchInput.model_validate({key: self.context[key] for key in fields if key in self.context})
scopes = []
for case_id in spec.case_ids:
scopes.append(
await blocking(
prepare_scope,
self.workspace.home_path,
CaseStudyInput(validation_path=spec.evaluation_path, case_id=case_id, focus=spec.focus),
),
)
if len({scope["candidate_commit"] for scope in scopes}) != 1:
raise ValueError("Evaluation candidate commit changed while preparing case scopes")
output_limit = self.kwargs.get("output_token_limit", 2500)
if isinstance(output_limit, bool) or not isinstance(output_limit, int) or output_limit < 1:
raise ValueError("output_token_limit must be a positive integer")
@ -187,9 +303,58 @@ class HomeCaseStudyStep(BaseStep):
for key in ("system_prompt", "task_prompt")
):
raise ValueError("Case-study YAML requires nonempty system_prompt and task_prompt strings")
response = self.context.response
response.metadata.update(
evaluation_path=scopes[0]["validation_path"],
candidate_commit=scopes[0]["candidate_commit"],
)
# Preserve the original text response and error handling for singular callers.
if "case_id" in self.context:
result = await self._investigate(scopes[0], spec.focus, prompts, output_limit)
response.answer = result["result"]
response.metadata.update(validation_path=scopes[0]["validation_path"], case_id=spec.case_ids[0])
response.metadata.update({key: result[key] for key in ("session_id", "usage") if key in result})
return response
results = []
response.answer = {"results": results}
for scope in scopes:
try:
result = await self._investigate(scope, spec.focus, prompts, output_limit)
results.append(
{
"case_id": scope["case_id"],
"success": True,
"report": result["result"],
**{key: result[key] for key in ("session_id", "usage") if key in result},
},
)
except Exception as exc: # A failed investigator must not discard or cancel sibling reports.
self.logger.exception(f"Case-study investigation failed: case_id={scope['case_id']}")
results.append({"case_id": scope["case_id"], "success": False, "error": f"{type(exc).__name__}: {exc}"})
response.success = all(item["success"] for item in results)
# Failed or unreported calls have unknown usage; never present a partial sum as a full total.
response.metadata["usage"] = (
TokenUsage.combine([TokenUsage.model_validate(item["usage"]) for item in results]).model_dump()
if all(item.get("usage") is not None for item in results)
else None
)
return response
async def _investigate(self, scope, focus, prompts, output_limit):
"""Build a fresh case-scoped toolkit and SDK agent for exactly one case."""
backend = create_case_study_sandbox(
cwd=scope["case_path"],
read_only_paths=scope["read_only_paths"],
writable_paths=[],
memory_limit_mib=self.kwargs.get("memory_limit_mib"),
)
probe = await probe_sandbox(backend, scope)
if not probe.ok():
raise RuntimeError("Case-study sandbox unavailable: " + probe.stderr.decode(errors="replace"))
toolkit = create_toolkit(self.agent_wrapper, scope, backend)
prompt = prompts["system_prompt"].format(**scope, output_token_limit=output_limit)
result = await self.agent_wrapper.reply(
prompts["task_prompt"].format(focus=spec.focus),
return await self.agent_wrapper.reply(
prompts["task_prompt"].format(focus=focus),
system_prompt=prompt,
toolkit=toolkit,
builtin_tools=[],
@ -201,9 +366,3 @@ class HomeCaseStudyStep(BaseStep):
output_schema=None,
permission_mode="bypass",
)
self.context.response.answer = result["result"]
self.context.response.metadata.update(
{key: scope[key] for key in ("validation_path", "case_id", "candidate_commit")},
)
self.context.response.metadata.update({key: result[key] for key in ("session_id", "usage") if key in result})
return self.context.response

View file

@ -3,7 +3,12 @@ system_prompt: |-
retrieval/ranking, final memory, construction checkpoints, original sessions, and the responsible implementation.
At each layer explain what should have happened, what happened, and the supporting or contradicting evidence.
Continue to the earliest evidence-supported mechanism; "not found", "ranked too low" and "wrong answer" are symptoms.
Compare plausible explanations and state the smallest missing observation when no cause is established.
Test the strongest alternative before attributing a cause; code/configuration consistent with a failure is only
a hypothesis without runtime evidence linking it to this outcome. Memory existence and search hit counts do not
prove which content reached the answerer. Your own analysis or printed conclusions are not independent evidence.
Distinguish factual recall from applying a user instruction; missing examples alone do not invalidate a rubric.
Leave causes unresolved when evidence is missing: name the remaining explanations and smallest distinguishing
observation. Do not exclude an explanation elsewhere described as unknown.
Assigned scope:
- Validation: {validation_path}
@ -16,14 +21,18 @@ system_prompt: |-
Treat construction partial_success as requiring repair even when answers succeed. Distinguish infrastructure,
candidate execution, answer-quality and judging failures without replacing their saved runtime statuses.
Use home_inspect_validation to select anomalous queries and locate result fields and logs. Start query evidence
tracing with home_trace_query_evidence. Respect evidence_source and evidence_is_certain: inferred or uncertain
session annotations are leads, not established facts. Original transcripts and derived memory may both be searchable;
For a selected query start with home_trace_query_evidence; use home_inspect_validation for status or selection.
Respect evidence_source and evidence_is_certain: inferred or uncertain session annotations are leads, not facts. Original transcripts and derived memory may both be searchable;
absence from a summary does not prove that the answerer could not retrieve the source session.
Use home_inspect_memory_history and home_compare_memory for construction checkpoints and focused diffs.
For a diff, supply distinct base_ref and target_ref from the returned history; path is memory-repository-relative.
Use home_list_code_tree, home_search_code and home_read_code_file to examine the evaluated commit; their revision
and case-tool scopes are fixed by the host. Follow returned paths for full source details and omitted evidence.
Code-tool paths are candidate-repository-relative; omit path to search/list the root, never use ".".
Read documentation and memory/artifact files with AutoBash or python_execute, not code tools.
A failed/denied read or truncated output cannot establish that information is missing. Keep stderr visible,
use set -o pipefail for shell pipelines, and retry with a suitable reader before drawing a conclusion.
AutoBash and python_execute share an OS file sandbox. The case artifacts, source case, validation manifest,
home/docs and all of home/code are readable; no host paths are writable. System tools and interpreter libraries
@ -36,7 +45,8 @@ system_prompt: |-
Return a concise report, conclusion and strongest evidence first, within approximately {output_token_limit} tokens.
Name affected queries and lifecycle stages, explain the causal chain, include contrary evidence and confidence,
and separate established causes, hypotheses and unknowns. Cite every error and causal claim with an exact
path:line location or path#key.path for structured JSON. Keep citations tight. Make no unsupported cross-case claims.
path:line location or path#JSON-Pointer for structured JSON; cite source locations, not tool-response fields.
Keep citations tight. Make no unsupported cross-case claims.
task_prompt: |-
Pay special attention to:
{focus}

View file

@ -0,0 +1,76 @@
"""Home-specific Git runtime setup around ReMe's unchanged OS file sandboxes."""
import os
import subprocess
import sys
from pathlib import Path
from reme.utils.file_sandbox import create_file_sandbox_backend
def _developer_directory():
"""Locate Apple's actual Git runtime without granting its parent directories."""
if sys.platform != "darwin":
return None
try:
result = subprocess.run(
["/usr/bin/xcode-select", "-p"],
capture_output=True,
text=True,
timeout=5,
check=False,
)
if result.returncode == 0 and result.stdout.strip():
path = Path(result.stdout.strip()).resolve(strict=True)
if (path / "usr/bin/git").is_file() and path != Path("/"):
return str(path)
except (OSError, subprocess.TimeoutExpired, ValueError, RuntimeError):
pass
return None
class CaseStudySandbox:
"""Delegate execution and cleanup to the OS sandbox, adding only child environment values."""
os_name = "posix"
def __init__(self, backend, environment):
self.backend = backend
self.environment = dict(environment)
async def getcwd(self):
"""Preserve the backend's contained working directory."""
return await self.backend.getcwd()
async def exec_shell(self, command, *, cwd=None, timeout=None):
"""Set the environment inside isolation using argv, never shell interpolation."""
if not command:
raise ValueError("command must not be empty")
return await self.backend.exec_shell(
["/usr/bin/env", *[f"{name}={value}" for name, value in self.environment.items()], *command],
cwd=cwd,
timeout=timeout,
)
def create_case_study_sandbox(*, read_only_paths, **kwargs):
"""Explicitly grant Home's Git runtime while preserving core isolation and resource limits."""
roots = list(read_only_paths)
developer = _developer_directory()
if developer and not any(Path(developer).is_relative_to(root) for root in roots):
roots.append(developer)
binary_paths = [str(Path(sys.prefix) / "bin"), str(Path(sys.base_prefix) / "bin")]
if developer:
binary_paths.append(str(Path(developer) / "usr/bin"))
binary_paths.extend(["/usr/local/bin", "/usr/bin", "/bin"])
environment = {
"GIT_CONFIG_NOSYSTEM": "1",
"GIT_CONFIG_GLOBAL": os.devnull,
"GIT_TERMINAL_PROMPT": "0",
"GIT_OPTIONAL_LOCKS": "0",
"PATH": os.pathsep.join(dict.fromkeys(binary_paths)),
}
if developer:
environment["DEVELOPER_DIR"] = developer
backend = create_file_sandbox_backend(read_only_paths=roots, **kwargs)
return CaseStudySandbox(backend, environment)

View file

@ -1,7 +1,11 @@
"""ReMe agent wrapper with Home's bounded investigation runtime."""
import json
from typing import Any
from agentscope.message import TextBlock, ToolResultState
from agentscope.tool import FunctionTool, ToolChunk
from reme.components.agent_wrapper import AsAgentWrapper
from reme.components.as_llm import BaseAsLLM
@ -31,7 +35,7 @@ class CaseStudyAgentWrapper(BoundedAgentWrapper):
rejects an absent investigation model after the judge is validated.
"""
def __init__(self, as_llm: str = "judge", **kwargs):
def __init__(self, as_llm: str = "home_case_study", **kwargs):
super().__init__(as_llm="", **kwargs)
self.home_judge = self.bind("default", HomeJudge, optional=False)
self.as_llm = self.bind(as_llm, BaseAsLLM, optional=True)
@ -40,3 +44,22 @@ class CaseStudyAgentWrapper(BoundedAgentWrapper):
await super()._start()
if self.as_llm is None:
raise ValueError("Case-study agent requires a configured as_llm component")
@classmethod
def _make_tool(cls, job, tool_context_id=None, injected_job_kwargs=None):
"""Expose structured investigation evidence as JSON without changing core Job tools."""
injected = cls._resolve_injected_job_kwargs(
{"tool_context_id": tool_context_id, "injected_job_kwargs": injected_job_kwargs},
)
async def run_job(**kwargs):
response = await job(**cls._merge_injected_job_kwargs(kwargs, injected))
answer = response.answer
text = json.dumps(answer, ensure_ascii=False) if isinstance(answer, (dict, list)) else str(answer)
state = ToolResultState.SUCCESS if response.success else ToolResultState.ERROR
return ToolChunk(content=[TextBlock(text=text)], state=state)
tool = FunctionTool(func=run_job, name=job.name, description=job.description, is_concurrency_safe=False)
if parameters := cls._strip_injected_parameters(job.parameters, injected):
tool.input_schema = parameters
return tool

View file

@ -65,9 +65,13 @@ def trace_query_evidence(home, spec):
if spec.query_id not in manifest.get("query_ids", {}).get(spec.case_id, []):
raise ValueError("query_id is not selected by this evaluation manifest")
source_path, case = _source_case(home, manifest, spec.case_id)
query = next((item for item in case.get("queries", []) if item.get("query_id") == spec.query_id), None)
if query is None:
query_index = next(
(i for i, item in enumerate(case.get("queries", [])) if item.get("query_id") == spec.query_id),
None,
)
if query_index is None:
raise ValueError(f"Normalized case has no query: {spec.query_id}")
query = case["queries"][query_index]
metadata = query.get("metadata") or {}
annotated = metadata.get("evidence_session_ids")
selected = TypeAdapter(SessionIds).validate_python(
@ -78,8 +82,8 @@ def trace_query_evidence(home, spec):
source = "independent_annotation" if annotated is not None else "diagnostician_inference"
if annotated is None and spec.evidence_session_ids is None:
source = "unavailable"
known = {session["session_id"] for session in case.get("sessions", [])}
if unknown := set(selected) - known:
known = {session["session_id"]: i for i, session in enumerate(case.get("sessions", []))}
if unknown := set(selected) - known.keys():
raise ValueError(f"Evidence sessions are not present in case: {sorted(unknown)}")
status, head = _head(repository)
@ -95,26 +99,40 @@ def trace_query_evidence(home, spec):
else []
)
files = [name for name in files if name]
row = {"id": session_id, "checkpoint": checkpoint, "changed_files": files[:MAX_CHANGED_FILES]}
row = {
"id": session_id,
"source_pointer": f"/sessions/{known[session_id]}",
"checkpoint": checkpoint,
"changed_files": files[:MAX_CHANGED_FILES],
}
if len(files) > MAX_CHANGED_FILES:
row["omitted_changed_file_count"] = len(files) - MAX_CHANGED_FILES
sessions.append(row)
results_path = contained(run, f"cases/{spec.case_id}/queries/result.json")
payload = _object(run, str(results_path.relative_to(run)))
result = next(
(item for item in (payload or {}).get("queries", []) if item.get("query_id") == spec.query_id),
result_index = next(
(i for i, item in enumerate((payload or {}).get("queries", [])) if item.get("query_id") == spec.query_id),
None,
)
outcome = result or {}
outcome = payload["queries"][result_index] if result_index is not None else {}
rubric = metadata.get("rubric")
expected_is_rubric = manifest.get("dataset") == "beam" and isinstance(query.get("golden_answer"), list)
if rubric is None and expected_is_rubric:
rubric = query["golden_answer"]
response = {
"query": {
"question": _preview(outcome.get("question", query.get("question")), 2000),
"expected": _preview(outcome.get("golden_answer", query.get("golden_answer")), 2000),
"expected_kind": "rubric" if expected_is_rubric else "reference_answer",
"actual": _preview(outcome.get("answer"), 2000),
"score": outcome.get("score"),
"error": _preview(outcome.get("error")),
"result_missing": result is None,
"result_missing": result_index is None,
},
"locations": {
"source_query": f"/queries/{query_index}",
"saved_query": f"/queries/{result_index}" if result_index is not None else None,
},
"evidence_source": source,
"evidence_is_certain": metadata.get("evidence_is_certain"),
@ -127,6 +145,23 @@ def trace_query_evidence(home, spec):
"query_artifacts": str(contained(run, f"cases/{spec.case_id}/queries/{spec.query_id}", directory=True)),
},
}
judged = ((outcome.get("judge_result") or {}).get("metadata") or {}).get("llm_judge_responses")
if isinstance(rubric, list):
field = "metadata/rubric" if metadata.get("rubric") is not None else "golden_answer"
response["locations"]["rubric"] = f"/queries/{query_index}/{field}"
if isinstance(rubric, list) and isinstance(judged, list) and len(rubric) == len(judged):
response["criteria"] = [
{
"requirement": _preview(requirement),
"score": item.get("score"),
"reason": _preview(item.get("reason")),
"pointer": f"/queries/{result_index}/judge_result/metadata/llm_judge_responses/{i}",
}
for i, (requirement, item) in enumerate(zip(rubric[:20], judged[:20]))
if isinstance(item, dict)
]
if len(rubric) > 20:
response["omitted_criterion_count"] = len(rubric) - 20
if len(selected) > MAX_SESSIONS:
response["omitted_session_count"] = len(selected) - MAX_SESSIONS
return response

View file

@ -19,10 +19,23 @@ backends:
application_defaults:
components:
as_llm:
home_case_study:
backend: ${LLM_BACKEND:-openai}
model: qwen3.7-max
stream: false
context_size: 200000
max_retries: 3
credential:
api_key: ${LLM_API_KEY:-}
base_url: ${LLM_BASE_URL:-}
parameters:
max_tokens: 65536
thinking_enable: false
agent_wrapper:
home_case_study:
backend: home_case_study_agent
as_llm: judge
as_llm: home_case_study
time_budget_minutes: 30
react_config:
max_iters: 60
@ -53,15 +66,29 @@ application_defaults:
jobs:
home_case_study:
backend: base
description: "Investigate one validation case with read-only sandboxed Bash/Python and commit-pinned evidence tools; return an evidence-supported diagnostic report"
description: "Investigate up to eight ordered evaluation cases sequentially with a shared focus; each case uses a fresh read-only sandboxed agent and returns its own report or error"
parameters:
type: object
additionalProperties: false
required: [validation_path, case_id]
allOf:
- oneOf:
- required: [evaluation_path]
- required: [validation_path]
- oneOf:
- required: [case_ids]
- required: [case_id]
properties:
validation_path: {type: string, minLength: 1}
case_id: {type: string, minLength: 1}
focus: {type: string, minLength: 1, description: "Questions or failure mechanisms to prioritize"}
evaluation_path: {type: string, minLength: 1, description: "Shared evaluation result directory"}
case_ids:
type: array
minItems: 1
maxItems: 8
uniqueItems: true
items: {type: string, minLength: 1}
description: "Ordered case IDs; results preserve this order, with one isolated agent per case"
focus: {type: string, minLength: 1, description: "Shared diagnostic request applied unchanged to every selected case"}
validation_path: {type: string, minLength: 1, description: "Deprecated alias for evaluation_path; do not supply both"}
case_id: {type: string, minLength: 1, description: "Legacy single-case selector returning a text report; do not combine with case_ids"}
steps:
- backend: home_case_study_step
agent_wrapper: home_case_study

View file

@ -199,7 +199,12 @@ Start a failed-query investigation with
`home_trace_query_evidence(validation_path, case_id, query_id)`.
It returns compact question/answer/score/error previews, source evidence session
IDs, matching Git checkpoints, changed files and paths to complete evidence. It
does not call a model or include transcripts, diffs or judge-result bodies.
does not call a model or include transcripts, diffs or full judge-result bodies.
`locations` contains JSON Pointers into the source case and saved query result;
each session's `source_pointer` locates it in the source case. `query.expected_kind`
distinguishes reference answers from BEAM rubric lists. Optional `criteria` pairs
saved rubric judgments only when counts match (up to 20, with 600-character
requirement/reason previews); missing judgments are never inferred.
Dataset `evidence_session_ids` annotations take precedence; only when absent may
you supply inferred `evidence_session_ids`. Respect `evidence_source` and
`evidence_is_certain`: inferred or uncertain sessions are leads, not established
@ -238,10 +243,15 @@ inspection paths.
## Single-case diagnostic agent
Call `home_case_study(validation_path, case_id, focus=...)` to investigate exactly
one case. It returns a text report with evidence citations and uncertainty, using
fresh agent state, bounded iterations and advisory time reminders. There is no
parallel case dispatcher. Scope and evaluated code revision are host-injected
Call `home_case_study(evaluation_path, case_ids, focus=...)` for up to eight
unique cases sharing one focus and evaluation. Cases run sequentially and results
preserve input order. Each gets fresh agent state, bounded iterations and advisory
time reminders. The answer contains `results`, with a report/session/usage or error
for each case; a failed case does not cancel the rest. Response success is false
if any case fails. Metadata includes `evaluation_path`, evaluated commit and total
usage (null if any call fails or usage is unavailable). `validation_path` remains
a path alias; legacy `case_id` calls retain a text report. Do not combine aliases.
Other inspection tools still use `validation_path`. There is no parallel case dispatcher. Scope and evaluated code revision are host-injected
into the structured tools. AutoBash and Python share a mandatory read-only file
sandbox covering the selected case artifacts, source JSON, validation manifest,
home/docs and all home/code, plus required system runtimes. Neither executor has
@ -250,5 +260,16 @@ and Python rather than a static command whitelist. The directly readable code
worktree can differ from the evaluated commit; use pinned code tools to verify
version-sensitive claims. No write/edit, validation or code-mutation tools are
exposed. Sandbox startup failure prevents model execution; it never falls back
to an unrestricted process. See the Home README for model, budget and platform
to an unrestricted process. Startup checks Python, source/manifest reads and
`git show` at the evaluated commit before spending model tokens. macOS grants
read-only access to the selected Xcode/Command Line Tools runtime and sets
`DEVELOPER_DIR`; Linux keeps Bubblewrap isolation and searches the virtual
environment and base interpreter's `bin` directories for tools.
Code tools accept candidate-repository-relative paths, not absolute artifact or
documentation paths. Omit `path` to list/search the root. Read logs, source JSON,
memory and home/docs with AutoBash or Python. Memory diffs require `base_ref`
and a different `target_ref` from the memory history. Keep stderr visible and use
`set -o pipefail` for shell pipelines: a failed/denied read is not evidence of
missing content. See the Home README for model, budget and platform
sandbox configuration.

View file

@ -5,14 +5,18 @@
import asyncio
import json
import shutil
import sys
from types import SimpleNamespace
import pytest
from jsonschema import Draft202012Validator
from pydantic import ValidationError
from agentscope.message import TextBlock, ToolCallBlock
from agentscope.model import ChatResponse, ChatUsage
from agentscope.tool import ExecResult
from reme_home.agent import case_study
from reme_home.agent.sandbox import create_case_study_sandbox
from reme_home.evaluation.artifacts import write_json
from reme_home.preparation.code import git
@ -26,7 +30,6 @@ from test_evaluation import ( # noqa: F401 # pylint: disable=unused-import
from test_evaluation_inspection import write_run, snapshot
from reme.utils.local_backend import LocalProcessBackend
from reme.utils.file_sandbox import create_file_sandbox_backend
class RecordingSandbox:
@ -52,7 +55,7 @@ def sandbox(monkeypatch):
instances.append(backend)
return backend
monkeypatch.setattr(case_study, "create_file_sandbox_backend", create)
monkeypatch.setattr(case_study, "create_case_study_sandbox", create)
return instances
@ -95,6 +98,16 @@ class InvestigationModel(RecordingModel):
return ChatResponse(content=content, is_last=True, usage=ChatUsage(input_tokens=2, output_tokens=3, time=0))
@pytest.mark.asyncio
async def test_case_study_uses_dedicated_model_without_changing_judge(app):
models = app.context.components["as_llm"]
wrapper = app.context.components["agent_wrapper"]["home_case_study"]
assert wrapper.as_llm is models["home_case_study"]
assert wrapper.as_llm.kwargs["model"] == "qwen3.7-max"
assert app.context.components["agent_wrapper"]["judge"].as_llm is models["judge"]
assert models["home_case_study"] is not models["judge"]
@pytest.mark.asyncio
async def test_single_run_real_agent_tools_scope_and_no_scratch(app, prepared, case_run, sandbox):
run, case_id, query_id, commit, source = case_run
@ -113,6 +126,9 @@ async def test_single_run_real_agent_tools_scope_and_no_scratch(app, prepared, c
assert "Evidence-supported report" in response.answer
assert response.metadata["candidate_commit"] == commit and response.metadata["usage"]["total_tokens"] == 25
assert response.metadata["session_id"]
results = [b for m in model.calls[1]["messages"] for b in m.content if b.type == "tool_result"]
evidence_result = json.loads(results[0].output[0].text)
assert evidence_result["locations"]["saved_query"] == "/queries/0"
assert "value = 1" in str(model.calls[2]["messages"])
assert "value = 999" not in str(model.calls[2]["messages"])
schemas = {item["function"]["name"]: item["function"]["parameters"] for item in model.calls[0]["tools"]}
@ -121,6 +137,13 @@ async def test_single_run_real_agent_tools_scope_and_no_scratch(app, prepared, c
assert "validation_path" not in schemas["home_inspect_validation"]["properties"]
assert "revision" not in schemas["home_read_code_file"]["properties"]
assert "target_validation_path" not in schemas["home_compare_memory"]["properties"]
assert "base_ref" in schemas["home_compare_memory"]["required"]
assert schemas["home_compare_memory"]["properties"]["base_ref"]["type"] == "string"
descriptions = {item["function"]["name"]: item["function"]["description"] for item in model.calls[0]["tools"]}
assert "repository-relative" in descriptions["home_read_code_file"]
assert "python_execute" in descriptions["home_read_code_file"]
assert "pipefail" in descriptions["AutoBash"]
assert "do not pass args" in descriptions["python_execute"]
policy = sandbox[0].policy
assert policy["writable_paths"] == [] and policy["cwd"] == str(run / f"cases/{case_id}")
assert str(prepared / "code") in policy["read_only_paths"] and str(source) in policy["read_only_paths"]
@ -140,7 +163,9 @@ async def test_scope_override_rejected_by_real_job_tool(app, prepared, case_run,
run, case_id, _query_id, _commit, _source = case_run
wrapper = app.context.components["agent_wrapper"]["home_case_study"]
scope = case_study.prepare_scope(prepared, case_study.CaseStudyInput(validation_path=str(run), case_id=case_id))
job_parameters = json.dumps(app.context.jobs["home_inspect_memory_history"].parameters, sort_keys=True)
toolkit = case_study.create_toolkit(wrapper, scope, RecordingSandbox())
assert json.dumps(app.context.jobs["home_inspect_memory_history"].parameters, sort_keys=True) == job_parameters
schemas = await toolkit.get_tool_schemas()
assert len(schemas) == 9
tool = wrapper._make_tool(
@ -181,7 +206,7 @@ async def test_invalid_scope_and_sandbox_failure_precede_llm(app, prepared, case
def unavailable(**kwargs):
raise RuntimeError("No sandbox backend")
monkeypatch.setattr(case_study, "create_file_sandbox_backend", unavailable)
monkeypatch.setattr(case_study, "create_case_study_sandbox", unavailable)
response = await app.run_job("home_case_study", validation_path=str(run), case_id=case_id)
assert not response.success and "No sandbox backend" in str(response.answer) and not model.calls
@ -207,9 +232,13 @@ async def test_os_sandbox_read_scope_and_no_writes(tmp_path):
case.mkdir()
code.mkdir()
(code / "source.py").write_text("visible code")
git(code, "init")
git(code, "add", "source.py")
git(code, "-c", "user.name=Test", "-c", "user.email=test@example.com", "commit", "-m", "initial")
outside = tmp_path / "secret.txt"
outside.write_text("outside scope")
backend = create_file_sandbox_backend(
(case / "manifest.json").write_text("{}")
backend = create_case_study_sandbox(
cwd=str(case),
read_only_paths=[str(case), str(code)],
writable_paths=[],
@ -221,9 +250,24 @@ async def test_os_sandbox_read_scope_and_no_writes(tmp_path):
pytest.skip(f"OS sandbox unavailable on this host: {error}")
if not probe.ok():
pytest.skip("OS sandbox cannot start on this host: " + probe.stderr.decode(errors="replace"))
ready = await case_study.probe_sandbox(
backend,
{
"case_path": str(case),
"source_case": str(code / "source.py"),
"validation_path": str(case),
"code_repository": str(code),
"candidate_commit": git(code, "rev-parse", "HEAD"),
},
)
assert ready.ok(), ready.stderr.decode(errors="replace")
tool = case_study.python_tool(backend, str(case))
result = await tool._func(code=f"from pathlib import Path; print(Path({str(code / 'source.py')!r}).read_text())")
assert result["returncode"] == 0 and "visible code" in result["stdout"]
result = await tool._func(
code=f"import subprocess; subprocess.run(['git', '-C', {str(code)!r}, 'show', 'HEAD:source.py'], check=True)",
)
assert result["returncode"] == 0 and "visible code" in result["stdout"], result
for script in (
f"open({str(outside)!r}).read()",
"open('written.txt', 'w').write('denied')",
@ -237,6 +281,17 @@ async def test_os_sandbox_read_scope_and_no_writes(tmp_path):
chunks = [chunk async for chunk in bash.call("echo changed > written.txt")]
assert "Command failed" in str(chunks)
assert not (case / "written.txt").exists() and (code / "source.py").read_text() == "visible code"
chunks = [chunk async for chunk in bash.call(f"git -C '{code}' show HEAD:source.py")]
assert "visible code" in str(chunks), chunks
git_config = (code / ".git/config").read_bytes()
result = await tool._func(
code=(
f"import subprocess; subprocess.run(['git', '-C', {str(code)!r},"
" 'config', 'test.blocked', 'yes'], check=True)"
),
)
assert result["returncode"] != 0
assert (code / ".git/config").read_bytes() == git_config
@pytest.mark.asyncio
@ -265,3 +320,154 @@ async def test_python_code_file_timeout_and_cancellation(tmp_path):
):
with pytest.raises(ValueError):
await tool._func(**arguments)
@pytest.fixture
def batch_run(case_run):
run, case_id, query_id, commit, source = case_run
case = json.loads(source.read_text())
case["case_id"] = "case2"
write_json(source.parent / "batch-second.json", case)
shutil.copytree(run / f"cases/{case_id}", run / "cases/case2")
manifest = json.loads((run / "manifest.json").read_text())
manifest["case_ids"].append("case2")
manifest["query_ids"]["case2"] = [query_id]
write_json(run / "manifest.json", manifest)
return run, case_id, query_id, commit
class BatchModel(RecordingModel):
"""Read each fresh agent's case-local evidence before returning its report."""
def __init__(self, query_id):
super().__init__()
self.query_id = query_id
async def __call__(self, **kwargs):
self.calls.append(kwargs)
results = [b for m in kwargs["messages"] for b in m.content if b.type == "tool_result"]
if not results:
content = [
ToolCallBlock(
id="batch-trace",
name="home_trace_query_evidence",
input=json.dumps({"query_id": self.query_id}),
),
]
else:
content = [TextBlock(text="case-local report")]
return ChatResponse(content=content, is_last=True, usage=ChatUsage(input_tokens=2, output_tokens=3, time=0))
@pytest.mark.asyncio
async def test_batch_shares_focus_but_isolates_agents_tools_and_sandboxes(app, prepared, batch_run, sandbox):
run, first, query_id, commit = batch_run
model = BatchModel(query_id)
app.context.components["agent_wrapper"]["home_case_study"].as_llm = SimpleNamespace(model=model)
before = snapshot(prepared)
response = await app.run_job(
"home_case_study",
evaluation_path=str(run),
case_ids=["case2", first],
focus="Shared hypothesis",
)
assert response.success, response.answer
results = response.answer["results"]
assert [item["case_id"] for item in results] == ["case2", first]
assert all(item["report"] == "case-local report" for item in results)
assert len({item["session_id"] for item in results}) == 2
assert response.metadata["evaluation_path"] == str(run) and response.metadata["candidate_commit"] == commit
assert response.metadata["usage"]["total_tokens"] == 20
assert len(model.calls) == 4
for i, case_id in enumerate(["case2", first]):
calls = model.calls[i * 2 : i * 2 + 2]
assert "Shared hypothesis" in str(calls[0]["messages"])
outputs = [b for m in calls[1]["messages"] for b in m.content if b.type == "tool_result"]
evidence = json.loads(outputs[0].output[0].text)
assert evidence["paths"]["query_result"] == str(run / f"cases/{case_id}/queries/result.json")
assert sandbox[i].policy["cwd"] == str(run / f"cases/{case_id}")
other = first if case_id == "case2" else "case2"
assert str(run / f"cases/{other}") not in sandbox[i].policy["read_only_paths"]
assert snapshot(prepared) == before
@pytest.mark.asyncio
async def test_batch_continues_after_case_failure_and_keeps_usage_unknown(app, batch_run, sandbox, monkeypatch):
run, first, query_id, _commit = batch_run
model = BatchModel(query_id)
app.context.components["agent_wrapper"]["home_case_study"].as_llm = SimpleNamespace(model=model)
original = RecordingSandbox.exec_shell
async def fail_second_case(self, command, *, cwd, timeout):
if str(cwd).endswith("/case2"):
return ExecResult(exit_code=1, stdout=b"", stderr=b"sandbox denied")
return await original(self, command, cwd=cwd, timeout=timeout)
monkeypatch.setattr(RecordingSandbox, "exec_shell", fail_second_case)
response = await app.run_job("home_case_study", validation_path=str(run), case_ids=["case2", first])
assert not response.success
failed, succeeded = response.answer["results"]
assert failed["case_id"] == "case2" and "sandbox unavailable" in failed["error"]
assert succeeded["case_id"] == first and succeeded["success"] and succeeded["session_id"]
assert response.metadata["usage"] is None
@pytest.mark.asyncio
async def test_entire_batch_is_validated_before_spending_model_tokens(app, batch_run, sandbox):
run, first, _query_id, _commit = batch_run
response = await app.run_job("home_case_study", evaluation_path=str(run), case_ids=[first, "missing"])
assert not response.success and "not selected" in response.answer and not sandbox
@pytest.mark.asyncio
async def test_batch_cancellation_propagates_without_starting_next_case(app, batch_run, sandbox, monkeypatch):
run, first, _query_id, _commit = batch_run
async def cancel(*args, **kwargs):
raise asyncio.CancelledError
monkeypatch.setattr(app.context.components["agent_wrapper"]["home_case_study"], "reply", cancel)
with pytest.raises(asyncio.CancelledError):
await app.run_job("home_case_study", evaluation_path=str(run), case_ids=[first, "case2"])
assert len(sandbox) == 1
@pytest.mark.parametrize(
"updates",
[
{"case_ids": []},
{"case_ids": ["case1", "case1"]},
{"case_ids": ["../escape"]},
{"case_ids": [str(i) for i in range(9)]},
{"case_ids": "case1"},
{"case_ids": [1]},
{"validation_path": "another"},
{"case_id": "case1"},
],
)
def test_batch_request_rejects_invalid_or_ambiguous_inputs(updates):
with pytest.raises(ValidationError):
case_study.CaseStudyBatchInput.model_validate(
{"evaluation_path": "evaluations/a/b/1", "case_ids": ["case1"], **updates},
)
@pytest.mark.parametrize("path_key", ["evaluation_path", "validation_path"])
@pytest.mark.parametrize("case_key", ["case_ids", "case_id"])
@pytest.mark.asyncio
async def test_new_and_legacy_selectors_match_job_schema(app, batch_run, sandbox, path_key, case_key):
run, first, query_id, _commit = batch_run
app.context.components["agent_wrapper"]["home_case_study"].as_llm = SimpleNamespace(model=BatchModel(query_id))
arguments = {path_key: str(run), case_key: [first] if case_key == "case_ids" else first}
schema = app.context.jobs["home_case_study"].parameters
Draft202012Validator.check_schema(schema)
validator = Draft202012Validator(schema)
assert validator.is_valid(arguments)
assert not validator.is_valid({**arguments, "evaluation_path": str(run), "validation_path": str(run)})
assert not validator.is_valid({**arguments, "case_ids": [first], "case_id": first})
response = await app.run_job("home_case_study", **arguments)
assert response.success, response.answer
if case_key == "case_id":
assert response.answer == "case-local report" and response.metadata["case_id"] == first
else:
assert response.answer["results"][0]["case_id"] == first

View file

@ -0,0 +1,96 @@
"""Home runtime adaptation preserves the core sandbox's path and process contracts."""
# pylint: disable=missing-function-docstring,protected-access
import asyncio
import os
import sys
from types import SimpleNamespace
from unittest.mock import AsyncMock
import pytest
from reme_home.agent import sandbox
from reme.utils.file_sandbox import PathBubbleBackend, PathSeatbeltBackend
def test_macos_grants_only_selected_developer_runtime(tmp_path, monkeypatch):
developer = tmp_path / "developer tools"
(developer / "usr/bin").mkdir(parents=True)
(developer / "usr/bin/git").touch()
workspace = tmp_path / "case"
workspace.mkdir()
monkeypatch.setattr(sys, "platform", "darwin")
monkeypatch.setattr(
sandbox.subprocess,
"run",
lambda *args, **kwargs: SimpleNamespace(returncode=0, stdout=str(developer)),
)
monkeypatch.setattr(sandbox, "create_file_sandbox_backend", PathSeatbeltBackend)
roots = [str(workspace)]
adapted = sandbox.create_case_study_sandbox(
cwd=workspace,
read_only_paths=roots,
writable_paths=[],
memory_limit_mib=0,
)
assert roots == [str(workspace)]
profile = adapted.backend._profile(str(workspace / "temporary"))
assert f'(allow file-read* (subpath "{developer}"))' in profile
assert f'(allow file-write* (subpath "{developer}"))' not in profile
assert f'(allow file-read* (subpath "{tmp_path}"))' not in profile
assert adapted.environment["DEVELOPER_DIR"] == str(developer)
paths = adapted.environment["PATH"].split(os.pathsep)
assert paths[0] == str(sandbox.Path(sys.prefix) / "bin")
assert paths.index(str(developer / "usr/bin")) < paths.index("/usr/bin")
@pytest.mark.parametrize("result", [(1, ""), (0, ""), (0, "/nonexistent/reme-toolchain")])
def test_failed_macos_lookup_adds_no_grant(tmp_path, monkeypatch, result):
monkeypatch.setattr(sys, "platform", "darwin")
monkeypatch.setattr(
sandbox.subprocess,
"run",
lambda *args, **kwargs: SimpleNamespace(returncode=result[0], stdout=result[1]),
)
def factory(**kwargs):
return SimpleNamespace(policy=kwargs)
monkeypatch.setattr(sandbox, "create_file_sandbox_backend", factory)
adapted = sandbox.create_case_study_sandbox(cwd=tmp_path, read_only_paths=[str(tmp_path)], writable_paths=[])
assert adapted.backend.policy["read_only_paths"] == [str(tmp_path)]
assert "DEVELOPER_DIR" not in adapted.environment
@pytest.mark.asyncio
async def test_linux_environment_is_inside_bubblewrap(tmp_path, monkeypatch):
monkeypatch.setattr(sys, "platform", "linux")
monkeypatch.setattr(sandbox, "create_file_sandbox_backend", PathBubbleBackend)
adapted = sandbox.create_case_study_sandbox(
cwd=tmp_path,
read_only_paths=[str(tmp_path)],
writable_paths=[],
memory_limit_mib=0,
)
backend = adapted.backend
backend.exec_shell = AsyncMock(return_value="result")
command = [sys.executable, "-I", "-B", "-c", "print('literal $HOME; `pwd`')"]
assert await adapted.exec_shell(command, cwd=str(tmp_path), timeout=7) == "result"
args, kwargs = backend.exec_shell.call_args
assert kwargs == {"cwd": str(tmp_path), "timeout": 7}
assert args[0][-len(command) :] == command
argv = backend._bwrap_argv(args[0], str(tmp_path))
assert "--clearenv" in argv and "--unshare-all" in argv
assert argv.index("--") < argv.index("/usr/bin/env")
assert "GIT_CONFIG_NOSYSTEM=1" in argv and "GIT_CONFIG_GLOBAL=/dev/null" in argv
assert adapted.environment["PATH"].split(":")[:2] == list(
dict.fromkeys([str(sandbox.Path(sys.prefix) / "bin"), str(sandbox.Path(sys.base_prefix) / "bin")]),
)
assert backend._writable_paths == []
assert await adapted.getcwd() == str(tmp_path)
backend.exec_shell.side_effect = asyncio.CancelledError
with pytest.raises(asyncio.CancelledError):
await adapted.exec_shell(command)
with pytest.raises(ValueError, match="empty"):
await adapted.exec_shell([])

View file

@ -63,6 +63,7 @@ def test_compact_annotation_root_checkpoint_and_read_only(query_evidence):
assert result["query"] == {
"question": "What is the fact?",
"expected": "old fact",
"expected_kind": "reference_answer",
"actual": "new fact",
"score": 0,
"error": None,
@ -71,10 +72,17 @@ def test_compact_annotation_root_checkpoint_and_read_only(query_evidence):
assert result["evidence_source"] == "independent_annotation"
assert result["evidence_is_certain"] is False and result["memory_status"] == "available"
assert result["sessions"] == [
{"id": "session1", "checkpoint": first, "changed_files": ["memory/note.md", "removed.txt"]},
{
"id": "session1",
"source_pointer": "/sessions/0",
"checkpoint": first,
"changed_files": ["memory/note.md", "removed.txt"],
},
]
assert result["paths"]["source_case"] == str(source)
assert result["paths"]["memory_repository"] == str(repo)
assert result["locations"] == {"source_query": "/queries/0", "saved_query": "/queries/0"}
assert "criteria" not in result
assert "source transcript" not in json.dumps(result) and "judge_result" not in json.dumps(result)
assert snapshot(home) == before
@ -94,7 +102,12 @@ def test_inference_missing_checkpoint_and_debug_dataset(query_evidence):
assert result["paths"]["source_case"] == str(debug_source)
assert result["sessions"][0]["checkpoint"] == second
assert result["sessions"][0]["changed_files"] == ["added.bin", "memory/note.md", "removed.txt"]
assert result["sessions"][1] == {"id": "not-constructed", "checkpoint": None, "changed_files": []}
assert result["sessions"][1] == {
"id": "not-constructed",
"source_pointer": "/sessions/2",
"checkpoint": None,
"changed_files": [],
}
assert trace(home, run)["evidence_source"] == "unavailable"
with pytest.raises(ValueError, match="not present"):
trace(home, run, evidence_session_ids=["unknown"])
@ -122,6 +135,7 @@ def test_missing_history_and_results(query_evidence, state):
(run / "cases/case1/queries/result.json").unlink()
result = trace(home, run)
assert result["memory_status"] == state and result["sessions"][0]["checkpoint"] is None
assert result["locations"]["saved_query"] is None and "criteria" not in result
assert result["query"]["result_missing"] and result["query"]["score"] is None
assert result["query"]["actual"] is None and result["query"]["expected"] == "old fact"
@ -219,3 +233,59 @@ async def test_registered_job_and_restricted_scope(app, prepared, query_evidence
)
assert not response.success and "unrestricted" in str(response.answer)
assert snapshot(prepared) == before
@pytest.mark.parametrize("count", [2, 1, 25])
def test_rubric_judgments_and_independent_source_pointers(query_evidence, count):
home, run, _repo, _first, _second, source, case = query_evidence
case["queries"].insert(0, {"query_id": "other"})
case["sessions"].insert(0, {"session_id": "other-session"})
case["queries"][1]["metadata"]["rubric"] = ["required" + str(i) for i in range(25 if count == 25 else 2)]
write_json(source, case)
result_path = run / "cases/case1/queries/result.json"
judged = [{"score": 0.5, "reason": "r" * 1000} for _ in range(count)]
write_json(
result_path,
{
"queries": [
{
"query_id": "q1",
"judge_result": {
"metadata": {"llm_judge_responses": judged},
},
},
],
},
)
result = trace(home, run)
assert result["locations"] == {
"source_query": "/queries/1",
"saved_query": "/queries/0",
"rubric": "/queries/1/metadata/rubric",
}
assert result["sessions"][0]["source_pointer"] == "/sessions/1"
if count == 1:
assert "criteria" not in result # Do not silently pair incomplete judgments.
else:
assert len(result["criteria"]) == min(count, 20)
item = result["criteria"][0]
assert item["requirement"] == "required0" and len(item["reason"]) < 700
value = json.loads(result_path.read_text())
for segment in item["pointer"].strip("/").split("/"):
value = value[int(segment)] if isinstance(value, list) else value[segment]
assert value == judged[0]
if count == 25:
assert result["omitted_criterion_count"] == 5
def test_beam_rubric_fallback_uses_explicit_dataset_contract(query_evidence):
home, run, _repo, _first, _second, source, case = query_evidence
case["queries"][0]["golden_answer"] = ["required fact"]
write_json(source, case)
manifest = json.loads((run / "manifest.json").read_text())
manifest["dataset"] = "beam"
write_json(run / "manifest.json", manifest)
result = trace(home, run)
assert result["query"]["expected_kind"] == "rubric"
assert result["locations"]["rubric"] == "/queries/0/golden_answer"
assert "criteria" not in result # A rubric is not a saved judgment.