ReMe/reme/steps/evolve/auto_memory_cc.py
jinliyl e7d44f6f3b
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382)
* feat(config): add environment variable configuration for agent subprocesses

- Add environment field to ApplicationConfig to store variables for agent subprocesses
- Remove dynamic loading of .env files in agent wrappers
- Introduce subprocess_environment property in base agent wrapper
- Pass application-level environment variables to Claude Code and Codex agents
- Load environment variables once at startup and pass to ReMe application
- Remove dependency on load_env utility in agent wrapper implementations
- Update tests to use configured environment instead of dynamic loading
- Remove unused environment loading utilities and related test cases

* refactor(mcp): remove channel notification system and related components

- Removed channel notification step implementation
- Removed claim channel step implementation
- Removed ChannelSink class from MCP service
- Removed channel-related documentation from AGENTS.md
- Removed channel instruction text from MCP service
- Removed all channel-related tests
- Updated application context metadata comment to remove channel sink reference
- Removed channel module initialization and imports

* feat(service): add job whitelisting capability to BaseService

- Add optional jobs parameter to BaseService.__init__ to configure job whitelist
- Store jobs as set in self.jobs attribute for efficient lookup operations
- Modify add_jobs method to filter jobs based on whitelist configuration
- Update documentation in both English and Chinese to describe new feature
- Add comprehensive unit tests for job whitelisting behavior
- Implement flowchart update showing new filtering logic
- Preserve existing enable_serve flag behavior alongside new whitelisting

* refactor(service): enhance service job validation and MCP tool injection

- Add strict validation for service jobs whitelist with detailed error messages
- Implement injected job arguments support for MCP services with conflict detection
- Add tool error handling for unsuccessful responses in MCP services
- Remove duplicate job names in Codex agent wrapper using dict.fromkeys
- Update MCP server argument format from single JSON array to repeated --job flags
- Add comprehensive test coverage for job injection and error handling scenarios
- Update documentation to reflect service job validation and MCP features
- Ensure application cleanup occurs even when service lifespan encounters errors

* feat(agent): update skill handling to preserve existing Claude skills

- Change skills parameter processing to use 'all' instead of filtered list
- Add logic to select project skills without restricting Claude's existing skills
- Update variable naming from 'skills' to 'selected_skills' for clarity
- Modify application context metadata documentation to clarify in-memory state usage
- Add test case to verify configured skills are added without filtering existing skills
- Update internal skill directory handling to use renamed variable consistently

* refactor(agent): restructure agent wrapper components and session storage

- Move CcFileSessionStore to separate module for better organization
- Add SDK package version logging in base agent wrapper
- Update Claude Code agent to use new session store structure with project keys
- Refactor Claude Code agent wrapper to use proper type hints and SDK integration
- Add support for server tool use events in Claude Code message processing
- Improve error handling and resource cleanup in streaming operations
- Update Codex agent wrapper with proper type annotations and configuration
- Remove deprecated system prompt mode handling from Claude Code wrapper
- Fix session path construction for Claude Code transcript storage
- Update dependency injection and configuration handling patterns

* fix(cc_agent_wrapper): resolve Claude Code SDK integration issues

- Added dataclass import and created _BlockState for content block metadata tracking
- Implemented proper MCP server name constant and tool context ID validation
- Fixed tool_context_id injection to prevent duplicate assignment errors
- Resolved skills parameter handling in build_options method
- Enhanced job tools integration with MCP servers mapping validation
- Replaced deprecated block_ids/block_types/tool_call_names with block_states dict
- Updated message_delta to emit USAGE chunks instead of REPLY_END
- Fixed stream result handling to ensure proper REPLY_END emission
- Improved error handling for session mirror failures and rate limits
- Added proper cleanup for expected trailing errors in streams
- Refactored Codex agent wrapper initialization and configuration management
- Removed obsolete system_prompt_mode from default config
- Enhanced test coverage for new block state and error handling features
- Fixed async generator handling with aclosing context manager
- Improved chunk type mapping for Claude Code SDK events

* refactor(tests): remove demo config tests from config parser test suite

- Removed test_demo_config_registers_llm_jobs function and its assertions
- Eliminated verification of LLM demo job configurations
- Removed checks for agent wrapper component settings
- Deleted assertions for model configurations and parameters
- Cleaned up deprecated test cases related to demo config parsing

* refactor(evolve): simplify Claude Code session store path structure

- Removed redundant project key subdirectory from session link generation
- Updated CcFileSessionStore initialization to use direct session directory path
- Maintained existing session layout compatibility for backward compatibility
- Added unit tests to verify session persistence behavior with existing transcripts
- Ensured UUID-based session files remain accessible at expected locations
- Preserved existing session directory structure without additional nesting

* refactor(agent): defer optional Codex SDK imports until first use

- Moved openai-codex imports inside functions to avoid mandatory dependencies
- Added TYPE_CHECKING guard for development time type checking only
- Implemented lazy loading mechanism with _get_async_codex_class function
- Updated AsyncCodex initialization to occur on demand rather than at module level
- Maintained backward compatibility while improving import performance
- Added test case to verify package import works without optional Codex SDK
- Updated agentscope dependency to version 2.0.4.post1 in pyproject.toml

* test(embedded): add compatibility tests for in-process ReMe embedding

- Add test suite for QwenPaw-style embedded configurations
- Verify optional defaults remain preserved in embedded configs
- Ensure in-process application API stays compatible
- Test model injection and lifecycle management compatibility
- Remove obsolete hermes agent plugin tests
- Update CLI import test to cover multiple optional SDKs
- Block claude_agent_sdk and openai_codex during import testing
2026-07-20 23:52:14 +08:00

184 lines
7.8 KiB
Python

"""auto_memory_cc — record a Claude Code session, resolved from its session_id.
The ReMe plugin's Stop hook hands the server only a ``session_id`` (never the
messages), and it fires on *every* stop. Unlike :class:`AutoMemoryStep` — whose
callers have no session management, so it re-serializes ``Msg`` history into its
own dialog store — Claude Code already manages the session as a transcript on
disk. So this step manages everything through the :class:`CcFileSessionStore`
abstraction and avoids the ``Msg`` round-trip entirely:
1. **load** the outer Claude Code session's transcript entries (Claude Code side).
2. **save** the *raw* entries into ReMe's own CC SessionStore — ``append`` dedups
by record ``uuid``, so this both copies the conversation into ReMe and tells
us the **increment** since the last stop.
3. render only that increment into plain ``{role, name, content}`` messages and
defer to :class:`AutoMemoryStep` for the daily-note write/merge.
Both the read (Claude Code side) and the copy (ReMe side) use the same
file-backed SessionStore with the SDK's project/session key layout.
"""
from __future__ import annotations
import json
import os
import re
from pathlib import Path
from typing import Any
from .auto_memory import AutoMemoryStep
from ...components import R
from ...components.agent_wrapper import CcFileSessionStore
# Whole-message-drop when a user turn is only Claude-Code-injected boilerplate.
_INJECTED_TAGS = (
"<local-command-caveat>",
"<local-command-stdout>",
"<local-command-stderr>",
"<command-name>",
"<command-message>",
"<command-args>",
"<system-reminder>",
"<bash-input>",
"<bash-stdout>",
"<bash-stderr>",
)
_TOOL_EXCERPT = 200
@R.register("auto_memory_cc_step")
class AutoMemoryCCStep(AutoMemoryStep):
"""Resolve a Claude Code session_id to its *new* turns, then reuse AutoMemoryStep."""
# Sub-directory under the session dir holding ReMe's copy of CC transcripts.
_CC_STORE_SUBDIR = "claude_code"
_REME_PROJECT_KEY = "claude_code"
async def execute(self):
assert self.context is not None
session_id: str = self.context.get("session_id", "")
cc_entries = await self._load_cc_session(session_id)
new_entries = await self._save_cc_session(session_id, cc_entries)
messages = self._entries_to_messages(new_entries)
self.logger.info(
f"[{self.name}] resolved Claude Code session session_id={session_id!r} "
f"transcript={len(cc_entries)} new_entries={len(new_entries)} messages={len(messages)}",
)
self.context["messages"] = messages
await super().execute()
# Claude Code owns the session (transcript + the CC SessionStore copy made in
# _save_cc_session); AutoMemoryStep's Msg-history dialog store does not apply.
async def _save_session_messages(self, session_id: str, messages) -> None: # noqa: D401
return
def _session_link(self, session_id: str) -> str:
return f"[[{self._session_dir()}/{self._CC_STORE_SUBDIR}/{session_id}.jsonl]]"
# ----- session: Claude Code side <-> ReMe CC SessionStore ----------------
async def _load_cc_session(self, session_id: str) -> list[dict]:
"""Load the outer Claude Code transcript entries (Claude Code side)."""
transcript_dir = self._resolve_transcript_dir(session_id)
if transcript_dir is None:
return []
store = CcFileSessionStore(self._projects_dir())
return await store.load({"project_key": transcript_dir.name, "session_id": session_id}) or []
async def _save_cc_session(self, session_id: str, cc_entries: list[dict]) -> list[dict]:
"""Copy raw CC entries into ReMe's CC SessionStore; return the increment.
Only identity-bearing entries are copied: every conversational entry
(user / assistant / attachment) carries a ``uuid``, while uuid-less rows
are CC control bookkeeping (queue operations, last-prompt) that would
otherwise re-copy on every stop. Dedup against the already-stored uuids
yields exactly the turns added since the previous stop.
"""
if not session_id:
return []
store = self._reme_cc_store()
key = {"project_key": self._REME_PROJECT_KEY, "session_id": session_id}
cc_entries = [e for e in cc_entries if isinstance(e, dict) and e.get("uuid")]
existing = await store.load(key) or []
seen = {e.get("uuid") for e in existing if isinstance(e, dict) and e.get("uuid")}
increment = [e for e in cc_entries if e.get("uuid") not in seen]
await store.append(key, increment)
return increment
def _reme_cc_store(self) -> CcFileSessionStore:
root = self.file_store.workspace_path / self._session_dir()
return CcFileSessionStore(root)
@staticmethod
def _projects_dir() -> Path:
base = Path(os.environ.get("CLAUDE_CONFIG_DIR") or "~/.claude").expanduser()
return base if base.name == "projects" else base / "projects"
def _resolve_transcript_dir(self, session_id: str) -> Path | None:
"""Return the project directory holding ``<session_id>.jsonl`` (newest)."""
projects = self._projects_dir()
if not session_id or not projects.is_dir():
return None
matches = list(projects.glob(f"*/{session_id}.jsonl"))
if not matches:
return None
matches.sort(key=lambda p: p.stat().st_mtime, reverse=True)
return matches[0].parent
# ----- rendering: raw CC entries -> plain agent messages -----------------
@classmethod
def _entries_to_messages(cls, entries: list[dict]) -> list[dict[str, str]]:
messages: list[dict[str, str]] = []
for record in entries:
if not isinstance(record, dict) or record.get("type") not in ("user", "assistant"):
continue
message = record.get("message") or {}
role = message.get("role")
if role not in ("user", "assistant"):
continue
text = cls._render_content(message.get("content", ""))
if not text or cls._is_injected_only(text):
continue
messages.append({"role": role, "name": role, "content": text})
return messages
@classmethod
def _render_content(cls, content: Any) -> str:
if isinstance(content, str):
return content.strip()
if not isinstance(content, list):
return ""
parts: list[str] = []
for block in content:
if not isinstance(block, dict):
continue
btype = block.get("type")
if btype == "text":
if t := (block.get("text") or "").strip():
parts.append(t)
elif btype == "tool_use":
name = block.get("name", "?")
try:
inp = json.dumps(block.get("input"), ensure_ascii=False)[:_TOOL_EXCERPT]
except (TypeError, ValueError):
inp = str(block.get("input"))[:_TOOL_EXCERPT]
parts.append(f"[tool {name}({inp})]")
elif btype == "tool_result":
inner = block.get("content")
excerpt = cls._render_content(inner) if isinstance(inner, list) else str(inner or "")
excerpt = excerpt.strip()
if len(excerpt) > _TOOL_EXCERPT:
excerpt = excerpt[:_TOOL_EXCERPT] + "..."
parts.append(f"[tool_result {excerpt}]")
# thinking blocks are private reasoning -> dropped
return "\n".join(p for p in parts if p).strip()
@staticmethod
def _is_injected_only(text: str) -> bool:
stripped = text.strip()
if not stripped.startswith(_INJECTED_TAGS):
return False
remaining = re.sub(r"<([a-z-]+)>.*?</\1>", "", stripped, flags=re.DOTALL)
return len(remaining.strip()) < 16