diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..7353cc4 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,47 @@ +# AGENTS + +## Codex Desktop Sidecar Evolution + +Use this workflow when the user is coding in Codex Desktop with their normal subscription login and wants OpenSpace to do post-task skill capture through the isolated `openspace_evolution` sidecar. + +Rules: +- Keep the main coding workflow unchanged. +- Do not switch the main Codex Desktop session to a provider-backed model. +- Do not modify code as part of sidecar evolution unless the user separately asks for code changes. +- Do not let OpenSpace take over the main task. +- Use the sidecar only for post-task skill capture. +- Prefer at most 1 new high-reuse skill per invocation unless the user explicitly asks for more. + +When the user asks for sidecar self-evolution, call: +- `openspace_evolution.evolve_from_context` + +Trigger phrases: +- `sidecar 自进化一下` +- `做一次 sidecar 自进化` +- `对当前这轮工作做一次 sidecar 自进化` +- `用 sidecar 沉淀一个 skill` +- `基于当前改动做一次 sidecar skill capture` +- `不要改代码，做一次 sidecar 自进化` + +If the user uses one of these phrases, default to this workflow automatically unless they explicitly ask for a different behavior. + +Derive the tool inputs from: +- the current conversation +- the current `git diff` +- the key changed files + +Behavior: +- Infer a concise `task` +- Infer a concise but specific `summary` +- Pass the most relevant changed files in `file_paths` +- Use `max_skills = 1` by default +- After the tool returns, report: + - the skill name + - the skill path + - why the skill is worth keeping + +Recommended user-facing invocation: + +```text +对当前这轮工作做一次 sidecar 自进化。不要改代码，不要接管任务。请调用 openspace_evolution.evolve_from_context，基于当前对话、git diff 和关键改动，自动提炼 task/summary，最多生成 1 个高复用 skill，并告诉我 skill 名称、路径、为什么值得保留。 +``` diff --git a/docs/codex-desktop-sidecar-evolution.md b/docs/codex-desktop-sidecar-evolution.md new file mode 100644 index 0000000..5f617f2 --- /dev/null +++ b/docs/codex-desktop-sidecar-evolution.md @@ -0,0 +1,221 @@ +# Codex Desktop Sidecar Evolution Integration + +## Goal + +This integration keeps the normal Codex Desktop workflow unchanged while moving OpenSpace skill capture and self-evolution onto a separate provider-backed sidecar path. + +The target user experience is: + +- Main coding still happens in Codex Desktop with the user's normal subscription login. +- OpenSpace does not take over the main task loop. +- Sidecar evolution can be invoked explicitly after a task and spend provider API tokens instead of the main Codex Desktop session. + +## Short Answer: Was this mainly an API-level dual routing change? + +No. + +The final effect does **not** come from a simple in-process "dual route" inside one OpenSpace runtime where: + +- coding uses Codex Desktop subscription auth, and +- evolution uses a provider API + +That approach is not viable because Codex Desktop subscription login is not exposed to the Python process as a reusable API credential. + +Instead, the final implementation uses **process-level split routing**: + +- the main coding session remains in Codex Desktop +- self-evolution runs through an isolated OpenSpace sidecar with its own provider-backed MCP server + +API compatibility work was still necessary, but it is only one part of the solution. + +## What Was Implemented + +### 1. OpenAI-compatible provider bridge for OpenSpace + +File: + +- `openspace/llm/client.py` + +Why it was needed: + +- The third-party relay worked with Codex's `/responses` path. +- OpenSpace uses LiteLLM / OpenAI-style chat completion flows. +- The relay was not reliable enough for OpenSpace's normal streaming path. + +What changed: + +- Added an OpenAI-compatible streaming fallback that talks directly to `/chat/completions`. +- Reconstructed streamed text, reasoning content, and tool calls into the shape OpenSpace already expects. +- Enabled this path through `OPENSPACE_LLM_OPENAI_STREAM_COMPAT`. + +Effect: + +- OpenSpace can use the relay provider for evolution workloads. + +### 2. Evolution-only MCP sidecar + +File: + +- `openspace/evolution_mcp_server.py` + +Why it was needed: + +- The user wanted OpenSpace to handle only post-task evolution and skill capture. +- The main coding loop had to stay outside OpenSpace. + +What changed: + +- Added a separate MCP server exposing only `evolve_from_context`. +- This server builds context from the current workspace, conversation summary, and git diff. +- It captures reusable skills without becoming the main task executor. + +Effect: + +- OpenSpace now has a narrow sidecar role instead of replacing the host coding agent. + +### 3. Sidecar-capable skill engine without full task recording + +File: + +- `openspace/tool_layer.py` + +Why it was needed: + +- The original skill evolution path assumed a fuller OpenSpace task/recording pipeline. +- The new sidecar path needed to create skills without enabling the normal OpenSpace recording flow. + +What changed: + +- Added `enable_skill_engine_without_recording`. +- Kept execution analysis tied to recording. +- Allowed skill evolution and skill store initialization in sidecar mode without enabling full task recordings. + +Effect: + +- Sidecar capture can work independently without creating full OpenSpace task sessions. + +### 4. Isolated Desktop launcher overlay + +File: + +- `scripts/codex-desktop-evolution` + +Why it was needed: + +- The main Codex Desktop session had to keep the user's normal login and defaults. +- The sidecar config had to be added without polluting `~/.codex`. + +What changed: + +- Created an overlay `CODEX_HOME` at `~/.codex-openspace-desktop`. +- Copied the primary Desktop auth and config base into the overlay. +- Added only one extra MCP server: `openspace_evolution`. +- Scrubbed `OPENSPACE_*` variables before launching the main Codex process. +- Avoided inheriting arbitrary shell state or leaking sidecar credentials into the main coding session. + +Effect: + +- Main Codex Desktop remains normal. +- The sidecar is available only in the isolated overlay profile. + +### 5. Agent instruction trigger for sidecar capture + +File: + +- `AGENTS.md` + +Why it was needed: + +- The sidecar should be callable naturally from the Desktop workflow. +- The user should not need to restate the full MCP call every time. + +What changed: + +- Added a repo-level instruction that maps phrases like `sidecar 自进化一下` to `openspace_evolution.evolve_from_context`. +- Limited the default behavior to: + - no code changes + - no main-task takeover + - at most one high-reuse skill by default + +Effect: + +- The sidecar behaves like a narrow post-task tool integrated into the normal Desktop workflow. + +## Other Supporting Changes + +### MCP stdout flush fix + +File: + +- `openspace/mcp_server.py` + +What changed: + +- Avoided a final stdout flush crash when the MCP stdio transport closes before Python exit. + +### Missing dependency for MCP backend + +Files: + +- `pyproject.toml` +- `requirements.txt` + +What changed: + +- Added `websockets>=15.0.0` +- Added `openspace-evolution-mcp` as a console entrypoint + +### Frontend dependency refresh + +File: + +- `frontend/package-lock.json` + +What changed: + +- Updated `lodash-es` +- Updated `vite` + +This was a maintenance fix and is not part of the sidecar architecture itself. + +## Architecture Summary + +The final architecture is: + +1. Codex Desktop remains the main coding agent. +2. Codex Desktop keeps using the user's normal subscription login. +3. A separate overlay profile adds an `openspace_evolution` MCP server. +4. That MCP server runs OpenSpace with provider-backed credentials. +5. OpenSpace uses the provider only for post-task evolution and skill capture. + +This means the practical "dual routing" exists at the workflow/process boundary, not as a single shared in-process auth router. + +## Usage + +Launch the Desktop profile that includes the sidecar: + +```bash +cd /Users/admin/PycharmProjects/openspace +./scripts/codex-desktop-evolution app +``` + +Inside that Desktop session, trigger sidecar capture with: + +```text +sidecar 自进化一下 +``` + +or the longer explicit form: + +```text +对当前这轮工作做一次 sidecar 自进化。不要改代码，不要接管任务。请调用 openspace_evolution.evolve_from_context，基于当前对话、git diff 和关键改动，自动提炼 task/summary，最多生成 1 个高复用 skill，并告诉我 skill 名称、路径、为什么值得保留。 +``` + +## Result + +The implemented effect is: + +- normal Codex Desktop coding stays unchanged +- OpenSpace self-evolution is available on demand +- provider token spend is isolated to the sidecar path +- the sidecar does not silently take over the main workflow diff --git a/frontend/package-lock.json b/frontend/package-lock.json index b4630ac..e2c75a4 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -2512,9 +2512,9 @@ "license": "MIT" }, "node_modules/lodash-es": { - "version": "4.17.23", - "resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.17.23.tgz", - "integrity": "sha512-kVI48u3PZr38HdYz98UmfPnXl2DXrpdctLrFLCd3kOx1xUkOmpFPx7gCWWM5MPkL/fD8zb+Ph0QzjGFs4+hHWg==", + "version": "4.18.1", + "resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.18.1.tgz", + "integrity": "sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==", "license": "MIT" }, "node_modules/loose-envify": { @@ -3455,9 +3455,9 @@ "license": "MIT" }, "node_modules/vite": { - "version": "6.4.1", - "resolved": "https://registry.npmjs.org/vite/-/vite-6.4.1.tgz", - "integrity": "sha512-+Oxm7q9hDoLMyJOYfUYBuHQo+dkAloi33apOPP56pzj+vsdJDzr+j1NISE5pyaAuKL4A3UD34qd0lx5+kfKp2g==", + "version": "6.4.2", + "resolved": "https://registry.npmjs.org/vite/-/vite-6.4.2.tgz", + "integrity": "sha512-2N/55r4JDJ4gdrCvGgINMy+HH3iRpNIz8K6SFwVsA+JbQScLiC+clmAxBgwiSPgcG9U15QmvqCGWzMbqda5zGQ==", "dev": true, "license": "MIT", "dependencies": { diff --git a/openspace/evolution_mcp_server.py b/openspace/evolution_mcp_server.py new file mode 100644 index 0000000..80cef38 --- /dev/null +++ b/openspace/evolution_mcp_server.py @@ -0,0 +1,606 @@ +"""OpenSpace evolution-only MCP server. + +This sidecar is designed for host-agent workflows where the main coding is +handled elsewhere (for example Codex Desktop with subscription auth), while +OpenSpace is only used to capture reusable skills via a separate provider. +""" + +from __future__ import annotations + +import asyncio +import inspect +import json +import logging +import os +import subprocess +import sys +import uuid +from datetime import datetime +from pathlib import Path +from typing import Any, Dict, Iterable, List, Optional + + +class _MCPSafeStdout: + """Stdout wrapper: binary (.buffer) -> real stdout, text (.write) -> stderr.""" + + def __init__(self, real_stdout, stderr): + self._real = real_stdout + self._stderr = stderr + + @property + def buffer(self): + return self._real.buffer + + def fileno(self): + return self._real.fileno() + + def write(self, s): + return self._stderr.write(s) + + def writelines(self, lines): + return self._stderr.writelines(lines) + + def flush(self): + self._stderr.flush() + try: + self._real.flush() + except ValueError: + pass + + def isatty(self): + return self._stderr.isatty() + + @property + def encoding(self): + return self._stderr.encoding + + @property + def errors(self): + return self._stderr.errors + + @property + def closed(self): + return self._stderr.closed + + def readable(self): + return False + + def writable(self): + return True + + def seekable(self): + return False + + def __getattr__(self, name): + return getattr(self._stderr, name) + + +_LOG_DIR = Path(__file__).resolve().parent.parent / "logs" +_LOG_DIR.mkdir(parents=True, exist_ok=True) + +_real_stdout = sys.stdout +if os.name == "nt": + _stderr_file = open( + _LOG_DIR / "evolution_mcp_stderr.log", "a", encoding="utf-8", buffering=1 + ) + sys.stderr = _stderr_file + +sys.stdout = _MCPSafeStdout(_real_stdout, sys.stderr) + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s - %(name)s - %(levelname)s - %(message)s", + handlers=[logging.FileHandler(_LOG_DIR / "evolution_mcp_server.log")], +) +logger = logging.getLogger("openspace.evolution_mcp_server") + +from mcp.server.fastmcp import FastMCP + +_fastmcp_kwargs: dict = {} +try: + if "description" in inspect.signature(FastMCP.__init__).parameters: + _fastmcp_kwargs["description"] = ( + "OpenSpace evolution sidecar: capture reusable skills from host-agent work." + ) +except (TypeError, ValueError): + pass + +mcp = FastMCP("OpenSpace Evolution", **_fastmcp_kwargs) + +_openspace_instance = None +_openspace_lock = asyncio.Lock() +_UPLOAD_META_FILENAME = ".upload_meta.json" + + +def _json_ok(data: Any) -> str: + return json.dumps(data, ensure_ascii=False, indent=2) + + +def _json_error(error: Any, **extra) -> str: + return json.dumps({"error": str(error), **extra}, ensure_ascii=False) + + +async def _get_openspace(): + global _openspace_instance + if _openspace_instance is not None and _openspace_instance.is_initialized(): + return _openspace_instance + + async with _openspace_lock: + if _openspace_instance is not None and _openspace_instance.is_initialized(): + return _openspace_instance + + logger.info("Initializing OpenSpace evolution engine ...") + from openspace.host_detection import ( + build_grounding_config_path, + build_llm_kwargs, + load_runtime_env, + ) + from openspace.tool_layer import OpenSpace, OpenSpaceConfig + + load_runtime_env() + + env_model = os.environ.get("OPENSPACE_MODEL", "") + workspace = os.environ.get("OPENSPACE_WORKSPACE") + enable_rec = os.environ.get("OPENSPACE_ENABLE_RECORDING", "false").lower() in ( + "true", + "1", + "yes", + ) + backend_scope_raw = os.environ.get("OPENSPACE_BACKEND_SCOPE", "shell,system") + backend_scope = [ + b.strip() for b in backend_scope_raw.split(",") if b.strip() + ] or None + + config_path = build_grounding_config_path() + model, llm_kwargs = build_llm_kwargs(env_model) + + config = OpenSpaceConfig( + llm_model=model, + llm_kwargs=llm_kwargs, + workspace_dir=workspace, + grounding_max_iterations=1, + enable_recording=enable_rec, + enable_skill_engine_without_recording=True, + recording_backends=["shell"] if enable_rec else None, + backend_scope=backend_scope, + grounding_config_path=config_path, + ) + + _openspace_instance = OpenSpace(config=config) + await _openspace_instance.initialize() + logger.info("OpenSpace evolution engine ready (model=%s).", model) + return _openspace_instance + + +def _write_upload_meta(skill_dir: Path, info: Dict[str, Any]) -> None: + meta = { + "origin": info.get("origin", "captured"), + "parent_skill_ids": info.get("parent_skill_ids", []), + "change_summary": info.get("change_summary", ""), + "created_by": info.get("created_by", "openspace"), + "tags": info.get("tags", []), + } + (skill_dir / _UPLOAD_META_FILENAME).write_text( + json.dumps(meta, ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + + +def _extract_json_object(text: str) -> Dict[str, Any]: + raw = (text or "").strip() + if raw.startswith("```"): + raw = raw.strip("`") + parts = raw.split("\n", 1) + raw = parts[1] if len(parts) == 2 else raw + if raw.endswith("```"): + raw = raw[:-3].rstrip() + + try: + data = json.loads(raw) + if isinstance(data, dict): + return data + except json.JSONDecodeError: + pass + + start = raw.find("{") + end = raw.rfind("}") + if start >= 0 and end > start: + data = json.loads(raw[start : end + 1]) + if isinstance(data, dict): + return data + + raise ValueError("LLM did not return a valid JSON object") + + +def _run_git(args: List[str], cwd: Path) -> str: + try: + completed = subprocess.run( + ["git", *args], + cwd=str(cwd), + text=True, + capture_output=True, + check=False, + ) + except Exception as exc: + logger.debug("git %s failed: %s", " ".join(args), exc) + return "" + + if completed.returncode != 0: + return "" + return completed.stdout.strip() + + +def _truncate(text: str, limit: int) -> str: + if len(text) <= limit: + return text + return text[: limit - 32].rstrip() + "\n...[truncated]..." + + +def _normalize_file_paths( + workspace: Path, + file_paths: Optional[Iterable[str]], +) -> List[Path]: + normalized: List[Path] = [] + for raw in file_paths or []: + if not raw: + continue + path = Path(raw) + if not path.is_absolute(): + path = workspace / path + normalized.append(path.resolve()) + return normalized + + +def _build_repo_context( + workspace: Path, + file_paths: List[Path], +) -> str: + sections: List[str] = [] + + if (workspace / ".git").exists(): + status = _run_git(["status", "--short"], workspace) + if status: + sections.append("## Git status\n" + _truncate(status, 4_000)) + + diff_stat = _run_git(["diff", "--stat"], workspace) + if diff_stat: + sections.append("## Git diff stat\n" + _truncate(diff_stat, 4_000)) + + staged_stat = _run_git(["diff", "--cached", "--stat"], workspace) + if staged_stat: + sections.append("## Git staged diff stat\n" + _truncate(staged_stat, 4_000)) + + if file_paths: + rel_paths = [] + for path in file_paths: + try: + rel_paths.append(str(path.relative_to(workspace))) + except ValueError: + rel_paths.append(str(path)) + scoped_diff = _run_git( + ["diff", "--unified=1", "--", *rel_paths], + workspace, + ) + if scoped_diff: + sections.append("## Focused diff\n" + _truncate(scoped_diff, 12_000)) + + if file_paths: + lines = ["## Mentioned files"] + for path in file_paths: + lines.append(f"- {path}") + sections.append("\n".join(lines)) + + return "\n\n".join(sections) if sections else "(no repository context available)" + + +def _existing_skill_names(registry) -> List[str]: + names = [] + for meta in registry.list_skills(): + names.append(meta.name) + return sorted(set(names)) + + +def _build_planning_prompt( + *, + task: str, + summary: str, + workspace: Path, + repo_context: str, + existing_skills: List[str], + max_skills: int, +) -> str: + skill_list = "\n".join(f"- {name}" for name in existing_skills[:200]) or "(none)" + return f"""You are deciding which reusable OpenSpace skills should be captured from a completed coding task. + +The main coding work was already completed by a host agent. Your job is ONLY to identify reusable patterns worth turning into new skills. + +Task: +{task} + +Execution summary: +{summary} + +Workspace: +{workspace} + +Repository context: +{repo_context} + +Existing local skill names: +{skill_list} + +Return exactly one JSON object with this shape: +{{ + "suggestions": [ + {{ + "category": "workflow", + "direction": "1-2 sentences describing the reusable pattern to capture." + }} + ] +}} + +Rules: +- Suggest at most {max_skills} skills. +- Only suggest skills that are reusable across future tasks. +- Categories must be one of: "tool_guide", "workflow", "reference". +- Do not suggest trivial one-step actions. +- Do not restate repo-specific one-off details as a reusable skill. +- Avoid duplicating an existing skill unless the new capability is clearly distinct. +- If nothing is worth capturing, return {{"suggestions": []}}. +""" + + +async def _plan_suggestions( + *, + openspace, + task: str, + summary: str, + workspace: Path, + repo_context: str, + max_skills: int, +) -> List[Dict[str, str]]: + registry = openspace._skill_registry + if not registry: + return [] + + logger.info( + "Planning evolution captures for task=%r (max_skills=%d)", + task[:120], + max_skills, + ) + prompt = _build_planning_prompt( + task=task, + summary=summary, + workspace=workspace, + repo_context=repo_context, + existing_skills=_existing_skill_names(registry), + max_skills=max_skills, + ) + + response = await openspace._llm_client.complete( + messages=prompt, + execute_tools=False, + model=openspace.config.llm_model, + ) + data = _extract_json_object(response["message"]["content"]) + raw_suggestions = data.get("suggestions", []) + if not isinstance(raw_suggestions, list): + raise ValueError("suggestions must be a list") + + deduped: List[Dict[str, str]] = [] + seen: set[tuple[str, str]] = set() + for item in raw_suggestions: + if not isinstance(item, dict): + continue + category = str(item.get("category", "")).strip() + direction = str(item.get("direction", "")).strip() + if category not in {"tool_guide", "workflow", "reference"} or not direction: + continue + key = (category, direction.lower()) + if key in seen: + continue + seen.add(key) + deduped.append({"category": category, "direction": direction}) + if len(deduped) >= max_skills: + break + logger.info("Planned %d capture suggestion(s)", len(deduped)) + return deduped + + +async def _prepend_output_dir(openspace, output_dir: Path) -> None: + output_dir.mkdir(parents=True, exist_ok=True) + registry = openspace._skill_registry + if not registry: + return + + if output_dir not in registry._skill_dirs: + registry._skill_dirs.insert(0, output_dir) + + skill_store = openspace._skill_store + metas = registry.discover_from_dirs([output_dir]) + if metas and skill_store: + await skill_store.sync_from_registry(metas) + + +async def _register_extra_skill_dirs(openspace, dirs: List[Path]) -> None: + registry = openspace._skill_registry + skill_store = openspace._skill_store + if not registry: + return + + metas = registry.discover_from_dirs(dirs) + if metas and skill_store: + await skill_store.sync_from_registry(metas) + + +@mcp.tool() +async def evolve_from_context( + task: str, + summary: str, + workspace_dir: str | None = None, + file_paths: list[str] | None = None, + max_skills: int = 3, + skill_dirs: list[str] | None = None, + output_dir: str | None = None, +) -> str: + """Capture reusable skills from a completed host-agent task. + + Use this when the main task was already handled by another agent + (for example Codex Desktop) and OpenSpace should only spend provider + tokens on post-task skill capture. + + Args: + task: Short description of the completed task. + summary: What changed, what was learned, and what seems reusable. + workspace_dir: Repository/workspace path. Defaults to OPENSPACE_WORKSPACE. + file_paths: Optional files worth emphasizing when planning captures. + max_skills: Maximum number of new skills to capture. + skill_dirs: Optional additional skill directories to register first. + output_dir: Override directory for new skills. Defaults to the first + OPENSPACE_HOST_SKILL_DIRS entry. + """ + try: + if not task.strip(): + return _json_error("task is required", status="error") + if not summary.strip(): + return _json_error("summary is required", status="error") + + openspace = await _get_openspace() + if not openspace._skill_evolver or not openspace._skill_registry: + return _json_error("Skill evolution is not enabled", status="error") + + workspace = Path(workspace_dir or openspace.config.workspace_dir or os.getcwd()).resolve() + normalized_paths = _normalize_file_paths(workspace, file_paths) + + if skill_dirs: + extra_dirs = [Path(p).expanduser().resolve() for p in skill_dirs if p] + if extra_dirs: + await _register_extra_skill_dirs(openspace, extra_dirs) + + if output_dir: + await _prepend_output_dir(openspace, Path(output_dir).expanduser().resolve()) + + repo_context = _build_repo_context(workspace, normalized_paths) + suggestions = await _plan_suggestions( + openspace=openspace, + task=task, + summary=summary, + workspace=workspace, + repo_context=repo_context, + max_skills=max(0, min(max_skills, 8)), + ) + + if not suggestions: + return _json_ok( + { + "status": "success", + "task": task, + "workspace_dir": str(workspace), + "suggestion_count": 0, + "created_skills": [], + "message": "No reusable skill captures were suggested.", + } + ) + + from openspace.skill_engine import EvolutionContext, EvolutionTrigger + from openspace.skill_engine.types import ( + EvolutionSuggestion, + EvolutionType, + ExecutionAnalysis, + SkillCategory, + ) + + evolver = openspace._skill_evolver + task_id = f"sidecar_{uuid.uuid4().hex[:12]}" + now = datetime.now() + analysis = ExecutionAnalysis( + task_id=task_id, + timestamp=now, + task_completed=True, + execution_note=_truncate(summary, 1_500), + analyzed_by=openspace.config.llm_model, + analyzed_at=now, + ) + + created_skills: List[Dict[str, Any]] = [] + skipped: List[Dict[str, str]] = [] + for suggestion in suggestions: + logger.info( + "Capturing skill (%s): %s", + suggestion["category"], + suggestion["direction"][:180], + ) + ctx = EvolutionContext( + trigger=EvolutionTrigger.ANALYSIS, + suggestion=EvolutionSuggestion( + evolution_type=EvolutionType.CAPTURED, + target_skill_ids=[], + category=SkillCategory(suggestion["category"]), + direction=suggestion["direction"], + ), + source_task_id=task_id, + recent_analyses=[analysis], + available_tools=[], + ) + new_record = await evolver.evolve(ctx) + if not new_record: + logger.info("Capture skipped by evolver") + skipped.append(suggestion) + continue + + skill_dir = Path(new_record.path).parent if new_record.path else None + if skill_dir: + _write_upload_meta( + skill_dir, + { + "origin": new_record.lineage.origin.value, + "parent_skill_ids": new_record.lineage.parent_skill_ids, + "change_summary": new_record.lineage.change_summary, + "created_by": new_record.lineage.created_by or "openspace", + "tags": new_record.tags, + }, + ) + + created_skills.append( + { + "name": new_record.name, + "skill_id": new_record.skill_id, + "skill_dir": str(skill_dir) if skill_dir else "", + "path": new_record.path, + "category": suggestion["category"], + "direction": suggestion["direction"], + "upload_ready": bool(skill_dir), + } + ) + + return _json_ok( + { + "status": "success", + "task": task, + "workspace_dir": str(workspace), + "suggestion_count": len(suggestions), + "created_skills": created_skills, + "skipped_suggestions": skipped, + } + ) + except Exception as e: + logger.error("evolve_from_context failed: %s", e, exc_info=True) + return _json_error(e, status="error") + + +def run_mcp_server() -> None: + import argparse + + parser = argparse.ArgumentParser(description="OpenSpace Evolution MCP Server") + parser.add_argument("--transport", choices=["stdio", "sse"], default="stdio") + parser.add_argument("--port", type=int, default=8080) + args = parser.parse_args() + + if args.transport == "sse": + mcp.run(transport="sse", sse_params={"port": args.port}) + else: + mcp.run(transport="stdio") + + +if __name__ == "__main__": + run_mcp_server() diff --git a/openspace/llm/client.py b/openspace/llm/client.py index 611cf87..5b969db 100644 --- a/openspace/llm/client.py +++ b/openspace/llm/client.py @@ -1,8 +1,12 @@ import litellm import json import asyncio +import os import time +from types import SimpleNamespace from typing import List, Sequence, Union, Dict, Optional + +import httpx from openai.types.chat import ChatCompletionToolParam from openspace.grounding.core.types import ToolSchema, ToolResult, ToolStatus @@ -20,6 +24,161 @@ litellm.suppress_debug_info = True logger = Logger.get_logger(__name__) +def _is_truthy(value: object) -> bool: + return str(value).strip().lower() in {"1", "true", "yes", "on"} + + +def _should_use_openai_stream_compat(litellm_kwargs: Optional[Dict] = None) -> bool: + if litellm_kwargs and _is_truthy(litellm_kwargs.get("openai_stream_compat")): + return True + return _is_truthy(os.environ.get("OPENSPACE_LLM_OPENAI_STREAM_COMPAT", "")) + + +def _build_stream_response( + *, + content: str, + reasoning_content: Optional[str], + tool_calls: List[Dict[str, object]], +): + response_tool_calls = [] + for tool_call in tool_calls: + response_tool_calls.append( + SimpleNamespace( + id=tool_call.get("id"), + type=tool_call.get("type", "function"), + function=SimpleNamespace( + name=tool_call.get("function", {}).get("name", ""), + arguments=tool_call.get("function", {}).get("arguments", ""), + ), + ) + ) + + message = SimpleNamespace( + content=content, + reasoning_content=reasoning_content, + tool_calls=response_tool_calls or None, + ) + return SimpleNamespace(choices=[SimpleNamespace(message=message)]) + + +async def _openai_compat_stream_completion( + *, + model: str, + messages: List[Dict], + timeout: float, + litellm_kwargs: Optional[Dict] = None, + tools: Optional[List[ChatCompletionToolParam]] = None, + tool_choice: Optional[str] = None, + reasoning_effort: Optional[str] = None, +): + kwargs = dict(litellm_kwargs or {}) + api_key = kwargs.pop("api_key", None) + api_base = kwargs.pop("api_base", None) + extra_headers = kwargs.pop("extra_headers", None) or {} + kwargs.pop("openai_stream_compat", None) + + if not api_key or not api_base: + raise ValueError( + "OpenAI stream compatibility mode requires api_key and api_base." + ) + + payload = { + "model": model, + "messages": messages, + "stream": True, + } + if tools: + payload["tools"] = tools + if tool_choice is not None: + payload["tool_choice"] = tool_choice + if reasoning_effort: + payload["reasoning_effort"] = reasoning_effort + + for key in ( + "temperature", + "top_p", + "presence_penalty", + "frequency_penalty", + "max_tokens", + "max_completion_tokens", + "parallel_tool_calls", + "response_format", + ): + if key in kwargs and kwargs[key] is not None: + payload[key] = kwargs[key] + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + headers.update(extra_headers) + + text_parts: List[str] = [] + reasoning_parts: List[str] = [] + streamed_tool_calls: Dict[int, Dict[str, object]] = {} + + async with httpx.AsyncClient(timeout=timeout) as client: + async with client.stream( + "POST", + f"{api_base.rstrip('/')}/chat/completions", + headers=headers, + json=payload, + ) as response: + if response.status_code >= 400: + body = await response.aread() + raise RuntimeError( + f"OpenAI-compat stream request failed: {response.status_code} " + f"{body.decode(errors='replace')}" + ) + + async for line in response.aiter_lines(): + if not line or not line.startswith("data: "): + continue + data = line[6:].strip() + if not data or data == "[DONE]": + break + + chunk = json.loads(data) + choices = chunk.get("choices") or [] + if not choices: + continue + + delta = choices[0].get("delta") or {} + content_delta = delta.get("content") + if content_delta: + text_parts.append(content_delta) + + reasoning_delta = delta.get("reasoning_content") + if reasoning_delta: + reasoning_parts.append(reasoning_delta) + + for tool_delta in delta.get("tool_calls") or []: + idx = tool_delta.get("index", 0) + state = streamed_tool_calls.setdefault( + idx, + { + "id": None, + "type": "function", + "function": {"name": "", "arguments": ""}, + }, + ) + if tool_delta.get("id"): + state["id"] = tool_delta["id"] + if tool_delta.get("type"): + state["type"] = tool_delta["type"] + fn = tool_delta.get("function") or {} + if fn.get("name"): + state["function"]["name"] += fn["name"] + if fn.get("arguments"): + state["function"]["arguments"] += fn["arguments"] + + return _build_stream_response( + content="".join(text_parts), + reasoning_content="".join(reasoning_parts) or None, + tool_calls=[streamed_tool_calls[i] for i in sorted(streamed_tool_calls)], + ) + + def _sanitize_schema(params: Dict) -> Dict: """Sanitize tool parameter schema to comply with Claude API requirements. @@ -263,15 +422,26 @@ Content: Concise summary:""" _extra = litellm_kwargs or {} - response = await asyncio.wait_for( - litellm.acompletion( - model=model, - messages=[{"role": "user", "content": prompt}], - timeout=timeout, - **_extra, - ), - timeout=timeout + 5 - ) + if _should_use_openai_stream_compat(_extra): + response = await asyncio.wait_for( + _openai_compat_stream_completion( + model=model, + messages=[{"role": "user", "content": prompt}], + timeout=timeout, + litellm_kwargs=_extra, + ), + timeout=timeout + 5, + ) + else: + response = await asyncio.wait_for( + litellm.acompletion( + model=model, + messages=[{"role": "user", "content": prompt}], + timeout=timeout, + **_extra, + ), + timeout=timeout + 5 + ) summary = response.choices[0].message.content.strip() result = f"[SUMMARY of {len(content):,} chars]\n{summary}" @@ -552,10 +722,24 @@ class LLMClient: for attempt in range(self.max_retries): try: # Add timeout to the completion call - response = await asyncio.wait_for( - litellm.acompletion(**completion_kwargs), - timeout=self.timeout - ) + if _should_use_openai_stream_compat(completion_kwargs): + response = await asyncio.wait_for( + _openai_compat_stream_completion( + model=completion_kwargs["model"], + messages=completion_kwargs["messages"], + timeout=self.timeout, + litellm_kwargs=completion_kwargs, + tools=completion_kwargs.get("tools"), + tool_choice=completion_kwargs.get("tool_choice"), + reasoning_effort=completion_kwargs.get("reasoning_effort"), + ), + timeout=self.timeout, + ) + else: + response = await asyncio.wait_for( + litellm.acompletion(**completion_kwargs), + timeout=self.timeout + ) return response except asyncio.TimeoutError: self._logger.error( diff --git a/openspace/mcp_server.py b/openspace/mcp_server.py index ba4ca31..ce49c3b 100644 --- a/openspace/mcp_server.py +++ b/openspace/mcp_server.py @@ -48,7 +48,11 @@ class _MCPSafeStdout: def flush(self): self._stderr.flush() - self._real.flush() + try: + self._real.flush() + except ValueError: + # The MCP stdio transport may close stdout before Python's final flush. + pass def isatty(self): return self._stderr.isatty() diff --git a/openspace/tool_layer.py b/openspace/tool_layer.py index c2eaca9..84a6457 100644 --- a/openspace/tool_layer.py +++ b/openspace/tool_layer.py @@ -58,6 +58,7 @@ class OpenSpaceConfig: enable_screenshot: bool = False enable_video: bool = False enable_conversation_log: bool = True # Save LLM conversations to conversations.jsonl + enable_skill_engine_without_recording: bool = False # Allow sidecar evolution without full task recording # Skill Evolution evolution_max_concurrent: int = 3 # Max parallel evolutions per trigger @@ -242,8 +243,13 @@ class OpenSpace: logger.info(f"✓ Skills: {len(skills)} discovered") self._grounding_agent.set_skill_registry(self._skill_registry) - # Initialize ExecutionAnalyzer (requires recording + skills) - if self.config.enable_recording and self._skill_registry: + # Initialize the skill engine whenever skills are available. + # Execution analysis still requires recordings, but skill capture + # can run without them (for host-agent sidecar workflows). + if self._skill_registry and ( + self.config.enable_recording + or self.config.enable_skill_engine_without_recording + ): try: skill_store = SkillStore() self._skill_store = skill_store # Expose for MCP server reuse @@ -257,19 +263,6 @@ class OpenSpace: # Bridge: pass quality_manager so analysis can feed back # LLM-identified tool issues to the tool quality system. - quality_mgr = ( - self._grounding_client.quality_manager - if self._grounding_client else None - ) - self._execution_analyzer = ExecutionAnalyzer( - store=skill_store, - llm_client=self._llm_client, - model=self.config.execution_analyzer_model, - skill_registry=self._skill_registry, - quality_manager=quality_mgr, - ) - logger.info("✓ Execution analysis enabled") - # Share store with GroundingAgent so retrieve_skill # can access quality metrics for LLM selection. self._grounding_agent._skill_store = skill_store @@ -287,8 +280,22 @@ class OpenSpace: f"✓ Skill evolution enabled " f"(concurrent={self.config.evolution_max_concurrent})" ) + + if self.config.enable_recording: + quality_mgr = ( + self._grounding_client.quality_manager + if self._grounding_client else None + ) + self._execution_analyzer = ExecutionAnalyzer( + store=skill_store, + llm_client=self._llm_client, + model=self.config.execution_analyzer_model, + skill_registry=self._skill_registry, + quality_manager=quality_mgr, + ) + logger.info("✓ Execution analysis enabled") except Exception as e: - logger.warning(f"Execution analyzer init failed (non-fatal): {e}") + logger.warning(f"Skill engine init failed (non-fatal): {e}") self._initialized = True logger.info("="*60) diff --git a/pyproject.toml b/pyproject.toml index 06c199c..a391cca 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,6 +19,7 @@ dependencies = [ "openai>=1.0.0", "jsonschema>=4.25.0", "mcp>=1.0.0", + "websockets>=15.0.0", "anthropic>=0.71.0", "pillow>=12.0.0", "numpy>=1.24.0", @@ -69,6 +70,7 @@ Repository = "https://github.com/HKUDS/OpenSpace" openspace = "openspace.__main__:run_main" openspace-server = "openspace.local_server.main:main" openspace-mcp = "openspace.mcp_server:run_mcp_server" +openspace-evolution-mcp = "openspace.evolution_mcp_server:run_mcp_server" openspace-download-skill = "openspace.cloud.cli.download_skill:main" openspace-upload-skill = "openspace.cloud.cli.upload_skill:main" openspace-dashboard = "openspace.dashboard_server:main" diff --git a/requirements.txt b/requirements.txt index 6371308..8683105 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,6 +4,7 @@ python-dotenv>=1.0.0 openai>=1.0.0 jsonschema>=4.25.0 mcp>=1.0.0 +websockets>=15.0.0 anthropic>=0.71.0 pillow>=12.0.0 numpy>=1.24.0 diff --git a/scripts/codex-desktop-evolution b/scripts/codex-desktop-evolution new file mode 100755 index 0000000..5a24b8a --- /dev/null +++ b/scripts/codex-desktop-evolution @@ -0,0 +1,170 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "$SCRIPT_DIR/.." && pwd)" +ENV_FILE="$REPO_ROOT/openspace/.env" +PRIMARY_CODEX_HOME="${PRIMARY_CODEX_HOME:-$HOME/.codex}" +PROFILE_HOME="${OPENSPACE_CODEX_HOME:-$HOME/.codex-openspace-desktop}" +PROJECT_NAME="$(basename "$REPO_ROOT")" +PROJECT_SKILL_DIR="$PROFILE_HOME/projects/$PROJECT_NAME/skills" +REPO_PYTHON="$REPO_ROOT/.venv/bin/python" + +if [[ ! -f "$PRIMARY_CODEX_HOME/config.toml" ]]; then + echo "Missing $PRIMARY_CODEX_HOME/config.toml" >&2 + exit 1 +fi + +if [[ ! -f "$PRIMARY_CODEX_HOME/auth.json" ]]; then + echo "Missing $PRIMARY_CODEX_HOME/auth.json" >&2 + exit 1 +fi + +read_env_value() { + local key="$1" + if [[ ! -f "$ENV_FILE" ]]; then + return 0 + fi + + python3 - <<'PY' "$ENV_FILE" "$key" +from pathlib import Path +import sys + +env_path = Path(sys.argv[1]) +target = sys.argv[2] + +for line in env_path.read_text(encoding="utf-8").splitlines(): + stripped = line.strip() + if not stripped or stripped.startswith("#") or "=" not in stripped: + continue + key, value = stripped.split("=", 1) + if key.strip() != target: + continue + value = value.strip().strip('"').strip("'") + print(value, end="") + break +PY +} + +OPENSPACE_MODEL="${OPENSPACE_MODEL:-$(read_env_value OPENSPACE_MODEL)}" +OPENSPACE_MODEL="${OPENSPACE_MODEL:-gpt-5.4}" +OPENSPACE_LLM_API_KEY="${OPENSPACE_LLM_API_KEY:-$(read_env_value OPENSPACE_LLM_API_KEY)}" +OPENSPACE_LLM_API_BASE="${OPENSPACE_LLM_API_BASE:-$(read_env_value OPENSPACE_LLM_API_BASE)}" +OPENSPACE_LLM_API_BASE="${OPENSPACE_LLM_API_BASE:-https://codexapi.space/v1}" +OPENSPACE_LLM_OPENAI_STREAM_COMPAT="${OPENSPACE_LLM_OPENAI_STREAM_COMPAT:-$(read_env_value OPENSPACE_LLM_OPENAI_STREAM_COMPAT)}" +OPENSPACE_LLM_OPENAI_STREAM_COMPAT="${OPENSPACE_LLM_OPENAI_STREAM_COMPAT:-true}" + +sync_profile_dir() { + local name="$1" + local src="$PRIMARY_CODEX_HOME/$name" + local dst="$PROFILE_HOME/$name" + + if [[ ! -d "$src" ]]; then + return + fi + + mkdir -p "$dst" + rsync -a --delete "$src/" "$dst/" +} + +bootstrap_profile_home() { + mkdir -p "$PROFILE_HOME" + mkdir -p "$PROJECT_SKILL_DIR" + mkdir -p "$PROFILE_HOME/skills" + + sync_profile_dir "plugins" + sync_profile_dir "skills" + sync_profile_dir "pua" + + cp "$PRIMARY_CODEX_HOME/auth.json" "$PROFILE_HOME/auth.json" + + python3 - <<'PY' "$PRIMARY_CODEX_HOME/config.toml" "$PROFILE_HOME/config.toml" "$REPO_ROOT" "$REPO_PYTHON" "$PROJECT_SKILL_DIR" "$PROFILE_HOME/skills" "$OPENSPACE_MODEL" "$OPENSPACE_LLM_API_KEY" "$OPENSPACE_LLM_API_BASE" "$OPENSPACE_LLM_OPENAI_STREAM_COMPAT" +from pathlib import Path +import os +import sys + +src = Path(sys.argv[1]) +dst = Path(sys.argv[2]) +repo_root = sys.argv[3] +python_cmd = sys.argv[4] +project_skill_dir = sys.argv[5] +profile_skill_dir = sys.argv[6] +model = sys.argv[7] +api_key = sys.argv[8] +api_base = sys.argv[9] +stream_compat = sys.argv[10] + +def strip_tables(text: str, table_names: set[str]) -> str: + kept = [] + skipping = False + for line in text.splitlines(): + stripped = line.strip() + header = stripped.split("#", 1)[0].rstrip() + if header.startswith("[") and header.endswith("]"): + skipping = header in table_names or header.startswith("[mcp_servers.openspace_evolution.") + if skipping: + continue + if skipping: + continue + kept.append(line) + return "\n".join(kept).rstrip() + "\n" + +base = strip_tables( + src.read_text(encoding="utf-8"), + { + "[mcp_servers.openspace_evolution]", + "[mcp_servers.openspace_evolution.env]", + }, +) + +project_marker = f'[projects."{repo_root}"]' +if project_marker not in base: + base += f'\n{project_marker}\ntrust_level = "trusted"\n' + +if api_key and Path(python_cmd).is_file() and os.access(python_cmd, os.X_OK): + base += f''' +[mcp_servers.openspace_evolution] +command = "{python_cmd}" +args = ["-m", "openspace.evolution_mcp_server", "--transport", "stdio"] + +[mcp_servers.openspace_evolution.env] +OPENSPACE_WORKSPACE = "{repo_root}" +OPENSPACE_HOST_SKILL_DIRS = "{project_skill_dir},{profile_skill_dir}" +OPENSPACE_MODEL = "{model}" +OPENSPACE_LLM_API_KEY = "{api_key}" +OPENSPACE_LLM_API_BASE = "{api_base}" +OPENSPACE_LLM_OPENAI_STREAM_COMPAT = "{stream_compat}" +OPENSPACE_ENABLE_RECORDING = "false" +OPENSPACE_BACKEND_SCOPE = "shell,system" +''' + +dst.write_text(base, encoding="utf-8") +PY +} + +bootstrap_profile_home + +clear_openspace_env() { + local var + for var in ${!OPENSPACE_@}; do + unset "$var" + done +} + +if [[ -z "$OPENSPACE_LLM_API_KEY" ]]; then + echo "Warning: OpenSpace evolution sidecar is disabled because no provider key was found in $ENV_FILE or the current environment." >&2 +elif [[ ! -x "$REPO_PYTHON" ]]; then + echo "Warning: OpenSpace evolution sidecar is disabled because $REPO_PYTHON is missing." >&2 +fi + +clear_openspace_env + +if [[ "${1:-}" == "app" ]]; then + shift + if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then + exec env CODEX_HOME="$PROFILE_HOME" codex app "$@" + fi + exec env CODEX_HOME="$PROFILE_HOME" codex app "$@" "$REPO_ROOT" +fi + +exec env CODEX_HOME="$PROFILE_HOME" codex -C "$REPO_ROOT" "$@" diff --git a/scripts/codex-openspace b/scripts/codex-openspace new file mode 100755 index 0000000..0650383 --- /dev/null +++ b/scripts/codex-openspace @@ -0,0 +1,136 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd -- "$SCRIPT_DIR/.." && pwd)" +ENV_FILE="$REPO_ROOT/openspace/.env" +PRIMARY_CODEX_HOME="${PRIMARY_CODEX_HOME:-$HOME/.codex}" +PROFILE_HOME="${CODEX_HOME:-$HOME/.codex-openspace}" + +if [[ ! -f "$ENV_FILE" ]]; then + echo "Missing $ENV_FILE" >&2 + exit 1 +fi + +set -a +source "$ENV_FILE" +set +a + +if [[ -z "${OPENSPACE_LLM_API_KEY:-}" ]]; then + echo "OPENSPACE_LLM_API_KEY is missing in $ENV_FILE" >&2 + exit 1 +fi + +OPENSPACE_MODEL="${OPENSPACE_MODEL:-gpt-5.4}" +OPENSPACE_LLM_API_BASE="${OPENSPACE_LLM_API_BASE:-https://codexapi.space/v1}" +OPENSPACE_LLM_OPENAI_STREAM_COMPAT="${OPENSPACE_LLM_OPENAI_STREAM_COMPAT:-true}" + +sync_profile_dir() { + local name="$1" + local src="$PRIMARY_CODEX_HOME/$name" + local dst="$PROFILE_HOME/$name" + + if [[ ! -d "$src" ]]; then + return + fi + + if [[ -L "$dst" ]]; then + rm -f "$dst" + fi + + mkdir -p "$dst" + rsync -a --delete "$src/" "$dst/" +} + +bootstrap_profile_home() { + mkdir -p "$PROFILE_HOME" + + sync_profile_dir "plugins" + sync_profile_dir "skills" + sync_profile_dir "pua" + + cat > "$PROFILE_HOME/auth.json" < "$PROFILE_HOME/config.toml" <&2 + exit 1 +fi + +export OPENSPACE_MODEL="${OPENSPACE_MODEL:-gpt-5.4}" +export OPENSPACE_LLM_API_KEY="$api_key" +export OPENSPACE_LLM_API_BASE="${OPENSPACE_LLM_API_BASE:-https://codexapi.space/v1}" +export OPENSPACE_LLM_OPENAI_STREAM_COMPAT="${OPENSPACE_LLM_OPENAI_STREAM_COMPAT:-true}" +export OPENSPACE_HOST_SKILL_DIRS="${OPENSPACE_HOST_SKILL_DIRS:-$ALT_HOME/skills}" +export OPENSPACE_WORKSPACE="${OPENSPACE_WORKSPACE:-$REPO_ROOT}" + +exec "$REPO_ROOT/.venv/bin/openspace" "$@"