diff --git a/README.md b/README.md index e41d69da..26a58740 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,23 @@ Agent Memory = Long-Term Memory + Short-Term Memory ## 📰 Latest Updates +- **[2026-02]** 💻 ReMeCli: A terminal-based AI chat assistant with built-in memory management. Automatically compacts long conversations into summaries to free up context space, and persists important information as Markdown files for retrieval in future sessions. Memory design inspired by [OpenClaw](https://github.com/openclaw/openclaw). + - [Quick Start](docs/cli/quick_start_en.md) + - Type `/horse` to trigger the Year of the Horse Easter egg -- fireworks, a galloping horse animation, and a random blessing. + + + + + + +
+ 马
上
有
钱 + 		+ + + 马
到
成
功 +
+ - **[2025-12]** 📄 Our procedural (task) memory paper has been released on [arXiv](https://arxiv.org/abs/2512.10696) - **[2025-11]** 🧠 React-agent with working-memory demo ([Intro](docs/work_memory/message_offload.md)) with ([Quick Start](docs/cookbook/working/quick_start.md)) and ([Code](cookbook/working_memory/work_memory_demo.py)) - **[2025-10]** 🚀 Direct Python import support: use `from reme_ai import ReMeApp` without HTTP/MCP service diff --git a/README_ZH.md b/README_ZH.md index 777e07d3..e9fd9477 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -41,6 +41,23 @@ Agent Memory = Long-Term Memory + Short-Term Memory ## 📰 最新进展 +- **[2026-02]** 💻 ReMeCli：终端 AI 聊天助手，内置记忆管理能力。当对话过长时自动将旧内容压缩为摘要以释放上下文空间，同时将重要信息以 Markdown 文件持久化存储，供未来会话自动检索使用。记忆设计灵感来源于 [OpenClaw](https://github.com/openclaw/openclaw)。 + - [快速开始](docs/cli/quick_start_en.md) + - 输入 `/horse` 触发马年彩蛋——烟花、奔马动画和随机马年祝福。 + + + + + + +
+ 马
上
有
钱 + 		+ + + 马
到
成
功 +
+ - **[2025-12]** 📄 我们的程序性（任务）记忆论文已在 [arXiv](https://arxiv.org/abs/2512.10696) 发布 - **[2025-11]** 🧠 基于工作记忆的 react-agent demo（[介绍](docs/work_memory/message_offload.md)、[Quick Start](docs/cookbook/working/quick_start.md)、[代码](cookbook/working_memory/work_memory_demo.py)） - **[2025-10]** 🚀 直接 Python 导入：支持 `from reme_ai import ReMeApp`，无需 HTTP/MCP 服务 diff --git a/docs/cli/quick_start_en.md b/docs/cli/quick_start_en.md new file mode 100644 index 00000000..59c77603 --- /dev/null +++ b/docs/cli/quick_start_en.md @@ -0,0 +1,296 @@ +# ReMe CLI Quick Start + +## Memory Management: Why Does AI Need This? + +Anyone who has used LLMs knows the context window is limited. As conversations grow longer: + +- The conversation gets cut off and can't continue +- Response quality drops noticeably — it forgets what was said earlier +- Start a new conversation? Everything from before is gone, back to square one + +Worse, **even if the context isn't full, a new conversation starts as a blank slate**. The technical decisions you made +last time, your personal preferences, work left half-done — all gone. + +ReMe solves this with two capabilities: + +| Capability | Purpose | +|------------------------|-------------------------------------------------------------------------------------------------------------------------| +| **Context compaction** | When conversations get too long, old content is automatically condensed into summaries to free up space for new content | +| **Long-term memory** | Important information is persisted to disk and automatically retrieved in future conversations | + +--- + +## File-Based Memory Design + +ReMe's long-term memory doesn't depend on an external database — **Markdown files are the memory itself**. You can open +and edit them at any time. + +> Memory design inspired by the [OpenClaw](https://github.com/openclaw/openclaw) memory architecture. + +### File Structure + +``` +.reme/ +├── MEMORY.md +└── memory/ + ├── 2025-02-12.md + ├── 2025-02-13.md + └── ... +``` + +### MEMORY.md — Long-Term Memory + +Stores key information that rarely changes — essentially your "profile": + +- **Location**: `{working_dir}/MEMORY.md` +- **Example content**: Project uses Python 3.12, prefers pytest, database is PostgreSQL +- **Written by**: Agent maintains it automatically via `write` / `edit` tools + +### memory/YYYY-MM-DD.md — Daily Logs + +One file per day, append-only, recording what happened: + +- **Location**: `{working_dir}/memory/YYYY-MM-DD.md` +- **Example content**: Fixed login bug, deployed v2.1, discussed caching strategy +- **Written by**: Agent tool writes + triggered automatically during compaction + +--- + +## ReMeCli Demo + + + +--- + +## Installation + +### PyPI (Recommended) + +```bash +pip install reme-ai==0.3.0.0b1 +``` + +### From Source + +```bash +git clone https://github.com/agentscope-ai/ReMe.git +cd ReMe +pip install -e . +``` + +> Python >= 3.10 + +--- + +## Configuration + +### Environment Variables + +In addition to the yaml config file, API keys are set via environment variables. You can put them in a `.env` file at +the project root: + +| Variable | Description | Example | +|---------------------------|--------------------|-----------------------------------------------------| +| `REME_LLM_API_KEY` | LLM API Key | `sk-xxx` | +| `REME_LLM_BASE_URL` | LLM Base URL | `https://dashscope.aliyuncs.com/compatible-mode/v1` | +| `REME_EMBEDDING_API_KEY` | Embedding API Key | `sk-xxx` | +| `REME_EMBEDDING_BASE_URL` | Embedding Base URL | `https://dashscope.aliyuncs.com/compatible-mode/v1` | + +> If you don't have an embedding service, search quality will be reduced. Make sure to also set `vector_enabled=false`. + +### Web Search (Optional) + +| Variable | Description | +|---------------------|-------------------------------------| +| `TAVILY_API_KEY` | Tavily Search API Key | +| `DASHSCOPE_API_KEY` | DashScope LLM (with search) API Key | + +> Pick one. If Tavily is available, it takes priority. + +--- + +### Config File: cli.yaml + +`remecli` loads [cli.yaml](https://github.com/agentscope-ai/ReMe/blob/main/reme/config/cli.yaml) on startup ( +`config_path="cli"`). All core parameters are managed in this single file. + +#### Parameter Reference + +**Basic Configuration** + +| Parameter | Value | Description | +|---------------|---------|---------------------------------------------------| +| `backend` | `cmd` | Runtime mode. CLI uses `cmd` | +| `working_dir` | `.reme` | Workspace directory where memory files are stored | + +**metadata — Context Window and Retrieval Parameters** + +Controls how context space is allocated and how memory is searched: + +| Parameter | Default | Description | +|-------------------------|----------|---------------------------------------------------------------------| +| `context_window_tokens` | `100000` | Total context window size (tokens) | +| `reserve_tokens` | `30000` | Space reserved for output and system overhead | +| `keep_recent_tokens` | `10000` | How many recent conversation tokens to keep after compaction | +| `vector_weight` | `0.7` | Vector search weight (BM25 = 1 - 0.7 = 0.3) | +| `candidate_multiplier` | `2` | Retrieval candidate pool multiplier. Higher = better recall, slower | + +> Auto-compaction triggers when total message tokens >= `context_window_tokens - reserve_tokens`, i.e. 70,000 tokens by +> default. + +**llms — LLM Models** + +| Parameter | Description | +|--------------------|-----------------------------------------------| +| `backend` | Backend type, uses OpenAI-compatible API | +| `model_name` | Model name, defaults to Qwen | +| `request_interval` | Request interval (seconds), for rate limiting | + +**embedding_models — Embedding Models** + +| Parameter | Description | +|--------------|---------------------------------------------| +| `backend` | Embedding backend type | +| `model_name` | Model name, defaults to `text-embedding-v4` | +| `dimensions` | Vector dimensions, `1024` | + +**memory_stores — Memory Storage** + +| Parameter | Description | +|-------------------|--------------------------------------------------| +| `backend` | Storage backend, defaults to `chroma` (ChromaDB) | +| `db_name` | Database file name | +| `store_name` | Collection name | +| `embedding_model` | Which embedding model to use | +| `fts_enabled` | Whether to enable BM25 full-text search | +| `vector_enabled` | Whether to enable vector semantic search | + +> Recommended to enable both `fts_enabled` and `vector_enabled` for the best hybrid retrieval results. + +**file_watchers — File Monitoring** + +| Parameter | Description | +|------------------|----------------------------------------| +| `backend` | Monitoring mode, `full` = full scan | +| `memory_store` | Corresponding memory store config | +| `watch_paths` | Directories/files to monitor | +| `suffix_filters` | Which file suffixes to watch (`.md`) | +| `recursive` | Whether to recurse into subdirectories | +| `scan_on_start` | Whether to do a full scan on startup | + +**token_counters — Token Counter** + +| Parameter | Description | +|-----------|---------------------------------------| +| `backend` | Counting method, `base` uses tiktoken | + +## Launch + +```bash +remecli config=cli +``` + +After launch, [cli.yaml](https://github.com/agentscope-ai/ReMe/blob/main/reme/config/cli.yaml) is loaded automatically +and you can start chatting with Remy. ReMe handles compaction and memory in the background. + +--- + +## System Commands + +Type `/`-prefixed commands during a conversation to control state: + +| Command | Description | Blocks | +|------------|---------------------------------------------------------------------------------------------|--------| +| `/compact` | Manually compact the current conversation; also saves to long-term memory in the background | Yes | +| `/new` | Start a new conversation; history is saved to long-term memory in the background | No | +| `/clear` | Clear everything, **without saving** | No | +| `/history` | View uncompacted messages in the current conversation | No | +| `/help` | Show command list | No | +| `/exit` | Exit | No | + +### Comparing the Three Commands + +| Command | Compaction Summary | Long-Term Memory | Message History | +|------------|-----------------------|------------------|-----------------------| +| `/compact` | Generates new summary | Saved | Keeps recent messages | +| `/new` | Cleared | Saved | Cleared | +| `/clear` | Cleared | Not saved | Cleared | + +> `/clear` is a hard delete — once cleared, it's gone and not saved anywhere. + +--- + +## ReMeCli Capabilities + +### When Does Memory Get Written? + +| Scenario | Written To | Trigger | +|---------------------------------------------------|--------------------------|-------------------------------------| +| Auto-compaction when context is too long | `memory/YYYY-MM-DD.md` | Automatic in background | +| User runs `/compact` | `memory/YYYY-MM-DD.md` | Manual compaction + background save | +| User runs `/new` | `memory/YYYY-MM-DD.md` | New conversation + background save | +| User says "remember this" | `MEMORY.md` or daily log | Agent writes via `write` tool | +| Agent identifies an important decision/preference | `MEMORY.md` | Agent writes proactively | + +### Memory Retrieval + +Two ways to find previously stored information: + +| Method | Tool | When to Use | Example | +|-----------------|-----------------|--------------------------------------------|----------------------------------------| +| Semantic search | `memory_search` | Don't know where it's stored, fuzzy lookup | "previous discussion about deployment" | +| Direct read | `read` | Know the date or file | Read `memory/2025-02-13.md` | + +Search uses **vector + BM25 hybrid retrieval** (vector weight 0.7, BM25 weight 0.3), so both natural language queries +and exact keywords work. + +### Built-in Tools + +| Tool | Function | Details | +|-----------------|----------------|--------------------------------------------------------------| +| `memory_search` | Search memory | Hybrid vector + BM25 search across MEMORY.md and memory/*.md | +| `bash` | Run commands | Execute bash commands with timeout and output truncation | +| `ls` | List directory | Show directory structure | +| `read` | Read files | Supports text and images, with partial reads | +| `edit` | Edit files | Exact text match and replace | +| `write` | Write files | Create or overwrite, auto-creates directories | +| `execute_code` | Run Python | Execute code snippets | +| `web_search` | Web search | Search via Tavily or DashScope | + +--- + +## How Context Compaction Works + +In short, long conversations are condensed into summaries while recent messages stay intact. Two trigger modes: + +### Auto-Compaction + +Before each conversation turn, ReMe checks current token usage. If it exceeds the threshold ( +`context_window_tokens - reserve_tokens`), old messages are automatically compacted: + +``` +Before compaction: After compaction: ++--------------------------+ +--------------------------+ +| Message 1: Hello | | Summary: Previously | +| Message 2: Write code | ──────> | helped user write code | +| Message 3: Tool output | | and make adjustments | +| (very long) | +--------------------------+ +| Message 4: Make changes | | Message 5: New request | +| Message 5: New request | +--------------------------+ ++--------------------------+ +``` + +### Manual Compaction + +Type `/compact` at any time to force-compact all current messages, regardless of the threshold. + +### What Gets Preserved in the Summary? + +| Content | Description | Example | +|-----------------------------|------------------------------------|----------------------------------------------------------| +| Goal | What the user wants to do | "Build a login system" | +| Constraints and preferences | Requirements the user specified | "Use TypeScript, no frameworks" | +| Progress | What's been done so far | "Login endpoint is done, registration still in progress" | +| Key decisions | What was decided and why | "Chose JWT over sessions for statelessness" | +| Next steps | What to do next | "Implement password reset" | +| Key context | File names, function names, errors | "Main file is src/auth.ts" | diff --git a/docs/cli/quick_start_zh.md b/docs/cli/quick_start_zh.md new file mode 100644 index 00000000..cf7c8ace --- /dev/null +++ b/docs/cli/quick_start_zh.md @@ -0,0 +1,300 @@ +# ReMe CLI 快速开始 + +## 记忆管理：AI 为什么需要这个？ + +用过大模型的人都知道，上下文窗口是有限的。聊着聊着就超长了，然后： + +- 对话直接断掉，没法继续 +- 回答质量明显变差，前面说的东西它不记得了 +- 开个新对话？之前聊的全忘了，从头来过 + +更烦的是，**就算上下文没满，新对话也是一张白纸**。上次定好的技术方案、你的个人偏好、干到一半的活——全没了。 + +ReMe 干了两件事来解决这个问题： + +| 能力 | 干嘛用的 | +|-----------|---------------------------| +| **上下文压缩** | 对话太长时，把旧内容自动浓缩成摘要，给新内容腾地方 | +| **长期记忆** | 重要信息落盘保存，下次对话自动搜出来用 | + +--- + +## 基于文件的记忆设计 + +ReMe 的长期记忆不依赖外部数据库——**Markdown 文件就是记忆本身**。你随时可以打开看、直接改。 + +> 记忆设计受 [OpenClaw](https://github.com/openclaw/openclaw) 记忆架构启发。 + +### 文件结构 + +``` +.reme/ +├── MEMORY.md +└── memory/ + ├── 2025-02-12.md + ├── 2025-02-13.md + └── ... +``` + +### MEMORY.md — 长期记忆 + +放那些不太会变的关键信息，相当于你的"个人档案"： + +- **位置**：`{working_dir}/MEMORY.md` +- **内容举例**：项目用 Python 3.12、偏好 pytest、数据库选了 PostgreSQL +- **谁来写**：Agent 通过 `write` / `edit` 工具自动维护 + +### memory/YYYY-MM-DD.md — 每日日志 + +一天一个文件，追加写入，记今天干了啥： + +- **位置**：`{working_dir}/memory/YYYY-MM-DD.md` +- **内容举例**：修了登录 Bug、部署了 v2.1、讨论了缓存方案 +- **谁来写**：Agent 工具写入 + 压缩时自动触发 + +--- + +## ReMeCli Demo + + + +--- + +## 安装 + +### PyPI（推荐） + +```bash +pip install reme-ai==0.3.0.0b1 +``` + +### 从源码装 + +```bash +git clone https://github.com/agentscope-ai/ReMe.git +cd ReMe +pip install -e . +``` + +> Python >= 3.10 + +--- + +## 配置 + +### 环境变量 + +除了 yaml 配置文件，API 密钥通过环境变量设置，可以写在项目根目录的 `.env` 里： + +| 环境变量 | 说明 | 示例 | +|---------------------------|----------------------|-----------------------------------------------------| +| `REME_LLM_API_KEY` | LLM 的 API Key | `sk-xxx` | +| `REME_LLM_BASE_URL` | LLM 的 Base URL | `https://dashscope.aliyuncs.com/compatible-mode/v1` | +| `REME_EMBEDDING_API_KEY` | Embedding 的 API Key | `sk-xxx` | +| `REME_EMBEDDING_BASE_URL` | Embedding 的 Base URL | `https://dashscope.aliyuncs.com/compatible-mode/v1` | + +> 没有 embedding 服务的话搜索效果会打折扣，记得同时设 `vector_enabled=false`。 + +### 联网搜索（可选） + +| 环境变量 | 说明 | +|---------------------|--------------------| +| `TAVILY_API_KEY` | Tavily 搜索 API Key | +| `DASHSCOPE_API_KEY` | 百炼 LLM（带搜索）API Key | + +> 二选一就行，有 Tavily 优先用 Tavily。 + +--- + +### 配置文件 cli.yaml + +`remecli` 启动时加载 [cli.yaml](https://github.com/agentscope-ai/ReMe/blob/main/reme/config/cli.yaml)（ +`config_path="cli"`），所有核心参数都在这一个文件里管。 + +#### 参数说明 + +**基础配置** + +| 参数 | 值 | 说明 | +|---------------|---------|------------------| +| `backend` | `cmd` | 运行模式，CLI 用 `cmd` | +| `working_dir` | `.reme` | 工作空间目录，记忆文件存这里 | + +**metadata — 上下文窗口与检索参数** + +控制上下文空间怎么分配、记忆怎么搜： + +| 参数 | 默认值 | 说明 | +|-------------------------|----------|------------------------------| +| `context_window_tokens` | `100000` | 上下文窗口总大小（token） | +| `reserve_tokens` | `30000` | 给输出和系统开销预留的空间 | +| `keep_recent_tokens` | `10000` | 压缩后保留多少最近的对话 | +| `vector_weight` | `0.7` | 向量搜索权重（BM25 = 1 - 0.7 = 0.3） | +| `candidate_multiplier` | `2` | 检索候选池倍数，越大召回越全、越慢 | + +> 自动压缩的触发点：消息总 token ≥ `context_window_tokens - reserve_tokens`，即默认 70000 token。 + +**llms — LLM 模型** + +| 参数 | 说明 | +|--------------------|--------------------| +| `backend` | 后端类型，走 OpenAI 兼容接口 | +| `model_name` | 模型名，默认通义千问 | +| `request_interval` | 请求间隔（秒），控速用 | + +**embedding_models — Embedding 模型** + +| 参数 | 说明 | +|--------------|----------------------------| +| `backend` | Embedding 后端类型 | +| `model_name` | 模型名，默认 `text-embedding-v4` | +| `dimensions` | 向量维度，`1024` | + +**memory_stores — 记忆存储** + +| 参数 | 说明 | +|-------------------|----------------------------| +| `backend` | 存储后端，默认 `chroma`（ChromaDB） | +| `db_name` | 数据库文件名 | +| `store_name` | 集合名 | +| `embedding_model` | 用哪个 Embedding 模型 | +| `fts_enabled` | 开不开 BM25 全文检索 | +| `vector_enabled` | 开不开向量语义搜索 | + +> 建议 `fts_enabled` 和 `vector_enabled` 都开，混合检索效果最好。 + +**file_watchers — 文件监控** + +| 参数 | 说明 | +|------------------|--------------------| +| `backend` | 监控模式，`full` = 全量扫描 | +| `memory_store` | 对应的记忆存储配置 | +| `watch_paths` | 要监控的目录/文件 | +| `suffix_filters` | 只关心哪些后缀（`.md`） | +| `recursive` | 是否递归子目录 | +| `scan_on_start` | 启动时先全量扫一遍 | + +**token_counters — Token 计数器** + +| 参数 | 说明 | +|-----------|------------------------| +| `backend` | 计数方式，`base` 用 tiktoken | + +## 启动 + +```bash +remecli config=cli +``` + +启动后自动加载 [cli.yaml](https://github.com/agentscope-ai/ReMe/blob/main/reme/config/cli.yaml)，然后就可以直接跟 Remy +聊了。ReMe 在后台自动处理压缩和记忆。 + +--- + +## 系统命令 + +对话里输入 `/` 开头的命令控制状态： + +| 命令 | 说明 | 需要等 | +|------------|---------------------|-----| +| `/compact` | 手动压缩当前对话，同时后台存到长期记忆 | 是 | +| `/new` | 开始新对话，历史后台保存到长期记忆 | 否 | +| `/clear` | 清空一切，**不保存** | 否 | +| `/history` | 看当前对话里未压缩的消息 | 否 | +| `/help` | 看命令列表 | 否 | +| `/exit` | 退出 | 否 | + +### 三个命令的区别 + +| 命令 | 压缩摘要 | 长期记忆 | 消息历史 | +|------------|-------|------|-------| +| `/compact` | 生成新摘要 | 保存 | 保留最近的 | +| `/new` | 清空 | 保存 | 清空 | +| `/clear` | 清空 | 不保存 | 清空 | + +> `/clear` 是真删，删了就没了，不会存到任何地方。 + +--- + +## ReMeCli 的能力 + +### 什么时候会写记忆？ + +| 场景 | 写到哪 | 怎么触发 | +|------------------|------------------------|----------------------| +| 上下文超长自动压缩 | `memory/YYYY-MM-DD.md` | 后台自动 | +| 用户执行 `/compact` | `memory/YYYY-MM-DD.md` | 手动压缩 + 后台保存 | +| 用户执行 `/new` | `memory/YYYY-MM-DD.md` | 新对话 + 后台保存 | +| 用户说"记住这个" | `MEMORY.md` 或日志 | Agent 用 `write` 工具写入 | +| Agent 发现了重要决策/偏好 | `MEMORY.md` | Agent 主动写 | + +### 记忆检索 + +两种方式找回之前的东西： + +| 方式 | 工具 | 什么时候用 | 举例 | +|------|-----------------|------------|--------------------------| +| 语义搜索 | `memory_search` | 不确定记在哪，模糊找 | "之前关于部署的讨论" | +| 直接读 | `read` | 知道是哪天、哪个文件 | 读 `memory/2025-02-13.md` | + +搜索用的是**向量 + BM25 混合检索**（向量权重 0.7，BM25 权重 0.3），自然语言和精确关键词都能搜到。 + +### 内置工具 + +| 工具 | 干什么 | 细节 | +|-----------------|----------|----------------------------------------| +| `memory_search` | 搜记忆 | MEMORY.md 和 memory/*.md 里做向量+BM25 混合检索 | +| `bash` | 跑命令 | 执行 bash 命令，有超时和输出截断 | +| `ls` | 看目录 | 列目录结构 | +| `read` | 读文件 | 文本和图片都行，支持分段读 | +| `edit` | 改文件 | 精确匹配文本后替换 | +| `write` | 写文件 | 创建或覆盖，自动建目录 | +| `execute_code` | 跑 Python | 运行代码片段 | +| `web_search` | 联网搜索 | 通过 Tavily 或 DashScope 搜 | + +--- + +## 上下文压缩怎么工作的 + +简单说就是把长对话浓缩成摘要，最近的对话保持原样。两种触发方式： + +### 自动压缩 + +每轮对话前 ReMe 会检查当前 token 用量。超过阈值（`context_window_tokens - reserve_tokens`）就自动压缩旧消息： + +```mermaid +graph TB + subgraph 压缩前 + A1[消息1: 你好] + A2[消息2: 帮我写代码] + A3[消息3: 工具调用结果...很长] + A4[消息4: 修改一下] + A5[消息5: 新需求] + end + + subgraph 压缩后 + B1[压缩摘要: 之前帮用户写了代码并完成调整] + B2[消息5: 新需求] + end + + A1 --> B1 + A2 --> B1 + A3 --> B1 + A4 --> B1 + A5 --> B2 +``` + +### 手动压缩 + +随时输入 `/compact`，强制压缩所有当前消息，不看阈值。 + +### 摘要里会留什么？ + +| 内容 | 说的是啥 | 例子 | +|-------|------------|-------------------------| +| 目标 | 用户想干什么 | "搞一个登录系统" | +| 约束和偏好 | 用户提的要求 | "用 TypeScript，不要框架" | +| 进展 | 做到哪了 | "登录接口好了，注册还在写" | +| 关键决策 | 定了什么、为什么 | "选 JWT 不选 Session，要无状态" | +| 下一步 | 接下来干嘛 | "做密码重置" | +| 关键上下文 | 文件名、函数名、报错 | "主文件 src/auth.ts" | diff --git a/docs/make_mp4.md b/docs/make_mp4.md new file mode 100644 index 00000000..c5b5d869 --- /dev/null +++ b/docs/make_mp4.md @@ -0,0 +1,19 @@ +```shell +ffmpeg -i /Users/yuli/Desktop/remecli_en.mov \ + -vf "scale=-2:1080,setpts=0.333*PTS" \ + -c:v libx264 \ + -crf 28 \ + -preset fast \ + -c:a aac \ + -b:a 96k \ + /Users/yuli/Desktop/remecli_en_1080p_3x.mp4 + +ffmpeg -i /Users/yuli/Desktop/remecli_zh.mov \ + -vf "scale=-2:1080,setpts=0.333*PTS" \ + -c:v libx264 \ + -crf 28 \ + -preset fast \ + -c:a aac \ + -b:a 96k \ + /Users/yuli/Desktop/remecli_zh_1080p_3x.mp4 +``` \ No newline at end of file diff --git a/example.env b/example.env index d09632d4..b988f2b2 100644 --- a/example.env +++ b/example.env @@ -2,3 +2,10 @@ FLOW_EMBEDDING_API_KEY=sk-xxxx FLOW_EMBEDDING_BASE_URL=https://xxxx/v1 FLOW_LLM_API_KEY=sk-xxxx FLOW_LLM_BASE_URL=https://xxxx/v1 + +REME_LLM_API_KEY=sk-xxxx +REME_LLM_BASE_URL=https://xxxx/v1 +REME_EMBEDDING_API_KEY=sk-xxxx +REME_EMBEDDING_BASE_URL=https://xxxx/v1 + +TAVILY_API_KEY=xxxx diff --git a/reme/__init__.py b/reme/__init__.py index 4ab6f36a..c280ab02 100644 --- a/reme/__init__.py +++ b/reme/__init__.py @@ -20,7 +20,7 @@ __all__ = [ "ReMeFs", ] -__version__ = "0.3.0.0a9" +__version__ = "0.3.0.0b1" """ diff --git a/reme/agent/chat/fs_cli.py b/reme/agent/chat/fs_cli.py index 671062e8..2ee1f352 100644 --- a/reme/agent/chat/fs_cli.py +++ b/reme/agent/chat/fs_cli.py @@ -1,5 +1,6 @@ """FsCli system prompt""" +import asyncio from datetime import datetime from pathlib import Path @@ -8,6 +9,7 @@ from loguru import logger from ...core.enumeration import Role, ChunkEnum from ...core.op import BaseReactStream from ...core.schema import Message, StreamChunk +from ...core.utils import format_messages from ...tool.fs import BashTool, LsTool, ReadTool, WriteTool, EditTool @@ -31,18 +33,23 @@ class FsCli(BaseReactStream): self.messages: list[Message] = [] self.previous_summary: str = "" + self.summary_tasks: list[asyncio.Task] = [] - async def reset(self) -> str: - """Reset conversation history using summary. + def add_summary_task(self, messages: list[Message]): + """Add summary task to queue.""" + remaining_tasks = [] + for task in self.summary_tasks: + if task.done(): + exc = task.exception() + if exc is not None: + logger.exception(f"Summary task failed: {exc}") + else: + result = task.result() + logger.info(f"Summary task completed: {result}") + else: + remaining_tasks.append(task) + self.summary_tasks = remaining_tasks - Summarizes current messages to memory files and clears history. - """ - if not self.messages: - self.messages.clear() - self.previous_summary = "" - return "No history to reset." - - # Import required modules from ..fs import FsSummarizer # Summarize current conversation and save to memory files @@ -59,14 +66,30 @@ class FsCli(BaseReactStream): language=self.language, ) - result = await summarizer.call( - messages=self.messages, - date=current_date, - service_context=self.service_context, + summary_task = asyncio.create_task( + summarizer.call( + messages=messages, + date=current_date, + service_context=self.service_context, + ), ) + self.summary_tasks.append(summary_task) + + async def new(self) -> str: + """Reset conversation history using summary. + + Summarizes current messages to memory files and clears history. + """ + if not self.messages: + self.messages.clear() + self.previous_summary = "" + return "No history to reset." + + self.add_summary_task(self.messages) + self.messages.clear() self.previous_summary = "" - return f"History saved to memory files and reset. Result: {result.get('answer', 'Done')}" + return "History saved to memory files and reset." async def context_check(self) -> dict: """Check if messages exceed token limits.""" @@ -104,20 +127,16 @@ class FsCli(BaseReactStream): tokens_before = cut_result.get("token_count", 0) if force_compact: - # Force compact: summarize all messages, leave only summary messages_to_summarize = self.messages turn_prefix_messages = [] left_messages = [] elif not cut_result.get("needs_compaction", False): - # No compaction needed return "History is within token limits, no compaction needed." else: - # Normal compaction: use cut point result messages_to_summarize = cut_result.get("messages_to_summarize", []) turn_prefix_messages = cut_result.get("turn_prefix_messages", []) left_messages = cut_result.get("left_messages", []) - # Step 2: Generate summary via Compactor compactor = FsCompactor(language=self.language) summary_content = await compactor.call( messages_to_summarize=messages_to_summarize, @@ -126,26 +145,37 @@ class FsCli(BaseReactStream): service_context=self.service_context, ) - # Step 3: Call reset_history to save and clear - reset_result = await self.reset() + self.add_summary_task(messages=messages_to_summarize) # Step 4: Assemble final messages self.messages = left_messages self.previous_summary = summary_content - return f"History compacted from {tokens_before} tokens. {reset_result}" + return f"History compacted from {tokens_before} tokens." + + def format_history(self) -> str: + """Format history messages.""" + return format_messages( + messages=self.messages, + add_index=False, + add_reasoning=False, + strip_markdown_headers=False, + ) async def build_messages(self) -> list[Message]: """Build system prompt message.""" current_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S %A") + has_web_search = any(t.name == "web_search" for t in self.tools) system_prompt = self.prompt_format( "system_prompt", workspace_dir=self.working_dir, current_time=current_time, + has_web_search=has_web_search, has_previous_summary=bool(self.previous_summary), previous_summary=self.previous_summary or "", ) + logger.info(f"[{self.__class__.__name__}] system_prompt: {system_prompt}") return [ Message(role=Role.SYSTEM, content=system_prompt), diff --git a/reme/agent/chat/fs_cli.yaml b/reme/agent/chat/fs_cli.yaml index a6b861a7..d18a9d49 100644 --- a/reme/agent/chat/fs_cli.yaml +++ b/reme/agent/chat/fs_cli.yaml @@ -1,70 +1,54 @@ system_prompt: | You are a personal assistant named Remy. - ## Workspace Dir + ## Working Directory {workspace_dir} ## Current Time {current_time} - ## Memory - You wake up fresh each session. These files are your continuity: - - **Daily notes:** `memory/YYYY-MM-DD.md` — raw logs of what happened - - **Long-term:** `MEMORY.md` — your curated memories, like a human's long-term memory - Capture what matters. Decisions, context, things to remember. Skip the secrets unless asked to keep them. + ## Tools + - `bash_tool` Run shell commands + - `ls_tool` List directory contents + - `read_tool` Read file contents + - `edit_tool` Edit file contents + - `write_tool` Write file contents + - `execute_code` Run Python code + - `memory_search` Search your memories via vector store + [has_web_search]- `web_search` Search the web - ### 🧠 MEMORY.md - Your Long-Term Memory - - You can **read, edit, and update** MEMORY.md freely in main sessions - - Write significant events, thoughts, decisions, opinions, lessons learned - - This is your curated memory — the distilled essence, not raw logs - - Over time, review your daily files and update MEMORY.md with what's worth keeping + **Don't give up easily** — if a tool doesn't return what you expect, try a different angle or approach. - ### 📝 Write It Down - No "Mental Notes"! - - "Mental notes" don't survive session restarts. Files do. - - **IMPORTANT: Always read the file first before writing** — understand what's already there, then append or update - - When someone says "remember this" → read then update `memory/YYYY-MM-DD.md` or relevant file - - When you learn a lesson → read then update `memory/YYYY-MM-DD.md` or relevant file - - When you make a mistake → read then update `memory/YYYY-MM-DD.md` or relevant file - - **Text > Brain** 📝 + ## Memory System + You are spun up fresh at the start of every session. These files are how you maintain continuity: + - **Long-term memory:** `MEMORY.md` — when you pick up a lesson or catch yourself making a mistake, feel free to **read, edit, and update** MEMORY.md + - **Daily notes:** `memory/YYYY-MM-DD.md` — jot things down often. When the user says "remember this," or whenever you feel something is worth noting or adding as a todo, feel free to **read, edit, and update** `memory/YYYY-MM-DD.md` + - **Read before you write** — always use `read_tool` to check existing content before updating with `edit_tool` or `write_tool` - ### 🔍 Recall Tools - Before answering questions about prior work, decisions, dates, people, preferences, or todos: - 1. Run `memory_search` on MEMORY.md + memory/*.md - 2. If you need to read the Daily Notes `memory/YYYY-MM-DD.md`, you can use the read tool to access it. + ### Memory Retrieval + 1. Start with `memory_search` — if nothing comes up, try rephrasing from a different angle + 2. To review a specific daily note (`memory/YYYY-MM-DD.md`), use `read_tool` - ### 🛠️ Other Tools - - **bash_tool** — execute shell commands - - **ls_tool** — list directory contents - - **read_tool** — read file contents - - **edit_tool** — modify existing files - - **write_tool** — create new files - - **execute_code** — run Python code - - **dashscope_search** — search the web + ## Response Style 😊 + - Keep it short and natural — talk like a friend, not a manual + - Use emoji sparingly for warmth — no more than 1–2 per reply + - For quick confirmations (yes/no, got it), an emoji is fine (👍, ✅, 🤔) + - When explaining or performing actions, lead with substance over flair - ## React Like a Human 😊 - **React when:** - - You appreciate something but don't need to reply (👍, ❤️, 🙌) - - Something made you laugh (😂, 💀) - - You find it interesting or thought-provoking (🤔, 💡) - - You want to acknowledge without interrupting the flow - - It's a simple yes/no or approval situation (✅, 👀) - **Why:** Reactions are lightweight social signals. Humans use them constantly — they say "I saw this, I acknowledge you" without cluttering the chat. - **Don't overdo it:** One reaction per message max. Pick the one that fits best. - - ## 🛡️ Safety Rules - - Don't run destructive commands without asking - - Prefer `trash` over `rm` (recoverable beats gone forever) + ## 🛡️ Safety + - Never run destructive commands without asking first + - Prefer `trash` over `rm` — recoverable beats permanent - When in doubt, ask ## Continuous Improvement - This is a starting point. Add your own conventions, style, and rules as you figure out what works. + This is just a starting point. When you spot useful patterns or lessons during your conversations, note them in `MEMORY.md`. Do not modify system-level config files. [has_previous_summary]## Previous Conversation Summary [has_previous_summary] [has_previous_summary]{previous_summary} [has_previous_summary] [has_previous_summary] - [has_previous_summary]The above is a summary of our previous conversation. Use it as context to maintain continuity. + [has_previous_summary]The above is a summary of our earlier conversation. Use it as context to maintain continuity. system_prompt_zh: | 你是一个名叫 Remy 的个人助手。 @@ -75,50 +59,33 @@ system_prompt_zh: | ## 当前时间 {current_time} + ## 工具集合 + - `bash_tool` 执行shell命令 + - `ls_tool` 列出目录内容 + - `read_tool` 读取文件内容 + - `edit_tool` 编辑文件内容 + - `write_tool` 写入文件内容 + - `execute_code` 运行python代码 + - `memory_search` 通过向量库检索你的记忆 + [has_web_search]- `web_search` 网络搜索 + + **不要轻易放弃**：如果工具执行结果不符合预期，可以从不同的维度进行不同的尝试。 + ## 记忆系统 - 每次会话你都会重新唤醒。这些文件是你保持连续性的关键： - - **每日笔记：** `memory/YYYY-MM-DD.md` — 发生的事情的原始记录 - - **长期记忆：** `MEMORY.md` — 你精心整理的记忆，就像人类的长期记忆一样 - 记录重要的事情。决策、上下文、需要记住的事情。除非被要求保留，否则跳过秘密信息。 + 每次新会话开始时，你都会被重新唤醒。以下文件是你保持连续性的关键： + - **长期记忆：** `MEMORY.md`：当你学到经验，或者当你犯了错误，可以**自由地阅读、编辑和更新** MEMORY.md + - **每日笔记：** `memory/YYYY-MM-DD.md`：要勤记笔记，当用户说"记住这个"，或者你觉得要记笔记/todo，可以**自由地阅读、编辑和更新** `memory/YYYY-MM-DD.md` + - **写入前先读取** — 务必先用 `read_tool` 读取已有内容，再用 `edit_tool` 或 `write_tool` 更新文件 - ### 🧠 MEMORY.md - 你的长期记忆 - - 在主会话中，你可以**自由地阅读、编辑和更新** MEMORY.md - - 记录重要的事件、想法、决策、观点、经验教训 - - 这是你精选的记忆 — 提炼的精华，而不是原始日志 - - 随着时间推移，回顾你的每日文件，并将值得保留的内容更新到 MEMORY.md + ### 记忆检索策略 + 1. 优先使用`memory_search`检索记忆，没有搜索结果可以从不同角度多次尝试 + 2. 如果你需要阅读每日笔记 `memory/YYYY-MM-DD.md`，可以使用`read_tool` - ### 📝 写下来 - 不要只在"脑中记住"！ - - "脑中记住"无法在会话重启后保留。文件可以。 - - **重要：写入之前务必先读取文件** — 了解已有内容，然后再追加或更新 - - 当有人说"记住这个" → 先读取再更新 `memory/YYYY-MM-DD.md` 或相关文件 - - 当你学到经验 → 先读取再更新 `memory/YYYY-MM-DD.md` 或相关文件 - - 当你犯了错误 → 先读取再更新 `memory/YYYY-MM-DD.md` 或相关文件 - - **文字 > 大脑** 📝 - - ### 🔍 检索工具 - 在回答关于过往工作、决策、日期、人员、偏好或待办事项的问题之前： - 1. 对 MEMORY.md + memory/*.md 运行 `memory_search`，没有搜索结果可以从不同角度多次尝试 - 2. 如果你需要阅读每日笔记 `memory/YYYY-MM-DD.md`，可以使用读取工具访问它。 - - ### 🛠️ 其他工具 - - **bash_tool** — 执行 shell 命令 - - **ls_tool** — 列出目录内容 - - **read_tool** — 读取文件内容 - - **edit_tool** — 修改现有文件 - - **write_tool** — 创建新文件 - - **execute_code** — 运行 Python 代码 - - **dashscope_search** — 网络搜索 - 如果对于工具结果不满意，可以混合使用多种工具，或者单个工具不同的使用参数。 - - ## 像人类一样回应 😊 - **何时使用表情回应：** - - 你欣赏某事但不需要文字回复时（👍, ❤️, 🙌） - - 某事让你发笑时（😂, 💀） - - 你觉得有趣或发人深省时（🤔, 💡） - - 你想要确认但不想打断对话流时 - - 这是一个简单的是/否或批准的情况（✅, 👀） - **原因：** 表情回应是轻量级的社交信号。人类经常使用它们 — 它们表示"我看到了，我认可你"而不会让对话变得混乱。 - **不要过度使用：** 每条消息最多一个表情回应。选择最合适的一个。 + ## 回应风格 😊 + - 保持简洁自然，像朋友对话一样 + - 适当使用 emoji 增加亲和力，但不要过度 — 每条回复最多 1-2 个 + - 简单确认类场景（是/否、收到）可以用 emoji 快速回应（👍, ✅, 🤔） + - 涉及操作或解释时，优先给出有实质内容的文字回复 ## 🛡️ 安全规则 - 不要在没有询问的情况下运行破坏性命令 @@ -126,7 +93,7 @@ system_prompt_zh: | - 有疑问时，先询问 ## 持续改进 - 这只是一个起点。随着你逐渐发现什么有效，添加你自己的约定、风格和规则。 + 这只是一个起点。当你在与用户的交互中发现有用的经验或模式，可以记录到 `MEMORY.md` 中。但不要修改系统级配置文件。 [has_previous_summary]## 之前的对话摘要 [has_previous_summary] diff --git a/reme/agent/fs/fs_summarizer.py b/reme/agent/fs/fs_summarizer.py index 91ff0510..38df4fb9 100644 --- a/reme/agent/fs/fs_summarizer.py +++ b/reme/agent/fs/fs_summarizer.py @@ -13,7 +13,7 @@ from ...core.utils import format_messages class FsSummarizer(BaseReact): """Retrieve personal memories through vector search and history reading.""" - def __init__(self, working_dir: str, memory_dir: str = "memory", version: str = "default", **kwargs): + def __init__(self, working_dir: str, memory_dir: str = "memory", version: str = "v1", **kwargs): super().__init__(**kwargs) self.working_dir: str = working_dir self.memory_dir: str = memory_dir diff --git a/reme/config/cli.yaml b/reme/config/cli.yaml new file mode 100644 index 00000000..a126ccfd --- /dev/null +++ b/reme/config/cli.yaml @@ -0,0 +1,48 @@ +backend: cmd +working_dir: .reme + +metadata: + context_window_tokens: 100000 + reserve_tokens: 30000 + keep_recent_tokens: 10000 + vector_weight: 0.7 + candidate_multiplier: 2 + +llms: + default: + backend: openai + model_name: qwen3-235b-a22b-thinking-2507 + request_interval: 1 + +embedding_models: + default: + backend: openai + model_name: text-embedding-v4 + dimensions: 1024 + +memory_stores: + default: + backend: chroma + db_name: reme.db + store_name: reme + embedding_model: default + fts_enabled: true + vector_enabled: true + +file_watchers: + default: + backend: full + memory_store: default + watch_paths: [".reme", ".reme/memory"] + suffix_filters: [".md"] + recursive: false + scan_on_start: true + +token_counters: + default: + backend: base + + hf: + backend: hf + model_name: Qwen/Qwen3-Coder-30B-A3B-Instruct + use_mirror: true diff --git a/reme/core/application.py b/reme/core/application.py index 389b96e9..2ac457e4 100644 --- a/reme/core/application.py +++ b/reme/core/application.py @@ -8,7 +8,7 @@ from .file_watcher import BaseFileWatcher from .flow import BaseFlow from .llm import BaseLLM from .memory_store import BaseMemoryStore -from .schema import Response +from .schema import Response, ServiceConfig from .token_counter import BaseTokenCounter from .utils import execute_stream_task, PydanticConfigParser from .vector_store import BaseVectorStore @@ -210,6 +210,11 @@ class Application: """Get the default token counter instance.""" return self.service_context.token_counters.get("default") + @property + def service_config(self) -> ServiceConfig: + """Get the service configuration.""" + return self.service_context.service_config + def get_token_counter(self, name: str): """Get a token counter instance by name.""" return self.service_context.token_counters.get(name) diff --git a/reme/core/context/service_context.py b/reme/core/context/service_context.py index 2de4b026..3f5a6403 100644 --- a/reme/core/context/service_context.py +++ b/reme/core/context/service_context.py @@ -170,7 +170,7 @@ class ServiceContext(BaseContext): async def start(self): """Start the service context by initializing all configured components.""" # Recreate thread pool if it was shut down - if self.thread_pool is None or self.thread_pool._shutdown: + if self.thread_pool is None or self.thread_pool._shutdown: # pylint: disable=protected-access self.thread_pool = ThreadPoolExecutor( max_workers=self.service_config.thread_pool_max_workers, ) diff --git a/reme/core/schema/service_config.py b/reme/core/schema/service_config.py index 0893197b..84c4e056 100644 --- a/reme/core/schema/service_config.py +++ b/reme/core/schema/service_config.py @@ -134,3 +134,5 @@ class ServiceConfig(BaseModel): memory_stores: dict[str, MemoryStoreConfig] = Field(default_factory=dict) token_counters: dict[str, TokenCounterConfig] = Field(default_factory=dict) file_watchers: dict[str, FileWatcherConfig] = Field(default_factory=dict) + + metadata: dict = Field(default_factory=dict) diff --git a/reme/core/utils/__init__.py b/reme/core/utils/__init__.py index fbe7202b..f8be9d4f 100644 --- a/reme/core/utils/__init__.py +++ b/reme/core/utils/__init__.py @@ -6,7 +6,7 @@ from .case_converter import snake_to_camel, camel_to_snake from .chunking_utils import chunk_markdown from .common_utils import run_coro_safely, execute_stream_task, hash_text, cosine_similarity, batch_cosine_similarity from .env_utils import load_env -from .execute_utils import exec_code, run_shell_command +from .execute_utils import exec_code, run_shell_command, async_exec_code from .http_client import HttpClient from .llm_utils import extract_content, format_messages, deduplicate_memories from .logger_utils import init_logger @@ -30,6 +30,7 @@ __all__ = [ "batch_cosine_similarity", "load_env", "exec_code", + "async_exec_code", "run_shell_command", "HttpClient", "extract_content", diff --git a/reme/core/utils/execute_utils.py b/reme/core/utils/execute_utils.py index a4d680ab..8ea00998 100644 --- a/reme/core/utils/execute_utils.py +++ b/reme/core/utils/execute_utils.py @@ -6,6 +6,7 @@ with support for async execution and output capture. import asyncio import contextlib +import concurrent.futures from io import StringIO @@ -18,6 +19,9 @@ async def run_shell_command(cmd: str, timeout: float | None = 30) -> tuple[str, Returns: A tuple containing (stdout, stderr, return_code) as strings and integer. + + Raises: + TimeoutError: If the command does not complete within the timeout. """ process = await asyncio.create_subprocess_shell( cmd, @@ -25,10 +29,16 @@ async def run_shell_command(cmd: str, timeout: float | None = 30) -> tuple[str, stderr=asyncio.subprocess.PIPE, ) - if timeout: - stdout, stderr = await asyncio.wait_for(process.communicate(), timeout=timeout) - else: - stdout, stderr = await process.communicate() + try: + if timeout: + stdout, stderr = await asyncio.wait_for(process.communicate(), timeout=timeout) + else: + stdout, stderr = await process.communicate() + except (asyncio.TimeoutError, TimeoutError) as e: + # Kill the child process to avoid orphaned / zombie processes + process.kill() + await process.wait() + raise TimeoutError(f"Shell command timed out after {timeout}s") from e return ( stdout.decode("utf-8", errors="ignore"), @@ -37,22 +47,94 @@ async def run_shell_command(cmd: str, timeout: float | None = 30) -> tuple[str, ) -def exec_code(code: str) -> str: +def exec_code( + code: str, + timeout: float | None = 30, + executor: concurrent.futures.ThreadPoolExecutor | None = None, +) -> str: """Execute Python code and capture the output. Args: code: The Python code string to execute. + timeout: Maximum time to wait for execution in seconds. None for no timeout. + executor: Optional thread pool executor to use. If None, a temporary + single-thread executor is created (and shut down after the call). + Pass a shared executor to amortize thread-creation overhead across + multiple calls. Returns: The captured stdout output, or the error message if execution fails. + + Raises: + TimeoutError: If execution exceeds the timeout. """ - try: + + def _run() -> str: redirected_output = StringIO() with contextlib.redirect_stdout(redirected_output): exec(code) - return redirected_output.getvalue() + def _submit(pool: concurrent.futures.ThreadPoolExecutor) -> str: + future = pool.submit(_run) + return future.result(timeout=timeout) + + try: + if executor is not None: + return _submit(executor) + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: + return _submit(pool) + + except concurrent.futures.TimeoutError as e: + raise TimeoutError(f"Code execution timed out after {timeout}s") from e + + except Exception as e: + return str(e) + + except BaseException as e: + return str(e) + + +async def async_exec_code( + code: str, + timeout: float | None = 30, + executor: concurrent.futures.ThreadPoolExecutor | None = None, +) -> str: + """Execute Python code asynchronously and capture the output. + + Runs the code in a thread executor to avoid blocking the event loop, + with async-friendly timeout via ``asyncio.wait_for``. + + Args: + code: The Python code string to execute. + timeout: Maximum time to wait for execution in seconds. None for no timeout. + executor: Optional thread pool executor. If None, the default event-loop + executor is used. + + Returns: + The captured stdout output, or the error message if execution fails. + + Raises: + TimeoutError: If execution exceeds the timeout. + """ + + def _run() -> str: + redirected_output = StringIO() + with contextlib.redirect_stdout(redirected_output): + exec(code) + return redirected_output.getvalue() + + loop = asyncio.get_running_loop() + + try: + coro = loop.run_in_executor(executor, _run) + if timeout is not None: + return await asyncio.wait_for(coro, timeout=timeout) + return await coro + + except asyncio.TimeoutError as e: + raise TimeoutError(f"Code execution timed out after {timeout}s") from e + except Exception as e: return str(e) diff --git a/reme/core/utils/logo_utils.py b/reme/core/utils/logo_utils.py index 5b4acd8e..d85a2073 100644 --- a/reme/core/utils/logo_utils.py +++ b/reme/core/utils/logo_utils.py @@ -81,4 +81,5 @@ def print_logo(service_config: "ServiceConfig"): expand=False, ) - Console().print(Group("\n", panel, "\n"), justify="center") + # use justify="center" to adjust position + Console().print(Group("\n", panel, "\n")) diff --git a/reme/core/utils/pydantic_config_parser.py b/reme/core/utils/pydantic_config_parser.py index 1fd0ab56..1f45e4ca 100644 --- a/reme/core/utils/pydantic_config_parser.py +++ b/reme/core/utils/pydantic_config_parser.py @@ -178,7 +178,7 @@ class PydanticConfigParser: # Merge all configs and validate self.config_dict = self.merge_configs(*configs_to_merge) - return self.config_class.model_validate(self.config_dict) + return self.config_class.model_validate(self.config_dict, extra="allow") def update_config(self, **kwargs) -> T: """Update current config with new values using kwargs. @@ -195,4 +195,4 @@ class PydanticConfigParser: # Merge with existing config final_config = self.merge_configs(self.config_dict, override_config) - return self.config_class.model_validate(final_config) + return self.config_class.model_validate(final_config, extra="allow") diff --git a/reme/horse.py b/reme/horse.py new file mode 100644 index 00000000..a11ac43c --- /dev/null +++ b/reme/horse.py @@ -0,0 +1,165 @@ +"""Horse Easter egg: fireworks, galloping horse animation, and a blessing.""" + +import math +import random +import shutil +import sys +import time + + +def _mirror_frame(frame: str) -> str: + """Mirror ASCII art horizontally.""" + mirror_map = str.maketrans(r"()/\<>[]{}", r")(\/><][}{") + lines = frame.split("\n") + max_len = max(len(line) for line in lines) if lines else 0 + mirrored = [] + for line in lines: + padded = line.ljust(max_len) + reversed_line = padded[::-1].translate(mirror_map) + mirrored.append(reversed_line) + return "\n".join(mirrored) + + +def _play_horse_easter_egg() -> None: + """Play the /horse Easter egg: fireworks, galloping horse, and a blessing.""" + cols = shutil.get_terminal_size((80, 24)).columns + rows = shutil.get_terminal_size((80, 24)).lines + + # -- Fireworks animation (~4 seconds at 8 fps = 32 frames) -- + firework_colors = [ + "\033[91m", # red + "\033[93m", # yellow + "\033[92m", # green + "\033[96m", # cyan + "\033[95m", # magenta + "\033[94m", # blue + ] + reset = "\033[0m" + particles_chars = ["*", ".", "o", "+", "x", "'", "`"] + + class Firework: + """A single firework burst with radial particles.""" + + def __init__(self, cx: int, cy: int, color: str, birth: int): + self.cx = cx + self.cy = cy + self.color = color + self.birth = birth + self.num = random.randint(12, 20) + self.angles = [random.uniform(0, 2 * math.pi) for _ in range(self.num)] + self.speeds = [random.uniform(0.5, 1.5) for _ in range(self.num)] + self.chars = [random.choice(particles_chars) for _ in range(self.num)] + + def particles(self, frame: int): + """Return list of (x, y, char) particle positions for the given frame.""" + age = frame - self.birth + if age < 0 or age > 10: + return [] + pts = [] + for i in range(self.num): + r = self.speeds[i] * age + px = self.cx + int(r * math.cos(self.angles[i]) * 2) # *2 for aspect ratio + py = self.cy + int(r * math.sin(self.angles[i])) + if 0 <= px < cols and 0 <= py < rows - 1: + pts.append((px, py, self.chars[i])) + return pts + + # Hide cursor + sys.stdout.write("\033[?25l") + sys.stdout.flush() + + try: + fireworks: list[Firework] = [] + total_frames = 32 + for f in range(total_frames): + # Spawn new fireworks periodically + if f % 4 == 0: + cx = random.randint(10, cols - 10) + cy = random.randint(2, rows // 2) + color = random.choice(firework_colors) + fireworks.append(Firework(cx, cy, color, f)) + + # Build frame buffer (blank) + buf: dict[tuple[int, int], tuple[str, str]] = {} + for fw in fireworks: + for px, py, ch in fw.particles(f): + buf[(px, py)] = (fw.color, ch) + + # Render + sys.stdout.write("\033[H\033[2J") # clear screen + for y in range(rows - 1): + line_parts: list[str] = [] + x = 0 + for x_pos in sorted(px for (px, py) in buf if py == y): + if x_pos >= x: + line_parts.append(" " * (x_pos - x)) + color, ch = buf[(x_pos, y)] + line_parts.append(f"{color}{ch}{reset}") + x = x_pos + 1 + sys.stdout.write("".join(line_parts) + "\n") + sys.stdout.flush() + time.sleep(1 / 8) # 8 fps + + # Prune old fireworks + fireworks.clear() + + # -- Horse ASCII art (bold yellow) -- + frame_1 = r""" + >>\. + /_ )`. + / _)`^)`. _.---. + (_,' \ `^---- `. + | | + \ / + / \ /___ / \ + / / | \ \ | +""" + frame_2 = r""" + >>\. + /_ )`. + / _)`^)`. _.---. + (_,' \ `^---- `. + | | + \ / + // / ___ / | + / / / | \ \ | +""" + + mirrored_1 = _mirror_frame(frame_1) + mirrored_2 = _mirror_frame(frame_2) + + bold_yellow = "\033[1;33m" + sys.stdout.write("\033[H\033[2J") # clear + + # Short galloping animation (8 cycles) + for i in range(8): + sys.stdout.write("\033[H\033[2J") + horse = mirrored_1 if i % 2 == 0 else mirrored_2 + indent = " " * (i * 3) + print("\n" * 3) + for line in horse.split("\n"): + if line.strip(): + print(f"{bold_yellow}{indent}{line}{reset}") + print(f"{bold_yellow}{'-' * min(i * 3 + 40, cols - 1)}{reset}") + sys.stdout.flush() + time.sleep(0.2) + + # -- Random blessing -- + blessings = [ + ("\u9a6c\u5230\u6210\u529f", "Succeed immediately"), + ("\u9f99\u9a6c\u7cbe\u795e", "Full of vitality"), + ("\u4e07\u9a6c\u5954\u817e", "Thousands of horses galloping"), + ("\u9a6c\u4e0d\u505c\u8e44", "Never stop striving"), + ("\u5feb\u9a6c\u52a0\u97ad", "Full speed ahead"), + ("\u4e00\u9a6c\u5f53\u5148", "Take the lead"), + ] + cn, en = random.choice(blessings) + print() + print(f"{bold_yellow} {cn} - {en}{reset}") + print(f"{bold_yellow} Happy Year of the Horse 2026!{reset}") + print() + + finally: + # Restore cursor + sys.stdout.write("\033[?25h") + sys.stdout.flush() diff --git a/reme/reme_cli.py b/reme/reme_cli.py index 6a429774..ee6021f0 100644 --- a/reme/reme_cli.py +++ b/reme/reme_cli.py @@ -1,11 +1,13 @@ """ReMe File System""" import asyncio +import os import sys from typing import AsyncGenerator from prompt_toolkit import PromptSession +from reme.core.op import BaseTool from .agent.chat import FsCli from .core.enumeration import ChunkEnum from .core.schema import StreamChunk @@ -20,42 +22,58 @@ from .tool.fs import ( WriteTool, ) from .tool.gallery import ExecuteCode -from .tool.search import DashscopeSearch +from .tool.search import DashscopeSearch, TavilySearch +from .horse import _play_horse_easter_egg class ReMeCli(ReMeFs): """ReMe Cli""" - def __init__(self, *args, **kwargs): + def __init__(self, *args, config_path: str = "cli", **kwargs): """Initialize ReMe with config.""" - super().__init__(*args, **kwargs) + super().__init__(*args, config_path=config_path, **kwargs) self.commands = { "/new": "Create a new conversation.", "/compact": "Compact messages into a summary.", "/exit": "Exit the application.", "/clear": "Clear the history.", "/help": "Show help.", + "/horse": "A surprise.", } + self.working_dir = self.service_config.working_dir - async def chat_with_remy(self, tool_result_max_size: int = 100, language: str = "zh", **kwargs): + async def chat_with_remy(self, tool_result_max_size: int = 100, **kwargs): """Interactive CLI chat with Remy using simple streaming output.""" + language = self.service_config.language + print(f"ReMe language={language or 'default'}") + tools: list[BaseTool] = [ + FsMemorySearch( + vector_weight=self.service_config.metadata["vector_weight"], + candidate_multiplier=self.service_config.metadata["candidate_multiplier"], + ), + BashTool(cwd=self.working_dir), + LsTool(cwd=self.working_dir), + ReadTool(cwd=self.working_dir), + EditTool(cwd=self.working_dir), + WriteTool(cwd=self.working_dir), + ExecuteCode(), + ] + tavily_api_key: str = os.getenv("TAVILY_API_KEY", "") + dashscope_api_key: str = os.getenv("DASHSCOPE_API_KEY", "") + if tavily_api_key: + tools.append(TavilySearch(name="web_search", language=language)) + print("find tavily_api_key, append Tavily search tool") + elif dashscope_api_key: + tools.append(DashscopeSearch(name="web_search", language=language)) + print("find dashscope_api_key, append Dashscope search tool") + else: + print("No Tavily or Dashscope API key found, skip Tavily and Dashscope search tool") + fs_cli = FsCli( - tools=[ - FsMemorySearch( - vector_weight=self.vector_weight, - candidate_multiplier=self.candidate_multiplier, - ), - BashTool(cwd=self.working_dir), - LsTool(cwd=self.working_dir), - ReadTool(cwd=self.working_dir), - EditTool(cwd=self.working_dir), - WriteTool(cwd=self.working_dir), - ExecuteCode(), - DashscopeSearch(), - ], - context_window_tokens=self.context_window_tokens, - reserve_tokens=self.reserve_tokens, - keep_recent_tokens=self.keep_recent_tokens, + tools=tools, + context_window_tokens=self.service_config.metadata["context_window_tokens"], + reserve_tokens=self.service_config.metadata["reserve_tokens"], + keep_recent_tokens=self.service_config.metadata["keep_recent_tokens"], working_dir=self.working_dir, language=language, **kwargs, @@ -98,7 +116,7 @@ class ReMeCli(ReMeFs): break if user_input == "/new": - result = await fs_cli.reset() + result = await fs_cli.new() print(f"{result}\nConversation reset\n") continue @@ -107,6 +125,11 @@ class ReMeCli(ReMeFs): print(f"{result}\nHistory compacted.\n") continue + if user_input == "/history": + result = fs_cli.format_history() + print(f"Formated History:\n{result}\n") + continue + if user_input == "/clear": fs_cli.messages.clear() print("History cleared.\n") @@ -118,6 +141,10 @@ class ReMeCli(ReMeFs): print(f" {command}: {description}") continue + if user_input == "/horse": + _play_horse_easter_egg() + continue + # Stream processing state in_thinking = False in_answer = False diff --git a/reme/reme_fs.py b/reme/reme_fs.py index ae597964..7a90adaa 100644 --- a/reme/reme_fs.py +++ b/reme/reme_fs.py @@ -2,6 +2,8 @@ from pathlib import Path +from loguru import logger + from .agent.fs import FsCompactor, FsContextChecker, FsSummarizer from .config import ReMeConfigParser from .core import Application @@ -76,18 +78,19 @@ class ReMeFs(Application): **kwargs, ) - self.context_window_tokens: int = context_window_tokens - self.reserve_tokens: int = reserve_tokens - self.keep_recent_tokens: int = keep_recent_tokens - self.vector_weight: float = vector_weight - self.candidate_multiplier: float = candidate_multiplier + self.service_config.metadata.setdefault("context_window_tokens", context_window_tokens) + self.service_config.metadata.setdefault("reserve_tokens", reserve_tokens) + self.service_config.metadata.setdefault("keep_recent_tokens", keep_recent_tokens) + self.service_config.metadata.setdefault("vector_weight", vector_weight) + self.service_config.metadata.setdefault("candidate_multiplier", candidate_multiplier) + logger.info(f"ReMe model_extra config: {self.service_config.metadata}") async def context_check(self, messages: list[Message | dict]) -> dict: """Check if messages exceed context limits.""" checker = FsContextChecker( - context_window_tokens=self.context_window_tokens, - reserve_tokens=self.reserve_tokens, - keep_recent_tokens=self.keep_recent_tokens, + context_window_tokens=self.service_config.metadata["context_window_tokens"], + reserve_tokens=self.service_config.metadata["reserve_tokens"], + keep_recent_tokens=self.service_config.metadata["keep_recent_tokens"], ) return await checker.call(messages=messages, service_context=self.service_context) @@ -147,8 +150,8 @@ class ReMeFs(Application): Search results as formatted string """ search_tool = FsMemorySearch( - vector_weight=self.vector_weight, - candidate_multiplier=self.candidate_multiplier, + vector_weight=self.service_config.metadata["vector_weight"], + candidate_multiplier=self.service_config.metadata["candidate_multiplier"], ) return await search_tool.call( query=query, @@ -177,8 +180,8 @@ class ReMeFs(Application): """Check if messages need compaction based on context window limits.""" messages = [Message(**message) if isinstance(message, dict) else message for message in messages] checker = FsContextChecker( - context_window_tokens=self.context_window_tokens, - reserve_tokens=self.reserve_tokens, + context_window_tokens=self.service_config.metadata["context_window_tokens"], + reserve_tokens=self.service_config.metadata["reserve_tokens"], ) result = await checker.call(messages=messages, service_context=self.service_context) return result["needs_compaction"] diff --git a/reme/tool/fs/fs_memory_search.py b/reme/tool/fs/fs_memory_search.py index 3edd344b..df8e561f 100644 --- a/reme/tool/fs/fs_memory_search.py +++ b/reme/tool/fs/fs_memory_search.py @@ -62,8 +62,16 @@ class FsMemorySearch(BaseFsTool): async def execute(self) -> str: """Execute the memory search operation.""" query: str = self.context.query.strip() - min_score = self.context.get("min_score", self.min_score) - max_results = self.context.get("max_results", self.max_results) + min_score: float = self.context.get("min_score", self.min_score) + max_results: int = self.context.get("max_results", self.max_results) + + assert query, "Query cannot be empty" + assert ( + isinstance(min_score, float) and 0.0 <= min_score <= 1.0 + ), f"min_score must be between 0 and 1, got {min_score}" + assert ( + isinstance(max_results, int) and max_results > 0 + ), f"max_results must be a positive integer, got {max_results}" # Use hybrid_search from memory_store results = await self.memory_store.hybrid_search( diff --git a/reme/tool/gallery/execute_code.py b/reme/tool/gallery/execute_code.py index 6cb77c85..262b7f2c 100644 --- a/reme/tool/gallery/execute_code.py +++ b/reme/tool/gallery/execute_code.py @@ -7,7 +7,7 @@ and return the output or error messages. from ...core.op import BaseTool from ...core.schema import ToolCall -from ...core.utils import exec_code +from ...core.utils import exec_code, async_exec_code class ExecuteCode(BaseTool): @@ -35,7 +35,7 @@ class ExecuteCode(BaseTool): ) async def execute(self): - self.execute_sync() + return await async_exec_code(self.context.code) def execute_sync(self): return exec_code(self.context.code) diff --git a/reme/tool/search/dashscope_search.yaml b/reme/tool/search/dashscope_search.yaml index 0dcef347..5a3dade5 100644 --- a/reme/tool/search/dashscope_search.yaml +++ b/reme/tool/search/dashscope_search.yaml @@ -10,11 +10,11 @@ role_prompt: | {query} # task - Extract the original content related to the user's query directly from the context, maintain accuracy, and avoid excessive processing. + Return all the original search results directly without processing. role_prompt_zh: | # 用户问题 {query} # task - 直接从上下文中提取与用户问题相关的原始内容，保持准确性，避免过度处理。 \ No newline at end of file + 直接返回所有的原始搜索结果，不要处理 \ No newline at end of file diff --git a/tests/test_execute_utils.py b/tests/test_execute_utils.py new file mode 100644 index 00000000..245e4f11 --- /dev/null +++ b/tests/test_execute_utils.py @@ -0,0 +1,138 @@ +"""Tests for reme.core.utils.execute_utils.""" + +import concurrent.futures + +import pytest + +from reme.core.utils import ( + async_exec_code, + exec_code, + run_shell_command, +) + + +# --------------------------------------------------------------------------- +# run_shell_command +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_run_shell_command_basic(): + """Basic shell command returns stdout and exit code 0.""" + stdout, _stderr, rc = await run_shell_command("echo hello") + assert stdout.strip() == "hello" + assert rc == 0 + + +@pytest.mark.asyncio +async def test_run_shell_command_stderr(): + """Shell command captures stderr output.""" + _stdout, stderr, rc = await run_shell_command("echo error >&2") + assert "error" in stderr + assert rc == 0 + + +@pytest.mark.asyncio +async def test_run_shell_command_nonzero_exit(): + """Shell command returns non-zero exit code.""" + _stdout, _stderr, rc = await run_shell_command("exit 42") + assert rc == 42 + + +@pytest.mark.asyncio +async def test_run_shell_command_timeout(): + """Shell command raises TimeoutError when exceeding timeout.""" + with pytest.raises(TimeoutError): + await run_shell_command("sleep 10", timeout=0.5) + + +# --------------------------------------------------------------------------- +# exec_code (sync) +# --------------------------------------------------------------------------- + + +def test_exec_code_basic(): + """exec_code captures print output.""" + result = exec_code("print('hello')") + assert result.strip() == "hello" + + +def test_exec_code_multiline(): + """exec_code handles multiline code.""" + code = "for i in range(3):\n print(i)" + result = exec_code(code) + assert result.strip() == "0\n1\n2" + + +def test_exec_code_exception_returns_message(): + """exec_code returns exception message on error.""" + result = exec_code("raise ValueError('boom')") + assert "boom" in result + + +def test_exec_code_no_output(): + """exec_code returns empty string when no output.""" + result = exec_code("x = 1 + 1") + assert result == "" + + +def test_exec_code_timeout(): + """exec_code raises TimeoutError when exceeding timeout.""" + with pytest.raises(TimeoutError, match="timed out"): + exec_code("import time; time.sleep(10)", timeout=0.5) + + +def test_exec_code_with_shared_executor(): + """exec_code works with a shared ThreadPoolExecutor.""" + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool: + r1 = exec_code("print('a')", executor=pool) + r2 = exec_code("print('b')", executor=pool) + assert r1.strip() == "a" + assert r2.strip() == "b" + + +def test_exec_code_no_timeout(): + """exec_code works with timeout=None.""" + result = exec_code("print('ok')", timeout=None) + assert result.strip() == "ok" + + +# --------------------------------------------------------------------------- +# async_exec_code +# --------------------------------------------------------------------------- + + +@pytest.mark.asyncio +async def test_async_exec_code_basic(): + """async_exec_code captures print output.""" + result = await async_exec_code("print('async hello')") + assert result.strip() == "async hello" + + +@pytest.mark.asyncio +async def test_async_exec_code_exception(): + """async_exec_code returns exception message on error.""" + result = await async_exec_code("raise RuntimeError('async boom')") + assert "async boom" in result + + +@pytest.mark.asyncio +async def test_async_exec_code_timeout(): + """async_exec_code raises TimeoutError when exceeding timeout.""" + with pytest.raises(TimeoutError, match="timed out"): + await async_exec_code("import time; time.sleep(10)", timeout=0.5) + + +@pytest.mark.asyncio +async def test_async_exec_code_with_executor(): + """async_exec_code works with a shared ThreadPoolExecutor.""" + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool: + result = await async_exec_code("print('pooled')", executor=pool) + assert result.strip() == "pooled" + + +@pytest.mark.asyncio +async def test_async_exec_code_no_timeout(): + """async_exec_code works with timeout=None.""" + result = await async_exec_code("print('no limit')", timeout=None) + assert result.strip() == "no limit" diff --git a/tests/test_horse.py b/tests/test_horse.py new file mode 100644 index 00000000..86f6c989 --- /dev/null +++ b/tests/test_horse.py @@ -0,0 +1,81 @@ +"""Interactive horse ASCII art animation demo.""" + +import os +import time + +from reme.horse import _mirror_frame + + +def clear_screen(): + """Clear the terminal screen.""" + os.system("cls" if os.name == "nt" else "clear") + + +def running_horse(): + """Run a galloping horse animation across the terminal.""" + # 定义两帧动画，模拟腿部动作 + frame_1 = r""" + >>\. + /_ )`. + / _)`^)`. _.---. + (_,' \ `^---- `. + | | + \ / + / \ /___ / \ + / / | \ \ | + """ + + frame_2 = r""" + >>\. + /_ )`. + / _)`^)`. _.---. + (_,' \ `^---- `. + | | + \ / + // / ___ / | + / / / | \ \ | + """ + + frame_1 = _mirror_frame(frame_1) + frame_2 = _mirror_frame(frame_2) + + frames = [frame_1, frame_2] + distance = 0 + + try: + while True: + # 1. 清理屏幕 + clear_screen() + + # 2. 获取当前帧（通过取余数在两帧之间切换） + current_frame = frames[distance % 2] + + # 3. 增加左侧空格，产生向右移动的效果 + indent = " " * distance + + # 4. 打印带缩进的每一行 + print("\n" * 5) # 顶部留白 + for line in current_frame.split("\n"): + # 只有非空行才打印，避免格式错乱 + if line.strip() != "": + print(indent + line) + else: + print() + + # 5. 打印地面 + print("-" * (distance + 40)) + + # 6. 更新距离并暂停 + distance += 1 + time.sleep(0.2) # 控制速度，0.2秒一帧 + + # 跑到屏幕边缘重置（可选） + if distance > 60: + distance = 0 + + except KeyboardInterrupt: + print("\n马儿休息了。(程序已停止)") + + +if __name__ == "__main__": + running_horse()