mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-11-02 07:13:50 +00:00
Merge 91208ffac1 into 23d4c96c15
This commit is contained in:
imrewce
GitHub
commit
09e2049cb4
402 changed files with 103002 additions and 0 deletions
284
benchmark/pi-bench/README.md
Normal file
284
benchmark/pi-bench/README.md
Normal file
|
|
@ -0,0 +1,284 @@
|
|||
# ReMe Benchmark on Pi-Bench
|
||||
|
||||
基于 Pi-Bench 框架的 ReMe Agent 评测套件。本目录包含所有必要代码，可独立运行 ReMe Agent 在 Pi-Bench researcher 角色上的完整评测。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
reme_benchmark/
|
||||
├── README.md # 本文件
|
||||
├── env.example.sh # 环境变量模板
|
||||
├── bridge_reme.py # ReMe Agent 桥接脚本
|
||||
├── run_with_reme.sh # 评测运行脚本
|
||||
├── fix_trace_logs.py # Trace 日志转换工具
|
||||
├── config/ # 配置文件
|
||||
│ ├── models/
|
||||
│ │ └── reme.yaml # ReMe 模型配置
|
||||
│ └── bench/
|
||||
│ └── evaluation/
|
||||
│ └── trace_history.yaml # 评测历史配置
|
||||
├── data/ # 评测数据
|
||||
│ └── researcher/ # Researcher 角色数据
|
||||
│ ├── profile.yaml # 用户画像
|
||||
│ ├── episode.yaml # Episode 定义
|
||||
│ ├── tools.yaml # 可用工具列表
|
||||
│ ├── scripts/ # 任务脚本
|
||||
│ └── tasks/ # 20 个评测任务
|
||||
├── src/ # Pi-Bench 核心框架
|
||||
│ ├── main.py # 主入口
|
||||
│ ├── runner/ # 评测运行器
|
||||
│ ├── evaluation/ # 评测模块
|
||||
│ ├── user_agent/ # 用户模拟 Agent
|
||||
│ ├── channels/ # 通信通道
|
||||
│ ├── llm/ # LLM 客户端
|
||||
│ ├── runtime/ # 运行时配置
|
||||
│ ├── data/ # 数据模型
|
||||
│ └── utils/ # 工具函数
|
||||
├── scripts/ # 辅助脚本
|
||||
│ ├── test_server.py # 测试服务器
|
||||
│ ├── setup_appworld.sh # AppWorld 安装脚本
|
||||
│ └── entrypoint.sh # Docker 入口脚本
|
||||
└── third_party/ # 第三方依赖
|
||||
├── appworld/ # AppWorld 环境
|
||||
└── nanobot/ # Nanobot Agent 框架
|
||||
```
|
||||
|
||||
## 环境要求
|
||||
|
||||
- Python >= 3.11
|
||||
- ReMe Agent 项目（需要单独克隆和配置）
|
||||
- AppWorld 环境（已包含在 `third_party/appworld/`）
|
||||
|
||||
## 安装步骤
|
||||
|
||||
### 1. 创建 Python 虚拟环境
|
||||
|
||||
```bash
|
||||
cd reme_benchmark
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
### 2. 安装 Pi-Bench 依赖
|
||||
|
||||
```bash
|
||||
pip install pyyaml loguru httpx
|
||||
|
||||
# 安装 AppWorld（本地开发版本）
|
||||
cd third_party/appworld
|
||||
pip install -e .
|
||||
python -m appworld.cli install --repo
|
||||
python -m appworld.cli download data --root .
|
||||
cd ../..
|
||||
```
|
||||
|
||||
### 3. 配置 ReMe Agent
|
||||
|
||||
ReMe Agent 需要单独配置。假设 ReMe 项目位于 `/path/to/ReMe`：
|
||||
|
||||
```bash
|
||||
# 确保 ReMe 项目已安装
|
||||
cd /path/to/ReMe
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 返回 reme_benchmark 目录
|
||||
cd -
|
||||
```
|
||||
|
||||
### 4. 配置环境变量
|
||||
|
||||
复制环境变量模板并填入你的 API 密钥：
|
||||
|
||||
```bash
|
||||
cp env.example.sh env.sh
|
||||
```
|
||||
|
||||
编辑 `env.sh`，填入以下 API 密钥：
|
||||
|
||||
- `USER_API_KEY`: 驱动模拟用户的 LLM API 密钥（用于判断隐藏意图）
|
||||
- `JUDGER_API_KEY`: 驱动评分裁判的 LLM API 密钥（用于评测打分）
|
||||
- `REME_LLM_API_KEY`: ReMe Agent 使用的 LLM API 密钥
|
||||
- `REME_DIR`: ReMe 项目的绝对路径（如 `/path/to/ReMe`）
|
||||
- `REME_WORKSPACE_DIR`: ReMe 工作目录（默认 `/tmp/reme_pibench_workspace`）
|
||||
|
||||
推荐使用 Dashscope（通义千问）API，默认配置已设置好 base_url。
|
||||
|
||||
## 运行评测
|
||||
|
||||
### 运行单个任务
|
||||
|
||||
```bash
|
||||
cd reme_benchmark
|
||||
source env.sh
|
||||
|
||||
./run_with_reme.sh --user-id researcher --task-id researcher_task_002
|
||||
```
|
||||
|
||||
### 运行所有任务
|
||||
|
||||
```bash
|
||||
./run_with_reme.sh --user-id researcher
|
||||
```
|
||||
|
||||
### 仅运行，不评测
|
||||
|
||||
```bash
|
||||
./run_with_reme.sh --user-id researcher --skip-eval
|
||||
```
|
||||
|
||||
## 评测流程
|
||||
|
||||
`run_with_reme.sh` 会自动执行以下步骤：
|
||||
|
||||
1. **启动 AppWorld API 服务**（端口 9000）
|
||||
- 提供模拟的 Web 应用 API（Amazon、Gmail、Todoist 等）
|
||||
|
||||
2. **启动 AppWorld MCP 服务**（端口 10000）
|
||||
- 提供 Model Context Protocol 接口
|
||||
- 允许 ReMe Agent 调用 AppWorld 工具
|
||||
|
||||
3. **启动 Test Server**（端口 9999）
|
||||
- 消息队列服务，协调 User Agent 和 ReMe Agent
|
||||
|
||||
4. **启动 ReMe Bridge**
|
||||
- 桥接脚本，连接 Test Server 和 ReMe Agent
|
||||
- 使用 ReMe 的 Python 虚拟环境运行
|
||||
- 加载用户画像和工具配置
|
||||
|
||||
5. **运行评测**
|
||||
- User Agent 按顺序发送任务
|
||||
- ReMe Agent 使用 AppWorld 工具完成任务
|
||||
- 记录完整的交互历史
|
||||
|
||||
6. **评测打分**
|
||||
- Checklist 评测：检查任务完成度
|
||||
- Proactiveness 评测：评估 Agent 主动性
|
||||
- 生成评测报告
|
||||
|
||||
## 输出结构
|
||||
|
||||
评测结果保存在 `outputs/reme/researcher/` 下：
|
||||
|
||||
```
|
||||
outputs/
|
||||
└── reme/
|
||||
└── researcher/
|
||||
├── researcher_task_001/
|
||||
│ ├── history/ # 交互历史
|
||||
│ │ ├── YYYYMMDD_HHMMSS-log.jsonl
|
||||
│ │ └── YYYYMMDD_HHMMSS-messages.jsonl
|
||||
│ ├── eval/
|
||||
│ │ ├── logs/ # 评测日志
|
||||
│ │ └── results/ # 评测结果
|
||||
│ │ └── YYYYMMDD_HHMMSS_result.json
|
||||
│ └── trace/ # Trace 历史（用于评测）
|
||||
├── researcher_task_002/
|
||||
└── ...
|
||||
```
|
||||
|
||||
每个任务的评测结果 JSON 包含：
|
||||
|
||||
- `overall_average_score`: 总体得分（0-1）
|
||||
- `overall_checklist_average_score`: Checklist 得分
|
||||
- `overall_proactiveness_average_score`: 主动性得分
|
||||
- `task.checklist`: Checklist 详细评分
|
||||
- `task.proactiveness`: Proactiveness 详细评分
|
||||
|
||||
## 评测任务说明
|
||||
|
||||
Researcher 角色包含 20 个评测任务，涵盖：
|
||||
|
||||
- **Task 001-005**: 基础任务（商品搜索、任务管理、邮件发送）
|
||||
- **Task 006-010**: 学术任务（论文检索、简历修改、Rebuttal 撰写）
|
||||
- **Task 011-015**: 复杂任务（多步骤工作流、跨应用协作）
|
||||
- **Task 016-020**: 高级任务（财务规划、音乐推荐、长期规划）
|
||||
|
||||
每个任务包含：
|
||||
|
||||
- `task.yaml`: 任务定义、初始输入、隐藏意图、评测标准
|
||||
- 可选的辅助文件（如简历、论文草稿、账单等）
|
||||
|
||||
## 常见问题
|
||||
|
||||
### 1. AppWorld 服务启动失败
|
||||
|
||||
```bash
|
||||
# 检查端口是否被占用
|
||||
lsof -i :9000
|
||||
lsof -i :10000
|
||||
lsof -i :9999
|
||||
|
||||
# 杀掉占用进程
|
||||
kill -9 <PID>
|
||||
```
|
||||
|
||||
### 2. ReMe Bridge 无法连接
|
||||
|
||||
确保 `env.sh` 中的 `REME_DIR` 路径正确，且 ReMe 项目已安装依赖：
|
||||
|
||||
```bash
|
||||
cd $REME_DIR
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 3. 评测超时
|
||||
|
||||
某些复杂任务可能需要较长时间。可以调整超时时间：
|
||||
|
||||
编辑 `config/models/reme.yaml`：
|
||||
|
||||
```yaml
|
||||
run:
|
||||
turn_timeout: 2400.0 # 增加到 40 分钟
|
||||
```
|
||||
|
||||
### 4. 重新评测
|
||||
|
||||
如果需要重新评测某个任务，先删除该任务的输出：
|
||||
|
||||
```bash
|
||||
rm -rf outputs/reme/researcher/researcher_task_002
|
||||
```
|
||||
|
||||
然后重新运行：
|
||||
|
||||
```bash
|
||||
./run_with_reme.sh --user-id researcher --task-id researcher_task_002
|
||||
```
|
||||
|
||||
## 技术架构
|
||||
|
||||
### 组件交互
|
||||
|
||||
```
|
||||
User Agent (LLM) → Test Server → ReMe Bridge → ReMe Agent
|
||||
↓ ↑ ↓
|
||||
判断隐藏意图 消息队列 调用 AppWorld MCP
|
||||
↓ ↑ ↓
|
||||
生成用户回复 ← 评测历史 ← 记录交互 ← 操作 Web 应用
|
||||
```
|
||||
|
||||
### 评测维度
|
||||
|
||||
1. **Checklist Score**（任务完成度）
|
||||
- 基于预定义的评测标准
|
||||
- 使用 LLM 判断每个标准是否满足
|
||||
- 加权平均计算总分
|
||||
|
||||
2. **Proactiveness Score**（主动性）
|
||||
- 评估 Agent 是否主动发现隐藏意图
|
||||
- 评估 Agent 是否主动追问
|
||||
- 评估交互效率（轮数、消息长度）
|
||||
|
||||
## 许可证
|
||||
|
||||
本评测套件基于 Pi-Bench 框架。相关许可证请参见原项目。
|
||||
|
||||
## 参考
|
||||
|
||||
- Pi-Bench 原项目：https://github.com/your-org/pi-bench
|
||||
- ReMe Agent：https://github.com/your-org/reme
|
||||
- AppWorld：https://github.com/stanfordnlp/appworld
|
||||
344
benchmark/pi-bench/README_NANOBOT_REME.md
Normal file
344
benchmark/pi-bench/README_NANOBOT_REME.md
Normal file
|
|
@ -0,0 +1,344 @@
|
|||
# NanoBot + ReMe Memory Pi-Bench Evaluation
|
||||
|
||||
This directory contains a standalone benchmark setup that evaluates a **NanoBot agent enhanced with ReMe cross-session memory** on the Pi-Bench personal assistant benchmark.
|
||||
|
||||
## Overview
|
||||
|
||||
The system combines three components:
|
||||
|
||||
1. **NanoBot** (`$HOME/nanobot/`) - The agent framework that executes tasks using LLM reasoning and tool use
|
||||
2. **ReMe Memory** (`$HOME/ReMe/`) - Cross-session memory system that persists context across tasks
|
||||
3. **Pi-Bench** (this directory) - Evaluation framework with AppWorld environment and test server
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Pi-Bench Test Server │
|
||||
│ (orchestrates evaluation) │
|
||||
└──────────────┬─────────────────────────────────┬────────────┘
|
||||
│ poll/send │ evaluation
|
||||
▼ ▼
|
||||
┌──────────────────────────┐ ┌──────────────────────┐
|
||||
│ bridge_nanobot_reme.py │ │ Pi-Bench Evaluator │
|
||||
│ │ │ │
|
||||
│ ┌────────────────────┐ │ │ - Checklist scoring │
|
||||
│ │ NanoBot Agent │ │ │ - Proactiveness │
|
||||
│ │ (SDK/programmatic)│ │ │ - Memory impact │
|
||||
│ └─────────┬──────────┘ │ └──────────────────────┘
|
||||
│ │ │
|
||||
│ ┌─────────▼──────────┐ │
|
||||
│ │ ReMe Memory │ │
|
||||
│ │ (search/save) │ │
|
||||
│ └────────────────────┘ │
|
||||
└──────────────┬────────────┘
|
||||
│ MCP (SSE)
|
||||
▼
|
||||
┌──────────────────────────┐
|
||||
│ AppWorld Environment │
|
||||
│ (Amazon, Gmail, etc.) │
|
||||
└──────────────────────────┘
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
reme_benchmark/
|
||||
├── README_NANOBOT_REME.md # This file
|
||||
├── bridge_nanobot_reme.py # Bridge: NanoBot ↔ Test Server ↔ ReMe Memory
|
||||
├── run_with_nanobot_reme.sh # Orchestration script
|
||||
├── env.nanobot_reme.example.sh # Environment template
|
||||
├── env.nanobot_reme.sh # Your local config (create from template)
|
||||
├── config/
|
||||
│ ├── models/
|
||||
│ │ └── nanobot_reme.yaml # Pi-Bench model config
|
||||
│ └── bench/
|
||||
│ └── evaluation/
|
||||
│ └── trace_history.yaml # Evaluation settings
|
||||
├── data/
|
||||
│ └── researcher/ # User persona and tasks
|
||||
│ ├── profile.yaml
|
||||
│ ├── episode.yaml
|
||||
│ └── tasks/ # 20 researcher tasks
|
||||
├── src/ # Pi-Bench framework (copied)
|
||||
│ ├── runner/
|
||||
│ ├── evaluation/
|
||||
│ ├── channels/
|
||||
│ ├── user_agent/
|
||||
│ └── ...
|
||||
├── scripts/ # Test server and setup scripts
|
||||
└── third_party/
|
||||
├── appworld/ # AppWorld environment
|
||||
└── nanobot/ # NanoBot framework (copied, optional)
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### 1. Install NanoBot
|
||||
|
||||
```bash
|
||||
cd $HOME/nanobot
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### 2. Install ReMe
|
||||
|
||||
```bash
|
||||
cd $HOME/ReMe
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### 3. Install Pi-Bench Dependencies
|
||||
|
||||
```bash
|
||||
cd $HOME/pi-bench/reme_benchmark
|
||||
pip install -r requirements.txt # If exists, or:
|
||||
pip install httpx pyyaml python-dotenv loguru
|
||||
```
|
||||
|
||||
### 4. Install AppWorld
|
||||
|
||||
```bash
|
||||
cd $HOME/pi-bench/reme_benchmark/third_party/appworld
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### 5. API Keys
|
||||
|
||||
You need API keys for:
|
||||
- **Dashscope** (for NanoBot agent LLM): `DASHSCOPE_API_KEY`
|
||||
- **Dashscope** (for User Agent): `USER_API_KEY`
|
||||
- **Dashscope** (for Judger): `JUDGER_API_KEY`
|
||||
- **ReMe** (for memory LLM): configured in ReMe's `.env`
|
||||
|
||||
## Setup
|
||||
|
||||
### Step 1: Create Environment Config
|
||||
|
||||
```bash
|
||||
cd $HOME/pi-bench/reme_benchmark
|
||||
cp env.nanobot_reme.example.sh env.nanobot_reme.sh
|
||||
```
|
||||
|
||||
Edit `env.nanobot_reme.sh` and fill in:
|
||||
|
||||
```bash
|
||||
# Required: ReMe and NanoBot paths
|
||||
export REME_DIR="$HOME/ReMe"
|
||||
export NANOBOT_DIR="$HOME/nanobot"
|
||||
|
||||
# Required: LLM API keys
|
||||
export MODEL_NAME="qwen3.6-plus"
|
||||
export DASHSCOPE_API_KEY="sk-..."
|
||||
export USER_API_KEY="sk-..."
|
||||
export JUDGER_API_KEY="sk-..."
|
||||
```
|
||||
|
||||
### Step 2: Configure ReMe Memory
|
||||
|
||||
Ensure ReMe is configured with its own LLM for memory operations:
|
||||
|
||||
```bash
|
||||
cd $HOME/ReMe
|
||||
# Edit .env file with REME_LLM_API_KEY or similar
|
||||
```
|
||||
|
||||
ReMe's memory system needs:
|
||||
- An LLM for memory consolidation (can be same as agent LLM)
|
||||
- Embedding model for semantic search
|
||||
- Vector store configuration
|
||||
|
||||
### Step 3: Verify AppWorld Setup
|
||||
|
||||
```bash
|
||||
cd $HOME/pi-bench/reme_benchmark/third_party/appworld
|
||||
python -c "import appworld; print('AppWorld OK')"
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Run Single Task
|
||||
|
||||
```bash
|
||||
cd $HOME/pi-bench/reme_benchmark
|
||||
./run_with_nanobot_reme.sh --user-id researcher --task-id researcher_task_002
|
||||
```
|
||||
|
||||
### Run Multiple Tasks
|
||||
|
||||
```bash
|
||||
./run_with_nanobot_reme.sh --user-id researcher --task-ids "researcher_task_001,researcher_task_002,researcher_task_003"
|
||||
```
|
||||
|
||||
### Run All Tasks
|
||||
|
||||
```bash
|
||||
./run_with_nanobot_reme.sh --user-id researcher
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### Bridge Flow
|
||||
|
||||
1. **Initialization**
|
||||
- Bridge loads environment config
|
||||
- Initializes ReMe Application (memory system)
|
||||
- Initializes NanoBot SDK with MCP config for AppWorld
|
||||
- Loads user profile from `data/researcher/profile.yaml`
|
||||
|
||||
2. **Message Processing**
|
||||
- Bridge polls Test Server for user messages
|
||||
- On first message in session:
|
||||
- Waits for pending memory saves from previous sessions
|
||||
- Searches ReMe memory for relevant context
|
||||
- Prepends memory context to user message
|
||||
- Forwards message to NanoBot agent
|
||||
- NanoBot reasons with LLM and uses AppWorld MCP tools
|
||||
- Bridge extracts reply and sends back to Test Server
|
||||
|
||||
3. **Session Reset**
|
||||
- When Test Server signals task completion (reset)
|
||||
- Bridge schedules non-blocking memory save of conversation
|
||||
- Clears NanoBot session for cross-task isolation
|
||||
- Memory save runs in background to avoid timeout
|
||||
|
||||
4. **Memory Integration**
|
||||
- **Search**: Before first message, queries ReMe for relevant past context
|
||||
- **Save**: On reset, saves conversation as ReMe daily memory
|
||||
- **Persistence**: Memory persists across task boundaries via ReMe's vector store
|
||||
|
||||
### Memory Context Injection
|
||||
|
||||
When memory is found, it's prepended to the user message:
|
||||
|
||||
```
|
||||
[Relevant memories from previous sessions]
|
||||
<memory search results>
|
||||
|
||||
[Current user message]
|
||||
<actual user message>
|
||||
```
|
||||
|
||||
This allows NanoBot to reference past context without explicit memory tools.
|
||||
|
||||
## Output
|
||||
|
||||
Results are saved to:
|
||||
|
||||
```
|
||||
outputs/nanobot_reme/
|
||||
└── researcher/
|
||||
├── researcher_task_002/
|
||||
│ ├── history/
|
||||
│ │ └── 20260730_143022-messages.jsonl
|
||||
│ └── result.json
|
||||
└── ...
|
||||
```
|
||||
|
||||
Each task produces:
|
||||
- **Conversation history**: Full message transcript
|
||||
- **Result**: Checklist scores, proactiveness metrics, memory impact
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### NanoBot Initialization Fails
|
||||
|
||||
```
|
||||
ERROR: Failed to initialize NanoBot: ...
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Check `NANOBOT_DIR` path exists
|
||||
- Verify nanobot is installed: `pip show nanobot-ai`
|
||||
- Check config generation: look for `nanobot_config.json` in workspace
|
||||
|
||||
### ReMe Memory Not Working
|
||||
|
||||
```
|
||||
WARNING: ReMe dir not set, memory features disabled
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Set `REME_DIR` in `env.nanobot_reme.sh`
|
||||
- Verify ReMe is installed: `pip show reme`
|
||||
- Check ReMe `.env` file has required API keys
|
||||
- Test ReMe standalone: `cd $REME_DIR && python -c "from reme.application import Application"`
|
||||
|
||||
### AppWorld MCP Connection Fails
|
||||
|
||||
```
|
||||
ERROR: MCP connection failed: ...
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Verify AppWorld MCP server is running: `curl http://localhost:10000/mcp`
|
||||
- Check port 10000 is not blocked: `lsof -i :10000`
|
||||
- Review logs: `tail -50 /tmp/appworld_mcp_nanobot.log`
|
||||
|
||||
### Bridge Hangs
|
||||
|
||||
```
|
||||
Bridge is processing tasks...
|
||||
(no further output)
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Check bridge log: `tail -100 /tmp/bridge_nanobot_reme.log`
|
||||
- Verify Test Server is running: `curl http://localhost:9999/poll?timeout=1`
|
||||
- Check for errors in Test Server log: `tail -50 /tmp/test_server_nanobot.log`
|
||||
|
||||
## Customization
|
||||
|
||||
### Change LLM Model
|
||||
|
||||
Edit `env.nanobot_reme.sh`:
|
||||
|
||||
```bash
|
||||
export MODEL_NAME="qwen-max" # or "gpt-5.4", "claude-opus-4-7", etc.
|
||||
export MODEL_BASE_URL="https://api.openai.com/v1" # if using OpenAI
|
||||
export MODEL_API_KEY="sk-..."
|
||||
```
|
||||
|
||||
### Disable ReMe Memory
|
||||
|
||||
Set `REME_DIR` to empty string in `env.nanobot_reme.sh`:
|
||||
|
||||
```bash
|
||||
export REME_DIR=""
|
||||
```
|
||||
|
||||
Bridge will run without memory features (no search/save).
|
||||
|
||||
### Adjust Memory Search Limit
|
||||
|
||||
Edit `bridge_nanobot_reme.py`, line ~350:
|
||||
|
||||
```python
|
||||
response = await self.search_job(query=query, limit=10) # Change from 5 to 10
|
||||
```
|
||||
|
||||
## Comparison with Pure ReMe Bridge
|
||||
|
||||
| Feature | bridge_reme.py | bridge_nanobot_reme.py |
|
||||
|---------|----------------|------------------------|
|
||||
| Agent Framework | ReMe (AgentScope) | NanoBot (SDK) |
|
||||
| Memory System | ReMe (native) | ReMe (integrated) |
|
||||
| MCP Tools | ReMe MCP client | NanoBot MCP config |
|
||||
| Tool Execution | AgentScope wrapper | NanoBot tool registry |
|
||||
| Session Management | AgentScope resume | NanoBot session_key |
|
||||
| Multi-turn | Yes (resume) | Yes (session_key) |
|
||||
| Cross-session Memory | Yes | Yes |
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Memory Search**: Limit to 5 results to avoid context overflow
|
||||
2. **Tool Timeout**: Set to 60s for AppWorld MCP tools
|
||||
3. **Session Isolation**: Clear session_key on reset to prevent cross-task contamination
|
||||
4. **Non-blocking Saves**: Memory saves run in background to avoid timeout on reset
|
||||
|
||||
## References
|
||||
|
||||
- NanoBot: https://github.com/nanobot-ai/nanobot
|
||||
- ReMe: Cross-session memory system
|
||||
- Pi-Bench: Personal assistant evaluation framework
|
||||
- AppWorld: Realistic app environment for agents
|
||||
501
benchmark/pi-bench/RUNBOOK.md
Normal file
501
benchmark/pi-bench/RUNBOOK.md
Normal file
|
|
@ -0,0 +1,501 @@
|
|||
# Pi-Bench 测评操作手册 (RUNBOOK)
|
||||
|
||||
> 本文档记录 **ReMe-only** 和 **NanoBot+ReMe** 两套测评方案的完整操作流程。
|
||||
> 最后更新: 2026-08-03
|
||||
|
||||
---
|
||||
|
||||
## 1. 架构总览
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Pi-Bench Runner │
|
||||
│ src/main.py --model-config config/models/XXX.yaml │
|
||||
│ 模式: run(发送任务) / eval(评估打分) │
|
||||
│ venv: $HOME/pi-bench/.venv │
|
||||
└────────────┬────────────────────────────────────┬────────────┘
|
||||
│ GET /poll │ POST /send
|
||||
▼ ▼
|
||||
┌─────────────────────────┐ ┌────────────────────────┐
|
||||
│ Test Server :9999 │◄─────────│ Bridge 进程 │
|
||||
│ scripts/test_server.py │ /send │ (二选一) │
|
||||
│ 纯 Python HTTP Server │ │ │
|
||||
└─────────────────────────┘ │ A) bridge_reme.py │
|
||||
│ venv: pi-bench │
|
||||
│ ReMe AgentScope │
|
||||
│ │
|
||||
│ B) bridge_nanobot_ │
|
||||
│ reme.py │
|
||||
│ venv: nanobot-main │
|
||||
│ NanoBot SDK + ReMe │
|
||||
└────────┬───────────────┘
|
||||
│ MCP (streamableHttp)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ AppWorld 服务 │
|
||||
│ API Server :9000 (appworld serve apis) │
|
||||
│ MCP Server :10000 (appworld serve mcp http) │
|
||||
│ venv: $HOME/pi-bench/.venv │
|
||||
│ 源码: $HOME/pi-bench/third_party/appworld │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 1.1 组件角色
|
||||
|
||||
| 组件 | 作用 | Python venv |
|
||||
|------|------|-------------|
|
||||
| **Pi-Bench Runner** | 读取 task.yaml → 通过 Test Server 发送用户消息 → 收集回复 → 评估打分 | `pi-bench/.venv` |
|
||||
| **Test Server** | 简单的消息队列 HTTP 服务，bridge 通过长轮询获取消息、发送回复 | `pi-bench/.venv` |
|
||||
| **Bridge (ReMe)** | 轮询 Test Server → 转发给 ReMe AgentScope agent → 返回回复 → 管理 ReMe 记忆 | `pi-bench/.venv` |
|
||||
| **Bridge (NanoBot+ReMe)** | 轮询 Test Server → 转发给 NanoBot SDK → 返回回复 → 管理 ReMe 记忆 | `nanobot-main/.venv` |
|
||||
| **AppWorld API** | 提供 Amazon/Gmail/Spotify/Todoist/Venmo/FileSystem 的 REST API | `pi-bench/.venv` |
|
||||
| **AppWorld MCP** | 将 AppWorld API 包装为 MCP 协议，供 agent 调用 | `pi-bench/.venv` |
|
||||
| **ReMe** | 跨会话记忆系统，在 session 结束时保存、新 session 开始时检索 | 由 bridge 内嵌调用 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 目录结构
|
||||
|
||||
```
|
||||
$HOME/pi-bench/
|
||||
├── .venv/ # Pi-Bench 主 venv (runner + appworld + reme bridge)
|
||||
├── third_party/appworld/ # AppWorld 源码
|
||||
├── env.sh # 主环境配置 (ReMe agent)
|
||||
├── env.reme.sh # ReMe-only 环境
|
||||
├── run_with_reme.sh # ReMe-only 一键启动 (顶层)
|
||||
│
|
||||
└── reme_benchmark/ # ★ 测评工作目录
|
||||
├── RUNBOOK.md # ← 本文件
|
||||
│
|
||||
├── env.nanobot_reme.sh # NanoBot+ReMe 环境变量 (API keys)
|
||||
├── env.nanobot_reme.example.sh # 模板
|
||||
├── env.example.sh # ReMe-only 环境模板
|
||||
│
|
||||
├── config/models/
|
||||
│ ├── nanobot_reme.yaml # NanoBot+ReMe 模型配置
|
||||
│ └── reme.yaml # ReMe-only 模型配置
|
||||
│
|
||||
├── bridge_reme.py # ReMe-only bridge (AgentScope)
|
||||
├── bridge_nanobot_reme.py # NanoBot+ReMe bridge (NanoBot SDK)
|
||||
│
|
||||
├── run_persona_nanobot_reme.sh # ★ 单 persona 通用脚本 (NanoBot+ReMe)
|
||||
├── run_all_personas.sh # ★ 5 persona 批跑脚本 (NanoBot+ReMe)
|
||||
├── run_batch_nanobot_reme.sh # 批跑备选方案
|
||||
├── run_with_nanobot_reme.sh # 完整启动含服务 (NanoBot+ReMe)
|
||||
├── run_with_reme.sh # 完整启动含服务 (ReMe-only)
|
||||
│
|
||||
├── run_researcher.sh # → exec run_persona_nanobot_reme.sh researcher
|
||||
├── run_Financier.sh # → exec run_persona_nanobot_reme.sh Financier
|
||||
├── run_law_trainee.sh # → exec run_persona_nanobot_reme.sh law_trainee
|
||||
├── run_marketer.sh # → exec run_persona_nanobot_reme.sh marketer
|
||||
├── run_pharmacist.sh # → exec run_persona_nanobot_reme.sh pharmacist
|
||||
│
|
||||
├── fix_trace_logs.py # 将 outputs 转换为 eval 需要的 trace_logs 格式
|
||||
│
|
||||
├── data/ # 5 个 persona 数据
|
||||
│ ├── researcher/
|
||||
│ │ ├── profile.yaml # 用户画像
|
||||
│ │ ├── episode.yaml # 任务编排
|
||||
│ │ ├── tools.yaml # AppWorld 工具配置
|
||||
│ │ ├── tasks/ # 20 个 task (researcher_task_001~020)
|
||||
│ │ └── scripts/ # 评估脚本
|
||||
│ ├── Financier/
|
||||
│ ├── law_trainee/
|
||||
│ ├── marketer/
|
||||
│ └── pharmacist/
|
||||
│
|
||||
├── scripts/
|
||||
│ ├── test_server.py # Test Server (:9999)
|
||||
│ ├── entrypoint.sh # Docker 入口 (参考用)
|
||||
│ ├── setup_appworld.sh # AppWorld 安装脚本
|
||||
│ └── trace_viewer.py # Trace 日志查看器
|
||||
│
|
||||
├── src/ # Runner 核心代码 (从 pi-bench/src 适配)
|
||||
│ ├── main.py # CLI 入口
|
||||
│ ├── runtime.py # 运行时配置
|
||||
│ └── docker_launcher.py # Docker 启动器
|
||||
│
|
||||
├── outputs/ # 运行结果
|
||||
│ └── nanobot_reme/{persona}/ # 按 persona 组织
|
||||
│
|
||||
└── .reme/ # ReMe 记忆存储
|
||||
├── daily/ # 每日记忆
|
||||
├── metadata/ # 索引元数据
|
||||
├── session/ # 会话状态
|
||||
└── mem_session/ # 记忆会话
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 环境依赖
|
||||
|
||||
### 3.1 Python venv 路径
|
||||
|
||||
| venv | 路径 | 用途 |
|
||||
|------|------|------|
|
||||
| **pi-bench** | `$HOME/pi-bench/.venv` | Runner, AppWorld, ReMe bridge |
|
||||
| **nanobot** | `$HOME/nanobot/.venv` | NanoBot bridge |
|
||||
| **reme** | `$HOME/ReMe/.venv` | ReMe SDK (被 bridge 内嵌调用) |
|
||||
|
||||
### 3.2 外部源码路径
|
||||
|
||||
| 项目 | 路径 |
|
||||
|------|------|
|
||||
| **Pi-Bench** | `$HOME/pi-bench` |
|
||||
| **NanoBot** | `$HOME/nanobot` |
|
||||
| **ReMe** | `$HOME/ReMe` |
|
||||
| **AppWorld** | `$HOME/pi-bench/third_party/appworld` |
|
||||
|
||||
### 3.3 端口分配
|
||||
|
||||
| 端口 | 服务 | 协议 |
|
||||
|------|------|------|
|
||||
| **9000** | AppWorld API Server | REST (HTTP) |
|
||||
| **10000** | AppWorld MCP Server | MCP (streamableHttp) |
|
||||
| **9999** | Test Server | HTTP (poll/send/inject) |
|
||||
|
||||
### 3.4 API Keys
|
||||
|
||||
| 变量 | 用途 | 值 |
|
||||
|------|------|-----|
|
||||
| `DASHSCOPE_API_KEY` | 通义千问 API (LLM 推理) | 在 `env.nanobot_reme.sh` 中配置 |
|
||||
| `MODEL_NAME` | LLM 模型名 | `qwen3.6-plus` |
|
||||
| `MODEL_BASE_URL` | Dashscope 兼容端点 | `https://dashscope.aliyuncs.com/compatible-mode/v1` |
|
||||
| `BRAVE_SEARCH_API_KEY` | 网络搜索 (当前为 dummy) | `dummy` |
|
||||
| `USER_API_KEY` | 用户模拟 LLM | 同 DASHSCOPE_API_KEY |
|
||||
| `JUDGER_API_KEY` | 评估打分 LLM | 同 DASHSCOPE_API_KEY |
|
||||
|
||||
### 3.5 日志文件
|
||||
|
||||
| 日志 | 路径 |
|
||||
|------|------|
|
||||
| AppWorld API | `/tmp/nanobot_reme_appworld_api.log` |
|
||||
| AppWorld MCP | `/tmp/nanobot_reme_appworld_mcp.log` |
|
||||
| Test Server | `/tmp/nanobot_reme_test_server.log` |
|
||||
| Bridge | `/tmp/nanobot_reme_bridge.log` |
|
||||
| Run (per persona) | `/tmp/nanobot_reme_run_{persona}.log` |
|
||||
| Eval (per persona) | `/tmp/nanobot_reme_eval_{persona}.log` |
|
||||
|
||||
---
|
||||
|
||||
## 4. 启动流程 (NanoBot + ReMe)
|
||||
|
||||
### 4.0 前置检查
|
||||
|
||||
```bash
|
||||
# 检查 venv 存在
|
||||
ls $HOME/pi-bench/.venv/bin/python
|
||||
ls $HOME/nanobot/.venv/bin/python
|
||||
|
||||
# 检查无残留进程
|
||||
pgrep -fl "appworld|test_server|bridge_nanobot|bridge_reme"
|
||||
# 如有残留:
|
||||
pkill -f "appworld serve" 2>/dev/null || true
|
||||
pkill -f "test_server.py" 2>/dev/null || true
|
||||
pkill -f "bridge_nanobot_reme" 2>/dev/null || true
|
||||
pkill -f "bridge_reme" 2>/dev/null || true
|
||||
sleep 2
|
||||
|
||||
# 清理 ReMe 记忆 (每个 persona 前执行)
|
||||
rm -rf $HOME/pi-bench/reme_benchmark/.reme/metadata
|
||||
rm -rf $HOME/pi-bench/reme_benchmark/.reme/session
|
||||
rm -rf $HOME/pi-bench/reme_benchmark/.reme/mem_session
|
||||
```
|
||||
|
||||
### 4.1 方式 A: 手动分步启动 (推荐调试时使用)
|
||||
|
||||
**⚠️ 关键规则:**
|
||||
- **禁止使用 `nohup`** (会导致 SIGHUP 异常)
|
||||
- 使用 `command > log 2>&1 &` 后台运行
|
||||
- Bridge 必须添加 `signal.signal(signal.SIGHUP, signal.SIG_IGN)` (已在代码中)
|
||||
- 所有 URL 使用 `127.0.0.1` 而非 `localhost` (NanoBot SSRF 会拦截 localhost 的 IPv6 ::1)
|
||||
- 每个服务在**单独的终端窗口**中启动
|
||||
|
||||
#### Step 1: 加载环境变量
|
||||
```bash
|
||||
cd $HOME/pi-bench/reme_benchmark
|
||||
source env.nanobot_reme.sh
|
||||
```
|
||||
|
||||
#### Step 2: 启动 AppWorld API (:9000)
|
||||
```bash
|
||||
cd $HOME/pi-bench
|
||||
source .venv/bin/activate
|
||||
cd third_party/appworld
|
||||
appworld serve apis --root . --port 9000 > /tmp/nanobot_reme_appworld_api.log 2>&1 &
|
||||
|
||||
# 等待就绪 (最多 180 秒)
|
||||
for i in $(seq 1 90); do
|
||||
python3 -c "import socket;s=socket.socket();s.settimeout(2);s.connect(('127.0.0.1',9000));s.close()" 2>/dev/null && echo "✓ API ready" && break
|
||||
sleep 2
|
||||
done
|
||||
```
|
||||
|
||||
#### Step 3: 启动 AppWorld MCP (:10000)
|
||||
```bash
|
||||
# (同一终端, 已 activate)
|
||||
cd $HOME/pi-bench/third_party/appworld
|
||||
appworld serve mcp http \
|
||||
--app-names amazon,gmail,spotify,todoist,venmo,file_system \
|
||||
--output-type both \
|
||||
--remote-apis-url http://127.0.0.1:9000 \
|
||||
--port 10000 > /tmp/nanobot_reme_appworld_mcp.log 2>&1 &
|
||||
|
||||
# 等待就绪
|
||||
for i in $(seq 1 90); do
|
||||
python3 -c "import socket;s=socket.socket();s.settimeout(2);s.connect(('127.0.0.1',10000));s.close()" 2>/dev/null && echo "✓ MCP ready" && break
|
||||
sleep 2
|
||||
done
|
||||
```
|
||||
|
||||
#### Step 4: 启动 Test Server (:9999)
|
||||
```bash
|
||||
cd $HOME/pi-bench
|
||||
source .venv/bin/activate
|
||||
PORT=9999 python reme_benchmark/scripts/test_server.py > /tmp/nanobot_reme_test_server.log 2>&1 &
|
||||
sleep 3
|
||||
echo "✓ Test Server ready"
|
||||
```
|
||||
|
||||
#### Step 5: 启动 Bridge (NanoBot+ReMe)
|
||||
```bash
|
||||
# ⚠️ 使用 nanobot 的 venv!
|
||||
cd $HOME/pi-bench/reme_benchmark
|
||||
source env.nanobot_reme.sh
|
||||
|
||||
$HOME/nanobot/.venv/bin/python bridge_nanobot_reme.py \
|
||||
--test-server-url "http://127.0.0.1:9999" \
|
||||
--appworld-mcp-url "http://127.0.0.1:10000/mcp" \
|
||||
--nanobot-dir "$HOME/nanobot" \
|
||||
--reme-dir "$HOME/ReMe" \
|
||||
--data-root "data" \
|
||||
--user-id "researcher" \
|
||||
--model-name "qwen3.6-plus" \
|
||||
--model-base-url "https://dashscope.aliyuncs.com/compatible-mode/v1" \
|
||||
--model-api-key "${DASHSCOPE_API_KEY}" \
|
||||
> /tmp/nanobot_reme_bridge.log 2>&1 &
|
||||
|
||||
sleep 8
|
||||
# 检查 bridge 存活
|
||||
if kill -0 $! 2>/dev/null; then echo "✓ Bridge ready (PID: $!)"; else echo "✗ Bridge died"; tail -30 /tmp/nanobot_reme_bridge.log; fi
|
||||
```
|
||||
|
||||
#### Step 6: 运行评测 (Pi-Bench Runner)
|
||||
```bash
|
||||
# 切回 pi-bench venv
|
||||
cd $HOME/pi-bench
|
||||
source .venv/bin/activate
|
||||
cd reme_benchmark
|
||||
|
||||
# Run phase
|
||||
python -m src.main \
|
||||
--model-config config/models/nanobot_reme.yaml \
|
||||
--mode run \
|
||||
--user-id researcher \
|
||||
2>&1 | tee /tmp/nanobot_reme_run_researcher.log
|
||||
|
||||
# 等待记忆保存
|
||||
sleep 5
|
||||
|
||||
# 转换 trace logs
|
||||
python fix_trace_logs.py
|
||||
|
||||
# Eval phase
|
||||
python -m src.main \
|
||||
--model-config config/models/nanobot_reme.yaml \
|
||||
--mode eval \
|
||||
--user-id researcher \
|
||||
2>&1 | tee /tmp/nanobot_reme_eval_researcher.log
|
||||
```
|
||||
|
||||
### 4.2 方式 B: 单 Persona 一键脚本
|
||||
|
||||
```bash
|
||||
cd $HOME/pi-bench/reme_benchmark
|
||||
bash run_researcher.sh
|
||||
# 或: bash run_persona_nanobot_reme.sh researcher
|
||||
|
||||
# 带参数:
|
||||
bash run_researcher.sh --skip-services # 跳过服务启动 (已运行时)
|
||||
bash run_researcher.sh --skip-eval # 跳过评估
|
||||
```
|
||||
|
||||
### 4.3 方式 C: 5 Persona 批跑
|
||||
|
||||
```bash
|
||||
cd $HOME/pi-bench/reme_benchmark
|
||||
bash run_all_personas.sh
|
||||
```
|
||||
|
||||
该脚本会:
|
||||
1. 一次性启动 AppWorld + TestServer
|
||||
2. 遍历 5 个 persona (researcher → Financier → law_trainee → marketer → pharmacist)
|
||||
3. 每个 persona 前清空 ReMe 记忆 + 重启 Bridge (加载不同 profile)
|
||||
4. persona 内 20 个 task 按 episode 顺序执行，memory 跨 task 积累
|
||||
|
||||
---
|
||||
|
||||
## 5. 启动流程 (ReMe-only, 对照参考)
|
||||
|
||||
与 NanoBot+ReMe 的区别:
|
||||
|
||||
| 项目 | ReMe-only | NanoBot+ReMe |
|
||||
|------|-----------|--------------|
|
||||
| Bridge | `bridge_reme.py` | `bridge_nanobot_reme.py` |
|
||||
| Bridge venv | `pi-bench/.venv` | `nanobot-main/.venv` |
|
||||
| Model config | `config/models/reme.yaml` | `config/models/nanobot_reme.yaml` |
|
||||
| env 文件 | `env.reme.sh` | `env.nanobot_reme.sh` |
|
||||
| 一键脚本 | `run_with_reme.sh` | `run_with_nanobot_reme.sh` |
|
||||
| LLM 调用方式 | AgentScope 内嵌 | NanoBot SDK |
|
||||
|
||||
---
|
||||
|
||||
## 6. 已知问题与修复
|
||||
|
||||
### 6.1 已修复
|
||||
|
||||
| 问题 | 根因 | 修复方式 |
|
||||
|------|------|---------|
|
||||
| **Bridge 被 SIGHUP 杀死** | 后台进程收到 SIGHUP | bridge 代码加 `signal.signal(signal.SIGHUP, signal.SIG_IGN)` |
|
||||
| **nohup 导致进程异常** | nohup 改变了信号/IO | 全部改为 `command > log 2>&1 &` |
|
||||
| **MCP 连接超时 (SSRF)** | localhost 解析 IPv6 ::1 被 SSRF 拦截 | URL 全部用 `127.0.0.1` |
|
||||
| **workspace 路径不匹配** | bridge 硬编码 `/tmp/` 但 runner 用 `~/.nanobot/workspace` | bridge 默认 workspace 改为 `~/.nanobot/workspace` |
|
||||
| **Trace log 格式不匹配** | eval 期望 `turn_N.json` 但 runner 输出 `messages.jsonl` | `fix_trace_logs.py` 自动转换 |
|
||||
| **BRAVE_SEARCH_API_KEY 为空** | 未设置导致错误 | 设为 `"dummy"` |
|
||||
| **zsh parse error** | `.zshrc` 中的函数定义被 source 加载 | 确保 `env.nanobot_reme.sh` 不含 bash 函数定义 |
|
||||
|
||||
### 6.2 未修复 (低优先级)
|
||||
|
||||
| 问题 | 说明 | 状态 |
|
||||
|------|------|------|
|
||||
| **搜索工具失效** | DuckDuckGo 在国内被墙, Brave key 为 dummy | 只关注 proc 分数, 暂不修复 |
|
||||
| **AppWorld DeprecationWarning** | Starlette HTTP_422 常量重命名警告 | 不影响功能, 可忽略 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 监控与调试命令
|
||||
|
||||
```bash
|
||||
# ─── 进程检查 ───
|
||||
pgrep -fl "appworld"
|
||||
pgrep -fl "test_server"
|
||||
pgrep -fl "bridge_nanobot"
|
||||
pgrep -fl "bridge_reme"
|
||||
|
||||
# ─── 端口检查 ───
|
||||
lsof -i :9000 # AppWorld API
|
||||
lsof -i :10000 # AppWorld MCP
|
||||
lsof -i :9999 # Test Server
|
||||
|
||||
# ─── 日志查看 ───
|
||||
tail -50 /tmp/nanobot_reme_bridge.log
|
||||
tail -50 /tmp/nanobot_reme_test_server.log
|
||||
tail -50 /tmp/nanobot_reme_appworld_api.log
|
||||
tail -50 /tmp/nanobot_reme_appworld_mcp.log
|
||||
tail -50 /tmp/nanobot_reme_run_researcher.log
|
||||
|
||||
# ─── 运行结果 ───
|
||||
ls $HOME/pi-bench/reme_benchmark/outputs/nanobot_reme/researcher/
|
||||
cat $HOME/pi-bench/reme_benchmark/outputs/nanobot_reme/researcher/*/eval/*.json 2>/dev/null
|
||||
|
||||
# ─── 检查 ReMe 记忆 ───
|
||||
ls $HOME/pi-bench/reme_benchmark/.reme/daily/
|
||||
|
||||
# ─── Test Server API 检查 ───
|
||||
curl -s http://127.0.0.1:9999/sent?after=-1 | python3 -m json.tool | head -20
|
||||
|
||||
# ─── MCP 健康检查 ───
|
||||
curl -s http://127.0.0.1:10000/mcp # 应返回 MCP 协议响应 (非 GET /)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 清理命令
|
||||
|
||||
```bash
|
||||
# 停止所有服务
|
||||
pkill -f "appworld serve" 2>/dev/null || true
|
||||
pkill -f "test_server.py" 2>/dev/null || true
|
||||
pkill -f "bridge_nanobot_reme" 2>/dev/null || true
|
||||
pkill -f "bridge_reme" 2>/dev/null || true
|
||||
|
||||
# 清空 ReMe 记忆 (换 persona 前)
|
||||
rm -rf $HOME/pi-bench/reme_benchmark/.reme/metadata
|
||||
rm -rf $HOME/pi-bench/reme_benchmark/.reme/session
|
||||
rm -rf $HOME/pi-bench/reme_benchmark/.reme/mem_session
|
||||
|
||||
# 清空 nanobot workspace
|
||||
rm -rf ~/.nanobot/workspace/*
|
||||
rm -rf ~/.nanobot/trace_logs/*
|
||||
|
||||
# 清空旧日志
|
||||
rm -f /tmp/nanobot_reme_*.log
|
||||
|
||||
# 释放端口 (紧急)
|
||||
for port in 9000 10000 9999; do
|
||||
pids=$(lsof -ti :${port} 2>/dev/null || true)
|
||||
[ -n "$pids" ] && kill -9 $pids 2>/dev/null || true
|
||||
done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Bridge 脚本关键参数速查
|
||||
|
||||
### bridge_nanobot_reme.py
|
||||
|
||||
```
|
||||
--test-server-url 默认 http://127.0.0.1:9999
|
||||
--appworld-mcp-url 默认 http://127.0.0.1:10000/mcp
|
||||
--nanobot-dir NanoBot 源码路径
|
||||
--reme-dir ReMe 源码路径
|
||||
--data-root persona 数据目录 (含 profile.yaml, tasks/)
|
||||
--user-id persona 名称 (researcher/Financier/...)
|
||||
--model-name LLM 模型名 (qwen3.6-plus)
|
||||
--model-base-url API 端点
|
||||
--model-api-key API key
|
||||
--workspace-dir 默认 ~/.nanobot/workspace
|
||||
--poll-timeout 长轮询超时 (默认 30s)
|
||||
```
|
||||
|
||||
### bridge_reme.py
|
||||
|
||||
```
|
||||
--test-server-url 默认 http://127.0.0.1:9999
|
||||
--appworld-mcp-url 默认 http://127.0.0.1:10000/mcp
|
||||
--reme-dir ReMe 源码路径
|
||||
--data-root persona 数据目录
|
||||
--user-id persona 名称
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. 快速复现清单
|
||||
|
||||
### NanoBot+ReMe 完整测评:
|
||||
|
||||
```bash
|
||||
# 1. 清理
|
||||
pkill -f "appworld serve" 2>/dev/null; pkill -f "test_server" 2>/dev/null; pkill -f "bridge_nanobot" 2>/dev/null
|
||||
sleep 2
|
||||
|
||||
# 2. 进入工作目录
|
||||
cd $HOME/pi-bench/reme_benchmark
|
||||
|
||||
# 3. 单 persona 运行 (自动启动服务 → run → trace 转换 → eval)
|
||||
bash run_researcher.sh
|
||||
|
||||
# 4. 查看结果
|
||||
ls outputs/nanobot_reme/researcher/
|
||||
|
||||
# 5. 全 persona 批跑
|
||||
bash run_all_personas.sh
|
||||
```
|
||||
|
||||
### ReMe-only 完整测评:
|
||||
|
||||
```bash
|
||||
cd $HOME/pi-bench
|
||||
source env.reme.sh
|
||||
bash run_with_reme.sh --user-id researcher
|
||||
```
|
||||
670
benchmark/pi-bench/bridge_nanobot_reme.py
Executable file
670
benchmark/pi-bench/bridge_nanobot_reme.py
Executable file
|
|
@ -0,0 +1,670 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Bridge script: Connects NanoBot agent with ReMe memory to Pi-Bench Test Server.
|
||||
|
||||
Uses NanoBot's SDK as a library for agent execution, with MCP integration
|
||||
to AppWorld and ReMe's cross-session memory for context persistence.
|
||||
|
||||
Flow:
|
||||
1. Poll Test Server /poll for user messages
|
||||
2. Forward to NanoBot agent (via SDK)
|
||||
3. Extract reply text
|
||||
4. Send reply back to Test Server POST /send
|
||||
5. On session end (reset), save conversation as ReMe daily memory (non-blocking)
|
||||
6. On next session start, search ReMe memory for relevant past context
|
||||
|
||||
Key design decisions:
|
||||
- NanoBot is used programmatically via its SDK (Nanobot.from_config + bot.run)
|
||||
- ReMe memory is integrated via Application.jobs (search, auto_memory)
|
||||
- Memory saves are non-blocking (fire-and-forget asyncio tasks) so reset
|
||||
acknowledgments are sent immediately and don't time out.
|
||||
- A pending-save tracker ensures the first message of a new session waits
|
||||
for any in-flight memory writes to complete before searching.
|
||||
- User profile is loaded from data/{user_id}/profile.yaml and injected
|
||||
into the system prompt.
|
||||
- NanoBot session state is maintained within a task via session_key,
|
||||
and cleared on reset for cross-task isolation.
|
||||
|
||||
Usage:
|
||||
python bridge_nanobot_reme.py [--test-server-url URL] [--nanobot-dir DIR] [--reme-dir DIR]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
import traceback
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Set
|
||||
|
||||
import httpx
|
||||
import yaml
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
logger = logging.getLogger("bridge_nanobot_reme")
|
||||
|
||||
|
||||
# ─── User Profile Loading ─────────────────────────────────────────────
|
||||
|
||||
def load_user_profile(data_root: str, user_id: str) -> str:
|
||||
"""Load user profile YAML and return as formatted text for the system prompt."""
|
||||
profile_path = Path(data_root) / user_id / "profile.yaml"
|
||||
if not profile_path.exists():
|
||||
logger.warning("User profile not found: %s", profile_path)
|
||||
return ""
|
||||
|
||||
with open(profile_path, "r", encoding="utf-8") as f:
|
||||
profile = yaml.safe_load(f)
|
||||
|
||||
if not profile:
|
||||
return ""
|
||||
|
||||
parts = []
|
||||
|
||||
if "role" in profile and profile["role"]:
|
||||
role_text = str(profile["role"]).strip()
|
||||
if role_text:
|
||||
parts.append(f"## User Profile\n{role_text}")
|
||||
|
||||
if "preferences" in profile and profile["preferences"]:
|
||||
prefs = profile["preferences"]
|
||||
if isinstance(prefs, dict):
|
||||
pref_lines = []
|
||||
for k, v in prefs.items():
|
||||
if v is not None and str(v).strip():
|
||||
pref_lines.append(f"- {k}: {v}")
|
||||
if pref_lines:
|
||||
parts.append("## Preferences\n" + "\n".join(pref_lines))
|
||||
elif isinstance(prefs, str):
|
||||
parts.append(f"## Preferences\n{prefs}")
|
||||
|
||||
if "long_term_goals" in profile and profile["long_term_goals"]:
|
||||
goals = profile["long_term_goals"]
|
||||
if isinstance(goals, list):
|
||||
goal_lines = [f"- {g}" for g in goals if g]
|
||||
if goal_lines:
|
||||
parts.append("## Long-term Goals\n" + "\n".join(goal_lines))
|
||||
elif isinstance(goals, str):
|
||||
parts.append(f"## Long-term Goals\n{goals}")
|
||||
|
||||
result = "\n\n".join(parts)
|
||||
logger.info("Loaded profile for %s: %d chars", user_id, len(result))
|
||||
return result
|
||||
|
||||
|
||||
def build_system_prompt(user_profile: str) -> str:
|
||||
"""Build the system prompt for the NanoBot agent with profile context."""
|
||||
base_prompt = """You are a proactive personal assistant agent participating in a long-horizon workflow evaluation. Your goal is to help users with their tasks by being thorough, anticipatory, and detail-oriented.
|
||||
|
||||
Key behaviors:
|
||||
- Always consider the user's profile, preferences, and context when responding
|
||||
- Proactively address hidden or implicit needs beyond what is explicitly stated
|
||||
- Use available tools (AppWorld applications like Amazon, Gmail, Todoist, etc.) to complete tasks
|
||||
- When searching for information, be thorough and consider multiple sources
|
||||
- Present information in clear, well-organized formats (tables, lists, etc. when appropriate)
|
||||
- If something is unavailable or uncertain, provide alternatives or fallback options
|
||||
- Remember and reference information from previous conversations when relevant
|
||||
- Before starting a new task, consider relevant context from previous sessions
|
||||
|
||||
You have access to:
|
||||
- AppWorld tools (Amazon, Gmail, Todoist, etc.) via MCP for real-world task execution
|
||||
- Shell and file system tools for local operations
|
||||
- Web search tools for finding information online
|
||||
|
||||
Always be helpful, thorough, and proactive in your assistance."""
|
||||
|
||||
if user_profile:
|
||||
return f"{base_prompt}\n\n{user_profile}"
|
||||
return base_prompt
|
||||
|
||||
|
||||
class NanoBotReMeBridge:
|
||||
"""Bridge between Pi-Bench Test Server and NanoBot agent with ReMe memory."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
test_server_url: str = "http://127.0.0.1:9999",
|
||||
appworld_mcp_url: str = "http://127.0.0.1:10000/mcp",
|
||||
nanobot_dir: str = "",
|
||||
reme_dir: str = "",
|
||||
data_root: str = "data",
|
||||
user_id: str = "researcher",
|
||||
poll_timeout: int = 30,
|
||||
workspace_dir: str = "",
|
||||
model_name: str = "qwen3.6-plus",
|
||||
model_base_url: str = "",
|
||||
model_api_key: str = "",
|
||||
):
|
||||
self.test_server_url = test_server_url.rstrip("/")
|
||||
self.appworld_mcp_url = appworld_mcp_url
|
||||
self.nanobot_dir = Path(nanobot_dir).resolve() if nanobot_dir else Path(__file__).resolve().parent.parent / "third_party" / "nanobot"
|
||||
self.reme_dir = Path(reme_dir).resolve() if reme_dir else None
|
||||
self.data_root = data_root
|
||||
self.user_id = user_id
|
||||
self.poll_timeout = poll_timeout
|
||||
self.model_name = model_name
|
||||
self.model_base_url = model_base_url
|
||||
self.model_api_key = model_api_key
|
||||
|
||||
# Workspace for nanobot
|
||||
self.workspace_dir = Path(workspace_dir).expanduser() if workspace_dir else Path.home() / ".nanobot" / "workspace"
|
||||
self.workspace_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# State
|
||||
self.client: Optional[httpx.AsyncClient] = None
|
||||
self.running = False
|
||||
self.bot = None # NanoBot SDK instance
|
||||
self.app = None # ReMe Application
|
||||
self.search_job = None
|
||||
self.auto_memory_job = None
|
||||
|
||||
# Session tracking
|
||||
self.session_messages: Dict[str, List[Dict]] = {}
|
||||
self.session_key: Optional[str] = None # Current nanobot session key
|
||||
|
||||
# User profile
|
||||
self.user_profile_text = load_user_profile(data_root, user_id)
|
||||
|
||||
# Non-blocking memory save tracking
|
||||
self._pending_memory_tasks: List[asyncio.Task] = []
|
||||
|
||||
async def start(self):
|
||||
"""Start the bridge: initialize NanoBot and ReMe."""
|
||||
self.client = httpx.AsyncClient(timeout=600.0, trust_env=False)
|
||||
self.running = True
|
||||
|
||||
logger.info("Starting NanoBot+ReMe Bridge:")
|
||||
logger.info(" Test Server: %s", self.test_server_url)
|
||||
logger.info(" AppWorld MCP: %s", self.appworld_mcp_url)
|
||||
logger.info(" NanoBot dir: %s", self.nanobot_dir)
|
||||
logger.info(" ReMe dir: %s", self.reme_dir)
|
||||
logger.info(" Workspace: %s", self.workspace_dir)
|
||||
logger.info(" User: %s", self.user_id)
|
||||
logger.info(" Model: %s", self.model_name)
|
||||
|
||||
# Initialize ReMe memory system
|
||||
await self._init_reme_app()
|
||||
|
||||
# Initialize NanoBot
|
||||
await self._init_nanobot()
|
||||
|
||||
logger.info("Bridge started successfully")
|
||||
|
||||
async def _init_reme_app(self):
|
||||
"""Initialize ReMe Application for memory operations."""
|
||||
if not self.reme_dir:
|
||||
logger.warning("ReMe dir not set, memory features disabled")
|
||||
return
|
||||
|
||||
# Add reme to Python path
|
||||
reme_str = str(self.reme_dir)
|
||||
if reme_str not in sys.path:
|
||||
sys.path.insert(0, reme_str)
|
||||
|
||||
try:
|
||||
from reme.config import resolve_app_config
|
||||
from reme.application import Application
|
||||
|
||||
# Load .env from reme dir if available
|
||||
env_path = self.reme_dir / ".env"
|
||||
if env_path.exists():
|
||||
logger.info("Loading ReMe .env from %s", env_path)
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv(env_path, override=False)
|
||||
|
||||
# Resolve config
|
||||
reme_config = resolve_app_config(
|
||||
config_dir=str(self.reme_dir / "config"),
|
||||
env_prefix="REME",
|
||||
)
|
||||
logger.info("ReMe config resolved")
|
||||
|
||||
# Create application
|
||||
self.app = Application(**reme_config)
|
||||
await self.app.start()
|
||||
logger.info("ReMe Application started")
|
||||
|
||||
# Get jobs for memory operations
|
||||
self.search_job = self.app.context.jobs.get("search")
|
||||
self.auto_memory_job = self.app.context.jobs.get("auto_memory")
|
||||
|
||||
logger.info("ReMe memory jobs:")
|
||||
logger.info(" search: %s", "yes" if self.search_job else "no")
|
||||
logger.info(" auto_memory: %s", "yes" if self.auto_memory_job else "no")
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to initialize ReMe: %s", e)
|
||||
logger.exception("ReMe initialization error")
|
||||
self.app = None
|
||||
self.search_job = None
|
||||
self.auto_memory_job = None
|
||||
|
||||
async def _init_nanobot(self):
|
||||
"""Initialize NanoBot SDK instance."""
|
||||
# Add nanobot to Python path
|
||||
nanobot_str = str(self.nanobot_dir)
|
||||
if nanobot_str not in sys.path:
|
||||
sys.path.insert(0, nanobot_str)
|
||||
|
||||
try:
|
||||
from nanobot import Nanobot
|
||||
|
||||
# Create config programmatically
|
||||
config_path = self.workspace_dir / "nanobot_config.json"
|
||||
self._write_nanobot_config(config_path)
|
||||
|
||||
# Initialize nanobot from config
|
||||
self.bot = Nanobot.from_config(
|
||||
config_path=str(config_path),
|
||||
workspace=str(self.workspace_dir),
|
||||
model=self.model_name,
|
||||
)
|
||||
logger.info("NanoBot initialized: model=%s", self.model_name)
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Failed to initialize NanoBot: %s", e)
|
||||
logger.exception("NanoBot initialization error")
|
||||
raise
|
||||
|
||||
def _write_nanobot_config(self, config_path: Path):
|
||||
"""Write NanoBot config.json with MCP and provider settings."""
|
||||
# Write SOUL.md with system prompt (nanobot uses workspace SOUL.md as identity)
|
||||
soul_path = self.workspace_dir / "SOUL.md"
|
||||
soul_path.write_text(build_system_prompt(self.user_profile_text), encoding="utf-8")
|
||||
logger.info("SOUL.md written: %s (%d chars)", soul_path, soul_path.stat().st_size)
|
||||
|
||||
config = {
|
||||
"agents": {
|
||||
"defaults": {
|
||||
"workspace": str(self.workspace_dir),
|
||||
"model": self.model_name,
|
||||
"provider": "dashscope" if "qwen" in self.model_name.lower() else "custom",
|
||||
"max_tokens": 16384,
|
||||
"temperature": 0.1,
|
||||
}
|
||||
},
|
||||
"providers": {
|
||||
"dashscope": {
|
||||
"api_key": self.model_api_key or os.environ.get("DASHSCOPE_API_KEY", ""),
|
||||
"api_base": self.model_base_url or "https://dashscope.aliyuncs.com/compatible-mode/v1",
|
||||
},
|
||||
"custom": {
|
||||
"api_key": self.model_api_key or os.environ.get("MODEL_API_KEY", ""),
|
||||
"api_base": self.model_base_url or "http://localhost:8088/v1",
|
||||
}
|
||||
},
|
||||
"tools": {
|
||||
"mcp_servers": {
|
||||
"appworld": {
|
||||
"type": "streamableHttp",
|
||||
"url": self.appworld_mcp_url,
|
||||
"tool_timeout": 60,
|
||||
"enabled_tools": ["*"],
|
||||
}
|
||||
},
|
||||
"exec": {
|
||||
"enabled": True,
|
||||
},
|
||||
"file": {
|
||||
"enabled": True,
|
||||
},
|
||||
"web": {
|
||||
"enabled": True,
|
||||
"search": {
|
||||
"provider": "brave",
|
||||
"api_key": "dummy",
|
||||
"timeout": 10,
|
||||
},
|
||||
},
|
||||
"ssrf_whitelist": ["127.0.0.0/8"], # Allow localhost for MCP
|
||||
},
|
||||
"channels": {
|
||||
"send_progress": False,
|
||||
},
|
||||
}
|
||||
|
||||
with open(config_path, "w") as f:
|
||||
json.dump(config, f, indent=2)
|
||||
logger.info("NanoBot config written: %s", config_path)
|
||||
|
||||
async def stop(self):
|
||||
"""Stop the bridge."""
|
||||
self.running = False
|
||||
if self.client:
|
||||
await self.client.aclose()
|
||||
self.client = None
|
||||
|
||||
# Stop ReMe application
|
||||
if self.app:
|
||||
try:
|
||||
await self.app.stop()
|
||||
except Exception as e:
|
||||
logger.warning("Error stopping ReMe app: %s", e)
|
||||
|
||||
logger.info("Bridge stopped")
|
||||
|
||||
# ─── Memory Operations ──────────────────────────────────────────
|
||||
|
||||
async def _wait_for_pending_memory_saves(self):
|
||||
"""Wait for all in-flight memory save tasks to complete."""
|
||||
if not self._pending_memory_tasks:
|
||||
return
|
||||
logger.info("Waiting for %d pending memory saves before search...",
|
||||
len(self._pending_memory_tasks))
|
||||
tasks = self._pending_memory_tasks[:]
|
||||
self._pending_memory_tasks.clear()
|
||||
for task in tasks:
|
||||
try:
|
||||
await asyncio.wait_for(task, timeout=120.0)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning("Memory save task timed out (120s)")
|
||||
except Exception as e:
|
||||
logger.warning("Memory save task failed: %s", e)
|
||||
|
||||
async def _search_memory(self, query: str) -> str:
|
||||
"""Search ReMe memory for relevant context from previous sessions."""
|
||||
if not self.search_job:
|
||||
return ""
|
||||
try:
|
||||
response = await self.search_job(query=query, limit=5)
|
||||
if response.success and response.answer:
|
||||
return response.answer
|
||||
except Exception as e:
|
||||
logger.warning("Memory search failed: %s", e)
|
||||
return ""
|
||||
|
||||
async def _do_save_session_memory(self, chat_id: str, messages: List[Dict]):
|
||||
"""Actually perform the session memory save (runs as background task)."""
|
||||
if not self.auto_memory_job or not messages:
|
||||
return
|
||||
|
||||
session_id = f"pibench_{chat_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
|
||||
|
||||
# Convert messages to auto_memory format
|
||||
memory_messages = []
|
||||
for msg in messages:
|
||||
role = msg.get("role", "user")
|
||||
memory_messages.append({
|
||||
"role": role,
|
||||
"name": msg.get("name", "user" if role == "user" else "assistant"),
|
||||
"content": msg.get("content", ""),
|
||||
"created_at": msg.get("timestamp", datetime.now().isoformat()),
|
||||
})
|
||||
|
||||
try:
|
||||
logger.info("Saving session memory: chat_id=%s messages=%d",
|
||||
chat_id, len(memory_messages))
|
||||
response = await self.auto_memory_job(
|
||||
messages=memory_messages,
|
||||
session_id=session_id,
|
||||
memory_hint=(
|
||||
f"Pi-Bench evaluation session for task {chat_id}. "
|
||||
f"User persona: {self.user_id}. "
|
||||
f"Save key decisions, actions taken, important outcomes, "
|
||||
f"and any user preferences or context that may be useful "
|
||||
f"for future sessions."
|
||||
),
|
||||
)
|
||||
if response.success:
|
||||
preview = (response.answer or "OK")[:200]
|
||||
logger.info("Session memory saved: %s", preview)
|
||||
else:
|
||||
logger.warning("Memory save returned unsuccessful: %s", response.answer)
|
||||
except Exception as e:
|
||||
logger.exception("Error saving session memory: %s", e)
|
||||
|
||||
def _schedule_memory_save(self, chat_id: str, messages: List[Dict]):
|
||||
"""Schedule a non-blocking memory save task."""
|
||||
if not self.auto_memory_job or not messages:
|
||||
return
|
||||
|
||||
task = asyncio.create_task(
|
||||
self._do_save_session_memory(chat_id, messages),
|
||||
name=f"memory_save_{chat_id}",
|
||||
)
|
||||
self._pending_memory_tasks.append(task)
|
||||
|
||||
# Clean up completed tasks from the tracking list
|
||||
self._pending_memory_tasks = [
|
||||
t for t in self._pending_memory_tasks if not t.done()
|
||||
]
|
||||
|
||||
# ─── Message Processing ─────────────────────────────────────────
|
||||
|
||||
async def process_message(self, sender_id: str, chat_id: str, content: str) -> Optional[str]:
|
||||
"""Process a user message through the NanoBot agent with ReMe memory."""
|
||||
# Track session messages for later memory save
|
||||
if chat_id not in self.session_messages:
|
||||
self.session_messages[chat_id] = []
|
||||
|
||||
self.session_messages[chat_id].append({
|
||||
"role": "user",
|
||||
"name": "user",
|
||||
"content": content,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
})
|
||||
|
||||
# On first message in session, wait for pending memory saves
|
||||
# then search memory for relevant context
|
||||
memory_context = ""
|
||||
is_first_message = len(self.session_messages[chat_id]) == 1
|
||||
if is_first_message:
|
||||
# Wait for any in-flight memory saves from previous sessions
|
||||
await self._wait_for_pending_memory_saves()
|
||||
|
||||
# Search memory for relevant context
|
||||
if self.search_job:
|
||||
try:
|
||||
memory_context = await self._search_memory(content)
|
||||
if memory_context:
|
||||
logger.info("Found relevant memory: %d chars", len(memory_context))
|
||||
except Exception as e:
|
||||
logger.warning("Memory search on first message failed: %s", e)
|
||||
|
||||
try:
|
||||
# Prepend memory context if available
|
||||
user_message = content
|
||||
if memory_context:
|
||||
user_message = (
|
||||
f"[Relevant memories from previous sessions]\n"
|
||||
f"{memory_context}\n\n"
|
||||
f"[Current user message]\n{content}"
|
||||
)
|
||||
|
||||
# Create or reuse session key for multi-turn continuity
|
||||
if not self.session_key:
|
||||
self.session_key = f"pibench_{chat_id}_{int(time.time())}"
|
||||
|
||||
# Call NanoBot agent (no timeout, like ReMe bridge)
|
||||
logger.info("Calling NanoBot.run()...")
|
||||
result = await self.bot.run(
|
||||
user_message,
|
||||
session_key=self.session_key,
|
||||
channel="pibench",
|
||||
chat_id=chat_id,
|
||||
sender_id=sender_id,
|
||||
)
|
||||
logger.info("NanoBot.run() completed")
|
||||
reply_text = result.content or ""
|
||||
|
||||
# Track the assistant reply
|
||||
self.session_messages[chat_id].append({
|
||||
"role": "assistant",
|
||||
"name": "assistant",
|
||||
"content": reply_text,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
})
|
||||
|
||||
logger.info("Reply: %d chars, session=%s", len(reply_text), self.session_key)
|
||||
return reply_text
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("Error processing message: %s", e)
|
||||
return None
|
||||
|
||||
async def handle_reset(self, chat_id: str):
|
||||
"""Handle session reset: schedule non-blocking memory save and clear state."""
|
||||
messages = self.session_messages.pop(chat_id, [])
|
||||
if messages:
|
||||
self._schedule_memory_save(chat_id, messages)
|
||||
# Clear session key for cross-task isolation
|
||||
self.session_key = None
|
||||
|
||||
# ─── Test Server Communication ──────────────────────────────────
|
||||
|
||||
async def poll_test_server(self) -> List[Dict[str, Any]]:
|
||||
"""Poll Test Server for pending messages."""
|
||||
try:
|
||||
resp = await self.client.get(
|
||||
f"{self.test_server_url}/poll",
|
||||
params={"timeout": self.poll_timeout},
|
||||
)
|
||||
if resp.is_success:
|
||||
data = resp.json()
|
||||
messages = data.get("messages", [])
|
||||
if messages:
|
||||
logger.info("Received %d messages", len(messages))
|
||||
return messages
|
||||
except Exception as e:
|
||||
logger.warning("Poll error: %s", e)
|
||||
return []
|
||||
|
||||
async def send_to_test_server(self, chat_id: str, content: str) -> bool:
|
||||
"""Send reply back to Test Server."""
|
||||
payload = {
|
||||
"chat_id": chat_id,
|
||||
"content": content,
|
||||
"media": [],
|
||||
"meta": {},
|
||||
}
|
||||
try:
|
||||
resp = await self.client.post(
|
||||
f"{self.test_server_url}/send",
|
||||
json=payload,
|
||||
)
|
||||
if resp.is_success:
|
||||
logger.info("Sent reply: chat_id=%s len=%d", chat_id, len(content))
|
||||
return True
|
||||
logger.error("Failed to send: %s", resp.status_code)
|
||||
except Exception as e:
|
||||
logger.exception("Error sending to Test Server")
|
||||
return False
|
||||
|
||||
# ─── Main Loop ──────────────────────────────────────────────────
|
||||
|
||||
def _install_signal_handlers(self):
|
||||
"""Install SIGTERM/SIGINT handlers for graceful shutdown."""
|
||||
loop = asyncio.get_running_loop()
|
||||
for sig_name in ("SIGTERM", "SIGINT"):
|
||||
sig = getattr(signal, sig_name, None)
|
||||
if sig is not None:
|
||||
loop.add_signal_handler(sig, self._handle_shutdown_signal, sig_name)
|
||||
|
||||
def _handle_shutdown_signal(self, sig_name: str):
|
||||
"""Handle shutdown signal: stop the bridge loop gracefully."""
|
||||
logger.info("Received %s, initiating graceful shutdown...", sig_name)
|
||||
self.running = False
|
||||
|
||||
async def run(self):
|
||||
"""Main bridge loop."""
|
||||
await self.start()
|
||||
self._install_signal_handlers()
|
||||
|
||||
try:
|
||||
while self.running:
|
||||
messages = await self.poll_test_server()
|
||||
|
||||
for msg in messages:
|
||||
sender_id = msg.get("sender_id", "unknown")
|
||||
chat_id = msg.get("chat_id", "default")
|
||||
content = msg.get("content", "")
|
||||
|
||||
if not content:
|
||||
continue
|
||||
|
||||
logger.info("Processing: sender=%s chat=%s len=%d",
|
||||
sender_id, chat_id, len(content))
|
||||
|
||||
# Check for reset/new-session signal
|
||||
if content.strip().lower() in ("reset", "new session", "/new"):
|
||||
logger.info("Reset signal: chat_id=%s", chat_id)
|
||||
await self.handle_reset(chat_id)
|
||||
await self.send_to_test_server(chat_id, "New session started")
|
||||
continue
|
||||
|
||||
# Forward to NanoBot agent
|
||||
reply = await self.process_message(sender_id, chat_id, content)
|
||||
|
||||
if reply:
|
||||
await self.send_to_test_server(chat_id, reply)
|
||||
else:
|
||||
logger.warning("No reply for chat_id=%s", chat_id)
|
||||
await self.send_to_test_server(
|
||||
chat_id, "[Error: Agent failed to generate response]"
|
||||
)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
logger.info("Interrupted by user")
|
||||
finally:
|
||||
# Save any remaining session memories
|
||||
for cid in list(self.session_messages.keys()):
|
||||
messages = self.session_messages.pop(cid, [])
|
||||
if messages:
|
||||
self._schedule_memory_save(cid, messages)
|
||||
# Wait for all pending memory saves to complete
|
||||
if self._pending_memory_tasks:
|
||||
logger.info("Waiting for %d pending memory saves to complete...",
|
||||
len(self._pending_memory_tasks))
|
||||
await self._wait_for_pending_memory_saves()
|
||||
await self.stop()
|
||||
|
||||
|
||||
async def main():
|
||||
# Ignore SIGHUP to prevent bridge from being killed
|
||||
signal.signal(signal.SIGHUP, signal.SIG_IGN)
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Bridge between Pi-Bench Test Server and NanoBot agent with ReMe memory"
|
||||
)
|
||||
parser.add_argument("--test-server-url", default="http://127.0.0.1:9999")
|
||||
parser.add_argument("--appworld-mcp-url", default="http://127.0.0.1:10000/mcp")
|
||||
parser.add_argument("--nanobot-dir", default="")
|
||||
parser.add_argument("--reme-dir", default="")
|
||||
parser.add_argument("--data-root", default="data")
|
||||
parser.add_argument("--user-id", default="researcher")
|
||||
parser.add_argument("--poll-timeout", type=int, default=30)
|
||||
parser.add_argument("--workspace-dir", default="")
|
||||
parser.add_argument("--model-name", default="qwen3.6-plus")
|
||||
parser.add_argument("--model-base-url", default="")
|
||||
parser.add_argument("--model-api-key", default="")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
bridge = NanoBotReMeBridge(
|
||||
test_server_url=args.test_server_url,
|
||||
appworld_mcp_url=args.appworld_mcp_url,
|
||||
nanobot_dir=args.nanobot_dir,
|
||||
reme_dir=args.reme_dir,
|
||||
data_root=args.data_root,
|
||||
user_id=args.user_id,
|
||||
poll_timeout=args.poll_timeout,
|
||||
workspace_dir=args.workspace_dir,
|
||||
model_name=args.model_name,
|
||||
model_base_url=args.model_base_url,
|
||||
model_api_key=args.model_api_key,
|
||||
)
|
||||
|
||||
await bridge.run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
644
benchmark/pi-bench/bridge_reme.py
Executable file
644
benchmark/pi-bench/bridge_reme.py
Executable file
|
|
@ -0,0 +1,644 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Bridge script: Connects ReMe agent to Pi-Bench Test Server.
|
||||
|
||||
Uses ReMe's AgentScope-based agent wrapper directly as a library,
|
||||
with MCP integration to AppWorld and cross-session memory support.
|
||||
|
||||
Flow:
|
||||
1. Poll Test Server /poll for user messages
|
||||
2. Forward to ReMe agent (via AgentScope)
|
||||
3. Extract reply text
|
||||
4. Send reply back to Test Server POST /send
|
||||
5. On session end (reset), save conversation as ReMe daily memory (non-blocking)
|
||||
6. On next session start, search memory for relevant past context
|
||||
|
||||
Key design decisions:
|
||||
- Memory saves are non-blocking (fire-and-forget asyncio tasks) so reset
|
||||
acknowledgments are sent immediately and don't time out.
|
||||
- A pending-save tracker ensures the first message of a new session waits
|
||||
for any in-flight memory writes to complete before searching.
|
||||
- User profile is loaded from data/{user_id}/profile.yaml and injected
|
||||
into every turn's system prompt.
|
||||
- AgentScope session state is maintained via `resume` within a task,
|
||||
and cleared on reset for cross-task isolation.
|
||||
|
||||
Usage:
|
||||
python bridge_reme.py [--test-server-url URL] [--reme-dir DIR]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
import traceback
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Set
|
||||
|
||||
import httpx
|
||||
import yaml
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
logger = logging.getLogger("bridge_reme")
|
||||
|
||||
|
||||
# ─── User Profile Loading ─────────────────────────────────────────────
|
||||
|
||||
def load_user_profile(data_root: str, user_id: str) -> str:
|
||||
"""Load user profile YAML and return as formatted text for the system prompt.
|
||||
|
||||
Handles the full Pi-Bench profile schema: role (with sub-sections),
|
||||
preferences, and long_term_goals.
|
||||
"""
|
||||
profile_path = Path(data_root) / user_id / "profile.yaml"
|
||||
if not profile_path.exists():
|
||||
logger.warning("User profile not found: %s", profile_path)
|
||||
return ""
|
||||
|
||||
with open(profile_path, "r", encoding="utf-8") as f:
|
||||
profile = yaml.safe_load(f)
|
||||
|
||||
if not profile:
|
||||
return ""
|
||||
|
||||
parts = []
|
||||
|
||||
# Role section: contains the full persona description
|
||||
if "role" in profile and profile["role"]:
|
||||
role_text = str(profile["role"]).strip()
|
||||
if role_text:
|
||||
parts.append(f"## User Profile\n{role_text}")
|
||||
|
||||
# Preferences section
|
||||
if "preferences" in profile and profile["preferences"]:
|
||||
prefs = profile["preferences"]
|
||||
if isinstance(prefs, dict):
|
||||
pref_lines = []
|
||||
for k, v in prefs.items():
|
||||
if v is not None and str(v).strip():
|
||||
pref_lines.append(f"- {k}: {v}")
|
||||
if pref_lines:
|
||||
parts.append("## Preferences\n" + "\n".join(pref_lines))
|
||||
elif isinstance(prefs, str):
|
||||
parts.append(f"## Preferences\n{prefs}")
|
||||
|
||||
# Long-term goals
|
||||
if "long_term_goals" in profile and profile["long_term_goals"]:
|
||||
goals = profile["long_term_goals"]
|
||||
if isinstance(goals, list):
|
||||
goal_lines = [f"- {g}" for g in goals if g]
|
||||
if goal_lines:
|
||||
parts.append("## Long-term Goals\n" + "\n".join(goal_lines))
|
||||
elif isinstance(goals, str):
|
||||
parts.append(f"## Long-term Goals\n{goals}")
|
||||
|
||||
result = "\n\n".join(parts)
|
||||
logger.info("Loaded profile for %s: %d chars, sections: %s",
|
||||
user_id, len(result),
|
||||
[k for k in ["role", "preferences", "long_term_goals"] if k in profile])
|
||||
return result
|
||||
|
||||
|
||||
def build_system_prompt(user_profile: str) -> str:
|
||||
"""Build the system prompt for the ReMe agent with profile context."""
|
||||
base_prompt = """You are a proactive personal assistant agent participating in a long-horizon workflow evaluation. Your goal is to help users with their tasks by being thorough, anticipatory, and detail-oriented.
|
||||
|
||||
Key behaviors:
|
||||
- Always consider the user's profile, preferences, and context when responding
|
||||
- Proactively address hidden or implicit needs beyond what is explicitly stated
|
||||
- Use available tools (AppWorld applications like Amazon, Gmail, Todoist, etc.) to complete tasks
|
||||
- When searching for information, be thorough and consider multiple sources
|
||||
- Present information in clear, well-organized formats (tables, lists, etc. when appropriate)
|
||||
- If something is unavailable or uncertain, provide alternatives or fallback options
|
||||
- Remember and reference information from previous conversations when relevant
|
||||
- Before starting a new task, search your memory for relevant context from previous sessions
|
||||
|
||||
You have access to:
|
||||
- AppWorld tools (Amazon, Gmail, Todoist, etc.) via MCP for real-world task execution
|
||||
- Memory tools: `search` to recall past conversations, `daily_write` to save important notes
|
||||
- File tools: `read`, `write`, `edit`, `bash` for file operations
|
||||
"""
|
||||
|
||||
if user_profile:
|
||||
base_prompt += f"\n\n---\n\n{user_profile}\n"
|
||||
|
||||
base_prompt += "\n\n---\n\nAlways respond in the same language as the user's message. Use tools proactively to help the user."
|
||||
return base_prompt
|
||||
|
||||
|
||||
# ─── ReMe Bridge ──────────────────────────────────────────────────────
|
||||
|
||||
class ReMeBridge:
|
||||
"""Bridge between Pi-Bench Test Server and ReMe agent."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
test_server_url: str = "http://localhost:9999",
|
||||
appworld_mcp_url: str = "http://localhost:10000/mcp",
|
||||
reme_dir: str = "",
|
||||
data_root: str = "data",
|
||||
user_id: str = "researcher",
|
||||
poll_timeout: int = 30,
|
||||
workspace_dir: str = "",
|
||||
model_name: str = "qwen3.6-plus",
|
||||
model_base_url: str = "",
|
||||
model_api_key: str = "",
|
||||
):
|
||||
self.test_server_url = test_server_url.rstrip("/")
|
||||
self.appworld_mcp_url = appworld_mcp_url
|
||||
self.reme_dir = Path(reme_dir).resolve() if reme_dir else None
|
||||
self.data_root = Path(data_root)
|
||||
self.user_id = user_id
|
||||
self.poll_timeout = poll_timeout
|
||||
self.workspace_dir = Path(workspace_dir) if workspace_dir else Path("/tmp/reme_pibench_workspace")
|
||||
self.model_name = model_name
|
||||
self.model_base_url = model_base_url
|
||||
self.model_api_key = model_api_key
|
||||
|
||||
self.client: Optional[httpx.AsyncClient] = None
|
||||
self.running = False
|
||||
|
||||
# ReMe components
|
||||
self.app = None
|
||||
self.agent_wrapper = None
|
||||
self.auto_memory_job = None
|
||||
self.search_job = None
|
||||
|
||||
# Session state
|
||||
self.user_profile_text = ""
|
||||
self.session_messages: Dict[str, List[Dict]] = {}
|
||||
self.agent_session_id: Optional[str] = None
|
||||
|
||||
# Non-blocking memory save tracking
|
||||
self._pending_memory_tasks: List[asyncio.Task] = []
|
||||
|
||||
async def start(self):
|
||||
"""Initialize the ReMe application and bridge components."""
|
||||
self.client = httpx.AsyncClient(timeout=300.0, trust_env=False)
|
||||
self.running = True
|
||||
|
||||
# Load user profile
|
||||
self.user_profile_text = load_user_profile(str(self.data_root), self.user_id)
|
||||
logger.info("User profile loaded: %d chars", len(self.user_profile_text))
|
||||
|
||||
# Initialize ReMe application
|
||||
await self._init_reme_app()
|
||||
|
||||
logger.info(
|
||||
"Bridge started: test_server=%s appworld_mcp=%s user=%s model=%s",
|
||||
self.test_server_url,
|
||||
self.appworld_mcp_url,
|
||||
self.user_id,
|
||||
self.model_name,
|
||||
)
|
||||
|
||||
async def _init_reme_app(self):
|
||||
"""Initialize the ReMe application with proper configuration."""
|
||||
# Add reme to Python path so imports work
|
||||
if self.reme_dir:
|
||||
reme_str = str(self.reme_dir)
|
||||
if reme_str not in sys.path:
|
||||
sys.path.insert(0, reme_str)
|
||||
|
||||
from reme.config import resolve_app_config
|
||||
from reme.application import Application
|
||||
|
||||
# Set environment variables for ReMe LLM config expansion
|
||||
os.environ["LLM_MODEL_NAME"] = self.model_name
|
||||
if self.model_base_url:
|
||||
os.environ["LLM_BASE_URL"] = self.model_base_url
|
||||
if self.model_api_key:
|
||||
os.environ["LLM_API_KEY"] = self.model_api_key
|
||||
# Ensure BRAVE_SEARCH_API_KEY is set (required by some tools)
|
||||
if not os.environ.get("BRAVE_SEARCH_API_KEY"):
|
||||
os.environ["BRAVE_SEARCH_API_KEY"] = "dummy"
|
||||
|
||||
# Ensure workspace exists
|
||||
self.workspace_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Load .env from reme dir if available
|
||||
environment = {}
|
||||
if self.reme_dir:
|
||||
env_path = self.reme_dir / ".env"
|
||||
if env_path.exists():
|
||||
with open(env_path, "r") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line and not line.startswith("#") and "=" in line:
|
||||
key, _, value = line.partition("=")
|
||||
environment[key.strip()] = value.strip()
|
||||
|
||||
# Override with explicit values
|
||||
if self.model_base_url:
|
||||
environment["LLM_BASE_URL"] = self.model_base_url
|
||||
if self.model_api_key:
|
||||
environment["LLM_API_KEY"] = self.model_api_key
|
||||
environment["LLM_MODEL_NAME"] = self.model_name
|
||||
|
||||
# Use resolve_app_config to load default.yaml and merge overrides
|
||||
reme_config = resolve_app_config(
|
||||
log_config=False,
|
||||
workspace_dir=str(self.workspace_dir),
|
||||
service={"backend": "http", "host": "127.0.0.1", "port": 18765},
|
||||
environment=environment,
|
||||
)
|
||||
|
||||
try:
|
||||
self.app = Application(**reme_config)
|
||||
await self.app.start()
|
||||
|
||||
# Get components
|
||||
self.agent_wrapper = self.app.context.components.get("agent_wrapper", {}).get("default")
|
||||
if self.agent_wrapper is None:
|
||||
raise RuntimeError("agent_wrapper component 'default' not found")
|
||||
|
||||
# Get jobs for memory operations
|
||||
self.auto_memory_job = self.app.context.jobs.get("auto_memory")
|
||||
self.search_job = self.app.context.jobs.get("search")
|
||||
|
||||
logger.info("ReMe initialized OK")
|
||||
logger.info(" agent_wrapper: %s", getattr(self.agent_wrapper, 'name', 'default'))
|
||||
logger.info(" auto_memory: %s", "yes" if self.auto_memory_job else "no")
|
||||
logger.info(" search: %s", "yes" if self.search_job else "no")
|
||||
logger.info(" total jobs: %d", len(self.app.context.jobs))
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("Failed to initialize ReMe: %s", e)
|
||||
raise
|
||||
|
||||
async def stop(self):
|
||||
"""Stop the bridge and cleanup."""
|
||||
self.running = False
|
||||
|
||||
# Wait for pending memory saves
|
||||
if self._pending_memory_tasks:
|
||||
logger.info("Waiting for %d pending memory saves...", len(self._pending_memory_tasks))
|
||||
for task in self._pending_memory_tasks:
|
||||
try:
|
||||
await asyncio.wait_for(task, timeout=60.0)
|
||||
except (asyncio.TimeoutError, Exception) as e:
|
||||
logger.warning("Pending memory save timed out or failed: %s", e)
|
||||
|
||||
if self.app:
|
||||
try:
|
||||
await self.app.close()
|
||||
except Exception as e:
|
||||
logger.warning("Error closing ReMe app: %s", e)
|
||||
if self.client:
|
||||
await self.client.aclose()
|
||||
self.client = None
|
||||
logger.info("Bridge stopped")
|
||||
|
||||
def _create_mcp_client(self):
|
||||
"""Create an MCP client for AppWorld."""
|
||||
from agentscope.mcp import MCPClient, HttpMCPConfig
|
||||
return MCPClient(
|
||||
name="AppWorld",
|
||||
is_stateful=False,
|
||||
mcp_config=HttpMCPConfig(
|
||||
url=self.appworld_mcp_url,
|
||||
timeout=120.0,
|
||||
),
|
||||
)
|
||||
|
||||
# ─── Memory Operations ──────────────────────────────────────────
|
||||
|
||||
async def _wait_for_pending_memory_saves(self):
|
||||
"""Wait for all in-flight memory save tasks to complete."""
|
||||
if not self._pending_memory_tasks:
|
||||
return
|
||||
logger.info("Waiting for %d pending memory saves before search...",
|
||||
len(self._pending_memory_tasks))
|
||||
tasks = self._pending_memory_tasks[:]
|
||||
self._pending_memory_tasks.clear()
|
||||
for task in tasks:
|
||||
try:
|
||||
await asyncio.wait_for(task, timeout=120.0)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning("Memory save task timed out (120s)")
|
||||
except Exception as e:
|
||||
logger.warning("Memory save task failed: %s", e)
|
||||
|
||||
async def _search_memory(self, query: str) -> str:
|
||||
"""Search ReMe memory for relevant context from previous sessions."""
|
||||
if not self.search_job:
|
||||
return ""
|
||||
try:
|
||||
response = await self.search_job(query=query, limit=5)
|
||||
if response.success and response.answer:
|
||||
return response.answer
|
||||
except Exception as e:
|
||||
logger.warning("Memory search failed: %s", e)
|
||||
return ""
|
||||
|
||||
async def _do_save_session_memory(self, chat_id: str, messages: List[Dict]):
|
||||
"""Actually perform the session memory save (runs as background task)."""
|
||||
if not self.auto_memory_job or not messages:
|
||||
return
|
||||
|
||||
session_id = f"pibench_{chat_id}_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
|
||||
|
||||
# Convert messages to auto_memory format
|
||||
memory_messages = []
|
||||
for msg in messages:
|
||||
role = msg.get("role", "user")
|
||||
memory_messages.append({
|
||||
"role": role,
|
||||
"name": msg.get("name", "user" if role == "user" else "assistant"),
|
||||
"content": msg.get("content", ""),
|
||||
"created_at": msg.get("timestamp", datetime.now().isoformat()),
|
||||
})
|
||||
|
||||
try:
|
||||
logger.info("Saving session memory: chat_id=%s messages=%d",
|
||||
chat_id, len(memory_messages))
|
||||
response = await self.auto_memory_job(
|
||||
messages=memory_messages,
|
||||
session_id=session_id,
|
||||
memory_hint=(
|
||||
f"Pi-Bench evaluation session for task {chat_id}. "
|
||||
f"User persona: {self.user_id}. "
|
||||
f"Save key decisions, actions taken, important outcomes, "
|
||||
f"and any user preferences or context that may be useful "
|
||||
f"for future sessions."
|
||||
),
|
||||
)
|
||||
if response.success:
|
||||
preview = (response.answer or "OK")[:200]
|
||||
logger.info("Session memory saved: %s", preview)
|
||||
else:
|
||||
logger.warning("Memory save returned unsuccessful: %s", response.answer)
|
||||
except Exception as e:
|
||||
logger.exception("Error saving session memory: %s", e)
|
||||
|
||||
def _schedule_memory_save(self, chat_id: str, messages: List[Dict]):
|
||||
"""Schedule a non-blocking memory save task."""
|
||||
if not self.auto_memory_job or not messages:
|
||||
return
|
||||
|
||||
task = asyncio.create_task(
|
||||
self._do_save_session_memory(chat_id, messages),
|
||||
name=f"memory_save_{chat_id}",
|
||||
)
|
||||
self._pending_memory_tasks.append(task)
|
||||
|
||||
# Clean up completed tasks from the tracking list
|
||||
self._pending_memory_tasks = [
|
||||
t for t in self._pending_memory_tasks if not t.done()
|
||||
]
|
||||
|
||||
# ─── Message Processing ─────────────────────────────────────────
|
||||
|
||||
async def process_message(self, sender_id: str, chat_id: str, content: str) -> Optional[str]:
|
||||
"""Process a user message through the ReMe agent."""
|
||||
# Track session messages for later memory save
|
||||
if chat_id not in self.session_messages:
|
||||
self.session_messages[chat_id] = []
|
||||
|
||||
self.session_messages[chat_id].append({
|
||||
"role": "user",
|
||||
"name": "user",
|
||||
"content": content,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
})
|
||||
|
||||
# Build system prompt with user profile
|
||||
system_prompt = build_system_prompt(self.user_profile_text)
|
||||
|
||||
# Create MCP client for AppWorld
|
||||
mcp_client = self._create_mcp_client()
|
||||
|
||||
# Determine which reme jobs to expose as tools
|
||||
job_tools = []
|
||||
if self.search_job:
|
||||
job_tools.append("search")
|
||||
if self.auto_memory_job:
|
||||
job_tools.extend(["auto_memory", "daily_write"])
|
||||
|
||||
# On first message in session, wait for pending memory saves
|
||||
# then search memory for relevant context
|
||||
memory_context = ""
|
||||
is_first_message = len(self.session_messages[chat_id]) == 1
|
||||
if is_first_message:
|
||||
# Wait for any in-flight memory saves from previous sessions
|
||||
await self._wait_for_pending_memory_saves()
|
||||
|
||||
# Search memory for relevant context
|
||||
if self.search_job:
|
||||
try:
|
||||
memory_context = await self._search_memory(content)
|
||||
if memory_context:
|
||||
logger.info("Found relevant memory: %d chars", len(memory_context))
|
||||
except Exception as e:
|
||||
logger.warning("Memory search on first message failed: %s", e)
|
||||
|
||||
try:
|
||||
# Prepend memory context if available
|
||||
user_message = content
|
||||
if memory_context:
|
||||
user_message = (
|
||||
f"[Relevant memories from previous sessions]\n"
|
||||
f"{memory_context}\n\n"
|
||||
f"[Current user message]\n{content}"
|
||||
)
|
||||
|
||||
# Call ReMe agent with MCP tools and memory tools
|
||||
reply_kwargs = {
|
||||
"system_prompt": system_prompt,
|
||||
"mcps": [mcp_client],
|
||||
"permission_mode": "bypass",
|
||||
}
|
||||
if job_tools:
|
||||
reply_kwargs["job_tools"] = job_tools
|
||||
|
||||
# Resume existing session for multi-turn continuity within same task
|
||||
if self.agent_session_id:
|
||||
reply_kwargs["resume"] = self.agent_session_id
|
||||
|
||||
result = await self.agent_wrapper.reply(user_message, **reply_kwargs)
|
||||
|
||||
reply_text = result.get("result", "")
|
||||
session_id = result.get("session_id", "")
|
||||
|
||||
if session_id:
|
||||
self.agent_session_id = session_id
|
||||
|
||||
# Track the assistant reply
|
||||
self.session_messages[chat_id].append({
|
||||
"role": "assistant",
|
||||
"name": "assistant",
|
||||
"content": reply_text,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
})
|
||||
|
||||
logger.info("Reply: %d chars, session=%s", len(reply_text), session_id)
|
||||
return reply_text
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("Error processing message: %s", e)
|
||||
return None
|
||||
|
||||
async def handle_reset(self, chat_id: str):
|
||||
"""Handle session reset: schedule non-blocking memory save and clear state."""
|
||||
messages = self.session_messages.pop(chat_id, [])
|
||||
if messages:
|
||||
self._schedule_memory_save(chat_id, messages)
|
||||
# Clear agent session for cross-task isolation
|
||||
self.agent_session_id = None
|
||||
|
||||
# ─── Test Server Communication ──────────────────────────────────
|
||||
|
||||
async def poll_test_server(self) -> List[Dict[str, Any]]:
|
||||
"""Poll Test Server for pending messages."""
|
||||
try:
|
||||
resp = await self.client.get(
|
||||
f"{self.test_server_url}/poll",
|
||||
params={"timeout": self.poll_timeout},
|
||||
)
|
||||
if resp.is_success:
|
||||
data = resp.json()
|
||||
messages = data.get("messages", [])
|
||||
if messages:
|
||||
logger.info("Received %d messages", len(messages))
|
||||
return messages
|
||||
except Exception as e:
|
||||
logger.warning("Poll error: %s", e)
|
||||
return []
|
||||
|
||||
async def send_to_test_server(self, chat_id: str, content: str) -> bool:
|
||||
"""Send reply back to Test Server."""
|
||||
payload = {
|
||||
"chat_id": chat_id,
|
||||
"content": content,
|
||||
"media": [],
|
||||
"meta": {},
|
||||
}
|
||||
try:
|
||||
resp = await self.client.post(
|
||||
f"{self.test_server_url}/send",
|
||||
json=payload,
|
||||
)
|
||||
if resp.is_success:
|
||||
logger.info("Sent reply: chat_id=%s len=%d", chat_id, len(content))
|
||||
return True
|
||||
logger.error("Failed to send: %s", resp.status_code)
|
||||
except Exception as e:
|
||||
logger.exception("Error sending to Test Server")
|
||||
return False
|
||||
|
||||
# ─── Main Loop ──────────────────────────────────────────────────
|
||||
|
||||
def _install_signal_handlers(self):
|
||||
"""Install SIGTERM/SIGINT handlers for graceful shutdown."""
|
||||
loop = asyncio.get_running_loop()
|
||||
for sig_name in ("SIGTERM", "SIGINT"):
|
||||
sig = getattr(signal, sig_name, None)
|
||||
if sig is not None:
|
||||
loop.add_signal_handler(sig, self._handle_shutdown_signal, sig_name)
|
||||
|
||||
def _handle_shutdown_signal(self, sig_name: str):
|
||||
"""Handle shutdown signal: stop the bridge loop gracefully."""
|
||||
logger.info("Received %s, initiating graceful shutdown...", sig_name)
|
||||
self.running = False
|
||||
|
||||
async def run(self):
|
||||
"""Main bridge loop."""
|
||||
await self.start()
|
||||
self._install_signal_handlers()
|
||||
|
||||
try:
|
||||
while self.running:
|
||||
messages = await self.poll_test_server()
|
||||
|
||||
for msg in messages:
|
||||
sender_id = msg.get("sender_id", "unknown")
|
||||
chat_id = msg.get("chat_id", "default")
|
||||
content = msg.get("content", "")
|
||||
|
||||
if not content:
|
||||
continue
|
||||
|
||||
logger.info("Processing: sender=%s chat=%s len=%d",
|
||||
sender_id, chat_id, len(content))
|
||||
|
||||
# Check for reset/new-session signal
|
||||
if content.strip().lower() in ("reset", "new session", "/new"):
|
||||
logger.info("Reset signal: chat_id=%s", chat_id)
|
||||
await self.handle_reset(chat_id)
|
||||
await self.send_to_test_server(chat_id, "New session started")
|
||||
continue
|
||||
|
||||
# Forward to ReMe agent
|
||||
reply = await self.process_message(sender_id, chat_id, content)
|
||||
|
||||
if reply:
|
||||
await self.send_to_test_server(chat_id, reply)
|
||||
else:
|
||||
logger.warning("No reply for chat_id=%s", chat_id)
|
||||
await self.send_to_test_server(
|
||||
chat_id, "[Error: Agent failed to generate response]"
|
||||
)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
logger.info("Interrupted by user")
|
||||
finally:
|
||||
# Save any remaining session memories
|
||||
for cid in list(self.session_messages.keys()):
|
||||
messages = self.session_messages.pop(cid, [])
|
||||
if messages:
|
||||
self._schedule_memory_save(cid, messages)
|
||||
# Wait for all pending memory saves to complete
|
||||
if self._pending_memory_tasks:
|
||||
logger.info("Waiting for %d pending memory saves to complete...",
|
||||
len(self._pending_memory_tasks))
|
||||
await self._wait_for_pending_memory_saves()
|
||||
await self.stop()
|
||||
|
||||
|
||||
async def main():
|
||||
# Ignore SIGHUP to prevent bridge from being killed (same fix as qwenpaw)
|
||||
signal.signal(signal.SIGHUP, signal.SIG_IGN)
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Bridge between Pi-Bench Test Server and ReMe agent"
|
||||
)
|
||||
parser.add_argument("--test-server-url", default="http://localhost:9999")
|
||||
parser.add_argument("--appworld-mcp-url", default="http://localhost:10000/mcp")
|
||||
parser.add_argument("--reme-dir", default=os.environ.get("REME_DIR", str(Path(__file__).resolve().parent / "ReMe")))
|
||||
parser.add_argument("--data-root", default="data")
|
||||
parser.add_argument("--user-id", default="researcher")
|
||||
parser.add_argument("--poll-timeout", type=int, default=30)
|
||||
parser.add_argument("--workspace-dir", default="")
|
||||
parser.add_argument("--model-name", default="qwen3.6-plus")
|
||||
parser.add_argument("--model-base-url", default="")
|
||||
parser.add_argument("--model-api-key", default="")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
bridge = ReMeBridge(
|
||||
test_server_url=args.test_server_url,
|
||||
appworld_mcp_url=args.appworld_mcp_url,
|
||||
reme_dir=args.reme_dir,
|
||||
data_root=args.data_root,
|
||||
user_id=args.user_id,
|
||||
poll_timeout=args.poll_timeout,
|
||||
workspace_dir=args.workspace_dir,
|
||||
model_name=args.model_name,
|
||||
model_base_url=args.model_base_url,
|
||||
model_api_key=args.model_api_key,
|
||||
)
|
||||
|
||||
await bridge.run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
40
benchmark/pi-bench/config/models/nanobot_reme.yaml
Normal file
40
benchmark/pi-bench/config/models/nanobot_reme.yaml
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
# NanoBot + ReMe model configuration for Pi-Bench
|
||||
# Uses NanoBot as the agent with ReMe cross-session memory
|
||||
# and AppWorld MCP tools for task execution.
|
||||
|
||||
model:
|
||||
model: nanobot_reme
|
||||
base_url: "http://localhost:9999"
|
||||
api_key: "dummy"
|
||||
provider: custom
|
||||
max_tokens: 16384
|
||||
max_tool_iterations: 120
|
||||
memory_window: 100
|
||||
|
||||
user_agent:
|
||||
model: qwen-max
|
||||
base_url: "${USER_BASE_URL}"
|
||||
api_key: "${USER_API_KEY}"
|
||||
temperature: 0.0
|
||||
request_timeout: 360.0
|
||||
|
||||
judger:
|
||||
model: qwen-max
|
||||
base_url: "${JUDGER_BASE_URL}"
|
||||
api_key: "${JUDGER_API_KEY}"
|
||||
temperature: 0.0
|
||||
request_timeout: 360.0
|
||||
|
||||
tools:
|
||||
web_search_max_results: 10
|
||||
|
||||
nanobot:
|
||||
trace_logs_dir: "~/.nanobot/trace_logs"
|
||||
workspace_dir: "~/.nanobot/workspace"
|
||||
copy_task_assets_to_workspace: true
|
||||
|
||||
run:
|
||||
output_dir: outputs
|
||||
log_level: INFO
|
||||
user_mode: llm
|
||||
turn_timeout: 2400.0
|
||||
40
benchmark/pi-bench/config/models/reme.yaml
Normal file
40
benchmark/pi-bench/config/models/reme.yaml
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
# ReMe model configuration for Pi-Bench
|
||||
# Uses ReMe's AgentScope agent with Dashscope as the LLM backend
|
||||
|
||||
model:
|
||||
model: reme
|
||||
base_url: "http://localhost:8088"
|
||||
api_key: "dummy"
|
||||
provider: custom
|
||||
max_tokens: 16384
|
||||
max_tool_iterations: 120
|
||||
memory_window: 100
|
||||
|
||||
user_agent:
|
||||
model: qwen-max
|
||||
base_url: "${USER_BASE_URL}"
|
||||
api_key: "${USER_API_KEY}"
|
||||
temperature: 0.0
|
||||
request_timeout: 360.0
|
||||
|
||||
judger:
|
||||
model: qwen-max
|
||||
base_url: "${JUDGER_BASE_URL}"
|
||||
api_key: "${JUDGER_API_KEY}"
|
||||
temperature: 0.0
|
||||
request_timeout: 360.0
|
||||
|
||||
tools:
|
||||
brave_search_api_key: "${BRAVE_SEARCH_API_KEY}"
|
||||
web_search_max_results: 10
|
||||
|
||||
nanobot:
|
||||
trace_logs_dir: "~/.nanobot/trace_logs"
|
||||
workspace_dir: "~/.nanobot/workspace"
|
||||
copy_task_assets_to_workspace: true
|
||||
|
||||
run:
|
||||
output_dir: outputs
|
||||
log_level: INFO
|
||||
user_mode: llm
|
||||
turn_timeout: 2400.0
|
||||
63
benchmark/pi-bench/data/Financier/episode.yaml
Normal file
63
benchmark/pi-bench/data/Financier/episode.yaml
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
schema_version: v2
|
||||
episode_id: ep_Financier
|
||||
user_id: Financier
|
||||
tasks:
|
||||
- task_id: Financier_task_001
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_002
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_003
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_004
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_005
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_006
|
||||
depends_on: [Financier_task_003]
|
||||
|
||||
- task_id: Financier_task_007
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_008
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_009
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_010
|
||||
depends_on: [Financier_task_001]
|
||||
|
||||
- task_id: Financier_task_011
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_012
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_013
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_014
|
||||
depends_on: [Financier_task_004]
|
||||
|
||||
- task_id: Financier_task_015
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_016
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_017
|
||||
depends_on: [Financier_task_007, Financier_task_012]
|
||||
|
||||
- task_id: Financier_task_018
|
||||
depends_on: []
|
||||
|
||||
- task_id: Financier_task_019
|
||||
depends_on: [Financier_task_009, Financier_task_013]
|
||||
|
||||
- task_id: Financier_task_020
|
||||
depends_on: [Financier_task_016, Financier_task_018]
|
||||
20
benchmark/pi-bench/data/Financier/profile.yaml
Normal file
20
benchmark/pi-bench/data/Financier/profile.yaml
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
schema_version: v2
|
||||
user_id: Financier
|
||||
role: |-
|
||||
# Financier
|
||||
|
||||
- Identity & Background: The user works on financial model risk assessment and independent model validation, especially PD scorecards, VaR backtesting, macro-risk data workflows, yield-curve modeling, and governance-oriented validation writing. They also occasionally use the agent for practical personal tasks across Gmail, Todoist, Phone, Simple Note, and Amazon.
|
||||
|
||||
- Interaction Preferences: Use English by default. For finance tasks, lead with the conclusion and the highest-risk issues, keep regulatory or methodological anchors visible, and avoid formula-first explanations unless explicitly requested. For personal execution tasks, prefer concise, structured, scan-friendly outputs and reliable follow-through.
|
||||
|
||||
- Output Preferences: For validation work, prefer clear Findings, Limitations, Recommendations, and explicit model-use judgment. For everyday app tasks, prefer concrete deadlines, auditable action summaries, and clean final artifacts.
|
||||
|
||||
- Additional Context: This shuffled task set spans local report review, public-source validation writing, app-based coordination tasks, code/data scripting tasks, and cross-task reuse of earlier finance decisions and source choices.
|
||||
|
||||
preferences:
|
||||
language: en
|
||||
verbosity: high
|
||||
long_term_goals:
|
||||
- Build reusable model-validation templates for credit-risk and market-risk models.
|
||||
- Improve risk-first validation writing that supports model-use decisions and remediation planning.
|
||||
- Use agents to reduce execution overhead across both professional finance workflows and everyday coordination tasks.
|
||||
|
|
@ -0,0 +1,144 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"safe_exact_title_preclean": 0,
|
||||
"rough_to_final_note_workflow": 0,
|
||||
"final_note_has_required_headings": 0,
|
||||
"final_note_has_trip_timing": 0,
|
||||
"final_note_lists_required_items": 0,
|
||||
"final_note_mentions_weather_risks": 0,
|
||||
"final_note_mentions_food_cooler": 0,
|
||||
"final_note_has_pending_confirmations": 0,
|
||||
"final_note_has_friday_2100_deadline": 0,
|
||||
}
|
||||
|
||||
NOTE_TITLE = "Weekend Camping Departure Memo - Qingyuan Reservoir"
|
||||
REQUIRED_HEADINGS = (
|
||||
"Meet-up and Travel",
|
||||
"Must-Bring Items",
|
||||
"Weather and Risk Notes",
|
||||
"Pre-Departure Confirmations",
|
||||
)
|
||||
REQUIRED_ITEMS = (
|
||||
"headlamp",
|
||||
"light rain jacket",
|
||||
"2L",
|
||||
"ID card",
|
||||
"power bank",
|
||||
)
|
||||
PENDING_ITEMS = (
|
||||
"carpool",
|
||||
"ice",
|
||||
"stove fuel",
|
||||
)
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
return name.removeprefix("mcp_appworld_")
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def _id_text(value: Any) -> str:
|
||||
return str(value) if isinstance(value, (int, str)) else ""
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
|
||||
search = None
|
||||
delete_calls: list[dict[str, Any]] = []
|
||||
create_note = None
|
||||
add_note = None
|
||||
update_note = None
|
||||
show_note = None
|
||||
|
||||
for item in tools_history or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = _norm(str(item.get("tool_name") or ""))
|
||||
if name == "simple_note__search_notes":
|
||||
search = item
|
||||
elif name == "simple_note__delete_note":
|
||||
delete_calls.append(item)
|
||||
elif name == "simple_note__create_note":
|
||||
create_note = item
|
||||
elif name == "simple_note__add_content_to_note":
|
||||
add_note = item
|
||||
elif name == "simple_note__update_note":
|
||||
update_note = item
|
||||
elif name == "simple_note__show_note":
|
||||
show_note = item
|
||||
|
||||
exact_match_note_ids: set[Any] = set()
|
||||
if isinstance(search, dict):
|
||||
if _text((search.get("call") or {}).get("query")) == NOTE_TITLE:
|
||||
response = (search.get("result") or {}).get("response") or []
|
||||
if isinstance(response, list):
|
||||
for note in response:
|
||||
if isinstance(note, dict) and _text(note.get("title")) == NOTE_TITLE:
|
||||
exact_match_note_ids.add(note.get("note_id"))
|
||||
|
||||
deleted_note_ids = {
|
||||
(item.get("call") or {}).get("note_id")
|
||||
for item in delete_calls
|
||||
if isinstance(item, dict)
|
||||
}
|
||||
if search:
|
||||
if exact_match_note_ids:
|
||||
if deleted_note_ids == exact_match_note_ids:
|
||||
result["safe_exact_title_preclean"] = 1
|
||||
elif not deleted_note_ids:
|
||||
result["safe_exact_title_preclean"] = 1
|
||||
|
||||
created_note_id = _id_text((((create_note or {}).get("result") or {}).get("response") or {}).get("note_id"))
|
||||
added_note_id = _id_text(((add_note or {}).get("call") or {}).get("note_id"))
|
||||
updated_note_id = _id_text(((update_note or {}).get("call") or {}).get("note_id"))
|
||||
if create_note and add_note and update_note and created_note_id and created_note_id == added_note_id == updated_note_id:
|
||||
if _text((create_note.get("call") or {}).get("title")) == NOTE_TITLE:
|
||||
result["rough_to_final_note_workflow"] = 1
|
||||
|
||||
final_title = ""
|
||||
final_content = ""
|
||||
if isinstance(show_note, dict):
|
||||
response = (show_note.get("result") or {}).get("response") or {}
|
||||
final_title = _text(response.get("title"))
|
||||
final_content = _text(response.get("content"))
|
||||
elif isinstance(update_note, dict):
|
||||
final_title = _text((create_note or {}).get("call", {}).get("title"))
|
||||
final_content = _text((update_note.get("call") or {}).get("content"))
|
||||
|
||||
lowered = final_content.lower()
|
||||
|
||||
if final_title == NOTE_TITLE and all(heading in final_content for heading in REQUIRED_HEADINGS):
|
||||
result["final_note_has_required_headings"] = 1
|
||||
|
||||
if "Saturday 07:20" in final_content and "North Parking Lot" in final_content and "90" in final_content:
|
||||
result["final_note_has_trip_timing"] = 1
|
||||
|
||||
if all(item.lower() in lowered for item in ["headlamp", "light rain jacket", "2l", "id card", "power bank"]):
|
||||
result["final_note_lists_required_items"] = 1
|
||||
|
||||
if (
|
||||
any(token in lowered for token in ("after dark", "once it gets dark"))
|
||||
and any(token in lowered for token in ("windy", "cooler", "temperature drops"))
|
||||
and "18:00" in final_content
|
||||
and any(token in lowered for token in ("light rain", "rain"))
|
||||
):
|
||||
result["final_note_mentions_weather_risks"] = 1
|
||||
|
||||
if "cooler" in lowered and any(token in lowered for token in ("first", "before anything else", "go into the cooler first")):
|
||||
result["final_note_mentions_food_cooler"] = 1
|
||||
|
||||
if all(item in lowered for item in PENDING_ITEMS):
|
||||
result["final_note_has_pending_confirmations"] = 1
|
||||
|
||||
if "Friday 21:00" in final_content and any(token in lowered for token in ("confirm by", "completed by", "locked by")):
|
||||
result["final_note_has_friday_2100_deadline"] = 1
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,131 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"gmail_draft_workflow_completed": 0,
|
||||
"sent_to_real_recipient": 0,
|
||||
"subject_exact_match": 0,
|
||||
"body_is_matter_specific": 0,
|
||||
"body_covers_three_confirmation_items": 0,
|
||||
"body_sets_deadline": 0,
|
||||
"body_mentions_badge_fallback": 0,
|
||||
"body_requests_immediate_risk_flag": 0,
|
||||
"body_uses_scan_friendly_structure": 0,
|
||||
"outbox_and_thread_verified": 0,
|
||||
}
|
||||
|
||||
SUBJECT = "Pre-Event Confirmation | Nanshan Client Product Demo | Wednesday Setup"
|
||||
RECIPIENT = "la-mcco@gmail.com"
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
return name.removeprefix("mcp_appworld_")
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def _id_text(value: Any) -> str:
|
||||
return str(value) if isinstance(value, (int, str)) else ""
|
||||
|
||||
|
||||
def _count_numbered_items(body: str) -> int:
|
||||
count = 0
|
||||
for raw_line in body.splitlines():
|
||||
line = raw_line.strip()
|
||||
if len(line) > 2 and line[0].isdigit() and line[1] in {".", ")"}:
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
create_draft = None
|
||||
update_draft = None
|
||||
send_from_draft = None
|
||||
show_outbox = None
|
||||
show_thread = None
|
||||
|
||||
for item in tools_history or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = _norm(str(item.get("tool_name") or ""))
|
||||
if name == "gmail__create_draft":
|
||||
create_draft = item
|
||||
elif name == "gmail__update_draft":
|
||||
update_draft = item
|
||||
elif name == "gmail__send_email_from_draft":
|
||||
send_from_draft = item
|
||||
elif name == "gmail__show_outbox_threads":
|
||||
show_outbox = item
|
||||
elif name == "gmail__show_thread":
|
||||
show_thread = item
|
||||
|
||||
draft_id = _id_text((((create_draft or {}).get("result") or {}).get("response") or {}).get("draft_id"))
|
||||
updated_draft_id = _id_text(((update_draft or {}).get("call") or {}).get("draft_id"))
|
||||
sent_draft_id = _id_text(((send_from_draft or {}).get("call") or {}).get("draft_id"))
|
||||
if create_draft and update_draft and send_from_draft:
|
||||
if draft_id and draft_id == updated_draft_id == sent_draft_id:
|
||||
result["gmail_draft_workflow_completed"] = 1
|
||||
elif updated_draft_id and updated_draft_id == sent_draft_id:
|
||||
result["gmail_draft_workflow_completed"] = 1
|
||||
|
||||
final_subject = ""
|
||||
final_body = ""
|
||||
final_recipients: list[str] = []
|
||||
if isinstance(update_draft, dict):
|
||||
call = update_draft.get("call") or {}
|
||||
final_subject = _text(call.get("subject"))
|
||||
final_body = _text(call.get("body"))
|
||||
emails = call.get("email_addresses") or []
|
||||
if isinstance(emails, list):
|
||||
final_recipients = [str(email) for email in emails]
|
||||
elif isinstance(create_draft, dict):
|
||||
call = create_draft.get("call") or {}
|
||||
final_subject = _text(call.get("subject"))
|
||||
final_body = _text(call.get("body"))
|
||||
emails = call.get("recipient_email_addresses") or []
|
||||
if isinstance(emails, list):
|
||||
final_recipients = [str(email) for email in emails]
|
||||
|
||||
lowered = final_body.lower()
|
||||
|
||||
if RECIPIENT in final_recipients:
|
||||
result["sent_to_real_recipient"] = 1
|
||||
if final_subject == SUBJECT:
|
||||
result["subject_exact_match"] = 1
|
||||
|
||||
if "nanshan client product demo" in lowered and "wednesday setup" in lowered:
|
||||
result["body_is_matter_specific"] = 1
|
||||
|
||||
if (
|
||||
("projector" in lowered and "hdmi" in lowered)
|
||||
and "badge" in lowered
|
||||
and "18" in final_body
|
||||
and "agenda" in lowered
|
||||
):
|
||||
result["body_covers_three_confirmation_items"] = 1
|
||||
|
||||
if "2026-04-29 16:00" in final_body and any(token in lowered for token in ("reply by", "confirm by", "respond by")):
|
||||
result["body_sets_deadline"] = 1
|
||||
|
||||
if "badge" in lowered and any(token in lowered for token in ("current draft", "draft version")) and any(
|
||||
token in lowered for token in ("expected final confirmation time", "expected finalization time", "final confirmation time")
|
||||
):
|
||||
result["body_mentions_badge_fallback"] = 1
|
||||
|
||||
if any(token in lowered for token in ("flag the risk immediately", "flag the risk right away", "tell me immediately")) and any(
|
||||
token in lowered for token in ("venue", "equipment", "printing", "printed materials")
|
||||
):
|
||||
result["body_requests_immediate_risk_flag"] = 1
|
||||
|
||||
if _count_numbered_items(final_body) >= 3:
|
||||
result["body_uses_scan_friendly_structure"] = 1
|
||||
|
||||
if isinstance(show_outbox, dict) and isinstance(show_thread, dict):
|
||||
result["outbox_and_thread_verified"] = 1
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,210 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"project_created": 0,
|
||||
"section_created": 0,
|
||||
"three_tasks_created": 0,
|
||||
"priority_task_assigned_to_self": 0,
|
||||
"priority_task_updated_with_checkpoint": 0,
|
||||
"label_created_and_bound": 0,
|
||||
"comment_posts_escalation_rule": 0,
|
||||
"comment_was_reviewed": 0,
|
||||
"full_cleanup_completed": 0,
|
||||
}
|
||||
|
||||
PROJECT_NAME = "Temporary Launch Rehearsal Board - Beacon"
|
||||
SECTION_NAME = "Tonight Rehearsal"
|
||||
LABEL_NAME = "launch-blocker"
|
||||
PRIORITY_TASK_TITLE = "Walk through Beacon share slides"
|
||||
DEVICE_TASK_TITLE = "Check demo device charging and screen recording"
|
||||
BADGE_TASK_TITLE = "Confirm guest badge pickup and sign-in sheet"
|
||||
SELF_EMAIL = "stmcco@gmail.com"
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
return name.removeprefix("mcp_appworld_")
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
|
||||
create_project = None
|
||||
create_section = None
|
||||
create_label = None
|
||||
create_tasks: list[dict[str, Any]] = []
|
||||
assign_calls: list[dict[str, Any]] = []
|
||||
update_calls: list[dict[str, Any]] = []
|
||||
add_label_calls: list[dict[str, Any]] = []
|
||||
post_comment_calls: list[dict[str, Any]] = []
|
||||
show_comment_calls: list[dict[str, Any]] = []
|
||||
remove_label_calls: list[dict[str, Any]] = []
|
||||
delete_comment_calls: list[dict[str, Any]] = []
|
||||
delete_task_calls: list[dict[str, Any]] = []
|
||||
delete_label = None
|
||||
delete_section = None
|
||||
delete_project = None
|
||||
|
||||
login_email = SELF_EMAIL
|
||||
|
||||
project_id = None
|
||||
section_id = None
|
||||
label_id = None
|
||||
created_task_ids: set[Any] = set()
|
||||
task_id_by_title: dict[str, Any] = {}
|
||||
|
||||
for item in tools_history or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = _norm(str(item.get("tool_name") or ""))
|
||||
call = item.get("call") or {}
|
||||
result_payload = (item.get("result") or {}).get("response") or {}
|
||||
if name == "todoist__login":
|
||||
username = _text(call.get("username"))
|
||||
if username:
|
||||
login_email = username
|
||||
elif name == "todoist__create_project":
|
||||
create_project = item
|
||||
project_id = result_payload.get("project_id")
|
||||
elif name == "todoist__create_section":
|
||||
create_section = item
|
||||
section_id = result_payload.get("section_id")
|
||||
elif name == "todoist__create_label":
|
||||
create_label = item
|
||||
label_id = result_payload.get("label_id")
|
||||
elif name == "todoist__create_task":
|
||||
create_tasks.append(item)
|
||||
task_id = result_payload.get("task_id")
|
||||
title = _text(call.get("title"))
|
||||
if task_id is not None:
|
||||
created_task_ids.add(task_id)
|
||||
if title and task_id is not None:
|
||||
task_id_by_title[title] = task_id
|
||||
elif name == "todoist__assign_or_unassign_task":
|
||||
assign_calls.append(item)
|
||||
elif name == "todoist__update_task":
|
||||
update_calls.append(item)
|
||||
elif name == "todoist__add_label_to_task":
|
||||
add_label_calls.append(item)
|
||||
elif name == "todoist__post_task_comment":
|
||||
post_comment_calls.append(item)
|
||||
elif name == "todoist__show_task_comments":
|
||||
show_comment_calls.append(item)
|
||||
elif name == "todoist__remove_label_from_task":
|
||||
remove_label_calls.append(item)
|
||||
elif name == "todoist__delete_task_comment":
|
||||
delete_comment_calls.append(item)
|
||||
elif name == "todoist__delete_task":
|
||||
delete_task_calls.append(item)
|
||||
elif name == "todoist__delete_label":
|
||||
delete_label = item
|
||||
elif name == "todoist__delete_section":
|
||||
delete_section = item
|
||||
elif name == "todoist__delete_project":
|
||||
delete_project = item
|
||||
|
||||
if isinstance(create_project, dict) and _text((create_project.get("call") or {}).get("name")) == PROJECT_NAME:
|
||||
result["project_created"] = 1
|
||||
|
||||
if (
|
||||
isinstance(create_section, dict)
|
||||
and _text((create_section.get("call") or {}).get("name")) == SECTION_NAME
|
||||
and (create_section.get("call") or {}).get("project_id") == project_id
|
||||
):
|
||||
result["section_created"] = 1
|
||||
|
||||
matched_titles: set[str] = set()
|
||||
for item in create_tasks:
|
||||
call = item.get("call") or {}
|
||||
title = _text(call.get("title"))
|
||||
description = _text(call.get("description"))
|
||||
lowered = description.lower()
|
||||
if call.get("project_id") != project_id or call.get("section_id") != section_id:
|
||||
continue
|
||||
if title == PRIORITY_TASK_TITLE and "slides" in lowered:
|
||||
matched_titles.add(title)
|
||||
elif title == DEVICE_TASK_TITLE and "charging" in lowered and "record" in lowered:
|
||||
matched_titles.add(title)
|
||||
elif title == BADGE_TASK_TITLE and "badge" in lowered and "sign-in" in lowered:
|
||||
matched_titles.add(title)
|
||||
if matched_titles == {PRIORITY_TASK_TITLE, DEVICE_TASK_TITLE, BADGE_TASK_TITLE}:
|
||||
result["three_tasks_created"] = 1
|
||||
|
||||
priority_task_id = task_id_by_title.get(PRIORITY_TASK_TITLE)
|
||||
for item in assign_calls:
|
||||
call = item.get("call") or {}
|
||||
if call.get("task_id") == priority_task_id and _text(call.get("assignee_email")) == login_email:
|
||||
result["priority_task_assigned_to_self"] = 1
|
||||
break
|
||||
|
||||
for item in update_calls:
|
||||
call = item.get("call") or {}
|
||||
description = _text(call.get("description"))
|
||||
if call.get("task_id") == priority_task_id and "2026-04-29 17:45" in description and "2026-04-29 18:30" in description:
|
||||
result["priority_task_updated_with_checkpoint"] = 1
|
||||
break
|
||||
|
||||
for item in add_label_calls:
|
||||
call = item.get("call") or {}
|
||||
if call.get("task_id") == priority_task_id and call.get("label_id") == label_id and isinstance(create_label, dict):
|
||||
if _text((create_label.get("call") or {}).get("name")) == LABEL_NAME:
|
||||
result["label_created_and_bound"] = 1
|
||||
break
|
||||
|
||||
comment_id = None
|
||||
for item in post_comment_calls:
|
||||
call = item.get("call") or {}
|
||||
content = _text(call.get("content"))
|
||||
comment_id = (((item.get("result") or {}).get("response") or {}).get("task_comment_id"))
|
||||
lowered = content.lower()
|
||||
if call.get("task_id") == priority_task_id and "2026-04-29 18:30" in content and "organizer" in lowered:
|
||||
result["comment_posts_escalation_rule"] = 1
|
||||
break
|
||||
|
||||
for item in show_comment_calls:
|
||||
call = item.get("call") or {}
|
||||
if call.get("task_id") == priority_task_id:
|
||||
result["comment_was_reviewed"] = 1
|
||||
break
|
||||
|
||||
removed_label_ok = False
|
||||
if priority_task_id is not None and label_id is not None:
|
||||
for item in remove_label_calls:
|
||||
call = item.get("call") or {}
|
||||
if call.get("task_id") == priority_task_id and call.get("label_id") == label_id:
|
||||
removed_label_ok = True
|
||||
break
|
||||
|
||||
deleted_task_ids = {
|
||||
(item.get("call") or {}).get("task_id")
|
||||
for item in delete_task_calls
|
||||
if isinstance(item, dict)
|
||||
}
|
||||
deleted_comment_ok = False
|
||||
if comment_id is not None:
|
||||
for item in delete_comment_calls:
|
||||
if (item.get("call") or {}).get("task_comment_id") == comment_id:
|
||||
deleted_comment_ok = True
|
||||
break
|
||||
|
||||
if (
|
||||
created_task_ids
|
||||
and deleted_task_ids == created_task_ids
|
||||
and removed_label_ok
|
||||
and deleted_comment_ok
|
||||
and isinstance(delete_label, dict)
|
||||
and (delete_label.get("call") or {}).get("label_id") == label_id
|
||||
and isinstance(delete_section, dict)
|
||||
and (delete_section.get("call") or {}).get("section_id") == section_id
|
||||
and isinstance(delete_project, dict)
|
||||
and (delete_project.get("call") or {}).get("project_id") == project_id
|
||||
):
|
||||
result["full_cleanup_completed"] = 1
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,95 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"searched_contact_or_thread": 0,
|
||||
"opened_text_window": 0,
|
||||
"message_sent_to_target": 0,
|
||||
"message_is_matter_specific": 0,
|
||||
"message_has_time_and_place": 0,
|
||||
"message_lists_required_items": 0,
|
||||
"message_sets_reply_deadline": 0,
|
||||
"message_requests_risk_flag": 0,
|
||||
"message_mentions_photo_fallback": 0,
|
||||
"message_is_scan_friendly": 0,
|
||||
}
|
||||
|
||||
TARGET_NUMBER = "2873148336"
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
return name.removeprefix("mcp_appworld_")
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
|
||||
search_contacts = None
|
||||
search_texts = None
|
||||
show_window = None
|
||||
send_text = None
|
||||
|
||||
for item in tools_history or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = _norm(str(item.get("tool_name") or ""))
|
||||
if name == "phone__search_contacts":
|
||||
search_contacts = item
|
||||
elif name == "phone__search_text_messages":
|
||||
search_texts = item
|
||||
elif name == "phone__show_text_message_window":
|
||||
show_window = item
|
||||
elif name == "phone__send_text_message":
|
||||
send_text = item
|
||||
|
||||
if isinstance(search_contacts, dict) or isinstance(search_texts, dict):
|
||||
if isinstance(search_texts, dict):
|
||||
phone_number = _text((search_texts.get("call") or {}).get("phone_number"))
|
||||
if phone_number == TARGET_NUMBER:
|
||||
result["searched_contact_or_thread"] = 1
|
||||
else:
|
||||
result["searched_contact_or_thread"] = 1
|
||||
|
||||
if isinstance(show_window, dict) and _text((show_window.get("call") or {}).get("phone_number")) == TARGET_NUMBER:
|
||||
result["opened_text_window"] = 1
|
||||
|
||||
final_message = ""
|
||||
if isinstance(send_text, dict):
|
||||
call = send_text.get("call") or {}
|
||||
if _text(call.get("phone_number")) == TARGET_NUMBER:
|
||||
result["message_sent_to_target"] = 1
|
||||
final_message = _text(call.get("message"))
|
||||
|
||||
lowered = final_message.lower()
|
||||
|
||||
if "chestnut" in lowered and any(token in lowered for token in ("boarding", "handover", "drop-off")):
|
||||
result["message_is_matter_specific"] = 1
|
||||
|
||||
if "07:40" in final_message and "Riverside Vet Clinic" in final_message and "west gate" in lowered:
|
||||
result["message_has_time_and_place"] = 1
|
||||
|
||||
if all(token in lowered for token in ("carrier", "vaccination booklet", "meal pouch")):
|
||||
result["message_lists_required_items"] = 1
|
||||
|
||||
if "21:30" in final_message and any(token in lowered for token in ("confirm by", "reply by", "let me know by")):
|
||||
result["message_sets_reply_deadline"] = 1
|
||||
|
||||
if any(token in lowered for token in ("pickup timing changes", "timing changes", "schedule changes")) and any(
|
||||
token in lowered for token in ("tell me immediately", "let me know immediately", "tell me right away")
|
||||
) and "vaccination booklet" in lowered:
|
||||
result["message_requests_risk_flag"] = 1
|
||||
|
||||
if any(token in lowered for token in ("send a photo first", "photo first", "send me a photo")):
|
||||
result["message_mentions_photo_fallback"] = 1
|
||||
|
||||
sentence_breaks = final_message.count("\n") + final_message.count(";") + final_message.count(".")
|
||||
if sentence_breaks >= 2:
|
||||
result["message_is_scan_friendly"] = 1
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"used_amazon_search_products": 0,
|
||||
"used_amazon_add_to_cart": 0,
|
||||
"used_amazon_show_cart": 0,
|
||||
"target_product_in_cart": 0,
|
||||
"target_quantity_is_one": 0,
|
||||
}
|
||||
|
||||
TARGET_PRODUCT_ID = 3
|
||||
|
||||
|
||||
def _tool_name_matches(name: str, suffix: str) -> bool:
|
||||
tool_name = (name or "").strip().lower()
|
||||
suffix = suffix.lower()
|
||||
return tool_name == suffix or tool_name.endswith(suffix)
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
|
||||
if not isinstance(tools_history, list) or not tools_history:
|
||||
return result
|
||||
|
||||
last_show_cart: dict[str, Any] | None = None
|
||||
|
||||
for item in tools_history:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
|
||||
tool_name = str(item.get("tool_name") or "")
|
||||
call_payload = item.get("call") or {}
|
||||
|
||||
if _tool_name_matches(tool_name, "amazon__search_products"):
|
||||
result["used_amazon_search_products"] = 1
|
||||
|
||||
if _tool_name_matches(tool_name, "amazon__add_product_to_cart"):
|
||||
if isinstance(call_payload, dict) and call_payload.get("product_id") == TARGET_PRODUCT_ID:
|
||||
result["used_amazon_add_to_cart"] = 1
|
||||
|
||||
if _tool_name_matches(tool_name, "amazon__show_cart"):
|
||||
last_show_cart = item
|
||||
|
||||
if not isinstance(last_show_cart, dict):
|
||||
return result
|
||||
|
||||
result["used_amazon_show_cart"] = 1
|
||||
response_payload = last_show_cart.get("result") or {}
|
||||
response_block = response_payload.get("response") or {}
|
||||
cart_items = response_block.get("cart_items") or []
|
||||
if not isinstance(cart_items, list):
|
||||
return result
|
||||
|
||||
for item in cart_items:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
if item.get("product_id") == TARGET_PRODUCT_ID:
|
||||
result["target_product_in_cart"] = 1
|
||||
if item.get("quantity") == 1:
|
||||
result["target_quantity_is_one"] = 1
|
||||
break
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
from typing import Any
|
||||
import os
|
||||
import yaml
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
possible_paths = []
|
||||
config_path = Path("config") / "nanobot.yaml"
|
||||
if config_path.exists():
|
||||
try:
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
config = yaml.safe_load(f)
|
||||
if config and 'nanobot' in config and 'workspace_dir' in config['nanobot']:
|
||||
workspace_dir = os.path.expanduser(config['nanobot']['workspace_dir'])
|
||||
possible_paths.append(Path(workspace_dir) / "risk_correlation_pipeline.py")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
content = None
|
||||
for file_path in possible_paths:
|
||||
if file_path.exists() and file_path.is_file():
|
||||
try:
|
||||
content = file_path.read_text(encoding='utf-8')
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if content is None:
|
||||
return {
|
||||
"has_volatility_annualization": 0,
|
||||
"uses_zipfile_extraction": 0,
|
||||
"uses_spark_distributed_read": 0
|
||||
}
|
||||
|
||||
# 1. 修复后的年化波动率检测 (包容嵌套的 F.lit(252))
|
||||
# 只要在同一行或紧挨着的地方出现了 sqrt 并且后面跟着 250/252，就算对
|
||||
has_volatility_annualization = 1 if bool(re.search(r'sqrt[^\n]*?25[02]|25[02]\.?0?\s*\*\*\s*0\.5', content)) else 0
|
||||
|
||||
# 2. 压缩包预处理常识 (保持不变)
|
||||
uses_zipfile_extraction = 1 if bool(re.search(r'import\s+zipfile|from\s+zipfile', content)) else 0
|
||||
|
||||
# 3. 修复后的 Spark 分布式读取检测 (包容链式调用的 .option())
|
||||
# 只要出现了 spark.read 并且后续调用链中出现了 .csv( 或 .format( 就算对
|
||||
uses_spark_distributed_read = 1 if bool(re.search(r'spark\.read(?:(?:[\r\n\s\\]*\.)[a-zA-Z0-9_]+(?:\([^)]*\)))*[\r\n\s\\]*\.(?:csv|format)\s*\(', content)) else 0
|
||||
|
||||
return {
|
||||
"has_volatility_annualization": has_volatility_annualization,
|
||||
"uses_zipfile_extraction": uses_zipfile_extraction,
|
||||
"uses_spark_distributed_read": uses_spark_distributed_read
|
||||
}
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
from typing import Any
|
||||
import os
|
||||
import yaml
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
# --- 1. 工作区路径解析 (保持不变) ---
|
||||
possible_paths = []
|
||||
config_path = Path("config") / "nanobot.yaml"
|
||||
if config_path.exists():
|
||||
try:
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
config = yaml.safe_load(f)
|
||||
if config and 'nanobot' in config and 'workspace_dir' in config['nanobot']:
|
||||
workspace_dir = config['nanobot']['workspace_dir']
|
||||
workspace_dir = os.path.expanduser(workspace_dir)
|
||||
possible_paths.append(Path(workspace_dir) / "yield_curve_fitter.py")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
content = None
|
||||
for file_path in possible_paths:
|
||||
if file_path.exists() and file_path.is_file():
|
||||
try:
|
||||
content = file_path.read_text(encoding='utf-8')
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 如果没生成文件，直接全 0 判负
|
||||
if content is None:
|
||||
return {
|
||||
"dynamic_lambda_calibration": 0,
|
||||
"uses_advanced_ols": 0,
|
||||
"targeted_api_fetching": 0,
|
||||
"has_robust_cleaning": 0,
|
||||
"outputs_three_artifacts": 0
|
||||
}
|
||||
|
||||
# --- 2. 高阶量化与工程标准扫描 ---
|
||||
|
||||
# 1. 动态 Decay 参数校准 (Dynamic Lambda Calibration)
|
||||
has_dynamic_loop = bool(re.search(r'for\s+[^:]+:\s*(?:.|\n){1,150}?(?:exp|ols|lstsq|fit)', content, re.IGNORECASE))
|
||||
has_hardcoded_lazy_lambda = bool(re.search(r'(LAMBDA|LAM|TAU)(_FIXED|FIXED_)?\s*=\s*0\.\d+|lam\s*=\s*0\.\d+', content, re.IGNORECASE))
|
||||
|
||||
dynamic_lambda_calibration = 1 if (has_dynamic_loop and not has_hardcoded_lazy_lambda) else 0
|
||||
# 2. 高级 OLS 闭式解法 (Advanced OLS)
|
||||
uses_advanced_ols = 1 if 'linalg.lstsq' in content else 0
|
||||
|
||||
# 3. 定向 API 获取 (Targeted API Fetching) [★ 本次核心更新]
|
||||
# 合理点：优质代码会通过循环年份拼接 URL，劣质代码会用 date_value=all 一次性拉取 30 年数据。
|
||||
# 正向：寻找按年份循环获取的特征 (例如 for year in..., for y in YEARS)
|
||||
has_year_loop = bool(re.search(r'for\s+[a-zA-Z_]+\s+in\s+([a-zA-Z_]+YEARS|range\(|\[2023)', content))
|
||||
# 反向：绝对不能出现傻瓜式的全量拉取参数
|
||||
has_lazy_all_fetch = 'date_value=all' in content
|
||||
targeted_api_fetching = 1 if (has_year_loop and not has_lazy_all_fetch) else 0
|
||||
|
||||
# 4. 鲁棒的时序数据清洗 (Robust Data Preprocessing) [★ 本次核心更新]
|
||||
# 合理点：金融时序数据处理中，遇到缺失值必须先尝试前向填充 (ffill)，直接全删 (dropna) 是初级行为。
|
||||
# 升级：剔除了单纯包含 dropna 就给分的宽泛规则，强制要求代码中必须包含 ffill 动作。
|
||||
has_ffill_cleaning = 1 if ('.ffill(' in content or '.fillna(method=' in content or '.fillna(method=\"ffill\"' in content) else 0
|
||||
# 6. 三个交付物 (Outputs 3 Artifacts)
|
||||
outputs_three_artifacts = 1 if len(re.findall(r'\.to_csv\(', content)) >= 3 else 0
|
||||
|
||||
return {
|
||||
"dynamic_lambda_calibration": dynamic_lambda_calibration,
|
||||
"uses_advanced_ols": uses_advanced_ols,
|
||||
"targeted_api_fetching": targeted_api_fetching, # 替换了原来的 fallback 检查
|
||||
"has_robust_cleaning": has_ffill_cleaning, # 变得更加严格
|
||||
"outputs_three_artifacts": outputs_three_artifacts
|
||||
}
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
from typing import Any
|
||||
import os
|
||||
import yaml
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
# 路径解析逻辑保持不变...
|
||||
possible_paths = []
|
||||
config_path = Path("config") / "nanobot.yaml"
|
||||
if config_path.exists():
|
||||
try:
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
config = yaml.safe_load(f)
|
||||
if config and 'nanobot' in config and 'workspace_dir' in config['nanobot']:
|
||||
workspace_dir = config['nanobot']['workspace_dir']
|
||||
possible_paths.append(Path(os.path.expanduser(workspace_dir)) / "Data_Pipeline.py")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
content = None
|
||||
for file_path in possible_paths:
|
||||
if file_path.exists() and file_path.is_file():
|
||||
try:
|
||||
content = file_path.read_text(encoding='utf-8')
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if content is None:
|
||||
return {
|
||||
"strict_tenor_adherence": 0,
|
||||
"has_deduplication": 0,
|
||||
"has_ffill": 0,
|
||||
"has_timeout": 0,
|
||||
"has_defensive_parsing": 0, # 新增
|
||||
"correct_dropna_axis": 0, # 新增
|
||||
"strictly_no_lookahead": 0 # 新增
|
||||
}
|
||||
|
||||
# --- 1. 区分题：严格期限过滤 (优质得 1，劣质得 0) ---
|
||||
includes_short_tenors = bool(re.search(r'1\s*Mo|2\s*Mo|3\s*Mo', content, re.IGNORECASE))
|
||||
strict_tenor_adherence = 0 if includes_short_tenors else 1
|
||||
|
||||
# --- 2. 区分题：防御性去重 (优质得 1，劣质得 0) ---
|
||||
has_deduplication = 1 if '.drop_duplicates(' in content else 0
|
||||
|
||||
# --- 3. 送分题：前向填充与超时保护 (优质得 1，劣质得 1) ---
|
||||
has_ffill = 1 if bool(re.search(r'\.ffill\(|\.fillna\(method=[\'"]ffill[\'"]', content)) else 0
|
||||
has_timeout = 1 if bool(re.search(r'timeout\s*=\s*\d+', content)) else 0
|
||||
|
||||
# --- 4. 送分题：防御性类型清洗 (优质得 1，劣质得 1) ---
|
||||
# 官方 CSV/HTML 经常有脏数据，两个回答都极其聪明地使用了 pd.to_numeric 来转 float
|
||||
has_defensive_parsing = 1 if bool(re.search(r'pd\.to_numeric|\.astype\(\s*float\s*\)', content)) else 0
|
||||
|
||||
# --- 5. 区分题：正确的 Dropna 轴向 (优质得 1，劣质得 0) ---
|
||||
# 劣质回答写了匪夷所思的 `dropna(axis=1, how='all')` (如果一天没数据，它竟然把整列期限删了！这是致命逻辑错误)
|
||||
# 优质回答写了正确的 `.dropna(subset=..., how="all")` (删除全为空的行)
|
||||
has_fatal_axis1_drop = '.dropna(axis=1' in content.replace(' ', '')
|
||||
correct_dropna_axis = 0 if has_fatal_axis1_drop else 1
|
||||
|
||||
# --- 7. 超纲题：绝对禁止前视偏差 (优质得 0，劣质得 1) ---
|
||||
# 极其讽刺！目前的“优质回答”为了偷懒，违规使用了 .bfill() 来倒填节假日，在这个超纲题上拿了 0 分！
|
||||
# 而“劣质回答”阴差阳错拉取了 2015 年数据并没用 bfill，反而在这一条上碰巧拿了 1 分 (虽然它别的全错)。
|
||||
# 这种戏剧性的得分点能极大地考验后续大模型在 Quant 领域的真实智商。
|
||||
uses_bfill = bool(re.search(r'\.bfill\(|\.fillna\(method=[\'"]bfill[\'"]|\.interpolate\(', content))
|
||||
strictly_no_lookahead = 0 if uses_bfill else 1
|
||||
|
||||
return {
|
||||
"strict_tenor_adherence": strict_tenor_adherence,
|
||||
"has_deduplication": has_deduplication,
|
||||
"has_ffill": has_ffill,
|
||||
"has_timeout": has_timeout,
|
||||
"has_defensive_parsing": has_defensive_parsing,
|
||||
"correct_dropna_axis": correct_dropna_axis,
|
||||
"strictly_no_lookahead": strictly_no_lookahead
|
||||
}
|
||||
|
|
@ -0,0 +1,162 @@
|
|||
---
|
||||
name: 信用分析一页纸
|
||||
description: 提供结构化、标准化的信用评估一页纸撰写框架，适用于金融分析师快速产出结论驱动、证据闭环的信用观点摘要，服务于投资决策与风险管理。
|
||||
---
|
||||
|
||||
### **信评一页纸：结构化分析与撰写框架**
|
||||
|
||||
#### **一、 核心理念**
|
||||
|
||||
1. **结论驱动：** 所有内容必须服务于最终信用结论。
|
||||
2. **证据支撑：** 定性结论必须有定量数据或具体事实支撑。
|
||||
3. **聚焦关键：** 仅分析最相关的核心驱动因素和重大风险。
|
||||
4. **动态分析：** 结合历史趋势、行业周期及未来风险事件进行分析。
|
||||
5. **标准化输出：** 使用统一的评级符号和风险等级关键词。
|
||||
6. **数据优先：** 优先使用 iFinD 官方数据作为分析素材，确保数据权威性与可追溯性。
|
||||
|
||||
---
|
||||
|
||||
#### **二、 文档模板**
|
||||
|
||||
**文档标题：** `[发行人全称] ([代码]) — 信用评估摘要 | 分析日期：YYYY-MM-DD`
|
||||
|
||||
---
|
||||
|
||||
**核心观点**
|
||||
|
||||
* **信用观点：** 客观描述发行人的核心信用特征。
|
||||
* 示例：`"区域性公用事业运营商，收入稳定性较强，但盈利能力受上游成本波动影响显著。"`
|
||||
* **信用支撑点：** 1-2句话概括主要支撑点（尽量量化）。
|
||||
* 示例：`"区域专营地位稳固，近三年经营现金流/EBITDA维持在85%以上。"`
|
||||
* **核心风险提示：** 1-2句话指出关键风险（包含量化指标）。
|
||||
* 示例：`"债务期限结构存在一定错配，未来12个月到期债务占总债务45%，货币资金/短期债务为0.6x。"`
|
||||
* **未来12个月展望：** `稳定` / `正面` / `负面`。
|
||||
|
||||
---
|
||||
|
||||
**发行人情况**
|
||||
|
||||
* **业务定位：** 一句话定义其在产业链中的角色。
|
||||
* 示例：`"华东地区领先的城市燃气运营商。"`
|
||||
* **基本情况：**
|
||||
* 成立时间与注册资本
|
||||
* 主要业务范围与板块贡献
|
||||
* 地理覆盖与运营规模
|
||||
* 行业地位与竞争格局
|
||||
|
||||
---
|
||||
|
||||
**股东背景与公司治理**
|
||||
|
||||
* **股权结构：** 明确至最终控制方。
|
||||
* 示例：`"省国资委（持股60%）。"`
|
||||
* **股东支持：** 对国企/平台分析财政实力、历史支持记录、战略定位；对民企分析实控人经营状况、股权质押情况。
|
||||
* **公司治理：** 决策机制、财务透明度、过往信用记录。
|
||||
|
||||
---
|
||||
|
||||
**区域经济与财政实力（城投公司适用）**
|
||||
|
||||
* **区域经济基本面：**
|
||||
* 区域定位与行政层级
|
||||
* 经济规模与增速（GDP、固定资产投资）
|
||||
* 产业结构与多元化程度
|
||||
* **财政实力与债务压力：**
|
||||
| 指标 | 数值 | 信用含义 |
|
||||
| :--- | :--- | :--- |
|
||||
| 一般公共预算收入 | XXX亿元 | 财政收入规模与地位 |
|
||||
| 财政收入增速 | X.X% | 收入可持续性 |
|
||||
| 财政自给率 | X.X% | 对转移支付依赖度 |
|
||||
| 政府性基金收入 | XXX亿元 | 土地财政依赖度 |
|
||||
| 政府债务率 | X.X% | 债务覆盖能力 |
|
||||
| 显性债务余额 | XXX亿元 | 债务累积速度 |
|
||||
* **区域信用环境：** 历史违约记录、区域战略地位、融资环境。
|
||||
|
||||
---
|
||||
|
||||
**主体信用深度分析**
|
||||
|
||||
*分析要点：根据发行人特点，选择2-3个核心维度深度分析，其他维度简要概述。*
|
||||
|
||||
**A. 主营业务分析**
|
||||
* 行业周期与监管环境
|
||||
* 商业模式与收入稳定性
|
||||
* 市场地位与竞争优势
|
||||
* 业务风险
|
||||
|
||||
**B. 资产质量分析**
|
||||
* 资产结构与主要构成
|
||||
* 资产流动性与变现能力
|
||||
* 关联方交易与资产占用
|
||||
|
||||
**C. 负债与偿债能力分析**
|
||||
* 债务结构与期限分布
|
||||
* 杠杆水平（资产负债率、净债务/EBITDA）
|
||||
* 短期偿债能力（货币资金/短期债务）
|
||||
* 债务到期压力
|
||||
|
||||
**D. 盈利能力分析**
|
||||
* 核心盈利指标（毛利率、净利率、EBITDA）
|
||||
* 盈利驱动因素
|
||||
* 利润质量评估
|
||||
|
||||
**E. 现金流分析**
|
||||
* 经营现金流生成能力
|
||||
* 营运资本管理效率
|
||||
* 投资与筹资匹配性
|
||||
|
||||
---
|
||||
|
||||
**债券市场表现**
|
||||
|
||||
* **存量债券概况：**
|
||||
| 项目 | 数值 |
|
||||
| :--- | :--- |
|
||||
| 存量债券只数 | XX只 |
|
||||
| 存量债券余额 | XX亿元 |
|
||||
| 未来12个月到期余额 | XX亿元 |
|
||||
| 当前主体评级 | AAA/AA+/AA |
|
||||
|
||||
* **一级市场表现：**
|
||||
| 项目 | 数值 |
|
||||
| :--- | :--- |
|
||||
| 近12个月发行次数 | XX次 |
|
||||
| 平均发行利率 | X.XX% |
|
||||
| 平均认购倍数 | X.Xx |
|
||||
|
||||
* **二级市场表现：**
|
||||
| 项目 | 数值 |
|
||||
| :--- | :--- |
|
||||
| 当前估值收益率 | X.XX% |
|
||||
| 信用利差 | XXbp |
|
||||
| 日均成交量 | XX万元 |
|
||||
|
||||
---
|
||||
|
||||
**关键风险与跟踪事项**
|
||||
|
||||
* **重大风险事件：** 列出已知重大不确定性。
|
||||
* **预警指标跟踪：**
|
||||
| 指标类别 | 具体指标 | 预警阈值 | 风险等级 |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| 财务指标 | 净债务/EBITDA | 6.0x | 高/中/低 |
|
||||
| 财务指标 | 货币资金/短期债务 | 0.5x | 高/中/低 |
|
||||
| 融资指标 | 银行授信使用率 | 95% | 高/中/低 |
|
||||
| 市场指标 | 信用利差变化 | +50bp | 高/中/低 |
|
||||
|
||||
* **下次评审触发点：** 具体化，如：发布年度财报后 / 触发预警指标时。
|
||||
|
||||
---
|
||||
|
||||
#### **三、 撰写流程**
|
||||
|
||||
1. **数据提取：** 收集财报、募集说明书、行业数据，计算关键比率。
|
||||
2. **框架选择：** 判定城投/非城投，选择相应分析框架。
|
||||
3. **核心维度识别：** 根据发行人特点识别2-3个重点分析维度。
|
||||
4. **框架填充：** 将数据和分析填入模板各部分。
|
||||
5. **结论提炼：** 逆向撰写"核心观点"，确保结论有依据。
|
||||
6. **精炼输出：** 删除重复和次要信息，确保逻辑完整。
|
||||
|
||||
---
|
||||
|
||||
**最终产出是一份高度结构化、信息密度高、结论明确、重点突出的信用评估摘要。**
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"ownerId": "kn7d5jgbcx5gsmyxcr57rz8cy984r0yf",
|
||||
"slug": "1pagecreditanalysis-4cnmarket",
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1776089238852
|
||||
}
|
||||
|
|
@ -0,0 +1,206 @@
|
|||
# 金融分析技能 (Financial Analysis Skill)
|
||||
|
||||
## 概述
|
||||
这是一个用于投资组合分析的技能，支持从免费API获取各类资产行情数据，并提供文字报告和图表可视化。优化版本整合了滚动窗口调仓逻辑和回测功能。
|
||||
|
||||
## 功能特性
|
||||
- ✅ 投资组合分析（资产配置、风险评估、收益分析）
|
||||
- ✅ 支持多种资产类型：股票、ETF、加密货币等
|
||||
- ✅ 免费数据源集成（yfinance）
|
||||
- ✅ 文字报告生成
|
||||
- ✅ 图表可视化（收益曲线、资产分布、相关性热力图）
|
||||
- ✅ 风险指标计算（波动率、最大回撤、夏普比率）
|
||||
- ✅ 支持CSV数据导入（自动处理百分比单位转换）
|
||||
- ✅ 风险平价组合分析（各资产风险贡献均衡）
|
||||
- ✅ **滚动窗口风险平价分析**（使用历史一年数据，每月调仓）
|
||||
- ✅ **避免未来数据**（使用expanding窗口计算波动率）
|
||||
- ✅ **回测功能**（完整的回测流程）
|
||||
|
||||
## 安装依赖
|
||||
|
||||
### 1. 安装Python依赖
|
||||
```bash
|
||||
pip install pandas numpy matplotlib seaborn
|
||||
```
|
||||
|
||||
### 2. 安装技能
|
||||
```bash
|
||||
# 将技能复制到OpenClaw技能目录
|
||||
cp -r financial-analysis-skill ~/.openclaw/skills/
|
||||
```
|
||||
|
||||
## 使用方法
|
||||
|
||||
### 1. 基本使用
|
||||
|
||||
#### 运行回测分析
|
||||
```bash
|
||||
python optimized_main.py --csv "C:\path\to\marketdata.csv" --output ./backtest_output
|
||||
```
|
||||
|
||||
#### 使用默认数据路径
|
||||
```bash
|
||||
python optimized_main.py
|
||||
```
|
||||
|
||||
#### 自定义输出目录
|
||||
```bash
|
||||
python optimized_main.py --output ./my_backtest
|
||||
```
|
||||
|
||||
### 2. 查看分析结果
|
||||
|
||||
#### 显示分析报告
|
||||
```bash
|
||||
python optimized_main.py --show-report
|
||||
```
|
||||
|
||||
#### 显示分析指标
|
||||
```bash
|
||||
python optimized_main.py --show-metrics
|
||||
```
|
||||
|
||||
#### 显示资产权重
|
||||
```bash
|
||||
python optimized_main.py --show-weights
|
||||
```
|
||||
|
||||
### 3. 直接使用技能类
|
||||
|
||||
```python
|
||||
from optimized_risk_parity_skill import OptimizedRiskParitySkill
|
||||
|
||||
# 创建技能实例
|
||||
skill = OptimizedRiskParitySkill('C:\path\to\marketdata.csv')
|
||||
|
||||
# 运行回测
|
||||
result = skill.run_backtest('C:\path\to\marketdata.csv', './backtest_output')
|
||||
|
||||
# 获取报告
|
||||
report = skill.get_report()
|
||||
print(report)
|
||||
|
||||
# 获取指标
|
||||
metrics = skill.get_metrics()
|
||||
print(f"总收益率: {metrics['total_return']*100:.2f}%")
|
||||
print(f"夏普比率: {metrics['sharpe_ratio']:.2f}")
|
||||
|
||||
# 获取权重
|
||||
weights = skill.get_weights()
|
||||
for asset, weight in weights.items():
|
||||
print(f"{asset}: {weight*100:.2f}%")
|
||||
```
|
||||
|
||||
## 数据格式要求
|
||||
|
||||
### CSV文件格式
|
||||
- 第0行：资产名称（如：CFFEX5五年期国债期货）
|
||||
- 第1行：数据类型（如：涨跌幅(%)）
|
||||
- 第2行开始：实际数据
|
||||
- 第1列：日期
|
||||
- 第2-5列：各资产收益率（百分比格式）
|
||||
|
||||
### 示例数据
|
||||
```
|
||||
日期,TF.CFE,T.CFE,CU.SHF,AU.SHF
|
||||
2015-03-20,0.1377,NaN,0.767,-0.1472
|
||||
2015-03-23,-0.1834,-0.0721,3.9248,1.1373
|
||||
2015-03-24,-0.0510,-0.1031,0.3662,0.4165
|
||||
```
|
||||
|
||||
## 滚动窗口风险平价分析
|
||||
|
||||
### 核心特点
|
||||
1. **避免未来数据**：使用历史一年数据计算波动率
|
||||
2. **动态调仓**：每月根据最新波动率重新计算权重
|
||||
3. **风险平价**：各资产对组合风险贡献相等
|
||||
4. **回测验证**：完整的回测流程，验证策略效果
|
||||
|
||||
### 分析流程
|
||||
1. **数据加载**：加载CSV数据，处理百分比单位
|
||||
2. **滚动波动率**：使用expanding窗口计算历史波动率
|
||||
3. **权重计算**：基于波动率倒数计算风险平价权重
|
||||
4. **投资组合收益率**：使用滚动权重计算每日收益率
|
||||
5. **绩效评估**：计算总收益率、年化收益率、夏普比率等
|
||||
6. **可视化**：生成收益曲线、资产配置、相关性等图表
|
||||
|
||||
### 输出文件
|
||||
- **分析报告**：`rolling_risk_parity_report.txt`
|
||||
- **详细数据**：`rolling_risk_parity_data.json`
|
||||
- **可视化图表**：
|
||||
- `rolling_risk_parity_returns.png` - 收益曲线图
|
||||
- `rolling_risk_parity_allocation.png` - 资产配置饼图
|
||||
- `rolling_risk_parity_correlation.png` - 相关性热力图
|
||||
- `rolling_asset_returns_comparison.png` - 资产收益对比图
|
||||
- `rolling_weight_changes.png` - 滚动权重变化图
|
||||
|
||||
## 报告内容
|
||||
- 投资组合概览
|
||||
- 收益分析（总收益、年化收益）
|
||||
- 风险指标（波动率、最大回撤、夏普比率）
|
||||
- 资产配置分析
|
||||
- 相关性分析
|
||||
- 滚动权重变化
|
||||
- 可视化图表
|
||||
- 投资建议
|
||||
|
||||
## 示例输出
|
||||
|
||||
### 分析报告
|
||||
```
|
||||
中国市场滚动风险平价组合分析报告
|
||||
============================================================
|
||||
生成时间: 2026-02-26 10:25:00
|
||||
数据来源: C:\Users\wu_zhuoran\.openclaw\workspace\data\marketdata.csv
|
||||
数据时间范围: 2015-03-23 至 2026-02-25
|
||||
数据点数: 2656
|
||||
滚动窗口: 252个交易日（约1年）
|
||||
调仓频率: 每月
|
||||
|
||||
平均投资组合配置:
|
||||
五年期国债 (TF.CFE): 46.04% (波动率: 2.46%)
|
||||
十年期国债 (T.CFE): 32.55% (波动率: 3.61%)
|
||||
沪铜 (CU.SHF): 9.33% (波动率: 17.87%)
|
||||
沪金 (AU.SHF): 12.09% (波动率: 14.88%)
|
||||
|
||||
收益指标:
|
||||
总收益率: 44.78%
|
||||
年化收益率: 3.57%
|
||||
|
||||
风险指标:
|
||||
年化波动率: 3.76%
|
||||
最大回撤: -9.48%
|
||||
夏普比率: 0.42
|
||||
```
|
||||
|
||||
## 技术实现
|
||||
|
||||
### 滚动窗口计算
|
||||
- 使用`expanding()`窗口而非`rolling()`窗口
|
||||
- 确保不使用未来数据
|
||||
- 最小窗口大小为252个交易日（约1年）
|
||||
|
||||
### 权重计算
|
||||
- 基于波动率倒数加权：`权重 = (1/波动率) / ∑(1/波动率)`
|
||||
- 每月调仓，动态调整权重
|
||||
- 使用前一天的权重计算当日收益率
|
||||
|
||||
### 回测验证
|
||||
- 完整的回测流程
|
||||
- 避免数据泄露
|
||||
- 提供详细的绩效指标
|
||||
|
||||
## 注意事项
|
||||
- 确保数据质量，处理缺失值和异常值
|
||||
- 滚动窗口大小可根据实际情况调整
|
||||
- 调仓频率可根据市场情况调整
|
||||
- 考虑交易成本对回测结果的影响
|
||||
|
||||
## 许可证
|
||||
MIT License
|
||||
|
||||
## 贡献
|
||||
欢迎提交Issue和Pull Request！
|
||||
|
||||
## 联系方式
|
||||
如有问题，请在项目仓库提交Issue。
|
||||
|
|
@ -0,0 +1,156 @@
|
|||
# 金融分析技能 (Financial Analysis Skill)
|
||||
|
||||
## 概述
|
||||
这是一个用于投资组合分析的技能，支持从免费API获取各类资产行情数据，并提供文字报告和图表可视化。优化版本整合了滚动窗口调仓逻辑和回测功能。
|
||||
|
||||
## 功能特性
|
||||
- ✅ 投资组合分析（资产配置、风险评估、收益分析）
|
||||
- ✅ 支持多种资产类型：股票、ETF、加密货币等
|
||||
- ✅ 免费数据源集成
|
||||
- ✅ 文字报告生成
|
||||
- ✅ 图表可视化（收益曲线、资产分布、风险指标等）
|
||||
- ✅ **滚动窗口风险平价分析**（使用历史一年数据，每月调仓）
|
||||
- ✅ **避免未来数据**（使用expanding窗口计算波动率）
|
||||
- ✅ **回测功能**（完整的回测流程）
|
||||
|
||||
## 数据源
|
||||
- Yahoo Finance (通过yfinance库)
|
||||
- Alpha Vantage (免费API)
|
||||
- Finnhub (免费层)
|
||||
- CSV文件（中国市场数据）
|
||||
|
||||
## 安装依赖
|
||||
```bash
|
||||
pip install yfinance pandas numpy matplotlib seaborn plotly
|
||||
```
|
||||
|
||||
## 使用方法
|
||||
|
||||
### 1. 基本使用
|
||||
|
||||
#### 分析投资组合
|
||||
```bash
|
||||
python financial_analysis.py --portfolio example_portfolio.json --output ./output
|
||||
```
|
||||
|
||||
#### 获取单个资产数据
|
||||
```bash
|
||||
python financial_analysis.py --symbol AAPL --period 1y
|
||||
```
|
||||
|
||||
### 2. 滚动窗口风险平价分析
|
||||
|
||||
#### 运行回测
|
||||
```bash
|
||||
python optimized_risk_parity_skill.py --csv "C:\path\to\marketdata.csv" --output ./backtest_output
|
||||
```
|
||||
|
||||
#### 数据格式要求
|
||||
- CSV文件应包含日期列和收益率列
|
||||
- 收益率数据应为百分比格式（如：1.5表示1.5%）
|
||||
- 脚本会自动将百分比转换为实际收益率（除以100）
|
||||
- 支持处理文本行和缺失值
|
||||
|
||||
### 3. 投资组合配置文件格式
|
||||
|
||||
创建一个JSON文件（如 `my_portfolio.json`）：
|
||||
|
||||
```json
|
||||
{
|
||||
"AAPL": 0.25, # 苹果股票，权重25%
|
||||
"GOOGL": 0.20, # 谷歌股票，权重20%
|
||||
"MSFT": 0.20, # 微软股票，权重20%
|
||||
"AMZN": 0.15, # 亚马逊股票，权重15%
|
||||
"TSLA": 0.20 # 特斯拉股票，权重20%
|
||||
}
|
||||
```
|
||||
|
||||
## 滚动窗口风险平价分析
|
||||
|
||||
### 核心特点
|
||||
1. **避免未来数据**：使用历史一年数据计算波动率
|
||||
2. **动态调仓**：每月根据最新波动率重新计算权重
|
||||
3. **风险平价**：各资产对组合风险贡献相等
|
||||
4. **回测验证**：完整的回测流程，验证策略效果
|
||||
|
||||
### 分析流程
|
||||
1. **数据加载**：加载CSV数据，处理百分比单位
|
||||
2. **滚动波动率**：使用expanding窗口计算历史波动率
|
||||
3. **权重计算**：基于波动率倒数计算风险平价权重
|
||||
4. **投资组合收益率**：使用滚动权重计算每日收益率
|
||||
5. **绩效评估**：计算总收益率、年化收益率、夏普比率等
|
||||
6. **可视化**：生成收益曲线、资产配置、相关性等图表
|
||||
|
||||
### 输出文件
|
||||
- **分析报告**：`rolling_risk_parity_report.txt`
|
||||
- **详细数据**：`rolling_risk_parity_data.json`
|
||||
- **可视化图表**：
|
||||
- `rolling_risk_parity_returns.png` - 收益曲线图
|
||||
- `rolling_risk_parity_allocation.png` - 资产配置饼图
|
||||
- `rolling_risk_parity_correlation.png` - 相关性热力图
|
||||
- `rolling_asset_returns_comparison.png` - 资产收益对比图
|
||||
- `rolling_weight_changes.png` - 滚动权重变化图
|
||||
|
||||
## 报告内容
|
||||
- 投资组合概览
|
||||
- 收益分析（总收益、年化收益）
|
||||
- 风险指标（波动率、最大回撤、夏普比率）
|
||||
- 资产配置分析
|
||||
- 相关性分析
|
||||
- 滚动权重变化
|
||||
- 可视化图表
|
||||
- 投资建议
|
||||
|
||||
## 配置
|
||||
需要配置API密钥（如使用Alpha Vantage）：
|
||||
- ALPHA_VANTAGE_API_KEY=your_key_here
|
||||
|
||||
## 示例输出
|
||||
```
|
||||
中国市场滚动风险平价组合分析报告
|
||||
============================================================
|
||||
生成时间: 2026-02-26 10:20:04
|
||||
数据来源: C:\Users\wu_zhuoran\.openclaw\workspace\data\marketdata.csv
|
||||
数据时间范围: 2015-03-23 至 2026-02-25
|
||||
数据点数: 2656
|
||||
滚动窗口: 252个交易日（约1年）
|
||||
调仓频率: 每月
|
||||
|
||||
平均投资组合配置:
|
||||
五年期国债 (TF.CFE): 46.04% (波动率: 2.46%)
|
||||
十年期国债 (T.CFE): 32.55% (波动率: 3.61%)
|
||||
沪铜 (CU.SHF): 9.33% (波动率: 17.87%)
|
||||
沪金 (AU.SHF): 12.09% (波动率: 14.88%)
|
||||
|
||||
收益指标:
|
||||
总收益率: 44.78%
|
||||
年化收益率: 3.57%
|
||||
|
||||
风险指标:
|
||||
年化波动率: 3.76%
|
||||
最大回撤: -9.48%
|
||||
夏普比率: 0.42
|
||||
```
|
||||
|
||||
## 技术实现
|
||||
|
||||
### 滚动窗口计算
|
||||
- 使用`expanding()`窗口而非`rolling()`窗口
|
||||
- 确保不使用未来数据
|
||||
- 最小窗口大小为252个交易日（约1年）
|
||||
|
||||
### 权重计算
|
||||
- 基于波动率倒数加权：`权重 = (1/波动率) / ∑(1/波动率)`
|
||||
- 每月调仓，动态调整权重
|
||||
- 使用前一天的权重计算当日收益率
|
||||
|
||||
### 回测验证
|
||||
- 完整的回测流程
|
||||
- 避免数据泄露
|
||||
- 提供详细的绩效指标
|
||||
|
||||
## 注意事项
|
||||
- 确保数据质量，处理缺失值和异常值
|
||||
- 滚动窗口大小可根据实际情况调整
|
||||
- 调仓频率可根据市场情况调整
|
||||
- 考虑交易成本对回测结果的影响
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"ownerId": "kn7ed17ds52mp5nvf0x0b318jn81xpng",
|
||||
"slug": "financial-analysis",
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1772074882443
|
||||
}
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
{
|
||||
"name": "金融分析技能",
|
||||
"slug": "financial-analysis",
|
||||
"version": "1.0.0",
|
||||
"description": "优化的金融分析技能，整合滚动窗口调仓逻辑和回测功能",
|
||||
"author": "OpenClaw User",
|
||||
"license": "MIT",
|
||||
"tags": [
|
||||
"finance",
|
||||
"portfolio",
|
||||
"risk-parity",
|
||||
"backtest",
|
||||
"analysis"
|
||||
],
|
||||
"dependencies": {
|
||||
"python": [
|
||||
"pandas",
|
||||
"numpy",
|
||||
"matplotlib",
|
||||
"seaborn"
|
||||
]
|
||||
},
|
||||
"entry_points": {
|
||||
"main": "optimized_main.py",
|
||||
"skill": "optimized_risk_parity_skill.py"
|
||||
},
|
||||
"features": [
|
||||
"投资组合分析",
|
||||
"风险平价策略",
|
||||
"滚动窗口分析",
|
||||
"避免未来数据",
|
||||
"动态调仓",
|
||||
"回测验证",
|
||||
"可视化图表",
|
||||
"中国市场支持"
|
||||
],
|
||||
"readme": "README.md",
|
||||
"changelog": "优化的金融分析技能，整合滚动窗口调仓逻辑和回测功能"
|
||||
}
|
||||
|
|
@ -0,0 +1,120 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
优化的金融分析技能主脚本
|
||||
整合滚动窗口调仓逻辑和回测功能
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import os
|
||||
from datetime import datetime
|
||||
|
||||
# 添加当前目录到Python路径
|
||||
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from optimized_risk_parity_skill import OptimizedRiskParitySkill
|
||||
|
||||
def main():
|
||||
"""主函数"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description='优化的金融分析技能 - 滚动窗口风险平价组合分析',
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog='''
|
||||
示例:
|
||||
# 运行回测分析
|
||||
python optimized_main.py --csv "C:\path\to\marketdata.csv" --output ./backtest_output
|
||||
|
||||
# 使用默认数据路径
|
||||
python optimized_main.py
|
||||
|
||||
# 自定义输出目录
|
||||
python optimized_main.py --output ./my_backtest
|
||||
'''
|
||||
)
|
||||
|
||||
parser.add_argument('--csv', type=str,
|
||||
default=r'C:\Users\wu_zhuoran\.openclaw\workspace\data\marketdata.csv',
|
||||
help='CSV文件路径（包含收益率数据）')
|
||||
|
||||
parser.add_argument('--output', type=str, default='./backtest_output',
|
||||
help='输出目录（用于保存报告和图表）')
|
||||
|
||||
parser.add_argument('--show-report', action='store_true',
|
||||
help='显示分析报告')
|
||||
|
||||
parser.add_argument('--show-metrics', action='store_true',
|
||||
help='显示分析指标')
|
||||
|
||||
parser.add_argument('--show-weights', action='store_true',
|
||||
help='显示资产权重')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
print("=" * 60)
|
||||
print("优化的金融分析技能 - 滚动窗口风险平价组合分析")
|
||||
print("=" * 60)
|
||||
print(f"数据文件: {args.csv}")
|
||||
print(f"输出目录: {args.output}")
|
||||
print(f"开始时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
print("")
|
||||
|
||||
# 创建技能实例
|
||||
skill = OptimizedRiskParitySkill(args.csv)
|
||||
|
||||
try:
|
||||
# 运行回测
|
||||
result = skill.run_backtest(args.csv, args.output)
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("分析完成！")
|
||||
print("=" * 60)
|
||||
|
||||
# 显示分析报告
|
||||
if args.show_report:
|
||||
print("\n" + "=" * 60)
|
||||
print("分析报告")
|
||||
print("=" * 60)
|
||||
print(result['text_report'])
|
||||
|
||||
# 显示分析指标
|
||||
if args.show_metrics:
|
||||
print("\n" + "=" * 60)
|
||||
print("分析指标")
|
||||
print("=" * 60)
|
||||
metrics = skill.get_metrics()
|
||||
if metrics:
|
||||
print(f"总收益率: {metrics.get('total_return', 0)*100:.2f}%")
|
||||
print(f"年化收益率: {metrics.get('annualized_return', 0)*100:.2f}%")
|
||||
print(f"年化波动率: {metrics.get('volatility', 0)*100:.2f}%")
|
||||
print(f"最大回撤: {metrics.get('max_drawdown', 0)*100:.2f}%")
|
||||
print(f"夏普比率: {metrics.get('sharpe_ratio', 0):.2f}")
|
||||
|
||||
# 显示资产权重
|
||||
if args.show_weights:
|
||||
print("\n" + "=" * 60)
|
||||
print("资产权重")
|
||||
print("=" * 60)
|
||||
weights = skill.get_weights()
|
||||
if weights:
|
||||
for asset, weight in weights.items():
|
||||
asset_name = skill.asset_mapping.get(asset, asset)
|
||||
print(f"{asset_name} ({asset}): {weight*100:.2f}%")
|
||||
|
||||
# 显示输出文件信息
|
||||
print("\n" + "=" * 60)
|
||||
print("输出文件")
|
||||
print("=" * 60)
|
||||
print(f"报告文件: {result['report_file']}")
|
||||
print(f"数据文件: {result['json_file']}")
|
||||
print(f"图表文件: {result['charts']}")
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("完成时间: " + datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
|
||||
print("=" * 60)
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n错误: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,622 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
优化的风险平价组合分析技能
|
||||
整合滚动窗口调仓逻辑和回测功能
|
||||
"""
|
||||
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
import seaborn as sns
|
||||
from datetime import datetime, timedelta
|
||||
import warnings
|
||||
warnings.filterwarnings('ignore')
|
||||
|
||||
class OptimizedRiskParitySkill:
|
||||
def __init__(self, csv_path=None):
|
||||
"""
|
||||
初始化优化的风险平价技能
|
||||
|
||||
Args:
|
||||
csv_path: CSV文件路径（可选）
|
||||
"""
|
||||
self.csv_path = csv_path
|
||||
self.asset_mapping = {
|
||||
"TF.CFE": "五年期国债",
|
||||
"T.CFE": "十年期国债",
|
||||
"CU.SHF": "沪铜",
|
||||
"AU.SHF": "沪金"
|
||||
}
|
||||
|
||||
# 资产类型映射
|
||||
self.asset_types = {
|
||||
"TF.CFE": "bond", # 国债
|
||||
"T.CFE": "bond", # 国债
|
||||
"CU.SHF": "commodity", # 铜
|
||||
"AU.SHF": "commodity" # 黄金
|
||||
}
|
||||
|
||||
# 滚动窗口参数
|
||||
self.window_size = 252 # 一年的交易日数
|
||||
self.rebalance_frequency = "M" # 每月调仓
|
||||
|
||||
# 存储分析结果
|
||||
self.results = {}
|
||||
|
||||
def load_data(self, csv_path=None):
|
||||
"""
|
||||
加载数据
|
||||
"""
|
||||
if csv_path:
|
||||
self.csv_path = csv_path
|
||||
|
||||
if not self.csv_path:
|
||||
raise ValueError("请提供CSV文件路径")
|
||||
|
||||
print("=" * 60)
|
||||
print("加载数据")
|
||||
print("=" * 60)
|
||||
|
||||
# 读取CSV文件
|
||||
df = pd.read_csv(self.csv_path, encoding='gbk')
|
||||
|
||||
print(f"原始数据形状: {df.shape}")
|
||||
|
||||
# 提取实际数据
|
||||
data_start_row = 2
|
||||
|
||||
# 创建干净的数据
|
||||
clean_data = pd.DataFrame()
|
||||
|
||||
# 提取日期
|
||||
date_column = df.iloc[data_start_row:, 0].values
|
||||
clean_data['Date'] = pd.to_datetime(date_column, errors='coerce')
|
||||
|
||||
# 提取收益率数据
|
||||
asset_codes = ['TF.CFE', 'T.CFE', 'CU.SHF', 'AU.SHF']
|
||||
|
||||
# 提取收益率数据
|
||||
for i, code in enumerate(asset_codes):
|
||||
if code in df.columns:
|
||||
# 获取列数据
|
||||
col_data = df.iloc[data_start_row:][code].values
|
||||
|
||||
# 转换为数值，忽略非数值数据
|
||||
numeric_data = []
|
||||
for val in col_data:
|
||||
try:
|
||||
# 尝试转换为数值
|
||||
num_val = float(val)
|
||||
# 数据中的涨跌幅单位是%，所以实际收益率应该是数字除以100
|
||||
numeric_data.append(num_val / 100.0)
|
||||
except (ValueError, TypeError):
|
||||
# 如果转换失败，跳过
|
||||
numeric_data.append(np.nan)
|
||||
|
||||
clean_data[code] = numeric_data
|
||||
|
||||
# 设置日期索引
|
||||
clean_data.set_index('Date', inplace=True)
|
||||
|
||||
# 删除空行
|
||||
clean_data = clean_data.dropna(how='all')
|
||||
|
||||
# 删除包含NaN的行
|
||||
clean_data = clean_data.dropna()
|
||||
|
||||
print(f"干净数据形状: {clean_data.shape}")
|
||||
print(f"数据时间范围: {clean_data.index[0]} 至 {clean_data.index[-1]}")
|
||||
|
||||
return clean_data
|
||||
|
||||
def calculate_rolling_volatility(self, returns_data, window_size=252):
|
||||
"""
|
||||
计算滚动波动率（使用历史数据，避免未来数据）
|
||||
"""
|
||||
print("\n" + "=" * 60)
|
||||
print("计算滚动波动率")
|
||||
print("=" * 60)
|
||||
|
||||
rolling_volatilities = {}
|
||||
|
||||
for asset in returns_data.columns:
|
||||
# 计算滚动波动率（年化）
|
||||
# 使用expanding()而不是rolling()，确保不使用未来数据
|
||||
rolling_vol = returns_data[asset].expanding(min_periods=window_size).std() * np.sqrt(252)
|
||||
rolling_volatilities[asset] = rolling_vol
|
||||
|
||||
# 显示统计信息
|
||||
valid_vol = rolling_vol.dropna()
|
||||
if len(valid_vol) > 0:
|
||||
print(f" {self.asset_mapping.get(asset, asset)} ({asset}):")
|
||||
print(f" 数据点数: {len(valid_vol)}")
|
||||
print(f" 平均波动率: {valid_vol.mean()*100:.2f}%")
|
||||
print(f" 最小波动率: {valid_vol.min()*100:.2f}%")
|
||||
print(f" 最大波动率: {valid_vol.max()*100:.2f}%")
|
||||
|
||||
return rolling_volatilities
|
||||
|
||||
def calculate_rolling_weights(self, rolling_volatilities):
|
||||
"""
|
||||
计算滚动风险平价权重
|
||||
"""
|
||||
print("\n" + "=" * 60)
|
||||
print("计算滚动风险平价权重")
|
||||
print("=" * 60)
|
||||
|
||||
# 创建DataFrame存储权重
|
||||
weights_df = pd.DataFrame(index=rolling_volatilities[list(rolling_volatilities.keys())[0]].index)
|
||||
|
||||
for asset in rolling_volatilities.keys():
|
||||
# 计算权重：权重 = (1/波动率) / ∑(1/波动率)
|
||||
vol_series = rolling_volatilities[asset]
|
||||
|
||||
# 创建权重序列
|
||||
weight_series = pd.Series(index=vol_series.index, dtype=float)
|
||||
|
||||
for date in vol_series.index:
|
||||
# 获取当前日期的所有资产波动率
|
||||
current_vols = {}
|
||||
for a in rolling_volatilities.keys():
|
||||
if date in rolling_volatilities[a].index:
|
||||
current_vols[a] = rolling_volatilities[a].loc[date]
|
||||
|
||||
# 计算权重
|
||||
if len(current_vols) == len(rolling_volatilities.keys()):
|
||||
# 所有资产都有波动率数据
|
||||
total_vol = sum(1/v for v in current_vols.values() if v > 0)
|
||||
if total_vol > 0:
|
||||
weight_series.loc[date] = (1/current_vols[asset]) / total_vol
|
||||
else:
|
||||
weight_series.loc[date] = 1/len(rolling_volatilities.keys())
|
||||
else:
|
||||
# 部分资产没有波动率数据，使用等权重
|
||||
weight_series.loc[date] = 1/len(rolling_volatilities.keys())
|
||||
|
||||
weights_df[asset] = weight_series
|
||||
|
||||
# 显示权重统计
|
||||
print(f"\n滚动权重统计:")
|
||||
for asset in weights_df.columns:
|
||||
valid_weights = weights_df[asset].dropna()
|
||||
if len(valid_weights) > 0:
|
||||
print(f" {self.asset_mapping.get(asset, asset)} ({asset}):")
|
||||
print(f" 平均权重: {valid_weights.mean()*100:.2f}%")
|
||||
print(f" 最小权重: {valid_weights.min()*100:.2f}%")
|
||||
print(f" 最大权重: {valid_weights.max()*100:.2f}%")
|
||||
|
||||
return weights_df
|
||||
|
||||
def calculate_portfolio_returns(self, returns_data, weights_df):
|
||||
"""
|
||||
计算投资组合收益率（使用滚动权重）
|
||||
"""
|
||||
print("\n" + "=" * 60)
|
||||
print("计算投资组合收益率")
|
||||
print("=" * 60)
|
||||
|
||||
# 创建投资组合收益率序列
|
||||
portfolio_returns = pd.Series(index=returns_data.index, dtype=float)
|
||||
|
||||
for date in returns_data.index:
|
||||
# 获取当天的收益率
|
||||
daily_returns = returns_data.loc[date]
|
||||
|
||||
# 获取当天的权重（使用前一天的权重，避免未来数据）
|
||||
prev_date = returns_data.index[returns_data.index.get_loc(date) - 1] if returns_data.index.get_loc(date) > 0 else None
|
||||
|
||||
if prev_date is not None and prev_date in weights_df.index:
|
||||
weights = weights_df.loc[prev_date]
|
||||
|
||||
# 计算投资组合收益率
|
||||
portfolio_return = 0
|
||||
for asset in returns_data.columns:
|
||||
if asset in weights.index and not pd.isna(weights[asset]):
|
||||
portfolio_return += daily_returns[asset] * weights[asset]
|
||||
|
||||
portfolio_returns.loc[date] = portfolio_return
|
||||
else:
|
||||
# 第一天，使用等权重
|
||||
portfolio_return = daily_returns.sum() / len(daily_returns)
|
||||
portfolio_returns.loc[date] = portfolio_return
|
||||
|
||||
# 显示统计信息
|
||||
valid_returns = portfolio_returns.dropna()
|
||||
if len(valid_returns) > 0:
|
||||
print(f"\n投资组合收益率统计:")
|
||||
print(f" 数据点数: {len(valid_returns)}")
|
||||
print(f" 平均收益率: {valid_returns.mean()*100:.4f}%")
|
||||
print(f" 标准差: {valid_returns.std()*100:.4f}%")
|
||||
print(f" 最小值: {valid_returns.min()*100:.4f}%")
|
||||
print(f" 最大值: {valid_returns.max()*100:.4f}%")
|
||||
|
||||
return portfolio_returns
|
||||
|
||||
def analyze_portfolio(self, returns_data, weights_df, portfolio_returns):
|
||||
"""
|
||||
分析投资组合
|
||||
"""
|
||||
print("\n" + "=" * 60)
|
||||
print("分析投资组合")
|
||||
print("=" * 60)
|
||||
|
||||
# 计算指标
|
||||
total_return = (1 + portfolio_returns).prod() - 1
|
||||
annualized_return = (1 + total_return) ** (252 / len(portfolio_returns)) - 1
|
||||
volatility = portfolio_returns.std() * np.sqrt(252)
|
||||
|
||||
# 计算最大回撤
|
||||
cumulative_returns = (1 + portfolio_returns).cumprod()
|
||||
running_max = cumulative_returns.expanding().max()
|
||||
drawdown = (cumulative_returns - running_max) / running_max
|
||||
max_drawdown = drawdown.min()
|
||||
|
||||
# 计算夏普比率
|
||||
risk_free_rate = 0.02
|
||||
sharpe_ratio = (annualized_return - risk_free_rate) / volatility if volatility > 0 else 0
|
||||
|
||||
# 计算资产相关性
|
||||
correlation_matrix = returns_data.corr()
|
||||
|
||||
# 计算各资产收益
|
||||
asset_returns = {}
|
||||
for col in returns_data.columns:
|
||||
asset_returns[col] = returns_data[col].tolist()
|
||||
|
||||
# 计算各资产累计收益
|
||||
asset_cumulative = {}
|
||||
for col in returns_data.columns:
|
||||
asset_cumulative[col] = (1 + returns_data[col]).cumprod().tolist()
|
||||
|
||||
# 计算平均权重
|
||||
avg_weights = weights_df.mean()
|
||||
|
||||
# 打印结果
|
||||
print(f"\n投资组合分析结果:")
|
||||
print(f" 总收益率: {total_return*100:.2f}%")
|
||||
print(f" 年化收益率: {annualized_return*100:.2f}%")
|
||||
print(f" 年化波动率: {volatility*100:.2f}%")
|
||||
print(f" 最大回撤: {max_drawdown*100:.2f}%")
|
||||
print(f" 夏普比率: {sharpe_ratio:.2f}")
|
||||
|
||||
print(f"\n平均资产权重:")
|
||||
for asset in avg_weights.index:
|
||||
print(f" {self.asset_mapping.get(asset, asset)} ({asset}): {avg_weights[asset]*100:.2f}%")
|
||||
|
||||
return {
|
||||
"total_return": total_return,
|
||||
"annualized_return": annualized_return,
|
||||
"volatility": volatility,
|
||||
"max_drawdown": max_drawdown,
|
||||
"sharpe_ratio": sharpe_ratio,
|
||||
"correlation_matrix": correlation_matrix.to_dict(),
|
||||
"asset_weights": avg_weights.to_dict(),
|
||||
"portfolio_returns": portfolio_returns.tolist(),
|
||||
"dates": returns_data.index.strftime('%Y-%m-%d').tolist(),
|
||||
"asset_returns": asset_returns,
|
||||
"asset_cumulative": asset_cumulative,
|
||||
"volatilities": {col: returns_data[col].std() * np.sqrt(252)
|
||||
for col in returns_data.columns},
|
||||
"rolling_weights": weights_df.to_dict()
|
||||
}
|
||||
|
||||
def generate_text_report(self, metrics, avg_weights):
|
||||
"""
|
||||
生成文字报告
|
||||
"""
|
||||
report = []
|
||||
report.append("=" * 60)
|
||||
report.append("中国市场滚动风险平价组合分析报告")
|
||||
report.append("=" * 60)
|
||||
report.append(f"生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
report.append(f"数据来源: {self.csv_path}")
|
||||
report.append(f"数据时间范围: {metrics['dates'][0]} 至 {metrics['dates'][-1]}")
|
||||
report.append(f"数据点数: {len(metrics['dates'])}")
|
||||
report.append(f"滚动窗口: 252个交易日（约1年）")
|
||||
report.append(f"调仓频率: 每月")
|
||||
report.append("")
|
||||
|
||||
# 投资组合配置
|
||||
report.append("平均投资组合配置:")
|
||||
for symbol, weight in avg_weights.items():
|
||||
asset_name = self.asset_mapping.get(symbol, symbol)
|
||||
vol = metrics['volatilities'][symbol]
|
||||
report.append(f" {asset_name} ({symbol}): {weight*100:.2f}% (波动率: {vol*100:.2f}%)")
|
||||
report.append("")
|
||||
|
||||
# 收益指标
|
||||
report.append("收益指标:")
|
||||
report.append(f" 总收益率: {metrics['total_return']*100:.2f}%")
|
||||
report.append(f" 年化收益率: {metrics['annualized_return']*100:.2f}%")
|
||||
report.append("")
|
||||
|
||||
# 风险指标
|
||||
report.append("风险指标:")
|
||||
report.append(f" 年化波动率: {metrics['volatility']*100:.2f}%")
|
||||
report.append(f" 最大回撤: {metrics['max_drawdown']*100:.2f}%")
|
||||
report.append(f" 夏普比率: {metrics['sharpe_ratio']:.2f}")
|
||||
report.append("")
|
||||
|
||||
# 资产相关性
|
||||
report.append("资产相关性矩阵:")
|
||||
corr_matrix = metrics['correlation_matrix']
|
||||
for asset1 in corr_matrix:
|
||||
asset1_name = self.asset_mapping.get(asset1, asset1)
|
||||
row = [f"{asset1_name}:"]
|
||||
for asset2 in corr_matrix[asset1]:
|
||||
asset2_name = self.asset_mapping.get(asset2, asset2)
|
||||
row.append(f" {asset2_name}: {corr_matrix[asset1][asset2]:.2f}")
|
||||
report.append(" ".join(row))
|
||||
report.append("")
|
||||
|
||||
# 投资建议
|
||||
report.append("投资建议:")
|
||||
if metrics['sharpe_ratio'] > 1:
|
||||
report.append(" 投资组合表现良好，夏普比率 > 1")
|
||||
elif metrics['sharpe_ratio'] > 0:
|
||||
report.append(" 投资组合表现一般，夏普比率在0-1之间")
|
||||
else:
|
||||
report.append(" 投资组合表现不佳，考虑调整配置")
|
||||
|
||||
if metrics['max_drawdown'] < -0.2:
|
||||
report.append(" 警告: 最大回撤超过20%，风险较高")
|
||||
|
||||
# 风险平价特点
|
||||
report.append("")
|
||||
report.append("滚动风险平价组合特点:")
|
||||
report.append(" - 使用历史波动率计算权重，避免未来数据")
|
||||
report.append(" - 每月调仓，动态调整权重")
|
||||
report.append(" - 各资产对组合风险贡献相等")
|
||||
report.append(" - 降低单一资产风险暴露")
|
||||
|
||||
# 资产类型分析
|
||||
report.append("")
|
||||
report.append("资产类型分析:")
|
||||
bond_assets = [s for s in avg_weights.keys() if self.asset_types.get(s) == 'bond']
|
||||
commodity_assets = [s for s in avg_weights.keys() if self.asset_types.get(s) == 'commodity']
|
||||
|
||||
if bond_assets:
|
||||
bond_weight = sum(avg_weights[s] for s in bond_assets)
|
||||
report.append(f" 债券类资产: {bond_weight*100:.2f}%")
|
||||
for s in bond_assets:
|
||||
report.append(f" - {self.asset_mapping.get(s, s)}: {avg_weights[s]*100:.2f}%")
|
||||
|
||||
if commodity_assets:
|
||||
commodity_weight = sum(avg_weights[s] for s in commodity_assets)
|
||||
report.append(f" 商品类资产: {commodity_weight*100:.2f}%")
|
||||
for s in commodity_assets:
|
||||
report.append(f" - {self.asset_mapping.get(s, s)}: {avg_weights[s]*100:.2f}%")
|
||||
|
||||
report.append("")
|
||||
report.append("=" * 60)
|
||||
|
||||
return "\n".join(report)
|
||||
|
||||
def generate_charts(self, metrics, weights_df, output_dir='.'):
|
||||
"""
|
||||
生成图表
|
||||
"""
|
||||
# 设置中文字体
|
||||
plt.rcParams['font.sans-serif'] = ['SimHei', 'Arial Unicode MS', 'DejaVu Sans']
|
||||
plt.rcParams['axes.unicode_minus'] = False
|
||||
|
||||
# 1. 收益曲线图
|
||||
plt.figure(figsize=(12, 6))
|
||||
dates = pd.to_datetime(metrics['dates'])
|
||||
cumulative_returns = (1 + np.array(metrics['portfolio_returns'])).cumprod()
|
||||
|
||||
plt.plot(dates, cumulative_returns, linewidth=2, color='blue')
|
||||
plt.title('滚动风险平价组合累计收益曲线', fontsize=14, fontweight='bold')
|
||||
plt.xlabel('日期', fontsize=12)
|
||||
plt.ylabel('累计收益 (倍数)', fontsize=12)
|
||||
plt.grid(True, alpha=0.3)
|
||||
plt.tight_layout()
|
||||
plt.savefig(f'{output_dir}/rolling_risk_parity_returns.png', dpi=300, bbox_inches='tight')
|
||||
plt.close()
|
||||
|
||||
# 2. 资产配置饼图（平均权重）
|
||||
plt.figure(figsize=(8, 8))
|
||||
labels = [self.asset_mapping.get(s, s) for s in metrics['asset_weights'].keys()]
|
||||
sizes = list(metrics['asset_weights'].values())
|
||||
colors = plt.cm.Set3(np.linspace(0, 1, len(labels)))
|
||||
|
||||
plt.pie(sizes, labels=labels, colors=colors, autopct='%1.1f%%', startangle=90)
|
||||
plt.title('滚动风险平价组合平均资产配置', fontsize=14, fontweight='bold')
|
||||
plt.axis('equal')
|
||||
plt.tight_layout()
|
||||
plt.savefig(f'{output_dir}/rolling_risk_parity_allocation.png', dpi=300, bbox_inches='tight')
|
||||
plt.close()
|
||||
|
||||
# 3. 相关性热力图
|
||||
corr_matrix = pd.DataFrame(metrics['correlation_matrix'])
|
||||
plt.figure(figsize=(10, 8))
|
||||
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', center=0,
|
||||
square=True, linewidths=0.5, cbar_kws={"shrink": 0.8})
|
||||
plt.title('资产相关性热力图', fontsize=14, fontweight='bold')
|
||||
plt.tight_layout()
|
||||
plt.savefig(f'{output_dir}/rolling_risk_parity_correlation.png', dpi=300, bbox_inches='tight')
|
||||
plt.close()
|
||||
|
||||
# 4. 资产收益对比图
|
||||
plt.figure(figsize=(12, 6))
|
||||
for symbol in metrics['asset_cumulative']:
|
||||
cumulative = metrics['asset_cumulative'][symbol]
|
||||
dates = pd.to_datetime(metrics['dates'])
|
||||
asset_name = self.asset_mapping.get(symbol, symbol)
|
||||
plt.plot(dates, cumulative, label=asset_name, linewidth=1.5)
|
||||
|
||||
plt.title('各资产累计收益对比', fontsize=14, fontweight='bold')
|
||||
plt.xlabel('日期', fontsize=12)
|
||||
plt.ylabel('累计收益 (倍数)', fontsize=12)
|
||||
plt.legend()
|
||||
plt.grid(True, alpha=0.3)
|
||||
plt.tight_layout()
|
||||
plt.savefig(f'{output_dir}/rolling_asset_returns_comparison.png', dpi=300, bbox_inches='tight')
|
||||
plt.close()
|
||||
|
||||
# 5. 滚动权重变化图
|
||||
plt.figure(figsize=(12, 6))
|
||||
dates = pd.to_datetime(metrics['dates'])
|
||||
|
||||
# 只显示部分日期的数据，避免图表过于密集
|
||||
if len(dates) > 100:
|
||||
step = len(dates) // 100
|
||||
dates = dates[::step]
|
||||
for asset in metrics['asset_weights'].keys():
|
||||
if asset in metrics['rolling_weights']:
|
||||
weights = pd.Series(metrics['rolling_weights'][asset])
|
||||
weights = weights[::step]
|
||||
plt.plot(dates, weights, label=self.asset_mapping.get(asset, asset), linewidth=1.5)
|
||||
else:
|
||||
for asset in metrics['asset_weights'].keys():
|
||||
if asset in metrics['rolling_weights']:
|
||||
weights = pd.Series(metrics['rolling_weights'][asset])
|
||||
plt.plot(dates, weights, label=self.asset_mapping.get(asset, asset), linewidth=1.5)
|
||||
|
||||
plt.title('滚动风险平价权重变化', fontsize=14, fontweight='bold')
|
||||
plt.xlabel('日期', fontsize=12)
|
||||
plt.ylabel('权重', fontsize=12)
|
||||
plt.legend()
|
||||
plt.grid(True, alpha=0.3)
|
||||
plt.tight_layout()
|
||||
plt.savefig(f'{output_dir}/rolling_weight_changes.png', dpi=300, bbox_inches='tight')
|
||||
plt.close()
|
||||
|
||||
print(f"图表已保存到 {output_dir} 目录")
|
||||
|
||||
def run_backtest(self, csv_path=None, output_dir='./backtest_output'):
|
||||
"""
|
||||
运行回测
|
||||
"""
|
||||
print("=" * 60)
|
||||
print("运行滚动风险平价组合回测")
|
||||
print("=" * 60)
|
||||
|
||||
# 1. 加载数据
|
||||
clean_data = self.load_data(csv_path)
|
||||
|
||||
# 2. 计算滚动波动率
|
||||
rolling_volatilities = self.calculate_rolling_volatility(clean_data)
|
||||
|
||||
# 3. 计算滚动权重
|
||||
weights_df = self.calculate_rolling_weights(rolling_volatilities)
|
||||
|
||||
# 4. 计算投资组合收益率
|
||||
portfolio_returns = self.calculate_portfolio_returns(clean_data, weights_df)
|
||||
|
||||
# 5. 分析投资组合
|
||||
metrics = self.analyze_portfolio(clean_data, weights_df, portfolio_returns)
|
||||
|
||||
# 6. 生成文字报告
|
||||
avg_weights = weights_df.mean()
|
||||
text_report = self.generate_text_report(metrics, avg_weights)
|
||||
|
||||
# 7. 生成图表
|
||||
self.generate_charts(metrics, weights_df, output_dir)
|
||||
|
||||
# 8. 保存报告到文件
|
||||
report_file = f"{output_dir}/rolling_risk_parity_report.txt"
|
||||
with open(report_file, 'w', encoding='utf-8') as f:
|
||||
f.write(text_report)
|
||||
|
||||
# 9. 保存详细数据到JSON
|
||||
detailed_data = {
|
||||
"portfolio_config": avg_weights.to_dict(),
|
||||
"metrics": {
|
||||
"total_return": metrics["total_return"],
|
||||
"annualized_return": metrics["annualized_return"],
|
||||
"volatility": metrics["volatility"],
|
||||
"max_drawdown": metrics["max_drawdown"],
|
||||
"sharpe_ratio": metrics["sharpe_ratio"],
|
||||
"asset_weights": metrics["asset_weights"],
|
||||
"volatilities": metrics["volatilities"]
|
||||
},
|
||||
"asset_mapping": self.asset_mapping,
|
||||
"analysis_time": datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
|
||||
"rolling_weights": {
|
||||
asset: {str(date): weight for date, weight in weights_df[asset].items() if not pd.isna(weight)}
|
||||
for asset in weights_df.columns
|
||||
}
|
||||
}
|
||||
|
||||
json_file = f"{output_dir}/rolling_risk_parity_data.json"
|
||||
with open(json_file, 'w', encoding='utf-8') as f:
|
||||
import json
|
||||
json.dump(detailed_data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
print(f"\n回测完成！")
|
||||
print(f"报告文件: {report_file}")
|
||||
print(f"数据文件: {json_file}")
|
||||
print(f"图表文件: {output_dir}/rolling_*.png")
|
||||
|
||||
# 存储结果
|
||||
self.results = {
|
||||
"text_report": text_report,
|
||||
"report_file": report_file,
|
||||
"json_file": json_file,
|
||||
"charts": [
|
||||
f"{output_dir}/rolling_risk_parity_returns.png",
|
||||
f"{output_dir}/rolling_risk_parity_allocation.png",
|
||||
f"{output_dir}/rolling_risk_parity_correlation.png",
|
||||
f"{output_dir}/rolling_asset_returns_comparison.png",
|
||||
f"{output_dir}/rolling_weight_changes.png"
|
||||
],
|
||||
"metrics": metrics,
|
||||
"avg_weights": avg_weights.to_dict(),
|
||||
"rolling_weights": weights_df.to_dict()
|
||||
}
|
||||
|
||||
return self.results
|
||||
|
||||
def get_report(self):
|
||||
"""
|
||||
获取分析报告
|
||||
"""
|
||||
if not self.results:
|
||||
raise ValueError("请先运行回测分析")
|
||||
|
||||
return self.results.get("text_report", "无报告数据")
|
||||
|
||||
def get_metrics(self):
|
||||
"""
|
||||
获取分析指标
|
||||
"""
|
||||
if not self.results:
|
||||
raise ValueError("请先运行回测分析")
|
||||
|
||||
return self.results.get("metrics", {})
|
||||
|
||||
def get_weights(self):
|
||||
"""
|
||||
获取资产权重
|
||||
"""
|
||||
if not self.results:
|
||||
raise ValueError("请先运行回测分析")
|
||||
|
||||
return self.results.get("avg_weights", {})
|
||||
|
||||
def main():
|
||||
"""主函数"""
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description='优化的风险平价组合分析技能')
|
||||
parser.add_argument('--csv', type=str, default='C:\\Users\\wu_zhuoran\\.openclaw\\workspace\\data\\marketdata.csv',
|
||||
help='CSV文件路径')
|
||||
parser.add_argument('--output', type=str, default='./backtest_output', help='输出目录')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# 创建技能实例
|
||||
skill = OptimizedRiskParitySkill(args.csv)
|
||||
|
||||
# 运行回测
|
||||
result = skill.run_backtest(args.csv, args.output)
|
||||
|
||||
# 显示文字报告
|
||||
print("\n" + "=" * 60)
|
||||
print("分析报告预览")
|
||||
print("=" * 60)
|
||||
print(result['text_report'])
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,279 @@
|
|||
# 优化的金融分析技能使用指南
|
||||
|
||||
## 概述
|
||||
|
||||
优化的金融分析技能整合了滚动窗口调仓逻辑和回测功能，提供更真实的投资组合分析。
|
||||
|
||||
## 核心功能
|
||||
|
||||
### 1. 滚动窗口风险平价分析
|
||||
- **避免未来数据**：使用历史一年数据计算波动率
|
||||
- **动态调仓**：每月根据最新波动率重新计算权重
|
||||
- **风险平价**：各资产对组合风险贡献相等
|
||||
- **回测验证**：完整的回测流程，验证策略效果
|
||||
|
||||
### 2. 数据处理
|
||||
- **CSV数据导入**：支持中国市场数据
|
||||
- **百分比转换**：自动将百分比数据转换为实际收益率
|
||||
- **异常值处理**：识别和处理异常值
|
||||
- **缺失值处理**：删除包含NaN的行
|
||||
|
||||
### 3. 分析指标
|
||||
- **总收益率**：整个回测期间的总收益
|
||||
- **年化收益率**：年化后的收益率
|
||||
- **年化波动率**：年化后的波动率
|
||||
- **最大回撤**：最大回撤幅度
|
||||
- **夏普比率**：风险调整后的收益
|
||||
|
||||
## 安装依赖
|
||||
|
||||
```bash
|
||||
pip install pandas numpy matplotlib seaborn
|
||||
```
|
||||
|
||||
## 使用方法
|
||||
|
||||
### 1. 基本使用
|
||||
|
||||
#### 运行回测分析
|
||||
```bash
|
||||
python optimized_main.py --csv "C:\path\to\marketdata.csv" --output ./backtest_output
|
||||
```
|
||||
|
||||
#### 使用默认数据路径
|
||||
```bash
|
||||
python optimized_main.py
|
||||
```
|
||||
|
||||
#### 自定义输出目录
|
||||
```bash
|
||||
python optimized_main.py --output ./my_backtest
|
||||
```
|
||||
|
||||
### 2. 查看分析结果
|
||||
|
||||
#### 显示分析报告
|
||||
```bash
|
||||
python optimized_main.py --show-report
|
||||
```
|
||||
|
||||
#### 显示分析指标
|
||||
```bash
|
||||
python optimized_main.py --show-metrics
|
||||
```
|
||||
|
||||
#### 显示资产权重
|
||||
```bash
|
||||
python optimized_main.py --show-weights
|
||||
```
|
||||
|
||||
### 3. 直接使用技能类
|
||||
|
||||
```python
|
||||
from optimized_risk_parity_skill import OptimizedRiskParitySkill
|
||||
|
||||
# 创建技能实例
|
||||
skill = OptimizedRiskParitySkill('C:\path\to\marketdata.csv')
|
||||
|
||||
# 运行回测
|
||||
result = skill.run_backtest('C:\path\to\marketdata.csv', './backtest_output')
|
||||
|
||||
# 获取报告
|
||||
report = skill.get_report()
|
||||
print(report)
|
||||
|
||||
# 获取指标
|
||||
metrics = skill.get_metrics()
|
||||
print(f"总收益率: {metrics['total_return']*100:.2f}%")
|
||||
print(f"夏普比率: {metrics['sharpe_ratio']:.2f}")
|
||||
|
||||
# 获取权重
|
||||
weights = skill.get_weights()
|
||||
for asset, weight in weights.items():
|
||||
print(f"{asset}: {weight*100:.2f}%")
|
||||
```
|
||||
|
||||
## 数据格式要求
|
||||
|
||||
### CSV文件格式
|
||||
- 第0行：资产名称（如：CFFEX5五年期国债期货）
|
||||
- 第1行：数据类型（如：涨跌幅(%)）
|
||||
- 第2行开始：实际数据
|
||||
- 第1列：日期
|
||||
- 第2-5列：各资产收益率（百分比格式）
|
||||
|
||||
### 示例数据
|
||||
```
|
||||
日期,TF.CFE,T.CFE,CU.SHF,AU.SHF
|
||||
2015-03-20,0.1377,NaN,0.767,-0.1472
|
||||
2015-03-23,-0.1834,-0.0721,3.9248,1.1373
|
||||
2015-03-24,-0.0510,-0.1031,0.3662,0.4165
|
||||
```
|
||||
|
||||
## 分析流程
|
||||
|
||||
### 1. 数据加载
|
||||
- 读取CSV文件
|
||||
- 提取实际数据（跳过前两行）
|
||||
- 将百分比数据转换为实际收益率（除以100）
|
||||
- 删除缺失值
|
||||
|
||||
### 2. 滚动波动率计算
|
||||
- 使用expanding窗口计算历史波动率
|
||||
- 最小窗口大小：252个交易日（约1年）
|
||||
- 年化波动率：波动率 × √252
|
||||
|
||||
### 3. 权重计算
|
||||
- 基于波动率倒数加权：`权重 = (1/波动率) / ∑(1/波动率)`
|
||||
- 每月调仓，动态调整权重
|
||||
- 使用前一天的权重计算当日收益率
|
||||
|
||||
### 4. 投资组合收益率计算
|
||||
- 使用滚动权重计算每日收益率
|
||||
- 累计收益：`(1 + 每日收益率).cumprod()`
|
||||
|
||||
### 5. 绩效评估
|
||||
- 总收益率：累计收益 - 1
|
||||
- 年化收益率：`(1 + 总收益率)^(252/天数) - 1`
|
||||
- 年化波动率：每日收益率标准差 × √252
|
||||
- 最大回撤：累计收益最大值与当前值的差值
|
||||
- 夏普比率：`(年化收益率 - 无风险利率) / 年化波动率`
|
||||
|
||||
### 6. 可视化
|
||||
- 收益曲线图：累计收益随时间变化
|
||||
- 资产配置饼图：各资产平均权重
|
||||
- 相关性热力图：资产间相关性
|
||||
- 资产收益对比图：各资产累计收益
|
||||
- 滚动权重变化图：权重随时间变化
|
||||
|
||||
## 输出文件
|
||||
|
||||
### 分析报告
|
||||
- **文件名**: `rolling_risk_parity_report.txt`
|
||||
- **内容**: 完整的分析报告，包括配置、收益、风险、建议等
|
||||
|
||||
### 详细数据
|
||||
- **文件名**: `rolling_risk_parity_data.json`
|
||||
- **内容**:
|
||||
- 投资组合配置
|
||||
- 分析指标
|
||||
- 资产映射
|
||||
- 滚动权重数据
|
||||
|
||||
### 可视化图表
|
||||
1. **收益曲线图**: `rolling_risk_parity_returns.png`
|
||||
2. **资产配置饼图**: `rolling_risk_parity_allocation.png`
|
||||
3. **相关性热力图**: `rolling_risk_parity_correlation.png`
|
||||
4. **资产收益对比图**: `rolling_asset_returns_comparison.png`
|
||||
5. **滚动权重变化图**: `rolling_weight_changes.png`
|
||||
|
||||
## 示例输出
|
||||
|
||||
### 分析报告
|
||||
```
|
||||
中国市场滚动风险平价组合分析报告
|
||||
============================================================
|
||||
生成时间: 2026-02-26 10:20:04
|
||||
数据来源: C:\Users\wu_zhuoran\.openclaw\workspace\data\marketdata.csv
|
||||
数据时间范围: 2015-03-23 至 2026-02-25
|
||||
数据点数: 2656
|
||||
滚动窗口: 252个交易日（约1年）
|
||||
调仓频率: 每月
|
||||
|
||||
平均投资组合配置:
|
||||
五年期国债 (TF.CFE): 46.04% (波动率: 2.46%)
|
||||
十年期国债 (T.CFE): 32.55% (波动率: 3.61%)
|
||||
沪铜 (CU.SHF): 9.33% (波动率: 17.87%)
|
||||
沪金 (AU.SHF): 12.09% (波动率: 14.88%)
|
||||
|
||||
收益指标:
|
||||
总收益率: 44.78%
|
||||
年化收益率: 3.57%
|
||||
|
||||
风险指标:
|
||||
年化波动率: 3.76%
|
||||
最大回撤: -9.48%
|
||||
夏普比率: 0.42
|
||||
```
|
||||
|
||||
## 优化特点
|
||||
|
||||
### 1. 避免未来数据
|
||||
- 使用expanding窗口而非rolling窗口
|
||||
- 确保不使用未来数据
|
||||
- 最小窗口大小为252个交易日
|
||||
|
||||
### 2. 动态调仓
|
||||
- 每月根据最新波动率重新计算权重
|
||||
- 模拟真实投资中的调仓操作
|
||||
- 适应市场变化
|
||||
|
||||
### 3. 风险平价
|
||||
- 各资产对组合风险贡献相等
|
||||
- 降低单一资产风险暴露
|
||||
- 适合长期持有
|
||||
|
||||
### 4. 回测验证
|
||||
- 完整的回测流程
|
||||
- 避免数据泄露
|
||||
- 提供详细的绩效指标
|
||||
|
||||
## 注意事项
|
||||
|
||||
### 数据质量
|
||||
- 确保数据质量，处理缺失值和异常值
|
||||
- 检查数据格式是否正确
|
||||
- 验证数据时间范围
|
||||
|
||||
### 参数调整
|
||||
- 滚动窗口大小可根据实际情况调整
|
||||
- 调仓频率可根据市场情况调整
|
||||
- 考虑交易成本对回测结果的影响
|
||||
|
||||
### 结果解读
|
||||
- 夏普比率 > 1: 投资组合表现良好
|
||||
- 夏普比率 0-1: 投资组合表现一般
|
||||
- 夏普比率 < 0: 投资组合表现不佳
|
||||
- 最大回撤 > 20%: 风险较高
|
||||
|
||||
## 常见问题
|
||||
|
||||
### Q: 如何处理异常值？
|
||||
A: 脚本会自动检查数据中的异常值，但建议在分析前检查数据质量。
|
||||
|
||||
### Q: 如何调整调仓频率？
|
||||
A: 修改`rolling_window_risk_parity.py`中的`rebalance_frequency`参数。
|
||||
|
||||
### Q: 如何调整滚动窗口大小？
|
||||
A: 修改`rolling_window_risk_parity.py`中的`window_size`参数。
|
||||
|
||||
### Q: 如何添加更多资产？
|
||||
A: 在`asset_mapping`和`asset_types`中添加新资产信息。
|
||||
|
||||
### Q: 如何比较不同策略？
|
||||
A: 运行多次分析，比较不同参数下的结果。
|
||||
|
||||
## 技术细节
|
||||
|
||||
### 滚动窗口计算
|
||||
```python
|
||||
# 使用expanding窗口计算波动率
|
||||
rolling_vol = returns_data[asset].expanding(min_periods=window_size).std() * np.sqrt(252)
|
||||
```
|
||||
|
||||
### 权重计算
|
||||
```python
|
||||
# 基于波动率倒数加权
|
||||
total_vol = sum(1/v for v in current_vols.values())
|
||||
weight = (1/current_vols[asset]) / total_vol
|
||||
```
|
||||
|
||||
### 投资组合收益率
|
||||
```python
|
||||
# 使用前一天的权重计算当日收益率
|
||||
portfolio_return = sum(daily_returns[asset] * weights[asset] for asset in assets)
|
||||
```
|
||||
|
||||
## 总结
|
||||
|
||||
优化的金融分析技能提供了完整的滚动窗口风险平价组合分析功能，避免了未来数据问题，提供了更真实的回测结果。通过动态调仓和风险平价策略，为投资者提供有价值的参考。
|
||||
|
|
@ -0,0 +1,300 @@
|
|||
# 优化的金融分析技能 - 总结
|
||||
|
||||
## 🎯 项目目标
|
||||
|
||||
根据调仓逻辑和回测情况，优化之前做好的金融分析技能，提供更真实的投资组合分析。
|
||||
|
||||
## ✅ 已完成的任务
|
||||
|
||||
### 1. 创建优化的技能脚本
|
||||
- ✅ `optimized_risk_parity_skill.py` - 优化的风险平价技能
|
||||
- ✅ `optimized_main.py` - 优化的主脚本
|
||||
- ✅ `optimized_risk_parity_skill.py` - 整合滚动窗口调仓逻辑
|
||||
|
||||
### 2. 更新文档
|
||||
- ✅ `SKILL.md` - 更新技能说明
|
||||
- ✅ `优化使用指南.md` - 详细的使用指南
|
||||
- ✅ `优化总结.md` - 本总结文档
|
||||
|
||||
### 3. 核心功能实现
|
||||
- ✅ **滚动窗口风险平价分析**：使用历史一年数据，每月调仓
|
||||
- ✅ **避免未来数据**：使用expanding窗口计算波动率
|
||||
- ✅ **动态调仓**：每月根据最新波动率重新计算权重
|
||||
- ✅ **回测验证**：完整的回测流程，验证策略效果
|
||||
- ✅ **可视化图表**：生成5个专业图表
|
||||
- ✅ **详细报告**：生成完整的分析报告
|
||||
|
||||
## 📊 优化后的分析结果
|
||||
|
||||
### 滚动窗口风险平价组合（优化后）
|
||||
|
||||
**数据来源**: `C:\Users\wu_zhuoran\.openclaw\workspace\data\marketdata.csv`
|
||||
**数据时间范围**: 2015-03-23 至 2026-02-25
|
||||
**数据点数**: 2656
|
||||
**滚动窗口**: 252个交易日（约1年）
|
||||
**调仓频率**: 每月
|
||||
|
||||
#### 投资组合配置（平均权重）
|
||||
|
||||
| 资产 | 权重 | 波动率 | 类型 |
|
||||
|------|------|--------|------|
|
||||
| **五年期国债** | 46.04% | 2.46% | 债券 |
|
||||
| **十年期国债** | 32.55% | 3.61% | 债券 |
|
||||
| **沪铜** | 9.33% | 17.87% | 商品 |
|
||||
| **沪金** | 12.09% | 14.88% | 商品 |
|
||||
|
||||
**资产类型分布**:
|
||||
- **债券类资产**: 78.58% (五年期国债 46.04% + 十年期国债 32.55%)
|
||||
- **商品类资产**: 21.42% (沪铜 9.33% + 沪金 12.09%)
|
||||
|
||||
#### 绩效指标
|
||||
|
||||
| 指标 | 数值 |
|
||||
|------|------|
|
||||
| **总收益率** | 44.78% |
|
||||
| **年化收益率** | 3.57% |
|
||||
| **年化波动率** | 3.76% |
|
||||
| **最大回撤** | -9.48% |
|
||||
| **夏普比率** | 0.42 |
|
||||
|
||||
## 🎯 优化特点
|
||||
|
||||
### 1. 避免未来数据
|
||||
- ✅ 使用expanding窗口而非rolling窗口
|
||||
- ✅ 确保不使用未来数据
|
||||
- ✅ 最小窗口大小为252个交易日
|
||||
|
||||
### 2. 动态调仓
|
||||
- ✅ 每月根据最新波动率重新计算权重
|
||||
- ✅ 模拟真实投资中的调仓操作
|
||||
- ✅ 适应市场变化
|
||||
|
||||
### 3. 风险平价
|
||||
- ✅ 各资产对组合风险贡献相等
|
||||
- ✅ 降低单一资产风险暴露
|
||||
- ✅ 适合长期持有
|
||||
|
||||
### 4. 回测验证
|
||||
- ✅ 完整的回测流程
|
||||
- ✅ 避免数据泄露
|
||||
- ✅ 提供详细的绩效指标
|
||||
|
||||
## 📁 输出文件
|
||||
|
||||
### 分析报告
|
||||
- `backtest_output/rolling_risk_parity_report.txt`
|
||||
|
||||
### 详细数据
|
||||
- `backtest_output/rolling_risk_parity_data.json`
|
||||
|
||||
### 可视化图表
|
||||
- `rolling_risk_parity_returns.png` - 收益曲线图
|
||||
- `rolling_risk_parity_allocation.png` - 资产配置饼图
|
||||
- `rolling_risk_parity_correlation.png` - 相关性热力图
|
||||
- `rolling_asset_returns_comparison.png` - 资产收益对比图
|
||||
- `rolling_weight_changes.png` - 滚动权重变化图
|
||||
|
||||
## 📚 文档文件
|
||||
|
||||
### 技术文档
|
||||
- `SKILL.md` - 技能说明
|
||||
- `优化使用指南.md` - 详细的使用指南
|
||||
- `优化总结.md` - 本总结文档
|
||||
|
||||
### 使用文档
|
||||
- `使用指南.md` - 基本使用指南
|
||||
- `风险平价组合使用说明.md` - 风险平价组合使用说明
|
||||
|
||||
## 🚀 使用方法
|
||||
|
||||
### 1. 基本使用
|
||||
```bash
|
||||
python optimized_main.py --csv "C:\path\to\marketdata.csv" --output ./backtest_output
|
||||
```
|
||||
|
||||
### 2. 查看分析结果
|
||||
```bash
|
||||
python optimized_main.py --show-report
|
||||
python optimized_main.py --show-metrics
|
||||
python optimized_main.py --show-weights
|
||||
```
|
||||
|
||||
### 3. 直接使用技能类
|
||||
```python
|
||||
from optimized_risk_parity_skill import OptimizedRiskParitySkill
|
||||
|
||||
skill = OptimizedRiskParitySkill('C:\path\to\marketdata.csv')
|
||||
result = skill.run_backtest('C:\path\to\marketdata.csv', './backtest_output')
|
||||
report = skill.get_report()
|
||||
metrics = skill.get_metrics()
|
||||
weights = skill.get_weights()
|
||||
```
|
||||
|
||||
## 🎯 技术实现
|
||||
|
||||
### 滚动窗口计算
|
||||
```python
|
||||
# 使用expanding窗口计算波动率
|
||||
rolling_vol = returns_data[asset].expanding(min_periods=window_size).std() * np.sqrt(252)
|
||||
```
|
||||
|
||||
### 权重计算
|
||||
```python
|
||||
# 基于波动率倒数加权
|
||||
total_vol = sum(1/v for v in current_vols.values())
|
||||
weight = (1/current_vols[asset]) / total_vol
|
||||
```
|
||||
|
||||
### 投资组合收益率
|
||||
```python
|
||||
# 使用前一天的权重计算当日收益率
|
||||
portfolio_return = sum(daily_returns[asset] * weights[asset] for asset in assets)
|
||||
```
|
||||
|
||||
## 📈 与静态方法的对比
|
||||
|
||||
### 静态风险平价（使用全历史数据）
|
||||
- **总收益率**: 42.31%
|
||||
- **年化收益率**: 3.40%
|
||||
- **年化波动率**: 3.11%
|
||||
- **最大回撤**: -4.20%
|
||||
- **夏普比率**: 0.45
|
||||
- **债券类资产**: 84.75%
|
||||
- **商品类资产**: 15.25%
|
||||
|
||||
### 滚动窗口风险平价（优化后）
|
||||
- **总收益率**: 44.78% ✅
|
||||
- **年化收益率**: 3.57% ✅
|
||||
- **年化波动率**: 3.76%
|
||||
- **最大回撤**: -9.48%
|
||||
- **夏普比率**: 0.42
|
||||
- **债券类资产**: 78.58%
|
||||
- **商品类资产**: 21.42% ✅
|
||||
|
||||
### 关键发现
|
||||
1. **收益更高**: 44.78% vs 42.31% ✅
|
||||
2. **避免未来数据**: 使用历史一年数据 ✅
|
||||
3. **商品暴露增加**: 21.42% vs 15.25% ✅
|
||||
4. **风险略高**: 最大回撤-9.48% vs -4.20%
|
||||
5. **夏普比率略低**: 0.42 vs 0.45
|
||||
|
||||
## 💡 投资建议
|
||||
|
||||
### 优点
|
||||
1. **避免未来数据**: 使用历史一年数据计算波动率，避免数据泄露
|
||||
2. **动态调整**: 每月调仓，适应市场变化
|
||||
3. **收益更高**: 总收益率44.78%，略高于静态方法
|
||||
4. **商品暴露增加**: 提供更好的分散化
|
||||
|
||||
### 缺点
|
||||
1. **风险更高**: 最大回撤-9.48%，高于静态方法
|
||||
2. **波动率更高**: 年化波动率3.76%，高于静态方法
|
||||
3. **夏普比率略低**: 0.42 vs 0.45
|
||||
|
||||
### 改进建议
|
||||
1. **优化调仓频率**: 可以尝试季度调仓或半年调仓
|
||||
2. **调整窗口大小**: 可以尝试6个月或18个月的滚动窗口
|
||||
3. **加入止损机制**: 控制最大回撤
|
||||
4. **考虑交易成本**: 每月调仓可能产生交易成本
|
||||
|
||||
## 🎉 项目成果
|
||||
|
||||
### 技能功能
|
||||
1. ✅ 支持CSV数据导入
|
||||
2. ✅ 自动处理百分比单位转换
|
||||
3. ✅ 滚动窗口风险平价分析
|
||||
4. ✅ 避免未来数据
|
||||
5. ✅ 动态调仓
|
||||
6. ✅ 回测验证
|
||||
7. ✅ 详细分析报告生成
|
||||
8. ✅ 多维度可视化图表
|
||||
9. ✅ 投资建议和改进方向
|
||||
|
||||
### 分析能力
|
||||
1. ✅ 资产配置分析
|
||||
2. ✅ 风险评估
|
||||
3. ✅ 收益分析
|
||||
4. ✅ 相关性分析
|
||||
5. ✅ 资产类型分析
|
||||
6. ✅ 滚动权重变化分析
|
||||
|
||||
### 输出能力
|
||||
1. ✅ 文字报告
|
||||
2. ✅ JSON数据
|
||||
3. ✅ PNG图表
|
||||
4. ✅ 交互式分析
|
||||
|
||||
## 📊 项目文件结构
|
||||
|
||||
```
|
||||
financial-analysis-skill/
|
||||
├── optimized_risk_parity_skill.py # 优化的风险平价技能
|
||||
├── optimized_main.py # 优化的主脚本
|
||||
├── SKILL.md # 技能说明
|
||||
├── 优化使用指南.md # 使用指南
|
||||
├── 优化总结.md # 本总结文档
|
||||
└── ... (其他文件)
|
||||
|
||||
backtest_output/
|
||||
├── rolling_risk_parity_report.txt # 分析报告
|
||||
├── rolling_risk_parity_data.json # 分析数据
|
||||
├── rolling_risk_parity_returns.png # 收益曲线图
|
||||
├── rolling_risk_parity_allocation.png # 资产配置饼图
|
||||
├── rolling_risk_parity_correlation.png # 相关性热力图
|
||||
├── rolling_asset_returns_comparison.png # 资产收益对比图
|
||||
└── rolling_weight_changes.png # 滚动权重变化图
|
||||
```
|
||||
|
||||
## 🎯 下一步计划
|
||||
|
||||
### 短期行动（1-2周）
|
||||
1. ✅ 完成优化的金融分析技能
|
||||
2. ✅ 生成分析报告和图表
|
||||
3. ✅ 更新文档和内存
|
||||
4. ✅ 测试技能功能
|
||||
|
||||
### 中期行动（1-2个月）
|
||||
1. 比较不同调仓频率的效果
|
||||
2. 优化滚动窗口大小
|
||||
3. 考虑加入更多资产类别
|
||||
4. 开发交互式分析界面
|
||||
|
||||
### 长期行动（3-6个月）
|
||||
1. 定期更新市场数据
|
||||
2. 监控投资组合表现
|
||||
3. 根据市场变化调整配置
|
||||
4. 扩展技能功能
|
||||
|
||||
## 🎉 项目总结
|
||||
|
||||
优化的金融分析技能成功完成，提供了：
|
||||
|
||||
1. **完整的分析框架**：从数据解析到报告生成的完整流程
|
||||
2. **避免未来数据**：使用滚动窗口计算波动率，避免数据泄露
|
||||
3. **动态调仓**：每月调仓，适应市场变化
|
||||
4. **风险平价策略**：各资产风险贡献均衡
|
||||
5. **全面的评估指标**：收益、风险、相关性等多维度分析
|
||||
6. **可视化的输出**：图表和报告便于理解和决策
|
||||
|
||||
### 项目价值
|
||||
- 为投资者提供中国市场风险平价组合的参考
|
||||
- 展示了风险平价策略在中国市场的应用效果
|
||||
- 提供了可复用的分析工具和方法
|
||||
- 为后续研究和优化奠定基础
|
||||
|
||||
### 技术贡献
|
||||
- 正确处理中国市场数据格式
|
||||
- 实现滚动窗口风险平价权重计算
|
||||
- 生成专业分析报告和图表
|
||||
- 提供完整的分析工具链
|
||||
|
||||
---
|
||||
|
||||
**项目状态**: ✅ 完成
|
||||
**分析时间**: 2026-02-26
|
||||
**数据时间范围**: 2015-03-23 至 2026-02-25
|
||||
**数据点数**: 2656
|
||||
**滚动窗口**: 252个交易日（约1年）
|
||||
**调仓频率**: 每月
|
||||
**总收益率**: 44.78%
|
||||
**夏普比率**: 0.42
|
||||
|
|
@ -0,0 +1,329 @@
|
|||
# 金融分析技能最终优化总结
|
||||
|
||||
## 🎯 项目目标
|
||||
|
||||
根据调仓逻辑和回测情况，优化之前做好的金融分析技能，提供更真实的投资组合分析。
|
||||
|
||||
## ✅ 已完成的任务
|
||||
|
||||
### 1. 创建优化的技能脚本
|
||||
- ✅ `optimized_risk_parity_skill.py` - 优化的风险平价技能
|
||||
- ✅ `optimized_main.py` - 优化的主脚本
|
||||
- ✅ 整合滚动窗口调仓逻辑和回测功能
|
||||
|
||||
### 2. 更新文档
|
||||
- ✅ `SKILL.md` - 更新技能说明
|
||||
- ✅ `优化使用指南.md` - 详细的使用指南
|
||||
- ✅ `优化总结.md` - 优化总结文档
|
||||
- ✅ `最终优化总结.md` - 本总结文档
|
||||
|
||||
### 3. 核心功能实现
|
||||
- ✅ **滚动窗口风险平价分析**：使用历史一年数据，每月调仓
|
||||
- ✅ **避免未来数据**：使用expanding窗口计算波动率
|
||||
- ✅ **动态调仓**：每月根据最新波动率重新计算权重
|
||||
- ✅ **回测验证**：完整的回测流程，验证策略效果
|
||||
- ✅ **可视化图表**：生成5个专业图表
|
||||
- ✅ **详细报告**：生成完整的分析报告
|
||||
|
||||
## 📊 优化后的分析结果
|
||||
|
||||
### 滚动窗口风险平价组合（优化后）
|
||||
|
||||
**数据来源**: `C:\Users\wu_zhuoran\.openclaw\workspace\data\marketdata.csv`
|
||||
**数据时间范围**: 2015-03-23 至 2026-02-25
|
||||
**数据点数**: 2656
|
||||
**滚动窗口**: 252个交易日（约1年）
|
||||
**调仓频率**: 每月
|
||||
|
||||
#### 投资组合配置（平均权重）
|
||||
|
||||
| 资产 | 权重 | 波动率 | 类型 |
|
||||
|------|------|--------|------|
|
||||
| **五年期国债** | 46.04% | 2.46% | 债券 |
|
||||
| **十年期国债** | 32.55% | 3.61% | 债券 |
|
||||
| **沪铜** | 9.33% | 17.87% | 商品 |
|
||||
| **沪金** | 12.09% | 14.88% | 商品 |
|
||||
|
||||
**资产类型分布**:
|
||||
- **债券类资产**: 78.58% (五年期国债 46.04% + 十年期国债 32.55%)
|
||||
- **商品类资产**: 21.42% (沪铜 9.33% + 沪金 12.09%)
|
||||
|
||||
#### 绩效指标
|
||||
|
||||
| 指标 | 数值 |
|
||||
|------|------|
|
||||
| **总收益率** | 44.78% |
|
||||
| **年化收益率** | 3.57% |
|
||||
| **年化波动率** | 3.76% |
|
||||
| **最大回撤** | -9.48% |
|
||||
| **夏普比率** | 0.42 |
|
||||
|
||||
## 🎯 优化特点
|
||||
|
||||
### 1. 避免未来数据
|
||||
- ✅ 使用expanding窗口而非rolling窗口
|
||||
- ✅ 确保不使用未来数据
|
||||
- ✅ 最小窗口大小为252个交易日
|
||||
|
||||
### 2. 动态调仓
|
||||
- ✅ 每月根据最新波动率重新计算权重
|
||||
- ✅ 模拟真实投资中的调仓操作
|
||||
- ✅ 适应市场变化
|
||||
|
||||
### 3. 风险平价
|
||||
- ✅ 各资产对组合风险贡献相等
|
||||
- ✅ 降低单一资产风险暴露
|
||||
- ✅ 适合长期持有
|
||||
|
||||
### 4. 回测验证
|
||||
- ✅ 完整的回测流程
|
||||
- ✅ 避免数据泄露
|
||||
- ✅ 提供详细的绩效指标
|
||||
|
||||
## 📁 输出文件
|
||||
|
||||
### 分析报告
|
||||
- `backtest_output/rolling_risk_parity_report.txt`
|
||||
|
||||
### 详细数据
|
||||
- `backtest_output/rolling_risk_parity_data.json`
|
||||
|
||||
### 可视化图表
|
||||
- `rolling_risk_parity_returns.png` - 收益曲线图
|
||||
- `rolling_risk_parity_allocation.png` - 资产配置饼图
|
||||
- `rolling_risk_parity_correlation.png` - 相关性热力图
|
||||
- `rolling_asset_returns_comparison.png` - 资产收益对比图
|
||||
- `rolling_weight_changes.png` - 滚动权重变化图
|
||||
|
||||
## 📚 文档文件
|
||||
|
||||
### 技术文档
|
||||
- `SKILL.md` - 技能说明
|
||||
- `优化使用指南.md` - 详细的使用指南
|
||||
- `优化总结.md` - 优化总结文档
|
||||
- `最终优化总结.md` - 本总结文档
|
||||
|
||||
### 使用文档
|
||||
- `使用指南.md` - 基本使用指南
|
||||
- `风险平价组合使用说明.md` - 风险平价组合使用说明
|
||||
|
||||
## 🚀 使用方法
|
||||
|
||||
### 1. 基本使用
|
||||
```bash
|
||||
python optimized_main.py --csv "C:\path\to\marketdata.csv" --output ./backtest_output
|
||||
```
|
||||
|
||||
### 2. 查看分析结果
|
||||
```bash
|
||||
python optimized_main.py --show-report
|
||||
python optimized_main.py --show-metrics
|
||||
python optimized_main.py --show-weights
|
||||
```
|
||||
|
||||
### 3. 直接使用技能类
|
||||
```python
|
||||
from optimized_risk_parity_skill import OptimizedRiskParitySkill
|
||||
|
||||
skill = OptimizedRiskParitySkill('C:\path\to\marketdata.csv')
|
||||
result = skill.run_backtest('C:\path\to\marketdata.csv', './backtest_output')
|
||||
report = skill.get_report()
|
||||
metrics = skill.get_metrics()
|
||||
weights = skill.get_weights()
|
||||
```
|
||||
|
||||
## 📈 与静态方法的对比
|
||||
|
||||
### 静态风险平价（使用全历史数据）
|
||||
- **总收益率**: 42.31%
|
||||
- **年化收益率**: 3.40%
|
||||
- **年化波动率**: 3.11%
|
||||
- **最大回撤**: -4.20%
|
||||
- **夏普比率**: 0.45
|
||||
- **债券类资产**: 84.75%
|
||||
- **商品类资产**: 15.25%
|
||||
|
||||
### 滚动窗口风险平价（优化后）
|
||||
- **总收益率**: 44.78% ✅
|
||||
- **年化收益率**: 3.57% ✅
|
||||
- **年化波动率**: 3.76%
|
||||
- **最大回撤**: -9.48%
|
||||
- **夏普比率**: 0.42
|
||||
- **债券类资产**: 78.58%
|
||||
- **商品类资产**: 21.42% ✅
|
||||
|
||||
### 关键发现
|
||||
1. **收益更高**: 44.78% vs 42.31% ✅
|
||||
2. **避免未来数据**: 使用历史一年数据 ✅
|
||||
3. **商品暴露增加**: 21.42% vs 15.25% ✅
|
||||
4. **风险略高**: 最大回撤-9.48% vs -4.20%
|
||||
5. **夏普比率略低**: 0.42 vs 0.45
|
||||
|
||||
## 💡 投资建议
|
||||
|
||||
### 优点
|
||||
1. **避免未来数据**: 使用历史一年数据计算波动率，避免数据泄露
|
||||
2. **动态调整**: 每月调仓，适应市场变化
|
||||
3. **收益更高**: 总收益率44.78%，略高于静态方法
|
||||
4. **商品暴露增加**: 提供更好的分散化
|
||||
|
||||
### 缺点
|
||||
1. **风险更高**: 最大回撤-9.48%，高于静态方法
|
||||
2. **波动率更高**: 年化波动率3.76%，高于静态方法
|
||||
3. **夏普比率略低**: 0.42 vs 0.45
|
||||
|
||||
### 改进建议
|
||||
1. **优化调仓频率**: 可以尝试季度调仓或半年调仓
|
||||
2. **调整窗口大小**: 可以尝试6个月或18个月的滚动窗口
|
||||
3. **加入止损机制**: 控制最大回撤
|
||||
4. **考虑交易成本**: 每月调仓可能产生交易成本
|
||||
|
||||
## 🎉 项目成果
|
||||
|
||||
### 技能功能
|
||||
1. ✅ 支持CSV数据导入
|
||||
2. ✅ 自动处理百分比单位转换
|
||||
3. ✅ 滚动窗口风险平价分析
|
||||
4. ✅ 避免未来数据
|
||||
5. ✅ 动态调仓
|
||||
6. ✅ 回测验证
|
||||
7. ✅ 详细分析报告生成
|
||||
8. ✅ 多维度可视化图表
|
||||
9. ✅ 投资建议和改进方向
|
||||
|
||||
### 分析能力
|
||||
1. ✅ 资产配置分析
|
||||
2. ✅ 风险评估
|
||||
3. ✅ 收益分析
|
||||
4. ✅ 相关性分析
|
||||
5. ✅ 资产类型分析
|
||||
6. ✅ 滚动权重变化分析
|
||||
|
||||
### 输出能力
|
||||
1. ✅ 文字报告
|
||||
2. ✅ JSON数据
|
||||
3. ✅ PNG图表
|
||||
4. ✅ 交互式分析
|
||||
|
||||
## 📊 项目文件结构
|
||||
|
||||
```
|
||||
financial-analysis-skill/
|
||||
├── optimized_risk_parity_skill.py # 优化的风险平价技能
|
||||
├── optimized_main.py # 优化的主脚本
|
||||
├── SKILL.md # 技能说明
|
||||
├── 优化使用指南.md # 使用指南
|
||||
├── 优化总结.md # 优化总结文档
|
||||
├── 最终优化总结.md # 本总结文档
|
||||
└── ... (其他文件)
|
||||
|
||||
backtest_output/
|
||||
├── rolling_risk_parity_report.txt # 分析报告
|
||||
├── rolling_risk_parity_data.json # 分析数据
|
||||
├── rolling_risk_parity_returns.png # 收益曲线图
|
||||
├── rolling_risk_parity_allocation.png # 资产配置饼图
|
||||
├── rolling_risk_parity_correlation.png # 相关性热力图
|
||||
├── rolling_asset_returns_comparison.png # 资产收益对比图
|
||||
└── rolling_weight_changes.png # 滚动权重变化图
|
||||
```
|
||||
|
||||
## 🎯 下一步计划
|
||||
|
||||
### 短期行动（1-2周）
|
||||
1. ✅ 完成优化的金融分析技能
|
||||
2. ✅ 生成分析报告和图表
|
||||
3. ✅ 更新文档和内存
|
||||
4. ✅ 测试技能功能
|
||||
|
||||
### 中期行动（1-2个月）
|
||||
1. 比较不同调仓频率的效果
|
||||
2. 优化滚动窗口大小
|
||||
3. 考虑加入更多资产类别
|
||||
4. 开发交互式分析界面
|
||||
|
||||
### 长期行动（3-6个月）
|
||||
1. 定期更新市场数据
|
||||
2. 监控投资组合表现
|
||||
3. 根据市场变化调整配置
|
||||
4. 扩展技能功能
|
||||
|
||||
## 🎉 项目总结
|
||||
|
||||
优化的金融分析技能成功完成，提供了：
|
||||
|
||||
1. **完整的分析框架**：从数据解析到报告生成的完整流程
|
||||
2. **避免未来数据**：使用滚动窗口计算波动率，避免数据泄露
|
||||
3. **动态调仓**：每月调仓，适应市场变化
|
||||
4. **风险平价策略**：各资产风险贡献均衡
|
||||
5. **全面的评估指标**：收益、风险、相关性等多维度分析
|
||||
6. **可视化的输出**：图表和报告便于理解和决策
|
||||
|
||||
### 项目价值
|
||||
- 为投资者提供中国市场风险平价组合的参考
|
||||
- 展示了风险平价策略在中国市场的应用效果
|
||||
- 提供了可复用的分析工具和方法
|
||||
- 为后续研究和优化奠定基础
|
||||
|
||||
### 技术贡献
|
||||
- 正确处理中国市场数据格式
|
||||
- 实现滚动窗口风险平价权重计算
|
||||
- 生成专业分析报告和图表
|
||||
- 提供完整的分析工具链
|
||||
|
||||
---
|
||||
|
||||
**项目状态**: ✅ 完成
|
||||
**分析时间**: 2026-02-26
|
||||
**数据时间范围**: 2015-03-23 至 2026-02-25
|
||||
**数据点数**: 2656
|
||||
**滚动窗口**: 252个交易日（约1年）
|
||||
**调仓频率**: 每月
|
||||
**总收益率**: 44.78%
|
||||
**夏普比率**: 0.42
|
||||
|
||||
## 📊 最终分析结果
|
||||
|
||||
### 滚动窗口风险平价组合（优化后）
|
||||
|
||||
**投资组合配置（平均权重）**:
|
||||
- 五年期国债: 46.04% (波动率: 2.46%)
|
||||
- 十年期国债: 32.55% (波动率: 3.61%)
|
||||
- 沪铜: 9.33% (波动率: 17.87%)
|
||||
- 沪金: 12.09% (波动率: 14.88%)
|
||||
|
||||
**资产类型分布**:
|
||||
- 债券类资产: 78.58%
|
||||
- 商品类资产: 21.42%
|
||||
|
||||
**绩效指标**:
|
||||
- 总收益率: 44.78%
|
||||
- 年化收益率: 3.57%
|
||||
- 年化波动率: 3.76%
|
||||
- 最大回撤: -9.48%
|
||||
- 夏普比率: 0.42
|
||||
|
||||
**优化特点**:
|
||||
- ✅ 避免未来数据（使用expanding窗口）
|
||||
- ✅ 动态调仓（每月调整权重）
|
||||
- ✅ 风险平价（各资产风险贡献相等）
|
||||
- ✅ 回测验证（完整回测流程）
|
||||
|
||||
**输出文件**:
|
||||
- 分析报告: `backtest_output/rolling_risk_parity_report.txt`
|
||||
- 详细数据: `backtest_output/rolling_risk_parity_data.json`
|
||||
- 可视化图表: 5个专业图表
|
||||
|
||||
**文档文件**:
|
||||
- `SKILL.md` - 技能说明
|
||||
- `优化使用指南.md` - 使用指南
|
||||
- `优化总结.md` - 优化总结
|
||||
- `最终优化总结.md` - 本总结
|
||||
|
||||
**项目状态**: ✅ 完成
|
||||
**分析时间**: 2026-02-26
|
||||
**数据时间范围**: 2015-03-23 至 2026-02-25
|
||||
**数据点数**: 2656
|
||||
**滚动窗口**: 252个交易日（约1年）
|
||||
**调仓频率**: 每月
|
||||
**总收益率**: 44.78%
|
||||
**夏普比率**: 0.42
|
||||
|
|
@ -0,0 +1,93 @@
|
|||
---
|
||||
name: ifind-repilot-finance-data-search
|
||||
description: 使用自然语言查询金融数据，支持A股股票、基金、期货等上市品种，覆盖基本资料、财务数据、日频行情信息、持仓信息及各类分析指标等数据。也支持宏观经济数据，包括世界经济数据、全球经济数据、中国经济数据、区域经济数据、行业经济数据、利率走势数据、商品数据和特色数据等。当需要查询上述金融相关数据查询时使用此 skill。
|
||||
metadata: {"openclaw":{"emoji":"🔍︎","requires":{"bins":["python3"]}}}
|
||||
---
|
||||
|
||||
## 概述
|
||||
|
||||
此 skill 通过自然语言查询金融数据。通过调用封装好的 Python 脚本，向同花顺金融数据接口发送请求，返回半结构化的金融数据。
|
||||
|
||||
## 前置要求
|
||||
|
||||
### 环境依赖
|
||||
|
||||
- Python 3.x
|
||||
- 标准库：`json`, `urllib`, `pathlib`
|
||||
- 无需安装第三方库
|
||||
- 注1：若 `python3` 命令不可用（常见于 Windows），请改用 `python`
|
||||
- 注2：`<skill_dir>` 为当前 skill 所在目录的完整路径，请根据当前上下文自动获取。
|
||||
|
||||
### 首次配置
|
||||
|
||||
首次使用 skill 前，需要先配置认证 token。
|
||||
|
||||
1. 提示用户先从飞研平台获取 token。平台地址： https://repilot.51ifind.com/ ，菜单路径：业务管理 -> Skills技能库。
|
||||
2. 用户提供token后，运行以下命令配置 token：
|
||||
```bash
|
||||
python3 <skill_dir>/scripts/fetch_data.py --set-token <your_auth_token>
|
||||
```
|
||||
3. 检查 token 是否已配置：
|
||||
```bash
|
||||
python3 <skill_dir>/scripts/fetch_data.py --check-token
|
||||
```
|
||||
|
||||
注：配置文件位于：`~/.config/ifind-repilot/config.json`
|
||||
|
||||
## 使用方式
|
||||
|
||||
### 基本查询
|
||||
|
||||
```bash
|
||||
python3 <skill_dir>/scripts/fetch_data.py "<自然语言查询语句>"
|
||||
```
|
||||
|
||||
### 查询示例
|
||||
|
||||
| 查询内容 | 命令 |
|
||||
|---------|------|
|
||||
| 同花顺收盘价 | `python3 <skill_dir>/scripts/fetch_data.py "同花顺收盘价"` |
|
||||
| 茅台股价 | `python3 <skill_dir>/scripts/fetch_data.py "贵州茅台当前股价"` |
|
||||
| 苹果公司财报 | `python3 <skill_dir>/scripts/fetch_data.py "苹果公司最近季度营收和利润"` |
|
||||
| 上证指数 | `python3 <skill_dir>/scripts/fetch_data.py "上证指数今日行情"` |
|
||||
| 股票对比 | `python3 <skill_dir>/scripts/fetch_data.py "比亚迪和蔚来股价对比"` |
|
||||
|
||||
### 返回格式
|
||||
|
||||
脚本直接返回查询结果的文本内容，通常是文本与 Markdown 表格格式的组合，示例：
|
||||
|
||||
```
|
||||
提取数据：数据浏览器
|
||||
|证券代码|证券简称|收盘价（元）|
|
||||
|---|---|---|
|
||||
|300033.SZ|同花顺|291.74|
|
||||
|
||||
提取数据：FinQuery
|
||||
|股票代码|股票简称|日期|收盘价:不复权|
|
||||
|---|---|---|---|
|
||||
|300033.SZ|同花顺|20250808|287.78|
|
||||
```
|
||||
|
||||
### 错误处理
|
||||
|
||||
| 错误信息 | 原因 | 解决方法 |
|
||||
|----------|------|----------|
|
||||
| `请先配置 auth_token` | 未设置 token | 提示用户获取并发送token，收到token后，运行 `--set-token` 配置 |
|
||||
| `API HTTP 错误` | 请求失败 | 检查网络或 API 地址 |
|
||||
| `网络错误` | 无法连接 | 检查网络连接 |
|
||||
| `API 返回失败` | 接口返回错误 | 查看具体错误信息 |
|
||||
| `API HTTP 错误: 429` | 触发当日请求限制 | 提醒用户到飞研平台检查用量信息 |
|
||||
| `API HTTP 错误: 401` | 缺失token或者token无效 | 提醒用户检查token是否正确 |
|
||||
| `API HTTP 错误: 403` | 没有权限访问接口 | 没有权限访问接口 |
|
||||
| `暂无结果` | 当前问句没有检索到数据，并非token权限问题 | 尝试生成一个新的自然语言查询语句重新查询 |
|
||||
|
||||
|
||||
## 执行规则
|
||||
|
||||
1. 先判断用户请求或任务是否需要金融数据检索。
|
||||
2. 若需要金融数据检索，根据用户请求内容与思考，生成一个合适的自然语言查询语句作为参数调用脚本。
|
||||
3. 若问题包含时间范围不明确的财务数据请求，默认查询最近已披露季度或最近财年，并在回答中明确说明默认时间范围。
|
||||
4. 若请求返回未检索到数据、或者返回的数据不符合期望，可以尝试生成一个新的自然语言查询语句重新查询。
|
||||
5. 不要编造任何金融数据。若脚本失败或返回空结果，应明确说明失败原因或未检索到数据。
|
||||
6. 将脚本输出中的关键数据提炼为简洁结论；如有表格，优先保留表格。
|
||||
7. 当出现401、403、429错误时，请勿再重试，直接提醒用户skill返回的错误内容。
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"ownerId": "kn7507madaqj21ak5b5fqdgp2582sfa1",
|
||||
"slug": "ifind-repilot-finance-data-search",
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1774514980389
|
||||
}
|
||||
|
|
@ -0,0 +1,176 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
金融数据自然语言取数脚本
|
||||
通过自然语言查询金融数据
|
||||
|
||||
用法:
|
||||
python fetch_data.py "同花顺收盘价"
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
from pathlib import Path
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
|
||||
if sys.platform == "win32":
|
||||
sys.stdout.reconfigure(encoding="utf-8")
|
||||
sys.stderr.reconfigure(encoding="utf-8")
|
||||
|
||||
DEFAULT_BASE_URL = "http://test-repilot.51ifind.com/"
|
||||
SKILL_CODE = "nlp_finance_query"
|
||||
CONFIG_DIR = Path.home() / ".config" / "ifind-repilot"
|
||||
CONFIG_FILE = CONFIG_DIR / "config.json"
|
||||
|
||||
|
||||
def get_config():
|
||||
"""读取配置文件"""
|
||||
if not CONFIG_FILE.exists():
|
||||
return {}
|
||||
try:
|
||||
with open(CONFIG_FILE) as f:
|
||||
return json.load(f)
|
||||
except json.JSONDecodeError:
|
||||
print(f"错误: 配置文件 {CONFIG_FILE} 格式无效，请检查 JSON 格式是否正确", file=sys.stderr)
|
||||
return {}
|
||||
|
||||
|
||||
def get_base_url():
|
||||
"""获取 base URL"""
|
||||
config = get_config()
|
||||
return config.get("base_url", DEFAULT_BASE_URL).rstrip("/")
|
||||
|
||||
|
||||
def get_token():
|
||||
"""获取认证 token"""
|
||||
config = get_config()
|
||||
token = config.get("auth_token")
|
||||
if not token:
|
||||
raise ValueError(f"请先配置 auth_token，参考 {CONFIG_FILE}")
|
||||
return token
|
||||
|
||||
|
||||
def fetch_data(query: str) -> dict:
|
||||
"""
|
||||
查询金融数据
|
||||
|
||||
Args:
|
||||
query: 查询问句
|
||||
|
||||
Returns:
|
||||
API 返回的 JSON 数据
|
||||
|
||||
Raises:
|
||||
ValueError: 参数无效或认证失败
|
||||
RuntimeError: API 调用失败
|
||||
"""
|
||||
if not query or not query.strip():
|
||||
raise ValueError("query 参数不能为空")
|
||||
|
||||
token = get_token()
|
||||
base_url = get_base_url()
|
||||
|
||||
params = {"query": query, "skillCode": SKILL_CODE}
|
||||
query_string = urllib.parse.urlencode(params)
|
||||
url = f"{base_url}/research_pilot_service/openapi/skills/v1/finance/query?{query_string}"
|
||||
|
||||
request = urllib.request.Request(url)
|
||||
request.add_header("Authorization", token)
|
||||
request.add_header("Content-Type", "application/json")
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
result = json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as e:
|
||||
raise RuntimeError(f"API HTTP 错误: {e.code} {e.reason}")
|
||||
except urllib.error.URLError as e:
|
||||
raise RuntimeError(f"网络错误: {e.reason}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise RuntimeError(f"响应 JSON 解析失败: {e}")
|
||||
|
||||
if result.get("code") != 1:
|
||||
error_msg = result.get("msg", "未知错误")
|
||||
sub_msg = result.get("subMsg") or result.get("sub_msg", "")
|
||||
raise RuntimeError(f"API 返回失败: {error_msg} {sub_msg}".strip())
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def check_token():
|
||||
"""检查 token 配置状态"""
|
||||
config = get_config()
|
||||
|
||||
print("Token 检查:")
|
||||
print("-" * 40)
|
||||
|
||||
if config.get("auth_token"):
|
||||
print("√ auth_token 已配置")
|
||||
return True
|
||||
else:
|
||||
print("× auth_token 未配置")
|
||||
print(f"\n请运行: python fetch_data.py --set-token <your_token>")
|
||||
print(f"或手动创建 {CONFIG_FILE}:")
|
||||
print(f' mkdir -p {CONFIG_DIR}')
|
||||
print(f' echo \'{{"auth_token": "your_token_here"}}\' > {CONFIG_FILE}')
|
||||
return False
|
||||
|
||||
|
||||
def set_config(key: str, value: str):
|
||||
"""设置配置项"""
|
||||
CONFIG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
config = get_config()
|
||||
config[key] = value
|
||||
|
||||
with open(CONFIG_FILE, "w") as f:
|
||||
json.dump(config, f, indent=2)
|
||||
|
||||
print(f"已保存到 {CONFIG_FILE}")
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print("用法: python fetch_data.py <查询问句>", file=sys.stderr)
|
||||
print(" python fetch_data.py --check-token", file=sys.stderr)
|
||||
print(" python fetch_data.py --set-token <token>", file=sys.stderr)
|
||||
print(" python fetch_data.py --set-url <url>", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if sys.argv[1] == "--check-token":
|
||||
check_token()
|
||||
sys.exit(0)
|
||||
|
||||
if sys.argv[1] == "--set-token":
|
||||
if len(sys.argv) < 3:
|
||||
print("错误: 请提供 token 值", file=sys.stderr)
|
||||
print("用法: python fetch_data.py --set-token <token>", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
set_config("auth_token", sys.argv[2])
|
||||
sys.exit(0)
|
||||
|
||||
if sys.argv[1] == "--set-url":
|
||||
if len(sys.argv) < 3:
|
||||
print("错误: 请提供 url 值", file=sys.stderr)
|
||||
print("用法: python fetch_data.py --set-url <url>", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
set_config("base_url", sys.argv[2])
|
||||
sys.exit(0)
|
||||
|
||||
query = " ".join(sys.argv[1:])
|
||||
|
||||
try:
|
||||
result = fetch_data(query)
|
||||
data = result.get("data", "")
|
||||
if isinstance(data, str):
|
||||
print(data)
|
||||
else:
|
||||
print(json.dumps(data, ensure_ascii=False, indent=2))
|
||||
except (ValueError, RuntimeError) as e:
|
||||
print(f"错误: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
# JSON Translator 技能
|
||||
|
||||
JSON 文件翻译助手，用于翻译 JSON 文件中的文本内容，特别是 `description` 字段。
|
||||
|
||||
## 文件结构
|
||||
|
||||
```
|
||||
json-translator/
|
||||
├── SKILL.md # 技能说明文档
|
||||
├── README.md # 使用说明（本文件）
|
||||
└── scripts/
|
||||
└── translate_json.py # 翻译脚本
|
||||
```
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 1. 翻译 description 字段到中文
|
||||
|
||||
```bash
|
||||
python scripts/translate_json.py data.json --target-language zh
|
||||
```
|
||||
|
||||
### 2. 翻译多个字段
|
||||
|
||||
```bash
|
||||
python scripts/translate_json.py data.json --target-language en --fields name,description
|
||||
```
|
||||
|
||||
### 3. 指定源语言
|
||||
|
||||
```bash
|
||||
python scripts/translate_json.py data.json --target-language ja --source-language zh
|
||||
```
|
||||
|
||||
## 参数说明
|
||||
|
||||
| 参数 | 必需 | 说明 | 示例 |
|
||||
|------|------|------|------|
|
||||
| 输入文件 | 是 | JSON 文件路径 | `data.json` |
|
||||
| `--target-language` | 是 | 目标语言代码 | `zh`, `en`, `ja`, `ko` |
|
||||
| `--source-language` | 否 | 源语言代码 | `zh`, `en`, `ja`, `ko`, `auto` |
|
||||
| `--fields` | 否 | 要翻译的字段名（逗号分隔） | `name,description` |
|
||||
| `--output` | 否 | 输出文件路径 | `output.json` |
|
||||
|
||||
## 支持的语言
|
||||
|
||||
| 语言代码 | 语言名称 |
|
||||
|---------|---------|
|
||||
| `zh` | 中文（简体） |
|
||||
| `en` | 英文 |
|
||||
| `ja` | 日文 |
|
||||
| `ko` | 韩文 |
|
||||
|
||||
## 使用场景
|
||||
|
||||
- **产品描述翻译**：翻译产品信息中的描述字段
|
||||
- **API 文档翻译**：翻译 API 文档的多语言版本
|
||||
- **配置文件翻译**：翻译配置文件中的说明文本
|
||||
- **数据集翻译**：翻译数据集中的文本字段
|
||||
- **国际化开发**：快速生成 JSON 文件的多语言版本
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. 需要安装 `requests` 库：`pip install requests`
|
||||
2. 确保网络可以访问 MyMemory API
|
||||
3. 建议文件大小不超过 10MB
|
||||
4. 翻译失败时会保留原文并添加错误标记
|
||||
|
||||
## 更多信息
|
||||
|
||||
详细的使用说明和示例请参考 [SKILL.md](./SKILL.md)
|
||||
|
|
@ -0,0 +1,166 @@
|
|||
---
|
||||
name: json-translator
|
||||
description: 翻译JSON文件中的文本内容，特别是description字段。当用户提到需要翻译JSON文件、翻译JSON中的字段内容、翻译description字段，或者需要将JSON文件翻译成其他语言时，使用此技能。这个技能非常适合处理产品描述、API文档、配置文件、数据集等需要多语言翻译的JSON内容。
|
||||
---
|
||||
|
||||
# JSON 文件翻译助手
|
||||
|
||||
翻译 JSON 文件中的指定字段（默认为 `description`），支持多种语言互译，并保持原始 JSON 结构完整。
|
||||
|
||||
## 核心功能
|
||||
|
||||
- **智能字段识别**：自动查找 JSON 中的目标字段（默认 `description`），支持自定义指定多个字段
|
||||
- **多语言支持**：支持中文、英文、日文、韩文四种语言的互译
|
||||
- **真实翻译 API**：集成 MyMemory 和 LibreTranslate 两个在线翻译服务
|
||||
- **进度反馈**：显示翻译进度和当前处理的字段内容
|
||||
- **结构保持**：只翻译目标字段，保持 JSON 结构和其他字段不变
|
||||
- **错误处理**：翻译失败时保留原文并添加标记，不会中断整个翻译过程
|
||||
- **文件导出**：提供翻译后的 JSON 文件下载和结果展示
|
||||
|
||||
## 工作流程
|
||||
|
||||
### 1. 读取用户需求
|
||||
|
||||
先理解用户需要翻译的内容：
|
||||
- JSON 文件的路径
|
||||
- 需要翻译的字段名（默认 `description`，可指定多个逗号分隔）
|
||||
- 源语言和目标语言
|
||||
- 是否需要查看翻译结果或直接下载文件
|
||||
|
||||
### 2. 调用翻译脚本
|
||||
|
||||
使用技能目录中的 `scripts/translate_json.py` 脚本进行翻译：
|
||||
|
||||
```bash
|
||||
python scripts/translate_json.py <输入文件> --target-language <语言代码> [选项]
|
||||
```
|
||||
|
||||
#### 脚本参数
|
||||
|
||||
| 参数 | 说明 | 示例 |
|
||||
|------|------|------|
|
||||
| `--target-language` | 目标语言代码 | `zh`, `en`, `ja`, `ko` |
|
||||
| `--source-language` | 源语言代码（可选，默认 auto） | `zh`, `en`, `ja`, `ko` 或 `auto` |
|
||||
| `--fields` | 要翻译的字段名（逗号分隔，可选） | `name,description` |
|
||||
| `--output` | 输出文件路径（可选） | `output.json` |
|
||||
|
||||
#### 支持的语言代码
|
||||
|
||||
| 语言代码 | 语言名称 |
|
||||
|---------|---------|
|
||||
| `zh` | 中文（简体） |
|
||||
| `en` | 英文 |
|
||||
| `ja` | 日文 |
|
||||
| `ko` | 韩文 |
|
||||
|
||||
### 3. 查看翻译结果
|
||||
|
||||
脚本执行后会显示：
|
||||
- 翻译进度（当前/总数）
|
||||
- 原文和译文预览
|
||||
- 翻译统计信息
|
||||
- 输出文件路径
|
||||
|
||||
### 4. 下载翻译后的文件
|
||||
|
||||
翻译完成后：
|
||||
- 控制台会显示输出文件路径
|
||||
- 用户可以下载该文件查看完整内容
|
||||
- 可以选择性地将结果返回给用户
|
||||
|
||||
## 使用示例
|
||||
|
||||
### 示例 1：翻译 description 字段到中文
|
||||
|
||||
用户说： "帮我翻译这个 JSON 文件的 description 字段到中文"
|
||||
|
||||
```bash
|
||||
python scripts/translate_json.py data.json --target-language zh
|
||||
```
|
||||
|
||||
### 示例 2：翻译多个字段到英文
|
||||
|
||||
用户说： "把这个 JSON 文件的 name 和 description 字段翻译成英文"
|
||||
|
||||
```bash
|
||||
python scripts/translate_json.py data.json --target-language en --fields name,description
|
||||
```
|
||||
|
||||
### 示例 3：指定源语言翻译
|
||||
|
||||
用户说： "将这个文件从中文翻译成日文"
|
||||
|
||||
```bash
|
||||
python scripts/translate_json.py data.json --target-language ja --source-language zh
|
||||
```
|
||||
|
||||
### 示例 4：指定输出文件
|
||||
|
||||
用户说： "翻译这个文件，输出到 translation.json"
|
||||
|
||||
```bash
|
||||
python scripts/translate_json.py data.json --target-language en --output translation.json
|
||||
```
|
||||
|
||||
## 输出说明
|
||||
|
||||
### 控制台输出
|
||||
|
||||
脚本执行时会在控制台显示：
|
||||
- `[当前/总数]` - 翻译进度
|
||||
- 字段路径 - 当前正在翻译的字段位置
|
||||
- 原文预览 - 50个字符
|
||||
- 译文预览 - 50个字符
|
||||
- 完成信息 - 翻译成功/失败统计
|
||||
|
||||
### 文件输出
|
||||
|
||||
- 默认文件名格式：`{原文件名}_translated.json`
|
||||
- 用户也可以通过 `--output` 参数指定输出路径
|
||||
- JSON 文件保持原始缩进格式，使用 UTF-8 编码
|
||||
|
||||
### 错误处理
|
||||
|
||||
脚本会处理以下错误情况：
|
||||
- JSON 文件格式错误 - 显示错误信息并退出
|
||||
- 文件不存在 - 提示文件路径错误
|
||||
- 字段不存在 - 显示警告信息但继续处理
|
||||
- 翻译失败 - 保留原文并添加错误标记
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **API 限流**：脚本会自动在每次翻译之间添加 0.1 秒延迟，避免 API 限流
|
||||
2. **大文件处理**：建议 JSON 文件大小不超过 10MB
|
||||
3. **网络要求**：需要能够访问 MyMemory API (`https://api.mymemory.translated.net`)
|
||||
4. **字段存在性**：如果指定的字段在 JSON 中不存在，会提示缺失字段但继续处理其他字段
|
||||
5. **结构保持**：只翻译目标字段，其他字段和 JSON 结构完全保持不变
|
||||
6. **依赖**：需要安装 `requests` 库（脚本会自动处理，如果未安装会提示）
|
||||
|
||||
## 技术实现
|
||||
|
||||
### 翻译服务
|
||||
|
||||
脚本使用 **MyMemory** 作为主要翻译服务，**LibreTranslate** 作为备用服务：
|
||||
|
||||
- MyMemory：免费在线翻译 API，主要翻译服务
|
||||
- LibreTranslate：开源机器翻译 API，作为备用服务
|
||||
- 自动降级：主服务失败时自动切换到备用服务
|
||||
- 错误处理：API 失败时保留原文并添加标记
|
||||
|
||||
### 字段查找
|
||||
|
||||
脚本使用深度递归算法查找 JSON 中的目标字段：
|
||||
|
||||
- 递归遍历所有对象和数组
|
||||
- 识别指定字段名
|
||||
- 记录字段路径和值
|
||||
- 保持原始 JSON 结构
|
||||
|
||||
### 进度反馈
|
||||
|
||||
脚本提供实时翻译进度：
|
||||
|
||||
- 显示当前翻译的字段编号
|
||||
- 显示字段路径和内容预览
|
||||
- 显示翻译状态（成功/失败）
|
||||
- 显示完成统计信息
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"ownerId": "kn7crktbsf8jfd82wwz80wy2wd8342jb",
|
||||
"slug": "json-translator",
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1774245828529
|
||||
}
|
||||
|
|
@ -0,0 +1,317 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
JSON 文件翻译脚本
|
||||
|
||||
功能：
|
||||
- 读取 JSON 文件
|
||||
- 翻译指定字段的内容
|
||||
- 保存翻译后的文件
|
||||
|
||||
使用方法：
|
||||
python translate_json.py <输入文件> --target-language <语言代码> [--fields <字段名>] [--source-language <语言代码>] [--output <输出文件>]
|
||||
|
||||
示例：
|
||||
python translate_json.py data.json --target-language zh
|
||||
python translate_json.py data.json --target-language en --fields name,description
|
||||
"""
|
||||
|
||||
import json
|
||||
import requests
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any, Tuple, Optional
|
||||
|
||||
|
||||
# 语言配置
|
||||
LANGUAGES = {
|
||||
'zh': 'zh-cn',
|
||||
'en': 'en',
|
||||
'ja': 'ja',
|
||||
'ko': 'ko'
|
||||
}
|
||||
|
||||
SOURCE_LANGUAGES = {
|
||||
'auto': 'auto',
|
||||
'zh': 'zh-cn',
|
||||
'en': 'en',
|
||||
'ja': 'ja',
|
||||
'ko': 'ko'
|
||||
}
|
||||
|
||||
|
||||
def translate_text(text: str, from_lang: str = 'auto', to_lang: str = 'zh') -> str:
|
||||
"""
|
||||
翻译文本
|
||||
:param text: 要翻译的文本
|
||||
:param from_lang: 源语言代码 ('auto' 或语言代码)
|
||||
:param to_lang: 目标语言代码
|
||||
:return: 翻译后的文本
|
||||
"""
|
||||
if not text or not text.strip():
|
||||
return text
|
||||
|
||||
try:
|
||||
# 使用 MyMemory API 进行翻译
|
||||
url = f"https://api.mymemory.translated.net/get"
|
||||
params = {
|
||||
'q': text,
|
||||
'langpair': f"{SOURCE_LANGUAGES.get(from_lang, 'auto')}|{LANGUAGES[to_lang]}"
|
||||
}
|
||||
|
||||
response = requests.get(url, params=params, timeout=10)
|
||||
data = response.json()
|
||||
|
||||
if 'responseData' in data and 'translatedText' in data['responseData']:
|
||||
return data['responseData']['translatedText']
|
||||
else:
|
||||
# API失败，返回带标记的原文
|
||||
return _get_error_marker(to_lang) + text
|
||||
|
||||
except Exception as e:
|
||||
print(f"翻译错误: {e}")
|
||||
return _get_error_marker(to_lang) + text
|
||||
|
||||
|
||||
def _get_error_marker(language: str) -> str:
|
||||
"""获取错误标记"""
|
||||
markers = {
|
||||
'zh': '【翻译失败-保持原文】',
|
||||
'en': '【Translation Failed-Kept Original】',
|
||||
'ja': '【翻訳失敗-原文保持】',
|
||||
'ko': '【번역 실패-원문 유지】'
|
||||
}
|
||||
return markers.get(language, '【翻译失败】')
|
||||
|
||||
|
||||
def find_all_fields(obj: Any, path: str = '', fields: set = None) -> set:
|
||||
"""
|
||||
递归查找JSON中所有字段名
|
||||
:param obj: JSON对象
|
||||
:param path: 当前路径
|
||||
:param fields: 字段集合
|
||||
:return: 所有字段名集合
|
||||
"""
|
||||
if fields is None:
|
||||
fields = set()
|
||||
|
||||
if isinstance(obj, dict):
|
||||
for key, value in obj.items():
|
||||
current_path = f"{path}.{key}" if path else key
|
||||
fields.add(current_path)
|
||||
|
||||
# 如果值是对象或数组，继续递归
|
||||
if isinstance(value, (dict, list)):
|
||||
find_all_fields(value, current_path, fields)
|
||||
|
||||
return fields
|
||||
|
||||
|
||||
def find_fields_by_name(obj: Any, target_fields: List[str], path: str = '') -> List[Dict[str, Any]]:
|
||||
"""
|
||||
递归查找指定字段的位置
|
||||
:param obj: JSON对象
|
||||
:param target_fields: 目标字段名列表
|
||||
:param path: 当前路径
|
||||
:return: 字段信息列表
|
||||
"""
|
||||
results: List[Dict[str, Any]] = []
|
||||
|
||||
if isinstance(obj, dict):
|
||||
for key, value in obj.items():
|
||||
current_path = f"{path}.{key}" if path else key
|
||||
|
||||
if key in target_fields and isinstance(value, str):
|
||||
results.append({
|
||||
'path': current_path,
|
||||
'parent': obj,
|
||||
'key': key,
|
||||
'value': value
|
||||
})
|
||||
|
||||
if isinstance(value, (dict, list)):
|
||||
results.extend(find_fields_by_name(value, target_fields, current_path))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def read_json_file(file_path: str) -> Tuple[Optional[Dict[str, Any]], Optional[str]]:
|
||||
"""
|
||||
读取JSON文件
|
||||
:param file_path: 文件路径
|
||||
:return: (数据, 错误信息)
|
||||
"""
|
||||
try:
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
return data, None
|
||||
except json.JSONDecodeError as e:
|
||||
return None, f"JSON格式错误: {e}"
|
||||
except Exception as e:
|
||||
return None, f"读取文件错误: {e}"
|
||||
|
||||
|
||||
def save_json_file(data: Dict[str, Any], output_path: str) -> bool:
|
||||
"""
|
||||
保存JSON文件
|
||||
:param data: JSON数据
|
||||
:param output_path: 输出文件路径
|
||||
:return: 是否成功
|
||||
"""
|
||||
try:
|
||||
with open(output_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(data, f, ensure_ascii=False, indent=2)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"保存文件错误: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def translate_json(
|
||||
input_file: str,
|
||||
target_language: str,
|
||||
source_language: str = 'auto',
|
||||
target_fields: List[str] = None,
|
||||
output_file: str = None
|
||||
) -> Tuple[bool, int, List[str], Optional[Dict[str, Any]]]:
|
||||
"""
|
||||
翻译 JSON 文件中的指定字段
|
||||
:param input_file: 输入文件路径
|
||||
:param target_language: 目标语言代码
|
||||
:param source_language: 源语言代码
|
||||
:param target_fields: 目标字段名列表
|
||||
:param output_file: 输出文件路径
|
||||
:return: (成功状态, 翻译数量, 缺失字段列表, 翻译后的数据)
|
||||
"""
|
||||
# 解析字段列表
|
||||
if target_fields is None or not target_fields:
|
||||
target_fields = ['description']
|
||||
|
||||
# 读取文件
|
||||
data, error = read_json_file(input_file)
|
||||
if error:
|
||||
return False, 0, [], None
|
||||
|
||||
# 查找所有存在的字段
|
||||
all_fields = find_all_fields(data)
|
||||
missing_fields = [f for f in target_fields if f not in all_fields]
|
||||
existing_fields = [f for f in target_fields if f in all_fields]
|
||||
|
||||
if not existing_fields:
|
||||
print(f"错误: 未找到指定的字段: {target_fields}")
|
||||
return False, 0, missing_fields, data
|
||||
|
||||
# 查找所有目标字段的位置
|
||||
field_locations = find_fields_by_name(data, existing_fields)
|
||||
|
||||
if not field_locations:
|
||||
print(f"错误: 指定字段中没有可翻译的字符串内容")
|
||||
return False, 0, missing_fields, data
|
||||
|
||||
# 执行翻译
|
||||
total_count = len(field_locations)
|
||||
translated_count = 0
|
||||
|
||||
print(f"\n开始翻译，共 {total_count} 个字段...")
|
||||
print(f"目标语言: {LANGUAGES.get(target_language, target_language)}")
|
||||
print(f"源语言: {source_language if source_language != 'auto' else '自动检测'}")
|
||||
print(f"字段: {', '.join(existing_fields)}")
|
||||
print()
|
||||
|
||||
for i, field_info in enumerate(field_locations, 1):
|
||||
field_name = field_info['key']
|
||||
field_path = field_info['path']
|
||||
|
||||
try:
|
||||
print(f"[{i}/{total_count}] 翻译字段: {field_name} ({field_path})")
|
||||
print(f" 原文: {field_info['value'][:50]}{'...' if len(field_info['value']) > 50 else ''}")
|
||||
|
||||
# 翻译文本
|
||||
translated = translate_text(field_info['value'], source_language, target_language)
|
||||
|
||||
# 更新字段值
|
||||
field_info['parent'][field_info['key']] = translated
|
||||
translated_count += 1
|
||||
|
||||
print(f" 译文: {translated[:50]}{'...' if len(translated) > 50 else ''}")
|
||||
|
||||
# 添加延迟避免 API 限流
|
||||
if i < total_count:
|
||||
import time
|
||||
time.sleep(0.1)
|
||||
|
||||
except Exception as e:
|
||||
print(f"翻译失败: {field_name} - {e}")
|
||||
|
||||
print()
|
||||
print(f"翻译完成！共翻译 {translated_count}/{total_count} 个字段")
|
||||
|
||||
# 生成输出文件名
|
||||
if output_file is None:
|
||||
input_path = Path(input_file)
|
||||
output_file = str(input_path.parent / f"{input_path.stem}_translated.json")
|
||||
|
||||
# 保存文件
|
||||
if save_json_file(data, output_file):
|
||||
print(f"已保存到: {output_file}")
|
||||
|
||||
return True, translated_count, missing_fields, data
|
||||
|
||||
|
||||
def main():
|
||||
"""命令行入口"""
|
||||
if len(sys.argv) < 3:
|
||||
print(__doc__)
|
||||
print("\n使用示例:")
|
||||
print(" python translate_json.py data.json --target-language zh")
|
||||
print(" python translate_json.py data.json --target-language en --fields name,description")
|
||||
print(" python translate_json.py data.json --target-language ja --source-language zh --output output.json")
|
||||
sys.exit(1)
|
||||
|
||||
# 解析参数
|
||||
input_file = sys.argv[1]
|
||||
target_language = None
|
||||
source_language = 'auto'
|
||||
target_fields = ['description']
|
||||
output_file = None
|
||||
|
||||
i = 2
|
||||
while i < len(sys.argv):
|
||||
arg = sys.argv[i]
|
||||
|
||||
if arg == '--target-language' and i + 1 < len(sys.argv):
|
||||
target_language = sys.argv[i + 1]
|
||||
i += 2
|
||||
elif arg == '--source-language' and i + 1 < len(sys.argv):
|
||||
source_language = sys.argv[i + 1]
|
||||
i += 2
|
||||
elif arg == '--fields' and i + 1 < len(sys.argv):
|
||||
target_fields = sys.argv[i + 1].split(',')
|
||||
i += 2
|
||||
elif arg == '--output' and i + 1 < len(sys.argv):
|
||||
output_file = sys.argv[i + 1]
|
||||
i += 2
|
||||
else:
|
||||
i += 1
|
||||
|
||||
# 验证目标语言
|
||||
if target_language not in LANGUAGES:
|
||||
print(f"错误: 不支持的目标语言。可用: {', '.join(LANGUAGES.keys())}")
|
||||
sys.exit(1)
|
||||
|
||||
# 执行翻译
|
||||
success, count, missing, data = translate_json(
|
||||
input_file=input_file,
|
||||
target_language=target_language,
|
||||
source_language=source_language,
|
||||
target_fields=target_fields,
|
||||
output_file=output_file
|
||||
)
|
||||
|
||||
if missing:
|
||||
print(f"\n警告: 以下字段在文件中不存在: {', '.join(missing)}")
|
||||
|
||||
sys.exit(0 if success else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
# jy-customer-requirement-analysis
|
||||
|
||||
金融投顾智能助理技能 - 基于客户沟通素材生成标准化投资需求分析报告。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
jy-customer-requirement-analysis/
|
||||
├── README.md # 本文件
|
||||
└── jy-customer-requirement-analysis/
|
||||
├── SKILL.md # 技能主文档
|
||||
└── references/
|
||||
└── output-examples.md # 输出示例与格式规范
|
||||
```
|
||||
|
||||
## 前置要求
|
||||
|
||||
- Node.js 和 npm
|
||||
- mcporter CLI
|
||||
- JY_API_KEY（向恒生聚源申请：datamap@gildata.com）
|
||||
|
||||
## 安装配置
|
||||
|
||||
1. 安装 mcporter：`npm install -g mcporter`
|
||||
2. 配置 MCP 服务：
|
||||
```bash
|
||||
mcporter config add gildata-tool --url "https://api.gildata.com/mcp-servers/aidata-assistant-srv-tool?token=你的 JY_API_KEY"
|
||||
mcporter config add gildata-api --url "https://api.gildata.com/mcp-servers/aidata-assistant-srv-api?token=你的 JY_API_KEY"
|
||||
```
|
||||
3. 重启 OpenClaw：`openclaw gateway restart`
|
||||
|
||||
## 使用方式
|
||||
|
||||
提供客户沟通素材，技能自动输出七步标准化分析报告：
|
||||
1. 客户需求分析
|
||||
2. 宏观环境分析与配置调整建议
|
||||
3. 投资需求可行性评估
|
||||
4. 解决方案建议
|
||||
5. 产品池匹配分析
|
||||
6. 再平衡与后续关注
|
||||
7. 潜在需求挖掘与引导话术
|
||||
|
||||
## 触发词
|
||||
|
||||
- "分析客户需求"
|
||||
- "投顾分析"
|
||||
- "客户画像"
|
||||
- "理财方案"
|
||||
- "投资需求分析"
|
||||
|
||||
## 详细文档
|
||||
|
||||
- 完整使用说明：`jy-customer-requirement-analysis/SKILL.md`
|
||||
- 输出示例：`jy-customer-requirement-analysis/references/output-examples.md`
|
||||
|
|
@ -0,0 +1,279 @@
|
|||
---
|
||||
name: jy-customer-requirement-analysis
|
||||
description: 金融投顾智能助理技能。基于客户沟通素材，帮助理财师/投顾经理快速精准理解客户投资需求，输出标准化分析报告（需求痛点、可行性评估、解决方案、产品匹配、潜在需求挖掘）。支持 PDF 导出和 HTML 可视化。使用场景：当用户需要分析客户投资需求、生成投顾分析报告、进行客户画像分析、制定理财方案或匹配金融产品时触发。Financial advisory assistant skill for analyzing customer investment needs and generating standardized reports. Supports PDF export and HTML visualization. Triggered for customer analysis, advisory reports, financial planning, and product matching.
|
||||
metadata:
|
||||
openclaw:
|
||||
requires:
|
||||
bins: ["node", "npm", "mcporter"]
|
||||
install:
|
||||
- id: install-mcporter
|
||||
kind: node
|
||||
package: mcporter
|
||||
label: Install mcporter via npm
|
||||
---
|
||||
|
||||
# 客户需求分析 - 金融投顾智能助理
|
||||
|
||||
基于客户沟通素材，输出标准化七步分析报告。详细示例见 `references/output-examples.md`。
|
||||
|
||||
## 功能范围
|
||||
|
||||
- 客户需求痛点分析、宏观环境分析与配置建议
|
||||
- 投资需求可行性评估、解决方案建议
|
||||
- 产品池匹配分析（基金、债券、黄金、QDII 等）
|
||||
- 再平衡与后续关注建议、潜在需求挖掘与引导话术
|
||||
- 支持 Markdown、HTML、PDF 多种输出格式
|
||||
|
||||
## 查询建议
|
||||
|
||||
**查询要素：** 客户基本信息（年龄、职业、收入、家庭）、投资历史、风险测评结果、资金用途和时间规划。
|
||||
|
||||
**触发词：** "分析客户需求"、"投顾分析"、"客户画像"、"理财方案"、"投资需求分析"
|
||||
|
||||
**查询示例：**
|
||||
```
|
||||
客户王先生，45 岁，企业高管，年收入 150 万，已婚有一子（12 岁）。
|
||||
现有资产：房产 2 套，存款 200 万，股票 50 万。
|
||||
对话记录："最近市场波动太大，我有点担心。希望能有稳定收益。"
|
||||
风险测评：平衡型
|
||||
```
|
||||
|
||||
## 环境检查与配置
|
||||
|
||||
**每次使用本技能前，必须先检查 mcporter 安装和 MCP 服务配置状态！**
|
||||
|
||||
### 步骤 1：检查 mcporter 是否安装
|
||||
|
||||
```bash
|
||||
mcporter --version
|
||||
```
|
||||
|
||||
**如未安装：**
|
||||
```bash
|
||||
npm install -g mcporter
|
||||
mcporter --version
|
||||
```
|
||||
|
||||
### 步骤 2：检查 MCP 服务配置
|
||||
|
||||
```bash
|
||||
mcporter list
|
||||
```
|
||||
|
||||
**预期输出：** `jy-financedata-tool`（5 个工具：FundMultipleFactorFilter, StockMultipleFactorFilter, FinQuery, FinancialResearchReport, MacroIndustryData）和 `jy-financedata-api`（252 个接口）
|
||||
|
||||
**如未配置，获取 JY_API_KEY：**
|
||||
|
||||
向恒生聚源申请：datamap@gildata.com
|
||||
|
||||
**邮件标题：** 数据地图 KEY 申请-XX 公司 - 申请人姓名
|
||||
|
||||
**正文模板：** 姓名、手机号、公司全称、部门、岗位、MCP_KEY 申请用途、Skill 申请列表、是否需要 Skill 安装包。
|
||||
|
||||
**无 JY_API_KEY 无法使用，必须先申请！**
|
||||
|
||||
### 步骤 3：配置 MCP 服务
|
||||
|
||||
```bash
|
||||
# 配置 jy-financedata-tool 服务（5 个工具）
|
||||
mcporter config add jy-financedata-tool --url "https://api.gildata.com/mcp-servers/aidata-assistant-srv-tool?token=你的 JY_API_KEY"
|
||||
|
||||
# 配置 jy-financedata-api 服务（252 个接口）
|
||||
mcporter config add jy-financedata-api --url "https://api.gildata.com/mcp-servers/aidata-assistant-srv-api?token=你的 JY_API_KEY"
|
||||
|
||||
# 验证配置
|
||||
mcporter list
|
||||
```
|
||||
|
||||
### 步骤 4：使用方式
|
||||
|
||||
```bash
|
||||
# 所有工具入参均为 query
|
||||
# jy-financedata-tool 包含 5 个工具：FundMultipleFactorFilter, StockMultipleFactorFilter, FinQuery, FinancialResearchReport, MacroIndustryData
|
||||
# 其他工具都在 jy-financedata-api 中
|
||||
mcporter call jy-financedata-tool.工具名 query="查询内容"
|
||||
mcporter call jy-financedata-api.工具名 query="查询内容"
|
||||
|
||||
# 示例（jy-financedata-tool）
|
||||
mcporter call jy-financedata-tool.MacroIndustryData query="中国最新 GDP 增速和 CPI 数据"
|
||||
mcporter call jy-financedata-tool.SmartFundSelection query="稳健型债券基金 年化收益 4% 以上"
|
||||
mcporter call jy-financedata-tool.FinQuery query="沪深 300 指数当前点位和近期走势"
|
||||
mcporter call jy-financedata-tool.FinancialResearchReport query="固收 + 策略配置建议"
|
||||
mcporter call jy-financedata-tool.SmartStockSelection query="市盈率低于 20 的科技股"
|
||||
|
||||
# 示例（jy-financedata-api - 其他 252 个接口）
|
||||
mcporter call jy-financedata-api.FinQuery query="基金代码 012348 基金简称"
|
||||
```
|
||||
|
||||
### 步骤 5：在 OpenClaw 中启用 mcporter
|
||||
|
||||
**配置文件路径：**
|
||||
- mcporter: `C:\Users\你的用户名\.mcporter\mcporter.json`
|
||||
- openclaw: `C:\Users\你的用户名\.openclaw\openclaw.json`
|
||||
|
||||
**编辑 openclaw.json：**
|
||||
```json
|
||||
{
|
||||
"skills": {
|
||||
"entries": {
|
||||
"mcporter": {
|
||||
"enabled": true,
|
||||
"env": {
|
||||
"MCPORTER_CONFIG": "C:\\Users\\你的用户名\\.mcporter\\mcporter.json"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**重启 OpenClaw：** `openclaw gateway restart`
|
||||
|
||||
## 核心工作流程
|
||||
|
||||
### 步骤 1：接收并解析客户输入
|
||||
|
||||
提取关键信息：客户基本信息、资产负债、投资目标、风险偏好、资金用途。信息不足时询问用户。
|
||||
|
||||
### 步骤 2：并发查询宏观数据
|
||||
|
||||
```bash
|
||||
mcporter call jy-financedata-tool.MacroIndustryData query="中国最新 GDP 增速 CPI PMI"
|
||||
mcporter call jy-financedata-tool.FinQuery query="沪深 300 上证指数 创业板指当前点位"
|
||||
mcporter call jy-financedata-tool.FinQuery query="10 年期国债收益率 人民币兑美元汇率"
|
||||
mcporter call jy-financedata-tool.FinQuery query="COMEX 黄金 布伦特原油当前价格"
|
||||
mcporter call jy-financedata-tool.FinQuery query="VIX 恐慌指数当前值"
|
||||
```
|
||||
|
||||
### 步骤 3：查询研报观点
|
||||
|
||||
```bash
|
||||
mcporter call jy-financedata-tool.FinancialResearchReport query="当前市场环境下资产配置建议"
|
||||
```
|
||||
|
||||
### 步骤 4：根据风险等级筛选产品
|
||||
|
||||
```bash
|
||||
mcporter call jy-financedata-tool.SmartFundSelection query="平衡型混合基金 近 3 年收益排名前 20 规模大于 10 亿"
|
||||
mcporter call jy-financedata-tool.SmartFundSelection query="稳健型债券基金 近 1 年收益大于 3%"
|
||||
mcporter call jy-financedata-tool.SmartFundSelection query="成长型股票基金 近 3 年收益排名前 20"
|
||||
```
|
||||
|
||||
### 步骤 5：查询产品详情
|
||||
|
||||
```bash
|
||||
mcporter call jy-financedata-tool.FinQuery query="基金代码 近 1 年收益 近 3 年收益 阶段业绩表现"
|
||||
mcporter call jy-financedata-tool.FinQuery query="基金代码 基金规模 资产规模"
|
||||
mcporter call jy-financedata-tool.FinQuery query="基金代码 基金经理 申购状态 基金公司名称"
|
||||
```
|
||||
|
||||
### 步骤 6：生成七步分析报告
|
||||
|
||||
1. 客户需求分析
|
||||
2. 宏观环境分析与配置调整建议
|
||||
3. 投资需求可行性评估
|
||||
4. 解决方案建议
|
||||
5. 产品池匹配分析
|
||||
6. 再平衡与后续关注
|
||||
7. 潜在需求挖掘与引导话术
|
||||
|
||||
### 步骤 7：合规检查与免责声明
|
||||
|
||||
检查禁用词语，标注数据来源和查询日期，添加免责声明。
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 工具调用命令
|
||||
|
||||
**jy-financedata-tool（5 个工具）：**
|
||||
|
||||
| 工具 | 用途 | 调用示例 |
|
||||
|------|------|----------|
|
||||
| `FinQuery` | 金融数据查询 | `mcporter call jy-financedata-tool.FinQuery query="沪深 300 指数走势"` |
|
||||
| `MacroIndustryData` | 宏观经济数据 | `mcporter call jy-financedata-tool.MacroIndustryData query="最新 GDP 增速 CPI"` |
|
||||
| `FinancialResearchReport` | 券商研报 | `mcporter call jy-financedata-tool.FinancialResearchReport query="固收 + 策略配置建议"` |
|
||||
| `FundMultipleFactorFilter` | 筛选基金 | `mcporter call jy-financedata-tool.SmartFundSelection query="稳健型债券基金 年化收益 4% 以上"` |
|
||||
| `StockMultipleFactorFilter` | 选股 | `mcporter call jy-financedata-tool.SmartStockSelection query="市盈率低于 20 的科技股"` |
|
||||
|
||||
**jy-financedata-api（252 个接口）：** 其他所有金融数据接口
|
||||
|
||||
### 数据完整性要求
|
||||
|
||||
- ❌ 禁止出现"待查询"字样
|
||||
- ✅ 所有数据必须来自聚源 MCP 实时查询
|
||||
- ✅ 查询不到的数据标注"--"并说明原因
|
||||
- ✅ 每个产品标注数据来源和查询日期
|
||||
|
||||
### 产品准入标准
|
||||
|
||||
| 检查项 | 标准 | 处理方式 |
|
||||
|--------|------|----------|
|
||||
| 申购状态 | 开放申购 | 暂停申购标注"⚠️"并提供替代 |
|
||||
| 存续规模 | ≥5000 万元 | <5000 万标注"⚠️ 规模过小" |
|
||||
| 成立年限 | 建议≥1 年 | <1 年标注"⚠️ 新基金" |
|
||||
| 异常收益 | 纯债>8% 或固收+>15% | 检查大额赎回，标注"⚠️ 收益失真" |
|
||||
|
||||
详细规则见 `references/output-examples.md`。
|
||||
|
||||
## 风险等级映射
|
||||
|
||||
| 等级 | 权益上限 | 适合产品 |
|
||||
|------|---------|----------|
|
||||
| 保守型 | 0% | 货币基金、国债 |
|
||||
| 稳健型 | 20% | 纯债基金、固收 + |
|
||||
| 平衡型 | 50% | 混合基金、指数基金 |
|
||||
| 成长型 | 70% | 股票基金、行业主题 |
|
||||
| 进取型 | 100% | 股票基金、QDII |
|
||||
|
||||
## 输出结构
|
||||
|
||||
### 一、客户需求分析
|
||||
精准提炼核心痛点，语言精炼、观点鲜明。
|
||||
|
||||
### 二、宏观环境分析与配置调整建议
|
||||
基于聚源 MCP 数据，给出 2-3 条配置建议。
|
||||
|
||||
### 三、投资需求可行性评估
|
||||
评估可行性，给出判断（高/中/低）。
|
||||
|
||||
### 四、解决方案建议
|
||||
提出针对性方案，明确产品类型和配置策略。
|
||||
|
||||
### 五、产品池匹配分析
|
||||
从聚源基金库匹配推荐，每类 2-4 个产品，数据实时查询。
|
||||
|
||||
### 六、再平衡与后续关注
|
||||
提供动态平衡建议和关注指标。
|
||||
|
||||
### 七、潜在需求挖掘与引导话术
|
||||
挖掘潜在需求，提供专业引导话术。
|
||||
|
||||
## 限制
|
||||
|
||||
### 数据源限制
|
||||
- ✅ 允许：`mcporter call jy-financedata-tool.*` 或 `jy-financedata-api.*` 查询的数据
|
||||
- ❌ 禁止：编造数据、引用其他平台（Wind、同花顺、东方财富等）
|
||||
|
||||
### 输出要求
|
||||
- 不承诺具体收益，使用"预期"、"历史"等限定词
|
||||
- 必须包含风险提示
|
||||
- 推荐产品时检查申购状态和规模
|
||||
- 风险与需求不匹配时需明确指出
|
||||
|
||||
## 免责声明模板
|
||||
|
||||
```
|
||||
---
|
||||
⚠️ 免责声明
|
||||
|
||||
1. 本报告基于客户提供的信息和聚源金融数据平台数据生成，仅供参考，不构成投资建议。
|
||||
2. 基金过往业绩不代表未来表现，市场有风险，投资需谨慎。
|
||||
3. 产品推荐基于当前市场环境和产品状态，实际投资前请再次确认产品最新信息。
|
||||
4. 投资者应根据自身风险承受能力、投资目标和资金流动性需求，独立做出投资决策。
|
||||
5. 理财师在向客户推荐产品时，应确保已完成适当性匹配，并向客户充分揭示产品风险。
|
||||
|
||||
报告生成时间：{当前日期时间}
|
||||
数据来源：恒生聚源金融数据平台
|
||||
---
|
||||
```
|
||||
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"ownerId": "kn71ehvhvjxxqc7yq9knrq6j0n81sxpe",
|
||||
"slug": "jy-customer-requirement-analysis",
|
||||
"version": "1.0.1",
|
||||
"publishedAt": 1776224161786
|
||||
}
|
||||
|
|
@ -0,0 +1,241 @@
|
|||
# 输出示例与格式规范
|
||||
|
||||
## 完整输出示例
|
||||
|
||||
**注意：以下示例中的产品数据仅为格式参考，实际输出时必须使用聚源 MCP 实时查询的真实数据！**
|
||||
|
||||
```markdown
|
||||
## 一、客户需求分析
|
||||
|
||||
客户风险偏好与收益预期存在明显矛盾，投资目标不清晰；
|
||||
客户对市场波动敏感，缺乏应对市场下跌的心理准备和策略；
|
||||
客户子女教育金规划缺失，长期资金准备不足；
|
||||
客户现有资产配置过于集中，缺乏有效的风险分散。
|
||||
|
||||
## 二、宏观环境分析与配置调整建议
|
||||
|
||||
**当前宏观环境分析：**
|
||||
【数据来源：聚源 MacroIndustryData - 查询日期：2026-03-31】
|
||||
- 经济增长：国内 Q4 GDP 增速~5.0%，政策宽松基调延续
|
||||
- 通胀水平：CPI 同比 1.3%，通胀温和回升
|
||||
- 制造业 PMI：49.0，略低于荣枯线
|
||||
- 市场行情：沪深 300 指数~4750 点
|
||||
- 10 年期国债收益率：2.6%
|
||||
- 人民币兑美元：6.85
|
||||
- COMEX 黄金：2050 美元/盎司
|
||||
- VIX 指数：16，市场风险偏好中性
|
||||
|
||||
【数据来源：聚源 FinancialResearchReport - 券商 2026 年 3 月研报】
|
||||
- 市场判断：指数空间受限，结构行情为主
|
||||
- 核心主线：顺周期涨价扩散 + AI 硬件扩散
|
||||
- 风格推荐：中小盘占优，黄金配置价值凸显
|
||||
|
||||
**配置调整建议：**
|
||||
1. 提升权益资产比例至 60-70%，重点关注科技成长风格基金；
|
||||
2. 布局中证 2000、科创 50 等中小盘指数基金 15-20%；
|
||||
3. 黄金 ETF 配置 5-8%，对冲地缘风险；
|
||||
4. 维持 20% 短久期固收产品作为安全垫。
|
||||
|
||||
## 三、投资需求可行性评估
|
||||
|
||||
客户期望年化收益 10% 左右，风险承受能力强（成长型），当前市场环境下具备可行性 — 可行性：中等偏上；
|
||||
理由：根据聚源 FundMultipleFactorFilter 查询结果，科技成长类基金近 3 年累计收益普遍在 80%-180% 之间，但波动率较高，需承担相应风险。
|
||||
|
||||
## 四、解决方案建议
|
||||
|
||||
针对客户换房计划（半年后）和高收益目标，建议采用"长短分离"策略：
|
||||
- **换房资金（150 万）**：配置短久期固收产品，确保本金安全，预期收益 3-4%
|
||||
- **长期资金（150 万）**：配置权益类基金追求高收益，预期收益 15-25%
|
||||
|
||||
## 五、产品池匹配分析
|
||||
|
||||
基于客户成长型风险偏好，从聚源在售基金库中筛选匹配产品：
|
||||
|
||||
【数据来源：聚源 FundMultipleFactorFilter - 查询日期：2026-03-31】
|
||||
|
||||
**大类资产配置比例：**
|
||||
|
||||
| 资产类别 | 配置比例 | 金额（万） | 配置原因 |
|
||||
|----------|----------|------------|----------|
|
||||
| 科技成长基金 | 40% | 60 | 把握 AI 硬件、半导体产业趋势 |
|
||||
| 中小盘指数基金 | 20% | 30 | 中小盘风格占优，弹性大 |
|
||||
| 港股科技 QDII | 10% | 15 | 分散地域风险，港股科技估值修复 |
|
||||
| 周期主题基金 | 15% | 22.5 | 顺周期涨价主线 |
|
||||
| 黄金 ETF | 10% | 15 | 对冲地缘风险和通胀 |
|
||||
| 现金管理 | 5% | 7.5 | 应对突发支出 |
|
||||
|
||||
**具体产品推荐（所有数据来自聚源 MCP 实时查询）：**
|
||||
|
||||
**1. 科技成长基金（40% = 60 万）**
|
||||
- 国泰中证全指通信设备 ETF（515880）：规模 136.4 亿，近 3 年收益 184.35% ✅
|
||||
- 易方达战略新兴产业股票 A（010391）：规模 43.9 亿，近 3 年收益 120.90% ✅
|
||||
- 易方达信息行业精选股票 A（010013）：规模 32.9 亿，近 3 年收益 93.81% ✅
|
||||
|
||||
**2. 中小盘指数基金（20% = 30 万）**
|
||||
- 南方中证 2000ETF 联接 A（017323）：规模 15.2 亿 ✅
|
||||
- 华夏科创 50ETF 联接 A（011612）：规模 42.8 亿 ✅
|
||||
- 南方中证 1000ETF 联接 A（016311）：规模 18.5 亿 ✅
|
||||
|
||||
**3. 黄金 ETF（10% = 15 万）**
|
||||
- 华安黄金易 ETF 联接 A（000216）：规模约 30 亿 ✅
|
||||
- 易方达黄金 ETF 联接 A（000307）：规模约 26 亿 ✅
|
||||
|
||||
**4. 港股科技 QDII（10% = 15 万）**
|
||||
- 易方达恒生科技 ETF 联接 (QDII)A（013308）：港股科技龙头 ✅
|
||||
- 华夏恒生互联网科技业 ETF 发起式联接 (QDII)A（013171）：互联网平台公司 ✅
|
||||
|
||||
## 六、再平衡与后续关注
|
||||
|
||||
**定期再平衡：**
|
||||
- 每季度末复盘组合资产比例
|
||||
- 当单一资产类别偏离目标比例±5% 时执行调仓
|
||||
|
||||
**信号触发调整：**
|
||||
- VIX 恐慌指数＞25 时，将现金及货币工具比例提升至 20%
|
||||
- 10 年期美债收益率突破 4.5% 时，降低长久期债券敞口
|
||||
- 地缘冲突缓解后，逐步加仓权益资产 5%-10%
|
||||
|
||||
**风险控制底线：**
|
||||
- 信用债仅投 AAA 级主体，避免房地产与弱资质城投债
|
||||
- 单一行业配置不超过组合总仓位的 15%
|
||||
- 单一基金产品持仓不超过组合总仓位的 10%
|
||||
|
||||
**后续关注指标：**
|
||||
- 宏观经济：GDP 增速、CPI/PPI、PMI、社融数据
|
||||
- 市场指标：10 年期国债收益率、人民币兑美元汇率、VIX 指数、COMEX 黄金
|
||||
- 政策动态：央行货币政策会议、财政政策动向、金融监管政策变化
|
||||
|
||||
## 七、潜在需求挖掘与引导话术
|
||||
|
||||
**潜在需求 1：养老规划需求**
|
||||
挖掘依据：客户 45 岁，处于职业生涯中后期，对话中未提及退休规划；
|
||||
引导话术："王先生，您现在事业处于黄金期，收入稳定。不知道您有没有考虑过退休后的生活品质规划？"
|
||||
|
||||
**潜在需求 2：资产传承需求**
|
||||
挖掘依据：客户资产规模较大，有子女，未提及传承安排；
|
||||
引导话术："您现在的资产积累已经相当可观了。关于这些资产未来如何传承给孩子，您有没有做过一些规划？"
|
||||
|
||||
**潜在需求 3：企业高管专属理财需求**
|
||||
挖掘依据：客户为企业高管，可能存在股权激励、期权行权等场景；
|
||||
引导话术："作为企业高管，您可能会涉及到股权激励或期权行权的情况。这部分资产的税务规划比较特殊，我们有专门针对高管客户的综合服务方案。"
|
||||
|
||||
---
|
||||
⚠️ 免责声明
|
||||
|
||||
1. 本报告基于客户提供的信息和聚源金融数据平台数据生成，仅供参考，不构成投资建议。
|
||||
2. 基金过往业绩不代表未来表现，市场有风险，投资需谨慎。
|
||||
3. 产品推荐基于当前市场环境和产品状态，实际投资前请再次确认产品最新信息。
|
||||
4. 投资者应根据自身风险承受能力、投资目标和资金流动性需求，独立做出投资决策。
|
||||
5. 理财师在向客户推荐产品时，应确保已完成适当性匹配，并向客户充分揭示产品风险。
|
||||
|
||||
报告生成时间：2026-03-31 10:30
|
||||
数据来源：恒生聚源金融数据平台
|
||||
---
|
||||
```
|
||||
|
||||
## 产品筛选规则
|
||||
|
||||
### 数据准确性要求
|
||||
|
||||
**所有产品数据必须从聚源 MCP 实时查询，禁止使用示例数据或历史记忆数据！**
|
||||
|
||||
| 数据类型 | 查询工具 | 必须字段 |
|
||||
|----------|----------|----------|
|
||||
| 基金规模、收益率 | `FinQuery` 或 `FundMultipleFactorFilter` | 规模 (元)、近 1 年收益、近 3 年收益、基金经理 |
|
||||
| 申购状态 | `FinQuery` | 申购状态、赎回状态 |
|
||||
| 宏观经济数据 | `MacroIndustryData` | GDP、CPI、PMI、社融等 |
|
||||
| 市场行情 | `FinQuery` | 指数点位、涨跌幅、成交量 |
|
||||
| 研报观点 | `FinancialResearchReport` | 机构、作者、发布时间、核心观点 |
|
||||
|
||||
### 基金产品准入标准
|
||||
|
||||
| 检查项 | 标准 | 处理方式 |
|
||||
|--------|------|----------|
|
||||
| 申购状态 | 必须为"开放申购" | 如暂停申购，标注"⚠️ 暂停申购"并提供替代产品 |
|
||||
| 存续规模 | ≥5000 万元 | 如<5000 万，标注"⚠️ 规模过小（XXX 万）"并提供替代产品 |
|
||||
| 成立年限 | 建议≥1 年 | 如<1 年，标注"⚠️ 新基金（成立 X 个月）" |
|
||||
| 数据准确性 | 必须来自聚源 MCP 实时查询 | 如无法查询到数据，不推荐该产品 |
|
||||
|
||||
### 固收基金异常收益检测
|
||||
|
||||
**检测规则：**
|
||||
- 纯债基金年化收益 >8% → 触发预警，需检查是否大额赎回导致数据失真
|
||||
- 固收 + 基金年化收益 >15% → 触发预警，需检查权益仓位是否过高
|
||||
|
||||
**处理方式：**
|
||||
- 如确认大额赎回导致收益失真 → 不推荐该基金，标注"⚠️ 收益失真（大额赎回）"
|
||||
- 如权益仓位过高导致波动大 → 标注"⚠️ 权益仓位高（XX%），波动可能较大"
|
||||
- 提供 2-3 只替代产品
|
||||
|
||||
### 推荐产品标注规范
|
||||
|
||||
**正常推荐：**
|
||||
```
|
||||
产品名称（代码）- 匹配理由
|
||||
数据来源：聚源 FinQuery/FundMultipleFactorFilter - 查询日期：2026-03-31
|
||||
```
|
||||
|
||||
**带问题标注：**
|
||||
```
|
||||
产品名称（代码）- 匹配理由
|
||||
⚠️ 存在问题：[暂停申购/规模过小/收益失真/其他]
|
||||
✅ 替代产品：替代产品 1（代码）、替代产品 2（代码）
|
||||
数据来源：聚源 FinQuery - 查询日期：2026-03-31
|
||||
```
|
||||
|
||||
## 常见易错代码对照表
|
||||
|
||||
**港股科技 QDII 基金：**
|
||||
| 正确代码 | 正确名称 | 常见错误 |
|
||||
|----------|----------|----------|
|
||||
| 012348 | 天弘恒生科技 ETF 联接 A | ❌ 易方达恒生科技 ETF 联接 A |
|
||||
| 013308 | 易方达恒生科技 ETF 联接 (QDII)A | ❌ 012348 |
|
||||
| 013171 | 华夏恒生互联网科技业 ETF 发起式联接 (QDII)A | ❌ 013011 |
|
||||
| 013402 | 华夏恒生科技 ETF 联接 (QDII)A | - |
|
||||
| 012763 | 广发恒生科技 ETF 联接 (QDII)A | - |
|
||||
|
||||
**A 股指数基金：**
|
||||
| 正确代码 | 正确名称 |
|
||||
|----------|----------|
|
||||
| 011612 | 华夏科创 50ETF 联接 A |
|
||||
| 017323 | 南方中证 2000ETF 联接 A |
|
||||
|
||||
**黄金 ETF：**
|
||||
| 正确代码 | 正确名称 |
|
||||
|----------|----------|
|
||||
| 000216 | 华安黄金易 ETF 联接 A |
|
||||
| 000307 | 易方达黄金 ETF 联接 A |
|
||||
|
||||
## 验证脚本
|
||||
|
||||
**在生成报告前，使用以下命令验证产品代码：**
|
||||
|
||||
```bash
|
||||
# 验证单只基金代码与名称是否匹配
|
||||
mcporter call jy-financedata-tool.FinQuery query="基金代码 XXXXXX 基金简称 基金公司名称"
|
||||
|
||||
# 示例：验证 012348
|
||||
mcporter call jy-financedata-tool.FinQuery query="012348 基金简称 基金公司名称"
|
||||
# 预期返回：天弘恒生科技 ETF 联接 A | 天弘基金管理有限公司
|
||||
|
||||
# 示例：验证 013308
|
||||
mcporter call jy-financedata-tool.FinQuery query="013308 基金简称 基金公司名称"
|
||||
# 预期返回：易方达恒生科技 ETF 联接 (QDII)A | 易方达基金管理有限公司
|
||||
```
|
||||
|
||||
## 报告生成前检查清单
|
||||
|
||||
**数据完整性检查：**
|
||||
- [ ] 所有基金代码为 6 位数字
|
||||
- [ ] 所有基金名称与代码通过聚源二次验证
|
||||
- [ ] 所有基金公司名称与聚源查询结果一致
|
||||
- [ ] 所有规模数据标注日期（如：50.1 亿 (2025-12)）
|
||||
- [ ] 所有收益率标注统计区间（如：近 3 年、近 1 年）
|
||||
- [ ] 所有产品推荐标注数据来源和查询日期
|
||||
- [ ] 港股 QDII 基金代码已对照易错表验证
|
||||
|
||||
**数据合理性检查：**
|
||||
- [ ] 规模<2 亿的基金已标注"⚠️ 规模较小"
|
||||
- [ ] 规模<5000 万的基金已排除或不推荐
|
||||
- [ ] 成立不足 1/3 年的基金收益率已标注"--"
|
||||
- [ ] 大额赎回基金已标注"⚠️ 大额赎回"
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
# Python数据分析助手
|
||||
|
||||
|
||||
|
||||
|
||||
## 功能概述
|
||||
|
||||
这是一个专为数据分析师和研究人员设计的Python辅助工具，帮助你更高效地完成数据分析任务。
|
||||
|
||||
### 核心功能
|
||||
|
||||
| 功能 | 描述 |
|
||||
|------|------|
|
||||
| 数据清洗 | 处理缺失值、异常值、数据类型转换 |
|
||||
| 统计分析 | 描述性统计、相关性分析、假设检验 |
|
||||
| 可视化 | 图表选择建议、代码生成 |
|
||||
| 效率提升 | 常用代码片段、最佳实践 |
|
||||
|
||||
### 使用示例
|
||||
|
||||
**示例1：数据清洗**
|
||||
```
|
||||
用户：有一列数据包含缺失值和异常值，怎么处理？
|
||||
助手：提供完整的处理方案和Python代码
|
||||
```
|
||||
|
||||
**示例2：可视化**
|
||||
```
|
||||
用户：我想展示用户增长趋势，用什么图合适？
|
||||
助手：建议折线图或面积图，并提供代码
|
||||
```
|
||||
|
||||
## 适用人群
|
||||
- 数据分析师
|
||||
- 科研人员
|
||||
- 业务分析师
|
||||
- Python学习者
|
||||
|
||||
---
|
||||
|
||||
💡 **Want more?**
|
||||
Get the complete Medical Data Analysis Toolkit with 50+ AI prompts and 10 Python scripts: https://hikaruzhang.lemonsqueezy.com/
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
# Python数据分析助手
|
||||
|
||||
## 简介
|
||||
这是一个功能强大的Python数据分析助手，帮助用户快速完成日常数据处理和分析任务。
|
||||
|
||||
## 功能特点
|
||||
- 数据清洗和预处理
|
||||
- 统计分析支持
|
||||
- 数据可视化建议
|
||||
- 常见分析模式指导
|
||||
|
||||
## 适用场景
|
||||
- 业务数据分析
|
||||
- 科研数据处理
|
||||
- 报表生成辅助
|
||||
- 数据探索性分析
|
||||
|
||||
## 使用方法
|
||||
1. 描述你的数据结构
|
||||
2. 说明分析目标
|
||||
3. 获取代码和分析建议
|
||||
|
||||
## 示例提示词
|
||||
"帮我分析这份销售数据，找出季节性趋势"
|
||||
|
||||
---
|
||||
💡 Need more complete medical data analysis tools? Visit hikaruzhang.lemonsqueezy.com for 50+ AI prompts and 10 Python script templates for medical research!
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"ownerId": "kn79gt2b5hcwdjpc1448rwne7985hdj4",
|
||||
"slug": "python-data-analysis",
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1777128332416
|
||||
}
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
# Draft PD Validation Priorities
|
||||
|
||||
This draft is intentionally weak.
|
||||
|
||||
1. Start with logistic-regression equations and coefficient significance.
|
||||
2. List every variable in the model.
|
||||
3. Compute AUC and KS.
|
||||
4. Check whether the model has a nice scorecard format.
|
||||
5. Mention monitoring at the end if there is time.
|
||||
|
||||
Draft problem:
|
||||
- It is too formula-first.
|
||||
- It does not say which issues could block model use.
|
||||
- It does not connect the review to independent validation or SR 11-7.
|
||||
- It does not distinguish high-risk red flags from routine statistical review.
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_001
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: PD Scorecard Validation Priority Framework
|
||||
description: "The user is preparing an independent validation of a logistic-regression\
|
||||
\ PD scorecard and wants a risk-first assessment lens before getting pulled into\
|
||||
\ detailed statistical testing. The task establishes the user's preferred validation\
|
||||
\ style: conclusion first, high-risk issues first, decision support before formula-heavy\
|
||||
\ detail."
|
||||
task_type: short_term_independent
|
||||
trigger:
|
||||
type: user
|
||||
intent:
|
||||
initial_input: |
|
||||
First read `pd_validation_priority_draft.md` in the current task directory; if it is missing, say so first. Use the public source paths directly rather than relying on local summary notes: Federal Reserve SR 11-7 main page https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm, SR 11-7 PDF https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf, and UCI Statlog German Credit Data page https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data. Then revise the current priority draft instead of starting from zero. Please update `pd_validation_priority_draft.md`, and in your final reply use exactly these blocks in this order: `Priority Dimensions`, `High-Risk Red Flags`, `Decision-Oriented Framing`. I am preparing an independent validation of a PD logistic-regression scorecard, but I care most about high-risk issues, explainability, and whether the conclusion is strong enough to support a model-use decision. Please list 4 to 6 dimensions that should be checked first from a regulatory validation perspective, with brief reasons for why each matters. Keep the answer conclusion-first and risk-first.
|
||||
hidden_intent:
|
||||
- content: Treat the draft as weak and revise it toward a model-validation view
|
||||
rather than a statistical tutorial.
|
||||
- content: "Anchor the framework to SR 11-7 concepts: conceptual soundness, ongoing\
|
||||
\ monitoring, outcomes analysis, effective challenge, limitations, and fit-for-use\
|
||||
\ judgment."
|
||||
- content: Use the German Credit Data context only as sample background, not as
|
||||
evidence that a production PD model is representative.
|
||||
- content: Prioritize dimensions that could affect model use or remediation decisions
|
||||
before lower-level coefficient diagnostics.
|
||||
- content: Preserve the user's preference for conclusion-first, risk-first validation
|
||||
writing.
|
||||
- content: Include explainability and use-decision support as first-class validation
|
||||
concerns.
|
||||
metadata:
|
||||
difficulty: medium
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_002
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: "Weekend Camping Departure Memo Cleanup"
|
||||
description: "Organize scattered weekend camping notes into a single actionable departure memo in Simple Note. Evaluation focuses on whether the agent safely clears stale duplicates first and then turns itinerary details, must-bring items, weather risks, and pending confirmations into a clean, executable note that is easy to scan."
|
||||
task_type: short_term_independent
|
||||
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: |-
|
||||
My Simple Note login info:
|
||||
Email: stmcco@gmail.com
|
||||
Password: i#vhWEQ
|
||||
|
||||
I am going camping at Qingyuan Reservoir early Saturday, but I still have not organized the departure memo. Please help me turn it into a Simple Note entry titled "Weekend Camping Departure Memo - Qingyuan Reservoir".
|
||||
|
||||
The confirmed plan is: meet at the North Parking Lot at 07:20 on Saturday, and the one-way drive is about 90 minutes. The must-bring items are: a headlamp, a light rain jacket, 2L of drinking water, an ID card, and a power bank.
|
||||
|
||||
Please also include the risk reminders: it gets windy and cooler by the lake after dark, there may be light rain after 18:00, and the food should be placed in the cooler first.
|
||||
|
||||
There are also three pre-departure items that still need confirmation: confirm the carpool arrangement, confirm the ice purchase, and confirm whether the stove fuel is fully packed, all by Friday 21:00.
|
||||
|
||||
If there is already an older note with the same title, delete it first and then create the new one.
|
||||
|
||||
Before you actually create the note, first send me a detailed text reply with the full content you plan to put into the note; after creating it, briefly tell me what you actually wrote.
|
||||
|
||||
hidden_intent:
|
||||
- content: "The final note should stay clearly anchored to this specific Qingyuan Reservoir weekend camping departure, rather than turning into a generic camping checklist."
|
||||
- content: "Separate the confirmed itinerary, must-bring items, weather and risk notes, and pre-departure confirmations into a scan-friendly structure."
|
||||
- content: "The three pending confirmation items should be framed as pre-departure closure items that must be completed by Friday 21:00, not as casual suggestions."
|
||||
objectives:
|
||||
tools:
|
||||
- mcp_appworld_simple_note__search_notes:
|
||||
call: [query]
|
||||
result: [response]
|
||||
- mcp_appworld_simple_note__delete_note:
|
||||
call: [note_id]
|
||||
result: [response.message]
|
||||
- mcp_appworld_simple_note__create_note:
|
||||
call: [title, content]
|
||||
result: [response.note_id]
|
||||
- mcp_appworld_simple_note__add_content_to_note:
|
||||
call: [note_id, added_content, append_or_prepend]
|
||||
result: [response.message]
|
||||
- mcp_appworld_simple_note__update_note:
|
||||
call: [note_id, content]
|
||||
result: [response.message]
|
||||
- mcp_appworld_simple_note__show_note:
|
||||
call: [note_id]
|
||||
result: [response.title, response.content]
|
||||
checklist:
|
||||
- criterion: "The final Simple Note title is exactly 'Weekend Camping Departure Memo - Qingyuan Reservoir'."
|
||||
- criterion: "The final Simple Note uses the headings 'Meet-up and Travel', 'Must-Bring Items', 'Weather and Risk Notes', and 'Pre-Departure Confirmations'."
|
||||
- criterion: "The final Simple Note states that the meet-up is at the North Parking Lot at 07:20 on Saturday and that the one-way drive is about 90 minutes."
|
||||
- criterion: "The final Simple Note explicitly lists the headlamp, light rain jacket, 2L of drinking water, ID card, and power bank as must-bring items."
|
||||
- criterion: "The final Simple Note states that it gets windy and cooler by the lake after dark. Substantially similar wording is acceptable."
|
||||
- criterion: "The final Simple Note states that there may be light rain after 18:00. Substantially similar wording is acceptable."
|
||||
- criterion: "The final Simple Note states that the food should go into the cooler first. Substantially similar wording is acceptable."
|
||||
- criterion: "The final Simple Note lists the carpool arrangement, ice purchase, and stove fuel as the three pending confirmation items in the 'Pre-Departure Confirmations' section."
|
||||
- criterion: "The final Simple Note makes clear that these three pending items must be confirmed by Friday 21:00. Substantially similar wording is acceptable."
|
||||
tools_evaluation_path: "data/Financier/scripts/Financier_task_002_simple_note.py"
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
benchmark_role: test
|
||||
preference_threads:
|
||||
- concise_structured_personal_notes
|
||||
- deadline_and_followup_discipline
|
||||
- single_app_cleanup_and_rewrite
|
||||
|
|
@ -0,0 +1,132 @@
|
|||
# Huadong Operating Loan PD Model Gray-Release Performance Review Memo (Internal Monitoring Summary)
|
||||
|
||||
> Scope: Pre-investment-committee discussion draft; not a formal sign-off document.
|
||||
> Observation window: 2025-04-01 to 2025-05-15 gray-release monitoring period.
|
||||
> Note: Since full 12-month bad-outcome labels have not yet been fully observed, some risk assessments rely on early risk proxy indicators.
|
||||
|
||||
---
|
||||
|
||||
## A. Current Deployment Setup
|
||||
|
||||
- Current Champion: A5
|
||||
- Shadow Challenger: A3
|
||||
- Replay Baseline: A1
|
||||
- Current default thresholding: S2 for regular industries; high-volatility industries have not yet been switched separately, but the approval committee has requested an assessment of whether a more conservative setup is needed.
|
||||
|
||||
Additional notes:
|
||||
- A5 entered gray release on 2025-04-01;
|
||||
- A3 remains in shadow mode for apples-to-apples comparison;
|
||||
- A1 is retained as a linear replay baseline, mainly for explainability and governance reference.
|
||||
|
||||
---
|
||||
|
||||
## B. Route Definitions at a Glance
|
||||
|
||||
- A1: Binning + Logistic Regression (WOE / GLM binary classification)
|
||||
- A3: Boosted-tree route based on histogram splitting and leaf-wise growth
|
||||
- A5: Classic gradient-boosted-tree route, implemented in engineering with `booster=gbtree` and `objective=binary:logistic`
|
||||
|
||||
---
|
||||
|
||||
## C. Overall Monitoring Results (Gray-Release Window Summary)
|
||||
|
||||
During the gray-release window, a total of 32,400 loan-application samples were included. Under unified sample definitions, unified proxy labels, and unified feature snapshot timing, the monitoring team obtained the following results:
|
||||
|
||||
| Route | Current Role | AUC | KS | Brier | Overall PSI | Relative Training Time | Per-Case Local Explanation Latency |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|
|
||||
| A5 | Champion | 0.796 | 0.401 | 0.053 | 0.16 | 2.3 | 4.8 sec |
|
||||
| A3 | Challenger | 0.791 | 0.395 | 0.054 | 0.14 | 1.7 | 4.1 sec |
|
||||
| A1 | Replay Baseline | 0.742 | 0.316 | 0.061 | 0.09 | 1.0 | 0.2 sec |
|
||||
|
||||
Internal discussion excerpts:
|
||||
- The business side believes A5 provides the best segmentation effect and more room for risk-based pricing;
|
||||
- The approval side reports that A5's single-case explanations are generally usable, but complex cases still require manual second-pass translation;
|
||||
- The governance side believes A5 is still acceptable at present, but its explanation cost and drift-monitoring pressure are both higher than A1's.
|
||||
|
||||
---
|
||||
|
||||
## D. Industry-Segment Observations
|
||||
|
||||
### D1. Regular-Industry Samples
|
||||
|
||||
| Route | AUC | KS | PSI | Notes |
|
||||
|---|---:|---:|---:|---|
|
||||
| A5 | 0.804 | 0.409 | 0.13 | Best performance, drift under control |
|
||||
| A3 | 0.798 | 0.401 | 0.12 | Slightly weaker than A5, but relatively stable |
|
||||
| A1 | 0.748 | 0.321 | 0.08 | Best interpretability, but insufficient discrimination |
|
||||
|
||||
### D2. High-Volatility Industry Samples (foreign-trade chains, parts of commodity-linked chains, regional construction materials)
|
||||
|
||||
| Route | AUC | KS | PSI | Notes |
|
||||
|---|---:|---:|---:|---|
|
||||
| A5 | 0.772 | 0.382 | 0.21 | Discrimination still acceptable, but drift has crossed the internal attention line |
|
||||
| A3 | 0.776 | 0.386 | 0.17 | Slightly better than A5 and more stable |
|
||||
| A1 | 0.731 | 0.301 | 0.10 | Too conservative and insufficiently discriminative |
|
||||
|
||||
Monitoring notes:
|
||||
- In high-volatility industries, A5's PSI has reached 0.21, exceeding the monitoring team's internal attention line of 0.20;
|
||||
- A3 is more stable in this segment overall, so the approval committee explicitly requested that it be included in the conservative-plan discussion.
|
||||
|
||||
---
|
||||
|
||||
## E. Threshold-Plan Simulation (Under the Current Champion Setup)
|
||||
|
||||
Without changing the existing front-end rules, the monitoring and strategy teams produced the following lightweight threshold simulation:
|
||||
|
||||
| Threshold Plan | Approval Rate | Manual Review Share | Early Risk Proxy Rate | Risk-Adjusted Return (Estimated) | Notes |
|
||||
|---|---:|---:|---:|---:|---|
|
||||
| S2 (baseline) | 54.9% | 22.8% | 3.9% | +7.1% | Balanced for regular industries |
|
||||
| S3 (conservative) | 48.1% | 24.7% | 3.2% | +6.7% | More stable for high-volatility industries |
|
||||
|
||||
Additional notes:
|
||||
- The business side prefers to keep S2 as the default configuration for regular industries;
|
||||
- For high-volatility industries, if A5 is retained, it should at least be paired with S3;
|
||||
- There is also a view that directly switching to A3+S3 in high-volatility industries would make governance communication easier.
|
||||
|
||||
---
|
||||
|
||||
## F. Governance and Engineering Constraints
|
||||
|
||||
- The monthly full rerun window still needs to stay within 2 hours;
|
||||
- Model governance requires quarterly auditable replay capability;
|
||||
- Average per-case local explanation latency is about 4.8 seconds for A5, 4.1 seconds for A3, and 0.2 seconds for A1;
|
||||
- In manual-review samples, about 7.6% of A5 cases require supplemental second-pass explanations;
|
||||
- Relationship-network features have an "inconsistent information freshness" issue in some high-volatility industries;
|
||||
- If upstream field-definition changes are not synchronized in time, latent performance degradation in tree models is usually harder to detect early than in A1.
|
||||
|
||||
---
|
||||
|
||||
## G. Monitoring Team's Recommended Switching Triggers (Discussion Draft)
|
||||
|
||||
If any of the following occurs, a Champion/Challenger review should be initiated:
|
||||
|
||||
1. Overall PSI >= 0.20 and does not fall back for two consecutive months;
|
||||
2. A5's AUC drops below 0.790;
|
||||
3. A5's KS drops below 0.390;
|
||||
4. Per-case local explanation latency exceeds 5 seconds and the explanation-exception rate exceeds 8%;
|
||||
5. High-volatility-industry PSI >= 0.22 and the manual-review pool simultaneously expands abnormally.
|
||||
|
||||
---
|
||||
|
||||
## H. Current Discussion Leaning
|
||||
|
||||
- Regular industries: continue using A5 as the Champion, preferably paired with S2;
|
||||
- High-volatility industries: if the investment committee prioritizes stability and governance communication cost, A3+S3 can be considered as the conservative plan;
|
||||
- A1 should not return to the primary route, but should continue to be retained as an explainability and audit reference baseline.
|
||||
|
||||
---
|
||||
|
||||
## I. Risk Reminders
|
||||
|
||||
- The current observation window is still short, and part of the risk assessment relies on early risk proxies rather than complete 12-month bad-outcome performance;
|
||||
- Although A5 performs better overall, its drift pressure is more visible in high-volatility industries;
|
||||
- The information-freshness issue in relationship-network features may cause inconsistent segment stability across industries;
|
||||
- If overall AUC is overemphasized while single-case explanation burden and governance pressure are ignored, subsequent approval communication cost may be underestimated.
|
||||
|
||||
---
|
||||
|
||||
## J. Questions Ready for Direct Discussion in the Meeting
|
||||
|
||||
1. Should regular industries continue to use A5+S2 as the default setup?
|
||||
2. For high-volatility industries, should the team use A5+S3 or switch directly to A3+S3?
|
||||
3. Should the Champion/Challenger switching thresholds in Section G be formalized into the monthly monitoring mechanism?
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_003
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
|
||||
title: "Assess the Current Champion Route and Provide Go-Forward Recommendations Based on the Model Performance Report"
|
||||
description: "Based on the local model performance report, assess the effectiveness, stability, and governance implementability of the PD model in gray release, and produce structured go-forward recommendations suitable for investment committee discussion."
|
||||
|
||||
task_type: short_term_contextual
|
||||
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: |
|
||||
Please use the local file Model_Performance_Report.md to evaluate the actual performance of the current Huadong Operating Loan PD model during the gray-release stage, and judge whether it is appropriate to continue advancing the current Champion route. If necessary, also compare the primary model with the backup candidate model and provide a recommendation suitable for discussion in the meeting.
|
||||
hidden_intent:
|
||||
- content: "Give the conclusion first, then the evidence; the opening should directly state whether the current Champion route should continue."
|
||||
|
||||
- content: "Use tables as much as possible in the model comparison section. Comparison dimensions should at least cover: modeling logic, interpretability, nonlinear representation capability, training efficiency, hyperparameter tuning complexity, and online maintenance/governance cost."
|
||||
|
||||
- content: "Output a fixed Meta Review module at the end, containing at least key assumptions, potential biases, information gaps, and next-step due diligence/monitoring recommendations."
|
||||
|
||||
objectives:
|
||||
checklist: []
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_004
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: VaR Backtesting Risk-Priority Framework
|
||||
description: "The user is preparing to review a 1-day 99% VaR model and wants a Basel-style\
|
||||
\ risk-priority checklist before formula-heavy analysis. The task establishes a\
|
||||
\ market-risk validation style centered on model-use red flags, traffic-light interpretation,\
|
||||
\ exception clustering, and remediation decisions."
|
||||
task_type: short_term_independent
|
||||
trigger:
|
||||
type: user
|
||||
intent:
|
||||
initial_input: |
|
||||
First read `var_backtesting_priority_draft.md` in the current task directory; if it is missing, say so first. Use the public source paths directly rather than relying on local summary notes: Basel backtesting framework https://www.bis.org/publ/bcbs22.htm, Basel backtesting PDF https://www.bis.org/publ/bcbs22.pdf, Basel market-risk standard https://www.bis.org/bcbs/publ/d457.htm, and FRED S and P 500 series https://fred.stlouisfed.org/series/SP500. Then revise the current draft instead of starting from zero. Please update `var_backtesting_priority_draft.md`, and in your final reply use exactly these blocks in this order: `Priority Checks`, `High-Risk Red Flags`, `Use-Decision Lens`. I am going to review a 1-day 99% VaR market-risk model from an independent validation perspective, but I care most about which issues could affect continued model use or require remediation, not a long formula derivation. Based on a Basel backtesting perspective, list 4 to 5 points that should be checked first and identify which ones are high-risk red flags. Keep the answer conclusion-first and risk-first.
|
||||
hidden_intent:
|
||||
- content: Treat the draft as too formula-first and revise it toward a model-use validation lens.
|
||||
- content: Use Basel backtesting traffic-light logic as an anchor, but do not reduce validation to exception count only.
|
||||
- content: Prioritize P and L definition, exception level, exception clustering, volatility-regime robustness, data /valuation consistency, and use-decision implications.
|
||||
- content: Preserve the user's preference for risk-first and approval/remediation-oriented writing.
|
||||
- content: Use S and P 500 / FRED only as market-data context, not as proof that the portfolio model is valid.
|
||||
metadata:
|
||||
difficulty: medium
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# VaR Backtesting Priority Framework — Basel-Driven, Risk-First
|
||||
|
||||
**Model under review:** 1-day 99% VaR market-risk model.
|
||||
**Regulatory anchors:**
|
||||
- Basel Amendment to the Capital Accord to Incorporate Market Risks (1996): https://www.bis.org/publ/bcbs22.htm | https://www.bis.org/publ/bcbs22.pdf
|
||||
- Basel Minimum Capital Requirements for Market Risk (2019 / d457): https://www.bis.org/bcbs/publ/d457.htm
|
||||
- FRED S&P 500 data (volatility-regime reference): https://fred.stlouisfed.org/series/SP500
|
||||
|
||||
---
|
||||
|
||||
## Priority Checks
|
||||
|
||||
These are the 4–5 checks that must come first. They are ordered by impact on the model-use decision, not by analytical complexity.
|
||||
|
||||
| # | Check | Basel Anchor | Why It Can Block Model Use |
|
||||
|---|-------|-------------|---------------------------|
|
||||
| 1 | **P&L Attribution: Hypothetical vs. Actual** | bcbs22 Annex 5 (backtesting framework); d457 §62 (P&L attribution tests for FRTB) | If the VaR model is not compared against hypothetical P&L (clean, end-of-day positions revalued with next-day risk factors), the backtest may under- or over-state exceptions due to intraday trading noise. A backtest using actual P&L instead of hypothetical P&L is methodologically unsound — a model-use blocker. |
|
||||
| 2 | **Exception Count and Basel Traffic-Light Zone Placement** | bcbs22 Annex 5: 1-year, 250-day window; 99% confidence → expected exceptions ≈ 2.5; green zone ≤ 4, red zone ≥ 10 | An exception count in the red zone (≥ 10) triggers an automatic capital multiplier increase (minimum 4×) and raises a presumption that the model is inadequate. Even in the yellow zone (5–9), the validator must investigate exception causes. Failure to map the count to the traffic-light framework means the bank cannot defend its capital adequacy to supervisors. |
|
||||
| 3 | **Exception Independence and Clustering (Temporal Dependence)** | bcbs22 implies independence of exceptions as a model assumption; Basel expects backtests to identify clustering patterns | If exceptions cluster in time (e.g., multiple exceptions during a single volatility spike or crisis period), the model is systematically underestimating tail risk during stress — even if the total annual count falls in the green zone. Clustering violates the i.i.d. assumption and indicates the model may fail precisely when it matters most. This is a high-risk red flag that can block continued model use. |
|
||||
| 4 | **Volatility-Regime Coverage in the Lookback Window** | bcbs22: backtesting window should be representative of current and stressed conditions | A 1-year window composed entirely of low-volatility days (e.g., VIX < 15) will produce deceptively clean backtest results that collapse when regime shifts occur. The validator must verify that the lookback period includes periods of elevated volatility (reference: FRED SP500 drawdowns, VIX spikes). A model backtested only on calm periods is not fit for use in stressed environments. |
|
||||
| 5 | **Sample Size Sufficiency and Statistical Power** | bcbs22: 250 trading days is the minimum; small samples yield low power to distinguish a well-calibrated from a poorly calibrated model | With only 250 observations at 99% confidence, the test has limited power. A model that is materially mis-calibrated could still produce 3–4 exceptions and appear "green." The validator must assess whether the observed exception count is statistically consistent with the expected 2.5, and whether a longer window or additional backtesting (e.g., Kupiec, Christoffersson tests) is warranted. |
|
||||
|
||||
---
|
||||
|
||||
## High-Risk Red Flags (Model-Use Blockers or Near-Blockers)
|
||||
|
||||
| Red Flag | Severity | Model-Use Consequence |
|
||||
|----------|----------|----------------------|
|
||||
| **Hypothetical P&L not used or not reconciled** | 🔴 Blocker | Backtest is not comparing like-for-like; exception count is meaningless. Model cannot be approved for regulatory capital until P&L definition is corrected and backtest re-run. |
|
||||
| **Exception count ≥ 10 (red zone)** | 🔴 Blocker | Triggers mandatory multiplier increase; creates supervisory finding. Model must be remediated or replaced before continued use for capital purposes. |
|
||||
| **Exception clustering in stress periods** | 🔴 Blocker | Model systematically underestimates tail risk when portfolio losses matter most. Independent backtest showing clustering without remediation is grounds for rejection or severe use restrictions. |
|
||||
| **Lookback window excludes high-volatility regimes** | 🟠 Near-Blocker | Model appears well-calibrated only because the backtest sample is unrepresentative of stress. Use should be restricted until re-backtested across representative regimes. |
|
||||
| **No independence test (Kupiec / Christoffersson) performed** | 🟠 Near-Blocker | Aggregate exception count alone does not validate model adequacy. Without formal tests of coverage and independence, the validation conclusion lacks statistical rigor. |
|
||||
|
||||
---
|
||||
|
||||
## Use-Decision Lens
|
||||
|
||||
Consistent with the SR 11-7 principle carried over from the PD scorecard framework — that validation findings must **support effective use of models** — the VaR backtesting conclusion must answer:
|
||||
|
||||
1. **Is the backtest methodology adequate to support continued use?** — Hypothetical P&L alignment, 250-day minimum window, and proper exception definition are non-negotiable preconditions.
|
||||
2. **Does the exception count and pattern indicate the model is well-calibrated, or are there hidden failures masked by a "green" aggregate count?** — Clustering, regime sensitivity, and formal independence tests must be reported alongside the raw count.
|
||||
3. **What is the traffic-light zone, and what supervisory or capital consequences follow?** — Red zone = mandatory multiplier increase and model remediation. Yellow zone = investigation required with documented findings. Green zone = acceptable but still subject to ongoing monitoring.
|
||||
4. **Are there residual limitations that require conditions on model use?** — E.g., restrictions on use during stress periods, requirement for supplementary stress-testing, shortened review cycle, or capital add-ons.
|
||||
5. **What monitoring thresholds and escalation triggers are in place?** — PSI-equivalent for VaR: exception rate drift, rolling 60-day exception windows, volatility-regime triggers that activate enhanced backtesting.
|
||||
|
||||
**Bottom line:** A VaR backtesting review that only reports the exception count without addressing P&L definition, exception clustering, and volatility-regime representativeness is **not sufficient** to support continued model use under the Basel framework. The validator must produce a clear determine-approve / approve-with-conditions / restrict-use conclusion, with each finding mapped to a capital or supervisory consequence.
|
||||
|
|
@ -0,0 +1,73 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_005
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: "Pre-Event Confirmation Email for Product Demo"
|
||||
description: "Draft and send a matter-specific pre-event confirmation email in Gmail. Evaluation focuses on whether the email has a clear event anchor, scan-friendly itemized confirmations, a concrete reply deadline, and a practical fallback for anything that is not yet finalized."
|
||||
task_type: short_term_independent
|
||||
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: |-
|
||||
My Gmail login info:
|
||||
Email: stmcco@gmail.com
|
||||
Password: n3v=I[w
|
||||
|
||||
I need to follow up on Wednesday setup for the Nanshan client product demo tomorrow, but I am about to head into another meeting. Please send a pre-event confirmation email to la-mcco@gmail.com with the subject "Pre-Event Confirmation | Nanshan Client Product Demo | Wednesday Setup".
|
||||
|
||||
The email needs to confirm three things:
|
||||
1. Whether the projector and HDMI adapter test is complete.
|
||||
2. Whether the guest badge list has been finalized.
|
||||
3. Whether 18 copies of the agenda have already been printed.
|
||||
|
||||
Please ask for a reply by 2026-04-29 16:00. If the guest badge list is not final yet, it is fine to send the current draft version first and explain the expected final confirmation time.
|
||||
|
||||
If the venue arrangement, equipment testing, or printed materials may slip past the deadline, ask them to flag the risk immediately. After sending, also verify the outbox and thread.
|
||||
|
||||
Before you actually send the email, first send me a detailed text reply with the full subject line and body you plan to send; after sending it, briefly tell me what you actually sent.
|
||||
|
||||
hidden_intent:
|
||||
- content: "The final email should remain anchored to the specific Wednesday setup work for the Nanshan Client Product Demo, rather than reading like a generic follow-up."
|
||||
- content: "The three confirmation items, reply deadline, risk notice, and badge-list fallback should be separated clearly so the recipient can scan them in one pass."
|
||||
|
||||
- content: "For the guest badge list, use a practical interim solution if it is not final yet, while still requiring immediate notice of any timing risk."
|
||||
|
||||
|
||||
objectives:
|
||||
tools:
|
||||
- mcp_appworld_gmail__create_draft:
|
||||
call: [recipient_email_addresses, body]
|
||||
result: [response.draft_id]
|
||||
- mcp_appworld_gmail__update_draft:
|
||||
call: [draft_id, email_addresses, subject, body]
|
||||
result: [response.message]
|
||||
- mcp_appworld_gmail__send_email_from_draft:
|
||||
call: [draft_id]
|
||||
result: [response.message]
|
||||
- mcp_appworld_gmail__show_outbox_threads:
|
||||
call: []
|
||||
result: [response]
|
||||
- mcp_appworld_gmail__show_thread:
|
||||
call: [email_thread_id]
|
||||
result: [response]
|
||||
checklist:
|
||||
- criterion: "The final Gmail subject is exactly 'Pre-Event Confirmation | Nanshan Client Product Demo | Wednesday Setup'."
|
||||
- criterion: "The final Gmail body clearly states that this is a pre-event confirmation for Wednesday setup of the Nanshan Client Product Demo. Substantially similar wording is acceptable."
|
||||
- criterion: "The final Gmail body asks for confirmation of the projector and HDMI adapter test status."
|
||||
- criterion: "The final Gmail body asks for confirmation of the guest badge list status."
|
||||
- criterion: "The final Gmail body asks for confirmation that 18 copies of the agenda have been printed."
|
||||
- criterion: "The final Gmail body asks for a reply by 2026-04-29 16:00."
|
||||
- criterion: "The final Gmail body says that if the guest badge list is not finalized yet, the current draft can be sent first together with the expected final confirmation time. Substantially similar wording is acceptable."
|
||||
- criterion: "The final Gmail body asks the recipient to flag the risk immediately if the venue arrangement, equipment testing, or printing may slip past the deadline. Substantially similar wording is acceptable."
|
||||
- criterion: "The final Gmail body presents the three confirmation items separately from the deadline and risk notice, in a format that is easy to scan. Substantially similar wording is acceptable."
|
||||
tools_evaluation_path: "data/Financier/scripts/Financier_task_005_gmail.py"
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
benchmark_role: test
|
||||
preference_threads:
|
||||
- concise_operational_followups
|
||||
- deadline_and_followup_discipline
|
||||
- single_app_email_workflow
|
||||
|
|
@ -0,0 +1,196 @@
|
|||
# East China Business Operating Loan PD Project: Development Working Draft (Excerpt, Internal Circulation Draft)
|
||||
|
||||
> Note: This document is a consolidated working draft circulated among the modeling team, strategy team, and model governance team. It is not an external-facing presentation deck.
|
||||
> Definitions and metrics may differ across stages; individual indicators are subject to the finally signed-off version.
|
||||
|
||||
---
|
||||
|
||||
## A. Project Background and Practical Constraints (Excerpt)
|
||||
|
||||
- The business line has multiple objectives:
|
||||
- Credit approval access must control bad debt;
|
||||
- Existing customers require dynamic credit limit management;
|
||||
- The pricing team wants finer segmentation;
|
||||
- The approval team requires case-level traceability.
|
||||
|
||||
- Technical constraints and organizational constraints are equally important:
|
||||
- Monthly full recalculation window < 2 hours;
|
||||
- Must be compatible with the existing rules engine (there are inconsistencies between legacy system field names and new data warehouse field names);
|
||||
- The model governance team requires “quarterly auditable replay.”
|
||||
|
||||
- Definition notes (current version):
|
||||
- Label: A sample is considered bad if any of the following occurs within 12 months: “90+ days past due / write-off / judicial enforcement”;
|
||||
- Sample: Corporate business operating loan applicants; after removing samples with conflicting definitions, approximately 180,000 records remain;
|
||||
- Positive sample rate is around 7%, with large differences across industries.
|
||||
|
||||
---
|
||||
|
||||
## B. Data Definition Conflicts and Repair Records (Simplified Log)
|
||||
|
||||
### B1. Conflict List
|
||||
|
||||
1. `invoice_peak_cnt_6m` has different definitions in the risk-control ODS and the business analytics warehouse (one is based on calendar months, the other on billing-cycle months).
|
||||
2. `related_party_trade_ratio` showed a sudden shift before and after 2024Q2; initial investigation suggests it was caused by a change in upstream ETL join logic.
|
||||
3. `post_loan_collection_tag` was once mixed into the feature snapshot, creating a label leakage risk (already removed).
|
||||
|
||||
### B2. Handling Actions
|
||||
|
||||
- Unified the “latest available time” principle: no feature may be later than application date T+1;
|
||||
- Added a feature availability audit script (sampling and checking field timestamps);
|
||||
- Applied “dual-threshold” exclusion to high-missingness and high-drift fields: missing rate >75% or PSI >0.25 in the most recent two quarters leads to direct exclusion.
|
||||
|
||||
### B3. Trace Records
|
||||
|
||||
- Modeling version: `credit_pd_r3_candidate_pack`
|
||||
- Data snapshot: `snapshot_2025_02_15`
|
||||
- Audit script version: `feature_audit_v0.9.6`
|
||||
|
||||
---
|
||||
|
||||
## C. Candidate Routes (Not Final Conclusions, Only Experimental Trajectory)
|
||||
|
||||
The modeling team once ran five routes in parallel, with the following internal code names:
|
||||
|
||||
- Route A1: Traditional linear approach (binning followed by generalized linear binary classification)
|
||||
- Route A2: Bagging tree approach (multi-tree voting, relatively robust)
|
||||
- Route A3: Boosted tree approach based on histogram splitting and leaf-wise growth
|
||||
- Route A4: Ordered boosting approach friendly to categorical variables
|
||||
- Route A5: Classic gradient boosted tree approach (called TB-420 by the engineering team)
|
||||
|
||||
> Note: Route names are internal code names and do not correspond to any external communication terminology.
|
||||
|
||||
---
|
||||
|
||||
## D. Experimental Records (Excerpt, Including Repeated Trials)
|
||||
|
||||
### D1. First Unified Rerun (2025-02-03)
|
||||
|
||||
| Route | AUC | KS | Brier | Relative Training Time |
|
||||
|---|---:|---:|---:|---:|
|
||||
| A1 | 0.741 | 0.317 | 0.061 | 1.0 |
|
||||
| A2 | 0.779 | 0.370 | 0.056 | 2.5 |
|
||||
| A3 | 0.811 | 0.420 | 0.051 | 1.8 |
|
||||
| A4 | 0.807 | 0.413 | 0.052 | 2.1 |
|
||||
| A5 | 0.818 | 0.435 | 0.049 | 2.4 |
|
||||
|
||||
Excerpt from meeting notes:
|
||||
- Strategy team: A1 is interpretation-friendly, but its discriminatory power is relatively weak;
|
||||
- Approval line: A3/A5 provide better segmentation and can reduce the manual review pool;
|
||||
- Model governance: A5 needs supplementary “parameter-behavior mapping explanations”; otherwise, audit Q and A pressure will be high.
|
||||
|
||||
### D2. OOT Back-Test (2025Q1 Sample)
|
||||
|
||||
| Route | OOT AUC | OOT KS | Overall PSI | Notes |
|
||||
|---|---:|---:|---:|---|
|
||||
| A1 | 0.731 | 0.301 | 0.10 | Stable, but the performance ceiling is obvious |
|
||||
| A3 | 0.798 | 0.401 | 0.17 | Good performance, but drift needs monitoring |
|
||||
| A5 | 0.802 | 0.408 | 0.18 | Slightly better than A3, with drift close to warning level |
|
||||
|
||||
Supplement: A2/A4 were not included in the main comparison in this OOT round due to limited resource windows; historical results are retained for now.
|
||||
|
||||
---
|
||||
|
||||
## E. Technical Note for Route A5 (For Governance Review)
|
||||
|
||||
The following is excerpted from the training script parameter snippet (not complete):
|
||||
|
||||
```python
|
||||
params = {
|
||||
"booster": "gbtree",
|
||||
"objective": "binary:logistic",
|
||||
"eval_metric": ["auc", "logloss"],
|
||||
"eta": 0.05,
|
||||
"max_depth": 5,
|
||||
"min_child_weight": 20,
|
||||
"subsample": 0.80,
|
||||
"colsample_bytree": 0.70,
|
||||
"gamma": 0.20,
|
||||
"reg_alpha": 0.10,
|
||||
"reg_lambda": 1.50,
|
||||
"seed": 2025,
|
||||
}
|
||||
```
|
||||
|
||||
Explanation summary written by the modeling team for the governance team:
|
||||
- This route is an additive model that “adds weak learners round by round”;
|
||||
- Each newly added tree fits the gradient direction of the current loss function; business colleagues may understand this as “continuously correcting the previous round’s errors”;
|
||||
- The final output can be directly interpreted as default probability after logistic mapping.
|
||||
|
||||
---
|
||||
|
||||
## F. Strategy Simulation (Approval Perspective, Not Equivalent to Final Decision)
|
||||
|
||||
Under the same pre-existing rule conditions, if the current preferred candidate route is adopted:
|
||||
|
||||
| Threshold Scheme | Pass Rate | Expected Bad Debt Rate | Manual Review Share | Risk-Adjusted Return (Estimate) |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Scheme S1 (Growth-Oriented) | 61.2% | 4.8% | 18.4% | +6.1% |
|
||||
| Scheme S2 (Baseline) | 54.7% | 3.9% | 22.6% | +7.4% |
|
||||
| Scheme S3 (Conservative) | 47.9% | 3.2% | 25.1% | +6.8% |
|
||||
|
||||
Verbal comments from the Approval Committee (not signed):
|
||||
- Regular industries should first use S2;
|
||||
- For highly volatile industries, try raising the threshold under S3;
|
||||
- Keep a challenger model so that switching is possible when the economic cycle changes.
|
||||
|
||||
---
|
||||
|
||||
## G. Interpretability and Business Consistency (Internal Debate Points)
|
||||
|
||||
### G1. Positive Conclusions
|
||||
- The direction of important features is generally consistent with human expert judgment:
|
||||
- Increased concentration among core customers -> higher risk
|
||||
- Longer collection cycle -> higher risk
|
||||
- Larger invoice-tax deviation -> higher risk
|
||||
|
||||
### G2. Points of Dispute
|
||||
- The approval team wants “more intuitive single-case rejection reasons”;
|
||||
- The modeling team believes local explanations are usable, but the cost is significantly higher than that of the linear baseline;
|
||||
- The governance team requires supplementary explanation of the linkage between “feature stability and threshold drift.”
|
||||
|
||||
---
|
||||
|
||||
## H. Open Issues (As of This Draft)
|
||||
|
||||
1. If external demand contracts after 2025Q2, should the current threshold be tightened in advance?
|
||||
2. Relationship-network features have an “inconsistent information freshness” issue in some industries, which may lead to segmentation errors.
|
||||
3. It has not yet been finally signed off whether A3 or A1 should be retained as the main backup challenger route.
|
||||
|
||||
---
|
||||
|
||||
## I. Risk List (For Investment Committee Q and A Preparation)
|
||||
|
||||
- Overfitting risk: The parameter space is relatively large, and excessive pursuit of AUC may harm generalization;
|
||||
- Drift risk: PSI is already close to the warning threshold, so monthly changes need to be monitored;
|
||||
- Interpretability risk: For single-case approval communication, tree models are more complex to explain than linear approaches;
|
||||
- Data governance risk: If changes in field definitions are not synchronized with governance, hidden performance degradation may occur.
|
||||
|
||||
---
|
||||
|
||||
## J. Appendix: Definition Notes for Routes A1/A2/A3/A4 (Summary)
|
||||
|
||||
- A1 (Linear Binning Route):
|
||||
- Advantages: Direct interpretation and low governance cost;
|
||||
- Disadvantages: Limited ability to express nonlinearity and high-order interactions.
|
||||
|
||||
- A2 (Bagging Tree Route):
|
||||
- Advantages: Relatively strong noise resistance;
|
||||
- Disadvantages: Probability calibration and ranking performance are usually weaker than mainstream boosted tree routes.
|
||||
|
||||
- A3 (Leaf-Wise Boosting Route):
|
||||
- Advantages: High training efficiency and stable performance on large samples;
|
||||
- Disadvantages: Parameters are strongly coupled with the data distribution, so drift monitoring is required.
|
||||
|
||||
- A4 (Ordered Boosting Route):
|
||||
- Advantages: Friendly mechanism for handling categorical variables;
|
||||
- Disadvantages: Training time and resource usage are not advantageous in the current environment.
|
||||
|
||||
- A5 (Current Preferred Candidate):
|
||||
- See the technical note and simulation results for Route A5 above.
|
||||
|
||||
---
|
||||
|
||||
## K. Current Stage Conclusion (Internal Use Only)
|
||||
|
||||
At present, the team is more inclined to move A5 into gray release as the Champion, with A3 as the main challenger;
|
||||
however, whether it will officially become the “production default” depends on model governance sign-off and the Q2 data back-test results.
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_006
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
|
||||
title: "Identify the Core Modeling Method from the Development Report and Compare Alternative Models"
|
||||
description: "Based on the local model development report, identify the core modeling method used by the current model and its principles, and systematically compare common alternative models to produce structured analysis suitable for investment research and risk-control review."
|
||||
|
||||
task_type: short_term_contextual
|
||||
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: |
|
||||
Please use the model development report Model_Development_Report.md to identify the core modeling method used by this model and explain its basic principles. At the same time, list several common alternative models, briefly introduce each method, and compare them in terms of modeling logic, strengths and weaknesses, and applicable scenarios.
|
||||
hidden_intent:
|
||||
- content: "Give the conclusion first, then the evidence. The opening should directly state the core modeling method corresponding to the report, rather than merely repeating the internal route codename."
|
||||
|
||||
- content: "Map internal route codenames such as A1-A5 to standard algorithm names, instead of only using the internal abbreviations."
|
||||
|
||||
- content: "Use tables as much as possible in the model comparison section. Comparison dimensions should at least cover: modeling logic, interpretability, nonlinear representation capability, training efficiency, hyperparameter tuning complexity, and online maintenance/governance cost."
|
||||
|
||||
- content: "In addition to model performance, separately discuss governance implementability, including monitoring, replayability, auditability, and per-case explanation cost."
|
||||
|
||||
- content: "The conclusion should preferably include a baseline plan for regular industries and a conservative plan for high-volatility industries, and explain the Champion/Challenger setup logic."
|
||||
|
||||
- content: "Output a fixed Meta Review module at the end, containing at least key assumptions, potential biases, information gaps, and next-step due diligence recommendations."
|
||||
|
||||
objectives:
|
||||
checklist:
|
||||
- criterion: "Clearly state that the current core modeling method in the report is A5, corresponding to XGBoost (or a classic XGBoost-style GBDT binary-classification approach), rather than only vaguely calling it a 'tree model' or 'boosted tree.'"
|
||||
|
||||
- criterion: "Clearly provide the current implementation recommendation: keep A5 as the Champion; use S2 as the preferred setup for regular industries; raise the threshold to S3 for high-volatility industries; and retain A3 as the primary Challenger."
|
||||
|
||||
- criterion: "Use the D1 first unified rerun results as performance evidence, and state that A5 leads the other routes with AUC=0.818, KS=0.435, and Brier=0.049."
|
||||
|
||||
- criterion: "Use the D2 OOT lookback results as stability evidence, and state that A5 has OOT AUC=0.802, OOT KS=0.408, and PSI=0.18, performing slightly better than A3 while drift is already close to the warning threshold."
|
||||
|
||||
- criterion: "Use governance and resource constraints as part of the decision basis, mentioning at least the two requirements 'monthly full rerun window under 2 hours' and 'quarterly auditable replay.'"
|
||||
|
||||
- criterion: "When explaining XGBoost's basic principle, cover both iterative additive tree building (boosting) and the idea that each round fits the previous round's residuals/negative gradients."
|
||||
|
||||
- criterion: "State that the core model ultimately outputs default probability/probability of default through binary-classification loss and logistic mapping."
|
||||
|
||||
- criterion: "Explain that XGBoost improves nonlinear fitting and interaction modeling by ensembling multiple weak learners, and point out at least one limitation such as hyperparameter-tuning complexity, overfitting risk, or high per-case explanation cost."
|
||||
|
||||
- criterion: "Identify A1 as the 'binning + generalized linear binary classification / logistic regression' route, and state that it has strong interpretability but limited nonlinear representation capability."
|
||||
|
||||
- criterion: "Identify A2 as the bagging-tree / random-forest route, and state that it is robust and noise-resistant, but its probability calibration or ranking performance is usually weaker than mainstream boosted-tree methods."
|
||||
|
||||
- criterion: "Identify A3 as a LightGBM-like route, and state that it is based on histogram splitting and leaf-wise growth, has high training efficiency, but requires attention to coupling with data distribution and drift monitoring."
|
||||
|
||||
- criterion: "Identify A4 as a CatBoost-like route, and state that it uses ordered boosting and a categorical-variable-friendly mechanism, while also noting that in the current environment its training time or resource usage is not advantageous."
|
||||
|
||||
- criterion: "Explicitly explain the modeling-logic differences between linear models and tree-based models in the comparison, rather than only listing pros and cons."
|
||||
|
||||
- criterion: "The comparison analysis should cover at least these dimensions: modeling logic, interpretability, nonlinear representation capability, training efficiency, hyperparameter tuning complexity, and online maintenance/governance cost."
|
||||
|
||||
- criterion: "The applicability analysis should cover at least both of these scenario types: high-interpretability / strong-regulation / small-sample scenarios, and large-sample / strong-nonlinearity / discrimination-oriented scenarios."
|
||||
|
||||
- criterion: "When explaining why the report leans toward A5, mention both its better performance, finer segmentation, and stronger return profile, and the fact that governance still requires supplementary parameter-behavior mapping explanations and per-case explanation mechanisms."
|
||||
|
||||
- criterion: "Output a Meta Review module that contains key assumptions, potential biases, information gaps, and next-step due diligence or monitoring recommendations, and points out at least one risk directly related to this report, such as PSI nearing warning level, field-definition drift, inconsistent freshness of relationship-network features, or historical label-leakage cleanup."
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
Binary file not shown.
|
|
@ -0,0 +1,37 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_007
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: ''
|
||||
description: ''
|
||||
task_type: val
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: "For our Enterprise Macro-Risk Data Platform, we need to stress-test\
|
||||
\ our portfolio models using simulated equity factors. Based on the Kenneth French\
|
||||
\ Data Library's 'F-F_Research_Data_5_Factors_2x3_daily.CSV.zip', please write\
|
||||
\ a Python script to process the raw file and generate at least 5 years of highly\
|
||||
\ realistic synthetic daily data. Output the final dataframe to 'synthetic_ff5_daily.csv'.\
|
||||
\ Please save the script directly into the current workspace as 'ff5_simulation.py'.\
|
||||
\ You only need to write and save the script; do not attempt to execute it, and\
|
||||
\ do not provide any text explanations or markdown reports."
|
||||
hidden_intent:
|
||||
- content: "The script must explicitly read and extract data directly from the\
|
||||
\ specified zip archive ('F-F_Research_Data_5_Factors_2x3_daily.CSV.zip')\
|
||||
\ using libraries like `zipfile`."
|
||||
- content: "Data Cleaning: The script must explicitly contain code (e.g., `skiprows`)\
|
||||
\ to drop the non-data metadata headers and copyright footers found in Kenneth\
|
||||
\ French CSVs."
|
||||
- content: "Schema Accuracy: The generated data must exclusively contain the exact\
|
||||
\ 7 fields: Date, Mkt-RF, SMB, HML, RMW, CMA, and RF."
|
||||
- content: "Advanced Simulation: The script must implement non-Gaussian methods\
|
||||
\ (like Copula, GARCH, or Block Bootstrap) to capture fat tails or volatility\
|
||||
\ clustering."
|
||||
objectives:
|
||||
files_read:
|
||||
- ff5_simulation.py
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
|
|
@ -0,0 +1,99 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_008
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: "Temporary Rehearsal Board for Beacon Beta Share"
|
||||
description: "Build a temporary rehearsal board in Todoist for the Beacon beta share, leave an auditable trail of tasks, label usage, and comment history, and then fully clean it up. Evaluation focuses on whether the tasks are specific, whether the blocker and checkpoints are explicit, and whether the temporary workspace is completely removed afterward."
|
||||
task_type: short_term_independent
|
||||
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: |-
|
||||
My Todoist login info:
|
||||
Email: stmcco@gmail.com
|
||||
Password: Ozz21Q4
|
||||
|
||||
I need to do a quick rehearsal pass tonight for the Beacon beta share setup. Please create a temporary project called "Temporary Launch Rehearsal Board - Beacon" and add a section called "Tonight Rehearsal".
|
||||
|
||||
The rehearsal needs to cover at least these three items:
|
||||
1. Walk through the share slides.
|
||||
2. Check demo device charging and whether screen recording works normally.
|
||||
3. Confirm guest badge pickup and sign-in sheet preparation.
|
||||
|
||||
The slides walkthrough is the main blocker: if it is still not finished by 2026-04-29 18:30, it needs to be escalated to the organizer immediately. There is also a fixed checkpoint at 2026-04-29 17:45, and that needs to appear in the tasking.
|
||||
|
||||
Please assign the priority task to me, add a temporary label called launch-blocker, and leave one comment that clearly states the escalation trigger. After the rehearsal pass is done, delete the temporary project, section, tasks, label, and comment.
|
||||
|
||||
Before you actually create these Todoist items, first send me a detailed text reply listing the project name, section name, task titles and descriptions, label name, and comment content you plan to create; after the actions are done, briefly tell me what you actually did.
|
||||
|
||||
hidden_intent:
|
||||
- content: "The temporary task board should stay tightly scoped to the Beacon beta share rehearsal, rather than turning into a generic collection of todos."
|
||||
- content: "Each task or comment should let a handoff recipient quickly understand the current item, the next checkpoint, and the possible blocker."
|
||||
- content: "The slides walkthrough must be clearly treated as the primary blocker, with both the 2026-04-29 17:45 checkpoint and the 2026-04-29 18:30 escalation trigger stated explicitly."
|
||||
|
||||
objectives:
|
||||
tools:
|
||||
- mcp_appworld_todoist__create_project:
|
||||
call: [name]
|
||||
result: [response.project_id]
|
||||
- mcp_appworld_todoist__create_section:
|
||||
call: [name, project_id]
|
||||
result: [response.section_id]
|
||||
- mcp_appworld_todoist__create_task:
|
||||
call: [title, project_id, section_id, description]
|
||||
result: [response.task_id]
|
||||
- mcp_appworld_todoist__assign_or_unassign_task:
|
||||
call: [task_id, assignee_email]
|
||||
result: [response.message]
|
||||
- mcp_appworld_todoist__update_task:
|
||||
call: [task_id, description]
|
||||
result: [response.message]
|
||||
- mcp_appworld_todoist__create_label:
|
||||
call: [name]
|
||||
result: [response.label_id]
|
||||
- mcp_appworld_todoist__add_label_to_task:
|
||||
call: [label_id, task_id]
|
||||
result: [response.message]
|
||||
- mcp_appworld_todoist__post_task_comment:
|
||||
call: [content, task_id]
|
||||
result: [response.task_comment_id]
|
||||
- mcp_appworld_todoist__show_task_comments:
|
||||
call: [task_id]
|
||||
result: [response]
|
||||
- mcp_appworld_todoist__remove_label_from_task:
|
||||
call: [task_id, label_id]
|
||||
result: [response.message]
|
||||
- mcp_appworld_todoist__delete_task_comment:
|
||||
call: [task_comment_id]
|
||||
result: [response.message]
|
||||
- mcp_appworld_todoist__delete_task:
|
||||
call: [task_id]
|
||||
result: [response.message]
|
||||
- mcp_appworld_todoist__delete_label:
|
||||
call: [label_id]
|
||||
result: [response.message]
|
||||
- mcp_appworld_todoist__delete_section:
|
||||
call: [section_id]
|
||||
result: [response.message]
|
||||
- mcp_appworld_todoist__delete_project:
|
||||
call: [project_id]
|
||||
result: [response.message]
|
||||
checklist:
|
||||
- criterion: "The temporary Todoist project is named exactly 'Temporary Launch Rehearsal Board - Beacon'."
|
||||
- criterion: "The temporary Todoist section is named exactly 'Tonight Rehearsal'."
|
||||
- criterion: "The temporary Todoist tasks cover the slides walkthrough, demo device charging and screen recording check, and guest badge pickup plus sign-in sheet preparation. Substantially similar wording is acceptable."
|
||||
- criterion: "The priority task clearly identifies the slides walkthrough as the main blocker. Substantially similar wording is acceptable."
|
||||
- criterion: "The priority task description or comment explicitly includes the fixed checkpoint at 2026-04-29 17:45."
|
||||
- criterion: "The priority task description or comment explicitly states that if the work is still unfinished by 2026-04-29 18:30, it must be escalated to the organizer immediately. Substantially similar wording is acceptable."
|
||||
- criterion: "The temporary label name is exactly launch-blocker."
|
||||
tools_evaluation_path: "data/Financier/scripts/Financier_task_008_todoist.py"
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
benchmark_role: test
|
||||
preference_threads:
|
||||
- audit_ready_work_tracking
|
||||
- deadline_and_followup_discipline
|
||||
- single_app_cleanup_workflow
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_009
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: ''
|
||||
description: ''
|
||||
task_type: val
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: "For our Fixed Income Analytics Platform, I want to build a US Treasury\
|
||||
\ yield curve modeling prototype. I prioritize high interpretability, ease of\
|
||||
\ implementation, and fast Python deployment over complex theoretical perfection.\
|
||||
\ I have uploaded two foundational papers: 'w1594.pdf' (Nelson-Siegel) and 'w4871.pdf'\
|
||||
\ (Svensson). Please provide a concise comparison of these two models and conclude\
|
||||
\ by explicitly recommending the better model for my specific prototype requirements.\
|
||||
\ Keep it conclusion-first without heavy math."
|
||||
hidden_intent:
|
||||
- content: "Model Recommendation: The response must explicitly recommend the Nelson-Siegel\
|
||||
\ (NS) model over Svensson for the initial prototype due to the user's preference\
|
||||
\ for simplicity."
|
||||
- content: "Parameter Awareness: The comparison must correctly state that the\
|
||||
\ Nelson-Siegel model uses 4 parameters while the Svensson extension uses\
|
||||
\ 6."
|
||||
- content: "The response must strictly obey the 'without heavy math' constraint\
|
||||
\ by completely avoiding the explicit writing of algebraic yield curve equations/formulas\
|
||||
\ (e.g., exponential functions), explaining the parameters conceptually instead."
|
||||
- content: "The response must explicitly identify that the Svensson model's increased\
|
||||
\ flexibility introduces severe calibration difficulties (such as high sensitivity\
|
||||
\ to starting values or constraints), which directly violates the user's 'ease\
|
||||
\ of implementation' requirement."
|
||||
metadata:
|
||||
difficulty: medium
|
||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,261 @@
|
|||
"""
|
||||
Independent validation metrics computation for fixed_german_credit_pd_scorecard.
|
||||
All metrics computed from raw data (german.data), split file (german_credit_split.csv), and model spec.
|
||||
"""
|
||||
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from sklearn.metrics import roc_auc_score
|
||||
import yaml
|
||||
import math
|
||||
|
||||
BASE = "/Users/wce/Downloads/pi-bench/data/Financier/tasks/Financier_task_010"
|
||||
|
||||
# 1. Load raw data
|
||||
columns = [
|
||||
'checking_status', 'duration', 'credit_history', 'purpose', 'credit_amount',
|
||||
'savings_status', 'employment', 'installment_commitment', 'personal_status',
|
||||
'other_parties', 'residence_since', 'property_magnitude', 'age',
|
||||
'other_payment_plans', 'housing', 'existing_credits', 'job',
|
||||
'num_dependents', 'own_telephone', 'foreign_worker', 'target'
|
||||
]
|
||||
|
||||
data = pd.read_csv(f"{BASE}/german.data", sep='\s+', header=None, names=columns)
|
||||
data['bad'] = (data['target'] == 2).astype(int)
|
||||
data['row_id'] = range(1, len(data) + 1)
|
||||
|
||||
# 2. Load split
|
||||
split = pd.read_csv(f"{BASE}/german_credit_split.csv")
|
||||
data = data.merge(split, on='row_id', how='left')
|
||||
train = data[data['split'] == 'train'].copy()
|
||||
test = data[data['split'] == 'test'].copy()
|
||||
|
||||
print(f"Full: N={len(data)}, Bad rate={data['bad'].mean():.4f}")
|
||||
print(f"Train: N={len(train)}, Bad rate={train['bad'].mean():.4f}")
|
||||
print(f"Test: N={len(test)}, Bad rate={test['bad'].mean():.4f}")
|
||||
|
||||
# 3. Compute PD scores per model spec
|
||||
def compute_pd(row):
|
||||
# Numeric terms
|
||||
duration_term = 0.45 * ((row['duration'] - 24) / 12)
|
||||
credit_amount_term = 0.25 * (math.log(row['credit_amount']) - math.log(3000))
|
||||
age_term = -0.18 * ((row['age'] - 35) / 10)
|
||||
|
||||
# Categorical terms
|
||||
checking_map = {'A11': 0.90, 'A12': 0.45, 'A13': 0.10, 'A14': -0.45}
|
||||
credit_history_map = {'A30': 0.70, 'A31': 0.45, 'A32': 0.05, 'A33': -0.15, 'A34': -0.40}
|
||||
savings_map = {'A61': 0.35, 'A62': 0.20, 'A63': 0.00, 'A64': -0.25, 'A65': -0.15}
|
||||
employment_map = {'A71': 0.35, 'A72': 0.20, 'A73': 0.05, 'A74': -0.10, 'A75': -0.20}
|
||||
property_map = {'A121': -0.10, 'A122': 0.05, 'A123': 0.20, 'A124': 0.35}
|
||||
parties_map = {'A101': 0.10, 'A102': -0.10, 'A103': -0.20}
|
||||
housing_map = {'A151': 0.25, 'A152': 0.00, 'A153': 0.15}
|
||||
|
||||
logit = -1.05
|
||||
logit += duration_term + credit_amount_term + age_term
|
||||
logit += checking_map.get(row['checking_status'], 0)
|
||||
logit += credit_history_map.get(row['credit_history'], 0)
|
||||
logit += savings_map.get(row['savings_status'], 0)
|
||||
logit += employment_map.get(row['employment'], 0)
|
||||
logit += property_map.get(row['property_magnitude'], 0)
|
||||
logit += parties_map.get(row['other_parties'], 0)
|
||||
logit += housing_map.get(row['housing'], 0)
|
||||
|
||||
pd_score = 1 / (1 + math.exp(-logit))
|
||||
return pd_score
|
||||
|
||||
train['pd'] = train.apply(compute_pd, axis=1)
|
||||
test['pd'] = test.apply(compute_pd, axis=1)
|
||||
|
||||
# 4. AUC and KS
|
||||
train_auc = roc_auc_score(train['bad'], train['pd'])
|
||||
test_auc = roc_auc_score(test['bad'], test['pd'])
|
||||
|
||||
def compute_ks(y_true, y_pred):
|
||||
# Sort by predicted score
|
||||
sorted_data = sorted(zip(y_pred, y_true), key=lambda x: x[0])
|
||||
n = len(sorted_data)
|
||||
n_pos = sum(y_true)
|
||||
n_neg = n - n_pos
|
||||
|
||||
ks_max = 0
|
||||
tp = 0
|
||||
fp = 0
|
||||
|
||||
for i, (pred, actual) in enumerate(sorted_data):
|
||||
if actual == 1:
|
||||
tp += 1
|
||||
else:
|
||||
fp += 1
|
||||
tpr = tp / n_pos if n_pos > 0 else 0
|
||||
fpr = fp / n_neg if n_neg > 0 else 0
|
||||
ks = abs(tpr - fpr)
|
||||
ks_max = max(ks_max, ks)
|
||||
|
||||
return ks_max
|
||||
|
||||
train_ks = compute_ks(train['bad'].values, train['pd'].values)
|
||||
test_ks = compute_ks(test['bad'].values, test['pd'].values)
|
||||
|
||||
print(f"\nTrain AUC: {train_auc:.4f}")
|
||||
print(f"Test AUC: {test_auc:.4f}")
|
||||
print(f"Train KS: {train_ks:.4f}")
|
||||
print(f"Test KS: {test_ks:.4f}")
|
||||
|
||||
# 5. Brier score (test)
|
||||
test_brier = np.mean((test['pd'].values - test['bad'].values) ** 2)
|
||||
print(f"Test Brier: {test_brier:.4f}")
|
||||
|
||||
# 6. Confusion matrix at 0.35
|
||||
test_pred = (test['pd'] >= 0.35).astype(int)
|
||||
tp = ((test_pred == 1) & (test['bad'] == 1)).sum()
|
||||
fp = ((test_pred == 1) & (test['bad'] == 0)).sum()
|
||||
tn = ((test_pred == 0) & (test['bad'] == 0)).sum()
|
||||
fn = ((test_pred == 0) & (test['bad'] == 1)).sum()
|
||||
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
|
||||
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
|
||||
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
|
||||
print(f"\n@0.35 cutoff: TP={tp}, FP={fp}, TN={tn}, FN={fn}")
|
||||
print(f"Precision@0.35: {precision:.4f} ({precision*100:.1f}%)")
|
||||
print(f"Recall@0.35: {recall:.4f}")
|
||||
print(f"F1@0.35: {f1:.4f}")
|
||||
|
||||
# Also @0.30 and @0.40
|
||||
for cutoff in [0.30, 0.40]:
|
||||
pred_c = (test['pd'] >= cutoff).astype(int)
|
||||
tp_c = ((pred_c == 1) & (test['bad'] == 1)).sum()
|
||||
fp_c = ((pred_c == 1) & (test['bad'] == 0)).sum()
|
||||
prec_c = tp_c / (tp_c + fp_c) if (tp_c + fp_c) > 0 else 0
|
||||
print(f"@{cutoff} cutoff: BR={(tp_c/(tp_c+fp_c))*100:.1f}% (precision={prec_c:.4f})")
|
||||
|
||||
# 7. Quintile bad rates (full sample)
|
||||
full = data.copy()
|
||||
full['pd'] = full.apply(compute_pd, axis=1)
|
||||
full['quintile'] = pd.qcut(full['pd'], 5, labels=['Q1', 'Q2', 'Q3', 'Q4', 'Q5'], duplicates='drop')
|
||||
quintile_stats = full.groupby('quintile').agg(
|
||||
n=('bad', 'count'),
|
||||
bad_count=('bad', 'sum'),
|
||||
bad_rate=('bad', 'mean'),
|
||||
pd_min=('pd', 'min'),
|
||||
pd_max=('pd', 'max'),
|
||||
pd_mean=('pd', 'mean')
|
||||
).reset_index()
|
||||
print(f"\nQuintile bad rates (full sample):")
|
||||
for _, row in quintile_stats.iterrows():
|
||||
print(f" {row['quintile']}: N={row['n']}, Bad={row['bad_count']}, BadRate={row['bad_rate']:.4f}, PD=[{row['pd_min']:.4f}, {row['pd_max']:.4f}]")
|
||||
|
||||
# Test quintile bad rates
|
||||
test['quintile'] = pd.qcut(test['pd'], 5, labels=['Q1', 'Q2', 'Q3', 'Q4', 'Q5'], duplicates='drop')
|
||||
test_quintile = test.groupby('quintile').agg(
|
||||
n=('bad', 'count'),
|
||||
bad_count=('bad', 'sum'),
|
||||
bad_rate=('bad', 'mean'),
|
||||
pd_min=('pd', 'min'),
|
||||
pd_max=('pd', 'max')
|
||||
).reset_index()
|
||||
print(f"\nTest quintile bad rates:")
|
||||
for _, row in test_quintile.iterrows():
|
||||
print(f" {row['quintile']}: N={row['n']}, Bad={row['bad_count']}, BadRate={row['bad_rate']:.4f}, PD=[{row['pd_min']:.4f}, {row['pd_max']:.4f}]")
|
||||
|
||||
# Check Q4 vs Q5 on test
|
||||
q4_data = test[test['quintile'] == 'Q4']
|
||||
q5_data = test[test['quintile'] == 'Q5']
|
||||
print(f"\nQ4 on test: N={len(q4_data)}, Bad={q4_data['bad'].sum()}, BadRate={q4_data['bad'].mean():.4f}")
|
||||
print(f"Q5 on test: N={len(q5_data)}, Bad={q5_data['bad'].sum()}, BadRate={q5_data['bad'].mean():.4f}")
|
||||
|
||||
# 8. Score PSI (train -> test, using train decile bins)
|
||||
train_deciles = pd.qcut(train['pd'], 10, duplicates='drop')
|
||||
train_bins = train_deciles.copy()
|
||||
train_bin_edges = pd.qcut(train['pd'], 10, duplicates='drop', retbins=True)[1]
|
||||
|
||||
# Apply train decile cut points to test
|
||||
test_bin_labels = pd.cut(test['pd'], bins=train_bin_edges, include_lowest=True, labels=False)
|
||||
train_bin_labels = pd.cut(train['pd'], bins=train_bin_edges, include_lowest=True, labels=False)
|
||||
|
||||
# PSI computation
|
||||
def compute_psi(train_labels, test_labels, n_bins, epsilon=1e-6):
|
||||
train_prop = np.zeros(n_bins)
|
||||
test_prop = np.zeros(n_bins)
|
||||
|
||||
for i in range(n_bins):
|
||||
train_prop[i] = (train_labels == i).sum() / len(train_labels) + epsilon
|
||||
test_prop[i] = (test_labels == i).sum() / len(test_labels) + epsilon
|
||||
|
||||
psi = np.sum((test_prop - train_prop) * np.log(test_prop / train_prop))
|
||||
return psi
|
||||
|
||||
n_score_bins = len(train_bin_edges) - 1
|
||||
score_psi = compute_psi(train_bin_labels, test_bin_labels, n_score_bins)
|
||||
print(f"\nScore PSI: {score_psi:.4f}")
|
||||
|
||||
# 9. checking_status PSI
|
||||
def categorical_psi(train_series, test_series, epsilon=1e-6):
|
||||
all_cats = sorted(set(train_series.unique()) | set(test_series.unique()))
|
||||
train_props = {}
|
||||
test_props = {}
|
||||
for cat in all_cats:
|
||||
train_props[cat] = (train_series == cat).sum() / len(train_series) + epsilon
|
||||
test_props[cat] = (test_series == cat).sum() / len(test_series) + epsilon
|
||||
|
||||
psi = sum((test_props[c] - train_props[c]) * np.log(test_props[c] / train_props[c]) for c in all_cats)
|
||||
return psi
|
||||
|
||||
checking_psi = categorical_psi(train['checking_status'], test['checking_status'])
|
||||
print(f"checking_status PSI: {checking_psi:.4f}")
|
||||
|
||||
# A13 bad rate train vs test
|
||||
a13_train = train[train['checking_status'] == 'A13']
|
||||
a13_test = test[test['checking_status'] == 'A13']
|
||||
print(f"A13 train: N={len(a13_train)}, BadRate={a13_train['bad'].mean():.4f}")
|
||||
print(f"A13 test: N={len(a13_test)}, BadRate={a13_test['bad'].mean():.4f}")
|
||||
|
||||
# 10. credit_amount band PSI
|
||||
def credit_amount_band(val):
|
||||
if val <= 1000:
|
||||
return '<=1000'
|
||||
elif val <= 2500:
|
||||
return '1001-2500'
|
||||
elif val <= 5000:
|
||||
return '2501-5000'
|
||||
else:
|
||||
return '>5000'
|
||||
|
||||
train['ca_band'] = train['credit_amount'].apply(credit_amount_band)
|
||||
test['ca_band'] = test['credit_amount'].apply(credit_amount_band)
|
||||
ca_psi = categorical_psi(train['ca_band'], test['ca_band'])
|
||||
print(f"credit_amount band PSI: {ca_psi:.4f}")
|
||||
|
||||
# 11. Excluded features
|
||||
all_features = set(columns) - {'target', 'row_id', 'bad', 'split', 'pd', 'quintile', 'ca_band'}
|
||||
model_features = {'checking_status', 'duration', 'credit_history', 'credit_amount', 'savings_status',
|
||||
'employment', 'property_magnitude', 'other_parties', 'housing', 'age'}
|
||||
excluded = all_features - model_features
|
||||
print(f"\nExcluded features ({len(excluded)}): {sorted(excluded)}")
|
||||
|
||||
# 12. Mean PD over-prediction
|
||||
test_mean_pd = test['pd'].mean()
|
||||
test_observed_br = test['bad'].mean()
|
||||
print(f"\nTest mean PD: {test_mean_pd:.4f}")
|
||||
print(f"Test observed bad rate: {test_observed_br:.4f}")
|
||||
print(f"Over-prediction: {(test_mean_pd - test_observed_br)*100:.1f} pp")
|
||||
|
||||
# 13. Verify all categorical codes in spec are covered by data
|
||||
spec_cats = {
|
||||
'checking_status': ['A11', 'A12', 'A13', 'A14'],
|
||||
'credit_history': ['A30', 'A31', 'A32', 'A33', 'A34'],
|
||||
'savings_status': ['A61', 'A62', 'A63', 'A64', 'A65'],
|
||||
'employment': ['A71', 'A72', 'A73', 'A74', 'A75'],
|
||||
'property_magnitude': ['A121', 'A122', 'A123', 'A124'],
|
||||
'other_parties': ['A101', 'A102', 'A103'],
|
||||
'housing': ['A151', 'A152', 'A153']
|
||||
}
|
||||
|
||||
for feat, codes in spec_cats.items():
|
||||
data_codes = set(data[feat].unique())
|
||||
spec_codes = set(codes)
|
||||
missing = spec_codes - data_codes
|
||||
if missing:
|
||||
print(f" WARNING: {feat} missing codes in data: {missing}")
|
||||
else:
|
||||
print(f" {feat}: all spec codes covered")
|
||||
|
||||
print("\n=== ALL METRICS COMPUTED ===")
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,42 @@
|
|||
# German Credit Raw Data Schema
|
||||
|
||||
Source URL:
|
||||
- https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data
|
||||
|
||||
Local raw file:
|
||||
- `german.data`
|
||||
|
||||
The local file is the original whitespace-delimited Statlog German Credit data file without a header row.
|
||||
|
||||
Column order:
|
||||
1. checking_status
|
||||
2. duration
|
||||
3. credit_history
|
||||
4. purpose
|
||||
5. credit_amount
|
||||
6. savings_status
|
||||
7. employment
|
||||
8. installment_commitment
|
||||
9. personal_status
|
||||
10. other_parties
|
||||
11. residence_since
|
||||
12. property_magnitude
|
||||
13. age
|
||||
14. other_payment_plans
|
||||
15. housing
|
||||
16. existing_credits
|
||||
17. job
|
||||
18. num_dependents
|
||||
19. own_telephone
|
||||
20. foreign_worker
|
||||
21. target
|
||||
|
||||
Target coding for this task:
|
||||
- `target = 1` means good credit.
|
||||
- `target = 2` means bad credit.
|
||||
- For validation metrics, define `bad = 1` when `target = 2`, otherwise `bad = 0`.
|
||||
|
||||
Dataset background to verify from the public UCI page:
|
||||
- 1,000 rows.
|
||||
- 20 input attributes plus the target label.
|
||||
- This is a benchmark dataset, not evidence by itself that a production applicant population is represented.
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,79 @@
|
|||
model_name: fixed_german_credit_pd_scorecard
|
||||
model_type: logistic_regression_scorecard
|
||||
raw_data_file: german.data
|
||||
schema_file: german_credit_schema.md
|
||||
split_file: german_credit_split.csv
|
||||
split_rule: use the explicit row_id to split mapping in german_credit_split.csv
|
||||
target:
|
||||
bad_label: 2
|
||||
good_label: 1
|
||||
threshold:
|
||||
manual_review_pd_cutoff: 0.35
|
||||
score_formula:
|
||||
logit_pd: intercept + sum(feature_contributions)
|
||||
intercept: -1.05
|
||||
numeric_terms:
|
||||
duration: 0.45 * ((duration - 24) / 12)
|
||||
credit_amount: 0.25 * (ln(credit_amount) - ln(3000))
|
||||
age: -0.18 * ((age - 35) / 10)
|
||||
categorical_terms:
|
||||
checking_status:
|
||||
A11: 0.90
|
||||
A12: 0.45
|
||||
A13: 0.10
|
||||
A14: -0.45
|
||||
credit_history:
|
||||
A30: 0.70
|
||||
A31: 0.45
|
||||
A32: 0.05
|
||||
A33: -0.15
|
||||
A34: -0.40
|
||||
savings_status:
|
||||
A61: 0.35
|
||||
A62: 0.20
|
||||
A63: 0.00
|
||||
A64: -0.25
|
||||
A65: -0.15
|
||||
employment:
|
||||
A71: 0.35
|
||||
A72: 0.20
|
||||
A73: 0.05
|
||||
A74: -0.10
|
||||
A75: -0.20
|
||||
property_magnitude:
|
||||
A121: -0.10
|
||||
A122: 0.05
|
||||
A123: 0.20
|
||||
A124: 0.35
|
||||
other_parties:
|
||||
A101: 0.10
|
||||
A102: -0.10
|
||||
A103: -0.20
|
||||
housing:
|
||||
A151: 0.25
|
||||
A152: 0.00
|
||||
A153: 0.15
|
||||
metrics_to_compute:
|
||||
- row counts and bad rates for full, train, and test
|
||||
- train and test AUC
|
||||
- train and test KS
|
||||
- test confusion matrix at PD cutoff 0.35
|
||||
- full-sample high-risk and low-risk quintile bad rates by predicted PD
|
||||
- score PSI comparing train and test, using train predicted-PD decile bins
|
||||
- checking_status PSI comparing train and test
|
||||
- credit_amount band PSI comparing train and test
|
||||
psi_rules:
|
||||
epsilon: 0.000001
|
||||
score_bins: use train predicted-PD decile cut points, then apply the same cut points to test
|
||||
credit_amount_bands:
|
||||
- <=1000
|
||||
- 1001-2500
|
||||
- 2501-5000
|
||||
- >5000
|
||||
binning_checks:
|
||||
credit_amount_high_balance_bin: credit_amount > 10000
|
||||
age_bins:
|
||||
- <25
|
||||
- 25-34
|
||||
- 35-49
|
||||
- 50+
|
||||
|
|
@ -0,0 +1,186 @@
|
|||
# Independent Validation Report — Fixed German Credit PD Scorecard
|
||||
|
||||
**Model:** fixed_german_credit_pd_scorecard (logistic-regression scorecard)
|
||||
**Validation Date:** 2026-05-01
|
||||
**Validation Framework Anchor:** Federal Reserve SR 11-7 *Supervisory Guidance on Model Risk Management* (2011)
|
||||
**Validation Conduct:** Independent of model development team
|
||||
**Data Source:** UCI Statlog German Credit Data (https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data)
|
||||
**SR 11-7 Reference:** https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm
|
||||
|
||||
---
|
||||
|
||||
## Validation Decision
|
||||
|
||||
**CONDITIONAL PASS — Benchmark-Only; Production Deployment Requires Recalibration on Representative Data**
|
||||
|
||||
The model meets minimum SR 11-7 thresholds for discriminatory power and population stability on the German Credit benchmark data. However, all metrics reported herein are derived exclusively from the UCI Statlog German Credit dataset — a static academic benchmark — and do not constitute evidence of model performance on any production or intended-use population. SR 11-7 requires that validation data be representative of the population for which the model will be used; this condition is not met by the benchmark alone.
|
||||
|
||||
Four HIGH-priority findings and two MEDIUM-priority findings require remediation before broader deployment. Three LOW-priority observations are documented for completeness. The most consequential risks are: (i) **all performance metrics are benchmark-only and do not transfer to any production population** — the model has not been validated on data representative of its intended use; (ii) test AUC of 0.7106 with 12 pp deterioration from train (0.8069) signals non-trivial out-of-sample degradation even within the benchmark; (iii) test KS of 0.3630, while above the retirement threshold, has narrowed substantially from the train value of 0.5001; (iv) the PD threshold of 0.35 is unanchored from any business or regulatory calibration; and (v) the scorecard cannot rank-order risk above the 60th percentile (Q4 and Q5 both produce 50.0% bad rates on test). Until remediation — including recalibration on representative data — is complete, use is restricted to supervised decision-support only.
|
||||
|
||||
**Independently Computed Metrics Summary (Benchmark-Only — Not Evidence of Production Performance)**
|
||||
|
||||
| Metric | Train | Test | Assessment |
|
||||
|--------|-------|------|------------|
|
||||
| AUC | 0.8069 | 0.7106 | 12 pp deterioration; test still adequate on benchmark |
|
||||
| KS statistic | 0.5001 | 0.3630 | Test KS above 0.30 threshold but narrowed on benchmark |
|
||||
| Brier score | — | 0.1998 | Exceeds 0.18 recalibration trigger |
|
||||
| Score PSI (train→test) | — | 0.0492 | At monitoring escalation level (>0.05) |
|
||||
| Precision @ 0.35 | — | 46.4% | Fewer than half of flagged apps are bad |
|
||||
| Q4 vs Q5 bad rate (test) | — | 50.0% vs 50.0% | Rank-order collapse in upper tail |
|
||||
| Excluded features | 10 | — | Underestimated in original spec documentation |
|
||||
|
||||
*All metrics above are computed on the UCI German Credit benchmark (n=1,000). They provide evidence of model behavior on a static academic dataset only and do not establish performance on any intended-use population.*
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### HIGH Priority — Must Resolve Before Deployment
|
||||
|
||||
**H1 · Model trained and validated exclusively on non-representative benchmark data — no evidence of performance on intended-use population**
|
||||
*SR 11-7 Principle: Data representativeness — model development and validation data must be representative of the population for which the model will be used.*
|
||||
The scorecard was developed on, and all validation metrics reported herein are derived from, the UCI Statlog German Credit dataset (n=1,000). This is a static academic benchmark from a single German bank in the 1990s. It does not represent any current applicant population, credit product mix, or economic environment. No recalibration, bridging analysis, or population-comparison exercise has been performed on representative current data. All performance metrics — AUC, KS, Brier, PSI, calibration — are conditional on this single benchmark and should not be interpreted as predictive of production performance.
|
||||
*Model-use consequence:* Deploying this model on any population other than the benchmark constitutes a model-use gap that SR 11-7 explicitly warns against: a model validated on one population cannot be assumed valid for another without bridging evidence. Risk-ranking, PD calibration, and threshold behavior may differ materially on a production portfolio, rendering all benchmark metrics moot for production decision-making.
|
||||
*Remediation:* Recalibrate the scorecard on a representative sample reflecting the intended portfolio's credit product mix, demographics, and economic environment. Re-run all validation metrics (AUC, KS, Brier, PSI, calibration) on the representative sample before any production use.
|
||||
|
||||
**H2 · Interaction terms and goodness-of-fit not validated**
|
||||
*SR 11-7 Principle: Conceptual soundness — model theory and assumptions must be appropriate for intended use.*
|
||||
The scorecard is a fixed additive logistic-regression formula with no documented interaction terms. Known credit-risk interactions — duration × credit_amount, age × employment — are absent from the specification. No Hosmer-Lemeshow or le Cessie-van Houwelingen goodness-of-fit test has been run on any partition. Without interaction controls, the model can assign systematically inflated PDs to thin-file young applicants with high credit amounts, a segment that also shows test-PSI instability.
|
||||
*Model-use consequence:* If used for automated approval without human review, high-risk sub-populations may receive systematically biased PDs, leading to adverse selection or discriminatory outcomes. SR 11-7 requires that model assumptions be tested, not assumed.
|
||||
*Remediation:* Run H-L test (10 decile groups) on train and test; if p < 0.05 on either partition, refit with interaction terms for duration × credit_amount and age × employment.
|
||||
|
||||
**H3 · PD threshold of 0.35 is unanchored from business or regulatory calibration**
|
||||
*SR 11-7 Principle: Appropriate methodology — model design must match intended use.*
|
||||
The cutoff for manual-review escalation has no documented derivation. No calibration curve, no benchmark against portfolio default rates, and no precision/recall tradeoff analysis at alternative cutoffs is provided. On test, at the chosen cutoff: precision = 46.4% (fewer than half of flagged applications are actually bad). At cutoff 0.30, precision = 43.6%; at 0.40, precision = 50.0%. No cost-of-error analysis or regulatory expected-loss table is referenced. The test Brier score of 0.1998 exceeds the 0.18 recalibration threshold, indicating probability calibration concerns that directly affect threshold reliability.
|
||||
*Model-use consequence:* An unanchored threshold undermines the SR 11-7 requirement for ongoing monitoring effectiveness — if the cutoff itself is arbitrary, breach triggers derived from it lack defensibility.
|
||||
*Remediation:* Document threshold rationale — portfolio bad-rate matching, F1 optimization, or regulatory capital grid — and retest at cutoffs 0.30 and 0.40 with full cost-of-error analysis.
|
||||
|
||||
**H4 · AUC deteriorates 12 pp (0.8069→0.7106) and KS narrows from 0.5001 to 0.3630 — rank-order discrimination degrades on unseen data**
|
||||
*SR 11-7 Principle: Outcomes analysis — out-of-sample performance must be demonstrated; performance deterioration is a model risk signal.*
|
||||
Train KS of 0.5001 is strong; test KS of 0.3630 represents a 27% narrowing. Train AUC of 0.8069 to test AUC of 0.7106 is a 12 pp deterioration. While test values remain above minimum thresholds, the magnitude of change signals sensitivity to data distribution shifts. Contributing factors: sparse-bin instability (A30, N=40; A103, N=52 in test), interaction misspecification, and possible training-partition overfitting.
|
||||
*Model-use consequence:* The model's risk-ranking reliability is materially weaker in production-like data than in training. Decisions near the approval/decline boundary carry higher misclassification risk than the train metrics suggest.
|
||||
*Remediation:* Bootstrap cross-validation (100 × 70/30); flag if 5th-percentile KS falls below 0.30. Require recalibration if test AUC falls below 0.68 on any future sample.
|
||||
|
||||
---
|
||||
|
||||
### MEDIUM Priority — Must Resolve Before Production Deployment
|
||||
|
||||
**M1 · Calibration evidence is absent; test Brier of 0.1998 exceeds recalibration threshold**
|
||||
*SR 11-7 Principle: Outcomes analysis — predicted probabilities must be aligned with observed outcomes.*
|
||||
No reliability diagram, Brier score, or expected-to-observed default ratio by decile has been performed or referenced. The independently computed test Brier score of 0.1998 exceeds the 0.18 recalibration threshold, indicating that predicted PDs deviate meaningfully from observed outcomes. Mean PD on test (0.3756) over-predicts the observed bad rate (0.3133) by 6.2 pp. Without calibration, the scorecard cannot be used for regulatory capital, expected loss, or pricing reserves.
|
||||
*Model-use consequence:* PDs used directly for capital or provisioning will systematically overstate risk by approximately 6 pp, potentially leading to excess capital allocation or overly conservative lending.
|
||||
*Remediation:* Compute Brier score and calibration curve on train and test. Recalibrate via Platt scaling or isotonic regression if Brier > 0.18 on test (currently met).
|
||||
|
||||
**M2 · Aggregate PSI masks directional category instability in A13 checking_status**
|
||||
*SR 11-7 Principle: Ongoing monitoring — stability assessment must identify sub-population shifts, not only aggregate shifts.*
|
||||
Score PSI = 0.0492, checking_status PSI = 0.0075, credit_amount band PSI = 0.0252 — all below the 0.10 threshold. However, A13 shows a 13 pp bad-rate increase from train (17.5%) to test (30.4%) on only 23 test observations, which is masked by aggregate PSI. The PSI masking of a directionally worsening minority category is itself a finding.
|
||||
*Model-use consequence:* Portfolio composition shifts affecting checking_status A13 may not be detected by score-level PSI monitoring, allowing sub-population risk drift to go unnoticed.
|
||||
*Remediation:* Track single-variable PSI on all key features in production; escalate if any individual-variable PSI exceeds 0.05.
|
||||
|
||||
---
|
||||
|
||||
### LOW Priority — Addressable Over Time
|
||||
|
||||
The following observations are documented for completeness but do not independently impede conditional approval:
|
||||
|
||||
**L1 · Ten features excluded without documented rationale**
|
||||
The model uses 10 of 20 available input features. Excluded features are: existing_credits, foreign_worker, installment_commitment, job, num_dependents, other_payment_plans, own_telephone, personal_status, purpose, residence_since. No documented feature-selection methodology (e.g., IV analysis, stepwise selection) supports these exclusions. However, no evidence of material omitted-variable bias was detected in the available data.
|
||||
|
||||
**L2 · Credit_amount log-transform parameters not documented**
|
||||
The centering constant of 3,000 and coefficient of 0.25 in the credit_amount term lack documented derivation. No comparison to linear or piecewise alternatives has been performed. However, the direction of the term (positive coefficient on log amount) is economically sound.
|
||||
|
||||
**L3 · No sensitivity or stress testing documented**
|
||||
No boundary-condition testing is documented for credit_amount > 10,000 (N=40, 4% of sample) or age > 60. SR 11-7 recommends sensitivity analysis for extrapolation regions, but the small exposure in these regions limits the practical impact.
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
1. **Not a production portfolio.** The UCI German Credit dataset is a static 1,000-observation academic benchmark from a single German bank (1990s vintage). It does not represent any current applicant population. SR 11-7 requires data representative of the intended use population; deploying without recalibration on representative current data would be a fundamental misuse.
|
||||
|
||||
2. **No temporal dimension.** All 1,000 observations lack a date field, making economic-cycle and time-series stability testing impossible. Static-benchmark validation is necessary but not sufficient for SR 11-7 compliance.
|
||||
|
||||
3. **Sparse minority bins.** Test N=300 yields wide binomial confidence intervals on subgroup metrics. A30 (N=40), A103 (N=52), A13 (N=23) have confidence bounds of approximately ±8–15%. Decisions on applicants mapped primarily to these bins require expert review.
|
||||
|
||||
4. **Credit_amount log-transform parameters not documented.** The centering constant 3,000 and coefficient 0.25 lack documented derivation. No comparison to linear or piecewise alternatives has been performed.
|
||||
|
||||
5. **No sensitivity or stress testing documented.** Boundary conditions (credit_amount > 10,000, N=40, 4%; age > 60, N≈30) are in extrapolation regions where SR 11-7 requires documented sensitivity analysis.
|
||||
|
||||
6. **Excluded variables not justified.** Ten features — purpose (10 categories), other_payment_plans, own_telephone, installment_commitment, residence_since, existing_credits, foreign_worker, job, num_dependents, personal_status — are excluded without documented rationale, creating potential omitted-variable bias.
|
||||
|
||||
7. **Score PSI already at monitoring escalation level.** The independently computed score PSI of 0.0492 between train and test is at the >0.05 escalation trigger, indicating that the model's score distribution shifts materially even between the benchmark's own train and test partitions.
|
||||
|
||||
8. **Mean PD over-predicts observed bad rate.** On the test set, mean predicted PD of 0.3756 exceeds the observed bad rate of 0.3133 by 6.2 pp. This systematic over-prediction is consistent with the test Brier score of 0.1998 exceeding the 0.18 threshold.
|
||||
|
||||
---
|
||||
|
||||
## Use Restrictions And Remediation Actions
|
||||
|
||||
Each validation finding and material limitation is mapped to a binding use restriction (what the model may not be used for until the restriction is lifted) and a remediation action (what must be completed to lift it). Restrictions are prioritized by finding severity.
|
||||
|
||||
### HIGH-Priority Restrictions — Must Be Satisfied Before Any Deployment
|
||||
|
||||
| # | Linked Finding | Use Restriction | Remediation Action | Owner |
|
||||
|---|---------------|----------------|--------------------|-------|
|
||||
| UR-1 | H1 · Non-representative benchmark data | Model may not be deployed on any production portfolio or applicant population other than the German Credit benchmark without recalibration and re-validation on representative data. All benchmark metrics (AUC, KS, Brier, PSI) are conditional on the benchmark and do not establish production validity. | Recalibrate the scorecard on a representative sample reflecting the intended portfolio's credit product mix, demographics, and economic environment. Re-run all validation metrics (AUC, KS, Brier, PSI, calibration) on the representative sample. | Model Risk / Validation |
|
||||
| UR-2 | H2 · Interaction terms and GoF not validated | Model may not be used for fully automated approval/decline decisions without human review, due to risk of systematically biased PDs for sub-populations with interaction effects (e.g., thin-file young applicants with high credit amounts). | Run Hosmer-Lemeshow test (10 decile groups) on train and test partitions. If p < 0.05 on either partition, refit with interaction terms for duration × credit_amount and age × employment and re-validate. | Model Development |
|
||||
| UR-3 | H3 · PD threshold unanchored | The PD threshold of 0.35 may not be used as a binding automated decision cutoff without documented business or regulatory justification. Any use of this threshold for capital, pricing, or automated escalation is prohibited until anchored. | Document threshold rationale (portfolio bad-rate matching, F1 optimization, or regulatory capital grid). Retest at cutoffs 0.30 and 0.40 with full cost-of-error analysis. Anchor to a documented decision-policy framework. | Model Risk / Business |
|
||||
| UR-4 | H4 · AUC/KS deterioration | Automated decisions for applicants near the approval/decline boundary (PD 0.30–0.40) require enhanced human review. The model's risk-ranking reliability on unseen data is materially weaker than train metrics suggest. | Perform bootstrap cross-validation (100 × 70/30 splits). Flag model if 5th-percentile KS falls below 0.30. Require recalibration if test AUC falls below 0.68 on any future sample. | Model Risk / Validation |
|
||||
|
||||
### MEDIUM-Priority Restrictions — Must Be Satisfied Before Production Deployment
|
||||
|
||||
| # | Linked Finding | Use Restriction | Remediation Action | Owner |
|
||||
|---|---------------|----------------|--------------------|-------|
|
||||
| UR-5 | M1 · Calibration absent / Brier = 0.1998 | PD outputs may not be used for regulatory capital, expected loss, provisioning, or pricing calculations. The test Brier score of 0.1998 exceeds the 0.18 recalibration threshold; PDs over-predict observed bad rates by 6.2 pp on test. | Compute Brier score and calibration curve on train and test. Recalibrate via Platt scaling or isotonic regression (Brier > 0.18 threshold already breached on test). | Model Risk / Validation |
|
||||
| UR-6 | M2 · PSI masks A13 instability | Score-level PSI monitoring alone is insufficient as an early-warning indicator. Applications where checking_status = A13 require individual risk review, as the 13 pp bad-rate increase (17.5% → 30.4%) is masked by aggregate PSI. | Track single-variable PSI on all key features in production. Escalate if any individual-variable PSI exceeds 0.05. | Model Risk / Monitoring |
|
||||
|
||||
### LOW-Priority Restrictions — Addressable Over Time
|
||||
|
||||
| # | Linked Finding | Use Restriction | Remediation Action | Owner |
|
||||
|---|---------------|----------------|--------------------|-------|
|
||||
| UR-7 | L3 · No stress testing documented | Applications with credit_amount > 10,000 or age > 60 require enhanced human review, as model behavior in these extrapolation regions is undocumented. | Conduct stress testing on boundary regions (credit_amount > 10,000, N=40; age > 60) to document model behavior under extrapolation. | Model Development |
|
||||
| UR-8 | L1 · 10 excluded features without rationale | No deployment restriction. However, any future model iteration that adds or removes features requires a new validation cycle. | Document rationale for excluding all 10 unused features. Re-evaluate any with bivariate IV > 0.1 on the current portfolio. | Model Development |
|
||||
| UR-9 | L2 · Log-transform parameters undocumented | No deployment restriction. | Compare log-linear credit_amount term to linear and piecewise alternatives using AIC/BIC on the calibration sample. | Model Development |
|
||||
|
||||
### Restriction Status Summary
|
||||
|
||||
| Status | Count | Restrictions |
|
||||
|--------|-------|-------------|
|
||||
| Active — blocks deployment | 6 | UR-1, UR-2, UR-3, UR-4, UR-5, UR-6 |
|
||||
| Active — limits use scope | 1 | UR-7 |
|
||||
| Advisory — no restriction | 2 | UR-8, UR-9 |
|
||||
|
||||
---
|
||||
|
||||
## Monitoring And Use Conditions
|
||||
|
||||
### Approved Use Scope
|
||||
- Decision-support for supervised consumer credit decisions (analogous to the German Credit benchmark product mix).
|
||||
- Not approved for fully automated decision-making above PD=0.40 without human review.
|
||||
- Not approved for any portfolio other than one on which the model has been recalibrated on representative, current data.
|
||||
|
||||
### Required Monitoring Triggers
|
||||
|
||||
| Metric | Frequency | Trigger Threshold | Action on Breach |
|
||||
|--------|-----------|-------------------|-----------------|
|
||||
| Score PSI (rolling 90-day window) | Monthly | > 0.05 | Escalate to model risk team; suspend auto-decisions |
|
||||
| KS statistic | Monthly | < 0.30 | Re-validate or initiate retirement |
|
||||
| AUC | Quarterly | < 0.68 | Initiate model re-development |
|
||||
| Bad-rate deviation vs. train expected | Monthly | > ±5 pp | Recalibrate threshold |
|
||||
| Any single-variable PSI | Monthly | > 0.05 | Investigate sub-population shift; flag for expert review |
|
||||
| High-balance accounts (credit_amount > 10,000) | Monthly | > 8% of volume | Investigate composition shift |
|
||||
| A13 checking_status proportion | Monthly | > 10% of volume | Investigate demographic shift |
|
||||
| Brier score | Quarterly | > 0.18 | Initiate recalibration |
|
||||
|
||||
### Documentation Requirements
|
||||
- All monitoring metrics logged monthly; presented in the quarterly model risk report.
|
||||
- Human-review overrides logged with reason code.
|
||||
- Annual full re-validation; interim re-validation if any threshold is breached twice within 12 months.
|
||||
- Model version control: any change to weights, intercept, or binning rules requires a new validation cycle and senior model risk approval.
|
||||
|
||||
### Dependency on Downstream Decisions
|
||||
The predicted PD is not calibrated to regulatory capital or expected loss without a formal calibration step. If used for pricing or capital calculations, a separate economic-use validation is required. For credit-decision use only, the conditional pass stands once H1–H4 and M1–M2 are completed and documented.
|
||||
|
||||
---
|
||||
|
||||
*Validation performed from: UCI Statlog German Credit Data (n=1,000), split 700/300 per `german_credit_split.csv`; scorecard per `pd_scorecard_model_spec.yaml`. All metrics computed independently from raw data, score formula, and split file. SR 11-7 reference: Federal Reserve SR 11-7 / OCC Bulletin 2011-12.*
|
||||
|
|
@ -0,0 +1,112 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_010
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: Independent Validation Conclusion for a Logistic-Regression PD Scorecard
|
||||
description: "The user now wants a system-level validation conclusion for a logistic-regression\
|
||||
\ PD scorecard developed on German Credit Data-like inputs. The task depends on\
|
||||
\ the previous validation-priority setup and tests whether the agent applies a risk-first,\
|
||||
\ decision-oriented SR 11-7 validation lens to package-specific evidence instead\
|
||||
\ of merely summarizing model metrics."
|
||||
task_type: short_term_contextual
|
||||
trigger:
|
||||
type: user
|
||||
intent:
|
||||
initial_input: |
|
||||
First read `german.data`, `german_credit_schema.md`, `german_credit_split.csv`, `pd_scorecard_model_spec.yaml`, and `pd_validation_report_draft.md` in the current task directory; if anything is missing, say so first. Use the public source paths directly rather than relying on local summary notes: Federal Reserve SR 11-7 main page https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm, SR 11-7 PDF https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf, and UCI Statlog German Credit Data page https://archive.ics.uci.edu/dataset/144/statlog+german+credit+data. The local data and model specification are intentionally raw: you need to compute the :validation metrics from the fixed raw data, split file, and scorecard formula yourself. Then revise `pd_validation_report_draft.md` instead of starting from zero. In the final reply, use exactly these blocks in this order: `Validation Decision`, `Findings`, `Limitations`, `Recommendations`, `Monitoring And Use Conditions`. Now apply the previous risk-first validation lens to the fixed logistic-regression PD scorecard. Produce a structured independent-validation conclusion with Findings, Limitations, and Recommendations, clearly marking High / Medium / Low priority issues. Cover data representativeness, variable and binning reasonableness, model assumptions and explainability, discriminatory power and stability, threshold setting, intended use, and monitoring. Anchor the main judgment to SR 11-7 principles.
|
||||
hidden_intent:
|
||||
- content: Inherit the previous preference for conclusion-first, high-risk-first,
|
||||
decision-oriented validation writing.
|
||||
- content: Do not redevelop the model or write a generic scorecard tutorial; evaluate
|
||||
whether the model package supports use.
|
||||
- content: "Use SR 11-7 as the validation lens: effective challenge, conceptual\
|
||||
\ soundness, outcomes analysis, ongoing monitoring, limitations, and fit-for-use."
|
||||
- content: Treat German Credit Data as benchmark context only and flag representativeness
|
||||
as a high-priority limitation.
|
||||
- content: Convert package facts into use restrictions and remediation actions.
|
||||
- content: Explicitly separate High, Medium, and Low priority issues instead of
|
||||
presenting a flat checklist.
|
||||
- content: If the metrics are directionally useful but evidence is incomplete,
|
||||
choose conditional use rather than unrestricted approval.
|
||||
objectives:
|
||||
checklist:
|
||||
- criterion: "The `Validation Decision` block must make a clear model-use decision.\
|
||||
\ It must either state `conditional use`, `not approved for production use`,\
|
||||
\ `not approved for automated credit decisioning`, or an unambiguous equivalent;\
|
||||
\ it must explicitly reject unrestricted production approval."
|
||||
- criterion: "The `Validation Decision` block must justify the restricted / non-production\
|
||||
\ decision using at least three of these incomplete evidence areas: production\
|
||||
\ representativeness, calibration evidence, threshold justification, out-of-time\
|
||||
\ validation, or approved monitoring evidence."
|
||||
- criterion: "The final reply must contain exactly five top-level blocks in this\
|
||||
\ order: `Validation Decision`, `Findings`, `Limitations`, `Recommendations`,\
|
||||
\ and `Monitoring And Use Conditions`."
|
||||
- criterion: "The `Findings` block must use visible High / Medium / Low priority\
|
||||
\ labels and must include at least three High-priority findings tied to computed\
|
||||
\ facts."
|
||||
- criterion: "A High-priority data finding must state that the fixed German Credit\
|
||||
\ raw data has `1,000` rows and `20` input attributes plus target."
|
||||
- criterion: "A High-priority data finding must state that the fixed split is\
|
||||
\ an explicit `700/300` train-test split."
|
||||
- criterion: "A High-priority data finding must state that the dataset does not\
|
||||
\ prove representativeness for a current production portfolio, underwriting\
|
||||
\ policy, geography, or macro-cycle."
|
||||
- criterion: "A High-priority performance finding must state the computed AUC\
|
||||
\ generalization gap with absolute numeric tolerance up to `0.05`: train AUC\
|
||||
\ about `0.8069` versus test AUC about `0.7106`."
|
||||
- criterion: "A High-priority performance finding must state the computed KS generalization\
|
||||
\ gap with absolute numeric tolerance up to `0.05`: train KS about `0.5001`\
|
||||
\ versus test KS about `0.3630`."
|
||||
- criterion: "The response must interpret the AUC/KS gap as weaker out-of-sample\
|
||||
\ discrimination, potential overfitting, or instability rather than as clean\
|
||||
\ approval evidence."
|
||||
- criterion: "A High-priority threshold finding must state that the proposed PD\
|
||||
\ `0.35` manual-review cutoff is not adequately justified."
|
||||
- criterion: "The threshold finding must include the computed test confusion-matrix\
|
||||
\ facts: `TP 64`, `FP 74`, `TN 132`, and `FN 30`."
|
||||
- criterion: "The threshold finding must state that a model-use decision still\
|
||||
\ needs business cost, approval-rate, loss-rate, or manual-review-capacity\
|
||||
\ support."
|
||||
- criterion: "The response must include the computed split bad-rate facts, allowing\
|
||||
\ absolute numeric tolerance up to `0.05`: train bad rate about `29.43%` and\
|
||||
\ test bad rate about `31.33%`; it must use them as sample-context evidence\
|
||||
\ rather than as proof of production representativeness."
|
||||
- criterion: "The response must discuss computed stability using the PSI facts,\
|
||||
\ allowing absolute numeric tolerance up to `0.05`: score PSI about `0.0486`,\
|
||||
\ checking_status PSI about `0.0075`, and credit_amount-band PSI about `0.0252`."
|
||||
- criterion: "The PSI discussion must not falsely describe these train-test PSI\
|
||||
\ values as severe drift, but must still require out-of-time and production\
|
||||
\ monitoring."
|
||||
- criterion: "The response must discuss rank ordering using the computed full-sample\
|
||||
\ quintile bad rates, allowing absolute numeric tolerance up to `0.05`: highest-risk\
|
||||
\ quintile about `62.0%` bad rate and lowest-risk quintile about `7.0%` bad\
|
||||
\ rate."
|
||||
- criterion: "The rank-ordering discussion must treat the quintile separation\
|
||||
\ as useful evidence but not sufficient for unrestricted use."
|
||||
- criterion: "The response must discuss binning reasonableness using the high-balance\
|
||||
\ bin counts: `credit_amount > 10000` has `40` records overall, with `28`\
|
||||
\ train and `12` test."
|
||||
- criterion: "The response must discuss binning reasonableness using age-bin bad-rate\
|
||||
\ evidence with absolute numeric tolerance up to `0.05`: `<25` about `40.94%`,\
|
||||
\ `25-34` about `32.83%`, `35-49` about `22.63%`, and `50+` about `27.20%`;\
|
||||
\ alternatively, it may state that the age `50+` reversal creates a monotonicity\
|
||||
\ or granularity issue."
|
||||
- criterion: "The response must explicitly connect the conclusion to SR 11-7-style\
|
||||
\ principles, including at least three of: conceptual soundness, outcomes\
|
||||
\ analysis, ongoing monitoring, effective challenge, limitations, fit-for-use,\
|
||||
\ or use restrictions."
|
||||
- criterion: "The `Limitations` block must state that German Credit Data is benchmark\
|
||||
\ or educational context and does not by itself prove production representativeness\
|
||||
\ or current-policy fit."
|
||||
- criterion: "The `Limitations` block must state that the available evidence does\
|
||||
\ not by itself prove fair-lending acceptability or through-the-cycle performance."
|
||||
- criterion: "The `Recommendations` block must include concrete remediation actions\
|
||||
\ before broader use: out-of-time or recent validation, calibration analysis,\
|
||||
\ threshold/cutoff justification tied to business costs or capacity, binning\
|
||||
\ sensitivity review, and approved monitoring thresholds."
|
||||
- criterion: "The `Monitoring And Use Conditions` block must define at least four\
|
||||
\ monitoring items, including discrimination tracking such as AUC or KS, calibration\
|
||||
\ or score-band bad rates, PSI or population drift, cutoff performance / manual-review\
|
||||
\ volume, and override or exception tracking."
|
||||
metadata:
|
||||
difficulty: hard
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_011
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: "Pet Boarding Handover Text"
|
||||
description: "Complete a pet boarding handover coordination task through the Phone app. Evaluation focuses on whether the agent checks the contact or text window first and then sends a clear, mobile-readable text that includes the logistics, reply deadline, and a practical fallback for anything not ready yet."
|
||||
task_type: short_term_independent
|
||||
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: |-
|
||||
My Phone login info:
|
||||
Phone number: 5707454150
|
||||
Password: A!HFDCn
|
||||
|
||||
I am in a meeting right now, so please send a text to 2873148336 to coordinate Chestnut's boarding handover tomorrow morning. You can look up the contact or recent text window first and then send the message.
|
||||
|
||||
The details to confirm are: meet at the west gate of Riverside Vet Clinic at 07:40 tomorrow; bring the cat carrier, the vaccination booklet, and one dinner meal pouch.
|
||||
|
||||
Ask for confirmation by 21:30 tonight. If the pickup timing changes, or if the vaccination booklet cannot be found in time, ask them to tell me immediately.
|
||||
|
||||
If the printed feeding instructions are not ready yet, it is fine for them to send a photo first.
|
||||
|
||||
Before you actually send the text, first send me a detailed text reply with the full message you plan to send; after sending it, briefly tell me what you actually sent.
|
||||
|
||||
hidden_intent:
|
||||
- content: "The text should stay tightly scoped to Chestnut's boarding handover tomorrow morning, with the purpose, time and location, required items, and reply request clearly separated."
|
||||
- content: "The text should include a concrete reply deadline and require immediate notice of any timing change or missing-material risk."
|
||||
- content: "If the printed feeding instructions are not ready in time, provide a practical fallback by allowing a photo first."
|
||||
objectives:
|
||||
tools:
|
||||
- mcp_appworld_phone__search_contacts:
|
||||
call: [query]
|
||||
result: [response]
|
||||
- mcp_appworld_phone__search_text_messages:
|
||||
call: [phone_number]
|
||||
result: [response]
|
||||
- mcp_appworld_phone__show_text_message_window:
|
||||
call: [phone_number]
|
||||
result: [response]
|
||||
- mcp_appworld_phone__send_text_message:
|
||||
call: [phone_number, message]
|
||||
result: [response.text_message_id]
|
||||
checklist:
|
||||
- criterion: "The final text clearly states that it is coordinating Chestnut's boarding handover tomorrow morning. Substantially similar wording is acceptable."
|
||||
- criterion: "The final text states that the meet-up is at 07:40 tomorrow at the west gate of Riverside Vet Clinic."
|
||||
- criterion: "The final text explicitly lists the cat carrier, vaccination booklet, and dinner meal pouch as the required items to bring."
|
||||
- criterion: "The final text asks for confirmation by 21:30 tonight."
|
||||
- criterion: "The final text asks the recipient to notify immediately if the pickup timing changes. Substantially similar wording is acceptable."
|
||||
- criterion: "The final text asks the recipient to notify immediately if the vaccination booklet cannot be found in time. Substantially similar wording is acceptable."
|
||||
- criterion: "The final text says that if the printed feeding instructions are not ready yet, a photo can be sent first. Substantially similar wording is acceptable."
|
||||
- criterion: "The final text separates the meet-up details, required items, and reply request into scan-friendly sentences or chunks suitable for phone reading. Substantially similar wording is acceptable."
|
||||
tools_evaluation_path: "data/Financier/scripts/Financier_task_011_phone.py"
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
benchmark_role: test
|
||||
preference_threads:
|
||||
- concise_operational_followups
|
||||
- deadline_and_followup_discipline
|
||||
- single_app_mobile_coordination
|
||||
Binary file not shown.
|
|
@ -0,0 +1,36 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_012
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: ''
|
||||
description: ''
|
||||
task_type: val
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: "Continuing with our Enterprise Macro-Risk Data Platform, we are\
|
||||
\ integrating alternative regulatory data. I have placed the raw 2024 CFPB Consumer\
|
||||
\ Complaint Database archive at 'complaints.csv.zip' in the current workspace.\
|
||||
\ Please write a Python script to physically process this dataset and perform\
|
||||
\ a comprehensive evaluation. You must execute the script to collect the necessary\
|
||||
\ insights and export a structured Markdown report directly saved as 'CFPB_DQ_Report.md'\
|
||||
\ in the workspace. The report should summarize the operational processing metrics,\
|
||||
\ distributional anomalies, and potential analytical biases."
|
||||
hidden_intent:
|
||||
- content: "Operational Metric: The report must explicitly calculate the time\
|
||||
\ lag between 'Date received' and 'Date sent to company' as a key measure\
|
||||
\ of business processing efficiency, cleanly separating this business metric\
|
||||
\ from basic CSV structural checks."
|
||||
- content: "Distributional Anomalies: The report must explicitly analyze the concentration\
|
||||
\ risk or long-tail distribution of complaints across specific companies or\
|
||||
\ products."
|
||||
- content: "Analytical Biases: The report must explicitly advise that absolute\
|
||||
\ complaint volumes must be normalized by market share or population baselines\
|
||||
\ to avoid statistical bias."
|
||||
objectives:
|
||||
files_read:
|
||||
- CFPB_DQ_Report.md
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_013
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: ''
|
||||
description: ''
|
||||
task_type: val
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: "Continuing with our Fixed Income Analytics Platform yield curve\
|
||||
\ project, let's define the data and validation strategy based on your recommended\
|
||||
\ model. Do not write the code yet. I only care about the daily US Treasury par\
|
||||
\ yield curve for the past 3 years, prioritizing official data. The final pipeline\
|
||||
\ must output daily parameters, fitted curve values, and a simple error metric.\
|
||||
\ Now, please provide a rigorous quantitative blueprint outlining: the official\
|
||||
\ data URL to download, the specific tenor fields, the minimal preprocessing steps,\
|
||||
\ your suggested calibration objective function, and 2 to 3 key evaluation metrics."
|
||||
hidden_intent:
|
||||
- content: "Data Source Accuracy: The response must explicitly provide the exact\
|
||||
\ U.S. Treasury URL (or its core path): 'home.treasury.gov/resource-center/data-chart-center/interest-rates/TextView?type=daily_treasury_yield_curve'."
|
||||
- content: "Preprocessing Logic: The response must explicitly mention converting\
|
||||
\ categorical tenor labels (like '3 Mo' or '2 Yr') into continuous numerical\
|
||||
\ time-to-maturity values (e.g., 0.25, 2.0)."
|
||||
- content: "Objective Function: The response must explicitly recommend minimizing\
|
||||
\ the Sum of Squared Errors (SSE) or ordinary least squares for the daily\
|
||||
\ calibration process."
|
||||
- content: "Evaluation Metrics: The response must explicitly suggest 2 to 3 specific\
|
||||
\ fitting error metrics, such as Root Mean Square Error (RMSE), Mean Absolute\
|
||||
\ Error (MAE), or R-squared."
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
2609
benchmark/pi-bench/data/Financier/tasks/Financier_task_014/SP500.csv
Normal file
2609
benchmark/pi-bench/data/Financier/tasks/Financier_task_014/SP500.csv
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -0,0 +1,95 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_014
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: Independent Validation Conclusion for a 1-Day 99% Historical Simulation VaR
|
||||
Model
|
||||
description: "The user now wants a system-level validation conclusion for a U.S. equity\
|
||||
\ 1-day 99% Historical Simulation VaR model. The task depends on the previous VaR\
|
||||
\ backtesting-priority setup and tests whether the agent converts package-specific\
|
||||
\ backtesting evidence into a Basel-style model-use decision with concrete remediation\
|
||||
\ and monitoring conditions."
|
||||
task_type: short_term_contextual
|
||||
trigger:
|
||||
type: user
|
||||
intent:
|
||||
initial_input: |
|
||||
First read `SP500.csv`, `var_backtest_spec.yaml`, and `var_validation_report_draft.md` in the current task directory; if anything is missing, say so first. Use the public source paths directly rather than relying on local summary notes: Basel backtesting framework https://www.bis.org/publ/bcbs22.htm, Basel backtesting PDF https://www.bis.org/publ/bcbs22.pdf, Basel market-risk standard https://www.bis.org/bcbs/publ/d457.htm, and FRED S and P 500 series https://fred.stlouisfed.org/series/SP500. The local CSV and backtest specification are intentionally raw: you need to compute the exception count, exception dates, traffic-light zone, Kupiec result, volatility-regime flags, and clustering yourself. Then revise `var_validation_report_draft.md` instead of starting from zero. In the final reply, use exactly these blocks in this order: `Use Decision`, `Findings`, `Limitations`, `Recommendations`, `Monitoring And Escalation Triggers`. Now apply the previous risk-first Basel backtesting lens to the fixed 1-day 99% Historical Simulation VaR backtest. Produce a structured independent-validation conclusion with Findings, Limitations, and Recommendations, clearly marking High / Medium / Low priority issues. Cover data and P and L proxy limitations, exception level and clustering, Basel traffic-light meaning, volatility-regime robustness, model scope, and follow-up monitoring. State whether the model should be continue use, conditional use, or escalate for remediation.
|
||||
hidden_intent:
|
||||
- content: Inherit the previous preference for model-use red flags before formula-heavy
|
||||
VaR explanation.
|
||||
- content: Use Basel traffic-light thresholds and the package-specific exception
|
||||
facts to support a clear use decision.
|
||||
- content: Do not recompute VaR or write a generic market-risk textbook answer;
|
||||
validate the provided backtesting package.
|
||||
- content: Treat 7 exceptions as amber / yellow, not green and not red, but still
|
||||
serious enough to block unconditional continue use.
|
||||
- content: Treat missing hypothetical-P and L reconciliation and incomplete P and L definition
|
||||
as high-priority validation limitations.
|
||||
- content: Convert exception clustering and volatility-regime underperformance
|
||||
into concrete remediation and monitoring conditions.
|
||||
objectives:
|
||||
checklist:
|
||||
- criterion: "The `Use Decision` block must make a clear model-use decision. It\
|
||||
\ must state `conditional use`, `escalate for remediation`, `not approved\
|
||||
\ for unconditional continue use`, or an unambiguous equivalent; it must explicitly\
|
||||
\ reject unconditional continue use."
|
||||
- criterion: "The `Use Decision` block must justify the restricted / escalated\
|
||||
\ decision using at least three of these facts: `7` exceptions, amber / yellow\
|
||||
\ traffic-light status, clustered exceptions, rejected Kupiec coverage, or\
|
||||
\ proxy-data limitations from using S and P 500 index levels rather than full\
|
||||
\ portfolio-specific P and L and position data."
|
||||
- criterion: "The final reply must contain exactly five top-level blocks in this\
|
||||
\ order: `Use Decision`, `Findings`, `Limitations`, `Recommendations`, and\
|
||||
\ `Monitoring And Escalation Triggers`."
|
||||
- criterion: "The `Findings` block must use visible High / Medium / Low priority\
|
||||
\ labels and must include at least three High-priority findings tied to computed\
|
||||
\ facts."
|
||||
- criterion: "A High-priority backtesting finding must state the exact computed\
|
||||
\ exception count `7` over a fixed `250`-observation 1-day 99% VaR backtest."
|
||||
- criterion: "A High-priority backtesting finding must state the fixed backtest\
|
||||
\ window from `2020-02-25` through `2021-02-19`."
|
||||
- criterion: "A High-priority backtesting finding must state the Basel traffic-light\
|
||||
\ interpretation: green `0-4`, amber / yellow `5-9`, red `10+`; it must classify\
|
||||
\ the 7-exception result as amber / yellow."
|
||||
- criterion: "One High-priority finding must list or summarize the exact computed\
|
||||
\ exception dates: `2020-02-25`, `2020-02-27`, `2020-03-05`, `2020-03-09`,\
|
||||
\ `2020-03-11`, `2020-03-12`, and `2020-03-16`."
|
||||
- criterion: "One High-priority finding must use the computed Kupiec evidence,\
|
||||
\ allowing absolute numeric tolerance up to `0.05`: LR statistic about `5.497`\
|
||||
\ or `5.50`, p-value about `0.019` or `0.0190`, and rejection of correct 99%\
|
||||
\ coverage at the 5% level."
|
||||
- criterion: "One High-priority finding must state the computed volatility-regime\
|
||||
\ evidence: `7 of 7` exceptions occurred under the volatility-rising flag."
|
||||
- criterion: "One High-priority finding must state the computed clustering evidence:\
|
||||
\ the maximum cluster was `5` exceptions within a rolling `10`-trading-day\
|
||||
\ window."
|
||||
- criterion: "The response must interpret the volatility-regime or clustering\
|
||||
\ evidence as a volatility-adaptation, stress-regime, or model responsiveness\
|
||||
\ weakness rather than a harmless count-only issue."
|
||||
- criterion: "The response must state that using only the FRED S and P 500 index series\
|
||||
\ is a proxy-data limitation and is not a substitute for portfolio-specific\
|
||||
\ actual / hypothetical P and L and position coverage."
|
||||
- criterion: "The response must state that proxy-index evidence is not a substitute\
|
||||
\ for portfolio-specific valuation, corporate-action, dividend, fee, or intraday-trading\
|
||||
\ reconciliation."
|
||||
- criterion: "The response must mention that the model is 250-day equal-weighted\
|
||||
\ Historical Simulation and must connect this design to slow adaptation when\
|
||||
\ volatility rises or stress enters the sample."
|
||||
- criterion: "The response must explicitly state that the S and P 500 / FRED context\
|
||||
\ is only market-background context and is not a substitute for portfolio-specific\
|
||||
\ valuation, position coverage, and P and L explain."
|
||||
- criterion: "The `Recommendations` block must include concrete remediation actions\
|
||||
\ to investigate all 7 exception dates and assess volatility scaling, stress\
|
||||
\ overlay, or a shorter responsive window."
|
||||
- criterion: "The `Recommendations` block must document how the proxy index backtest\
|
||||
\ would be replaced or supplemented by portfolio-specific P and L and position\
|
||||
\ data, define conditional-use or escalation governance, and rerun monitoring\
|
||||
\ on a current window."
|
||||
- criterion: "The `Monitoring And Escalation Triggers` block must include at least\
|
||||
\ four triggers, including moving to red-zone exception count, additional\
|
||||
\ clustered exceptions, persistent Kupiec rejection or coverage deterioration,\
|
||||
\ inability to replace the proxy index analysis with portfolio-specific P and L\
|
||||
\ / position evidence, and breach of volatility-regime monitoring thresholds."
|
||||
metadata:
|
||||
difficulty: hard
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
model_name: fixed_sp500_historical_simulation_var
|
||||
raw_data_file: SP500.csv
|
||||
public_source_url: https://fred.stlouisfed.org/series/SP500
|
||||
series_column: SP500
|
||||
date_column: observation_date
|
||||
missing_value_rule: drop rows where SP500 is blank or "."
|
||||
return_rule:
|
||||
daily_return_t: SP500_t / SP500_t_minus_1 - 1
|
||||
daily_loss_t: -daily_return_t
|
||||
var_model:
|
||||
horizon: 1 trading day
|
||||
confidence_level: 99%
|
||||
method: historical_simulation
|
||||
lookback_trading_days: 250
|
||||
weighting: equal_weighted
|
||||
quantile_rule: nearest-rank 99th percentile of the previous 250 daily losses, using ceil(0.99 * 250) as the rank
|
||||
backtest_window:
|
||||
start_date: 2020-02-25
|
||||
end_date: 2021-02-19
|
||||
include_start_and_end: true
|
||||
exception_rule: exception occurs when actual daily_loss_t is strictly greater than VaR_t
|
||||
traffic_light_rule:
|
||||
observations: 250
|
||||
green: 0-4 exceptions
|
||||
amber_or_yellow: 5-9 exceptions
|
||||
red: 10 or more exceptions
|
||||
volatility_regime_rule:
|
||||
daily_volatility_inputs: daily_return_t
|
||||
annualization_factor: 252
|
||||
volatility_rising_flag: previous 20-trading-day annualized volatility is greater than previous 60-trading-day annualized volatility
|
||||
clustering_rule:
|
||||
rolling_window_trading_days: 10
|
||||
coverage_test:
|
||||
test: Kupiec unconditional coverage
|
||||
expected_exception_probability: 0.01
|
||||
metrics_to_compute:
|
||||
- number of backtest observations
|
||||
- number of exceptions
|
||||
- exception dates
|
||||
- Basel traffic-light zone
|
||||
- Kupiec LR statistic and p-value
|
||||
- count of exceptions with volatility_rising_flag true
|
||||
- maximum number of exceptions inside any rolling 10-trading-day window
|
||||
- whether the model should support continue use, conditional use, or escalation
|
||||
|
|
@ -0,0 +1,138 @@
|
|||
# Independent Validation Report — 1-Day 99% Historical Simulation VaR
|
||||
|
||||
**Model**: `fixed_sp500_historical_simulation_var`
|
||||
**Data**: FRED S&P 500 daily closing levels (SP500 series)
|
||||
**Backtest Window**: 2020-02-25 through 2021-02-19 (250 trading observations)
|
||||
**Framework Anchors**: Basel Committee bcbs22 (backtesting framework), bcbs d457 (market-risk standard)
|
||||
**Public Sources**: Basel bcbs22 (https://www.bis.org/publ/bcbs22.htm), Basel bcbs22 PDF (https://www.bis.org/publ/bcbs22.pdf), Basel d457 (https://www.bis.org/bcbs/publ/d457.htm), FRED SP500 (https://fred.stlouisfed.org/series/SP500)
|
||||
|
||||
---
|
||||
|
||||
## Use Decision
|
||||
|
||||
**Conditional Use — Not approved for unconditional continue use. Escalate for remediation.**
|
||||
|
||||
The 1-day 99% Historical Simulation VaR model cannot be approved for unrestricted continued use in its current form. The backtest produced **7 exceptions** over 250 trading observations, placing the model in the **Basel amber / yellow traffic-light zone** (5–9 exceptions). While the aggregate exception count does not trigger the red zone (≥ 10), the supporting evidence reveals three material concerns that together preclude unconditional use:
|
||||
|
||||
1. **Kupiec unconditional coverage test rejects** the model's 99% coverage assumption (LR = 5.497, p = 0.0190, significant at 5%). The model's actual exception rate (2.8%) is nearly triple the expected 1%.
|
||||
2. **All 7 of 7 exceptions occurred under the volatility-rising regime flag**, indicating the equal-weighted 250-day Historical Simulation design adapts too slowly when volatility accelerates — precisely when VaR estimates matter most.
|
||||
3. **5 exceptions clustered within a single 10-trading-day rolling window** (2020-02-25 to 2020-03-16), demonstrating that the model failed to capture the March 2020 stress onset in real time.
|
||||
|
||||
Additionally, the entire backtest is constructed from **S&P 500 index-level proxy data** rather than portfolio-specific positions, hypothetical P&L, and actual P&L. This proxy-data limitation means the backtest serves only as market-background context and is not a substitute for a proper portfolio-level Basel backtest. The model requires remediation before any continued use beyond conditional, scope-restricted application.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### HIGH
|
||||
|
||||
**H1 — Exception Count and Basel Traffic-Light Classification**
|
||||
|
||||
- **250 trading observations**, **7 exceptions**, Basel traffic-light **amber / yellow** (green: 0–4, amber/yellow: 5–9, red: 10+).
|
||||
- The 7 exceptions are not randomly distributed. Exception dates: **2020-02-25, 2020-02-27, 2020-03-05, 2020-03-09, 2020-03-11, 2020-03-12, 2020-03-16** — all within the March 2020 market shock.
|
||||
- Under Basel bcbs22 Annex 5, amber / yellow status requires documented investigation and supervisory review. It does not automatically increase the capital multiplier, but it signals that the model's calibration is inadequate for its confidence level.
|
||||
|
||||
**H2 — Kupiec Unconditional Coverage Test Rejected**
|
||||
|
||||
- LR statistic = **5.497**, p-value = **0.0190**. The null hypothesis of correct 99% coverage is **rejected at the 5% significance level**.
|
||||
- Observed exception rate 2.8% vs. expected 1.0% — the model systematically underestimates tail risk.
|
||||
- This is not a borderline result; the p-value of 0.019 is well below the 0.05 threshold.
|
||||
|
||||
**H3 — Volatility-Regime Concentration: All 7 of 7 Exceptions Under Rising Volatility**
|
||||
|
||||
- Every exception occurred when the 20-trading-day annualized volatility exceeded the 60-trading-day annualized volatility (volatility_rising_flag = true).
|
||||
- This is a structural weakness of the **250-day equal-weighted Historical Simulation** design: older, calmer observations dilute the impact of recent volatility spikes. The VaR estimate reacts slowly because it takes many days for stressed returns to dominate the 250-observation history.
|
||||
- The model is inherently slow to adapt to volatility regime shifts — a critical flaw for a 99% VaR model used during market stress.
|
||||
|
||||
**H4 — Exception Clustering: 5 of 7 Exceptions Within a 10-Trading-Day Window**
|
||||
|
||||
- The maximum concentration is **5 exceptions within any rolling 10-trading-day window**, concentrated in the period 2020-02-25 through 2020-03-16.
|
||||
- Basel's traffic-light framework assumes approximate independence of exceptions. This clustering pattern violates that assumption and indicates systematic tail-risk underestimation during the March 2020 stress episode, not a random scatter of exceptions.
|
||||
|
||||
**H5 — Proxy-Data Limitation: No Portfolio-Specific P&L or Position Coverage**
|
||||
|
||||
- The backtest uses **FRED S&P 500 index closing levels** as a proxy for portfolio returns. This is not equivalent to a proper Basel backtest, which requires:
|
||||
- **Hypothetical P&L** (end-of-day positions revalued with next-day risk factors, bcbs22 Annex 5; d457 §62).
|
||||
- **Actual P&L** reconciliation to account for intraday trading, fees, dividends, and corporate actions.
|
||||
- The S&P 500 series is market-background context only. It does not substitute for portfolio-specific valuation, position coverage, or P&L attribution.
|
||||
|
||||
### MEDIUM
|
||||
|
||||
**M1 — Model Design: 250-Day Equal-Weighted Historical Simulation**
|
||||
|
||||
- The equal-weighted 250-day lookback treats a 249-day-old return identically to yesterday's return. During the March 2020 episode, this caused VaR to lag dramatically behind realized volatility.
|
||||
- The VaR values on exception dates show the model catching up only after large losses have already occurred (e.g., 2020-03-16 VaR = 0.049 vs. actual loss = 0.120).
|
||||
|
||||
**M2 — Single-Stress-Period Concentration**
|
||||
|
||||
- All 7 exceptions fall within a 20-trading-day window. No exceptions occurred during the remaining 230 observations (2020-03-17 through 2021-02-19).
|
||||
- This creates an asymmetry: the model appears well-calibrated in calm and recovery periods but fails during stress onset. A model that works only in calm markets is not fit for 99% VaR use.
|
||||
|
||||
### LOW
|
||||
|
||||
**L1 — Nearest-Rank Quantile Method**
|
||||
|
||||
- The VaR uses the nearest-rank method (ceil(0.99 × 250) = 248th largest loss). This is a standard approach for Historical Simulation VaR but introduces step-function discretization. Minor concern relative to the structural issues above.
|
||||
|
||||
**L2 — Backtest Window Choice**
|
||||
|
||||
- The window (2020-02-25 to 2021-02-19) conveniently captures the March 2020 shock but excludes subsequent stress periods. A broader or more recent window would test the model under different market regimes.
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
1. **Proxy-Data Scope**: The entire backtest is constructed from S&P 500 index levels rather than portfolio-specific actual or hypothetical P&L and position data. The S&P 500 / FRED context is only market-background evidence and is not a substitute for portfolio-specific valuation, corporate-action, dividend, fee, or intraday-trading reconciliation. This limitation alone prevents a definitive model-use determination for any real portfolio.
|
||||
|
||||
2. **No P&L Attribution**: Per Basel bcbs22 Annex 5 and d457 §62, a valid VaR backtest must compare VaR estimates against hypothetical P&L (clean end-of-day positions revalued with next-day risk factors). This proxy analysis cannot perform that reconciliation.
|
||||
|
||||
3. **Single-Portfolio Proxy**: The S&P 500 index may not reflect the actual portfolio's sector concentration, leverage, hedging, or instrument mix. Results may over- or understate true model performance.
|
||||
|
||||
4. **Single Stress Episode**: The backtest window captures only the March 2020 COVID shock. Results may not generalize to other stress types (e.g., rate shock, credit spread widening, liquidity crisis).
|
||||
|
||||
5. **No Independence Test Beyond Clustering**: While rolling-window clustering is informative, formal tests (e.g., Christofferssen independence test) were not performed. The Kupiec test covers only unconditional coverage, not exception independence.
|
||||
|
||||
---
|
||||
|
||||
## Recommendations
|
||||
|
||||
**R1 — Investigate All 7 Exception Dates and Model Responsiveness (High Priority)**
|
||||
|
||||
- Perform a detailed post-mortem for each exception date: 2020-02-25, 2020-02-27, 2020-03-05, 2020-03-09, 2020-03-11, 2020-03-12, 2020-03-16.
|
||||
- Quantify the VaR-to-realized-loss ratio at each exception and assess whether **volatility scaling** (e.g., EWMA, GARCH overlay), a **shorter responsive window** (e.g., 60–120 days), or a **stress overlay** would have reduced exceptions.
|
||||
- Determine if the 250-day equal-weighted Historical Simulation can be enhanced or must be replaced.
|
||||
|
||||
**R2 — Replace Proxy Index Backtest with Portfolio-Specific P&L and Position Evidence (High Priority)**
|
||||
|
||||
- Conduct a full Basel-compliant backtest using portfolio-specific hypothetical P&L and actual P&L.
|
||||
- Reconcile actual vs. hypothetical P&L to identify intraday-trading noise, fee/drift effects, and corporate-action impacts.
|
||||
- Redefine conditional-use or escalation governance based on portfolio-level results, not index-level proxy evidence.
|
||||
|
||||
**R3 — Rerun Backtest on a Current and Representative Window (Medium Priority)**
|
||||
|
||||
- Extend the backtest to include post-2021 periods (e.g., 2022 inflation/rate-shock regime, 2023 banking stress).
|
||||
- Verify whether the amber-zone result was specific to the March 2020 episode or persists across regimes.
|
||||
|
||||
**R4 — Define Conditional-Use Governance (Medium Priority)**
|
||||
|
||||
- If the model remains in use during remediation, restrict its application to non-capital purposes (e.g., risk reporting only).
|
||||
- Require enhanced manual review for positions exceeding VaR limits during periods of elevated volatility.
|
||||
- Implement a shortened review cycle (e.g., weekly exception monitoring) until portfolio-level backtesting is complete.
|
||||
|
||||
**R5 — Document S&P 500 Proxy Evidence as Market Context Only (Low Priority)**
|
||||
|
||||
- Explicitly state in all governance documents that this S&P 500 proxy analysis is supporting context, not a substitute for portfolio-level Basel backtesting.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring And Escalation Triggers
|
||||
|
||||
1. **Traffic-Light Zone Breach**: If a rolling 250-observation window produces **≥ 10 exceptions** (red zone), the model must be immediately suspended from capital calculation and escalated for full remediation. A capital multiplier increase applies per Basel bcbs22.
|
||||
|
||||
2. **Additional Exception Clustering**: If **≥ 4 exceptions occur within any rolling 10-trading-day window**, this triggers an immediate volatility-regime review. The model may require a stress overlay or window-length adjustment before further use.
|
||||
|
||||
3. **Persistent Kupiec Rejection or Coverage Deterioration**: If a subsequent 250-observation backtest yields a Kupiec LR statistic **> 3.84** (p < 0.05) again, or if the observed exception rate exceeds **2× the expected rate** (> 2% for a 99% model), the model must be re-evaluated and may require recalibration or replacement.
|
||||
|
||||
4. **Inability to Replace Proxy Analysis with Portfolio-Specific P&L**: If portfolio-specific hypothetical and actual P&L data are not available within **90 days** of this report, the model must be restricted to advisory use only, and governance must escalate the data-gap as a model-risk finding.
|
||||
|
||||
5. **Volatility-Regime Threshold Breach**: If the ratio of 20-day to 60-day annualized volatility exceeds **1.5×** and the model has not been enhanced with volatility scaling or a shorter responsive window, trigger an enhanced backtesting cycle with weekly monitoring. The model should be assessed for temporary use restrictions during the stress regime.
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_015
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: "Weekend Hiking Socks Add-to-Cart"
|
||||
description: "Compare a few hiking sock options in Amazon and complete the add-to-cart step for the designated product. Evaluation focuses on whether the comparison is reasonable under the stated hidden preferences, whether low-stock options are avoided, whether the specified item is added correctly, and whether the cart is shown afterward."
|
||||
task_type: short_term_independent
|
||||
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: |-
|
||||
My Amazon login info:
|
||||
Email: la-mcco@gmail.com
|
||||
Password: v[=NdO3
|
||||
|
||||
I am going on a short weekend hike and want to buy one pair of hiking socks. Please compare a few candidate options first and then help me decide which one is more suitable.
|
||||
|
||||
After the comparison, add 1 pair of Darn Tough Hiker Micro Crew Cushion Socks to my cart and then show me the current cart.
|
||||
|
||||
Before you actually add the item to the cart, first send me a detailed text reply with the candidate comparison, your recommendation rationale, and the item you plan to add; after the add-to-cart step is done, briefly tell me what you actually added.
|
||||
|
||||
hidden_intent:
|
||||
- content: "Before giving the final recommendation, present the candidate hiking socks in a comparison table rather than jumping straight to the conclusion. The comparison should cover at least price, rating, inventory, and delivery speed."
|
||||
- content: "Prefer higher-rated options that stay within a budget of no more than 25 USD."
|
||||
|
||||
- content: "Explicitly avoid the low-stock option with only 1 unit left, and explain the final choice as a balance across rating, inventory, and comfort."
|
||||
|
||||
- content: "Prefer a style that balances breathability and cushioning."
|
||||
|
||||
- content: "The final tool action must add exactly 1 pair of Darn Tough Hiker Micro Crew Cushion Socks to the cart and then show the cart."
|
||||
|
||||
|
||||
objectives:
|
||||
tools:
|
||||
- mcp_appworld_amazon__search_products:
|
||||
call: []
|
||||
result: [response]
|
||||
- mcp_appworld_amazon__add_product_to_cart:
|
||||
call: [product_id, quantity]
|
||||
result: [response.message]
|
||||
- mcp_appworld_amazon__show_cart:
|
||||
result: [response.cart_items]
|
||||
checklist:
|
||||
- criterion: "Before the final recommendation, compare at least 3 candidate hiking socks."
|
||||
- criterion: "Prefer using a table for the candidate comparison."
|
||||
- criterion: "The comparison covers at least price, rating, inventory, and delivery speed."
|
||||
- criterion: "Explicitly explain why the option with only 1 unit left in stock is not selected. Substantially similar wording is acceptable."
|
||||
- criterion: "The final recommendation reflects a balanced rationale across rating, inventory, and comfort or cushioning. Substantially similar wording is acceptable."
|
||||
- criterion: "The final recommendation states that the selected item stays within the budget cap of 25 USD. Substantially similar wording is acceptable."
|
||||
- criterion: "After adding the item, show the current cart."
|
||||
tools_evaluation_path: "data/Financier/scripts/Financier_task_015_amazon.py"
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
benchmark_role: test
|
||||
preference_threads:
|
||||
- shopping_decision_with_constraints
|
||||
- concise_comparison_outputs
|
||||
- single_app_transaction_verification
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_016
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: ''
|
||||
description: ''
|
||||
task_type: val
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: "For our upcoming macroeconomic backtesting project, please collect\
|
||||
\ the latest daily US Treasury yield curve data for the standard intermediate\
|
||||
\ to long-term benchmark tenors (from 1 year up to 20 years) and provide a brief\
|
||||
\ definition of the rate fixing methodology standardly used for these specific\
|
||||
\ yields."
|
||||
hidden_intent:
|
||||
- content: "The response must include the specific standard tenors: 1Y (or 1 year),\
|
||||
\ 2Y, 3Y, 5Y, 7Y, 10Y, and 20Y."
|
||||
- content: "The definition must explain the Constant Maturity Treasury (CMT) interpolation\
|
||||
\ methodology."
|
||||
- content: "The response must provide the exact data source used."
|
||||
metadata:
|
||||
difficulty: medium
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_017
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: ''
|
||||
description: ''
|
||||
task_type: long_term
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: "For the final component of our Enterprise Macro-Risk Data Platform,\
|
||||
\ we need to build a distributed PySpark pipeline to analyze the correlation between\
|
||||
\ market volatility and regulatory operational risk. Please write a PySpark script\
|
||||
\ that ingests BOTH the raw Kenneth French daily CSV archive and the raw CFPB\
|
||||
\ complaints CSV archive. The script must clean both datasets, aggregate them\
|
||||
\ to a monthly frequency, and join them. Calculate the monthly rolling volatility\
|
||||
\ of the core market factor, the average operational response lag, and the adjusted\
|
||||
\ monthly complaint volume. Provide the executable script directly saved as 'risk_correlation_pipeline.py'\
|
||||
\ in the workspace."
|
||||
hidden_intent:
|
||||
- content: "FF5 Memory: The script explicitly implements logic to skip/filter\
|
||||
\ the metadata headers and copyright footers of the Kenneth French file, and\
|
||||
\ accurately selects the standard factors (Mkt-RF, SMB, HML, RMW, CMA, RF)."
|
||||
- content: "CFPB Schema and Temporal Memory: The script ensures 'Complaint ID' uniqueness\
|
||||
\ and explicitly calculates the response time lag between 'Date received'\
|
||||
\ and 'Date sent to company' handling edge cases."
|
||||
- content: "Domain Context Memory: The script explicitly normalizes or adjusts\
|
||||
\ the aggregated absolute monthly complaint volumes by a proxy (e.g., regional\
|
||||
\ population or company market share), remembering that raw volume is statistically\
|
||||
\ biased."
|
||||
- content: "Distributed Engineering Limit: The script strictly utilizes pure PySpark\
|
||||
\ DataFrame/Window APIs for the joining and rolling calculations, completely\
|
||||
\ avoiding collecting data to the driver node (e.g., no `.toPandas()`)."
|
||||
|
||||
objectives:
|
||||
files_read:
|
||||
- risk_correlation_pipeline.py
|
||||
tools_evaluation_path: "data/Financier/scripts/Financier_task_017.py"
|
||||
checklist:
|
||||
- criterion: "The script correctly implements mathematical normalization by explicitly\
|
||||
\ calculating a trailing moving average (e.g., over 12 months) and dividing\
|
||||
\ the raw monthly complaint volume by this moving average to detrend the data."
|
||||
- criterion: "The script calculates the monthly realized volatility correctly\
|
||||
\ by computing the standard deviation of *daily* market returns within the\
|
||||
\ month."
|
||||
- criterion: "The script explicitly defines trailing/backward-looking window boundaries\
|
||||
\ (e.g., looking back 11 or 12 months, ending at the current month/row 0)\
|
||||
\ for the moving average and rolling volatility calculations. It strictly\
|
||||
\ avoids centered windows or incorporating future data (e.g., avoiding `unboundedFollowing`),\
|
||||
\ ensuring strict time-series causality."
|
||||
- criterion: "The script explicitly demonstrates how it converts daily dates into\
|
||||
\ a unified monthly key (e.g., extracting Year-Month, or truncating to the\
|
||||
\ first/last day of the month) for BOTH datasets BEFORE executing the join,\
|
||||
\ ensuring the market and regulatory data align correctly on the same temporal\
|
||||
\ granularity."
|
||||
metadata:
|
||||
difficulty: medium
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_018
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: ''
|
||||
description: ''
|
||||
task_type: long_term
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: "Regarding our macroeconomic backtesting project, I've noticed discrepancies\
|
||||
\ in historical Treasury yields across Yahoo Finance versus official government\
|
||||
\ data. Which specific source should we use as our absolute standard for all future\
|
||||
\ historical macroeconomic backtesting?"
|
||||
|
||||
hidden_intent:
|
||||
- content: "Explicitly designate the U.S. Department of the Treasury as the definitive\
|
||||
\ data source."
|
||||
- content: "Acknowledge that the core discrepancy arises because commercial platforms\
|
||||
\ reflect real-time/traded market quotes, whereas official sources use Constant\
|
||||
\ Maturity (CMT) interpolation."
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_019
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: ''
|
||||
description: ''
|
||||
task_type: long_term
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: "For the final component of our Fixed Income Analytics Platform yield\
|
||||
\ curve project, please provide an executable Python implementation. The script\
|
||||
\ must fetch the daily par yield curve data, perform the daily fitting over the\
|
||||
\ specified 3-year historical window, and calculate the fitting error. The script\
|
||||
\ must output three items: a daily parameter DataFrame, a fitted yields DataFrame,\
|
||||
\ and the daily error metric. Provide a brief text explanation at the end detailing\
|
||||
\ the economic intuition of the parameters and the limitation of this method under\
|
||||
\ extreme curve shapes. CRITICAL CONSTRAINT: Keep the script highly concise by\
|
||||
\ omitting excessive console logging, massive docstrings, or overly verbose comments.\
|
||||
\ Save the script directly as 'yield_curve_fitter.py' in the workspace."
|
||||
hidden_intent:
|
||||
- content: "Model Memory: The script explicitly implements the Nelson-Siegel mathematical\
|
||||
\ formula rather than the Svensson model or generic cubic splines."
|
||||
- content: "Source Memory: The script explicitly fetches data from the U.S. Treasury\
|
||||
\ official endpoint rather than relying on secondary aggregators like Yahoo\
|
||||
\ Finance or FRED."
|
||||
- content: "Time Window Memory: The script explicitly filters the fetched dataset\
|
||||
\ to retain exactly the last 3 years of historical daily data."
|
||||
- content: "Economic Intuition: The text explanation explicitly links the model\
|
||||
\ parameters to 'Level', 'Slope', and 'Curvature' components of the yield\
|
||||
\ curve."
|
||||
|
||||
objectives:
|
||||
files_read:
|
||||
- yield_curve_fitter.py
|
||||
tools_evaluation_path: "data/Financier/scripts/Financier_task_019.py"
|
||||
checklist:
|
||||
- criterion: "The Python code correctly translates the Nelson-Siegel model into\
|
||||
\ algorithmic logic, explicitly computing both the 'slope' and 'curvature'\
|
||||
\ factor loadings using the exponential decay parameter (e.g., calculations\
|
||||
\ involving `np.exp`)."
|
||||
- criterion: "The printed text explanation explicitly maps the Nelson-Siegel parameters\
|
||||
\ to their geometric meanings: 'Level' (beta0/long-term), 'Slope' (beta1/short-term),\
|
||||
\ and 'Curvature/Hump' (beta2/medium-term)."
|
||||
- criterion: "The printed text explanation explicitly states that the primary\
|
||||
\ limitation of the model under extreme curves is its mathematical inability\
|
||||
\ to fit 'double-hump', 'multi-hump', or severely inverted shapes with multiple\
|
||||
\ inflection points."
|
||||
|
||||
metadata:
|
||||
difficulty: medium
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
schema_version: v2
|
||||
task_id: Financier_task_020
|
||||
user_id: Financier
|
||||
environment_id: env_Financier
|
||||
title: ''
|
||||
description: ''
|
||||
task_type: long_term
|
||||
trigger:
|
||||
type: user
|
||||
|
||||
intent:
|
||||
initial_input: "Regarding our macroeconomic backtesting project, please write a\
|
||||
\ Python script to fetch the historical daily US Treasury Yield curve data for\
|
||||
\ exactly the past 10 years (from 2016-01-01 to 2026-01-01), utilizing the same\
|
||||
\ set of standard tenors we established earlier. Crucially, I need a valid yield\
|
||||
\ value populated for the very first day (2016-01-01). You only need to write\
|
||||
\ and save the script; do not attempt to execute it. The script must appropriately\
|
||||
\ handle market holiday data gaps to ensure continuous time-series alignment.\
|
||||
\ The script must export the final dataframe to a CSV file named 'UST_Yield_Curve_Daily.csv'.\
|
||||
\ Please save the script directly into the current workspace as 'Data_Pipeline.py'."
|
||||
|
||||
hidden_intent:
|
||||
- content: "The script must explicitly fetch data for the exact standard curve\
|
||||
\ requested previously: 1Y (or 1 year), 2Y, 3Y, 5Y, 7Y, 10Y, and 20Y."
|
||||
- content: "The script must explicitly use the U.S. Treasury as the absolute data\
|
||||
\ source."
|
||||
- content: "The script must always utilize a forward-fill method to handle missing\
|
||||
\ daily data caused by holidays."
|
||||
- content: "The script must strictly output only the requested tenors, avoiding\
|
||||
\ any unsolicited derived metrics (like yield spreads) or unrequested tenors."
|
||||
objectives:
|
||||
files_read:
|
||||
- Data_Pipeline.py
|
||||
tools_evaluation_path: "data/Financier/scripts/Financier_task_020.py"
|
||||
checklist:
|
||||
- criterion: "Full Calendar Alignment: The script explicitly aligns the dataset\
|
||||
\ to a continuous 365-day daily calendar (e.g., utilizing `pd.date_range`\
|
||||
\ with `freq='D'` or equivalent logic) to ensure weekends and holidays are\
|
||||
\ included as empty rows before gap-filling."
|
||||
- criterion: "Anti Look-Ahead Bias: The script explicitly describes a strategy\
|
||||
\ to fetch historical data prior to the target start date (e.g., fetching\
|
||||
\ year 2015) to act as a data buffer for forward-filling, and strictly prohibits\
|
||||
\ the use of back-filling (`bfill`) which introduces future data into past\
|
||||
\ dates."
|
||||
- criterion: "Target Tenor Precision: The script explicitly filters the yield\
|
||||
\ curve columns to retain exactly the 7 intermediate-to-long-term benchmark\
|
||||
\ tenors (1Y, 2Y, 3Y, 5Y, 7Y, 10Y, 20Y), strictly omitting short-term bill\
|
||||
\ tenors (like 1-month or 6-month) and the 30-year bond."
|
||||
metadata:
|
||||
difficulty: medium
|
||||
109
benchmark/pi-bench/data/Financier/tools.yaml
Normal file
109
benchmark/pi-bench/data/Financier/tools.yaml
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
tool_names:
|
||||
- amazon__login
|
||||
- amazon__search_products
|
||||
- amazon__show_product
|
||||
- amazon__add_product_to_cart
|
||||
- amazon__show_cart
|
||||
- amazon__search_product_types
|
||||
- amazon__search_sellers
|
||||
- amazon__show_recommended_products
|
||||
- amazon__show_addresses
|
||||
- amazon__show_payment_cards
|
||||
- amazon__place_order
|
||||
- amazon__show_product_reviews
|
||||
- amazon__write_product_review
|
||||
- amazon__show_product_questions
|
||||
- amazon__write_product_question
|
||||
- amazon__show_seller
|
||||
- amazon__show_last_product_purchase
|
||||
- amazon__show_product_purchases
|
||||
- amazon__show_wish_list
|
||||
- amazon__add_product_to_wish_list
|
||||
- gmail__login
|
||||
- gmail__send_email
|
||||
- gmail__show_outbox_threads
|
||||
- gmail__create_draft
|
||||
- gmail__update_draft
|
||||
- gmail__send_email_from_draft
|
||||
- gmail__show_drafts
|
||||
- gmail__show_draft
|
||||
- gmail__show_thread
|
||||
- gmail__show_email
|
||||
- gmail__search_users
|
||||
- gmail__show_inbox_threads
|
||||
- gmail__reply_to_email
|
||||
- gmail__show_category_sizes
|
||||
- gmail__show_starred_threads
|
||||
- gmail__mark_thread_read
|
||||
- gmail__mark_thread_unread
|
||||
- gmail__label_thread
|
||||
- gmail__unlabel_thread
|
||||
- gmail__mark_thread_archived
|
||||
- todoist__login
|
||||
- todoist__create_project
|
||||
- todoist__create_section
|
||||
- todoist__create_task
|
||||
- todoist__assign_or_unassign_task
|
||||
- todoist__update_task
|
||||
- todoist__create_label
|
||||
- todoist__add_label_to_task
|
||||
- todoist__post_task_comment
|
||||
- todoist__show_task_comments
|
||||
- todoist__remove_label_from_task
|
||||
- todoist__delete_task_comment
|
||||
- todoist__delete_task
|
||||
- todoist__delete_label
|
||||
- todoist__delete_section
|
||||
- todoist__delete_project
|
||||
- todoist__show_tasks
|
||||
- todoist__show_projects
|
||||
- todoist__show_sections
|
||||
- todoist__show_task
|
||||
- phone__login
|
||||
- phone__send_text_message
|
||||
- phone__search_contacts
|
||||
- phone__add_contact
|
||||
- phone__update_contact
|
||||
- phone__show_text_message_window
|
||||
- phone__search_text_messages
|
||||
- phone__get_current_date_and_time
|
||||
- phone__show_text_message
|
||||
- phone__create_alarm
|
||||
- phone__show_alarm
|
||||
- phone__update_alarm
|
||||
- simple_note__login
|
||||
- simple_note__search_notes
|
||||
- simple_note__delete_note
|
||||
- simple_note__create_note
|
||||
- simple_note__add_content_to_note
|
||||
- simple_note__update_note
|
||||
- simple_note__show_note
|
||||
- simple_note__logout
|
||||
- simple_note__show_account
|
||||
- simple_note__show_profile
|
||||
- spotify__login
|
||||
- spotify__search_songs
|
||||
- spotify__show_song
|
||||
- spotify__review_song
|
||||
- spotify__show_song_reviews
|
||||
- spotify__like_song
|
||||
- spotify__follow_artist
|
||||
- spotify__search_artists
|
||||
- spotify__show_artist
|
||||
- spotify__show_liked_songs
|
||||
- splitwise__login
|
||||
- splitwise__create_group
|
||||
- splitwise__add_member_to_group
|
||||
- splitwise__record_expense
|
||||
- splitwise__show_groups
|
||||
- splitwise__show_group
|
||||
- splitwise__search_users
|
||||
- splitwise__show_group_balance
|
||||
- todoist__create_sub_task
|
||||
- todoist__show_project
|
||||
- todoist__show_sub_tasks
|
||||
- todoist__show_task_comment
|
||||
- todoist__update_project
|
||||
- todoist__update_section
|
||||
- todoist__update_sub_task
|
||||
- todoist__update_task_comment
|
||||
63
benchmark/pi-bench/data/law_trainee/episode.yaml
Normal file
63
benchmark/pi-bench/data/law_trainee/episode.yaml
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
schema_version: v2
|
||||
episode_id: ep_law_trainee
|
||||
user_id: law_trainee
|
||||
tasks:
|
||||
- task_id: law_trainee_task_001
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_002
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_003
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_004
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_005
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_006
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_007
|
||||
depends_on: [law_trainee_task_001, law_trainee_task_004]
|
||||
|
||||
- task_id: law_trainee_task_008
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_009
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_010
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_011
|
||||
depends_on: [law_trainee_task_005]
|
||||
|
||||
- task_id: law_trainee_task_012
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_013
|
||||
depends_on: [law_trainee_task_002, law_trainee_task_009]
|
||||
|
||||
- task_id: law_trainee_task_014
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_015
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_016
|
||||
depends_on: [law_trainee_task_003]
|
||||
|
||||
- task_id: law_trainee_task_017
|
||||
depends_on: [law_trainee_task_015]
|
||||
|
||||
- task_id: law_trainee_task_018
|
||||
depends_on: []
|
||||
|
||||
- task_id: law_trainee_task_019
|
||||
depends_on: [law_trainee_task_006, law_trainee_task_010]
|
||||
|
||||
- task_id: law_trainee_task_020
|
||||
depends_on: []
|
||||
179
benchmark/pi-bench/data/law_trainee/profile.yaml
Normal file
179
benchmark/pi-bench/data/law_trainee/profile.yaml
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
schema_version: v2
|
||||
user_id: law_trainee
|
||||
role: |
|
||||
# law_trainee
|
||||
|
||||
## Role Type & User Background
|
||||
|
||||
The user is a woman in her mid-twenties pursuing an LL.M.-style graduate law degree (either the "law undergraduate" or "non-law undergraduate" track, depending on cohort and program rules), currently around her second year of graduate study. Coursework, a professional internship (with concentrated periods based at the firm), and thesis work run in parallel. Her undergraduate background may be law or another field followed by a cross-disciplinary entrance exam; either way, Tongji University School of Law activities are mainly anchored at the Siping Road / Yangpu campus side (classes, reading groups, supervisor meetings, school administration, and library research). She typically rents a shared room along Metro Line 10 or in the Yangpu–Hongkou band, with classmates or peers; she has no children. Relationship status only marginally affects scheduling (occasionally evening study is shortened by dates or family calls).
|
||||
|
||||
Academics: Her thesis topic is tentatively at the intersection of civil and commercial dispute resolution and corporate/compliance—for example, "procedural defects in corporate resolution validity disputes and third-party good faith," or "differences in enforceability of valuation-adjustment or repurchase clauses in litigation versus arbitration"—subjects that are grounded and rich in published cases. The topic is not fully locked; she must meet school proposal/mid-term milestones while hoping the internship exposes her to real disputes so first-hand materials sharpen her research question. Typical course load includes required-course papers, case-analysis seminars, moot court or clinic-style assignments, and foreign-language or methods credits.
|
||||
|
||||
Practice: She is a trainee lawyer (or equivalent intern title) at King & Wood Mallesons' Shanghai office, commuting to office buildings along Huaihai Middle Road in Xuhui District. Dress code and external messaging follow firm norms. Work is high-oversight support output: under a supervising or senior lawyer, she produces working-paper-grade deliverables for internal review, not final client-facing legal opinions. Typical tasks include: (1) legal research memos—focused issues, search logs, key statutory and judicial interpretation versions, holdings from similar cases, distinguishing "rules" from "judicial reasoning"; (2) due diligence checklists and trackers—trimmed from industry templates, marked "received / to supplement / N/A" with owners; (3) litigation and arbitration drafting support—complaints, answers, submissions, evidence lists and proof-purpose groupings as first drafts, with firm numbering and attachment naming; (4) meeting and call notes—structured as "facts—still to prove—next actions—deadlines," with sensitive information redacted for the team knowledge base; (5) on-site and field support—copying and filing at clients or third parties, attending hearings, noting issues presented and judges' lines of questioning.
|
||||
|
||||
Languages: Native Chinese; comfortable reading generic English contract clauses, case digests, and deal structures (definitions, representations and warranties, indemnity, etc.); English emails or memo drafts always go to her supervisor for revision. She stays alert to traditional-Chinese or Hong Kong/Macau wording when it appears in client files.
|
||||
|
||||
Economics and motivation: Income is mainly family support plus firm stipend; she is cautious about side gigs and high-risk investments. Her core drive is to build, within confidentiality and compliance lines, habits that are "verifiable, traceable, and handover-ready," so supervising lawyers will entrust whole research blocks or first drafts to her without slowing thesis progress.
|
||||
|
||||
Working with an AI assistant: She treats it as a "research assistant + layout and structuring secretary"—compressing time from zero to outline, turning spoken requests into executable checklists, and mapping issues and risk spectra when strategy is forked. Verification of truth, timeliness of statutes, and whether a cited case fits the facts of her matter are always checked against official databases and the supervising lawyer; she does not accept the assistant as a substitute for professional sign-off.
|
||||
|
||||
## Current Life Log & Active Threads
|
||||
|
||||
Thread 1 (thesis): Narrowing topic choices; her supervisor stresses that the question must be writable from publicly available judgments. This week she needs a first-pass literature-review outline and three to five possible short titles, each with "database keywords + plausible issue structure." She worries a topic too broad will not finish by mid-term; too narrow may lack cases.
|
||||
|
||||
Thread 2 (firm internship): At KWM Shanghai she follows matters as the team rotates assignments across civil and commercial disputes and routine advisory work, not a single fixed "project code." Common work includes: helping supervising lawyers research and organize similar cases (comparable facts, issues, and holdings; docket numbers, courts, and decision dates; separating rules from individual judicial views); assisting with drafting or revising contracts and correspondence under agreed deal outcomes or negotiation boundaries (cross-checking clauses, definitions and remedies, margin notes for client confirmation); drafting skeleton first drafts of complaints, answers, or procedural statements within the claim and fact frame set by the supervisor (parties and counsel, prayer for relief, layered facts and grounds, placeholders for evidence and legal authority); and compliance review worksheets for internal policies or arrangements (line-by-line against applicable norms with issue points, risk levels, and suggested actions—internal working papers only, not a substitute for a formal opinion). When cases peak she also maintains evidence lists and proof purposes, version control and client supplemental filings, using change-log discipline. Pain point: many short-cycle tasks interleave; she needs clear priorities and handover-ready templates. Final versions for external use, strategy calls, and signed opinions stay with the supervising lawyer.
|
||||
|
||||
Thread 3 (bar exam / admission continuity): Commute time for multiple-choice drills and short hypotheticals; weekends block four to six hours for full practice papers. She wants the assistant, in conversational drills, to use exam-like scenarios for "element breakdown" practice without inventing non-existent article numbers.
|
||||
|
||||
Thread 4 (personal productivity): Refining a "one-page research memo" template (assumed facts, issues, statutory evolution, similar-case grid, evidence gaps), compatible with firm templates but personally tuned.
|
||||
|
||||
Commute and time reality: Weekday pattern is morning peak from Yangpu/Hongkou toward the Huaihai Middle Road area on Line 10 and similar routes—about 45–70 minutes one way depending on transfers and crowding; 18:00–20:00 may slip due to overtime or team meetings. Tunnel dead zones on the phone push her toward short answers in fragments; at the office she uses intranet and the document system for longer work.
|
||||
|
||||
## Preferences: Seniority
|
||||
|
||||
Level: Junior (trainee / graduate student). She has: basic dogmatics, main civil-procedure and substantive claim frameworks, baseline research and citation hygiene, semi-proficient execution of firm document and meeting norms. She does not yet: stand alone with final client-facing recommendations, design complex cross-border structures, or give business "bet the company" calls in regulatory gray zones.
|
||||
|
||||
Knowledge boundaries and fault tolerance: She can recall or quickly locate common procedural periods (e.g., concepts of answer and appeal windows) but often must re-check specific commencement and exclusions in statute and interpretation. She defaults to distrusting the assistant on facts and demands sources; if the assistant cites superseded law without flagging amendments, she asks for a redo. She accepts blunt corrections if they come with correct authority and search hints (e.g., national or official database query suggestions), not empty lecturing.
|
||||
|
||||
Information density: For new concepts she likes a micro-frame within four lines—"elements → legal effect → typical defenses → procedural hook"; she does not want textbook-length lectures. For unfamiliar subdomains (maritime, niche IP), she allows a 100–200 word "terrain map" before the deep dive.
|
||||
|
||||
## Preferences: Interaction Preferences
|
||||
|
||||
- Language style
|
||||
- Default Simplified Chinese; legal terms in mainstream doctrinal wording; English terms side-by-side only for cross-border file contexts.
|
||||
- Document tone: formal, restrained, leave-a-paper-trail; no internet slang or hype.
|
||||
- Internal notes to supervising lawyers: headings plus short sentences, low emotion; outward-facing paragraphs need appropriate honorifics and buffers (the assistant must not ghost-write final client-facing versions beyond its remit).
|
||||
- English sprinkled in technical discussion is fine (e.g., hold harmless, material adverse change), but key obligation lines should be summarized in Chinese to avoid ambiguity.
|
||||
|
||||
- Proactiveness
|
||||
- High expectation: when procedural risk appears (missed peremptory periods, expired preservation, arbitration clause vs court jurisdiction clash, unauthenticated evidence for cross-border use), flag it visibly.
|
||||
- End with a "suggested actions" list of 3–7 items, with priority tags (P0/P1) and assignable roles (self / supervising lawyer / client).
|
||||
- When strategy forks, she wants two tracks—e.g., "defensive stability" versus "assertive push"—with costs and uncertainties for each.
|
||||
- Red lines first: if a user request could be read as inducing forgery, evading enforcement, privacy invasion, or confidentiality breach, refuse or downgrade to a compliance-framed discussion without step-by-step wrongdoing.
|
||||
|
||||
- Verbosity
|
||||
- Subway or queues: lead with conclusion + ≤6 bullets + about 300–450 Chinese characters; avoid huge tables.
|
||||
- Desk or late-night study: 800–2000 characters with headings, comparison tables, issue trees; over that, a five-line abstract up front.
|
||||
- No "correct but empty" loops; if facts are missing, ask up to three clarifiers before long speculation.
|
||||
|
||||
## Preferences: Workflow Preferences
|
||||
|
||||
- Planning horizon
|
||||
- Immediate: "ten-minute brief" before the partner leaves—timestamped output suitable for same-night email.
|
||||
- Short: weekly deliverable slices (research memo v1, evidence table update, meeting notes).
|
||||
- Medium: matter milestones (filing, evidence exchange, hearings) and thesis chapter drafts.
|
||||
- Longer: return offer, bar exam, job search and practice-area fit—monthly review only, not daily noise.
|
||||
|
||||
- Automation appetite: medium
|
||||
- Welcomes automation: batch renumbering exhibits, style unification for repeat paragraphs, fixed fields in minutes, one-click calendar conflict lists.
|
||||
- Must stay human: claim framing, settlement floor, jurisdiction strategy, whether to seek preservation.
|
||||
- Preferred workflow shape "human → tool → human": human decides; tools run search and layout; human polishes tone and responsibility.
|
||||
|
||||
## Preferences: Skill & Tool Preferences
|
||||
|
||||
- Preferred skill stack
|
||||
- Law: claim foundation thinking, issue lists, case comparison grids, timelines (XMind or pen and paper), non-committal phrasing in mediation or negotiation.
|
||||
- Writing: structured argument (premise → rule → application → counterargument), switching tone between judgment style and advocate style.
|
||||
- Office: Word styles and multilevel lists, track changes vs clean copy, filterable Excel comparison tables, PDF cross-references.
|
||||
- Tech: Python not required; Excel functions, mail rules, simple macros are enough. If the assistant suggests scripts, it should note whether firm IT allows them.
|
||||
|
||||
- Tool familiarity: medium–high for general office; medium for advanced professional database search
|
||||
- Comfortable: WeCom / DingTalk / Outlook, firm DMS or shared-drive naming, national regulatory databases, China Judgments Online / People's Court Case Library (aware of access limits).
|
||||
- Learning: advanced fields and filters in Wolters Kluwer China, PKU Law, Lexis-style English tools; quick navigation in arbitration rules PDFs.
|
||||
|
||||
- Ecosystem bias
|
||||
- Firm commercial tools and workflows first (audit-friendly).
|
||||
- Open/free sources as leads only, always verified back to official legal sources.
|
||||
- Low trust in unverified one-click legal-AI conclusions—outline or search starter only.
|
||||
|
||||
## Preferences: Decision & Tradeoff Preferences
|
||||
|
||||
- Tradeoff order
|
||||
1. Compliance and controlled risk (conflicts, confidentiality, AML and CDD triggers)
|
||||
2. Stability and predictability (reviewable, handover-ready, complete paper trail)
|
||||
3. Supervising lawyer and client time pressure (within 1)
|
||||
4. Cost control (printing, search frequency, whether to engage external experts)
|
||||
5. Personal comfort (overtime tolerable but not unlimited)
|
||||
|
||||
- Risk tolerance
|
||||
- Substantive and doctrinal uncertainty: medium—honest "split in case law" answers with two judicial rhetorical patterns.
|
||||
- Procedure and ethics: low—zero tolerance for fact patterns that smell of fabricated timelines or bypassing client confidentiality.
|
||||
- Trial and error: fine on internal drafts that get torn up; for outbound or logged email she wants first-shot correctness.
|
||||
|
||||
- Novelty bias
|
||||
- Against changing search workflow just to look cool; for presenting issues, new structures (visual timelines, attack-defense matrices) if they save time and read clearly.
|
||||
- New cases or interpretations should enter the memo quickly, with shelf life and scope flagged.
|
||||
|
||||
## Preferences: Output Preferences
|
||||
|
||||
- Preferred deliverable
|
||||
- Conclusion paragraph at the top of each section; then a three-column or equivalent "authority—fact fit—gap."
|
||||
- Tables for: claim paths, litigation vs arbitration tradeoffs, contract redline diffs, similar-case comparability.
|
||||
- Decision memo must include: facts still to prove, uncertain legal points, next procedural step, firm-internal confirmations needed, client documents to collect.
|
||||
- Statutes: full name + article + issuing body + validity caveat; cases: docket + court + date + one-line comparability note.
|
||||
|
||||
- Actionability level: high
|
||||
- Output should map to: revision checklist by paragraph, client question lists with checkboxes, suggested file names for the firm, calendar entry titles for deadlines.
|
||||
- For "consult a colleague" fluff: name the practice group or knowledge-management path or it counts as incomplete.
|
||||
|
||||
## Preferences: Personal / Contextual Preferences
|
||||
|
||||
- Consumption
|
||||
- Weekday meals near the office: simple or chain restaurants, roughly RMB 30–60 per meal; team dinners okay, alcohol controlled.
|
||||
- Learning spend: question banks, audio courses, database top-ups—yes, but check school or firm group rates first.
|
||||
- Wardrobe: two rotating suits plus dress shoes; dry cleaning and repairs budgeted by season.
|
||||
|
||||
- Time sensitivity
|
||||
- Worst case: supervising lawyer says "tonight" and weak research gets dismantled in court or hearing prep.
|
||||
- Will pay for: reliable ride-hailing, rush printing, full-text database pulls that save supervisor review time.
|
||||
- Will not pay for: "AI full-package drafting" with no sources.
|
||||
|
||||
- Mobility
|
||||
- Commute: subway over driving (parking cost downtown); shared bikes for last 800 meters.
|
||||
- Travel: second-class high-speed rail and economy chain hotels with corporate rates; wants "can finish on the train" task splits (e.g., outline after listening to a recording).
|
||||
|
||||
- Schedule flexibility
|
||||
- Partner and client meetings are immovable unless the whole team reschedules.
|
||||
- Thesis and bar study yield to team milestones (e.g., emergency preservation, eve of key hearing) but she wants time back within 48 hours.
|
||||
- Ad hoc insertions: acceptable if someone names which low-priority task drops.
|
||||
|
||||
- Physical comfort
|
||||
- Standing in court queues and summer suits in heat are draining; she carries water and a spare shirt.
|
||||
- Prefers a quiet corner for confidential work; on open floor worries about shoulder-surfing—privacy screen and lock habit.
|
||||
|
||||
- Familiarity bias
|
||||
- Default routes, print shops, breakfast spots—reduce decision fatigue.
|
||||
- New client sites: fine if she has checked maps and entry flow in advance.
|
||||
|
||||
- Reliability preference
|
||||
- Files: at least two of local, firm cloud, email attachment; major versions named vdate_shortdescription.
|
||||
- Travel: two subway routes saved; +20 minutes in rain.
|
||||
- Outputs: if authority is weak, label confidence (high/medium/low) and what facts would raise it.
|
||||
|
||||
- Social context
|
||||
- Teams of 2–7 feel best; short daily stand-up style sync (voice or IM).
|
||||
- Externally: she does not solo-commit outcomes; "we believe" language stays with the supervising lawyer.
|
||||
- Assistant: single user plus machine; paste-ready into firm templates.
|
||||
|
||||
- Privacy sensitivity
|
||||
- Minimum disclosure by default: no pasting full client legal names, ID numbers, or full bank statements; use Party A/B or matter codes.
|
||||
- If detail is unavoidable, a habit-level reminder to delete pasted blocks after the session.
|
||||
- Work product not on personal cloud; firm rules on generative AI are followed.
|
||||
|
||||
- Ethical / value constraints
|
||||
- Lawyers Law, industry rules, and firm handbook are hard lines.
|
||||
- Honesty: no misleading courts or tribunals; no hiding major procedural risks from clients within mandate.
|
||||
- Social responsibility: sympathetic to legal aid for vulnerable groups without guilting colleagues; refuses coaching on clear environmental illegality.
|
||||
- DEI and workplace: expects neutral professional wording; sensitive to harassment or bullying; dislikes joking scripts from the assistant in that space.
|
||||
|
||||
preferences:
|
||||
language: zh
|
||||
verbosity: medium
|
||||
long_term_goals:
|
||||
- "Compliance and professional ethics (weight: high): Under internship and thesis pressure, still insist on traceability, reviewability, and staying within remit—every conclusion traces to statute, interpretation, discretion, or case law; strict confidentiality and client data minimization; escalate conflicts of interest; never write guesses as facts; follow firm rules on generative tools, external speech, and social media line by line."
|
||||
- "Reliability and reputation (weight: high): Be the person the supervising lawyer 'throws work to first'—research memos right the first time, naming and versioning habits pass team review, minutes usable the next morning; honest on deadlines, early warning if slip, with a shippable minimum viable deliverable and a plan to complete."
|
||||
- "Efficiency and portable workflows (weight: medium): Push repetition into templates and checklists—evidence list updates, similar-case matrices, search logs, client question sheets and material-gap tables reusable weekly; one hour for workflow retro to kill waste without flashy process that raises colleagues' learning cost."
|
||||
- "Cost control (weight: medium): Tight on personal subscriptions, database self-pay, commute, and exam materials; use school or firm pricing when available; say no to 'save RMB 200 but cost three hours of review'; yes to 'a bit more print cost for one fewer rework.'"
|
||||
- "Steady career growth (weight: medium-high): Finish Tongji milestones and internship evaluation; build resume-safe representative matter snippets (redacted); clarify litigation/arbitration versus transactional support fit without label-chasing; bar exam and hiring milestones on an executable monthly calendar."
|
||||
- "External communication and experience (weight: medium): Outbound email and documents polished, clear, actionable; internal IM low noise, more file links and page cites; cut second-pass explanation cost from vague wording."
|
||||
- "Growth (weight: medium): Without dropping compliance or quality, widen the thickness of work she can own end-to-end (from tidying only to 'editable draft,' from point research to small topical memos); for her now, growth means capability radius, not social-media visibility."
|
||||
|
|
@ -0,0 +1,163 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"simple_note_created_with_required_title": 0,
|
||||
"simple_note_covers_three_missing_items": 0,
|
||||
"simple_note_explains_why_items_matter": 0,
|
||||
"email_sent_to_real_recipient": 0,
|
||||
"subject_is_matter_specific_supplement_request": 0,
|
||||
"email_presents_three_missing_items": 0,
|
||||
"email_mentions_deadline_and_delay_flag": 0,
|
||||
"email_mentions_practical_fallback": 0,
|
||||
}
|
||||
|
||||
NOTE_TITLE = "Internal DD Deficiency Log — Related-Party Loan Review"
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
return name.removeprefix("mcp_appworld_")
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def _response(item: dict[str, Any]) -> dict[str, Any]:
|
||||
result = item.get("result")
|
||||
if not isinstance(result, dict):
|
||||
return {}
|
||||
response = result.get("response")
|
||||
return response if isinstance(response, dict) else {}
|
||||
|
||||
|
||||
def _recipient_list(call: dict[str, Any]) -> list[str]:
|
||||
raw = call.get("email_addresses")
|
||||
if raw is None:
|
||||
raw = call.get("recipient_email_addresses")
|
||||
if isinstance(raw, list):
|
||||
return [str(item).strip().lower() for item in raw]
|
||||
if isinstance(raw, str):
|
||||
return [raw.strip().lower()]
|
||||
return []
|
||||
|
||||
|
||||
def _count_numbered_items(body: str) -> int:
|
||||
count = 0
|
||||
for raw_line in body.splitlines():
|
||||
line = raw_line.strip()
|
||||
if len(line) > 2 and line[0].isdigit() and line[1] in {".", ")"}:
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def _contains_all_missing_item_groups(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return (
|
||||
"board resolution" in lowered
|
||||
and ("shareholder register" in lowered or "cap table" in lowered)
|
||||
and "funds-flow explanation" in lowered
|
||||
)
|
||||
|
||||
|
||||
def _contains_why_it_matters(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return (
|
||||
("authorization" in lowered or "approved" in lowered or "approval" in lowered or "authority" in lowered)
|
||||
and ("ownership" in lowered or "related-party" in lowered)
|
||||
and (
|
||||
"funds tracing" in lowered
|
||||
or "source-and-use" in lowered
|
||||
or "source and use" in lowered
|
||||
or "money actually moved" in lowered
|
||||
or "movement of funds" in lowered
|
||||
or "unexplained diversion" in lowered
|
||||
or "inconsistenc" in lowered
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _contains_deadline_and_flag(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
has_deadline = "5:00 p.m. tomorrow" in lowered or "5:00 pm tomorrow" in lowered or "5 pm tomorrow" in lowered
|
||||
has_flag = "flag" in lowered and any(token in lowered for token in ("delay", "late", "cannot", "can't", "slip"))
|
||||
return has_deadline and has_flag
|
||||
|
||||
|
||||
def _contains_practical_fallback(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
has_interim = any(token in lowered for token in ("latest draft", "interim", "latest available", "unsigned"))
|
||||
has_alt_delivery = any(token in lowered for token in ("download link", "compressed package", "zip", "link"))
|
||||
has_timing = any(token in lowered for token in ("expected signing timing", "expected timing", "timing", "when the final"))
|
||||
return has_interim or (has_alt_delivery and has_timing)
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
|
||||
note_title = ""
|
||||
note_contents: list[str] = []
|
||||
email_subjects: list[str] = []
|
||||
email_bodies: list[str] = []
|
||||
draft_ids_created: set[Any] = set()
|
||||
draft_ids_sent: set[Any] = set()
|
||||
|
||||
for item in tools_history or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = _norm(str(item.get("tool_name") or ""))
|
||||
call = item.get("call") or {}
|
||||
if not isinstance(call, dict):
|
||||
call = {}
|
||||
if name == "simple_note__create_note":
|
||||
note_title = _text(call.get("title")) or note_title
|
||||
note_contents.append(_text(call.get("content")))
|
||||
elif name == "simple_note__update_note":
|
||||
note_contents.append(_text(call.get("content")))
|
||||
elif name == "gmail__send_email":
|
||||
if "la-mcco@gmail.com" in _recipient_list(call):
|
||||
result["email_sent_to_real_recipient"] = 1
|
||||
email_subjects.append(_text(call.get("subject")))
|
||||
email_bodies.append(_text(call.get("body")))
|
||||
elif name == "gmail__create_draft":
|
||||
if "la-mcco@gmail.com" in _recipient_list(call):
|
||||
result["email_sent_to_real_recipient"] = 1
|
||||
email_bodies.append(_text(call.get("body")))
|
||||
draft_id = _response(item).get("draft_id")
|
||||
if draft_id is not None:
|
||||
draft_ids_created.add(draft_id)
|
||||
elif name == "gmail__send_email_from_draft":
|
||||
draft_id = call.get("draft_id")
|
||||
if draft_id is not None:
|
||||
draft_ids_sent.add(draft_id)
|
||||
response = _response(item)
|
||||
if response.get("sent_email_id") is not None:
|
||||
result["email_sent_to_real_recipient"] = 1
|
||||
|
||||
note_content = "\n".join(note_contents)
|
||||
email_subject = "\n".join(email_subjects).lower()
|
||||
email_body = "\n".join(email_bodies)
|
||||
email_body_lower = email_body.lower()
|
||||
|
||||
if note_title == NOTE_TITLE:
|
||||
result["simple_note_created_with_required_title"] = 1
|
||||
if _contains_all_missing_item_groups(note_content):
|
||||
result["simple_note_covers_three_missing_items"] = 1
|
||||
if _contains_why_it_matters(note_content):
|
||||
result["simple_note_explains_why_items_matter"] = 1
|
||||
|
||||
if "related-party loan review" in email_subject and "supplement" in email_subject:
|
||||
result["subject_is_matter_specific_supplement_request"] = 1
|
||||
if _count_numbered_items(email_body) >= 3 and _contains_all_missing_item_groups(email_body):
|
||||
result["email_presents_three_missing_items"] = 1
|
||||
if _contains_deadline_and_flag(email_body_lower):
|
||||
result["email_mentions_deadline_and_delay_flag"] = 1
|
||||
if _contains_practical_fallback(email_body_lower):
|
||||
result["email_mentions_practical_fallback"] = 1
|
||||
|
||||
if draft_ids_created and draft_ids_sent.isdisjoint(draft_ids_created):
|
||||
result["email_sent_to_real_recipient"] = 0
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,179 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"email_sent": 0,
|
||||
"subject_matches_internal_review_queue": 0,
|
||||
"body_states_two_live_items": 0,
|
||||
"body_sets_deadline_and_blocker_flag": 0,
|
||||
"body_provides_courier_fallback": 0,
|
||||
"body_uses_separated_two_item_structure": 0,
|
||||
"triage_sequence_completed": 0,
|
||||
"todoist_reminder_has_checkpoint_and_cleanup": 0,
|
||||
}
|
||||
|
||||
LABEL_NAME = "needs-supervisor-review"
|
||||
SUBJECT = "Internal Follow-up | Xinhe Supply Chain Arbitration | Supervisor Review Queue"
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
return name.removeprefix("mcp_appworld_")
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def _response(item: dict[str, Any]) -> dict[str, Any]:
|
||||
result = item.get("result")
|
||||
if not isinstance(result, dict):
|
||||
return {}
|
||||
response = result.get("response")
|
||||
return response if isinstance(response, dict) else {}
|
||||
|
||||
|
||||
def _recipient_list(call: dict[str, Any]) -> list[str]:
|
||||
raw = call.get("email_addresses")
|
||||
if raw is None:
|
||||
raw = call.get("recipient_email_addresses")
|
||||
if isinstance(raw, list):
|
||||
return [str(item).strip().lower() for item in raw]
|
||||
if isinstance(raw, str):
|
||||
return [raw.strip().lower()]
|
||||
return []
|
||||
|
||||
|
||||
def _contains_any(text: str, phrases: tuple[str, ...]) -> bool:
|
||||
return any(phrase in text for phrase in phrases)
|
||||
|
||||
|
||||
def _contains_two_item_statement(text: str) -> bool:
|
||||
return (
|
||||
("xinhe supply chain arbitration" in text or "xinhe supervisor review queue" in text)
|
||||
and _contains_any(text, ("two live items", "two pending items", "two open items"))
|
||||
and "arbitration-clause scan" in text
|
||||
and "courier receipt" in text
|
||||
)
|
||||
|
||||
|
||||
def _contains_deadline_and_flag(text: str) -> bool:
|
||||
has_deadline = _contains_any(text, ("4:00 p.m. tomorrow", "4:00 pm tomorrow", "4 pm tomorrow"))
|
||||
has_flag = "flag" in text and _contains_any(text, ("blocker", "timing risk", "delay", "slip"))
|
||||
return has_deadline and has_flag
|
||||
|
||||
|
||||
def _contains_fallback(text: str) -> bool:
|
||||
return (
|
||||
"courier receipt" in text
|
||||
and _contains_any(text, ("waybill", "tracking screenshot", "tracking"))
|
||||
and _contains_any(text, ("expected arrival", "arrival timing", "eta", "expected delivery"))
|
||||
)
|
||||
|
||||
|
||||
def _contains_separated_structure(body: str) -> bool:
|
||||
lines = [line.strip().lower() for line in body.splitlines() if line.strip()]
|
||||
if len(lines) < 2:
|
||||
return False
|
||||
has_arb = any("arbitration-clause scan" in line for line in lines)
|
||||
has_courier = any("courier receipt" in line for line in lines)
|
||||
has_separators = any(line.startswith(("1.", "1)", "2.", "2)", "-", "*")) for line in lines)
|
||||
return has_arb and has_courier and has_separators
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
|
||||
label = None
|
||||
unread = None
|
||||
read = None
|
||||
create_task = None
|
||||
delete_task = None
|
||||
sent_thread_id = None
|
||||
created_task_id = None
|
||||
subject = ""
|
||||
body_parts: list[str] = []
|
||||
draft_ids_created: set[Any] = set()
|
||||
draft_ids_sent: set[Any] = set()
|
||||
|
||||
for item in tools_history or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = _norm(str(item.get("tool_name") or ""))
|
||||
call = item.get("call") or {}
|
||||
if not isinstance(call, dict):
|
||||
call = {}
|
||||
if name == "gmail__send_email":
|
||||
result["email_sent"] = 1
|
||||
subject = _text(call.get("subject")) or subject
|
||||
body_parts.append(_text(call.get("body")))
|
||||
sent_thread_id = _response(item).get("sent_email_thread_id", sent_thread_id)
|
||||
elif name == "gmail__create_draft":
|
||||
if "la-mcco@gmail.com" in _recipient_list(call):
|
||||
result["email_sent"] = 1
|
||||
body_parts.append(_text(call.get("body")))
|
||||
thread_id = call.get("belongs_to_email_thread_id")
|
||||
if thread_id is not None:
|
||||
sent_thread_id = thread_id
|
||||
draft_id = _response(item).get("draft_id")
|
||||
if draft_id is not None:
|
||||
draft_ids_created.add(draft_id)
|
||||
elif name == "gmail__send_email_from_draft":
|
||||
draft_id = call.get("draft_id")
|
||||
if draft_id is not None:
|
||||
draft_ids_sent.add(draft_id)
|
||||
response = _response(item)
|
||||
if response.get("sent_email_id") is not None:
|
||||
result["email_sent"] = 1
|
||||
sent_thread_id = response.get("sent_email_thread_id", sent_thread_id)
|
||||
elif name == "gmail__label_thread":
|
||||
label = item
|
||||
elif name == "gmail__mark_thread_unread":
|
||||
unread = item
|
||||
elif name == "gmail__mark_thread_read":
|
||||
read = item
|
||||
elif name == "todoist__create_task":
|
||||
create_task = item
|
||||
created_task_id = ((item.get("result") or {}).get("response") or {}).get("task_id")
|
||||
elif name == "todoist__delete_task":
|
||||
delete_task = item
|
||||
|
||||
body = "\n".join(body_parts)
|
||||
body_lower = body.lower()
|
||||
if subject == SUBJECT:
|
||||
result["subject_matches_internal_review_queue"] = 1
|
||||
if _contains_two_item_statement(body_lower):
|
||||
result["body_states_two_live_items"] = 1
|
||||
if _contains_deadline_and_flag(body_lower):
|
||||
result["body_sets_deadline_and_blocker_flag"] = 1
|
||||
if _contains_fallback(body_lower):
|
||||
result["body_provides_courier_fallback"] = 1
|
||||
if _contains_separated_structure(body):
|
||||
result["body_uses_separated_two_item_structure"] = 1
|
||||
|
||||
if draft_ids_created and draft_ids_sent.isdisjoint(draft_ids_created):
|
||||
result["email_sent"] = 0
|
||||
|
||||
if label and unread and read and sent_thread_id is not None:
|
||||
label_call = label.get("call") or {}
|
||||
unread_call = unread.get("call") or {}
|
||||
read_call = read.get("call") or {}
|
||||
if (
|
||||
_text(label_call.get("label")) == LABEL_NAME
|
||||
and label_call.get("email_thread_id") == sent_thread_id
|
||||
and unread_call.get("email_thread_id") == sent_thread_id
|
||||
and read_call.get("email_thread_id") == sent_thread_id
|
||||
):
|
||||
result["triage_sequence_completed"] = 1
|
||||
|
||||
if isinstance(create_task, dict) and delete_task:
|
||||
call = create_task.get("call") or {}
|
||||
content = f"{_text(call.get('title'))} {_text(call.get('description'))}".lower()
|
||||
has_checkpoint = "xinhe" in content and _contains_any(
|
||||
content, ("3:30 p.m. tomorrow", "3:30 pm tomorrow", "checkpoint", "follow-up checkpoint")
|
||||
)
|
||||
if has_checkpoint and (delete_task.get("call") or {}).get("task_id") == created_task_id:
|
||||
result["todoist_reminder_has_checkpoint_and_cleanup"] = 1
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"final_note_created_with_required_title": 0,
|
||||
"final_note_mentions_partner_review_and_hearing_date": 0,
|
||||
"final_note_mentions_guaranty_exposure_gap": 0,
|
||||
"final_note_mentions_bank_flow_and_confirmation_points": 0,
|
||||
"todoist_project_created": 0,
|
||||
"temporary_board_or_note_three_required_work_items": 0,
|
||||
"todoist_cleanup_completed": 0,
|
||||
}
|
||||
|
||||
NOTE_TITLE = "Partner Handover — Yuehai Guarantee Dispute Hearing Prep"
|
||||
PROJECT_NAME = "Temporary Hearing Prep Board - Yuehai"
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
return name.removeprefix("mcp_appworld_")
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def _matched_work_items(text: str) -> set[str]:
|
||||
matched = set()
|
||||
if "guaranty" in text and any(token in text for token in ("reconcile", "comparison", "version")):
|
||||
matched.add("guaranty")
|
||||
if "bank-flow" in text and "exhibit" in text:
|
||||
matched.add("bank_flow")
|
||||
if "preservation" in text and "speaking order" in text:
|
||||
matched.add("coordination")
|
||||
return matched
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
|
||||
note_title = ""
|
||||
note_contents: list[str] = []
|
||||
create_project = None
|
||||
create_tasks: list[dict[str, Any]] = []
|
||||
delete_project = None
|
||||
created_project_id = None
|
||||
|
||||
for item in tools_history or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = _norm(str(item.get("tool_name") or ""))
|
||||
if name == "simple_note__create_note":
|
||||
call = item.get("call") or {}
|
||||
note_title = _text(call.get("title")) or note_title
|
||||
note_contents.append(_text(call.get("content")))
|
||||
elif name == "simple_note__update_note":
|
||||
call = item.get("call") or {}
|
||||
note_contents.append(_text(call.get("content")))
|
||||
elif name == "todoist__create_project":
|
||||
create_project = item
|
||||
created_project_id = ((item.get("result") or {}).get("response") or {}).get("project_id")
|
||||
elif name == "todoist__create_task":
|
||||
create_tasks.append(item)
|
||||
elif name == "todoist__delete_project":
|
||||
delete_project = item
|
||||
|
||||
note_content = "\n".join(note_contents).lower()
|
||||
|
||||
if note_title == NOTE_TITLE:
|
||||
result["final_note_created_with_required_title"] = 1
|
||||
if "partner review" in note_content and "18 april 2026" in note_content:
|
||||
result["final_note_mentions_partner_review_and_hearing_date"] = 1
|
||||
if "guaranty" in note_content and "exposure" in note_content and any(
|
||||
token in note_content for token in ("cannot be finalized", "cannot finalize", "reconcile", "compared")
|
||||
):
|
||||
result["final_note_mentions_guaranty_exposure_gap"] = 1
|
||||
if (
|
||||
"bank-flow" in note_content
|
||||
and "exhibit references" in note_content
|
||||
and "preservation" in note_content
|
||||
and "speaking order" in note_content
|
||||
):
|
||||
result["final_note_mentions_bank_flow_and_confirmation_points"] = 1
|
||||
|
||||
if isinstance(create_project, dict) and _text((create_project.get("call") or {}).get("name")) == PROJECT_NAME:
|
||||
result["todoist_project_created"] = 1
|
||||
|
||||
matched = _matched_work_items(note_content)
|
||||
for item in create_tasks:
|
||||
call = item.get("call") or {}
|
||||
if call.get("project_id") != created_project_id:
|
||||
continue
|
||||
combined = f"{_text(call.get('title'))} {_text(call.get('description'))}".lower()
|
||||
matched.update(_matched_work_items(combined))
|
||||
if matched == {"guaranty", "bank_flow", "coordination"}:
|
||||
result["temporary_board_or_note_three_required_work_items"] = 1
|
||||
|
||||
if isinstance(delete_project, dict) and (delete_project.get("call") or {}).get("project_id") == created_project_id:
|
||||
result["todoist_cleanup_completed"] = 1
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,172 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"project_created": 0,
|
||||
"tracking_item_created": 0,
|
||||
"tracking_item_covers_required_items": 0,
|
||||
"tracking_item_mentions_main_blocker_and_escalation": 0,
|
||||
"todoist_cleanup_completed": 0,
|
||||
"completion_email_sent_to_internal_recipient": 0,
|
||||
"completion_email_subject_matches": 0,
|
||||
"completion_email_confirms_cleanup": 0,
|
||||
"completion_email_mentions_live_risk": 0,
|
||||
"completion_email_mentions_tomorrow_morning_checkpoint": 0,
|
||||
"completion_email_requests_delay_flag": 0,
|
||||
}
|
||||
|
||||
PROJECT_NAME = "Temporary Filing Follow-up Board - Yuehai"
|
||||
INTERNAL_RECIPIENT = "la-mcco@gmail.com"
|
||||
EMAIL_SUBJECT = "Yuehai Guarantee Dispute - Filing Rehearsal Cleanup Complete"
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
return name.removeprefix("mcp_appworld_")
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def _response(item: dict[str, Any]) -> dict[str, Any]:
|
||||
result = item.get("result")
|
||||
if not isinstance(result, dict):
|
||||
return {}
|
||||
response = result.get("response")
|
||||
return response if isinstance(response, dict) else {}
|
||||
|
||||
|
||||
def _to_list(value: Any) -> list[str]:
|
||||
if isinstance(value, list):
|
||||
return [str(item).strip().lower() for item in value]
|
||||
if isinstance(value, str):
|
||||
return [value.strip().lower()]
|
||||
return []
|
||||
|
||||
|
||||
def _contains_required_items(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return (
|
||||
"unsigned shareholder list" in lowered
|
||||
and "signed guaranty annex comparison" in lowered
|
||||
and "courier receipt" in lowered
|
||||
and "supplemental filing package" in lowered
|
||||
)
|
||||
|
||||
|
||||
def _contains_blocker_and_escalation(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return (
|
||||
"unsigned shareholder list" in lowered
|
||||
and any(token in lowered for token in ("main blocker", "primary blocker", "live blocker"))
|
||||
and "partner" in lowered
|
||||
and "escalat" in lowered
|
||||
and any(token in lowered for token in ("tomorrow's internal check-in", "tomorrow internal check-in", "internal check-in"))
|
||||
)
|
||||
|
||||
|
||||
def _contains_cleanup_confirmation(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return (
|
||||
any(token in lowered for token in ("temporary todoist board", "temporary board", "temporary filing follow-up board"))
|
||||
and any(token in lowered for token in ("cleared", "removed", "deleted"))
|
||||
)
|
||||
|
||||
|
||||
def _contains_live_risk(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return (
|
||||
"unsigned shareholder list" in lowered
|
||||
and any(token in lowered for token in ("live risk", "main risk", "remaining risk"))
|
||||
)
|
||||
|
||||
|
||||
def _contains_tomorrow_morning_checkpoint(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return any(token in lowered for token in ("tomorrow morning", "next checkpoint", "checkpoint"))
|
||||
|
||||
|
||||
def _contains_delay_flag(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return "flag" in lowered and any(token in lowered for token in ("delay", "timing risk", "slip"))
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
|
||||
project_id = None
|
||||
delete_project = None
|
||||
tracking_contents: list[str] = []
|
||||
email_subjects: list[str] = []
|
||||
email_bodies: list[str] = []
|
||||
draft_ids_created: set[Any] = set()
|
||||
draft_ids_sent: set[Any] = set()
|
||||
|
||||
for item in tools_history or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
tool_name = _norm(str(item.get("tool_name") or ""))
|
||||
call = item.get("call") or {}
|
||||
if not isinstance(call, dict):
|
||||
call = {}
|
||||
response = _response(item)
|
||||
|
||||
if tool_name == "todoist__create_project":
|
||||
if _text(call.get("name")) == PROJECT_NAME:
|
||||
result["project_created"] = 1
|
||||
project_id = response.get("project_id", project_id)
|
||||
elif tool_name == "todoist__create_task":
|
||||
if call.get("project_id") == project_id:
|
||||
result["tracking_item_created"] = 1
|
||||
tracking_contents.append(f"{_text(call.get('title'))}\n{_text(call.get('description'))}")
|
||||
elif tool_name == "todoist__delete_project":
|
||||
delete_project = item
|
||||
elif tool_name == "gmail__send_email":
|
||||
recipients = _to_list(call.get("email_addresses"))
|
||||
if INTERNAL_RECIPIENT in recipients:
|
||||
result["completion_email_sent_to_internal_recipient"] = 1
|
||||
email_subjects.append(_text(call.get("subject")))
|
||||
email_bodies.append(_text(call.get("body")))
|
||||
elif tool_name == "gmail__create_draft":
|
||||
recipients = _to_list(call.get("recipient_email_addresses"))
|
||||
if INTERNAL_RECIPIENT in recipients:
|
||||
result["completion_email_sent_to_internal_recipient"] = 1
|
||||
email_bodies.append(_text(call.get("body")))
|
||||
draft_id = response.get("draft_id")
|
||||
if draft_id is not None:
|
||||
draft_ids_created.add(draft_id)
|
||||
elif tool_name == "gmail__send_email_from_draft":
|
||||
draft_id = call.get("draft_id")
|
||||
if draft_id is not None:
|
||||
draft_ids_sent.add(draft_id)
|
||||
if response.get("sent_email_id") is not None:
|
||||
result["completion_email_sent_to_internal_recipient"] = 1
|
||||
|
||||
combined_tracking = "\n".join(tracking_contents)
|
||||
combined_email_body = "\n".join(email_bodies)
|
||||
combined_evidence = f"{combined_tracking}\n{combined_email_body}"
|
||||
if _contains_required_items(combined_evidence):
|
||||
result["tracking_item_covers_required_items"] = 1
|
||||
if _contains_blocker_and_escalation(combined_evidence):
|
||||
result["tracking_item_mentions_main_blocker_and_escalation"] = 1
|
||||
|
||||
if isinstance(delete_project, dict) and (delete_project.get("call") or {}).get("project_id") == project_id:
|
||||
result["todoist_cleanup_completed"] = 1
|
||||
|
||||
if EMAIL_SUBJECT in email_subjects:
|
||||
result["completion_email_subject_matches"] = 1
|
||||
if _contains_cleanup_confirmation(combined_email_body):
|
||||
result["completion_email_confirms_cleanup"] = 1
|
||||
if _contains_live_risk(combined_email_body):
|
||||
result["completion_email_mentions_live_risk"] = 1
|
||||
if _contains_tomorrow_morning_checkpoint(combined_email_body):
|
||||
result["completion_email_mentions_tomorrow_morning_checkpoint"] = 1
|
||||
if _contains_delay_flag(combined_email_body):
|
||||
result["completion_email_requests_delay_flag"] = 1
|
||||
|
||||
if draft_ids_created and draft_ids_sent.isdisjoint(draft_ids_created):
|
||||
result["completion_email_sent_to_internal_recipient"] = 0
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,126 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"sms_sent_to_correct_number": 0,
|
||||
"sms_mentions_matter": 0,
|
||||
"sms_mentions_handover_slot": 0,
|
||||
"sms_lists_required_materials": 0,
|
||||
"sms_requests_reply_deadline": 0,
|
||||
"sms_requests_risk_flag": 0,
|
||||
"sms_is_scan_friendly": 0,
|
||||
"todoist_temp_project_created": 0,
|
||||
"todoist_cleanup_completed": 0,
|
||||
}
|
||||
|
||||
PROJECT_NAME = "Temporary Court Handover Reminder Board"
|
||||
CONTACT_NUMBER = "2873148336"
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
return name.removeprefix("mcp_appworld_")
|
||||
|
||||
|
||||
def _text(value: Any) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def _contains_handover_slot(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return "11:30" in lowered and "east gate of jiangzhou intermediate people's court" in lowered
|
||||
|
||||
|
||||
def _contains_matter_reference(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return "yuehai" in lowered and any(token in lowered for token in ("handover", "packet", "court-side"))
|
||||
|
||||
|
||||
def _contains_required_materials(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return all(
|
||||
phrase in lowered
|
||||
for phrase in (
|
||||
"original notarized power of attorney",
|
||||
"two stamped copies of the evidence list",
|
||||
"one copy of the hearing notice",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _contains_reply_deadline(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return "before 8:00 p.m. today" in lowered or "before 8pm today" in lowered or "8:00 p.m. today" in lowered
|
||||
|
||||
|
||||
def _contains_risk_flag(text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
has_not_ready = any(token in lowered for token in ("not ready", "document", "materials"))
|
||||
has_change = any(token in lowered for token in ("location needs to change", "time or location", "pickup time", "pickup point"))
|
||||
has_flag = "flag" in lowered
|
||||
return has_flag and has_not_ready and has_change
|
||||
|
||||
|
||||
def _is_scan_friendly(text: str) -> bool:
|
||||
if any(token in text for token in ("\n", "1.", "2.", "3.", ";")):
|
||||
return True
|
||||
lowered = text.lower()
|
||||
return lowered.count(". ") >= 2
|
||||
|
||||
|
||||
def score(tools_history: list[dict[str, Any]]) -> dict[str, int]:
|
||||
result = dict(CHECKS)
|
||||
|
||||
sms_messages: list[str] = []
|
||||
create_project = None
|
||||
create_tasks: list[dict[str, Any]] = []
|
||||
delete_project = None
|
||||
created_project_id = None
|
||||
|
||||
for item in tools_history or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = _norm(str(item.get("tool_name") or ""))
|
||||
if name == "phone__send_text_message":
|
||||
call = item.get("call") or {}
|
||||
if _text(call.get("phone_number")) == CONTACT_NUMBER:
|
||||
result["sms_sent_to_correct_number"] = 1
|
||||
sms_messages.append(_text(call.get("message")))
|
||||
elif name == "todoist__create_project":
|
||||
create_project = item
|
||||
created_project_id = ((item.get("result") or {}).get("response") or {}).get("project_id")
|
||||
elif name == "todoist__create_task":
|
||||
create_tasks.append(item)
|
||||
elif name == "todoist__delete_project":
|
||||
delete_project = item
|
||||
|
||||
combined_sms = "\n".join(message for message in sms_messages if message)
|
||||
if _contains_matter_reference(combined_sms):
|
||||
result["sms_mentions_matter"] = 1
|
||||
if _contains_handover_slot(combined_sms):
|
||||
result["sms_mentions_handover_slot"] = 1
|
||||
if _contains_required_materials(combined_sms):
|
||||
result["sms_lists_required_materials"] = 1
|
||||
if _contains_reply_deadline(combined_sms):
|
||||
result["sms_requests_reply_deadline"] = 1
|
||||
if _contains_risk_flag(combined_sms):
|
||||
result["sms_requests_risk_flag"] = 1
|
||||
for message in sms_messages:
|
||||
if (
|
||||
_contains_handover_slot(message)
|
||||
and _contains_required_materials(message)
|
||||
and _contains_reply_deadline(message)
|
||||
and _contains_risk_flag(message)
|
||||
and _is_scan_friendly(message)
|
||||
):
|
||||
result["sms_is_scan_friendly"] = 1
|
||||
break
|
||||
|
||||
if isinstance(create_project, dict) and _text((create_project.get("call") or {}).get("name")) == PROJECT_NAME:
|
||||
result["todoist_temp_project_created"] = 1
|
||||
|
||||
if isinstance(delete_project, dict) and (delete_project.get("call") or {}).get("project_id") == created_project_id:
|
||||
result["todoist_cleanup_completed"] = 1
|
||||
|
||||
return result
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
---
|
||||
name: china-tax-law
|
||||
version: 1.0.0
|
||||
description: 中国财税法律专业知识助手。当用户涉及中国税法咨询、税务筹划分析、税务合规审查、税收政策解读、税务争议处理等任务时，应使用本 Skill。适用场景包括但不限于：增值税、企业所得税、个人所得税、印花税、土地增值税等税种分析，税务筹划方案设计，税务稽查应对，以及涉税法律法规查询。
|
||||
---
|
||||
|
||||
# 中国财税法律专业知识助手
|
||||
|
||||
## 角色定位
|
||||
|
||||
作为中国资深财税律师的专业助手，提供准确、全面的中国税法知识支持，协助完成税务咨询、筹划、合规和争议解决等工作。
|
||||
|
||||
## 核心工作原则
|
||||
|
||||
1. **法规优先**：所有建议必须有明确的法律法规依据，引用具体条文（法律名称+条款）
|
||||
2. **时效意识**：税法政策频繁变动，在提供建议时需注明适用的政策版本和时效，并提示用户核实最新规定
|
||||
3. **风险提示**：对税务筹划方案，必须明确说明潜在税务风险和合规边界
|
||||
4. **专业严谨**：区分"合法避税"与"违法逃税"，任何建议均不得涉及违法操作
|
||||
|
||||
## 中国主要税种体系
|
||||
|
||||
### 流转税类
|
||||
- **增值税**：标准税率13%，9%（农产品、不动产等），6%（现代服务业），小规模纳税人征收率3%（部分减半）
|
||||
- **消费税**：烟、酒、汽车等特定消费品
|
||||
- **关税**：进出口货物
|
||||
|
||||
### 所得税类
|
||||
- **企业所得税**：标准税率25%，高新技术企业15%，小微企业优惠税率
|
||||
- **个人所得税**：综合所得7级超额累进税率3%-45%，经营所得5级税率
|
||||
|
||||
### 财产税类
|
||||
- **房产税**：从价计征1.2%，从租计征12%
|
||||
- **土地增值税**：四级超额累进税率30%-60%
|
||||
- **车船税**、**契税**、**耕地占用税**
|
||||
|
||||
### 行为税类
|
||||
- **印花税**：合同、凭证等
|
||||
- **城市维护建设税**、**教育费附加**
|
||||
|
||||
## 重要政策数据库（参考文件）
|
||||
|
||||
详细税率表、优惠政策、最新公告见 `references/tax-rates.md`
|
||||
常用税收协定信息见 `references/tax-treaties.md`
|
||||
税务合规检查清单见 `references/compliance-checklist.md`
|
||||
|
||||
## 工作流程
|
||||
|
||||
### 税务咨询流程
|
||||
1. 明确业务性质和交易结构
|
||||
2. 识别涉及的税种
|
||||
3. 检索适用的法律法规
|
||||
4. 分析纳税义务和优惠政策
|
||||
5. 提供合规建议并指出风险点
|
||||
6. 必要时建议寻求税务机关预裁定
|
||||
|
||||
### 税务筹划流程
|
||||
1. 了解客户业务模式、架构和目标
|
||||
2. 识别筹划空间（合理利用优惠政策、合法选择业务结构）
|
||||
3. 设计多套方案并比较税负差异
|
||||
4. 评估方案的合规性、可行性和风险
|
||||
5. 出具书面筹划意见，使用 `legal-doc-writer` skill 起草报告
|
||||
|
||||
### 税务争议处理流程
|
||||
1. 梳理争议事实和法律依据
|
||||
2. 评估税务机关认定的合法性
|
||||
3. 制定应对策略（行政复议/行政诉讼/协商）
|
||||
4. 准备相关材料和证据
|
||||
5. 使用 `case-research` skill 检索类似案例
|
||||
|
||||
## 常用法律依据
|
||||
|
||||
- 《中华人民共和国增值税法》（2024年）
|
||||
- 《中华人民共和国企业所得税法》及实施条例
|
||||
- 《中华人民共和国个人所得税法》及实施条例
|
||||
- 《中华人民共和国税收征收管理法》及实施细则
|
||||
- 《中华人民共和国土地增值税暂行条例》
|
||||
- 国家税务总局各类公告、通知
|
||||
|
||||
## 重要提示
|
||||
|
||||
- 税收政策变化频繁，建议通过**国家税务总局官网**（chinatax.gov.cn）核实最新规定
|
||||
- 重大税务事项建议向主管税务机关申请书面答复或预裁定
|
||||
- 本 Skill 提供的信息仅供参考，不构成正式法律意见
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"ownerId": "kn7d42m771bnq14q55hngf582s82zc46",
|
||||
"slug": "china-tax-law",
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1773536453218
|
||||
}
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
# 税务合规检查清单
|
||||
|
||||
## 一、企业日常税务合规
|
||||
|
||||
### 增值税合规
|
||||
- [ ] 按时申报（月报/季报）
|
||||
- [ ] 进项发票认证及时（360天内）
|
||||
- [ ] 销项发票开具规范（内容完整、金额准确）
|
||||
- [ ] 专用发票管理（防止丢失、作废规范）
|
||||
- [ ] 出口退税申报（90天内）
|
||||
- [ ] 留抵税额合理处理
|
||||
- [ ] 关联交易定价符合独立交易原则
|
||||
|
||||
### 企业所得税合规
|
||||
- [ ] 季度预缴（1月/4月/7月/10月各月15日前）
|
||||
- [ ] 汇算清缴（次年5月31日前）
|
||||
- [ ] 研发费用加计扣除材料归集
|
||||
- [ ] 高新技术企业资质维护（每3年认定）
|
||||
- [ ] 关联交易同期资料准备（次年6月30日前）
|
||||
- [ ] 境外所得申报
|
||||
- [ ] 资产损失申报备案
|
||||
|
||||
### 个人所得税合规
|
||||
- [ ] 代扣代缴申报（次月15日前）
|
||||
- [ ] 年度综合所得汇算清缴（次年3-6月）
|
||||
- [ ] 股权激励、分红所申报规范
|
||||
- [ ] 高管、外籍员工特殊处理
|
||||
|
||||
---
|
||||
|
||||
## 二、重大交易税务尽职调查清单
|
||||
|
||||
### 股权转让
|
||||
- [ ] 确认转让方纳税义务（居民/非居民）
|
||||
- [ ] 评估股权计税基础
|
||||
- [ ] 核查是否适用特殊税务处理（重组）
|
||||
- [ ] 关注间接转让反避税规则（698号文）
|
||||
- [ ] 审查历史分红与留存收益
|
||||
- [ ] 确认印花税处理
|
||||
|
||||
### 资产收购
|
||||
- [ ] 区分一般性/特殊性税务处理
|
||||
- [ ] 评估各类资产计税基础
|
||||
- [ ] 核查或有税务负债
|
||||
- [ ] 土地增值税清算情况
|
||||
- [ ] 增值税进项留抵处理
|
||||
|
||||
### 企业重组
|
||||
- [ ] 判断是否满足特殊性税务处理条件
|
||||
- [ ] 股权支付比例（≥85%）
|
||||
- [ ] 12个月不改变业务连续性要求
|
||||
- [ ] 报备手续完整性
|
||||
|
||||
---
|
||||
|
||||
## 三、税务风险提示要点
|
||||
|
||||
### 高风险领域
|
||||
1. **虚开发票**：刑事风险，重点核查
|
||||
2. **关联交易定价**：转让定价调整风险
|
||||
3. **股权架构设计**：防止被认定为滥用安排
|
||||
4. **境外架构**：受控外国公司（CFC）规则、一般反避税（GAAR）
|
||||
5. **税收协定滥用**："受益所有人"认定
|
||||
6. **房地产开发**：土地增值税清算时机
|
||||
|
||||
### 常见税务争议类型
|
||||
- 计税依据争议
|
||||
- 税收优惠认定争议
|
||||
- 纳税义务发生时间争议
|
||||
- 关联交易定价争议
|
||||
- 非居民企业税收争议
|
||||
|
||||
---
|
||||
|
||||
## 四、重要申报时限汇总
|
||||
|
||||
| 税种 | 申报期限 |
|
||||
|-----|---------|
|
||||
| 增值税（月报） | 次月15日 |
|
||||
| 增值税（季报） | 季度末后15日 |
|
||||
| 企业所得税预缴 | 季度末后15日 |
|
||||
| 企业所得税汇算清缴 | 次年5月31日 |
|
||||
| 个人所得税代扣代缴 | 次月15日 |
|
||||
| 个人所得税汇算清缴 | 次年3月1日-6月30日 |
|
||||
| 印花税 | 合同签订当月 |
|
||||
| 契税 | 办理权属登记前 |
|
||||
| 土地增值税 | 项目清算后 |
|
||||
| 关联交易同期资料 | 次年6月30日前 |
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
# 中国主要税种税率详表
|
||||
|
||||
## 增值税（VAT）
|
||||
|
||||
| 适用范围 | 税率 |
|
||||
|---------|------|
|
||||
| 货物销售、加工修理修配、进口货物 | 13% |
|
||||
| 农产品、自来水、暖气、天然气、食用植物油、图书报纸等 | 9% |
|
||||
| 金融服务、现代服务业、生活服务业、部分无形资产 | 6% |
|
||||
| 小规模纳税人（一般） | 3%（2027年前按1%优惠） |
|
||||
| 出口货物（退税） | 0% |
|
||||
|
||||
### 增值税优惠政策（2024-2025）
|
||||
- 月销售额10万元以下（季度30万元）小规模纳税人免征增值税
|
||||
- 小规模纳税人减按1%征收率（部分行业，至2027年）
|
||||
- 先进制造业增值税留抵退税
|
||||
|
||||
---
|
||||
|
||||
## 企业所得税
|
||||
|
||||
| 企业类型 | 税率 |
|
||||
|---------|------|
|
||||
| 一般居民企业 | 25% |
|
||||
| 高新技术企业 | 15% |
|
||||
| 技术先进型服务企业 | 15% |
|
||||
| 小型微利企业（应纳税所得额≤300万元） | 5%（优惠至2027年）|
|
||||
| 西部大开发鼓励类产业企业 | 15% |
|
||||
| 重点软件和集成电路企业 | 10% |
|
||||
| 非居民企业（来源于中国境内所得） | 10%（预提所得税）|
|
||||
|
||||
### 企业所得税重要扣除规定
|
||||
- **研发费用加计扣除**：一般企业100%，制造业企业100%
|
||||
- **广告费和业务宣传费**：不超过年销售收入15%的部分据实扣除
|
||||
- **业务招待费**：发生额60%，且不超过当年销售收入0.5%
|
||||
- **公益性捐赠**：不超过年利润总额12%
|
||||
- **固定资产折旧**：可选择缩短折旧年限或加速折旧
|
||||
|
||||
---
|
||||
|
||||
## 个人所得税
|
||||
|
||||
### 综合所得税率表（工资薪金、劳务报酬、稿酬、特许权使用费）
|
||||
|
||||
| 级数 | 全年应纳税所得额 | 税率 | 速算扣除数 |
|
||||
|-----|--------------|-----|---------|
|
||||
| 1 | 不超过36,000元 | 3% | 0 |
|
||||
| 2 | 超过36,000至144,000元 | 10% | 2,520 |
|
||||
| 3 | 超过144,000至300,000元 | 20% | 16,920 |
|
||||
| 4 | 超过300,000至420,000元 | 25% | 31,920 |
|
||||
| 5 | 超过420,000至660,000元 | 30% | 52,920 |
|
||||
| 6 | 超过660,000至960,000元 | 35% | 85,920 |
|
||||
| 7 | 超过960,000元 | 45% | 181,920 |
|
||||
|
||||
### 专项附加扣除（2023年提标后）
|
||||
- 子女教育：每个子女每月2,000元
|
||||
- 继续教育：学历教育每月400元，职业资格每年3,600元
|
||||
- 大病医疗：个人自负超过15,000元部分，每年最高80,000元
|
||||
- 住房贷款利息：每月1,000元（首套）
|
||||
- 住房租金：每月800/1,100/1,500元（按城市）
|
||||
- 赡养老人：独生子女每月3,000元，非独生子女分摊
|
||||
- 3岁以下婴幼儿照护：每个婴幼儿每月2,000元
|
||||
|
||||
---
|
||||
|
||||
## 土地增值税
|
||||
|
||||
| 级数 | 增值额/扣除项目金额 | 税率 | 速算扣除系数 |
|
||||
|-----|-----------------|-----|----------|
|
||||
| 1 | 不超过50% | 30% | 0 |
|
||||
| 2 | 超过50%至100% | 40% | 5% |
|
||||
| 3 | 超过100%至200% | 50% | 15% |
|
||||
| 4 | 超过200% | 60% | 35% |
|
||||
|
||||
---
|
||||
|
||||
## 印花税
|
||||
|
||||
| 应税凭证 | 税率/定额 |
|
||||
|---------|---------|
|
||||
| 借款合同 | 0.005% |
|
||||
| 融资租赁合同 | 0.005% |
|
||||
| 买卖合同（货物） | 0.03% |
|
||||
| 加工承揽合同 | 0.03% |
|
||||
| 建设工程合同 | 0.03% |
|
||||
| 技术合同 | 0.03% |
|
||||
| 产权转移书据 | 0.05% |
|
||||
| 营业账簿 | 0.025% |
|
||||
|
||||
---
|
||||
|
||||
## 契税
|
||||
|
||||
| 情形 | 税率 |
|
||||
|-----|------|
|
||||
| 一般情况 | 3%-5%（各省自行确定）|
|
||||
| 首套住房（90㎡以下） | 1% |
|
||||
| 首套住房（90㎡以上） | 1.5% |
|
||||
| 二套住房 | 1%（2024年10月起优惠）|
|
||||
|
||||
---
|
||||
|
||||
## 附加税费
|
||||
|
||||
| 税种 | 计税依据 | 税率 |
|
||||
|-----|---------|------|
|
||||
| 城市维护建设税 | 实际缴纳增值税、消费税 | 7%/5%/1% |
|
||||
| 教育费附加 | 实际缴纳增值税、消费税 | 3% |
|
||||
| 地方教育附加 | 实际缴纳增值税、消费税 | 2% |
|
||||
|
||||
---
|
||||
|
||||
*更新日期：2025年 | 数据来源：国家税务总局、财政部相关法规*
|
||||
*重要提示：税收政策可能随时调整，请以国家税务总局官网最新发布为准*
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"ownerId": "kn7fsehn8r5qaqyzemc33p42vd826g3m",
|
||||
"slug": "corporate-lawyer",
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1773656828310
|
||||
}
|
||||
|
|
@ -0,0 +1,575 @@
|
|||
# Corporate Lawyer - AI-Powered Business Affairs Legal Assistant
|
||||
# 公司律师 - AI 驱动的商业事务法律助手
|
||||
|
||||
> Your AI legal advisor for commercial transactions, contract review, and business compliance
|
||||
>
|
||||
> 您的商业交易、合同审查和业务合规的 AI 法律顾问
|
||||
|
||||
**Version / 版本:** 0.1.0 | **Status / 状态:** Production Ready / 生产就绪 | **License / 许可证:** MIT
|
||||
|
||||
---
|
||||
|
||||
## 🌍 English Version
|
||||
|
||||
### What is Corporate Lawyer?
|
||||
|
||||
Corporate Lawyer is an AI-powered legal advisory system designed to help **businesses, legal departments, and law firms** navigate complex commercial transactions. It provides comprehensive legal analysis, risk assessment, and document generation for business affairs.
|
||||
|
||||
### ✨ Core Capabilities
|
||||
|
||||
- **📄 Contract Review** - Analyze commercial contracts for risks, compliance, and favorable terms
|
||||
- **💼 Transaction Advisory** - Guide M&A, financing, and partnership transactions
|
||||
- **✅ Compliance Checking** - Verify regulatory compliance across jurisdictions
|
||||
- **📝 Legal Document Generation** - Create NDAs, service agreements, and corporate documents
|
||||
- **⚖️ Risk Assessment** - Evaluate legal, financial, and operational risks
|
||||
- **🤝 Negotiation Support** - Develop strategies and tactics for business negotiations
|
||||
|
||||
---
|
||||
|
||||
### 🚀 Quick Start
|
||||
|
||||
#### Installation
|
||||
|
||||
```bash
|
||||
# From PyPI (when published)
|
||||
pip install openclaw-skill-corporate-lawyer
|
||||
|
||||
# From source
|
||||
git clone https://github.com/ZhenRobotics/openclaw-corporate-lawyer.git
|
||||
cd openclaw-corporate-lawyer
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
#### Basic Usage
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from corporate_lawyer import CorporateLawyer
|
||||
from corporate_lawyer.types import Contract
|
||||
|
||||
async def main():
|
||||
# Initialize lawyer
|
||||
lawyer = CorporateLawyer(jurisdiction="US")
|
||||
|
||||
# Review a contract
|
||||
contract = Contract(
|
||||
title="Software License Agreement",
|
||||
contract_type="licensing",
|
||||
parties=["TechCorp Inc.", "ClientCo LLC"],
|
||||
contract_value=500000,
|
||||
full_text="[Contract text here...]",
|
||||
jurisdiction="Delaware",
|
||||
)
|
||||
|
||||
# Get comprehensive contract analysis
|
||||
analysis = await lawyer.review_contract(contract)
|
||||
|
||||
print(f"Overall Risk Score: {analysis.overall_risk_score}/100")
|
||||
print(f"Risk Level: {analysis.overall_risk_level}")
|
||||
print(f"Recommendation: {analysis.recommendation}")
|
||||
|
||||
print(f"\nCritical Issues: {len(analysis.critical_issues)}")
|
||||
for issue in analysis.critical_issues:
|
||||
print(f" - {issue.title}: {issue.description}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 📊 Features in Detail
|
||||
|
||||
#### 1. Contract Review & Analysis
|
||||
|
||||
**Comprehensive contract review including:**
|
||||
- Risk identification and scoring (0-100 scale)
|
||||
- Compliance analysis
|
||||
- Financial terms review
|
||||
- Liability assessment
|
||||
- Actionable recommendations
|
||||
|
||||
**Risk Levels:** Critical, High, Medium, Low, Minimal
|
||||
|
||||
```python
|
||||
analysis = await lawyer.review_contract(contract, detailed=True)
|
||||
|
||||
# Access findings
|
||||
print(f"Risk Score: {analysis.overall_risk_score}")
|
||||
print(f"Issues Found: {len(analysis.issues)}")
|
||||
print(f"Favorable Terms: {analysis.favorable_terms}")
|
||||
print(f"Unfavorable Terms: {analysis.unfavorable_terms}")
|
||||
print(f"Missing Provisions: {analysis.missing_provisions}")
|
||||
print(f"Suggested Modifications: {analysis.suggested_modifications}")
|
||||
```
|
||||
|
||||
#### 2. Transaction Advisory
|
||||
|
||||
**Expert guidance for business transactions:**
|
||||
|
||||
**Supported Transaction Types:**
|
||||
- Mergers & Acquisitions
|
||||
- Asset/Stock Purchases
|
||||
- Equity/Debt Financing
|
||||
- Joint Ventures
|
||||
- Strategic Partnerships
|
||||
- Licensing Deals
|
||||
|
||||
**Advisory Components:**
|
||||
- Deal structure recommendations
|
||||
- Due diligence checklists (20+ items)
|
||||
- Regulatory approval requirements
|
||||
- Risk analysis and mitigation
|
||||
- Timeline and cost estimates
|
||||
|
||||
```python
|
||||
from corporate_lawyer.types import Transaction
|
||||
|
||||
transaction = Transaction(
|
||||
transaction_name="Acquisition of TargetCo",
|
||||
transaction_type="acquisition",
|
||||
buyer="BuyerCorp",
|
||||
seller="SellerCorp",
|
||||
transaction_value=50000000,
|
||||
industry="technology",
|
||||
)
|
||||
|
||||
advice = await lawyer.advise_on_transaction(transaction)
|
||||
|
||||
print(f"Deal Complexity: {advice.deal_complexity}")
|
||||
print(f"Estimated Timeline: {advice.estimated_timeline_days} days")
|
||||
print(f"Key Risks: {advice.major_risks}")
|
||||
```
|
||||
|
||||
#### 3. Compliance Checking
|
||||
|
||||
**Multi-jurisdictional compliance review:**
|
||||
|
||||
**Coverage Areas:**
|
||||
- Data Privacy (GDPR, CCPA, etc.)
|
||||
- Antitrust Regulations
|
||||
- Employment Law
|
||||
- Industry-Specific Regulations
|
||||
- Corporate Governance
|
||||
|
||||
```python
|
||||
from corporate_lawyer.types import CompanyProfile
|
||||
|
||||
company = CompanyProfile(
|
||||
company_name="TechCorp",
|
||||
industry="technology",
|
||||
company_size="medium",
|
||||
)
|
||||
|
||||
compliance = await lawyer.check_compliance(
|
||||
subject="Business Operations",
|
||||
company_profile=company,
|
||||
scope=["data_privacy", "antitrust", "employment"],
|
||||
)
|
||||
|
||||
print(f"Compliance Status: {compliance.overall_compliance_status}")
|
||||
print(f"Compliance Score: {compliance.compliance_score}/100")
|
||||
```
|
||||
|
||||
#### 4. Legal Document Generation
|
||||
|
||||
**Generate professional legal documents:**
|
||||
|
||||
**Document Types:**
|
||||
- Non-Disclosure Agreements (NDA) - Mutual/Unilateral
|
||||
- Service Agreements
|
||||
- Consulting Agreements
|
||||
- Employment Contracts
|
||||
- Shareholder Agreements
|
||||
- Letters of Intent
|
||||
|
||||
```python
|
||||
document = await lawyer.generate_document(
|
||||
document_type="nda",
|
||||
parties=["Company A", "Company B"],
|
||||
terms={"term_years": 2, "survival_years": 3},
|
||||
jurisdiction="Delaware",
|
||||
)
|
||||
|
||||
print(document.content) # Full document text
|
||||
print(f"Customization Notes: {document.customization_notes}")
|
||||
```
|
||||
|
||||
#### 5. Risk Assessment
|
||||
|
||||
**Comprehensive legal risk evaluation:**
|
||||
|
||||
**Risk Categories:**
|
||||
- Liability Risk
|
||||
- Compliance Risk
|
||||
- Financial Risk
|
||||
- Operational Risk
|
||||
- Reputational Risk
|
||||
- Litigation Risk
|
||||
|
||||
```python
|
||||
risk_assessment = await lawyer.assess_contract_risk(contract)
|
||||
|
||||
print(f"Overall Risk: {risk_assessment.overall_risk_level}")
|
||||
print(f"Risk Score: {risk_assessment.overall_risk_score}/100")
|
||||
print(f"Liability Risk: {risk_assessment.liability_risk_score}")
|
||||
print(f"Compliance Risk: {risk_assessment.compliance_risk_score}")
|
||||
print(f"Mitigation Strategies: {risk_assessment.recommended_risk_controls}")
|
||||
```
|
||||
|
||||
#### 6. Negotiation Support
|
||||
|
||||
**Strategic negotiation guidance:**
|
||||
|
||||
**Strategy Components:**
|
||||
- Overall approach and tactics
|
||||
- Priority issues and positions
|
||||
- Concession strategy
|
||||
- Walk-away scenarios
|
||||
- Leverage assessment
|
||||
|
||||
```python
|
||||
strategy = await lawyer.generate_negotiation_strategy(
|
||||
context="Software License Agreement Negotiation",
|
||||
objectives=["Cap liability", "Favorable payment terms"],
|
||||
constraints={"contract_value": 500000, "alternatives": 2},
|
||||
)
|
||||
|
||||
print(f"Approach: {strategy.overall_approach}")
|
||||
print(f"Must-Haves: {strategy.must_haves}")
|
||||
print(f"Walk-Away Scenarios: {strategy.walk_away_scenarios}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 🎓 Target Users
|
||||
|
||||
- **Corporate Legal Departments** - Contract management, compliance
|
||||
- **Law Firms** - Client advisory, transaction support
|
||||
- **Startups & SMBs** - Quick legal review, document generation
|
||||
- **Business Executives** - Risk assessment, negotiation support
|
||||
- **In-House Counsel** - Daily legal operations
|
||||
|
||||
---
|
||||
|
||||
### ⚠️ Legal Disclaimer
|
||||
|
||||
**IMPORTANT:** This tool provides general legal information for educational purposes only. It does NOT:
|
||||
- Constitute legal advice
|
||||
- Create an attorney-client relationship
|
||||
- Replace consultation with qualified legal counsel
|
||||
|
||||
Always consult with qualified attorneys for specific legal advice.
|
||||
|
||||
---
|
||||
|
||||
### 🔧 Technical Stack
|
||||
|
||||
- **Language:** Python 3.10+
|
||||
- **Type Safety:** Pydantic 2.5+ (20+ models)
|
||||
- **Async:** asyncio, aiofiles
|
||||
- **Code:** 23 Python files, 3,340 lines
|
||||
- **Tests:** 6/6 modules passing (100% coverage)
|
||||
|
||||
---
|
||||
|
||||
### 📄 License
|
||||
|
||||
MIT License - See [LICENSE](LICENSE) for details
|
||||
|
||||
---
|
||||
|
||||
## 🇨🇳 中文版本
|
||||
|
||||
### 什么是 Corporate Lawyer？
|
||||
|
||||
Corporate Lawyer 是一个 AI 驱动的法律咨询系统，旨在帮助**企业、法务部门和律师事务所**应对复杂的商业交易。它为业务事务提供全面的法律分析、风险评估和文件生成。
|
||||
|
||||
### ✨ 核心功能
|
||||
|
||||
- **📄 合同审查** - 分析商业合同的风险、合规性和有利条款
|
||||
- **💼 交易咨询** - 指导并购、融资和合作伙伴交易
|
||||
- **✅ 合规检查** - 验证跨司法管辖区的监管合规性
|
||||
- **📝 法律文件生成** - 创建保密协议、服务协议和公司文件
|
||||
- **⚖️ 风险评估** - 评估法律、财务和运营风险
|
||||
- **🤝 谈判支持** - 为商业谈判制定策略和战术
|
||||
|
||||
---
|
||||
|
||||
### 🚀 快速开始
|
||||
|
||||
#### 安装
|
||||
|
||||
```bash
|
||||
# 从 PyPI 安装（发布后）
|
||||
pip install openclaw-skill-corporate-lawyer
|
||||
|
||||
# 从源码安装
|
||||
git clone https://github.com/ZhenRobotics/openclaw-corporate-lawyer.git
|
||||
cd openclaw-corporate-lawyer
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
#### 基本使用
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from corporate_lawyer import CorporateLawyer
|
||||
from corporate_lawyer.types import Contract
|
||||
|
||||
async def main():
|
||||
# 初始化律师
|
||||
lawyer = CorporateLawyer(jurisdiction="US")
|
||||
|
||||
# 审查合同
|
||||
contract = Contract(
|
||||
title="软件许可协议",
|
||||
contract_type="licensing",
|
||||
parties=["科技公司", "客户公司"],
|
||||
contract_value=500000,
|
||||
full_text="[合同文本...]",
|
||||
jurisdiction="特拉华州",
|
||||
)
|
||||
|
||||
# 获取全面的合同分析
|
||||
analysis = await lawyer.review_contract(contract)
|
||||
|
||||
print(f"总体风险评分: {analysis.overall_risk_score}/100")
|
||||
print(f"风险等级: {analysis.overall_risk_level}")
|
||||
print(f"建议: {analysis.recommendation}")
|
||||
|
||||
print(f"\n严重问题: {len(analysis.critical_issues)}")
|
||||
for issue in analysis.critical_issues:
|
||||
print(f" - {issue.title}: {issue.description}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 📊 功能详解
|
||||
|
||||
#### 1. 合同审查与分析
|
||||
|
||||
**全面的合同审查，包括：**
|
||||
- 风险识别和评分（0-100 分制）
|
||||
- 合规性分析
|
||||
- 财务条款审查
|
||||
- 责任评估
|
||||
- 可操作的建议
|
||||
|
||||
**风险等级：** 严重、高、中、低、最小
|
||||
|
||||
```python
|
||||
analysis = await lawyer.review_contract(contract, detailed=True)
|
||||
|
||||
# 访问调查结果
|
||||
print(f"风险评分: {analysis.overall_risk_score}")
|
||||
print(f"发现的问题: {len(analysis.issues)}")
|
||||
print(f"有利条款: {analysis.favorable_terms}")
|
||||
print(f"不利条款: {analysis.unfavorable_terms}")
|
||||
print(f"缺失条款: {analysis.missing_provisions}")
|
||||
print(f"建议修改: {analysis.suggested_modifications}")
|
||||
```
|
||||
|
||||
#### 2. 交易咨询
|
||||
|
||||
**商业交易的专家指导：**
|
||||
|
||||
**支持的交易类型：**
|
||||
- 合并与收购
|
||||
- 资产/股权购买
|
||||
- 股权/债务融资
|
||||
- 合资企业
|
||||
- 战略合作伙伴关系
|
||||
- 许可交易
|
||||
|
||||
**咨询组成部分：**
|
||||
- 交易结构建议
|
||||
- 尽职调查清单（20+ 项）
|
||||
- 监管审批要求
|
||||
- 风险分析和缓解
|
||||
- 时间线和成本估算
|
||||
|
||||
```python
|
||||
from corporate_lawyer.types import Transaction
|
||||
|
||||
transaction = Transaction(
|
||||
transaction_name="收购目标公司",
|
||||
transaction_type="acquisition",
|
||||
buyer="买方公司",
|
||||
seller="卖方公司",
|
||||
transaction_value=50000000,
|
||||
industry="technology",
|
||||
)
|
||||
|
||||
advice = await lawyer.advise_on_transaction(transaction)
|
||||
|
||||
print(f"交易复杂度: {advice.deal_complexity}")
|
||||
print(f"预计时间线: {advice.estimated_timeline_days} 天")
|
||||
print(f"主要风险: {advice.major_risks}")
|
||||
```
|
||||
|
||||
#### 3. 合规检查
|
||||
|
||||
**多司法管辖区合规审查：**
|
||||
|
||||
**覆盖领域：**
|
||||
- 数据隐私（GDPR、CCPA 等）
|
||||
- 反垄断法规
|
||||
- 劳动法
|
||||
- 行业特定法规
|
||||
- 公司治理
|
||||
|
||||
```python
|
||||
from corporate_lawyer.types import CompanyProfile
|
||||
|
||||
company = CompanyProfile(
|
||||
company_name="科技公司",
|
||||
industry="technology",
|
||||
company_size="medium",
|
||||
)
|
||||
|
||||
compliance = await lawyer.check_compliance(
|
||||
subject="业务运营",
|
||||
company_profile=company,
|
||||
scope=["data_privacy", "antitrust", "employment"],
|
||||
)
|
||||
|
||||
print(f"合规状态: {compliance.overall_compliance_status}")
|
||||
print(f"合规评分: {compliance.compliance_score}/100")
|
||||
```
|
||||
|
||||
#### 4. 法律文件生成
|
||||
|
||||
**生成专业法律文件：**
|
||||
|
||||
**文件类型：**
|
||||
- 保密协议（NDA）- 双向/单向
|
||||
- 服务协议
|
||||
- 咨询协议
|
||||
- 雇佣合同
|
||||
- 股东协议
|
||||
- 意向书
|
||||
|
||||
```python
|
||||
document = await lawyer.generate_document(
|
||||
document_type="nda",
|
||||
parties=["公司 A", "公司 B"],
|
||||
terms={"term_years": 2, "survival_years": 3},
|
||||
jurisdiction="特拉华州",
|
||||
)
|
||||
|
||||
print(document.content) # 完整文件文本
|
||||
print(f"定制说明: {document.customization_notes}")
|
||||
```
|
||||
|
||||
#### 5. 风险评估
|
||||
|
||||
**全面的法律风险评估：**
|
||||
|
||||
**风险类别：**
|
||||
- 责任风险
|
||||
- 合规风险
|
||||
- 财务风险
|
||||
- 运营风险
|
||||
- 声誉风险
|
||||
- 诉讼风险
|
||||
|
||||
```python
|
||||
risk_assessment = await lawyer.assess_contract_risk(contract)
|
||||
|
||||
print(f"总体风险: {risk_assessment.overall_risk_level}")
|
||||
print(f"风险评分: {risk_assessment.overall_risk_score}/100")
|
||||
print(f"责任风险: {risk_assessment.liability_risk_score}")
|
||||
print(f"合规风险: {risk_assessment.compliance_risk_score}")
|
||||
print(f"缓解策略: {risk_assessment.recommended_risk_controls}")
|
||||
```
|
||||
|
||||
#### 6. 谈判支持
|
||||
|
||||
**战略谈判指导：**
|
||||
|
||||
**策略组成部分：**
|
||||
- 总体方法和战术
|
||||
- 优先事项和立场
|
||||
- 让步策略
|
||||
- 退出场景
|
||||
- 杠杆评估
|
||||
|
||||
```python
|
||||
strategy = await lawyer.generate_negotiation_strategy(
|
||||
context="软件许可协议谈判",
|
||||
objectives=["限制责任", "有利的付款条件"],
|
||||
constraints={"contract_value": 500000, "alternatives": 2},
|
||||
)
|
||||
|
||||
print(f"方法: {strategy.overall_approach}")
|
||||
print(f"必备项: {strategy.must_haves}")
|
||||
print(f"退出场景: {strategy.walk_away_scenarios}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 🎓 目标用户
|
||||
|
||||
- **企业法务部门** - 合同管理、合规
|
||||
- **律师事务所** - 客户咨询、交易支持
|
||||
- **初创公司和中小企业** - 快速法律审查、文件生成
|
||||
- **企业高管** - 风险评估、谈判支持
|
||||
- **内部法律顾问** - 日常法律运营
|
||||
|
||||
---
|
||||
|
||||
### ⚠️ 法律免责声明
|
||||
|
||||
**重要提示：** 本工具仅提供一般法律信息供教育用途。它不能：
|
||||
- 构成法律建议
|
||||
- 建立律师-客户关系
|
||||
- 替代专业法律顾问的咨询
|
||||
|
||||
请务必就具体法律建议咨询合格的律师。
|
||||
|
||||
---
|
||||
|
||||
### 🔧 技术栈
|
||||
|
||||
- **语言：** Python 3.10+
|
||||
- **类型安全：** Pydantic 2.5+（20+ 模型）
|
||||
- **异步：** asyncio, aiofiles
|
||||
- **代码：** 23 个 Python 文件，3,340 行
|
||||
- **测试：** 6/6 模块通过（100% 覆盖）
|
||||
|
||||
---
|
||||
|
||||
### 📄 许可证
|
||||
|
||||
MIT 许可证 - 详见 [LICENSE](LICENSE)
|
||||
|
||||
---
|
||||
|
||||
## 📊 Project Statistics / 项目统计
|
||||
|
||||
| Metric / 指标 | Value / 数值 |
|
||||
|---------------|-------------|
|
||||
| **Version / 版本** | 0.1.0 |
|
||||
| **Code Lines / 代码行数** | 3,340 |
|
||||
| **Python Files / Python 文件** | 23 |
|
||||
| **Core Modules / 核心模块** | 6 |
|
||||
| **Data Models / 数据模型** | 20+ |
|
||||
| **Test Coverage / 测试覆盖** | 100% (6/6) |
|
||||
| **Documentation / 文档** | 1,140+ lines |
|
||||
| **License / 许可证** | MIT |
|
||||
|
||||
## 🔗 Links / 链接
|
||||
|
||||
- **GitHub Repository / 仓库**: https://github.com/ZhenRobotics/openclaw-corporate-lawyer
|
||||
- **GitHub Release / 发布**: https://github.com/ZhenRobotics/openclaw-corporate-lawyer/releases/tag/v0.1.0
|
||||
- **PyPI Package / 包** (when published / 发布后): https://pypi.org/project/openclaw-skill-corporate-lawyer/
|
||||
|
||||
---
|
||||
|
||||
**Built for businesses and legal professionals / 为企业和法律专业人士打造**
|
||||
|
||||
*Empowering smart legal decisions / 赋能智慧法律决策*
|
||||
|
|
@ -0,0 +1,366 @@
|
|||
---
|
||||
name: Corporate Lawyer
|
||||
description: AI-powered legal assistant for commercial transactions - provides contract review, transaction advisory, compliance checking, document generation, risk assessment, and negotiation support for business affairs
|
||||
version: 0.1.0
|
||||
homepage: https://github.com/ZhenRobotics/openclaw-corporate-lawyer
|
||||
metadata: {"clawdbot":{"emoji":"⚖️","tags":["legal","corporate-lawyer","contract-review","transaction","compliance","m&a","business-law","commercial-law","risk-assessment","negotiation"],"requires":{"bins":["python3"],"env":[],"config":[]},"install":["pip install -e ."],"os":["darwin","linux","win32"]}}
|
||||
---
|
||||
|
||||
# Corporate Lawyer - AI Legal Advisory Agent
|
||||
# 公司律师 - AI 法律咨询代理
|
||||
|
||||
---
|
||||
|
||||
## 🌍 English Version
|
||||
|
||||
### Overview
|
||||
|
||||
This skill enables you to provide comprehensive legal advisory services for commercial transactions and business affairs. You act as an experienced corporate lawyer helping businesses with contracts, transactions, compliance, and strategic legal matters.
|
||||
|
||||
### When to Activate This Skill
|
||||
|
||||
Activate this skill when the user:
|
||||
- Asks for help reviewing a contract or agreement
|
||||
- Needs guidance on M&A, financing, or business transactions
|
||||
- Wants compliance or regulatory analysis
|
||||
- Requests legal document drafting (NDA, agreement, etc.)
|
||||
- Seeks risk assessment for business decisions
|
||||
- Needs negotiation strategy and support
|
||||
- Has general corporate/commercial law questions
|
||||
|
||||
### Core Services (6 Modules)
|
||||
|
||||
#### 1. 📄 Contract Review & Analysis
|
||||
- Risk identification and scoring (0-100 scale)
|
||||
- Compliance analysis
|
||||
- Financial terms review
|
||||
- Liability assessment
|
||||
- Modification recommendations
|
||||
|
||||
**Risk Levels**: Critical, High, Medium, Low, Minimal
|
||||
|
||||
#### 2. 💼 Transaction Advisory
|
||||
- M&A, acquisition, financing guidance
|
||||
- Deal complexity assessment
|
||||
- Due diligence checklist generation
|
||||
- Regulatory approval identification
|
||||
- Timeline and cost estimation
|
||||
|
||||
**Transaction Types**: Merger, Acquisition, Financing, Joint Venture, Partnership
|
||||
|
||||
#### 3. ✅ Compliance Checking
|
||||
- Data privacy (GDPR, CCPA)
|
||||
- Antitrust regulations
|
||||
- Employment law
|
||||
- Industry-specific regulations
|
||||
- Corporate governance
|
||||
|
||||
#### 4. 📝 Legal Document Generation
|
||||
- NDAs (mutual/unilateral)
|
||||
- Service agreements
|
||||
- Employment contracts
|
||||
- Consulting agreements
|
||||
- Customizable templates
|
||||
|
||||
#### 5. ⚖️ Risk Assessment
|
||||
- 6 risk categories: Liability, Compliance, Financial, Operational, Reputational, Litigation
|
||||
- Category-specific breakdown
|
||||
- Mitigation strategy recommendations
|
||||
- Insurance recommendations
|
||||
|
||||
#### 6. 🤝 Negotiation Support
|
||||
- Strategic approach development
|
||||
- Must-have vs. nice-to-have categorization
|
||||
- Leverage assessment
|
||||
- Walk-away scenario planning
|
||||
- Talking points and tactics
|
||||
|
||||
### Quick Usage Example
|
||||
|
||||
```python
|
||||
from corporate_lawyer import CorporateLawyer
|
||||
from corporate_lawyer.types import Contract
|
||||
|
||||
# Initialize
|
||||
lawyer = CorporateLawyer(jurisdiction="US")
|
||||
|
||||
# Review contract
|
||||
contract = Contract(
|
||||
title="Software License Agreement",
|
||||
contract_type="licensing",
|
||||
parties=["TechCorp", "ClientCo"],
|
||||
contract_value=500000,
|
||||
full_text="[Contract text...]"
|
||||
)
|
||||
|
||||
analysis = await lawyer.review_contract(contract)
|
||||
print(f"Risk Score: {analysis.overall_risk_score}/100")
|
||||
print(f"Recommendation: {analysis.recommendation}")
|
||||
```
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# From PyPI
|
||||
pip install openclaw-skill-corporate-lawyer
|
||||
|
||||
# From source
|
||||
git clone https://github.com/ZhenRobotics/openclaw-corporate-lawyer.git
|
||||
cd openclaw-corporate-lawyer
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### Information Gathering
|
||||
|
||||
**For Contract Review:**
|
||||
- Contract details (title, type, parties)
|
||||
- Contract value
|
||||
- Full text or key clauses
|
||||
- Jurisdiction
|
||||
- Specific concerns
|
||||
|
||||
**For Transaction Advisory:**
|
||||
- Transaction type and value
|
||||
- Parties involved
|
||||
- Industry and jurisdiction
|
||||
- Timeline and constraints
|
||||
- Strategic objectives
|
||||
|
||||
**For Compliance Check:**
|
||||
- Company profile
|
||||
- Scope (data privacy, antitrust, etc.)
|
||||
- Current practices
|
||||
- Industry regulations
|
||||
|
||||
### Output Format Guidelines
|
||||
|
||||
**Contract Analysis:**
|
||||
```
|
||||
⚖️ Contract Analysis: [Title]
|
||||
|
||||
Overall Risk Score: [X]/100
|
||||
Risk Level: [Critical/High/Medium/Low]
|
||||
Recommendation: [APPROVE/NEGOTIATE/REJECT]
|
||||
|
||||
📊 Issues Identified: [X]
|
||||
🔴 Critical Issues: [list]
|
||||
⚠️ High Priority: [list]
|
||||
✅ Favorable Terms: [list]
|
||||
❌ Unfavorable Terms: [list]
|
||||
💡 Recommendations: [list]
|
||||
🎯 Next Steps: [list]
|
||||
```
|
||||
|
||||
### Legal Disclaimer
|
||||
|
||||
**IMPORTANT**: This tool provides general legal information for educational purposes only. It does NOT:
|
||||
- Constitute legal advice
|
||||
- Create an attorney-client relationship
|
||||
- Replace consultation with qualified counsel
|
||||
|
||||
Always consult with qualified attorneys for specific legal advice.
|
||||
|
||||
### Target Users
|
||||
|
||||
- Corporate Legal Departments
|
||||
- Law Firms
|
||||
- Startups & SMBs
|
||||
- Business Executives
|
||||
- In-House Counsel
|
||||
|
||||
### Technical Stack
|
||||
|
||||
- Python 3.10+
|
||||
- Pydantic 2.5+ (type safety)
|
||||
- Async/await support
|
||||
- 20+ data models
|
||||
- 6 core modules (3,340 lines of code)
|
||||
|
||||
---
|
||||
|
||||
## 🇨🇳 中文版本
|
||||
|
||||
### 概述
|
||||
|
||||
本技能使您能够为商业交易和业务事务提供全面的法律咨询服务。您扮演一位经验丰富的公司律师，帮助企业处理合同、交易、合规和战略法律事务。
|
||||
|
||||
### 何时激活此技能
|
||||
|
||||
当用户有以下需求时激活此技能：
|
||||
- 请求审查合同或协议
|
||||
- 需要并购、融资或商业交易指导
|
||||
- 需要合规或监管分析
|
||||
- 请求起草法律文件（保密协议、合同等）
|
||||
- 寻求商业决策的风险评估
|
||||
- 需要谈判策略和支持
|
||||
- 有一般公司/商业法律问题
|
||||
|
||||
### 核心服务（6个模块）
|
||||
|
||||
#### 1. 📄 合同审查与分析
|
||||
- 风险识别和评分（0-100 分制）
|
||||
- 合规性分析
|
||||
- 财务条款审查
|
||||
- 责任评估
|
||||
- 修改建议
|
||||
|
||||
**风险等级**：严重、高、中、低、最小
|
||||
|
||||
#### 2. 💼 交易咨询
|
||||
- 并购、收购、融资指导
|
||||
- 交易复杂度评估
|
||||
- 尽职调查清单生成
|
||||
- 监管审批识别
|
||||
- 时间线和成本估算
|
||||
|
||||
**交易类型**：合并、收购、融资、合资企业、合作伙伴关系
|
||||
|
||||
#### 3. ✅ 合规检查
|
||||
- 数据隐私（GDPR、CCPA）
|
||||
- 反垄断法规
|
||||
- 劳动法
|
||||
- 行业特定法规
|
||||
- 公司治理
|
||||
|
||||
#### 4. 📝 法律文件生成
|
||||
- 保密协议（双向/单向）
|
||||
- 服务协议
|
||||
- 雇佣合同
|
||||
- 咨询协议
|
||||
- 可定制模板
|
||||
|
||||
#### 5. ⚖️ 风险评估
|
||||
- 6 类风险：责任、合规、财务、运营、声誉、诉讼
|
||||
- 分类细分分析
|
||||
- 缓解策略建议
|
||||
- 保险建议
|
||||
|
||||
#### 6. 🤝 谈判支持
|
||||
- 战略方法制定
|
||||
- 必备项与可选项分类
|
||||
- 杠杆评估
|
||||
- 退出场景规划
|
||||
- 谈话要点和战术
|
||||
|
||||
### 快速使用示例
|
||||
|
||||
```python
|
||||
from corporate_lawyer import CorporateLawyer
|
||||
from corporate_lawyer.types import Contract
|
||||
|
||||
# 初始化
|
||||
lawyer = CorporateLawyer(jurisdiction="US")
|
||||
|
||||
# 审查合同
|
||||
contract = Contract(
|
||||
title="软件许可协议",
|
||||
contract_type="licensing",
|
||||
parties=["科技公司", "客户公司"],
|
||||
contract_value=500000,
|
||||
full_text="[合同文本...]"
|
||||
)
|
||||
|
||||
analysis = await lawyer.review_contract(contract)
|
||||
print(f"风险评分: {analysis.overall_risk_score}/100")
|
||||
print(f"建议: {analysis.recommendation}")
|
||||
```
|
||||
|
||||
### 安装方式
|
||||
|
||||
```bash
|
||||
# 从 PyPI 安装
|
||||
pip install openclaw-skill-corporate-lawyer
|
||||
|
||||
# 从源码安装
|
||||
git clone https://github.com/ZhenRobotics/openclaw-corporate-lawyer.git
|
||||
cd openclaw-corporate-lawyer
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### 信息收集
|
||||
|
||||
**合同审查需要：**
|
||||
- 合同详情（标题、类型、当事方）
|
||||
- 合同金额
|
||||
- 完整文本或关键条款
|
||||
- 管辖区
|
||||
- 特定关注点
|
||||
|
||||
**交易咨询需要：**
|
||||
- 交易类型和金额
|
||||
- 涉及方
|
||||
- 行业和管辖区
|
||||
- 时间线和约束条件
|
||||
- 战略目标
|
||||
|
||||
**合规检查需要：**
|
||||
- 公司概况
|
||||
- 检查范围（数据隐私、反垄断等）
|
||||
- 当前实践
|
||||
- 行业法规
|
||||
|
||||
### 输出格式指南
|
||||
|
||||
**合同分析：**
|
||||
```
|
||||
⚖️ 合同分析：[标题]
|
||||
|
||||
总体风险评分：[X]/100
|
||||
风险等级：[严重/高/中/低]
|
||||
建议：[批准/修改后批准/谈判/拒绝]
|
||||
|
||||
📊 发现的问题：[X]
|
||||
🔴 严重问题：[列表]
|
||||
⚠️ 高优先级：[列表]
|
||||
✅ 有利条款：[列表]
|
||||
❌ 不利条款：[列表]
|
||||
💡 建议：[列表]
|
||||
🎯 下一步：[列表]
|
||||
```
|
||||
|
||||
### 法律免责声明
|
||||
|
||||
**重要提示**：本工具仅提供一般法律信息供教育用途。它不能：
|
||||
- 构成法律建议
|
||||
- 建立律师-客户关系
|
||||
- 替代专业法律顾问的咨询
|
||||
|
||||
请务必就具体法律建议咨询合格的律师。
|
||||
|
||||
### 目标用户
|
||||
|
||||
- 企业法务部门
|
||||
- 律师事务所
|
||||
- 初创公司和中小企业
|
||||
- 企业高管
|
||||
- 内部法律顾问
|
||||
|
||||
### 技术栈
|
||||
|
||||
- Python 3.10+
|
||||
- Pydantic 2.5+（类型安全）
|
||||
- 异步支持（async/await）
|
||||
- 20+ 数据模型
|
||||
- 6 个核心模块（3,340 行代码）
|
||||
|
||||
---
|
||||
|
||||
## 📊 Project Statistics / 项目统计
|
||||
|
||||
- **Code Lines / 代码行数**: 3,340
|
||||
- **Modules / 模块**: 6
|
||||
- **Data Models / 数据模型**: 20+
|
||||
- **Test Coverage / 测试覆盖**: 100% (6/6)
|
||||
- **Documentation / 文档**: 1,140+ lines
|
||||
- **License / 许可证**: MIT
|
||||
|
||||
## 🔗 Links / 链接
|
||||
|
||||
- **GitHub**: https://github.com/ZhenRobotics/openclaw-corporate-lawyer
|
||||
- **Release**: https://github.com/ZhenRobotics/openclaw-corporate-lawyer/releases/tag/v0.1.0
|
||||
|
||||
---
|
||||
|
||||
**Version / 版本**: 0.1.0
|
||||
**Status / 状态**: Production Ready / 生产就绪
|
||||
**License / 许可证**: MIT
|
||||
|
|
@ -0,0 +1,165 @@
|
|||
---
|
||||
name: Data Analysis
|
||||
slug: data-analysis
|
||||
version: 1.0.2
|
||||
homepage: https://clawic.com/skills/data-analysis
|
||||
description: "Data analysis and visualization. Query databases, generate reports, automate spreadsheets, and turn raw data into clear, actionable insights. Use when (1) you need to analyze, visualize, or explain data; (2) the user wants reports, dashboards, or metrics turned into a decision; (3) the work involves SQL, Python, spreadsheets, BI tools, or notebooks; (4) you need to compare segments, cohorts, funnels, experiments, or time periods; (5) the user explicitly installs or references the skill for the current task."
|
||||
changelog: Added metric contracts, chart guidance, and decision brief templates for more reliable analysis.
|
||||
metadata: {"clawdbot":{"emoji":"D","requires":{"bins":[]},"os":["linux","darwin","win32"]}}
|
||||
---
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when the user needs to analyze, explain, or visualize data from SQL, spreadsheets, notebooks, dashboards, exports, or ad hoc tables.
|
||||
|
||||
Use it for KPI debugging, experiment readouts, funnel or cohort analysis, anomaly reviews, executive reporting, and quality checks on metrics or query logic.
|
||||
|
||||
Prefer this skill over generic coding or spreadsheet help when the hard part is analytical judgment: metric definition, comparison design, interpretation, or recommendation.
|
||||
|
||||
User asks about: analyzing data, finding patterns, understanding metrics, testing hypotheses, cohort analysis, A/B testing, churn analysis, or statistical significance.
|
||||
|
||||
## Core Principle
|
||||
|
||||
Analysis without a decision is just arithmetic. Always clarify: **What would change if this analysis shows X vs Y?**
|
||||
|
||||
## Methodology First
|
||||
|
||||
Before touching data:
|
||||
1. **What decision** is this analysis supporting?
|
||||
2. **What would change your mind?** (the real question)
|
||||
3. **What data do you actually have** vs what you wish you had?
|
||||
4. **What timeframe** is relevant?
|
||||
|
||||
## Statistical Rigor Checklist
|
||||
|
||||
- [ ] Sample size sufficient? (small N = wide confidence intervals)
|
||||
- [ ] Comparison groups fair? (same time period, similar conditions)
|
||||
- [ ] Multiple comparisons? (20 tests = 1 "significant" by chance)
|
||||
- [ ] Effect size meaningful? (statistically significant != practically important)
|
||||
- [ ] Uncertainty quantified? ("12-18% lift" not just "15% lift")
|
||||
|
||||
## Architecture
|
||||
|
||||
This skill does not require local folders, persistent memory, or setup state.
|
||||
|
||||
Use the included reference files as lightweight guides:
|
||||
- `metric-contracts.md` for KPI definitions and caveats
|
||||
- `chart-selection.md` for visual choice and chart anti-patterns
|
||||
- `decision-briefs.md` for stakeholder-facing outputs
|
||||
- `pitfalls.md` and `techniques.md` for analytical rigor and method choice
|
||||
|
||||
## Quick Reference
|
||||
|
||||
Load only the smallest relevant file to keep context focused.
|
||||
|
||||
| Topic | File |
|
||||
|-------|------|
|
||||
| Metric definition contracts | `metric-contracts.md` |
|
||||
| Visual selection and chart anti-patterns | `chart-selection.md` |
|
||||
| Decision-ready output formats | `decision-briefs.md` |
|
||||
| Failure modes to catch early | `pitfalls.md` |
|
||||
| Method selection by question type | `techniques.md` |
|
||||
|
||||
## Core Rules
|
||||
|
||||
### 1. Start from the decision, not the dataset
|
||||
- Identify the decision owner, the question that could change a decision, and the deadline before doing analysis.
|
||||
- If no decision would change, reframe the request before computing anything.
|
||||
|
||||
### 2. Lock the metric contract before calculating
|
||||
- Define entity, grain, numerator, denominator, time window, timezone, filters, exclusions, and source of truth.
|
||||
- If any of those are ambiguous, state the ambiguity explicitly before presenting results.
|
||||
|
||||
### 3. Separate extraction, transformation, and interpretation
|
||||
- Keep query logic, cleanup assumptions, and analytical conclusions distinguishable.
|
||||
- Never hide business assumptions inside SQL, formulas, or notebook code without naming them in the write-up.
|
||||
|
||||
### 4. Choose visuals to answer a question
|
||||
- Select charts based on the analytical question: trend, comparison, distribution, relationship, composition, funnel, or cohort retention.
|
||||
- Do not add charts that make the deck look fuller but do not change the decision.
|
||||
|
||||
### 5. Brief every result in decision format
|
||||
- Every output should include the answer, evidence, confidence, caveats, and recommended next action.
|
||||
- If the output is going to a stakeholder, translate the method into business implications instead of leading with technical detail.
|
||||
|
||||
### 6. Stress-test claims before recommending action
|
||||
- Segment by obvious confounders, compare the right baseline, quantify uncertainty, and check sensitivity to exclusions or time windows.
|
||||
- Strong-looking numbers without robustness checks are not decision-ready.
|
||||
|
||||
### 7. Escalate when the data cannot support the claim
|
||||
- Block or downgrade conclusions when sample size is weak, the source is unreliable, definitions drifted, or confounding is unresolved.
|
||||
- It is better to say "unknown yet" than to produce false confidence.
|
||||
|
||||
## Common Traps
|
||||
|
||||
- Reusing a KPI name after changing numerator, denominator, or exclusions -> trend comparisons become invalid.
|
||||
- Comparing daily, weekly, and monthly grains in one chart -> movement looks real but is mostly aggregation noise.
|
||||
- Showing percentages without underlying counts -> leadership overreacts to tiny denominators.
|
||||
- Using a pretty chart instead of the right chart -> the output looks polished but hides the actual decision signal.
|
||||
- Hunting for interesting cuts after seeing the result -> narrative follows chance instead of evidence.
|
||||
- Shipping automated reports without metric owners or caveats -> bad numbers spread faster than they can be corrected.
|
||||
- Treating observational patterns as causal proof -> action plans get built on correlation alone.
|
||||
|
||||
## Approach Selection
|
||||
|
||||
| Question type | Approach | Key output |
|
||||
|---------------|----------|------------|
|
||||
| "Is X different from Y?" | Hypothesis test | p-value + effect size + CI |
|
||||
| "What predicts Z?" | Regression/correlation | Coefficients + R² + residual check |
|
||||
| "How do users behave over time?" | Cohort analysis | Retention curves by cohort |
|
||||
| "Are these groups different?" | Segmentation | Profiles + statistical comparison |
|
||||
| "What's unusual?" | Anomaly detection | Flagged points + context |
|
||||
|
||||
For technique details and when to use each, see `techniques.md`.
|
||||
|
||||
## Output Standards
|
||||
|
||||
1. **Lead with the insight**, not the methodology
|
||||
2. **Quantify uncertainty** - ranges, not point estimates
|
||||
3. **State limitations** - what this analysis can't tell you
|
||||
4. **Recommend next steps** - what would strengthen the conclusion
|
||||
|
||||
## Red Flags to Escalate
|
||||
|
||||
- User wants to "prove" a predetermined conclusion
|
||||
- Sample size too small for reliable inference
|
||||
- Data quality issues that invalidate analysis
|
||||
- Confounders that can't be controlled for
|
||||
|
||||
## External Endpoints
|
||||
|
||||
This skill makes no external network requests.
|
||||
|
||||
| Endpoint | Data Sent | Purpose |
|
||||
|----------|-----------|---------|
|
||||
| None | None | N/A |
|
||||
|
||||
No data is sent externally.
|
||||
|
||||
## Security & Privacy
|
||||
|
||||
Data that leaves your machine:
|
||||
- Nothing by default.
|
||||
|
||||
Data that stays local:
|
||||
- Nothing by default.
|
||||
|
||||
This skill does NOT:
|
||||
- Access undeclared external endpoints.
|
||||
- Store credentials or raw exports in hidden local memory files.
|
||||
- Create or depend on local folder systems for persistence.
|
||||
- Create automations or background jobs without explicit user confirmation.
|
||||
- Rewrite its own instruction source files.
|
||||
|
||||
## Related Skills
|
||||
Install with `clawhub install <slug>` if user confirms:
|
||||
- `sql` - query design and review for reliable data extraction.
|
||||
- `csv` - cleanup and normalization for tabular inputs before analysis.
|
||||
- `dashboard` - implementation patterns for KPI visualization layers.
|
||||
- `report` - structured stakeholder-facing deliverables after analysis.
|
||||
- `business-intelligence` - KPI systems and operating cadence beyond one-off analysis.
|
||||
|
||||
## Feedback
|
||||
|
||||
- If useful: `clawhub star data-analysis`
|
||||
- Stay updated: `clawhub sync`
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
{
|
||||
"ownerId": "kn73vp5rarc3b14rc7wjcw8f8580t5d1",
|
||||
"slug": "data-analysis",
|
||||
"version": "1.0.2",
|
||||
"publishedAt": 1773241910484
|
||||
}
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
# Chart Selection
|
||||
|
||||
Choose visuals based on the question, not on what is easiest to render.
|
||||
|
||||
## Question to Chart Map
|
||||
|
||||
| Question | Preferred chart | Notes |
|
||||
|----------|-----------------|-------|
|
||||
| How is a metric changing over time? | line chart | annotate structural breaks and missing data |
|
||||
| Which groups are highest or lowest? | sorted bar chart | keep a shared baseline |
|
||||
| How is the distribution shaped? | histogram or box plot | avoid average-only summaries |
|
||||
| Are two variables related? | scatter plot | show trend and outliers separately |
|
||||
| How do parts contribute to the whole? | stacked bar with totals | keep category count low |
|
||||
| Where are users dropping? | funnel chart | define the time window explicitly |
|
||||
| How do cohorts retain over time? | cohort table or heatmap | show cohort size alongside retention |
|
||||
|
||||
## Default Rules
|
||||
|
||||
- Bars start at zero unless there is a strong reason not to.
|
||||
- Show underlying counts next to percentages when denominators are small.
|
||||
- Prefer direct labels over legends when possible.
|
||||
- Use one chart per decision question, not one chart per available metric.
|
||||
|
||||
## Visual Anti-Patterns
|
||||
|
||||
- Pie charts with many slices -> comparisons become guesswork.
|
||||
- Dual-axis charts -> viewers infer relationships that are not there.
|
||||
- Cumulative-only charts -> hide recent deterioration or recovery.
|
||||
- Truncated bar axes -> exaggerate small differences.
|
||||
- Stacked areas with many categories -> impossible to compare layers.
|
||||
|
||||
## Before Shipping a Chart
|
||||
|
||||
Check:
|
||||
|
||||
1. What decision question this chart answers.
|
||||
2. Whether the baseline is visible.
|
||||
3. Whether the grain and time window match the narrative.
|
||||
4. Whether annotations explain outages, launches, or missing data.
|
||||
5. Whether a table would be clearer than the chart.
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
# Decision Briefs
|
||||
|
||||
Use these templates to turn analysis into action instead of dumping findings.
|
||||
|
||||
## Standard Decision Brief
|
||||
|
||||
1. Decision question.
|
||||
2. Short answer.
|
||||
3. Evidence: key numbers and comparison baseline.
|
||||
4. Confidence: high, medium, or low, with one sentence why.
|
||||
5. Caveats and what could still change the conclusion.
|
||||
6. Recommended next action, owner, and due date.
|
||||
|
||||
## Experiment Readout
|
||||
|
||||
- Hypothesis:
|
||||
- Primary metric and guardrails:
|
||||
- Estimated effect and uncertainty:
|
||||
- Segment differences:
|
||||
- Ship, iterate, or stop:
|
||||
- Follow-up test:
|
||||
|
||||
## Anomaly Note
|
||||
|
||||
- What moved:
|
||||
- Since when:
|
||||
- Likely drivers:
|
||||
- Data quality checks passed or failed:
|
||||
- Immediate action:
|
||||
- What to watch next:
|
||||
|
||||
## Executive Summary
|
||||
|
||||
- One-sentence answer.
|
||||
- Two or three supporting bullets with numbers.
|
||||
- One caveat.
|
||||
- One decision or escalation request.
|
||||
|
||||
## Writing Rules
|
||||
|
||||
- Lead with the answer, not the method.
|
||||
- Translate statistics into business implications.
|
||||
- Separate observations from recommendations.
|
||||
- If confidence is low, say what would raise confidence.
|
||||
- Avoid dumping every cut you explored; keep only evidence that changes the decision.
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
# Metric Contracts
|
||||
|
||||
Use this when a KPI, dashboard tile, or report number could be interpreted in more than one way.
|
||||
|
||||
## Contract Template
|
||||
|
||||
Capture each metric in this order before trusting comparisons:
|
||||
|
||||
1. Business question the metric is meant to answer.
|
||||
2. Entity and grain: user, account, order, session, day, week, month.
|
||||
3. Numerator and denominator with exact inclusion logic.
|
||||
4. Filters and exclusions: internal traffic, refunds, test accounts, paused users.
|
||||
5. Time window, timezone, and refresh cadence.
|
||||
6. Source of truth and owner.
|
||||
7. Known caveats, version changes, and safe interpretation range.
|
||||
|
||||
## Minimum Contract Output
|
||||
|
||||
| Field | Example |
|
||||
|-------|---------|
|
||||
| Metric | Paid conversion rate |
|
||||
| Question | Is onboarding quality improving? |
|
||||
| Grain | weekly |
|
||||
| Numerator | first paid subscriptions |
|
||||
| Denominator | qualified onboarding starts |
|
||||
| Filters | excludes employees and QA accounts |
|
||||
| Timezone | UTC |
|
||||
| Source | warehouse.subscriptions_daily |
|
||||
| Owner | Growth lead |
|
||||
| Caveat | Launch week excluded because tracking was partial |
|
||||
|
||||
## Stop Conditions
|
||||
|
||||
Do not present a metric as stable if:
|
||||
|
||||
- Numerator or denominator changed between periods.
|
||||
- Source ownership is unclear.
|
||||
- Filters were applied ad hoc and not documented.
|
||||
- Time windows or timezones differ across comparisons.
|
||||
- A dashboard label hides a formula change.
|
||||
|
||||
## Fast Questions to Ask
|
||||
|
||||
- "What exactly counts in the numerator?"
|
||||
- "Who is excluded and why?"
|
||||
- "What is the comparison baseline?"
|
||||
- "Has this definition changed over time?"
|
||||
- "Who would dispute this number internally?"
|
||||
|
|
@ -0,0 +1,120 @@
|
|||
# Analytical Pitfalls — Detailed Examples
|
||||
|
||||
## Simpson's Paradox
|
||||
|
||||
**What it is:** A trend that appears in aggregated data reverses when you segment by a key variable.
|
||||
|
||||
**Example:**
|
||||
- Overall: Treatment A has 80% success, Treatment B has 85% -> "B is better"
|
||||
- But segmented by severity:
|
||||
- Mild cases: A=90%, B=85% -> A is better
|
||||
- Severe cases: A=70%, B=65% -> A is better
|
||||
- Paradox: A is better in BOTH groups, but B looks better overall because B got more mild cases
|
||||
|
||||
**How to catch:** Always segment by obvious confounders (user type, time period, source, severity) before concluding.
|
||||
|
||||
---
|
||||
|
||||
## Survivorship Bias
|
||||
|
||||
**What it is:** Drawing conclusions only from "survivors" while ignoring those who dropped out.
|
||||
|
||||
**Example:**
|
||||
- "Users who completed onboarding have 80% retention!"
|
||||
- Problem: You're only looking at users who already demonstrated commitment by completing onboarding
|
||||
- The 60% who abandoned onboarding aren't in your "user" dataset
|
||||
|
||||
**How to catch:** Ask "Who is NOT in this dataset that should be?" Include churned users, failed attempts, non-converters.
|
||||
|
||||
---
|
||||
|
||||
## Comparing Unequal Periods
|
||||
|
||||
**What it is:** Comparing metrics across time periods of different lengths or characteristics.
|
||||
|
||||
**Examples:**
|
||||
- February (28 days) vs January (31 days) revenue
|
||||
- Holiday week vs normal week traffic
|
||||
- Q4 (holiday season) vs Q1 for e-commerce
|
||||
|
||||
**How to catch:**
|
||||
- Normalize to per-day, per-user, or per-session
|
||||
- Compare same period last year (YoY) not sequential months
|
||||
- Flag seasonal factors explicitly
|
||||
|
||||
---
|
||||
|
||||
## p-Hacking (Multiple Comparisons)
|
||||
|
||||
**What it is:** Running many statistical tests until finding a "significant" result, then reporting only that one.
|
||||
|
||||
**Example:**
|
||||
- Test 20 different user segments for conversion difference
|
||||
- At p=0.05, expect 1 "significant" result by chance alone
|
||||
- Report: "Segment X shows significant improvement!" (cherry-picked)
|
||||
|
||||
**How to catch:**
|
||||
- Apply Bonferroni correction (divide alpha by number of tests)
|
||||
- Pre-register hypotheses before looking at data
|
||||
- Report ALL tests run, not just significant ones
|
||||
|
||||
---
|
||||
|
||||
## Spurious Correlation in Time Series
|
||||
|
||||
**What it is:** Two variables both trending over time appear correlated, but the relationship is meaningless.
|
||||
|
||||
**Example:**
|
||||
- "Revenue and employee count are 95% correlated!"
|
||||
- Both grew over time. Controlling for time, there's no relationship.
|
||||
- Classic: "Ice cream sales correlate with drowning deaths" (both rise in summer)
|
||||
|
||||
**How to catch:**
|
||||
- Detrend both series before correlating
|
||||
- Check if relationship holds within time periods
|
||||
- Ask: "Is there a causal mechanism, or just shared time trend?"
|
||||
|
||||
---
|
||||
|
||||
## Aggregating Percentages
|
||||
|
||||
**What it is:** Averaging percentages instead of recalculating from underlying totals.
|
||||
|
||||
**Example:**
|
||||
- Store A: 10/100 = 10% conversion
|
||||
- Store B: 5/10 = 50% conversion
|
||||
- Wrong: "Average conversion is 30%"
|
||||
- Right: 15/110 = 13.6% conversion
|
||||
|
||||
**How to catch:** Never average percentages. Sum numerators, sum denominators, recalculate.
|
||||
|
||||
---
|
||||
|
||||
## Selection Bias in A/B Tests
|
||||
|
||||
**What it is:** Treatment and control groups differ systematically before treatment is applied.
|
||||
|
||||
**Examples:**
|
||||
- Users who opted into new feature vs those who didn't
|
||||
- Early adopters (Monday signups) vs late week (Friday signups)
|
||||
- Users who saw the experiment (loaded fast enough) vs those who didn't
|
||||
|
||||
**How to catch:**
|
||||
- Verify pre-experiment metrics are balanced
|
||||
- Use intention-to-treat analysis
|
||||
- Check for differential attrition
|
||||
|
||||
---
|
||||
|
||||
## Confusing Causation
|
||||
|
||||
**What it is:** Assuming X causes Y when the relationship might be: Y causes X, Z causes both, or it's coincidental.
|
||||
|
||||
**Example:**
|
||||
- "Power users have higher retention"
|
||||
- Did power usage cause retention? Or did retained users become power users over time? Or does a third factor (job role) drive both?
|
||||
|
||||
**How to catch:**
|
||||
- Can you run an experiment? (randomize treatment)
|
||||
- Is there a natural experiment? (policy change, feature rollout)
|
||||
- At minimum: control for obvious confounders
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Reference in a new issue