mirror of
https://github.com/iflytek/skillhub.git
synced 2026-10-04 02:34:22 +00:00
feat : 调整skillhub-scanner 服务启动 root_path路径问题
This commit is contained in:
翟二远
parent
048b137ee8
commit
29093a452d
18 changed files with 6124 additions and 5 deletions
674
docs/agent-skills-search/agent_teams/01-architecture-design.md
Normal file
674
docs/agent-skills-search/agent_teams/01-architecture-design.md
Normal file
|
|
@ -0,0 +1,674 @@
|
|||
# Agent+Skill 检索系统架构设计
|
||||
|
||||
## 1. 系统概述
|
||||
|
||||
### 1.1 设计目标
|
||||
|
||||
构建一个基于 PostgreSQL + pgvector 的智能技能检索系统，通过意图识别、多轮对话上下文、场景描述、用户角色、标签和元数据等多维度信息，为 Agent 精准匹配适合的 Skills。
|
||||
|
||||
### 1.2 核心能力
|
||||
|
||||
- **语义理解**：基于向量嵌入的语义相似度检索
|
||||
- **意图识别**：识别用户查询的真实意图类别
|
||||
- **上下文感知**：结合多轮对话历史，理解用户当前需求
|
||||
- **场景适配**：根据使用场景和用户角色推荐技能
|
||||
- **多维度融合**：综合向量、意图、标签、流行度、评分等多维度评分
|
||||
|
||||
## 2. 系统分层架构
|
||||
|
||||
### 2.1 整体分层
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "表现层 (Presentation Layer)"
|
||||
A1[REST API Gateway]
|
||||
A2[WebSocket Gateway<br/>实时对话]
|
||||
end
|
||||
|
||||
subgraph "应用层 (Application Layer)"
|
||||
B1[Agent Skill Search Controller]
|
||||
B2[Agent Session Controller]
|
||||
B3[Skill Embedding Sync Controller]
|
||||
end
|
||||
|
||||
subgraph "服务层 (Service Layer)"
|
||||
C1[Context Builder<br/>上下文构建器]
|
||||
C2[Intent Classifier<br/>意图分类器]
|
||||
C3[Query Embedder<br/>查询向量化]
|
||||
C4[Hybrid Search Service<br/>混合检索服务]
|
||||
C5[Skill Reranker<br/>技能重排序器]
|
||||
C6[Agent Session Service<br/>会话管理服务]
|
||||
C7[Embedding Sync Service<br/>嵌入同步服务]
|
||||
end
|
||||
|
||||
subgraph "领域层 (Domain Layer)"
|
||||
D1[Agent Session]
|
||||
D2[Agent Message]
|
||||
D3[Skill Embedding]
|
||||
D4[Intent Mapping]
|
||||
D5[Search Context]
|
||||
D6[Match Result]
|
||||
end
|
||||
|
||||
subgraph "基础设施层 (Infrastructure Layer)"
|
||||
E1[(PostgreSQL<br/>+ pgvector)]
|
||||
E2[(Redis Cache)]
|
||||
E3[OpenAI Embedding API]
|
||||
E4[Message Queue<br/>异步处理]
|
||||
end
|
||||
|
||||
A1 --> B1
|
||||
A1 --> B2
|
||||
A1 --> B3
|
||||
A2 --> B2
|
||||
|
||||
B1 --> C1
|
||||
B1 --> C4
|
||||
B2 --> C6
|
||||
B3 --> C7
|
||||
|
||||
C1 --> C6
|
||||
C1 --> D5
|
||||
C2 --> D4
|
||||
C3 --> E3
|
||||
C4 --> C3
|
||||
C4 --> C2
|
||||
C4 --> C5
|
||||
C6 --> D1
|
||||
C6 --> D2
|
||||
C7 --> D3
|
||||
|
||||
C1 --> E2
|
||||
C2 --> E2
|
||||
C4 --> E1
|
||||
C5 --> E1
|
||||
C6 --> E1
|
||||
C6 --> E2
|
||||
C7 --> E4
|
||||
|
||||
style A1 fill:#e1f5ff
|
||||
style A2 fill:#e1f5ff
|
||||
style E3 fill:#ffebee
|
||||
style E1 fill:#c8e6c9
|
||||
style E2 fill:#fff9c4
|
||||
```
|
||||
|
||||
### 2.2 分层职责
|
||||
|
||||
| 层级 | 职责 | 核心组件 |
|
||||
|------|------|----------|
|
||||
| 表现层 | 暴露 API，处理请求响应 | REST API、WebSocket |
|
||||
| 应用层 | 编排业务流程，处理应用逻辑 | Controller、DTO |
|
||||
| 服务层 | 核心业务逻辑实现 | 各种 Service |
|
||||
| 领域层 | 领域模型和业务规则 | Entity、Domain Object |
|
||||
| 基础设施层 | 数据持久化、外部服务调用 | 数据库、缓存、API |
|
||||
|
||||
## 3. 核心模块设计
|
||||
|
||||
### 3.1 模块关系图
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "输入模块"
|
||||
IN1[Query Input]
|
||||
IN2[Session Context]
|
||||
IN3[User Context]
|
||||
end
|
||||
|
||||
subgraph "处理模块"
|
||||
P1[Context Builder]
|
||||
P2[Intent Classifier]
|
||||
P3[Query Embedder]
|
||||
P4[Hybrid Search]
|
||||
P5[Reranker]
|
||||
end
|
||||
|
||||
subgraph "存储模块"
|
||||
S1[Skill Embeddings]
|
||||
S2[Intent Mapping]
|
||||
S3[Session Store]
|
||||
end
|
||||
|
||||
subgraph "输出模块"
|
||||
OUT1[Ranked Skills]
|
||||
OUT2[Match Reasons]
|
||||
OUT3[Confidence Scores]
|
||||
end
|
||||
|
||||
IN1 --> P1
|
||||
IN2 --> P1
|
||||
IN3 --> P1
|
||||
P1 --> P2
|
||||
P1 --> P3
|
||||
P2 --> P4
|
||||
P3 --> P4
|
||||
P4 --> S1
|
||||
P4 --> S2
|
||||
P5 --> P4
|
||||
P4 --> P5
|
||||
P5 --> OUT1
|
||||
P5 --> OUT2
|
||||
P5 --> OUT3
|
||||
P1 --> S3
|
||||
|
||||
style IN1 fill:#e1f5ff
|
||||
style S1 fill:#c8e6c9
|
||||
style OUT1 fill:#c8e6c9
|
||||
```
|
||||
|
||||
### 3.2 模块详细设计
|
||||
|
||||
#### 3.2.1 Context Builder（上下文构建器）
|
||||
|
||||
**职责**：收集并整合所有上下文信息
|
||||
|
||||
**输入**：
|
||||
- 查询文本
|
||||
- Session ID
|
||||
- 用户 ID
|
||||
- 场景描述（可选）
|
||||
- 用户角色（可选）
|
||||
- 元数据（可选）
|
||||
|
||||
**输出**：`AgentSearchContext`
|
||||
|
||||
**核心逻辑**：
|
||||
1. 从缓存/数据库加载会话历史
|
||||
2. 提取最近 N 轮对话作为上下文
|
||||
3. 整合用户角色和场景信息
|
||||
4. 构建统一的上下文对象
|
||||
|
||||
#### 3.2.2 Intent Classifier（意图分类器）
|
||||
|
||||
**职责**：识别用户查询的意图类型
|
||||
|
||||
**实现方式**：基于向量相似度的分类
|
||||
|
||||
**流程**：
|
||||
1. 将查询文本转换为向量
|
||||
2. 与预定义的意图嵌入计算余弦相似度
|
||||
3. 返回 Top-N 意图及置信度
|
||||
|
||||
**输出**：`IntentClassificationResult`
|
||||
|
||||
#### 3.2.3 Query Embedder（查询向量化）
|
||||
|
||||
**职责**：将综合查询上下文转换为向量
|
||||
|
||||
**向量融合公式**：
|
||||
```
|
||||
query_vector = normalize(
|
||||
w1 * embed(query_text) +
|
||||
w2 * embed(conversation_context) +
|
||||
w3 * embed(scenario_description) +
|
||||
w4 * embed(user_role) +
|
||||
w5 * intent_embedding
|
||||
)
|
||||
```
|
||||
|
||||
**权重配置**：
|
||||
- w1 (查询文本): 0.4
|
||||
- w2 (对话上下文): 0.2
|
||||
- w3 (场景描述): 0.15
|
||||
- w4 (用户角色): 0.1
|
||||
- w5 (意图): 0.15
|
||||
|
||||
#### 3.2.4 Hybrid Search Service（混合检索服务）
|
||||
|
||||
**职责**：执行多维度检索，综合多种检索策略
|
||||
|
||||
**检索维度**：
|
||||
1. 向量检索：pgvector 语义相似度
|
||||
2. 意图过滤：基于意图类别过滤
|
||||
3. 标签匹配：精确匹配用户指定标签
|
||||
4. 权限过滤：根据用户角色过滤可见技能
|
||||
|
||||
**检索策略**：先向量检索获得候选集，再逐步过滤
|
||||
|
||||
#### 3.2.5 Skill Reranker（技能重排序器）
|
||||
|
||||
**职责**：对检索结果进行多维度重排序
|
||||
|
||||
**评分公式**：
|
||||
```
|
||||
final_score = α * vector_similarity
|
||||
+ β * intent_match_score
|
||||
+ γ * tag_match_score
|
||||
+ δ * popularity_score
|
||||
+ ε * rating_score
|
||||
```
|
||||
|
||||
**权重配置**：
|
||||
- α (向量相似度): 0.4
|
||||
- β (意图匹配): 0.25
|
||||
- γ (标签匹配): 0.2
|
||||
- δ (流行度): 0.1
|
||||
- ε (评分): 0.05
|
||||
|
||||
#### 3.2.6 Agent Session Service（会话管理服务）
|
||||
|
||||
**职责**：管理 Agent 对话会话的生命周期
|
||||
|
||||
**核心功能**：
|
||||
- 创建新会话
|
||||
- 追加对话消息
|
||||
- 获取会话历史
|
||||
- 会话过期管理
|
||||
- 异步持久化
|
||||
|
||||
## 4. 数据流设计
|
||||
|
||||
### 4.1 主检索流程
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant API
|
||||
participant ContextBuilder
|
||||
participant IntentClassifier
|
||||
participant QueryEmbedder
|
||||
participant HybridSearch
|
||||
participant Reranker
|
||||
participant Redis
|
||||
participant PostgreSQL
|
||||
|
||||
Client->>API: POST /api/agent/skills/search
|
||||
API->>ContextBuilder: buildContext(request)
|
||||
ContextBuilder->>Redis: getSession(sessionId)
|
||||
Redis-->>ContextBuilder: session data
|
||||
ContextBuilder-->>API: AgentSearchContext
|
||||
|
||||
API->>IntentClassifier: classify(query, context)
|
||||
IntentClassifier->>PostgreSQL: query intents by vector
|
||||
PostgreSQL-->>IntentClassifier: top intents
|
||||
IntentClassifier-->>API: IntentClassificationResult
|
||||
|
||||
API->>QueryEmbedder: embed(query, context, intent)
|
||||
QueryEmbedder->>Redis: get cached embedding
|
||||
alt cache miss
|
||||
QueryEmbedder->>QueryEmbedder: call OpenAI API
|
||||
QueryEmbedder->>Redis: cache embedding
|
||||
end
|
||||
QueryEmbedder-->>API: query_vector
|
||||
|
||||
API->>HybridSearch: search(vector, context, intent)
|
||||
HybridSearch->>PostgreSQL: vector search with filters
|
||||
PostgreSQL-->>HybridSearch: candidate skills
|
||||
HybridSearch-->>API: candidate results
|
||||
|
||||
API->>Reranker: rerank(candidates, context)
|
||||
Reranker-->>API: ranked results
|
||||
|
||||
API-->>Client: SearchResponse
|
||||
|
||||
Note over API,Redis: async save message
|
||||
API->>Redis: saveMessage(sessionId, message)
|
||||
```
|
||||
|
||||
### 4.2 技能向量同步流程
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant SkillService
|
||||
participant MessageQueue
|
||||
participant EmbeddingSync
|
||||
participant OpenAI
|
||||
participant PostgreSQL
|
||||
|
||||
SkillService->>MessageQueue: publish skill update event
|
||||
MessageQueue->>EmbeddingSync: consume event
|
||||
EmbeddingSync->>PostgreSQL: get skill metadata
|
||||
PostgreSQL-->>EmbeddingSync: skill data
|
||||
EmbeddingSync->>OpenAI: generate embeddings
|
||||
OpenAI-->>EmbeddingSync: embedding vectors
|
||||
EmbeddingSync->>PostgreSQL: upsert embeddings
|
||||
EmbeddingSync->>PostgreSQL: rebuild vector index
|
||||
PostgreSQL-->>EmbeddingSync: success
|
||||
```
|
||||
|
||||
## 5. 架构图
|
||||
|
||||
### 5.1 系统架构图
|
||||
|
||||
```mermaid
|
||||
C4Context
|
||||
title Agent+Skill 检索系统架构
|
||||
|
||||
Person(agent_user, "Agent 用户", "使用 Agent 进行技能搜索")
|
||||
System(skillhub_system, "SkillHub 系统", "技能注册与发现平台")
|
||||
System(external_ai, "OpenAI API", "嵌入向量生成服务")
|
||||
SystemDb(database, "PostgreSQL", "数据持久化与向量检索")
|
||||
|
||||
Rel(agent_user, skillhub_system, "使用", "HTTPS/JSON")
|
||||
Rel(skillhub_system, external_ai, "调用", "HTTPS")
|
||||
Rel(skillhub_system, database, "读写", "JDBC")
|
||||
|
||||
System_Boundary(skillhub_boundary, "SkillHub 系统") {
|
||||
System_Ext(api_gateway, "API Gateway", "REST API 接入层")
|
||||
System(application, "Application Layer", "业务应用层")
|
||||
System(service_layer, "Service Layer", "核心服务层")
|
||||
System(cache, "Redis", "缓存层")
|
||||
}
|
||||
|
||||
Rel(agent_user, api_gateway, "调用")
|
||||
Rel(api_gateway, application, "路由")
|
||||
Rel(application, service_layer, "调用")
|
||||
Rel(service_layer, cache, "缓存读写")
|
||||
```
|
||||
|
||||
### 5.2 容器架构图
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "用户接入层"
|
||||
LB[Load Balancer]
|
||||
end
|
||||
|
||||
subgraph "应用服务层"
|
||||
GW[API Gateway]
|
||||
APP1[SkillHub App 1]
|
||||
APP2[SkillHub App 2]
|
||||
APP_N[SkillHub App N]
|
||||
end
|
||||
|
||||
subgraph "数据处理层"
|
||||
WORKER[Embedding Worker]
|
||||
end
|
||||
|
||||
subgraph "存储层"
|
||||
PG[(PostgreSQL Primary)]
|
||||
PG_REPLICA[(PostgreSQL Replica)]
|
||||
REDIS[(Redis Cluster)]
|
||||
MQ[Message Queue]
|
||||
end
|
||||
|
||||
subgraph "外部服务"
|
||||
OPENAI[OpenAI API]
|
||||
end
|
||||
|
||||
LB --> GW
|
||||
GW --> APP1
|
||||
GW --> APP2
|
||||
GW --> APP_N
|
||||
|
||||
APP1 --> PG
|
||||
APP1 --> PG_REPLICA
|
||||
APP1 --> REDIS
|
||||
APP2 --> PG
|
||||
APP2 --> PG_REPLICA
|
||||
APP2 --> REDIS
|
||||
APP_N --> PG
|
||||
APP_N --> PG_REPLICA
|
||||
APP_N --> REDIS
|
||||
|
||||
APP1 --> MQ
|
||||
APP2 --> MQ
|
||||
APP_N --> MQ
|
||||
|
||||
MQ --> WORKER
|
||||
WORKER --> OPENAI
|
||||
WORKER --> PG
|
||||
|
||||
PG -.->|Replication| PG_REPLICA
|
||||
|
||||
APP1 --> OPENAI
|
||||
APP2 --> OPENAI
|
||||
APP_N --> OPENAI
|
||||
|
||||
style LB fill:#e1f5ff
|
||||
style PG fill:#c8e6c9
|
||||
style REDIS fill:#fff9c4
|
||||
style OPENAI fill:#ffebee
|
||||
```
|
||||
|
||||
## 6. 可扩展性设计
|
||||
|
||||
### 6.1 水平扩展
|
||||
|
||||
| 组件 | 扩展方式 | 说明 |
|
||||
|------|----------|------|
|
||||
| 应用服务 | 多实例部署 | 通过负载均衡分发请求 |
|
||||
| 数据库 | 读写分离 | 主库写入，从库读取 |
|
||||
| 缓存 | 集群模式 | Redis Cluster 分片存储 |
|
||||
| 向量索引 | 分区表 | 按技能类别分区 |
|
||||
|
||||
### 6.2 垂直扩展
|
||||
|
||||
| 资源 | 优化方向 | 说明 |
|
||||
|------|----------|------|
|
||||
| CPU | 异步处理 | 嵌入生成异步化 |
|
||||
| 内存 | 缓存优化 | 热数据缓存 |
|
||||
| 网络 | 批量请求 | 减少 API 调用次数 |
|
||||
| 存储 | 索引优化 | ivfflat 参数调优 |
|
||||
|
||||
### 6.3 扩展点
|
||||
|
||||
1. **嵌入提供者接口**：支持切换不同的嵌入模型
|
||||
2. **意图分类器接口**：支持自定义意图分类逻辑
|
||||
3. **重排序策略接口**：支持业务特定的排序规则
|
||||
4. **过滤器插件接口**：支持添加自定义过滤逻辑
|
||||
|
||||
## 7. 可靠性设计
|
||||
|
||||
### 7.1 故障处理策略
|
||||
|
||||
| 故障场景 | 处理策略 | 降级方案 |
|
||||
|----------|----------|----------|
|
||||
| OpenAI API 调用失败 | 重试 3 次，指数退避 | 使用本地哈希向量 |
|
||||
| 向量索引不可用 | 记录告警，使用全文搜索 | 基于关键词检索 |
|
||||
| 意图识别失败 | 使用默认意图 "general" | 无降级 |
|
||||
| 数据库连接失败 | 切换到只读副本 | 返回缓存结果 |
|
||||
| Redis 不可用 | 直接查询数据库 | 稍微增加响应时间 |
|
||||
| 消息队列不可用 | 同步处理嵌入 | 稍微增加发布时间 |
|
||||
|
||||
### 7.2 数据一致性
|
||||
|
||||
| 场景 | 策略 |
|
||||
|------|------|
|
||||
| 会话消息保存 | 写后读一致性 |
|
||||
| 嵌入向量更新 | 最终一致性 |
|
||||
| 缓存更新 | Write-Through |
|
||||
| 搜索索引更新 | 异步重建 |
|
||||
|
||||
### 7.3 容灾设计
|
||||
|
||||
- 数据库定期备份（每日全量 + 每小时增量）
|
||||
- Redis AOF 持久化 + RDB 快照
|
||||
- 应用服务多可用区部署
|
||||
- 关键接口重试机制
|
||||
|
||||
## 8. 性能设计
|
||||
|
||||
### 8.1 性能目标
|
||||
|
||||
| 指标 | 目标值 |
|
||||
|------|--------|
|
||||
| P95 查询响应时间 | < 500ms |
|
||||
| P99 查询响应时间 | < 1000ms |
|
||||
| 并发 QPS | > 100 |
|
||||
| 支持 Agent 数量 | > 1000 |
|
||||
| 单会话消息数 | > 1000 |
|
||||
| 技能库规模 | > 10000 |
|
||||
|
||||
### 8.2 性能优化策略
|
||||
|
||||
1. **缓存策略**
|
||||
- 活跃会话：Redis 缓存，TTL 1 小时
|
||||
- 意图嵌入：Redis 缓存，永久
|
||||
- 技能嵌入：PostgreSQL 存储，版本更新时失效
|
||||
- 查询结果：Redis 缓存，TTL 5 分钟
|
||||
|
||||
2. **数据库优化**
|
||||
- 向量索引：ivfflat，lists = sqrt(行数)
|
||||
- 查询优化：限制候选集大小
|
||||
- 连接池：合理配置连接池大小
|
||||
|
||||
3. **异步处理**
|
||||
- 嵌入生成：异步处理
|
||||
- 会话消息保存：异步持久化
|
||||
- 指标收集：异步上报
|
||||
|
||||
4. **批量操作**
|
||||
- 批量向量生成
|
||||
- 批量数据库写入
|
||||
- 批量缓存更新
|
||||
|
||||
## 9. 安全设计
|
||||
|
||||
### 9.1 安全层次
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "网络安全"
|
||||
N1[TLS/HTTPS 加密传输]
|
||||
N2[防火墙规则]
|
||||
end
|
||||
|
||||
subgraph "应用安全"
|
||||
A1[身份认证<br/>JWT/OAuth2]
|
||||
A2[权限控制<br/>RBAC]
|
||||
A3[输入验证]
|
||||
A4[输出编码]
|
||||
end
|
||||
|
||||
subgraph "数据安全"
|
||||
D1[SQL 注入防护]
|
||||
D2[敏感数据加密]
|
||||
D3[访问审计]
|
||||
end
|
||||
|
||||
subgraph "API 安全"
|
||||
API1[速率限制]
|
||||
API2[API Key 管理]
|
||||
API3[请求签名]
|
||||
end
|
||||
|
||||
N1 --> A1
|
||||
A1 --> A2
|
||||
A2 --> D1
|
||||
D1 --> API1
|
||||
|
||||
style A1 fill:#fff9c4
|
||||
style D1 fill:#ffebee
|
||||
```
|
||||
|
||||
### 9.2 安全措施
|
||||
|
||||
| 安全领域 | 措施 |
|
||||
|----------|------|
|
||||
| 认证 | JWT Token，有效期 1 小时 |
|
||||
| 授权 | 基于角色的访问控制 (RBAC) |
|
||||
| 加密 | HTTPS 传输，数据库字段加密 |
|
||||
| 审计 | 记录关键操作日志 |
|
||||
| 限流 | 用户级别 + IP 级别限流 |
|
||||
| 输入验证 | 所有输入参数校验 |
|
||||
|
||||
## 10. 监控与可观测性
|
||||
|
||||
### 10.1 监控指标
|
||||
|
||||
| 类别 | 指标 |
|
||||
|------|------|
|
||||
| 业务 | 搜索请求量、Top-10 准确率、意图识别准确率 |
|
||||
| 性能 | 响应时间 P50/P95/P99、QPS、错误率 |
|
||||
| 系统 | CPU 使用率、内存使用率、磁盘 I/O |
|
||||
| 数据库 | 连接数、查询耗时、慢查询数 |
|
||||
| 缓存 | 命中率、内存使用、键数量 |
|
||||
|
||||
### 10.2 日志策略
|
||||
|
||||
| 日志类型 | 级别 | 内容 |
|
||||
|----------|------|------|
|
||||
| 访问日志 | INFO | 请求/响应摘要 |
|
||||
| 业务日志 | INFO | 关键业务操作 |
|
||||
| 错误日志 | ERROR | 异常堆栈 |
|
||||
| 审计日志 | WARN | 敏感操作 |
|
||||
|
||||
### 10.3 告警规则
|
||||
|
||||
| 场景 | 阈值 | 级别 |
|
||||
|------|------|------|
|
||||
| P95 响应时间 | > 1000ms | WARNING |
|
||||
| 错误率 | > 1% | WARNING |
|
||||
| 错误率 | > 5% | CRITICAL |
|
||||
| 数据库连接数 | > 80% | WARNING |
|
||||
| OpenAI API 失败率 | > 10% | CRITICAL |
|
||||
|
||||
## 11. 部署架构
|
||||
|
||||
### 11.1 开发环境
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
DEV[开发环境] --> DEV_APP[单实例应用]
|
||||
DEV_APP --> DEV_PG[(PostgreSQL)]
|
||||
DEV_APP --> DEV_REDIS[(Redis)]
|
||||
|
||||
style DEV_APP fill:#e1f5ff
|
||||
style DEV_PG fill:#c8e6c9
|
||||
```
|
||||
|
||||
### 11.2 生产环境
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "生产环境"
|
||||
LB[Load Balancer]
|
||||
APP1[App Instance 1]
|
||||
APP2[App Instance 2]
|
||||
APP3[App Instance N]
|
||||
PG_MASTER[(PostgreSQL Master)]
|
||||
PG_SLAVE1[(PostgreSQL Slave 1)]
|
||||
PG_SLAVE2[(PostgreSQL Slave 2)]
|
||||
REDIS_CLUSTER[Redis Cluster]
|
||||
MQ_CLUSTER[Message Queue Cluster]
|
||||
end
|
||||
|
||||
LB --> APP1
|
||||
LB --> APP2
|
||||
LB --> APP3
|
||||
|
||||
APP1 --> PG_MASTER
|
||||
APP1 --> PG_SLAVE1
|
||||
APP1 --> REDIS_CLUSTER
|
||||
APP1 --> MQ_CLUSTER
|
||||
|
||||
APP2 --> PG_MASTER
|
||||
APP2 --> PG_SLAVE2
|
||||
APP2 --> REDIS_CLUSTER
|
||||
APP2 --> MQ_CLUSTER
|
||||
|
||||
APP3 --> PG_MASTER
|
||||
APP3 --> PG_SLAVE1
|
||||
APP3 --> REDIS_CLUSTER
|
||||
APP3 --> MQ_CLUSTER
|
||||
|
||||
PG_MASTER -.->|Replication| PG_SLAVE1
|
||||
PG_MASTER -.->|Replication| PG_SLAVE2
|
||||
|
||||
style LB fill:#e1f5ff
|
||||
style PG_MASTER fill:#c8e6c9
|
||||
style REDIS_CLUSTER fill:#fff9c4
|
||||
```
|
||||
|
||||
### 11.3 部署策略
|
||||
|
||||
| 环境 | 实例数 | 数据库 | 缓存 |
|
||||
|------|--------|--------|------|
|
||||
| 开发 | 1 | 单实例 | 单实例 |
|
||||
| 测试 | 2 | 1主1从 | 单实例 |
|
||||
| 预发 | 2 | 1主2从 | 哨兵模式 |
|
||||
| 生产 | 3+ | 1主N从 | 集群模式 |
|
||||
|
||||
## 12. 总结
|
||||
|
||||
本架构设计遵循以下原则：
|
||||
|
||||
1. **分层清晰**：表现层、应用层、服务层、领域层、基础设施层职责分明
|
||||
2. **高内聚低耦合**：各模块职责单一，依赖关系清晰
|
||||
3. **可扩展**：通过接口抽象支持多种扩展方式
|
||||
4. **高可用**：多实例部署、读写分离、故障降级
|
||||
5. **高性能**：多层缓存、异步处理、批量操作
|
||||
6. **安全可靠**：多层次安全防护、完善的监控告警
|
||||
|
||||
该架构能够满足 Agent+Skill 检索系统的核心需求，为后续实现提供清晰的指导。
|
||||
1472
docs/agent-skills-search/agent_teams/02-agent-module-design.md
Normal file
1472
docs/agent-skills-search/agent_teams/02-agent-module-design.md
Normal file
File diff suppressed because it is too large
Load diff
321
docs/agent-skills-search/agent_teams/03-skills-module-design.md
Normal file
321
docs/agent-skills-search/agent_teams/03-skills-module-design.md
Normal file
|
|
@ -0,0 +1,321 @@
|
|||
# Skills 模块设计方案
|
||||
|
||||
## 1. Skill 数据模型设计
|
||||
|
||||
### 1.1 核心数据表结构
|
||||
|
||||
#### skills 表
|
||||
| 字段名 | 类型 | 说明 | 索引 |
|
||||
|--------|------|------|------|
|
||||
| id | uuid | 主键 | PK |
|
||||
| name | varchar(255) | Skill 名称 | |
|
||||
| description | text | Skill 描述 | |
|
||||
| embedding | vector(1536) | 语义向量（基于 description 生成） | pgvector |
|
||||
| category_id | uuid | 所属分类 | FK |
|
||||
| is_active | boolean | 是否启用 | |
|
||||
| created_at | timestamp | 创建时间 | |
|
||||
| updated_at | timestamp | 更新时间 | |
|
||||
|
||||
#### skill_tags 表
|
||||
| 字段名 | 类型 | 说明 | 索引 |
|
||||
|--------|------|------|------|
|
||||
| id | uuid | 主键 | PK |
|
||||
| skill_id | uuid | 关联的 skill | FK |
|
||||
| tag | varchar(100) | 标签名称 | |
|
||||
| tag_type | varchar(50) | 标签类型（domain/role/action） | |
|
||||
|
||||
#### skill_metadata 表
|
||||
| 字段名 | 类型 | 说明 | 索引 |
|
||||
|--------|------|------|------|
|
||||
| id | uuid | 主键 | PK |
|
||||
| skill_id | uuid | 关联的 skill | FK |
|
||||
| key | varchar(100) | 元数据键 | |
|
||||
| value | text | 元数据值（JSON 格式） | |
|
||||
|
||||
#### skill_categories 表
|
||||
| 字段名 | 类型 | 说明 | 索引 |
|
||||
|--------|------|------|------|
|
||||
| id | uuid | 主键 | PK |
|
||||
| name | varchar(100) | 分类名称 | |
|
||||
| parent_id | uuid | 父分类 | FK |
|
||||
| level | int | 分类层级 | |
|
||||
|
||||
### 1.2 向量化方案
|
||||
|
||||
#### 向量化内容来源
|
||||
1. **主要向量**：基于 Skill 的 `description` 字段
|
||||
2. **增强向量**：合并 `name` + `description` + `tags` 生成更丰富的语义表示
|
||||
3. **多维度向量**：针对不同场景生成独立向量（如：技术栈向量、业务场景向量）
|
||||
|
||||
#### 向量维度选择
|
||||
- 推荐维度：1536（OpenAI text-embedding-3-small/large）
|
||||
- 备选维度：768（轻量级场景）
|
||||
- 存储类型：`vector(1536)` 或 `vector(768)`
|
||||
|
||||
#### 向量更新策略
|
||||
- 创建 Skill 时自动生成向量
|
||||
- 当 `description`、`name`、`tags` 变更时，异步重新计算向量
|
||||
- 支持手动触发向量重新生成
|
||||
|
||||
## 2. Skill 标签和元数据结构
|
||||
|
||||
### 2.1 标签分类体系
|
||||
|
||||
#### Domain 标签（领域标签）
|
||||
- 描述 Skill 所属业务领域
|
||||
- 示例：`database`、`api`、`auth`、`ui`、`testing`、`deployment`
|
||||
|
||||
#### Role 标签（角色标签）
|
||||
- 描述适合使用该 Skill 的角色
|
||||
- 示例：`developer`、`devops`、`analyst`、`designer`、`manager`
|
||||
|
||||
#### Action 标签（动作标签）
|
||||
- 描述 Skill 的主要动作类型
|
||||
- 示例：`create`、`read`、`update`、`delete`、`analyze`、`deploy`
|
||||
|
||||
#### Tech Stack 标签（技术栈标签）
|
||||
- 描述 Skill 涉及的技术栈
|
||||
- 示例：`python`、`java`、`react`、`kubernetes`、`postgresql`
|
||||
|
||||
### 2.2 元数据结构设计
|
||||
|
||||
#### 标准元数据字段
|
||||
```json
|
||||
{
|
||||
"complexity": "low|medium|high",
|
||||
"estimated_time": "5m",
|
||||
"dependencies": ["skill_id_1", "skill_id_2"],
|
||||
"required_permissions": ["read_code", "write_files"],
|
||||
"compatibility": {
|
||||
"min_agent_version": "1.0.0",
|
||||
"supported_models": ["claude-sonnet-4.6", "claude-opus-4.6"]
|
||||
},
|
||||
"usage_stats": {
|
||||
"total_calls": 1000,
|
||||
"success_rate": 0.95
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 扩展元数据
|
||||
- 自定义字段支持 JSON 格式存储
|
||||
- 支持元数据查询和过滤
|
||||
|
||||
## 3. Skill 匹配算法设计
|
||||
|
||||
### 3.1 匹配流程架构
|
||||
|
||||
```
|
||||
输入：查询上下文（意图、历史、场景、用户角色）
|
||||
↓
|
||||
阶段 1：规则过滤
|
||||
├── 活跃状态过滤（is_active = true）
|
||||
├── 角色权限过滤
|
||||
├── 技术栈兼容性过滤
|
||||
└── 标签前置过滤
|
||||
↓
|
||||
阶段 2：语义相似度计算
|
||||
├── 查询向量生成（意图 + 场景 + 上下文）
|
||||
├── 向量相似度检索（ANN 搜索）
|
||||
└── 候选集排序
|
||||
↓
|
||||
阶段 3：多维度评分
|
||||
├── 语义相似度分（40%）
|
||||
├── 标签匹配分（30%）
|
||||
├── 历史使用频率分（20%）
|
||||
└── 上下文相关性分（10%）
|
||||
↓
|
||||
阶段 4：结果重排
|
||||
├── 多样性保证（避免重复类型）
|
||||
├── 相关性阈值过滤
|
||||
└── Top-K 返回
|
||||
↓
|
||||
输出：匹配的 Skills 列表（带评分）
|
||||
```
|
||||
|
||||
### 3.2 查询向量生成策略
|
||||
|
||||
#### 组合查询向量
|
||||
将以下内容合并生成查询向量：
|
||||
1. **当前意图文本**：用户当前的请求意图
|
||||
2. **场景描述**：当前工作场景的上下文描述
|
||||
3. **对话历史摘要**：最近 N 轮对话的语义摘要
|
||||
4. **用户角色信息**：用户角色相关的描述文本
|
||||
|
||||
#### 向量生成方式
|
||||
- 方式 A：直接拼接文本，生成单一向量
|
||||
- 方式 B：分别生成向量后加权平均
|
||||
- 方式 C：生成多个独立向量，分别检索后合并结果
|
||||
|
||||
### 3.3 相似度计算方法
|
||||
|
||||
#### 向量相似度
|
||||
- 使用余弦相似度（Cosine Similarity）
|
||||
- 公式：`similarity = (A · B) / (||A|| × ||B||)`
|
||||
- pgvector 操作符：`<=>`（余弦距离，越小越相似）
|
||||
|
||||
#### 标签匹配度
|
||||
- 精确匹配：标签完全一致
|
||||
- 层次匹配：父子标签层级匹配
|
||||
- 权重计算：不同标签类型设置不同权重
|
||||
|
||||
#### 上下文相关性
|
||||
- 基于对话历史中的 Skill 使用模式
|
||||
- 计算 Skill 与历史上下文的关联度
|
||||
|
||||
### 3.4 多维度评分公式
|
||||
|
||||
```
|
||||
Total Score = w1 × SemanticScore + w2 × TagScore + w3 × HistoryScore + w4 × ContextScore
|
||||
|
||||
其中：
|
||||
- w1 = 0.4（语义相似度权重）
|
||||
- w2 = 0.3（标签匹配权重）
|
||||
- w3 = 0.2（历史使用权重）
|
||||
- w4 = 0.1（上下文相关性权重）
|
||||
```
|
||||
|
||||
## 4. 场景描述处理策略
|
||||
|
||||
### 4.1 场景描述来源
|
||||
|
||||
1. **显式场景描述**：用户提供的场景文本
|
||||
2. **隐式场景推断**：从对话历史和上下文自动提取
|
||||
3. **环境上下文**：当前工作目录、文件类型、Git 状态等
|
||||
|
||||
### 4.2 场景描述处理流程
|
||||
|
||||
```
|
||||
原始场景输入
|
||||
↓
|
||||
文本预处理
|
||||
├── 去除噪声和冗余
|
||||
├── 提取关键实体
|
||||
└── 识别技术关键词
|
||||
↓
|
||||
场景分类
|
||||
├── 开发场景（coding/debugging/refactoring）
|
||||
├── 部署场景（deploy/monitor/maintenance）
|
||||
└── 分析场景（review/analysis/report）
|
||||
↓
|
||||
场景增强
|
||||
├── 关联历史相似场景
|
||||
├── 添加相关技术标签
|
||||
└── 生成场景向量
|
||||
↓
|
||||
输出：结构化场景描述
|
||||
```
|
||||
|
||||
### 4.3 场景描述缓存策略
|
||||
|
||||
- 热门场景描述缓存（LRU 策略）
|
||||
- 场景向量预计算
|
||||
- 定期清理过期缓存
|
||||
|
||||
## 5. pgvector 索引策略和检索优化
|
||||
|
||||
### 5.1 索引类型选择
|
||||
|
||||
#### HNSW 索引（推荐）
|
||||
- 优点：查询速度快，适合大规模数据
|
||||
- 适用场景：实时检索，查询频繁
|
||||
- 参数配置：
|
||||
- `m = 16`（每个节点的连接数）
|
||||
- `ef_construction = 64`（构建时的搜索宽度）
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_skills_embedding_hnsw
|
||||
ON skills USING hnsw (embedding vector_cosine_ops)
|
||||
WITH (m = 16, ef_construction = 64);
|
||||
```
|
||||
|
||||
#### IVFFlat 索引
|
||||
- 优点：构建速度快，内存占用低
|
||||
- 适用场景：数据更新频繁
|
||||
- 参数配置：
|
||||
- `lists = 100`（聚类中心数量）
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_skills_embedding_ivfflat
|
||||
ON skills USING ivfflat (embedding vector_cosine_ops)
|
||||
WITH (lists = 100);
|
||||
```
|
||||
|
||||
### 5.2 检索优化策略
|
||||
|
||||
#### 批量检索优化
|
||||
- 使用 `array_agg` 批量获取结果
|
||||
- 减少 SQL 查询次数
|
||||
|
||||
#### 预过滤优化
|
||||
- 先用规则过滤缩小候选集
|
||||
- 再在候选集上进行向量检索
|
||||
|
||||
#### 查询参数调优
|
||||
- `ef_search`：控制搜索精度与速度的权衡
|
||||
- 推荐值：`ef_search = 40`（平衡性能）
|
||||
|
||||
```sql
|
||||
SET hnsw.ef_search = 40;
|
||||
```
|
||||
|
||||
#### 分页优化
|
||||
- 使用游标分页代替 OFFSET
|
||||
- 避免深度分页性能问题
|
||||
|
||||
### 5.3 索引维护策略
|
||||
|
||||
#### 定期重建索引
|
||||
- 当数据变更超过 20% 时重建 HNSW 索引
|
||||
- 使用 `REINDEX INDEX` 命令
|
||||
|
||||
#### 向量更新优化
|
||||
- 批量更新向量，减少索引重建频率
|
||||
- 使用异步任务处理向量计算
|
||||
|
||||
#### 索引监控
|
||||
- 监控索引大小和查询性能
|
||||
- 定期收集统计信息
|
||||
|
||||
```sql
|
||||
ANALYZE skills;
|
||||
```
|
||||
|
||||
### 5.4 查询性能优化
|
||||
|
||||
#### 查询计划分析
|
||||
- 使用 `EXPLAIN ANALYZE` 分析查询性能
|
||||
- 识别性能瓶颈并优化
|
||||
|
||||
#### 结果缓存
|
||||
- 缓存热门查询结果
|
||||
- 设置合理的缓存过期时间
|
||||
|
||||
#### 连接池配置
|
||||
- 合理配置数据库连接池大小
|
||||
- 避免连接频繁创建和销毁
|
||||
|
||||
## 6. 扩展性设计
|
||||
|
||||
### 6.1 插件化标签系统
|
||||
- 支持自定义标签类型
|
||||
- 支持标签权重动态配置
|
||||
|
||||
### 6.2 多语言向量支持
|
||||
- 支持不同语言的向量嵌入
|
||||
- 支持多语言混合查询
|
||||
|
||||
### 6.3 A/B 测试框架
|
||||
- 支持不同匹配算法的对比测试
|
||||
- 支持实时算法切换
|
||||
|
||||
### 6.4 可观测性
|
||||
- 记录匹配过程的详细日志
|
||||
- 统计各项指标（召回率、准确率、响应时间）
|
||||
- 支持算法效果分析
|
||||
|
||||
---
|
||||
|
||||
**设计版本**：v1.0
|
||||
**设计日期**：2026-04-05
|
||||
**设计人**：Skills 开发工程师
|
||||
96
docs/agent-skills-search/doc-01/01-overview.md
Normal file
96
docs/agent-skills-search/doc-01/01-overview.md
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
# Agent Skills 智能检索系统 - 系统概述
|
||||
|
||||
## 1. 项目背景
|
||||
|
||||
SkillHub 是一个 Agent 技能注册中心，用于管理 Agent 技能包的发布、版本控制和发现。为了提升 Agent 选择技能的智能化水平，需要构建一个基于向量检索的智能匹配系统。
|
||||
|
||||
## 2. 核心目标
|
||||
|
||||
构建一个稳定可靠、精准匹配的技能检索系统，通过以下能力实现：
|
||||
|
||||
- **意图识别**：理解用户查询的真实意图
|
||||
- **上下文感知**：结合多轮对话历史，理解用户当前需求
|
||||
- **场景适配**：根据使用场景和用户角色，推荐最合适的技能
|
||||
- **精准匹配**：结合语义向量、标签、类别等多维度信息进行匹配
|
||||
|
||||
## 3. 核心输入
|
||||
|
||||
| 输入项 | 说明 | 示例 |
|
||||
|--------|------|------|
|
||||
| 意图识别 | 用户查询的意图类型 | "代码生成"、"数据分析"、"文本处理" |
|
||||
| 多轮对话上下文 | 用户与 Agent 的历史对话 | 过去 N 轮的问答记录 |
|
||||
| 场景描述 | 当前使用场景 | "Web 开发"、"数据可视化" |
|
||||
| 用户角色信息 | 用户身份和权限 | "开发者"、"数据分析师" |
|
||||
| 标签信息 | 技能的标签分类 | ["python", "react", "api"] |
|
||||
| 元数据 | 技能的详细元信息 | 版本、依赖、示例代码等 |
|
||||
|
||||
## 4. 技术选型
|
||||
|
||||
| 组件 | 选型 | 说明 |
|
||||
|------|------|------|
|
||||
| 向量数据库 | PostgreSQL + pgvector | 利用现有数据库，减少架构复杂度 |
|
||||
| 嵌入模型 | OpenAI text-embedding-3-small | 1536 维向量，高质量语义表示 |
|
||||
| 意图识别 | 基于向量相似度 | 预定义意图类别，通过向量匹配 |
|
||||
| 向量索引 | ivfflat | 平衡精度和性能 |
|
||||
| 对话管理 | Redis + PostgreSQL | 热数据缓存，冷数据持久化 |
|
||||
| 后端框架 | Spring Boot 3.2.3 | 现有技术栈，无缝集成 |
|
||||
|
||||
## 5. 系统特性
|
||||
|
||||
### 5.1 稳定性
|
||||
- 基于成熟的 PostgreSQL 数据库
|
||||
- pgvector 扩展经过生产验证
|
||||
- 完善的错误处理和降级机制
|
||||
|
||||
### 5.2 可靠性
|
||||
- 多维度评分，避免单点失败
|
||||
- 权限控制确保数据安全
|
||||
- 异步处理保障响应性能
|
||||
|
||||
### 5.3 精准匹配
|
||||
- 语义向量相似度检索
|
||||
- 意图类别过滤
|
||||
- 标签精确匹配
|
||||
- 多维度权重融合
|
||||
|
||||
## 6. 性能指标
|
||||
|
||||
| 指标 | 目标值 | 说明 |
|
||||
|------|--------|------|
|
||||
| 查询响应时间 | < 500ms | 95 分位 |
|
||||
| 并发 QPS | > 100 | 稳定支持 |
|
||||
| Top-10 准确率 | > 80% | 相关技能在前 10 |
|
||||
| 意图匹配准确率 | > 85% | 意图分类正确率 |
|
||||
|
||||
## 7. 与现有系统集成
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ SkillHub 现有系统 │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Skill 管理 │ │ 搜索服务 │ │ 权限系统 │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ 复用现有能力
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Agent Skills 智能检索系统 │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ 对话管理 │ │ 意图识别 │ │ 智能匹配 │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 8. 文档结构
|
||||
|
||||
| 文档 | 内容 |
|
||||
|------|------|
|
||||
| 01-overview.md | 系统概述（本文档） |
|
||||
| 02-architecture.md | 架构设计 |
|
||||
| 03-data-model.md | 数据模型 |
|
||||
| 04-algorithms.md | 算法设计 |
|
||||
| 05-flow-diagrams.md | 流程图 |
|
||||
| 06-architecture-diagrams.md | 架构图 |
|
||||
| 07-api-design.md | API 设计 |
|
||||
| 08-implementation-guide.md | 实施指南 |
|
||||
266
docs/agent-skills-search/doc-01/02-architecture.md
Normal file
266
docs/agent-skills-search/doc-01/02-architecture.md
Normal file
|
|
@ -0,0 +1,266 @@
|
|||
# Agent Skills 智能检索系统 - 架构设计
|
||||
|
||||
## 1. 系统架构分层
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ API Gateway Layer │
|
||||
│ ┌──────────────────────────────────────────────────────────┐ │
|
||||
│ │ Agent Skill Search Controller │ │
|
||||
│ │ - POST /api/agent/skills/search │ │
|
||||
│ │ - POST /api/agent/sessions │ │
|
||||
│ │ - GET /api/agent/sessions/{id} │ │
|
||||
│ └──────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────▼───────────────────────────────────┐
|
||||
│ Service Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Context │ │ Intent │ │ Query │ │
|
||||
│ │ Builder │ │ Classifier │ │ Embedder │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Hybrid │ │ Skill │ │ Agent │ │
|
||||
│ │ Search │ │ Reranker │ │ Session │ │
|
||||
│ │ Service │ │ │ │ Service │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────▼───────────────────────────────────┐
|
||||
│ Domain Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Agent │ │ Skill │ │ Intent │ │
|
||||
│ │ Session │ │ Embedding │ │ Mapping │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────▼───────────────────────────────────┐
|
||||
│ Infrastructure Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ PostgreSQL │ │ Redis │ │ OpenAI │ │
|
||||
│ │ + pgvector │ │ Cache │ │ Embedding │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 2. 核心组件详解
|
||||
|
||||
### 2.1 Context Builder（上下文构建器）
|
||||
|
||||
**职责**：收集并构建完整的查询上下文
|
||||
|
||||
**输入**：
|
||||
- 用户查询文本
|
||||
- Session ID
|
||||
- 用户 ID
|
||||
- 场景描述
|
||||
- 用户角色
|
||||
|
||||
**输出**：`AgentSearchContext`
|
||||
|
||||
```java
|
||||
public record AgentSearchContext(
|
||||
String query, // 查询文本
|
||||
List<AgentMessage> conversationHistory, // 对话历史
|
||||
String scenarioDescription, // 场景描述
|
||||
String userRole, // 用户角色
|
||||
Map<String, Object> metadata // 元数据
|
||||
) {}
|
||||
```
|
||||
|
||||
### 2.2 Intent Classifier（意图分类器）
|
||||
|
||||
**职责**：识别用户查询的意图类型
|
||||
|
||||
**实现方式**：基于向量相似度的分类
|
||||
|
||||
**流程**：
|
||||
1. 将查询文本转换为向量
|
||||
2. 与预定义的意图嵌入计算余弦相似度
|
||||
3. 返回 Top-N 意图及置信度
|
||||
|
||||
```java
|
||||
public interface IntentClassifier {
|
||||
IntentClassificationResult classify(
|
||||
String query,
|
||||
AgentSearchContext context
|
||||
);
|
||||
}
|
||||
|
||||
public record IntentClassificationResult(
|
||||
String intentLabel, // 主要意图
|
||||
double confidence, // 置信度
|
||||
List<String> alternativeIntents // 备选意图
|
||||
) {}
|
||||
```
|
||||
|
||||
### 2.3 Query Embedder（查询向量化）
|
||||
|
||||
**职责**：将综合查询上下文转换为向量
|
||||
|
||||
**向量构建公式**：
|
||||
```
|
||||
query_vector = normalize(
|
||||
w1 * embed(query_text) +
|
||||
w2 * embed(conversation_context) +
|
||||
w3 * embed(scenario_description) +
|
||||
w4 * embed(user_role) +
|
||||
w5 * intent_embedding
|
||||
)
|
||||
```
|
||||
|
||||
**权重配置**：
|
||||
- w1 (查询文本): 0.4
|
||||
- w2 (对话上下文): 0.2
|
||||
- w3 (场景描述): 0.15
|
||||
- w4 (用户角色): 0.1
|
||||
- w5 (意图): 0.15
|
||||
|
||||
### 2.4 Hybrid Search Service（混合检索服务）
|
||||
|
||||
**职责**：执行多维度检索
|
||||
|
||||
**检索流程**：
|
||||
1. **向量检索**：使用 pgvector 进行语义相似度检索
|
||||
2. **意图过滤**：根据识别的意图过滤技能类别
|
||||
3. **标签匹配**：匹配用户指定或相关的标签
|
||||
4. **权限过滤**：根据用户角色过滤可见技能
|
||||
|
||||
### 2.5 Skill Reranker（技能重排序器）
|
||||
|
||||
**职责**：对检索结果进行多维度重排序
|
||||
|
||||
**评分公式**：
|
||||
```
|
||||
final_score = α * vector_similarity
|
||||
+ β * intent_match_score
|
||||
+ γ * tag_match_score
|
||||
+ δ * popularity_score
|
||||
+ ε * rating_score
|
||||
```
|
||||
|
||||
**权重配置**：
|
||||
- α (向量相似度): 0.4
|
||||
- β (意图匹配): 0.25
|
||||
- γ (标签匹配): 0.2
|
||||
- δ (流行度): 0.1
|
||||
- ε (评分): 0.05
|
||||
|
||||
### 2.6 Agent Session Service（会话管理服务）
|
||||
|
||||
**职责**：管理 Agent 对话会话
|
||||
|
||||
**功能**：
|
||||
- 创建新会话
|
||||
- 保存对话消息
|
||||
- 获取会话历史
|
||||
- 会话过期管理
|
||||
|
||||
## 3. 数据流
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ Agent Request│
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────┐
|
||||
│ 1. Context Builder │
|
||||
│ - 收集查询、历史、角色 │
|
||||
│ - 构建 SearchContext │
|
||||
└──────┬──────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────┐
|
||||
│ 2. Intent Classifier │
|
||||
│ - 向量化查询 │
|
||||
│ - 匹配意图 │
|
||||
└──────┬──────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────┐
|
||||
│ 3. Query Embedder │
|
||||
│ - 融合多维度信息 │
|
||||
│ - 生成查询向量 │
|
||||
└──────┬──────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────┐
|
||||
│ 4. Hybrid Search │
|
||||
│ - 向量检索 (pgvector) │
|
||||
│ - 意图/标签/权限过滤 │
|
||||
└──────┬──────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────┐
|
||||
│ 5. Skill Reranker │
|
||||
│ - 多维度评分 │
|
||||
│ - 重排序结果 │
|
||||
└──────┬──────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────┐
|
||||
│ 6. Return Results │
|
||||
│ - 返回匹配技能列表 │
|
||||
│ - 包含匹配原因 │
|
||||
└─────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 4. 模块依赖关系
|
||||
|
||||
```
|
||||
Agent Skill Search Controller
|
||||
│
|
||||
├─► AgentContextBuilder
|
||||
│ └─► AgentSessionService
|
||||
│ └─► Redis / PostgreSQL
|
||||
│
|
||||
├─► IntentClassifier
|
||||
│ ├─► QueryEmbedder
|
||||
│ └─► IntentMappingRepository
|
||||
│ └─► PostgreSQL
|
||||
│
|
||||
├─► HybridSkillSearchService
|
||||
│ ├─► QueryEmbedder
|
||||
│ ├─► SkillEmbeddingRepository
|
||||
│ ├─► IntentClassifier
|
||||
│ └─► SkillReranker
|
||||
│
|
||||
└─► SkillReranker
|
||||
├─► SkillRepository
|
||||
└─► LabelRepository
|
||||
```
|
||||
|
||||
## 5. 外部依赖
|
||||
|
||||
| 依赖 | 用途 | 调用方式 |
|
||||
|------|------|----------|
|
||||
| PostgreSQL | 数据持久化、向量检索 | JPA / Native SQL |
|
||||
| Redis | 会话缓存、热数据存储 | Spring Data Redis |
|
||||
| OpenAI API | 嵌入向量生成 | HTTP REST API |
|
||||
|
||||
## 6. 缓存策略
|
||||
|
||||
| 数据类型 | 缓存位置 | TTL | 更新策略 |
|
||||
|----------|----------|-----|----------|
|
||||
| 活跃会话 | Redis | 1小时 | 每次访问更新 |
|
||||
| 意图嵌入 | Redis | 永久 | 管理接口更新 |
|
||||
| 技能嵌入 | PostgreSQL | - | 版本更新时 |
|
||||
| 查询结果 | Redis | 5分钟 | 主动失效 |
|
||||
|
||||
## 7. 错误处理
|
||||
|
||||
| 场景 | 处理策略 |
|
||||
|------|----------|
|
||||
| OpenAI API 调用失败 | 降级到基于关键词的检索 |
|
||||
| 向量索引未就绪 | 使用全文搜索 |
|
||||
| 会话不存在 | 创建新会话 |
|
||||
| 意图识别失败 | 使用通用意图 "general" |
|
||||
|
||||
## 8. 扩展点
|
||||
|
||||
1. **嵌入模型切换**：支持配置不同的嵌入模型
|
||||
2. **意图库扩展**：通过数据库动态添加新意图
|
||||
3. **评分策略**：可配置的权重参数
|
||||
4. **检索后处理**：支持自定义业务规则过滤
|
||||
573
docs/agent-skills-search/doc-01/03-data-model.md
Normal file
573
docs/agent-skills-search/doc-01/03-data-model.md
Normal file
|
|
@ -0,0 +1,573 @@
|
|||
# Agent Skills 智能检索系统 - 数据模型
|
||||
|
||||
## 1. 数据库扩展
|
||||
|
||||
### 1.1 pgvector 扩展安装
|
||||
|
||||
```sql
|
||||
-- 在 PostgreSQL 中安装 pgvector 扩展
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
|
||||
-- 验证安装
|
||||
SELECT * FROM pg_extension WHERE extname = 'vector';
|
||||
```
|
||||
|
||||
## 2. 核心数据表
|
||||
|
||||
### 2.1 agent_session（Agent 会话表）
|
||||
|
||||
存储 Agent 与用户的对话会话信息。
|
||||
|
||||
```sql
|
||||
CREATE TABLE agent_session (
|
||||
-- 主键
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
|
||||
-- 标识信息
|
||||
agent_id VARCHAR(128) NOT NULL,
|
||||
user_id VARCHAR(128) NOT NULL,
|
||||
session_id VARCHAR(128) UNIQUE NOT NULL,
|
||||
|
||||
-- 上下文信息
|
||||
scenario_description TEXT,
|
||||
user_role VARCHAR(128),
|
||||
|
||||
-- 元数据（JSONB 格式，灵活扩展）
|
||||
metadata JSONB DEFAULT '{}',
|
||||
|
||||
-- 时间戳
|
||||
started_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
ended_at TIMESTAMP WITH TIME ZONE,
|
||||
|
||||
-- 索引
|
||||
CONSTRAINT fk_agent_session_user FOREIGN KEY (user_id) REFERENCES "user"(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- 创建索引
|
||||
CREATE INDEX idx_agent_session_agent_id ON agent_session(agent_id);
|
||||
CREATE INDEX idx_agent_session_user_id ON agent_session(user_id);
|
||||
CREATE INDEX idx_agent_session_session_id ON agent_session(session_id);
|
||||
CREATE INDEX idx_agent_session_started_at ON agent_session(started_at DESC);
|
||||
|
||||
-- GIN 索引用于 JSONB 查询
|
||||
CREATE INDEX idx_agent_session_metadata ON agent_session USING GIN(metadata);
|
||||
```
|
||||
|
||||
**字段说明**：
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| id | BIGSERIAL | 主键 |
|
||||
| agent_id | VARCHAR(128) | Agent 标识符 |
|
||||
| user_id | VARCHAR(128) | 用户 ID |
|
||||
| session_id | VARCHAR(128) | 会话唯一标识 |
|
||||
| scenario_description | TEXT | 场景描述 |
|
||||
| user_role | VARCHAR(128) | 用户角色 |
|
||||
| metadata | JSONB | 扩展元数据 |
|
||||
| started_at | TIMESTAMPTZ | 会话开始时间 |
|
||||
| updated_at | TIMESTAMPTZ | 会话更新时间 |
|
||||
| ended_at | TIMESTAMPTZ | 会话结束时间 |
|
||||
|
||||
### 2.2 agent_message（对话消息表）
|
||||
|
||||
存储会话中的每条消息。
|
||||
|
||||
```sql
|
||||
CREATE TABLE agent_message (
|
||||
-- 主键
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
|
||||
-- 关联会话
|
||||
session_id BIGINT NOT NULL,
|
||||
|
||||
-- 消息内容
|
||||
role VARCHAR(32) NOT NULL, -- 'user' | 'assistant' | 'system'
|
||||
content TEXT NOT NULL,
|
||||
|
||||
-- 意图识别结果
|
||||
intent_label VARCHAR(128),
|
||||
intent_confidence DECIMAL(5,4),
|
||||
|
||||
-- 向量嵌入
|
||||
embedding vector(1536),
|
||||
|
||||
-- 元数据
|
||||
metadata JSONB DEFAULT '{}',
|
||||
|
||||
-- 时间戳
|
||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
|
||||
-- 外键约束
|
||||
CONSTRAINT fk_agent_message_session
|
||||
FOREIGN KEY (session_id)
|
||||
REFERENCES agent_session(id)
|
||||
ON DELETE CASCADE,
|
||||
|
||||
-- 检查约束
|
||||
CONSTRAINT chk_agent_message_role
|
||||
CHECK (role IN ('user', 'assistant', 'system')),
|
||||
|
||||
CONSTRAINT chk_agent_message_confidence
|
||||
CHECK (intent_confidence IS NULL OR
|
||||
(intent_confidence >= 0 AND intent_confidence <= 1))
|
||||
);
|
||||
|
||||
-- 创建索引
|
||||
CREATE INDEX idx_agent_message_session_id ON agent_message(session_id);
|
||||
CREATE INDEX idx_agent_message_created_at ON agent_message(created_at DESC);
|
||||
CREATE INDEX idx_agent_message_intent ON agent_message(intent_label);
|
||||
|
||||
-- 向量索引（需要先有足够数据）
|
||||
-- CREATE INDEX idx_agent_message_embedding ON agent_message
|
||||
-- USING ivfflat(embedding vector_cosine_ops) WITH (lists = 100);
|
||||
|
||||
-- GIN 索引用于 JSONB 查询
|
||||
CREATE INDEX idx_agent_message_metadata ON agent_message USING GIN(metadata);
|
||||
```
|
||||
|
||||
**字段说明**：
|
||||
|
||||
| 字段 | 类型 | 说明 |
|
||||
|------|------|------|
|
||||
| id | BIGSERIAL | 主键 |
|
||||
| session_id | BIGINT | 关联的会话 ID |
|
||||
| role | VARCHAR(32) | 消息角色 |
|
||||
| content | TEXT | 消息内容 |
|
||||
| intent_label | VARCHAR(128) | 识别的意图标签 |
|
||||
| intent_confidence | DECIMAL(5,4) | 意图置信度 |
|
||||
| embedding | vector(1536) | 消息向量 |
|
||||
| metadata | JSONB | 扩展元数据 |
|
||||
| created_at | TIMESTAMPTZ | 创建时间 |
|
||||
|
||||
### 2.3 skill_embedding（技能向量表）
|
||||
|
||||
存储技能的向量嵌入，用于语义检索。
|
||||
|
||||
```sql
|
||||
CREATE TABLE skill_embedding (
|
||||
-- 主键
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
|
||||
-- 关联技能
|
||||
skill_id BIGINT NOT NULL,
|
||||
skill_version_id BIGINT,
|
||||
|
||||
-- 嵌入类型和向量
|
||||
embedding_type VARCHAR(32) NOT NULL,
|
||||
embedding vector(1536) NOT NULL,
|
||||
|
||||
-- 元数据
|
||||
metadata JSONB DEFAULT '{}',
|
||||
|
||||
-- 时间戳
|
||||
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
|
||||
-- 外键约束
|
||||
CONSTRAINT fk_skill_embedding_skill
|
||||
FOREIGN KEY (skill_id)
|
||||
REFERENCES skill(id)
|
||||
ON DELETE CASCADE,
|
||||
|
||||
CONSTRAINT fk_skill_embedding_version
|
||||
FOREIGN KEY (skill_version_id)
|
||||
REFERENCES skill_version(id)
|
||||
ON DELETE SET NULL,
|
||||
|
||||
-- 唯一约束
|
||||
CONSTRAINT uk_skill_embedding UNIQUE(skill_id, embedding_type),
|
||||
|
||||
-- 检查约束
|
||||
CONSTRAINT chk_skill_embedding_type
|
||||
CHECK (embedding_type IN ('description', 'example', 'usage', 'combined'))
|
||||
);
|
||||
|
||||
-- 创建索引
|
||||
CREATE INDEX idx_skill_embedding_skill_id ON skill_embedding(skill_id);
|
||||
CREATE INDEX idx_skill_embedding_version_id ON skill_embedding(skill_version_id);
|
||||
|
||||
-- 向量索引（ivfflat 类型，平衡精度和性能）
|
||||
CREATE INDEX idx_skill_embedding_embedding ON skill_embedding
|
||||
USING ivfflat(embedding vector_cosine_ops)
|
||||
WITH (lists = 100);
|
||||
|
||||
-- GIN 索引用于 JSONB 查询
|
||||
CREATE INDEX idx_skill_embedding_metadata ON skill_embedding USING GIN(metadata);
|
||||
```
|
||||
|
||||
**embedding_type 枚举值**：
|
||||
|
||||
| 类型 | 说明 |
|
||||
|------|------|
|
||||
| description | 技能描述向量化 |
|
||||
| example | 示例代码向量化 |
|
||||
| usage | 使用说明向量化 |
|
||||
| combined | 综合向量 |
|
||||
|
||||
### 2.4 intent_mapping（意图映射表）
|
||||
|
||||
存储意图类别及其相关配置。
|
||||
|
||||
```sql
|
||||
CREATE TABLE intent_mapping (
|
||||
-- 主键
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
|
||||
-- 意图信息
|
||||
intent_label VARCHAR(128) UNIQUE NOT NULL,
|
||||
description TEXT,
|
||||
|
||||
-- 关联的技能类别
|
||||
skill_categories TEXT[],
|
||||
|
||||
-- 示例查询（用于意图识别）
|
||||
example_queries TEXT[],
|
||||
|
||||
-- 意图向量
|
||||
embedding vector(1536) NOT NULL,
|
||||
|
||||
-- 优先级和状态
|
||||
priority INTEGER DEFAULT 0,
|
||||
active BOOLEAN DEFAULT true,
|
||||
|
||||
-- 时间戳
|
||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
|
||||
-- 检查约束
|
||||
CONSTRAINT chk_intent_mapping_priority
|
||||
CHECK (priority >= 0)
|
||||
);
|
||||
|
||||
-- 创建索引
|
||||
CREATE INDEX idx_intent_mapping_label ON intent_mapping(intent_label);
|
||||
CREATE INDEX idx_intent_mapping_active ON intent_mapping(active);
|
||||
|
||||
-- 向量索引
|
||||
CREATE INDEX idx_intent_mapping_embedding ON intent_mapping
|
||||
USING ivfflat(embedding vector_cosine_ops)
|
||||
WITH (lists = 50);
|
||||
|
||||
-- GIN 索引用于数组查询
|
||||
CREATE INDEX idx_intent_mapping_categories ON intent_mapping
|
||||
USING GIN(skill_categories);
|
||||
```
|
||||
|
||||
## 3. Java 领域模型
|
||||
|
||||
### 3.1 AgentSession
|
||||
|
||||
```java
|
||||
package com.iflytek.skillhub.domain.agent;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
|
||||
@Entity
|
||||
@Table(name = "agent_session")
|
||||
public class AgentSession {
|
||||
|
||||
@Id
|
||||
@GeneratedValue(strategy = GenerationType.IDENTITY)
|
||||
private Long id;
|
||||
|
||||
@Column(name = "agent_id", nullable = false, length = 128)
|
||||
private String agentId;
|
||||
|
||||
@Column(name = "user_id", nullable = false, length = 128)
|
||||
private String userId;
|
||||
|
||||
@Column(name = "session_id", nullable = false, unique = true, length = 128)
|
||||
private String sessionId;
|
||||
|
||||
@Column(name = "scenario_description", columnDefinition = "TEXT")
|
||||
private String scenarioDescription;
|
||||
|
||||
@Column(name = "user_role", length = 128)
|
||||
private String userRole;
|
||||
|
||||
@JdbcTypeCode(SqlTypes.JSON)
|
||||
@Column(name = "metadata", columnDefinition = "jsonb")
|
||||
private Map<String, Object> metadata = Map.of();
|
||||
|
||||
@Column(name = "started_at", nullable = false)
|
||||
private Instant startedAt = Instant.now();
|
||||
|
||||
@Column(name = "updated_at", nullable = false)
|
||||
private Instant updatedAt = Instant.now();
|
||||
|
||||
@Column(name = "ended_at")
|
||||
private Instant endedAt;
|
||||
|
||||
@PreUpdate
|
||||
protected void onUpdate() {
|
||||
updatedAt = Instant.now();
|
||||
}
|
||||
|
||||
// Getters and Setters...
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 AgentMessage
|
||||
|
||||
```java
|
||||
package com.iflytek.skillhub.domain.agent;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import java.math.BigDecimal;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
|
||||
@Entity
|
||||
@Table(name = "agent_message")
|
||||
public class AgentMessage {
|
||||
|
||||
@Id
|
||||
@GeneratedValue(strategy = GenerationType.IDENTITY)
|
||||
private Long id;
|
||||
|
||||
@Column(name = "session_id", nullable = false)
|
||||
private Long sessionId;
|
||||
|
||||
@Enumerated(EnumType.STRING)
|
||||
@Column(name = "role", nullable = false, length = 32)
|
||||
private MessageRole role;
|
||||
|
||||
@Column(name = "content", nullable = false, columnDefinition = "TEXT")
|
||||
private String content;
|
||||
|
||||
@Column(name = "intent_label", length = 128)
|
||||
private String intentLabel;
|
||||
|
||||
@Column(name = "intent_confidence", precision = 5, scale = 4)
|
||||
private BigDecimal intentConfidence;
|
||||
|
||||
// 注意：pgvector 的 Java 类型需要自定义类型处理器
|
||||
// 这里使用 String 存储序列化后的向量
|
||||
@Column(name = "embedding", columnDefinition = "vector(1536)")
|
||||
private String embedding;
|
||||
|
||||
@JdbcTypeCode(SqlTypes.JSON)
|
||||
@Column(name = "metadata", columnDefinition = "jsonb")
|
||||
private Map<String, Object> metadata = Map.of();
|
||||
|
||||
@Column(name = "created_at", nullable = false)
|
||||
private Instant createdAt = Instant.now();
|
||||
|
||||
public enum MessageRole {
|
||||
USER, ASSISTANT, SYSTEM
|
||||
}
|
||||
|
||||
// Getters and Setters...
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 SkillEmbedding
|
||||
|
||||
```java
|
||||
package com.iflytek.skillhub.domain.agent;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
|
||||
@Entity
|
||||
@Table(name = "skill_embedding")
|
||||
public class SkillEmbedding {
|
||||
|
||||
@Id
|
||||
@GeneratedValue(strategy = GenerationType.IDENTITY)
|
||||
private Long id;
|
||||
|
||||
@Column(name = "skill_id", nullable = false)
|
||||
private Long skillId;
|
||||
|
||||
@Column(name = "skill_version_id")
|
||||
private Long skillVersionId;
|
||||
|
||||
@Enumerated(EnumType.STRING)
|
||||
@Column(name = "embedding_type", nullable = false, length = 32)
|
||||
private EmbeddingType embeddingType;
|
||||
|
||||
@Column(name = "embedding", nullable = false, columnDefinition = "vector(1536)")
|
||||
private String embedding;
|
||||
|
||||
@JdbcTypeCode(SqlTypes.JSON)
|
||||
@Column(name = "metadata", columnDefinition = "jsonb")
|
||||
private Map<String, Object> metadata = Map.of();
|
||||
|
||||
@Column(name = "updated_at", nullable = false)
|
||||
private Instant updatedAt = Instant.now();
|
||||
|
||||
@PreUpdate
|
||||
protected void onUpdate() {
|
||||
updatedAt = Instant.now();
|
||||
}
|
||||
|
||||
public enum EmbeddingType {
|
||||
DESCRIPTION, EXAMPLE, USAGE, COMBINED
|
||||
}
|
||||
|
||||
// Getters and Setters...
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 IntentMapping
|
||||
|
||||
```java
|
||||
package com.iflytek.skillhub.domain.agent;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
@Entity
|
||||
@Table(name = "intent_mapping")
|
||||
public class IntentMapping {
|
||||
|
||||
@Id
|
||||
@GeneratedValue(strategy = GenerationType.IDENTITY)
|
||||
private Long id;
|
||||
|
||||
@Column(name = "intent_label", nullable = false, unique = true, length = 128)
|
||||
private String intentLabel;
|
||||
|
||||
@Column(name = "description", columnDefinition = "TEXT")
|
||||
private String description;
|
||||
|
||||
@ElementCollection
|
||||
@CollectionTable(name = "intent_skill_categories", joinColumns = @JoinColumn(name = "intent_id"))
|
||||
@Column(name = "category")
|
||||
private List<String> skillCategories;
|
||||
|
||||
@ElementCollection
|
||||
@CollectionTable(name = "intent_example_queries", joinColumns = @JoinColumn(name = "intent_id"))
|
||||
@Column(name = "query")
|
||||
private List<String> exampleQueries;
|
||||
|
||||
@Column(name = "embedding", nullable = false, columnDefinition = "vector(1536)")
|
||||
private String embedding;
|
||||
|
||||
@Column(name = "priority", nullable = false)
|
||||
private Integer priority = 0;
|
||||
|
||||
@Column(name = "active", nullable = false)
|
||||
private Boolean active = true;
|
||||
|
||||
@Column(name = "created_at", nullable = false)
|
||||
private Instant createdAt = Instant.now();
|
||||
|
||||
@Column(name = "updated_at", nullable = false)
|
||||
private Instant updatedAt = Instant.now();
|
||||
|
||||
@PreUpdate
|
||||
protected void onUpdate() {
|
||||
updatedAt = Instant.now();
|
||||
}
|
||||
|
||||
// Getters and Setters...
|
||||
}
|
||||
```
|
||||
|
||||
## 4. 初始数据
|
||||
|
||||
### 4.1 预定义意图类别
|
||||
|
||||
```sql
|
||||
-- 插入预定义意图
|
||||
INSERT INTO intent_mapping (intent_label, description, skill_categories, example_queries, priority, active) VALUES
|
||||
('code_generation', '代码生成相关技能', ARRAY['coding', 'development'], ARRAY['写一个函数', '生成代码', '创建类'], 10, true),
|
||||
('data_analysis', '数据分析相关技能', ARRAY['data', 'analysis', 'visualization'], ARRAY['分析数据', '绘制图表', '数据统计'], 10, true),
|
||||
('text_processing', '文本处理相关技能', ARRAY['text', 'nlp', 'language'], ARRAY['处理文本', '提取关键词', '文本分类'], 10, true),
|
||||
('api_integration', 'API 集成相关技能', ARRAY['api', 'integration', 'web'], ARRAY['调用 API', 'HTTP 请求', 'REST API'], 10, true),
|
||||
('file_processing', '文件处理相关技能', ARRAY['file', 'io', 'storage'], ARRAY['读取文件', '写入文件', '文件转换'], 10, true),
|
||||
('database', '数据库相关技能', ARRAY['database', 'sql', 'storage'], ARRAY['查询数据库', 'SQL 语句', '数据库操作'], 10, true),
|
||||
('general', '通用查询', ARRAY['general'], ARRAY['帮助', '能做什么', '功能列表'], 0, true);
|
||||
```
|
||||
|
||||
## 5. 数据迁移
|
||||
|
||||
### 5.1 Flyway 迁移脚本
|
||||
|
||||
```sql
|
||||
-- V__create_agent_tables.sql
|
||||
|
||||
-- 1. 安装 pgvector 扩展
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
|
||||
-- 2. 创建 agent_session 表
|
||||
CREATE TABLE agent_session (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
agent_id VARCHAR(128) NOT NULL,
|
||||
user_id VARCHAR(128) NOT NULL,
|
||||
session_id VARCHAR(128) UNIQUE NOT NULL,
|
||||
scenario_description TEXT,
|
||||
user_role VARCHAR(128),
|
||||
metadata JSONB DEFAULT '{}',
|
||||
started_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
ended_at TIMESTAMP WITH TIME ZONE
|
||||
);
|
||||
|
||||
-- 3. 创建 agent_message 表
|
||||
CREATE TABLE agent_message (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
session_id BIGINT NOT NULL,
|
||||
role VARCHAR(32) NOT NULL CHECK (role IN ('user', 'assistant', 'system')),
|
||||
content TEXT NOT NULL,
|
||||
intent_label VARCHAR(128),
|
||||
intent_confidence DECIMAL(5,4) CHECK (intent_confidence IS NULL OR (intent_confidence >= 0 AND intent_confidence <= 1)),
|
||||
embedding vector(1536),
|
||||
metadata JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT fk_agent_message_session FOREIGN KEY (session_id) REFERENCES agent_session(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- 4. 创建 skill_embedding 表
|
||||
CREATE TABLE skill_embedding (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
skill_id BIGINT NOT NULL,
|
||||
skill_version_id BIGINT,
|
||||
embedding_type VARCHAR(32) NOT NULL CHECK (embedding_type IN ('description', 'example', 'usage', 'combined')),
|
||||
embedding vector(1536) NOT NULL,
|
||||
metadata JSONB DEFAULT '{}',
|
||||
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT uk_skill_embedding UNIQUE(skill_id, embedding_type),
|
||||
CONSTRAINT fk_skill_embedding_skill FOREIGN KEY (skill_id) REFERENCES skill(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- 5. 创建 intent_mapping 表
|
||||
CREATE TABLE intent_mapping (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
intent_label VARCHAR(128) UNIQUE NOT NULL,
|
||||
description TEXT,
|
||||
skill_categories TEXT[],
|
||||
example_queries TEXT[],
|
||||
embedding vector(1536) NOT NULL,
|
||||
priority INTEGER DEFAULT 0 CHECK (priority >= 0),
|
||||
active BOOLEAN DEFAULT true,
|
||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- 6. 创建索引
|
||||
CREATE INDEX idx_agent_session_agent_id ON agent_session(agent_id);
|
||||
CREATE INDEX idx_agent_session_user_id ON agent_session(user_id);
|
||||
CREATE INDEX idx_agent_session_session_id ON agent_session(session_id);
|
||||
CREATE INDEX idx_agent_session_metadata ON agent_session USING GIN(metadata);
|
||||
|
||||
CREATE INDEX idx_agent_message_session_id ON agent_message(session_id);
|
||||
CREATE INDEX idx_agent_message_intent ON agent_message(intent_label);
|
||||
CREATE INDEX idx_agent_message_metadata ON agent_message USING GIN(metadata);
|
||||
|
||||
CREATE INDEX idx_skill_embedding_skill_id ON skill_embedding(skill_id);
|
||||
CREATE INDEX idx_skill_embedding_embedding ON skill_embedding USING ivfflat(embedding vector_cosine_ops) WITH (lists = 100);
|
||||
CREATE INDEX idx_skill_embedding_metadata ON skill_embedding USING GIN(metadata);
|
||||
|
||||
CREATE INDEX idx_intent_mapping_label ON intent_mapping(intent_label);
|
||||
CREATE INDEX idx_intent_mapping_active ON intent_mapping(active);
|
||||
CREATE INDEX idx_intent_mapping_embedding ON intent_mapping USING ivfflat(embedding vector_cosine_ops) WITH (lists = 50);
|
||||
CREATE INDEX idx_intent_mapping_categories ON intent_mapping USING GIN(skill_categories);
|
||||
```
|
||||
445
docs/agent-skills-search/doc-01/04-algorithms.md
Normal file
445
docs/agent-skills-search/doc-01/04-algorithms.md
Normal file
|
|
@ -0,0 +1,445 @@
|
|||
# Agent Skills 智能检索系统 - 算法设计
|
||||
|
||||
## 1. 向量嵌入生成
|
||||
|
||||
### 1.1 嵌入模型配置
|
||||
|
||||
使用 OpenAI `text-embedding-3-small` 模型：
|
||||
- 维度：1536
|
||||
- 输入长度：最多 8191 tokens
|
||||
- 输出格式：浮点数数组
|
||||
|
||||
### 1.2 技能向量生成
|
||||
|
||||
**输入文本构建**：
|
||||
```
|
||||
skill_text = skill_name + "\n" +
|
||||
skill_description + "\n" +
|
||||
skill_keywords + "\n" +
|
||||
usage_examples
|
||||
```
|
||||
|
||||
**生成流程**：
|
||||
```python
|
||||
def generate_skill_embedding(skill):
|
||||
# 构建输入文本
|
||||
text = build_skill_text(skill)
|
||||
|
||||
# 调用 OpenAI API
|
||||
response = openai.Embedding.create(
|
||||
model="text-embedding-3-small",
|
||||
input=text
|
||||
)
|
||||
|
||||
# 返回向量
|
||||
return response['data'][0]['embedding']
|
||||
```
|
||||
|
||||
**存储策略**：
|
||||
- 为每个技能生成 4 种类型的嵌入
|
||||
- `description`: 仅技能描述
|
||||
- `example`: 仅示例代码
|
||||
- `usage`: 仅使用说明
|
||||
- `combined`: 综合所有内容
|
||||
|
||||
## 2. 意图识别算法
|
||||
|
||||
### 2.1 基于向量相似度的意图分类
|
||||
|
||||
**算法流程**：
|
||||
|
||||
```
|
||||
1. 将用户查询转换为向量
|
||||
query_vector = embed(query_text)
|
||||
|
||||
2. 与所有活跃意图的嵌入计算余弦相似度
|
||||
for each intent in active_intents:
|
||||
similarity = cosine_similarity(query_vector, intent.embedding)
|
||||
|
||||
3. 按相似度排序，取 Top-N
|
||||
top_intents = sort_by_similarity(all_similarities)[:N]
|
||||
|
||||
4. 返回结果
|
||||
return top_intents[0] as primary_intent
|
||||
```
|
||||
|
||||
**相似度计算**：
|
||||
```sql
|
||||
-- 使用 pgvector 的 <=> 操作符
|
||||
SELECT
|
||||
intent_label,
|
||||
1 - (embedding <=> :query_vector) as similarity
|
||||
FROM intent_mapping
|
||||
WHERE active = true
|
||||
ORDER BY embedding <=> :query_vector
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
**置信度处理**：
|
||||
- 最高相似度 < 0.3：返回默认意图 "general"
|
||||
- 最高相似度 >= 0.3：返回识别的意图
|
||||
- 返回 Top-3 备选意图
|
||||
|
||||
### 2.2 意图库更新
|
||||
|
||||
当添加新意图时：
|
||||
1. 收集该意图的示例查询
|
||||
2. 生成意图嵌入向量（使用示例查询的平均向量）
|
||||
3. 插入 `intent_mapping` 表
|
||||
|
||||
## 3. 查询向量构建
|
||||
|
||||
### 3.1 多维度向量融合
|
||||
|
||||
**公式**：
|
||||
```
|
||||
query_vector = normalize(
|
||||
w1 * embed(query_text) +
|
||||
w2 * embed(conversation_context) +
|
||||
w3 * embed(scenario_description) +
|
||||
w4 * embed(user_role) +
|
||||
w5 * intent_embedding
|
||||
)
|
||||
```
|
||||
|
||||
**权重配置**：
|
||||
|
||||
| 权重 | 值 | 说明 |
|
||||
|------|-----|------|
|
||||
| w1 | 0.4 | 查询文本（最重要） |
|
||||
| w2 | 0.2 | 对话上下文 |
|
||||
| w3 | 0.15 | 场景描述 |
|
||||
| w4 | 0.1 | 用户角色 |
|
||||
| w5 | 0.15 | 意图向量 |
|
||||
|
||||
**归一化**：
|
||||
```python
|
||||
def normalize(vector):
|
||||
norm = np.linalg.norm(vector)
|
||||
if norm == 0:
|
||||
return vector
|
||||
return vector / norm
|
||||
```
|
||||
|
||||
### 3.2 对话上下文编码
|
||||
|
||||
**策略**：使用最近 N 轮对话的摘要
|
||||
|
||||
```python
|
||||
def encode_conversation(messages, max_turns=5):
|
||||
# 取最近 N 轮对话
|
||||
recent_messages = messages[-2*max_turns:]
|
||||
|
||||
# 构建上下文文本
|
||||
context_text = "\n".join([
|
||||
f"{msg.role}: {msg.content}"
|
||||
for msg in recent_messages
|
||||
])
|
||||
|
||||
# 生成嵌入
|
||||
return embed(context_text)
|
||||
```
|
||||
|
||||
## 4. 混合检索算法
|
||||
|
||||
### 4.1 向量检索（Vector Search）
|
||||
|
||||
**SQL 查询**：
|
||||
```sql
|
||||
SELECT
|
||||
se.skill_id,
|
||||
s.name,
|
||||
s.summary,
|
||||
1 - (se.embedding <=> :query_vector) as similarity
|
||||
FROM skill_embedding se
|
||||
JOIN skill s ON s.id = se.skill_id
|
||||
WHERE se.embedding_type = 'description'
|
||||
AND s.status = 'ACTIVE'
|
||||
ORDER BY se.embedding <=> :query_vector
|
||||
LIMIT 200;
|
||||
```
|
||||
|
||||
**向量索引**：
|
||||
```sql
|
||||
CREATE INDEX idx_skill_embedding_embedding ON skill_embedding
|
||||
USING ivfflat(embedding vector_cosine_ops)
|
||||
WITH (lists = 100);
|
||||
```
|
||||
|
||||
### 4.2 意图过滤（Intent Filter）
|
||||
|
||||
**过滤逻辑**：
|
||||
```sql
|
||||
SELECT vc.skill_id, vc.similarity
|
||||
FROM vector_candidates vc
|
||||
JOIN skill s ON s.id = vc.skill_id
|
||||
WHERE s.category = ANY(:intent_categories)
|
||||
OR vc.similarity > 0.7; -- 高相似度可跨意图
|
||||
```
|
||||
|
||||
### 4.3 标签匹配（Tag Matching）
|
||||
|
||||
**精确匹配**：
|
||||
```sql
|
||||
SELECT s.id, COUNT(*) as match_count
|
||||
FROM skill s
|
||||
JOIN skill_label sl ON sl.skill_id = s.id
|
||||
JOIN label_definition ld ON ld.id = sl.label_id
|
||||
WHERE s.id IN (:candidate_ids)
|
||||
AND ld.slug = ANY(:user_tags)
|
||||
GROUP BY s.id;
|
||||
```
|
||||
|
||||
**模糊匹配**：
|
||||
- 计算标签向量与查询向量的相似度
|
||||
- 相似度 > 0.6 的标签视为匹配
|
||||
|
||||
### 4.4 权限过滤（Permission Filter）
|
||||
|
||||
```sql
|
||||
-- 公开技能
|
||||
visibility = 'PUBLIC'
|
||||
|
||||
-- 用户私有技能
|
||||
visibility = 'PRIVATE' AND (owner_id = :user_id OR namespace_id IN (:admin_namespaces))
|
||||
|
||||
-- 命名空间内可见
|
||||
visibility = 'NAMESPACE_ONLY' AND namespace_id IN (:member_namespaces)
|
||||
```
|
||||
|
||||
### 4.5 综合检索 SQL
|
||||
|
||||
```sql
|
||||
WITH vector_candidates AS (
|
||||
SELECT
|
||||
se.skill_id,
|
||||
1 - (se.embedding <=> :query_vector) as vector_score
|
||||
FROM skill_embedding se
|
||||
JOIN skill s ON s.id = se.skill_id
|
||||
WHERE se.embedding_type = 'description'
|
||||
AND s.status = 'ACTIVE'
|
||||
ORDER BY se.embedding <=> :query_vector
|
||||
LIMIT 200
|
||||
),
|
||||
intent_filtered AS (
|
||||
SELECT
|
||||
vc.skill_id,
|
||||
vc.vector_score
|
||||
FROM vector_candidates vc
|
||||
JOIN skill s ON s.id = vc.skill_id
|
||||
WHERE s.category = ANY(:intent_categories)
|
||||
OR vc.vector_score > 0.7
|
||||
),
|
||||
permission_filtered AS (
|
||||
SELECT
|
||||
if.skill_id,
|
||||
if.vector_score
|
||||
FROM intent_filtered if
|
||||
JOIN skill s ON s.id = if.skill_id
|
||||
WHERE (s.visibility = 'PUBLIC')
|
||||
OR (s.visibility = 'NAMESPACE_ONLY' AND s.namespace_id = ANY(:member_namespace_ids))
|
||||
OR (s.visibility = 'PRIVATE' AND (s.owner_id = :user_id OR s.namespace_id = ANY(:admin_namespace_ids)))
|
||||
),
|
||||
tag_matched AS (
|
||||
SELECT
|
||||
pf.skill_id,
|
||||
pf.vector_score,
|
||||
COUNT(*) FILTER (WHERE ld.slug = ANY(:user_tags)) as tag_count
|
||||
FROM permission_filtered pf
|
||||
LEFT JOIN skill_label sl ON sl.skill_id = pf.skill_id
|
||||
LEFT JOIN label_definition ld ON ld.id = sl.label_id
|
||||
GROUP BY pf.skill_id, pf.vector_score
|
||||
)
|
||||
SELECT
|
||||
tm.skill_id,
|
||||
s.name,
|
||||
s.summary,
|
||||
s.description,
|
||||
tm.vector_score,
|
||||
tm.tag_count,
|
||||
s.download_count,
|
||||
s.rating_avg
|
||||
FROM tag_matched tm
|
||||
JOIN skill s ON s.id = tm.skill_id
|
||||
ORDER BY tm.vector_score DESC
|
||||
LIMIT :limit;
|
||||
```
|
||||
|
||||
## 5. 重排序算法
|
||||
|
||||
### 5.1 多维度评分
|
||||
|
||||
**评分公式**：
|
||||
```
|
||||
final_score = α * normalize(vector_score)
|
||||
+ β * normalize(intent_match_score)
|
||||
+ γ * normalize(tag_match_score)
|
||||
+ δ * normalize(popularity_score)
|
||||
+ ε * normalize(rating_score)
|
||||
```
|
||||
|
||||
**权重配置**：
|
||||
|
||||
| 权重 | 值 | 说明 |
|
||||
|------|-----|------|
|
||||
| α | 0.4 | 向量相似度 |
|
||||
| β | 0.25 | 意图匹配度 |
|
||||
| γ | 0.2 | 标签匹配度 |
|
||||
| δ | 0.1 | 流行度（下载次数） |
|
||||
| ε | 0.05 | 用户评分 |
|
||||
|
||||
### 5.2 各维度计算
|
||||
|
||||
**1. 向量相似度**：
|
||||
```
|
||||
vector_score = 1 - (skill_embedding <=> query_embedding)
|
||||
范围：[0, 1]
|
||||
```
|
||||
|
||||
**2. 意图匹配度**：
|
||||
```
|
||||
intent_match_score = 1.0 # 如果技能类别匹配意图
|
||||
0.7 # 如果技能类别在意图的相关类别中
|
||||
0.3 # 其他
|
||||
```
|
||||
|
||||
**3. 标签匹配度**：
|
||||
```
|
||||
tag_match_score = min(tag_count / max_user_tags, 1.0)
|
||||
范围：[0, 1]
|
||||
```
|
||||
|
||||
**4. 流行度**：
|
||||
```
|
||||
popularity_score = download_count / max_download_count
|
||||
使用对数缩放：log(1 + download_count) / log(1 + max_download_count)
|
||||
```
|
||||
|
||||
**5. 用户评分**：
|
||||
```
|
||||
rating_score = rating_avg / 5.0
|
||||
范围：[0, 1]
|
||||
```
|
||||
|
||||
### 5.3 重排序实现
|
||||
|
||||
```java
|
||||
public class SkillReranker {
|
||||
|
||||
private final double wVector = 0.4;
|
||||
private final double wIntent = 0.25;
|
||||
private final double wTag = 0.2;
|
||||
private final double wPopularity = 0.1;
|
||||
private final double wRating = 0.05;
|
||||
|
||||
public List<SkillMatchResult> rerank(
|
||||
List<SkillCandidate> candidates,
|
||||
IntentClassificationResult intent,
|
||||
Set<String> userTags,
|
||||
double maxDownloadCount
|
||||
) {
|
||||
// 计算每个候选技能的综合分数
|
||||
return candidates.stream()
|
||||
.map(candidate -> calculateScore(candidate, intent, userTags, maxDownloadCount))
|
||||
.sorted(Comparator.comparingDouble(SkillMatchResult::finalScore).reversed())
|
||||
.collect(Collectors.toList());
|
||||
}
|
||||
|
||||
private SkillMatchResult calculateScore(
|
||||
SkillCandidate candidate,
|
||||
IntentClassificationResult intent,
|
||||
Set<String> userTags,
|
||||
double maxDownloadCount
|
||||
) {
|
||||
double vectorScore = candidate.vectorSimilarity();
|
||||
double intentScore = calculateIntentScore(candidate, intent);
|
||||
double tagScore = calculateTagScore(candidate, userTags);
|
||||
double popularityScore = Math.log(1 + candidate.downloadCount())
|
||||
/ Math.log(1 + maxDownloadCount);
|
||||
double ratingScore = candidate.ratingAvg() / 5.0;
|
||||
|
||||
double finalScore = wVector * vectorScore
|
||||
+ wIntent * intentScore
|
||||
+ wTag * tagScore
|
||||
+ wPopularity * popularityScore
|
||||
+ wRating * ratingScore;
|
||||
|
||||
return new SkillMatchResult(
|
||||
candidate.skillId(),
|
||||
candidate.name(),
|
||||
candidate.summary(),
|
||||
vectorScore,
|
||||
intentScore,
|
||||
tagScore,
|
||||
finalScore,
|
||||
buildMatchReason(vectorScore, intentScore, tagScore)
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 6. 缓存算法
|
||||
|
||||
### 6.1 查询结果缓存
|
||||
|
||||
**缓存键**：
|
||||
```
|
||||
cache_key = "agent_search:" +
|
||||
hash(query_text + ":" +
|
||||
intent_label + ":" +
|
||||
sorted(user_tags) + ":" +
|
||||
user_role + ":" +
|
||||
scenario_description)
|
||||
```
|
||||
|
||||
**TTL**：5 分钟
|
||||
|
||||
### 6.2 嵌入向量缓存
|
||||
|
||||
**缓存策略**：
|
||||
- 技能嵌入：永久缓存（版本更新时失效）
|
||||
- 意图嵌入：永久缓存（意图更新时失效）
|
||||
- 查询嵌入：TTL 1 小时
|
||||
|
||||
## 7. 降级策略
|
||||
|
||||
| 场景 | 降级方案 |
|
||||
|------|----------|
|
||||
| OpenAI API 调用失败 | 使用本地哈希向量 |
|
||||
| 向量索引未就绪 | 使用全文搜索 |
|
||||
| 意图识别失败 | 使用默认意图 |
|
||||
| 响应超时 | 返回部分结果 |
|
||||
|
||||
## 8. 性能优化
|
||||
|
||||
### 8.1 向量索引调优
|
||||
|
||||
```sql
|
||||
-- ivfflat 索引参数
|
||||
-- lists = sqrt(行数)
|
||||
-- 1000 行：lists = 32
|
||||
-- 10000 行：lists = 100
|
||||
-- 100000 行：lists = 316
|
||||
```
|
||||
|
||||
### 8.2 查询优化
|
||||
|
||||
1. **限制候选集**：向量检索先返回 Top-200，再过滤
|
||||
2. **异步处理**：非阻塞式嵌入生成
|
||||
3. **批量查询**：减少数据库往返
|
||||
|
||||
### 8.3 批量向量化
|
||||
|
||||
```python
|
||||
def batch_embed(texts, batch_size=100):
|
||||
"""批量生成嵌入向量"""
|
||||
all_embeddings = []
|
||||
for i in range(0, len(texts), batch_size):
|
||||
batch = texts[i:i + batch_size]
|
||||
response = openai.Embedding.create(
|
||||
model="text-embedding-3-small",
|
||||
input=batch
|
||||
)
|
||||
all_embeddings.extend([item['embedding'] for item in response['data']])
|
||||
return all_embeddings
|
||||
```
|
||||
255
docs/agent-skills-search/doc-01/05-flow-diagrams.md
Normal file
255
docs/agent-skills-search/doc-01/05-flow-diagrams.md
Normal file
|
|
@ -0,0 +1,255 @@
|
|||
# Agent Skills 智能检索系统 - 流程图
|
||||
|
||||
## 1. 主检索流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Agent 发起技能搜索请求] --> B[Context Builder<br/>收集上下文]
|
||||
B --> C{检查 Session}
|
||||
C -->|存在| D[从 Redis/PG 加载历史对话]
|
||||
C -->|不存在| E[创建新 Session]
|
||||
D --> F[Intent Classifier<br/>意图识别]
|
||||
E --> F
|
||||
F --> G[Query Embedder<br/>构建查询向量]
|
||||
G --> H[Hybrid Search Service<br/>混合检索]
|
||||
H --> I[向量检索<br/>pgvector 相似度计算]
|
||||
I --> J[意图过滤]
|
||||
J --> K[标签匹配]
|
||||
K --> L[权限过滤]
|
||||
L --> M[Skill Reranker<br/>多维度重排序]
|
||||
M --> N[Result Enricher<br/>结果丰富化]
|
||||
N --> O[异步保存对话记录]
|
||||
O --> P[返回匹配技能列表]
|
||||
|
||||
style A fill:#e1f5ff
|
||||
style P fill:#c8e6c9
|
||||
style F fill:#fff9c4
|
||||
style H fill:#fff9c4
|
||||
style M fill:#fff9c4
|
||||
```
|
||||
|
||||
## 2. 意图识别流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[接收用户查询] --> B[文本预处理<br/>清理、分词]
|
||||
B --> C[调用 OpenAI Embedding API<br/>生成查询向量]
|
||||
C --> D[查询活跃意图列表]
|
||||
D --> E[遍历意图嵌入]
|
||||
E --> F[计算余弦相似度]
|
||||
F --> G{是否还有意图?}
|
||||
G -->|是| E
|
||||
G -->|否| H[按相似度降序排序]
|
||||
H --> I{最高相似度 >= 0.3?}
|
||||
I -->|是| J[返回 Top-N 意图<br/>包含置信度]
|
||||
I -->|否| K[返回默认意图 'general']
|
||||
J --> L[输出意图分类结果]
|
||||
K --> L
|
||||
|
||||
style C fill:#ffebee
|
||||
style F fill:#e3f2fd
|
||||
style J fill:#c8e6c9
|
||||
```
|
||||
|
||||
## 3. 向量检索流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[接收查询向量] --> B[构建 pgvector 查询]
|
||||
B --> C[执行 SQL 查询<br/>使用 <=> 操作符]
|
||||
C --> D{向量索引可用?}
|
||||
D -->|是| E[使用 ivfflat 索引<br/>快速检索]
|
||||
D -->|否| F[暴力扫描计算]
|
||||
E --> G[返回 Top-K 候选]
|
||||
F --> G
|
||||
G --> H[过滤非活跃技能]
|
||||
H --> I[应用可见性过滤]
|
||||
I --> J[返回候选技能列表]
|
||||
|
||||
style C fill:#e3f2fd
|
||||
style E fill:#c8e6c9
|
||||
style F fill:#fff3e0
|
||||
```
|
||||
|
||||
## 4. 混合检索详细流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[开始混合检索] --> B[阶段 1: 向量检索]
|
||||
B --> B1[查询 pgvector]
|
||||
B1 --> B2[获取 Top-200 候选]
|
||||
B2 --> C[阶段 2: 意图过滤]
|
||||
C --> C1{技能类别匹配意图?}
|
||||
C1 -->|是| C2[保留候选]
|
||||
C1 -->|否| C3{向量相似度 > 0.7?}
|
||||
C3 -->|是| C2
|
||||
C3 -->|否| C4[丢弃候选]
|
||||
C2 --> D[阶段 3: 标签匹配]
|
||||
D --> D1[统计匹配标签数]
|
||||
D1 --> E[阶段 4: 权限过滤]
|
||||
E --> E1{用户有权限访问?}
|
||||
E1 -->|是| E2[保留候选]
|
||||
E1 -->|否| C4
|
||||
E2 --> F[返回过滤后的候选]
|
||||
|
||||
style B fill:#e3f2fd
|
||||
style C fill:#fff9c4
|
||||
style D fill:#fff9c4
|
||||
style E fill:#fff9c4
|
||||
style F fill:#c8e6c9
|
||||
style C4 fill:#ffcdd2
|
||||
```
|
||||
|
||||
## 5. 重排序流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[接收候选技能列表] --> B[计算各维度分数]
|
||||
B --> B1[向量相似度: 40%]
|
||||
B --> B2[意图匹配度: 25%]
|
||||
B --> B3[标签匹配度: 20%]
|
||||
B --> B4[流行度: 10%]
|
||||
B --> B5[用户评分: 5%]
|
||||
B1 --> C[计算综合分数]
|
||||
B2 --> C
|
||||
B3 --> C
|
||||
B4 --> C
|
||||
B5 --> C
|
||||
C --> D[按综合分数降序排序]
|
||||
D --> E[取 Top-N 结果]
|
||||
E --> F[生成分数明细]
|
||||
F --> G[生成匹配原因]
|
||||
G --> H[返回重排序结果]
|
||||
|
||||
style B fill:#e1f5ff
|
||||
style C fill:#fff9c4
|
||||
style H fill:#c8e6c9
|
||||
```
|
||||
|
||||
## 6. 会话管理流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[接收消息] --> B{Session ID 存在?}
|
||||
B -->|是| C[从 Redis 获取 Session]
|
||||
B -->|否| D[创建新 Session]
|
||||
C --> E{Session 有效?}
|
||||
D --> F[保存到 Redis]
|
||||
E -->|是| G[追加消息到历史]
|
||||
E -->|否| F
|
||||
F --> G
|
||||
G --> H[更新 updated_at]
|
||||
H --> I[异步保存到 PostgreSQL]
|
||||
I --> J[处理业务逻辑]
|
||||
J --> K[返回响应]
|
||||
|
||||
style D fill:#fff9c4
|
||||
style F fill:#e3f2fd
|
||||
style I fill:#e1f5ff
|
||||
style K fill:#c8e6c9
|
||||
```
|
||||
|
||||
## 7. 技能向量生成流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[技能版本发布] --> B[提取技能元数据]
|
||||
B --> C[构建嵌入文本]
|
||||
C --> C1[技能名称]
|
||||
C --> C2[技能描述]
|
||||
C --> C3[关键词]
|
||||
C --> C4[使用示例]
|
||||
C1 --> D[调用 OpenAI Embedding API]
|
||||
C2 --> D
|
||||
C3 --> D
|
||||
C4 --> D
|
||||
D --> E[接收向量响应]
|
||||
E --> F[存储到 skill_embedding 表]
|
||||
F --> G[创建 4 种类型嵌入]
|
||||
G --> G1[description]
|
||||
G --> G2[example]
|
||||
G --> G3[usage]
|
||||
G --> G4[combined]
|
||||
G1 --> H[更新向量索引]
|
||||
G2 --> H
|
||||
G3 --> H
|
||||
G4 --> H
|
||||
H --> I[完成]
|
||||
|
||||
style D fill:#ffebee
|
||||
style F fill:#e3f2fd
|
||||
style H fill:#c8e6c9
|
||||
```
|
||||
|
||||
## 8. 错误处理流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[执行操作] --> B{成功?}
|
||||
B -->|是| C[返回结果]
|
||||
B -->|否| D{错误类型}
|
||||
D -->|OpenAI API 失败| E[降级到哈希向量]
|
||||
D -->|向量索引不可用| F[使用全文搜索]
|
||||
D -->|意图识别失败| G[使用默认意图]
|
||||
D -->|数据库连接失败| H[返回缓存结果或错误]
|
||||
D -->|超时| I[返回部分结果]
|
||||
E --> J[记录降级日志]
|
||||
F --> J
|
||||
G --> J
|
||||
H --> J
|
||||
I --> J
|
||||
J --> C
|
||||
|
||||
style E fill:#fff3e0
|
||||
style F fill:#fff3e0
|
||||
style G fill:#fff3e0
|
||||
style H fill:#ffcdd2
|
||||
style I fill:#fff3e0
|
||||
```
|
||||
|
||||
## 9. 缓存查询流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[接收查询请求] --> B[生成缓存键]
|
||||
B --> C{缓存命中?}
|
||||
C -->|是| D[返回缓存结果]
|
||||
C -->|否| E[执行完整检索]
|
||||
E --> F[保存到缓存]
|
||||
F --> G[返回结果]
|
||||
D --> H{缓存 TTL 过期?}
|
||||
H -->|是| I[异步刷新缓存]
|
||||
H -->|否| J[完成]
|
||||
I --> J
|
||||
G --> J
|
||||
|
||||
style C fill:#e1f5ff
|
||||
style D fill:#c8e6c9
|
||||
style I fill:#fff9c4
|
||||
```
|
||||
|
||||
## 10. 批量向量更新流程
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[定时任务触发] --> B[查询待更新技能]
|
||||
B --> C{有待更新技能?}
|
||||
C -->|否| D[结束]
|
||||
C -->|是| E[批量获取技能元数据]
|
||||
E --> F[批量调用 Embedding API<br/>batch_size=100]
|
||||
F --> G{API 调用成功?}
|
||||
G -->|是| H[批量更新数据库]
|
||||
G -->|否| I[记录失败日志<br/>重试队列]
|
||||
H --> J[更新向量索引]
|
||||
I --> K{重试次数 < 3?}
|
||||
K -->|是| F
|
||||
K -->|否| L[标记为失败]
|
||||
J --> B
|
||||
L --> B
|
||||
D --> M[完成]
|
||||
|
||||
style F fill:#e3f2fd
|
||||
style H fill:#e3f2fd
|
||||
style I fill:#fff3e0
|
||||
style L fill:#ffcdd2
|
||||
```
|
||||
526
docs/agent-skills-search/doc-01/06-architecture-diagrams.md
Normal file
526
docs/agent-skills-search/doc-01/06-architecture-diagrams.md
Normal file
|
|
@ -0,0 +1,526 @@
|
|||
# Agent Skills 智能检索系统 - 架构图
|
||||
|
||||
## 1. 系统整体架构
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Client Layer"
|
||||
A[Agent Client]
|
||||
end
|
||||
|
||||
subgraph "API Gateway"
|
||||
B[REST API]
|
||||
C[Authentication]
|
||||
end
|
||||
|
||||
subgraph "Application Layer"
|
||||
D[Agent Skill Search Controller]
|
||||
E[Agent Session Controller]
|
||||
end
|
||||
|
||||
subgraph "Service Layer"
|
||||
F[AgentContextBuilder]
|
||||
G[IntentClassifier]
|
||||
H[QueryEmbedder]
|
||||
I[HybridSkillSearchService]
|
||||
J[SkillReranker]
|
||||
K[AgentSessionService]
|
||||
end
|
||||
|
||||
subgraph "Domain Layer"
|
||||
L[AgentSession]
|
||||
M[AgentMessage]
|
||||
N[SkillEmbedding]
|
||||
O[IntentMapping]
|
||||
end
|
||||
|
||||
subgraph "Infrastructure Layer"
|
||||
P[(PostgreSQL<br/>+ pgvector)]
|
||||
Q[(Redis Cache)]
|
||||
R[OpenAI Embedding API]
|
||||
end
|
||||
|
||||
A -->|HTTP/JSON| B
|
||||
B --> C
|
||||
C --> D
|
||||
C --> E
|
||||
D --> F
|
||||
D --> I
|
||||
E --> K
|
||||
F --> K
|
||||
F --> G
|
||||
G --> H
|
||||
H --> R
|
||||
G --> O
|
||||
I --> H
|
||||
I --> N
|
||||
I --> J
|
||||
J --> L
|
||||
K --> L
|
||||
K --> M
|
||||
L --> P
|
||||
M --> P
|
||||
N --> P
|
||||
O --> P
|
||||
K --> Q
|
||||
|
||||
style A fill:#e1f5ff
|
||||
style R fill:#ffebee
|
||||
style P fill:#c8e6c9
|
||||
style Q fill:#fff9c4
|
||||
```
|
||||
|
||||
## 2. 数据流架构
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph "Input"
|
||||
A1[Query Text]
|
||||
A2[Conversation History]
|
||||
A3[Scenario Description]
|
||||
A4[User Role]
|
||||
end
|
||||
|
||||
subgraph "Processing"
|
||||
B1[Context Builder]
|
||||
B2[Intent Classifier]
|
||||
B3[Query Embedder]
|
||||
B4[Hybrid Search]
|
||||
B5[Reranker]
|
||||
end
|
||||
|
||||
subgraph "Storage"
|
||||
C1[(PostgreSQL)]
|
||||
C2[(Redis)]
|
||||
C3[OpenAI API]
|
||||
end
|
||||
|
||||
subgraph "Output"
|
||||
D1[Matched Skills]
|
||||
D2[Match Reasons]
|
||||
D3[Confidence Scores]
|
||||
end
|
||||
|
||||
A1 --> B1
|
||||
A2 --> B1
|
||||
A3 --> B1
|
||||
A4 --> B1
|
||||
B1 --> B2
|
||||
B1 --> B3
|
||||
B2 --> C3
|
||||
B3 --> C3
|
||||
B2 --> B4
|
||||
B3 --> B4
|
||||
B4 --> C1
|
||||
B4 --> B5
|
||||
B5 --> D1
|
||||
B5 --> D2
|
||||
B5 --> D3
|
||||
B1 --> C2
|
||||
B2 --> C2
|
||||
|
||||
style A1 fill:#e1f5ff
|
||||
style C3 fill:#ffebee
|
||||
style D1 fill:#c8e6c9
|
||||
```
|
||||
|
||||
## 3. 模块依赖关系
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[AgentSkillSearchController] --> B[AgentContextBuilder]
|
||||
A --> C[HybridSkillSearchService]
|
||||
|
||||
B --> D[AgentSessionService]
|
||||
D --> E[AgentSessionRepository]
|
||||
E --> F[(PostgreSQL)]
|
||||
|
||||
C --> G[QueryEmbedder]
|
||||
C --> H[IntentClassifier]
|
||||
C --> I[SkillReranker]
|
||||
C --> J[SkillEmbeddingRepository]
|
||||
|
||||
G --> K[OpenAIEmbeddingClient]
|
||||
K --> L[OpenAI API]
|
||||
|
||||
H --> G
|
||||
H --> M[IntentMappingRepository]
|
||||
M --> F
|
||||
|
||||
I --> N[SkillRepository]
|
||||
I --> O[LabelRepository]
|
||||
N --> F
|
||||
O --> F
|
||||
|
||||
J --> F
|
||||
|
||||
D --> P[RedisTemplate]
|
||||
P --> Q[(Redis)]
|
||||
|
||||
style A fill:#e1f5ff
|
||||
style L fill:#ffebee
|
||||
style F fill:#c8e6c9
|
||||
style Q fill:#fff9c4
|
||||
```
|
||||
|
||||
## 4. 数据库架构
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
agent_session ||--o{ agent_message : contains
|
||||
skill ||--o{ skill_embedding : has
|
||||
skill ||--o{ skill_label : tagged_with
|
||||
label_definition ||--o{ skill_label : used_in
|
||||
|
||||
agent_session {
|
||||
bigint id PK
|
||||
varchar agent_id
|
||||
varchar user_id FK
|
||||
varchar session_id UK
|
||||
text scenario_description
|
||||
varchar user_role
|
||||
jsonb metadata
|
||||
timestamptz started_at
|
||||
timestamptz updated_at
|
||||
timestamptz ended_at
|
||||
}
|
||||
|
||||
agent_message {
|
||||
bigint id PK
|
||||
bigint session_id FK
|
||||
varchar role
|
||||
text content
|
||||
varchar intent_label
|
||||
decimal intent_confidence
|
||||
vector embedding
|
||||
jsonb metadata
|
||||
timestamptz created_at
|
||||
}
|
||||
|
||||
skill {
|
||||
bigint id PK
|
||||
bigint namespace_id FK
|
||||
varchar slug
|
||||
varchar display_name
|
||||
text summary
|
||||
varchar owner_id
|
||||
varchar visibility
|
||||
varchar status
|
||||
bigint latest_version_id
|
||||
bigint download_count
|
||||
int star_count
|
||||
decimal rating_avg
|
||||
int rating_count
|
||||
}
|
||||
|
||||
skill_embedding {
|
||||
bigint id PK
|
||||
bigint skill_id FK
|
||||
bigint skill_version_id FK
|
||||
varchar embedding_type
|
||||
vector embedding
|
||||
jsonb metadata
|
||||
timestamptz updated_at
|
||||
}
|
||||
|
||||
skill_label {
|
||||
bigint id PK
|
||||
bigint skill_id FK
|
||||
bigint label_id FK
|
||||
}
|
||||
|
||||
label_definition {
|
||||
bigint id PK
|
||||
varchar name
|
||||
varchar slug UK
|
||||
text description
|
||||
varchar color
|
||||
int priority
|
||||
}
|
||||
|
||||
intent_mapping {
|
||||
bigint id PK
|
||||
varchar intent_label UK
|
||||
text description
|
||||
text[] skill_categories
|
||||
text[] example_queries
|
||||
vector embedding
|
||||
int priority
|
||||
boolean active
|
||||
timestamptz created_at
|
||||
timestamptz updated_at
|
||||
}
|
||||
```
|
||||
|
||||
## 5. 部署架构
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Load Balancer"
|
||||
LB[Nginx / ALB]
|
||||
end
|
||||
|
||||
subgraph "Application Servers"
|
||||
APP1[SkillHub App 1]
|
||||
APP2[SkillHub App 2]
|
||||
APP3[SkillHub App N]
|
||||
end
|
||||
|
||||
subgraph "Data Layer"
|
||||
PG[PostgreSQL<br/>Primary]
|
||||
PG_REPLICA[PostgreSQL<br/>Read Replica]
|
||||
REDIS[Redis Cluster]
|
||||
end
|
||||
|
||||
subgraph "External Services"
|
||||
OPENAI[OpenAI API]
|
||||
end
|
||||
|
||||
LB --> APP1
|
||||
LB --> APP2
|
||||
LB --> APP3
|
||||
|
||||
APP1 --> PG
|
||||
APP1 --> PG_REPLICA
|
||||
APP1 --> REDIS
|
||||
APP1 --> OPENAI
|
||||
|
||||
APP2 --> PG
|
||||
APP2 --> PG_REPLICA
|
||||
APP2 --> REDIS
|
||||
APP2 --> OPENAI
|
||||
|
||||
APP3 --> PG
|
||||
APP3 --> PG_REPLICA
|
||||
APP3 --> REDIS
|
||||
APP3 --> OPENAI
|
||||
|
||||
PG -.->|Replication| PG_REPLICA
|
||||
|
||||
style LB fill:#e1f5ff
|
||||
style PG fill:#c8e6c9
|
||||
style REDIS fill:#fff9c4
|
||||
style OPENAI fill:#ffebee
|
||||
```
|
||||
|
||||
## 6. 缓存架构
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph "Application"
|
||||
A[Service Layer]
|
||||
end
|
||||
|
||||
subgraph "Cache Layer - Redis"
|
||||
B1[Session Cache<br/>TTL: 1h]
|
||||
B2[Query Result Cache<br/>TTL: 5min]
|
||||
B3[Embedding Cache<br/>TTL: 1h/Permanent]
|
||||
end
|
||||
|
||||
subgraph "Database"
|
||||
C[(PostgreSQL)]
|
||||
end
|
||||
|
||||
A -->|Get| B1
|
||||
A -->|Get| B2
|
||||
A -->|Get| B3
|
||||
|
||||
B1 -.->|Miss| C
|
||||
B2 -.->|Miss| C
|
||||
B3 -.->|Miss| C
|
||||
|
||||
C -->|Load| B1
|
||||
C -->|Load| B2
|
||||
C -->|Load| B3
|
||||
|
||||
A -->|Update| B1
|
||||
A -->|Update| B2
|
||||
A -->|Update| B3
|
||||
|
||||
B1 -.->|Write Through| C
|
||||
B2 -.->|Write Back| C
|
||||
B3 -.->|Write Through| C
|
||||
|
||||
style A fill:#e1f5ff
|
||||
style B1 fill:#fff9c4
|
||||
style B2 fill:#fff9c4
|
||||
style B3 fill:#fff9c4
|
||||
style C fill:#c8e6c9
|
||||
```
|
||||
|
||||
## 7. 搜索流程架构
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Query Processing"
|
||||
A[User Query] --> B[Context Builder]
|
||||
B --> C[Intent Classifier]
|
||||
C --> D[Query Embedder]
|
||||
end
|
||||
|
||||
subgraph "Search Pipeline"
|
||||
D --> E[Vector Search]
|
||||
E --> F[Intent Filter]
|
||||
F --> G[Tag Matcher]
|
||||
G --> H[Permission Filter]
|
||||
H --> I[Reranker]
|
||||
end
|
||||
|
||||
subgraph "Data Sources"
|
||||
J[(Skill Embeddings)]
|
||||
K[(Intent Mapping)]
|
||||
L[(Skill Metadata)]
|
||||
M[(User Permissions)]
|
||||
end
|
||||
|
||||
subgraph "Result"
|
||||
I --> N[Ranked Skills]
|
||||
N --> O[Response Builder]
|
||||
end
|
||||
|
||||
E --> J
|
||||
F --> K
|
||||
G --> L
|
||||
H --> M
|
||||
I --> L
|
||||
|
||||
style A fill:#e1f5ff
|
||||
style J fill:#c8e6c9
|
||||
style K fill:#c8e6c9
|
||||
style L fill:#c8e6c9
|
||||
style M fill:#c8e6c9
|
||||
style N fill:#c8e6c9
|
||||
```
|
||||
|
||||
## 8. 安全架构
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Client"
|
||||
A[Agent Client]
|
||||
end
|
||||
|
||||
subgraph "Security Layer"
|
||||
B[Authentication<br/>JWT/OAuth2]
|
||||
C[Authorization<br/>RBAC]
|
||||
D[Rate Limiting]
|
||||
end
|
||||
|
||||
subgraph "Application"
|
||||
E[API Controller]
|
||||
end
|
||||
|
||||
subgraph "Data Protection"
|
||||
F[Input Validation]
|
||||
G[SQL Injection Prevention]
|
||||
H[Data Encryption]
|
||||
end
|
||||
|
||||
subgraph "Infrastructure"
|
||||
I[PostgreSQL]
|
||||
J[Redis]
|
||||
end
|
||||
|
||||
A --> B
|
||||
B --> C
|
||||
C --> D
|
||||
D --> E
|
||||
E --> F
|
||||
F --> G
|
||||
G --> H
|
||||
H --> I
|
||||
H --> J
|
||||
|
||||
style B fill:#fff9c4
|
||||
style C fill:#fff9c4
|
||||
style D fill:#fff9c4
|
||||
style G fill:#ffebee
|
||||
style H fill:#ffebee
|
||||
```
|
||||
|
||||
## 9. 监控与日志架构
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Application"
|
||||
A[Services]
|
||||
end
|
||||
|
||||
subgraph "Logging"
|
||||
B[Structured Logs]
|
||||
C[Log Aggregation]
|
||||
end
|
||||
|
||||
subgraph "Metrics"
|
||||
D[Prometheus]
|
||||
E[Custom Metrics]
|
||||
end
|
||||
|
||||
subgraph "Tracing"
|
||||
F[OpenTelemetry]
|
||||
G[Jaeger/Tempo]
|
||||
end
|
||||
|
||||
subgraph "Alerting"
|
||||
H[Alert Manager]
|
||||
I[Notifications]
|
||||
end
|
||||
|
||||
A --> B
|
||||
A --> E
|
||||
A --> F
|
||||
B --> C
|
||||
E --> D
|
||||
F --> G
|
||||
D --> H
|
||||
C --> H
|
||||
H --> I
|
||||
|
||||
style A fill:#e1f5ff
|
||||
style D fill:#c8e6c9
|
||||
style G fill:#fff9c4
|
||||
style H fill:#ffebee
|
||||
```
|
||||
|
||||
## 10. 扩展性架构
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Current System"
|
||||
A[Core Services]
|
||||
end
|
||||
|
||||
subgraph "Extension Points"
|
||||
B1[Embedding Provider<br/>Interface]
|
||||
B2[Intent Classifier<br/>Interface]
|
||||
B3[Reranking Strategy<br/>Interface]
|
||||
B4[Filter Plugin<br/>Interface]
|
||||
end
|
||||
|
||||
subgraph "Possible Extensions"
|
||||
C1[Local Embedding<br/>Models]
|
||||
C2[Custom Intent<br/>Classifiers]
|
||||
C3[Business-specific<br/>Reranking]
|
||||
C4[Domain Filters]
|
||||
end
|
||||
|
||||
A --> B1
|
||||
A --> B2
|
||||
A --> B3
|
||||
A --> B4
|
||||
|
||||
B1 -.->|Implement| C1
|
||||
B2 -.->|Implement| C2
|
||||
B3 -.->|Implement| C3
|
||||
B4 -.->|Implement| C4
|
||||
|
||||
style A fill:#e1f5ff
|
||||
style B1 fill:#fff9c4
|
||||
style B2 fill:#fff9c4
|
||||
style B3 fill:#fff9c4
|
||||
style B4 fill:#fff9c4
|
||||
style C1 fill:#e8f5e9
|
||||
style C2 fill:#e8f5e9
|
||||
style C3 fill:#e8f5e9
|
||||
style C4 fill:#e8f5e9
|
||||
```
|
||||
606
docs/agent-skills-search/doc-01/07-api-design.md
Normal file
606
docs/agent-skills-search/doc-01/07-api-design.md
Normal file
|
|
@ -0,0 +1,606 @@
|
|||
# Agent Skills 智能检索系统 - API 设计
|
||||
|
||||
## 1. 概述
|
||||
|
||||
Agent Skills 智能检索系统提供 RESTful API，支持 Agent 进行智能技能检索和会话管理。
|
||||
|
||||
## 2. API 基础信息
|
||||
|
||||
### 2.1 基础 URL
|
||||
|
||||
```
|
||||
生产环境: https://api.skillhub.example.com/api/agent
|
||||
开发环境: http://localhost:8080/api/agent
|
||||
```
|
||||
|
||||
### 2.2 认证方式
|
||||
|
||||
使用 Bearer Token 认证：
|
||||
|
||||
```
|
||||
Authorization: Bearer {access_token}
|
||||
```
|
||||
|
||||
### 2.3 通用响应格式
|
||||
|
||||
**成功响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": { ... },
|
||||
"timestamp": "2024-04-05T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**错误响应**：
|
||||
```json
|
||||
{
|
||||
"code": 40001,
|
||||
"message": "Invalid request parameter",
|
||||
"details": {
|
||||
"field": "query",
|
||||
"error": "Query text cannot be empty"
|
||||
},
|
||||
"timestamp": "2024-04-05T10:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## 3. 会话管理 API
|
||||
|
||||
### 3.1 创建会话
|
||||
|
||||
**请求**：
|
||||
```
|
||||
POST /api/agent/sessions
|
||||
```
|
||||
|
||||
**请求体**：
|
||||
```json
|
||||
{
|
||||
"agentId": "agent-001",
|
||||
"userId": "user-123",
|
||||
"scenarioDescription": "Web 前端开发",
|
||||
"userRole": "frontend-developer",
|
||||
"metadata": {
|
||||
"projectId": "project-456",
|
||||
"environment": "development"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"sessionId": "sess-789-xyz",
|
||||
"agentId": "agent-001",
|
||||
"userId": "user-123",
|
||||
"scenarioDescription": "Web 前端开发",
|
||||
"userRole": "frontend-developer",
|
||||
"metadata": {
|
||||
"projectId": "project-456",
|
||||
"environment": "development"
|
||||
},
|
||||
"startedAt": "2024-04-05T10:30:00Z",
|
||||
"updatedAt": "2024-04-05T10:30:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 获取会话
|
||||
|
||||
**请求**：
|
||||
```
|
||||
GET /api/agent/sessions/{sessionId}
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"sessionId": "sess-789-xyz",
|
||||
"agentId": "agent-001",
|
||||
"userId": "user-123",
|
||||
"scenarioDescription": "Web 前端开发",
|
||||
"userRole": "frontend-developer",
|
||||
"messageCount": 5,
|
||||
"startedAt": "2024-04-05T10:30:00Z",
|
||||
"updatedAt": "2024-04-05T10:35:00Z",
|
||||
"endedAt": null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 获取会话历史
|
||||
|
||||
**请求**：
|
||||
```
|
||||
GET /api/agent/sessions/{sessionId}/messages?limit=10&offset=0
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"messages": [
|
||||
{
|
||||
"id": 1,
|
||||
"role": "user",
|
||||
"content": "我需要一个处理 JSON 数据的技能",
|
||||
"intentLabel": "data_processing",
|
||||
"intentConfidence": 0.92,
|
||||
"createdAt": "2024-04-05T10:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"role": "assistant",
|
||||
"content": "我为您找到了几个相关技能...",
|
||||
"createdAt": "2024-04-05T10:30:05Z"
|
||||
}
|
||||
],
|
||||
"total": 5,
|
||||
"limit": 10,
|
||||
"offset": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.4 结束会话
|
||||
|
||||
**请求**：
|
||||
```
|
||||
POST /api/agent/sessions/{sessionId}/end
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"sessionId": "sess-789-xyz",
|
||||
"endedAt": "2024-04-05T11:00:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 4. 技能搜索 API
|
||||
|
||||
### 4.1 智能技能搜索
|
||||
|
||||
**请求**：
|
||||
```
|
||||
POST /api/agent/skills/search
|
||||
```
|
||||
|
||||
**请求体**：
|
||||
```json
|
||||
{
|
||||
"query": "我需要一个处理 JSON 数据的技能",
|
||||
"sessionId": "sess-789-xyz",
|
||||
"limit": 10,
|
||||
"options": {
|
||||
"enableIntentRecognition": true,
|
||||
"enableContextAware": true,
|
||||
"tags": ["json", "data-processing"],
|
||||
"intent": "data_processing",
|
||||
"minConfidence": 0.3
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**参数说明**：
|
||||
|
||||
| 参数 | 类型 | 必填 | 说明 |
|
||||
|------|------|------|------|
|
||||
| query | string | 是 | 搜索查询文本 |
|
||||
| sessionId | string | 否 | 会话 ID，用于上下文感知 |
|
||||
| limit | integer | 否 | 返回结果数量，默认 10 |
|
||||
| options.enableIntentRecognition | boolean | 否 | 是否启用意图识别，默认 true |
|
||||
| options.enableContextAware | boolean | 否 | 是否启用上下文感知，默认 true |
|
||||
| options.tags | string[] | 否 | 指定标签过滤 |
|
||||
| options.intent | string | 否 | 指定意图（跳过自动识别） |
|
||||
| options.minConfidence | number | 否 | 最小置信度，默认 0.3 |
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"query": "我需要一个处理 JSON 数据的技能",
|
||||
"intent": {
|
||||
"label": "data_processing",
|
||||
"confidence": 0.92,
|
||||
"alternatives": [
|
||||
{
|
||||
"label": "file_processing",
|
||||
"confidence": 0.75
|
||||
}
|
||||
]
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"skillId": 123,
|
||||
"name": "json-parser",
|
||||
"displayName": "JSON Parser",
|
||||
"namespace": "default",
|
||||
"summary": "强大的 JSON 解析和处理库",
|
||||
"version": "1.2.3",
|
||||
"scores": {
|
||||
"vectorSimilarity": 0.89,
|
||||
"intentMatchScore": 1.0,
|
||||
"tagMatchScore": 0.8,
|
||||
"popularityScore": 0.65,
|
||||
"ratingScore": 0.88,
|
||||
"finalScore": 0.87
|
||||
},
|
||||
"matchReason": "高度匹配：查询向量相似度 0.89，意图完全匹配，标签匹配度 80%",
|
||||
"tags": ["json", "parser", "data"],
|
||||
"downloadCount": 1520,
|
||||
"ratingAvg": 4.4,
|
||||
"ratingCount": 45
|
||||
},
|
||||
{
|
||||
"skillId": 456,
|
||||
"name": "data-transformer",
|
||||
"displayName": "Data Transformer",
|
||||
"namespace": "utils",
|
||||
"summary": "通用数据转换工具",
|
||||
"version": "2.1.0",
|
||||
"scores": {
|
||||
"vectorSimilarity": 0.76,
|
||||
"intentMatchScore": 1.0,
|
||||
"tagMatchScore": 0.6,
|
||||
"popularityScore": 0.45,
|
||||
"ratingScore": 0.8,
|
||||
"finalScore": 0.72
|
||||
},
|
||||
"matchReason": "良好匹配：意图完全匹配，部分标签匹配",
|
||||
"tags": ["data", "transform"],
|
||||
"downloadCount": 890,
|
||||
"ratingAvg": 4.0,
|
||||
"ratingCount": 28
|
||||
}
|
||||
],
|
||||
"total": 45,
|
||||
"limit": 10,
|
||||
"processingTimeMs": 245
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 快速搜索（简化版）
|
||||
|
||||
**请求**：
|
||||
```
|
||||
GET /api/agent/skills/search?q={query}&limit={limit}
|
||||
```
|
||||
|
||||
**示例**：
|
||||
```
|
||||
GET /api/agent/skills/search?q=json%20parser&limit=5
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"query": "json parser",
|
||||
"results": [
|
||||
{
|
||||
"skillId": 123,
|
||||
"name": "json-parser",
|
||||
"displayName": "JSON Parser",
|
||||
"namespace": "default",
|
||||
"summary": "强大的 JSON 解析和处理库",
|
||||
"version": "1.2.3",
|
||||
"score": 0.87
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 5. 意图管理 API
|
||||
|
||||
### 5.1 获取所有意图
|
||||
|
||||
**请求**：
|
||||
```
|
||||
GET /api/agent/intents?activeOnly=true
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"intents": [
|
||||
{
|
||||
"id": 1,
|
||||
"label": "code_generation",
|
||||
"description": "代码生成相关技能",
|
||||
"skillCategories": ["coding", "development"],
|
||||
"exampleQueries": [
|
||||
"写一个函数",
|
||||
"生成代码",
|
||||
"创建类"
|
||||
],
|
||||
"priority": 10,
|
||||
"active": true
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"label": "data_analysis",
|
||||
"description": "数据分析相关技能",
|
||||
"skillCategories": ["data", "analysis"],
|
||||
"exampleQueries": [
|
||||
"分析数据",
|
||||
"绘制图表",
|
||||
"数据统计"
|
||||
],
|
||||
"priority": 10,
|
||||
"active": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 识别意图
|
||||
|
||||
**请求**：
|
||||
```
|
||||
POST /api/agent/intents/recognize
|
||||
```
|
||||
|
||||
**请求体**：
|
||||
```json
|
||||
{
|
||||
"query": "我需要生成一个处理用户认证的函数"
|
||||
}
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"primaryIntent": {
|
||||
"label": "code_generation",
|
||||
"confidence": 0.88
|
||||
},
|
||||
"alternativeIntents": [
|
||||
{
|
||||
"label": "authentication",
|
||||
"confidence": 0.72
|
||||
},
|
||||
{
|
||||
"label": "general",
|
||||
"confidence": 0.15
|
||||
}
|
||||
],
|
||||
"processingTimeMs": 85
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 6. 批量操作 API
|
||||
|
||||
### 6.1 批量生成技能嵌入
|
||||
|
||||
**请求**：
|
||||
```
|
||||
POST /api/agent/skills/embeddings/batch
|
||||
```
|
||||
|
||||
**请求体**：
|
||||
```json
|
||||
{
|
||||
"skillIds": [123, 456, 789],
|
||||
"embeddingTypes": ["description", "combined"]
|
||||
}
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"taskId": "task-001-xyz",
|
||||
"status": "processing",
|
||||
"totalCount": 3,
|
||||
"successCount": 0,
|
||||
"failureCount": 0,
|
||||
"estimatedCompletion": "2024-04-05T10:35:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6.2 查询批量任务状态
|
||||
|
||||
**请求**：
|
||||
```
|
||||
GET /api/agent/tasks/{taskId}
|
||||
```
|
||||
|
||||
**响应**：
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "success",
|
||||
"data": {
|
||||
"taskId": "task-001-xyz",
|
||||
"status": "completed",
|
||||
"totalCount": 3,
|
||||
"successCount": 3,
|
||||
"failureCount": 0,
|
||||
"results": [
|
||||
{
|
||||
"skillId": 123,
|
||||
"status": "success",
|
||||
"embeddingTypes": ["description", "combined"]
|
||||
}
|
||||
],
|
||||
"startedAt": "2024-04-05T10:30:00Z",
|
||||
"completedAt": "2024-04-05T10:31:15Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 7. 错误码
|
||||
|
||||
| 错误码 | 说明 | HTTP 状态码 |
|
||||
|--------|------|-------------|
|
||||
| 0 | 成功 | 200 |
|
||||
| 40001 | 请求参数错误 | 400 |
|
||||
| 40002 | 缺少必填参数 | 400 |
|
||||
| 40101 | 未认证 | 401 |
|
||||
| 40301 | 无权限访问 | 403 |
|
||||
| 40401 | 资源不存在 | 404 |
|
||||
| 40901 | 资源冲突 | 409 |
|
||||
| 42901 | 请求频率超限 | 429 |
|
||||
| 50001 | 服务器内部错误 | 500 |
|
||||
| 50301 | 服务不可用 | 503 |
|
||||
| 50002 | OpenAI API 调用失败 | 500 |
|
||||
| 50003 | 向量索引不可用 | 500 |
|
||||
|
||||
## 8. 速率限制
|
||||
|
||||
| 端点 | 限制 | 时间窗口 |
|
||||
|------|------|----------|
|
||||
| POST /api/agent/skills/search | 100 | 1分钟 |
|
||||
| GET /api/agent/sessions/{id} | 200 | 1分钟 |
|
||||
| POST /api/agent/sessions | 50 | 1分钟 |
|
||||
| POST /api/agent/intents/recognize | 60 | 1分钟 |
|
||||
|
||||
**响应头**：
|
||||
```
|
||||
X-RateLimit-Limit: 100
|
||||
X-RateLimit-Remaining: 95
|
||||
X-RateLimit-Reset: 1712289000
|
||||
```
|
||||
|
||||
## 9. Webhook 通知
|
||||
|
||||
### 9.1 向量生成完成通知
|
||||
|
||||
**请求体**：
|
||||
```json
|
||||
{
|
||||
"eventType": "embedding.completed",
|
||||
"timestamp": "2024-04-05T10:30:00Z",
|
||||
"data": {
|
||||
"skillId": 123,
|
||||
"skillVersionId": 456,
|
||||
"embeddingTypes": ["description", "combined"],
|
||||
"status": "success"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 10. SDK 示例
|
||||
|
||||
### 10.1 JavaScript/TypeScript
|
||||
|
||||
```typescript
|
||||
import { SkillHubAgentClient } from '@skillhub/agent-sdk';
|
||||
|
||||
const client = new SkillHubAgentClient({
|
||||
baseURL: 'https://api.skillhub.example.com/api/agent',
|
||||
apiKey: 'your-api-key'
|
||||
});
|
||||
|
||||
// 创建会话
|
||||
const session = await client.createSession({
|
||||
agentId: 'agent-001',
|
||||
userId: 'user-123',
|
||||
scenarioDescription: 'Web 前端开发',
|
||||
userRole: 'frontend-developer'
|
||||
});
|
||||
|
||||
// 搜索技能
|
||||
const results = await client.searchSkills({
|
||||
query: '我需要一个处理 JSON 数据的技能',
|
||||
sessionId: session.sessionId,
|
||||
limit: 10
|
||||
});
|
||||
|
||||
console.log(results.results);
|
||||
```
|
||||
|
||||
### 10.2 Python
|
||||
|
||||
```python
|
||||
from skillhub_agent import SkillHubAgentClient
|
||||
|
||||
client = SkillHubAgentClient(
|
||||
base_url='https://api.skillhub.example.com/api/agent',
|
||||
api_key='your-api-key'
|
||||
)
|
||||
|
||||
# 创建会话
|
||||
session = client.create_session(
|
||||
agent_id='agent-001',
|
||||
user_id='user-123',
|
||||
scenario_description='Web 前端开发',
|
||||
user_role='frontend-developer'
|
||||
)
|
||||
|
||||
# 搜索技能
|
||||
results = client.search_skills(
|
||||
query='我需要一个处理 JSON 数据的技能',
|
||||
session_id=session['sessionId'],
|
||||
limit=10
|
||||
)
|
||||
|
||||
print(results['results'])
|
||||
```
|
||||
|
||||
### 10.3 Java
|
||||
|
||||
```java
|
||||
import com.iflytek.skillhub.agent.SkillHubAgentClient;
|
||||
import com.iflytek.skillhub.agent.dto.*;
|
||||
|
||||
SkillHubAgentClient client = new SkillHubAgentClient(
|
||||
"https://api.skillhub.example.com/api/agent",
|
||||
"your-api-key"
|
||||
);
|
||||
|
||||
// 创建会话
|
||||
CreateSessionRequest sessionRequest = CreateSessionRequest.builder()
|
||||
.agentId("agent-001")
|
||||
.userId("user-123")
|
||||
.scenarioDescription("Web 前端开发")
|
||||
.userRole("frontend-developer")
|
||||
.build();
|
||||
|
||||
AgentSession session = client.createSession(sessionRequest);
|
||||
|
||||
// 搜索技能
|
||||
SearchSkillsRequest searchRequest = SearchSkillsRequest.builder()
|
||||
.query("我需要一个处理 JSON 数据的技能")
|
||||
.sessionId(session.getSessionId())
|
||||
.limit(10)
|
||||
.build();
|
||||
|
||||
SearchSkillsResponse results = client.searchSkills(searchRequest);
|
||||
|
||||
results.getResults().forEach(skill -> {
|
||||
System.out.println(skill.getName() + ": " + skill.getFinalScore());
|
||||
});
|
||||
```
|
||||
648
docs/agent-skills-search/doc-01/08-implementation-guide.md
Normal file
648
docs/agent-skills-search/doc-01/08-implementation-guide.md
Normal file
|
|
@ -0,0 +1,648 @@
|
|||
# Agent Skills 智能检索系统 - 实施指南
|
||||
|
||||
## 1. 实施阶段
|
||||
|
||||
### 阶段 1：基础设施准备（1-2 周）
|
||||
|
||||
#### 1.1 数据库配置
|
||||
|
||||
**安装 pgvector 扩展**：
|
||||
|
||||
```bash
|
||||
# 连接到 PostgreSQL
|
||||
psql -U postgres -d skillhub
|
||||
|
||||
# 安装 pgvector 扩展
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
|
||||
# 验证安装
|
||||
SELECT * FROM pg_extension WHERE extname = 'vector';
|
||||
|
||||
# 查看可用版本
|
||||
SELECT * FROM pg_available_extensions WHERE name = 'vector';
|
||||
```
|
||||
|
||||
**配置连接池**（application.yml）：
|
||||
```yaml
|
||||
spring:
|
||||
datasource:
|
||||
hikari:
|
||||
maximum-pool-size: 20
|
||||
minimum-idle: 5
|
||||
connection-timeout: 30000
|
||||
idle-timeout: 600000
|
||||
max-lifetime: 1800000
|
||||
```
|
||||
|
||||
#### 1.2 Redis 配置
|
||||
|
||||
```yaml
|
||||
spring:
|
||||
data:
|
||||
redis:
|
||||
host: localhost
|
||||
port: 6379
|
||||
password: ${REDIS_PASSWORD}
|
||||
database: 0
|
||||
timeout: 5000ms
|
||||
lettuce:
|
||||
pool:
|
||||
max-active: 20
|
||||
max-idle: 10
|
||||
min-idle: 5
|
||||
```
|
||||
|
||||
#### 1.3 OpenAI API 配置
|
||||
|
||||
```yaml
|
||||
openai:
|
||||
api-key: ${OPENAI_API_KEY}
|
||||
embedding:
|
||||
model: text-embedding-3-small
|
||||
dimension: 1536
|
||||
timeout: 30000
|
||||
max-retries: 3
|
||||
```
|
||||
|
||||
### 阶段 2：数据库迁移（1 周）
|
||||
|
||||
#### 2.1 创建迁移脚本
|
||||
|
||||
创建 `V__create_agent_tables.sql`：
|
||||
|
||||
```sql
|
||||
-- 参见 03-data-model.md 中的完整 SQL
|
||||
|
||||
-- 1. 安装 pgvector
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
|
||||
-- 2. 创建 agent_session 表
|
||||
...
|
||||
|
||||
-- 3. 创建 agent_message 表
|
||||
...
|
||||
|
||||
-- 4. 创建 skill_embedding 表
|
||||
...
|
||||
|
||||
-- 5. 创建 intent_mapping 表
|
||||
...
|
||||
|
||||
-- 6. 创建索引
|
||||
...
|
||||
|
||||
-- 7. 插入初始数据
|
||||
INSERT INTO intent_mapping (intent_label, description, skill_categories, example_queries, priority, active)
|
||||
VALUES
|
||||
('code_generation', '代码生成相关技能', ARRAY['coding', 'development'], ARRAY['写一个函数', '生成代码', '创建类'], 10, true),
|
||||
('data_analysis', '数据分析相关技能', ARRAY['data', 'analysis'], ARRAY['分析数据', '绘制图表', '数据统计'], 10, true),
|
||||
...
|
||||
;
|
||||
```
|
||||
|
||||
#### 2.2 执行迁移
|
||||
|
||||
```bash
|
||||
# 使用 Flyway
|
||||
mvn flyway:migrate
|
||||
|
||||
# 或使用 Spring Boot 自动迁移
|
||||
mvn spring-boot:run -Dspring.flyway.enabled=true
|
||||
```
|
||||
|
||||
### 阶段 3：核心服务开发（2-3 周）
|
||||
|
||||
#### 3.1 领域模型
|
||||
|
||||
创建 `server/skillhub-domain/src/main/java/com/iflytek/skillhub/domain/agent/` 目录：
|
||||
|
||||
```java
|
||||
// AgentSession.java
|
||||
@Entity
|
||||
@Table(name = "agent_session")
|
||||
public class AgentSession {
|
||||
@Id
|
||||
@GeneratedValue(strategy = GenerationType.IDENTITY)
|
||||
private Long id;
|
||||
|
||||
@Column(name = "agent_id", nullable = false)
|
||||
private String agentId;
|
||||
|
||||
@Column(name = "user_id", nullable = false)
|
||||
private String userId;
|
||||
|
||||
@Column(name = "session_id", nullable = false, unique = true)
|
||||
private String sessionId;
|
||||
|
||||
// ... 其他字段
|
||||
|
||||
@PrePersist
|
||||
protected void onCreate() {
|
||||
startedAt = Instant.now();
|
||||
updatedAt = Instant.now();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 3.2 Repository 接口
|
||||
|
||||
```java
|
||||
// AgentSessionRepository.java
|
||||
@Repository
|
||||
public interface AgentSessionRepository extends JpaRepository<AgentSession, Long> {
|
||||
Optional<AgentSession> findBySessionId(String sessionId);
|
||||
|
||||
List<AgentSession> findByUserIdAndEndedAtIsNull(String userId);
|
||||
|
||||
@Query("SELECT a FROM AgentSession a WHERE a.sessionId = :sessionId AND a.endedAt IS NULL")
|
||||
Optional<AgentSession> findActiveBySessionId(@Param("sessionId") String sessionId);
|
||||
}
|
||||
```
|
||||
|
||||
#### 3.3 服务实现
|
||||
|
||||
**AgentContextBuilder.java**：
|
||||
```java
|
||||
@Service
|
||||
public class AgentContextBuilderImpl implements AgentContextBuilder {
|
||||
|
||||
private final AgentSessionService sessionService;
|
||||
private final int maxHistorySize;
|
||||
|
||||
@Override
|
||||
public AgentSearchContext build(AgentSearchRequest request) {
|
||||
// 获取会话历史
|
||||
List<AgentMessage> history = request.sessionId() != null
|
||||
? sessionService.getRecentMessages(request.sessionId(), maxHistorySize)
|
||||
: List.of();
|
||||
|
||||
return new AgentSearchContext(
|
||||
request.query(),
|
||||
history,
|
||||
request.scenarioDescription(),
|
||||
request.userRole(),
|
||||
request.metadata()
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**IntentClassifier.java**：
|
||||
```java
|
||||
@Service
|
||||
public class VectorIntentClassifier implements IntentClassifier {
|
||||
|
||||
private final IntentMappingRepository intentRepository;
|
||||
private final OpenAiEmbeddingClient embeddingClient;
|
||||
private final double minConfidence = 0.3;
|
||||
|
||||
@Override
|
||||
public IntentClassificationResult classify(String query, AgentSearchContext context) {
|
||||
// 生成查询向量
|
||||
float[] queryVector = embeddingClient.embed(query);
|
||||
|
||||
// 获取所有活跃意图
|
||||
List<IntentMapping> intents = intentRepository.findByActiveTrue();
|
||||
|
||||
// 计算相似度
|
||||
List<IntentScore> scores = intents.stream()
|
||||
.map(intent -> new IntentScore(
|
||||
intent.getIntentLabel(),
|
||||
cosineSimilarity(queryVector, parseVector(intent.getEmbedding()))
|
||||
))
|
||||
.sorted(Comparator.comparingDouble(IntentScore::score).reversed())
|
||||
.toList();
|
||||
|
||||
if (scores.isEmpty() || scores.get(0).score() < minConfidence) {
|
||||
return new IntentClassificationResult("general", 0.0, List.of());
|
||||
}
|
||||
|
||||
return new IntentClassificationResult(
|
||||
scores.get(0).label(),
|
||||
scores.get(0).score(),
|
||||
scores.subList(1, Math.min(3, scores.size()))
|
||||
.stream()
|
||||
.map(IntentScore::label)
|
||||
.toList()
|
||||
);
|
||||
}
|
||||
|
||||
private double cosineSimilarity(float[] a, float[] b) {
|
||||
double dot = 0, normA = 0, normB = 0;
|
||||
for (int i = 0; i < a.length; i++) {
|
||||
dot += a[i] * b[i];
|
||||
normA += a[i] * a[i];
|
||||
normB += b[i] * b[i];
|
||||
}
|
||||
return dot / (Math.sqrt(normA) * Math.sqrt(normB));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 3.4 OpenAI 集成
|
||||
|
||||
**OpenAiEmbeddingClient.java**：
|
||||
```java
|
||||
@Service
|
||||
public class OpenAiEmbeddingClient {
|
||||
|
||||
private final String apiKey;
|
||||
private final String model;
|
||||
private final RestTemplate restTemplate;
|
||||
|
||||
public float[] embed(String text) {
|
||||
HttpHeaders headers = new HttpHeaders();
|
||||
headers.setContentType(MediaType.APPLICATION_JSON);
|
||||
headers.setBearerAuth(apiKey);
|
||||
|
||||
Map<String, Object> body = Map.of(
|
||||
"model", model,
|
||||
"input", text
|
||||
);
|
||||
|
||||
HttpEntity<Map<String, Object>> request = new HttpEntity<>(body, headers);
|
||||
|
||||
OpenAiResponse response = restTemplate.postForObject(
|
||||
"https://api.openai.com/v1/embeddings",
|
||||
request,
|
||||
OpenAiResponse.class
|
||||
);
|
||||
|
||||
return response.data().get(0).embedding();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 阶段 4：向量生成（1 周）
|
||||
|
||||
#### 4.1 批量向量生成任务
|
||||
|
||||
```java
|
||||
@Service
|
||||
public class SkillEmbeddingGenerator {
|
||||
|
||||
@Scheduled(fixedDelay = 3600000) // 每小时检查一次
|
||||
public void generatePendingEmbeddings() {
|
||||
List<Skill> skills = skillRepository.findSkillsWithoutEmbeddings();
|
||||
|
||||
for (Skill skill : skills) {
|
||||
try {
|
||||
generateEmbeddings(skill);
|
||||
} catch (Exception e) {
|
||||
log.error("Failed to generate embeddings for skill {}", skill.getId(), e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void generateEmbeddings(Skill skill) {
|
||||
String description = skill.getSummary() != null ? skill.getSummary() : "";
|
||||
float[] descVector = embeddingClient.embed(description);
|
||||
|
||||
SkillEmbedding embedding = new SkillEmbedding();
|
||||
embedding.setSkillId(skill.getId());
|
||||
embedding.setEmbeddingType(EmbeddingType.DESCRIPTION);
|
||||
embedding.setEmbedding(serializeVector(descVector));
|
||||
embedding.setUpdatedAt(Instant.now());
|
||||
|
||||
embeddingRepository.save(embedding);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 4.2 手动触发向量生成
|
||||
|
||||
```java
|
||||
@RestController
|
||||
@RequestMapping("/api/admin/skills")
|
||||
public class AdminSkillController {
|
||||
|
||||
@PostMapping("/{skillId}/embeddings")
|
||||
public ResponseEntity<?> generateEmbeddings(@PathVariable Long skillId) {
|
||||
Skill skill = skillRepository.findById(skillId)
|
||||
.orElseThrow(() -> new NotFoundException("Skill not found"));
|
||||
|
||||
embeddingGenerator.generateEmbeddings(skill);
|
||||
|
||||
return ResponseEntity.ok(Map.of("status", "processing"));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 阶段 5：API 开发（1 周）
|
||||
|
||||
#### 5.1 Controller 实现
|
||||
|
||||
**AgentSkillSearchController.java**：
|
||||
```java
|
||||
@RestController
|
||||
@RequestMapping("/api/agent")
|
||||
@RequiredArgsConstructor
|
||||
public class AgentSkillSearchController {
|
||||
|
||||
private final HybridSkillSearchService searchService;
|
||||
|
||||
@PostMapping("/skills/search")
|
||||
public ResponseEntity<ApiResponse<SearchResult>> searchSkills(
|
||||
@Valid @RequestBody AgentSearchRequest request
|
||||
) {
|
||||
SearchResult result = searchService.search(request);
|
||||
return ResponseEntity.ok(ApiResponse.success(result));
|
||||
}
|
||||
|
||||
@PostMapping("/sessions")
|
||||
public ResponseEntity<ApiResponse<AgentSession>> createSession(
|
||||
@Valid @RequestBody CreateSessionRequest request
|
||||
) {
|
||||
AgentSession session = sessionService.createSession(request);
|
||||
return ResponseEntity.ok(ApiResponse.success(session));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 5.2 DTO 定义
|
||||
|
||||
```java
|
||||
public record AgentSearchRequest(
|
||||
@NotBlank String query,
|
||||
String sessionId,
|
||||
@Min(1) @Max(100) Integer limit,
|
||||
SearchOptions options
|
||||
) {
|
||||
public AgentSearchRequest {
|
||||
if (limit == null) limit = 10;
|
||||
if (options == null) options = new SearchOptions();
|
||||
}
|
||||
}
|
||||
|
||||
public record SearchOptions(
|
||||
Boolean enableIntentRecognition,
|
||||
Boolean enableContextAware,
|
||||
List<String> tags,
|
||||
String intent,
|
||||
Double minConfidence
|
||||
) {
|
||||
public SearchOptions {
|
||||
if (enableIntentRecognition == null) enableIntentRecognition = true;
|
||||
if (enableContextAware == null) enableContextAware = true;
|
||||
if (minConfidence == null) minConfidence = 0.3;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 阶段 6：测试（1-2 周）
|
||||
|
||||
#### 6.1 单元测试
|
||||
|
||||
```java
|
||||
@SpringBootTest
|
||||
class IntentClassifierTest {
|
||||
|
||||
@Autowired
|
||||
private IntentClassifier intentClassifier;
|
||||
|
||||
@Test
|
||||
void testClassifyCodeGenerationIntent() {
|
||||
String query = "写一个快速排序算法";
|
||||
|
||||
IntentClassificationResult result = intentClassifier.classify(query, null);
|
||||
|
||||
assertEquals("code_generation", result.intentLabel());
|
||||
assertTrue(result.confidence() > 0.5);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 6.2 集成测试
|
||||
|
||||
```java
|
||||
@SpringBootTest
|
||||
@AutoConfigureMockMvc
|
||||
class AgentSkillSearchIntegrationTest {
|
||||
|
||||
@Autowired
|
||||
private MockMvc mockMvc;
|
||||
|
||||
@Test
|
||||
void testSearchSkillsEndpoint() throws Exception {
|
||||
String request = """
|
||||
{
|
||||
"query": "处理 JSON 数据",
|
||||
"sessionId": "test-session-123",
|
||||
"limit": 5
|
||||
}
|
||||
""";
|
||||
|
||||
mockMvc.perform(post("/api/agent/skills/search")
|
||||
.contentType(MediaType.APPLICATION_JSON)
|
||||
.content(request))
|
||||
.andExpect(status().isOk())
|
||||
.andExpect(jsonPath("$.code").value(0))
|
||||
.andExpect(jsonPath("$.data.results").isArray());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 阶段 7：部署（1 周）
|
||||
|
||||
#### 7.1 Docker 配置
|
||||
|
||||
**Dockerfile**：
|
||||
```dockerfile
|
||||
FROM openjdk:21-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY target/skillhub-app.jar /app/
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
ENTRYPOINT ["java", "-jar", "skillhub-app.jar"]
|
||||
```
|
||||
|
||||
#### 7.2 Kubernetes 配置
|
||||
|
||||
**deployment.yaml**：
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: skillhub-agent
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: skillhub-agent
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: skillhub-agent
|
||||
spec:
|
||||
containers:
|
||||
- name: skillhub
|
||||
image: skillhub/agent:latest
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
env:
|
||||
- name: SPRING_DATASOURCE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: skillhub-secrets
|
||||
key: database-url
|
||||
- name: OPENAI_API_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: skillhub-secrets
|
||||
key: openai-api-key
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "1000m"
|
||||
```
|
||||
|
||||
### 阶段 8：监控与调优（持续）
|
||||
|
||||
#### 8.1 监控指标
|
||||
|
||||
```java
|
||||
@Component
|
||||
public class SearchMetrics {
|
||||
|
||||
private final MeterRegistry meterRegistry;
|
||||
|
||||
public void recordSearch(SearchResult result, long duration) {
|
||||
meterRegistry.counter("skill.search.total",
|
||||
"intent", result.intent().label()
|
||||
).increment();
|
||||
|
||||
meterRegistry.timer("skill.search.duration").record(duration, TimeUnit.MILLISECONDS);
|
||||
|
||||
result.results().forEach(r ->
|
||||
meterRegistry.histogram("skill.search.score").record(r.finalScore())
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 8.2 性能调优
|
||||
|
||||
**向量索引调优**：
|
||||
```sql
|
||||
-- 根据数据量调整 lists 参数
|
||||
-- 建议值：sqrt(行数)
|
||||
CREATE INDEX idx_skill_embedding_embedding ON skill_embedding
|
||||
USING ivfflat(embedding vector_cosine_ops)
|
||||
WITH (lists = 100);
|
||||
```
|
||||
|
||||
**缓存配置优化**：
|
||||
```yaml
|
||||
skillhub:
|
||||
agent:
|
||||
search:
|
||||
cache:
|
||||
enabled: true
|
||||
ttl: 300 # 5分钟
|
||||
max-size: 10000
|
||||
```
|
||||
|
||||
## 9. 检查清单
|
||||
|
||||
### 准备阶段
|
||||
- [ ] PostgreSQL 17 安装完成
|
||||
- [ ] pgvector 扩展安装完成
|
||||
- [ ] Redis 7.0 配置完成
|
||||
- [ ] OpenAI API Key 获取
|
||||
- [ ] 数据库迁移脚本准备
|
||||
|
||||
### 开发阶段
|
||||
- [ ] 领域模型定义完成
|
||||
- [ ] Repository 接口定义完成
|
||||
- [ ] 核心服务实现完成
|
||||
- [ ] OpenAI 集成完成
|
||||
- [ ] 混合检索实现完成
|
||||
- [ ] API 端点实现完成
|
||||
|
||||
### 测试阶段
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
- [ ] 集成测试通过
|
||||
- [ ] 性能测试达标（< 500ms）
|
||||
- [ ] 负载测试通过（100+ QPS）
|
||||
|
||||
### 部署阶段
|
||||
- [ ] Docker 镜像构建完成
|
||||
- [ ] Kubernetes 配置完成
|
||||
- [ ] 环境变量配置完成
|
||||
- [ ] 健康检查配置完成
|
||||
- [ ] 日志收集配置完成
|
||||
|
||||
### 上线后
|
||||
- [ ] 监控指标配置完成
|
||||
- [ ] 告警规则配置完成
|
||||
- [ ] 文档更新完成
|
||||
- [ ] 团队培训完成
|
||||
|
||||
## 10. 常见问题
|
||||
|
||||
### Q1: pgvector 索引创建失败
|
||||
|
||||
**原因**：数据量不足或参数不当
|
||||
|
||||
**解决**：
|
||||
```sql
|
||||
-- 确保有足够的数据（至少 100 行）
|
||||
-- 调整 lists 参数
|
||||
CREATE INDEX idx_skill_embedding_embedding ON skill_embedding
|
||||
USING ivfflat(embedding vector_cosine_ops)
|
||||
WITH (lists = 32);
|
||||
```
|
||||
|
||||
### Q2: OpenAI API 调用超时
|
||||
|
||||
**原因**：网络延迟或 API 限制
|
||||
|
||||
**解决**：
|
||||
```yaml
|
||||
openai:
|
||||
embedding:
|
||||
timeout: 60000 # 增加超时时间
|
||||
max-retries: 5 # 增加重试次数
|
||||
```
|
||||
|
||||
### Q3: 搜索结果不准确
|
||||
|
||||
**原因**：向量未生成或权重配置不当
|
||||
|
||||
**解决**：
|
||||
1. 检查技能向量是否生成
|
||||
2. 调整重排序权重
|
||||
3. 增加意图类别和示例
|
||||
|
||||
### Q4: 内存占用过高
|
||||
|
||||
**原因**：向量数据量大，缓存过多
|
||||
|
||||
**解决**：
|
||||
```yaml
|
||||
spring:
|
||||
jpa:
|
||||
properties:
|
||||
hibernate:
|
||||
jdbc:
|
||||
batch_size: 50
|
||||
|
||||
skillhub:
|
||||
agent:
|
||||
search:
|
||||
cache:
|
||||
max-size: 5000 # 减少缓存大小
|
||||
```
|
||||
|
||||
## 11. 参考资源
|
||||
|
||||
- [pgvector 官方文档](https://github.com/pgvector/pgvector)
|
||||
- [OpenAI Embeddings API](https://platform.openai.com/docs/guides/embeddings)
|
||||
- [Spring Boot 文档](https://spring.io/projects/spring-boot)
|
||||
- [PostgreSQL 性能调优](https://wiki.postgresql.org/wiki/Performance_Optimization)
|
||||
231
docs/sso-build-ceph/ceph-migration-plan.md
Normal file
231
docs/sso-build-ceph/ceph-migration-plan.md
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
# MinIO 迁移到 Ceph 方案
|
||||
|
||||
## 背景
|
||||
|
||||
当前项目使用 MinIO 作为 S3 兼容对象存储，现需迁移到已部署的 Ceph RGW。由于测试环境暂无生产数据，无需数据迁移，直接配置切换即可。
|
||||
|
||||
## 关键发现
|
||||
|
||||
**代码层面无需修改**：
|
||||
- 项目已有 `ObjectStorageService` 抽象接口
|
||||
- `S3StorageService` 使用 AWS S3 SDK v2，完全兼容 Ceph RGW
|
||||
- 通过 `@ConditionalOnProperty(name = "skillhub.storage.provider")` 条件装配
|
||||
|
||||
## 改动点
|
||||
|
||||
### 1. K8s ConfigMap 配置
|
||||
|
||||
**文件**: `deploy/k8s/base/configmap.yaml`
|
||||
|
||||
```yaml
|
||||
# 修改存储提供商
|
||||
skillhub-storage-provider: s3 # 原值: local
|
||||
|
||||
# 添加 Ceph S3 配置
|
||||
skillhub-storage-s3-endpoint: http://ceph-rgw:7480
|
||||
skillhub-storage-s3-public-endpoint: https://ceph.example.com:7480
|
||||
skillhub-storage-s3-bucket: skillhub
|
||||
skillhub-storage-s3-region: default
|
||||
skillhub-storage-s3-force-path-style: "true"
|
||||
skillhub-storage-s3-auto-create-bucket: "true"
|
||||
skillhub-storage-s3-presign-expiry: PT10M
|
||||
```
|
||||
|
||||
**移除本地存储配置**（可选）：
|
||||
```yaml
|
||||
# 注释或删除
|
||||
# storage-base-path: /var/lib/skillhub/storage
|
||||
```
|
||||
|
||||
### 2. K8s Secret 配置
|
||||
|
||||
**文件**: `deploy/k8s/base/secret.yaml.example`
|
||||
|
||||
```yaml
|
||||
# 添加 Ceph 凭证
|
||||
skillhub-storage-s3-access-key: your-ceph-access-key
|
||||
skillhub-storage-s3-secret-key: your-ceph-secret-key
|
||||
```
|
||||
|
||||
创建实际 Secret：
|
||||
```bash
|
||||
kubectl create secret generic skillhub-secret \
|
||||
--from-literal=skillhub-storage-s3-access-key=your-access-key \
|
||||
--from-literal=skillhub-storage-s3-secret-key=your-secret-key \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
### 3. Backend Deployment 配置
|
||||
|
||||
**文件**: `deploy/k8s/base/backend-deployment.yaml`
|
||||
|
||||
**添加 S3 环境变量**（在 env 部分添加）：
|
||||
```yaml
|
||||
# Storage - S3
|
||||
- name: SKILLHUB_STORAGE_S3_ENDPOINT
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillhub-config
|
||||
key: skillhub-storage-s3-endpoint
|
||||
- name: SKILLHUB_STORAGE_S3_PUBLIC_ENDPOINT
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillhub-config
|
||||
key: skillhub-storage-s3-public-endpoint
|
||||
- name: SKILLHUB_STORAGE_S3_BUCKET
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillhub-config
|
||||
key: skillhub-storage-s3-bucket
|
||||
- name: SKILLHUB_STORAGE_S3_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: skillhub-secret
|
||||
key: skillhub-storage-s3-access-key
|
||||
- name: SKILLHUB_STORAGE_S3_SECRET_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: skillhub-secret
|
||||
key: skillhub-storage-s3-secret-key
|
||||
- name: SKILLHUB_STORAGE_S3_REGION
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillhub-config
|
||||
key: skillhub-storage-s3-region
|
||||
- name: SKILLHUB_STORAGE_S3_FORCE_PATH_STYLE
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillhub-config
|
||||
key: skillhub-storage-s3-force-path-style
|
||||
- name: SKILLHUB_STORAGE_S3_AUTO_CREATE_BUCKET
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillhub-config
|
||||
key: skillhub-storage-s3-auto-create-bucket
|
||||
- name: SKILLHUB_STORAGE_S3_PRESIGN_EXPIRY
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: skillhub-config
|
||||
key: skillhub-storage-s3-presign-expiry
|
||||
```
|
||||
|
||||
**移除本地存储挂载**（可选）：
|
||||
```yaml
|
||||
# 注释或删除 volumeMounts 中的 skillhub-storage
|
||||
# 注释或删除 volumes 中的 skillhub-storage-pvc
|
||||
```
|
||||
|
||||
### 4. Docker Compose 配置
|
||||
|
||||
**文件**: `docker-compose.yml`
|
||||
|
||||
**移除 MinIO 服务**：
|
||||
```yaml
|
||||
# minio:
|
||||
# image: minio/minio:RELEASE.2025-09-07T16-13-09Z
|
||||
# ports:
|
||||
# - "9000:9000"
|
||||
# - "9001:9001"
|
||||
# ...
|
||||
```
|
||||
|
||||
### 5. ConfigMap 中移除 PVC 定义
|
||||
|
||||
**文件**: `deploy/k8s/base/configmap.yaml`
|
||||
|
||||
```yaml
|
||||
# 注释或删除 PersistentVolumeClaim 部分
|
||||
# ---
|
||||
# apiVersion: v1
|
||||
# kind: PersistentVolumeClaim
|
||||
# metadata:
|
||||
# name: skillhub-storage-pvc
|
||||
# ...
|
||||
```
|
||||
|
||||
## 执行步骤
|
||||
|
||||
1. **更新 ConfigMap**
|
||||
```bash
|
||||
kubectl apply -f deploy/k8s/base/configmap.yaml
|
||||
```
|
||||
|
||||
2. **创建/更新 Secret**
|
||||
```bash
|
||||
kubectl create secret generic skillhub-secret \
|
||||
--from-literal=skillhub-storage-s3-access-key=<your-key> \
|
||||
--from-literal=skillhub-storage-s3-secret-key=<your-secret> \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
```
|
||||
|
||||
3. **更新 Deployment**
|
||||
```bash
|
||||
kubectl apply -f deploy/k8s/base/backend-deployment.yaml
|
||||
```
|
||||
|
||||
4. **重启 Pod**
|
||||
```bash
|
||||
kubectl rollout restart deployment/skillhub-server
|
||||
```
|
||||
|
||||
5. **验证配置**
|
||||
```bash
|
||||
kubectl logs -f deployment/skillhub-server
|
||||
```
|
||||
|
||||
## 验证测试
|
||||
|
||||
### 功能验证清单
|
||||
- [ ] Pod 正常启动，无错误日志
|
||||
- [ ] 上传技能包
|
||||
- [ ] 下载技能包
|
||||
- [ ] 生成预签名 URL
|
||||
- [ ] 删除技能包
|
||||
|
||||
### 验证命令
|
||||
```bash
|
||||
# 检查 Pod 日志
|
||||
kubectl logs deployment/skillhub-server
|
||||
|
||||
# 检查环境变量
|
||||
kubectl exec deployment/skillhub-server -- env | grep STORAGE
|
||||
|
||||
# 测试上传（通过 API）
|
||||
curl -X POST http://<service>/api/skills/{namespace}/publish \
|
||||
-F "file=@test.zip" \
|
||||
-F "visibility=public"
|
||||
```
|
||||
|
||||
## 关键文件清单
|
||||
|
||||
| 文件路径 | 操作 |
|
||||
|---------|------|
|
||||
| `deploy/k8s/base/configmap.yaml` | 修改：添加 S3 配置 |
|
||||
| `deploy/k8s/base/secret.yaml.example` | 修改：添加 S3 凭证模板 |
|
||||
| `deploy/k8s/base/backend-deployment.yaml` | 修改：添加 S3 环境变量 |
|
||||
| `docker-compose.yml` | 修改：移除 MinIO 服务 |
|
||||
|
||||
## 注意事项
|
||||
|
||||
1. **Ceph 网络连通性**：确保集群能访问 Ceph RGW 的 endpoint
|
||||
2. **Bucket 创建**：建议提前创建 bucket，或启用 `auto-create-bucket`
|
||||
3. **Region 配置**：Ceph 通常使用 `default` region
|
||||
4. **Path Style**：Ceph RGW 需要启用 `force-path-style`
|
||||
5. **SSL/TLS**：生产环境建议使用 HTTPS
|
||||
|
||||
## 回滚方案
|
||||
|
||||
如果出现问题，快速回滚：
|
||||
|
||||
```bash
|
||||
# 1. 恢复 ConfigMap
|
||||
git checkout deploy/k8s/base/configmap.yaml
|
||||
kubectl apply -f deploy/k8s/base/configmap.yaml
|
||||
|
||||
# 2. 恢复 Deployment
|
||||
git checkout deploy/k8s/base/backend-deployment.yaml
|
||||
kubectl apply -f deploy/k8s/base/backend-deployment.yaml
|
||||
|
||||
# 3. 重启服务
|
||||
kubectl rollout restart deployment/skillhub-server
|
||||
```
|
||||
|
|
@ -65,4 +65,4 @@ EXPOSE 8000
|
|||
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD wget --no-verbose --tries=1 --spider http://127.0.0.1:8000/health || exit 1
|
||||
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000"]
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000", "--root-path", "/skillhub"]
|
||||
|
|
|
|||
|
|
@ -39,4 +39,4 @@ RUN mkdir -p /tmp/skillhub-scans
|
|||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000"]
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000", "--root-path", "/skillhub"]
|
||||
|
|
|
|||
|
|
@ -63,4 +63,4 @@ EXPOSE 8000
|
|||
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD wget --no-verbose --tries=1 --spider http://127.0.0.1:8000/health || exit 1
|
||||
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000"]
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000", "--root-path", "/skillhub"]
|
||||
|
|
|
|||
|
|
@ -45,4 +45,4 @@ EXPOSE 8000
|
|||
HEALTHCHECK --interval=10s --timeout=3s \
|
||||
CMD wget -qO- http://127.0.0.1:8000/health || exit 1
|
||||
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000"]
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000", "--root-path", "/skillhub"]
|
||||
|
|
|
|||
|
|
@ -46,4 +46,4 @@ EXPOSE 8000
|
|||
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD wget --no-verbose --tries=1 --spider http://127.0.0.1:8000/health || exit 1
|
||||
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000"]
|
||||
CMD ["skill-scanner-api", "--host", "0.0.0.0", "--port", "8000", "--root-path", "/skillhub"]
|
||||
|
|
|
|||
|
|
@ -3,6 +3,10 @@ spring:
|
|||
hibernate:
|
||||
ddl-auto: validate
|
||||
show-sql: true
|
||||
properties:
|
||||
hibernate:
|
||||
format_sql: true
|
||||
use_sql_comments: true
|
||||
flyway:
|
||||
enabled: true
|
||||
datasource:
|
||||
|
|
@ -51,3 +55,5 @@ logging:
|
|||
level:
|
||||
com.iflytek.skillhub: INFO
|
||||
org.springframework.security: WARN
|
||||
org.hibernate.SQL: DEBUG
|
||||
org.hibernate.type.descriptor.sql.BasicBinder: TRACE
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue