mirror of
https://github.com/iflytek/skillhub.git
synced 2026-09-24 00:55:35 +00:00
146 lines
4.7 KiB
Markdown
146 lines
4.7 KiB
Markdown
# Skill Scanner Backend Runtime Guide
|
|
|
|
## Overview
|
|
|
|
SkillHub now supports a backend-only security scanning chain around `skill-scanner`.
|
|
The publish flow changes are:
|
|
|
|
1. publish request enters `SkillPublishService`
|
|
2. if scanner is enabled, the version moves to `SCANNING`
|
|
3. the backend enqueues a `ScanTask`
|
|
4. `ScanTaskConsumer` calls `skill-scanner`
|
|
5. the scan result is stored in `security_audit`
|
|
6. the version moves to `PENDING_REVIEW`, or to `SCAN_FAILED` after final retry exhaustion
|
|
7. review still happens through the existing review workflow
|
|
|
|
Frontend is intentionally out of scope here. The frontend should fetch audit details through the dedicated backend API instead of expecting scanner data inside existing review detail payloads.
|
|
|
|
## Runtime Modes
|
|
|
|
Two runtime modes are supported:
|
|
|
|
- `local`
|
|
Use `POST /scan` and pass a filesystem path. This only works when SkillHub and `skill-scanner` can see the same files.
|
|
- `upload`
|
|
Use `POST /scan-upload` and upload the package archive. This is the safer default for split deployments.
|
|
|
|
Recommended usage:
|
|
|
|
- local development with shared filesystem: `local`
|
|
- Kubernetes or any split-service deployment: `upload`
|
|
|
|
## Backend Configuration
|
|
|
|
Application properties:
|
|
|
|
```yaml
|
|
skillhub:
|
|
security:
|
|
scanner:
|
|
enabled: false
|
|
base-url: http://localhost:8000
|
|
health-path: /health
|
|
scan-path: /scan-upload
|
|
mode: local
|
|
connect-timeout-ms: 5000
|
|
read-timeout-ms: 900000
|
|
retry-max-attempts: 3
|
|
stream:
|
|
key: skillhub:scan:requests
|
|
group: skillhub-scanners
|
|
reclaim-min-idle: PT16M
|
|
max-unavailable-age: PT1H
|
|
```
|
|
|
|
Important environment variables:
|
|
|
|
- `SKILLHUB_SECURITY_SCANNER_ENABLED`
|
|
- `SKILLHUB_SECURITY_SCANNER_URL`
|
|
- `SKILLHUB_SECURITY_SCANNER_MODE`
|
|
- `SKILLHUB_SECURITY_SCANNER_READ_TIMEOUT`
|
|
- `SKILLHUB_SCAN_STREAM_KEY`
|
|
- `SKILLHUB_SCAN_STREAM_GROUP`
|
|
- `SKILLHUB_SCAN_STREAM_RECLAIM_MIN_IDLE`
|
|
- `SKILLHUB_SECURITY_STREAM_MAX_UNAVAILABLE_AGE`
|
|
|
|
Scanner-side optional environment variables:
|
|
|
|
- `SKILL_SCANNER_LLM_API_KEY`
|
|
- `SKILL_SCANNER_LLM_BASE_URL`
|
|
- `SKILL_SCANNER_LLM_MODEL`
|
|
- `SKILLHUB_SCANNER_MAX_CONCURRENT_SCANS` (default `1`)
|
|
- `SKILLHUB_SCANNER_HARD_TIMEOUT_SECONDS` (default `930`)
|
|
|
|
If the LLM variables are absent, the scanner should still run with non-LLM analyzers.
|
|
The default timeout ordering is server read timeout (900 seconds), scanner hard timeout
|
|
(930 seconds), then pending-message reclaim (960 seconds). A hard timeout exits the scanner process
|
|
with status `124`; Compose or Kubernetes restarts it and the Redis pending task is retried.
|
|
|
|
## Kubernetes Notes
|
|
|
|
Current repository manifests assume **separate** `skillhub-server` and `skillhub-scanner` deployments.
|
|
Because these deployments do not share a writable package directory, Kubernetes should use:
|
|
|
|
```text
|
|
SKILLHUB_SECURITY_SCANNER_MODE=upload
|
|
SKILLHUB_SECURITY_SCANNER_URL=http://skillhub-scanner:8000
|
|
```
|
|
|
|
Relevant manifests:
|
|
|
|
- `deploy/k8s/scanner-deployment.yaml`
|
|
- `deploy/k8s/services.yaml`
|
|
- `deploy/k8s/backend-deployment.yaml`
|
|
- `deploy/k8s/configmap.yaml`
|
|
|
|
The scanner service is internal-only by default and is consumed by the backend through cluster DNS.
|
|
|
|
## Verification
|
|
|
|
Verify the scanner service itself:
|
|
|
|
```bash
|
|
sh scripts/verify-scanner.sh http://localhost:8000
|
|
sh scripts/verify-scanner.sh http://localhost:8000 /path/to/skill.zip
|
|
```
|
|
|
|
Recommended backend checks after enabling the feature:
|
|
|
|
1. publish a test package
|
|
2. confirm the version status becomes `SCANNING`
|
|
3. confirm a `security_audit` row is created
|
|
4. confirm the version eventually moves to `PENDING_REVIEW` or `SCAN_FAILED`
|
|
5. call `GET /api/v1/skills/{skillId}/versions/{versionId}/security-audit`
|
|
|
|
## Audit Query API
|
|
|
|
Backend audit data is available from:
|
|
|
|
```text
|
|
GET /api/v1/skills/{skillId}/versions/{versionId}/security-audit
|
|
```
|
|
|
|
Response fields include:
|
|
|
|
- `scanId`
|
|
- `scannerType`
|
|
- `verdict`
|
|
- `isSafe`
|
|
- `maxSeverity`
|
|
- `findingsCount`
|
|
- `findings`
|
|
- `scanDurationSeconds`
|
|
- `scannedAt`
|
|
- `createdAt`
|
|
|
|
## Failure Semantics
|
|
|
|
- scan task retries are handled by `AbstractStreamConsumer`
|
|
- scanner connection failures, HTTP 429, and HTTP 5xx remain pending for automatic recovery
|
|
- unavailable tasks older than `max-unavailable-age` are marked `SCAN_FAILED`, acknowledged, and removed from the Redis Stream
|
|
- the timeout is evaluated during pending reclaim; terminal handling can occur roughly one `reclaim-min-idle` plus one `reclaim-interval` after the configured age
|
|
- terminal failures retain a failure reason in the security audit response for operators and authorized users
|
|
- other final failures mark the version as `SCAN_FAILED` after retry exhaustion
|
|
- even after scan failure, a review task is still created so the package does not get stuck forever
|
|
|
|
This keeps the existing human review path intact while making scanner failures visible.
|