mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-10 08:28:25 +00:00
feat(skills): add optional manual security scans (#50194)
This commit is contained in:
tin-berri
GitHub
parent
639625d679
commit
79e63d3e06
24 changed files with 1811 additions and 2 deletions
60
cookbook/skill_scanner/README.md
Normal file
60
cookbook/skill_scanner/README.md
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
# Manual skill security scans
|
||||
|
||||
This standalone adapter runs NVIDIA SkillSpector on a separate machine. It is not installed in the LiteLLM proxy and does not require a shared Helm or Compose deployment
|
||||
|
||||
Start it with Python 3.12 or newer and uv. The script pins SkillSpector and its HTTP dependencies
|
||||
|
||||
```sh
|
||||
export SKILL_SCANNER_API_KEY="$(openssl rand -hex 32)"
|
||||
uv run server.py
|
||||
```
|
||||
|
||||
For another host, place the loopback service behind an HTTPS reverse proxy that forwards the Authorization header and enforces a 65 MiB request limit. Run it under an unprivileged account with a writable temporary directory and no production files or credentials. The scanner host needs Git and network access to requested sources and optional model providers. Apply network egress restrictions suitable for those sources
|
||||
|
||||
Configure the same scanner secret in the gateway's environment, then add this to LiteLLM's configuration
|
||||
|
||||
```yaml
|
||||
general_settings:
|
||||
skill_scanner:
|
||||
endpoint: https://scanner.example.com/scan
|
||||
api_key_env: SKILL_SCANNER_API_KEY
|
||||
use_llm: false
|
||||
timeout_seconds: 180
|
||||
```
|
||||
|
||||
The endpoint must be reachable from the gateway. A customer's laptop works while it is reachable through their network, VPN or authenticated HTTPS tunnel. `localhost` refers to the gateway machine, not the admin's browser. HTTP is accepted only for loopback development
|
||||
|
||||
Open Skills, Submissions, and Review as a proxy admin. Click **Scan for threats**, inspect the report, then independently approve or reject the proposal. Scanning is optional and never changes approval permissions or blocks publication. The report shows findings, locations, remediation, risk score, scanner version and whether semantic analysis actually ran. Incomplete reports do not establish safety
|
||||
|
||||
Uploaded ZIPs are sent privately to the scanner with their exact stored bytes. Metadata-only edits reuse the matching stored archive. If that archive is missing or changed, submit a new ZIP before scanning. The displayed SHA-256 identifies that archive, and the scan examines its files rather than proposed catalog metadata. Remote URL reports describe the source at scan time and do not freeze its contents. GitHub branch/tag refs and explicit-ref subdirectories are supported. Other ref/subdirectory combinations require a ZIP so the adapter does not silently scan the wrong revision
|
||||
|
||||
The latest scan result is retained with the submission. Running scans have a deadline. If a gateway restarts or the scanner becomes unavailable, the attempt expires and an admin can retry. Retries replace the latest report. Approving or rejecting during a scan does not cancel it or change the decision
|
||||
|
||||
Set `use_llm: true` to request semantic analysis and configure the provider on the scanner machine using [SkillSpector's provider instructions](https://github.com/NVIDIA/skillspector#llm-analysis). Provider credentials stay on that machine. Static analysis can also call OSV for dependency checks
|
||||
|
||||
## HTTP contract
|
||||
|
||||
The gateway sends an authenticated `POST` to the configured endpoint. The body contains form fields `source` (JSON source descriptor), `use_llm` (boolean), and `expires_at` (timezone-aware ISO timestamp), plus `file` (ZIP) when bytes are available. The service returns HTTP 200 with this JSON shape
|
||||
|
||||
```json
|
||||
{
|
||||
"scanner_version": "2.12.0",
|
||||
"complete": true,
|
||||
"scan_mode": "static-only",
|
||||
"risk_score": 70,
|
||||
"findings": [{
|
||||
"rule_id": "example-rule",
|
||||
"severity": "HIGH",
|
||||
"message": "Potential data exfiltration",
|
||||
"file": "SKILL.md",
|
||||
"start_line": 8,
|
||||
"explanation": "Instructions request sending local files to an external host",
|
||||
"remediation": "Remove the external upload instruction"
|
||||
}],
|
||||
"limitations": []
|
||||
}
|
||||
```
|
||||
|
||||
Reports are limited to 1 MiB and 500 findings. Scanner uploads allow 64 MiB because the gateway can expand compressed skill uploads into stored compatibility ZIPs. The gateway rejects malformed reports, refuses redirects and verifies TLS. Only the configured scanner credential is sent. A timeout, HTTP error or incomplete report remains visible to the reviewer
|
||||
|
||||
The bundled adapter runs at most two child scans concurrently. Each child and its descendants are terminated at the gateway deadline, capped at 570 seconds. Keep the gateway and scanner clocks synchronized. Direct adapter clients may omit `expires_at` to use the cap. Child processes never execute submitted skill scripts. Raise the gateway deadline up to 600 seconds when needed for semantic analysis and configure the reverse proxy timeout to match
|
||||
172
cookbook/skill_scanner/server.py
Normal file
172
cookbook/skill_scanner/server.py
Normal file
|
|
@ -0,0 +1,172 @@
|
|||
# /// script
|
||||
# requires-python = ">=3.12"
|
||||
# dependencies = [
|
||||
# "fastapi==0.162.0", "uvicorn==0.54.0", "python-multipart==0.0.33",
|
||||
# "skillspector @ git+https://github.com/NVIDIA/skillspector.git@62c2fe8622f580ac292d4078c3b71f790194dbbd",
|
||||
# ]
|
||||
# ///
|
||||
import asyncio
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import tempfile
|
||||
from contextlib import suppress
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Annotated, Final
|
||||
from urllib.parse import quote, urlsplit
|
||||
|
||||
from fastapi import Depends, FastAPI, File, Form, HTTPException, UploadFile
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
from pydantic import AwareDatetime, TypeAdapter, ValidationError
|
||||
|
||||
MAX_UPLOAD: Final = 64 * 1024 * 1024
|
||||
MAX_REPORT: Final = 1024 * 1024
|
||||
SCAN_TIMEOUT: Final = 570
|
||||
SOURCE: Final = TypeAdapter(dict[str, str])
|
||||
SLOTS: Final = asyncio.Semaphore(2)
|
||||
app: Final = FastAPI(title="SkillSpector adapter", docs_url=None, redoc_url=None, openapi_url=None)
|
||||
|
||||
|
||||
def authenticate(credentials: Annotated[HTTPAuthorizationCredentials, Depends(HTTPBearer())]) -> None:
|
||||
token: Final = os.environ.get("SKILL_SCANNER_API_KEY", "")
|
||||
if not token or not hmac.compare_digest(credentials.credentials, token):
|
||||
raise HTTPException(401, "Invalid scanner credentials")
|
||||
|
||||
|
||||
def source_target(source: dict[str, str]) -> str:
|
||||
target: Final = (
|
||||
"https://github.com/" + source.get("repo", "") if source.get("source") == "github" else source.get("url", "")
|
||||
)
|
||||
parsed: Final = urlsplit(target)
|
||||
if parsed.scheme != "https" or not parsed.hostname or parsed.username or parsed.password or parsed.fragment:
|
||||
raise HTTPException(400, "Remote sources require an HTTPS URL without credentials or a fragment")
|
||||
ref: Final = source.get("ref")
|
||||
subdir: Final = source.get("path")
|
||||
if ref or subdir:
|
||||
if parsed.hostname != "github.com" or len(parsed.path.strip("/").split("/")) != 2 or not ref:
|
||||
raise HTTPException(400, "Upload a ZIP to scan this source's exact ref and subdirectory")
|
||||
suffix: Final = "/" + quote(subdir, safe="/") if subdir else ""
|
||||
return target.removesuffix(".git").rstrip("/") + "/tree/" + quote(ref, safe="/") + suffix
|
||||
return target
|
||||
|
||||
|
||||
async def worker(target: str, output: Path, use_llm: bool) -> None:
|
||||
from skillspector.mcp_server import run_scan
|
||||
|
||||
result: Final = await run_scan(target, use_llm=use_llm, allow_local_targets=not target.startswith("https://"))
|
||||
coverage: Final = result.get("analysis_completeness", {})
|
||||
findings: Final = result["findings"]
|
||||
complete: Final = (
|
||||
result.get("execution_successful") is True
|
||||
and coverage.get("is_complete") is True
|
||||
and (not use_llm or (result.get("llm_available") is True and result.get("llm_used") is True))
|
||||
and len(findings) <= 500
|
||||
)
|
||||
limitations: Final = tuple(str(value)[:2000] for value in coverage.get("limitations", ()))
|
||||
report: Final = {
|
||||
"scanner_version": result["version"],
|
||||
"complete": complete,
|
||||
"scan_mode": result["scan_mode"],
|
||||
"risk_score": result["risk_score"],
|
||||
"findings": [
|
||||
{
|
||||
"rule_id": finding["id"],
|
||||
"severity": finding["severity"],
|
||||
"message": finding.get("pattern") or finding.get("finding") or "Security finding",
|
||||
"file": finding["location"]["file"],
|
||||
"start_line": finding["location"].get("start_line"),
|
||||
"explanation": finding.get("explanation"),
|
||||
"remediation": finding.get("remediation"),
|
||||
}
|
||||
for finding in findings[:500]
|
||||
],
|
||||
"limitations": (
|
||||
*limitations,
|
||||
*(("Only the first 500 findings are displayed.",) if len(findings) > 500 else ()),
|
||||
),
|
||||
}
|
||||
encoded: Final = json.dumps(report).encode()
|
||||
if len(encoded) > MAX_REPORT:
|
||||
raise ValueError("Scanner report exceeded the response limit")
|
||||
output.write_bytes(encoded)
|
||||
|
||||
|
||||
async def run_worker(target: str, output: Path, use_llm: bool, expires_at: datetime | None = None) -> dict[str, object]:
|
||||
timeout: Final = (
|
||||
SCAN_TIMEOUT
|
||||
if expires_at is None
|
||||
else min(SCAN_TIMEOUT, (expires_at - datetime.now(timezone.utc)).total_seconds())
|
||||
)
|
||||
if timeout <= 0:
|
||||
raise HTTPException(504, "SkillSpector scan deadline expired")
|
||||
process: Final = await asyncio.create_subprocess_exec(
|
||||
sys.executable,
|
||||
str(Path(__file__).resolve()),
|
||||
"--scan",
|
||||
target,
|
||||
str(output),
|
||||
str(use_llm).lower(),
|
||||
stdout=asyncio.subprocess.DEVNULL,
|
||||
stderr=asyncio.subprocess.DEVNULL,
|
||||
start_new_session=True,
|
||||
env={
|
||||
**os.environ,
|
||||
"HOME": str(output.parent),
|
||||
"TMPDIR": str(output.parent),
|
||||
"GIT_CONFIG_GLOBAL": os.devnull,
|
||||
"GIT_CONFIG_NOSYSTEM": "1",
|
||||
"GIT_TERMINAL_PROMPT": "0",
|
||||
"GIT_ALLOW_PROTOCOL": "https",
|
||||
},
|
||||
)
|
||||
try:
|
||||
code: Final = await asyncio.wait_for(process.wait(), timeout)
|
||||
if code != 0 or not output.is_file() or output.stat().st_size > MAX_REPORT:
|
||||
raise HTTPException(502, "SkillSpector could not produce a report for this source")
|
||||
return TypeAdapter(dict[str, object]).validate_json(output.read_bytes())
|
||||
except TimeoutError:
|
||||
raise HTTPException(504, "SkillSpector scan timed out") from None
|
||||
finally:
|
||||
with suppress(ProcessLookupError):
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
await process.wait()
|
||||
|
||||
|
||||
@app.post("/scan", dependencies=[Depends(authenticate)])
|
||||
async def scan(
|
||||
source: Annotated[str, Form(max_length=16384)],
|
||||
use_llm: Annotated[bool, Form()] = False,
|
||||
expires_at: Annotated[AwareDatetime | None, Form()] = None,
|
||||
file: Annotated[UploadFile | None, File()] = None,
|
||||
) -> dict[str, object]:
|
||||
if SLOTS.locked():
|
||||
raise HTTPException(429, "Scanner is busy; try again shortly")
|
||||
try:
|
||||
parsed: Final = SOURCE.validate_json(source)
|
||||
except ValidationError:
|
||||
raise HTTPException(400, "Invalid source metadata") from None
|
||||
async with SLOTS:
|
||||
with tempfile.TemporaryDirectory(prefix="skill-scan-") as directory:
|
||||
root: Final = Path(directory)
|
||||
if file is not None:
|
||||
content: Final = await file.read(MAX_UPLOAD + 1)
|
||||
if len(content) > MAX_UPLOAD:
|
||||
raise HTTPException(413, "Upload exceeds 64 MiB")
|
||||
archive: Final = root / "skill.zip"
|
||||
archive.write_bytes(content)
|
||||
return await run_worker(str(archive), root / "report.json", use_llm, expires_at)
|
||||
return await run_worker(source_target(parsed), root / "report.json", use_llm, expires_at)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) > 1 and sys.argv[1] == "--scan":
|
||||
asyncio.run(worker(sys.argv[2], Path(sys.argv[3]), sys.argv[4] == "true"))
|
||||
else:
|
||||
import uvicorn
|
||||
|
||||
if not os.environ.get("SKILL_SCANNER_API_KEY"):
|
||||
raise SystemExit("Set SKILL_SCANNER_API_KEY before starting the scanner")
|
||||
uvicorn.run(app, host="127.0.0.1", port=int(os.environ.get("PORT", "8099")))
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
SET LOCAL lock_timeout = '10s';
|
||||
|
||||
ALTER TABLE "LiteLLM_SkillSubmission" ADD COLUMN IF NOT EXISTS "scan_json" TEXT;
|
||||
|
|
@ -1652,6 +1652,7 @@ model LiteLLM_SkillSubmission {
|
|||
reviewed_by String?
|
||||
reviewed_at DateTime?
|
||||
rejection_reason String?
|
||||
scan_json String?
|
||||
|
||||
@@index([submitted_by, created_at(sort: Desc), id(sort: Desc)])
|
||||
@@index([submitted_by, status, created_at(sort: Desc), id(sort: Desc)])
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ from litellm.models.plugin_upload import (
|
|||
PluginPublicationError,
|
||||
PublishedPlugin,
|
||||
)
|
||||
from litellm.types.proxy.skill_scanning import SkillScan
|
||||
|
||||
SubmissionStatus: TypeAlias = Literal["pending_review", "approved", "rejected"]
|
||||
SubmissionOperation: TypeAlias = Literal["create", "update"]
|
||||
|
|
@ -70,7 +71,22 @@ class SubmissionReviewed:
|
|||
SubmissionReviewResult: TypeAlias = SubmissionReviewed | SubmissionMissing | SubmissionConflict | PluginPublicationError
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class SkillScanInput:
|
||||
scan: SkillScan
|
||||
source: Mapping[str, str]
|
||||
content: bytes | None
|
||||
|
||||
|
||||
class SkillSubmissionStore(Protocol):
|
||||
async def get_scan(self, submission_id: str) -> SkillScan | SubmissionMissing | None: ...
|
||||
|
||||
async def start_scan(
|
||||
self, submission_id: str, scan: SkillScan
|
||||
) -> SkillScanInput | SubmissionMissing | SubmissionConflict: ...
|
||||
|
||||
async def finish_scan(self, submission_id: str, scan: SkillScan) -> None: ...
|
||||
|
||||
async def submit(
|
||||
self, upload: PluginPublication, submitted_by: str, base: PluginPublicationBase | None
|
||||
) -> SkillSubmission | SubmissionLimitExceeded: ...
|
||||
|
|
|
|||
|
|
@ -50,9 +50,11 @@ def _mount_app(prefix: str, attr_name: str = "app") -> Callable[["FastAPI", obje
|
|||
|
||||
def _include_skill_hub(app: "FastAPI", module: object) -> None:
|
||||
from litellm.proxy.discovery_endpoints.skill_hub_submissions import router
|
||||
from litellm.proxy.discovery_endpoints.skill_scanning import router as scan_router
|
||||
|
||||
_include_router()(app, module)
|
||||
app.include_router(router)
|
||||
app.include_router(scan_router)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
|
|||
|
|
@ -59741,6 +59741,248 @@
|
|||
"title": "SkillHubVersions",
|
||||
"type": "object"
|
||||
},
|
||||
"SkillScan": {
|
||||
"properties": {
|
||||
"archive_sha256": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"title": "Archive Sha256"
|
||||
},
|
||||
"completed_at": {
|
||||
"anyOf": [
|
||||
{
|
||||
"format": "date-time",
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"title": "Completed At"
|
||||
},
|
||||
"error": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"title": "Error"
|
||||
},
|
||||
"expires_at": {
|
||||
"format": "date-time",
|
||||
"title": "Expires At",
|
||||
"type": "string"
|
||||
},
|
||||
"id": {
|
||||
"title": "Id",
|
||||
"type": "string"
|
||||
},
|
||||
"report": {
|
||||
"anyOf": [
|
||||
{
|
||||
"$ref": "#/components/schemas/SkillScanReport"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
},
|
||||
"requested_by": {
|
||||
"title": "Requested By",
|
||||
"type": "string"
|
||||
},
|
||||
"started_at": {
|
||||
"format": "date-time",
|
||||
"title": "Started At",
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"enum": [
|
||||
"running",
|
||||
"completed",
|
||||
"failed"
|
||||
],
|
||||
"title": "Status",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"id",
|
||||
"status",
|
||||
"requested_by",
|
||||
"started_at",
|
||||
"expires_at"
|
||||
],
|
||||
"title": "SkillScan",
|
||||
"type": "object"
|
||||
},
|
||||
"SkillScanFinding": {
|
||||
"properties": {
|
||||
"explanation": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 8000,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"title": "Explanation"
|
||||
},
|
||||
"file": {
|
||||
"maxLength": 2048,
|
||||
"title": "File",
|
||||
"type": "string"
|
||||
},
|
||||
"message": {
|
||||
"maxLength": 8000,
|
||||
"title": "Message",
|
||||
"type": "string"
|
||||
},
|
||||
"remediation": {
|
||||
"anyOf": [
|
||||
{
|
||||
"maxLength": 8000,
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"title": "Remediation"
|
||||
},
|
||||
"rule_id": {
|
||||
"maxLength": 200,
|
||||
"title": "Rule Id",
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"enum": [
|
||||
"LOW",
|
||||
"MEDIUM",
|
||||
"HIGH",
|
||||
"CRITICAL"
|
||||
],
|
||||
"title": "Severity",
|
||||
"type": "string"
|
||||
},
|
||||
"start_line": {
|
||||
"anyOf": [
|
||||
{
|
||||
"minimum": 1.0,
|
||||
"type": "integer"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"title": "Start Line"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"rule_id",
|
||||
"severity",
|
||||
"message",
|
||||
"file"
|
||||
],
|
||||
"title": "SkillScanFinding",
|
||||
"type": "object"
|
||||
},
|
||||
"SkillScanReport": {
|
||||
"properties": {
|
||||
"complete": {
|
||||
"title": "Complete",
|
||||
"type": "boolean"
|
||||
},
|
||||
"findings": {
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/SkillScanFinding"
|
||||
},
|
||||
"maxItems": 500,
|
||||
"title": "Findings",
|
||||
"type": "array"
|
||||
},
|
||||
"limitations": {
|
||||
"default": [],
|
||||
"items": {
|
||||
"maxLength": 2000,
|
||||
"type": "string"
|
||||
},
|
||||
"maxItems": 100,
|
||||
"title": "Limitations",
|
||||
"type": "array"
|
||||
},
|
||||
"risk_score": {
|
||||
"maximum": 100.0,
|
||||
"minimum": 0.0,
|
||||
"title": "Risk Score",
|
||||
"type": "integer"
|
||||
},
|
||||
"scan_mode": {
|
||||
"enum": [
|
||||
"static-only",
|
||||
"static+llm"
|
||||
],
|
||||
"title": "Scan Mode",
|
||||
"type": "string"
|
||||
},
|
||||
"scanner_version": {
|
||||
"maxLength": 100,
|
||||
"minLength": 1,
|
||||
"title": "Scanner Version",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"scanner_version",
|
||||
"complete",
|
||||
"scan_mode",
|
||||
"risk_score",
|
||||
"findings"
|
||||
],
|
||||
"title": "SkillScanReport",
|
||||
"type": "object"
|
||||
},
|
||||
"SkillScanRequest": {
|
||||
"additionalProperties": false,
|
||||
"properties": {},
|
||||
"title": "SkillScanRequest",
|
||||
"type": "object"
|
||||
},
|
||||
"SkillScanResponse": {
|
||||
"properties": {
|
||||
"configured": {
|
||||
"title": "Configured",
|
||||
"type": "boolean"
|
||||
},
|
||||
"scan": {
|
||||
"anyOf": [
|
||||
{
|
||||
"$ref": "#/components/schemas/SkillScan"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"configured",
|
||||
"scan"
|
||||
],
|
||||
"title": "SkillScanResponse",
|
||||
"type": "object"
|
||||
},
|
||||
"ValidationError": {
|
||||
"properties": {
|
||||
"ctx": {
|
||||
|
|
@ -60890,6 +61132,117 @@
|
|||
"skill_hub"
|
||||
]
|
||||
}
|
||||
},
|
||||
"/skill-hub/submissions/{submission_id}/scan": {
|
||||
"get": {
|
||||
"description": "Read the latest manual security scan. Only proxy admins may read reports.",
|
||||
"operationId": "get_skill_scan_skill_hub_submissions__submission_id__scan_get",
|
||||
"parameters": [
|
||||
{
|
||||
"in": "path",
|
||||
"name": "submission_id",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"title": "Submission Id",
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SkillScanResponse"
|
||||
}
|
||||
}
|
||||
},
|
||||
"description": "Successful Response"
|
||||
},
|
||||
"422": {
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/HTTPValidationError"
|
||||
}
|
||||
}
|
||||
},
|
||||
"description": "Validation Error"
|
||||
}
|
||||
},
|
||||
"security": [
|
||||
{
|
||||
"APIKeyHeader": []
|
||||
}
|
||||
],
|
||||
"summary": "Get Skill Scan",
|
||||
"tags": [
|
||||
"skill_hub"
|
||||
]
|
||||
},
|
||||
"post": {
|
||||
"description": "Start an optional scan of a pending submission. Scans never change approval decisions.",
|
||||
"operationId": "start_skill_scan_skill_hub_submissions__submission_id__scan_post",
|
||||
"parameters": [
|
||||
{
|
||||
"in": "path",
|
||||
"name": "submission_id",
|
||||
"required": true,
|
||||
"schema": {
|
||||
"title": "Submission Id",
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"requestBody": {
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"anyOf": [
|
||||
{
|
||||
"$ref": "#/components/schemas/SkillScanRequest"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"title": " Request"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/SkillScanResponse"
|
||||
}
|
||||
}
|
||||
},
|
||||
"description": "Successful Response"
|
||||
},
|
||||
"422": {
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/HTTPValidationError"
|
||||
}
|
||||
}
|
||||
},
|
||||
"description": "Validation Error"
|
||||
}
|
||||
},
|
||||
"security": [
|
||||
{
|
||||
"APIKeyHeader": []
|
||||
}
|
||||
],
|
||||
"summary": "Start Skill Scan",
|
||||
"tags": [
|
||||
"skill_hub"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
|
|
|
|||
|
|
@ -88,6 +88,7 @@ from litellm.types.proxy.carried_budget_state import (
|
|||
)
|
||||
from litellm.types.proxy.control_plane_endpoints import WorkerRegistryEntry
|
||||
from litellm.types.proxy.request_tag_mappings import RequestTagMapping
|
||||
from litellm.types.proxy.skill_scanning import SkillScannerConfig
|
||||
from litellm.types.proxy.spend_capture_rate import SpendCaptureRateCheckSettings
|
||||
from litellm.types.router import AllowedModelRegion, RouterErrors, UpdateRouterConfig
|
||||
from litellm.types.router_weights import validate_router_settings_dict
|
||||
|
|
@ -467,6 +468,7 @@ class LiteLLMRoutes(enum.Enum):
|
|||
"/skill-hub/submissions/{submission_id}/archive",
|
||||
"/skill-hub/submissions/{submission_id}/approve",
|
||||
"/skill-hub/submissions/{submission_id}/reject",
|
||||
"/skill-hub/submissions/{submission_id}/scan",
|
||||
)
|
||||
|
||||
litellm_native_routes = [
|
||||
|
|
@ -3041,6 +3043,9 @@ class ConfigGeneralSettings(LiteLLMPydanticObjectBase):
|
|||
None,
|
||||
description="If True, forwards client headers (e.g. Authorization) to the LLM API. Required for Claude Code with Max subscription.",
|
||||
)
|
||||
skill_scanner: SkillScannerConfig | None = Field(
|
||||
None, description="Opt-in external scanner for manual admin review of skill submissions"
|
||||
)
|
||||
mcp_required_fields: list[str] | None = Field(
|
||||
None,
|
||||
description="List of MCP server fields that must be filled in for a submission to pass standards checks (e.g. ['description', 'source_url', 'alias']).",
|
||||
|
|
|
|||
186
litellm/proxy/discovery_endpoints/skill_scanning.py
Normal file
186
litellm/proxy/discovery_endpoints/skill_scanning.py
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
import asyncio
|
||||
import io
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Annotated, Final
|
||||
from uuid import uuid4
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, BackgroundTasks, Body, Depends, HTTPException
|
||||
from pydantic import ValidationError
|
||||
|
||||
from litellm.llms.custom_httpx.http_handler import (
|
||||
get_async_httpx_client, # pyright: ignore[reportUnknownVariableType] # shared HTTP factory has untyped optional params
|
||||
)
|
||||
from litellm.models.skill_submission import SkillScanInput, SkillSubmissionStore, SubmissionConflict, SubmissionMissing
|
||||
from litellm.proxy._types import UserAPIKeyAuth
|
||||
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
|
||||
from litellm.proxy.common_utils.resource_ownership import is_proxy_admin
|
||||
from litellm.proxy.deps.repositories import SkillSubmissionRepositoryDep
|
||||
from litellm.secret_managers.main import get_secret_str
|
||||
from litellm.types.proxy.skill_scanning import (
|
||||
SkillScan,
|
||||
SkillScannerConfig,
|
||||
SkillScanReport,
|
||||
SkillScanRequest,
|
||||
SkillScanResponse,
|
||||
)
|
||||
|
||||
router: Final = APIRouter()
|
||||
_MAX_REPORT_BYTES: Final = 1024 * 1024
|
||||
|
||||
|
||||
def scanner_config() -> SkillScannerConfig | str | None:
|
||||
from litellm.proxy.proxy_server import general_settings_view
|
||||
|
||||
value: Final = general_settings_view().get("skill_scanner")
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
return SkillScannerConfig.model_validate(value)
|
||||
except ValidationError:
|
||||
return "The skill scanner configuration is invalid"
|
||||
|
||||
|
||||
async def scan_report(
|
||||
client: httpx.AsyncClient, config: SkillScannerConfig, api_key: str, target: SkillScanInput
|
||||
) -> SkillScanReport | str:
|
||||
try:
|
||||
async with client.stream(
|
||||
"POST",
|
||||
config.endpoint,
|
||||
headers={"Authorization": f"Bearer {api_key}", "Accept-Encoding": "identity"},
|
||||
data={
|
||||
"source": json.dumps(dict(target.source)),
|
||||
"use_llm": str(config.use_llm).lower(),
|
||||
"expires_at": target.scan.expires_at.isoformat(),
|
||||
},
|
||||
files=None if target.content is None else {"file": ("skill.zip", target.content, "application/zip")},
|
||||
timeout=httpx.Timeout(config.timeout_seconds, connect=10),
|
||||
follow_redirects=False,
|
||||
) as response:
|
||||
if response.status_code != 200:
|
||||
return f"Scanner returned HTTP {response.status_code}. Check the scanner service and authentication."
|
||||
output: Final = io.BytesIO()
|
||||
async for chunk in response.aiter_bytes(chunk_size=65536):
|
||||
if output.tell() + len(chunk) > _MAX_REPORT_BYTES:
|
||||
return "Scanner report exceeded the 1 MiB limit."
|
||||
output.write(chunk)
|
||||
report: Final = SkillScanReport.model_validate_json(output.getvalue())
|
||||
if config.use_llm and report.scan_mode != "static+llm":
|
||||
return report.model_copy(
|
||||
update={
|
||||
"complete": False,
|
||||
"limitations": (*report.limitations[:99], "Requested semantic analysis did not run."),
|
||||
}
|
||||
)
|
||||
return report
|
||||
except (httpx.HTTPError, ValidationError):
|
||||
return "The scanner was unreachable or returned an invalid report."
|
||||
|
||||
|
||||
async def complete_scan(
|
||||
submissions: SkillSubmissionStore,
|
||||
submission_id: str,
|
||||
target: SkillScanInput,
|
||||
config: SkillScannerConfig,
|
||||
api_key: str,
|
||||
) -> None:
|
||||
try:
|
||||
client: Final = get_async_httpx_client(
|
||||
"skill_scanner", params={"ssl_verify": True, "follow_redirects": False}
|
||||
).client
|
||||
report: Final = await asyncio.wait_for(
|
||||
scan_report(client, config, api_key, target),
|
||||
timeout=(target.scan.expires_at - datetime.now(timezone.utc)).total_seconds(),
|
||||
)
|
||||
completed: Final = target.scan.model_copy(
|
||||
update={
|
||||
"status": "completed" if isinstance(report, SkillScanReport) else "failed",
|
||||
"report": report if isinstance(report, SkillScanReport) else None,
|
||||
"error": report if isinstance(report, str) else None,
|
||||
"completed_at": datetime.now(timezone.utc),
|
||||
}
|
||||
)
|
||||
await submissions.finish_scan(submission_id, completed)
|
||||
except (TimeoutError, asyncio.CancelledError) as error:
|
||||
await submissions.finish_scan(
|
||||
submission_id,
|
||||
target.scan.model_copy(
|
||||
update={
|
||||
"status": "failed",
|
||||
"error": "Scan timed out or was interrupted. Try scanning again.",
|
||||
"completed_at": datetime.now(timezone.utc),
|
||||
}
|
||||
),
|
||||
)
|
||||
if isinstance(error, asyncio.CancelledError):
|
||||
raise
|
||||
|
||||
|
||||
@router.get(
|
||||
"/skill-hub/submissions/{submission_id}/scan",
|
||||
tags=["Skills Hub"],
|
||||
response_model=SkillScanResponse,
|
||||
)
|
||||
async def get_skill_scan(
|
||||
submission_id: str,
|
||||
submissions: SkillSubmissionRepositoryDep,
|
||||
caller: Annotated[UserAPIKeyAuth, Depends(user_api_key_auth)],
|
||||
) -> SkillScanResponse:
|
||||
"""Read the latest manual security scan. Only proxy admins may read reports."""
|
||||
if not is_proxy_admin(caller):
|
||||
raise HTTPException(403, "Proxy admin access is required to review security scans")
|
||||
if submissions is None:
|
||||
raise HTTPException(503, "Database is not connected")
|
||||
scan: Final = await submissions.get_scan(submission_id)
|
||||
if isinstance(scan, SubmissionMissing):
|
||||
raise HTTPException(404, "Skill submission not found")
|
||||
return SkillScanResponse(
|
||||
configured=isinstance(scanner_config(), SkillScannerConfig),
|
||||
scan=None if scan is None else scan.at(datetime.now(timezone.utc)),
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/skill-hub/submissions/{submission_id}/scan",
|
||||
tags=["Skills Hub"],
|
||||
response_model=SkillScanResponse,
|
||||
)
|
||||
async def start_skill_scan(
|
||||
submission_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
submissions: SkillSubmissionRepositoryDep,
|
||||
caller: Annotated[UserAPIKeyAuth, Depends(user_api_key_auth)],
|
||||
_request: Annotated[SkillScanRequest | None, Body()] = None,
|
||||
) -> SkillScanResponse:
|
||||
"""Start an optional scan of a pending submission. Scans never change approval decisions."""
|
||||
if not is_proxy_admin(caller):
|
||||
raise HTTPException(403, "Proxy admin access is required to scan skills")
|
||||
if submissions is None:
|
||||
raise HTTPException(503, "Database is not connected")
|
||||
config: Final = scanner_config()
|
||||
if config is None:
|
||||
raise HTTPException(503, "Security scanning is not configured")
|
||||
if isinstance(config, str):
|
||||
raise HTTPException(503, config)
|
||||
api_key: Final = get_secret_str(config.api_key_env)
|
||||
if not api_key:
|
||||
raise HTTPException(503, "The scanner authentication secret is not configured")
|
||||
now: Final = datetime.now(timezone.utc)
|
||||
target: Final = await submissions.start_scan(
|
||||
submission_id,
|
||||
SkillScan(
|
||||
id=str(uuid4()),
|
||||
status="running",
|
||||
requested_by=caller.user_id or "proxy-admin",
|
||||
started_at=now,
|
||||
expires_at=now + timedelta(seconds=config.timeout_seconds),
|
||||
),
|
||||
)
|
||||
if isinstance(target, SubmissionMissing):
|
||||
raise HTTPException(404, "Skill submission not found")
|
||||
if isinstance(target, SubmissionConflict):
|
||||
raise HTTPException(409, target.message)
|
||||
background_tasks.add_task(complete_scan, submissions, submission_id, target, config, api_key)
|
||||
return SkillScanResponse(configured=True, scan=target.scan)
|
||||
|
|
@ -1652,6 +1652,7 @@ model LiteLLM_SkillSubmission {
|
|||
reviewed_by String?
|
||||
reviewed_at DateTime?
|
||||
rejection_reason String?
|
||||
scan_json String?
|
||||
|
||||
@@index([submitted_by, created_at(sort: Desc), id(sort: Desc)])
|
||||
@@index([submitted_by, status, created_at(sort: Desc), id(sort: Desc)])
|
||||
|
|
|
|||
|
|
@ -1,9 +1,11 @@
|
|||
import hashlib
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from contextlib import AbstractAsyncContextManager
|
||||
from dataclasses import replace
|
||||
from datetime import datetime, timezone
|
||||
from typing import TYPE_CHECKING, Final, Protocol, cast # noqa: TID251 # Prisma's generated delegates are dynamic
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, TypeAdapter
|
||||
|
||||
|
|
@ -16,6 +18,7 @@ from litellm.models.plugin_upload import (
|
|||
)
|
||||
from litellm.models.skill_submission import (
|
||||
SkillProposal,
|
||||
SkillScanInput,
|
||||
SkillSubmission,
|
||||
SubmissionConflict,
|
||||
SubmissionLimitExceeded,
|
||||
|
|
@ -26,8 +29,9 @@ from litellm.models.skill_submission import (
|
|||
SubmissionReviewResult,
|
||||
SubmissionStatus,
|
||||
)
|
||||
from litellm.repositories.plugin_publication import PluginPublicationTransaction, publish_plugin
|
||||
from litellm.repositories.plugin_publication import PluginPublicationTransaction, publish_plugin, verified_archive
|
||||
from litellm.repositories.prisma_protocols import TableActions
|
||||
from litellm.types.proxy.skill_scanning import SkillScan
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from prisma import models as prisma_models
|
||||
|
|
@ -90,6 +94,7 @@ class _SubmissionRow(BaseModel):
|
|||
)
|
||||
|
||||
|
||||
_SCAN_ROWS: Final = TypeAdapter(tuple[Mapping[str, str | None], ...])
|
||||
_ROWS: Final = TypeAdapter(tuple[_SubmissionRow, ...])
|
||||
_DESCRIPTION: Final = TypeAdapter[str | None](str | None)
|
||||
_SOURCE: Final = TypeAdapter(dict[str, str])
|
||||
|
|
@ -125,6 +130,81 @@ class SkillSubmissionRepository:
|
|||
SubmissionDatabase, prisma_client.writer_db
|
||||
)
|
||||
|
||||
async def get_scan(self, submission_id: str) -> SkillScan | SubmissionMissing | None:
|
||||
rows: Final = _SCAN_ROWS.validate_python(
|
||||
await self._database.query_raw(
|
||||
'SELECT scan_json FROM "LiteLLM_SkillSubmission" WHERE id = $1', submission_id
|
||||
)
|
||||
)
|
||||
if not rows:
|
||||
return SubmissionMissing()
|
||||
value: Final = rows[0]["scan_json"]
|
||||
return None if value is None else SkillScan.model_validate_json(value)
|
||||
|
||||
async def start_scan(
|
||||
self, submission_id: str, scan: SkillScan
|
||||
) -> SkillScanInput | SubmissionMissing | SubmissionConflict:
|
||||
async with self._database.tx() as transaction:
|
||||
await transaction.query_raw(
|
||||
'SELECT id FROM "LiteLLM_SkillSubmission" WHERE id = $1 FOR UPDATE', submission_id
|
||||
)
|
||||
row: Final = await transaction.litellm_skillsubmission.find_unique(where={"id": submission_id})
|
||||
if row is None:
|
||||
return SubmissionMissing()
|
||||
if row.status != "pending_review":
|
||||
return SubmissionConflict("Only pending submissions can be scanned.")
|
||||
previous: Final = None if row.scan_json is None else SkillScan.model_validate_json(row.scan_json)
|
||||
if previous is not None and previous.at(scan.started_at).status == "running":
|
||||
return SubmissionConflict("A scan is already running for this submission.")
|
||||
stored: Final = _StoredProposal.model_validate_json(row.proposal_json)
|
||||
source: Final = _SOURCE.validate_python(stored.manifest.get("source", {}))
|
||||
inherited: Final = (
|
||||
await transaction.litellm_claudecodepluginupload.find_unique(where={"plugin_name": row.plugin_name})
|
||||
if row.archive_content is None and source.get("source") == "archive"
|
||||
else None
|
||||
)
|
||||
content: Final = (
|
||||
row.archive_content.decode()
|
||||
if row.archive_content is not None
|
||||
else verified_archive(
|
||||
None if inherited is None else inherited.content.decode(), json.dumps(stored.manifest)
|
||||
)
|
||||
)
|
||||
managed_archive: Final = urlsplit(source.get("url", "")).path.endswith(
|
||||
f"/claude-code/plugins/{row.plugin_name}/archive/{source.get('sha256', '')}.zip"
|
||||
)
|
||||
if source.get("source") == "archive" and managed_archive and content is None:
|
||||
return SubmissionConflict("The proposal's archive is no longer available. Submit a new ZIP to scan it.")
|
||||
started: Final = scan.model_copy(
|
||||
update={"archive_sha256": None if content is None else hashlib.sha256(content).hexdigest()}
|
||||
)
|
||||
await transaction.litellm_skillsubmission.update(
|
||||
where={"id": submission_id}, data={"scan_json": started.model_dump_json()}
|
||||
)
|
||||
return SkillScanInput(started, source, content)
|
||||
|
||||
async def finish_scan(self, submission_id: str, scan: SkillScan) -> None:
|
||||
async with self._database.tx() as transaction:
|
||||
rows: Final = _SCAN_ROWS.validate_python(
|
||||
await transaction.query_raw(
|
||||
'SELECT scan_json FROM "LiteLLM_SkillSubmission" WHERE id = $1 FOR UPDATE', submission_id
|
||||
)
|
||||
)
|
||||
value: Final = rows[0]["scan_json"] if rows else None
|
||||
if value is None:
|
||||
return
|
||||
current: Final = SkillScan.model_validate_json(value)
|
||||
if current.id != scan.id or current.status != "running":
|
||||
return
|
||||
result: Final = (
|
||||
current.at(scan.completed_at)
|
||||
if scan.completed_at is not None and scan.completed_at >= current.expires_at
|
||||
else scan
|
||||
)
|
||||
await transaction.litellm_skillsubmission.update(
|
||||
where={"id": submission_id}, data={"scan_json": result.model_dump_json()}
|
||||
)
|
||||
|
||||
async def submit(
|
||||
self, upload: PluginPublication, submitted_by: str, base: PluginPublicationBase | None
|
||||
) -> SkillSubmission | SubmissionLimitExceeded:
|
||||
|
|
|
|||
91
litellm/types/proxy/skill_scanning.py
Normal file
91
litellm/types/proxy/skill_scanning.py
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
from datetime import datetime
|
||||
from typing import Annotated, Final, Literal
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
|
||||
class SkillScannerConfig(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
endpoint: str = Field(max_length=2048, description="External scanner's authenticated POST endpoint")
|
||||
api_key_env: str = Field(
|
||||
pattern=r"^[A-Za-z_][A-Za-z0-9_]*$", description="Environment secret name for scanner auth"
|
||||
)
|
||||
use_llm: bool = False
|
||||
timeout_seconds: int = Field(default=180, ge=10, le=600)
|
||||
|
||||
@field_validator("endpoint")
|
||||
@classmethod
|
||||
def validate_endpoint(cls, value: str) -> str:
|
||||
parsed: Final = urlsplit(value)
|
||||
if (
|
||||
not parsed.hostname
|
||||
or parsed.username is not None
|
||||
or parsed.password is not None
|
||||
or parsed.query
|
||||
or parsed.fragment
|
||||
or not (
|
||||
parsed.scheme == "https"
|
||||
or (parsed.scheme == "http" and parsed.hostname in {"127.0.0.1", "::1", "localhost"})
|
||||
)
|
||||
):
|
||||
raise ValueError(
|
||||
"Scanner endpoint must use HTTPS without credentials, query or fragment (HTTP loopback allowed)"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
class SkillScanFinding(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="ignore")
|
||||
|
||||
rule_id: str = Field(max_length=200)
|
||||
severity: Literal["LOW", "MEDIUM", "HIGH", "CRITICAL"]
|
||||
message: str = Field(max_length=8000)
|
||||
file: str = Field(max_length=2048)
|
||||
start_line: int | None = Field(default=None, ge=1)
|
||||
explanation: str | None = Field(default=None, max_length=8000)
|
||||
remediation: str | None = Field(default=None, max_length=8000)
|
||||
|
||||
|
||||
class SkillScanReport(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="ignore")
|
||||
|
||||
scanner_version: str = Field(min_length=1, max_length=100)
|
||||
complete: bool
|
||||
scan_mode: Literal["static-only", "static+llm"]
|
||||
risk_score: int = Field(ge=0, le=100)
|
||||
findings: tuple[SkillScanFinding, ...] = Field(max_length=500)
|
||||
limitations: tuple[Annotated[str, Field(max_length=2000)], ...] = Field(default=(), max_length=100)
|
||||
|
||||
|
||||
class SkillScan(BaseModel):
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
id: str
|
||||
status: Literal["running", "completed", "failed"]
|
||||
requested_by: str
|
||||
started_at: datetime
|
||||
expires_at: datetime
|
||||
completed_at: datetime | None = None
|
||||
archive_sha256: str | None = None
|
||||
report: SkillScanReport | None = None
|
||||
error: str | None = None
|
||||
|
||||
def at(self, now: datetime) -> "SkillScan":
|
||||
if self.status == "running" and self.expires_at <= now:
|
||||
return self.model_copy(
|
||||
update={"status": "failed", "error": "Scan timed out or was interrupted. Try scanning again."}
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
class SkillScanRequest(BaseModel):
|
||||
model_config = ConfigDict(frozen=True, extra="forbid")
|
||||
|
||||
|
||||
class SkillScanResponse(BaseModel):
|
||||
model_config = ConfigDict(frozen=True)
|
||||
|
||||
configured: bool
|
||||
scan: SkillScan | None
|
||||
|
|
@ -1652,6 +1652,7 @@ model LiteLLM_SkillSubmission {
|
|||
reviewed_by String?
|
||||
reviewed_at DateTime?
|
||||
rejection_reason String?
|
||||
scan_json String?
|
||||
|
||||
@@index([submitted_by, created_at(sort: Desc), id(sort: Desc)])
|
||||
@@index([submitted_by, status, created_at(sort: Desc), id(sort: Desc)])
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ import os
|
|||
import uuid
|
||||
import zipfile
|
||||
from collections.abc import AsyncIterator, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, replace
|
||||
from typing import Final, Literal
|
||||
|
||||
import psycopg
|
||||
|
|
@ -371,3 +371,84 @@ async def test_submission_pages_filter_owner_and_status_before_limit(pending_upd
|
|||
global_pending: Final = await case.submissions.list(None, "pending_review", 3, 0)
|
||||
all_pending: Final = tuple(sorted((*own_pending, submitted[1].id, submitted[3].id), reverse=True))
|
||||
assert (tuple(row.id for row in global_pending.submissions), global_pending.has_more) == (all_pending[:3], True)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("completion_seconds", (13, 21), ids=("before-deadline", "at-deadline"))
|
||||
async def test_scan_claim_retry_and_late_result_preserve_proposal(
|
||||
pending_update: PendingUpdate, completion_seconds: int,
|
||||
) -> None:
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from litellm.models.skill_submission import SkillScanInput
|
||||
from litellm.types.proxy.skill_scanning import SkillScan, SkillScanReport
|
||||
|
||||
case: Final = pending_update
|
||||
now: Final = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
first: Final = SkillScan(
|
||||
id="first", status="running", requested_by="reviewer", started_at=now,
|
||||
expires_at=now + timedelta(seconds=10),
|
||||
)
|
||||
started: Final = await case.submissions.start_scan(case.submission.id, first)
|
||||
assert isinstance(started, SkillScanInput)
|
||||
assert started.content == case.proposal.content
|
||||
assert started.scan.archive_sha256 == hashlib.sha256(case.proposal.content or b"").hexdigest()
|
||||
duplicate: Final = await case.submissions.start_scan(case.submission.id, first.model_copy(update={"id": "duplicate"}))
|
||||
assert isinstance(duplicate, SubmissionConflict)
|
||||
second: Final = first.model_copy(update={
|
||||
"id": "second", "started_at": now + timedelta(seconds=11), "expires_at": now + timedelta(seconds=21),
|
||||
})
|
||||
retry: Final = await case.submissions.start_scan(case.submission.id, second)
|
||||
assert isinstance(retry, SkillScanInput)
|
||||
report: Final = SkillScanReport(
|
||||
scanner_version="test-version", complete=False, scan_mode="static-only", risk_score=90, findings=(),
|
||||
)
|
||||
await case.submissions.finish_scan(case.submission.id, started.scan.model_copy(update={
|
||||
"status": "completed", "report": report, "completed_at": now + timedelta(seconds=12),
|
||||
}))
|
||||
assert await case.submissions.get_scan(case.submission.id) == retry.scan
|
||||
completed: Final = retry.scan.model_copy(update={
|
||||
"status": "completed", "report": report, "completed_at": now + timedelta(seconds=completion_seconds),
|
||||
})
|
||||
expected: Final = completed if completion_seconds < 21 else retry.scan.at(now + timedelta(seconds=21))
|
||||
await case.submissions.finish_scan(case.submission.id, completed)
|
||||
assert await case.submissions.get_scan(case.submission.id) == expected
|
||||
assert await case.submissions.get(case.submission.id, None) == case.submission
|
||||
approved: Final = await case.submissions.review(case.submission.id, "reviewer", True, None, publication_base_url="https://gateway.test")
|
||||
assert isinstance(approved, SubmissionReviewed)
|
||||
assert await case.submissions.get_scan(case.submission.id) == expected
|
||||
denied: Final = await case.submissions.start_scan(case.submission.id, second)
|
||||
assert isinstance(denied, SubmissionConflict)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("available, remote", ((True, False), (False, False), (False, True)))
|
||||
async def test_metadata_scan_requires_the_matching_managed_archive(
|
||||
pending_update: PendingUpdate, available: bool, remote: bool
|
||||
) -> None:
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from litellm.models.skill_submission import SkillScanInput
|
||||
from litellm.types.proxy.skill_scanning import SkillScan
|
||||
|
||||
case: Final = pending_update
|
||||
submission: Final = await case.submissions.submit(
|
||||
replace(case.original, content=None, manifest={
|
||||
**case.original.manifest,
|
||||
**({"source": {"source": "archive", "url": "https://example.org/skill.zip"}} if remote else {}),
|
||||
}), "member", PluginPublicationBase(case.base.id, case.base.updated_at)
|
||||
)
|
||||
if not available:
|
||||
write_rows('DELETE FROM "LiteLLM_ClaudeCodePluginUpload" WHERE plugin_name=%s', (case.original.name,))
|
||||
now: Final = datetime.now(timezone.utc)
|
||||
result: Final = await case.submissions.start_scan(submission.id, SkillScan(
|
||||
id="metadata-scan", status="running", requested_by="admin", started_at=now,
|
||||
expires_at=now + timedelta(seconds=10),
|
||||
))
|
||||
if available or remote:
|
||||
assert isinstance(result, SkillScanInput)
|
||||
assert result.content == (None if remote else case.original.content)
|
||||
if remote:
|
||||
assert result.source == {"source": "archive", "url": "https://example.org/skill.zip"}
|
||||
assert result.scan.archive_sha256 is None
|
||||
else:
|
||||
assert isinstance(result, SubmissionConflict)
|
||||
assert await case.submissions.get_scan(submission.id) is None
|
||||
|
|
|
|||
58
tests/integration/security/test_skill_scanner.py
Normal file
58
tests/integration/security/test_skill_scanner.py
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
import asyncio
|
||||
import sys
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from typing import Final
|
||||
|
||||
import psutil
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
|
||||
from cookbook.skill_scanner import server
|
||||
from tests.integration._support.process import _wait_until_gone, group_members, stop_processes
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", ("deadline", "cap", "complete"))
|
||||
async def test_adapter_bounds_worker_lifetime_and_reaps_its_process_group(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, mode: str,
|
||||
) -> None:
|
||||
script: Final = (
|
||||
"import subprocess, sys, time\n"
|
||||
"child = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(30)'], "
|
||||
"stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)\n"
|
||||
"print(child.pid, flush=True)\n"
|
||||
+ ("pass\n" if mode == "complete" else "time.sleep(30)\n")
|
||||
)
|
||||
child: Final = await asyncio.create_subprocess_exec(
|
||||
sys.executable, "-c", script, stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.DEVNULL, start_new_session=True,
|
||||
)
|
||||
try:
|
||||
assert child.stdout is not None
|
||||
descendant: Final = psutil.Process(int(await asyncio.wait_for(child.stdout.readline(), 5)))
|
||||
if mode == "complete":
|
||||
await asyncio.wait_for(child.wait(), 5)
|
||||
output: Final = tmp_path / "report.json"
|
||||
output.write_text('{"complete": true}')
|
||||
|
||||
async def launch(*args: object, **kwargs: object) -> asyncio.subprocess.Process:
|
||||
return child
|
||||
|
||||
monkeypatch.setattr(server.asyncio, "create_subprocess_exec", launch)
|
||||
monkeypatch.setattr(server, "SCAN_TIMEOUT", 0.05 if mode == "cap" else 30)
|
||||
deadline: Final = (
|
||||
None if mode == "cap"
|
||||
else datetime.now(timezone.utc) + timedelta(seconds=0.05 if mode == "deadline" else 30)
|
||||
)
|
||||
result: Final = server.run_worker("unused.zip", output, False, deadline)
|
||||
if mode == "complete":
|
||||
assert await asyncio.wait_for(result, 5) == {"complete": True}
|
||||
else:
|
||||
with pytest.raises(HTTPException) as raised:
|
||||
await asyncio.wait_for(result, 5)
|
||||
assert raised.value.status_code == 504
|
||||
assert child.returncode is not None
|
||||
_wait_until_gone(descendant, 3)
|
||||
finally:
|
||||
await asyncio.to_thread(stop_processes, group_members(child.pid), 0.1)
|
||||
await child.wait()
|
||||
|
|
@ -684,3 +684,72 @@ async def test_replacement_rejections_do_not_stage_or_change_files(pending_hub:
|
|||
assert hub.database.litellm_skillsubmission.rows == ()
|
||||
hub.plugins.update.assert_not_awaited()
|
||||
hub.archives.upsert.assert_not_awaited()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("key", (MEMBER_KEY, OTHER_KEY, VIEWER_KEY))
|
||||
@pytest.mark.parametrize("method", ("GET", "POST"))
|
||||
async def test_skill_scan_requires_full_admin(hub: HubClient, key: str, method: str) -> None:
|
||||
response = await hub.client.request(method, "/skill-hub/submissions/absent/scan", headers=_headers(key))
|
||||
assert response.status_code == 403, response.text
|
||||
|
||||
|
||||
async def test_unconfigured_scanning_preserves_manual_approval(pending_hub: HubClient) -> None:
|
||||
hub = pending_hub
|
||||
created = await _upload(hub, _zip(FILES), MEMBER_KEY, version=None)
|
||||
path = f"/skill-hub/submissions/{created.json()['submission_id']}"
|
||||
state = await hub.client.get(path + "/scan", headers=_headers(MASTER_KEY))
|
||||
assert state.status_code == 200, state.text
|
||||
assert state.json() == {"configured": False, "scan": None}
|
||||
unavailable = await hub.client.post(path + "/scan", headers=_headers(MASTER_KEY))
|
||||
assert unavailable.status_code == 503, unavailable.text
|
||||
approved = await hub.client.put(path + "/approve", headers=_headers(MASTER_KEY))
|
||||
assert approved.status_code == 200, approved.text
|
||||
assert approved.json()["submission"]["status"] == "approved"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("scanner_status", (200, 503), ids=("report", "error"))
|
||||
async def test_configured_scan_persists_result_without_changing_manual_approval(
|
||||
pending_hub: HubClient, monkeypatch: pytest.MonkeyPatch, scanner_status: int,
|
||||
) -> None:
|
||||
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler
|
||||
from litellm.proxy.discovery_endpoints import skill_scanning
|
||||
|
||||
hub: Final = pending_hub
|
||||
report: Final = {
|
||||
"scanner_version": "test-version", "complete": True, "scan_mode": "static-only",
|
||||
"risk_score": 0, "findings": [], "limitations": [],
|
||||
}
|
||||
|
||||
def respond(request: httpx.Request) -> httpx.Response:
|
||||
assert request.headers["authorization"] == "Bearer scanner-only-token"
|
||||
return httpx.Response(scanner_status, json=report)
|
||||
|
||||
handler: Final = AsyncHTTPHandler(transport=httpx.MockTransport(respond))
|
||||
|
||||
def scanner_client(provider: str, params: Mapping[str, object]) -> AsyncHTTPHandler:
|
||||
return handler
|
||||
|
||||
monkeypatch.setattr(skill_scanning, "get_async_httpx_client", scanner_client)
|
||||
monkeypatch.setattr(proxy_server, "general_settings", {"skill_scanner": {
|
||||
"endpoint": "https://scanner.example/scan", "api_key_env": "SCANNER_TEST_KEY",
|
||||
}})
|
||||
monkeypatch.setenv("SCANNER_TEST_KEY", "scanner-only-token")
|
||||
created: Final = await _upload(hub, _zip(FILES), MEMBER_KEY, version=None)
|
||||
path: Final = f"/skill-hub/submissions/{created.json()['submission_id']}"
|
||||
before: Final = (await hub.client.get(path, headers=_headers(MASTER_KEY))).json()
|
||||
async with handler.client:
|
||||
started: Final = await hub.client.post(path + "/scan", headers=_headers(MASTER_KEY))
|
||||
assert started.status_code == 200, started.text
|
||||
completed: Final = (await hub.client.get(path + "/scan", headers=_headers(MASTER_KEY))).json()["scan"]
|
||||
assert completed["id"] == started.json()["scan"]["id"]
|
||||
expected: Final = ("completed", report) if scanner_status == 200 else ("failed", None)
|
||||
assert (completed["status"], completed["report"]) == expected
|
||||
assert completed["error"] is None if scanner_status == 200 else "HTTP 503" in completed["error"]
|
||||
monkeypatch.setattr(proxy_server, "general_settings", {"skill_scanner": {"endpoint": "invalid"}})
|
||||
retained: Final = await hub.client.get(path + "/scan", headers=_headers(MASTER_KEY))
|
||||
assert retained.status_code == 200 and retained.json() == {"configured": False, "scan": completed}
|
||||
unavailable: Final = await hub.client.post(path + "/scan", headers=_headers(MASTER_KEY))
|
||||
assert unavailable.status_code == 503 and "configuration is invalid" in unavailable.text
|
||||
assert (await hub.client.get(path, headers=_headers(MASTER_KEY))).json() == before
|
||||
approved: Final = await hub.client.put(path + "/approve", headers=_headers(MASTER_KEY))
|
||||
assert approved.status_code == 200 and approved.json()["submission"]["status"] == "approved"
|
||||
|
|
|
|||
155
tests/unit/proxy/discovery_endpoints/test_skill_scanning.py
Normal file
155
tests/unit/proxy/discovery_endpoints/test_skill_scanning.py
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
import hashlib
|
||||
import io
|
||||
import zipfile
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from typing import Final
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from pydantic import ValidationError
|
||||
|
||||
from litellm.models.skill_submission import SkillScanInput
|
||||
from litellm.proxy.discovery_endpoints.skill_scanning import scan_report
|
||||
from litellm.types.proxy.skill_scanning import SkillScan, SkillScannerConfig, SkillScanReport
|
||||
|
||||
NOW: Final = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
CONFIG: Final = SkillScannerConfig(endpoint="https://scanner.example/scan", api_key_env="SCANNER_TEST_KEY")
|
||||
REPORT: Final = SkillScanReport(
|
||||
scanner_version="test-version", complete=True, scan_mode="static-only", risk_score=0, findings=(),
|
||||
)
|
||||
TARGET: Final = SkillScanInput(
|
||||
SkillScan(id="scan-1", status="running", requested_by="admin", started_at=NOW, expires_at=NOW + timedelta(seconds=10)),
|
||||
{"source": "archive", "url": "https://gateway.example/private.zip"},
|
||||
b"exact-private-archive",
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_scan_posts_stored_bytes_with_only_scanner_credentials() -> None:
|
||||
def respond(request: httpx.Request) -> httpx.Response:
|
||||
assert request.url == CONFIG.endpoint
|
||||
assert request.headers["authorization"] == "Bearer scanner-only-token"
|
||||
assert b"exact-private-archive" in request.content
|
||||
assert b'name="use_llm"\r\n\r\nfalse' in request.content
|
||||
assert f'name="expires_at"\r\n\r\n{TARGET.scan.expires_at.isoformat()}'.encode() in request.content
|
||||
return httpx.Response(200, json=REPORT.model_dump())
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(respond)) as client:
|
||||
result: Final = await scan_report(client, CONFIG, "scanner-only-token", TARGET)
|
||||
assert result == REPORT
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("failure", ("redirect", "unavailable", "malformed", "oversized", "network"))
|
||||
async def test_scanner_failures_never_become_clean_reports(failure: str) -> None:
|
||||
def respond(request: httpx.Request) -> httpx.Response:
|
||||
assert request.url.host == "scanner.example"
|
||||
if failure == "redirect":
|
||||
return httpx.Response(302, headers={"Location": "https://untrusted.example/scan"})
|
||||
if failure == "unavailable":
|
||||
return httpx.Response(503, text="provider-internal-secret")
|
||||
if failure == "malformed":
|
||||
return httpx.Response(200, json={"complete": True})
|
||||
if failure == "oversized":
|
||||
return httpx.Response(200, content=b" " * (1024 * 1024 + 1))
|
||||
raise httpx.ConnectError("provider-internal-secret", request=request)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(respond), follow_redirects=True) as client:
|
||||
result: Final = await scan_report(client, CONFIG, "scanner-only-token", TARGET)
|
||||
assert isinstance(result, str)
|
||||
assert "provider-internal-secret" not in result
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("existing_limitations", (0, 100), ids=("empty", "full"))
|
||||
async def test_missing_requested_semantic_pass_remains_incomplete(existing_limitations: int) -> None:
|
||||
report: Final = REPORT.model_copy(update={"limitations": ("Existing limitation",) * existing_limitations})
|
||||
|
||||
def respond(request: httpx.Request) -> httpx.Response:
|
||||
assert b'name="use_llm"\r\n\r\ntrue' in request.content
|
||||
return httpx.Response(200, json=report.model_dump())
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(respond)) as client:
|
||||
result: Final = await scan_report(client, CONFIG.model_copy(update={"use_llm": True}), "token", TARGET)
|
||||
assert isinstance(result, SkillScanReport)
|
||||
assert result == report.model_copy(update={
|
||||
"complete": False, "limitations": (*report.limitations[:99], "Requested semantic analysis did not run."),
|
||||
})
|
||||
assert SkillScanReport.model_validate(result.model_dump()) == result
|
||||
|
||||
|
||||
def test_interrupted_scan_becomes_retryable_without_changing_completed_evidence() -> None:
|
||||
assert TARGET.scan.at(NOW) == TARGET.scan
|
||||
assert TARGET.scan.at(NOW + timedelta(seconds=10)).status == "failed"
|
||||
complete: Final = TARGET.scan.model_copy(update={"status": "completed", "report": REPORT})
|
||||
assert complete.at(NOW + timedelta(days=1)) == complete
|
||||
|
||||
|
||||
@pytest.mark.parametrize("endpoint", (
|
||||
"http://scanner.example/scan", "https://user:secret@scanner.example/scan", "https://scanner.example/scan?token=secret",
|
||||
))
|
||||
def test_scanner_config_rejects_unsafe_credential_destinations(endpoint: str) -> None:
|
||||
with pytest.raises(ValidationError):
|
||||
SkillScannerConfig(endpoint=endpoint, api_key_env="SCANNER_TEST_KEY")
|
||||
|
||||
|
||||
def test_adapter_authentication_and_exact_remote_source(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from cookbook.skill_scanner import server
|
||||
|
||||
monkeypatch.setenv("SKILL_SCANNER_API_KEY", "adapter-secret")
|
||||
clock: Final = MagicMock()
|
||||
clock.now.return_value = NOW + timedelta(seconds=1)
|
||||
monkeypatch.setattr(server, "datetime", clock)
|
||||
launch: Final = AsyncMock(side_effect=AssertionError("Invalid deadline launched a worker"))
|
||||
monkeypatch.setattr(server.asyncio, "create_subprocess_exec", launch)
|
||||
with TestClient(server.app) as client:
|
||||
assert client.post("/scan", data={"source": "{}"}).status_code == 401
|
||||
assert client.post("/scan", headers={"Authorization": "Bearer wrong"}, data={"source": "{}"}).status_code == 401
|
||||
response = client.post("/scan", headers={"Authorization": "Bearer adapter-secret"}, data={
|
||||
"source": '{"source":"git","url":"https://example.org/repo.git","ref":"release"}',
|
||||
})
|
||||
assert response.status_code == 400
|
||||
for deadline, status in ((NOW.isoformat(), 504), (NOW.replace(tzinfo=None).isoformat(), 422)):
|
||||
rejected: Final = client.post("/scan", headers={"Authorization": "Bearer adapter-secret"}, data={
|
||||
"source": '{"source":"archive","url":"https://example.org/skill.zip"}', "expires_at": deadline,
|
||||
})
|
||||
assert rejected.status_code == status
|
||||
launch.assert_not_called()
|
||||
assert server.source_target({"source": "archive", "url": "https://example.org/skill.zip"}) == "https://example.org/skill.zip"
|
||||
assert server.source_target({"source": "github", "repo": "example/skills", "ref": "v2", "path": "skills/review"}) == "https://github.com/example/skills/tree/v2/skills/review"
|
||||
|
||||
|
||||
def test_adapter_accepts_expanded_stored_skill_bytes(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from cookbook.skill_scanner import server
|
||||
from litellm.proxy.anthropic_endpoints.claude_code_endpoints.skill_upload_archive import (
|
||||
build_uploaded_skill_archive,
|
||||
)
|
||||
|
||||
uploaded: Final = io.BytesIO()
|
||||
with zipfile.ZipFile(uploaded, "w", zipfile.ZIP_DEFLATED) as archive:
|
||||
archive.writestr("SKILL.md", "---\nname: large-skill\ndescription: Synthetic archive\n---\n")
|
||||
for name in ("first.bin", "second.bin"):
|
||||
archive.writestr(name, b"x" * (6 * 1024 * 1024))
|
||||
stored: Final = build_uploaded_skill_archive(uploaded.getvalue(), "large-skill")
|
||||
assert stored is not None
|
||||
stored_size: Final = len(stored)
|
||||
assert stored_size > 10 * 1024 * 1024
|
||||
expected: Final = (stored_size, hashlib.sha256(stored).hexdigest())
|
||||
|
||||
async def run_worker(
|
||||
target: str, output: Path, use_llm: bool, expires_at: datetime | None = None,
|
||||
) -> dict[str, object]:
|
||||
received: Final = Path(target).read_bytes()
|
||||
actual: Final = (len(received), hashlib.sha256(received).hexdigest())
|
||||
assert actual == expected
|
||||
return {"complete": True}
|
||||
|
||||
monkeypatch.setattr(server, "run_worker", run_worker)
|
||||
monkeypatch.setenv("SKILL_SCANNER_API_KEY", "adapter-secret")
|
||||
with TestClient(server.app) as client:
|
||||
response: Final = client.post("/scan", headers={"Authorization": "Bearer adapter-secret"},
|
||||
data={"source": "{}"}, files={"file": ("skill.zip", stored, "application/zip")})
|
||||
assert response.status_code == 200 and response.json() == {"complete": True}
|
||||
|
|
@ -41,6 +41,8 @@ class SubmissionRows:
|
|||
return updated
|
||||
|
||||
async def query_raw(self, query: str, *args: object) -> tuple[Mapping[str, object], ...]:
|
||||
if query.startswith("SELECT scan_json"):
|
||||
return tuple({"scan_json": row.scan_json} for row in self.rows if row.id == args[0])
|
||||
if 'FROM "LiteLLM_SkillSubmission"' not in query or "FOR UPDATE" in query:
|
||||
return ()
|
||||
if "COUNT(*) AS count" in query:
|
||||
|
|
|
|||
|
|
@ -115,7 +115,48 @@ const makeSubmissions = (count: number): SkillSubmission[] =>
|
|||
proposal: { ...pending.proposal, description: `Instructions ${index + 1}` },
|
||||
}));
|
||||
|
||||
let scanFailure = false;
|
||||
let scanRequested = false;
|
||||
|
||||
function scanResponse(method: string) {
|
||||
if (method === "POST") scanRequested = true;
|
||||
return Response.json({
|
||||
configured: true,
|
||||
scan: !scanRequested
|
||||
? null
|
||||
: {
|
||||
id: "scan-1",
|
||||
status: scanFailure ? "failed" : "completed",
|
||||
requested_by: "admin",
|
||||
started_at: "2026-01-01T00:00:00Z",
|
||||
expires_at: "2026-01-01T00:03:00Z",
|
||||
archive_sha256: "a".repeat(64),
|
||||
error: scanFailure ? "Scanner was unreachable" : null,
|
||||
report: scanFailure
|
||||
? null
|
||||
: {
|
||||
scanner_version: "test-version",
|
||||
complete: false,
|
||||
scan_mode: "static-only",
|
||||
risk_score: 85,
|
||||
findings: [
|
||||
{
|
||||
rule_id: "unsafe-upload",
|
||||
severity: "HIGH",
|
||||
message: "Potential data exfiltration",
|
||||
file: "SKILL.md",
|
||||
start_line: 8,
|
||||
},
|
||||
],
|
||||
limitations: [],
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
scanFailure = false;
|
||||
scanRequested = false;
|
||||
testQueryClient.clear();
|
||||
vi.clearAllMocks();
|
||||
catalog = [published];
|
||||
|
|
@ -129,6 +170,9 @@ beforeEach(() => {
|
|||
const url = new URL(request?.url ?? String(input), "http://localhost");
|
||||
const path = url.pathname;
|
||||
const method = request?.method ?? options?.method ?? "GET";
|
||||
if (path === `/skill-hub/submissions/${pending.id}/scan`) {
|
||||
return scanResponse(method);
|
||||
}
|
||||
if (method !== "GET") {
|
||||
const body = request ? await request.text() : options?.body;
|
||||
writes.push({ path, method, body });
|
||||
|
|
@ -630,3 +674,71 @@ it("ignores an in-flight admin submission list that resolves after the same acco
|
|||
expect(screen.queryByText("Proposed instructions")).not.toBeInTheDocument();
|
||||
expect(screen.queryByRole("button", { name: "Review" })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it.each([false, true])("keeps approval available after manual scanning, failure=%s", async (failure) => {
|
||||
scanFailure = failure;
|
||||
renderSubmissions();
|
||||
fireEvent.click(await screen.findByRole("button", { name: "Review" }));
|
||||
const scan = await screen.findByRole("button", { name: "Scan for threats" });
|
||||
await waitFor(() => expect(scan).toBeEnabled());
|
||||
expect(screen.getByRole("button", { name: "Approve and publish" })).toBeEnabled();
|
||||
fireEvent.click(scan);
|
||||
expect(
|
||||
await screen.findByText(failure ? "Scanner was unreachable" : "Potential data exfiltration"),
|
||||
).toBeInTheDocument();
|
||||
if (!failure) {
|
||||
expect(screen.getByText("SKILL.md:8")).toBeInTheDocument();
|
||||
expect(
|
||||
screen.getByText("Analysis was incomplete. This report does not establish that the skill is safe."),
|
||||
).toBeInTheDocument();
|
||||
}
|
||||
expect(screen.getByRole("button", { name: "Reject" })).toBeEnabled();
|
||||
fireEvent.click(screen.getByRole("button", { name: "Approve and publish" }));
|
||||
expect(await screen.findByText("Approved")).toBeInTheDocument();
|
||||
expect(writes).toHaveLength(1);
|
||||
expect(writes[0].path).toBe(`/skill-hub/submissions/${pending.id}/approve`);
|
||||
});
|
||||
|
||||
it.each([200, 409])("refreshes the running scan after POST %s and ignores an older GET", async (status) => {
|
||||
renderSubmissions();
|
||||
fireEvent.click(await screen.findByRole("button", { name: "Review" }));
|
||||
const scan = await screen.findByRole("button", { name: "Scan for threats" });
|
||||
await waitFor(() => expect(scan).toBeEnabled());
|
||||
const older = Promise.withResolvers<Response>();
|
||||
const refreshed = Promise.withResolvers<Response>();
|
||||
const running = {
|
||||
configured: true,
|
||||
scan: {
|
||||
id: "current-scan",
|
||||
status: "running",
|
||||
requested_by: "another-admin",
|
||||
started_at: "2026-01-01T00:00:00Z",
|
||||
expires_at: "2026-01-01T00:03:00Z",
|
||||
},
|
||||
};
|
||||
const originalFetch = fetchMock.getMockImplementation()!;
|
||||
let reads = 0;
|
||||
fetchMock.mockImplementation((input, options) => {
|
||||
const url = input instanceof Request ? input.url : String(input);
|
||||
if (!url.endsWith(`/submissions/${pending.id}/scan`)) return originalFetch(input, options);
|
||||
if ((input instanceof Request ? input.method : options?.method) === "POST") {
|
||||
return Promise.resolve(
|
||||
Response.json(status === 409 ? { detail: "A scan is already running" } : running, { status }),
|
||||
);
|
||||
}
|
||||
return ++reads === 1 ? older.promise : refreshed.promise;
|
||||
});
|
||||
void testQueryClient.refetchQueries({ type: "active" });
|
||||
await waitFor(() => expect(reads).toBe(1));
|
||||
fireEvent.click(scan);
|
||||
await waitFor(() => expect(reads).toBe(2));
|
||||
expect(screen.getByRole("button", { name: "Scanning…" })).toBeDisabled();
|
||||
expect(screen.getByRole("button", { name: "Approve and publish" })).toBeEnabled();
|
||||
refreshed.resolve(Response.json(running));
|
||||
expect(await screen.findByText("Requested by another-admin")).toBeInTheDocument();
|
||||
await act(async () => older.resolve(Response.json({ configured: true, scan: null })));
|
||||
expect(screen.getByRole("button", { name: "Scanning…" })).toBeDisabled();
|
||||
expect(screen.getByText("Requested by another-admin")).toBeInTheDocument();
|
||||
fireEvent.click(screen.getByRole("button", { name: "Approve and publish" }));
|
||||
expect(await screen.findByText("Approved")).toBeInTheDocument();
|
||||
});
|
||||
|
|
|
|||
|
|
@ -0,0 +1,131 @@
|
|||
import { useMutation, useQuery, useQueryClient, type QueryKey } from "@tanstack/react-query";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { getSkillScan, startSkillScan } from "@/components/networking";
|
||||
import type { components } from "@/lib/http/schema";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import { Button } from "@/components/ui/button";
|
||||
|
||||
interface SkillScanPanelProps {
|
||||
id: string;
|
||||
accessToken: string;
|
||||
queryScope: QueryKey;
|
||||
pending: boolean;
|
||||
}
|
||||
|
||||
export default function SkillScanPanel({ id, accessToken, queryScope, pending }: SkillScanPanelProps) {
|
||||
const { t } = useTranslation();
|
||||
const queryClient = useQueryClient();
|
||||
const queryKey = [...queryScope, "submission", id, "scan"];
|
||||
const result = useQuery({
|
||||
queryKey,
|
||||
queryFn: () => getSkillScan(accessToken, id),
|
||||
refetchInterval: (query) => (query.state.data?.scan?.status === "running" ? 2000 : false),
|
||||
});
|
||||
const start = useMutation({
|
||||
mutationFn: () => startSkillScan(accessToken, id),
|
||||
onSettled: () => queryClient.invalidateQueries({ queryKey, exact: true }),
|
||||
});
|
||||
const scan = result.data?.scan;
|
||||
const report = scan?.report;
|
||||
const running = start.isPending || scan?.status === "running";
|
||||
return (
|
||||
<section aria-label={t("skills.scanning.title")} className="space-y-3 rounded-lg border p-4">
|
||||
<div className="flex items-center justify-between gap-3">
|
||||
<h3 className="text-sm font-semibold">{t("skills.scanning.title")}</h3>
|
||||
{pending && (
|
||||
<Button
|
||||
variant="outline"
|
||||
disabled={running || result.isError || !result.data?.configured}
|
||||
onClick={() => start.mutate()}
|
||||
>
|
||||
{t(running ? "skills.scanning.running" : "skills.scanning.scan")}
|
||||
</Button>
|
||||
)}
|
||||
</div>
|
||||
<p className="text-xs text-muted-foreground">{t("skills.scanning.advisory")}</p>
|
||||
{result.isPending && <p className="text-sm">{t("skills.scanning.loading")}</p>}
|
||||
{result.data && !result.data.configured && (
|
||||
<p className="text-sm text-muted-foreground">{t("skills.scanning.unconfigured")}</p>
|
||||
)}
|
||||
{result.isError && (
|
||||
<div role="alert" className="text-sm text-destructive">
|
||||
{t("skills.scanning.loadFailed")}
|
||||
<Button variant="link" onClick={() => void result.refetch()}>
|
||||
{t("skills.submissions.retry")}
|
||||
</Button>
|
||||
</div>
|
||||
)}
|
||||
{start.isError && (
|
||||
<p role="alert" className="text-sm text-destructive">
|
||||
{start.error.message}
|
||||
</p>
|
||||
)}
|
||||
{scan && (
|
||||
<>
|
||||
<div className="flex flex-wrap items-center gap-2 text-xs text-muted-foreground">
|
||||
<Badge variant="outline">{t(`skills.scanning.status.${scan.status}`)}</Badge>
|
||||
<span>{t("skills.scanning.requestedBy", { name: scan.requested_by })}</span>
|
||||
<span>{new Date(scan.started_at).toLocaleString()}</span>
|
||||
</div>
|
||||
<p className="break-all text-xs text-muted-foreground">
|
||||
{scan.archive_sha256
|
||||
? t("skills.scanning.archiveDigest", { digest: scan.archive_sha256 })
|
||||
: t("skills.scanning.mutableSource")}
|
||||
</p>
|
||||
{scan.error && (
|
||||
<p role="alert" className="text-sm text-destructive">
|
||||
{scan.error}
|
||||
</p>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
{report && <ScanReport report={report} />}
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
function ScanReport({ report }: { report: components["schemas"]["SkillScanReport"] }) {
|
||||
const { t } = useTranslation();
|
||||
return (
|
||||
<>
|
||||
<p className="text-sm">
|
||||
{t("skills.scanning.summary", {
|
||||
count: report.findings.length,
|
||||
score: report.risk_score,
|
||||
version: report.scanner_version,
|
||||
})}
|
||||
</p>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
{t(report.scan_mode === "static+llm" ? "skills.scanning.semantic" : "skills.scanning.static")}
|
||||
</p>
|
||||
{!report.complete && (
|
||||
<p role="alert" className="text-sm text-warning">
|
||||
{t("skills.scanning.incomplete")}
|
||||
</p>
|
||||
)}
|
||||
{(report.limitations ?? []).map((limitation, index) => (
|
||||
<p key={index} className="text-xs text-muted-foreground">
|
||||
{limitation}
|
||||
</p>
|
||||
))}
|
||||
{report.findings.length === 0 && <p className="text-sm">{t("skills.scanning.noFindings")}</p>}
|
||||
<div className="max-h-80 space-y-3 overflow-y-auto">
|
||||
{report.findings.map((finding, index) => (
|
||||
<article key={index} className="space-y-1 rounded border p-3 text-sm">
|
||||
<div className="flex flex-wrap items-center gap-2">
|
||||
<Badge variant="outline">{finding.severity}</Badge>
|
||||
<span className="font-mono text-xs">{finding.rule_id}</span>
|
||||
</div>
|
||||
<p className="whitespace-pre-wrap break-words font-medium">{finding.message}</p>
|
||||
<p className="break-all font-mono text-xs text-muted-foreground">
|
||||
{finding.file}
|
||||
{finding.start_line ? `:${finding.start_line}` : ""}
|
||||
</p>
|
||||
{finding.explanation && <p className="whitespace-pre-wrap break-words">{finding.explanation}</p>}
|
||||
{finding.remediation && <p className="whitespace-pre-wrap break-words">{finding.remediation}</p>}
|
||||
</article>
|
||||
))}
|
||||
</div>
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
|
@ -14,6 +14,7 @@ import { Button } from "@/components/ui/button";
|
|||
import { Dialog, DialogContent, DialogHeader, DialogTitle } from "@/components/ui/dialog";
|
||||
import { Textarea } from "@/components/ui/textarea";
|
||||
import { useUrlTableState, type UrlTableStateOptions } from "@/components/shared/DataTable/useUrlTableState";
|
||||
import SkillScanPanel from "./SkillScanPanel";
|
||||
import { toast } from "@/lib/toast";
|
||||
|
||||
const METADATA_FIELDS = ["name", "version", "description", "source", "keywords", "domain", "namespace"] as const;
|
||||
|
|
@ -208,6 +209,14 @@ function SubmissionReview({ id, accessToken, queryScope, canReview, onClose, onC
|
|||
{t(downloading ? "skills.submissions.downloading" : "skills.submissions.download")}
|
||||
</Button>
|
||||
)}
|
||||
{canReview && (
|
||||
<SkillScanPanel
|
||||
id={id}
|
||||
accessToken={accessToken}
|
||||
queryScope={queryScope}
|
||||
pending={submission.status === "pending_review"}
|
||||
/>
|
||||
)}
|
||||
{canDecide && (
|
||||
<>
|
||||
<label htmlFor="skill-rejection-reason" className="text-sm font-medium">
|
||||
|
|
|
|||
|
|
@ -6975,6 +6975,12 @@ export const reviewSkillSubmission = (
|
|||
): Promise<components["schemas"]["SkillHubReviewResponse"]> =>
|
||||
apiClient.put(`/skill-hub/submissions/${encodeURIComponent(id)}/${action}`, { accessToken, body });
|
||||
|
||||
export const getSkillScan = (accessToken: string, id: string): Promise<components["schemas"]["SkillScanResponse"]> =>
|
||||
apiClient.get(`/skill-hub/submissions/${encodeURIComponent(id)}/scan`, { accessToken });
|
||||
|
||||
export const startSkillScan = (accessToken: string, id: string): Promise<components["schemas"]["SkillScanResponse"]> =>
|
||||
apiClient.post(`/skill-hub/submissions/${encodeURIComponent(id)}/scan`, { accessToken });
|
||||
|
||||
export const downloadSkillSubmission = (accessToken: string, id: string): Promise<Blob> =>
|
||||
apiClient.getBlob(`/skill-hub/submissions/${encodeURIComponent(id)}/archive`, { accessToken, redirect: "error" });
|
||||
|
||||
|
|
|
|||
|
|
@ -1755,6 +1755,28 @@
|
|||
"importHint": "Enter the original source URL. Use a skill selector or path when the repository contains several skills. Leave the ref blank to use HEAD.",
|
||||
"saving": "Saving..."
|
||||
},
|
||||
"scanning": {
|
||||
"title": "Security scan",
|
||||
"scan": "Scan for threats",
|
||||
"running": "Scanning…",
|
||||
"advisory": "Scanning is optional. You can approve or reject this submission at any time.",
|
||||
"loading": "Loading scan status…",
|
||||
"unconfigured": "Security scanning is not configured. Ask your gateway administrator to connect a scanner.",
|
||||
"loadFailed": "Could not load the scan status.",
|
||||
"requestedBy": "Requested by {{name}}",
|
||||
"archiveDigest": "Scanned archive SHA-256: {{digest}}",
|
||||
"mutableSource": "This scan uses an external source that may change after scanning.",
|
||||
"summary": "{{count}} findings · Risk score {{score}}/100 · SkillSpector {{version}}",
|
||||
"semantic": "Static and semantic analysis",
|
||||
"static": "Static analysis only",
|
||||
"incomplete": "Analysis was incomplete. This report does not establish that the skill is safe.",
|
||||
"noFindings": "No findings were reported in the analyzed content.",
|
||||
"status": {
|
||||
"running": "In progress",
|
||||
"completed": "Completed",
|
||||
"failed": "Failed"
|
||||
}
|
||||
},
|
||||
"submissions": {
|
||||
"adminIntro": "Review submitted skills and changes before publishing them",
|
||||
"authorIntro": "Your submissions and review outcomes. Pending changes are not published",
|
||||
|
|
|
|||
193
ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
193
ui/litellm-dashboard/src/lib/http/schema.d.ts
generated
vendored
|
|
@ -16019,6 +16019,30 @@ export interface paths {
|
|||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/skill-hub/submissions/{submission_id}/scan": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
/**
|
||||
* Get Skill Scan
|
||||
* @description Read the latest manual security scan. Only proxy admins may read reports.
|
||||
*/
|
||||
get: operations["get_skill_scan_skill_hub_submissions__submission_id__scan_get"];
|
||||
put?: never;
|
||||
/**
|
||||
* Start Skill Scan
|
||||
* @description Start an optional scan of a pending submission. Scans never change approval decisions.
|
||||
*/
|
||||
post: operations["start_skill_scan_skill_hub_submissions__submission_id__scan_post"];
|
||||
delete?: never;
|
||||
options?: never;
|
||||
head?: never;
|
||||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/spend/calculate": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
|
|
@ -31226,6 +31250,8 @@ export interface components {
|
|||
* @description Lets a non-admin change max_budget or budget_limits on a key they created and still own via /key/update. 'disabled' (default) requires an admin for any budget change. 'lower_only' lets the owner add or lower limits. 'ceiling' also lets the owner raise or remove limits when they could create that key with the new budget via /key/generate.
|
||||
*/
|
||||
self_serve_budget_policy?: ("disabled" | "lower_only" | "ceiling") | null;
|
||||
/** @description Opt-in external scanner for manual admin review of skill submissions */
|
||||
skill_scanner?: components["schemas"]["SkillScannerConfig"] | null;
|
||||
/** @description Daily check of the spend LiteLLM captured against the provider's own bill (OpenAI via OPENAI_ADMIN_KEY). Publishes litellm_spend_capture_rate per provider and alerts when the ratio over the lookback window falls under the threshold (default 0.9). Off unless set. */
|
||||
spend_capture_rate_check?: components["schemas"]["SpendCaptureRateCheckSettings"] | null;
|
||||
/** @description Which keys of LiteLLM_SpendLogs.metadata are written to the database. Set exactly one of 'include' (write only these keys) or 'exclude' (drop these keys). 'status' and 'cold_storage_object_key' are always written. Daily spend tables, budgets and logging callbacks still see every key. Unset writes every key */
|
||||
|
|
@ -47034,6 +47060,107 @@ export interface components {
|
|||
/** Version */
|
||||
version?: string;
|
||||
};
|
||||
/** SkillScan */
|
||||
SkillScan: {
|
||||
/** Archive Sha256 */
|
||||
archive_sha256?: string | null;
|
||||
/** Completed At */
|
||||
completed_at?: string | null;
|
||||
/** Error */
|
||||
error?: string | null;
|
||||
/**
|
||||
* Expires At
|
||||
* Format: date-time
|
||||
*/
|
||||
expires_at: string;
|
||||
/** Id */
|
||||
id: string;
|
||||
report?: components["schemas"]["SkillScanReport"] | null;
|
||||
/** Requested By */
|
||||
requested_by: string;
|
||||
/**
|
||||
* Started At
|
||||
* Format: date-time
|
||||
*/
|
||||
started_at: string;
|
||||
/**
|
||||
* Status
|
||||
* @enum {string}
|
||||
*/
|
||||
status: "running" | "completed" | "failed";
|
||||
};
|
||||
/** SkillScanFinding */
|
||||
SkillScanFinding: {
|
||||
/** Explanation */
|
||||
explanation?: string | null;
|
||||
/** File */
|
||||
file: string;
|
||||
/** Message */
|
||||
message: string;
|
||||
/** Remediation */
|
||||
remediation?: string | null;
|
||||
/** Rule Id */
|
||||
rule_id: string;
|
||||
/**
|
||||
* Severity
|
||||
* @enum {string}
|
||||
*/
|
||||
severity: "LOW" | "MEDIUM" | "HIGH" | "CRITICAL";
|
||||
/** Start Line */
|
||||
start_line?: number | null;
|
||||
};
|
||||
/** SkillScanReport */
|
||||
SkillScanReport: {
|
||||
/** Complete */
|
||||
complete: boolean;
|
||||
/** Findings */
|
||||
findings: components["schemas"]["SkillScanFinding"][];
|
||||
/**
|
||||
* Limitations
|
||||
* @default []
|
||||
*/
|
||||
limitations: string[];
|
||||
/** Risk Score */
|
||||
risk_score: number;
|
||||
/**
|
||||
* Scan Mode
|
||||
* @enum {string}
|
||||
*/
|
||||
scan_mode: "static-only" | "static+llm";
|
||||
/** Scanner Version */
|
||||
scanner_version: string;
|
||||
};
|
||||
/** SkillScanRequest */
|
||||
SkillScanRequest: Record<string, never>;
|
||||
/** SkillScanResponse */
|
||||
SkillScanResponse: {
|
||||
/** Configured */
|
||||
configured: boolean;
|
||||
scan: components["schemas"]["SkillScan"] | null;
|
||||
};
|
||||
/** SkillScannerConfig */
|
||||
SkillScannerConfig: {
|
||||
/**
|
||||
* Api Key Env
|
||||
* @description Environment secret name for scanner auth
|
||||
*/
|
||||
api_key_env: string;
|
||||
/**
|
||||
* Endpoint
|
||||
* @description External scanner's authenticated POST endpoint
|
||||
*/
|
||||
endpoint: string;
|
||||
/**
|
||||
* Timeout Seconds
|
||||
* @default 180
|
||||
*/
|
||||
timeout_seconds: number;
|
||||
/**
|
||||
* Use Llm
|
||||
* @default false
|
||||
*/
|
||||
use_llm: boolean;
|
||||
};
|
||||
/** Span */
|
||||
Span: {
|
||||
/** Agent */
|
||||
|
|
@ -74756,6 +74883,72 @@ export interface operations {
|
|||
};
|
||||
};
|
||||
};
|
||||
get_skill_scan_skill_hub_submissions__submission_id__scan_get: {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path: {
|
||||
submission_id: string;
|
||||
};
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody?: never;
|
||||
responses: {
|
||||
/** @description Successful Response */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["SkillScanResponse"];
|
||||
};
|
||||
};
|
||||
/** @description Validation Error */
|
||||
422: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["HTTPValidationError"];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
start_skill_scan_skill_hub_submissions__submission_id__scan_post: {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path: {
|
||||
submission_id: string;
|
||||
};
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody?: {
|
||||
content: {
|
||||
"application/json": components["schemas"]["SkillScanRequest"] | null;
|
||||
};
|
||||
};
|
||||
responses: {
|
||||
/** @description Successful Response */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["SkillScanResponse"];
|
||||
};
|
||||
};
|
||||
/** @description Validation Error */
|
||||
422: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["HTTPValidationError"];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
calculate_spend_spend_calculate_post: {
|
||||
parameters: {
|
||||
query?: never;
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue