diff --git a/docs/09-deployment.md b/docs/09-deployment.md index 9299b9bff..04706d6d9 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -617,6 +617,7 @@ override 或部署平台环境变量把上述 `SPRING_SECURITY_*` 变量注入 ` | 请求关联 | 响应头和日志中的 `X-Request-Id` / `request.id` | | 日志 | 文本或 ECS 风格 JSON,均输出到容器 stdout / stderr | | Trace | `none`、Micrometer + OTel SDK、或外部 Java Agent 三选一 | +| 业务事实 | 默认关闭的独立 OTLP Logs 出口;[事件目录、配置、Secret、Collector 实收查询与可靠性边界](./business-event-logs.md) | | 指标 | Spring Boot Actuator;Prometheus 是可选后端,不是 Trace 前置条件 | ### 10.1 通用配置 @@ -643,7 +644,7 @@ JSON 日志使用以下稳定字段: ### 10.2 三种 Tracing 模式 -三种模式只能选择一种,切换后需要重启: +三种 Trace 模式只能选择一种,切换后需要重启。该限制不禁止独立配置业务 Logs;下表及本节的 endpoint 限制仅指 TRACE endpoint: | 模式 | 适用场景 | 必需配置 | |------|----------|----------| diff --git a/docs/business-event-logs.md b/docs/business-event-logs.md new file mode 100644 index 000000000..bb2ff56c6 --- /dev/null +++ b/docs/business-event-logs.md @@ -0,0 +1,163 @@ +# Business-event Logs over OTLP / OTLP 业务事件日志 + +Default off. This is lossy analytics, NOT audit, billing, notification or installation confirmation. Use the official server image built from this change; no custom application code, vendor SDK or Collector is required. The receiver must support OTLP/HTTP protobuf Logs. / 默认关闭;用于有损分析,不替代审计、计费、可靠通知或安装确认。只需对应版本官方镜像与运维配置,Collector 可选。 + +## Event catalog / 事件目录 (contract version 1) + +Every row requires `skillhub.event.name`, `skillhub.skill.id`, `skillhub.namespace.id`, `skillhub.visibility`, `skillhub.source`, `skillhub.actor.type`, `log.record.uid` and producer occurrence `Timestamp`. IDs are numeric except the local opaque human `user.id` and UUID UID. Actor type is `user` (including anonymous/unknown user requests) or `system`. A known human adds `user.id`; anonymous/unknown and system actors omit it. / 所有事件都有上述字段;主体类型为 user(含匿名或未知请求)或 system;已知人类操作者额外使用本实例不透明 `user.id`,匿名、未知及系统主体省略该字段。 + +| Event | When / 时机 | Extra fields / 额外字段 | Actor and source / 操作者与来源 | +| --- | --- | --- | --- | +| `skill.publication.effective` | Actual transition into PUBLISHED, exported only after commit / 实际发布且事务提交 | `skillhub.version.id` | Auto-publication: submitting privileged human, `auto_publish`; internal forced publication: `system`, no `user.id`. PRIVATE confirmation: confirmer, `private_confirm`. Review/promotion approval: actual approver, `review` / `promotion`, NOT necessarily uploader or owner / 审核或提升时为实际批准人而非上传者 | +| `skill.download.prepared` | Authorized PUBLISHED request has built its download result and updated existing counters; before stream opens/redirect / 已授权的已发布版本已构建下载结果,尚未打开流或跳转 | `skillhub.version.id` | Current authenticated local user, otherwise anonymous (`user` without `user.id`); `download` / 当前调用人,否则 user 类型且省略 ID | +| `skill.review.approved` | Existing content-version review approved, after commit / 内容版本审核通过并提交 | `skillhub.version.id`, `skillhub.review.id` | Actual reviewer; `review` / 实际审核人 | +| `skill.review.rejected` | Existing content-version review rejected, after commit / 内容版本审核拒绝并提交 | `skillhub.version.id`, `skillhub.review.id` | Actual reviewer; `review`; no reason/comment / 不导出拒绝原因 | +| `skill.promotion.approved` | Initial or update promotion approved, after commit / 首次或更新提升批准并提交 | `skillhub.promotion.id` | Actual approver; `promotion`; skill/namespace refer to SOURCE, not derived target / 技能及空间指来源而非全局目标 | +| `skill.promotion.rejected` | Promotion rejected, after commit / 提升拒绝并提交 | `skillhub.promotion.id` | Actual reviewer; `promotion`; SOURCE scope; no reason / 来源范围,不导出原因 | +| `skill.status.changed` | Archive/unarchive actually changes status, after commit / 归档或取消归档状态实际变化并提交 | `skillhub.status.old`, `skillhub.status.new` | Owner, namespace manager or authorized admin invoking operation; `governance` / 实际执行的所有者、空间管理者或管理员 | +| `skill.visibility.changed` | Effective access-scope change: immediate narrowing OR approved expansion, after commit / 缩小范围立即生效,扩大范围审核通过后生效;均提交后导出 | `skillhub.visibility.old`, `skillhub.visibility.new` | Narrowing requester or expansion approver; `visibility` / 缩小时请求人,扩大时批准人 | +| `skill.version.yanked` | Existing published-version yank, after commit / 已发布版本撤回分发并提交 | `skillhub.version.id` | Actual owner/manager/admin; `governance` / 实际执行人 | + +Publication covers all four producers: `SkillPublishService`, `SkillReviewSubmitService`, `ReviewService`, `PromotionService`. The reserved `builtin-skill-publisher` identity is classified as `system` even through the ordinary privileged auto-publication path (bootstrap verifies that it is a system account). Approval may correctly create TWO different facts: publication plus review/promotion approval. These are not duplicate records. Promotion approval's publication references the TARGET skill/version/namespace; its promotion fact references the SOURCE. Namespace filtering may therefore keep only one. / 发布覆盖四条生产路径;内置技能的保留系统身份即便走普通特权直发,也标记为 system,启动器会校验其系统账号属性;批准同时产生发布及审核/提升两个不同事实,不是重复。提升的发布事件指目标,提升决定事件指来源,空间过滤可能只保留其一。 + +Timestamp is captured immediately before the producer publishes its domain event, not `publishedAt`, transaction commit time or async-send time. Transaction synchronization captures the record and valid span identity at event dispatch, admits only after commit, and never emits on rollback. Outside a transaction (notably the shared download service), admission is immediate. No entity lookup happens in the telemetry listener. / 时间是生产点发布领域事件前的时刻,不是提交或发送时刻;事务内仅提交后入队,回滚不发;无事务下载直接入队。 + +Download means prepared, NOT delivered, complete, installed or executed. Web/v1/CLI share the producer; authorized previews of unpublished versions and review packages do not emit. Compat redirects alone, Suite plans, resolve/detail polling, direct object-storage access, URL reuse and client caches are not new registry download-prepared facts. / 下载准备不等于送达、传输完成、安装或执行;未发布预览、审核包、套装计划、查询轮询不计入。对象存储直取、URL 重用及缓存可绕过注册表。 + +Visibility facts cover the existing dedicated `SkillVisibilityService` event producers. Visibility assigned as part of uploading/publishing a content version has no separate visibility event in the existing domain model; its final value is included on the publication fact. / 可见范围事实覆盖独立范围调整的既有生产点;内容上传或发布时顺带赋予的可见范围没有独立领域事件,只在发布事实中携带最终值。 + +Deferred: metadata/avatar review decisions without an existing decision event (effective visibility changes ARE included), hide/unhide, promotion revocation, review submission/withdrawal, Suite events, searches/views, transfer completion, installation and durable replay. Existing audit/notification/search listeners and authorization rules are unchanged. Legacy events constructed without a producer snapshot are intentionally ignored, not enriched from a database at export time. / 其余事件分期;不变更既有审计、通知、搜索或授权;缺少生产点快照的旧式事件不导出,不在导出时查库补全。 + +## Representation and data boundary / 数据模型与边界 + +- Dedicated scope `com.iflytek.skillhub.business`, instrumentation version `1`; Spring Boot 3.2.3 BOM / OTel Java 1.31.0 unchanged. `skillhub.event.name` is a documented custom attribute because 1.31 lacks `setEventName`. This is standard OTLP LogRecord transport, not a claim of implementing the latest experimental OTel Events semantic conventions. `log.record.uid` borrows the experimental uniqueness convention; contract version 1 locks its meaning here, not an unpinned global semconv schema. +- Resource is ONLY `service.name` (existing `spring.application.name`), `service.version` (existing SkillHub setting), `deployment.environment` (existing SkillHub environment). No host/cloud/process/resource autodetection. Occurrence uses Timestamp; ObservedTimestamp is left to SDK. Body is empty. +- Only an already-valid current OTel span context is associated, including unsampled spans. No span is created and no MDC/request ID is converted into trace identity. Trace absence or sampling never filters business records; an associated unsampled trace may not be queryable. +- The sole mapper copies explicit fixed fields, never an event/entity map. Maximum 16 attributes, 256 UTF-16 code units per string (SDK limit as defense-in-depth), three bounded Resource strings, no arbitrary arrays/maps. Unknown event types/sources are ignored. IDs of private skills and namespaces ARE included by default when this instance-level feature is enabled. Empty namespace allowlist means all namespaces. This keeps review/governance analytics interpretable without exporting names or performing enrichment queries. IDs remain linkable/personal data, NOT anonymized; restrict the allowlist, receiver access, retention and destination accordingly. +- Never exported: skill/namespace names or slugs, package/body/metadata content, titles, summaries, rejection comments/reasons, submitter/owner identity, email/name, query terms, tokens/passwords/cookies, signed URLs, IP, UA, arbitrary request headers, request IDs, MDC or baggage. Authentication headers are transport-only, not attributes. + +中文:独立记录器,不桥接应用日志,不更改 Trace;Resource 仅包含服务名、版本及环境。白名单最多 16 个属性、字符串最多 256 个 UTF-16 单元。默认包括私有技能及空间的 ID,但不包括名称/内容;ID 仍可关联个人或资源,须按敏感数据管理。命名空间白名单建议上线前明确限制。正文、原因、密钥、签名 URL、IP/UA、任意请求头和 baggage 一律不输出。 + +## Complete configuration / 完整配置 + +All settings are startup-only: recreate/restart the server after changes. Property prefix is `skillhub.observability.business-logs.`. Environment suffixes below use `SKILLHUB_BUSINESS_LOGS_`. `application.yml` and release Compose explicitly map them; manual SDK builders DO NOT automatically read `OTEL_*`. / 所有配置启动时读取,需重建/重启;以下环境变量使用统一前缀,不自动读取 `OTEL_*`。 + +| Property suffix / 属性 | Env suffix / 环境变量后缀 | Default / 默认 | Meaning / 范围与含义 | +| --- | --- | --- | --- | +| `enabled` | `ENABLED` | `false` | Opt-in, independent of tracing / 独立开关 | +| `endpoint` | `ENDPOINT` | empty | Required when enabled; full HTTP(S) Logs URL, used AS-IS, normally `/v1/logs`; NOT `/v1/traces`. No userinfo/query/fragment or known metadata literal targets / 启用必填完整 Logs URL,不追加路径 | +| `headers` | `HEADERS` | empty | At most 16 comma-separated URL-encoded `name=value` pairs; total 8192 characters; ASCII values, no controls / 最多 16 对 URL 编码请求头,总长 8192;使用 Secret | +| `timeout` | `TIMEOUT` | `3s` | SDK HTTP timeout; 100ms..30s / 有界超时 | +| `compression` | `COMPRESSION` | `gzip` | `gzip` or `none` | +| `max-queue-size` | `MAX_QUEUE_SIZE` | `2048` | 1..65536 admitted records INCLUDING queued and in-flight, not bytes or precommit callbacks / 接纳后的队列及在途记录,不按字节,不包括提交前回调 | +| `max-export-batch-size` | `MAX_EXPORT_BATCH_SIZE` | `256` | 1..min(4096,queue size) / 每批上限 | +| `schedule-delay` | `SCHEDULE_DELAY` | `1s` | 10ms..60s; SDK batch schedule / 批处理调度间隔 | +| `shutdown-timeout` | `SHUTDOWN_TIMEOUT` | `5s` | 100ms..30s; maximum caller wait for graceful drain / 退出等待上限 | +| `retry-max-attempts` | `RETRY_MAX_ATTEMPTS` | `3` | 1..5 INCLUDING first request; 1 disables retry / 含首次请求,1 为不重试 | +| `retry-initial-backoff` | `RETRY_INITIAL_BACKOFF` | `200ms` | 1ms..5s, no greater than max backoff / 初始退避 | +| `retry-max-backoff` | `RETRY_MAX_BACKOFF` | `1s` | 1ms..10s, SDK exponential multiplier 2 with jitter / 最大退避 | +| `event-allowlist` | `EVENT_ALLOWLIST` | all 9 catalog names | Comma-separated exact names; empty exports none; unknown name fails startup / 精确事件名,空为不导出,未知值启动失败 | +| `namespace-allowlist` | `NAMESPACE_ALLOWLIST` | empty | Empty = all including private; otherwise up to 1000 positive numeric namespace IDs, comma-separated / 空为全部(含私有),否则最多 1000 个正整数 ID | + +Related existing Resource settings: `spring.application.name` (`skillhub`, environment `SPRING_APPLICATION_NAME`), `skillhub.observability.service-version` (`SKILLHUB_SERVICE_VERSION`, `unknown`), `skillhub.observability.service-environment` (`SKILLHUB_SERVICE_ENVIRONMENT`, `local`; release Compose `production`). These strings are bounded to 256 code units. Do not put credentials in service identity. / 沿用既有服务标识配置;不要在服务标识中放凭证。 + +A generic OTLP base endpoint normally appends a signal path in OTel autoconfiguration; THIS setting is Logs-specific, uses the given URL unchanged, and requires an explicit non-root path. Invalid enabled configuration fails startup without echoing endpoint/header values. Localhost/private-network receivers are valid. Destination is trusted operator configuration, not an end-user setting: use egress firewall/NetworkPolicy and DNS controls to allow only the authorized receiver and block cloud metadata, including aliases/rebinding. The managed 1.31 OkHttp sender follows redirects and exposes no redirect switch on its public Logs exporter builder: configure the receiver not to redirect and enforce destination/HTTPS restrictions at egress. Literal validation is not a complete SSRF firewall. / 本配置不是通用 base endpoint,不追加信号路径;允许自托管内网目标。1.31 发送器会跟随重定向,公开 builder 无关闭开关,接收端应禁止重定向,并配合出口/DNS/HTTPS 策略;字面量校验不能替代防火墙。 + +## Direct receiver / 直连部署 + +For an OTLP/HTTP Logs-compatible receiver, set these on the official server container; tracing may remain off: + +```dotenv +SKILLHUB_BUSINESS_LOGS_ENABLED=true +SKILLHUB_BUSINESS_LOGS_ENDPOINT=https://analytics.example.com/v1/logs +SKILLHUB_BUSINESS_LOGS_NAMESPACE_ALLOWLIST=7,12 +SKILLHUB_BUSINESS_LOGS_EVENT_ALLOWLIST=skill.publication.effective,skill.download.prepared,skill.review.approved +``` + +Use the release Compose file plus deployment-controlled environment/Secret injection. Never commit real `HEADERS`; e.g. its format is `Authorization=Bearer%20` (encode comma, equals, percent and literal plus as needed; `+` decodes to space). Do not run `docker compose config` into shared logs with secrets loaded. Across trust boundaries require HTTPS and receiver-supported authentication. JVM default trusted certificates apply; for private CAs use a deployment-provided JVM truststore, not an insecure TLS bypass. mTLS can be terminated by a local Collector/proxy. / 通过部署侧 Secret 注入请求头;跨信任边界必须 HTTPS/鉴权,私有 CA 使用 JVM truststore;不要关闭 TLS 校验,不要公开渲染含密钥的 Compose 配置。 + +Docker Secret alternative (no credential in Compose interpolation): mount a Secret whose target filename is `skillhub.observability.business-logs.headers`, and set `SPRING_CONFIG_IMPORT=optional:configtree:/run/secrets/`. Spring config-tree keys override the lower-priority application default. Do not simultaneously define a nonempty `SKILLHUB_BUSINESS_LOGS_HEADERS`. Manage the actual Secret file outside the repository and protect access to Actuator/config diagnostics. / 可用 Docker Secret + Spring configtree,文件名即完整属性名;真实 Secret 文件放仓库外,不与非空环境变量重复设置。 + +Helm exposes `server.businessLogs.enabled`, `endpoint`, and `headersSecretRef.{name,key}`. The chart references an EXISTING Secret and does not create inline header secrets. Use existing `server.extraEnv` for every tuning/filter/Resource/Trace environment variable in the table. Do not duplicate the three built-in env names in extraEnv. / Helm 提供开关、目标及现有 Secret 引用;预算/过滤等使用 `server.extraEnv`,不要重复内建环境变量。 + +```yaml +server: + businessLogs: + enabled: true + endpoint: https://analytics.example.com/v1/logs + headersSecretRef: + name: business-logs-credentials + key: headers + extraEnv: + - name: SKILLHUB_BUSINESS_LOGS_NAMESPACE_ALLOWLIST + value: "7,12" +``` + +## Optional Collector and real receive/query example / 可选 Collector 实收查询 + +The pinned example is `docs/examples/business-event-logs/compose.yaml` (Collector contrib `0.111.0`, 128 MiB / 0.5 CPU, only local HTTP port). It writes OTLP JSON to an ephemeral file, NOT production durable storage. Do not expose its unauthenticated port externally. It is separate from release Compose and is not started by default. / 示例仅本地验证,JSON 文件随容器删除,不是生产持久存储;无鉴权端口不得对外开放,不默认随正式部署启动。 + +From repository root, with port 4318 free: + +```bash +docker compose -f docs/examples/business-event-logs/compose.yaml up -d +# Host-run app: SKILLHUB_BUSINESS_LOGS_ENDPOINT=http://127.0.0.1:4318/v1/logs +# Docker Desktop server container: use host.docker.internal:4318 instead of localhost. +# Linux containers: use a shared private network / explicit host-gateway routing. +# Enable business Logs on the app; publish/approve or request an authorized published download. +docker cp skillhub-1009-collector:/business-logs.json ./business-logs.jsonl +jq -r '.resourceLogs[].scopeLogs[].logRecords[] | (.attributes | map({key:.key,value:(.value.stringValue // .value.intValue // .value.boolValue)}) | from_entries) as $a | [.timeUnixNano, $a["skillhub.event.name"], $a["skillhub.skill.id"], $a["log.record.uid"]] | @tsv' business-logs.jsonl +docker compose -f docs/examples/business-event-logs/compose.yaml down +``` + +For a source-verifiable run of the REAL production exporter wiring against the same Collector config, without bootstrapping business accounts: + +```bash +cd server +TESTCONTAINERS_RYUK_DISABLED=true ./mvnw -pl skillhub-app -am test \ + -Dtest=BusinessLogsCollectorTest -Dsurefire.failIfNoSpecifiedTests=false \ + -Dskillhub.test.evidence-dir=/absolute/operator-owned/evidence +``` + +Create that evidence directory first. The test uses named disposable containers and copies the actual sink output to `collector-received.jsonl`; apply the same jq query. It exercises all nine catalog records in all three Trace configurations (SDK trace sampling 0), checks IDs, UID uniqueness, field exclusion and receiver ACK metrics. This is real exporter/receiver integration, not a claim of a complete browser/CLI deployment smoke or loading an external Java agent. / 该测试使用真实应用导出装配及真实 Collector,保存实收结果,可用相同 jq 查询;不等同于完整 Web/CLI 部署验收或真实 Java Agent 注入测试。 + +Collector is optional. Production operators choose exporters, retention, TLS/auth, memory limiter and persistent sending queues appropriate to their sink. Collector persistence cannot recover records lost before application admission. File demo has no retention policy and must be stopped after verification. / 生产出口、保留策略、持久队列由运维选择;Collector 持久化不能补回应用侧漏报,示例用后停止。 + +## Reliability, metrics and troubleshooting / 可靠性、指标与排障 + +Business threads only map fixed bounded fields, capture context, register a transaction callback and attempt nonwaiting SDK admission. No DB enrichment or network IO is performed by the adapter on business threads. Standard BatchLogRecordProcessor handles a bounded record queue; a conservative admission permit counts queued PLUS in-flight records to expose exact queue-full drops without SDK internals. Only SDK exporters implement network retry (retryable HTTP status/IO); no caller-runs fallback. Batch export budget is `(timeout + retry-max-backoff) * retry-max-attempts + 1s`; receiver `Retry-After` and sender behavior still depend on the managed SDK. / 业务线程仅做固定字段映射及非等待入队,不发送网络请求;队列预算包含在途,重试交标准 SDK。 + +Loss is expected on commit-to-listener/admission gaps, crashes, forced exits, queue saturation, timeout or exhausted retry, and incomplete shutdown. No disk spool, outbox, replay, backfill or exactly-once. Ambiguous acknowledgement can duplicate the same UID; deduplicate by `log.record.uid`, not `(skill,version)` (multiple legitimate downloads), request ID or trace ID. Mapping a new domain-event dispatch creates a new UID; idempotency of repeated business operations is unchanged. / 提交到入队间隙、崩溃、满队列及重试耗尽都可能漏报;超时重试可能重复,接收方按 UID 去重,不按技能版本或 Trace 去重。 + +The queue budget starts at after-commit admission. Before commit, transaction synchronizations retain one bounded mapped snapshot per dispatched event until completion, outside that SDK budget. Avoid unbounded bulk work in one outer transaction; this configuration is not a process-wide heap cap. / 队列预算从提交后接纳开始;提交前每个事件的有界快照由事务回调保留,不计入 SDK 预算。不要在单个外层事务里执行无界批量操作,这不是整个进程的堆内存上限。 + +Graceful shutdown stops new admission, gives SDK drain a `shutdown-timeout` wait budget, then shuts down the exporter to cancel pending network work. Local cleanup and scheduling add some overhead; this is not a hard-real-time deadline. Remaining records may be lost; daemon completion may continue briefly. This differs deliberately from the draft RFC's discard-unsent-on-disable wording: configuration is NOT hot-reloaded; setting `enabled=false` takes effect on restart, whose old process gets a bounded drain opportunity. Already accepted receiver data cannot be recalled. / 正常退出停止接纳并限时等待排空,另有本地清理和调度开销,不是硬实时期限;剩余记录可丢失。关闭开关需重启,旧进程会限时发送,不承诺立即撤销已入队或接收的数据。 + +OTel 1.31 treats all HTTP 2xx responses as full success and does not expose `partial_success` rejected counts. `exported` means SDK/receiver ACK, NOT downstream storage confirmation. Do not replay whole partially accepted batches. Partial rejection diagnostics and durable retries belong at the receiver; this change does not upgrade the BOM. / 1.31 不读取部分拒收数量;成功指标不代表最终落库,不整批重发部分接收的数据。 + +Micrometer names on the existing protected Actuator metrics surface (Prometheus renders dots as underscores): + +| Metric | Unit / 含义 | +| --- | --- | +| `skillhub.business.logs.enqueued` | Records admitted / 已接纳记录 | +| `skillhub.business.logs.dropped{reason=queue_full}` | Records refused because outstanding budget is full / 满队列丢弃 | +| `skillhub.business.logs.dropped{reason=shutdown}` | New records refused during closing / 退出时拒绝新记录 | +| `skillhub.business.logs.errors` | Adapter mapping/admission/lifecycle exceptions, swallowed / 适配器异常(不传播) | +| `skillhub.business.logs.export.failures` | Batches whose SDK export fails/throws after retries / 最终失败批次 | +| `skillhub.business.logs.exported` | Records in SDK-successful batches; may include receiver partial rejections / SDK 成功记录,含部分拒收盲区 | + +No per-user/skill/namespace metric tags. Adapter diagnostics are fixed text, at most once per minute per pipeline, without exception messages, headers or event payloads; SDK diagnostics use its own throttling. / 指标不带高基数用户/资源标签,适配器诊断每分钟最多一次固定文字。 + +| Symptom / 现象 | Check / 排查 | +| --- | --- | +| Startup fails / 启动失败 | Enabled without endpoint, bad full URL, invalid bounds/allowlist/header encoding; do not paste config secrets / 检查目标、范围、编码,不分享 Secret | +| No records, no exporter metrics / 无记录或指标 | Default off? Environment explicitly forwarded? Container recreated? No historical backfill / 开关、透传、重建、新事件 | +| Enqueued but failed / 已入队但失败 | `/v1/logs`, HTTP/protobuf (not gRPC 4317), DNS/network/TLS/auth and receiver Logs pipeline / 路径、协议、网络及鉴权 | +| Drops / 丢弃 | Outstanding queue + in-flight, receiver throughput; tune bounded budgets, not an unbounded queue / 检查接收吞吐与在途占用,不改无界 | +| Missing one approval fact / 少一条批准事实 | Source vs target namespace allowlist; content vs metadata review; transaction rollback / 来源与目标空间过滤、内容与元数据审核区别、回滚 | +| Duplicate counts / 重复 | UID dedup, receiver ack ambiguity, agent bridges; never bridge ordinary app logs into this scope / UID 去重、确认歧义、Agent 日志桥接 | +| Exported count exceeds sink count / 指标大于入库量 | 2xx partial rejection blind spot, downstream filters/retention; inspect receiver / 部分拒收盲区及下游过滤 | + +Trace modes remain independent: `none`/`external-agent` still forbid the TRACE endpoint, not the business Logs endpoint. External agents must not bridge/re-export this dedicated scope or attach a second business Logs processor; verify real deployment agent plugins for duplicates. No actual external-agent installation is implied by application configuration tests. / Trace 限制仅针对 Trace 目标;部署方须检查 Agent 插件是否重复导出业务记录。 + +For executed verification and explicit remaining limits, see [acceptance matrix](../openspec/changes/add-otlp-business-event-logs/acceptance.md). diff --git a/docs/observability-developer-guide.md b/docs/observability-developer-guide.md index 405105edd..fae365a7e 100644 --- a/docs/observability-developer-guide.md +++ b/docs/observability-developer-guide.md @@ -18,15 +18,19 @@ 通过 `SKILLHUB_TRACING_MODE` 选择一种模式,修改后重启应用: -- `none`:默认模式。无应用内 OTel SDK 和 OTLP 导出,只保留 `request.id`。 +- `none`:默认 Trace 模式。无应用内 Trace SDK 和 Trace OTLP 导出,只保留 `request.id`;独立业务 Logs 开关不受此限制。 - `otel-sdk`:使用 Micrometer Tracing + OTel Bridge;配置 `MANAGEMENT_OTLP_TRACING_ENDPOINT` 后才向 Collector 导出。 - `external-agent`:应用内 Tracer 为 NOOP,由部署环境提供唯一的外部 Agent。 SkillHub 只能校验自身配置,不能识别任意 JVM Agent;唯一 Agent 是部署检查项。 -`none`/`external-agent` 不能配置 OTLP endpoint;`otel-sdk` 与外部 Tracing Agent +`none`/`external-agent` 不能配置 TRACE OTLP endpoint;`otel-sdk` 与外部 Tracing Agent 不得在同一进程中叠加。 +业务分析使用独立、默认关闭的 LoggerProvider,不桥接应用日志或更改 Trace 采样。 +生产点只增加领域快照,提交后才入队;事件与字段必须通过统一 mapper 白名单。 +开发/部署契约见[业务事件 Logs](./business-event-logs.md),不要把审计或可靠通知改接为有损遥测。 + ## 3. 开发者接入方式 ### 3.1 普通 HTTP 请求 diff --git a/docs/skillhub/en/guide/deployment.md b/docs/skillhub/en/guide/deployment.md index 275b3b92c..70b64a8cb 100644 --- a/docs/skillhub/en/guide/deployment.md +++ b/docs/skillhub/en/guide/deployment.md @@ -38,6 +38,16 @@ Resolve validation errors before starting. Backend startup runs Flyway database Suite review writes are enabled by default in the single-server release Compose topology; Suite Bundle confirmation is disabled by default. See [Bundle import](suite-bundle.md) for enabling it. For rolling upgrades where old and new backends coexist, follow the [Suite deployment order](https://github.com/iflytek/skillhub/blob/main/docs/25-skill-suites.md). +## Optional business-event OTLP Logs + +Business analytics is disabled by default and independent of all three Trace modes. With the matching +official server image, enable `SKILLHUB_BUSINESS_LOGS_ENABLED=true` and a full +`SKILLHUB_BUSINESS_LOGS_ENDPOINT`; Collector is optional and authentication headers belong in deployment +Secrets. See the [business-event operations guide](https://github.com/iflytek/skillhub/blob/main/docs/business-event-logs.md) +for publication, download preparation, review, promotion, archive, visibility and yank semantics, private-ID +scope, every setting, a real receive/query example and loss/duplicate/shutdown limits. Download preparation +is not completion or installation; this lossy stream is not audit, billing or reliable notification. + ## Upgrade, stop, and troubleshoot Before upgrading, record the current image version and configuration, and back up both the database and skill object storage. Set `SKILLHUB_VERSION` to the target release and repeat validation, `pull`, `up -d`, and workflow verification. Rolling back an image does not roll back the database; check migration compatibility first. diff --git a/docs/skillhub/guide/deployment.md b/docs/skillhub/guide/deployment.md index ae8f69703..4b17983ed 100644 --- a/docs/skillhub/guide/deployment.md +++ b/docs/skillhub/guide/deployment.md @@ -38,6 +38,15 @@ docker compose --env-file .env.release -f compose.release.yml logs --tail=100 se Suite 审核写入在单机 release Compose 中默认开启;Suite Bundle 确认默认关闭。需要批量导入时,按 [Bundle 导入](suite-bundle.md#部署开关)启用对应开关。滚动升级新旧后端共存时,另按[套装部署顺序](https://github.com/iflytek/skillhub/blob/main/docs/25-skill-suites.md#部署顺序)操作。 +## 可选的业务事件 OTLP Logs + +业务分析默认关闭,与三种 Trace 模式独立。对应版本官方镜像可通过 +`SKILLHUB_BUSINESS_LOGS_ENABLED=true` 和完整 `SKILLHUB_BUSINESS_LOGS_ENDPOINT` 启用; +Collector 可选,请求头凭证使用部署 Secret。发布、下载准备、审核、提升、归档、可见性和 yank 的 +精确定义、私有 ID 范围、完整配置表、实收查询示例及丢失/重复/退出限制见 +[业务事件运行手册](https://github.com/iflytek/skillhub/blob/main/docs/business-event-logs.md)。 +下载准备不是下载完成或安装成功,不能用于审计、计费或可靠通知。 + ## 升级、停止与排障 升级前保存当前镜像版本、配置,并备份数据库和技能对象存储。修改 `SKILLHUB_VERSION` 为目标发布版本,重新执行校验、`pull`、`up -d` 和业务验证。回退镜像不等于回退数据库;先检查目标迁移的兼容性。 diff --git a/openspec/changes/add-otlp-business-event-logs/acceptance.md b/openspec/changes/add-otlp-business-event-logs/acceptance.md new file mode 100644 index 000000000..bfac53fb7 --- /dev/null +++ b/openspec/changes/add-otlp-business-event-logs/acceptance.md @@ -0,0 +1,60 @@ +# Acceptance evidence — OTLP business-event Logs + +Base: `9ebb99226dd1b3e1d4fd8dd3e36baae70a4ed273`; branch `feat/otlp-business-events-1009`. These are executed results, not a claim of a deployed browser/CLI production stack. Raw logs and receiver output are retained in the implementation worker's evidence directory; the worker report gives absolute paths. / 以下为实际运行结果,不等同完整生产部署验收;原始输出路径见 worker report。 + +## Event/producer matrix + +`BusinessLogsProducerPostgresTest` uses actual Spring service proxies, real JPA repositories, Flyway migrations and PostgreSQL 16. Storage/scanner/avatar/notification/audit/compensation collaborators and package-validation/metadata-parsing collaborators are isolated mocks; business state and transaction boundaries are real. Each row has a committed scenario and a rollback scenario. Rollback emits zero records. / 服务代理、仓储、迁移和事务为真实依赖;存储、扫描、头像、通知、审计、补偿及包校验/解析协作组件用 mock 隔离,不宣称验证它们的端到端副作用。 + +| Producer path | Committed facts (exactly) | Rollback | Actor | +| --- | --- | --- | --- | +| Privileged auto-publication | publication effective | 0 | privileged submitting user | +| PRIVATE confirmation | publication effective | 0 | confirmer | +| Content review approval | publication effective + review approved | 0 | actual reviewer, not owner | +| Content review rejection | review rejected | 0 | actual reviewer | +| Initial promotion approval | target publication effective + source promotion approved | 0 | actual approver | +| Promotion update approval | target publication effective + source promotion approved | 0 | actual approver | +| Promotion rejection | source promotion rejected | 0 | actual reviewer | +| Archive | status changed ACTIVE → ARCHIVED | 0 | owner | +| Unarchive | status changed ARCHIVED → ACTIVE | 0 | owner | +| Immediate narrowing | visibility changed to PRIVATE | 0 | requester | +| Metadata-review expansion | visibility changed to PUBLIC, no new publication | 0 | expansion approver | +| Published version yank | version yanked | 0 | admin actor | +| Published download inside outer transaction | download prepared | 0 | anonymous: user type without user.id | + +Result: 39 real-service/JPA scenarios passed in `final-focused-3.log`: the 26 commit/rollback scenarios above plus 13 denied-actor cases, one for every path, asserting no exported fact and unchanged persisted skill/version/review/promotion state. Includes PRIVATE publication and review-task rollback assertions, actor checks and sensitive-field exclusion. Dedicated nontransactional real-download-service test separately passed: one anonymous preparation; no facts for review package or denied anonymous/unauthorized PRIVATE access. / 上表 26 个提交/回滚场景及逐路径 13 个拒绝授权场景通过;拒绝时不发事件、持久状态不变。无事务真实下载服务另验证一次匿名准备,以及审核包、匿名/无权限私有请求不发事件。 + +## Cross-cutting acceptance + +| Requirement | Executed assertion | Result / evidence | +| --- | --- | --- | +| TDD | New mapper/config tests compiled before implementation and failed on missing implementation symbols | RED `red-unit.log`; initial GREEN 12 tests in `green-unit-2.log` | +| Mapper boundary | Exact fixed keys; no rejection reason/submitter/unknown event serialization; namespace/event allowlists; anonymous omission; system actor; length limits; reserved built-in publisher | PASS `BusinessEventMapperTest` | +| Default off | No pipeline/listener/provider bean; no default receiver | PASS `BusinessLogsConfigurationTest` | +| Invalid configuration | Enabled+empty URL rejected; metadata/userinfo/query/file URL and bad budgets/event names rejected without echoing supplied URL; URL-encoded headers and control-character rejection | PASS `BusinessLogsConfigurationTest` | +| Capture before commit | Valid unsampled span survives scope end before PostgreSQL commit; implementation reads standard SpanContext only, not request/MDC strings | PASS `BusinessLogsTransactionTest`; mapper/listener source boundary | +| Trace coexistence | Actual production configuration exports 9 records each with none, otel-sdk (sampling=0), external-agent mode | PASS, actual Collector output 27 records / 9 names / 27 unique UIDs | +| Actual OTLP receive/query | Pinned Collector contrib 0.111.0 file exporter; documented jq query run against real received JSON | PASS `collector-received.jsonl`, `collector-query.tsv`, `collector-query-summary.json` | +| Queue full, deterministic | 1000 caller events with blocked exporter; exactly 999 drops; bounded shutdown | PASS final focused run: 12 ms caller loop, 169 ms close for 100 ms wait budget (includes scheduling/cleanup overhead) | +| Live receiver stalled | Pause real Collector; queue=2, batch=1, HTTP timeout=500ms; 1000 caller events; failures counted | PASS final focused run: 10 ms caller loop, 998 drops | +| Live receiver down | Stop real Collector; publishing returns without exception; export failure counter increments | PASS focused run: caller 0 ms | +| Duplicate semantics | HTTP receiver first responds 503 then 200; retry bytes identical and include the same UID; auth appears only in transport header | PASS `BusinessLogsHttpTest`, 2 requests / 1 SDK-exported record | +| Partial rejection blind spot | Real protobuf HTTP 200 partial_success rejected=1; no whole-batch replay; SDK reports success | PASS expected SDK limitation, 1 request / 1 SDK-success counter | +| Resource bounds | Java test fork capped at 512 MiB; Collector Docker inspect verifies 128 MiB and 0.5 CPU; at most 16 record attributes; string cap tested | PASS focused command with `-DargLine=-Xmx512m`; not a throughput/byte-accurate heap benchmark | +| Existing consumers | All affected domain service tests, 20 notification tests and 13 Trace configuration/filter tests | PASS final focused run: 289 total tests, no failures/errors/skips | +| Release config | Actual make target with a synthetic valid fixture (removed afterward); existing release shell regression suite | PASS `validate-release-config.log`, `release-config-tests.log`; no real credential file used | +| Helm/Compose | Helm 3.17.3 lint/render, existing chart contracts; new Python deployment contracts assert default-off, endpoint requirement, Secret reference and env forwarding | PASS 1 chart, 4 Python tests without skips (`business-logs-config-with-helm-final.log`), `helm-contracts.log` | +| Backend packaging | Reactor app+dependencies package with tests explicitly skipped (tests verified separately) | PASS `backend-package-final.log` | +| Local doc/source checks | Changed Markdown relative targets and code-fence pairing; required OpenSpec headings | PASS 11 files, 19 relative targets, 5 requirements / 9 scenarios (`docs-source-check.json`); not a site build or OpenSpec CLI run | + +The raw measurement values above are from the named focused run, not promised production latency percentiles. Shutdown has a bounded SDK wait budget plus local cleanup/scheduling overhead. / 延迟数字是该次测试实测,不是生产百分位承诺;退出等待预算之外有本地清理和调度开销。 + +## Broad suite and environment limits + +Broad `make test-backend` runs are NOT green. Initial candidate: 2732 tests, 2 failures, 1 error, 30 skips. Final candidate broad run: 2762 tests, 3 failures, 11 errors, 30 skips. Frozen base: 2712 tests, 2 failures, 2 errors, 30 skips. Failures are in GitHub integration tests; exact failing methods are retained in the count JSONs. Four methods occur in both the final candidate's and base's broad failure sets (Go executor, credential privacy, two budget cases), but the full failure sets and error modes vary. Do NOT infer that every candidate failure was proven pre-existing. Subsequent identical five-class GitHub focused commands PASS all 37 tests on BOTH the candidate and frozen base. An independent SQL/host clock probe found the Docker database clock ahead of the host in every sample, consistent with timing-sensitive eligibility checks, but it is NOT asserted as the proven cause of every failure. No clock/global configuration was changed. The authorized detached base worktree was removed. / 全量检查不是绿色;基线与 candidate 有四个相同失败方法,但集合及错误形式不同,不冒称全部故障已证实为基线问题。随后相同五类 GitHub 定向测试在两边均 37 项通过。独立探测发现 Docker 数据库时钟领先宿主机,与时间筛选类故障相符,但未证明所有故障根因;未改系统时钟/全局配置。基线临时工作树已移除。 + +The 13 added denial cases and two mapper edge assertions were finalized after the last broad run compiled its test classes; all final sources were then exercised by the successful 289-test focused run. / 最后一次全量编译测试类后补齐 13 项拒绝授权及两个 mapper 边界断言,最终源码由 289 项定向测试实际验证。 + +Docs site build was attempted. VitePress was absent; `npm ci` was blocked by the environment's network guard for both the lockfile registry and an explicit npmjs alternative. This is reported as BLOCKED, not passing. Markdown/source/link checks are separate from a built site. / 文档站构建受依赖下载网络限制,未冒称通过。 + +Not executed: real external Java-agent/plugin installation, TLS/mTLS production deployment, a browser/CLI deployment smoke, forced-process-crash recovery or sustained throughput/heap profiling. There is no outbox/replay guarantee to test. Agent duplicate suppression, authorized egress/DNS/redirect policy, production Secret provisioning, receiver persistence and retention remain deployment responsibilities. All application mode combinations were exercised without installing a Java agent. / 未执行真实 Agent 注入、生产 TLS/mTLS、完整 Web/CLI 部署或持续负载测试;相关限制和部署责任明确保留。 diff --git a/openspec/changes/add-otlp-business-event-logs/design.md b/openspec/changes/add-otlp-business-event-logs/design.md new file mode 100644 index 000000000..5a5ffd3e0 --- /dev/null +++ b/openspec/changes/add-otlp-business-event-logs/design.md @@ -0,0 +1,39 @@ +## Architecture + +Domain records retain their original accessors and compatibility constructors, and gain an immutable `BusinessEventSnapshot` (namespace ID, visibility, opaque actor/type, fixed source, Instant). Every production publisher of the nine included event types supplies this snapshot. No domain OTel dependency is introduced. Notification/search listeners keep their existing semantics. Missing legacy snapshots are ignored, not enriched asynchronously. The promotion rejection path adds one scalar-source lookup because its old event/request lacks source namespace/visibility; it does not export reasons or submitters. + +The conditional app configuration owns a dedicated `SdkLoggerProvider`, BatchLogRecordProcessor and OTLP/HTTP Logs exporter. It exposes only the pipeline/listener as beans, not a global/provider bean that Boot might consume for Trace autoconfiguration. Resource is explicitly constructed from existing service settings, not detected. No application logs are bridged; no trace sampling/configuration is altered. + +## Commit and context boundary + +A synchronous Spring `@EventListener` maps fixed scalar data and captures only valid OTel SpanContext when the producer dispatches. It registers `TransactionSynchronization.afterCommit` for actual transactions, otherwise admits immediately. This is equivalent to AFTER_COMMIT + fallbackExecution but preserves the producer's span even if the span ends before commit. Each boundary catches telemetry RuntimeExceptions and counts them with bounded fixed-text diagnostics. No network IO, enrichment queries or space-waiting occurs in the listener. Disabled configuration creates no listener, provider, worker, exporter or OTel context objects. + +Occurrence is `Instant.now()` immediately before producer publication, not the independently clocked persisted publishedAt or commit time. Sources identify domain branch, not caller-controlled Web/CLI labels. Forced internal auto-publication is a system actor; privileged human auto-publication remains a human actor. Anonymous downloads have actor type user and no user.id (user means user-request origin, not proof of an identified human). Review and promotion use the actual approver, not owner/uploader. Promotion decisions use source skill/namespace; effective publication uses the derived target. + +## Bounds and observability + +Standard SDK batching is retained. A nonwaiting semaphore around processor admission conservatively bounds queued PLUS in-flight records to `max-queue-size`; SDK queue capacity is at least that large, so accepted records cannot hit its hidden overflow boundary. A measured exporter releases permits when the SDK result completes and counts exported records or failed batches. This supplies exact admission/full metrics without reflection or a custom network queue/retry engine. SDK MeterProvider is explicitly noop; exporter health uses the existing Micrometer registry without global registration. + +A single mapper allows only known event classes, fixed sources/actor types, scalar IDs, enums and bounded user ID. The SDK also limits 16 attributes and 256 code units. Three Resource strings are separately bounded. Configuration has hard count/duration/header/list bounds. Admitted pipeline memory is bounded by record count and fixed record shape, not a promised byte-accurate or zero-allocation budget. Before commit, transaction synchronizations retain one bounded mapped snapshot per dispatched event outside the SDK admission window; callers must not wrap unbounded bulk work in one transaction. This is not a process-wide heap cap. Saturation latency and receiver container limits are measured in tests; this is not a production throughput benchmark. + +## SDK compatibility verified against 1.31.0 + +Boot remains 3.2.3, OTel remains BOM-managed 1.31.0. Only an explicit managed sdk-logs dependency is added. EventName API is absent: use custom `skillhub.event.name`. No global semconv schema is advertised; scope contract version is 1 and experimental log.record.uid semantics are documented locally. + +The public HTTP Logs exporter builder supports RetryPolicy; set explicit finite attempts/backoff, or leave retry disabled for attempts=1. No retry code is implemented here. Logs endpoint is explicit and used as-is. Builders do not read OTEL_* automatically. The SDK treats 2xx as success without inspecting partial-success counts; counters reflect that limitation and tests send a real protobuf partial-success response. Retry tests verify identical payload/UID and transport-only headers. + +The managed OkHttp sender uses its default redirect policy; the public 1.31 Logs builder has no redirect-policy customization. Do not fork the sender, use reflection or introduce an unrelated HTTP dependency: document nonredirecting receivers and enforce authorized destinations/TLS with deployment egress policy. This is an explicit limit relative to the draft RFC's redirect-validation aspiration, not a claim of full in-process SSRF prevention. Its retry interceptor bounds jittered backoff and does not honor Retry-After; SDK timeout/attempt limits may terminate before the configured maximum attempts. + +## Data and destination policy + +Private resource IDs are included by default only after instance operators opt in; empty namespace allowlist means all namespaces. No names/slugs/content are exported, avoiding entity serialization/enrichment and keeping governance facts available. IDs are linkable data, not anonymized. Namespace and exact event allowlists are startup-only. Credentials are deployment headers/Secrets only; endpoint userinfo/query/fragment and known metadata literal destinations are rejected without echoing input. Localhost/private receivers are allowed. Operators still own egress/DNS policy, TLS/truststore, retention and receiver access. Literal URL validation cannot prevent all metadata aliases/rebinding. + +## Reliability and RFC deviations + +Best-effort analytics only; commit-to-admission gaps, crashes, overflow, timeout, shutdown and receiver failures can lose data. No outbox/replay/exactly-once. Same SDK retry reuses mapped UID; a new domain dispatch gets a new UID. Approval and publication are intentionally distinct facts. + +No runtime hot-reload. On restart to disable, the old process gets a bounded graceful drain opportunity rather than the RFC draft's immediate unsent cancellation. New admission stops on close; provider shutdown is joined for shutdown-timeout and the exporter is canceled at deadline. Accepted receiver data cannot be withdrawn. An active external-agent JVM/plugin installation, TLS/mTLS deployment and browser/CLI end-to-end smoke are separate deployment verifications, not inferred from application mode tests. + +## Scope + +All nine existing facts are included. Metadata/avatar review decisions with no existing decision event, hide/unhide, promotion revocation, submission/withdrawal, Suite events, search/view, transfer completion, installation, durable delivery and historical backfill are deferred. Existing metadata-review effective visibility events ARE included. The event catalog and full operator configuration live in docs/business-event-logs.md; executed results and limitations live in acceptance.md and the worker report. diff --git a/openspec/changes/add-otlp-business-event-logs/proposal.md b/openspec/changes/add-otlp-business-event-logs/proposal.md new file mode 100644 index 000000000..3a9531fac --- /dev/null +++ b/openspec/changes/add-otlp-business-event-logs/proposal.md @@ -0,0 +1,23 @@ +## Why + +Self-hosted deployments need vendor-neutral business analytics using the official image and deployment configuration, without a fork or a bespoke webhook protocol. Issue #1009 and maintainer comment 6463066865 authorize existing domain events → dedicated OpenTelemetry Logs → OTLP → operator receiver. + +## What Changes + +- Add a default-off, independent OTLP/HTTP protobuf business Logs adapter in app. +- Snapshot occurrence time, opaque actor, namespace ID and visibility at existing domain producers; keep notification/search semantics and permissions unchanged. +- Include publication (all four producers), published-download preparation, content-review decisions, promotion decisions, archive/unarchive, effective visibility change and version yank. +- Bound admission, attributes, retry, timeout and shutdown; expose Micrometer analytics-export health counters. +- Document event semantics, deployment, data boundary, real receiver/query example and acceptance evidence alongside implementation. + +## Capabilities + +### New Capabilities +- `business-event-logs`: operator-controlled, lossy business analytics using OTLP Logs. + +### Modified Capabilities +None. No API, package, CLI, storage schema, authorization or audit contract changes. + +## Scope and deferrals + +Not audit, billing, notification, outbox, replay, historical backfill or exactly-once delivery. No application-log bridge, new trace pipeline, vendor SDK or BOM upgrade. Metadata/avatar review decisions without existing decision events, hide/unhide, promotion revocation, review submission/withdrawal, Suite events, search/views, stream completion and installation are deferred. Effective metadata-review visibility changes ARE included through their existing visibility event. No rejection comments, content, names/slugs, network identifiers or request headers are exported. diff --git a/openspec/changes/add-otlp-business-event-logs/specs/business-event-logs/spec.md b/openspec/changes/add-otlp-business-event-logs/specs/business-event-logs/spec.md new file mode 100644 index 000000000..dc9aa3f7c --- /dev/null +++ b/openspec/changes/add-otlp-business-event-logs/specs/business-event-logs/spec.md @@ -0,0 +1,64 @@ +## ADDED Requirements + +### Requirement: Business Logs SHALL be opt-in and independent of Trace + +The application SHALL create a dedicated LoggerProvider only when `skillhub.observability.business-logs.enabled=true`. It SHALL NOT register global OpenTelemetry, bridge application logs, change trace sampling or start a trace exporter. Empty endpoint while enabled SHALL fail startup without disclosing configuration values. + +#### Scenario: Default deployment +- **WHEN** no business Logs settings are provided +- **THEN** no business LoggerProvider, processor, exporter, worker or listener is created + +#### Scenario: Trace independence +- **WHEN** business Logs is enabled with tracing `none`, `otel-sdk` or `external-agent` +- **THEN** business records are accepted independently of trace sampling +- **AND** only valid current span context is associated; request ID/MDC is never converted into trace identity + +### Requirement: Existing business facts SHALL retain producer meaning + +The event catalog in `docs/business-event-logs.md` SHALL define all included events, producer paths, occurrence boundary, actor and fields. Publication SHALL cover auto-publication, PRIVATE confirmation, review approval and promotion approval. Download preparation SHALL NOT mean transfer, installation or execution success. + +#### Scenario: Commit and rollback +- **WHEN** a supported event is published within a transaction +- **THEN** the adapter captures a scalar snapshot and trace context before handoff +- **AND** export admission occurs only after commit; rollback emits no record + +#### Scenario: Nontransactional download +- **WHEN** an authorized PUBLISHED download successfully builds its result without an outer transaction +- **THEN** one download-prepared record is admitted +- **AND** a rejected request, preview/review package, Suite plan or metadata poll emits no such record + +### Requirement: Export SHALL have bounded, nonblocking business-thread cost + +The adapter SHALL use asynchronous SDK batching and standard bounded exporter retry, never caller-runs network IO. Record count, batch size, attribute count/length, timeout, retry attempts/backoff and shutdown wait SHALL be bounded. Telemetry exceptions SHALL NOT propagate into business calls; failures SHALL be counted with bounded diagnostic logging. + +#### Scenario: Receiver outage and saturation +- **WHEN** the receiver stalls or fails and capacity is exhausted +- **THEN** business calls do not wait for receiver IO or queue space +- **AND** admission drops and failed exports are observable through existing Actuator metrics + +#### Scenario: Ambiguous acknowledgement +- **WHEN** an accepted batch is retried after an ambiguous network failure +- **THEN** the same `log.record.uid` survives retries +- **AND** receivers may deduplicate, without an exactly-once promise + +### Requirement: Data export SHALL use a single explicit allowlist + +A single mapper SHALL export only fixed documented scalar fields, including IDs of private resources. Namespace IDs and event names SHALL be operator-filterable. Human actors SHALL use only local opaque `user.id`; anonymous/unknown and system actors SHALL omit it. No entity serialization, enrichment queries, resource autodetection, content, credentials, URLs, IP/UA, request IDs or raw headers is permitted. + +#### Scenario: Sensitive rejection fields +- **WHEN** a domain rejection carries arbitrary reason or submitter information +- **THEN** neither field is copied into business attributes or body + +#### Scenario: Private resource scope +- **WHEN** a private resource is within the configured namespace/event allowlists +- **THEN** only IDs, visibility and documented operational fields are exported +- **AND** names, slugs and content remain excluded + +### Requirement: Deployment and delivery limitations SHALL be explicit + +User documentation SHALL include every configuration key/default/bound, direct and Collector deployment, Secret injection, loss/duplicate/shutdown semantics, SDK compatibility limits, troubleshooting, a reproducible real receiver/query example and an acceptance matrix separating executed from unverified scenarios. + +#### Scenario: Graceful exit +- **WHEN** the process shuts down +- **THEN** new admission stops and queued records get only the configured bounded drain opportunity +- **AND** remaining records may be lost; already accepted receiver data cannot be recalled diff --git a/openspec/changes/add-otlp-business-event-logs/tasks.md b/openspec/changes/add-otlp-business-event-logs/tasks.md new file mode 100644 index 000000000..c38dc9706 --- /dev/null +++ b/openspec/changes/add-otlp-business-event-logs/tasks.md @@ -0,0 +1,26 @@ +## Implementation + +- [x] Inventory every existing producer/consumer for the nine included event types; preserve notification/audit/permission behavior. +- [x] Add scalar producer snapshots (time, namespace ID, visibility, actor/type, fixed source), without domain telemetry dependencies. +- [x] Add a dedicated default-off Logs pipeline; no global registration, application-log bridge or second Trace pipeline. +- [x] Capture valid span identity before handoff; admit only after commit with nontransactional fallback. +- [x] Central allowlist mapper, private-ID policy, stable per-record UUID, fixed attribute/string limits. +- [x] Standard SDK batching/retry, bounded outstanding admission, timeout and shutdown; fixed-cardinality Micrometer counters and rate-bounded safe diagnostics. +- [x] Explicit application/release environment mappings, release validator, Helm defaults and existing-Secret reference; no new vendor/third-party dependency or BOM upgrade. +- [x] Bilingual operator documentation, deployment links, event catalog, configuration table, direct/Collector and Secret guidance. + +## Verification + +- [x] Write initial mapper/configuration tests before implementation; retain RED compiler output and subsequent GREEN output. +- [x] Execute mapper/filter/anonymous/system/UID/config validation tests. +- [x] Execute PostgreSQL transaction listener commit/rollback tests for all nine catalog facts, nontransactional real download/permission tests and unsampled context capture. +- [x] Execute SDK HTTP retry/identical-payload UID/auth-only tests and partial-success blind-spot test. +- [x] Execute real Collector receive-and-query with all nine events and three Trace configurations. +- [x] Execute real-service/JPA commit/rollback and denied-actor matrix for all producer paths (39 scenarios). +- [x] Execute paused/stopped live Collector saturation/outage and record measured latency/resource limits. +- [x] Execute synthetic release validation, existing release regression checks, Helm lint/render/contracts and new deployment configuration tests. +- [x] Finish broad backend/base runs and report reproduced versus unreproduced failures explicitly (full suite remains FAIL; identical GitHub focused checks pass on both). +- [x] Run final focused checks after the last code edits, query actual receiver evidence with documented jq command, validate local docs/links and diff (site build BLOCKED by dependency-network guard). +- [x] Verify cleanup of every owned temporary container/network/volume and the authorized baseline worktree; leave changes uncommitted. + +See acceptance.md for final executed results and explicit unverified deployment scenarios; a task checkbox is not itself acceptance evidence.