diff --git a/docs/business-event-logs.md b/docs/business-event-logs.md index 01d8cbfdd..2188caa97 100644 --- a/docs/business-event-logs.md +++ b/docs/business-event-logs.md @@ -150,13 +150,13 @@ Micrometer names on the existing protected Actuator metrics surface (Prometheus | `skillhub.business.logs.export.failures` | Batches whose SDK export fails/throws, exceeds the export budget, or is still outstanding at close / 最终失败、超时或关闭时仍未结束的批次 | | `skillhub.business.logs.exported` | Records in SDK-successful batches; may include receiver partial rejections / SDK 成功记录，含部分拒收盲区 | -No per-user/skill/namespace metric tags. Adapter diagnostics are fixed text, at most once per minute per pipeline, without exception messages, headers or event payloads: `Business Logs telemetry failed; inspect skillhub.business.logs metrics and receiver health`. Shipped `logback-spring.xml` sets `io.opentelemetry.exporter` and `io.opentelemetry.sdk.logs` to OFF by default, including their child loggers routed from JUL by Spring Boot. Raw SDK diagnostics can echo receiver error bodies or transport exception text containing credentials; throttling or ERROR level alone is NOT safe. This also silences shared OTel exporter diagnostics for Trace; Trace export behavior is unchanged. Custom logging configurations must retain these OFF rules. / 指标无高基数标签；适配器每分钟最多一次固定文字。默认关闭 OTel exporter 与 sdk.logs 原始诊断（包括子记录器及 JUL 桥接），其响应或异常文本可能回显凭证，ERROR 级别也不安全；同时静默共享 Trace exporter 诊断但不改变导出。自定义日志配置需保留这些规则。 +No per-user/skill/namespace metric tags. Adapter diagnostics are fixed text, at most once per minute per pipeline, without exception messages, headers or event payloads: `Business Logs telemetry failed; inspect skillhub.business.logs metrics and receiver health`. Raw SDK diagnostics can echo receiver error bodies or transport exception text containing credentials; throttling or ERROR level alone is NOT safe. Shipped `logback-spring.xml` therefore installs `BusinessLogsDiagnosticsFilter`, which drops `io.opentelemetry.*` / `okhttp3.*` events (including those routed from JUL by Spring Boot) only on the business exporter's own network threads, and sets `io.opentelemetry.sdk.logs` (used only by this pipeline) to OFF. Shared exporter loggers keep their normal levels, so existing Trace export diagnostics such as `Failed to export spans ... 401` are unchanged whether business Logs are on or off. Custom logging configurations must retain the filter and the `sdk.logs` OFF rule. / 指标无高基数标签；适配器每分钟最多一次固定文字。原始 SDK 诊断可能回显凭证，ERROR 级别也不安全；默认配置只在业务 exporter 自己的网络线程上丢弃 OTel/OkHttp 诊断（含 JUL 桥接），并关闭仅本管道使用的 sdk.logs。共享 exporter 记录器级别不变，Trace 导出错误提示无论业务 Logs 开关均保持原样。自定义日志配置需保留该过滤器和 sdk.logs 规则。 -Logger OFF alone does not contain OTel 1.31's uncaught asynchronous callback failures (for example, an HTTP 401 with a malformed chunked body). The production business exporter is constructed in a private ThreadGroup: only its `okhttp-dispatch-*` threads' uncaught exceptions are discarded without rendering or forwarding the throwable. This increments `errors` and retains the same fixed warning; the existing completion deadline separately counts one failed batch and recovers permits. The JVM default handler is never changed. Other threads, including same-named threads outside that group and shared OkHttp housekeeping threads, delegate to their normal parent/default handler unchanged. Pipeline shutdown cancels its dispatcher executor; containment stays attached until its last callback exits, then no owned dispatcher remains. An inert group can outlive the pipeline for shared OkHttp housekeeping, but delegates those exceptions and only weakly references metrics. / 仅关闭 logger 不足以阻止 1.31 异步回调异常直写 stderr。业务 exporter 在私有线程组内构建，只隔离该组导出 dispatcher 的未捕获异常，计 errors 并保留固定告警；预算到期另计一批失败并恢复许可。不改 JVM 默认 handler，其它线程仍按原处理链执行。关闭时取消导出线程，隔离保留至回调退出；共享 housekeeping 可保留空闲边界组，但不受抑制且不强引用指标。 +Log filtering alone does not contain OTel 1.31's uncaught asynchronous callback failures (for example, an HTTP 401 with a malformed chunked body). The production business exporter is constructed in a private ThreadGroup: only its `okhttp-dispatch-*` threads' uncaught exceptions are discarded without rendering or forwarding the throwable. This increments `errors` and retains the same fixed warning; the existing completion deadline separately counts one failed batch and recovers permits. The JVM default handler is never changed. Other threads, including same-named threads outside that group and shared OkHttp housekeeping threads, delegate to their normal parent/default handler unchanged. Pipeline shutdown cancels its dispatcher executor; containment stays attached until its last callback exits, then no owned dispatcher remains. An inert group can outlive the pipeline for shared OkHttp housekeeping, but delegates those exceptions and only weakly references metrics. / 仅过滤日志不足以阻止 1.31 异步回调异常直写 stderr。业务 exporter 在私有线程组内构建，只隔离该组导出 dispatcher 的未捕获异常，计 errors 并保留固定告警；预算到期另计一批失败并恢复许可。不改 JVM 默认 handler，其它线程仍按原处理链执行。关闭时取消导出线程，隔离保留至回调退出；共享 housekeeping 可保留空闲边界组，但不受抑制且不强引用指标。 -This safe-diagnostics boundary covers this dedicated business Logs pipeline with the shipped logger OFF defaults and the verified managed 1.31 sender. It is not process-wide exception redaction: unrelated Trace/agent exporters, application failures, custom logging/SDK overrides, agents intercepting exceptions, heap dumps and receiver-side logs are outside it. SDK upgrades must reverify dispatcher ownership/naming and the captured stdout/stderr tests; no BOM upgrade is included here. / 安全诊断范围限于默认日志规则和已验证的 1.31 业务管道，不是整个 JVM 的异常脱敏；其它 exporter、应用异常、配置覆盖、Agent、堆转储和接收端日志不在范围内。升级 SDK 必须重验线程归属与输出回归。 +This safe-diagnostics boundary covers this dedicated business Logs pipeline with the shipped diagnostics filter defaults and the verified managed 1.31 sender. It is not process-wide exception redaction: unrelated Trace/agent exporters keep their own (unchanged) diagnostics, and application failures, custom logging/SDK overrides, agents intercepting exceptions, heap dumps and receiver-side logs are outside it. SDK upgrades must reverify that the sender reports results on dispatcher threads owned by the private group, plus the captured stdout/stderr and Trace-coexistence tests; no BOM upgrade is included here. / 安全诊断范围限于默认日志规则和已验证的 1.31 业务管道，不是整个 JVM 的异常脱敏；Trace 等其它 exporter 保留各自原有诊断，应用异常、配置覆盖、Agent、堆转储和接收端日志不在范围内。升级 SDK 必须重验回调线程归属、输出回归与 Trace 共存测试。 -Troubleshoot using the protected Actuator counters above, this fixed warning, receiver-side status/health, and independent DNS/network/TLS/auth checks with secret-safe tools. If raw SDK diagnostics are indispensable, first remove production credentials or reproduce with synthetic credentials in an isolated environment, restrict log access/retention, and knowingly accept disclosure risk. Temporarily set the exact Spring property `logging.level.io.opentelemetry.exporter.internal.http.HttpExporter=DEBUG` (e.g. a deployment-owned config file or `SPRING_APPLICATION_JSON`); `logging.level.io.opentelemetry.sdk.logs=DEBUG` opts into Logs processor diagnostics separately. Logger names are case-sensitive; do not rely on relaxed uppercase env names for the class logger. Restore both package defaults to OFF and remove any child-logger override after diagnosis, then restart; do not share unredacted SDK output. Raising root logging alone does not override explicit package OFF. / 排障优先使用受保护指标、固定告警及接收端健康，不输出 Secret。确需 SDK 原始细节时，在隔离环境用假凭证复现，限制访问与保留并明确接受泄露风险；临时提高精确日志属性（类名区分大小写），结束后恢复 OFF、删除子记录器覆盖并重启，勿分享未脱敏输出。 +Troubleshoot using the protected Actuator counters above, this fixed warning, receiver-side status/health, and independent DNS/network/TLS/auth checks with secret-safe tools. If raw SDK diagnostics are indispensable, first remove production credentials or reproduce with synthetic credentials in an isolated environment, restrict log access/retention, and knowingly accept disclosure risk. Business exporter raw diagnostics are dropped by the filter regardless of logger level; reproduce with a deployment-owned logging config that omits `BusinessLogsDiagnosticsFilter` (and, for processor diagnostics, sets `io.opentelemetry.sdk.logs` to DEBUG), then restore the shipped config and restart; do not share unredacted SDK output. / 排障优先使用受保护指标、固定告警及接收端健康，不输出 Secret。确需 SDK 原始细节时，在隔离环境用假凭证复现，限制访问与保留并明确接受泄露风险；业务 exporter 的原始诊断不受日志级别影响，需用部署方自有日志配置临时去掉过滤器复现，结束后恢复默认配置并重启，勿分享未脱敏输出。 | Symptom / 现象 | Check / 排查 | | --- | --- | diff --git a/openspec/changes/add-otlp-business-event-logs/acceptance.md b/openspec/changes/add-otlp-business-event-logs/acceptance.md index 2ff94f91a..d0987a53e 100644 --- a/openspec/changes/add-otlp-business-event-logs/acceptance.md +++ b/openspec/changes/add-otlp-business-event-logs/acceptance.md @@ -86,6 +86,20 @@ An initial RED fixture run (`red-chunked-diagnostics.log`) failed 2 startup asse Scope: containment is private to this verified managed business sender, with shipped SDK logger OFF defaults. Non-dispatcher housekeeping delegates unchanged; its inert group may remain alive but weakly references metrics. No global handler is installed/restored; containment lasts through owned callback exit. Unrelated exporters/application failures, explicit logging/SDK overrides, exception-intercepting agents, heap dumps and receiver logs remain outside the guarantee. SDK upgrades require revalidation. Broad backend/base comparisons, packaging and docs-site build were not rerun in Repair 2; previous limitations below remain unchanged. `git diff --check` passes; owned containers/networks/volumes are cleaned up. / 隔离非全 JVM 脱敏，版本与默认日志范围及剩余部署责任详见设计和运维文档；本轮未重跑全量后端、打包或文档站构建，不冒称消除既有环境限制。 +## Repair 3 — maintainer review (PR #1029, `bd30ac1`) + +Finding: the global `io.opentelemetry.exporter = OFF` rule also hid existing Trace exporter errors when business Logs were disabled (reviewer reproduced 2 Trace 401s with 0 error lines vs. 2 `Failed to export spans` on main's logback config). / 维护者复现：全局 OFF 在业务 Logs 关闭时也屏蔽了既有 Trace 导出错误。 + +Change: removed the global exporter OFF; added `BusinessLogsDiagnosticsFilter` (Logback TurboFilter) that drops `io.opentelemetry.*` / `okhttp3.*` events only on threads of the business exporter's private `BusinessLogsExporterBoundary` group; kept `io.opentelemetry.sdk.logs` OFF (business pipeline only). / 去掉全局 OFF，改为只在业务 exporter 私有线程组上过滤 SDK 诊断。 + +| Scenario | Assertion | RED → GREEN | +| --- | --- | --- | +| Logs disabled + Trace 401 | Real `OtlpHttpSpanExporter` → local receiver returns 401 on `/v1/traces`; captured output contains `Failed to export spans`, `401` and the receiver message, text and JSON formats | RED with `bd30ac1` logback (await timeout, 0 lines); GREEN | +| Logs enabled + Trace 401 coexist | Business Logs 401 echoing synthetic credential and Trace 401 in one logging setup: Trace diagnostic present; credential and `Failed to export logs` absent; fixed business warning present | RED with `bd30ac1` logback (Trace line missing); GREEN | +| Sensitivity of the filter | Same tests with the global OFF removed but no filter: existing credential tests fail (6 failures: credential / `Failed to export logs` printed) | Proves the filter, not the logger level, provides the business containment | + +Focused command (Repair 1 set, JDK 21): **311 tests, 0 failures, 0 errors, 0 skips** (adds 4 Trace-coexistence cases). Broad suite, deployment contracts and real Collector run were not rerun in Repair 3; no deployment setting changed. / 定向 311 项全部通过；本轮未重跑全量与部署契约。 + ## Broad suite and environment limits Broad `make test-backend` runs are NOT green. Initial candidate: 2732 tests, 2 failures, 1 error, 30 skips. Final candidate broad run: 2762 tests, 3 failures, 11 errors, 30 skips. Frozen base: 2712 tests, 2 failures, 2 errors, 30 skips. Failures are in GitHub integration tests; exact failing methods are retained in the count JSONs. Four methods occur in both the final candidate's and base's broad failure sets (Go executor, credential privacy, two budget cases), but the full failure sets and error modes vary. Do NOT infer that every candidate failure was proven pre-existing. Subsequent identical five-class GitHub focused commands PASS all 37 tests on BOTH the candidate and frozen base. An independent SQL/host clock probe found the Docker database clock ahead of the host in every sample, consistent with timing-sensitive eligibility checks, but it is NOT asserted as the proven cause of every failure. No clock/global configuration was changed. The authorized detached base worktree was removed. / 全量检查不是绿色；基线与 candidate 有四个相同失败方法，但集合及错误形式不同，不冒称全部故障已证实为基线问题。随后相同五类 GitHub 定向测试在两边均 37 项通过。独立探测发现 Docker 数据库时钟领先宿主机，与时间筛选类故障相符，但未证明所有故障根因；未改系统时钟/全局配置。基线临时工作树已移除。 diff --git a/openspec/changes/add-otlp-business-event-logs/design.md b/openspec/changes/add-otlp-business-event-logs/design.md index bf9344d7b..3ff9f8493 100644 --- a/openspec/changes/add-otlp-business-event-logs/design.md +++ b/openspec/changes/add-otlp-business-event-logs/design.md @@ -16,17 +16,17 @@ Occurrence is `Instant.now()` immediately before producer publication, not the i Standard SDK batching is retained. A nonwaiting semaphore around processor admission conservatively bounds queued PLUS in-flight records to `max-queue-size`; SDK queue capacity is at least that large, so accepted records cannot hit its hidden overflow boundary. A measured exporter owns a first-completion-wins result: SDK completion or the existing `(timeout + retry-max-backoff) * retry-max-attempts + 1s` deadline releases permits and updates counters exactly once. This handles OTel 1.31 async response-body exceptions that abandon their result; deadline/close completion counts one failed batch and late SDK completion is ignored. One daemon scheduler removes canceled deadlines; close stops it and fails outstanding guards. It bounds local accounting, not remote delivery/cancellation. This supplies exact admission/full metrics without reflection or a custom network queue/retry engine. SDK MeterProvider is explicitly noop; exporter health uses the existing Micrometer registry without global registration. -Shipped Logback configuration sets `io.opentelemetry.exporter` and `io.opentelemetry.sdk.logs` (and inherited child loggers) OFF. These JUL diagnostics are bridged by Boot and may otherwise disclose arbitrary receiver Status messages or transport exception text echoing headers; ERROR-only is insufficient. SkillHub's fixed warning (at most once/minute/pipeline) and Micrometer counters remain the operator signal. This also suppresses shared Trace exporter diagnostics, not its exports. Custom logging must retain these defaults; a deliberate case-sensitive `logging.level` override can temporarily enable unsafe details, preferably only with synthetic credentials and restricted logs, then restore OFF/remove child overrides. The operator guide documents that troubleshooting path. +OTel exporter diagnostics are bridged from JUL by Boot and may disclose arbitrary receiver Status messages or transport exception text echoing headers; ERROR-only is insufficient. The shipped Logback configuration installs `BusinessLogsDiagnosticsFilter`, a TurboFilter that denies `io.opentelemetry.*` / `okhttp3.*` events only when the current thread belongs to the business exporter's private `BusinessLogsExporterBoundary` group (the managed 1.31 sender reports results on those dispatcher threads), and sets `io.opentelemetry.sdk.logs`, used only by this pipeline, OFF. Shared exporter loggers keep their configured level, so Trace (`otel-sdk`) export failures such as HTTP 401 keep their original `Failed to export spans` diagnostics whether business Logs are disabled or enabled. SkillHub's fixed warning (at most once/minute/pipeline) and Micrometer counters remain the business Logs operator signal. Custom logging must retain the filter and the `sdk.logs` rule; the operator guide documents the isolated troubleshooting path. ### Scoped asynchronous exception boundary (Repair 2) -The 1.31 `HttpExporter` can throw from OkHttp's callback while parsing a malformed non-2xx response body; logger OFF and the accounting deadline do not stop the JVM uncaught handler printing that throwable to stderr. `BusinessLogsExporterBoundary` constructs only this exporter on a short-lived initializer in a private ThreadGroup. Its uncaught handler suppresses only threads whose group is that exact object AND whose restored factory name starts `okhttp-dispatch-`. It never inspects/formats/forwards the throwable, increments the existing adapter error counter and emits only the fixed warning; the completion guard independently records one failed batch and releases permits at its deadline. Initializer failures pass through a FutureTask and become a fixed cause-free startup exception; an interrupted caller finishes ownership transfer and restores its interrupt flag. Failure while constructing the pipeline cancels the built exporter and throws fixed text without the raw cause. +The 1.31 `HttpExporter` can throw from OkHttp's callback while parsing a malformed non-2xx response body; log filtering and the accounting deadline do not stop the JVM uncaught handler printing that throwable to stderr. `BusinessLogsExporterBoundary` constructs only this exporter on a short-lived initializer in a private ThreadGroup. Its uncaught handler suppresses only threads whose group is that exact object AND whose restored factory name starts `okhttp-dispatch-`. It never inspects/formats/forwards the throwable, increments the existing adapter error counter and emits only the fixed warning; the completion guard independently records one failed batch and releases permits at its deadline. Initializer failures pass through a FutureTask and become a fixed cause-free startup exception; an interrupted caller finishes ownership transfer and restores its interrupt flag. Failure while constructing the pipeline cancels the built exporter and throws fixed text without the raw cause. Why this rather than a global handler or new HTTP stack: `javap` against the actual managed 1.31 artifacts confirms the public `OtlpHttpLogRecordExporterBuilder` has NO transport/executor/thread-factory setter. `OkHttpUtil.newDispatcher()` constructs a per-exporter executor with `DaemonThreadFactory("okhttp-dispatch")`; its `Executors.defaultThreadFactory()` captures the constructing thread's group. This JDK ThreadGroup mechanism needs no SDK-private imports/reflection, dependency or BOM change and retains standard HTTP/retry behavior. The narrow version-specific assumption is dispatcher construction/group capture and restored naming, protected by the real receiver regression. Future SDK upgrades must rerun it and revisit the boundary, not assume compatibility. There is no process-wide installation or restoration: the default handler is never set by production code, no thread scans mutate handlers, and multiple pipelines own independent groups. The scope is attached at exporter construction, before any export, and retired with that exporter's dispatcher executor at shutdown. It MUST remain attached to late callbacks even after close returns rather than exposing a teardown race. Group members other than dispatchers delegate through `super.uncaughtException` unchanged. OkHttp may first initialize shared housekeeping in this group; such threads are NOT suppressed and can keep the otherwise inert group alive. Its metrics reference is weak so this cannot retain the closed pipeline/registry. Java 21 reclaims groups when unreachable; no deprecated `ThreadGroup.destroy()` or global cleanup is needed. Tests assert actual production dispatchers use the private group, vanish after close, and never change the previous default handler. Guard tests additionally throw on an unrelated same-named thread outside the group and a non-dispatcher inside it: both reach the previous default handler with the original throwable. -Together with the shipped logger OFF rules and synchronous cause-free catches, this prevents raw business-exporter failure text in application stdout/stderr/logs, including malformed chunked 401 responses. It is deliberately NOT process-wide redaction: unrelated Trace/agent exporters, application failures, explicit logging/SDK overrides, exception-intercepting agents, heap dumps and receiver logs are outside the guarantee. No previously installed handler is silenced for those threads. The existing shared Trace logger suppression remains the disclosed operational tradeoff; the new exception boundary adds no global handler side effect. +Together with the shipped diagnostics filter and synchronous cause-free catches, this prevents raw business-exporter failure text in application stdout/stderr/logs, including malformed chunked 401 responses. It is deliberately NOT process-wide redaction: unrelated Trace/agent exporters, application failures, explicit logging/SDK overrides, exception-intercepting agents, heap dumps and receiver logs are outside the guarantee. No previously installed handler is silenced for those threads. Trace exporter diagnostics are not suppressed; the new exception boundary adds no global handler side effect. A single mapper allows only known event classes, fixed sources/actor types, scalar IDs, enums and bounded user ID. The SDK also limits 16 attributes and 256 code units. Three Resource strings are separately bounded. Configuration has hard count/duration/header/list bounds. Admitted pipeline memory is bounded by record count and fixed record shape, not a promised byte-accurate or zero-allocation budget. Before commit, transaction synchronizations retain one bounded mapped snapshot per dispatched event outside the SDK admission window; callers must not wrap unbounded bulk work in one transaction. This is not a process-wide heap cap. Saturation latency and receiver container limits are measured in tests; this is not a production throughput benchmark. diff --git a/openspec/changes/add-otlp-business-event-logs/specs/business-event-logs/spec.md b/openspec/changes/add-otlp-business-event-logs/specs/business-event-logs/spec.md index 4725b1791..b04f4b9ef 100644 --- a/openspec/changes/add-otlp-business-event-logs/specs/business-event-logs/spec.md +++ b/openspec/changes/add-otlp-business-event-logs/specs/business-event-logs/spec.md @@ -76,7 +76,8 @@ A single mapper SHALL export only fixed documented scalar fields, including IDs - **WHEN** MDC and OTel Baggage contain distinct sensitive sentinels - **THEN** neither keys nor values SHALL appear in exported attributes, body or resource - **WHEN** receiver Status messages or transport exceptions echo transport credentials -- **THEN** shipped logging SHALL suppress raw `io.opentelemetry.exporter` and `io.opentelemetry.sdk.logs` diagnostics, including child loggers, by default +- **THEN** shipped logging SHALL suppress raw OTel/OkHttp diagnostics emitted on the business exporter's own threads and `io.opentelemetry.sdk.logs` diagnostics by default +- **AND** existing Trace exporter diagnostics SHALL remain unchanged whether business Logs are disabled or enabled - **AND** fixed-text rate-bounded SkillHub diagnostics and counters SHALL remain available - **AND** operator documentation SHALL warn that explicitly raising SDK logging opts into possible credential disclosure diff --git a/server/skillhub-app/src/main/java/com/iflytek/skillhub/observability/business/BusinessLogsDiagnosticsFilter.java b/server/skillhub-app/src/main/java/com/iflytek/skillhub/observability/business/BusinessLogsDiagnosticsFilter.java new file mode 100644 index 000000000..7b77ccb2c --- /dev/null +++ b/server/skillhub-app/src/main/java/com/iflytek/skillhub/observability/business/BusinessLogsDiagnosticsFilter.java @@ -0,0 +1,28 @@ +package com.iflytek.skillhub.observability.business; + +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.Logger; +import ch.qos.logback.classic.turbo.TurboFilter; +import ch.qos.logback.core.spi.FilterReply; +import org.slf4j.Marker; + +/** + * Drops raw OTel/OkHttp diagnostics emitted on the business exporter's own network threads. + * Those messages can echo receiver bodies or transport exception text containing credentials. + * The managed 1.31 sender reports results on its dispatcher threads, which are all created in + * {@link BusinessLogsExporterBoundary}; Trace and other exporters share the same logger names but + * run on other threads, so their diagnostics keep the logging configuration's normal behavior. + */ +public final class BusinessLogsDiagnosticsFilter extends TurboFilter { + @Override + public FilterReply decide(Marker marker, Logger logger, Level level, String format, Object[] params, + Throwable throwable) { + if (logger == null) { return FilterReply.NEUTRAL; } + String name = logger.getName(); + if ((name.startsWith("io.opentelemetry.") || name.startsWith("okhttp3.")) + && Thread.currentThread().getThreadGroup() instanceof BusinessLogsExporterBoundary) { + return FilterReply.DENY; + } + return FilterReply.NEUTRAL; + } +} diff --git a/server/skillhub-app/src/main/resources/logback-spring.xml b/server/skillhub-app/src/main/resources/logback-spring.xml index 803ac9180..5217a0491 100644 --- a/server/skillhub-app/src/main/resources/logback-spring.xml +++ b/server/skillhub-app/src/main/resources/logback-spring.xml @@ -45,10 +45,10 @@ - - + + diff --git a/server/skillhub-app/src/test/java/com/iflytek/skillhub/observability/business/BusinessLogsDiagnosticsTest.java b/server/skillhub-app/src/test/java/com/iflytek/skillhub/observability/business/BusinessLogsDiagnosticsTest.java index 4b88b7beb..19296d80a 100644 --- a/server/skillhub-app/src/test/java/com/iflytek/skillhub/observability/business/BusinessLogsDiagnosticsTest.java +++ b/server/skillhub-app/src/test/java/com/iflytek/skillhub/observability/business/BusinessLogsDiagnosticsTest.java @@ -36,15 +36,83 @@ class BusinessLogsDiagnosticsTest { environment.getPropertySources().addFirst(new MapPropertySource("diagnostic-test", java.util.Map.of("skillhub.observability.log-format", format))); logging.initialize(new LoggingInitializationContext(environment), "classpath:logback-spring.xml", null); - for (String name : java.util.List.of("io.opentelemetry.exporter.internal.http.HttpExporter", - "io.opentelemetry.exporter.sender.okhttp.internal.OkHttpHttpSender", - "io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor")) { - org.slf4j.LoggerFactory.getLogger(name); - assertThat(logging.getLoggerConfiguration(name).getEffectiveLevel()) - .isEqualTo(org.springframework.boot.logging.LogLevel.OFF); + String processor = "io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor"; + org.slf4j.LoggerFactory.getLogger(processor); + assertThat(logging.getLoggerConfiguration(processor).getEffectiveLevel()) + .isEqualTo(org.springframework.boot.logging.LogLevel.OFF); + } + + @ParameterizedTest + @ValueSource(strings = {"text", "json"}) + void businessLogsDisabledKeepsExistingTraceExportFailureDiagnostics(String format, CapturedOutput output) + throws Exception { + configureShippedLogging(format); + try (var receiver = new UnauthorizedReceiver("/v1/traces")) { + exportSpanFailure(receiver.endpoint()); + assertThat(receiver.requests.get()).isEqualTo(1); + org.awaitility.Awaitility.await().untilAsserted(() -> assertThat(output.getAll()) + .contains("Failed to export spans", "401", "trace-receiver-rejected")); } } + @ParameterizedTest + @ValueSource(strings = {"text", "json"}) + void businessLogsEnabledKeepsTraceDiagnosticsAndHidesBusinessReceiverResponse(String format, + CapturedOutput output) throws Exception { + configureShippedLogging(format); + try (var traces = new UnauthorizedReceiver("/v1/traces"); + var logs = new UnauthorizedReceiver("/v1/logs")) { + exportFailure(logs.endpoint()); + exportSpanFailure(traces.endpoint()); + assertThat(logs.requests.get()).isEqualTo(1); + assertThat(traces.requests.get()).isEqualTo(1); + org.awaitility.Awaitility.await().untilAsserted(() -> assertThat(output.getAll()) + .contains("Business Logs telemetry failed", "Failed to export spans")); + assertThat(output.getAll()).doesNotContain(CREDENTIAL, "Failed to export logs"); + } + } + + private static void exportSpanFailure(String endpoint) { + var exporter = io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter.builder() + .setEndpoint(endpoint).setTimeout(java.time.Duration.ofSeconds(2)).build(); + var provider = io.opentelemetry.sdk.trace.SdkTracerProvider.builder() + .addSpanProcessor(io.opentelemetry.sdk.trace.export.SimpleSpanProcessor.create(exporter)).build(); + try { + provider.get("trace-diagnostics-test").spanBuilder("trace-401").startSpan().end(); + assertThat(provider.forceFlush().join(5, TimeUnit.SECONDS).isSuccess()).isFalse(); + } finally { provider.shutdown().join(5, TimeUnit.SECONDS); } + } + + /** Receiver that rejects with 401 and echoes a marker (the synthetic credential for /v1/logs). */ + private static final class UnauthorizedReceiver implements AutoCloseable { + private final HttpServer server; + private final String path; + final java.util.concurrent.atomic.AtomicInteger requests = new java.util.concurrent.atomic.AtomicInteger(); + + UnauthorizedReceiver(String path) throws java.io.IOException { + this.path = path; + server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); + server.createContext(path, exchange -> { + exchange.getRequestBody().readAllBytes(); + requests.incrementAndGet(); + String echoed = path.endsWith("/logs") ? CREDENTIAL : "trace-receiver-rejected"; + byte[] text = ("auth failed for Bearer " + echoed).getBytes(StandardCharsets.UTF_8); + byte[] body = new byte[text.length + 4]; + body[0] = 0x08; body[1] = 0x10; body[2] = 0x12; body[3] = (byte) text.length; + System.arraycopy(text, 0, body, 4, text.length); + exchange.getResponseHeaders().add("Content-Type", "application/x-protobuf"); + exchange.sendResponseHeaders(401, body.length); + exchange.getResponseBody().write(body); + exchange.close(); + }); + server.start(); + } + + String endpoint() { return "http://127.0.0.1:" + server.getAddress().getPort() + path; } + + @Override public void close() { server.stop(0); } + } + @AfterEach void cleanupLogging() { if (logging != null) {