Commit graph

3 commits

Author SHA1 Message Date
ishaan-berri
03ba79d261
feat(lens): show end-user feedback on traces, stored in ClickHouse (#45171)
* feat(lens): add lens_feedback ClickHouse table for human trace scores

ReplacingMergeTree(UpdatedAt, IsDeleted) keyed like agent_traces_by_key so
feedback joins traces in sort order, with a bloom filter on TraceId, a score
CHECK, and the same retention as traces

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): add scoped ClickHouse reads for trace feedback

feedback_target resolves a visible trace's team, key and trace_ref before a
write; feedback lists the latest live entry per author; feedback_summary
returns count, average and lowest score for a batch of traces

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lens): pin the Claude session to trace id hash shared with feedback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): expose feedback queries to the Python trace bridge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): add typed feedback models and ClickHouse feedback store

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): add /lens/feedback API for 0-10 human scores on traces

PUT saves or replaces the caller's score and comment, GET lists every
entry, DELETE removes the caller's, POST /summary feeds the trace list.
Callers can target a trace_id or a Claude Code session_id

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): add typed feedback calls to the traces API

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): flag rated traces and filter the list by feedback

A Feedback column shows each run's average score and rating count, marked
red when any score is 4 or below, and a filter narrows to rated or
low-score runs through the URL

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): show and edit human feedback inside a trace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): let app keys post end-user feedback on their own traces

Any key can now PUT/DELETE feedback on traces its team or key sent, naming
the end user in an optional user field. Reading feedback stays with Lens
admins

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): show end-user feedback first in a run and flag low-scored rows

Replace the edit popover and feedback filter with a read-only view: the
list shows each run's score in a fixed-width badge and tints the row red
when a user scored it 4 or below, and opening a run shows what users said
with their score right under the header

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:14:30 -07:00
ishaan-berri
bf9bd35469
feat(lens): scope traces to one agent with a header picker (#45202)
* feat(lens): add trace_agents rollup query

* feat(lens): register the trace_agents read query

* feat(lens): add trace_agents params and row types

* feat(lens): dispatch the trace_agents query

* feat(lens): export trace_agents wire schemas

* test(lens): pin the trace_agents query name

* test(lens): cover trace_agents scope, window and failure counts

* feat(lens): cap the agent list size

* feat(lens): declare the trace_agents bridge query

* feat(lens): read trace agents from clickhouse storage

* feat(lens): add trace agent response models

* feat(lens): list agents within the reader's trace scope

* feat(lens): add GET /v1/traces/agents

* chore(lens): regenerate trace models with trace_agents

* chore(lens): regenerate trace types with trace_agents

* chore(lens): regenerate read query name schema

* chore(lens): add trace agent row schema

* chore(lens): add trace agents params schema

* test(lens): cover the trace agents route

* test(lens): cover agent listing scope and timestamps

* chore(ui): regenerate api types with trace agents route

* feat(lens): derive trace agent types from the schema

* feat(lens): fetch the agent list from the traces api

* feat(lens): roll up demo runs into agents

* feat(lens): serve the agent list in demo data

* feat(lens): load agents seen in the last two weeks

* feat(lens): remember the selected agent per browser

* feat(lens): add the agent picker

* feat(lens): wire agent selection into the lens header

* feat(lens): show the agent picker next to the lens title

* refactor(lens): drop the toolbar agent filter in favor of the header picker

* test(lens): remove tests for the toolbar agent filter

* test(lens): cover agent resolution and rollup

* test(lens): cover scoping, switching and remembering the agent

* test(lens): read only the create request in guided setup

* test(lens): assert the toolbar agent filter is gone

* test(lens): read stubbed requests without their abort signal
2026-10-07 23:48:04 +00:00
devin-ai-integration[bot]
e340e546e2
feat(traces): tracing development seed (#44363)
* feat(dev): seed linked tracing and spend fixtures

* chore(dev): use OpenAI model in tracing config

* chore(dev): align tracing credentials with UI E2E

* fix(dev): update fixture seeder query scope

* feat(dev): seed linked tracing and spend fixtures

* chore(dev): use OpenAI model in tracing config

* chore(dev): align tracing credentials with UI E2E

* fix(dev): update fixture seeder query scope

* wip

* wip

* wip

* chore(trace): checkpoint ongoing Rust migration

* refactor(trace): group Python bridge under trace package

* refactor(traces): read span conventions through a Convention trait

Each span format (Claude Code, LangSmith, OpenInference, gen_ai) now lives under
normalize/convention/ as a unit struct implementing Convention, owning both its
detection and its extraction. Precedence is one ordered registry instead of an
if-chain in mod.rs that reached into each module differently.

The modules now share one way to read attributes: present() for the first
non-empty key and Payload for a text that also reports the key it consumed,
replacing three different idioms and the &mut Vec threaded through payload
readers. Instrumentation::adjust returns a new Extraction instead of mutating
one, with each SDK rule as its own function, and the LangChain middleware
suffix list exists once.

* feat(trace): export Rust-owned wire schemas and enforce contract bounds

* fix(trace): bound quoted counts in ClickHouse wire schemas

* feat(trace): generate Python wire contracts with datamodel-code-generator

* test(trace): validate migrated callers and generated contracts at the native boundary

* refactor(traces): rename normalization convention to format

* fix(traces): reconcile spend evidence and preserve unknown costs

* feat(traces): normalize additional telemetry formats

* test(traces): cover captured normalization fixtures

* refactor(traces): isolate SDK normalization rules

* feat(tracing): seed all trace exports for local dashboard

* fix(clickhouse): preserve custom LiteLLM request metadata

* docs(traces): define normalization module boundaries

* docs(traces): define resolution and OTLP boundaries

* fix(ui): normalize nullable trace message names

* refactor(traces): split resolver modules and cover resolution behavior

* test(traces): replace normalization snapshots with behavior assertions

* fix(ui): align dashboard API contracts with generated types

* refactor(traces): type normalization and storage boundaries

* fix(traces): seed captured SDK spend and preserve provider identities

* wip

* test(traces): verify guide discovery and content ordering

---------

Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-03 09:59:51 +00:00