litellm/scripts
ishaan-berri 03ba79d261
feat(lens): show end-user feedback on traces, stored in ClickHouse (#45171)
* feat(lens): add lens_feedback ClickHouse table for human trace scores

ReplacingMergeTree(UpdatedAt, IsDeleted) keyed like agent_traces_by_key so
feedback joins traces in sort order, with a bloom filter on TraceId, a score
CHECK, and the same retention as traces

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): add scoped ClickHouse reads for trace feedback

feedback_target resolves a visible trace's team, key and trace_ref before a
write; feedback lists the latest live entry per author; feedback_summary
returns count, average and lowest score for a batch of traces

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(lens): pin the Claude session to trace id hash shared with feedback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): expose feedback queries to the Python trace bridge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): add typed feedback models and ClickHouse feedback store

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): add /lens/feedback API for 0-10 human scores on traces

PUT saves or replaces the caller's score and comment, GET lists every
entry, DELETE removes the caller's, POST /summary feeds the trace list.
Callers can target a trace_id or a Claude Code session_id

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): add typed feedback calls to the traces API

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): flag rated traces and filter the list by feedback

A Feedback column shows each run's average score and rating count, marked
red when any score is 4 or below, and a filter narrows to rated or
low-score runs through the URL

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): show and edit human feedback inside a trace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens): let app keys post end-user feedback on their own traces

Any key can now PUT/DELETE feedback on traces its team or key sent, naming
the end user in an optional user field. Reading feedback stays with Lens
admins

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(lens-ui): show end-user feedback first in a run and flag low-scored rows

Replace the edit popover and feedback filter with a read-only view: the
list shows each run's score in a fixed-width badge and tints the row red
when a user scored it 4 or below, and opening a run shows what users said
with their score right under the header

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:14:30 -07:00
..
adaptive_router_demo fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
health_check fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
trace_codegen feat(lens): show end-user feedback on traces, stored in ClickHouse (#45171) 2026-10-07 20:14:30 -07:00
auto-close-duplicates.test.ts fix(ci): keep earlier objections when the duplicate notice is re-posted 2026-08-29 17:30:39 -04:00
auto-close-duplicates.ts ci: classify new issues into domain, provider, kind, priority and lift labels 2026-09-18 11:37:55 -07:00
benchmark_anthropic_messages_perf.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
benchmark_chat_completions_perf.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
benchmark_mock.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
benchmark_model_response_creator.py fix: improve bedrock streaming hot path perf (#28720) 2026-05-28 11:31:37 -07:00
benchmark_ocr_callbacks.py refactor(ocr): complete native lifecycle and preserve Azure auth (#40734) 2026-09-12 11:56:49 -07:00
benchmark_proxy_vs_provider.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
benchmark_streaming_chunk_overhead.py feat: litellm oss 110626 (#30202) 2026-06-11 22:30:26 -07:00
budget_ratchet_check.py test(ci): prove RULE_CODES by running every checker rule 2026-09-20 08:27:02 +00:00
check_mcp_operation_boundary.py refactor(mcp): extract explicit operation context and dispatch 2026-09-21 12:24:15 -07:00
check_mcp_sdk_install.py fix(mcp): preserve legacy behavior on SDK2 and streamline verification 2026-09-18 22:28:31 -07:00
check_test_quality.py fix(ci): excuse retired test-quality rules in the budget ratchet 2026-09-20 08:12:39 +00:00
check_type_discipline.py chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
classify-issue.test.ts fix(issue-classifier): queue runs per issue and judge edits by live labels 2026-09-18 11:37:55 -07:00
classify-issue.ts fix(issue-classifier): queue runs per issue and judge edits by live labels 2026-09-18 11:37:55 -07:00
comment-fixed-issue.test.ts chore(ci): drop litellm_internal_staging and litellm_oss_staging references, main is the only trunk (#42745) 2026-09-23 08:14:11 -07:00
comment-fixed-issue.ts feat(ci): close open pull requests superseded by a merged fix on their linked issue (#42609) 2026-09-22 18:24:35 -07:00
create_litellm_branch.ps1 feat: add script to create branches with litellm_ prefix (#17606) 2025-12-06 10:41:39 -08:00
create_litellm_branch.sh enhance: create_litellm_branch tool to be more robust (#17874) 2025-12-12 05:35:50 -08:00
create_team_key_and_submit_guardrail.sh feat(guardrails): team-based guardrail registration and approval workflow (#22459) 2026-03-02 22:06:49 -08:00
default_branch.py ci: follow the default branch in development tooling 2026-09-07 14:34:45 -07:00
eval_compression.py Prompt Compression - add it to the proxy (#25729) 2026-04-20 15:08:00 -07:00
flag-duplicate-issue.test.ts ci(duplicate-check): drop the unused considered field from the verdict schema 2026-09-13 15:41:30 -07:00
flag-duplicate-issue.ts ci(duplicate-check): only warn about the auto close the sweep will actually do 2026-09-12 20:27:37 -07:00
gate_slot_lock.py refactor(make): stop queueing bootstrap for a machine-wide gate slot 2026-08-15 21:53:28 +00:00
generate_lens_contract.py feat(lens): isolate ingestion and investigations in a Rust service (#45148) 2026-10-07 17:18:58 -07:00
generate_trace_types.py perf(types): defer pydantic schema builds via shared LiteLLMBaseModel (#44720) 2026-10-05 22:10:08 -07:00
install-cli.sh fix(install): pass an explicit Python version request to uv tool install 2026-07-26 21:02:35 -07:00
install.sh fix(install): pass an explicit Python version request to uv tool install 2026-07-26 21:02:35 -07:00
install_git_hooks.sh build(lint): rename make pre-commit to make check with a working-tree fallback 2026-08-08 03:25:35 -07:00
issue-labels.ts ci(issue-classifier): name every workflow, script and job after the issue it works on 2026-09-18 11:37:55 -07:00
label-issue.test.ts ci(issue-classifier): parse only form headings and keep the claude code label 2026-09-18 11:37:55 -07:00
label-issue.ts ci(issue-classifier): name every workflow, script and job after the issue it works on 2026-09-18 11:37:55 -07:00
lens_dev.sh feat(lens): isolate ingestion and investigations in a Rust service (#45148) 2026-10-07 17:18:58 -07:00
mock_bedrock_passthrough_target.py Refactor Bedrock response stream shape handling (#27257) 2026-05-06 17:39:38 -07:00
mock_grayswan_timeout_server.py implement failopen option default to True on grayswan guardrail (#18266) 2026-01-06 15:17:05 +05:30
mutation_report.py fix: address cross-version CI failures 2026-09-02 14:17:19 -07:00
pre_commit_lint.sh test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
prisma_generate_if_needed.py fix(lint): generate the prisma client into the gate-owned venv 2026-08-06 01:54:26 -07:00
quickstart.ps1 feat(docker): add a Windows quickstart in PowerShell, and a styled terminal for both quickstarts (#44310) 2026-10-03 13:35:32 -07:00
quickstart.sh feat(docker): add a Windows quickstart in PowerShell, and a styled terminal for both quickstarts (#44310) 2026-10-03 13:35:32 -07:00
ruff_strict_gate.py ci: follow the default branch in development tooling 2026-09-07 14:34:45 -07:00
run_tracing_proxy_local.sh feat: improve Lens dev seeding and live UI (#44468) 2026-10-04 01:03:11 +00:00
seed_request_logs.py refactor(lens): storage-independent trace reads, shared keyset pager, typed read failures (#44422) 2026-10-04 12:39:47 -07:00
seed_tracing_fixtures.py feat(lens): isolate ingestion and investigations in a Rust service (#45148) 2026-10-07 17:18:58 -07:00
sync-issue-labels.test.ts ci(issue-classifier): name every workflow, script and job after the issue it works on 2026-09-18 11:37:55 -07:00
sync-issue-labels.ts ci(issue-classifier): name every workflow, script and job after the issue it works on 2026-09-18 11:37:55 -07:00
sync_together_ai_models.py fix(together_ai): stop writing context_length as max_output_tokens in the serverless sync 2026-08-29 15:26:44 -07:00
test_agent_mcp_endpoints.sh Agents - assign tools (#22064) 2026-02-25 11:44:30 -08:00
test_guardrails_register_endpoints.sh feat(guardrails): team-based guardrail registration and approval workflow (#22459) 2026-03-02 22:06:49 -08:00
test_quality_gate.py ci: follow the default branch in development tooling 2026-09-07 14:34:45 -07:00
test_tool_allowlist_script.py test(proxy): move middleware, spend_tracking, pass_through, common_utils and root proxy tests into tests/unit/proxy (#44015) 2026-10-01 18:23:31 +00:00
tpm_headline_test.sh fix: atomic TPM rate limit (#27001) 2026-05-05 16:58:07 -07:00
type_check_gate.py chore(ci): drop litellm_internal_staging and litellm_oss_staging references, main is the only trunk (#42745) 2026-09-23 08:14:11 -07:00
type_discipline_gate.py chore(lint): remove the LIT002 mutable-construction rule (#43971) 2026-10-01 12:24:02 -07:00
verify_adaptive_router.py feat: add adaptive routing to litellm 2026-04-18 16:35:17 -07:00
with_dashboard_node.sh fix(bootstrap): fail fast when nvm cannot activate the pinned node 2026-08-04 21:18:58 -07:00