veritas-kanban/docs/SOP-output-evaluation.md

SOP: Agent Output Evaluation & Scoring

Define scoring profiles and evaluate agent outputs for quality.

---

Overview

The Scoring Framework lets you define profiles with weighted criteria that evaluate agent outputs. Each evaluation runs the output through all scorers, combines scores using a composite method, and returns a per-scorer breakdown for diagnostics.

Scorer types:

Type	What it checks
RegexMatch	Whether bounded worker-isolated regex evaluation matches
KeywordContains	Whether the output contains required keywords
NumericRange	Whether a numeric field in the output falls within a range
OccurrenceRatio	Literal occurrence density with optional numeric-path normalization

RegexMatch accepts patterns up to 256 characters and any valid JavaScript regex flag set supported by the active Node runtime. Evaluation uses a globally bounded four-worker pool outside the server event loop, a bounded wait queue, and a 100 ms limit. Output is limited to 100,000 characters, action text to 10,000 characters, and their combined scoring target to 110,001 characters.

OccurrenceRatio is the declarative replacement for legacy custom expressions. It accepts one to 32 literal needles and divides their occurrence count by either a fixed denominator or a numeric denominatorPath, optionally scaled with denominatorScale. wholeWord, caseSensitive, minimumDenominator, and invert provide bounded transformations without executing code.

Persisted profiles containing the removed CustomExpression scorer fail closed during evaluation. Replace those scorers through the profile API before retrying; the server never evaluates or silently converts the stored expression.

Composite methods:

Method	Behavior
weightedAvg	Weighted average of all scorer scores
minimum	Score is the lowest individual scorer score
geometricMean	Geometric mean — penalizes any single low scorer

---

Prerequisites

• VK server running (v4.0+)
• Know what "good" looks like for the outputs you want to evaluate

---

Step-by-Step Procedure

1. Create a Scoring Profile

curl -X POST http://localhost:3001/api/scoring/profiles \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Task Summary Quality",
    "description": "Evaluate quality of agent task completion summaries.",
    "compositeMethod": "weightedAvg",
    "scorers": [
      {
        "id": "has-action-items",
        "name": "Contains action items",
        "type": "KeywordContains",
        "weight": 0.4,
        "target": "output",
        "keywords": ["completed", "implemented", "fixed", "added"],
        "matchMode": "any",
        "caseSensitive": false
      },
      {
        "id": "length-check",
        "name": "Summary length",
        "type": "NumericRange",
        "weight": 0.3,
        "target": "output",
        "valuePath": "length",
        "min": 50,
        "max": 500
      },
      {
        "id": "no-apologies",
        "name": "No apology language",
        "type": "RegexMatch",
        "weight": 0.3,
        "target": "output",
        "pattern": "I apologize|I\\'m sorry|unfortunately|I cannot",
        "flags": "i",
        "scoreOnMatch": 0,
        "scoreOnMiss": 1,
        "invert": false
      }
    ]
  }'

Response: 201 with profile object including id.

2. List Available Profiles

curl http://localhost:3001/api/scoring/profiles

3. Evaluate an Output

curl -X POST http://localhost:3001/api/scoring/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "profileId": "prof_abc123",
    "output": "Implemented the OAuth login feature using Passport.js. Added Google and GitHub providers. Tests pass.",
    "action": "task-completion",
    "agent": "TARS",
    "taskId": "task_20260321_abc"
  }'

Response:

{
  "id": "eval_xyz789",
  "profileId": "prof_abc123",
  "score": 0.87,
  "passed": true,
  "breakdown": [
    { "scorerId": "has-action-items", "score": 1.0, "weight": 0.4, "weighted": 0.4 },
    { "scorerId": "length-check", "score": 0.85, "weight": 0.3, "weighted": 0.255 },
    { "scorerId": "no-apologies", "score": 1.0, "weight": 0.3, "weighted": 0.3 }
  ],
  "agent": "TARS",
  "taskId": "task_20260321_abc",
  "createdAt": "2026-03-21T14:00:00.000Z"
}

4. Review Evaluation History

# All history
curl http://localhost:3001/api/scoring/history

# Filter by profile and agent
curl "http://localhost:3001/api/scoring/history?profileId=prof_abc123&agent=TARS&limit=20"

5. Update a Profile

curl -X PUT http://localhost:3001/api/scoring/profiles/prof_abc123 \
  -H "Content-Type: application/json" \
  -d '{
    "compositeMethod": "minimum"
  }'

6. Delete a Profile

Built-in profiles cannot be deleted.

curl -X DELETE http://localhost:3001/api/scoring/profiles/prof_abc123

Response: 204 No Content.

---

API Endpoints

Method	Path	Description
GET	/api/scoring/profiles	List all scoring profiles
POST	/api/scoring/profiles	Create a scoring profile
GET	/api/scoring/profiles/:id	Get a specific profile
PUT	/api/scoring/profiles/:id	Update a profile
DELETE	/api/scoring/profiles/:id	Delete a profile (non-built-in only)
POST	/api/scoring/evaluate	Evaluate an output against a profile
GET	/api/scoring/history	List evaluation history

---

Common Issues

Cannot delete built-in profile: Built-in profiles ship with VK and cannot be removed. Create a custom profile instead.

Score unexpectedly low: Check the breakdown field in the evaluation response — it shows which scorer penalized the score.

NumericRange scorer always returns 0: Ensure valuePath correctly addresses a numeric property of the output object. For string length, use length.

KeywordContains with matchMode: "all" is too strict: Switch to "any" if you want partial credit, or use partialCredit: true to award fractional scores.

---

Related Docs

• FEATURES.md — Output Evaluation
• API-REFERENCE.md — Scoring
• SOP: Decision Audit Trail
• SOP: Behavioral Drift Detection