mirror of
https://github.com/BradGroux/veritas-kanban.git
synced 2026-08-28 02:44:59 +00:00
fix: harden input validation
This commit is contained in:
Brad Groux
parent
1cdcd6ec60
commit
c45f6f2917
14 changed files with 719 additions and 163 deletions
|
|
@ -163,7 +163,7 @@ When the board is working, use [Setup Paths](docs/SETUP-PATHS.md) to choose the
|
|||
|
||||
### 🛡️ Agent Governance
|
||||
|
||||
**Policy Engine** — Define what agents can and can't do. Configurable tool/action policies with `allow`, `deny`, and `require-approval` guard rules. Every policy decision is logged. **Sandbox Policy Presets** — Assign reusable filesystem, network, environment, and credential rules to agents, workflow agents, or one-off runs; unsupported required controls fail closed before launch with redacted audit traces. **Decision Audit Trail** — Log agent decisions with confidence scores, supporting evidence, and stated assumptions. Record outcomes afterward to see whether assumptions held. **Behavioral Drift Detection** — Set metric baselines and thresholds; get alerted when an agent's behavior deviates. **User Feedback Loop** — Collect feedback on agent outputs with sentiment tagging and category analytics. **Output Evaluation** — Score agent outputs against weighted criteria profiles (regex, keyword, numeric range, custom expressions).
|
||||
**Policy Engine** — Define what agents can and can't do. Configurable tool/action policies with `allow`, `deny`, and `require-approval` guard rules. Every policy decision is logged. **Sandbox Policy Presets** — Assign reusable filesystem, network, environment, and credential rules to agents, workflow agents, or one-off runs; unsupported required controls fail closed before launch with redacted audit traces. **Decision Audit Trail** — Log agent decisions with confidence scores, supporting evidence, and stated assumptions. Record outcomes afterward to see whether assumptions held. **Behavioral Drift Detection** — Set metric baselines and thresholds; get alerted when an agent's behavior deviates. **User Feedback Loop** — Collect feedback on agent outputs with sentiment tagging and category analytics. **Output Evaluation** — Score agent outputs against weighted bounded criteria profiles (regex, keyword, numeric range, occurrence ratio).
|
||||
|
||||
### 🤖 Agent Orchestration
|
||||
|
||||
|
|
|
|||
|
|
@ -5799,6 +5799,13 @@ POST /api/scoring/profiles
|
|||
|
||||
**Response:** `201` with created profile.
|
||||
|
||||
Scoring profiles accept `KeywordContains`, `NumericRange`, bounded `RegexMatch`, and declarative
|
||||
`OccurrenceRatio` scorers. Regex patterns are limited to 256 characters and execute through a
|
||||
globally bounded four-worker pool and wait queue with a 100 ms limit. Valid JavaScript regex flags
|
||||
supported by the active Node runtime remain accepted. `OccurrenceRatio` counts literal `needles`
|
||||
and normalizes them with a fixed `denominator` or numeric `denominatorPath`; it does not execute
|
||||
expression strings. Persisted legacy custom-expression profiles must be migrated before evaluation.
|
||||
|
||||
#### Get Scoring Profile
|
||||
|
||||
```
|
||||
|
|
|
|||
|
|
@ -2523,7 +2523,9 @@ Define scoring profiles with weighted criteria and evaluate agent outputs agains
|
|||
|
||||
**Key capabilities:**
|
||||
|
||||
- Four scorer types: `RegexMatch`, `KeywordContains`, `NumericRange`, `CustomExpression`
|
||||
- Four bounded scorer types: `RegexMatch`, `KeywordContains`, `NumericRange`, `OccurrenceRatio`
|
||||
- Regex evaluation runs outside the server event loop with input, pattern, and time limits
|
||||
- Occurrence ratios use literal values and optional numeric normalization; arbitrary code is never evaluated
|
||||
- Weighted scorers with optional `target`: `action`, `output`, or `combined`
|
||||
- Composite scoring methods: `weightedAvg`, `minimum`, `geometricMean`
|
||||
- Per-evaluation history with scorer-level breakdowns
|
||||
|
|
|
|||
|
|
@ -10,12 +10,26 @@ The Scoring Framework lets you define profiles with weighted criteria that evalu
|
|||
|
||||
**Scorer types:**
|
||||
|
||||
| Type | What it checks |
|
||||
| ------------------- | ------------------------------------------------------------ |
|
||||
| `RegexMatch` | Whether the output matches a regular expression |
|
||||
| `KeywordContains` | Whether the output contains required keywords |
|
||||
| `NumericRange` | Whether a numeric field in the output falls within a range |
|
||||
| `CustomExpression` | A custom evaluation expression |
|
||||
| Type | What it checks |
|
||||
| ------------------ | --------------------------------------------------------------------- |
|
||||
| `RegexMatch` | Whether bounded worker-isolated regex evaluation matches |
|
||||
| `KeywordContains` | Whether the output contains required keywords |
|
||||
| `NumericRange` | Whether a numeric field in the output falls within a range |
|
||||
| `OccurrenceRatio` | Literal occurrence density with optional numeric-path normalization |
|
||||
|
||||
`RegexMatch` accepts patterns up to 256 characters and any valid JavaScript regex flag set supported
|
||||
by the active Node runtime. Evaluation uses a globally bounded four-worker pool outside the server
|
||||
event loop, a bounded wait queue, and a 100 ms limit. Output is limited to 100,000 characters,
|
||||
action text to 10,000 characters, and their combined scoring target to 110,001 characters.
|
||||
|
||||
`OccurrenceRatio` is the declarative replacement for legacy custom expressions. It accepts one to
|
||||
32 literal `needles` and divides their occurrence count by either a fixed `denominator` or a numeric
|
||||
`denominatorPath`, optionally scaled with `denominatorScale`. `wholeWord`, `caseSensitive`,
|
||||
`minimumDenominator`, and `invert` provide bounded transformations without executing code.
|
||||
|
||||
Persisted profiles containing the removed `CustomExpression` scorer fail closed during evaluation.
|
||||
Replace those scorers through the profile API before retrying; the server never evaluates or
|
||||
silently converts the stored expression.
|
||||
|
||||
**Composite methods:**
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,13 @@
|
|||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import { mkdtemp, rm } from 'fs/promises';
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { mkdir, mkdtemp, rm, writeFile } from 'fs/promises';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { ScoringService } from '../services/scoring-service.js';
|
||||
import {
|
||||
boundedRegexTest,
|
||||
getScoringRegexRuntimeSnapshot,
|
||||
} from '../services/scoring-runtime.js';
|
||||
import { SCORING_REGEX_MAX_CONCURRENCY } from '../config/scoring.js';
|
||||
|
||||
describe('ScoringService', () => {
|
||||
const originalCwd = process.cwd();
|
||||
|
|
@ -27,6 +32,37 @@ describe('ScoringService', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('migrates legacy built-in profiles to bounded scorers', async () => {
|
||||
const profilesDir = join(tempDir, 'storage', 'scoring');
|
||||
await mkdir(profilesDir, { recursive: true });
|
||||
await writeFile(
|
||||
join(profilesDir, 'task-efficiency.json'),
|
||||
JSON.stringify({
|
||||
id: 'task-efficiency',
|
||||
name: 'Task Efficiency',
|
||||
builtIn: true,
|
||||
compositeMethod: 'weightedAvg',
|
||||
created: '2026-01-01T00:00:00.000Z',
|
||||
updated: '2026-01-01T00:00:00.000Z',
|
||||
scorers: [
|
||||
{
|
||||
id: 'legacy',
|
||||
name: 'Legacy',
|
||||
type: 'CustomExpression',
|
||||
weight: 1,
|
||||
expression: 'process.env',
|
||||
},
|
||||
],
|
||||
})
|
||||
);
|
||||
|
||||
const profiles = await service.listProfiles();
|
||||
const migrated = profiles.find((profile) => profile.id === 'task-efficiency');
|
||||
expect(migrated?.created).toBe('2026-01-01T00:00:00.000Z');
|
||||
expect(migrated?.scorers.some((scorer) => scorer.type === 'OccurrenceRatio')).toBe(true);
|
||||
expect(migrated?.scorers.some((scorer) => scorer.type === 'CustomExpression')).toBe(false);
|
||||
});
|
||||
|
||||
it('evaluates profiles and stores history', async () => {
|
||||
const profile = await service.createProfile({
|
||||
name: 'Weighted score',
|
||||
|
|
@ -77,16 +113,18 @@ describe('ScoringService', () => {
|
|||
{
|
||||
id: 'pass',
|
||||
name: 'Pass',
|
||||
type: 'CustomExpression',
|
||||
type: 'OccurrenceRatio',
|
||||
weight: 1,
|
||||
expression: '1',
|
||||
needles: ['anything'],
|
||||
denominator: 1,
|
||||
},
|
||||
{
|
||||
id: 'fail',
|
||||
name: 'Fail',
|
||||
type: 'CustomExpression',
|
||||
type: 'OccurrenceRatio',
|
||||
weight: 1,
|
||||
expression: '0',
|
||||
needles: ['never-present'],
|
||||
denominator: 1,
|
||||
},
|
||||
],
|
||||
});
|
||||
|
|
@ -98,4 +136,120 @@ describe('ScoringService', () => {
|
|||
|
||||
expect(result.compositeScore).toBe(0);
|
||||
});
|
||||
|
||||
it.each([
|
||||
'typeof process === "object"',
|
||||
'({}).constructor.constructor("return process")()',
|
||||
'globalThis.process',
|
||||
])('rejects legacy executable scorer state: %s', async (expression) => {
|
||||
const profilesDir = join(tempDir, 'storage', 'scoring');
|
||||
await service.listProfiles();
|
||||
await writeFile(
|
||||
join(profilesDir, 'legacy-profile.json'),
|
||||
JSON.stringify({
|
||||
id: 'legacy-profile',
|
||||
name: 'Legacy profile',
|
||||
compositeMethod: 'weightedAvg',
|
||||
created: new Date().toISOString(),
|
||||
updated: new Date().toISOString(),
|
||||
scorers: [
|
||||
{
|
||||
id: 'legacy',
|
||||
name: 'Legacy',
|
||||
type: 'CustomExpression',
|
||||
weight: 1,
|
||||
expression,
|
||||
},
|
||||
],
|
||||
})
|
||||
);
|
||||
|
||||
await expect(
|
||||
service.evaluate({ profileId: 'legacy-profile', output: 'ordinary output' })
|
||||
).rejects.toThrow('legacy custom expression');
|
||||
});
|
||||
|
||||
it('terminates regex evaluation that exceeds the bounded runtime', async () => {
|
||||
const profile = await service.createProfile({
|
||||
name: 'Bounded regex',
|
||||
compositeMethod: 'weightedAvg',
|
||||
scorers: [
|
||||
{
|
||||
id: 'regex',
|
||||
name: 'Regex',
|
||||
type: 'RegexMatch',
|
||||
weight: 1,
|
||||
pattern: '^(a+)+$',
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
await expect(
|
||||
service.evaluate({ profileId: profile.id, output: `${'a'.repeat(50_000)}!` })
|
||||
).rejects.toThrow('100ms limit');
|
||||
});
|
||||
|
||||
it('bounds regex workers across concurrent evaluations', async () => {
|
||||
const evaluations = Array.from({ length: SCORING_REGEX_MAX_CONCURRENCY + 1 }, () =>
|
||||
boundedRegexTest('^(a+)+$', undefined, `${'a'.repeat(50_000)}!`).catch((error) => error)
|
||||
);
|
||||
|
||||
await vi.waitFor(() => {
|
||||
const snapshot = getScoringRegexRuntimeSnapshot();
|
||||
expect(snapshot.activeWorkers).toBe(SCORING_REGEX_MAX_CONCURRENCY);
|
||||
expect(snapshot.queuedEvaluations).toBe(1);
|
||||
});
|
||||
|
||||
await Promise.all(evaluations);
|
||||
expect(getScoringRegexRuntimeSnapshot()).toEqual({ activeWorkers: 0, queuedEvaluations: 0 });
|
||||
});
|
||||
|
||||
it('preserves ordinary regex and bounded occurrence scoring', async () => {
|
||||
const profile = await service.createProfile({
|
||||
name: 'Safe scoring',
|
||||
compositeMethod: 'weightedAvg',
|
||||
scorers: [
|
||||
{
|
||||
id: 'regex',
|
||||
name: 'Regex',
|
||||
type: 'RegexMatch',
|
||||
weight: 1,
|
||||
pattern: '\\bverified\\b',
|
||||
flags: 'gi',
|
||||
},
|
||||
{
|
||||
id: 'ratio',
|
||||
name: 'Ratio',
|
||||
type: 'OccurrenceRatio',
|
||||
weight: 1,
|
||||
needles: ['verified'],
|
||||
wholeWord: true,
|
||||
denominator: 1,
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const result = await service.evaluate({ profileId: profile.id, output: 'Verified result' });
|
||||
expect(result.scores.map((score) => score.score)).toEqual([1, 1]);
|
||||
});
|
||||
|
||||
it('does not traverse reserved metadata path segments', async () => {
|
||||
const profile = await service.createProfile({
|
||||
name: 'Contained metadata path',
|
||||
compositeMethod: 'weightedAvg',
|
||||
scorers: [
|
||||
{
|
||||
id: 'ratio',
|
||||
name: 'Ratio',
|
||||
type: 'OccurrenceRatio',
|
||||
weight: 1,
|
||||
needles: ['verified'],
|
||||
denominatorPath: 'metadata.constructor.prototype.polluted',
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const result = await service.evaluate({ profileId: profile.id, output: 'verified' });
|
||||
expect(result.scores[0]?.score).toBe(1);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -278,6 +278,40 @@ describe('SQLite governance repositories', () => {
|
|||
result.id,
|
||||
]);
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const legacyProfile = {
|
||||
id: 'legacy-sqlite-profile',
|
||||
name: 'Legacy SQLite profile',
|
||||
builtIn: false,
|
||||
compositeMethod: 'weightedAvg',
|
||||
created: now,
|
||||
updated: now,
|
||||
scorers: [
|
||||
{
|
||||
id: 'legacy',
|
||||
name: 'Legacy',
|
||||
type: 'CustomExpression',
|
||||
weight: 1,
|
||||
expression: 'globalThis.process',
|
||||
},
|
||||
],
|
||||
};
|
||||
fixture.database
|
||||
.getConnection()
|
||||
.prepare(
|
||||
`
|
||||
INSERT INTO scoring_profiles (
|
||||
id, workspace_id, name, built_in, profile_json, created_at, updated_at
|
||||
)
|
||||
VALUES (?, 'local', ?, 0, ?, ?, ?)
|
||||
`
|
||||
)
|
||||
.run(legacyProfile.id, legacyProfile.name, JSON.stringify(legacyProfile), now, now);
|
||||
|
||||
await expect(
|
||||
service.evaluate({ profileId: legacyProfile.id, output: 'ordinary output' })
|
||||
).rejects.toThrow('legacy custom expression');
|
||||
|
||||
await expect(fs.access(profilesDir)).rejects.toThrow();
|
||||
await expect(fs.access(evaluationsDir)).rejects.toThrow();
|
||||
});
|
||||
|
|
|
|||
8
server/src/config/scoring.ts
Normal file
8
server/src/config/scoring.ts
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
export const SCORING_MAX_ACTION_LENGTH = 10_000;
|
||||
export const SCORING_MAX_OUTPUT_LENGTH = 100_000;
|
||||
export const SCORING_MAX_COMBINED_LENGTH =
|
||||
SCORING_MAX_ACTION_LENGTH + SCORING_MAX_OUTPUT_LENGTH + 1;
|
||||
export const SCORING_MAX_PATTERN_LENGTH = 256;
|
||||
export const SCORING_REGEX_TIMEOUT_MS = 100;
|
||||
export const SCORING_REGEX_MAX_CONCURRENCY = 4;
|
||||
export const SCORING_REGEX_MAX_QUEUE = 64;
|
||||
|
|
@ -4,76 +4,14 @@ import { asyncHandler } from '../middleware/async-handler.js';
|
|||
import { BadRequestError, NotFoundError, ValidationError } from '../middleware/error-handler.js';
|
||||
import { scoringService } from '../services/scoring-service.js';
|
||||
import { paramStr, qNum, qStr } from '../lib/query-helpers.js';
|
||||
import {
|
||||
createScoringProfileSchema,
|
||||
evaluateScoringSchema,
|
||||
updateScoringProfileSchema,
|
||||
} from '../schemas/scoring-schemas.js';
|
||||
|
||||
const router: RouterType = Router();
|
||||
|
||||
const scorerSchema = z.discriminatedUnion('type', [
|
||||
z.object({
|
||||
id: z.string().min(1),
|
||||
name: z.string().min(1),
|
||||
description: z.string().optional(),
|
||||
weight: z.number().min(0),
|
||||
target: z.enum(['action', 'output', 'combined']).optional(),
|
||||
type: z.literal('RegexMatch'),
|
||||
pattern: z.string().min(1),
|
||||
flags: z.string().optional(),
|
||||
scoreOnMatch: z.number().min(0).max(1).optional(),
|
||||
scoreOnMiss: z.number().min(0).max(1).optional(),
|
||||
invert: z.boolean().optional(),
|
||||
}),
|
||||
z.object({
|
||||
id: z.string().min(1),
|
||||
name: z.string().min(1),
|
||||
description: z.string().optional(),
|
||||
weight: z.number().min(0),
|
||||
target: z.enum(['action', 'output', 'combined']).optional(),
|
||||
type: z.literal('KeywordContains'),
|
||||
keywords: z.array(z.string().min(1)).min(1),
|
||||
matchMode: z.enum(['all', 'any']).optional(),
|
||||
caseSensitive: z.boolean().optional(),
|
||||
partialCredit: z.boolean().optional(),
|
||||
}),
|
||||
z.object({
|
||||
id: z.string().min(1),
|
||||
name: z.string().min(1),
|
||||
description: z.string().optional(),
|
||||
weight: z.number().min(0),
|
||||
target: z.enum(['action', 'output', 'combined']).optional(),
|
||||
type: z.literal('NumericRange'),
|
||||
valuePath: z.string().min(1),
|
||||
min: z.number().optional(),
|
||||
max: z.number().optional(),
|
||||
scoreOnMiss: z.number().min(0).max(1).optional(),
|
||||
}),
|
||||
z.object({
|
||||
id: z.string().min(1),
|
||||
name: z.string().min(1),
|
||||
description: z.string().optional(),
|
||||
weight: z.number().min(0),
|
||||
target: z.enum(['action', 'output', 'combined']).optional(),
|
||||
type: z.literal('CustomExpression'),
|
||||
expression: z.string().min(1),
|
||||
}),
|
||||
]);
|
||||
|
||||
const createProfileSchema = z.object({
|
||||
name: z.string().min(1),
|
||||
description: z.string().optional(),
|
||||
scorers: z.array(scorerSchema).min(1),
|
||||
compositeMethod: z.enum(['weightedAvg', 'minimum', 'geometricMean']),
|
||||
});
|
||||
|
||||
const updateProfileSchema = createProfileSchema.partial();
|
||||
|
||||
const evaluateSchema = z.object({
|
||||
profileId: z.string().min(1),
|
||||
action: z.string().optional(),
|
||||
output: z.string().min(1),
|
||||
agent: z.string().optional(),
|
||||
taskId: z.string().optional(),
|
||||
metadata: z.record(z.string(), z.unknown()).optional(),
|
||||
});
|
||||
|
||||
const parseOrThrow = <T>(schema: z.ZodType<T>, value: unknown): T => {
|
||||
try {
|
||||
return schema.parse(value);
|
||||
|
|
@ -107,7 +45,7 @@ router.get(
|
|||
router.post(
|
||||
'/profiles',
|
||||
asyncHandler(async (req, res) => {
|
||||
const input = parseOrThrow(createProfileSchema, req.body);
|
||||
const input = parseOrThrow(createScoringProfileSchema, req.body);
|
||||
const profile = await scoringService.createProfile(input);
|
||||
res.status(201).json(profile);
|
||||
})
|
||||
|
|
@ -116,7 +54,7 @@ router.post(
|
|||
router.put(
|
||||
'/profiles/:id',
|
||||
asyncHandler(async (req, res) => {
|
||||
const input = parseOrThrow(updateProfileSchema, req.body);
|
||||
const input = parseOrThrow(updateScoringProfileSchema, req.body);
|
||||
let profile;
|
||||
try {
|
||||
profile = await scoringService.updateProfile(paramStr(req.params.id), input);
|
||||
|
|
@ -154,7 +92,7 @@ router.delete(
|
|||
router.post(
|
||||
'/evaluate',
|
||||
asyncHandler(async (req, res) => {
|
||||
const input = parseOrThrow(evaluateSchema, req.body);
|
||||
const input = parseOrThrow(evaluateScoringSchema, req.body);
|
||||
const result = await scoringService.evaluate(input);
|
||||
res.status(201).json(result);
|
||||
})
|
||||
|
|
|
|||
105
server/src/schemas/scoring-schemas.ts
Normal file
105
server/src/schemas/scoring-schemas.ts
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
import { z } from 'zod';
|
||||
import type {
|
||||
CreateScoringProfileInput,
|
||||
EvaluationRequest,
|
||||
Scorer,
|
||||
UpdateScoringProfileInput,
|
||||
} from '@veritas-kanban/shared';
|
||||
import {
|
||||
SCORING_MAX_ACTION_LENGTH,
|
||||
SCORING_MAX_OUTPUT_LENGTH,
|
||||
SCORING_MAX_PATTERN_LENGTH,
|
||||
} from '../config/scoring.js';
|
||||
|
||||
const regexFlagsSchema = z
|
||||
.string()
|
||||
.max(8)
|
||||
.refine((flags) => {
|
||||
try {
|
||||
void new RegExp('', flags);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}, 'Regex flags must be a valid, non-duplicated JavaScript flag set');
|
||||
|
||||
const metadataFitsLimit = (value: Record<string, unknown>): boolean => {
|
||||
try {
|
||||
return new TextEncoder().encode(JSON.stringify(value)).byteLength <= 64_000;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
const baseScorerShape = {
|
||||
id: z.string().min(1).max(128),
|
||||
name: z.string().min(1).max(256),
|
||||
description: z.string().max(2_000).optional(),
|
||||
weight: z.number().min(0).max(1_000_000),
|
||||
target: z.enum(['action', 'output', 'combined']).optional(),
|
||||
};
|
||||
|
||||
export const scorerSchema: z.ZodType<Scorer> = z.discriminatedUnion('type', [
|
||||
z.object({
|
||||
...baseScorerShape,
|
||||
type: z.literal('RegexMatch'),
|
||||
pattern: z.string().min(1).max(SCORING_MAX_PATTERN_LENGTH),
|
||||
flags: regexFlagsSchema.optional(),
|
||||
scoreOnMatch: z.number().min(0).max(1).optional(),
|
||||
scoreOnMiss: z.number().min(0).max(1).optional(),
|
||||
invert: z.boolean().optional(),
|
||||
}),
|
||||
z.object({
|
||||
...baseScorerShape,
|
||||
type: z.literal('KeywordContains'),
|
||||
keywords: z.array(z.string().min(1).max(256)).min(1).max(64),
|
||||
matchMode: z.enum(['all', 'any']).optional(),
|
||||
caseSensitive: z.boolean().optional(),
|
||||
partialCredit: z.boolean().optional(),
|
||||
}),
|
||||
z.object({
|
||||
...baseScorerShape,
|
||||
type: z.literal('NumericRange'),
|
||||
valuePath: z.string().min(1).max(256),
|
||||
min: z.number().finite().optional(),
|
||||
max: z.number().finite().optional(),
|
||||
scoreOnMiss: z.number().min(0).max(1).optional(),
|
||||
}),
|
||||
z.object({
|
||||
...baseScorerShape,
|
||||
type: z.literal('OccurrenceRatio'),
|
||||
needles: z.array(z.string().min(1).max(64)).min(1).max(32),
|
||||
caseSensitive: z.boolean().optional(),
|
||||
wholeWord: z.boolean().optional(),
|
||||
denominator: z.number().positive().max(1_000_000).optional(),
|
||||
denominatorPath: z.string().min(1).max(256).optional(),
|
||||
denominatorScale: z.number().positive().max(1_000_000).optional(),
|
||||
minimumDenominator: z.number().positive().max(1_000_000).optional(),
|
||||
invert: z.boolean().optional(),
|
||||
}),
|
||||
]);
|
||||
|
||||
const createScoringProfileObjectSchema = z.object({
|
||||
name: z.string().min(1).max(256),
|
||||
description: z.string().max(4_000).optional(),
|
||||
scorers: z.array(scorerSchema).min(1).max(64),
|
||||
compositeMethod: z.enum(['weightedAvg', 'minimum', 'geometricMean']),
|
||||
});
|
||||
|
||||
export const createScoringProfileSchema: z.ZodType<CreateScoringProfileInput> =
|
||||
createScoringProfileObjectSchema;
|
||||
|
||||
export const updateScoringProfileSchema: z.ZodType<UpdateScoringProfileInput> =
|
||||
createScoringProfileObjectSchema.partial();
|
||||
|
||||
export const evaluateScoringSchema: z.ZodType<EvaluationRequest> = z.object({
|
||||
profileId: z.string().min(1).max(128),
|
||||
action: z.string().max(SCORING_MAX_ACTION_LENGTH).optional(),
|
||||
output: z.string().min(1).max(SCORING_MAX_OUTPUT_LENGTH),
|
||||
agent: z.string().optional(),
|
||||
taskId: z.string().max(128).optional(),
|
||||
metadata: z
|
||||
.record(z.string(), z.unknown())
|
||||
.refine(metadataFitsLimit, 'Metadata must be JSON serializable and cannot exceed 64,000 bytes')
|
||||
.optional(),
|
||||
});
|
||||
198
server/src/services/scoring-runtime.ts
Normal file
198
server/src/services/scoring-runtime.ts
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
import { Worker } from 'node:worker_threads';
|
||||
import type { OccurrenceRatioScorer, ScoringTarget } from '@veritas-kanban/shared';
|
||||
import { ValidationError } from '../middleware/error-handler.js';
|
||||
import {
|
||||
SCORING_MAX_COMBINED_LENGTH,
|
||||
SCORING_MAX_PATTERN_LENGTH,
|
||||
SCORING_REGEX_MAX_CONCURRENCY,
|
||||
SCORING_REGEX_MAX_QUEUE,
|
||||
SCORING_REGEX_TIMEOUT_MS,
|
||||
} from '../config/scoring.js';
|
||||
|
||||
let activeRegexWorkers = 0;
|
||||
const regexWorkerWaiters: Array<() => void> = [];
|
||||
|
||||
async function acquireRegexWorkerSlot(): Promise<void> {
|
||||
if (activeRegexWorkers < SCORING_REGEX_MAX_CONCURRENCY) {
|
||||
activeRegexWorkers += 1;
|
||||
return;
|
||||
}
|
||||
if (regexWorkerWaiters.length >= SCORING_REGEX_MAX_QUEUE) {
|
||||
throw new ValidationError('Regex evaluation capacity is temporarily exhausted');
|
||||
}
|
||||
await new Promise<void>((resolve) => regexWorkerWaiters.push(resolve));
|
||||
}
|
||||
|
||||
function releaseRegexWorkerSlot(): void {
|
||||
const next = regexWorkerWaiters.shift();
|
||||
if (next) {
|
||||
next();
|
||||
return;
|
||||
}
|
||||
activeRegexWorkers = Math.max(0, activeRegexWorkers - 1);
|
||||
}
|
||||
|
||||
export function getScoringRegexRuntimeSnapshot(): {
|
||||
activeWorkers: number;
|
||||
queuedEvaluations: number;
|
||||
} {
|
||||
return {
|
||||
activeWorkers: activeRegexWorkers,
|
||||
queuedEvaluations: regexWorkerWaiters.length,
|
||||
};
|
||||
}
|
||||
|
||||
const REGEX_WORKER_SOURCE = String.raw`
|
||||
const { parentPort, workerData } = require('node:worker_threads');
|
||||
try {
|
||||
const regex = new RegExp(workerData.pattern, workerData.flags);
|
||||
parentPort.postMessage({ matched: regex.test(workerData.text) });
|
||||
} catch (error) {
|
||||
parentPort.postMessage({ error: error instanceof Error ? error.message : 'Invalid pattern' });
|
||||
}
|
||||
`;
|
||||
|
||||
export interface ScoringRuntimeContext {
|
||||
action: string;
|
||||
output: string;
|
||||
combined: string;
|
||||
metadata: Record<string, unknown>;
|
||||
}
|
||||
|
||||
function targetText(target: ScoringTarget | undefined, context: ScoringRuntimeContext): string {
|
||||
switch (target) {
|
||||
case 'action':
|
||||
return context.action;
|
||||
case 'combined':
|
||||
return context.combined;
|
||||
case 'output':
|
||||
default:
|
||||
return context.output;
|
||||
}
|
||||
}
|
||||
|
||||
function valueAtPath(root: Record<string, unknown>, path: string): unknown {
|
||||
return path.split('.').reduce<unknown>((current, segment) => {
|
||||
if (
|
||||
!current ||
|
||||
typeof current !== 'object' ||
|
||||
segment === '__proto__' ||
|
||||
segment === 'prototype' ||
|
||||
segment === 'constructor' ||
|
||||
!Object.prototype.hasOwnProperty.call(current, segment)
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
return (current as Record<string, unknown>)[segment];
|
||||
}, root);
|
||||
}
|
||||
|
||||
function isWordCharacter(value: string | undefined): boolean {
|
||||
return value !== undefined && /[A-Za-z0-9_]/.test(value);
|
||||
}
|
||||
|
||||
function countOccurrences(text: string, needle: string, wholeWord: boolean): number {
|
||||
let count = 0;
|
||||
let offset = 0;
|
||||
while (offset <= text.length - needle.length) {
|
||||
const index = text.indexOf(needle, offset);
|
||||
if (index < 0) break;
|
||||
const before = index > 0 ? text[index - 1] : undefined;
|
||||
const after = text[index + needle.length];
|
||||
if (!wholeWord || (!isWordCharacter(before) && !isWordCharacter(after))) count += 1;
|
||||
offset = index + Math.max(needle.length, 1);
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
export function evaluateOccurrenceRatio(
|
||||
scorer: OccurrenceRatioScorer,
|
||||
context: ScoringRuntimeContext
|
||||
): number {
|
||||
const rawText = targetText(scorer.target, context);
|
||||
const text = scorer.caseSensitive ? rawText : rawText.toLowerCase();
|
||||
const needles = scorer.caseSensitive
|
||||
? scorer.needles
|
||||
: scorer.needles.map((needle) => needle.toLowerCase());
|
||||
const occurrences = needles.reduce(
|
||||
(sum, needle) => sum + countOccurrences(text, needle, scorer.wholeWord === true),
|
||||
0
|
||||
);
|
||||
const pathValue = scorer.denominatorPath
|
||||
? valueAtPath(
|
||||
{ action: context.action, output: context.output, metadata: context.metadata },
|
||||
scorer.denominatorPath
|
||||
)
|
||||
: undefined;
|
||||
const numericPathValue = typeof pathValue === 'number' ? pathValue : Number(pathValue);
|
||||
const scaledPathValue = Number.isFinite(numericPathValue)
|
||||
? numericPathValue / (scorer.denominatorScale ?? 1)
|
||||
: undefined;
|
||||
const denominator = Math.max(
|
||||
scorer.minimumDenominator ?? 1,
|
||||
scaledPathValue ?? scorer.denominator ?? 1
|
||||
);
|
||||
const ratio = occurrences / denominator;
|
||||
return scorer.invert ? 1 - ratio : ratio;
|
||||
}
|
||||
|
||||
export async function boundedRegexTest(
|
||||
pattern: string,
|
||||
flags: string | undefined,
|
||||
text: string
|
||||
): Promise<boolean> {
|
||||
if (pattern.length > SCORING_MAX_PATTERN_LENGTH) {
|
||||
throw new ValidationError(`Regex patterns cannot exceed ${SCORING_MAX_PATTERN_LENGTH} characters`);
|
||||
}
|
||||
if (text.length > SCORING_MAX_COMBINED_LENGTH) {
|
||||
throw new ValidationError(
|
||||
`Combined scoring text cannot exceed ${SCORING_MAX_COMBINED_LENGTH} characters`
|
||||
);
|
||||
}
|
||||
|
||||
await acquireRegexWorkerSlot();
|
||||
try {
|
||||
return await new Promise<boolean>((resolve, reject) => {
|
||||
const worker = new Worker(REGEX_WORKER_SOURCE, {
|
||||
eval: true,
|
||||
workerData: { pattern, flags: flags ?? '', text },
|
||||
});
|
||||
let settled = false;
|
||||
let timer: ReturnType<typeof setTimeout>;
|
||||
const finish = (callback: () => void, terminate = true) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
if (!terminate) {
|
||||
callback();
|
||||
return;
|
||||
}
|
||||
void worker.terminate().then(callback, callback);
|
||||
};
|
||||
timer = setTimeout(() => {
|
||||
finish(() => reject(new ValidationError('Regex evaluation exceeded the 100ms limit')));
|
||||
}, SCORING_REGEX_TIMEOUT_MS);
|
||||
worker.once('message', (message: { matched?: boolean; error?: string }) => {
|
||||
if (message.error) {
|
||||
finish(() => reject(new ValidationError(`Invalid regex pattern: ${message.error}`)));
|
||||
return;
|
||||
}
|
||||
const matched = message.matched;
|
||||
if (typeof matched !== 'boolean') {
|
||||
finish(() => reject(new ValidationError('Regex worker returned an invalid response')));
|
||||
return;
|
||||
}
|
||||
finish(() => resolve(matched));
|
||||
});
|
||||
worker.once('error', (error) => finish(() => reject(error)));
|
||||
worker.once('exit', () => {
|
||||
finish(
|
||||
() => reject(new ValidationError('Regex evaluation stopped unexpectedly')),
|
||||
false
|
||||
);
|
||||
});
|
||||
});
|
||||
} finally {
|
||||
releaseRegexWorkerSlot();
|
||||
}
|
||||
}
|
||||
|
|
@ -2,13 +2,13 @@ import { join } from 'path';
|
|||
import { nanoid } from 'nanoid';
|
||||
import type {
|
||||
CreateScoringProfileInput,
|
||||
CustomExpressionScorer,
|
||||
EvaluationDimensionScore,
|
||||
EvaluationHistoryQuery,
|
||||
EvaluationRequest,
|
||||
EvaluationResult,
|
||||
KeywordContainsScorer,
|
||||
NumericRangeScorer,
|
||||
OccurrenceRatioScorer,
|
||||
RegexMatchScorer,
|
||||
Scorer,
|
||||
ScoringProfile,
|
||||
|
|
@ -17,9 +17,15 @@ import { fileExists, mkdir, readdir, readFile, unlink, writeFile } from '../stor
|
|||
import { withFileLock } from './file-lock.js';
|
||||
import { createLogger } from '../lib/logger.js';
|
||||
import { ensureWithinBase, validatePathSegment } from '../utils/sanitize.js';
|
||||
import { NotFoundError } from '../middleware/error-handler.js';
|
||||
import { NotFoundError, ValidationError } from '../middleware/error-handler.js';
|
||||
import { SqliteDatabase, type SqliteConnectionOptions } from '../storage/sqlite/database.js';
|
||||
import { SqliteScoringRepository } from '../storage/sqlite/governance-repositories.js';
|
||||
import {
|
||||
createScoringProfileSchema,
|
||||
evaluateScoringSchema,
|
||||
updateScoringProfileSchema,
|
||||
} from '../schemas/scoring-schemas.js';
|
||||
import { boundedRegexTest, evaluateOccurrenceRatio } from './scoring-runtime.js';
|
||||
|
||||
const log = createLogger('scoring-service');
|
||||
|
||||
|
|
@ -55,13 +61,34 @@ const getTargetText = (scorer: Scorer, context: EvaluationContext): string => {
|
|||
|
||||
const getValueAtPath = (root: Record<string, unknown>, path: string): unknown => {
|
||||
return path.split('.').reduce<unknown>((current, segment) => {
|
||||
if (current && typeof current === 'object' && segment in current) {
|
||||
if (
|
||||
current &&
|
||||
typeof current === 'object' &&
|
||||
segment !== '__proto__' &&
|
||||
segment !== 'prototype' &&
|
||||
segment !== 'constructor' &&
|
||||
Object.prototype.hasOwnProperty.call(current, segment)
|
||||
) {
|
||||
return (current as Record<string, unknown>)[segment];
|
||||
}
|
||||
return undefined;
|
||||
}, root);
|
||||
};
|
||||
|
||||
const hasLegacyCustomExpression = (profile: unknown): boolean => {
|
||||
if (!profile || typeof profile !== 'object') return false;
|
||||
const scorers = (profile as { scorers?: unknown }).scorers;
|
||||
return (
|
||||
Array.isArray(scorers) &&
|
||||
scorers.some(
|
||||
(scorer) =>
|
||||
scorer &&
|
||||
typeof scorer === 'object' &&
|
||||
(scorer as { type?: unknown }).type === 'CustomExpression'
|
||||
)
|
||||
);
|
||||
};
|
||||
|
||||
const BUILT_IN_PROFILES: Array<Omit<ScoringProfile, 'created' | 'updated'>> = [
|
||||
{
|
||||
id: 'code-quality',
|
||||
|
|
@ -129,10 +156,15 @@ const BUILT_IN_PROFILES: Array<Omit<ScoringProfile, 'created' | 'updated'>> = [
|
|||
{
|
||||
id: 'limited-filler',
|
||||
name: 'Limited filler',
|
||||
type: 'CustomExpression',
|
||||
type: 'OccurrenceRatio',
|
||||
weight: 0.25,
|
||||
expression:
|
||||
'Math.max(0, 1 - (((output.match(/\\b(?:just|really|very|basically|simply)\\b/gi) || []).length) / Math.max(1, metadata.outputWordCount / 50)))',
|
||||
target: 'output',
|
||||
needles: ['just', 'really', 'very', 'basically', 'simply'],
|
||||
wholeWord: true,
|
||||
denominatorPath: 'metadata.outputWordCount',
|
||||
denominatorScale: 50,
|
||||
minimumDenominator: 1,
|
||||
invert: true,
|
||||
},
|
||||
],
|
||||
},
|
||||
|
|
@ -164,11 +196,11 @@ const BUILT_IN_PROFILES: Array<Omit<ScoringProfile, 'created' | 'updated'>> = [
|
|||
{
|
||||
id: 'has-structure',
|
||||
name: 'Has concise structure',
|
||||
type: 'CustomExpression',
|
||||
type: 'OccurrenceRatio',
|
||||
target: 'output',
|
||||
weight: 0.33,
|
||||
expression:
|
||||
'Math.min(1, ((output.match(/\\n/g) || []).length + (output.match(/\\./g) || []).length) / 4)',
|
||||
needles: ['\n', '.'],
|
||||
denominator: 4,
|
||||
},
|
||||
],
|
||||
},
|
||||
|
|
@ -213,19 +245,49 @@ export class ScoringService {
|
|||
await this.seedBuiltIns();
|
||||
}
|
||||
|
||||
private parseProfileInput(input: unknown): CreateScoringProfileInput {
|
||||
const parsed = createScoringProfileSchema.safeParse(input);
|
||||
if (parsed.success) return parsed.data;
|
||||
if (hasLegacyCustomExpression(input)) {
|
||||
throw new ValidationError(
|
||||
'Scoring profile contains a legacy custom expression. Replace it with a bounded OccurrenceRatio, KeywordContains, RegexMatch, or NumericRange scorer before evaluation.'
|
||||
);
|
||||
}
|
||||
throw new ValidationError('Invalid scoring profile', parsed.error.issues);
|
||||
}
|
||||
|
||||
private parseProfileUpdate(input: unknown): Partial<CreateScoringProfileInput> {
|
||||
const parsed = updateScoringProfileSchema.safeParse(input);
|
||||
if (parsed.success) return parsed.data;
|
||||
if (hasLegacyCustomExpression(input)) {
|
||||
throw new ValidationError(
|
||||
'Scoring profile contains a legacy custom expression. Replace it with a bounded OccurrenceRatio, KeywordContains, RegexMatch, or NumericRange scorer before evaluation.'
|
||||
);
|
||||
}
|
||||
throw new ValidationError('Invalid scoring profile update', parsed.error.issues);
|
||||
}
|
||||
|
||||
private parseEvaluationInput(input: unknown): EvaluationRequest {
|
||||
const parsed = evaluateScoringSchema.safeParse(input);
|
||||
if (parsed.success) return parsed.data;
|
||||
throw new ValidationError('Invalid scoring evaluation request', parsed.error.issues);
|
||||
}
|
||||
|
||||
private async seedBuiltIns(): Promise<void> {
|
||||
if (this.builtInsSeeded) return;
|
||||
|
||||
for (const profile of BUILT_IN_PROFILES) {
|
||||
const existing = this.repository
|
||||
? await this.repository.getProfile(profile.id)
|
||||
: await fileExists(this.profilePath(profile.id));
|
||||
if (existing) continue;
|
||||
: (await fileExists(this.profilePath(profile.id)))
|
||||
? await this.readProfileFile(this.profilePath(profile.id))
|
||||
: null;
|
||||
if (existing && !(existing.builtIn && hasLegacyCustomExpression(existing))) continue;
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const seededProfile: ScoringProfile = {
|
||||
...profile,
|
||||
created: now,
|
||||
created: existing?.created ?? now,
|
||||
updated: now,
|
||||
};
|
||||
|
||||
|
|
@ -311,15 +373,16 @@ export class ScoringService {
|
|||
|
||||
async createProfile(input: CreateScoringProfileInput): Promise<ScoringProfile> {
|
||||
await this.ensureDirs();
|
||||
const parsed = this.parseProfileInput(input);
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const id = `${slugify(input.name)}-${nanoid(6)}`;
|
||||
const id = `${slugify(parsed.name)}-${nanoid(6)}`;
|
||||
const profile: ScoringProfile = {
|
||||
id,
|
||||
name: input.name,
|
||||
description: input.description,
|
||||
scorers: input.scorers,
|
||||
compositeMethod: input.compositeMethod,
|
||||
name: parsed.name,
|
||||
description: parsed.description,
|
||||
scorers: parsed.scorers,
|
||||
compositeMethod: parsed.compositeMethod,
|
||||
builtIn: false,
|
||||
created: now,
|
||||
updated: now,
|
||||
|
|
@ -348,10 +411,11 @@ export class ScoringService {
|
|||
if (existing.builtIn) {
|
||||
throw new Error('Built-in scoring profiles cannot be modified');
|
||||
}
|
||||
const parsed = this.parseProfileUpdate(input);
|
||||
|
||||
const updated: ScoringProfile = {
|
||||
...existing,
|
||||
...input,
|
||||
...parsed,
|
||||
id: existing.id,
|
||||
builtIn: existing.builtIn,
|
||||
updated: new Date().toISOString(),
|
||||
|
|
@ -386,13 +450,12 @@ export class ScoringService {
|
|||
return true;
|
||||
}
|
||||
|
||||
private evaluateRegex(
|
||||
private async evaluateRegex(
|
||||
scorer: RegexMatchScorer,
|
||||
context: EvaluationContext
|
||||
): EvaluationDimensionScore {
|
||||
): Promise<EvaluationDimensionScore> {
|
||||
const text = getTargetText(scorer, context);
|
||||
const regex = new RegExp(scorer.pattern, scorer.flags);
|
||||
const didMatch = regex.test(text);
|
||||
const didMatch = await boundedRegexTest(scorer.pattern, scorer.flags, text);
|
||||
const matched = scorer.invert ? !didMatch : didMatch;
|
||||
const score = matched ? (scorer.scoreOnMatch ?? 1) : (scorer.scoreOnMiss ?? 0);
|
||||
|
||||
|
|
@ -472,51 +535,27 @@ export class ScoringService {
|
|||
};
|
||||
}
|
||||
|
||||
private evaluateCustom(
|
||||
scorer: CustomExpressionScorer,
|
||||
private evaluateOccurrence(
|
||||
scorer: OccurrenceRatioScorer,
|
||||
context: EvaluationContext
|
||||
): EvaluationDimensionScore {
|
||||
const evaluator = new Function(
|
||||
'action',
|
||||
'output',
|
||||
'combined',
|
||||
'metadata',
|
||||
`return (${scorer.expression});`
|
||||
) as (
|
||||
action: string,
|
||||
output: string,
|
||||
combined: string,
|
||||
metadata: Record<string, unknown>
|
||||
) => unknown;
|
||||
|
||||
let rawResult: unknown = 0;
|
||||
try {
|
||||
rawResult = evaluator(context.action, context.output, context.combined, context.metadata);
|
||||
} catch (error) {
|
||||
log.warn({ err: error, scorerId: scorer.id }, 'Custom scorer expression failed');
|
||||
}
|
||||
|
||||
const score =
|
||||
typeof rawResult === 'boolean'
|
||||
? rawResult
|
||||
? 1
|
||||
: 0
|
||||
: typeof rawResult === 'number'
|
||||
? rawResult
|
||||
: 0;
|
||||
const score = clampScore(evaluateOccurrenceRatio(scorer, context));
|
||||
|
||||
return {
|
||||
scorerId: scorer.id,
|
||||
scorerName: scorer.name,
|
||||
scorerType: scorer.type,
|
||||
weight: scorer.weight,
|
||||
score: clampScore(score),
|
||||
matched: clampScore(score) > 0,
|
||||
explanation: `Custom expression returned ${String(rawResult)}`,
|
||||
score,
|
||||
matched: score > 0,
|
||||
explanation: `Observed occurrence ratio ${score.toFixed(3)}`,
|
||||
};
|
||||
}
|
||||
|
||||
private evaluateScorer(scorer: Scorer, context: EvaluationContext): EvaluationDimensionScore {
|
||||
private async evaluateScorer(
|
||||
scorer: Scorer,
|
||||
context: EvaluationContext
|
||||
): Promise<EvaluationDimensionScore> {
|
||||
switch (scorer.type) {
|
||||
case 'RegexMatch':
|
||||
return this.evaluateRegex(scorer, context);
|
||||
|
|
@ -524,8 +563,12 @@ export class ScoringService {
|
|||
return this.evaluateKeywords(scorer, context);
|
||||
case 'NumericRange':
|
||||
return this.evaluateNumeric(scorer, context);
|
||||
case 'CustomExpression':
|
||||
return this.evaluateCustom(scorer, context);
|
||||
case 'OccurrenceRatio':
|
||||
return this.evaluateOccurrence(scorer, context);
|
||||
default:
|
||||
throw new ValidationError(
|
||||
'Scoring profile uses an unsupported legacy scorer. Replace it with KeywordContains, RegexMatch, NumericRange, or OccurrenceRatio.'
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -575,23 +618,35 @@ export class ScoringService {
|
|||
|
||||
async evaluate(input: EvaluationRequest): Promise<EvaluationResult> {
|
||||
await this.ensureDirs();
|
||||
const parsedInput = this.parseEvaluationInput(input);
|
||||
|
||||
const profile = await this.getProfile(input.profileId);
|
||||
const profile = await this.getProfile(parsedInput.profileId);
|
||||
if (!profile) {
|
||||
throw new NotFoundError('Scoring profile not found');
|
||||
}
|
||||
|
||||
const context = this.buildContext(input);
|
||||
const scores = profile.scorers.map((scorer) => this.evaluateScorer(scorer, context));
|
||||
const validatedProfile = this.parseProfileInput({
|
||||
name: profile.name,
|
||||
description: profile.description,
|
||||
scorers: profile.scorers,
|
||||
compositeMethod: profile.compositeMethod,
|
||||
});
|
||||
const context = this.buildContext(parsedInput);
|
||||
const scores: EvaluationDimensionScore[] = [];
|
||||
for (const scorer of validatedProfile.scorers) {
|
||||
// Regex scorers execute in worker threads. Keep evaluation sequential so
|
||||
// one request cannot fan out into dozens of concurrent workers.
|
||||
scores.push(await this.evaluateScorer(scorer, context));
|
||||
}
|
||||
const compositeScore = this.computeComposite(profile, scores);
|
||||
const result: EvaluationResult = {
|
||||
id: `evaluation_${Date.now()}_${nanoid(6)}`,
|
||||
profileId: profile.id,
|
||||
profileName: profile.name,
|
||||
action: input.action,
|
||||
output: input.output,
|
||||
agent: input.agent,
|
||||
taskId: input.taskId,
|
||||
action: parsedInput.action,
|
||||
output: parsedInput.output,
|
||||
agent: parsedInput.agent,
|
||||
taskId: parsedInput.taskId,
|
||||
metadata: context.metadata,
|
||||
scores,
|
||||
compositeScore,
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import type { AgentType } from './task.types.js';
|
||||
|
||||
export type ScorerType = 'RegexMatch' | 'KeywordContains' | 'NumericRange' | 'CustomExpression';
|
||||
export type ScorerType = 'RegexMatch' | 'KeywordContains' | 'NumericRange' | 'OccurrenceRatio';
|
||||
|
||||
export type ScoringCompositeMethod = 'weightedAvg' | 'minimum' | 'geometricMean';
|
||||
|
||||
|
|
@ -39,16 +39,23 @@ export interface NumericRangeScorer extends BaseScorer {
|
|||
scoreOnMiss?: number;
|
||||
}
|
||||
|
||||
export interface CustomExpressionScorer extends BaseScorer {
|
||||
type: 'CustomExpression';
|
||||
expression: string;
|
||||
export interface OccurrenceRatioScorer extends BaseScorer {
|
||||
type: 'OccurrenceRatio';
|
||||
needles: string[];
|
||||
caseSensitive?: boolean;
|
||||
wholeWord?: boolean;
|
||||
denominator?: number;
|
||||
denominatorPath?: string;
|
||||
denominatorScale?: number;
|
||||
minimumDenominator?: number;
|
||||
invert?: boolean;
|
||||
}
|
||||
|
||||
export type Scorer =
|
||||
| RegexMatchScorer
|
||||
| KeywordContainsScorer
|
||||
| NumericRangeScorer
|
||||
| CustomExpressionScorer;
|
||||
| OccurrenceRatioScorer;
|
||||
|
||||
export interface ScoringProfile {
|
||||
id: string;
|
||||
|
|
|
|||
|
|
@ -44,7 +44,7 @@ export type {
|
|||
RegexMatchScorer,
|
||||
KeywordContainsScorer,
|
||||
NumericRangeScorer,
|
||||
CustomExpressionScorer,
|
||||
OccurrenceRatioScorer,
|
||||
Scorer,
|
||||
ScoringProfile,
|
||||
CreateScoringProfileInput,
|
||||
|
|
|
|||
|
|
@ -51,8 +51,8 @@ const createScorer = (type: ScorerType = 'KeywordContains'): Scorer => {
|
|||
return { ...base, type, pattern: '', flags: '', invert: false };
|
||||
case 'NumericRange':
|
||||
return { ...base, type, valuePath: 'metadata.outputWordCount', min: 1, max: 500 };
|
||||
case 'CustomExpression':
|
||||
return { ...base, type, expression: 'output.length > 0 ? 1 : 0' };
|
||||
case 'OccurrenceRatio':
|
||||
return { ...base, type, needles: ['verified'], denominator: 1 };
|
||||
case 'KeywordContains':
|
||||
default:
|
||||
return {
|
||||
|
|
@ -83,7 +83,7 @@ const scorerTypeOptions: ScorerType[] = [
|
|||
'KeywordContains',
|
||||
'RegexMatch',
|
||||
'NumericRange',
|
||||
'CustomExpression',
|
||||
'OccurrenceRatio',
|
||||
];
|
||||
|
||||
const scorerTypeSelectData = scorerTypeOptions.map((type) => ({ value: type, label: type }));
|
||||
|
|
@ -805,23 +805,57 @@ export function ScoringProfiles({ onBack }: ScoringProfilesProps) {
|
|||
</>
|
||||
)}
|
||||
|
||||
{'expression' in scorer && (
|
||||
{scorer.type === 'OccurrenceRatio' && (
|
||||
<div className="space-y-2 lg:col-span-2">
|
||||
<label className="text-xs font-medium uppercase tracking-wide text-muted-foreground">
|
||||
Expression
|
||||
Literal values, one per line
|
||||
</label>
|
||||
<Textarea
|
||||
aria-label={`Scorer ${index + 1} expression`}
|
||||
aria-label={`Scorer ${index + 1} literal values`}
|
||||
rows={3}
|
||||
value={scorer.expression}
|
||||
value={scorer.needles.join('\n')}
|
||||
onChange={(event) =>
|
||||
updateScorer(index, (current) => ({
|
||||
...current,
|
||||
expression: event.target.value,
|
||||
needles: event.target.value
|
||||
.split('\n')
|
||||
.map((value) => value.trim())
|
||||
.filter(Boolean),
|
||||
}))
|
||||
}
|
||||
disabled={draftReadOnly}
|
||||
/>
|
||||
<div className="grid gap-3 sm:grid-cols-2">
|
||||
<TextInput
|
||||
aria-label={`Scorer ${index + 1} denominator`}
|
||||
type="number"
|
||||
min={1}
|
||||
placeholder="Denominator"
|
||||
value={scorer.denominator ?? ''}
|
||||
onChange={(event) =>
|
||||
updateScorer(index, (current) => ({
|
||||
...current,
|
||||
denominator:
|
||||
event.target.value === ''
|
||||
? undefined
|
||||
: Number(event.target.value),
|
||||
}))
|
||||
}
|
||||
disabled={draftReadOnly}
|
||||
/>
|
||||
<TextInput
|
||||
aria-label={`Scorer ${index + 1} denominator value path`}
|
||||
placeholder="metadata.outputWordCount"
|
||||
value={scorer.denominatorPath ?? ''}
|
||||
onChange={(event) =>
|
||||
updateScorer(index, (current) => ({
|
||||
...current,
|
||||
denominatorPath: event.target.value || undefined,
|
||||
}))
|
||||
}
|
||||
disabled={draftReadOnly}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue