6 commits

Author	SHA1	Message	Date
tin-berri	2dcd453860	feat(shadow_eval)!: gate the per-key budget on dollar spend instead of turns (#37555)	2026-08-20 14:55:21 -07:00
tin-berri	b20314efcf	fix(shadow_eval): schema-constrain the judge verdict like the classifier (#37239)	2026-08-17 18:12:07 -07:00
tin-berri	5277dab4f2	fix(shadow_eval): copy messages before router call and raise judge output cap (#37232) ... * fix(shadow_eval): copy messages before router call and raise judge output cap * fix(shadow_eval): lead failure detail with location and pin post-failure continuation	2026-08-17 17:10:52 -07:00
tin-berri	f338cfb531	feat: shadow eval samples /v1/messages and /v1/responses traffic (#36830)	2026-08-15 12:15:23 -07:00
tin-berri	2d3c3e3098	feat(shadow_eval): add reverse-direction shadow eval jobs (#36865) ... Shadow eval only answered "should this key adopt this auto-router". Once a key is on the router it is invisible to the feature, because the sampling gate skips any request the shadowed router already served, so post-adoption quality regressions go unmeasured. Reverse mode inverts the arms: sample the traffic the router did serve and duplicate it against a fixed baseline_model, judged by the same blind pairwise judge. Same job table, same attempt rows, same aggregates. real_* stays the arm the caller was served and shadow_* the duplicated one, so in reverse real_model is the router's pick and shadow_model is the baseline. The active-job slot becomes one per (key, direction) so both directions can run at once, and tier attribution in reverse reads the control request's routing decision rather than the shadow call's write-back.	2026-08-14 17:05:55 -07:00
tin-berri	d8fda675cc	feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, derived state) (#36587)	2026-08-13 13:15:45 -07:00