docs(router): publish trained V2 benchmark results

cookbook/auto_router_roi_training/ABLATIONS.md

@@ -0,0 +1,44 @@
+# Card and calibration ablations
+
+These comparisons hold the boundary at the original default: capability base 0.5 with step 0.1, or V2 quality gap 0.05. All coefficients were fitted on the training split. This table reports every raw card and the fixed middle regularization strength of 10 for calibrated variants; the JSON contains all strengths. No variant is chosen using these held-out results
+
+| Pair | Classifier | Card | Adjustment | Solved | Cost | Savings | Efficient Brier |
+|---|---|---|---|---|---:|---:|---:|
+| sonnet_opus | v2 | original | none | 23/25 | $8.619 | 0.9% | 0.080 |
+| sonnet_opus | v2 | original | per_model | 23/25 | $8.716 | -0.2% | 0.279 |
+| sonnet_opus | v2 | original | task_conditioned | 23/25 | $8.716 | -0.2% | 0.296 |
+| sonnet_opus | cap | original | none | 24/25 | $7.439 | 14.5% | 0.074 |
+| sonnet_opus | cap | original | per_model | 23/25 | $8.716 | -0.2% | 0.288 |
+| sonnet_opus | cap | original | task_conditioned | 23/25 | $8.716 | -0.2% | 0.288 |
+| sonnet_opus | v2 | research | none | 23/25 | $8.380 | 3.7% | 0.076 |
+| sonnet_opus | v2 | research | per_model | 23/25 | $8.716 | -0.2% | 0.288 |
+| sonnet_opus | v2 | research | task_conditioned | 23/25 | $8.716 | -0.2% | 0.297 |
+| sonnet_opus | cap | research | none | 24/25 | $7.433 | 14.6% | 0.066 |
+| sonnet_opus | cap | research | per_model | 23/25 | $8.710 | -0.1% | 0.288 |
+| sonnet_opus | cap | research | task_conditioned | 23/25 | $8.710 | -0.1% | 0.303 |
+| sonnet_opus | v2 | trained_card | none | 23/25 | $8.711 | -0.1% | 0.169 |
+| sonnet_opus | v2 | trained_card | per_model | 23/25 | $8.711 | -0.1% | 0.288 |
+| sonnet_opus | v2 | trained_card | task_conditioned | 23/25 | $8.711 | -0.1% | 0.304 |
+| sonnet_opus | cap | trained_card | none | 24/25 | $7.705 | 11.4% | 0.125 |
+| sonnet_opus | cap | trained_card | per_model | 23/25 | $8.710 | -0.1% | 0.288 |
+| sonnet_opus | cap | trained_card | task_conditioned | 23/25 | $8.710 | -0.1% | 0.299 |
+| luna_sol | v2 | original | none | 25/25 | $5.887 | 23.5% | 0.114 |
+| luna_sol | v2 | original | per_model | 25/25 | $7.713 | -0.2% | 0.293 |
+| luna_sol | v2 | original | task_conditioned | 25/25 | $7.713 | -0.2% | 0.309 |
+| luna_sol | cap | original | none | 23/25 | $0.454 | 94.1% | 0.095 |
+| luna_sol | cap | original | per_model | 25/25 | $7.548 | 1.9% | 0.265 |
+| luna_sol | cap | original | task_conditioned | 25/25 | $7.504 | 2.5% | 0.276 |
+| luna_sol | v2 | research | none | 25/25 | $7.132 | 7.3% | 0.114 |
+| luna_sol | v2 | research | per_model | 25/25 | $7.714 | -0.2% | 0.268 |
+| luna_sol | v2 | research | task_conditioned | 25/25 | $7.714 | -0.2% | 0.274 |
+| luna_sol | cap | research | none | 23/25 | $0.454 | 94.1% | 0.080 |
+| luna_sol | cap | research | per_model | 25/25 | $7.707 | -0.1% | 0.293 |
+| luna_sol | cap | research | task_conditioned | 25/25 | $7.707 | -0.1% | 0.315 |
+| luna_sol | v2 | trained_card | none | 25/25 | $7.708 | -0.1% | 0.228 |
+| luna_sol | v2 | trained_card | per_model | 25/25 | $7.708 | -0.1% | 0.272 |
+| luna_sol | v2 | trained_card | task_conditioned | 25/25 | $7.708 | -0.1% | 0.250 |
+| luna_sol | cap | trained_card | none | 23/25 | $1.936 | 74.9% | 0.131 |
+| luna_sol | cap | trained_card | per_model | 25/25 | $7.707 | -0.1% | 0.293 |
+| luna_sol | cap | trained_card | task_conditioned | 25/25 | $7.707 | -0.1% | 0.304 |
+
+Lower Brier means more accurate probabilities on these realized attempts. A lower Brier score can still yield worse routing at a fixed boundary. Compare these fixed-boundary controls with the separately frozen validation-selected profiles in REPORT.md. These exploratory comparisons reuse the same 25 tasks, so selecting a new winner here would require another holdout

cookbook/auto_router_roi_training/BENCHMARK.md

@@ -0,0 +1,55 @@
+# Auto Router training experiment
+
+Profiles were fitted on 83 DeepSWE tasks and selected on 30 repository-disjoint validation tasks before inspecting live grades. The live comparison uses 25 native-image-eligible SWE-bench Verified tasks and four current solver models at high effort. Each solver runs once per task; frozen task-pinned policies reuse those attempts and add their judge cost
+
+The baselines use the original classifier prompts and boundaries under the same judge settings: capability base 0.5 with step 0.1, and V2 gap 0.05 with neutral model-name profiles. This isolates classifier changes. It does not reproduce a production router that reclassifies every turn or escalates during an attempt
+
+| Pair | Policy | Solved | Cost | Savings vs capable | Efficient tasks | Lost / gained |
+|---|---|---:|---:|---:|---:|---:|
+| Sonnet / Opus | Sonnet only | 24/25 | $7.422 | 14.7% | 25 | 1 / 2 |
+| Sonnet / Opus | Opus only | 23/25 | $8.699 | 0.0% | 0 | 0 / 0 |
+| Sonnet / Opus | Original capability | 24/25 | $7.439 | 14.5% | 25 | 1 / 2 |
+| Sonnet / Opus | Original V2 | 23/25 | $8.619 | 0.9% | 1 | 0 / 0 |
+| Sonnet / Opus | Trained capability, 0 pp validation allowance | 24/25 | $7.705 | 11.4% | 22 | 1 / 2 |
+| Sonnet / Opus | Trained capability, 2 pp validation allowance | 24/25 | $7.633 | 12.3% | 22 | 1 / 2 |
+| Sonnet / Opus | Trained capability, 5 pp validation allowance | 24/25 | $7.633 | 12.3% | 22 | 1 / 2 |
+| Sonnet / Opus | Trained V2, 0 pp validation allowance | 23/25 | $8.711 | -0.1% | 0 | 0 / 0 |
+| Sonnet / Opus | Trained V2, 2 pp validation allowance | 23/25 | $8.711 | -0.1% | 0 | 0 / 0 |
+| Sonnet / Opus | Trained V2, 5 pp validation allowance | 23/25 | $8.711 | -0.1% | 0 | 0 / 0 |
+| Luna / Sol | Luna only | 23/25 | $0.444 | 94.2% | 25 | 2 / 0 |
+| Luna / Sol | Sol only | 25/25 | $7.697 | 0.0% | 0 | 0 / 0 |
+| Luna / Sol | Original capability | 23/25 | $0.454 | 94.1% | 25 | 2 / 0 |
+| Luna / Sol | Original V2 | 25/25 | $5.887 | 23.5% | 7 | 0 / 0 |
+| Luna / Sol | Trained capability, 0 pp validation allowance | 23/25 | $1.234 | 84.0% | 23 | 2 / 0 |
+| Luna / Sol | Trained capability, 2 pp validation allowance | 24/25 | $5.643 | 26.7% | 8 | 1 / 0 |
+| Luna / Sol | Trained capability, 5 pp validation allowance | 23/25 | $1.234 | 84.0% | 23 | 2 / 0 |
+| Luna / Sol | Trained V2, 0 pp validation allowance | 23/25 | $2.086 | 72.9% | 21 | 2 / 0 |
+| Luna / Sol | Trained V2, 2 pp validation allowance | 23/25 | $0.891 | 88.4% | 23 | 2 / 0 |
+| Luna / Sol | Trained V2, 5 pp validation allowance | 25/25 | $7.713 | -0.2% | 0 | 0 / 0 |
+
+The fitted settings below were selected on validation. The allowance is a constraint on average net validation loss, not the V2 gap threshold and not a production guarantee
+
+| Pair | Classifier | Validation allowance | Card | Probability adjustment | Boundary |
+|---|---|---:|---|---|---|
+| sonnet_opus | cap | 0 pp | trained_card | none | base_threshold=0.5, threshold_step=0 |
+| sonnet_opus | cap | 2 pp | research | none | base_threshold=0.72, threshold_step=0 |
+| sonnet_opus | cap | 5 pp | research | none | base_threshold=0.72, threshold_step=0 |
+| sonnet_opus | v2 | 0 pp | trained_card | task_conditioned (regularization 1) | max_quality_gap=0.1649 |
+| sonnet_opus | v2 | 2 pp | trained_card | task_conditioned (regularization 1) | max_quality_gap=0.1649 |
+| sonnet_opus | v2 | 5 pp | trained_card | task_conditioned (regularization 1) | max_quality_gap=0.1649 |
+| luna_sol | cap | 0 pp | original | task_conditioned (regularization 100) | base_threshold=0.4565, threshold_step=0 |
+| luna_sol | cap | 2 pp | original | task_conditioned (regularization 1) | base_threshold=0.5, threshold_step=0 |
+| luna_sol | cap | 5 pp | original | task_conditioned (regularization 10) | base_threshold=0.4544, threshold_step=0 |
+| luna_sol | v2 | 0 pp | original | none | max_quality_gap=0.08 |
+| luna_sol | v2 | 2 pp | research | none | max_quality_gap=0.09 |
+| luna_sol | v2 | 5 pp | original | per_model (regularization 1) | max_quality_gap=0.2561 |
+
+Equal solve counts can hide different successful tasks. Twenty-five tasks cannot establish small quality differences; the JSON includes paired repository-cluster intervals. These intervals are exploratory with few repositories. If every observed paired difference is zero, the empirical bootstrap interval is also zero and cannot estimate unseen failures. Even zero lost successes in 25 independent trials permits an 11.3% one-sided 95% binomial upper bound on that event rate; repository dependence weakens that inference. Public-data training uses different budgets and serving configurations, so this is a transfer test
+
+The initial x86 runs were excluded because of environment activation and emulator startup failures. Native tasks were selected by the same seeded repository/task ordering, skipping unavailable ARM images before any live grade inspection. The sample is not representative of every SWE-bench platform or repository
+
+The OpenAI adapter pilot was excluded because the gateway split sequential output blocks across choices and the stock harness discarded its tool calls. A later accounting correction restarted all three in-flight Anthropic attempts to capture the billed cost of malformed responses. Four completed Anthropic attempts were retained after confirming complete accounting. Grading container-name conflicts were retried with model-specific containers, preserving the solver attempts. One incomplete Sol attempt was archived and retried after gateway rate limits exhausted transport retries before submission; its $0.345 cost is excluded from the policy comparison and recorded separately. The final Sol grade exceeded 900 seconds during host slowdown; an unmodified-image CLI control also took 102 seconds. Only the saved patch was regraded with a 3600-second infrastructure deadline. These amendments, excluded attempts and unchanged frozen-selection hash are recorded in protocol.json
+
+Costs include billed solver responses, including malformed replies, plus the applicable classifier forecast. Host compute, image downloads and excluded infrastructure pilots are separate experiment expenses. The replay does not demonstrate mid-task escalation, inherited-state rescue, or a new repeated stochastic router run
+
+The JSON also reports held-out Brier scores for per-model probabilities and mean squared error for the V2 predicted gap. Those diagnostics measure probability accuracy separately from the routing threshold. One attempt per model per task does not reveal a task's true solve probability

cookbook/auto_router_roi_training/FINDINGS.md

@@ -0,0 +1,33 @@
+# What the training experiment showed
+
+Both implementations now support runnable, opt-in trained profiles. Training covered three cards, raw probabilities, per-model calibration, task-dependent calibration, and pair-specific boundaries, including combinations. All fitting and policy selection used the DeepSWE training and validation splits. The 25 fresh SWE-bench tasks were used only for evaluation
+
+The selected profiles did not establish a consistent improvement over the original routers. Most paid more for the same solve count or traded away solves for savings. They remain experimental configurations, with the original defaults preserved
+
+## Comparison with the existing routers
+
+For Sonnet 5 / Opus 5, Sonnet-only solved 24/25 for $7.422 and Opus-only solved 23/25 for $8.699. Original capability matched Sonnet's 24 solves at $7.439. The primary trained capability profile also solved 24, but cost $7.705. Original V2 solved 23 for $8.619; trained V2 selected Opus for every task, solved 23, and cost $8.711
+
+For Luna / Sol, Luna-only solved 23/25 for $0.444 and Sol-only solved 25/25 for $7.697. Original V2 solved the same 25 tasks for $5.887, saving 23.5% versus Sol. The primary trained V2 profile kept the original card and raw probabilities but raised the gap boundary from 0.05 to 0.08. It cost $2.086 and solved 23, losing both tasks Luna failed. Its lower cost therefore came with an observed quality loss. The primary trained capability profile also solved 23 and cost $1.234, versus original capability's 23 for $0.454
+
+The other validation allowances are reported in BENCHMARK.md in the PRs and REPORT.md in the full bundle. The Luna/Sol capability profile selected with a two-point validation allowance solved 24 for $5.643. That is a quality/cost tradeoff, not a profile that maximizes both. The most permissive selected V2 calibration chose Sol everywhere and added judge cost
+
+## Cards, probability calibration, and boundaries had different effects
+
+The research rewrite improved the capability classifier's efficient-model Brier score from 0.0736 to 0.0662 for Sonnet, and from 0.0953 to 0.0804 for Luna. With the original boundary, it still routed all 25 tasks to the efficient solver. This is a descriptive signal for better card wording, without an observed solve-rate improvement or a meaningful solver-cost advantage. It was not promoted to a new winner using the held-out outcomes
+
+The training-derived priors and learned probability adjustments transferred poorly. The primary trained V2 profile predicted a mean 37-point Opus advantage, while the realized paired difference was a 4-point Sonnet advantage. Its Sonnet probability averaged 0.434 against observed success of 0.96. The primary Luna capability calibration averaged 0.463 against observed success of 0.92. These estimates and Brier scores show a calibration problem on this workload, independently of the threshold
+
+Threshold tuning cannot repair that probability error. The Luna/Sol 0.05-to-0.08 comparison also shows that a boundary can be too permissive even when the mean predicted gap is close to the observed average gap. Request-level ranking and the placement of harmful downgrades matter, as well as average calibration
+
+## What to train next
+
+Use paired outcomes collected with the same agent, model effort, tools and budget as the intended deployment, with a new repository-held-out evaluation set. Treat these 25 tasks as used evaluation data. DeepSWE supplied useful fitting data, but this experiment does not establish whether its domain, task difficulty, budget or serving differences caused the failed transfer
+
+Train a model-pair estimate of the expected quality difference and the risk that only the stronger solver succeeds, alongside per-model expected total attempt cost. Some Sonnet attempts needed 71-84 model calls, so nominal token prices alone do not capture the routing opportunity. Keep task labels and model-specific probabilities as inputs, and test shrinking learned corrections toward the original probabilities when matching evidence is sparse
+
+Keep the research-only card rewrite as a candidate for that next evaluation. Do not increase the boundary merely to make these held-out results look better. The unchanged Luna/Sol V2 configuration is a useful quality-preserving control on this sample, and the efficient-only model is an essential savings control
+
+## Scope of the evidence
+
+These results come from 100 fresh solver attempts, one attempt per model per task, plus 300 fresh classifier forecasts. The policies are frozen, task-pinned paired replays over those attempts, with measured judge cost added. They do not measure per-turn reclassification, escalation or additional independently sampled router runs. Twenty-five tasks cannot establish a small production quality-loss guarantee

cookbook/auto_router_roi_training/README.md

@@ -28,6 +28,8 @@ curl http://localhost:4000/v1/chat/completions \
 
 `profiles.json` contains the exact cards, coefficients, thresholds and validation results for each alias. `manifest.json` records model versions, effort, sources and limitations. `training_records.jsonl` retains the numerical forecasts and paired outcome evidence used for fitting and selection
 
+For controlled comparisons, `fixed_boundary_profiles.json` contains 18 additional configurations: each model pair, each of the three cards, and raw, per-model, or task-dependent calibration. Calibration uses the fixed middle regularization strength of 10. These keep the original routing boundary unchanged and were not selected using live outcomes. Replace one alias's `complexity_router_config` with the chosen entry to benchmark it
+
 ## What was fitted
 
 DeepSWE v1.1 supplies 113 tasks, with repeated attempts for each solver. We split repositories into 83 training tasks and 30 validation tasks. Within each task, repeated outcomes are averaged; each task receives equal fitting weight. Missing bills remain missing and are excluded as matched pairs only from cost calculations
@@ -45,3 +47,9 @@ The raw model probabilities and calibrated probabilities are separate outputs. B
 These are small-sample experimental snapshots. Model, judge, prompt, effort, task distribution and harness changes can invalidate the coefficients. Published DeepSWE outcomes use different budgets and serving configurations from the live SWE-bench pilot, so that pilot measures transfer. Zero observed validation loss is a point estimate, not statistical noninferiority
 
 The capability classifier uses an absolute efficient-model probability threshold plus its boundary step. V2 compares the two models' predicted probabilities. Their threshold values have different meanings and should not be copied between classifiers
+
+## Fresh benchmark results
+
+Read [FINDINGS.md](FINDINGS.md) for the interpretation and next experiments. See [BENCHMARK.md](BENCHMARK.md) for the 25-task comparison against the original classifiers and each fixed model. [ABLATIONS.md](ABLATIONS.md) holds the boundary fixed to isolate card and calibration changes. The JSON files include per-task routes, outcomes, cost, lost and gained solves, probability diagnostics, and the frozen selection hash
+
+The learned Sonnet/Opus V2 profiles choose Opus on every live task and add classifier cost. That configuration provides no savings on this sample. Use the complete tables to compare other profiles with both fixed-model baselines; validation gains do not establish transfer to this workload

cookbook/auto_router_roi_training/manifest.json

@@ -25,7 +25,31 @@
     "83 fitting tasks and 30 validation tasks support only exploratory estimates",
     "Published training outcomes use different task budgets and serving configurations from the live pilot",
     "Missing training cost records remain missing and are excluded as pairs only from cost metrics",
-    "Luna/Sol profiles have public-data validation only unless a corresponding fresh live evaluation is listed",
-    "Training retains the source included_in_score filter; excluded provider/infrastructure failures are not modeled as solver failures"
-  ]
+    "Training retains the source included_in_score filter; excluded provider/infrastructure failures are not modeled as solver failures",
+    "One attempt per solver per task; 25 tasks do not establish a small production quality-loss bound"
+  ],
+  "live_evaluation": {
+    "tasks": 25,
+    "solver_attempts": 100,
+    "judge_forecasts": 300,
+    "dataset": "SWE-bench Verified",
+    "dataset_revision": "c104f840cc67f8b6eec6f759ebc8b2693d585d4a",
+    "architecture": "native ARM64",
+    "harness": "mini-swe-agent==2.0.0",
+    "grader": "swebench==4.1.0",
+    "solver_models": [
+      "anthropic/claude-sonnet-5",
+      "anthropic/claude-opus-5",
+      "openai/gpt-5.6-luna",
+      "openai/gpt-5.6-sol"
+    ],
+    "comparison": "Task-pinned paired replay of frozen policies, with recorded solver and judge cost",
+    "files": [
+      "FINDINGS.md",
+      "BENCHMARK.md",
+      "benchmark_results.json",
+      "ABLATIONS.md",
+      "ablation_diagnostics.json"
+    ]
+  }
 }