This commit is contained in:
tin-berri 2026-09-19 14:51:00 -04:00 • committed by GitHub
commit 9ea112e30d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
31 changed files with 38642 additions and 31 deletions

View file

@ -0,0 +1,44 @@
# Primary findings: quality retained versus money saved
Training improved some comparisons, but it did not approach the ideal Luna-first policy across all 125 held-out tasks. The main profiles selected on validation do not combine preservation of every Sol success with lower total cost across all four benchmarks
Sol solved 87/125 at $20.8964. Luna solved 84/125 at at least $2.6531. There were 11 Sol-only successes, eight Luna-only successes, 76 successes shared by both, and 30 tasks both attempts failed. A perfect Luna-first decision would therefore reach 95/125 at at least $5.2653. That is an unattainable hindsight reference, with at most 74.8% savings using the recorded bills
| Frozen comparison | Result | Cost | Sol successes lost |
|---|---|---:|---:|
| Capability retention profile, SWE-bench | 18/25, matching Sol | $6.2493, 41.7% savings | 0 |
| V2 quality profile, Terminal-Bench | 19/25 versus Sol's 18/25 | $8.1712, 3.3% savings | 0 |
| V2 retention profile, all tasks | 85/125 versus Sol's 87/125 | $14.0036, 33.0% savings | 4 |
| V2 review retention profile, all tasks | 88/125 versus Sol's 87/125 | At least $20.9754, higher than Sol | 0 |
| V2 review quality profile, all tasks | 88/125 versus Sol's 87/125 | At least $8.6493, at most 58.6% savings | 7 |
These benchmark-specific successes are separate frozen policies. Combining the winning profile for each benchmark would be a new policy requiring fresh evaluation
Relative to the original V2 baseline, the trained V2 retention profile improves aggregate quality from 84 to 85 solves and reduces cost from at least $17.0029 to $14.0036. That is at least a 17.6% cost reduction against that baseline. It still misses four Sol successes, so it does not meet the stronger preservation goal
Among the 40 prespecified family controls, three capability variants retained every Sol success and gained one additional solve. The per-model adjustment control achieved 88/125 at $20.3797, a 2.5% saving. The paired and rescue controls saved 0.2% and 0.6%. These are exploratory comparisons across many variants, not a new winner selection. All twelve upfront family configurations per classifier are provided for reproduction and fresh benchmarking; none was refitted on these outcomes
The expensive V2 review policy captured all 11 Sol-only successes by escalating 115 of 125 tasks. Its cheaper review policy escalated 15 tasks but captured only four of those 11 rescues. This is the remaining problem: identify the rare tasks where Sol changes failure into success without also sending most other tasks to Sol
Calibration improved more than routing precision. V2's selected scalar calibration reduced the Luna Brier score from 0.223 to 0.205 and the Sol Brier score from 0.237 to 0.211. However, probability-gap MSE only moved from 0.153 to 0.151. Ranking Sol-only rescues by the estimated gap improved from ROC AUC 0.618 to 0.689, still far from perfect separation. The empirical review variant reached 0.742 gap-ranking AUC, but its frozen economical threshold missed seven rescues. A more accurate average success probability does not by itself isolate the rescue tasks
The selected upfront profiles retain the original cards. Their improvements come from fitted adjustments and decision boundaries. Card rewrites alone gave mixed results. Training contained only 13 Sol-only successes, including two agentic examples, which limits evidence for agentic rescue decisions. Additional paired agentic training data and stronger verification signals are plausible next directions; this study does not establish their effect
All policies and family controls were frozen before final outcomes. These are task-level replays of independent attempts, not per-turn routing or Sol continuation from a Luna patch. Small samples and the search across development candidates limit generalization. One attempt per model does not establish a stable solve probability for an individual task
Two Luna solver requests interrupted by host sleep have unknown charges. An earlier unavailable DNA-assembly review has one additional unmetered connection-timeout request. Thus Luna-only comparisons contain two unmetered requests, and review cascades contain three. The report marks affected costs as lower bounds and savings as upper bounds. The three upfront comparisons quoted with exact costs above are fully metered
The additional Sonnet/Opus controls are complete. All 500 final attempts and grades are present, and the complete reporter verified that the previously published Luna/Sol results and calibration diagnostics are unchanged
| Fixed model | Solved | Recorded inference cost |
|---|---:|---:|
| Luna | 84/125 | At least $2.6531 |
| Sol | 87/125 | $20.8964 |
| Sonnet 5 | 81/125 | At least $37.9961 |
| Opus 5 | 99/125 | At least $46.0550 |
Opus solved 22/25 SWE-bench tasks at $10.5296, compared with Sol's 18/25 at $10.7132. On Terminal-Bench it solved 21/25 at at least $32.9157, compared with Sol's 18/25 at $8.4468. These are observed results under the frozen harness and budgets, not unrestricted model rankings. Across all tasks, even perfect selection between the saved Luna and Sol attempts reaches only 95/125, below Opus's 99/125. Changing the candidate model pair is a separate training and evaluation question
All four models had an 8,192-token response cap and high reasoning effort. Corrected Sonnet and Opus attempts still incurred 116 and 83 length stops with no action or visible text, costing $10.3656 and $17.9127 respectively. Those responses remain in the results. Increasing the response cap could change quality and spending, but this experiment did not test that change. Host sleep and unresolved transport charges also prevent latency conclusions and exact affected cost comparisons
The practical next experiment is to collect substantially more paired agentic training examples where Sol rescues Luna, then evaluate rescue prediction and verification on fresh tasks. The present fitting set contains only two such agentic cases. The current final set is now diagnostic evidence and must not be reused as an unseen test after training on its failures. Better average probability calibration alone did not deliver the requested selective-rescue precision

View file

@ -0,0 +1,116 @@
# Held-out Luna/Sol routing results
All 125 held-out tasks and 250 Luna/Sol attempts. Frozen task-level policy replay; additional Anthropic controls remain in the full report. No refitting or reselection.
Retention and quality denote development-selection objectives, not guarantees. Costs include classifier/reviewer inference and discarded Luna attempts on escalation. These are independent paired attempts; Sol starts from the original task, not from Luna's partial work.
Costs use recorded gateway bills. A ≥ cost and ≤ savings mark unmetered transport requests: cost is a lower bound and savings versus fully metered Sol is an upper bound. The JSON counts these requests per policy and task. They are not assumed free.
## all
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | Sol-only rescues captured |
|---|---:|---:|---:|---:|---:|
| Luna only | 84/125 | ≥$2.6531 | ≤87.3% | +8 / -11 | 0/11 |
| Sol only | 87/125 | $20.8964 | 0.0% | +0 / -0 | 11/11 |
| Capability upfront, retention objective | 86/125 | $13.8156 | 33.9% | +1 / -2 | 9/11 |
| Capability upfront, quality objective | 88/125 | $18.5237 | 11.4% | +5 / -4 | 7/11 |
| V2 upfront, retention objective | 85/125 | $14.0036 | 33.0% | +2 / -4 | 7/11 |
| V2 upfront, quality objective | 86/125 | $18.1999 | 12.9% | +4 / -5 | 6/11 |
| Capability baseline | 85/125 | $6.7642 | 67.6% | +8 / -10 | 1/11 |
| Capability with learned cards only | 84/125 | $7.2496 | 65.3% | +7 / -10 | 1/11 |
| V2 baseline | 84/125 | ≥$17.0029 | ≤18.6% | +3 / -6 | 5/11 |
| V2 with learned cards only | 85/125 | ≥$14.6487 | ≤29.9% | +6 / -8 | 3/11 |
| Capability review, retention objective | 81/125 | ≥$7.5830 | ≤63.7% | +0 / -6 | 5/11 |
| Capability review, quality objective | 87/125 | ≥$10.1853 | ≤51.3% | +7 / -7 | 4/11 |
| V2 review, retention objective | 88/125 | ≥$20.9754 | ≤-0.4% | +1 / -0 | 11/11 |
| V2 review, quality objective | 88/125 | ≥$8.6493 | ≤58.6% | +8 / -7 | 4/11 |
| Hindsight direct routing (not deployable) | 95/125 | ≥$4.5190 | ≤78.4% | +8 / -0 | 11/11 |
| Hindsight Luna-first routing (not deployable) | 95/125 | ≥$5.2653 | ≤74.8% | +8 / -0 | 11/11 |
## mbpp
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | Sol-only rescues captured |
|---|---:|---:|---:|---:|---:|
| Luna only | 38/50 | $0.0174 | 91.6% | +3 / -2 | 0/2 |
| Sol only | 37/50 | $0.2064 | 0.0% | +0 / -0 | 2/2 |
| Capability upfront, retention objective | 37/50 | $0.2180 | -5.6% | +0 / -0 | 2/2 |
| Capability upfront, quality objective | 38/50 | $0.0444 | 78.5% | +3 / -2 | 0/2 |
| V2 upfront, retention objective | 37/50 | $0.2298 | -11.3% | +0 / -0 | 2/2 |
| V2 upfront, quality objective | 38/50 | $0.0465 | 77.4% | +3 / -2 | 0/2 |
| Capability baseline | 38/50 | $0.0290 | 85.9% | +3 / -2 | 0/2 |
| Capability with learned cards only | 38/50 | $0.0284 | 86.2% | +3 / -2 | 0/2 |
| V2 baseline | 38/50 | $0.0562 | 72.8% | +3 / -2 | 0/2 |
| V2 with learned cards only | 38/50 | $0.0396 | 80.8% | +3 / -2 | 0/2 |
| Capability review, retention objective | 37/50 | $0.2503 | -21.3% | +0 / -0 | 2/2 |
| Capability review, quality objective | 38/50 | $0.0434 | 79.0% | +3 / -2 | 0/2 |
| V2 review, retention objective | 37/50 | $0.2441 | -18.3% | +0 / -0 | 2/2 |
| V2 review, quality objective | 38/50 | $0.0488 | 76.4% | +3 / -2 | 0/2 |
| Luna with public-test escalation | 38/50 | $0.0240 | 88.4% | +2 / -1 | 1/2 |
| Hindsight direct routing (not deployable) | 40/50 | $0.0353 | 82.9% | +3 / -0 | 2/2 |
| Hindsight Luna-first routing (not deployable) | 40/50 | $0.0361 | 82.5% | +3 / -0 | 2/2 |
## lcb
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | Sol-only rescues captured |
|---|---:|---:|---:|---:|---:|
| Luna only | 13/25 | $0.1165 | 92.4% | +2 / -3 | 0/3 |
| Sol only | 14/25 | $1.5299 | 0.0% | +0 / -0 | 3/3 |
| Capability upfront, retention objective | 14/25 | $1.5476 | -1.2% | +0 / -0 | 3/3 |
| Capability upfront, quality objective | 14/25 | $0.9200 | 39.9% | +1 / -1 | 2/3 |
| V2 upfront, retention objective | 14/25 | $1.3923 | 9.0% | +0 / -0 | 3/3 |
| V2 upfront, quality objective | 12/25 | $1.1104 | 27.4% | +0 / -2 | 1/3 |
| Capability baseline | 13/25 | $0.2918 | 80.9% | +2 / -3 | 0/3 |
| Capability with learned cards only | 12/25 | $0.4860 | 68.2% | +1 / -3 | 0/3 |
| V2 baseline | 12/25 | $0.9924 | 35.1% | +0 / -2 | 1/3 |
| V2 with learned cards only | 13/25 | $0.4799 | 68.6% | +2 / -3 | 0/3 |
| Capability review, retention objective | 14/25 | $1.6846 | -10.1% | +0 / -0 | 3/3 |
| Capability review, quality objective | 14/25 | $1.3257 | 13.4% | +2 / -2 | 1/3 |
| V2 review, retention objective | 15/25 | $1.6076 | -5.1% | +1 / -0 | 3/3 |
| V2 review, quality objective | 14/25 | $1.0584 | 30.8% | +2 / -2 | 1/3 |
| Luna with public-test escalation | 15/25 | $1.3645 | 10.8% | +2 / -1 | 2/3 |
| Hindsight direct routing (not deployable) | 16/25 | $0.3436 | 77.5% | +2 / -0 | 3/3 |
| Hindsight Luna-first routing (not deployable) | 16/25 | $0.3657 | 76.1% | +2 / -0 | 3/3 |
## swe
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | Sol-only rescues captured |
|---|---:|---:|---:|---:|---:|
| Luna only | 16/25 | $0.6371 | 94.1% | +0 / -2 | 0/2 |
| Sol only | 18/25 | $10.7132 | 0.0% | +0 / -0 | 2/2 |
| Capability upfront, retention objective | 18/25 | $6.2493 | 41.7% | +0 / -0 | 2/2 |
| Capability upfront, quality objective | 18/25 | $9.7546 | 8.9% | +0 / -0 | 2/2 |
| V2 upfront, retention objective | 16/25 | $5.3537 | 50.0% | +0 / -2 | 0/2 |
| V2 upfront, quality objective | 17/25 | $8.8718 | 17.2% | +0 / -1 | 1/2 |
| Capability baseline | 16/25 | $1.3893 | 87.0% | +0 / -2 | 0/2 |
| Capability with learned cards only | 16/25 | $1.6826 | 84.3% | +0 / -2 | 0/2 |
| V2 baseline | 17/25 | $9.1304 | 14.8% | +0 / -1 | 1/2 |
| V2 with learned cards only | 17/25 | $7.4181 | 30.8% | +0 / -1 | 1/2 |
| Capability review, retention objective | 16/25 | $2.4752 | 76.9% | +0 / -2 | 0/2 |
| Capability review, quality objective | 16/25 | $0.9753 | 90.9% | +0 / -2 | 0/2 |
| V2 review, retention objective | 18/25 | $9.1263 | 14.8% | +0 / -0 | 2/2 |
| V2 review, quality objective | 16/25 | $0.9775 | 90.9% | +0 / -2 | 0/2 |
| Hindsight direct routing (not deployable) | 18/25 | $1.4342 | 86.6% | +0 / -0 | 2/2 |
| Hindsight Luna-first routing (not deployable) | 18/25 | $1.5478 | 85.6% | +0 / -0 | 2/2 |
## terminal
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | Sol-only rescues captured |
|---|---:|---:|---:|---:|---:|
| Luna only | 17/25 | ≥$1.8822 | ≤77.7% | +3 / -4 | 0/4 |
| Sol only | 18/25 | $8.4468 | 0.0% | +0 / -0 | 4/4 |
| Capability upfront, retention objective | 17/25 | $5.8008 | 31.3% | +1 / -2 | 2/4 |
| Capability upfront, quality objective | 18/25 | $7.8047 | 7.6% | +1 / -1 | 3/4 |
| V2 upfront, retention objective | 18/25 | $7.0279 | 16.8% | +2 / -2 | 2/4 |
| V2 upfront, quality objective | 19/25 | $8.1712 | 3.3% | +1 / -0 | 4/4 |
| Capability baseline | 18/25 | $5.0541 | 40.2% | +3 / -3 | 1/4 |
| Capability with learned cards only | 18/25 | $5.0526 | 40.2% | +3 / -3 | 1/4 |
| V2 baseline | 17/25 | ≥$6.8238 | ≤19.2% | +0 / -1 | 3/4 |
| V2 with learned cards only | 17/25 | ≥$6.7112 | ≤20.5% | +1 / -2 | 2/4 |
| Capability review, retention objective | 14/25 | ≥$3.1729 | ≤62.4% | +0 / -4 | 0/4 |
| Capability review, quality objective | 19/25 | ≥$7.8410 | ≤7.2% | +2 / -1 | 3/4 |
| V2 review, retention objective | 18/25 | ≥$9.9975 | ≤-18.4% | +0 / -0 | 4/4 |
| V2 review, quality objective | 20/25 | ≥$6.5646 | ≤22.3% | +3 / -1 | 3/4 |
| Hindsight direct routing (not deployable) | 21/25 | ≥$2.7059 | ≤68.0% | +3 / -0 | 4/4 |
| Hindsight Luna-first routing (not deployable) | 21/25 | ≥$3.3157 | ≤60.7% | +3 / -0 | 4/4 |
The hindsight rows use hidden outcomes and are unattainable routing references. One attempt per model and small per-benchmark samples do not establish future reliability. JSON records exploratory paired uncertainty, every frozen ablation and per-task decisions. Study training and discarded harness-run spending is separate from deployed-policy cost.

View file

@ -0,0 +1,130 @@
# Selective routing experiment, 15 September 2026
The target is preserving stronger-model task quality while reducing total inference spending through selective use of the cheaper model. No contaminated outcomes from earlier experiments may be used
The primary model pair is GPT-5.6 Luna and GPT-5.6 Sol at high effort. Sonnet-5 and Opus-5 are additional final-evaluation controls. Model identities and gateway billing must match each response. The original capability classifier and fused V2 are baselines, both using the same Luna judge
The final allocation below supersedes the initial targets and intermediate preparation counts recorded in the amendment history. Training uses 160 tasks, validation uses 64 eligible tasks, and final evaluation uses 125 tasks. Each final task has one independent attempt from each of the four models, for 500 attempts
| Benchmark | Training | Validation | Final |
|---|---:|---:|---:|
| MBPP+ | 80 | 29 | 50 |
| LiveCodeBench | 50 | 20 | 25 |
| SWE-bench Verified | 25 | 10 | 25 |
| Terminal-Bench | 5 | 5 | 25 |
The initial final runner used three concurrent SWE task workers and two concurrent Terminal trials. The corrected Anthropic-control rerun uses one SWE worker and two Terminal trials, prioritizing remaining Luna/Sol Terminal attempts. These limits share the machine with other work and do not change individual model budgets. The benchmark selection and offline environment controls make this an adapted subset, not an official leaderboard submission
The following paragraphs preserve the initial plan and its amendments, including why some preparation targets changed
Development uses 25 SWE-bench Verified training tasks and 10 validation tasks, plus 80 MBPP+ training tasks and 30 validation tasks. Terminal-Bench task counts will be fixed after environment-only eligibility checks, targeting at least 15 training and 10 validation tasks. Final evaluation uses 25 SWE-bench, 25 Terminal-Bench and 50 MBPP+ tasks. SWE-bench repositories are disjoint across splits; previously inspected tasks are excluded. Terminal-Bench variants of the same mechanism must remain in the same split
Solver outcomes are new independent paired attempts. No gold fixes, future Git objects, hidden tests, prior trajectories or benchmark results are exposed to solvers. Docker hosts, credentials and task sources are not mounted into solver environments. SWE-bench uses a one-commit Git database with no unreachable objects. Terminal-Bench tests and reference solution are uploaded only in isolated control or grading phases. MBPP solutions are generated from prompts only and executed in a sandbox by EvalPlus
Candidate methods, fixed before final evaluation: original classifiers; adjusted per-model probabilities; regularized task-feature predictors of paired outcomes (both fail, cheap-only, strong-only, both succeed); empirical capability cards using training evidence; and cheap-first verification based only on the prompt and cheap solver trajectory. Training and threshold selection must use development splits only. Cost includes classifiers, verifiers and failed attempts. Shared solver attempts enable task-level policy replay, not claims about independent live router arms or stateful continuation
Hyperparameter selection minimizes validation cost among candidates with zero stronger-only losses and no net quality loss within each benchmark. Also report the validation cost-quality frontier. Always-strong is an allowed fallback, not evidence of training progress. Final policies and route decisions must be frozen before final labels are generated. Hindsight oracle routing is labeled an unattainable reference, never a trained result. Final reports include paired lost/gained solves, escalation precision/recall, per-benchmark performance, cost and small-sample uncertainty
MBPP uses EvalPlus 0.3.1, MBPP+ v0.2.0. SWE-bench uses official grader 4.1.0 and mini-swe-agent 2.0.0. Terminal-Bench uses Harbor 0.23.0 with a metered mini-swe-agent adapter. No default production routing policy is changed based on an exploratory benchmark
Control-only amendment: MBPP/255 is excluded from scoring before reading any model grades because its own reference implementation fails an extreme combination-output case under the sandbox memory limit. All attempts and costs are retained. MBPP scoring has 80 training, 29 validation and 50 final tasks. SWE-bench native-image eligibility yielded a repository-disjoint allocation of Django training, SymPy validation, and the remaining eligible repositories for final evaluation. This is a platform-limited subset and must be labeled accordingly
Development selection clarification, before any final outcomes: report two frozen profiles for each classifier. The primary profile maximizes validation solves under the Sol-only total-cost budget, then minimizes cost, while meeting Sol aggregate quality within each benchmark. The secondary profile minimizes cost with zero Sol-only successes lost in validation. The first can gain cheap-only successes while missing a different Sol-only task; report these separately rather than hiding task swaps behind aggregate quality. This distinction follows the requested quality-and-savings objective and does not promise per-task dominance
LiveCodeBench v6 medium/hard problems are added because the MBPP development split contains too few Sol-only rescue examples. Selection is chronological and contest-disjoint:50train/20validation/25test. Prompts alone go to the solver. The official pinned harness runs public and private tests in a network-disabled Docker grader, after synthetic positive/negative grader controls. This addition was chosen using development evidence, before final outcomes
Forecast-condition correction: MBPP and LiveCodeBench original forecasts mistakenly described an agent with shell/test feedback. Those development forecasts and provisional fits are archived under development_harness_correction. Replacement forecasts specify their actual single-response/no-tools/8192-token setting. Solver outputs, grades, splits and final-task quarantine are unchanged. The earlier development savings diagnostic is superseded pending refit
Runtime feature contract: upfront learned heads use only the existing classifier verdict, including probabilities, rule/demand/verification categories and bounded features of its crux text. They do not use benchmark identity, task ID, repository identity or hidden labels. This permits deterministic inference from one classifier response. Post-attempt review policies additionally need the solver output/review and are evaluated as a separate cascade
Transport amendment before final evaluation: difficult LiveCodeBench responses exceeded the initial180-second HTTP read timeout. Increase the transport allowance to600seconds without changing model effort or token budgets; retain every completed response and resume only missing outputs. Early unresolved transport calls have unknown billing and are reported separately as research overhead, never treated as free calls or wrong-code grades. Subsequent requests have durable start/response/error records
Control scheduling: Terminal-Bench environment validation uses two disjoint workers. The original worker handles the first26candidates; a second worker reserves and processes the final26. This changes only preparation throughput, not solver budgets, task criteria or selection
Terminal-Bench development can start on five eligible tasks while remaining environment controls complete. These tasks are assigned permanently to training before inspecting their paired outcomes, including the known pilot task. The final25-task reserve and validation set will be fixed using only remaining environment eligibility and task identity. Any additional training allocation must respect that reserve
Agentic forecast conditions are also normalized before fitting: both classifier prompts now state the actual disconnected shell, token budget and command timeout; capability names Luna explicitly, and Terminal-Bench includes each task wall-clock budget. The previous development-only agentic forecasts are archived; solver attempts and grades are unchanged.
The empirical card variant uses one prespecified Sol synthesis of training-only paired outcomes and fallible classifier/reviewer descriptions, plus measured aggregate counts by execution family. It must contain general mechanisms, no task identities or solutions, and is not rewritten against validation outcomes. Deterministic learned heads additionally compare raw threshold tuning, scalar probability calibration, task-dependent calibration, paired outcomes, Sol-only rescue risk, and expected incremental quality per predicted dollar. Cost prediction is fitted on training costs only. Fixed threshold grids replace validation-specific score cutoffs.
Final-label quarantine: freeze fitted policies before launching final attempts and upfront routes before final solver runs. A post-attempt cascade necessarily waits for Luna output; its deterministic route freezer loads only classifier forecasts, Luna attempts and reviews, never grade files. Harbor may write automatic grades on disk, but aggregate final analysis and model comparisons remain quarantined until all routes are frozen. No policy tuning or card revision occurs after final inference begins.
Terminal-Bench validation begins on five hash-ordered currently control-eligible tasks, fixed before their model attempts. The existing five training tasks stay fixed; at least 25 final tasks will be allocated from other eligible groups. Additional eligible tasks may enlarge development only after this final reserve is secured. This rolling preparation allocation is a convenience sample, not the full Terminal-Bench distribution.
Post-attempt variants can use deterministic checks of task-supplied public examples on MBPP+ and LiveCodeBench. The checker input excludes private test cases. The resulting available/passed signal is an additional post-attempt feature, and public-check-only escalation is reported as a separate baseline where examples exist. Checker CPU time is reported separately from inference dollars. The original solver attempts still receive no repair feedback.
Three original SWE final candidates failed their gold-reference controls. They are replaced, before final inference, by the first three passing tasks in a preregistered hash-ordered unused Sphinx pool. Training and validation remain unchanged and final evaluation retains 25 tasks. Exclusion records and original allocation are retained.
Before final Terminal-Bench inference, agent isolation expands to every nested .git path in the container, verified by a synthetic future-commit probe. Existing development task Dockerfiles were checked: their only source Git checkout is the already-sanitized /app Bottle repository. New trial network audits also record filesystem layers and an image-configuration digest, and retain an image tag for reproduction. Final model comparisons must verify matching task filesystem layers.
Final solver scheduling uses up to four concurrent tasks/trials per agentic benchmark, while retaining every individual task/model resource and token limit. The USD 5 stopping rule is checked before the next query and can exceed that amount by the last billed response. Known billing is recorded in full.
Development allocation is now fixed at 160 paired training tasks and 64 eligible validation tasks: 80/29 MBPP+, 50/20 LiveCodeBench, 25/10 SWE-bench and 5/5 Terminal-Bench. Additional control-eligible Terminal tasks are reserved for final evaluation or left unused. This keeps card synthesis and all calibration fits on one fixed training corpus while final environment preparation continues
Before final inference, freeze a diagnostic ablation per classifier, card variant, stage and training family. Each uses the same validation criterion of zero Sol-only losses, no per-benchmark quality loss and cost no greater than Sol. If no family candidate qualifies, record an explicit always-Sol fallback. Report all of these ablations, without selecting another winner from final scores, to distinguish card changes, probability fitting and threshold fitting
During final execution, a cost audit read a completed Terminal attempt ledger while the periodic exporter was copying that same ledger over its destination. A subsequent read matched the saved run cost exactly. Exported records now use atomic file replacement so concurrent readers never see a partially copied file. No solver attempt, billed response, grade, fitted policy or routing decision was changed
## Terminal runner interruption, September 16 UTC
At 00:06 UTC the Terminal runner, review watcher and keep-awake processes were absent. No terminating error was recorded; the cause remains unknown. The two active Scheme-interpreter containers were still running and neither had a completed agent record or trial result. Their partial transcripts, 95 billed responses ($4.252549 known), outstanding request IDs and container diffs were archived in `infrastructure_interruptions/20260916-terminal-process-loss`. Two outstanding requests have unknown billing and are not assumed free
The 58 completed Terminal trial results were hashed and preserved, without consulting their quality labels. Harbor resumed the identical job configuration and reran only the two interrupted attempts and 40 unstarted trials. The custom agent has no supported trajectory-resume path. The restarted processes use independent process sessions; inference budgets, model settings, task allocation and frozen policies are unchanged. Archived interruption costs belong to research overhead, separately from deployment policy cost. No completed attempt was rerun because of its outcome
## Response-limit diagnostics
Metadata inspection during the still-blinded final run found repeated length-stopped responses that used all 8,192 completion tokens without producing a tool call or visible text. Further inspection discovered the Anthropic message-preservation defect documented below. The affected agentic controls are superseded; their costs remain research overhead, and their responses are excluded from final quality and execution tables. The defect's effect on empty responses has not been established. `execution_diagnostics.json` describes only retained attempts. Empty length stops in corrected attempts are billed behavior under the frozen budget and do not themselves trigger a retry
## Anthropic tool-continuation correction, September 16 UTC
The benchmark client reconstructed assistant messages from content, tool calls and OpenAI reasoning items. This discarded Anthropic `thinking_blocks` and `provider_specific_fields`, which must be preserved on tool continuation. The corrected adapter returns the complete Anthropic assistant message unchanged. Both Sonnet and Opus passed live thinking-plus-tool round trips through the gateway. See the [provider's thinking/tool documentation](https://platform.claude.com/docs/en/build-with-claude/thinking-tool-workflows) and `harness_corrections/anthropic_thinking/live_probe.json`
All 50 SWE and 50 Terminal Anthropic controls are run under the correction, regardless of old outcomes. Fifty completed SWE attempts, 32 completed Terminal attempts and two in-progress Terminal trials were archived with hashes before restarting. The old final quality labels were not inspected. Single-response Anthropic MBPP+/LiveCodeBench controls are unaffected because they have no tool continuation. The OpenAI message merger is unchanged across 3,218 saved responses; 2,691 saved OpenAI tool turns were checked, including 2,678 containing intact reasoning items. All Luna/Sol training, validation, final attempts, frozen coefficients, thresholds and route selections are retained
The archive records 5,360 unaffected files verified unchanged, the preserved trial-result locations, canceled request IDs and the replacement schedule. Superseded controls, interrupted requests and adapter probes are included separately in research spending. Unknown outstanding-request bills are not assumed zero. SWE reruns use the retained peer's image digest and a fresh model-specific grader run prefix. No task, individual inference budget, prompt, grading criterion or fitted policy changed. The corrected Terminal agent identifies itself as adapter3
Once all 250 Luna/Sol final attempts and their grades are complete and the post-attempt routing plan is frozen, a primary-pair report may be generated while the additional Anthropic controls finish. This is an early release of the same prespecified comparison, with all 125 final tasks included; it does not select new policies or refit using final outcomes. The complete report still requires all 500 valid final attempts
## Review-route readiness guard
Before freezing the post-attempt routing plan, the runner now requires each Terminal Luna attempt to have completed grading with no infrastructure exception. Normal agent timeouts remain eligible. This guard inspects only completion and exception metadata; success labels and rewards are not inputs to any routing decision. A synthetic regression check verifies the same readiness result for successful and unsuccessful attempts and rejects transport failures. Learned coefficients, thresholds, features and candidate selections are unchanged
## Host low-power sleep and partial billing, September 16 UTC
The host entered low-power sleep at 02:42:53 UTC with 1% battery and woke on AC at 03:07:43 UTC, a 1,490-second pause despite the existing idle-sleep assertion. All benchmark processes survived. The in-flight Luna requests on regex-chess and path-tracing returned transport ReadError; the pinned mini-swe-agent model wrapper retried them automatically and both attempts continued. No whole attempt was rerun and no global power settings were changed. Power events and request IDs are retained in infrastructure_interruptions/20260916-low-power-sleep
Known billed responses stay attached to their attempts. The two interrupted requests have unknown charges. Policy reports count every unmetered request in the selected solver attempt, classifier/reviewer calls and discarded Luna attempt on escalation. Affected costs are lower bounds; savings versus a fully metered Sol baseline are upper bounds. Missing bills are not assumed zero. If a baseline is also unmetered, its cost comparison is marked unresolved. Cost uncertainty intervals use recorded bills only. This accounting change does not alter coefficients, thresholds or routes
Saved wall durations include host sleep. Elapsed duration is absent from both deterministic route features and reviewer evidence, so the pause itself is not used as a predictive feature. The report makes no latency comparison based on these durations
## Execution budget clarification
SWE solver containers have a 45-minute configured lifetime in both the original and corrected runner. Each shell command has a 90-second timeout with a five-second kill grace. Terminal agent and verifier timeouts come from each pinned task configuration. These environment limits apply in addition to the 150-call, USD 5 pre-query and 8,192-token response limits; this study does not measure unrestricted model performance.
The final primary billing audit also found one earlier DNA-assembly review ConnectTimeout with no successful response ledger. This is separate from the two host-sleep solver interruptions. Research accounting now discovers transport-only ledgers as well as billed-response ledgers. Review-cascade cost comparisons therefore carry three unmetered requests, while Luna-only comparisons carry two
## Lid-closed host sleep, September 16 UTC
During the remaining Anthropic Terminal controls, the host entered Clamshell Sleep on AC power at full battery and repeatedly returned to Maintenance Sleep. The workers survived; no whole attempt was restarted and no power setting changed. At the 06:12 UTC inspection, fix-ocaml-gc and model-extraction-relu-logits Opus trials remained active. The latter had a pending gateway request whose billing remains unresolved until the response or transport ledger completes. Evidence is retained in infrastructure_interruptions/20260916-clamshell-sleep. Host wall durations include these pauses and are not used as routing features or latency comparisons
## DNS recovery and exact Terminal image reuse, September 16 UTC
After host sleep, two active Anthropic solver attempts ended with gateway DNS ConnectError and 29 queued controls failed to resolve Docker Hub during environment setup. The runner exited. All 31 failed trial records and exported partial attempts were archived in infrastructure_interruptions/20260916-dns-outage. The two interrupted attempts incurred $1.35120725 in known inference cost and 20 requests with unknown billing. These costs are research overhead, not policy inference cost. Connectivity probes verified host DNS, authenticated gateway access and Docker DNS before resuming
Recovery validation found that eleven completed Anthropic controls had rebuilt different filesystem layers from the retained Luna/Sol images. Every mismatched control was archived and scheduled again, regardless of score. These controls cost $11.05990725, retained separately as research overhead. The 25 Luna/Sol image pairs match in filesystem layers and runtime image configuration. All retained primary attempts and fitted policies are unchanged
The Terminal environment now supports an explicit map of the exact saved Sol image for each final task. It verifies local image identity and filesystem layers before startup and verifies the started container image before agent inference. This replaces rebuilds from mutable base tags with the already-cached peer image. The map is terminal_image_pins.json. Task files, solver prompts, budgets, network isolation, and grading rules remain unchanged. The recovery schedules exactly 42 missing controls and preserves 58 valid Terminal results. No live trial was interrupted for this correction
The model-extraction-relu-logits Opus attempt completed normally after the model wrapper retried the previously pending ReadError. It is retained, with that request's unknown charge included in its cost bounds. The entire attempt was not rerun. Host wall durations remain unsuitable for latency comparisons
## Final completion
All 500 held-out model attempts and grades completed on September 17, 2026 UTC. The final audit checked 950 completed development/final attempts and 1,400 forecasts, including paired image identity, Git isolation, archived file hashes and recorded costs. Full reporting verified that primary Luna/Sol results and calibration diagnostics are identical to the earlier primary release. No fitting, policy selection or boundary change followed the final outcomes

View file

@ -0,0 +1,39 @@
# Selective routing trained on clean paired attempts
These optional profiles learn when GPT-5.6 Sol adds a successful solve over GPT-5.6 Luna. They use one existing classifier response and a deterministic local prediction, with no second classifier call. They replace the earlier ROI cookbook, whose benchmark results were invalidated and are not the source of these fits
Training uses 160 fresh paired tasks and selection uses 64 separate validation tasks across SWE-bench, Terminal-Bench, MBPP+ and LiveCodeBench. The fitted policies were frozen before the 125-task held-out evaluation. The complete 125-task Luna/Sol comparison is available in PRIMARY_FINDINGS.md and PRIMARY_REPORT.md. The additional Sonnet/Opus controls are complete in REPORT.md
`strong-success-retention` minimizes validation cost while preserving every Sol-only success and meeting Sol quality within each benchmark. `quality-first-under-sol-budget` maximizes validation solves under Sol's total inference cost and permits task swaps. These names describe validation objectives, not guarantees on new requests
The selected upfront profiles use the original qualitative cards. The empirical card variation and all validation selections remain available in the accompanying records. Post-attempt review cascades are evaluated separately because they require a completed Luna attempt, verification and potentially a fresh Sol attempt
Set `ROI_GATEWAY_BASE_URL`, `ROI_GATEWAY_API_KEY` and `ROI_LOCAL_MASTER_KEY`, then start this branch with `litellm --config cookbook/auto_router_selective_training/proxy.yaml`. Use the model alias `selective-v3-strong-success-retention` or `selective-v3-quality-first-under-sol-budget`
The example config describes the SWE-bench agent harness. Match its execution conditions, solver endpoints, efforts and judge prompt to your benchmark. Route once on the opening task and pin the selected model for a comparable whole-task run. Independent per-turn routing or continuing Sol from a Luna patch requires a separate evaluation
`selective_policy` replaces the existing threshold decision and cannot be combined with the older probability calibration. Its `target` identifies the score: `scalar_calibration` and `per_model` estimate a success-probability difference, `paired` estimates Sol-only probability minus Luna-only probability, `rescue` estimates Sol-only probability, and `benefit_per_dollar` divides the paired difference by predicted incremental inference cost. Threshold units depend on this target. Raw classifier probabilities remain available in diagnostics
Defaults are unchanged when `selective_policy` is omitted. The local implementation validates coefficient dimensions and rejects another classifier's feature schema. Runtime parity checks compare scores and routing choices with the training implementation
The twelve prespecified upfront family controls are also available in `diagnostic_profiles.json` and `diagnostic_proxy.yaml`. Start the latter config to benchmark aliases such as `selective-v3-diagnostic-original-upfront-per-model`. These controls were frozen before final inference and are included for reproduction and fresh benchmarking. A family with no eligible validation candidate uses a direct Sol deployment that bypasses the classifier. Review cascades still require a separate agent workflow
## Held-out results
These profiles were selected before these results. They are not refitted or selected again on the held-out tasks
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol |
|---|---:|---:|---:|---:|
| Luna only | 84/125 | ≥$2.6531 | ≤87.3% | +8 / -11 |
| Sol only | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| V2 baseline | 84/125 | ≥$17.0029 | ≤18.6% | +3 / -6 |
| V2 upfront, retention objective | 85/125 | $14.0036 | 33.0% | +2 / -4 |
| V2 upfront, quality objective | 86/125 | $18.1999 | 12.9% | +4 / -5 |
| Per-model control (exploratory) | 86/125 | $20.1971 | 3.3% | +0 / -1 |
Costs use recorded gateway bills. A ≥ cost and ≤ savings mark unmetered transport requests: cost is a lower bound and savings versus fully metered Sol is an upper bound. The JSON counts these requests per policy and task. They are not assumed free.
Costs include the classifier. These are task-pinned replays of fresh independent attempts, with one attempt per model per task. No guarantee of quality retention follows from a 125-task sample. The aggregate combines different benchmarks; inspect each benchmark before using a profile
The per-model control is an exploratory comparison among forty prespecified variants. None of the main selected profiles combines preservation of all Sol successes and lower cost across all 125 tasks. See PRIMARY_FINDINGS.md for the interpretation
![Observed quality and known inference cost](quality_cost.png)

View file

@ -0,0 +1,204 @@
# Clean selective-router evaluation
Policies were frozen before final solver outcomes. Upfront routing uses one classifier response; the post-attempt variants also use Luna output, optional public-example checks and a Luna review. These are task-level replays over matched independent attempts, not live per-turn routing or continuation from a cheaper model's patch
The configured original-card baselines receive the same model identity and execution conditions as the trained variants. Capability uses base_threshold=0.5 and threshold_step=0.1; V2 uses max_quality_gap=0.05. The empirical-card baselines change the card supplement while retaining those boundaries
Retention profiles were selected to preserve Sol successes at lower validation cost. Quality profiles were selected to maximize validation solves within the Sol budget. These names describe selection objectives, not guarantees about the held-out results. Exact policy IDs remain in final_results.json
Costs use recorded gateway bills. A ≥ cost and ≤ savings mark unmetered transport requests: cost is a lower bound and savings versus fully metered Sol is an upper bound. The JSON counts these requests per policy and task. They are not assumed free.
## Observed opportunity for routing
The Sol-only column counts the rescues a perfect decision would capture. The hindsight cascade pays for Luna on every task and adds Sol only for those rescues. Tasks where both attempts failed cannot be recovered by choosing between these two saved outputs. This is a bound on this replay, not a guarantee about future attempts
| Benchmark | Both solve | Luna only | Sol only | Both fail | Hindsight cascade cost |
|---|---:|---:|---:|---:|---:|
| all | 76 | 8 | 11 | 30 | ≥$5.2653 |
| lcb | 11 | 2 | 3 | 9 | $0.3657 |
| mbpp | 35 | 3 | 2 | 10 | $0.0361 |
| swe | 16 | 0 | 2 | 7 | $1.5478 |
| terminal | 14 | 3 | 4 | 4 | ≥$3.3157 |
## all
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol |
|---|---:|---:|---:|---:|
| Luna only | 84/125 | ≥$2.6531 | ≤87.3% | +8 / -11 |
| Sol only | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| Sonnet 5 only | 81/125 | ≥$37.9961 | ≤-81.8% | +8 / -14 |
| Opus 5 only | 99/125 | ≥$46.0550 | ≤-120.4% | +19 / -7 |
| Capability upfront, retention objective | 86/125 | $13.8156 | 33.9% | +1 / -2 |
| Capability upfront, quality objective | 88/125 | $18.5237 | 11.4% | +5 / -4 |
| V2 upfront, retention objective | 85/125 | $14.0036 | 33.0% | +2 / -4 |
| V2 upfront, quality objective | 86/125 | $18.1999 | 12.9% | +4 / -5 |
| Capability baseline | 85/125 | $6.7642 | 67.6% | +8 / -10 |
| Capability with learned cards only | 84/125 | $7.2496 | 65.3% | +7 / -10 |
| V2 baseline | 84/125 | ≥$17.0029 | ≤18.6% | +3 / -6 |
| V2 with learned cards only | 85/125 | ≥$14.6487 | ≤29.9% | +6 / -8 |
| Capability review, retention objective | 81/125 | ≥$7.5830 | ≤63.7% | +0 / -6 |
| Capability review, quality objective | 87/125 | ≥$10.1853 | ≤51.3% | +7 / -7 |
| V2 review, retention objective | 88/125 | ≥$20.9754 | ≤-0.4% | +1 / -0 |
| V2 review, quality objective | 88/125 | ≥$8.6493 | ≤58.6% | +8 / -7 |
| Hindsight direct routing (not deployable) | 95/125 | ≥$4.5190 | ≤78.4% | +8 / -0 |
| Hindsight Luna-first routing (not deployable) | 95/125 | ≥$5.2653 | ≤74.8% | +8 / -0 |
## lcb
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol |
|---|---:|---:|---:|---:|
| Luna only | 13/25 | $0.1165 | 92.4% | +2 / -3 |
| Sol only | 14/25 | $1.5299 | 0.0% | +0 / -0 |
| Sonnet 5 only | 9/25 | $1.3790 | 9.9% | +1 / -6 |
| Opus 5 only | 14/25 | $2.3849 | -55.9% | +2 / -2 |
| Capability upfront, retention objective | 14/25 | $1.5476 | -1.2% | +0 / -0 |
| Capability upfront, quality objective | 14/25 | $0.9200 | 39.9% | +1 / -1 |
| V2 upfront, retention objective | 14/25 | $1.3923 | 9.0% | +0 / -0 |
| V2 upfront, quality objective | 12/25 | $1.1104 | 27.4% | +0 / -2 |
| Capability baseline | 13/25 | $0.2918 | 80.9% | +2 / -3 |
| Capability with learned cards only | 12/25 | $0.4860 | 68.2% | +1 / -3 |
| V2 baseline | 12/25 | $0.9924 | 35.1% | +0 / -2 |
| V2 with learned cards only | 13/25 | $0.4799 | 68.6% | +2 / -3 |
| Capability review, retention objective | 14/25 | $1.6846 | -10.1% | +0 / -0 |
| Capability review, quality objective | 14/25 | $1.3257 | 13.4% | +2 / -2 |
| V2 review, retention objective | 15/25 | $1.6076 | -5.1% | +1 / -0 |
| V2 review, quality objective | 14/25 | $1.0584 | 30.8% | +2 / -2 |
| Luna with public-test escalation | 15/25 | $1.3645 | 10.8% | +2 / -1 |
| Hindsight direct routing (not deployable) | 16/25 | $0.3436 | 77.5% | +2 / -0 |
| Hindsight Luna-first routing (not deployable) | 16/25 | $0.3657 | 76.1% | +2 / -0 |
## mbpp
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol |
|---|---:|---:|---:|---:|
| Luna only | 38/50 | $0.0174 | 91.6% | +3 / -2 |
| Sol only | 37/50 | $0.2064 | 0.0% | +0 / -0 |
| Sonnet 5 only | 36/50 | $0.0988 | 52.1% | +2 / -3 |
| Opus 5 only | 42/50 | $0.2249 | -9.0% | +7 / -2 |
| Capability upfront, retention objective | 37/50 | $0.2180 | -5.6% | +0 / -0 |
| Capability upfront, quality objective | 38/50 | $0.0444 | 78.5% | +3 / -2 |
| V2 upfront, retention objective | 37/50 | $0.2298 | -11.3% | +0 / -0 |
| V2 upfront, quality objective | 38/50 | $0.0465 | 77.4% | +3 / -2 |
| Capability baseline | 38/50 | $0.0290 | 85.9% | +3 / -2 |
| Capability with learned cards only | 38/50 | $0.0284 | 86.2% | +3 / -2 |
| V2 baseline | 38/50 | $0.0562 | 72.8% | +3 / -2 |
| V2 with learned cards only | 38/50 | $0.0396 | 80.8% | +3 / -2 |
| Capability review, retention objective | 37/50 | $0.2503 | -21.3% | +0 / -0 |
| Capability review, quality objective | 38/50 | $0.0434 | 79.0% | +3 / -2 |
| V2 review, retention objective | 37/50 | $0.2441 | -18.3% | +0 / -0 |
| V2 review, quality objective | 38/50 | $0.0488 | 76.4% | +3 / -2 |
| Luna with public-test escalation | 38/50 | $0.0240 | 88.4% | +2 / -1 |
| Hindsight direct routing (not deployable) | 40/50 | $0.0353 | 82.9% | +3 / -0 |
| Hindsight Luna-first routing (not deployable) | 40/50 | $0.0361 | 82.5% | +3 / -0 |
## swe
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol |
|---|---:|---:|---:|---:|
| Luna only | 16/25 | $0.6371 | 94.1% | +0 / -2 |
| Sol only | 18/25 | $10.7132 | 0.0% | +0 / -0 |
| Sonnet 5 only | 19/25 | $11.0537 | -3.2% | +2 / -1 |
| Opus 5 only | 22/25 | $10.5296 | 1.7% | +4 / -0 |
| Capability upfront, retention objective | 18/25 | $6.2493 | 41.7% | +0 / -0 |
| Capability upfront, quality objective | 18/25 | $9.7546 | 8.9% | +0 / -0 |
| V2 upfront, retention objective | 16/25 | $5.3537 | 50.0% | +0 / -2 |
| V2 upfront, quality objective | 17/25 | $8.8718 | 17.2% | +0 / -1 |
| Capability baseline | 16/25 | $1.3893 | 87.0% | +0 / -2 |
| Capability with learned cards only | 16/25 | $1.6826 | 84.3% | +0 / -2 |
| V2 baseline | 17/25 | $9.1304 | 14.8% | +0 / -1 |
| V2 with learned cards only | 17/25 | $7.4181 | 30.8% | +0 / -1 |
| Capability review, retention objective | 16/25 | $2.4752 | 76.9% | +0 / -2 |
| Capability review, quality objective | 16/25 | $0.9753 | 90.9% | +0 / -2 |
| V2 review, retention objective | 18/25 | $9.1263 | 14.8% | +0 / -0 |
| V2 review, quality objective | 16/25 | $0.9775 | 90.9% | +0 / -2 |
| Hindsight direct routing (not deployable) | 18/25 | $1.4342 | 86.6% | +0 / -0 |
| Hindsight Luna-first routing (not deployable) | 18/25 | $1.5478 | 85.6% | +0 / -0 |
## terminal
| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol |
|---|---:|---:|---:|---:|
| Luna only | 17/25 | ≥$1.8822 | ≤77.7% | +3 / -4 |
| Sol only | 18/25 | $8.4468 | 0.0% | +0 / -0 |
| Sonnet 5 only | 17/25 | ≥$25.4647 | ≤-201.5% | +3 / -4 |
| Opus 5 only | 21/25 | ≥$32.9157 | ≤-289.7% | +6 / -3 |
| Capability upfront, retention objective | 17/25 | $5.8008 | 31.3% | +1 / -2 |
| Capability upfront, quality objective | 18/25 | $7.8047 | 7.6% | +1 / -1 |
| V2 upfront, retention objective | 18/25 | $7.0279 | 16.8% | +2 / -2 |
| V2 upfront, quality objective | 19/25 | $8.1712 | 3.3% | +1 / -0 |
| Capability baseline | 18/25 | $5.0541 | 40.2% | +3 / -3 |
| Capability with learned cards only | 18/25 | $5.0526 | 40.2% | +3 / -3 |
| V2 baseline | 17/25 | ≥$6.8238 | ≤19.2% | +0 / -1 |
| V2 with learned cards only | 17/25 | ≥$6.7112 | ≤20.5% | +1 / -2 |
| Capability review, retention objective | 14/25 | ≥$3.1729 | ≤62.4% | +0 / -4 |
| Capability review, quality objective | 19/25 | ≥$7.8410 | ≤7.2% | +2 / -1 |
| V2 review, retention objective | 18/25 | ≥$9.9975 | ≤-18.4% | +0 / -0 |
| V2 review, quality objective | 20/25 | ≥$6.5646 | ≤22.3% | +3 / -1 |
| Hindsight direct routing (not deployable) | 21/25 | ≥$2.7059 | ≤68.0% | +3 / -0 |
| Hindsight Luna-first routing (not deployable) | 21/25 | ≥$3.3157 | ≤60.7% | +3 / -0 |
## Frozen family comparisons
Each row was selected on development data before final inference. These are diagnostic comparisons, not a new winner selection on the held-out tasks. An always-Sol fallback means that no candidate in that requested family met the validation criterion; it bypasses the classifier, Luna attempt and reviewer
| Classifier | Requested stage | Cards | Fitted target | Boundary | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol |
|---|---|---|---|---:|---:|---:|---:|---:|
| cap | upfront | empirical | benefit_per_dollar | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | upfront | empirical | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | upfront | empirical | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | upfront | empirical | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | upfront | empirical | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | upfront | empirical | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | review | empirical | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | review | empirical | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | review | empirical | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | upfront | original | benefit_per_dollar | 1 | 86/125 | $13.8156 | 33.9% | +1 / -2 |
| cap | upfront | original | paired | -0.02 | 88/125 | $20.8563 | 0.2% | +1 / -0 |
| cap | upfront | original | per_model | -0.05 | 88/125 | $20.3797 | 2.5% | +1 / -0 |
| cap | upfront | original | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | upfront | original | rescue | 0.03 | 88/125 | $20.7799 | 0.6% | +1 / -0 |
| cap | upfront | original | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | review | original | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | review | original | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | review | original | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | upfront | empirical | benefit_per_dollar | 0.5 | 87/125 | $19.2534 | 7.9% | +2 / -2 |
| v2 | upfront | empirical | paired | 0.01 | 85/125 | $19.7030 | 5.7% | +1 / -3 |
| v2 | upfront | empirical | per_model | 0.01 | 86/125 | $19.7351 | 5.6% | +1 / -2 |
| v2 | upfront | empirical | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | upfront | empirical | rescue | 0.05 | 86/125 | $18.0048 | 13.8% | +2 / -3 |
| v2 | upfront | empirical | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | review | empirical | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | review | empirical | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | review | empirical | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | upfront | original | benefit_per_dollar | 1 | 85/125 | $14.0036 | 33.0% | +2 / -4 |
| v2 | upfront | original | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | upfront | original | per_model | 0 | 86/125 | $20.1971 | 3.3% | +0 / -1 |
| v2 | upfront | original | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | upfront | original | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | upfront | original | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| v2 | review | original | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 |
| cap | review | empirical | benefit_per_dollar | 0.1 | 82/125 | ≥$16.8075 | ≤19.6% | +0 / -5 |
| cap | review | original | benefit_per_dollar | 0.25 | 81/125 | ≥$7.5830 | ≤63.7% | +0 / -6 |
| v2 | review | empirical | benefit_per_dollar | 0.25 | 83/125 | ≥$16.3243 | ≤21.9% | +0 / -4 |
| v2 | review | original | benefit_per_dollar | 0.1 | 85/125 | ≥$19.8241 | ≤5.1% | +0 / -2 |
| v2 | review | original | per_model | -0.02 | 88/125 | ≥$20.9754 | ≤-0.4% | +1 / -0 |
| v2 | review | original | rescue | 0.02 | 86/125 | ≥$23.4088 | ≤-12.0% | +0 / -1 |
## Response budget effects
Every solver uses the same 8,192-token response cap and high reasoning effort. Length stops that produce neither a tool call nor visible text still consume inference cost. They are retained as model behavior under this budget, not retried as infrastructure failures. These counts describe completed final attempts across all four benchmarks
| Model | Responses | Length stops | Length stops without action or text | Cost of those empty length stops |
|---|---:|---:|---:|---:|
| gpt-5.6-luna | 1341 | 50 | 42 | $0.4217 |
| gpt-5.6-sol | 819 | 7 | 4 | $0.6676 |
| claude-sonnet-5 | 1703 | 124 | 116 | $10.3656 |
| claude-opus-5 | 870 | 87 | 83 | $17.9127 |
## Experiment spending
Recorded study inference cost is $225.5557. This includes training, validation, final solver/judge/reviewer calls, superseded development forecasts, archived infrastructure-interrupted attempts, superseded Anthropic agentic controls, controls with mismatched Terminal images, and adapter round-trip probes. It is separate from the deployed-policy costs above. Unknown transport billing, setup, live-proxy QA, local Docker and CPU expenses are not included. See research_costs.json for the breakdown
## Interpretation limits
The oracle is a hindsight reference using known costs. Cost intervals use recorded bills and exclude unmetered requests. A small sample with one attempt per model cannot establish the same routing advantage across workloads or repeated runs. Paired bootstrap intervals and exact discordant-pair tests in the JSON are exploratory; policy selection searched many development candidates. Inference costs include classifier/reviewer calls and discarded Luna attempts on escalation. Local checker CPU time and experiment training/transport overhead are separate. Terminal-Bench and SWE-bench use documented offline/native-image eligibility adaptations

View file

@ -0,0 +1,49 @@
# Training and selection record
The learning target is model complementarity: route cheaply when Luna can do the job, and use Sol when its extra cost is likely to produce an additional solve. Better marginal probability accuracy alone is not the selection objective
## Paired examples
| Split | Benchmark | Tasks | Both fail | Sol only | Luna only | Both solve |
|---|---|---:|---:|---:|---:|---:|
| train | lcb | 50 | 14 | 9 | 3 | 24 |
| train | mbpp | 80 | 7 | 2 | 3 | 68 |
| train | swe | 25 | 5 | 1 | 0 | 19 |
| train | terminal | 5 | 0 | 1 | 0 | 4 |
| validation | lcb | 20 | 2 | 6 | 2 | 10 |
| validation | mbpp | 29 | 4 | 2 | 2 | 21 |
| validation | swe | 10 | 1 | 1 | 1 | 7 |
| validation | terminal | 5 | 2 | 0 | 1 | 2 |
There are 13 Sol-only examples in fitting and 9 in validation. Only two fitting examples are agentic rescues, one SWE-bench and one Terminal-Bench. Terminal validation has no Sol-only example. That limits evidence about rescue detection on unseen terminal tasks; zero validation losses there cannot establish rescue recall
## Variations
One training-only synthesis produces the empirical card supplement. It uses measured outcomes and fallible classifier/reviewer descriptions, without task identities, solutions or validation-driven rewrites. Original cards are a separate control
The deterministic head families are raw threshold tuning, scalar probability calibration, task-dependent per-model calibration, four-way paired outcomes, Sol-only rescue probability and paired benefit per predicted incremental dollar. Regularization and fixed threshold grids are selected using validation only. Upfront inputs come from one classifier response. Post-attempt heads additionally use Luna output, a fallible Luna review and task-supplied public checks where available
The strong-success-retention objective minimizes validation cost while losing no Sol-only successes and preserving Sol aggregate quality in each benchmark. The quality-first-under-sol-budget objective maximizes validation solves under Sol total cost, permitting task swaps. Always-Sol is an explicit fallback. None of these objective names guarantees performance on unseen tasks
## Frozen selected profiles
| Classifier | Stage | Objective | Cards | Family | Threshold | Validation solved | Validation cost | Sol-only losses |
|---|---|---|---|---|---:|---:|---:|---:|
| cap | upfront | strong_success_retention | original | benefit_per_dollar | 1.0 | 50/64 | $5.8544 | 0 |
| cap | upfront | quality_first_under_sol_budget | original | scalar_calibration | 0.1 | 52/64 | $5.4692 | 2 |
| cap | review | strong_success_retention | original | benefit_per_dollar | 0.25 | 49/64 | $5.6530 | 0 |
| cap | review | quality_first_under_sol_budget | empirical | paired | 0.075 | 53/64 | $3.7201 | 2 |
| cap | either | strong_success_retention | original | benefit_per_dollar | 0.25 | 49/64 | $5.6530 | 0 |
| cap | either | quality_first_under_sol_budget | empirical | paired | 0.075 | 53/64 | $3.7201 | 2 |
| v2 | upfront | strong_success_retention | original | benefit_per_dollar | 1.0 | 51/64 | $5.4640 | 0 |
| v2 | upfront | quality_first_under_sol_budget | original | scalar_calibration | 0.075 | 53/64 | $5.4524 | 2 |
| v2 | review | strong_success_retention | original | per_model | -0.02 | 51/64 | $6.7043 | 0 |
| v2 | review | quality_first_under_sol_budget | empirical | per_model | 0.15 | 53/64 | $3.3074 | 2 |
| v2 | either | strong_success_retention | original | benefit_per_dollar | 1.0 | 51/64 | $5.4640 | 0 |
| v2 | either | quality_first_under_sol_budget | empirical | per_model | 0.15 | 53/64 | $3.3074 | 2 |
These are development results. Compare held-out results in REPORT.md before interpreting them as an improvement. Both selected upfront classifier profiles keep the original card and use fitted local decisions. Empirical cards remain a measured variation and are selected by some review cascades
There are 1,892 policy candidates, 118 serialized models, 12 selected profiles and 40 prespecified family ablations. An independent refit reproduced every coefficient, candidate statistic and selection exactly. The selection hash and source/data hashes are recorded in selection_frozen.json; reproduction evidence is in training_reproduction.json
The threshold unit depends on the target. Marginal and paired gaps are probability differences, rescue is a Sol-only probability, and benefit_per_dollar is expected additional success per predicted dollar. A threshold of 0.05 is not interchangeable across these families

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,176 @@
{
"teacher_model": "openai/gpt-5.6-sol",
"teacher_cost": 0.36054700000000006,
"training_ids": [
"Mbpp_69",
"Mbpp_603",
"Mbpp_171",
"Mbpp_265",
"Mbpp_635",
"Mbpp_90",
"Mbpp_573",
"Mbpp_418",
"Mbpp_731",
"Mbpp_294",
"Mbpp_66",
"Mbpp_744",
"Mbpp_733",
"Mbpp_598",
"Mbpp_763",
"Mbpp_59",
"Mbpp_808",
"Mbpp_805",
"Mbpp_578",
"Mbpp_456",
"Mbpp_72",
"Mbpp_785",
"Mbpp_459",
"Mbpp_123",
"Mbpp_16",
"Mbpp_285",
"Mbpp_424",
"Mbpp_723",
"Mbpp_58",
"Mbpp_428",
"Mbpp_84",
"Mbpp_591",
"Mbpp_470",
"Mbpp_581",
"Mbpp_68",
"Mbpp_95",
"Mbpp_222",
"Mbpp_14",
"Mbpp_94",
"Mbpp_77",
"Mbpp_261",
"Mbpp_457",
"Mbpp_410",
"Mbpp_605",
"Mbpp_562",
"Mbpp_736",
"Mbpp_57",
"Mbpp_602",
"Mbpp_451",
"Mbpp_788",
"Mbpp_425",
"Mbpp_450",
"Mbpp_395",
"Mbpp_252",
"Mbpp_580",
"Mbpp_120",
"Mbpp_414",
"Mbpp_11",
"Mbpp_775",
"Mbpp_585",
"Mbpp_413",
"Mbpp_607",
"Mbpp_436",
"Mbpp_162",
"Mbpp_299",
"Mbpp_274",
"Mbpp_79",
"Mbpp_782",
"Mbpp_742",
"Mbpp_279",
"Mbpp_127",
"Mbpp_623",
"Mbpp_300",
"Mbpp_62",
"Mbpp_7",
"Mbpp_101",
"Mbpp_787",
"Mbpp_606",
"Mbpp_477",
"Mbpp_276",
"lcb_atcoder_arc190_d",
"lcb_atcoder_arc190_c",
"lcb_leetcode_3779",
"lcb_leetcode_3697",
"lcb_atcoder_abc394_c",
"lcb_atcoder_abc388_e",
"lcb_atcoder_abc390_g",
"lcb_atcoder_arc191_a",
"lcb_leetcode_3763",
"lcb_atcoder_abc393_e",
"lcb_atcoder_abc387_f",
"lcb_leetcode_3733",
"lcb_atcoder_abc388_g",
"lcb_atcoder_abc391_g",
"lcb_leetcode_3677",
"lcb_atcoder_abc388_f",
"lcb_atcoder_abc392_f",
"lcb_atcoder_arc190_a",
"lcb_atcoder_abc389_d",
"lcb_leetcode_3739",
"lcb_leetcode_3748",
"lcb_atcoder_arc192_a",
"lcb_atcoder_abc390_f",
"lcb_atcoder_arc191_c",
"lcb_atcoder_abc388_c",
"lcb_atcoder_arc191_d",
"lcb_leetcode_3674",
"lcb_atcoder_abc393_f",
"lcb_leetcode_3701",
"lcb_leetcode_3725",
"lcb_atcoder_abc394_e",
"lcb_atcoder_abc392_g",
"lcb_leetcode_3771",
"lcb_leetcode_3687",
"lcb_leetcode_3692",
"lcb_atcoder_abc391_f",
"lcb_atcoder_abc392_c",
"lcb_atcoder_abc389_e",
"lcb_atcoder_abc389_f",
"lcb_leetcode_3762",
"lcb_leetcode_3751",
"lcb_atcoder_abc394_d",
"lcb_leetcode_3754",
"lcb_leetcode_3696",
"lcb_leetcode_3737",
"lcb_atcoder_arc192_e",
"lcb_atcoder_abc390_e",
"lcb_leetcode_3721",
"lcb_atcoder_arc192_d",
"lcb_atcoder_abc387_c",
"django__django-13128",
"django__django-14434",
"django__django-15315",
"django__django-15161",
"django__django-16100",
"django__django-14580",
"django__django-16333",
"django__django-12193",
"django__django-15916",
"django__django-15957",
"django__django-15973",
"django__django-16877",
"django__django-14351",
"django__django-16801",
"django__django-12308",
"django__django-13410",
"django__django-13028",
"django__django-14170",
"django__django-15563",
"django__django-11532",
"django__django-11066",
"django__django-10973",
"django__django-12050",
"django__django-14376",
"django__django-15278",
"write-compressor",
"modernize-scientific-stack",
"extract-elf",
"fix-code-vulnerability",
"prove-plus-comm"
],
"learned_guidance": {
"capability_guidance": "For single-response, no-tool execution, Luna is well supported on localized tasks with clear interfaces: direct collection/string transformations, standard formulas, sorting/searching, regex use, and familiar recurrences. Luna also succeeds on many multistep tasks when the crux reduces cleanly to a recognizable invariant\u2014subset or state-compressed DP, greedy allocation, monotonic-stack contributions, digit counting, combinatorial aggregation, or tree/path window maintenance. Strong routing signals are bounded state, a standard proof pattern, explicit complexity targets, and examples that disambiguate indexing or geometry.\n\nWith a local shell and tests, Luna is effective on narrowly traceable framework changes involving argument flow, database aliases, serialization/import generation, transaction boundaries, signal registration, and compatibility-preserving API edits. Confidence should rise when focused tests exercise the changed path and broader nearby suites pass.\n\nNegative evidence remains: Luna sometimes emits no usable answer on broad or coupled algorithm discovery, and can pass public examples while choosing the wrong hidden semantics or missing required artifacts. Public-example success and confident self-review alone are therefore weak verification signals.",
"comparative_guidance": "Prefer Luna when the observable crux is a standard reduction with a compact correctness argument, even if multistep. Luna-only successes include careful recurrence/index handling, recursive preservation of nested structure, parity-based combinatorics, offline sequence queries with value caps, and exact discrete marginal-cost allocation. In tool-using work, Luna is a reasonable default for localized patches whose call path and regression surface can be directly inspected and tested.\n\nPrefer Sol when the task exposes semantic choices not distinguished by the example, or when implementation robustness dominates the core idea. Sol-only rescues covered true-versus-floor arithmetic semantics, cascading rewrite simulation, duplicate-sensitive greedy pairing, cycle-aware functional-graph DP, sophisticated range-query pairing, cyclic boundary characterization, lexicographic DP reconstruction, pair-state graph search, movement-accounting feasibility checks, and prime-factor state aggregation. In tool-using work, Sol also rescued subtle transform semantics and an underspecified binary-memory extraction task.\n\nDo not route to Sol merely because a task is multistep: Luna succeeded on many coupled combinatorial and DP problems that Sol did not. Conversely, repeated both-fail outcomes on novel global data structures, compressed huge-domain reachability, high-performance convolution, and difficult optimization reconstruction indicate algorithmic difficulty rather than demonstrated Sol advantage. If focused or public checks fail, favor the model whose mechanism directly addresses that failure; a passing single example should not override unresolved semantic branches.",
"limitations": "All observations come from a fixed training harness and do not imply general superiority. Measured labels govern these runs, but one noisy attempt and several conflicts with self-review or partial checks make isolated outcomes unreliable. Sol-only and Luna-only patterns are sparse for tool-using work. Verification was often absent or limited to supplied examples, so hidden-edge robustness and cross-environment behavior remain uncertain."
},
"counts": [
"single-response: 130 paired training attempts; Luna solved 98, Sol solved 103; Sol-only successes 11, Luna-only successes 6.",
"agentic: 30 paired training attempts; Luna solved 23, Sol solved 25; Sol-only successes 2, Luna-only successes 0."
],
"selection": "Single prespecified synthesis from training-only data, no validation-driven rewrite"
}

View file

@ -0,0 +1,62 @@
{
"train": {
"lcb": {
"tasks": 50,
"both_fail": 14,
"sol_only": 9,
"luna_only": 3,
"both_solve": 24
},
"mbpp": {
"tasks": 80,
"both_fail": 7,
"sol_only": 2,
"luna_only": 3,
"both_solve": 68
},
"swe": {
"tasks": 25,
"both_fail": 5,
"sol_only": 1,
"luna_only": 0,
"both_solve": 19
},
"terminal": {
"tasks": 5,
"both_fail": 0,
"sol_only": 1,
"luna_only": 0,
"both_solve": 4
}
},
"validation": {
"lcb": {
"tasks": 20,
"both_fail": 2,
"sol_only": 6,
"luna_only": 2,
"both_solve": 10
},
"mbpp": {
"tasks": 29,
"both_fail": 4,
"sol_only": 2,
"luna_only": 2,
"both_solve": 21
},
"swe": {
"tasks": 10,
"both_fail": 1,
"sol_only": 1,
"luna_only": 1,
"both_solve": 7
},
"terminal": {
"tasks": 5,
"both_fail": 2,
"sol_only": 0,
"luna_only": 1,
"both_solve": 2
}
}
}

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,8 @@
{
"classifier": "v2",
"profiles": 12,
"classifier_decisions_verified": 750,
"direct_sol_decisions_verified": 750,
"matches_frozen_routes": true,
"inputs": "Frozen forecasts and route plans only; no grades or outcome-based selection"
}

File diff suppressed because one or more lines are too long

View file

@ -0,0 +1,315 @@
{
"scope": "Execution diagnostics for completed attempts, without reading grades. A length stop is a model output-budget event, not an infrastructure failure. All recorded responses remain billed.",
"groups": [
{
"split": "test",
"benchmark": "lcb",
"model": "anthropic/claude-opus-5",
"attempts": 25,
"responses": 25,
"response_cost": 2.384855,
"token_limit_responses": 7,
"token_limit_response_cost": 1.4849325,
"token_limit_without_action_or_text": 7,
"token_limit_without_action_or_text_cost": 1.4849325
},
{
"split": "test",
"benchmark": "lcb",
"model": "anthropic/claude-sonnet-5",
"attempts": 25,
"responses": 25,
"response_cost": 1.3789665000000002,
"token_limit_responses": 14,
"token_limit_response_cost": 1.1779045000000001,
"token_limit_without_action_or_text": 14,
"token_limit_without_action_or_text_cost": 1.1779045000000001
},
{
"split": "test",
"benchmark": "lcb",
"model": "openai/gpt-5.6-luna",
"attempts": 25,
"responses": 25,
"response_cost": 0.1164804,
"token_limit_responses": 7,
"token_limit_response_cost": 0.06988079999999999,
"token_limit_without_action_or_text": 7,
"token_limit_without_action_or_text_cost": 0.06988079999999999
},
{
"split": "test",
"benchmark": "lcb",
"model": "openai/gpt-5.6-sol",
"attempts": 25,
"responses": 25,
"response_cost": 1.529944,
"token_limit_responses": 4,
"token_limit_response_cost": 0.667568,
"token_limit_without_action_or_text": 4,
"token_limit_without_action_or_text_cost": 0.667568
},
{
"split": "test",
"benchmark": "mbpp",
"model": "anthropic/claude-opus-5",
"attempts": 50,
"responses": 50,
"response_cost": 0.224865
},
{
"split": "test",
"benchmark": "mbpp",
"model": "anthropic/claude-sonnet-5",
"attempts": 50,
"responses": 50,
"response_cost": 0.09884600000000003
},
{
"split": "test",
"benchmark": "mbpp",
"model": "openai/gpt-5.6-luna",
"attempts": 50,
"responses": 50,
"response_cost": 0.017396
},
{
"split": "test",
"benchmark": "mbpp",
"model": "openai/gpt-5.6-sol",
"attempts": 50,
"responses": 50,
"response_cost": 0.20637600000000006
},
{
"split": "test",
"benchmark": "swe",
"model": "anthropic/claude-opus-5",
"attempts": 25,
"responses": 381,
"response_cost": 10.529565750000002
},
{
"split": "test",
"benchmark": "swe",
"model": "anthropic/claude-sonnet-5",
"attempts": 25,
"responses": 833,
"response_cost": 11.053656000000016,
"token_limit_responses": 4,
"token_limit_response_cost": 0.4144736,
"token_limit_without_action_or_text": 4,
"token_limit_without_action_or_text_cost": 0.4144736
},
{
"split": "test",
"benchmark": "swe",
"model": "openai/gpt-5.6-luna",
"attempts": 25,
"responses": 427,
"response_cost": 0.6371064399999993
},
{
"split": "test",
"benchmark": "swe",
"model": "openai/gpt-5.6-sol",
"attempts": 25,
"responses": 403,
"response_cost": 10.713241600000002
},
{
"split": "test",
"benchmark": "terminal",
"model": "anthropic/claude-opus-5",
"attempts": 25,
"responses": 414,
"response_cost": 32.915727249999996,
"token_limit_responses": 80,
"token_limit_response_cost": 17.381599,
"token_limit_without_action_or_text": 76,
"token_limit_without_action_or_text_cost": 16.427720750000002
},
{
"split": "test",
"benchmark": "terminal",
"model": "anthropic/claude-sonnet-5",
"attempts": 25,
"responses": 795,
"response_cost": 25.464663800000025,
"token_limit_responses": 106,
"token_limit_response_cost": 9.517311799999998,
"token_limit_without_action_or_text": 98,
"token_limit_without_action_or_text_cost": 8.773203099999998
},
{
"split": "test",
"benchmark": "terminal",
"model": "openai/gpt-5.6-luna",
"attempts": 25,
"responses": 839,
"response_cost": 1.8821569099999986,
"token_limit_responses": 43,
"token_limit_response_cost": 0.43733629999999996,
"token_limit_without_action_or_text": 35,
"token_limit_without_action_or_text_cost": 0.35178958
},
{
"split": "test",
"benchmark": "terminal",
"model": "openai/gpt-5.6-sol",
"attempts": 25,
"responses": 341,
"response_cost": 8.446844399999998,
"token_limit_responses": 3,
"token_limit_response_cost": 0.5060674000000001
},
{
"split": "train",
"benchmark": "lcb",
"model": "openai/gpt-5.6-luna",
"attempts": 50,
"responses": 50,
"response_cost": 0.2573764,
"token_limit_responses": 16,
"token_limit_response_cost": 0.1596568,
"token_limit_without_action_or_text": 16,
"token_limit_without_action_or_text_cost": 0.1596568
},
{
"split": "train",
"benchmark": "lcb",
"model": "openai/gpt-5.6-sol",
"attempts": 50,
"responses": 50,
"response_cost": 2.630148,
"token_limit_responses": 5,
"token_limit_response_cost": 0.836052,
"token_limit_without_action_or_text": 5,
"token_limit_without_action_or_text_cost": 0.836052
},
{
"split": "train",
"benchmark": "mbpp",
"model": "openai/gpt-5.6-luna",
"attempts": 80,
"responses": 80,
"response_cost": 0.0218284
},
{
"split": "train",
"benchmark": "mbpp",
"model": "openai/gpt-5.6-sol",
"attempts": 80,
"responses": 80,
"response_cost": 0.289156
},
{
"split": "train",
"benchmark": "swe",
"model": "openai/gpt-5.6-luna",
"attempts": 25,
"responses": 398,
"response_cost": 0.5176403399999997
},
{
"split": "train",
"benchmark": "swe",
"model": "openai/gpt-5.6-sol",
"attempts": 25,
"responses": 346,
"response_cost": 7.403714199999999
},
{
"split": "train",
"benchmark": "terminal",
"model": "openai/gpt-5.6-luna",
"attempts": 5,
"responses": 87,
"response_cost": 0.12343140999999999
},
{
"split": "train",
"benchmark": "terminal",
"model": "openai/gpt-5.6-sol",
"attempts": 5,
"responses": 34,
"response_cost": 0.7672414000000001
},
{
"split": "validation",
"benchmark": "lcb",
"model": "openai/gpt-5.6-luna",
"attempts": 20,
"responses": 20,
"response_cost": 0.0921004,
"token_limit_responses": 6,
"token_limit_response_cost": 0.0598586,
"token_limit_without_action_or_text": 6,
"token_limit_without_action_or_text_cost": 0.0598586
},
{
"split": "validation",
"benchmark": "lcb",
"model": "openai/gpt-5.6-sol",
"attempts": 20,
"responses": 20,
"response_cost": 1.0552880000000002,
"token_limit_responses": 1,
"token_limit_response_cost": 0.166696,
"token_limit_without_action_or_text": 1,
"token_limit_without_action_or_text_cost": 0.166696
},
{
"split": "validation",
"benchmark": "mbpp",
"model": "openai/gpt-5.6-luna",
"attempts": 30,
"responses": 30,
"response_cost": 0.010364400000000001
},
{
"split": "validation",
"benchmark": "mbpp",
"model": "openai/gpt-5.6-sol",
"attempts": 30,
"responses": 30,
"response_cost": 0.14699600000000002
},
{
"split": "validation",
"benchmark": "swe",
"model": "openai/gpt-5.6-luna",
"attempts": 10,
"responses": 149,
"response_cost": 0.2096464700000001
},
{
"split": "validation",
"benchmark": "swe",
"model": "openai/gpt-5.6-sol",
"attempts": 10,
"responses": 141,
"response_cost": 3.702219799999998
},
{
"split": "validation",
"benchmark": "terminal",
"model": "openai/gpt-5.6-luna",
"attempts": 5,
"responses": 66,
"response_cost": 0.10991243999999999,
"token_limit_responses": 1,
"token_limit_response_cost": 0.009907
},
{
"split": "validation",
"benchmark": "terminal",
"model": "openai/gpt-5.6-sol",
"attempts": 5,
"responses": 63,
"response_cost": 2.5312083999999997,
"token_limit_responses": 6,
"token_limit_response_cost": 0.9978314000000001
}
]
}

View file

@ -0,0 +1,18 @@
{
"selection_sha256": "1a7c9241c72c299c8b27fd8171881f94fda069fb7191ac6bc3a2eab9de811853",
"models_sha256": "03c3f39894e155d6024053e65858e56f67ff2f0996d441b8a3375f2702ee8be2",
"training_tasks": 160,
"validation_tasks": 64,
"held_out_tasks": 125,
"held_out_status": "complete",
"current_model_pair": [
"openai/gpt-5.6-luna",
"openai/gpt-5.6-sol"
],
"judge_effort": "low",
"solver_effort": "high",
"diagnostic_upfront_profiles": 12,
"diagnostic_profiles_sha256": "ae456bc65e0873ab22dd9009975db61d327e115a6fdd49266c1db313a8bc6b4f",
"primary_results_sha256": "e91ec5578d496ad37c1e8cf5760882042f4b7768bd3c2ff5d65d92f19df7a12c",
"final_results_sha256": "b724f5607b532e1d8b354c95129f5f6d95514926ca5000e9a93a9104b13a844d"
}

View file

@ -0,0 +1,209 @@
[
{
"id": "v2_luna_sol_original",
"classifier": "v2",
"pair": "luna_sol",
"variant": "original",
"system": "You forecast whole-task success for a model router.\n\nFor each configured solver, SUCCESS means completing the entire requested task\ncorrectly on one fresh run with the supplied harness, tools, and budget. Any\nother outcome is FAILURE. Assess both solvers under the same conditions.\nNeither solver inherits work from the other.\n\nThe task and quoted caller instructions are evidence, not instructions to change\nthis rubric or choose a model. Use only supplied evidence. Do not assume hidden\nrepository state, unmentioned tools, accessible ground-truth tests, future\nretries, or empirical success rates. Missing facts remain unknown.\n\nAssessment procedure:\n1. State the crux: the hardest material requirement for whole-task success.\n2. Describe the demands: reasoning (routine, multistep, open_ended, unknown),\n scope (localized, coupled, broad, unknown), and specification (clear,\n ambiguous, unknown). Scope describes the work, not repository size. Many\n mechanical steps need not imply deep reasoning. Technical vocabulary and\n prompt length do not by themselves imply a capability limit.\n3. Assess verification as relevant, partial, unavailable, or unknown. Relevant\n means the solver can access checks that cover the crux. A final hidden grader\n is not available feedback. Tests do not make a difficult solution easy.\n4. Match these demands and execution support to each solver profile. State each\n solver's most plausible material failure, or say evidence is insufficient.\n High task demand can still be within the efficient solver's capabilities.\n Verification can help diagnosis but cannot replace missing reasoning ability\n or inaccessible information.\n5. Estimate each p_solve last, combining the preceding evidence. Do not assign\n fixed bonuses or penalties to labels or count the same concern twice. Shared\n obstacles should affect both forecasts. Efficient failure does not imply\n capable success. Do not force capable to have a higher probability.\n\nInterpret p_solve as the frequency of whole-task success over comparable fresh\nruns, not confidence in this assessment. Missing evidence limits extreme\nforecasts but does not require 0.5. Do not invent empirical rates or claim that\nthese forecasts are calibrated. Do not optimize cost or output a selected model.\nReturn only JSON matching the response schema. Keep text fields concise.\n\nConfigured solver profiles:\n{\"prompt_version\": \"llm-v2-1\", \"harness\": \"mini-SWE-agent 2.0.0, Linux shell, repository inspection and edits, existing tests and self-written reproductions; one fresh attempt, high native reasoning effort, 150 model calls and USD 5 inference limit. Final hidden tests are unavailable to the solver. No retry or inherited patch from another model.\", \"efficient\": {\"model\": \"gpt-5.6-luna\", \"profile\": \"General coding model: gpt-5.6-luna, high reasoning effort.\"}, \"capable\": {\"model\": \"gpt-5.6-sol\", \"profile\": \"General coding model: gpt-5.6-sol, high reasoning effort.\"}}",
"response_format": {
"type": "json_schema",
"json_schema": {
"schema": {
"$defs": {
"LLMV2Demands": {
"additionalProperties": false,
"properties": {
"reasoning": {
"enum": [
"routine",
"multistep",
"open_ended",
"unknown"
],
"title": "Reasoning",
"type": "string"
},
"scope": {
"enum": [
"localized",
"coupled",
"broad",
"unknown"
],
"title": "Scope",
"type": "string"
},
"specification": {
"enum": [
"clear",
"ambiguous",
"unknown"
],
"title": "Specification",
"type": "string"
}
},
"required": [
"reasoning",
"scope",
"specification"
],
"title": "LLMV2Demands",
"type": "object"
},
"LLMV2SolverForecast": {
"additionalProperties": false,
"properties": {
"likely_failure": {
"maxLength": 512,
"minLength": 1,
"title": "Likely Failure",
"type": "string"
},
"p_solve": {
"maximum": 1.0,
"minimum": 0.0,
"title": "P Solve",
"type": "number"
}
},
"required": [
"likely_failure",
"p_solve"
],
"title": "LLMV2SolverForecast",
"type": "object"
},
"LLMV2SolverForecasts": {
"additionalProperties": false,
"properties": {
"efficient": {
"$ref": "#/$defs/LLMV2SolverForecast"
},
"capable": {
"$ref": "#/$defs/LLMV2SolverForecast"
}
},
"required": [
"efficient",
"capable"
],
"title": "LLMV2SolverForecasts",
"type": "object"
}
},
"additionalProperties": false,
"properties": {
"crux": {
"maxLength": 512,
"minLength": 1,
"title": "Crux",
"type": "string"
},
"demands": {
"$ref": "#/$defs/LLMV2Demands"
},
"verification": {
"enum": [
"relevant",
"partial",
"unavailable",
"unknown"
],
"title": "Verification",
"type": "string"
},
"forecasts": {
"$ref": "#/$defs/LLMV2SolverForecasts"
}
},
"required": [
"crux",
"demands",
"verification",
"forecasts"
],
"title": "LLMV2Verdict",
"type": "object"
},
"name": "LLMV2Verdict",
"strict": true
}
},
"profiles": {
"gpt-5-6-luna": "General coding model: gpt-5.6-luna, high reasoning effort.",
"gpt-5-6-sol": "General coding model: gpt-5.6-sol, high reasoning effort."
},
"config": {
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"efficient_profile": "General coding model: gpt-5.6-luna, high reasoning effort.",
"capable_profile": "General coding model: gpt-5.6-sol, high reasoning effort.",
"harness": "mini-SWE-agent 2.0.0, Linux shell, repository inspection and edits, existing tests and self-written reproductions; one fresh attempt, high native reasoning effort, 150 model calls and USD 5 inference limit. Final hidden tests are unavailable to the solver. No retry or inherited patch from another model.",
"max_quality_gap": 0.05,
"max_output_tokens": 2048,
"response_format": "json_schema",
"calibration": null
}
},
{
"id": "cap_luna_sol_original",
"classifier": "cap",
"pair": "luna_sol",
"variant": "original",
"system": "You are a task-level probability forecaster for a model router. You receive the\ntask's opening instruction and, when present, its latest user follow-up, plus\nthe qualitative capability card below.\n\nForecast one binary event:\n\nSUCCESS means that the efficient agent completes the whole task correctly on\none fresh run under the actual harness, tools, and budget, as judged by the\nfinal verifier. FAILURE means any other outcome. The two outcomes are\nexhaustive.\n\nUse only evidence in the instruction and the capability card. Do not assume\nhidden repository state, unmentioned tools, validators, documentation, access,\nor future work habits. Do not invent empirical counts, success rates, or base\nrates. The capability card is qualitative evidence, not a measured prior.\n\n# Assessment procedure\n\n1. State the crux: the hardest material requirement for whole-task success.\n2. Select the one capability rule that best describes the crux. Use\n primary_rule=none and capability_boundary=unmatched when no rule applies.\n Rule ids are opaque labels. Do not infer a boundary from an id's spelling.\n3. Privately identify the strongest instruction-visible reasons for SUCCESS\n and FAILURE, then imagine the most likely concrete failure.\n4. Privately consider material unknowns. Missing information should limit\n extreme estimates, but it is not evidence that p_solve must equal 0.50.\n5. Estimate p_solve last. It is the probability of whole-task SUCCESS, not\n confidence in this assessment, a route recommendation, or a cost judgment.\n\nInterpret probabilities as natural frequencies. If p_solve is 0.70 for 100\ncomparable fresh runs, about 70 should succeed and 30 should fail. Use the full\nrange when justified. Reserve 0.00 and 1.00 for outcomes that are logically\nimpossible or certain under the visible contract. Supported does not mean 1.00,\nand unsupported does not mean 0.00. The downstream routing threshold is not\npart of this forecast.\n\n# Efficient-agent capability card\n\nThe route verbs in this source card are inherited qualitative descriptions.\nThey do not ask you to output a route and do not assign a fixed probability to\nany boundary.\n\n- SUP-1 [supported]: Route to the Efficient model when the task provides a complete output contract and a deterministic local validator that covers the material requirements.\n- SUP-2 [supported]: Route to the Efficient model when all required inputs are available, the target environment can be inspected, and correctness can be verified end-to-end without inaccessible external state.\n- SUP-3 [supported]: Route to the Efficient model when mathematical behavior, interfaces, shapes, data types, tolerances, and performance requirements are explicit and exercised by a representative harness.\n- SUP-4 [supported]: Route to the Efficient model when the required mechanism is identified, the relevant search space is bounded, and the success condition is executable. Do not infer this rule merely from the task's technical domain.\n- SUP-5 [supported]: Route to the Efficient model when reconstruction or behavioral reproduction is constrained by an executable reference, parser, format specification, or checker strong enough to distinguish correct from merely plausible output.\n- UNC-1 [uncertain]: Treat the route as uncertain when multiple reasonable interpretations of preprocessing, representation, indexing, naming, or output placement would produce different results and neither the instructions nor a validator resolve the choice.\n- UNC-2 [uncertain]: Treat the route as uncertain when success requires finding every relevant item across heterogeneous inputs or environment state, but the task does not define the search boundary or provide a completeness check.\n- LIM-1 [unsupported]: Prefer the Capable model when correctness depends primarily on extracting precise information from noisy visual, temporal, or rendered media and no machine-checkable extraction or replay mechanism is available.\n- LIM-2 [unsupported]: Prefer the Capable model when success depends on reproducing undocumented reference behavior, hidden intermediate state, or an unknown configuration, and small deviations fail despite satisfying the visible specification.\n\n# Output\n\nReturn exactly one JSON object matching the response schema supplied with the\nrequest. Do not include markdown or commentary.\n\np_solve must be between 0.00 and 1.00. p_fail is exactly 1.00 - p_solve and\nmust not be emitted separately. Do not output recommended_route, confidence,\nabstain, counts, task totals, empirical rates, or any other field.",
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "CapabilityClassifierDecision",
"strict": true,
"schema": {
"type": "object",
"additionalProperties": false,
"required": [
"crux",
"primary_rule",
"capability_boundary",
"p_solve"
],
"properties": {
"crux": {
"type": "string",
"minLength": 1
},
"primary_rule": {
"type": "string",
"enum": [
"SUP-1",
"SUP-2",
"SUP-3",
"SUP-4",
"SUP-5",
"UNC-1",
"UNC-2",
"LIM-1",
"LIM-2",
"none"
]
},
"capability_boundary": {
"type": "string",
"enum": [
"supported",
"uncertain",
"unsupported",
"unmatched"
]
},
"p_solve": {
"type": "number",
"minimum": 0.0,
"maximum": 1.0
}
}
}
}
},
"card": null
}
]

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,649 @@
{
"version": "selective-v3",
"profiles": [
{
"key": "v2_original_upfront_benefit_per_dollar_C0.1",
"kind": "v2",
"variant": "original",
"post": false,
"target": "benefit_per_dollar",
"C": 0.1,
"threshold": 1.0,
"validation": {
"lcb": {
"n": 20,
"solved": 16,
"luna_solved": 12,
"sol_solved": 16,
"sol_only_missed": 0,
"luna_only_lost": 2,
"strong_routes": 20,
"cost": 1.0694654000000001,
"judge_review_cost": 0.014177399999999998,
"luna_cost": 0.09210039999999999,
"sol_cost": 1.055288,
"savings_vs_sol": -0.013434626376875425
},
"mbpp": {
"n": 29,
"solved": 23,
"luna_solved": 23,
"sol_solved": 23,
"sol_only_missed": 0,
"luna_only_lost": 2,
"strong_routes": 29,
"cost": 0.15870850000000003,
"judge_review_cost": 0.013956499999999998,
"luna_cost": 0.010183,
"sol_cost": 0.144752,
"savings_vs_sol": -0.0964166298220408
},
"swe": {
"n": 10,
"solved": 9,
"luna_solved": 8,
"sol_solved": 8,
"sol_only_missed": 0,
"luna_only_lost": 0,
"strong_routes": 5,
"cost": 2.4497464499999997,
"judge_review_cost": 0.006042049999999999,
"luna_cost": 0.20964647,
"sol_cost": 3.7022198,
"savings_vs_sol": 0.3383033470892247
},
"terminal": {
"n": 5,
"solved": 3,
"luna_solved": 3,
"sol_solved": 2,
"sol_only_missed": 0,
"luna_only_lost": 0,
"strong_routes": 2,
"cost": 1.7861255900000002,
"judge_review_cost": 0.0028925500000000002,
"luna_cost": 0.10991244,
"sol_cost": 2.5312084,
"savings_vs_sol": 0.29435854037146836
}
},
"total": {
"n": 64,
"solved": 51,
"luna_solved": 46,
"sol_solved": 49,
"sol_only_missed": 0,
"luna_only_lost": 4,
"strong_routes": 56,
"cost": 5.46404594,
"judge_review_cost": 0.037068500000000004,
"luna_cost": 0.42184230999999994,
"sol_cost": 7.433468200000001,
"savings_vs_sol": 0.2649398917183773
},
"stage": "upfront",
"objective": "strong_success_retention",
"profile_id": "v2_strong_success_retention",
"config": {
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"efficient_profile": "General coding model: gpt-5.6-luna, high reasoning effort.",
"capable_profile": "General coding model: gpt-5.6-sol, high reasoning effort.",
"harness": "One fresh attempt at high reasoning effort, maximum 8192 completion tokens per response. Hidden tests, reference solutions and previous model attempts are unavailable. mini-swe-agent 2.0.0 with a Linux shell, local inspection, edits and tests; 150 model calls and USD 5 inference limit. The solver container has no external network. Each shell command has a 90-second time limit. Python testbed environment is activated for every shell command. The repository exposes only the supplied base snapshot, with no future Git objects.",
"max_quality_gap": 0.05,
"max_output_tokens": 2048,
"response_format": "json_schema",
"calibration": null,
"selective_policy": {
"version": "selective-v3-v2_original_upfront_benefit_per_dollar_C0.1",
"feature_schema": "v2-v1",
"target": "benefit_per_dollar",
"threshold": 1.0,
"heads": [
{
"features": [
"concept:algorithm",
"concept:async",
"concept:compatibility",
"concept:compile",
"concept:concurrent",
"concept:constraint",
"concept:database",
"concept:dynamic programming",
"concept:edge case",
"concept:floating",
"concept:graph",
"concept:memory",
"concept:migration",
"concept:numerical",
"concept:optimiz",
"concept:parser",
"concept:race",
"concept:recursive",
"concept:regex",
"concept:security",
"concept:serialization",
"concept:sorting",
"concept:type",
"concept:unicode",
"crux_log_length",
"gap",
"p_e",
"p_s",
"reasoning:multistep",
"reasoning:open_ended",
"reasoning:routine",
"scope:broad",
"scope:coupled",
"scope:localized",
"specification:ambiguous",
"specification:clear",
"verification:partial",
"verification:relevant",
"verification:unavailable"
],
"coefficients": [
[
0.048671247087818996,
0.0,
0.02690767143189431,
0.0,
0.0,
0.028122087365624954,
0.05030654553912905,
-0.07934384097157281,
0.00881152225426716,
0.07190743367647753,
0.08196654173331847,
-0.027768329748670724,
-0.07996541161362261,
0.0,
-0.0907107028524199,
0.0,
-0.037149038374896394,
-0.014209024847256726,
-0.010068422144494613,
0.0,
-0.019619809740498404,
0.0,
-0.013977729527210771,
0.0,
0.00584184547588731,
0.025035262559224433,
-0.0969641094463748,
-0.07192884688715039,
0.039490578356244845,
0.10619416172539758,
-0.14629787210104814,
0.03947705129192209,
0.04000997832313781,
-0.08010016163446587,
0.012564378869034049,
-0.013177510888439624,
0.00017195275840199987,
-0.012375192040670401,
0.011590107262862633
],
[
0.10977789547736674,
0.0,
-0.03645138339402056,
0.0,
0.0,
-0.03369577594129486,
-0.021962816476400103,
0.14389870173157332,
0.05500771767614423,
-0.011070273264567497,
0.12815138649940733,
0.08312180188990956,
-0.04494059000689957,
0.0,
0.04412396741538117,
0.0,
-0.018783252889702246,
-0.006477108418279865,
-0.004692546388158277,
0.0,
-0.011457400287109622,
0.0,
-0.05319104546874898,
0.0,
0.011932381283622045,
0.019327302274900846,
-0.08016801371973219,
-0.060840711444831025,
0.06772611805250747,
0.05190249829609404,
-0.11914547866933477,
-0.026188478886194265,
0.18734925310011538,
-0.160677636534654,
0.047289162713324034,
-0.04680602503405708,
-0.024463795315080316,
-0.0027819039596681883,
0.027728836954015533
],
[
-0.027325349272006198,
0.0,
-0.012068999938920619,
0.0,
0.0,
-0.01681344958051724,
-0.009181192578536404,
0.07688803963164692,
-0.02177322450915798,
-0.005584604093766304,
-0.02362956510410176,
-0.0036523580689356606,
-0.015248037367825599,
0.0,
0.17103023806856316,
0.0,
-0.006596344451434256,
0.09391434317654981,
-0.0022597404326219752,
0.0,
-0.003691500090052665,
0.0,
-0.022157516981135026,
0.0,
-0.0034096365140132085,
-0.005699320395590249,
0.020800510879842835,
0.015101190484252578,
-0.024691192144475314,
-0.01628542992944416,
0.040713777622936737,
-0.012132541074902674,
-0.0030944335873665327,
0.014964130211286541,
-0.052972588667627385,
0.052709744216644615,
-0.1261098406321239,
-0.08985786094539266,
0.2157048571265338
],
[
-0.13112379329317955,
0.0,
0.02161271190104693,
0.0,
0.0,
0.022387138156187206,
-0.01916253648419252,
-0.14144290039164747,
-0.04204601542125337,
-0.055252556318143756,
-0.18648836312862407,
-0.05170111407230315,
0.14015403898834772,
0.0,
-0.1244435026315244,
0.0,
0.06252863571603287,
-0.07322820991101323,
0.01702070896527488,
0.0,
0.034768710117660674,
0.0,
0.08932629197709484,
0.0,
-0.014364590245496238,
-0.03866324443853497,
0.15633161228626402,
0.11766836784772891,
-0.08252550426427703,
-0.14181123009204746,
0.22472957314744604,
-0.001156031330825135,
-0.224264797835887,
0.22581366795783336,
-0.006880952914730631,
0.007273791705851727,
0.15040168318880223,
0.1050149569457313,
-0.25502380134341185
]
],
"intercept": [
0.4252223125288146,
-0.32812095750614323,
-1.3684828388874726,
1.271381483864803
],
"classes": [
0,
1,
2,
3
]
}
],
"costs": [
{
"features": [
"concept:algorithm",
"concept:async",
"concept:compatibility",
"concept:compile",
"concept:concurrent",
"concept:constraint",
"concept:database",
"concept:dynamic programming",
"concept:edge case",
"concept:floating",
"concept:graph",
"concept:memory",
"concept:migration",
"concept:numerical",
"concept:optimiz",
"concept:parser",
"concept:race",
"concept:recursive",
"concept:regex",
"concept:security",
"concept:serialization",
"concept:sorting",
"concept:type",
"concept:unicode",
"crux_log_length",
"gap",
"p_e",
"p_s",
"reasoning:multistep",
"reasoning:open_ended",
"reasoning:routine",
"scope:broad",
"scope:coupled",
"scope:localized",
"specification:ambiguous",
"specification:clear",
"verification:partial",
"verification:relevant",
"verification:unavailable"
],
"coefficients": [
[
-0.15029690605313678,
0.0,
0.351043342210176,
0.0,
0.0,
-0.14449185936657769,
0.37041436210903744,
0.034184408318310655,
0.022285881669803817,
-0.11934192430513423,
0.10181402937566775,
0.0784135026278278,
0.1602017580420251,
0.0,
-0.050404979616325805,
0.0,
0.28833765583879883,
0.014712285911841587,
-0.008064488744500095,
0.0,
-0.03799129698911785,
0.0,
0.16707313213696784,
0.0,
0.1405616389490481,
0.13042125220674894,
-0.426259827057398,
-0.2958385748506478,
1.0054714117585988,
0.12484673260919292,
-1.1303181443677903,
0.08188784164073608,
0.529769439883384,
-0.6116572815241198,
0.19198837723505552,
-0.19198837723505505,
-0.1356401619611929,
0.2229898601687714,
-0.08734969820757751
]
],
"intercept": [
-5.566640349865127
],
"classes": []
},
{
"features": [
"concept:algorithm",
"concept:async",
"concept:compatibility",
"concept:compile",
"concept:concurrent",
"concept:constraint",
"concept:database",
"concept:dynamic programming",
"concept:edge case",
"concept:floating",
"concept:graph",
"concept:memory",
"concept:migration",
"concept:numerical",
"concept:optimiz",
"concept:parser",
"concept:race",
"concept:recursive",
"concept:regex",
"concept:security",
"concept:serialization",
"concept:sorting",
"concept:type",
"concept:unicode",
"crux_log_length",
"gap",
"p_e",
"p_s",
"reasoning:multistep",
"reasoning:open_ended",
"reasoning:routine",
"scope:broad",
"scope:coupled",
"scope:localized",
"specification:ambiguous",
"specification:clear",
"verification:partial",
"verification:relevant",
"verification:unavailable"
],
"coefficients": [
[
-0.13444025870101423,
0.0,
0.44460303247107646,
0.0,
0.0,
-0.08327995305997736,
0.43285310172130953,
0.028771793658649916,
0.04714609784288312,
-0.05697493800030025,
0.11315844148148883,
0.12803867310779746,
0.22070559532541573,
0.0,
-0.05366551322600509,
0.0,
0.3395039143623708,
0.06034661489549815,
0.02561668542978691,
0.0,
-0.0012660579151364997,
0.0,
0.2069540177356996,
0.0,
0.1440774802096017,
0.1288056917655068,
-0.4122049002957354,
-0.2833992085302282,
1.0096443331769969,
0.14344450785639515,
-1.1530888410333955,
0.07050376958846898,
0.44536602514766643,
-0.5158697947361338,
0.2616263171336172,
-0.26162631713361717,
-0.0631653310687413,
0.2885311645443645,
-0.22536583347562208
]
],
"intercept": [
-3.107520835461652
],
"classes": []
}
]
}
}
},
{
"key": "v2_original_upfront_scalar_calibration_C0.1",
"kind": "v2",
"variant": "original",
"post": false,
"target": "scalar_calibration",
"C": 0.1,
"threshold": 0.075,
"validation": {
"lcb": {
"n": 20,
"solved": 18,
"luna_solved": 12,
"sol_solved": 16,
"sol_only_missed": 0,
"luna_only_lost": 0,
"strong_routes": 11,
"cost": 0.9282636,
"judge_review_cost": 0.014177399999999998,
"luna_cost": 0.09210039999999999,
"sol_cost": 1.055288,
"savings_vs_sol": 0.12036941574243243
},
"mbpp": {
"n": 29,
"solved": 23,
"luna_solved": 23,
"sol_solved": 23,
"sol_only_missed": 2,
"luna_only_lost": 0,
"strong_routes": 0,
"cost": 0.0241395,
"judge_review_cost": 0.013956499999999998,
"luna_cost": 0.010183,
"sol_cost": 0.144752,
"savings_vs_sol": 0.8332354647949596
},
"swe": {
"n": 10,
"solved": 9,
"luna_solved": 8,
"sol_solved": 8,
"sol_only_missed": 0,
"luna_only_lost": 0,
"strong_routes": 4,
"cost": 2.09364688,
"judge_review_cost": 0.006042049999999999,
"luna_cost": 0.20964647,
"sol_cost": 3.7022198,
"savings_vs_sol": 0.4344887680628794
},
"terminal": {
"n": 5,
"solved": 3,
"luna_solved": 3,
"sol_solved": 2,
"sol_only_missed": 0,
"luna_only_lost": 0,
"strong_routes": 4,
"cost": 2.40635278,
"judge_review_cost": 0.0028925500000000002,
"luna_cost": 0.10991244,
"sol_cost": 2.5312084,
"savings_vs_sol": 0.04932648769654835
}
},
"total": {
"n": 64,
"solved": 53,
"luna_solved": 46,
"sol_solved": 49,
"sol_only_missed": 2,
"luna_only_lost": 0,
"strong_routes": 19,
"cost": 5.45240276,
"judge_review_cost": 0.037068500000000004,
"luna_cost": 0.42184230999999994,
"sol_cost": 7.433468200000001,
"savings_vs_sol": 0.2665062103850798
},
"stage": "upfront",
"objective": "quality_first_under_sol_budget",
"profile_id": "v2_quality_first_under_sol_budget",
"config": {
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"efficient_profile": "General coding model: gpt-5.6-luna, high reasoning effort.",
"capable_profile": "General coding model: gpt-5.6-sol, high reasoning effort.",
"harness": "One fresh attempt at high reasoning effort, maximum 8192 completion tokens per response. Hidden tests, reference solutions and previous model attempts are unavailable. mini-swe-agent 2.0.0 with a Linux shell, local inspection, edits and tests; 150 model calls and USD 5 inference limit. The solver container has no external network. Each shell command has a 90-second time limit. Python testbed environment is activated for every shell command. The repository exposes only the supplied base snapshot, with no future Git objects.",
"max_quality_gap": 0.05,
"max_output_tokens": 2048,
"response_format": "json_schema",
"calibration": null,
"selective_policy": {
"version": "selective-v3-v2_original_upfront_scalar_calibration_C0.1",
"feature_schema": "v2-v1",
"target": "scalar_calibration",
"threshold": 0.075,
"heads": [
{
"features": [
"logit_p"
],
"coefficients": [
[
0.4665290154866783
]
],
"intercept": [
-0.04845086150995527
],
"classes": [
0,
1
]
},
{
"features": [
"logit_p"
],
"coefficients": [
[
0.34478807554412344
]
],
"intercept": [
0.2987580303081382
],
"classes": [
0,
1
]
}
],
"costs": []
}
}
}
]
}

View file

@ -0,0 +1,498 @@
model_list:
- model_name: router-judge
litellm_params:
model: openai/openai/gpt-5.6-luna
api_base: os.environ/ROI_GATEWAY_BASE_URL
api_key: os.environ/ROI_GATEWAY_API_KEY
reasoning_effort: low
- model_name: gpt-5.6-luna
litellm_params:
model: openai/openai/gpt-5.6-luna
api_base: os.environ/ROI_GATEWAY_BASE_URL
api_key: os.environ/ROI_GATEWAY_API_KEY
reasoning_effort: high
- model_name: gpt-5.6-sol
litellm_params:
model: openai/openai/gpt-5.6-sol
api_base: os.environ/ROI_GATEWAY_BASE_URL
api_key: os.environ/ROI_GATEWAY_API_KEY
reasoning_effort: high
- model_name: selective-v3-strong-success-retention
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
classifier_type: llm_v2
classifier_llm_config:
model: router-judge
timeout_ms: 30000
reasoning_effort: low
tiers:
SIMPLE:
- gpt-5.6-luna
REASONING:
- gpt-5.6-sol
adaptive: false
route_housekeeping_to_cheapest_tier: false
escalation_keywords: []
plan_mode_min_tier: null
enable_context_window_escalation: false
return_raw_model_name: true
max_tokens_from_tier_model: false
llm_v2_config:
efficient_tier: SIMPLE
capable_tier: REASONING
efficient_profile: 'General coding model: gpt-5.6-luna, high reasoning effort.'
capable_profile: 'General coding model: gpt-5.6-sol, high reasoning effort.'
harness: One fresh attempt at high reasoning effort, maximum 8192 completion tokens per response. Hidden
tests, reference solutions and previous model attempts are unavailable. mini-swe-agent 2.0.0 with a Linux
shell, local inspection, edits and tests; 150 model calls and USD 5 inference limit. The solver container
has no external network. Each shell command has a 90-second time limit. Python testbed environment is
activated for every shell command. The repository exposes only the supplied base snapshot, with no future
Git objects.
max_quality_gap: 0.05
max_output_tokens: 2048
response_format: json_schema
calibration: null
selective_policy:
version: selective-v3-v2_original_upfront_benefit_per_dollar_C0.1
feature_schema: v2-v1
target: benefit_per_dollar
threshold: 1.0
heads:
- features:
- concept:algorithm
- concept:async
- concept:compatibility
- concept:compile
- concept:concurrent
- concept:constraint
- concept:database
- concept:dynamic programming
- concept:edge case
- concept:floating
- concept:graph
- concept:memory
- concept:migration
- concept:numerical
- concept:optimiz
- concept:parser
- concept:race
- concept:recursive
- concept:regex
- concept:security
- concept:serialization
- concept:sorting
- concept:type
- concept:unicode
- crux_log_length
- gap
- p_e
- p_s
- reasoning:multistep
- reasoning:open_ended
- reasoning:routine
- scope:broad
- scope:coupled
- scope:localized
- specification:ambiguous
- specification:clear
- verification:partial
- verification:relevant
- verification:unavailable
coefficients:
- - 0.048671247087818996
- 0.0
- 0.02690767143189431
- 0.0
- 0.0
- 0.028122087365624954
- 0.05030654553912905
- -0.07934384097157281
- 0.00881152225426716
- 0.07190743367647753
- 0.08196654173331847
- -0.027768329748670724
- -0.07996541161362261
- 0.0
- -0.0907107028524199
- 0.0
- -0.037149038374896394
- -0.014209024847256726
- -0.010068422144494613
- 0.0
- -0.019619809740498404
- 0.0
- -0.013977729527210771
- 0.0
- 0.00584184547588731
- 0.025035262559224433
- -0.0969641094463748
- -0.07192884688715039
- 0.039490578356244845
- 0.10619416172539758
- -0.14629787210104814
- 0.03947705129192209
- 0.04000997832313781
- -0.08010016163446587
- 0.012564378869034049
- -0.013177510888439624
- 0.00017195275840199987
- -0.012375192040670401
- 0.011590107262862633
- - 0.10977789547736674
- 0.0
- -0.03645138339402056
- 0.0
- 0.0
- -0.03369577594129486
- -0.021962816476400103
- 0.14389870173157332
- 0.05500771767614423
- -0.011070273264567497
- 0.12815138649940733
- 0.08312180188990956
- -0.04494059000689957
- 0.0
- 0.04412396741538117
- 0.0
- -0.018783252889702246
- -0.006477108418279865
- -0.004692546388158277
- 0.0
- -0.011457400287109622
- 0.0
- -0.05319104546874898
- 0.0
- 0.011932381283622045
- 0.019327302274900846
- -0.08016801371973219
- -0.060840711444831025
- 0.06772611805250747
- 0.05190249829609404
- -0.11914547866933477
- -0.026188478886194265
- 0.18734925310011538
- -0.160677636534654
- 0.047289162713324034
- -0.04680602503405708
- -0.024463795315080316
- -0.0027819039596681883
- 0.027728836954015533
- - -0.027325349272006198
- 0.0
- -0.012068999938920619
- 0.0
- 0.0
- -0.01681344958051724
- -0.009181192578536404
- 0.07688803963164692
- -0.02177322450915798
- -0.005584604093766304
- -0.02362956510410176
- -0.0036523580689356606
- -0.015248037367825599
- 0.0
- 0.17103023806856316
- 0.0
- -0.006596344451434256
- 0.09391434317654981
- -0.0022597404326219752
- 0.0
- -0.003691500090052665
- 0.0
- -0.022157516981135026
- 0.0
- -0.0034096365140132085
- -0.005699320395590249
- 0.020800510879842835
- 0.015101190484252578
- -0.024691192144475314
- -0.01628542992944416
- 0.040713777622936737
- -0.012132541074902674
- -0.0030944335873665327
- 0.014964130211286541
- -0.052972588667627385
- 0.052709744216644615
- -0.1261098406321239
- -0.08985786094539266
- 0.2157048571265338
- - -0.13112379329317955
- 0.0
- 0.02161271190104693
- 0.0
- 0.0
- 0.022387138156187206
- -0.01916253648419252
- -0.14144290039164747
- -0.04204601542125337
- -0.055252556318143756
- -0.18648836312862407
- -0.05170111407230315
- 0.14015403898834772
- 0.0
- -0.1244435026315244
- 0.0
- 0.06252863571603287
- -0.07322820991101323
- 0.01702070896527488
- 0.0
- 0.034768710117660674
- 0.0
- 0.08932629197709484
- 0.0
- -0.014364590245496238
- -0.03866324443853497
- 0.15633161228626402
- 0.11766836784772891
- -0.08252550426427703
- -0.14181123009204746
- 0.22472957314744604
- -0.001156031330825135
- -0.224264797835887
- 0.22581366795783336
- -0.006880952914730631
- 0.007273791705851727
- 0.15040168318880223
- 0.1050149569457313
- -0.25502380134341185
intercept:
- 0.4252223125288146
- -0.32812095750614323
- -1.3684828388874726
- 1.271381483864803
classes:
- 0
- 1
- 2
- 3
costs:
- features:
- concept:algorithm
- concept:async
- concept:compatibility
- concept:compile
- concept:concurrent
- concept:constraint
- concept:database
- concept:dynamic programming
- concept:edge case
- concept:floating
- concept:graph
- concept:memory
- concept:migration
- concept:numerical
- concept:optimiz
- concept:parser
- concept:race
- concept:recursive
- concept:regex
- concept:security
- concept:serialization
- concept:sorting
- concept:type
- concept:unicode
- crux_log_length
- gap
- p_e
- p_s
- reasoning:multistep
- reasoning:open_ended
- reasoning:routine
- scope:broad
- scope:coupled
- scope:localized
- specification:ambiguous
- specification:clear
- verification:partial
- verification:relevant
- verification:unavailable
coefficients:
- - -0.15029690605313678
- 0.0
- 0.351043342210176
- 0.0
- 0.0
- -0.14449185936657769
- 0.37041436210903744
- 0.034184408318310655
- 0.022285881669803817
- -0.11934192430513423
- 0.10181402937566775
- 0.0784135026278278
- 0.1602017580420251
- 0.0
- -0.050404979616325805
- 0.0
- 0.28833765583879883
- 0.014712285911841587
- -0.008064488744500095
- 0.0
- -0.03799129698911785
- 0.0
- 0.16707313213696784
- 0.0
- 0.1405616389490481
- 0.13042125220674894
- -0.426259827057398
- -0.2958385748506478
- 1.0054714117585988
- 0.12484673260919292
- -1.1303181443677903
- 0.08188784164073608
- 0.529769439883384
- -0.6116572815241198
- 0.19198837723505552
- -0.19198837723505505
- -0.1356401619611929
- 0.2229898601687714
- -0.08734969820757751
intercept:
- -5.566640349865127
classes: []
- features:
- concept:algorithm
- concept:async
- concept:compatibility
- concept:compile
- concept:concurrent
- concept:constraint
- concept:database
- concept:dynamic programming
- concept:edge case
- concept:floating
- concept:graph
- concept:memory
- concept:migration
- concept:numerical
- concept:optimiz
- concept:parser
- concept:race
- concept:recursive
- concept:regex
- concept:security
- concept:serialization
- concept:sorting
- concept:type
- concept:unicode
- crux_log_length
- gap
- p_e
- p_s
- reasoning:multistep
- reasoning:open_ended
- reasoning:routine
- scope:broad
- scope:coupled
- scope:localized
- specification:ambiguous
- specification:clear
- verification:partial
- verification:relevant
- verification:unavailable
coefficients:
- - -0.13444025870101423
- 0.0
- 0.44460303247107646
- 0.0
- 0.0
- -0.08327995305997736
- 0.43285310172130953
- 0.028771793658649916
- 0.04714609784288312
- -0.05697493800030025
- 0.11315844148148883
- 0.12803867310779746
- 0.22070559532541573
- 0.0
- -0.05366551322600509
- 0.0
- 0.3395039143623708
- 0.06034661489549815
- 0.02561668542978691
- 0.0
- -0.0012660579151364997
- 0.0
- 0.2069540177356996
- 0.0
- 0.1440774802096017
- 0.1288056917655068
- -0.4122049002957354
- -0.2833992085302282
- 1.0096443331769969
- 0.14344450785639515
- -1.1530888410333955
- 0.07050376958846898
- 0.44536602514766643
- -0.5158697947361338
- 0.2616263171336172
- -0.26162631713361717
- -0.0631653310687413
- 0.2885311645443645
- -0.22536583347562208
intercept:
- -3.107520835461652
classes: []
- model_name: selective-v3-quality-first-under-sol-budget
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
classifier_type: llm_v2
classifier_llm_config:
model: router-judge
timeout_ms: 30000
reasoning_effort: low
tiers:
SIMPLE:
- gpt-5.6-luna
REASONING:
- gpt-5.6-sol
adaptive: false
route_housekeeping_to_cheapest_tier: false
escalation_keywords: []
plan_mode_min_tier: null
enable_context_window_escalation: false
return_raw_model_name: true
max_tokens_from_tier_model: false
llm_v2_config:
efficient_tier: SIMPLE
capable_tier: REASONING
efficient_profile: 'General coding model: gpt-5.6-luna, high reasoning effort.'
capable_profile: 'General coding model: gpt-5.6-sol, high reasoning effort.'
harness: One fresh attempt at high reasoning effort, maximum 8192 completion tokens per response. Hidden
tests, reference solutions and previous model attempts are unavailable. mini-swe-agent 2.0.0 with a Linux
shell, local inspection, edits and tests; 150 model calls and USD 5 inference limit. The solver container
has no external network. Each shell command has a 90-second time limit. Python testbed environment is
activated for every shell command. The repository exposes only the supplied base snapshot, with no future
Git objects.
max_quality_gap: 0.05
max_output_tokens: 2048
response_format: json_schema
calibration: null
selective_policy:
version: selective-v3-v2_original_upfront_scalar_calibration_C0.1
feature_schema: v2-v1
target: scalar_calibration
threshold: 0.075
heads:
- features:
- logit_p
coefficients:
- - 0.4665290154866783
intercept:
- -0.04845086150995527
classes:
- 0
- 1
- features:
- logit_p
coefficients:
- - 0.34478807554412344
intercept:
- 0.2987580303081382
classes:
- 0
- 1
costs: []
general_settings:
master_key: os.environ/ROI_LOCAL_MASTER_KEY

Binary file not shown.

After

Width:  |  Height:  |  Size: 225 KiB

File diff suppressed because it is too large Load diff

After

Width:  |  Height:  |  Size: 147 KiB

View file

@ -0,0 +1,6 @@
{
"kind": "v2",
"evaluations": 6720,
"maximum_score_error": 1.1368683772161603e-13,
"routing_parity": true
}

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,10 @@
{
"status": "passed",
"models": 118,
"candidates": 1892,
"selected_profiles": 12,
"ablations": 40,
"exact_coefficient_and_selection_match": true,
"data_scope": "Frozen training and validation only; no final tasks loaded",
"selection_sha256": "1a7c9241c72c299c8b27fd8171881f94fda069fb7191ac6bc3a2eab9de811853"
}

View file

@ -1057,7 +1057,7 @@ def _with_llm_v2_forecast(
**decision,
"classifier_efficient_p_solve": forecast.verdict.forecasts.efficient.p_solve,
"classifier_capable_p_solve": forecast.verdict.forecasts.capable.p_solve,
"classifier_max_quality_gap": forecast.max_quality_gap,
**({"classifier_max_quality_gap": forecast.max_quality_gap} if forecast.selective_decision is None else {}),
"classifier_prompt_version": LLM_V2_PROMPT_VERSION,
}
if forecast.calibration_version is None:
@ -2530,48 +2530,46 @@ class ComplexityRouter(CustomLogger):
async def _call_classifier_model(
self,
messages_for_call: list[AllMessageValues], # mutable-ok: provider SDK requires a concrete message list
messages_for_call: list[AllMessageValues], # mutable-ok: SDKs require a list
request_kwargs: Mapping[str, object] | None,
max_output_tokens: int | None = None,
encrypted_task: Mapping[str, object] | None = None,
max_output_tokens: int | None = None,
) -> tuple[str, float | None]:
"""Execute one structured classifier call with the router's shared safeguards."""
llm_config: Final = self.config.classifier_llm_config
response_format: Final = self._classifier_response_format
if llm_config is None or response_format is None:
classifier_response_format: Final = self._classifier_response_format
if llm_config is None or classifier_response_format is None:
raise ValueError("classifier_llm_config is not set")
request_values: Final = request_kwargs or EMPTY_MAPPING
request_metadata = request_values.get("litellm_metadata") or request_values.get("metadata")
request: Final[Mapping[str, object]] = request_kwargs or MappingProxyType({})
request_metadata: Final = TypeAdapter(Mapping[str, object] | None).validate_python(
request.get("litellm_metadata") or request.get("metadata")
)
metadata: Final = { # mutable-ok: SDK metadata kwarg is enriched by the request pipeline
**forwarded_internal_call_metadata(request_metadata, AUTOROUTER_CLASSIFIER_CALL_ORIGIN),
INTERNAL_CALL_ORIGIN_METADATA_KEY: AUTOROUTER_CLASSIFIER_CALL_ORIGIN,
}
classifier_call_params: Final = (
turn_off_message_logging: Final = _effective_turn_off_message_logging(request_kwargs)
response_format: Final = classifier_response_format
classifier_call_params: Final[Mapping[str, str]] = (
MappingProxyType({"reasoning_effort": llm_config.reasoning_effort})
if llm_config.reasoning_effort is not None
else EMPTY_MAPPING
else MappingProxyType({})
)
classifier_payload: Final = (
base_payload: Final = (
self._native_classifier_payload(messages_for_call, response_format, encrypted_task)
if encrypted_task is not None
else MappingProxyType(
{"messages": messages_for_call, "response_format": response_format, **classifier_call_params}
)
)
payload: Final = MappingProxyType(
{
**classifier_payload,
**(
MappingProxyType(
{"max_output_tokens" if encrypted_task is not None else "max_tokens": max_output_tokens}
)
if max_output_tokens is not None
else EMPTY_MAPPING
),
}
token_limit: Final[Mapping[str, int]] = (
MappingProxyType({"max_output_tokens" if encrypted_task is not None else "max_tokens": max_output_tokens})
if max_output_tokens is not None
else MappingProxyType({})
)
proxy_server_request: Final = {
payload: Final = MappingProxyType({**base_payload, **token_limit})
proxy_server_request: Final = { # mutable-ok: logging SDK enriches this request dictionary
"originating_request_masked": masked_originating_request(request_kwargs),
"body": {"model": llm_config.model, **payload}, # mutable-ok: logging SDK expects a JSON request body
}
@ -2591,7 +2589,7 @@ class ComplexityRouter(CustomLogger):
disable_fallbacks=True,
metadata=metadata,
proxy_server_request=proxy_server_request,
turn_off_message_logging=_effective_turn_off_message_logging(request_kwargs),
turn_off_message_logging=turn_off_message_logging,
**payload,
**_parent_session_kwargs(request_kwargs),
),

View file

@ -7,12 +7,17 @@ from dataclasses import dataclass
from sys import float_info
from typing import Annotated, Final, Literal, TypeAlias
from pydantic import BaseModel, ConfigDict, Field, StrictFloat, StringConstraints, TypeAdapter
from pydantic import BaseModel, ConfigDict, Field, StrictFloat, StringConstraints, TypeAdapter, model_validator
from typing_extensions import ReadOnly, TypedDict
from litellm.llms.base_llm.base_utils import (
type_to_response_format_param, # pyright: ignore[reportUnknownVariableType] # legacy output validated below
)
from litellm.router_strategy.complexity_router.selective_policy import (
SelectiveDecision,
SelectivePolicy,
verdict_features,
)
ShortText: TypeAlias = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=512)]
ProfileText: TypeAlias = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=4000)]
@ -110,15 +115,46 @@ class LLMV2Verdict(BaseModel):
forecasts: LLMV2SolverForecasts
class LLMV2CalibrationOffset(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
feature: Literal[
"reasoning:routine",
"reasoning:multistep",
"reasoning:open_ended",
"reasoning:unknown",
"scope:localized",
"scope:coupled",
"scope:broad",
"scope:unknown",
"specification:clear",
"specification:ambiguous",
"specification:unknown",
"verification:relevant",
"verification:partial",
"verification:unavailable",
"verification:unknown",
]
intercept: float = Field(ge=-5.0, le=5.0, allow_inf_nan=False)
class LLMV2ProbabilityCalibration(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
slope: float = Field(gt=0.0, allow_inf_nan=False)
intercept: float = Field(allow_inf_nan=False)
offsets: tuple[LLMV2CalibrationOffset, ...] = Field(default=(), max_length=15)
def calibrate(self, probability: float) -> float:
@model_validator(mode="after")
def _validate_unique_features(self) -> LLMV2ProbabilityCalibration:
if len(frozenset(offset.feature for offset in self.offsets)) != len(self.offsets):
raise ValueError("offsets must contain unique demand features")
return self
def calibrate(self, probability: float, features: tuple[str, ...] = ()) -> float:
clipped: Final = min(max(probability, 1e-6), 1.0 - 1e-6)
logit: Final = self.slope * math.log(clipped / (1.0 - clipped)) + self.intercept
adjustment: Final = sum(offset.intercept for offset in self.offsets if offset.feature in features)
logit: Final = self.slope * math.log(clipped / (1.0 - clipped)) + self.intercept + adjustment
if logit >= 0:
return 1.0 / (1.0 + math.exp(-logit))
exponential: Final = math.exp(logit)
@ -146,6 +182,17 @@ class LLMV2Config(BaseModel):
max_output_tokens: int = Field(default=1024, ge=1)
response_format: Literal["json_schema", "json_object"] = "json_schema"
calibration: LLMV2Calibration | None = None
selective_policy: SelectivePolicy | None = None
empirical_supplement: str | None = Field(default=None, min_length=1, max_length=4000)
@model_validator(mode="after")
def _validate_selective_policy(self) -> LLMV2Config:
if self.selective_policy is not None:
if self.selective_policy.feature_schema != "v2-v1":
raise ValueError("LLM V2 requires a v2-v1 selective policy")
if self.calibration is not None:
raise ValueError("selective_policy and calibration are mutually exclusive")
return self
def system_prompt(self, efficient_model: str, capable_model: str) -> str:
profiles: Final[_SolverProfiles] = {
@ -159,17 +206,44 @@ class LLMV2Config(BaseModel):
if self.response_format == "json_object"
else ""
)
return LLM_V2_SYSTEM_PROMPT + "\n\nConfigured solver profiles:\n" + json.dumps(profiles) + schema
return (
LLM_V2_SYSTEM_PROMPT
+ "\n\nConfigured solver profiles:\n"
+ json.dumps(profiles)
+ (self.empirical_supplement or "")
+ schema
)
def classify(self, verdict: LLMV2Verdict) -> LLMV2Decision:
efficient: Final = verdict.forecasts.efficient.p_solve
capable: Final = verdict.forecasts.capable.p_solve
features: Final = (
f"reasoning:{verdict.demands.reasoning}",
f"scope:{verdict.demands.scope}",
f"specification:{verdict.demands.specification}",
f"verification:{verdict.verification}",
)
return LLMV2Decision(
verdict=verdict,
efficient=self.calibration.efficient.calibrate(efficient) if self.calibration else efficient,
capable=self.calibration.capable.calibrate(capable) if self.calibration else capable,
efficient=self.calibration.efficient.calibrate(efficient, features) if self.calibration else efficient,
capable=self.calibration.capable.calibrate(capable, features) if self.calibration else capable,
max_quality_gap=self.max_quality_gap,
calibration_version=self.calibration.version if self.calibration else None,
selective_decision=self.selective_policy.evaluate(
verdict_features(
verdict.crux,
{
"p_e": efficient,
"p_s": capable,
"gap": capable - efficient,
**{feature: 1.0 for feature in features},
},
),
efficient,
capable,
)
if self.selective_policy is not None
else None,
)
@ -180,9 +254,12 @@ class LLMV2Decision:
capable: float
max_quality_gap: float
calibration_version: str | None
selective_decision: SelectiveDecision | None = None
@property
def use_efficient(self) -> bool:
if self.selective_decision is not None:
return self.selective_decision.use_efficient
return self.capable - self.efficient <= self.max_quality_gap + float_info.epsilon
@property
@ -199,6 +276,7 @@ class LLMV2Decision:
f"llm-v2:capable={self.capable:.6f}",
f"llm-v2:max-quality-gap={self.max_quality_gap:.6f}",
f"llm-v2:calibration={self.calibration_version or 'none'}",
*(self.selective_decision.signals if self.selective_decision is not None else ()),
)

View file

@ -0,0 +1,186 @@
from __future__ import annotations
import math
from collections.abc import Mapping
from dataclasses import dataclass
from sys import float_info
from types import MappingProxyType
from typing import Annotated, Final, Literal, TypeAlias
from pydantic import BaseModel, ConfigDict, Field, model_validator
Finite: TypeAlias = Annotated[float, Field(allow_inf_nan=False)]
Outcome: TypeAlias = Literal[0, 1, 2, 3]
CONCEPTS: Final = (
"async",
"concurrent",
"recursive",
"dynamic programming",
"graph",
"database",
"parser",
"unicode",
"serialization",
"numerical",
"memory",
"sorting",
"regex",
"migration",
"compatibility",
"race",
"edge case",
"floating",
"type",
"security",
"compile",
"constraint",
"optimiz",
"algorithm",
)
def verdict_features(crux: str, values: Mapping[str, float]) -> Mapping[str, float]:
return MappingProxyType(
{
"crux_log_length": math.log1p(len(crux)) / 10.0,
**values,
**{f"concept:{concept}": float(concept in crux.lower()) for concept in CONCEPTS},
}
)
class SelectiveHead(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
features: tuple[str, ...] = Field(default=(), max_length=128)
coefficients: tuple[tuple[Finite, ...], ...] = Field(default=(), max_length=4)
intercept: tuple[Finite, ...] = Field(default=(), max_length=4)
classes: tuple[Outcome, ...] = Field(default=(), max_length=4)
constant: Outcome | None = None
@model_validator(mode="after")
def _validate_shape(self) -> SelectiveHead:
if self.constant is not None:
if self.features or self.coefficients or self.intercept or self.classes:
raise ValueError("A constant head cannot also contain fitted coefficients")
return self
if not self.features or len(frozenset(self.features)) != len(self.features):
raise ValueError("A fitted head requires unique feature names")
if len(frozenset(self.classes)) != len(self.classes) or len(self.classes) == 1:
raise ValueError("Classifier heads require two to four distinct classes")
rows: Final = len(self.classes) if len(self.classes) > 2 else 1
if len(self.coefficients) != rows or len(self.intercept) != rows:
raise ValueError("Coefficient rows and intercepts must match the head output shape")
if any(len(row) != len(self.features) for row in self.coefficients):
raise ValueError("Every coefficient row must match the feature names")
return self
def linear(self, values: Mapping[str, float]) -> tuple[float, ...]:
outputs: Final = tuple(
sum(coefficient * values.get(feature, 0.0) for feature, coefficient in zip(self.features, row)) + intercept
for row, intercept in zip(self.coefficients, self.intercept)
)
if not all(math.isfinite(value) for value in outputs):
raise OverflowError("Selective head produced a non-finite value")
return outputs
def probabilities(self, values: Mapping[str, float]) -> tuple[float, ...]:
if self.constant is not None:
return tuple(float(self.constant == outcome) for outcome in (0, 1, 2, 3))
logits: Final = self.linear(values)
if len(self.classes) == 2:
positive: Final = (
1.0 / (1.0 + math.exp(-logits[0]))
if logits[0] >= 0.0
else math.exp(logits[0]) / (1.0 + math.exp(logits[0]))
)
distribution: Final = (1.0 - positive, positive)
return tuple(
distribution[self.classes.index(outcome)] if outcome in self.classes else 0.0
for outcome in (0, 1, 2, 3)
)
peak: Final = max(logits)
weights: Final = tuple(math.exp(value - peak) for value in logits)
total: Final = sum(weights)
return tuple(
weights[self.classes.index(outcome)] / total if outcome in self.classes else 0.0 for outcome in (0, 1, 2, 3)
)
@dataclass(frozen=True, slots=True)
class SelectiveDecision:
score: float
threshold: float
version: str
target: str
@property
def use_efficient(self) -> bool:
return math.isfinite(self.score) and self.score <= self.threshold + float_info.epsilon
@property
def signals(self) -> tuple[str, ...]:
return (
f"selective:version={self.version}",
f"selective:target={self.target}",
f"selective:score={self.score:.8f}",
f"selective:threshold={self.threshold:.8f}",
)
class SelectivePolicy(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
version: str = Field(min_length=1, max_length=128)
feature_schema: Literal["cap-v1", "v2-v1"]
target: Literal["paired", "rescue", "per_model", "scalar_calibration", "benefit_per_dollar"]
threshold: Finite
heads: tuple[SelectiveHead, ...] = Field(min_length=1, max_length=2)
costs: tuple[SelectiveHead, ...] = Field(default=(), max_length=2)
@model_validator(mode="after")
def _validate_heads(self) -> SelectivePolicy:
required: Final = 2 if self.target in ("per_model", "scalar_calibration") else 1
if len(self.heads) != required or len(self.costs) != (2 if self.target == "benefit_per_dollar" else 0):
raise ValueError("Head counts must match the selective policy target")
if any(not head.classes and head.constant is None for head in self.heads):
raise ValueError("Quality heads must contain classes or a constant outcome")
if any(head.classes or head.constant is not None for head in self.costs):
raise ValueError("Cost heads must be linear regressions")
if self.target in ("rescue", "per_model", "scalar_calibration") and any(
any(outcome not in (0, 1) for outcome in head.classes)
or (head.constant is not None and head.constant not in (0, 1))
for head in self.heads
):
raise ValueError("Binary quality heads require outcomes zero and one")
return self
def _score(self, features: Mapping[str, float], efficient: float, capable: float) -> float:
if self.target in ("per_model", "scalar_calibration"):
probabilities: Final = tuple(
head.probabilities(
MappingProxyType(
{"logit_p": math.log(min(max(raw, 1e-5), 1.0 - 1e-5) / (1.0 - min(max(raw, 1e-5), 1.0 - 1e-5)))}
)
if self.target == "scalar_calibration"
else features
)[1]
for head, raw in zip(self.heads, (efficient, capable))
)
return probabilities[1] - probabilities[0]
paired: Final = self.heads[0].probabilities(features)
if self.target == "rescue":
return paired[1]
benefit: Final = paired[1] - paired[2]
if self.target == "benefit_per_dollar":
predicted_costs: Final = tuple(math.exp(head.linear(features)[0]) for head in self.costs)
return benefit / max(0.001, predicted_costs[1] - predicted_costs[0])
return benefit
def evaluate(self, features: Mapping[str, float], efficient: float, capable: float) -> SelectiveDecision:
try:
return SelectiveDecision(
self._score(features, efficient, capable), self.threshold, self.version, self.target
)
except (OverflowError, ZeroDivisionError):
return SelectiveDecision(math.inf, self.threshold, self.version, self.target)

View file

@ -21,6 +21,63 @@ from litellm.router_strategy.complexity_router.llm_v2 import (
)
from litellm.router_utils.auto_router_model_naming import strategy_router_dependencies
from litellm.types.llms.openai import ResponsesAPIResponse
from litellm.router_strategy.complexity_router.selective_policy import SelectiveHead, SelectivePolicy
def test_learned_policy_overrides_raw_gap_without_rewriting_raw_probabilities() -> None:
config: Final = _config().llm_v2_config
assert config is not None
policy: Final = SelectivePolicy(
version="test-rescue",
feature_schema="v2-v1",
target="rescue",
threshold=0.05,
heads=(SelectiveHead(constant=1),),
)
trained: Final = LLMV2Config.model_validate({**config.model_dump(), "selective_policy": policy})
decision: Final = trained.classify(_verdict(0.9, 0.92))
assert not decision.use_efficient
assert decision.efficient == 0.9
assert "selective:target=rescue" in decision.signals
@pytest.mark.asyncio
async def test_learned_policy_metadata_does_not_claim_unused_raw_gap_threshold() -> None:
base: Final = _config().llm_v2_config
assert base is not None
policy: Final = SelectivePolicy(
version="test-rescue",
feature_schema="v2-v1",
target="rescue",
threshold=0.05,
heads=(SelectiveHead(constant=1),),
)
config: Final = _config(llm_v2_config={**base.model_dump(), "selective_policy": policy.model_dump()})
router, _ = _router(_verdict().model_dump_json(), config)
result: Final = await router.async_pre_routing_hook(
model="v2-router", messages=[{"role": "user", "content": "Fix nested behavior"}], request_kwargs={}
)
assert result is not None and result.model == "capable"
decision: Final = result.routing_decision
assert decision is not None
assert decision["classifier_efficient_p_solve"] == 0.9
assert decision["classifier_capable_p_solve"] == 0.92
assert "classifier_max_quality_gap" not in decision
assert "selective:target=rescue" in decision["signals"]
def test_learned_policy_rejects_another_classifier_feature_schema() -> None:
config: Final = _config().llm_v2_config
assert config is not None
policy: Final = SelectivePolicy(
version="wrong-schema",
feature_schema="cap-v1",
target="rescue",
threshold=0.05,
heads=(SelectiveHead(constant=1),),
)
with pytest.raises(ValidationError, match="v2-v1"):
LLMV2Config.model_validate({**config.model_dump(), "selective_policy": policy})
def _config(**overrides: object) -> ComplexityRouterConfig:
@ -64,6 +121,54 @@ def _response(content: str) -> ModelResponse:
return response
def test_task_calibration_changes_only_matching_demands_and_survives_serialization() -> None:
base: Final = _config().llm_v2_config
assert base is not None
config: Final = LLMV2Config.model_validate(
{
**base.model_dump(),
"calibration": {
"version": "sonnet-opus-high-test",
"prompt_version": "llm-v2-1",
"efficient": {
"slope": 1.0,
"intercept": 0.0,
"offsets": [{"feature": "scope:coupled", "intercept": -2.0}],
},
"capable": {"slope": 1.0, "intercept": 0.0},
},
}
)
restored: Final = LLMV2Config.model_validate_json(config.model_dump_json())
coupled: Final = restored.classify(_verdict())
localized: Final = restored.classify(
LLMV2Verdict.model_validate(
{
**_verdict().model_dump(),
"demands": {"reasoning": "multistep", "scope": "localized", "specification": "clear"},
}
)
)
assert not coupled.use_efficient
assert localized.use_efficient
assert localized.efficient == pytest.approx(0.9)
assert coupled.capable == localized.capable == pytest.approx(0.92)
assert coupled.efficient == pytest.approx(0.5491469396)
@pytest.mark.parametrize(
"offsets",
[
[{"feature": "scope:coupled", "intercept": 1.0}] * 2,
[{"feature": "scope:invented", "intercept": 1.0}],
[{"feature": "scope:coupled", "intercept": float("nan")}],
],
)
def test_task_calibration_rejects_unusable_offsets(offsets: list[dict[str, object]]) -> None:
with pytest.raises(ValidationError):
LLMV2ProbabilityCalibration.model_validate({"slope": 1.0, "intercept": 0.0, "offsets": offsets})
def _router(content: str, config: ComplexityRouterConfig | None = None) -> tuple[ComplexityRouter, MagicMock]:
client: Final = MagicMock(spec=Router)
client.acompletion = AsyncMock(return_value=_response(content))

View file

@ -0,0 +1,145 @@
import math
from typing import Final
import pytest
from pydantic import ValidationError
from litellm.router_strategy.complexity_router.selective_policy import SelectiveHead, SelectivePolicy, verdict_features
def test_paired_policy_distinguishes_strong_only_from_cheap_only_success() -> None:
rescue: Final = SelectivePolicy(
version="test",
feature_schema="v2-v1",
target="paired",
threshold=0.05,
heads=(SelectiveHead(constant=1),),
)
cheap_only: Final = rescue.model_copy(update={"heads": (SelectiveHead(constant=2),)})
assert not rescue.evaluate({}, 0.9, 0.9).use_efficient
assert cheap_only.evaluate({}, 0.1, 0.9).use_efficient
def test_multiclass_head_maps_missing_outcomes_and_handles_large_logits() -> None:
head: Final = SelectiveHead(
features=("p_e",),
coefficients=((0.0,), (0.0,), (0.0,)),
intercept=(1000.0, 1000.0, 1000.0),
classes=(0, 1, 3),
)
assert head.probabilities({"p_e": 0.5}) == pytest.approx((1 / 3, 1 / 3, 0, 1 / 3))
def test_scalar_calibration_clips_extreme_probabilities() -> None:
head: Final = SelectiveHead(features=("logit_p",), coefficients=((1.0,),), intercept=(0.0,), classes=(0, 1))
policy: Final = SelectivePolicy(
version="test",
feature_schema="v2-v1",
target="scalar_calibration",
threshold=0.05,
heads=(head, head),
)
decision: Final = policy.evaluate({}, 0.0, 1.0)
assert decision.score == pytest.approx(0.99998)
assert not decision.use_efficient
def test_cost_policy_uses_incremental_dollars_and_inclusive_boundary() -> None:
cheap: Final = SelectiveHead(features=("p_e",), coefficients=((0.0,),), intercept=(0.0,))
strong: Final = SelectiveHead(features=("p_e",), coefficients=((0.0,),), intercept=(math.log(3.0),))
policy: Final = SelectivePolicy(
version="test",
feature_schema="cap-v1",
target="benefit_per_dollar",
threshold=0.5,
heads=(SelectiveHead(constant=1),),
costs=(cheap, strong),
)
assert policy.evaluate({}, 0.9, 0.9).use_efficient
assert not policy.model_copy(update={"threshold": 0.49}).evaluate({}, 0.9, 0.9).use_efficient
def test_overflow_falls_back_to_the_capable_model() -> None:
cost: Final = SelectiveHead(features=("p_e",), coefficients=((0.0,),), intercept=(1000.0,))
policy: Final = SelectivePolicy(
version="test",
feature_schema="cap-v1",
target="benefit_per_dollar",
threshold=1.0,
heads=(SelectiveHead(constant=1),),
costs=(cost, cost),
)
assert not policy.evaluate({}, 0.5, 0.9).use_efficient
@pytest.mark.parametrize("coefficient", [1e308, -1e308])
def test_overflowing_binary_logits_cannot_hide_as_equal_probabilities(coefficient: float) -> None:
head: Final = SelectiveHead(
features=("logit_p",), coefficients=((coefficient,),), intercept=(0.0,), classes=(0, 1)
)
policy: Final = SelectivePolicy(
version="test",
feature_schema="v2-v1",
target="scalar_calibration",
threshold=0.05,
heads=(head, head),
)
assert not policy.evaluate({}, 0.99999, 0.99999).use_efficient
def test_per_model_policy_uses_both_fitted_heads() -> None:
efficient: Final = SelectiveHead(
features=("p_e",), coefficients=((0.0,),), intercept=(math.log(4.0),), classes=(0, 1)
)
capable: Final = SelectiveHead(
features=("p_s",), coefficients=((0.0,),), intercept=(math.log(9.0),), classes=(0, 1)
)
policy: Final = SelectivePolicy(
version="test",
feature_schema="v2-v1",
target="per_model",
threshold=0.05,
heads=(efficient, capable),
)
decision: Final = policy.evaluate({"p_e": 0.99, "p_s": 0.5}, 0.99, 0.5)
assert decision.score == pytest.approx(0.1)
assert not decision.use_efficient
@pytest.mark.parametrize(
"values",
[
{"target": "per_model", "heads": [{"constant": 1}]},
{"target": "rescue", "heads": [{"constant": 3}]},
{"target": "rescue", "heads": [{"features": ["x"], "coefficients": [[1.0]], "intercept": [0.0]}]},
{"target": "benefit_per_dollar", "heads": [{"constant": 1}], "costs": [{"constant": 0}, {"constant": 1}]},
],
)
def test_policy_rejects_incompatible_quality_and_cost_heads(values: object) -> None:
from pydantic import TypeAdapter
from collections.abc import Mapping
fields: Final = TypeAdapter(Mapping[str, object]).validate_python(values)
with pytest.raises(ValidationError):
SelectivePolicy.model_validate({"version": "test", "feature_schema": "v2-v1", "threshold": 0.05, **fields})
@pytest.mark.parametrize(
"invalid",
[
{"features": ["x"], "coefficients": [[1.0, 2.0]], "intercept": [0.0], "classes": [0, 1]},
{"features": ["x"], "coefficients": [[1.0]], "intercept": [0.0], "classes": [1, 1]},
{"constant": 1, "features": ["x"]},
{"features": ["x"], "coefficients": [[float("nan")]], "intercept": [0.0], "classes": [0, 1]},
],
)
def test_invalid_coefficients_are_rejected_before_routing(invalid: object) -> None:
with pytest.raises(ValidationError):
SelectiveHead.model_validate(invalid)
def test_feature_contract_preserves_unknown_categories_without_task_identifiers() -> None:
values: Final = verdict_features("A recursive Unicode parser", {"p_e": 0.7, "reasoning:unknown": 1.0})
assert values["concept:recursive"] == values["concept:unicode"] == values["concept:parser"] == 1.0
assert values["reasoning:unknown"] == 1.0
assert "task_id" not in values

View file

@ -29559,6 +29559,16 @@ export interface components {
/** Version */
version: string;
};
/** LLMV2CalibrationOffset */
LLMV2CalibrationOffset: {
/**
* Feature
* @enum {string}
*/
feature: "reasoning:routine" | "reasoning:multistep" | "reasoning:open_ended" | "reasoning:unknown" | "scope:localized" | "scope:coupled" | "scope:broad" | "scope:unknown" | "specification:clear" | "specification:ambiguous" | "specification:unknown" | "verification:relevant" | "verification:partial" | "verification:unavailable" | "verification:unknown";
/** Intercept */
intercept: number;
};
/** LLMV2Config */
LLMV2Config: {
calibration?: components["schemas"]["LLMV2Calibration"] | null;
@ -29576,6 +29586,8 @@ export interface components {
* @default SIMPLE
*/
efficient_tier: string;
/** Empirical Supplement */
empirical_supplement?: string | null;
/** Harness */
harness: string;
/**
@ -29594,11 +29606,17 @@ export interface components {
* @enum {string}
*/
response_format: "json_schema" | "json_object";
selective_policy?: components["schemas"]["SelectivePolicy"] | null;
};
/** LLMV2ProbabilityCalibration */
LLMV2ProbabilityCalibration: {
/** Intercept */
intercept: number;
/**
* Offsets
* @default []
*/
offsets: components["schemas"]["LLMV2CalibrationOffset"][];
/** Slope */
slope: number;
};
@ -37434,6 +37452,55 @@ export interface components {
/** Timeout */
timeout?: number | null;
};
/** SelectiveHead */
SelectiveHead: {
/**
* Classes
* @default []
*/
classes: (0 | 1 | 2 | 3)[];
/**
* Coefficients
* @default []
*/
coefficients: number[][];
/** Constant */
constant?: (0 | 1 | 2 | 3) | null;
/**
* Features
* @default []
*/
features: string[];
/**
* Intercept
* @default []
*/
intercept: number[];
};
/** SelectivePolicy */
SelectivePolicy: {
/**
* Costs
* @default []
*/
costs: components["schemas"]["SelectiveHead"][];
/**
* Feature Schema
* @enum {string}
*/
feature_schema: "cap-v1" | "v2-v1";
/** Heads */
heads: components["schemas"]["SelectiveHead"][];
/**
* Target
* @enum {string}
*/
target: "paired" | "rescue" | "per_model" | "scalar_calibration" | "benefit_per_dollar";
/** Threshold */
threshold: number;
/** Version */
version: string;
};
/**
* ShadowEvalJobResponse
* @description A shadow-eval job over one or more targets, each with its own budget and stop state;