diff --git a/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md b/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md index 1f641d6bb6b..70cb0b329dc 100644 --- a/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md +++ b/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md @@ -28,4 +28,17 @@ All policies and family controls were frozen before final outcomes. These are ta Two Luna solver requests interrupted by host sleep have unknown charges. An earlier unavailable DNA-assembly review has one additional unmetered connection-timeout request. Thus Luna-only comparisons contain two unmetered requests, and review cascades contain three. The report marks affected costs as lower bounds and savings as upper bounds. The three upfront comparisons quoted with exact costs above are fully metered -The additional Sonnet/Opus controls are still running. Their completion will extend the comparison; it cannot change these frozen Luna/Sol results +The additional Sonnet/Opus controls are complete. All 500 final attempts and grades are present, and the complete reporter verified that the previously published Luna/Sol results and calibration diagnostics are unchanged + +| Fixed model | Solved | Recorded inference cost | +|---|---:|---:| +| Luna | 84/125 | At least $2.6531 | +| Sol | 87/125 | $20.8964 | +| Sonnet 5 | 81/125 | At least $37.9961 | +| Opus 5 | 99/125 | At least $46.0550 | + +Opus solved 22/25 SWE-bench tasks at $10.5296, compared with Sol's 18/25 at $10.7132. On Terminal-Bench it solved 21/25 at at least $32.9157, compared with Sol's 18/25 at $8.4468. These are observed results under the frozen harness and budgets, not unrestricted model rankings. Across all tasks, even perfect selection between the saved Luna and Sol attempts reaches only 95/125, below Opus's 99/125. Changing the candidate model pair is a separate training and evaluation question + +All four models had an 8,192-token response cap and high reasoning effort. Corrected Sonnet and Opus attempts still incurred 116 and 83 length stops with no action or visible text, costing $10.3656 and $17.9127 respectively. Those responses remain in the results. Increasing the response cap could change quality and spending, but this experiment did not test that change. Host sleep and unresolved transport charges also prevent latency conclusions and exact affected cost comparisons + +The practical next experiment is to collect substantially more paired agentic training examples where Sol rescues Luna, then evaluate rescue prediction and verification on fresh tasks. The present fitting set contains only two such agentic cases. The current final set is now diagnostic evidence and must not be reused as an unseen test after training on its failures. Better average probability calibration alone did not deliver the requested selective-rescue precision diff --git a/cookbook/auto_router_selective_training/PROTOCOL.md b/cookbook/auto_router_selective_training/PROTOCOL.md index c6f7b3cf596..562b8e01e79 100644 --- a/cookbook/auto_router_selective_training/PROTOCOL.md +++ b/cookbook/auto_router_selective_training/PROTOCOL.md @@ -108,3 +108,23 @@ SWE solver containers have a 45-minute configured lifetime in both the original The final primary billing audit also found one earlier DNA-assembly review ConnectTimeout with no successful response ledger. This is separate from the two host-sleep solver interruptions. Research accounting now discovers transport-only ledgers as well as billed-response ledgers. Review-cascade cost comparisons therefore carry three unmetered requests, while Luna-only comparisons carry two + + +## Lid-closed host sleep, September 16 UTC + +During the remaining Anthropic Terminal controls, the host entered Clamshell Sleep on AC power at full battery and repeatedly returned to Maintenance Sleep. The workers survived; no whole attempt was restarted and no power setting changed. At the 06:12 UTC inspection, fix-ocaml-gc and model-extraction-relu-logits Opus trials remained active. The latter had a pending gateway request whose billing remains unresolved until the response or transport ledger completes. Evidence is retained in infrastructure_interruptions/20260916-clamshell-sleep. Host wall durations include these pauses and are not used as routing features or latency comparisons + + +## DNS recovery and exact Terminal image reuse, September 16 UTC + +After host sleep, two active Anthropic solver attempts ended with gateway DNS ConnectError and 29 queued controls failed to resolve Docker Hub during environment setup. The runner exited. All 31 failed trial records and exported partial attempts were archived in infrastructure_interruptions/20260916-dns-outage. The two interrupted attempts incurred $1.35120725 in known inference cost and 20 requests with unknown billing. These costs are research overhead, not policy inference cost. Connectivity probes verified host DNS, authenticated gateway access and Docker DNS before resuming + +Recovery validation found that eleven completed Anthropic controls had rebuilt different filesystem layers from the retained Luna/Sol images. Every mismatched control was archived and scheduled again, regardless of score. These controls cost $11.05990725, retained separately as research overhead. The 25 Luna/Sol image pairs match in filesystem layers and runtime image configuration. All retained primary attempts and fitted policies are unchanged + +The Terminal environment now supports an explicit map of the exact saved Sol image for each final task. It verifies local image identity and filesystem layers before startup and verifies the started container image before agent inference. This replaces rebuilds from mutable base tags with the already-cached peer image. The map is terminal_image_pins.json. Task files, solver prompts, budgets, network isolation, and grading rules remain unchanged. The recovery schedules exactly 42 missing controls and preserves 58 valid Terminal results. No live trial was interrupted for this correction + +The model-extraction-relu-logits Opus attempt completed normally after the model wrapper retried the previously pending ReadError. It is retained, with that request's unknown charge included in its cost bounds. The entire attempt was not rerun. Host wall durations remain unsuitable for latency comparisons + +## Final completion + +All 500 held-out model attempts and grades completed on September 17, 2026 UTC. The final audit checked 950 completed development/final attempts and 1,400 forecasts, including paired image identity, Git isolation, archived file hashes and recorded costs. Full reporting verified that primary Luna/Sol results and calibration diagnostics are identical to the earlier primary release. No fitting, policy selection or boundary change followed the final outcomes diff --git a/cookbook/auto_router_selective_training/README.md b/cookbook/auto_router_selective_training/README.md index 674e2cbfeff..0b38d1d46a7 100644 --- a/cookbook/auto_router_selective_training/README.md +++ b/cookbook/auto_router_selective_training/README.md @@ -2,7 +2,7 @@ These optional profiles learn when GPT-5.6 Sol adds a successful solve over GPT-5.6 Luna. They use one existing classifier response and a deterministic local prediction, with no second classifier call. They replace the earlier ROI cookbook, whose benchmark results were invalidated and are not the source of these fits -Training uses 160 fresh paired tasks and selection uses 64 separate validation tasks across SWE-bench, Terminal-Bench, MBPP+ and LiveCodeBench. The fitted policies were frozen before the 125-task held-out evaluation. The complete 125-task Luna/Sol comparison is available in PRIMARY_FINDINGS.md and PRIMARY_REPORT.md. Additional Sonnet/Opus controls are still running +Training uses 160 fresh paired tasks and selection uses 64 separate validation tasks across SWE-bench, Terminal-Bench, MBPP+ and LiveCodeBench. The fitted policies were frozen before the 125-task held-out evaluation. The complete 125-task Luna/Sol comparison is available in PRIMARY_FINDINGS.md and PRIMARY_REPORT.md. The additional Sonnet/Opus controls are complete in REPORT.md `strong-success-retention` minimizes validation cost while preserving every Sol-only success and meeting Sol quality within each benchmark. `quality-first-under-sol-budget` maximizes validation solves under Sol's total inference cost and permits task swaps. These names describe validation objectives, not guarantees on new requests @@ -21,15 +21,19 @@ The twelve prespecified upfront family controls are also available in `diagnosti These profiles were selected before these results. They are not refitted or selected again on the held-out tasks -| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost Sol solves | +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | |---|---:|---:|---:|---:| | Luna only | 84/125 | ≥$2.6531 | ≤87.3% | +8 / -11 | | Sol only | 87/125 | $20.8964 | 0.0% | +0 / -0 | | Capability baseline | 85/125 | $6.7642 | 67.6% | +8 / -10 | | Capability upfront, retention objective | 86/125 | $13.8156 | 33.9% | +1 / -2 | | Capability upfront, quality objective | 88/125 | $18.5237 | 11.4% | +5 / -4 | -| ablation_cap_original_upfront_per_model | 88/125 | $20.3797 | 2.5% | +1 / -0 | +| Per-model control (exploratory) | 88/125 | $20.3797 | 2.5% | +1 / -0 | Costs use recorded gateway bills. A ≥ cost and ≤ savings mark unmetered transport requests: cost is a lower bound and savings versus fully metered Sol is an upper bound. The JSON counts these requests per policy and task. They are not assumed free. -The per-model family control is an exploratory comparison among forty prespecified variants, not a newly selected winner. No validation-selected profile combines preservation of all Sol successes and lower cost across all 125 tasks. Task-level replay does not measure per-turn routing or Sol continuation from a Luna patch +Costs include the classifier. These are task-pinned replays of fresh independent attempts, with one attempt per model per task. No guarantee of quality retention follows from a 125-task sample. The aggregate combines different benchmarks; inspect each benchmark before using a profile + +The per-model control is an exploratory comparison among forty prespecified variants. No validation-selected profile combines preservation of all Sol successes and lower cost across all 125 tasks. See PRIMARY_FINDINGS.md for the interpretation + +![Observed quality and known inference cost](quality_cost.png) diff --git a/cookbook/auto_router_selective_training/REPORT.md b/cookbook/auto_router_selective_training/REPORT.md new file mode 100644 index 00000000000..bcd8470de8f --- /dev/null +++ b/cookbook/auto_router_selective_training/REPORT.md @@ -0,0 +1,204 @@ +# Clean selective-router evaluation + +Policies were frozen before final solver outcomes. Upfront routing uses one classifier response; the post-attempt variants also use Luna output, optional public-example checks and a Luna review. These are task-level replays over matched independent attempts, not live per-turn routing or continuation from a cheaper model's patch + +The configured original-card baselines receive the same model identity and execution conditions as the trained variants. Capability uses base_threshold=0.5 and threshold_step=0.1; V2 uses max_quality_gap=0.05. The empirical-card baselines change the card supplement while retaining those boundaries + +Retention profiles were selected to preserve Sol successes at lower validation cost. Quality profiles were selected to maximize validation solves within the Sol budget. These names describe selection objectives, not guarantees about the held-out results. Exact policy IDs remain in final_results.json + +Costs use recorded gateway bills. A ≥ cost and ≤ savings mark unmetered transport requests: cost is a lower bound and savings versus fully metered Sol is an upper bound. The JSON counts these requests per policy and task. They are not assumed free. + +## Observed opportunity for routing + +The Sol-only column counts the rescues a perfect decision would capture. The hindsight cascade pays for Luna on every task and adds Sol only for those rescues. Tasks where both attempts failed cannot be recovered by choosing between these two saved outputs. This is a bound on this replay, not a guarantee about future attempts + +| Benchmark | Both solve | Luna only | Sol only | Both fail | Hindsight cascade cost | +|---|---:|---:|---:|---:|---:| +| all | 76 | 8 | 11 | 30 | ≥$5.2653 | +| lcb | 11 | 2 | 3 | 9 | $0.3657 | +| mbpp | 35 | 3 | 2 | 10 | $0.0361 | +| swe | 16 | 0 | 2 | 7 | $1.5478 | +| terminal | 14 | 3 | 4 | 4 | ≥$3.3157 | + +## all + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 84/125 | ≥$2.6531 | ≤87.3% | +8 / -11 | +| Sol only | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| Sonnet 5 only | 81/125 | ≥$37.9961 | ≤-81.8% | +8 / -14 | +| Opus 5 only | 99/125 | ≥$46.0550 | ≤-120.4% | +19 / -7 | +| Capability upfront, retention objective | 86/125 | $13.8156 | 33.9% | +1 / -2 | +| Capability upfront, quality objective | 88/125 | $18.5237 | 11.4% | +5 / -4 | +| V2 upfront, retention objective | 85/125 | $14.0036 | 33.0% | +2 / -4 | +| V2 upfront, quality objective | 86/125 | $18.1999 | 12.9% | +4 / -5 | +| Capability baseline | 85/125 | $6.7642 | 67.6% | +8 / -10 | +| Capability with learned cards only | 84/125 | $7.2496 | 65.3% | +7 / -10 | +| V2 baseline | 84/125 | ≥$17.0029 | ≤18.6% | +3 / -6 | +| V2 with learned cards only | 85/125 | ≥$14.6487 | ≤29.9% | +6 / -8 | +| Capability review, retention objective | 81/125 | ≥$7.5830 | ≤63.7% | +0 / -6 | +| Capability review, quality objective | 87/125 | ≥$10.1853 | ≤51.3% | +7 / -7 | +| V2 review, retention objective | 88/125 | ≥$20.9754 | ≤-0.4% | +1 / -0 | +| V2 review, quality objective | 88/125 | ≥$8.6493 | ≤58.6% | +8 / -7 | +| Hindsight direct routing (not deployable) | 95/125 | ≥$4.5190 | ≤78.4% | +8 / -0 | +| Hindsight Luna-first routing (not deployable) | 95/125 | ≥$5.2653 | ≤74.8% | +8 / -0 | + +## lcb + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 13/25 | $0.1165 | 92.4% | +2 / -3 | +| Sol only | 14/25 | $1.5299 | 0.0% | +0 / -0 | +| Sonnet 5 only | 9/25 | $1.3790 | 9.9% | +1 / -6 | +| Opus 5 only | 14/25 | $2.3849 | -55.9% | +2 / -2 | +| Capability upfront, retention objective | 14/25 | $1.5476 | -1.2% | +0 / -0 | +| Capability upfront, quality objective | 14/25 | $0.9200 | 39.9% | +1 / -1 | +| V2 upfront, retention objective | 14/25 | $1.3923 | 9.0% | +0 / -0 | +| V2 upfront, quality objective | 12/25 | $1.1104 | 27.4% | +0 / -2 | +| Capability baseline | 13/25 | $0.2918 | 80.9% | +2 / -3 | +| Capability with learned cards only | 12/25 | $0.4860 | 68.2% | +1 / -3 | +| V2 baseline | 12/25 | $0.9924 | 35.1% | +0 / -2 | +| V2 with learned cards only | 13/25 | $0.4799 | 68.6% | +2 / -3 | +| Capability review, retention objective | 14/25 | $1.6846 | -10.1% | +0 / -0 | +| Capability review, quality objective | 14/25 | $1.3257 | 13.4% | +2 / -2 | +| V2 review, retention objective | 15/25 | $1.6076 | -5.1% | +1 / -0 | +| V2 review, quality objective | 14/25 | $1.0584 | 30.8% | +2 / -2 | +| Luna with public-test escalation | 15/25 | $1.3645 | 10.8% | +2 / -1 | +| Hindsight direct routing (not deployable) | 16/25 | $0.3436 | 77.5% | +2 / -0 | +| Hindsight Luna-first routing (not deployable) | 16/25 | $0.3657 | 76.1% | +2 / -0 | + +## mbpp + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 38/50 | $0.0174 | 91.6% | +3 / -2 | +| Sol only | 37/50 | $0.2064 | 0.0% | +0 / -0 | +| Sonnet 5 only | 36/50 | $0.0988 | 52.1% | +2 / -3 | +| Opus 5 only | 42/50 | $0.2249 | -9.0% | +7 / -2 | +| Capability upfront, retention objective | 37/50 | $0.2180 | -5.6% | +0 / -0 | +| Capability upfront, quality objective | 38/50 | $0.0444 | 78.5% | +3 / -2 | +| V2 upfront, retention objective | 37/50 | $0.2298 | -11.3% | +0 / -0 | +| V2 upfront, quality objective | 38/50 | $0.0465 | 77.4% | +3 / -2 | +| Capability baseline | 38/50 | $0.0290 | 85.9% | +3 / -2 | +| Capability with learned cards only | 38/50 | $0.0284 | 86.2% | +3 / -2 | +| V2 baseline | 38/50 | $0.0562 | 72.8% | +3 / -2 | +| V2 with learned cards only | 38/50 | $0.0396 | 80.8% | +3 / -2 | +| Capability review, retention objective | 37/50 | $0.2503 | -21.3% | +0 / -0 | +| Capability review, quality objective | 38/50 | $0.0434 | 79.0% | +3 / -2 | +| V2 review, retention objective | 37/50 | $0.2441 | -18.3% | +0 / -0 | +| V2 review, quality objective | 38/50 | $0.0488 | 76.4% | +3 / -2 | +| Luna with public-test escalation | 38/50 | $0.0240 | 88.4% | +2 / -1 | +| Hindsight direct routing (not deployable) | 40/50 | $0.0353 | 82.9% | +3 / -0 | +| Hindsight Luna-first routing (not deployable) | 40/50 | $0.0361 | 82.5% | +3 / -0 | + +## swe + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 16/25 | $0.6371 | 94.1% | +0 / -2 | +| Sol only | 18/25 | $10.7132 | 0.0% | +0 / -0 | +| Sonnet 5 only | 19/25 | $11.0537 | -3.2% | +2 / -1 | +| Opus 5 only | 22/25 | $10.5296 | 1.7% | +4 / -0 | +| Capability upfront, retention objective | 18/25 | $6.2493 | 41.7% | +0 / -0 | +| Capability upfront, quality objective | 18/25 | $9.7546 | 8.9% | +0 / -0 | +| V2 upfront, retention objective | 16/25 | $5.3537 | 50.0% | +0 / -2 | +| V2 upfront, quality objective | 17/25 | $8.8718 | 17.2% | +0 / -1 | +| Capability baseline | 16/25 | $1.3893 | 87.0% | +0 / -2 | +| Capability with learned cards only | 16/25 | $1.6826 | 84.3% | +0 / -2 | +| V2 baseline | 17/25 | $9.1304 | 14.8% | +0 / -1 | +| V2 with learned cards only | 17/25 | $7.4181 | 30.8% | +0 / -1 | +| Capability review, retention objective | 16/25 | $2.4752 | 76.9% | +0 / -2 | +| Capability review, quality objective | 16/25 | $0.9753 | 90.9% | +0 / -2 | +| V2 review, retention objective | 18/25 | $9.1263 | 14.8% | +0 / -0 | +| V2 review, quality objective | 16/25 | $0.9775 | 90.9% | +0 / -2 | +| Hindsight direct routing (not deployable) | 18/25 | $1.4342 | 86.6% | +0 / -0 | +| Hindsight Luna-first routing (not deployable) | 18/25 | $1.5478 | 85.6% | +0 / -0 | + +## terminal + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 17/25 | ≥$1.8822 | ≤77.7% | +3 / -4 | +| Sol only | 18/25 | $8.4468 | 0.0% | +0 / -0 | +| Sonnet 5 only | 17/25 | ≥$25.4647 | ≤-201.5% | +3 / -4 | +| Opus 5 only | 21/25 | ≥$32.9157 | ≤-289.7% | +6 / -3 | +| Capability upfront, retention objective | 17/25 | $5.8008 | 31.3% | +1 / -2 | +| Capability upfront, quality objective | 18/25 | $7.8047 | 7.6% | +1 / -1 | +| V2 upfront, retention objective | 18/25 | $7.0279 | 16.8% | +2 / -2 | +| V2 upfront, quality objective | 19/25 | $8.1712 | 3.3% | +1 / -0 | +| Capability baseline | 18/25 | $5.0541 | 40.2% | +3 / -3 | +| Capability with learned cards only | 18/25 | $5.0526 | 40.2% | +3 / -3 | +| V2 baseline | 17/25 | ≥$6.8238 | ≤19.2% | +0 / -1 | +| V2 with learned cards only | 17/25 | ≥$6.7112 | ≤20.5% | +1 / -2 | +| Capability review, retention objective | 14/25 | ≥$3.1729 | ≤62.4% | +0 / -4 | +| Capability review, quality objective | 19/25 | ≥$7.8410 | ≤7.2% | +2 / -1 | +| V2 review, retention objective | 18/25 | ≥$9.9975 | ≤-18.4% | +0 / -0 | +| V2 review, quality objective | 20/25 | ≥$6.5646 | ≤22.3% | +3 / -1 | +| Hindsight direct routing (not deployable) | 21/25 | ≥$2.7059 | ≤68.0% | +3 / -0 | +| Hindsight Luna-first routing (not deployable) | 21/25 | ≥$3.3157 | ≤60.7% | +3 / -0 | + +## Frozen family comparisons + +Each row was selected on development data before final inference. These are diagnostic comparisons, not a new winner selection on the held-out tasks. An always-Sol fallback means that no candidate in that requested family met the validation criterion; it bypasses the classifier, Luna attempt and reviewer + +| Classifier | Requested stage | Cards | Fitted target | Boundary | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---|---|---|---:|---:|---:|---:|---:| +| cap | upfront | empirical | benefit_per_dollar | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | empirical | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | empirical | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | empirical | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | original | benefit_per_dollar | 1 | 86/125 | $13.8156 | 33.9% | +1 / -2 | +| cap | upfront | original | paired | -0.02 | 88/125 | $20.8563 | 0.2% | +1 / -0 | +| cap | upfront | original | per_model | -0.05 | 88/125 | $20.3797 | 2.5% | +1 / -0 | +| cap | upfront | original | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | original | rescue | 0.03 | 88/125 | $20.7799 | 0.6% | +1 / -0 | +| cap | upfront | original | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | original | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | original | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | original | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | empirical | benefit_per_dollar | 0.5 | 87/125 | $19.2534 | 7.9% | +2 / -2 | +| v2 | upfront | empirical | paired | 0.01 | 85/125 | $19.7030 | 5.7% | +1 / -3 | +| v2 | upfront | empirical | per_model | 0.01 | 86/125 | $19.7351 | 5.6% | +1 / -2 | +| v2 | upfront | empirical | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | empirical | rescue | 0.05 | 86/125 | $18.0048 | 13.8% | +2 / -3 | +| v2 | upfront | empirical | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | review | empirical | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | review | empirical | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | review | empirical | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | original | benefit_per_dollar | 1 | 85/125 | $14.0036 | 33.0% | +2 / -4 | +| v2 | upfront | original | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | original | per_model | 0 | 86/125 | $20.1971 | 3.3% | +0 / -1 | +| v2 | upfront | original | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | original | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | original | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | review | original | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | empirical | benefit_per_dollar | 0.1 | 82/125 | ≥$16.8075 | ≤19.6% | +0 / -5 | +| cap | review | original | benefit_per_dollar | 0.25 | 81/125 | ≥$7.5830 | ≤63.7% | +0 / -6 | +| v2 | review | empirical | benefit_per_dollar | 0.25 | 83/125 | ≥$16.3243 | ≤21.9% | +0 / -4 | +| v2 | review | original | benefit_per_dollar | 0.1 | 85/125 | ≥$19.8241 | ≤5.1% | +0 / -2 | +| v2 | review | original | per_model | -0.02 | 88/125 | ≥$20.9754 | ≤-0.4% | +1 / -0 | +| v2 | review | original | rescue | 0.02 | 86/125 | ≥$23.4088 | ≤-12.0% | +0 / -1 | + +## Response budget effects + +Every solver uses the same 8,192-token response cap and high reasoning effort. Length stops that produce neither a tool call nor visible text still consume inference cost. They are retained as model behavior under this budget, not retried as infrastructure failures. These counts describe completed final attempts across all four benchmarks + +| Model | Responses | Length stops | Length stops without action or text | Cost of those empty length stops | +|---|---:|---:|---:|---:| +| gpt-5.6-luna | 1341 | 50 | 42 | $0.4217 | +| gpt-5.6-sol | 819 | 7 | 4 | $0.6676 | +| claude-sonnet-5 | 1703 | 124 | 116 | $10.3656 | +| claude-opus-5 | 870 | 87 | 83 | $17.9127 | + +## Experiment spending + +Recorded study inference cost is $225.5557. This includes training, validation, final solver/judge/reviewer calls, superseded development forecasts, archived infrastructure-interrupted attempts, superseded Anthropic agentic controls, controls with mismatched Terminal images, and adapter round-trip probes. It is separate from the deployed-policy costs above. Unknown transport billing, setup, live-proxy QA, local Docker and CPU expenses are not included. See research_costs.json for the breakdown + +## Interpretation limits + +The oracle is a hindsight reference using known costs. Cost intervals use recorded bills and exclude unmetered requests. A small sample with one attempt per model cannot establish the same routing advantage across workloads or repeated runs. Paired bootstrap intervals and exact discordant-pair tests in the JSON are exploratory; policy selection searched many development candidates. Inference costs include classifier/reviewer calls and discarded Luna attempts on escalation. Local checker CPU time and experiment training/transport overhead are separate. Terminal-Bench and SWE-bench use documented offline/native-image eligibility adaptations diff --git a/cookbook/auto_router_selective_training/TRAINING.md b/cookbook/auto_router_selective_training/TRAINING.md new file mode 100644 index 00000000000..1113d7d8333 --- /dev/null +++ b/cookbook/auto_router_selective_training/TRAINING.md @@ -0,0 +1,49 @@ +# Training and selection record + +The learning target is model complementarity: route cheaply when Luna can do the job, and use Sol when its extra cost is likely to produce an additional solve. Better marginal probability accuracy alone is not the selection objective + +## Paired examples + +| Split | Benchmark | Tasks | Both fail | Sol only | Luna only | Both solve | +|---|---|---:|---:|---:|---:|---:| +| train | lcb | 50 | 14 | 9 | 3 | 24 | +| train | mbpp | 80 | 7 | 2 | 3 | 68 | +| train | swe | 25 | 5 | 1 | 0 | 19 | +| train | terminal | 5 | 0 | 1 | 0 | 4 | +| validation | lcb | 20 | 2 | 6 | 2 | 10 | +| validation | mbpp | 29 | 4 | 2 | 2 | 21 | +| validation | swe | 10 | 1 | 1 | 1 | 7 | +| validation | terminal | 5 | 2 | 0 | 1 | 2 | + +There are 13 Sol-only examples in fitting and 9 in validation. Only two fitting examples are agentic rescues, one SWE-bench and one Terminal-Bench. Terminal validation has no Sol-only example. That limits evidence about rescue detection on unseen terminal tasks; zero validation losses there cannot establish rescue recall + +## Variations + +One training-only synthesis produces the empirical card supplement. It uses measured outcomes and fallible classifier/reviewer descriptions, without task identities, solutions or validation-driven rewrites. Original cards are a separate control + +The deterministic head families are raw threshold tuning, scalar probability calibration, task-dependent per-model calibration, four-way paired outcomes, Sol-only rescue probability and paired benefit per predicted incremental dollar. Regularization and fixed threshold grids are selected using validation only. Upfront inputs come from one classifier response. Post-attempt heads additionally use Luna output, a fallible Luna review and task-supplied public checks where available + +The strong-success-retention objective minimizes validation cost while losing no Sol-only successes and preserving Sol aggregate quality in each benchmark. The quality-first-under-sol-budget objective maximizes validation solves under Sol total cost, permitting task swaps. Always-Sol is an explicit fallback. None of these objective names guarantees performance on unseen tasks + +## Frozen selected profiles + +| Classifier | Stage | Objective | Cards | Family | Threshold | Validation solved | Validation cost | Sol-only losses | +|---|---|---|---|---|---:|---:|---:|---:| +| cap | upfront | strong_success_retention | original | benefit_per_dollar | 1.0 | 50/64 | $5.8544 | 0 | +| cap | upfront | quality_first_under_sol_budget | original | scalar_calibration | 0.1 | 52/64 | $5.4692 | 2 | +| cap | review | strong_success_retention | original | benefit_per_dollar | 0.25 | 49/64 | $5.6530 | 0 | +| cap | review | quality_first_under_sol_budget | empirical | paired | 0.075 | 53/64 | $3.7201 | 2 | +| cap | either | strong_success_retention | original | benefit_per_dollar | 0.25 | 49/64 | $5.6530 | 0 | +| cap | either | quality_first_under_sol_budget | empirical | paired | 0.075 | 53/64 | $3.7201 | 2 | +| v2 | upfront | strong_success_retention | original | benefit_per_dollar | 1.0 | 51/64 | $5.4640 | 0 | +| v2 | upfront | quality_first_under_sol_budget | original | scalar_calibration | 0.075 | 53/64 | $5.4524 | 2 | +| v2 | review | strong_success_retention | original | per_model | -0.02 | 51/64 | $6.7043 | 0 | +| v2 | review | quality_first_under_sol_budget | empirical | per_model | 0.15 | 53/64 | $3.3074 | 2 | +| v2 | either | strong_success_retention | original | benefit_per_dollar | 1.0 | 51/64 | $5.4640 | 0 | +| v2 | either | quality_first_under_sol_budget | empirical | per_model | 0.15 | 53/64 | $3.3074 | 2 | + +These are development results. Compare held-out results in REPORT.md before interpreting them as an improvement. Both selected upfront classifier profiles keep the original card and use fitted local decisions. Empirical cards remain a measured variation and are selected by some review cascades + +There are 1,892 policy candidates, 118 serialized models, 12 selected profiles and 40 prespecified family ablations. An independent refit reproduced every coefficient, candidate statistic and selection exactly. The selection hash and source/data hashes are recorded in selection_frozen.json; reproduction evidence is in training_reproduction.json + +The threshold unit depends on the target. Marginal and paired gaps are probability differences, rescue is a Sol-only probability, and benefit_per_dollar is expected additional success per predicted dollar. A threshold of 0.05 is not interchangeable across these families diff --git a/cookbook/auto_router_selective_training/benchmark_summary.json b/cookbook/auto_router_selective_training/benchmark_summary.json new file mode 100644 index 00000000000..c51b6e153bc --- /dev/null +++ b/cookbook/auto_router_selective_training/benchmark_summary.json @@ -0,0 +1,11707 @@ +{ + "protocol": "Fresh independent paired solver attempts; fixed task-level policy replay. Training and threshold selection use development only. The hindsight oracle uses final labels and is not deployable.", + "paired_outcomes": { + "all": { + "both_fail": 30, + "sol_only": 11, + "luna_only": 8, + "both_solve": 76 + }, + "lcb": { + "both_fail": 9, + "sol_only": 3, + "luna_only": 2, + "both_solve": 11 + }, + "mbpp": { + "both_fail": 10, + "sol_only": 2, + "luna_only": 3, + "both_solve": 35 + }, + "swe": { + "both_fail": 7, + "sol_only": 2, + "luna_only": 0, + "both_solve": 16 + }, + "terminal": { + "both_fail": 4, + "sol_only": 4, + "luna_only": 3, + "both_solve": 14 + } + }, + "results": { + "fixed_gpt-5.6-luna": { + "total": { + "tasks": 125, + "solved": 84, + "cost": 2.65313975, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.8730336810071551, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 11, + "paired_exact_p": 0.6476058959960938, + "quality_delta_bootstrap_95": [ + -0.088, + 0.048 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18976455058400002, + -0.10695287329400002 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.2849141529181544 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 0.1164804, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9238662330124501, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.079881704, + -0.03457668540000001 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.7075982261787133 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.017396, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9157072527813311, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0049195134000000005, + -0.002844990800000001 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.6371064399999999, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.940530936966828, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5143941064600001, + -0.31072451521000005 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 1.88215691, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.7771763251611454, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 4, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.24, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.37199028632000003, + -0.16931580716000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.6023646356164746 + ] + } + } + } + }, + "fixed_gpt-5.6-sol": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "fixed_claude-sonnet-5": { + "total": { + "tasks": 125, + "solved": 81, + "cost": 37.9961323, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.8183094403889355, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 14, + "paired_exact_p": 0.28627872467041016, + "quality_delta_bootstrap_95": [ + -0.12, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + 0.04082165667999999, + 0.24607147657999995 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 9, + "cost": 1.3789665, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.09868171645498136, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 6, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.4, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.023796100500000004, + 0.011813104999999997 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "mbpp": { + "tasks": 50, + "solved": 36, + "cost": 0.098846, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.5210392681319533, + "baseline_solved": 37, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.1, + 0.06 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0028876400000000003, + -0.0015387350000000007 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "swe": { + "tasks": 25, + "solved": 19, + "cost": 11.053656, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.031775107172044015, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16099999999998546 + ], + "mean_cost_delta_bootstrap_95": [ + -0.17330483480000003, + 0.24415650309999992 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 25.4646638, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -2.014695499777408, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 4, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.24, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + 0.29976725349999994, + 1.0925644450999998 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + } + } + }, + "fixed_claude-opus-5": { + "total": { + "tasks": 125, + "solved": 99, + "cost": 46.055013, + "unknown_billing_requests": 4, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -1.203968136913113, + "baseline_solved": 87, + "gained_sol_failures": 19, + "lost_sol_successes": 7, + "paired_exact_p": 0.028959274291992188, + "quality_delta_bootstrap_95": [ + 0.024, + 0.176 + ], + "mean_cost_delta_bootstrap_95": [ + 0.07898218682999997, + 0.34375395494999983 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 2.384855, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.5587858117682738, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0175889235, + 0.05212979624999998 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "mbpp": { + "tasks": 50, + "solved": 42, + "cost": 0.224865, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.08958890568670785, + "baseline_solved": 37, + "gained_sol_failures": 7, + "lost_sol_successes": 2, + "paired_exact_p": 0.1796875, + "quality_delta_bootstrap_95": [ + 0.0, + 0.22 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0002537405, + 0.0010139794999999997 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "swe": { + "tasks": 25, + "solved": 22, + "cost": 10.52956575, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.017144750100660522, + "baseline_solved": 18, + "gained_sol_failures": 4, + "lost_sol_successes": 0, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + 0.04, + 0.32 + ], + "mean_cost_delta_bootstrap_95": [ + -0.10402242440000004, + 0.10201380229999983 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "terminal": { + "tasks": 25, + "solved": 21, + "cost": 32.91572725, + "unknown_billing_requests": 4, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -2.896807575856376, + "baseline_solved": 18, + "gained_sol_failures": 6, + "lost_sol_successes": 3, + "paired_exact_p": 0.5078125, + "quality_delta_bootstrap_95": [ + -0.12, + 0.36 + ], + "mean_cost_delta_bootstrap_95": [ + 0.4561114184000001, + 1.5875788229999999 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + } + } + }, + "cap_upfront_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 13.815640460000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.33885087895018884, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.088707189458, + -0.029987009136000043 + ], + "strong_routes": 102, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.08823529411764706, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.041143831377718536, + 0.16089819225628654 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.54757188, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.011521911913115801, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0005666024999999994, + 0.0008554352799999992 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.21798, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.05622746831026859, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00020418355, + 0.00026104007500000006 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 6.24925476, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.4166793774164488, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.30450702318000006, + -0.07650369019000006 + ], + "strong_routes": 15, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.13333333333333333, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.016575913440083623, + 0.4046026966033721 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 5.80083382, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.31325432962870725, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18159462757, + -0.04581437822000005 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.020862525460092384, + 0.484137748685967 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "cap_upfront_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 18.52372147, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.11354510100923576, + "baseline_solved": 87, + "gained_sol_failures": 5, + "lost_sol_successes": 4, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.056 + ], + "mean_cost_delta_bootstrap_95": [ + -0.028897788182000003, + -0.010285258102000018 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.14, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.05819170034037209, + 0.2673960024970084 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 0.92004308, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.39864264312942166, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.04378357345, + -0.008312728250000003 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.25, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.03185402624994424, + 0.650855794412824 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.044412, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7848005582044424, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0043592915, + -0.0023561677250000003 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.841886116991581 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.75459163, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.08948271735046087, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07487978142000001, + -0.009760154500000014 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.1, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.012348527170294804, + 0.3169827140190823 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 7.80467476, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.07602479808909468, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.05543588585, + -0.0045051321900000055 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.15, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.032070937185463715, + 0.3789268265453139 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "v2_upfront_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 14.003645670000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.3298538672152522, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 4, + "paired_exact_p": 0.6875, + "quality_delta_bootstrap_95": [ + -0.056, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.08716027666, + -0.03046360189800001 + ], + "strong_routes": 100, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.07, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.028605288907438704, + 0.13891972845585732 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3922581500000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.08999404553369261, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.018160890399999997, + 0.0008559290000000007 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2297896, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.11345117649339076, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00045055567499999996, + 0.0004872066999999999 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 5.35368162, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.5002743502022768, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.33043864058, + -0.1220960099900001 + ], + "strong_routes": 10, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3084971078187608 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 7.0279163, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.16798321749599177, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.1264185382, + -0.011905893240000002 + ], + "strong_routes": 16, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.015513603815413892, + 0.38347623684926346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "v2_upfront_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 18.19994713, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.12903936064412225, + "baseline_solved": 87, + "gained_sol_failures": 4, + "lost_sol_successes": 5, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.056, + 0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.03653304908999999, + -0.009390257962000001 + ], + "strong_routes": 51, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 6, + "sol_route_rescue_precision": 0.11764705882352941, + "sol_only_recall": 0.5454545454545454, + "sol_route_rescue_precision_exact_95": [ + 0.044420141856314814, + 0.23867604726407454 + ], + "sol_only_recall_exact_95": [ + 0.23379359765934518, + 0.8325119059362929 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 12, + "cost": 1.1104201500000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.274208631165585, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0330455756, + -0.005133099700000001 + ], + "strong_routes": 10, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.002528578544461787, + 0.4450161170281954 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.0465402, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7744883125944877, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004358442625000001, + -0.00225688455 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 8.87179983, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.17188464880694931, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.14395534461999998, + -0.01956562445 + ], + "strong_routes": 19, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.05263157894736842, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.001331628804179424, + 0.2602806541952142 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 8.171186950000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.03263437053487084, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.023058544700000003, + -0.0017085895500000016 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.19047619047619047, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.054463568178406814, + 0.41906604118352786 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "v2_either_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 14.003645670000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.3298538672152522, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 4, + "paired_exact_p": 0.6875, + "quality_delta_bootstrap_95": [ + -0.056, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.08716027666, + -0.03046360189800001 + ], + "strong_routes": 100, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.07, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.028605288907438704, + 0.13891972845585732 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3922581500000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.08999404553369261, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.018160890399999997, + 0.0008559290000000007 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2297896, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.11345117649339076, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00045055567499999996, + 0.0004872066999999999 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 5.35368162, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.5002743502022768, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.33043864058, + -0.1220960099900001 + ], + "strong_routes": 10, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3084971078187608 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 7.0279163, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.16798321749599177, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.1264185382, + -0.011905893240000002 + ], + "strong_routes": 16, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.015513603815413892, + 0.38347623684926346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_per_model": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_raw": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_scalar_calibration": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_review_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_review_per_model": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_review_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 13.815640460000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.33885087895018884, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.088707189458, + -0.029987009136000043 + ], + "strong_routes": 102, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.08823529411764706, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.041143831377718536, + 0.16089819225628654 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.54757188, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.011521911913115801, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0005666024999999994, + 0.0008554352799999992 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.21798, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.05622746831026859, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00020418355, + 0.00026104007500000006 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 6.24925476, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.4166793774164488, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.30450702318000006, + -0.07650369019000006 + ], + "strong_routes": 15, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.13333333333333333, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.016575913440083623, + 0.4046026966033721 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 5.80083382, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.31325432962870725, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18159462757, + -0.04581437822000005 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.020862525460092384, + 0.484137748685967 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "ablation_cap_original_upfront_paired": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.85627941, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0019202627475750633, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0012403747780000008, + 0.00028191264799999605 + ], + "strong_routes": 118, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.09322033898305085, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.047458978550390885, + 0.16067637565779969 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.46044488, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.045425924086110236, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.006978613070000001, + 0.00012139414999999822 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.15, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.032070937185463715, + 0.3789268265453139 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2123152, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.028778540140326436, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -5.851392499999999e-05, + 0.00024433614999999994 + ], + "strong_routes": 48, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.041666666666666664, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.005086467930032977, + 0.14254097320137743 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.724427200000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.001044091080705245, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0004037624999999898, + 0.0004933144499999826 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.45909213, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.0014499769878559832, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00042307479999999757, + 0.0005730975699999966 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_per_model": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.379702, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.024726931511571903, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.010380521006000008, + 1.0734627999997192e-05 + ], + "strong_routes": 115, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.09565217391304348, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.048722767492542195, + 0.1647044453125498 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.46044488, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.045425924086110236, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.006978613070000001, + 0.00012139414999999822 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.15, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.032070937185463715, + 0.3789268265453139 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2088462, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.011969415048261434, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.00017778097500000003, + 0.00022795382499999996 + ], + "strong_routes": 47, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.0425531914893617, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.005195583176743832, + 0.14540524548821393 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.251318790000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0431169973801393, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.04796722185000002, + 0.0004733589499999808 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08695652173913043, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.010709966037875893, + 0.2803793359094109 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.45909213, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.0014499769878559832, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00042307479999999757, + 0.0005730975699999966 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_raw": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_rescue": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.779895630000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.005575617644488773, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0036691405380000013, + 0.0004582755239999995 + ], + "strong_routes": 124, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.08870967741935484, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04511844911596335, + 0.1531829791414111 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.54757188, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.011521911913115801, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0005666024999999994, + 0.0008554352799999992 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.21798, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.05622746831026859, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00020418355, + 0.00026104007500000006 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.724427200000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.001044091080705245, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0004037624999999898, + 0.0004933144499999826 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 8.289916550000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.01857828113893023, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.019854608400000005, + 0.0005552486399999959 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.047353626607257136, + 0.37384171348868267 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_scalar_calibration": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_review_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_review_per_model": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_review_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 19.25344261, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0786242088711332, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.032, + 0.032 + ], + "mean_cost_delta_bootstrap_95": [ + -0.025640446206000003, + -0.003413582256000004 + ], + "strong_routes": 117, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.07692307692307693, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.03577697786951805, + 0.14100438661143383 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.4366962, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.06094850530476925, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.012602240650000002, + 0.0007569143499999969 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.22282945, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.0797255979377447, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00030838077499999996, + 0.0003543357749999998 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 9.674128730000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.09699331993035598, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0953708952, + 0.0004907219000000017 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.045454545454545456, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0011501475265735762, + 0.2284443976676335 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 7.91978823, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.06239681294472521, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.048092758770000014, + -0.0015903870300000168 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.14285714285714285, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.030488967621515563, + 0.36342398701698087 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_paired": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 19.702987890000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.057111165910539774, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 3, + "paired_exact_p": 0.625, + "quality_delta_bootstrap_95": [ + -0.048, + 0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.017677389673999996, + -0.0031051263940000047 + ], + "strong_routes": 103, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 8, + "sol_route_rescue_precision": 0.07766990291262135, + "sol_only_recall": 0.7272727272727273, + "sol_route_rescue_precision_exact_95": [ + 0.03412864449825134, + 0.14731394223104977 + ], + "sol_only_recall_exact_95": [ + 0.3902574404275788, + 0.9397822658270933 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 1.2205622, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.2022177282305757, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.027744909950000004, + -0.0009071294500000018 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.05, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.001265089497949805, + 0.2487327627720277 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 36, + "cost": 0.19468825, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.05663328100166687, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0005602417, + 6.0092474999999715e-05 + ], + "strong_routes": 38, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.02631578947368421, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0006660362042023661, + 0.13809902979242009 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.9587556, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0704255563507501, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.06546539239999999, + 0.0004996763000000104 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.011205586024150999, + 0.29161274155393346 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.32898184, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.01395344277917554, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.012773785860000011, + 0.0005436093800000007 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.17391304347826086, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.049507645305793435, + 0.3878118899547976 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_per_model": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 19.735080890000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.05557535157002591, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.032, + 0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.017406569784, + -0.002787731114000002 + ], + "strong_routes": 103, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.08737864077669903, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.040736426646704606, + 0.15939919045550544 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.2564414000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.1787664123654198, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.026266256300000002, + 0.00047061504999999765 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09523809523809523, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.0117493178844458, + 0.30377440691392416 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 36, + "cost": 0.19090205000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0749794065201379, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0006591471500000002, + 4.965949999999577e-06 + ], + "strong_routes": 37, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.02702702702702703, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0006840310246946654, + 0.14160309561115805 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.9587556, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0704255563507501, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.06546539239999999, + 0.0004996763000000104 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.011205586024150999, + 0.29161274155393346 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.32898184, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.01395344277917554, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.012773785860000011, + 0.0005436093800000007 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.17391304347826086, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.049507645305793435, + 0.3878118899547976 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_raw": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_rescue": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 18.00477004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.13837958355135338, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.03901592546799999, + -0.010119537734 + ], + "strong_routes": 97, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 8, + "sol_route_rescue_precision": 0.08247422680412371, + "sol_only_recall": 0.7272727272727273, + "sol_route_rescue_precision_exact_95": [ + 0.03628024982016327, + 0.15605269680266584 + ], + "sol_only_recall_exact_95": [ + 0.3902574404275788, + 0.9397822658270933 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.2564414000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.1787664123654198, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.026266256300000002, + 0.00047061504999999765 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09523809523809523, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.0117493178844458, + 0.30377440691392416 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 36, + "cost": 0.19468825, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.05663328100166687, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0005602417, + 6.0092474999999715e-05 + ], + "strong_routes": 38, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.02631578947368421, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0006660362042023661, + 0.13809902979242009 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 8.39968576, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.2159529231563302, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.16257429948000002, + -0.03598645919000014 + ], + "strong_routes": 17, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.058823529411764705, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.001488174391266636, + 0.28688939666722063 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 8.153954630000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.034674460204333624, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.02422112356000001, + -0.0018927489900000074 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.19047619047619047, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.054463568178406814, + 0.41906604118352786 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_scalar_calibration": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_review_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_review_per_model": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_review_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 14.003645670000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.3298538672152522, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 4, + "paired_exact_p": 0.6875, + "quality_delta_bootstrap_95": [ + -0.056, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.08716027666, + -0.03046360189800001 + ], + "strong_routes": 100, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.07, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.028605288907438704, + 0.13891972845585732 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3922581500000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.08999404553369261, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.018160890399999997, + 0.0008559290000000007 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2297896, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.11345117649339076, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00045055567499999996, + 0.0004872066999999999 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 5.35368162, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.5002743502022768, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.33043864058, + -0.1220960099900001 + ], + "strong_routes": 10, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3084971078187608 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 7.0279163, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.16798321749599177, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.1264185382, + -0.011905893240000002 + ], + "strong_routes": 16, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.015513603815413892, + 0.38347623684926346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "ablation_v2_original_upfront_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_per_model": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 20.19712788, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.03346403778716789, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.024, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.012449875198, + -0.0005903775300000025 + ], + "strong_routes": 119, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 10, + "sol_route_rescue_precision": 0.08403361344537816, + "sol_only_recall": 0.9090909090909091, + "sol_route_rescue_precision_exact_95": [ + 0.041034387787323594, + 0.14911334056870384 + ], + "sol_only_recall_exact_95": [ + 0.5872200830116174, + 0.9977010277861857 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 1.24396835, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.18691903102335772, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.029562692249999994, + 0.0007949600000000009 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.011205586024150999, + 0.29161274155393346 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2263206, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.09664205140132576, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0002497678749999999, + 0.00048172807500000003 + ], + "strong_routes": 49, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04081632653061224, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004981841834971302, + 0.1397871870224637 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.263600330000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.04197060859712143, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.04842432615, + 0.0007329489000000062 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08695652173913043, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.010709966037875893, + 0.2803793359094109 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4632386, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.001940866816488418, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006238924000000015, + 0.0006923091999999893 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_raw": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_scalar_calibration": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_review_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "cap_original_raw_baseline": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 6.76416331, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.6763001585057259, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 10, + "paired_exact_p": 0.8145294189453125, + "quality_delta_bootstrap_95": [ + -0.08, + 0.048 + ], + "mean_cost_delta_bootstrap_95": [ + -0.15178693768999998, + -0.08009726906600002 + ], + "strong_routes": 9, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1111111111111111, + "sol_only_recall": 0.09090909090909091, + "sol_route_rescue_precision_exact_95": [ + 0.0028091367465992167, + 0.482496514917337 + ], + "sol_only_recall_exact_95": [ + 0.002298972213814269, + 0.4127799169883826 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 0.29180008, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8092740126436, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07231712887, + -0.029113650460000026 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.7075982261787133 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.028999999999999998, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8594797844710625, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004671895275000001, + -0.0026354289000000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 1.3892767799999999, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.870321529946641, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.49086965658000004, + -0.27021434123000004 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.841886116991581 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 5.054086450000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.4016598139300399, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.2 + ], + "mean_cost_delta_bootstrap_95": [ + -0.21025275142999997, + -0.07596707077000003 + ], + "strong_routes": 6, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.25, + "sol_route_rescue_precision_exact_95": [ + 0.004210744514489473, + 0.6412345789976748 + ], + "sol_only_recall_exact_95": [ + 0.006309463209709872, + 0.8058795503167566 + ] + } + } + } + }, + "cap_empirical_raw_baseline": { + "total": { + "tasks": 125, + "solved": 84, + "cost": 7.249637870000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.6530677155679308, + "baseline_solved": 87, + "gained_sol_failures": 7, + "lost_sol_successes": 10, + "paired_exact_p": 0.629058837890625, + "quality_delta_bootstrap_95": [ + -0.088, + 0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.14845173568000006, + -0.07582031650200001 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.04935316, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.08333333333333333, + "sol_only_recall": 0.09090909090909091, + "sol_route_rescue_precision_exact_95": [ + 0.002107593231860228, + 0.38479616515094406 + ], + "sol_only_recall_exact_95": [ + 0.002298972213814269, + 0.4127799169883826 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 12, + "cost": 0.4859902, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.682347719916546, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 3, + "paired_exact_p": 0.625, + "quality_delta_bootstrap_95": [ + -0.24, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0633542286, + -0.022207698250000015 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0.017241, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.7075982261787133 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.7075982261787133 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.028446199999999998, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8621632360351978, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.00468852185, + -0.0026388404500000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.0110502, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 1.68262264, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8429399146566432, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.48227658493000003, + -0.25442487609000003 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0.01032185, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.7075982261787133 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 5.052578830000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.4018382971515373, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.2 + ], + "mean_cost_delta_bootstrap_95": [ + -0.21032360977, + -0.07603086512000003 + ], + "strong_routes": 6, + "classifier_reviewer_cost": 0.010740109999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.25, + "sol_route_rescue_precision_exact_95": [ + 0.004210744514489473, + 0.6412345789976748 + ], + "sol_only_recall_exact_95": [ + 0.006309463209709872, + 0.8058795503167566 + ] + } + } + } + }, + "v2_original_raw_baseline": { + "total": { + "tasks": 125, + "solved": 84, + "cost": 17.00290026, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.18632418129701356, + "baseline_solved": 87, + "gained_sol_failures": 3, + "lost_sol_successes": 6, + "paired_exact_p": 0.5078125, + "quality_delta_bootstrap_95": [ + -0.072, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.051681949914000005, + -0.01354233937400001 + ], + "strong_routes": 51, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 5, + "sol_route_rescue_precision": 0.09803921568627451, + "sol_only_recall": 0.45454545454545453, + "sol_route_rescue_precision_exact_95": [ + 0.03260648955189092, + 0.21414498254568087 + ], + "sol_only_recall_exact_95": [ + 0.16748809406370715, + 0.7662064023406548 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 12, + "cost": 0.99244255, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.3513209960626009, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.03863867435, + -0.007342218450000004 + ], + "strong_routes": 9, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1111111111111111, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.0028091367465992167, + 0.482496514917337 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.056221600000000004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7275768500213202, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0041246895000000006, + -0.0021125469000000006 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.841886116991581 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 9.13040559, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.14774575885603092, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.13240011395, + -0.012091017849999993 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.05, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.001265089497949805, + 0.2487327627720277 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 6.8238305200000005, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.19214440365445817, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.14368180549, + -0.005013580750000003 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.15, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.032070937185463715, + 0.3789268265453139 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "v2_empirical_raw_baseline": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 14.64873408, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.29898308445959565, + "baseline_solved": 87, + "gained_sol_failures": 6, + "lost_sol_successes": 8, + "paired_exact_p": 0.79052734375, + "quality_delta_bootstrap_95": [ + -0.072, + 0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07110927915399999, + -0.03075588462600001 + ], + "strong_routes": 33, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 0.2727272727272727, + "sol_route_rescue_precision_exact_95": [ + 0.019154942987404933, + 0.24331635075915498 + ], + "sol_only_recall_exact_95": [ + 0.06021773417290667, + 0.6097425595724212 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 0.47988020000000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.6863413301401882, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.06521872845, + -0.022116862650000006 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.7075982261787133 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.7075982261787133 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.03958005, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8082138911501338, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004503112775000001, + -0.002399418000000001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 7.418084090000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.30757800794859325, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.20128575522, + -0.07104596217000003 + ], + "strong_routes": 13, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.07692307692307693, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0019456284973467497, + 0.3602974352678774 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 6.71118974, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.2054796534431248, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.14393527747999998, + -0.017818168519999997 + ], + "strong_routes": 16, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.015513603815413892, + 0.38347623684926346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "cap_review_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 81, + "cost": 7.582970260000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6371160543109662, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 6, + "paired_exact_p": 0.03125, + "quality_delta_bootstrap_95": [ + -0.088, + -0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.15162677911000003, + -0.0657057803420001 + ], + "strong_routes": 95, + "classifier_reviewer_cost": 0.14986411, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 5, + "sol_route_rescue_precision": 0.05263157894736842, + "sol_only_recall": 0.45454545454545453, + "sol_route_rescue_precision_exact_95": [ + 0.01730787584643997, + 0.11856278880488623 + ], + "sol_only_recall_exact_95": [ + 0.16748809406370715, + 0.7662064023406548 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68462508, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1011024455797076, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00456608113, + 0.007829581510000003 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.03820068, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.25031600000000004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.21291235414970755, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006985228249999999, + 0.001141979325 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.026543999999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 2.47515794, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7689627441987307, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.4611561338500001, + -0.21285575223000017 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.046706899999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3694166475528192 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 14, + "cost": 3.17287124, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6243720033483746, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 4, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.32, + -0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.3343958345, + -0.10576620506000002 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03841253, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.2646484693970512 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.6023646356164746 + ] + } + } + } + }, + "cap_review_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 10.18532141, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.5125802298251672, + "baseline_solved": 87, + "gained_sol_failures": 7, + "lost_sol_successes": 7, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.056, + 0.056 + ], + "mean_cost_delta_bootstrap_95": [ + -0.12620953304200003, + -0.049324036836 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.14655205999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.18181818181818182, + "sol_only_recall": 0.36363636363636365, + "sol_route_rescue_precision_exact_95": [ + 0.05186729931243693, + 0.4028457830778714 + ], + "sol_only_recall_exact_95": [ + 0.10926344381909812, + 0.6920952849883228 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3256502000000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.1335302468587085, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.015286975900000005, + -0.0016845305000000009 + ], + "strong_routes": 9, + "classifier_reviewer_cost": 0.037813799999999995, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1111111111111111, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.0028091367465992167, + 0.482496514917337 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.0433862, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7897710974144281, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004362987525000001, + -0.0023665260000000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.025990199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.9752569899999999, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9089671430540687, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5038894888000001, + -0.2904808679400001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.04584315, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 7.84102802, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.07172102992686824, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.11185988509999999, + 0.055466048719999966 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03690491, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.25, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.054860644527992754, + 0.5718584618781891 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "cap_either_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 81, + "cost": 7.582970260000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6371160543109662, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 6, + "paired_exact_p": 0.03125, + "quality_delta_bootstrap_95": [ + -0.088, + -0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.15162677911000003, + -0.0657057803420001 + ], + "strong_routes": 95, + "classifier_reviewer_cost": 0.14986411, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 5, + "sol_route_rescue_precision": 0.05263157894736842, + "sol_only_recall": 0.45454545454545453, + "sol_route_rescue_precision_exact_95": [ + 0.01730787584643997, + 0.11856278880488623 + ], + "sol_only_recall_exact_95": [ + 0.16748809406370715, + 0.7662064023406548 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68462508, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1011024455797076, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00456608113, + 0.007829581510000003 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.03820068, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.25031600000000004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.21291235414970755, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006985228249999999, + 0.001141979325 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.026543999999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 2.47515794, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7689627441987307, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.4611561338500001, + -0.21285575223000017 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.046706899999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3694166475528192 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 14, + "cost": 3.17287124, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6243720033483746, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 4, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.32, + -0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.3343958345, + -0.10576620506000002 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03841253, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.2646484693970512 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.6023646356164746 + ] + } + } + } + }, + "cap_either_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 10.18532141, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.5125802298251672, + "baseline_solved": 87, + "gained_sol_failures": 7, + "lost_sol_successes": 7, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.056, + 0.056 + ], + "mean_cost_delta_bootstrap_95": [ + -0.12620953304200003, + -0.049324036836 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.14655205999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.18181818181818182, + "sol_only_recall": 0.36363636363636365, + "sol_route_rescue_precision_exact_95": [ + 0.05186729931243693, + 0.4028457830778714 + ], + "sol_only_recall_exact_95": [ + 0.10926344381909812, + 0.6920952849883228 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3256502000000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.1335302468587085, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.015286975900000005, + -0.0016845305000000009 + ], + "strong_routes": 9, + "classifier_reviewer_cost": 0.037813799999999995, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1111111111111111, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.0028091367465992167, + 0.482496514917337 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.0433862, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7897710974144281, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004362987525000001, + -0.0023665260000000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.025990199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.9752569899999999, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9089671430540687, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5038894888000001, + -0.2904808679400001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.04584315, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 7.84102802, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.07172102992686824, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.11185988509999999, + 0.055466048719999966 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03690491, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.25, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.054860644527992754, + 0.5718584618781891 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "v2_review_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.9753999, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.0037802625006422286, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.026018115068, + 0.02134152558 + ], + "strong_routes": 115, + "classifier_reviewer_cost": 0.17438415, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.09565217391304348, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.048722767492542195, + 0.1647044453125498 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.60758355, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.050746661315708375, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0014359300500000006, + 0.006706794349999999 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.04041915, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.13636363636363635, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.029055851128746705, + 0.349122097257408 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.24409760000000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1827809435205645, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00032057892499999985, + 0.0010412571749999998 + ], + "strong_routes": 48, + "classifier_reviewer_cost": 0.0383536, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.041666666666666664, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.005086467930032977, + 0.14254097320137743 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.12625244, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.14813342396758788, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18424675188000003, + 0.01805392697 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.0530524, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09523809523809523, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0117493178844458, + 0.30377440691392416 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 9.99746631, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.18357410608865954, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.01264921694, + 0.10895171352 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.042559, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.047353626607257136, + 0.37384171348868267 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "v2_review_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 8.649280130000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.5860876683770405, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 7, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.048, + 0.072 + ], + "mean_cost_delta_bootstrap_95": [ + -0.137710579418, + -0.06163882030400005 + ], + "strong_routes": 15, + "classifier_reviewer_cost": 0.15609798, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.26666666666666666, + "sol_only_recall": 0.36363636363636365, + "sol_route_rescue_precision_exact_95": [ + 0.0778715462910436, + 0.551003241036971 + ], + "sol_only_recall_exact_95": [ + 0.10926344381909812, + 0.6920952849883228 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.0584044000000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.30820709777612765, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0339924741, + -0.006823776650000001 + ], + "strong_routes": 6, + "classifier_reviewer_cost": 0.037696, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.004210744514489473, + 0.6412345789976748 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.04878945, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7635895162228166, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0042552748250000005, + -0.0022587952250000005 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.031393449999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.97752454, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9087554844277945, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5038272815000002, + -0.2903748879100001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.0481107, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 20, + "cost": 6.56456174, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.2228385620551977, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 1, + "paired_exact_p": 0.625, + "quality_delta_bootstrap_95": [ + -0.08, + 0.24 + ], + "mean_cost_delta_bootstrap_95": [ + -0.16652745669, + 0.009938520709999963 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.03889783, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.375, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.08523341413725358, + 0.7551367836334484 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "v2_either_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 8.649280130000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.5860876683770405, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 7, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.048, + 0.072 + ], + "mean_cost_delta_bootstrap_95": [ + -0.137710579418, + -0.06163882030400005 + ], + "strong_routes": 15, + "classifier_reviewer_cost": 0.15609798, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.26666666666666666, + "sol_only_recall": 0.36363636363636365, + "sol_route_rescue_precision_exact_95": [ + 0.0778715462910436, + 0.551003241036971 + ], + "sol_only_recall_exact_95": [ + 0.10926344381909812, + 0.6920952849883228 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.0584044000000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.30820709777612765, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0339924741, + -0.006823776650000001 + ], + "strong_routes": 6, + "classifier_reviewer_cost": 0.037696, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.004210744514489473, + 0.6412345789976748 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.04878945, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7635895162228166, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0042552748250000005, + -0.0022587952250000005 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.031393449999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.97752454, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9087554844277945, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5038272815000002, + -0.2903748879100001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.0481107, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 20, + "cost": 6.56456174, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.2228385620551977, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 1, + "paired_exact_p": 0.625, + "quality_delta_bootstrap_95": [ + -0.08, + 0.24 + ], + "mean_cost_delta_bootstrap_95": [ + -0.16652745669, + 0.009938520709999963 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.03889783, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.375, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.08523341413725358, + 0.7551367836334484 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "ablation_cap_empirical_review_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 82, + "cost": 16.807459010000002, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.19567704561253263, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 5, + "paired_exact_p": 0.0625, + "quality_delta_bootstrap_95": [ + -0.08, + -0.008 + ], + "mean_cost_delta_bootstrap_95": [ + -0.06854206789400004, + -0.0038170915060000052 + ], + "strong_routes": 115, + "classifier_reviewer_cost": 0.14655205999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 6, + "sol_route_rescue_precision": 0.05217391304347826, + "sol_only_recall": 0.5454545454545454, + "sol_route_rescue_precision_exact_95": [ + 0.019385005671561636, + 0.11010465960377613 + ], + "sol_only_recall_exact_95": [ + 0.23379359765934518, + 0.8325119059362929 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.6842382, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.10084957357916369, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0045572192499999985, + 0.007806525299999999 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.037813799999999995, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.24976220000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.21022890258557192, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006922497249999998, + 0.0011280457999999997 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.025990199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 7.90735119, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.2619086281037478, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.24478701523000004, + -0.007153033600000035 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.04584315, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.1684334709830853 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 15, + "cost": 6.96610742, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.17530061048597034, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 3, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + -0.24, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.17648076557000003, + 0.03294136301999995 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.03690491, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.05, + "sol_only_recall": 0.25, + "sol_route_rescue_precision_exact_95": [ + 0.001265089497949805, + 0.2487327627720277 + ], + "sol_only_recall_exact_95": [ + 0.006309463209709872, + 0.8058795503167566 + ] + } + } + } + }, + "ablation_cap_original_review_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 81, + "cost": 7.582970260000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6371160543109662, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 6, + "paired_exact_p": 0.03125, + "quality_delta_bootstrap_95": [ + -0.088, + -0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.15162677911000003, + -0.0657057803420001 + ], + "strong_routes": 95, + "classifier_reviewer_cost": 0.14986411, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 5, + "sol_route_rescue_precision": 0.05263157894736842, + "sol_only_recall": 0.45454545454545453, + "sol_route_rescue_precision_exact_95": [ + 0.01730787584643997, + 0.11856278880488623 + ], + "sol_only_recall_exact_95": [ + 0.16748809406370715, + 0.7662064023406548 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68462508, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1011024455797076, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00456608113, + 0.007829581510000003 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.03820068, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.25031600000000004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.21291235414970755, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006985228249999999, + 0.001141979325 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.026543999999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 2.47515794, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7689627441987307, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.4611561338500001, + -0.21285575223000017 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.046706899999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3694166475528192 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 14, + "cost": 3.17287124, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6243720033483746, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 4, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.32, + -0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.3343958345, + -0.10576620506000002 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03841253, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.2646484693970512 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.6023646356164746 + ] + } + } + } + }, + "ablation_v2_empirical_review_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 83, + "cost": 16.32433933, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.21879679548722408, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 4, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.064, + -0.008 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07417190475199999, + -0.006337188472000025 + ], + "strong_routes": 114, + "classifier_reviewer_cost": 0.15609798, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.06140350877192982, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.025041319674763633, + 0.12242691621974269 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.6841204, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.10077257729694677, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0045775936500000005, + 0.0077651657 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.037696, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.25516545, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.23641048377718343, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.000798436425, + 0.0012493406249999995 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.031393449999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 8.98205814, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.16159287026626945, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.19684950881, + 0.02339831545999997 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.0481107, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.045454545454545456, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0011501475265735762, + 0.2284443976676335 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 15, + "cost": 5.40299534, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.360353395405271, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 3, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + -0.24, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.24350404569, + -0.014465777570000041 + ], + "strong_routes": 17, + "classifier_reviewer_cost": 0.03889783, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.058823529411764705, + "sol_only_recall": 0.25, + "sol_route_rescue_precision_exact_95": [ + 0.001488174391266636, + 0.28688939666722063 + ], + "sol_only_recall_exact_95": [ + 0.006309463209709872, + 0.8058795503167566 + ] + } + } + } + }, + "ablation_v2_original_review_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 19.8241285, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.05131396757892248, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.04, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.03935038515, + 0.014239168169999995 + ], + "strong_routes": 120, + "classifier_reviewer_cost": 0.17438415, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.075, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.03486739061196212, + 0.13760105789036722 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68684355, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.10255247904498477, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.004706471149999998, + 0.007860030849999998 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.04041915, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2621256, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.2701360623328295, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0009425592999999999, + 0.0013770193499999996 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0383536, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.30107364, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.13181518841132078, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18580700010999998, + 0.02750766961 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.0530524, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08695652173913043, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.010709966037875893, + 0.2803793359094109 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 16, + "cost": 8.57408571, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.015063768666083277, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07362103302000002, + 0.06466245526999996 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.042559, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.011205586024150999, + 0.29161274155393346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "ablation_v2_original_review_per_model": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.9753999, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.0037802625006422286, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.026018115068, + 0.02134152558 + ], + "strong_routes": 115, + "classifier_reviewer_cost": 0.17438415, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.09565217391304348, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.048722767492542195, + 0.1647044453125498 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.60758355, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.050746661315708375, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0014359300500000006, + 0.006706794349999999 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.04041915, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.13636363636363635, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.029055851128746705, + 0.349122097257408 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.24409760000000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1827809435205645, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00032057892499999985, + 0.0010412571749999998 + ], + "strong_routes": 48, + "classifier_reviewer_cost": 0.0383536, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.041666666666666664, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.005086467930032977, + 0.14254097320137743 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.12625244, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.14813342396758788, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18424675188000003, + 0.01805392697 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.0530524, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09523809523809523, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0117493178844458, + 0.30377440691392416 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 9.99746631, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.18357410608865954, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.01264921694, + 0.10895171352 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.042559, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.047353626607257136, + 0.37384171348868267 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_review_rescue": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 23.408775900000002, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.12022976104120486, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.024, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.012625334294, + 0.02913148761 + ], + "strong_routes": 124, + "classifier_reviewer_cost": 0.17438415, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 10, + "sol_route_rescue_precision": 0.08064516129032258, + "sol_only_recall": 0.9090909090909091, + "sol_route_rescue_precision_exact_95": [ + 0.03935028219135281, + 0.14331003914481275 + ], + "sol_only_recall_exact_95": [ + 0.5872200830116174, + 0.9977010277861857 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68684355, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.10255247904498477, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.004706471149999998, + 0.007860030849999998 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.04041915, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2621256, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.2701360623328295, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0009425592999999999, + 0.0013770193499999996 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0383536, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 11.40340044, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.06442110294609615, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.021655114940000005, + 0.034285069830000015 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0530524, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 10.05640631, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.1905518598164304, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.03293031734, + 0.10278958102999998 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.042559, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "public_examples_cascade": { + "total": { + "tasks": 75, + "solved": 53, + "cost": 1.3885004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.20031998709915222, + "baseline_solved": 51, + "gained_sol_failures": 4, + "lost_sol_successes": 2, + "paired_exact_p": 0.6875, + "quality_delta_bootstrap_95": [ + -0.04, + 0.09333333333333334 + ], + "mean_cost_delta_bootstrap_95": [ + -0.006993307266666667, + -0.0024630961333333354 + ], + "strong_routes": 13, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 5, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.23076923076923078, + "sol_only_recall": 0.6, + "sol_route_rescue_precision_exact_95": [ + 0.05038107349115152, + 0.538131539234041 + ], + "sol_only_recall_exact_95": [ + 0.14663279963467318, + 0.9472550494736831 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.3645444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.10810827062951323, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.013030371000000002, + -0.0006957486000000024 + ], + "strong_routes": 11, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.18181818181818182, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.022831198299959672, + 0.5177558523601726 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.023955999999999998, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8839206109237508, + "baseline_solved": 37, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.1 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0048078653, + -0.0026865571000000005 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.5, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + } + } + }, + "hindsight_oracle_unattainable": { + "total": { + "tasks": 125, + "solved": 95, + "cost": 4.51896464, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.7837444084882348, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 0, + "paired_exact_p": 0.0078125, + "quality_delta_bootstrap_95": [ + 0.024, + 0.112 + ], + "mean_cost_delta_bootstrap_95": [ + -0.173399360712, + -0.092329002966 + ], + "strong_routes": 11, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.7150858470818456, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 16, + "cost": 0.3436374, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7753921712167242, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 0, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + 0.0, + 0.2 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07116694220000001, + -0.025871578400000025 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.2924017738212867, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 40, + "cost": 0.0352694, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8291012520835757, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 0, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + 0.0, + 0.14 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0045176579, + -0.0025793627 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.15811388300841905, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 1.4341749300000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8661306275404075, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.48791458023, + -0.2698511783400001 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.15811388300841905, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 21, + "cost": 2.70588291, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6796575405129992, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 0, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + 0.0, + 0.24 + ], + "mean_cost_delta_bootstrap_95": [ + -0.34443187702, + -0.13137291555000002 + ], + "strong_routes": 4, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.3976353643835254, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "hindsight_cascade_oracle_unattainable": { + "total": { + "tasks": 125, + "solved": 95, + "cost": 5.26530215, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.7480283379830963, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 0, + "paired_exact_p": 0.0078125, + "quality_delta_bootstrap_95": [ + 0.024, + 0.112 + ], + "mean_cost_delta_bootstrap_95": [ + -0.16898772801800002, + -0.085152777052 + ], + "strong_routes": 11, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.7150858470818456, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 16, + "cost": 0.3657364, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7609478516860748, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 0, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + 0.0, + 0.2 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07068738919999999, + -0.024637301600000005 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.2924017738212867, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 40, + "cost": 0.036092, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8251153234872273, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 0, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + 0.0, + 0.14 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0045073966000000005, + -0.0025570753 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.15811388300841905, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 1.54781684, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8555230155548812, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.48565299367000003, + -0.26237691106 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.15811388300841905, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 21, + "cost": 3.31565691, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6074679782191796, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 0, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + 0.0, + 0.24 + ], + "mean_cost_delta_bootstrap_95": [ + -0.33003004255, + -0.09622135258999999 + ], + "strong_routes": 4, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.3976353643835254, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + } + } +} diff --git a/cookbook/auto_router_selective_training/calibration_results.json b/cookbook/auto_router_selective_training/calibration_results.json new file mode 100644 index 00000000000..882ad358d4a --- /dev/null +++ b/cookbook/auto_router_selective_training/calibration_results.json @@ -0,0 +1,5849 @@ +{ + "scope": "Held-out diagnostics only; no fitting or policy selection from these results. AUC for the signed success-probability gap measures rescue ranking, not calibration of a Sol-only probability.", + "results": { + "cap_original_raw": { + "efficient": { + "n": 125, + "brier": 0.238939608, + "auc": 0.6713124274099884, + "bins": [ + { + "n": 2, + "mean_prediction": 0.105, + "observed_frequency": 0.5 + }, + { + "n": 2, + "mean_prediction": 0.295, + "observed_frequency": 0.5 + }, + { + "n": 6, + "mean_prediction": 0.5133333333333333, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 33, + "mean_prediction": 0.7396969696969697, + "observed_frequency": 0.5757575757575758 + }, + { + "n": 82, + "mean_prediction": 0.943890243902439, + "observed_frequency": 0.7439024390243902 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.312881, + "auc": 0.8205128205128206, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.73, + "observed_frequency": 0.25 + }, + { + "n": 19, + "mean_prediction": 0.9302631578947367, + "observed_frequency": 0.631578947368421 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.22063651999999997, + "auc": 0.6633771929824561, + "bins": [ + { + "n": 2, + "mean_prediction": 0.7, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.9744583333333333, + "observed_frequency": 0.75 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.22333199999999997, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5700000000000001, + "observed_frequency": 0.5 + }, + { + "n": 14, + "mean_prediction": 0.7378571428571429, + "observed_frequency": 0.5 + }, + { + "n": 9, + "mean_prediction": 0.8444444444444446, + "observed_frequency": 0.8888888888888888 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21721200000000004, + "auc": 0.625, + "bins": [ + { + "n": 2, + "mean_prediction": 0.105, + "observed_frequency": 0.5 + }, + { + "n": 2, + "mean_prediction": 0.295, + "observed_frequency": 0.5 + }, + { + "n": 2, + "mean_prediction": 0.47, + "observed_frequency": 0.5 + }, + { + "n": 13, + "mean_prediction": 0.7507692307692307, + "observed_frequency": 0.6923076923076923 + }, + { + "n": 6, + "mean_prediction": 0.8916666666666666, + "observed_frequency": 0.8333333333333334 + } + ] + } + } + } + }, + "v2_original_raw": { + "efficient": { + "n": 125, + "brier": 0.222738496, + "auc": 0.6702961672473867, + "bins": [ + { + "n": 3, + "mean_prediction": 0.06999999999999999, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.3133333333333334, + "observed_frequency": 0.5 + }, + { + "n": 14, + "mean_prediction": 0.49714285714285716, + "observed_frequency": 0.5 + }, + { + "n": 32, + "mean_prediction": 0.6912499999999999, + "observed_frequency": 0.6875 + }, + { + "n": 70, + "mean_prediction": 0.9539142857142855, + "observed_frequency": 0.7428571428571429 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.236602904, + "auc": 0.617816091954023, + "bins": [ + { + "n": 2, + "mean_prediction": 0.035, + "observed_frequency": 0.5 + }, + { + "n": 2, + "mean_prediction": 0.32999999999999996, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.5191666666666667, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 26, + "mean_prediction": 0.7296153846153847, + "observed_frequency": 0.5769230769230769 + }, + { + "n": 83, + "mean_prediction": 0.9463253012048195, + "observed_frequency": 0.7590361445783133 + } + ] + }, + "gap_mse": 0.15295239200000002, + "predicted_mean_gap": 0.052247999999999996, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6176236044657097, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.249792, + "auc": 0.7532051282051282, + "bins": [ + { + "n": 1, + "mean_prediction": 0.18, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.32, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.5066666666666666, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 5, + "mean_prediction": 0.6779999999999999, + "observed_frequency": 0.6 + }, + { + "n": 14, + "mean_prediction": 0.9107142857142858, + "observed_frequency": 0.6428571428571429 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22572799999999998, + "auc": 0.8149350649350648, + "bins": [ + { + "n": 1, + "mean_prediction": 0.3, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.52, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.72, + "observed_frequency": 0.4 + }, + { + "n": 15, + "mean_prediction": 0.9253333333333332, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.20014400000000002, + "predicted_mean_gap": 0.05520000000000001, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5909090909090908 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.22922823999999997, + "auc": 0.6622807017543859, + "bins": [ + { + "n": 2, + "mean_prediction": 0.74, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.9827916666666666, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.25340925999999997, + "auc": 0.6330561330561331, + "bins": [ + { + "n": 50, + "mean_prediction": 0.9825, + "observed_frequency": 0.74 + } + ] + }, + "gap_mse": 0.10050498, + "predicted_mean_gap": 0.009420000000000006, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.75 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.19096400000000002, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 1, + "mean_prediction": 0.28, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5, + "observed_frequency": 0.0 + }, + { + "n": 19, + "mean_prediction": 0.69, + "observed_frequency": 0.7368421052631579 + }, + { + "n": 3, + "mean_prediction": 0.8333333333333334, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.17622800000000002, + "auc": 0.8174603174603174, + "bins": [ + { + "n": 1, + "mean_prediction": 0.36, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.5, + "observed_frequency": 1.0 + }, + { + "n": 13, + "mean_prediction": 0.7569230769230769, + "observed_frequency": 0.6153846153846154 + }, + { + "n": 10, + "mean_prediction": 0.8450000000000001, + "observed_frequency": 0.9 + } + ] + }, + "gap_mse": 0.07872799999999999, + "predicted_mean_gap": 0.0904, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.2282608695652174 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21448, + "auc": 0.6544117647058824, + "bins": [ + { + "n": 2, + "mean_prediction": 0.015, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.32000000000000006, + "observed_frequency": 1.0 + }, + { + "n": 9, + "mean_prediction": 0.4933333333333333, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 6, + "mean_prediction": 0.69, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.8699999999999999, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.27424, + "auc": 0.36904761904761907, + "bins": [ + { + "n": 2, + "mean_prediction": 0.035, + "observed_frequency": 0.5 + }, + { + "n": 7, + "mean_prediction": 0.5214285714285715, + "observed_frequency": 1.0 + }, + { + "n": 8, + "mean_prediction": 0.6912499999999999, + "observed_frequency": 0.625 + }, + { + "n": 8, + "mean_prediction": 0.88625, + "observed_frequency": 0.625 + } + ] + }, + "gap_mse": 0.28488, + "predicted_mean_gap": 0.09680000000000001, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.494047619047619 + } + } + }, + "cap_empirical_raw": { + "efficient": { + "n": 125, + "brier": 0.235516, + "auc": 0.6782810685249709, + "bins": [ + { + "n": 3, + "mean_prediction": 0.11333333333333333, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 2, + "mean_prediction": 0.365, + "observed_frequency": 0.5 + }, + { + "n": 7, + "mean_prediction": 0.49571428571428566, + "observed_frequency": 0.42857142857142855 + }, + { + "n": 37, + "mean_prediction": 0.7337837837837837, + "observed_frequency": 0.5675675675675675 + }, + { + "n": 76, + "mean_prediction": 0.9486842105263157, + "observed_frequency": 0.7631578947368421 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.318396, + "auc": 0.7564102564102565, + "bins": [ + { + "n": 3, + "mean_prediction": 0.4433333333333333, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 4, + "mean_prediction": 0.74, + "observed_frequency": 0.0 + }, + { + "n": 18, + "mean_prediction": 0.9255555555555556, + "observed_frequency": 0.6666666666666666 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.21883999999999998, + "auc": 0.7050438596491229, + "bins": [ + { + "n": 3, + "mean_prediction": 0.6933333333333334, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.9765957446808511, + "observed_frequency": 0.7659574468085106 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.21998799999999996, + "auc": 0.736111111111111, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.49, + "observed_frequency": 0.5 + }, + { + "n": 17, + "mean_prediction": 0.7570588235294117, + "observed_frequency": 0.6470588235294118 + }, + { + "n": 5, + "mean_prediction": 0.844, + "observed_frequency": 0.8 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.20151599999999997, + "auc": 0.6948529411764706, + "bins": [ + { + "n": 3, + "mean_prediction": 0.11333333333333333, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 1, + "mean_prediction": 0.35, + "observed_frequency": 1.0 + }, + { + "n": 2, + "mean_prediction": 0.58, + "observed_frequency": 0.5 + }, + { + "n": 13, + "mean_prediction": 0.7107692307692308, + "observed_frequency": 0.6153846153846154 + }, + { + "n": 6, + "mean_prediction": 0.8866666666666666, + "observed_frequency": 1.0 + } + ] + } + } + } + }, + "v2_empirical_raw": { + "efficient": { + "n": 125, + "brier": 0.230588872, + "auc": 0.6743612078977933, + "bins": [ + { + "n": 3, + "mean_prediction": 0.07, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.2833333333333334, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 11, + "mean_prediction": 0.49545454545454537, + "observed_frequency": 0.45454545454545453 + }, + { + "n": 35, + "mean_prediction": 0.712, + "observed_frequency": 0.6285714285714286 + }, + { + "n": 70, + "mean_prediction": 0.9573571428571429, + "observed_frequency": 0.7571428571428571 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.240495128, + "auc": 0.6199334543254689, + "bins": [ + { + "n": 2, + "mean_prediction": 0.060000000000000005, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.28600000000000003, + "observed_frequency": 0.6 + }, + { + "n": 7, + "mean_prediction": 0.5157142857142858, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 31, + "mean_prediction": 0.7187096774193551, + "observed_frequency": 0.5806451612903226 + }, + { + "n": 80, + "mean_prediction": 0.9424125, + "observed_frequency": 0.7625 + } + ] + }, + "gap_mse": 0.15314735999999995, + "predicted_mean_gap": 0.028304, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.5283094098883573, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.26556599999999997, + "auc": 0.7660256410256411, + "bins": [ + { + "n": 1, + "mean_prediction": 0.14, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.34, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.495, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.7183333333333333, + "observed_frequency": 0.5 + }, + { + "n": 15, + "mean_prediction": 0.9146666666666665, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23513432, + "auc": 0.7922077922077921, + "bins": [ + { + "n": 2, + "mean_prediction": 0.29500000000000004, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.5, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.7183333333333334, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 16, + "mean_prediction": 0.90625, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.20596031999999997, + "predicted_mean_gap": 0.015999999999999997, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.3181818181818182 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.22757317999999999, + "auc": 0.6984649122807017, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7600000000000001, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.9862765957446806, + "observed_frequency": 0.7659574468085106 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.25562665999999995, + "auc": 0.6642411642411643, + "bins": [ + { + "n": 2, + "mean_prediction": 0.77, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.9852708333333334, + "observed_frequency": 0.7291666666666666 + } + ] + }, + "gap_mse": 0.09847623999999999, + "predicted_mean_gap": 0.003960000000000002, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.8385416666666667 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.200724, + "auc": 0.7569444444444444, + "bins": [ + { + "n": 2, + "mean_prediction": 0.495, + "observed_frequency": 0.0 + }, + { + "n": 20, + "mean_prediction": 0.71, + "observed_frequency": 0.7 + }, + { + "n": 3, + "mean_prediction": 0.8333333333333334, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18122800000000003, + "auc": 0.7261904761904762, + "bins": [ + { + "n": 1, + "mean_prediction": 0.49, + "observed_frequency": 0.0 + }, + { + "n": 14, + "mean_prediction": 0.7299999999999999, + "observed_frequency": 0.6428571428571429 + }, + { + "n": 10, + "mean_prediction": 0.8220000000000001, + "observed_frequency": 0.9 + } + ] + }, + "gap_mse": 0.07663199999999999, + "predicted_mean_gap": 0.04959999999999999, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.28260869565217395 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.231508, + "auc": 0.6580882352941176, + "bins": [ + { + "n": 2, + "mean_prediction": 0.034999999999999996, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.27199999999999996, + "observed_frequency": 0.8 + }, + { + "n": 7, + "mean_prediction": 0.49571428571428566, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 6, + "mean_prediction": 0.6883333333333334, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.8879999999999999, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.27486, + "auc": 0.5, + "bins": [ + { + "n": 2, + "mean_prediction": 0.060000000000000005, + "observed_frequency": 0.5 + }, + { + "n": 3, + "mean_prediction": 0.28, + "observed_frequency": 1.0 + }, + { + "n": 5, + "mean_prediction": 0.524, + "observed_frequency": 0.8 + }, + { + "n": 9, + "mean_prediction": 0.69, + "observed_frequency": 0.5555555555555556 + }, + { + "n": 6, + "mean_prediction": 0.8966666666666666, + "observed_frequency": 0.8333333333333334 + } + ] + }, + "gap_mse": 0.286192, + "predicted_mean_gap": 0.068, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.3035714285714286 + } + } + }, + "cap_original_upfront_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.21578879236003057, + "auc": 0.6536004645760743, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5055888529225132, + "observed_frequency": 0.0 + }, + { + "n": 90, + "mean_prediction": 0.7176629375856297, + "observed_frequency": 0.6444444444444445 + }, + { + "n": 33, + "mean_prediction": 0.8345212917151, + "observed_frequency": 0.7878787878787878 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2131703163517031, + "auc": 0.5931639443436176, + "bins": [ + { + "n": 1, + "mean_prediction": 0.555770699456167, + "observed_frequency": 0.0 + }, + { + "n": 84, + "mean_prediction": 0.7661084084960381, + "observed_frequency": 0.6904761904761905 + }, + { + "n": 40, + "mean_prediction": 0.8504392225830284, + "observed_frequency": 0.725 + } + ] + }, + "gap_mse": 0.15128313352119796, + "predicted_mean_gap": 0.04629120961035594, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6523125996810207, + "sol_only_rescue": { + "n": 125, + "brier": 0.07955130967379873, + "auc": 0.6523125996810207, + "bins": [ + { + "n": 125, + "mean_prediction": 0.08377164419969689, + "observed_frequency": 0.088 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.26436027656729394, + "auc": 0.6858974358974359, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5055888529225132, + "observed_frequency": 0.0 + }, + { + "n": 22, + "mean_prediction": 0.6976614387317333, + "observed_frequency": 0.5454545454545454 + }, + { + "n": 1, + "mean_prediction": 0.8341471865750809, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2672982373384959, + "auc": 0.5714285714285714, + "bins": [ + { + "n": 1, + "mean_prediction": 0.555770699456167, + "observed_frequency": 0.0 + }, + { + "n": 22, + "mean_prediction": 0.7529751123092933, + "observed_frequency": 0.5909090909090909 + }, + { + "n": 2, + "mean_prediction": 0.8354998573964358, + "observed_frequency": 0.5 + } + ] + }, + "gap_mse": 0.19613405616550308, + "predicted_mean_gap": 0.06393385362141006, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5909090909090908, + "sol_only_rescue": { + "n": 25, + "brier": 0.10598046514143906, + "auc": 0.6060606060606061, + "bins": [ + { + "n": 25, + "mean_prediction": 0.1089674247207968, + "observed_frequency": 0.12 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1841850742046408, + "auc": 0.5866228070175439, + "bins": [ + { + "n": 19, + "mean_prediction": 0.7211839376102072, + "observed_frequency": 0.7368421052631579 + }, + { + "n": 31, + "mean_prediction": 0.8346594628532378, + "observed_frequency": 0.7741935483870968 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.19598177198078268, + "auc": 0.5769230769230771, + "bins": [ + { + "n": 17, + "mean_prediction": 0.7699827029241958, + "observed_frequency": 0.6470588235294118 + }, + { + "n": 33, + "mean_prediction": 0.8569082359937822, + "observed_frequency": 0.7878787878787878 + } + ] + }, + "gap_mse": 0.10527414754717614, + "predicted_mean_gap": 0.035814791489236844, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.5104166666666666, + "sol_only_rescue": { + "n": 50, + "brier": 0.04088844390444613, + "auc": 0.53125, + "bins": [ + { + "n": 50, + "mean_prediction": 0.06893827316716425, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.2341796766580581, + "auc": 0.6805555555555556, + "bins": [ + { + "n": 24, + "mean_prediction": 0.7303836916583736, + "observed_frequency": 0.625 + }, + { + "n": 1, + "mean_prediction": 0.8306120915728509, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20078602119445169, + "auc": 0.7142857142857143, + "bins": [ + { + "n": 23, + "mean_prediction": 0.775783039816873, + "observed_frequency": 0.7391304347826086 + }, + { + "n": 2, + "mean_prediction": 0.8285393863546553, + "observed_frequency": 0.5 + } + ] + }, + "gap_mse": 0.07219176419405957, + "predicted_mean_gap": 0.045610719884943, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.9347826086956521, + "sol_only_rescue": { + "n": 25, + "brier": 0.07067574794325598, + "auc": 0.9347826086956521, + "bins": [ + { + "n": 25, + "mean_prediction": 0.08383557123821561, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21203386016551926, + "auc": 0.6102941176470589, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7203763726485458, + "observed_frequency": 0.68 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20580377926400273, + "auc": 0.4920634920634921, + "bins": [ + { + "n": 22, + "mean_prediction": 0.7661335444256061, + "observed_frequency": 0.7727272727272727 + }, + { + "n": 3, + "mean_prediction": 0.8038395426747105, + "observed_frequency": 0.3333333333333333 + } + ] + }, + "gap_mse": 0.27754155215207504, + "predicted_mean_gap": 0.05028189156695295, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5238095238095238, + "sol_only_rescue": { + "n": 25, + "brier": 0.13932344747540631, + "auc": 0.5119047619047619, + "bins": [ + { + "n": 25, + "mean_prediction": 0.08817867870514366, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "cap_original_upfront_scalar_calibration_C1.0": { + "efficient": { + "n": 125, + "brier": 0.204343872017707, + "auc": 0.6713124274099884, + "bins": [ + { + "n": 2, + "mean_prediction": 0.12415807062382743, + "observed_frequency": 0.5 + }, + { + "n": 6, + "mean_prediction": 0.31665388316333204, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 42, + "mean_prediction": 0.5225772167682418, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 23, + "mean_prediction": 0.6771396736131469, + "observed_frequency": 0.5652173913043478 + }, + { + "n": 52, + "mean_prediction": 0.8755975141269776, + "observed_frequency": 0.8461538461538461 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21018152489753747, + "auc": 0.6300665456745311, + "bins": [ + { + "n": 2, + "mean_prediction": 0.2932246265086425, + "observed_frequency": 1.0 + }, + { + "n": 9, + "mean_prediction": 0.528432999443439, + "observed_frequency": 0.4444444444444444 + }, + { + "n": 61, + "mean_prediction": 0.6900385181760702, + "observed_frequency": 0.6229508196721312 + }, + { + "n": 53, + "mean_prediction": 0.876520093194308, + "observed_frequency": 0.8113207547169812 + } + ] + }, + "gap_mse": 0.15308437289594812, + "predicted_mean_gap": 0.06950796019078283, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6535087719298246, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.21194578519747873, + "auc": 0.8205128205128206, + "bins": [ + { + "n": 1, + "mean_prediction": 0.31702625485635816, + "observed_frequency": 0.0 + }, + { + "n": 7, + "mean_prediction": 0.5170020859114358, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 6, + "mean_prediction": 0.650597617214943, + "observed_frequency": 0.5 + }, + { + "n": 11, + "mean_prediction": 0.8600026540064601, + "observed_frequency": 0.8181818181818182 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23582117078528128, + "auc": 0.8246753246753247, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5398896593836647, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.6971397131389456, + "observed_frequency": 0.4166666666666667 + }, + { + "n": 11, + "mean_prediction": 0.8667288110681942, + "observed_frequency": 0.8181818181818182 + } + ] + }, + "gap_mse": 0.20285508657256632, + "predicted_mean_gap": 0.06719268178350733, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6590909090909092 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.17696213686411177, + "auc": 0.6633771929824561, + "bins": [ + { + "n": 2, + "mean_prediction": 0.48992704352365574, + "observed_frequency": 1.0 + }, + { + "n": 7, + "mean_prediction": 0.7072015801321168, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 41, + "mean_prediction": 0.8797815009885797, + "observed_frequency": 0.8536585365853658 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.19874376048749023, + "auc": 0.6361746361746362, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5919496252045799, + "observed_frequency": 1.0 + }, + { + "n": 8, + "mean_prediction": 0.7558572229972363, + "observed_frequency": 0.375 + }, + { + "n": 41, + "mean_prediction": 0.8804667827201775, + "observed_frequency": 0.8048780487804879 + } + ] + }, + "gap_mse": 0.10028427296896268, + "predicted_mean_gap": 0.014732776244116974, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.7447916666666666 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.21914075548246842, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 1, + "mean_prediction": 0.39505570458862127, + "observed_frequency": 0.0 + }, + { + "n": 19, + "mean_prediction": 0.5231216435409408, + "observed_frequency": 0.5789473684210527 + }, + { + "n": 5, + "mean_prediction": 0.642393779838937, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19146512742697916, + "auc": 0.753968253968254, + "bins": [ + { + "n": 2, + "mean_prediction": 0.570131757785125, + "observed_frequency": 0.5 + }, + { + "n": 23, + "mean_prediction": 0.6728242556554616, + "observed_frequency": 0.7391304347826086 + } + ] + }, + "gap_mse": 0.07526325810684778, + "predicted_mean_gap": 0.12275542258338723, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.5652173913043479 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2367085456803643, + "auc": 0.625, + "bins": [ + { + "n": 2, + "mean_prediction": 0.12415807062382743, + "observed_frequency": 0.5 + }, + { + "n": 4, + "mean_prediction": 0.2969603348837531, + "observed_frequency": 0.5 + }, + { + "n": 14, + "mean_prediction": 0.5292902277543512, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 5, + "mean_prediction": 0.7016493659386438, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2261338053004462, + "auc": 0.5277777777777778, + "bins": [ + { + "n": 2, + "mean_prediction": 0.2932246265086425, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.4859761338621977, + "observed_frequency": 0.5 + }, + { + "n": 18, + "mean_prediction": 0.6780476326121916, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 1, + "mean_prediction": 0.8224099260209196, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.2867349738624011, + "predicted_mean_gap": 0.12812614409878564, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5119047619047619 + } + } + }, + "cap_original_review_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.1914107556813511, + "auc": 0.6878629500580721, + "bins": [ + { + "n": 7, + "mean_prediction": 0.32213130893111247, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5420956711656675, + "observed_frequency": 0.5 + }, + { + "n": 55, + "mean_prediction": 0.7377653233839958, + "observed_frequency": 0.6727272727272727 + }, + { + "n": 55, + "mean_prediction": 0.8377278940115311, + "observed_frequency": 0.7818181818181819 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2038560003807696, + "auc": 0.647005444646098, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38350868842862773, + "observed_frequency": 0.0 + }, + { + "n": 7, + "mean_prediction": 0.5451755825208928, + "observed_frequency": 0.2857142857142857 + }, + { + "n": 34, + "mean_prediction": 0.7359347238019713, + "observed_frequency": 0.5882352941176471 + }, + { + "n": 83, + "mean_prediction": 0.8459653650194651, + "observed_frequency": 0.7831325301204819 + } + ] + }, + "gap_mse": 0.1481425974333536, + "predicted_mean_gap": 0.04954265746688325, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7615629984051037, + "sol_only_rescue": { + "n": 125, + "brier": 0.07799011548857751, + "auc": 0.7248803827751196, + "bins": [ + { + "n": 119, + "mean_prediction": 0.07480052526589055, + "observed_frequency": 0.08403361344537816 + }, + { + "n": 6, + "mean_prediction": 0.24507485827055805, + "observed_frequency": 0.16666666666666666 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.12957033814377258, + "auc": 0.9423076923076923, + "bins": [ + { + "n": 7, + "mean_prediction": 0.32213130893111247, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5220924689126838, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.7162147302860672, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 10, + "mean_prediction": 0.8146838920339935, + "observed_frequency": 0.9 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19524390639329517, + "auc": 0.8961038961038961, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38350868842862773, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.5408249033452184, + "observed_frequency": 0.16666666666666666 + }, + { + "n": 5, + "mean_prediction": 0.6779739491315399, + "observed_frequency": 0.2 + }, + { + "n": 13, + "mean_prediction": 0.8314592152124622, + "observed_frequency": 0.9230769230769231 + } + ] + }, + "gap_mse": 0.18975451296155385, + "predicted_mean_gap": 0.08336264998080616, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6212121212121211, + "sol_only_rescue": { + "n": 25, + "brier": 0.10591855099378641, + "auc": 0.606060606060606, + "bins": [ + { + "n": 19, + "mean_prediction": 0.09522965045841934, + "observed_frequency": 0.10526315789473684 + }, + { + "n": 6, + "mean_prediction": 0.24507485827055805, + "observed_frequency": 0.16666666666666666 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.181247091176039, + "auc": 0.6030701754385965, + "bins": [ + { + "n": 10, + "mean_prediction": 0.7679947868944556, + "observed_frequency": 0.6 + }, + { + "n": 40, + "mean_prediction": 0.8463055694722218, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20037523275837466, + "auc": 0.5966735966735968, + "bins": [ + { + "n": 5, + "mean_prediction": 0.7786485718232278, + "observed_frequency": 0.6 + }, + { + "n": 45, + "mean_prediction": 0.8590560269296759, + "observed_frequency": 0.7555555555555555 + } + ] + }, + "gap_mse": 0.10152808797950025, + "predicted_mean_gap": 0.020371868462362528, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9791666666666667, + "sol_only_rescue": { + "n": 50, + "brier": 0.03698189800261555, + "auc": 0.8541666666666666, + "bins": [ + { + "n": 50, + "mean_prediction": 0.05711060801812077, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.264293879624284, + "auc": 0.36111111111111116, + "bins": [ + { + "n": 1, + "mean_prediction": 0.46929269429144527, + "observed_frequency": 1.0 + }, + { + "n": 19, + "mean_prediction": 0.7579112042425926, + "observed_frequency": 0.6842105263157895 + }, + { + "n": 5, + "mean_prediction": 0.8151944942810779, + "observed_frequency": 0.4 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22245102108155135, + "auc": 0.3650793650793651, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5712796575749394, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.7561514741196076, + "observed_frequency": 0.5 + }, + { + "n": 20, + "mean_prediction": 0.8306263911696631, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.07197077496742654, + "predicted_mean_gap": 0.050513413245621426, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.8260869565217391, + "sol_only_rescue": { + "n": 25, + "brier": 0.07156211035442267, + "auc": 0.7391304347826086, + "bins": [ + { + "n": 25, + "mean_prediction": 0.07386940067886485, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.20069537828662096, + "auc": 0.6985294117647058, + "bins": [ + { + "n": 5, + "mean_prediction": 0.5646575474417054, + "observed_frequency": 0.6 + }, + { + "n": 20, + "mean_prediction": 0.7099771827424776, + "observed_frequency": 0.7 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20083460891225208, + "auc": 0.5634920634920635, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7357031054007379, + "observed_frequency": 0.7 + }, + { + "n": 5, + "mean_prediction": 0.8272212927249818, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.2759315232787871, + "predicted_mean_gap": 0.0730934871832636, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6071428571428571, + "sol_only_rescue": { + "n": 25, + "brier": 0.13850612008944743, + "auc": 0.5238095238095237, + "bins": [ + { + "n": 25, + "mean_prediction": 0.09558534920213385, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "cap_empirical_review_paired_C0.1": { + "efficient": { + "n": 125, + "brier": 0.1884210186201437, + "auc": 0.7110917537746807, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3016674163230186, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5483539094554382, + "observed_frequency": 0.375 + }, + { + "n": 55, + "mean_prediction": 0.7388728916157172, + "observed_frequency": 0.6727272727272727 + }, + { + "n": 55, + "mean_prediction": 0.8411380031183535, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.20574455245647424, + "auc": 0.6013309134906231, + "bins": [ + { + "n": 8, + "mean_prediction": 0.49402970509838184, + "observed_frequency": 0.25 + }, + { + "n": 34, + "mean_prediction": 0.7461213580731263, + "observed_frequency": 0.6176470588235294 + }, + { + "n": 83, + "mean_prediction": 0.8467219090870934, + "observed_frequency": 0.7710843373493976 + } + ] + }, + "gap_mse": 0.1479538572232481, + "predicted_mean_gap": 0.04959343895378859, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7687400318979267, + "sol_only_rescue": { + "n": 125, + "brier": 0.0779799699311674, + "auc": 0.7256778309409888, + "bins": [ + { + "n": 118, + "mean_prediction": 0.07402949252697953, + "observed_frequency": 0.0847457627118644 + }, + { + "n": 7, + "mean_prediction": 0.2280473214835302, + "observed_frequency": 0.14285714285714285 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.13420378327392754, + "auc": 0.9935897435897436, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3016674163230186, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5844449187014562, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.7569869779976703, + "observed_frequency": 0.4 + }, + { + "n": 11, + "mean_prediction": 0.8356245061354722, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19903158584621417, + "auc": 0.8636363636363636, + "bins": [ + { + "n": 7, + "mean_prediction": 0.48630229151021614, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 4, + "mean_prediction": 0.7254938389078004, + "observed_frequency": 0.25 + }, + { + "n": 14, + "mean_prediction": 0.8456425485269531, + "observed_frequency": 0.8571428571428571 + } + ] + }, + "gap_mse": 0.1931233530095146, + "predicted_mean_gap": 0.07550883465749872, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6212121212121211, + "sol_only_rescue": { + "n": 25, + "brier": 0.1082483777519406, + "auc": 0.6212121212121211, + "bins": [ + { + "n": 18, + "mean_prediction": 0.07825027382251246, + "observed_frequency": 0.1111111111111111 + }, + { + "n": 7, + "mean_prediction": 0.2280473214835302, + "observed_frequency": 0.14285714285714285 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1772992624031761, + "auc": 0.5328947368421053, + "bins": [ + { + "n": 12, + "mean_prediction": 0.7500658239208748, + "observed_frequency": 0.5833333333333334 + }, + { + "n": 38, + "mean_prediction": 0.8467148452242442, + "observed_frequency": 0.8157894736842105 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20304421268401235, + "auc": 0.4303534303534303, + "bins": [ + { + "n": 6, + "mean_prediction": 0.7627406484622891, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 44, + "mean_prediction": 0.8584286521195373, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.09983372322843556, + "predicted_mean_gap": 0.023427011569231916, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9895833333333333, + "sol_only_rescue": { + "n": 50, + "brier": 0.03569536629394268, + "auc": 0.9791666666666667, + "bins": [ + { + "n": 50, + "mean_prediction": 0.061181062669169686, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.26710729772582764, + "auc": 0.3125, + "bins": [ + { + "n": 1, + "mean_prediction": 0.439851863525283, + "observed_frequency": 1.0 + }, + { + "n": 18, + "mean_prediction": 0.7649098051223528, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 6, + "mean_prediction": 0.8159260809163279, + "observed_frequency": 0.3333333333333333 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22251407086765682, + "auc": 0.3492063492063492, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5481216002155417, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.7711520757599395, + "observed_frequency": 0.5 + }, + { + "n": 20, + "mean_prediction": 0.8284150887312146, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.07541550609767562, + "predicted_mean_gap": 0.043889873466159655, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.5, + "sol_only_rescue": { + "n": 25, + "brier": 0.07507170965033697, + "auc": 0.4347826086956522, + "bins": [ + { + "n": 25, + "mean_prediction": 0.06587682979248362, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.18619548729461116, + "auc": 0.8235294117647058, + "bins": [ + { + "n": 5, + "mean_prediction": 0.5556179149430621, + "observed_frequency": 0.4 + }, + { + "n": 20, + "mean_prediction": 0.7041953884811629, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20108868020047574, + "auc": 0.5555555555555556, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7402549312520803, + "observed_frequency": 0.7 + }, + { + "n": 5, + "mean_prediction": 0.8199520613934961, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.27156298055217926, + "predicted_mean_gap": 0.08171446350682078, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.619047619047619, + "sol_only_rescue": { + "n": 25, + "brier": 0.13518902966567417, + "auc": 0.6071428571428571, + "bins": [ + { + "n": 25, + "mean_prediction": 0.10484005244431145, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_original_upfront_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.20955101845143284, + "auc": 0.6591173054587689, + "bins": [ + { + "n": 16, + "mean_prediction": 0.5667534503435131, + "observed_frequency": 0.4375 + }, + { + "n": 54, + "mean_prediction": 0.6789223136887862, + "observed_frequency": 0.6481481481481481 + }, + { + "n": 55, + "mean_prediction": 0.853313330691114, + "observed_frequency": 0.7636363636363637 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2120795262896164, + "auc": 0.6321839080459771, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5932803491724423, + "observed_frequency": 0.0 + }, + { + "n": 53, + "mean_prediction": 0.7131101509259069, + "observed_frequency": 0.6226415094339622 + }, + { + "n": 70, + "mean_prediction": 0.8599280022004958, + "observed_frequency": 0.7714285714285715 + } + ] + }, + "gap_mse": 0.1518496395206646, + "predicted_mean_gap": 0.052114124150005804, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6339712918660287, + "sol_only_rescue": { + "n": 125, + "brier": 0.07999504149700941, + "auc": 0.6228070175438597, + "bins": [ + { + "n": 125, + "mean_prediction": 0.08489086207563169, + "observed_frequency": 0.088 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.23665019595851228, + "auc": 0.782051282051282, + "bins": [ + { + "n": 9, + "mean_prediction": 0.5484291815802487, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 12, + "mean_prediction": 0.7395154157522984, + "observed_frequency": 0.5833333333333334 + }, + { + "n": 4, + "mean_prediction": 0.8570040691886698, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23604831208369462, + "auc": 0.8506493506493508, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5932803491724423, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.6964671564970546, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 11, + "mean_prediction": 0.8374086915518695, + "observed_frequency": 0.9090909090909091 + } + ] + }, + "gap_mse": 0.20840532574641726, + "predicted_mean_gap": 0.060703931335024296, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.48484848484848486, + "sol_only_rescue": { + "n": 25, + "brier": 0.1119211071362905, + "auc": 0.5, + "bins": [ + { + "n": 25, + "mean_prediction": 0.10122056872175196, + "observed_frequency": 0.12 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19295227968276996, + "auc": 0.5877192982456141, + "bins": [ + { + "n": 1, + "mean_prediction": 0.6813398775411287, + "observed_frequency": 1.0 + }, + { + "n": 49, + "mean_prediction": 0.8531639371020696, + "observed_frequency": 0.7551020408163265 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.21156910489292113, + "auc": 0.5446985446985447, + "bins": [ + { + "n": 1, + "mean_prediction": 0.735504011160592, + "observed_frequency": 1.0 + }, + { + "n": 49, + "mean_prediction": 0.8731227105981716, + "observed_frequency": 0.7346938775510204 + } + ] + }, + "gap_mse": 0.10130703391819111, + "predicted_mean_gap": 0.020642880698569174, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.8229166666666667, + "sol_only_rescue": { + "n": 50, + "brier": 0.03845277659040781, + "auc": 0.8229166666666667, + "bins": [ + { + "n": 50, + "mean_prediction": 0.046373446628495525, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.20882354664311553, + "auc": 0.8402777777777778, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5941141668655936, + "observed_frequency": 0.0 + }, + { + "n": 24, + "mean_prediction": 0.6667613159417245, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19101930004530196, + "auc": 0.8492063492063493, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7274520936924024, + "observed_frequency": 0.65 + }, + { + "n": 5, + "mean_prediction": 0.810470069447004, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.07101647541319869, + "predicted_mean_gap": 0.0802002588646435, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6956521739130435, + "sol_only_rescue": { + "n": 25, + "brier": 0.0721550946043794, + "auc": 0.7173913043478262, + "bins": [ + { + "n": 25, + "mean_prediction": 0.11549946293873621, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21637679028999643, + "auc": 0.4779411764705882, + "bins": [ + { + "n": 6, + "mean_prediction": 0.5896797340680631, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 17, + "mean_prediction": 0.6531769700014324, + "observed_frequency": 0.6470588235294118 + }, + { + "n": 2, + "mean_prediction": 0.8495919966275949, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21019180953324298, + "auc": 0.3412698412698413, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7076343118049885, + "observed_frequency": 0.75 + }, + { + "n": 5, + "mean_prediction": 0.8296202760837431, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.27721232860732486, + "predicted_mean_gap": 0.07838066915322284, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.40476190476190477, + "sol_only_rescue": { + "n": 25, + "brier": 0.13899345256356152, + "auc": 0.36904761904761907, + "bins": [ + { + "n": 25, + "mean_prediction": 0.11498738546067915, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_original_upfront_scalar_calibration_C0.1": { + "efficient": { + "n": 125, + "brier": 0.20454289048147975, + "auc": 0.6702961672473867, + "bins": [ + { + "n": 2, + "mean_prediction": 0.11733927578309869, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.3676547804526239, + "observed_frequency": 0.25 + }, + { + "n": 42, + "mean_prediction": 0.5306628758263982, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 25, + "mean_prediction": 0.6766111515699391, + "observed_frequency": 0.76 + }, + { + "n": 52, + "mean_prediction": 0.8801589054214638, + "observed_frequency": 0.7692307692307693 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2108054170799802, + "auc": 0.617816091954023, + "bins": [ + { + "n": 2, + "mean_prediction": 0.2943642691374838, + "observed_frequency": 0.5 + }, + { + "n": 9, + "mean_prediction": 0.5533227554610396, + "observed_frequency": 0.4444444444444444 + }, + { + "n": 60, + "mean_prediction": 0.6830970782840848, + "observed_frequency": 0.6833333333333333 + }, + { + "n": 54, + "mean_prediction": 0.8769448478815192, + "observed_frequency": 0.7592592592592593 + } + ] + }, + "gap_mse": 0.15120245748014532, + "predicted_mean_gap": 0.0578623959265715, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6885964912280702, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.22091798281245, + "auc": 0.7532051282051282, + "bins": [ + { + "n": 2, + "mean_prediction": 0.35519679173027074, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5162717553668691, + "observed_frequency": 0.375 + }, + { + "n": 10, + "mean_prediction": 0.6957773174493485, + "observed_frequency": 0.6 + }, + { + "n": 5, + "mean_prediction": 0.8707386746370119, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2246470342419736, + "auc": 0.8149350649350648, + "bins": [ + { + "n": 4, + "mean_prediction": 0.5541291747850261, + "observed_frequency": 0.0 + }, + { + "n": 15, + "mean_prediction": 0.6992917836050431, + "observed_frequency": 0.6 + }, + { + "n": 6, + "mean_prediction": 0.8467414195646222, + "observed_frequency": 0.8333333333333334 + } + ] + }, + "gap_mse": 0.20532577111426073, + "predicted_mean_gap": 0.06537231186117777, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6060606060606061 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1898629366301952, + "auc": 0.6622807017543859, + "bins": [ + { + "n": 1, + "mean_prediction": 0.596804463889429, + "observed_frequency": 1.0 + }, + { + "n": 3, + "mean_prediction": 0.6947712497458647, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 46, + "mean_prediction": 0.8823113755502815, + "observed_frequency": 0.7608695652173914 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20786062055512197, + "auc": 0.6330561330561331, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7097151707629702, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.8816373441177446, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "gap_mse": 0.10079426640602042, + "predicted_mean_gap": 0.005973183947658731, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.7083333333333334 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.20982837369936858, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 1, + "mean_prediction": 0.3801127691749771, + "observed_frequency": 0.0 + }, + { + "n": 17, + "mean_prediction": 0.5597635631691373, + "observed_frequency": 0.5882352941176471 + }, + { + "n": 7, + "mean_prediction": 0.6456432545484561, + "observed_frequency": 0.8571428571428571 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18885350649032856, + "auc": 0.8174603174603174, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5496063311842023, + "observed_frequency": 0.5 + }, + { + "n": 23, + "mean_prediction": 0.6848065109012914, + "observed_frequency": 0.7391304347826086 + } + ] + }, + "gap_mse": 0.07488918328015506, + "predicted_mean_gap": 0.09736665152834414, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.4782608695652174 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21224222263518988, + "auc": 0.6544117647058824, + "bins": [ + { + "n": 2, + "mean_prediction": 0.11733927578309869, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.3801127691749771, + "observed_frequency": 1.0 + }, + { + "n": 16, + "mean_prediction": 0.5028051065005626, + "observed_frequency": 0.625 + }, + { + "n": 5, + "mean_prediction": 0.6707378167356419, + "observed_frequency": 1.0 + }, + { + "n": 1, + "mean_prediction": 0.8282464334181043, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22480530355735504, + "auc": 0.36904761904761907, + "bins": [ + { + "n": 2, + "mean_prediction": 0.2943642691374838, + "observed_frequency": 0.5 + }, + { + "n": 3, + "mean_prediction": 0.5547251458802822, + "observed_frequency": 1.0 + }, + { + "n": 19, + "mean_prediction": 0.6640396147342544, + "observed_frequency": 0.6842105263157895 + }, + { + "n": 1, + "mean_prediction": 0.8376180946802971, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.27420880019426996, + "predicted_mean_gap": 0.11462664834801807, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6726190476190477 + } + } + }, + "v2_original_review_per_model_C1.0": { + "efficient": { + "n": 125, + "brier": 0.19932787533774815, + "auc": 0.6492450638792102, + "bins": [ + { + "n": 8, + "mean_prediction": 0.09808319476498092, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.28536439122575186, + "observed_frequency": 0.4 + }, + { + "n": 8, + "mean_prediction": 0.48886282725063324, + "observed_frequency": 0.75 + }, + { + "n": 34, + "mean_prediction": 0.733993887146487, + "observed_frequency": 0.6764705882352942 + }, + { + "n": 70, + "mean_prediction": 0.8715403372888845, + "observed_frequency": 0.7571428571428571 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.20931594252899682, + "auc": 0.6091954022988506, + "bins": [ + { + "n": 6, + "mean_prediction": 0.3403154042414572, + "observed_frequency": 0.16666666666666666 + }, + { + "n": 2, + "mean_prediction": 0.5074102884175007, + "observed_frequency": 0.5 + }, + { + "n": 23, + "mean_prediction": 0.7316869758668929, + "observed_frequency": 0.6521739130434783 + }, + { + "n": 94, + "mean_prediction": 0.867781103974221, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "gap_mse": 0.16328838238192742, + "predicted_mean_gap": 0.07496745052274353, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6443381180223285, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.05652455210980535, + "auc": 0.9743589743589743, + "bins": [ + { + "n": 8, + "mean_prediction": 0.09808319476498092, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.25617641805972485, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.41798011212615066, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.7562765355320514, + "observed_frequency": 1.0 + }, + { + "n": 13, + "mean_prediction": 0.8763633997087588, + "observed_frequency": 0.9230769230769231 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18002485102375843, + "auc": 0.8701298701298701, + "bins": [ + { + "n": 6, + "mean_prediction": 0.3403154042414572, + "observed_frequency": 0.16666666666666666 + }, + { + "n": 2, + "mean_prediction": 0.5074102884175007, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.7353023838701571, + "observed_frequency": 0.2 + }, + { + "n": 12, + "mean_prediction": 0.874839672100801, + "observed_frequency": 0.9166666666666666 + } + ] + }, + "gap_mse": 0.20595801309115422, + "predicted_mean_gap": 0.13469206994931118, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6515151515151515 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19091964059448877, + "auc": 0.5021929824561403, + "bins": [ + { + "n": 2, + "mean_prediction": 0.29503917005045877, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.7805277780914377, + "observed_frequency": 0.8 + }, + { + "n": 43, + "mean_prediction": 0.8735118156150351, + "observed_frequency": 0.7674418604651163 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.21240803577351028, + "auc": 0.40540540540540543, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7392149518637621, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.884324929859315, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "gap_mse": 0.10885331642033774, + "predicted_mean_gap": 0.03454382513948935, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9791666666666666 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.27579565498158387, + "auc": 0.3958333333333333, + "bins": [ + { + "n": 3, + "mean_prediction": 0.5323266144146128, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 11, + "mean_prediction": 0.7354168760459815, + "observed_frequency": 0.7272727272727273 + }, + { + "n": 11, + "mean_prediction": 0.8569469095234605, + "observed_frequency": 0.5454545454545454 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21960534537584622, + "auc": 0.39682539682539686, + "bins": [ + { + "n": 5, + "mean_prediction": 0.7302710591423467, + "observed_frequency": 0.6 + }, + { + "n": 20, + "mean_prediction": 0.846885721543498, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.08086696652085992, + "predicted_mean_gap": 0.05904352968295964, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.5652173913043479 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.28247988840837396, + "auc": 0.3014705882352941, + "bins": [ + { + "n": 1, + "mean_prediction": 0.32439077990839177, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.4739856656587692, + "observed_frequency": 1.0 + }, + { + "n": 17, + "mean_prediction": 0.7180759470873835, + "observed_frequency": 0.5882352941176471 + }, + { + "n": 3, + "mean_prediction": 0.8758917792678234, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22213344469835924, + "auc": 0.3650793650793651, + "bins": [ + { + "n": 10, + "mean_prediction": 0.7283288374284733, + "observed_frequency": 0.9 + }, + { + "n": 15, + "mean_prediction": 0.8381574382739623, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.31191029945694765, + "predicted_mean_gap": 0.11201400270246818, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.1547619047619048 + } + } + }, + "v2_empirical_review_per_model_C0.1": { + "efficient": { + "n": 125, + "brier": 0.19010512226877627, + "auc": 0.7047038327526133, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3334240172369359, + "observed_frequency": 0.0 + }, + { + "n": 9, + "mean_prediction": 0.5417594715537408, + "observed_frequency": 0.4444444444444444 + }, + { + "n": 60, + "mean_prediction": 0.7254281408821367, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 49, + "mean_prediction": 0.8410865458465858, + "observed_frequency": 0.8163265306122449 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.20735573624589884, + "auc": 0.6007259528130672, + "bins": [ + { + "n": 7, + "mean_prediction": 0.5373237575440344, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 33, + "mean_prediction": 0.7638175292576284, + "observed_frequency": 0.6363636363636364 + }, + { + "n": 85, + "mean_prediction": 0.8353217086986139, + "observed_frequency": 0.7647058823529411 + } + ] + }, + "gap_mse": 0.14802323408721804, + "predicted_mean_gap": 0.06416685954911215, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7416267942583731, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.13994229066694372, + "auc": 1.0, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3334240172369359, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.5696121158980051, + "observed_frequency": 0.0 + }, + { + "n": 14, + "mean_prediction": 0.7574108461100196, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 3, + "mean_prediction": 0.8465217062183243, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21306367488670708, + "auc": 0.8701298701298701, + "bins": [ + { + "n": 7, + "mean_prediction": 0.5373237575440344, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 10, + "mean_prediction": 0.7609076761326313, + "observed_frequency": 0.5 + }, + { + "n": 8, + "mean_prediction": 0.8234084087310922, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.19308006491920632, + "predicted_mean_gap": 0.07642852532925948, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.18284109550990832, + "auc": 0.5021929824561404, + "bins": [ + { + "n": 4, + "mean_prediction": 0.7287123247106313, + "observed_frequency": 0.25 + }, + { + "n": 46, + "mean_prediction": 0.8407320788658201, + "observed_frequency": 0.8043478260869565 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.2031037287625894, + "auc": 0.41372141372141374, + "bins": [ + { + "n": 2, + "mean_prediction": 0.789903402752905, + "observed_frequency": 0.5 + }, + { + "n": 48, + "mean_prediction": 0.8484833906880502, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.10064951069561175, + "predicted_mean_gap": 0.014369692637239322, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9791666666666666 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.2466896387796678, + "auc": 0.5555555555555556, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5572098889724613, + "observed_frequency": 1.0 + }, + { + "n": 23, + "mean_prediction": 0.7432932681777755, + "observed_frequency": 0.6086956521739131 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2167319530762399, + "auc": 0.26190476190476186, + "bins": [ + { + "n": 7, + "mean_prediction": 0.7745524256235462, + "observed_frequency": 0.8571428571428571 + }, + { + "n": 18, + "mean_prediction": 0.8161483637753537, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "gap_mse": 0.07097323173566336, + "predicted_mean_gap": 0.07609490325149727, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.8695652173913043 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.1982114908774532, + "auc": 0.7058823529411764, + "bins": [ + { + "n": 6, + "mean_prediction": 0.5319672250234567, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 19, + "mean_prediction": 0.6795443231819248, + "observed_frequency": 0.7894736842105263 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2007755957413684, + "auc": 0.626984126984127, + "bins": [ + { + "n": 14, + "mean_prediction": 0.7568019942360563, + "observed_frequency": 0.6428571428571429 + }, + { + "n": 11, + "mean_prediction": 0.8179276971409692, + "observed_frequency": 0.8181818181818182 + } + ] + }, + "gap_mse": 0.274763852389997, + "predicted_mean_gap": 0.13957148389032537, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.7261904761904762 + } + } + }, + "cap_empirical_review_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.1884210186201437, + "auc": 0.7110917537746807, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3016674163230186, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5483539094554382, + "observed_frequency": 0.375 + }, + { + "n": 55, + "mean_prediction": 0.7388728916157172, + "observed_frequency": 0.6727272727272727 + }, + { + "n": 55, + "mean_prediction": 0.8411380031183535, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.20574455245647424, + "auc": 0.6013309134906231, + "bins": [ + { + "n": 8, + "mean_prediction": 0.49402970509838184, + "observed_frequency": 0.25 + }, + { + "n": 34, + "mean_prediction": 0.7461213580731263, + "observed_frequency": 0.6176470588235294 + }, + { + "n": 83, + "mean_prediction": 0.8467219090870934, + "observed_frequency": 0.7710843373493976 + } + ] + }, + "gap_mse": 0.1479538572232481, + "predicted_mean_gap": 0.04959343895378859, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7687400318979267, + "sol_only_rescue": { + "n": 125, + "brier": 0.0779799699311674, + "auc": 0.7256778309409888, + "bins": [ + { + "n": 118, + "mean_prediction": 0.07402949252697953, + "observed_frequency": 0.0847457627118644 + }, + { + "n": 7, + "mean_prediction": 0.2280473214835302, + "observed_frequency": 0.14285714285714285 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.13420378327392754, + "auc": 0.9935897435897436, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3016674163230186, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5844449187014562, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.7569869779976703, + "observed_frequency": 0.4 + }, + { + "n": 11, + "mean_prediction": 0.8356245061354722, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19903158584621417, + "auc": 0.8636363636363636, + "bins": [ + { + "n": 7, + "mean_prediction": 0.48630229151021614, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 4, + "mean_prediction": 0.7254938389078004, + "observed_frequency": 0.25 + }, + { + "n": 14, + "mean_prediction": 0.8456425485269531, + "observed_frequency": 0.8571428571428571 + } + ] + }, + "gap_mse": 0.1931233530095146, + "predicted_mean_gap": 0.07550883465749872, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6212121212121211, + "sol_only_rescue": { + "n": 25, + "brier": 0.1082483777519406, + "auc": 0.6212121212121211, + "bins": [ + { + "n": 18, + "mean_prediction": 0.07825027382251246, + "observed_frequency": 0.1111111111111111 + }, + { + "n": 7, + "mean_prediction": 0.2280473214835302, + "observed_frequency": 0.14285714285714285 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1772992624031761, + "auc": 0.5328947368421053, + "bins": [ + { + "n": 12, + "mean_prediction": 0.7500658239208748, + "observed_frequency": 0.5833333333333334 + }, + { + "n": 38, + "mean_prediction": 0.8467148452242442, + "observed_frequency": 0.8157894736842105 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20304421268401235, + "auc": 0.4303534303534303, + "bins": [ + { + "n": 6, + "mean_prediction": 0.7627406484622891, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 44, + "mean_prediction": 0.8584286521195373, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.09983372322843556, + "predicted_mean_gap": 0.023427011569231916, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9895833333333333, + "sol_only_rescue": { + "n": 50, + "brier": 0.03569536629394268, + "auc": 0.9791666666666667, + "bins": [ + { + "n": 50, + "mean_prediction": 0.061181062669169686, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.26710729772582764, + "auc": 0.3125, + "bins": [ + { + "n": 1, + "mean_prediction": 0.439851863525283, + "observed_frequency": 1.0 + }, + { + "n": 18, + "mean_prediction": 0.7649098051223528, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 6, + "mean_prediction": 0.8159260809163279, + "observed_frequency": 0.3333333333333333 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22251407086765682, + "auc": 0.3492063492063492, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5481216002155417, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.7711520757599395, + "observed_frequency": 0.5 + }, + { + "n": 20, + "mean_prediction": 0.8284150887312146, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.07541550609767562, + "predicted_mean_gap": 0.043889873466159655, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.5, + "sol_only_rescue": { + "n": 25, + "brier": 0.07507170965033697, + "auc": 0.4347826086956522, + "bins": [ + { + "n": 25, + "mean_prediction": 0.06587682979248362, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.18619548729461116, + "auc": 0.8235294117647058, + "bins": [ + { + "n": 5, + "mean_prediction": 0.5556179149430621, + "observed_frequency": 0.4 + }, + { + "n": 20, + "mean_prediction": 0.7041953884811629, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20108868020047574, + "auc": 0.5555555555555556, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7402549312520803, + "observed_frequency": 0.7 + }, + { + "n": 5, + "mean_prediction": 0.8199520613934961, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.27156298055217926, + "predicted_mean_gap": 0.08171446350682078, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.619047619047619, + "sol_only_rescue": { + "n": 25, + "brier": 0.13518902966567417, + "auc": 0.6071428571428571, + "bins": [ + { + "n": 25, + "mean_prediction": 0.10484005244431145, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "cap_original_upfront_paired_C1.0": { + "efficient": { + "n": 125, + "brier": 0.2162848443236792, + "auc": 0.6445993031358885, + "bins": [ + { + "n": 2, + "mean_prediction": 0.1765799121631378, + "observed_frequency": 0.0 + }, + { + "n": 13, + "mean_prediction": 0.48784999388299466, + "observed_frequency": 0.6153846153846154 + }, + { + "n": 72, + "mean_prediction": 0.7327722309375719, + "observed_frequency": 0.6527777777777778 + }, + { + "n": 38, + "mean_prediction": 0.8700950590095132, + "observed_frequency": 0.7631578947368421 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21684750902033487, + "auc": 0.5486993345432547, + "bins": [ + { + "n": 1, + "mean_prediction": 0.2814455913620738, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.5368912598347659, + "observed_frequency": 0.5 + }, + { + "n": 51, + "mean_prediction": 0.7404760777754543, + "observed_frequency": 0.6862745098039216 + }, + { + "n": 67, + "mean_prediction": 0.8591020916406562, + "observed_frequency": 0.7313432835820896 + } + ] + }, + "gap_mse": 0.15175632352505936, + "predicted_mean_gap": 0.0504679251373675, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6459330143540671, + "sol_only_rescue": { + "n": 125, + "brier": 0.08154577784631287, + "auc": 0.616427432216906, + "bins": [ + { + "n": 116, + "mean_prediction": 0.06741066524163526, + "observed_frequency": 0.08620689655172414 + }, + { + "n": 9, + "mean_prediction": 0.2884753665190133, + "observed_frequency": 0.1111111111111111 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.253659251872235, + "auc": 0.6858974358974358, + "bins": [ + { + "n": 2, + "mean_prediction": 0.1765799121631378, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.4682494528044025, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 16, + "mean_prediction": 0.7354111933302958, + "observed_frequency": 0.5 + }, + { + "n": 1, + "mean_prediction": 0.8773105057960244, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.24908676767101326, + "auc": 0.6103896103896105, + "bins": [ + { + "n": 1, + "mean_prediction": 0.2814455913620738, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.5178774123086126, + "observed_frequency": 0.25 + }, + { + "n": 10, + "mean_prediction": 0.7417186180949691, + "observed_frequency": 0.6 + }, + { + "n": 10, + "mean_prediction": 0.8319429948483291, + "observed_frequency": 0.7 + } + ] + }, + "gap_mse": 0.20363943995631115, + "predicted_mean_gap": 0.09132100919184234, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5303030303030303, + "sol_only_rescue": { + "n": 25, + "brier": 0.11789359780682673, + "auc": 0.5454545454545455, + "bins": [ + { + "n": 20, + "mean_prediction": 0.10607970040615036, + "observed_frequency": 0.15 + }, + { + "n": 5, + "mean_prediction": 0.2855816159271335, + "observed_frequency": 0.0 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19009889910930744, + "auc": 0.5603070175438597, + "bins": [ + { + "n": 19, + "mean_prediction": 0.7379037794053194, + "observed_frequency": 0.7368421052631579 + }, + { + "n": 31, + "mean_prediction": 0.8762443664853552, + "observed_frequency": 0.7741935483870968 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20380262054017337, + "auc": 0.5353430353430353, + "bins": [ + { + "n": 7, + "mean_prediction": 0.7107600299026597, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 43, + "mean_prediction": 0.8724268734854672, + "observed_frequency": 0.7441860465116279 + } + ] + }, + "gap_mse": 0.10508667079941304, + "predicted_mean_gap": 0.026118571988932623, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.5625, + "sol_only_rescue": { + "n": 50, + "brier": 0.040928686216825626, + "auc": 0.46875, + "bins": [ + { + "n": 50, + "mean_prediction": 0.05367778464962154, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.23708523366794856, + "auc": 0.6736111111111112, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5866178031538105, + "observed_frequency": 0.5 + }, + { + "n": 17, + "mean_prediction": 0.7264681255964354, + "observed_frequency": 0.6470588235294118 + }, + { + "n": 6, + "mean_prediction": 0.8371210625865765, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20270167191546734, + "auc": 0.5634920634920635, + "bins": [ + { + "n": 16, + "mean_prediction": 0.7384115132483009, + "observed_frequency": 0.6875 + }, + { + "n": 9, + "mean_prediction": 0.8294199795518544, + "observed_frequency": 0.7777777777777778 + } + ] + }, + "gap_mse": 0.07052358338676709, + "predicted_mean_gap": 0.02933775643892083, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.9347826086956521, + "sol_only_rescue": { + "n": 25, + "brier": 0.06743215967349744, + "auc": 0.9347826086956521, + "bins": [ + { + "n": 25, + "mean_prediction": 0.06216253801825453, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21048193785959757, + "auc": 0.6176470588235294, + "bins": [ + { + "n": 5, + "mean_prediction": 0.471863519468979, + "observed_frequency": 0.6 + }, + { + "n": 20, + "mean_prediction": 0.7311445795189984, + "observed_frequency": 0.7 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.224843864434847, + "auc": 0.3015873015873016, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5749189548870726, + "observed_frequency": 1.0 + }, + { + "n": 18, + "mean_prediction": 0.7531771869059471, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 5, + "mean_prediction": 0.8522549631197791, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.27444525268339237, + "predicted_mean_gap": 0.07944371607820912, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5238095238095238, + "sol_only_rescue": { + "n": 25, + "brier": 0.14054575931758895, + "auc": 0.5238095238095238, + "bins": [ + { + "n": 21, + "mean_prediction": 0.0695281179509163, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 4, + "mean_prediction": 0.2920925547588631, + "observed_frequency": 0.25 + } + ] + } + } + } + }, + "cap_original_upfront_per_model_C1.0": { + "efficient": { + "n": 125, + "brier": 0.21487591950282664, + "auc": 0.643437862950058, + "bins": [ + { + "n": 1, + "mean_prediction": 0.16121382253171718, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.29434527988269404, + "observed_frequency": 0.0 + }, + { + "n": 13, + "mean_prediction": 0.4991731160013399, + "observed_frequency": 0.6923076923076923 + }, + { + "n": 72, + "mean_prediction": 0.7299767055000576, + "observed_frequency": 0.6388888888888888 + }, + { + "n": 37, + "mean_prediction": 0.8651218558885555, + "observed_frequency": 0.7837837837837838 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21545835040915215, + "auc": 0.5565638233514821, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38229007133429044, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.48540807315296636, + "observed_frequency": 0.0 + }, + { + "n": 62, + "mean_prediction": 0.7449979650518022, + "observed_frequency": 0.6935483870967742 + }, + { + "n": 60, + "mean_prediction": 0.8619209259445029, + "observed_frequency": 0.7333333333333333 + } + ] + }, + "gap_mse": 0.1549157397759275, + "predicted_mean_gap": 0.05960999402661531, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6299840510366825, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.251852547495908, + "auc": 0.6666666666666666, + "bins": [ + { + "n": 1, + "mean_prediction": 0.16121382253171718, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.21639030958729122, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.4830239547920591, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 16, + "mean_prediction": 0.7335550902460288, + "observed_frequency": 0.5 + }, + { + "n": 1, + "mean_prediction": 0.8437907198728529, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2549590803118468, + "auc": 0.5909090909090909, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38229007133429044, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.48540807315296636, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.7217046395253081, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 10, + "mean_prediction": 0.8343312541280412, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.20977700883433306, + "predicted_mean_gap": 0.10001817634174617, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19049473966512714, + "auc": 0.5603070175438597, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5488948385357809, + "observed_frequency": 1.0 + }, + { + "n": 18, + "mean_prediction": 0.7413171701165587, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 31, + "mean_prediction": 0.8684711986080795, + "observed_frequency": 0.7741935483870968 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20024743170144074, + "auc": 0.5374220374220374, + "bins": [ + { + "n": 8, + "mean_prediction": 0.7115201910282071, + "observed_frequency": 0.625 + }, + { + "n": 42, + "mean_prediction": 0.8691817574451719, + "observed_frequency": 0.7619047619047619 + } + ] + }, + "gap_mse": 0.10582504813726028, + "predicted_mean_gap": 0.027651685668771468, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.5729166666666666 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.23444262481708145, + "auc": 0.6805555555555556, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5843762128843636, + "observed_frequency": 0.5 + }, + { + "n": 18, + "mean_prediction": 0.7276852530794122, + "observed_frequency": 0.6111111111111112 + }, + { + "n": 5, + "mean_prediction": 0.8486221582306458, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19940750075049063, + "auc": 0.6666666666666666, + "bins": [ + { + "n": 22, + "mean_prediction": 0.7599564854719414, + "observed_frequency": 0.6818181818181818 + }, + { + "n": 3, + "mean_prediction": 0.8330987285102216, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.07079617907245593, + "predicted_mean_gap": 0.02832564374247998, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.9347826086956521 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2070949458708894, + "auc": 0.6764705882352942, + "bins": [ + { + "n": 1, + "mean_prediction": 0.37230025017809687, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.4683648787401388, + "observed_frequency": 0.75 + }, + { + "n": 20, + "mean_prediction": 0.7189698867270101, + "observed_frequency": 0.7 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22243030758054202, + "auc": 0.23015873015873017, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7559106975149834, + "observed_frequency": 0.75 + }, + { + "n": 5, + "mean_prediction": 0.8734026034323744, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.28235541469832803, + "predicted_mean_gap": 0.11440277871130744, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.48809523809523814 + } + } + }, + "cap_original_upfront_rescue_C1.0": { + "sol_only_rescue": { + "n": 125, + "brier": 0.08098389746060136, + "auc": 0.6116427432216905, + "bins": [ + { + "n": 117, + "mean_prediction": 0.07044439549245314, + "observed_frequency": 0.08547008547008547 + }, + { + "n": 8, + "mean_prediction": 0.24371366434908592, + "observed_frequency": 0.125 + } + ] + } + }, + "v2_empirical_upfront_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.20614602966847304, + "auc": 0.679732868757259, + "bins": [ + { + "n": 14, + "mean_prediction": 0.5592501035333568, + "observed_frequency": 0.35714285714285715 + }, + { + "n": 58, + "mean_prediction": 0.6849112164846393, + "observed_frequency": 0.6724137931034483 + }, + { + "n": 53, + "mean_prediction": 0.8262493609452123, + "observed_frequency": 0.7547169811320755 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21078160725794215, + "auc": 0.6358136721113127, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5469473101052138, + "observed_frequency": 0.0 + }, + { + "n": 68, + "mean_prediction": 0.7315550309308635, + "observed_frequency": 0.6617647058823529 + }, + { + "n": 56, + "mean_prediction": 0.8500830125525453, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.15106011643393696, + "predicted_mean_gap": 0.052414159845393084, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6467304625199362, + "sol_only_rescue": { + "n": 125, + "brier": 0.07915810297684357, + "auc": 0.6539074960127592, + "bins": [ + { + "n": 125, + "mean_prediction": 0.09147465916930164, + "observed_frequency": 0.088 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.23267478328247318, + "auc": 0.8076923076923077, + "bins": [ + { + "n": 7, + "mean_prediction": 0.548700819886402, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 15, + "mean_prediction": 0.6814580100271866, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 3, + "mean_prediction": 0.824160379060679, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.24226287696249313, + "auc": 0.8246753246753247, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5469473101052138, + "observed_frequency": 0.0 + }, + { + "n": 20, + "mean_prediction": 0.7131054223387527, + "observed_frequency": 0.55 + }, + { + "n": 4, + "mean_prediction": 0.8389054225050308, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.20305957306776407, + "predicted_mean_gap": 0.0651768168042296, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.4545454545454545, + "sol_only_rescue": { + "n": 25, + "brier": 0.1080654158532295, + "auc": 0.4545454545454545, + "bins": [ + { + "n": 25, + "mean_prediction": 0.11380770464791952, + "observed_frequency": 0.12 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.18690428069273377, + "auc": 0.5701754385964912, + "bins": [ + { + "n": 2, + "mean_prediction": 0.7958290172277925, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.8257839939314301, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20471339507554998, + "auc": 0.6133056133056133, + "bins": [ + { + "n": 50, + "mean_prediction": 0.8507768113197909, + "observed_frequency": 0.74 + } + ] + }, + "gap_mse": 0.10223690699459223, + "predicted_mean_gap": 0.026191016456506268, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.53125, + "sol_only_rescue": { + "n": 50, + "brier": 0.039114126318372536, + "auc": 0.5, + "bins": [ + { + "n": 50, + "mean_prediction": 0.05598018946360002, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.21704544995216324, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 25, + "mean_prediction": 0.685798204703094, + "observed_frequency": 0.64 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19692586930075023, + "auc": 0.7222222222222222, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7493935961465694, + "observed_frequency": 0.72 + } + ] + }, + "gap_mse": 0.0727382215111854, + "predicted_mean_gap": 0.0635953914434754, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6521739130434783, + "sol_only_rescue": { + "n": 25, + "brier": 0.07459963628267795, + "auc": 0.6304347826086957, + "bins": [ + { + "n": 25, + "mean_prediction": 0.10674067594073243, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.207201353722261, + "auc": 0.6323529411764706, + "bins": [ + { + "n": 7, + "mean_prediction": 0.5697993871803118, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 16, + "mean_prediction": 0.6728979533542716, + "observed_frequency": 0.6875 + }, + { + "n": 2, + "mean_prediction": 0.8405516421027848, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20529249987536755, + "auc": 0.5079365079365079, + "bins": [ + { + "n": 23, + "mean_prediction": 0.7282084240373664, + "observed_frequency": 0.6956521739130435 + }, + { + "n": 2, + "mean_prediction": 0.8550932234664312, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.275028973601551, + "predicted_mean_gap": 0.08091655806624784, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5595238095238095, + "sol_only_rescue": { + "n": 25, + "brier": 0.13489721011156525, + "auc": 0.5357142857142857, + "bins": [ + { + "n": 25, + "mean_prediction": 0.12486453633065611, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_empirical_upfront_paired_C1.0": { + "efficient": { + "n": 125, + "brier": 0.2080943989531877, + "auc": 0.6736353077816493, + "bins": [ + { + "n": 4, + "mean_prediction": 0.15644375818272682, + "observed_frequency": 0.5 + }, + { + "n": 10, + "mean_prediction": 0.31150298086087785, + "observed_frequency": 0.4 + }, + { + "n": 8, + "mean_prediction": 0.5588278855410702, + "observed_frequency": 0.25 + }, + { + "n": 55, + "mean_prediction": 0.7094333493962423, + "observed_frequency": 0.7090909090909091 + }, + { + "n": 48, + "mean_prediction": 0.8537685801458172, + "observed_frequency": 0.7708333333333334 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21360032306533672, + "auc": 0.6137326073805203, + "bins": [ + { + "n": 1, + "mean_prediction": 0.17075662235704414, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.33976772838680547, + "observed_frequency": 0.75 + }, + { + "n": 5, + "mean_prediction": 0.5151525354022158, + "observed_frequency": 0.2 + }, + { + "n": 55, + "mean_prediction": 0.7426988360322682, + "observed_frequency": 0.7090909090909091 + }, + { + "n": 60, + "mean_prediction": 0.8613185302743246, + "observed_frequency": 0.7333333333333333 + } + ] + }, + "gap_mse": 0.15682659187926692, + "predicted_mean_gap": 0.06737587217351008, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.5454545454545454, + "sol_only_rescue": { + "n": 125, + "brier": 0.09036826846799315, + "auc": 0.5574162679425837, + "bins": [ + { + "n": 112, + "mean_prediction": 0.07325571273365372, + "observed_frequency": 0.08928571428571429 + }, + { + "n": 9, + "mean_prediction": 0.30635966893978644, + "observed_frequency": 0.1111111111111111 + }, + { + "n": 3, + "mean_prediction": 0.428030746619577, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.6652938490985272, + "observed_frequency": 0.0 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.18543689404289726, + "auc": 0.8269230769230769, + "bins": [ + { + "n": 2, + "mean_prediction": 0.1487132700934445, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.31977350532684157, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.5688527980972494, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 14, + "mean_prediction": 0.696359329420184, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 3, + "mean_prediction": 0.851002535914096, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20632419886038997, + "auc": 0.7662337662337662, + "bins": [ + { + "n": 1, + "mean_prediction": 0.17075662235704414, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.3523191492394685, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.5151525354022158, + "observed_frequency": 0.2 + }, + { + "n": 12, + "mean_prediction": 0.7536107186692327, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 6, + "mean_prediction": 0.834100926820931, + "observed_frequency": 0.8333333333333334 + } + ] + }, + "gap_mse": 0.21189705150266153, + "predicted_mean_gap": 0.07525715853919776, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.303030303030303, + "sol_only_rescue": { + "n": 25, + "brier": 0.12866363917485774, + "auc": 0.3333333333333333, + "bins": [ + { + "n": 21, + "mean_prediction": 0.09084986563564264, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 2, + "mean_prediction": 0.26677127148198065, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.436230798411804, + "observed_frequency": 0.0 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19168814317670066, + "auc": 0.5745614035087719, + "bins": [ + { + "n": 8, + "mean_prediction": 0.7490222205011471, + "observed_frequency": 0.75 + }, + { + "n": 42, + "mean_prediction": 0.8545431022757461, + "observed_frequency": 0.7619047619047619 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20938010339620747, + "auc": 0.604989604989605, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7604430685421785, + "observed_frequency": 1.0 + }, + { + "n": 47, + "mean_prediction": 0.8648773647651561, + "observed_frequency": 0.723404255319149 + } + ] + }, + "gap_mse": 0.1015502040973808, + "predicted_mean_gap": 0.020951545799966984, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.5104166666666667, + "sol_only_rescue": { + "n": 50, + "brier": 0.03981500899372741, + "auc": 0.5, + "bins": [ + { + "n": 50, + "mean_prediction": 0.04846643458664766, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.20137012683964148, + "auc": 0.7847222222222222, + "bins": [ + { + "n": 3, + "mean_prediction": 0.5540030399384587, + "observed_frequency": 0.0 + }, + { + "n": 22, + "mean_prediction": 0.6992186345662055, + "observed_frequency": 0.7272727272727273 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.190760341718989, + "auc": 0.746031746031746, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7444086986569519, + "observed_frequency": 0.72 + } + ] + }, + "gap_mse": 0.07131746482173262, + "predicted_mean_gap": 0.06261593544607624, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6739130434782609, + "sol_only_rescue": { + "n": 25, + "brier": 0.07580317490467339, + "auc": 0.5217391304347826, + "bins": [ + { + "n": 25, + "mean_prediction": 0.09726600462104401, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.27028868752999846, + "auc": 0.5514705882352942, + "bins": [ + { + "n": 2, + "mean_prediction": 0.1641742462720091, + "observed_frequency": 1.0 + }, + { + "n": 7, + "mean_prediction": 0.3079584703754648, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 2, + "mean_prediction": 0.5510277851107188, + "observed_frequency": 0.5 + }, + { + "n": 11, + "mean_prediction": 0.7177105345859156, + "observed_frequency": 0.6363636363636364 + }, + { + "n": 3, + "mean_prediction": 0.8456913145585326, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.25215686795488956, + "auc": 0.48412698412698413, + "bins": [ + { + "n": 3, + "mean_prediction": 0.3355839214359178, + "observed_frequency": 1.0 + }, + { + "n": 15, + "mean_prediction": 0.727570712379575, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 7, + "mean_prediction": 0.8607528730816514, + "observed_frequency": 0.7142857142857143 + } + ] + }, + "gap_mse": 0.2978180348771788, + "predicted_mean_gap": 0.15710317528234244, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.48809523809523814, + "sol_only_rescue": { + "n": 25, + "brier": 0.16774451027297987, + "auc": 0.47619047619047616, + "bins": [ + { + "n": 16, + "mean_prediction": 0.09011380018513991, + "observed_frequency": 0.1875 + }, + { + "n": 7, + "mean_prediction": 0.31767063964201664, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 1, + "mean_prediction": 0.4116306430351231, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.6652938490985272, + "observed_frequency": 0.0 + } + ] + } + } + } + }, + "v2_empirical_upfront_per_model_C1.0": { + "efficient": { + "n": 125, + "brier": 0.20657867476950909, + "auc": 0.6750871080139373, + "bins": [ + { + "n": 2, + "mean_prediction": 0.17533662530467625, + "observed_frequency": 0.5 + }, + { + "n": 10, + "mean_prediction": 0.30144249797903244, + "observed_frequency": 0.4 + }, + { + "n": 9, + "mean_prediction": 0.5314527188685427, + "observed_frequency": 0.2222222222222222 + }, + { + "n": 55, + "mean_prediction": 0.700527687016309, + "observed_frequency": 0.7090909090909091 + }, + { + "n": 49, + "mean_prediction": 0.8497572027666681, + "observed_frequency": 0.7755102040816326 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21170684744219537, + "auc": 0.6110102843315185, + "bins": [ + { + "n": 2, + "mean_prediction": 0.3286678524251529, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5251891302489499, + "observed_frequency": 0.625 + }, + { + "n": 57, + "mean_prediction": 0.753289270092932, + "observed_frequency": 0.7017543859649122 + }, + { + "n": 58, + "mean_prediction": 0.8627523548403426, + "observed_frequency": 0.7241379310344828 + } + ] + }, + "gap_mse": 0.15730952554883526, + "predicted_mean_gap": 0.07616540240958886, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6204146730462521, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.19446320585991278, + "auc": 0.8012820512820513, + "bins": [ + { + "n": 1, + "mean_prediction": 0.17030812553329366, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.2804262563941234, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.4982774513801156, + "observed_frequency": 0.0 + }, + { + "n": 15, + "mean_prediction": 0.6833517333570753, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 4, + "mean_prediction": 0.8454265758755718, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2157219226480885, + "auc": 0.7272727272727273, + "bins": [ + { + "n": 2, + "mean_prediction": 0.3286678524251529, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.5244952757091477, + "observed_frequency": 0.4 + }, + { + "n": 15, + "mean_prediction": 0.7548579011867973, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 3, + "mean_prediction": 0.8637521691101347, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "gap_mse": 0.2154156477616112, + "predicted_mean_gap": 0.06215252028776371, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.3181818181818181 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19024135884503435, + "auc": 0.5986842105263158, + "bins": [ + { + "n": 7, + "mean_prediction": 0.751118023184539, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 43, + "mean_prediction": 0.8496762066956742, + "observed_frequency": 0.7674418604651163 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.208579449311202, + "auc": 0.6133056133056133, + "bins": [ + { + "n": 2, + "mean_prediction": 0.7631408772085947, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.8634623986342778, + "observed_frequency": 0.7291666666666666 + } + ] + }, + "gap_mse": 0.10214024588690591, + "predicted_mean_gap": 0.023571476773134994, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.53125 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.20418640315421943, + "auc": 0.7847222222222222, + "bins": [ + { + "n": 3, + "mean_prediction": 0.5549689153865006, + "observed_frequency": 0.0 + }, + { + "n": 22, + "mean_prediction": 0.6907296381938236, + "observed_frequency": 0.7272727272727273 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.1935848290660298, + "auc": 0.7380952380952381, + "bins": [ + { + "n": 24, + "mean_prediction": 0.7488403567480294, + "observed_frequency": 0.7083333333333334 + }, + { + "n": 1, + "mean_prediction": 0.8011884587899738, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.06779457375523315, + "predicted_mean_gap": 0.0764959293727624, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.8260869565217391 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2537610471433447, + "auc": 0.5588235294117647, + "bins": [ + { + "n": 1, + "mean_prediction": 0.1803651250760588, + "observed_frequency": 1.0 + }, + { + "n": 7, + "mean_prediction": 0.3104494586582792, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 4, + "mean_prediction": 0.5304032052242879, + "observed_frequency": 0.5 + }, + { + "n": 11, + "mean_prediction": 0.7113516893622699, + "observed_frequency": 0.7272727272727273 + }, + { + "n": 2, + "mean_prediction": 0.8601598720752253, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23206858687445464, + "auc": 0.4444444444444444, + "bins": [ + { + "n": 3, + "mean_prediction": 0.5263455544819534, + "observed_frequency": 1.0 + }, + { + "n": 16, + "mean_prediction": 0.7572605975703292, + "observed_frequency": 0.6875 + }, + { + "n": 6, + "mean_prediction": 0.8668327466956933, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "gap_mse": 0.29905691445352006, + "predicted_mean_gap": 0.195035608841148, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5833333333333334 + } + } + }, + "v2_empirical_upfront_rescue_C1.0": { + "sol_only_rescue": { + "n": 125, + "brier": 0.08525474469007004, + "auc": 0.5789473684210527, + "bins": [ + { + "n": 112, + "mean_prediction": 0.07622955214302686, + "observed_frequency": 0.08928571428571429 + }, + { + "n": 12, + "mean_prediction": 0.28427142383378873, + "observed_frequency": 0.08333333333333333 + }, + { + "n": 1, + "mean_prediction": 0.47459195550358957, + "observed_frequency": 0.0 + } + ] + } + }, + "v2_empirical_review_benefit_per_dollar_C0.01": { + "efficient": { + "n": 125, + "brier": 0.2112880942819632, + "auc": 0.6933797909407665, + "bins": [ + { + "n": 125, + "mean_prediction": 0.748423428112692, + "observed_frequency": 0.672 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2135266089139855, + "auc": 0.6237144585601936, + "bins": [ + { + "n": 63, + "mean_prediction": 0.7682466772610972, + "observed_frequency": 0.6190476190476191 + }, + { + "n": 62, + "mean_prediction": 0.8215671846480648, + "observed_frequency": 0.7741935483870968 + } + ] + }, + "gap_mse": 0.1507635135305867, + "predicted_mean_gap": 0.04627022081234104, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7519936204146731, + "sol_only_rescue": { + "n": 125, + "brier": 0.07894204770048503, + "auc": 0.7432216905901117, + "bins": [ + { + "n": 125, + "mean_prediction": 0.08353897476842373, + "observed_frequency": 0.088 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.24649849180878186, + "auc": 0.9551282051282051, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7141349171022814, + "observed_frequency": 0.52 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.26302867995468027, + "auc": 0.8701298701298701, + "bins": [ + { + "n": 16, + "mean_prediction": 0.7425919983569584, + "observed_frequency": 0.375 + }, + { + "n": 9, + "mean_prediction": 0.8102934385002505, + "observed_frequency": 0.8888888888888888 + } + ] + }, + "gap_mse": 0.1973426180626784, + "predicted_mean_gap": 0.05282959970626206, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6363636363636364, + "sol_only_rescue": { + "n": 25, + "brier": 0.10542548757799036, + "auc": 0.6363636363636364, + "bins": [ + { + "n": 25, + "mean_prediction": 0.09453413502816466, + "observed_frequency": 0.12 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1813557390507603, + "auc": 0.49122807017543857, + "bins": [ + { + "n": 50, + "mean_prediction": 0.7880444684447534, + "observed_frequency": 0.76 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.19896197510709127, + "auc": 0.4490644490644491, + "bins": [ + { + "n": 50, + "mean_prediction": 0.8243777135690532, + "observed_frequency": 0.74 + } + ] + }, + "gap_mse": 0.10266858123475395, + "predicted_mean_gap": 0.03633324512429957, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.96875, + "sol_only_rescue": { + "n": 50, + "brier": 0.03887840062044573, + "auc": 0.96875, + "bins": [ + { + "n": 50, + "mean_prediction": 0.07164732277157736, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.23898826861324188, + "auc": 0.625, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7360993887082864, + "observed_frequency": 0.64 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2068442735792015, + "auc": 0.507936507936508, + "bins": [ + { + "n": 24, + "mean_prediction": 0.7855037419478069, + "observed_frequency": 0.7083333333333334 + }, + { + "n": 1, + "mean_prediction": 0.8024856236167205, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.07418679787593897, + "predicted_mean_gap": 0.05008362850627704, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6739130434782609, + "sol_only_rescue": { + "n": 25, + "brier": 0.07351623992325741, + "auc": 0.5652173913043479, + "bins": [ + { + "n": 25, + "mean_prediction": 0.08684112692043212, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2082422328862718, + "auc": 0.7647058823529411, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7157938978633855, + "observed_frequency": 0.68 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19983614082186324, + "auc": 0.6587301587301587, + "bins": [ + { + "n": 23, + "mean_prediction": 0.7680860385648011, + "observed_frequency": 0.6956521739130435 + }, + { + "n": 2, + "mean_prediction": 0.811576599804194, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.27695098924480827, + "predicted_mean_gap": 0.05577138560056696, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6190476190476191, + "sol_only_rescue": { + "n": 25, + "brier": 0.13801170976028596, + "auc": 0.5952380952380952, + "bins": [ + { + "n": 25, + "mean_prediction": 0.0930249663503671, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_original_upfront_per_model_C1.0": { + "efficient": { + "n": 125, + "brier": 0.20586976539836235, + "auc": 0.6689895470383276, + "bins": [ + { + "n": 2, + "mean_prediction": 0.16734055590238106, + "observed_frequency": 0.0 + }, + { + "n": 10, + "mean_prediction": 0.31257471880364485, + "observed_frequency": 0.5 + }, + { + "n": 17, + "mean_prediction": 0.525050269638746, + "observed_frequency": 0.5882352941176471 + }, + { + "n": 42, + "mean_prediction": 0.685490493237321, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 54, + "mean_prediction": 0.867196084861678, + "observed_frequency": 0.7592592592592593 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21463596402764343, + "auc": 0.6046581972171808, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38349530742347737, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.552574380463959, + "observed_frequency": 0.5833333333333334 + }, + { + "n": 41, + "mean_prediction": 0.7310730169579913, + "observed_frequency": 0.6341463414634146 + }, + { + "n": 71, + "mean_prediction": 0.8737483687005557, + "observed_frequency": 0.7605633802816901 + } + ] + }, + "gap_mse": 0.15835986324432882, + "predicted_mean_gap": 0.08815234851048069, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.665071770334928, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.19488912439376446, + "auc": 0.8076923076923077, + "bins": [ + { + "n": 2, + "mean_prediction": 0.16734055590238106, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.2802147089293126, + "observed_frequency": 0.0 + }, + { + "n": 7, + "mean_prediction": 0.5046435849841482, + "observed_frequency": 0.42857142857142855 + }, + { + "n": 11, + "mean_prediction": 0.7439326104067401, + "observed_frequency": 0.6363636363636364 + }, + { + "n": 4, + "mean_prediction": 0.8810929519268931, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21868500464071708, + "auc": 0.7922077922077922, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38349530742347737, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.5529075674542637, + "observed_frequency": 0.2 + }, + { + "n": 7, + "mean_prediction": 0.6993408522286815, + "observed_frequency": 0.42857142857142855 + }, + { + "n": 12, + "mean_prediction": 0.8521454465974614, + "observed_frequency": 0.8333333333333334 + } + ] + }, + "gap_mse": 0.23149067001288734, + "predicted_mean_gap": 0.09656532126641114, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.4090909090909091 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19701885570569297, + "auc": 0.5877192982456141, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7361446911347711, + "observed_frequency": 1.0 + }, + { + "n": 47, + "mean_prediction": 0.866992066203904, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.2168710265280743, + "auc": 0.49896049896049893, + "bins": [ + { + "n": 1, + "mean_prediction": 0.7152948899908204, + "observed_frequency": 1.0 + }, + { + "n": 49, + "mean_prediction": 0.8868058951771391, + "observed_frequency": 0.7346938775510204 + } + ] + }, + "gap_mse": 0.10274702190273528, + "predicted_mean_gap": 0.02423445137365687, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.8229166666666667 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.1825976903237712, + "auc": 0.861111111111111, + "bins": [ + { + "n": 1, + "mean_prediction": 0.3066542216250509, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.547373057749475, + "observed_frequency": 0.5 + }, + { + "n": 19, + "mean_prediction": 0.6659394144995158, + "observed_frequency": 0.6842105263157895 + }, + { + "n": 1, + "mean_prediction": 0.8515193573703993, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18390186166529454, + "auc": 0.880952380952381, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7422146865505383, + "observed_frequency": 0.65 + }, + { + "n": 5, + "mean_prediction": 0.8203560846538114, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.06890436328025466, + "predicted_mean_gap": 0.11782237875182698, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6956521739130435 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2578243008628901, + "auc": 0.49264705882352944, + "bins": [ + { + "n": 8, + "mean_prediction": 0.3173597821852606, + "observed_frequency": 0.625 + }, + { + "n": 6, + "mean_prediction": 0.5339762096619575, + "observed_frequency": 0.8333333333333334 + }, + { + "n": 9, + "mean_prediction": 0.6384510058442484, + "observed_frequency": 0.5555555555555556 + }, + { + "n": 2, + "mean_prediction": 0.8520351529345824, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23685090077605683, + "auc": 0.380952380952381, + "bins": [ + { + "n": 7, + "mean_prediction": 0.5523363897565986, + "observed_frequency": 0.8571428571428571 + }, + { + "n": 13, + "mean_prediction": 0.7322322391288681, + "observed_frequency": 0.6923076923076923 + }, + { + "n": 5, + "mean_prediction": 0.8510239063242085, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.2859102391230315, + "predicted_mean_gap": 0.17790513978685155, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6309523809523809 + } + } + }, + "v2_original_review_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.19475731872208132, + "auc": 0.6823461091753774, + "bins": [ + { + "n": 7, + "mean_prediction": 0.2730479048664671, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.572792908294837, + "observed_frequency": 0.5 + }, + { + "n": 48, + "mean_prediction": 0.7167708857902779, + "observed_frequency": 0.6041666666666666 + }, + { + "n": 62, + "mean_prediction": 0.8532704382769977, + "observed_frequency": 0.8225806451612904 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.205236316674579, + "auc": 0.6167574107683, + "bins": [ + { + "n": 7, + "mean_prediction": 0.45844405121046133, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 42, + "mean_prediction": 0.7462895669734516, + "observed_frequency": 0.6428571428571429 + }, + { + "n": 76, + "mean_prediction": 0.8621683001695367, + "observed_frequency": 0.7763157894736842 + } + ] + }, + "gap_mse": 0.15083555656813358, + "predicted_mean_gap": 0.050212901541694555, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7073365231259968, + "sol_only_rescue": { + "n": 125, + "brier": 0.08031896724608772, + "auc": 0.6842105263157895, + "bins": [ + { + "n": 120, + "mean_prediction": 0.07198762815203195, + "observed_frequency": 0.09166666666666666 + }, + { + "n": 5, + "mean_prediction": 0.2500931349617063, + "observed_frequency": 0.0 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.13181318837313602, + "auc": 0.9487179487179488, + "bins": [ + { + "n": 7, + "mean_prediction": 0.2730479048664671, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5547376465205437, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.7248448731838749, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 10, + "mean_prediction": 0.8399584582796091, + "observed_frequency": 0.9 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18787617751354888, + "auc": 0.9090909090909091, + "bins": [ + { + "n": 7, + "mean_prediction": 0.45844405121046133, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 7, + "mean_prediction": 0.7329031940881119, + "observed_frequency": 0.2857142857142857 + }, + { + "n": 11, + "mean_prediction": 0.8544242032361381, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.20039222751558342, + "predicted_mean_gap": 0.07874530014727336, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5909090909090908, + "sol_only_rescue": { + "n": 25, + "brier": 0.11162071470745126, + "auc": 0.5909090909090908, + "bins": [ + { + "n": 20, + "mean_prediction": 0.08852120639588114, + "observed_frequency": 0.15 + }, + { + "n": 5, + "mean_prediction": 0.2500931349617063, + "observed_frequency": 0.0 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.18695068562083517, + "auc": 0.5043859649122807, + "bins": [ + { + "n": 4, + "mean_prediction": 0.7378031217945231, + "observed_frequency": 0.5 + }, + { + "n": 46, + "mean_prediction": 0.8613292739755435, + "observed_frequency": 0.782608695652174 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.208890426637236, + "auc": 0.3866943866943867, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7717796010734302, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.8742327268387793, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "gap_mse": 0.10035736842153892, + "predicted_mean_gap": 0.016638357491796322, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9791666666666666, + "sol_only_rescue": { + "n": 50, + "brier": 0.0361568271447332, + "auc": 0.9583333333333334, + "bins": [ + { + "n": 50, + "mean_prediction": 0.04666064024782396, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.2505947865898087, + "auc": 0.4722222222222222, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5984650151506499, + "observed_frequency": 1.0 + }, + { + "n": 20, + "mean_prediction": 0.7368278035525606, + "observed_frequency": 0.55 + }, + { + "n": 4, + "mean_prediction": 0.8139818632578771, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21767436620765138, + "auc": 0.40476190476190477, + "bins": [ + { + "n": 12, + "mean_prediction": 0.7745779866915775, + "observed_frequency": 0.75 + }, + { + "n": 13, + "mean_prediction": 0.8351767748528173, + "observed_frequency": 0.6923076923076923 + } + ] + }, + "gap_mse": 0.07268589804642063, + "predicted_mean_gap": 0.062451414966087394, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.7391304347826086, + "sol_only_rescue": { + "n": 25, + "brier": 0.07280073531343963, + "auc": 0.6086956521739131, + "bins": [ + { + "n": 25, + "mean_prediction": 0.08365296583491949, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2174772474057914, + "auc": 0.5808823529411765, + "bins": [ + { + "n": 5, + "mean_prediction": 0.5748805916333917, + "observed_frequency": 0.6 + }, + { + "n": 18, + "mean_prediction": 0.6871202622555995, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 2, + "mean_prediction": 0.8130542672356273, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20285018637722257, + "auc": 0.5317460317460317, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7301782405374482, + "observed_frequency": 0.7 + }, + { + "n": 5, + "mean_prediction": 0.835977668555605, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.2803849204355858, + "predicted_mean_gap": 0.07659107761151943, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.33333333333333337, + "sol_only_rescue": { + "n": 25, + "brier": 0.1448597319200813, + "auc": 0.2619047619047619, + "bins": [ + { + "n": 25, + "mean_prediction": 0.09774940368248107, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_original_review_rescue_C1.0": { + "sol_only_rescue": { + "n": 125, + "brier": 0.08576811223362082, + "auc": 0.5773524720893142, + "bins": [ + { + "n": 117, + "mean_prediction": 0.05753457988877527, + "observed_frequency": 0.09401709401709402 + }, + { + "n": 6, + "mean_prediction": 0.29215656863558076, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.4430515794724206, + "observed_frequency": 0.0 + } + ] + } + } + } +} \ No newline at end of file diff --git a/cookbook/auto_router_selective_training/development_pair_counts.json b/cookbook/auto_router_selective_training/development_pair_counts.json new file mode 100644 index 00000000000..07a56eb2715 --- /dev/null +++ b/cookbook/auto_router_selective_training/development_pair_counts.json @@ -0,0 +1,62 @@ +{ + "train": { + "lcb": { + "tasks": 50, + "both_fail": 14, + "sol_only": 9, + "luna_only": 3, + "both_solve": 24 + }, + "mbpp": { + "tasks": 80, + "both_fail": 7, + "sol_only": 2, + "luna_only": 3, + "both_solve": 68 + }, + "swe": { + "tasks": 25, + "both_fail": 5, + "sol_only": 1, + "luna_only": 0, + "both_solve": 19 + }, + "terminal": { + "tasks": 5, + "both_fail": 0, + "sol_only": 1, + "luna_only": 0, + "both_solve": 4 + } + }, + "validation": { + "lcb": { + "tasks": 20, + "both_fail": 2, + "sol_only": 6, + "luna_only": 2, + "both_solve": 10 + }, + "mbpp": { + "tasks": 29, + "both_fail": 4, + "sol_only": 2, + "luna_only": 2, + "both_solve": 21 + }, + "swe": { + "tasks": 10, + "both_fail": 1, + "sol_only": 1, + "luna_only": 1, + "both_solve": 7 + }, + "terminal": { + "tasks": 5, + "both_fail": 2, + "sol_only": 0, + "luna_only": 1, + "both_solve": 2 + } + } +} \ No newline at end of file diff --git a/cookbook/auto_router_selective_training/execution_diagnostics.json b/cookbook/auto_router_selective_training/execution_diagnostics.json new file mode 100644 index 00000000000..fc9a782f93a --- /dev/null +++ b/cookbook/auto_router_selective_training/execution_diagnostics.json @@ -0,0 +1,315 @@ +{ + "scope": "Execution diagnostics for completed attempts, without reading grades. A length stop is a model output-budget event, not an infrastructure failure. All recorded responses remain billed.", + "groups": [ + { + "split": "test", + "benchmark": "lcb", + "model": "anthropic/claude-opus-5", + "attempts": 25, + "responses": 25, + "response_cost": 2.384855, + "token_limit_responses": 7, + "token_limit_response_cost": 1.4849325, + "token_limit_without_action_or_text": 7, + "token_limit_without_action_or_text_cost": 1.4849325 + }, + { + "split": "test", + "benchmark": "lcb", + "model": "anthropic/claude-sonnet-5", + "attempts": 25, + "responses": 25, + "response_cost": 1.3789665000000002, + "token_limit_responses": 14, + "token_limit_response_cost": 1.1779045000000001, + "token_limit_without_action_or_text": 14, + "token_limit_without_action_or_text_cost": 1.1779045000000001 + }, + { + "split": "test", + "benchmark": "lcb", + "model": "openai/gpt-5.6-luna", + "attempts": 25, + "responses": 25, + "response_cost": 0.1164804, + "token_limit_responses": 7, + "token_limit_response_cost": 0.06988079999999999, + "token_limit_without_action_or_text": 7, + "token_limit_without_action_or_text_cost": 0.06988079999999999 + }, + { + "split": "test", + "benchmark": "lcb", + "model": "openai/gpt-5.6-sol", + "attempts": 25, + "responses": 25, + "response_cost": 1.529944, + "token_limit_responses": 4, + "token_limit_response_cost": 0.667568, + "token_limit_without_action_or_text": 4, + "token_limit_without_action_or_text_cost": 0.667568 + }, + { + "split": "test", + "benchmark": "mbpp", + "model": "anthropic/claude-opus-5", + "attempts": 50, + "responses": 50, + "response_cost": 0.224865 + }, + { + "split": "test", + "benchmark": "mbpp", + "model": "anthropic/claude-sonnet-5", + "attempts": 50, + "responses": 50, + "response_cost": 0.09884600000000003 + }, + { + "split": "test", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-luna", + "attempts": 50, + "responses": 50, + "response_cost": 0.017396 + }, + { + "split": "test", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-sol", + "attempts": 50, + "responses": 50, + "response_cost": 0.20637600000000006 + }, + { + "split": "test", + "benchmark": "swe", + "model": "anthropic/claude-opus-5", + "attempts": 25, + "responses": 381, + "response_cost": 10.529565750000002 + }, + { + "split": "test", + "benchmark": "swe", + "model": "anthropic/claude-sonnet-5", + "attempts": 25, + "responses": 833, + "response_cost": 11.053656000000016, + "token_limit_responses": 4, + "token_limit_response_cost": 0.4144736, + "token_limit_without_action_or_text": 4, + "token_limit_without_action_or_text_cost": 0.4144736 + }, + { + "split": "test", + "benchmark": "swe", + "model": "openai/gpt-5.6-luna", + "attempts": 25, + "responses": 427, + "response_cost": 0.6371064399999993 + }, + { + "split": "test", + "benchmark": "swe", + "model": "openai/gpt-5.6-sol", + "attempts": 25, + "responses": 403, + "response_cost": 10.713241600000002 + }, + { + "split": "test", + "benchmark": "terminal", + "model": "anthropic/claude-opus-5", + "attempts": 25, + "responses": 414, + "response_cost": 32.915727249999996, + "token_limit_responses": 80, + "token_limit_response_cost": 17.381599, + "token_limit_without_action_or_text": 76, + "token_limit_without_action_or_text_cost": 16.427720750000002 + }, + { + "split": "test", + "benchmark": "terminal", + "model": "anthropic/claude-sonnet-5", + "attempts": 25, + "responses": 795, + "response_cost": 25.464663800000025, + "token_limit_responses": 106, + "token_limit_response_cost": 9.517311799999998, + "token_limit_without_action_or_text": 98, + "token_limit_without_action_or_text_cost": 8.773203099999998 + }, + { + "split": "test", + "benchmark": "terminal", + "model": "openai/gpt-5.6-luna", + "attempts": 25, + "responses": 839, + "response_cost": 1.8821569099999986, + "token_limit_responses": 43, + "token_limit_response_cost": 0.43733629999999996, + "token_limit_without_action_or_text": 35, + "token_limit_without_action_or_text_cost": 0.35178958 + }, + { + "split": "test", + "benchmark": "terminal", + "model": "openai/gpt-5.6-sol", + "attempts": 25, + "responses": 341, + "response_cost": 8.446844399999998, + "token_limit_responses": 3, + "token_limit_response_cost": 0.5060674000000001 + }, + { + "split": "train", + "benchmark": "lcb", + "model": "openai/gpt-5.6-luna", + "attempts": 50, + "responses": 50, + "response_cost": 0.2573764, + "token_limit_responses": 16, + "token_limit_response_cost": 0.1596568, + "token_limit_without_action_or_text": 16, + "token_limit_without_action_or_text_cost": 0.1596568 + }, + { + "split": "train", + "benchmark": "lcb", + "model": "openai/gpt-5.6-sol", + "attempts": 50, + "responses": 50, + "response_cost": 2.630148, + "token_limit_responses": 5, + "token_limit_response_cost": 0.836052, + "token_limit_without_action_or_text": 5, + "token_limit_without_action_or_text_cost": 0.836052 + }, + { + "split": "train", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-luna", + "attempts": 80, + "responses": 80, + "response_cost": 0.0218284 + }, + { + "split": "train", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-sol", + "attempts": 80, + "responses": 80, + "response_cost": 0.289156 + }, + { + "split": "train", + "benchmark": "swe", + "model": "openai/gpt-5.6-luna", + "attempts": 25, + "responses": 398, + "response_cost": 0.5176403399999997 + }, + { + "split": "train", + "benchmark": "swe", + "model": "openai/gpt-5.6-sol", + "attempts": 25, + "responses": 346, + "response_cost": 7.403714199999999 + }, + { + "split": "train", + "benchmark": "terminal", + "model": "openai/gpt-5.6-luna", + "attempts": 5, + "responses": 87, + "response_cost": 0.12343140999999999 + }, + { + "split": "train", + "benchmark": "terminal", + "model": "openai/gpt-5.6-sol", + "attempts": 5, + "responses": 34, + "response_cost": 0.7672414000000001 + }, + { + "split": "validation", + "benchmark": "lcb", + "model": "openai/gpt-5.6-luna", + "attempts": 20, + "responses": 20, + "response_cost": 0.0921004, + "token_limit_responses": 6, + "token_limit_response_cost": 0.0598586, + "token_limit_without_action_or_text": 6, + "token_limit_without_action_or_text_cost": 0.0598586 + }, + { + "split": "validation", + "benchmark": "lcb", + "model": "openai/gpt-5.6-sol", + "attempts": 20, + "responses": 20, + "response_cost": 1.0552880000000002, + "token_limit_responses": 1, + "token_limit_response_cost": 0.166696, + "token_limit_without_action_or_text": 1, + "token_limit_without_action_or_text_cost": 0.166696 + }, + { + "split": "validation", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-luna", + "attempts": 30, + "responses": 30, + "response_cost": 0.010364400000000001 + }, + { + "split": "validation", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-sol", + "attempts": 30, + "responses": 30, + "response_cost": 0.14699600000000002 + }, + { + "split": "validation", + "benchmark": "swe", + "model": "openai/gpt-5.6-luna", + "attempts": 10, + "responses": 149, + "response_cost": 0.2096464700000001 + }, + { + "split": "validation", + "benchmark": "swe", + "model": "openai/gpt-5.6-sol", + "attempts": 10, + "responses": 141, + "response_cost": 3.702219799999998 + }, + { + "split": "validation", + "benchmark": "terminal", + "model": "openai/gpt-5.6-luna", + "attempts": 5, + "responses": 66, + "response_cost": 0.10991243999999999, + "token_limit_responses": 1, + "token_limit_response_cost": 0.009907 + }, + { + "split": "validation", + "benchmark": "terminal", + "model": "openai/gpt-5.6-sol", + "attempts": 5, + "responses": 63, + "response_cost": 2.5312083999999997, + "token_limit_responses": 6, + "token_limit_response_cost": 0.9978314000000001 + } + ] +} \ No newline at end of file diff --git a/cookbook/auto_router_selective_training/manifest.json b/cookbook/auto_router_selective_training/manifest.json index d1eca8caed0..8de8767b0f1 100644 --- a/cookbook/auto_router_selective_training/manifest.json +++ b/cookbook/auto_router_selective_training/manifest.json @@ -4,7 +4,7 @@ "training_tasks": 160, "validation_tasks": 64, "held_out_tasks": 125, - "held_out_status": "primary_pair_complete_anthropic_controls_in_progress", + "held_out_status": "complete", "current_model_pair": [ "openai/gpt-5.6-luna", "openai/gpt-5.6-sol" @@ -13,5 +13,6 @@ "solver_effort": "high", "diagnostic_upfront_profiles": 12, "diagnostic_profiles_sha256": "468de3b22e309ca3ef800ef90bd9b03a8aca3eebef32bcc2155377b4cf7e1f88", - "primary_results_sha256": "e91ec5578d496ad37c1e8cf5760882042f4b7768bd3c2ff5d65d92f19df7a12c" + "primary_results_sha256": "e91ec5578d496ad37c1e8cf5760882042f4b7768bd3c2ff5d65d92f19df7a12c", + "final_results_sha256": "b724f5607b532e1d8b354c95129f5f6d95514926ca5000e9a93a9104b13a844d" } diff --git a/cookbook/auto_router_selective_training/quality_cost.png b/cookbook/auto_router_selective_training/quality_cost.png new file mode 100644 index 00000000000..aa381d853bc Binary files /dev/null and b/cookbook/auto_router_selective_training/quality_cost.png differ diff --git a/cookbook/auto_router_selective_training/quality_cost.svg b/cookbook/auto_router_selective_training/quality_cost.svg new file mode 100644 index 00000000000..c703fe285a4 --- /dev/null +++ b/cookbook/auto_router_selective_training/quality_cost.svg @@ -0,0 +1,4023 @@ + + + + + + + + 2026-09-16T21:07:55.079486 + image/svg+xml + + + Matplotlib v3.10.6, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/cookbook/auto_router_selective_training/training_reproduction.json b/cookbook/auto_router_selective_training/training_reproduction.json new file mode 100644 index 00000000000..c2fab082498 --- /dev/null +++ b/cookbook/auto_router_selective_training/training_reproduction.json @@ -0,0 +1,10 @@ +{ + "status": "passed", + "models": 118, + "candidates": 1892, + "selected_profiles": 12, + "ablations": 40, + "exact_coefficient_and_selection_match": true, + "data_scope": "Frozen training and validation only; no final tasks loaded", + "selection_sha256": "1a7c9241c72c299c8b27fd8171881f94fda069fb7191ac6bc3a2eab9de811853" +} \ No newline at end of file