From 5523acc0c484b5add07416bcc99ae0e7f6f9a0aa Mon Sep 17 00:00:00 2001 From: Tin Chi Lo Date: Thu, 17 Sep 2026 07:07:40 -0400 Subject: [PATCH] docs(router): publish complete selective routing evaluation --- .../PRIMARY_FINDINGS.md | 15 +- .../PROTOCOL.md | 20 + .../auto_router_selective_training/README.md | 12 +- .../auto_router_selective_training/REPORT.md | 204 + .../TRAINING.md | 49 + .../benchmark_summary.json | 11707 ++++++++++++++++ .../calibration_results.json | 5849 ++++++++ .../development_pair_counts.json | 62 + .../execution_diagnostics.json | 315 + .../manifest.json | 5 +- .../quality_cost.png | Bin 0 -> 230338 bytes .../quality_cost.svg | 4023 ++++++ .../training_reproduction.json | 10 + 13 files changed, 22264 insertions(+), 7 deletions(-) create mode 100644 cookbook/auto_router_selective_training/REPORT.md create mode 100644 cookbook/auto_router_selective_training/TRAINING.md create mode 100644 cookbook/auto_router_selective_training/benchmark_summary.json create mode 100644 cookbook/auto_router_selective_training/calibration_results.json create mode 100644 cookbook/auto_router_selective_training/development_pair_counts.json create mode 100644 cookbook/auto_router_selective_training/execution_diagnostics.json create mode 100644 cookbook/auto_router_selective_training/quality_cost.png create mode 100644 cookbook/auto_router_selective_training/quality_cost.svg create mode 100644 cookbook/auto_router_selective_training/training_reproduction.json diff --git a/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md b/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md index 1f641d6bb6b..70cb0b329dc 100644 --- a/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md +++ b/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md @@ -28,4 +28,17 @@ All policies and family controls were frozen before final outcomes. These are ta Two Luna solver requests interrupted by host sleep have unknown charges. An earlier unavailable DNA-assembly review has one additional unmetered connection-timeout request. Thus Luna-only comparisons contain two unmetered requests, and review cascades contain three. The report marks affected costs as lower bounds and savings as upper bounds. The three upfront comparisons quoted with exact costs above are fully metered -The additional Sonnet/Opus controls are still running. Their completion will extend the comparison; it cannot change these frozen Luna/Sol results +The additional Sonnet/Opus controls are complete. All 500 final attempts and grades are present, and the complete reporter verified that the previously published Luna/Sol results and calibration diagnostics are unchanged + +| Fixed model | Solved | Recorded inference cost | +|---|---:|---:| +| Luna | 84/125 | At least $2.6531 | +| Sol | 87/125 | $20.8964 | +| Sonnet 5 | 81/125 | At least $37.9961 | +| Opus 5 | 99/125 | At least $46.0550 | + +Opus solved 22/25 SWE-bench tasks at $10.5296, compared with Sol's 18/25 at $10.7132. On Terminal-Bench it solved 21/25 at at least $32.9157, compared with Sol's 18/25 at $8.4468. These are observed results under the frozen harness and budgets, not unrestricted model rankings. Across all tasks, even perfect selection between the saved Luna and Sol attempts reaches only 95/125, below Opus's 99/125. Changing the candidate model pair is a separate training and evaluation question + +All four models had an 8,192-token response cap and high reasoning effort. Corrected Sonnet and Opus attempts still incurred 116 and 83 length stops with no action or visible text, costing $10.3656 and $17.9127 respectively. Those responses remain in the results. Increasing the response cap could change quality and spending, but this experiment did not test that change. Host sleep and unresolved transport charges also prevent latency conclusions and exact affected cost comparisons + +The practical next experiment is to collect substantially more paired agentic training examples where Sol rescues Luna, then evaluate rescue prediction and verification on fresh tasks. The present fitting set contains only two such agentic cases. The current final set is now diagnostic evidence and must not be reused as an unseen test after training on its failures. Better average probability calibration alone did not deliver the requested selective-rescue precision diff --git a/cookbook/auto_router_selective_training/PROTOCOL.md b/cookbook/auto_router_selective_training/PROTOCOL.md index c6f7b3cf596..562b8e01e79 100644 --- a/cookbook/auto_router_selective_training/PROTOCOL.md +++ b/cookbook/auto_router_selective_training/PROTOCOL.md @@ -108,3 +108,23 @@ SWE solver containers have a 45-minute configured lifetime in both the original The final primary billing audit also found one earlier DNA-assembly review ConnectTimeout with no successful response ledger. This is separate from the two host-sleep solver interruptions. Research accounting now discovers transport-only ledgers as well as billed-response ledgers. Review-cascade cost comparisons therefore carry three unmetered requests, while Luna-only comparisons carry two + + +## Lid-closed host sleep, September 16 UTC + +During the remaining Anthropic Terminal controls, the host entered Clamshell Sleep on AC power at full battery and repeatedly returned to Maintenance Sleep. The workers survived; no whole attempt was restarted and no power setting changed. At the 06:12 UTC inspection, fix-ocaml-gc and model-extraction-relu-logits Opus trials remained active. The latter had a pending gateway request whose billing remains unresolved until the response or transport ledger completes. Evidence is retained in infrastructure_interruptions/20260916-clamshell-sleep. Host wall durations include these pauses and are not used as routing features or latency comparisons + + +## DNS recovery and exact Terminal image reuse, September 16 UTC + +After host sleep, two active Anthropic solver attempts ended with gateway DNS ConnectError and 29 queued controls failed to resolve Docker Hub during environment setup. The runner exited. All 31 failed trial records and exported partial attempts were archived in infrastructure_interruptions/20260916-dns-outage. The two interrupted attempts incurred $1.35120725 in known inference cost and 20 requests with unknown billing. These costs are research overhead, not policy inference cost. Connectivity probes verified host DNS, authenticated gateway access and Docker DNS before resuming + +Recovery validation found that eleven completed Anthropic controls had rebuilt different filesystem layers from the retained Luna/Sol images. Every mismatched control was archived and scheduled again, regardless of score. These controls cost $11.05990725, retained separately as research overhead. The 25 Luna/Sol image pairs match in filesystem layers and runtime image configuration. All retained primary attempts and fitted policies are unchanged + +The Terminal environment now supports an explicit map of the exact saved Sol image for each final task. It verifies local image identity and filesystem layers before startup and verifies the started container image before agent inference. This replaces rebuilds from mutable base tags with the already-cached peer image. The map is terminal_image_pins.json. Task files, solver prompts, budgets, network isolation, and grading rules remain unchanged. The recovery schedules exactly 42 missing controls and preserves 58 valid Terminal results. No live trial was interrupted for this correction + +The model-extraction-relu-logits Opus attempt completed normally after the model wrapper retried the previously pending ReadError. It is retained, with that request's unknown charge included in its cost bounds. The entire attempt was not rerun. Host wall durations remain unsuitable for latency comparisons + +## Final completion + +All 500 held-out model attempts and grades completed on September 17, 2026 UTC. The final audit checked 950 completed development/final attempts and 1,400 forecasts, including paired image identity, Git isolation, archived file hashes and recorded costs. Full reporting verified that primary Luna/Sol results and calibration diagnostics are identical to the earlier primary release. No fitting, policy selection or boundary change followed the final outcomes diff --git a/cookbook/auto_router_selective_training/README.md b/cookbook/auto_router_selective_training/README.md index 674e2cbfeff..0b38d1d46a7 100644 --- a/cookbook/auto_router_selective_training/README.md +++ b/cookbook/auto_router_selective_training/README.md @@ -2,7 +2,7 @@ These optional profiles learn when GPT-5.6 Sol adds a successful solve over GPT-5.6 Luna. They use one existing classifier response and a deterministic local prediction, with no second classifier call. They replace the earlier ROI cookbook, whose benchmark results were invalidated and are not the source of these fits -Training uses 160 fresh paired tasks and selection uses 64 separate validation tasks across SWE-bench, Terminal-Bench, MBPP+ and LiveCodeBench. The fitted policies were frozen before the 125-task held-out evaluation. The complete 125-task Luna/Sol comparison is available in PRIMARY_FINDINGS.md and PRIMARY_REPORT.md. Additional Sonnet/Opus controls are still running +Training uses 160 fresh paired tasks and selection uses 64 separate validation tasks across SWE-bench, Terminal-Bench, MBPP+ and LiveCodeBench. The fitted policies were frozen before the 125-task held-out evaluation. The complete 125-task Luna/Sol comparison is available in PRIMARY_FINDINGS.md and PRIMARY_REPORT.md. The additional Sonnet/Opus controls are complete in REPORT.md `strong-success-retention` minimizes validation cost while preserving every Sol-only success and meeting Sol quality within each benchmark. `quality-first-under-sol-budget` maximizes validation solves under Sol's total inference cost and permits task swaps. These names describe validation objectives, not guarantees on new requests @@ -21,15 +21,19 @@ The twelve prespecified upfront family controls are also available in `diagnosti These profiles were selected before these results. They are not refitted or selected again on the held-out tasks -| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost Sol solves | +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | |---|---:|---:|---:|---:| | Luna only | 84/125 | ≥$2.6531 | ≤87.3% | +8 / -11 | | Sol only | 87/125 | $20.8964 | 0.0% | +0 / -0 | | Capability baseline | 85/125 | $6.7642 | 67.6% | +8 / -10 | | Capability upfront, retention objective | 86/125 | $13.8156 | 33.9% | +1 / -2 | | Capability upfront, quality objective | 88/125 | $18.5237 | 11.4% | +5 / -4 | -| ablation_cap_original_upfront_per_model | 88/125 | $20.3797 | 2.5% | +1 / -0 | +| Per-model control (exploratory) | 88/125 | $20.3797 | 2.5% | +1 / -0 | Costs use recorded gateway bills. A ≥ cost and ≤ savings mark unmetered transport requests: cost is a lower bound and savings versus fully metered Sol is an upper bound. The JSON counts these requests per policy and task. They are not assumed free. -The per-model family control is an exploratory comparison among forty prespecified variants, not a newly selected winner. No validation-selected profile combines preservation of all Sol successes and lower cost across all 125 tasks. Task-level replay does not measure per-turn routing or Sol continuation from a Luna patch +Costs include the classifier. These are task-pinned replays of fresh independent attempts, with one attempt per model per task. No guarantee of quality retention follows from a 125-task sample. The aggregate combines different benchmarks; inspect each benchmark before using a profile + +The per-model control is an exploratory comparison among forty prespecified variants. No validation-selected profile combines preservation of all Sol successes and lower cost across all 125 tasks. See PRIMARY_FINDINGS.md for the interpretation + +![Observed quality and known inference cost](quality_cost.png) diff --git a/cookbook/auto_router_selective_training/REPORT.md b/cookbook/auto_router_selective_training/REPORT.md new file mode 100644 index 00000000000..bcd8470de8f --- /dev/null +++ b/cookbook/auto_router_selective_training/REPORT.md @@ -0,0 +1,204 @@ +# Clean selective-router evaluation + +Policies were frozen before final solver outcomes. Upfront routing uses one classifier response; the post-attempt variants also use Luna output, optional public-example checks and a Luna review. These are task-level replays over matched independent attempts, not live per-turn routing or continuation from a cheaper model's patch + +The configured original-card baselines receive the same model identity and execution conditions as the trained variants. Capability uses base_threshold=0.5 and threshold_step=0.1; V2 uses max_quality_gap=0.05. The empirical-card baselines change the card supplement while retaining those boundaries + +Retention profiles were selected to preserve Sol successes at lower validation cost. Quality profiles were selected to maximize validation solves within the Sol budget. These names describe selection objectives, not guarantees about the held-out results. Exact policy IDs remain in final_results.json + +Costs use recorded gateway bills. A ≥ cost and ≤ savings mark unmetered transport requests: cost is a lower bound and savings versus fully metered Sol is an upper bound. The JSON counts these requests per policy and task. They are not assumed free. + +## Observed opportunity for routing + +The Sol-only column counts the rescues a perfect decision would capture. The hindsight cascade pays for Luna on every task and adds Sol only for those rescues. Tasks where both attempts failed cannot be recovered by choosing between these two saved outputs. This is a bound on this replay, not a guarantee about future attempts + +| Benchmark | Both solve | Luna only | Sol only | Both fail | Hindsight cascade cost | +|---|---:|---:|---:|---:|---:| +| all | 76 | 8 | 11 | 30 | ≥$5.2653 | +| lcb | 11 | 2 | 3 | 9 | $0.3657 | +| mbpp | 35 | 3 | 2 | 10 | $0.0361 | +| swe | 16 | 0 | 2 | 7 | $1.5478 | +| terminal | 14 | 3 | 4 | 4 | ≥$3.3157 | + +## all + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 84/125 | ≥$2.6531 | ≤87.3% | +8 / -11 | +| Sol only | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| Sonnet 5 only | 81/125 | ≥$37.9961 | ≤-81.8% | +8 / -14 | +| Opus 5 only | 99/125 | ≥$46.0550 | ≤-120.4% | +19 / -7 | +| Capability upfront, retention objective | 86/125 | $13.8156 | 33.9% | +1 / -2 | +| Capability upfront, quality objective | 88/125 | $18.5237 | 11.4% | +5 / -4 | +| V2 upfront, retention objective | 85/125 | $14.0036 | 33.0% | +2 / -4 | +| V2 upfront, quality objective | 86/125 | $18.1999 | 12.9% | +4 / -5 | +| Capability baseline | 85/125 | $6.7642 | 67.6% | +8 / -10 | +| Capability with learned cards only | 84/125 | $7.2496 | 65.3% | +7 / -10 | +| V2 baseline | 84/125 | ≥$17.0029 | ≤18.6% | +3 / -6 | +| V2 with learned cards only | 85/125 | ≥$14.6487 | ≤29.9% | +6 / -8 | +| Capability review, retention objective | 81/125 | ≥$7.5830 | ≤63.7% | +0 / -6 | +| Capability review, quality objective | 87/125 | ≥$10.1853 | ≤51.3% | +7 / -7 | +| V2 review, retention objective | 88/125 | ≥$20.9754 | ≤-0.4% | +1 / -0 | +| V2 review, quality objective | 88/125 | ≥$8.6493 | ≤58.6% | +8 / -7 | +| Hindsight direct routing (not deployable) | 95/125 | ≥$4.5190 | ≤78.4% | +8 / -0 | +| Hindsight Luna-first routing (not deployable) | 95/125 | ≥$5.2653 | ≤74.8% | +8 / -0 | + +## lcb + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 13/25 | $0.1165 | 92.4% | +2 / -3 | +| Sol only | 14/25 | $1.5299 | 0.0% | +0 / -0 | +| Sonnet 5 only | 9/25 | $1.3790 | 9.9% | +1 / -6 | +| Opus 5 only | 14/25 | $2.3849 | -55.9% | +2 / -2 | +| Capability upfront, retention objective | 14/25 | $1.5476 | -1.2% | +0 / -0 | +| Capability upfront, quality objective | 14/25 | $0.9200 | 39.9% | +1 / -1 | +| V2 upfront, retention objective | 14/25 | $1.3923 | 9.0% | +0 / -0 | +| V2 upfront, quality objective | 12/25 | $1.1104 | 27.4% | +0 / -2 | +| Capability baseline | 13/25 | $0.2918 | 80.9% | +2 / -3 | +| Capability with learned cards only | 12/25 | $0.4860 | 68.2% | +1 / -3 | +| V2 baseline | 12/25 | $0.9924 | 35.1% | +0 / -2 | +| V2 with learned cards only | 13/25 | $0.4799 | 68.6% | +2 / -3 | +| Capability review, retention objective | 14/25 | $1.6846 | -10.1% | +0 / -0 | +| Capability review, quality objective | 14/25 | $1.3257 | 13.4% | +2 / -2 | +| V2 review, retention objective | 15/25 | $1.6076 | -5.1% | +1 / -0 | +| V2 review, quality objective | 14/25 | $1.0584 | 30.8% | +2 / -2 | +| Luna with public-test escalation | 15/25 | $1.3645 | 10.8% | +2 / -1 | +| Hindsight direct routing (not deployable) | 16/25 | $0.3436 | 77.5% | +2 / -0 | +| Hindsight Luna-first routing (not deployable) | 16/25 | $0.3657 | 76.1% | +2 / -0 | + +## mbpp + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 38/50 | $0.0174 | 91.6% | +3 / -2 | +| Sol only | 37/50 | $0.2064 | 0.0% | +0 / -0 | +| Sonnet 5 only | 36/50 | $0.0988 | 52.1% | +2 / -3 | +| Opus 5 only | 42/50 | $0.2249 | -9.0% | +7 / -2 | +| Capability upfront, retention objective | 37/50 | $0.2180 | -5.6% | +0 / -0 | +| Capability upfront, quality objective | 38/50 | $0.0444 | 78.5% | +3 / -2 | +| V2 upfront, retention objective | 37/50 | $0.2298 | -11.3% | +0 / -0 | +| V2 upfront, quality objective | 38/50 | $0.0465 | 77.4% | +3 / -2 | +| Capability baseline | 38/50 | $0.0290 | 85.9% | +3 / -2 | +| Capability with learned cards only | 38/50 | $0.0284 | 86.2% | +3 / -2 | +| V2 baseline | 38/50 | $0.0562 | 72.8% | +3 / -2 | +| V2 with learned cards only | 38/50 | $0.0396 | 80.8% | +3 / -2 | +| Capability review, retention objective | 37/50 | $0.2503 | -21.3% | +0 / -0 | +| Capability review, quality objective | 38/50 | $0.0434 | 79.0% | +3 / -2 | +| V2 review, retention objective | 37/50 | $0.2441 | -18.3% | +0 / -0 | +| V2 review, quality objective | 38/50 | $0.0488 | 76.4% | +3 / -2 | +| Luna with public-test escalation | 38/50 | $0.0240 | 88.4% | +2 / -1 | +| Hindsight direct routing (not deployable) | 40/50 | $0.0353 | 82.9% | +3 / -0 | +| Hindsight Luna-first routing (not deployable) | 40/50 | $0.0361 | 82.5% | +3 / -0 | + +## swe + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 16/25 | $0.6371 | 94.1% | +0 / -2 | +| Sol only | 18/25 | $10.7132 | 0.0% | +0 / -0 | +| Sonnet 5 only | 19/25 | $11.0537 | -3.2% | +2 / -1 | +| Opus 5 only | 22/25 | $10.5296 | 1.7% | +4 / -0 | +| Capability upfront, retention objective | 18/25 | $6.2493 | 41.7% | +0 / -0 | +| Capability upfront, quality objective | 18/25 | $9.7546 | 8.9% | +0 / -0 | +| V2 upfront, retention objective | 16/25 | $5.3537 | 50.0% | +0 / -2 | +| V2 upfront, quality objective | 17/25 | $8.8718 | 17.2% | +0 / -1 | +| Capability baseline | 16/25 | $1.3893 | 87.0% | +0 / -2 | +| Capability with learned cards only | 16/25 | $1.6826 | 84.3% | +0 / -2 | +| V2 baseline | 17/25 | $9.1304 | 14.8% | +0 / -1 | +| V2 with learned cards only | 17/25 | $7.4181 | 30.8% | +0 / -1 | +| Capability review, retention objective | 16/25 | $2.4752 | 76.9% | +0 / -2 | +| Capability review, quality objective | 16/25 | $0.9753 | 90.9% | +0 / -2 | +| V2 review, retention objective | 18/25 | $9.1263 | 14.8% | +0 / -0 | +| V2 review, quality objective | 16/25 | $0.9775 | 90.9% | +0 / -2 | +| Hindsight direct routing (not deployable) | 18/25 | $1.4342 | 86.6% | +0 / -0 | +| Hindsight Luna-first routing (not deployable) | 18/25 | $1.5478 | 85.6% | +0 / -0 | + +## terminal + +| Policy | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---:|---:|---:|---:| +| Luna only | 17/25 | ≥$1.8822 | ≤77.7% | +3 / -4 | +| Sol only | 18/25 | $8.4468 | 0.0% | +0 / -0 | +| Sonnet 5 only | 17/25 | ≥$25.4647 | ≤-201.5% | +3 / -4 | +| Opus 5 only | 21/25 | ≥$32.9157 | ≤-289.7% | +6 / -3 | +| Capability upfront, retention objective | 17/25 | $5.8008 | 31.3% | +1 / -2 | +| Capability upfront, quality objective | 18/25 | $7.8047 | 7.6% | +1 / -1 | +| V2 upfront, retention objective | 18/25 | $7.0279 | 16.8% | +2 / -2 | +| V2 upfront, quality objective | 19/25 | $8.1712 | 3.3% | +1 / -0 | +| Capability baseline | 18/25 | $5.0541 | 40.2% | +3 / -3 | +| Capability with learned cards only | 18/25 | $5.0526 | 40.2% | +3 / -3 | +| V2 baseline | 17/25 | ≥$6.8238 | ≤19.2% | +0 / -1 | +| V2 with learned cards only | 17/25 | ≥$6.7112 | ≤20.5% | +1 / -2 | +| Capability review, retention objective | 14/25 | ≥$3.1729 | ≤62.4% | +0 / -4 | +| Capability review, quality objective | 19/25 | ≥$7.8410 | ≤7.2% | +2 / -1 | +| V2 review, retention objective | 18/25 | ≥$9.9975 | ≤-18.4% | +0 / -0 | +| V2 review, quality objective | 20/25 | ≥$6.5646 | ≤22.3% | +3 / -1 | +| Hindsight direct routing (not deployable) | 21/25 | ≥$2.7059 | ≤68.0% | +3 / -0 | +| Hindsight Luna-first routing (not deployable) | 21/25 | ≥$3.3157 | ≤60.7% | +3 / -0 | + +## Frozen family comparisons + +Each row was selected on development data before final inference. These are diagnostic comparisons, not a new winner selection on the held-out tasks. An always-Sol fallback means that no candidate in that requested family met the validation criterion; it bypasses the classifier, Luna attempt and reviewer + +| Classifier | Requested stage | Cards | Fitted target | Boundary | Solved | Inference cost | Savings vs Sol | Gained / lost vs Sol | +|---|---|---|---|---:|---:|---:|---:|---:| +| cap | upfront | empirical | benefit_per_dollar | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | empirical | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | empirical | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | empirical | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | empirical | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | original | benefit_per_dollar | 1 | 86/125 | $13.8156 | 33.9% | +1 / -2 | +| cap | upfront | original | paired | -0.02 | 88/125 | $20.8563 | 0.2% | +1 / -0 | +| cap | upfront | original | per_model | -0.05 | 88/125 | $20.3797 | 2.5% | +1 / -0 | +| cap | upfront | original | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | upfront | original | rescue | 0.03 | 88/125 | $20.7799 | 0.6% | +1 / -0 | +| cap | upfront | original | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | original | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | original | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | original | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | empirical | benefit_per_dollar | 0.5 | 87/125 | $19.2534 | 7.9% | +2 / -2 | +| v2 | upfront | empirical | paired | 0.01 | 85/125 | $19.7030 | 5.7% | +1 / -3 | +| v2 | upfront | empirical | per_model | 0.01 | 86/125 | $19.7351 | 5.6% | +1 / -2 | +| v2 | upfront | empirical | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | empirical | rescue | 0.05 | 86/125 | $18.0048 | 13.8% | +2 / -3 | +| v2 | upfront | empirical | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | review | empirical | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | review | empirical | per_model | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | review | empirical | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | original | benefit_per_dollar | 1 | 85/125 | $14.0036 | 33.0% | +2 / -4 | +| v2 | upfront | original | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | original | per_model | 0 | 86/125 | $20.1971 | 3.3% | +0 / -1 | +| v2 | upfront | original | raw | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | original | rescue | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | upfront | original | scalar_calibration | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| v2 | review | original | paired | always Sol fallback | 87/125 | $20.8964 | 0.0% | +0 / -0 | +| cap | review | empirical | benefit_per_dollar | 0.1 | 82/125 | ≥$16.8075 | ≤19.6% | +0 / -5 | +| cap | review | original | benefit_per_dollar | 0.25 | 81/125 | ≥$7.5830 | ≤63.7% | +0 / -6 | +| v2 | review | empirical | benefit_per_dollar | 0.25 | 83/125 | ≥$16.3243 | ≤21.9% | +0 / -4 | +| v2 | review | original | benefit_per_dollar | 0.1 | 85/125 | ≥$19.8241 | ≤5.1% | +0 / -2 | +| v2 | review | original | per_model | -0.02 | 88/125 | ≥$20.9754 | ≤-0.4% | +1 / -0 | +| v2 | review | original | rescue | 0.02 | 86/125 | ≥$23.4088 | ≤-12.0% | +0 / -1 | + +## Response budget effects + +Every solver uses the same 8,192-token response cap and high reasoning effort. Length stops that produce neither a tool call nor visible text still consume inference cost. They are retained as model behavior under this budget, not retried as infrastructure failures. These counts describe completed final attempts across all four benchmarks + +| Model | Responses | Length stops | Length stops without action or text | Cost of those empty length stops | +|---|---:|---:|---:|---:| +| gpt-5.6-luna | 1341 | 50 | 42 | $0.4217 | +| gpt-5.6-sol | 819 | 7 | 4 | $0.6676 | +| claude-sonnet-5 | 1703 | 124 | 116 | $10.3656 | +| claude-opus-5 | 870 | 87 | 83 | $17.9127 | + +## Experiment spending + +Recorded study inference cost is $225.5557. This includes training, validation, final solver/judge/reviewer calls, superseded development forecasts, archived infrastructure-interrupted attempts, superseded Anthropic agentic controls, controls with mismatched Terminal images, and adapter round-trip probes. It is separate from the deployed-policy costs above. Unknown transport billing, setup, live-proxy QA, local Docker and CPU expenses are not included. See research_costs.json for the breakdown + +## Interpretation limits + +The oracle is a hindsight reference using known costs. Cost intervals use recorded bills and exclude unmetered requests. A small sample with one attempt per model cannot establish the same routing advantage across workloads or repeated runs. Paired bootstrap intervals and exact discordant-pair tests in the JSON are exploratory; policy selection searched many development candidates. Inference costs include classifier/reviewer calls and discarded Luna attempts on escalation. Local checker CPU time and experiment training/transport overhead are separate. Terminal-Bench and SWE-bench use documented offline/native-image eligibility adaptations diff --git a/cookbook/auto_router_selective_training/TRAINING.md b/cookbook/auto_router_selective_training/TRAINING.md new file mode 100644 index 00000000000..1113d7d8333 --- /dev/null +++ b/cookbook/auto_router_selective_training/TRAINING.md @@ -0,0 +1,49 @@ +# Training and selection record + +The learning target is model complementarity: route cheaply when Luna can do the job, and use Sol when its extra cost is likely to produce an additional solve. Better marginal probability accuracy alone is not the selection objective + +## Paired examples + +| Split | Benchmark | Tasks | Both fail | Sol only | Luna only | Both solve | +|---|---|---:|---:|---:|---:|---:| +| train | lcb | 50 | 14 | 9 | 3 | 24 | +| train | mbpp | 80 | 7 | 2 | 3 | 68 | +| train | swe | 25 | 5 | 1 | 0 | 19 | +| train | terminal | 5 | 0 | 1 | 0 | 4 | +| validation | lcb | 20 | 2 | 6 | 2 | 10 | +| validation | mbpp | 29 | 4 | 2 | 2 | 21 | +| validation | swe | 10 | 1 | 1 | 1 | 7 | +| validation | terminal | 5 | 2 | 0 | 1 | 2 | + +There are 13 Sol-only examples in fitting and 9 in validation. Only two fitting examples are agentic rescues, one SWE-bench and one Terminal-Bench. Terminal validation has no Sol-only example. That limits evidence about rescue detection on unseen terminal tasks; zero validation losses there cannot establish rescue recall + +## Variations + +One training-only synthesis produces the empirical card supplement. It uses measured outcomes and fallible classifier/reviewer descriptions, without task identities, solutions or validation-driven rewrites. Original cards are a separate control + +The deterministic head families are raw threshold tuning, scalar probability calibration, task-dependent per-model calibration, four-way paired outcomes, Sol-only rescue probability and paired benefit per predicted incremental dollar. Regularization and fixed threshold grids are selected using validation only. Upfront inputs come from one classifier response. Post-attempt heads additionally use Luna output, a fallible Luna review and task-supplied public checks where available + +The strong-success-retention objective minimizes validation cost while losing no Sol-only successes and preserving Sol aggregate quality in each benchmark. The quality-first-under-sol-budget objective maximizes validation solves under Sol total cost, permitting task swaps. Always-Sol is an explicit fallback. None of these objective names guarantees performance on unseen tasks + +## Frozen selected profiles + +| Classifier | Stage | Objective | Cards | Family | Threshold | Validation solved | Validation cost | Sol-only losses | +|---|---|---|---|---|---:|---:|---:|---:| +| cap | upfront | strong_success_retention | original | benefit_per_dollar | 1.0 | 50/64 | $5.8544 | 0 | +| cap | upfront | quality_first_under_sol_budget | original | scalar_calibration | 0.1 | 52/64 | $5.4692 | 2 | +| cap | review | strong_success_retention | original | benefit_per_dollar | 0.25 | 49/64 | $5.6530 | 0 | +| cap | review | quality_first_under_sol_budget | empirical | paired | 0.075 | 53/64 | $3.7201 | 2 | +| cap | either | strong_success_retention | original | benefit_per_dollar | 0.25 | 49/64 | $5.6530 | 0 | +| cap | either | quality_first_under_sol_budget | empirical | paired | 0.075 | 53/64 | $3.7201 | 2 | +| v2 | upfront | strong_success_retention | original | benefit_per_dollar | 1.0 | 51/64 | $5.4640 | 0 | +| v2 | upfront | quality_first_under_sol_budget | original | scalar_calibration | 0.075 | 53/64 | $5.4524 | 2 | +| v2 | review | strong_success_retention | original | per_model | -0.02 | 51/64 | $6.7043 | 0 | +| v2 | review | quality_first_under_sol_budget | empirical | per_model | 0.15 | 53/64 | $3.3074 | 2 | +| v2 | either | strong_success_retention | original | benefit_per_dollar | 1.0 | 51/64 | $5.4640 | 0 | +| v2 | either | quality_first_under_sol_budget | empirical | per_model | 0.15 | 53/64 | $3.3074 | 2 | + +These are development results. Compare held-out results in REPORT.md before interpreting them as an improvement. Both selected upfront classifier profiles keep the original card and use fitted local decisions. Empirical cards remain a measured variation and are selected by some review cascades + +There are 1,892 policy candidates, 118 serialized models, 12 selected profiles and 40 prespecified family ablations. An independent refit reproduced every coefficient, candidate statistic and selection exactly. The selection hash and source/data hashes are recorded in selection_frozen.json; reproduction evidence is in training_reproduction.json + +The threshold unit depends on the target. Marginal and paired gaps are probability differences, rescue is a Sol-only probability, and benefit_per_dollar is expected additional success per predicted dollar. A threshold of 0.05 is not interchangeable across these families diff --git a/cookbook/auto_router_selective_training/benchmark_summary.json b/cookbook/auto_router_selective_training/benchmark_summary.json new file mode 100644 index 00000000000..c51b6e153bc --- /dev/null +++ b/cookbook/auto_router_selective_training/benchmark_summary.json @@ -0,0 +1,11707 @@ +{ + "protocol": "Fresh independent paired solver attempts; fixed task-level policy replay. Training and threshold selection use development only. The hindsight oracle uses final labels and is not deployable.", + "paired_outcomes": { + "all": { + "both_fail": 30, + "sol_only": 11, + "luna_only": 8, + "both_solve": 76 + }, + "lcb": { + "both_fail": 9, + "sol_only": 3, + "luna_only": 2, + "both_solve": 11 + }, + "mbpp": { + "both_fail": 10, + "sol_only": 2, + "luna_only": 3, + "both_solve": 35 + }, + "swe": { + "both_fail": 7, + "sol_only": 2, + "luna_only": 0, + "both_solve": 16 + }, + "terminal": { + "both_fail": 4, + "sol_only": 4, + "luna_only": 3, + "both_solve": 14 + } + }, + "results": { + "fixed_gpt-5.6-luna": { + "total": { + "tasks": 125, + "solved": 84, + "cost": 2.65313975, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.8730336810071551, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 11, + "paired_exact_p": 0.6476058959960938, + "quality_delta_bootstrap_95": [ + -0.088, + 0.048 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18976455058400002, + -0.10695287329400002 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.2849141529181544 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 0.1164804, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9238662330124501, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.079881704, + -0.03457668540000001 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.7075982261787133 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.017396, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9157072527813311, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0049195134000000005, + -0.002844990800000001 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.6371064399999999, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.940530936966828, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5143941064600001, + -0.31072451521000005 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 1.88215691, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.7771763251611454, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 4, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.24, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.37199028632000003, + -0.16931580716000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.6023646356164746 + ] + } + } + } + }, + "fixed_gpt-5.6-sol": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "fixed_claude-sonnet-5": { + "total": { + "tasks": 125, + "solved": 81, + "cost": 37.9961323, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.8183094403889355, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 14, + "paired_exact_p": 0.28627872467041016, + "quality_delta_bootstrap_95": [ + -0.12, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + 0.04082165667999999, + 0.24607147657999995 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 9, + "cost": 1.3789665, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.09868171645498136, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 6, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.4, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.023796100500000004, + 0.011813104999999997 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "mbpp": { + "tasks": 50, + "solved": 36, + "cost": 0.098846, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.5210392681319533, + "baseline_solved": 37, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.1, + 0.06 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0028876400000000003, + -0.0015387350000000007 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "swe": { + "tasks": 25, + "solved": 19, + "cost": 11.053656, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.031775107172044015, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16099999999998546 + ], + "mean_cost_delta_bootstrap_95": [ + -0.17330483480000003, + 0.24415650309999992 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 25.4646638, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -2.014695499777408, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 4, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.24, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + 0.29976725349999994, + 1.0925644450999998 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + } + } + }, + "fixed_claude-opus-5": { + "total": { + "tasks": 125, + "solved": 99, + "cost": 46.055013, + "unknown_billing_requests": 4, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -1.203968136913113, + "baseline_solved": 87, + "gained_sol_failures": 19, + "lost_sol_successes": 7, + "paired_exact_p": 0.028959274291992188, + "quality_delta_bootstrap_95": [ + 0.024, + 0.176 + ], + "mean_cost_delta_bootstrap_95": [ + 0.07898218682999997, + 0.34375395494999983 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 2.384855, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.5587858117682738, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0175889235, + 0.05212979624999998 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "mbpp": { + "tasks": 50, + "solved": 42, + "cost": 0.224865, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.08958890568670785, + "baseline_solved": 37, + "gained_sol_failures": 7, + "lost_sol_successes": 2, + "paired_exact_p": 0.1796875, + "quality_delta_bootstrap_95": [ + 0.0, + 0.22 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0002537405, + 0.0010139794999999997 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "swe": { + "tasks": 25, + "solved": 22, + "cost": 10.52956575, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.017144750100660522, + "baseline_solved": 18, + "gained_sol_failures": 4, + "lost_sol_successes": 0, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + 0.04, + 0.32 + ], + "mean_cost_delta_bootstrap_95": [ + -0.10402242440000004, + 0.10201380229999983 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + }, + "terminal": { + "tasks": 25, + "solved": 21, + "cost": 32.91572725, + "unknown_billing_requests": 4, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -2.896807575856376, + "baseline_solved": 18, + "gained_sol_failures": 6, + "lost_sol_successes": 3, + "paired_exact_p": 0.5078125, + "quality_delta_bootstrap_95": [ + -0.12, + 0.36 + ], + "mean_cost_delta_bootstrap_95": [ + 0.4561114184000001, + 1.5875788229999999 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": null + } + } + }, + "cap_upfront_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 13.815640460000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.33885087895018884, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.088707189458, + -0.029987009136000043 + ], + "strong_routes": 102, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.08823529411764706, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.041143831377718536, + 0.16089819225628654 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.54757188, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.011521911913115801, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0005666024999999994, + 0.0008554352799999992 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.21798, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.05622746831026859, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00020418355, + 0.00026104007500000006 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 6.24925476, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.4166793774164488, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.30450702318000006, + -0.07650369019000006 + ], + "strong_routes": 15, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.13333333333333333, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.016575913440083623, + 0.4046026966033721 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 5.80083382, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.31325432962870725, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18159462757, + -0.04581437822000005 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.020862525460092384, + 0.484137748685967 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "cap_upfront_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 18.52372147, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.11354510100923576, + "baseline_solved": 87, + "gained_sol_failures": 5, + "lost_sol_successes": 4, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.056 + ], + "mean_cost_delta_bootstrap_95": [ + -0.028897788182000003, + -0.010285258102000018 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.14, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.05819170034037209, + 0.2673960024970084 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 0.92004308, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.39864264312942166, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.04378357345, + -0.008312728250000003 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.25, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.03185402624994424, + 0.650855794412824 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.044412, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7848005582044424, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0043592915, + -0.0023561677250000003 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.841886116991581 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.75459163, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.08948271735046087, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07487978142000001, + -0.009760154500000014 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.1, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.012348527170294804, + 0.3169827140190823 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 7.80467476, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.07602479808909468, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.05543588585, + -0.0045051321900000055 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.15, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.032070937185463715, + 0.3789268265453139 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "v2_upfront_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 14.003645670000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.3298538672152522, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 4, + "paired_exact_p": 0.6875, + "quality_delta_bootstrap_95": [ + -0.056, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.08716027666, + -0.03046360189800001 + ], + "strong_routes": 100, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.07, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.028605288907438704, + 0.13891972845585732 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3922581500000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.08999404553369261, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.018160890399999997, + 0.0008559290000000007 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2297896, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.11345117649339076, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00045055567499999996, + 0.0004872066999999999 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 5.35368162, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.5002743502022768, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.33043864058, + -0.1220960099900001 + ], + "strong_routes": 10, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3084971078187608 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 7.0279163, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.16798321749599177, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.1264185382, + -0.011905893240000002 + ], + "strong_routes": 16, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.015513603815413892, + 0.38347623684926346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "v2_upfront_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 18.19994713, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.12903936064412225, + "baseline_solved": 87, + "gained_sol_failures": 4, + "lost_sol_successes": 5, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.056, + 0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.03653304908999999, + -0.009390257962000001 + ], + "strong_routes": 51, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 6, + "sol_route_rescue_precision": 0.11764705882352941, + "sol_only_recall": 0.5454545454545454, + "sol_route_rescue_precision_exact_95": [ + 0.044420141856314814, + 0.23867604726407454 + ], + "sol_only_recall_exact_95": [ + 0.23379359765934518, + 0.8325119059362929 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 12, + "cost": 1.1104201500000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.274208631165585, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0330455756, + -0.005133099700000001 + ], + "strong_routes": 10, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.002528578544461787, + 0.4450161170281954 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.0465402, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7744883125944877, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004358442625000001, + -0.00225688455 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 8.87179983, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.17188464880694931, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.14395534461999998, + -0.01956562445 + ], + "strong_routes": 19, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.05263157894736842, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.001331628804179424, + 0.2602806541952142 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 8.171186950000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.03263437053487084, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.023058544700000003, + -0.0017085895500000016 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.19047619047619047, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.054463568178406814, + 0.41906604118352786 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "v2_either_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 14.003645670000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.3298538672152522, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 4, + "paired_exact_p": 0.6875, + "quality_delta_bootstrap_95": [ + -0.056, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.08716027666, + -0.03046360189800001 + ], + "strong_routes": 100, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.07, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.028605288907438704, + 0.13891972845585732 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3922581500000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.08999404553369261, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.018160890399999997, + 0.0008559290000000007 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2297896, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.11345117649339076, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00045055567499999996, + 0.0004872066999999999 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 5.35368162, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.5002743502022768, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.33043864058, + -0.1220960099900001 + ], + "strong_routes": 10, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3084971078187608 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 7.0279163, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.16798321749599177, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.1264185382, + -0.011905893240000002 + ], + "strong_routes": 16, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.015513603815413892, + 0.38347623684926346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_per_model": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_raw": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_upfront_scalar_calibration": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_review_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_review_per_model": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_empirical_review_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 13.815640460000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.33885087895018884, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.088707189458, + -0.029987009136000043 + ], + "strong_routes": 102, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.08823529411764706, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.041143831377718536, + 0.16089819225628654 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.54757188, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.011521911913115801, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0005666024999999994, + 0.0008554352799999992 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.21798, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.05622746831026859, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00020418355, + 0.00026104007500000006 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 6.24925476, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.4166793774164488, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.30450702318000006, + -0.07650369019000006 + ], + "strong_routes": 15, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.13333333333333333, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.016575913440083623, + 0.4046026966033721 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 5.80083382, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.31325432962870725, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18159462757, + -0.04581437822000005 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.020862525460092384, + 0.484137748685967 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "ablation_cap_original_upfront_paired": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.85627941, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0019202627475750633, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0012403747780000008, + 0.00028191264799999605 + ], + "strong_routes": 118, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.09322033898305085, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.047458978550390885, + 0.16067637565779969 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.46044488, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.045425924086110236, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.006978613070000001, + 0.00012139414999999822 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.15, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.032070937185463715, + 0.3789268265453139 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2123152, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.028778540140326436, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -5.851392499999999e-05, + 0.00024433614999999994 + ], + "strong_routes": 48, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.041666666666666664, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.005086467930032977, + 0.14254097320137743 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.724427200000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.001044091080705245, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0004037624999999898, + 0.0004933144499999826 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.45909213, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.0014499769878559832, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00042307479999999757, + 0.0005730975699999966 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_per_model": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.379702, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.024726931511571903, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.010380521006000008, + 1.0734627999997192e-05 + ], + "strong_routes": 115, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.09565217391304348, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.048722767492542195, + 0.1647044453125498 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.46044488, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.045425924086110236, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.006978613070000001, + 0.00012139414999999822 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.15, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.032070937185463715, + 0.3789268265453139 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2088462, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.011969415048261434, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.00017778097500000003, + 0.00022795382499999996 + ], + "strong_routes": 47, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.0425531914893617, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.005195583176743832, + 0.14540524548821393 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.251318790000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0431169973801393, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.04796722185000002, + 0.0004733589499999808 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08695652173913043, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.010709966037875893, + 0.2803793359094109 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.45909213, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.0014499769878559832, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00042307479999999757, + 0.0005730975699999966 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_raw": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_rescue": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.779895630000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.005575617644488773, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0036691405380000013, + 0.0004582755239999995 + ], + "strong_routes": 124, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.08870967741935484, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04511844911596335, + 0.1531829791414111 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.54757188, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.011521911913115801, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0005666024999999994, + 0.0008554352799999992 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.21798, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.05622746831026859, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00020418355, + 0.00026104007500000006 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.724427200000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.001044091080705245, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0004037624999999898, + 0.0004933144499999826 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 8.289916550000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.01857828113893023, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.019854608400000005, + 0.0005552486399999959 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.047353626607257136, + 0.37384171348868267 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_upfront_scalar_calibration": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_review_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_review_per_model": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_cap_original_review_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 19.25344261, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0786242088711332, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.032, + 0.032 + ], + "mean_cost_delta_bootstrap_95": [ + -0.025640446206000003, + -0.003413582256000004 + ], + "strong_routes": 117, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.07692307692307693, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.03577697786951805, + 0.14100438661143383 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.4366962, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.06094850530476925, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.012602240650000002, + 0.0007569143499999969 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.22282945, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.0797255979377447, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00030838077499999996, + 0.0003543357749999998 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 9.674128730000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.09699331993035598, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0953708952, + 0.0004907219000000017 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.045454545454545456, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0011501475265735762, + 0.2284443976676335 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 7.91978823, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.06239681294472521, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.048092758770000014, + -0.0015903870300000168 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.14285714285714285, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.030488967621515563, + 0.36342398701698087 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_paired": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 19.702987890000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.057111165910539774, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 3, + "paired_exact_p": 0.625, + "quality_delta_bootstrap_95": [ + -0.048, + 0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.017677389673999996, + -0.0031051263940000047 + ], + "strong_routes": 103, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 8, + "sol_route_rescue_precision": 0.07766990291262135, + "sol_only_recall": 0.7272727272727273, + "sol_route_rescue_precision_exact_95": [ + 0.03412864449825134, + 0.14731394223104977 + ], + "sol_only_recall_exact_95": [ + 0.3902574404275788, + 0.9397822658270933 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 1.2205622, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.2022177282305757, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.027744909950000004, + -0.0009071294500000018 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.05, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.001265089497949805, + 0.2487327627720277 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 36, + "cost": 0.19468825, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.05663328100166687, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0005602417, + 6.0092474999999715e-05 + ], + "strong_routes": 38, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.02631578947368421, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0006660362042023661, + 0.13809902979242009 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.9587556, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0704255563507501, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.06546539239999999, + 0.0004996763000000104 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.011205586024150999, + 0.29161274155393346 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.32898184, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.01395344277917554, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.012773785860000011, + 0.0005436093800000007 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.17391304347826086, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.049507645305793435, + 0.3878118899547976 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_per_model": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 19.735080890000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.05557535157002591, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.032, + 0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.017406569784, + -0.002787731114000002 + ], + "strong_routes": 103, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.08737864077669903, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.040736426646704606, + 0.15939919045550544 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.2564414000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.1787664123654198, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.026266256300000002, + 0.00047061504999999765 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09523809523809523, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.0117493178844458, + 0.30377440691392416 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 36, + "cost": 0.19090205000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0749794065201379, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0006591471500000002, + 4.965949999999577e-06 + ], + "strong_routes": 37, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.02702702702702703, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0006840310246946654, + 0.14160309561115805 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.9587556, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0704255563507501, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.06546539239999999, + 0.0004996763000000104 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.011205586024150999, + 0.29161274155393346 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.32898184, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.01395344277917554, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.012773785860000011, + 0.0005436093800000007 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.17391304347826086, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.049507645305793435, + 0.3878118899547976 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_raw": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_rescue": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 18.00477004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.13837958355135338, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.03901592546799999, + -0.010119537734 + ], + "strong_routes": 97, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 8, + "sol_route_rescue_precision": 0.08247422680412371, + "sol_only_recall": 0.7272727272727273, + "sol_route_rescue_precision_exact_95": [ + 0.03628024982016327, + 0.15605269680266584 + ], + "sol_only_recall_exact_95": [ + 0.3902574404275788, + 0.9397822658270933 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.2564414000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.1787664123654198, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.026266256300000002, + 0.00047061504999999765 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09523809523809523, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.0117493178844458, + 0.30377440691392416 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 36, + "cost": 0.19468825, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.05663328100166687, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0005602417, + 6.0092474999999715e-05 + ], + "strong_routes": 38, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.02631578947368421, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0006660362042023661, + 0.13809902979242009 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 8.39968576, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.2159529231563302, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.16257429948000002, + -0.03598645919000014 + ], + "strong_routes": 17, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.058823529411764705, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.001488174391266636, + 0.28688939666722063 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 8.153954630000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.034674460204333624, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.02422112356000001, + -0.0018927489900000074 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.19047619047619047, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.054463568178406814, + 0.41906604118352786 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_upfront_scalar_calibration": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_review_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_review_per_model": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_empirical_review_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 14.003645670000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.3298538672152522, + "baseline_solved": 87, + "gained_sol_failures": 2, + "lost_sol_successes": 4, + "paired_exact_p": 0.6875, + "quality_delta_bootstrap_95": [ + -0.056, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.08716027666, + -0.03046360189800001 + ], + "strong_routes": 100, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.07, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.028605288907438704, + 0.13891972845585732 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3922581500000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.08999404553369261, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.018160890399999997, + 0.0008559290000000007 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2297896, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.11345117649339076, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00045055567499999996, + 0.0004872066999999999 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 5.35368162, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.5002743502022768, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.33043864058, + -0.1220960099900001 + ], + "strong_routes": 10, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3084971078187608 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 7.0279163, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.16798321749599177, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.1264185382, + -0.011905893240000002 + ], + "strong_routes": 16, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.015513603815413892, + 0.38347623684926346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "ablation_v2_original_upfront_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_per_model": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 20.19712788, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.03346403778716789, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.024, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.012449875198, + -0.0005903775300000025 + ], + "strong_routes": 119, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 10, + "sol_route_rescue_precision": 0.08403361344537816, + "sol_only_recall": 0.9090909090909091, + "sol_route_rescue_precision_exact_95": [ + 0.041034387787323594, + 0.14911334056870384 + ], + "sol_only_recall_exact_95": [ + 0.5872200830116174, + 0.9977010277861857 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 1.24396835, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.18691903102335772, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.029562692249999994, + 0.0007949600000000009 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.011205586024150999, + 0.29161274155393346 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2263206, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.09664205140132576, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0002497678749999999, + 0.00048172807500000003 + ], + "strong_routes": 49, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04081632653061224, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004981841834971302, + 0.1397871870224637 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.263600330000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.04197060859712143, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.04842432615, + 0.0007329489000000062 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08695652173913043, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.010709966037875893, + 0.2803793359094109 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4632386, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.001940866816488418, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006238924000000015, + 0.0006923091999999893 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_raw": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_rescue": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_upfront_scalar_calibration": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_review_paired": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 20.896406000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 125, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.088, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.04475062974201752, + 0.15200141787376736 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.529944, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.206376, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 10.7132416, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 8.4468444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.0, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0453794523717096, + 0.3608284544592721 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "cap_original_raw_baseline": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 6.76416331, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.6763001585057259, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 10, + "paired_exact_p": 0.8145294189453125, + "quality_delta_bootstrap_95": [ + -0.08, + 0.048 + ], + "mean_cost_delta_bootstrap_95": [ + -0.15178693768999998, + -0.08009726906600002 + ], + "strong_routes": 9, + "classifier_reviewer_cost": 0.05266521, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1111111111111111, + "sol_only_recall": 0.09090909090909091, + "sol_route_rescue_precision_exact_95": [ + 0.0028091367465992167, + 0.482496514917337 + ], + "sol_only_recall_exact_95": [ + 0.002298972213814269, + 0.4127799169883826 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 0.29180008, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8092740126436, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07231712887, + -0.029113650460000026 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.01762788, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.7075982261787133 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.028999999999999998, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8594797844710625, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004671895275000001, + -0.0026354289000000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.011604, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 1.3892767799999999, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.870321529946641, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.49086965658000004, + -0.27021434123000004 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0.0111856, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.841886116991581 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 5.054086450000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.4016598139300399, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.2 + ], + "mean_cost_delta_bootstrap_95": [ + -0.21025275142999997, + -0.07596707077000003 + ], + "strong_routes": 6, + "classifier_reviewer_cost": 0.01224773, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.25, + "sol_route_rescue_precision_exact_95": [ + 0.004210744514489473, + 0.6412345789976748 + ], + "sol_only_recall_exact_95": [ + 0.006309463209709872, + 0.8058795503167566 + ] + } + } + } + }, + "cap_empirical_raw_baseline": { + "total": { + "tasks": 125, + "solved": 84, + "cost": 7.249637870000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.6530677155679308, + "baseline_solved": 87, + "gained_sol_failures": 7, + "lost_sol_successes": 10, + "paired_exact_p": 0.629058837890625, + "quality_delta_bootstrap_95": [ + -0.088, + 0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.14845173568000006, + -0.07582031650200001 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.04935316, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.08333333333333333, + "sol_only_recall": 0.09090909090909091, + "sol_route_rescue_precision_exact_95": [ + 0.002107593231860228, + 0.38479616515094406 + ], + "sol_only_recall_exact_95": [ + 0.002298972213814269, + 0.4127799169883826 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 12, + "cost": 0.4859902, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.682347719916546, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 3, + "paired_exact_p": 0.625, + "quality_delta_bootstrap_95": [ + -0.24, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0633542286, + -0.022207698250000015 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0.017241, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.7075982261787133 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.7075982261787133 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.028446199999999998, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8621632360351978, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.00468852185, + -0.0026388404500000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.0110502, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 1.68262264, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8429399146566432, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.48227658493000003, + -0.25442487609000003 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0.01032185, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.7075982261787133 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 5.052578830000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.4018382971515373, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.2 + ], + "mean_cost_delta_bootstrap_95": [ + -0.21032360977, + -0.07603086512000003 + ], + "strong_routes": 6, + "classifier_reviewer_cost": 0.010740109999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.25, + "sol_route_rescue_precision_exact_95": [ + 0.004210744514489473, + 0.6412345789976748 + ], + "sol_only_recall_exact_95": [ + 0.006309463209709872, + 0.8058795503167566 + ] + } + } + } + }, + "v2_original_raw_baseline": { + "total": { + "tasks": 125, + "solved": 84, + "cost": 17.00290026, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.18632418129701356, + "baseline_solved": 87, + "gained_sol_failures": 3, + "lost_sol_successes": 6, + "paired_exact_p": 0.5078125, + "quality_delta_bootstrap_95": [ + -0.072, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.051681949914000005, + -0.01354233937400001 + ], + "strong_routes": 51, + "classifier_reviewer_cost": 0.07718525, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 5, + "sol_route_rescue_precision": 0.09803921568627451, + "sol_only_recall": 0.45454545454545453, + "sol_route_rescue_precision_exact_95": [ + 0.03260648955189092, + 0.21414498254568087 + ], + "sol_only_recall_exact_95": [ + 0.16748809406370715, + 0.7662064023406548 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 12, + "cost": 0.99244255, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.3513209960626009, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.03863867435, + -0.007342218450000004 + ], + "strong_routes": 9, + "classifier_reviewer_cost": 0.01984635, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1111111111111111, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.0028091367465992167, + 0.482496514917337 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.056221600000000004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7275768500213202, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0041246895000000006, + -0.0021125469000000006 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0.0234136, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.841886116991581 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 9.13040559, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.14774575885603092, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.13240011395, + -0.012091017849999993 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.0175311, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.05, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.001265089497949805, + 0.2487327627720277 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 6.8238305200000005, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.19214440365445817, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.14368180549, + -0.005013580750000003 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.0163942, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.15, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.032070937185463715, + 0.3789268265453139 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "v2_empirical_raw_baseline": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 14.64873408, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.29898308445959565, + "baseline_solved": 87, + "gained_sol_failures": 6, + "lost_sol_successes": 8, + "paired_exact_p": 0.79052734375, + "quality_delta_bootstrap_95": [ + -0.072, + 0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07110927915399999, + -0.03075588462600001 + ], + "strong_routes": 33, + "classifier_reviewer_cost": 0.05889908, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 0.2727272727272727, + "sol_route_rescue_precision_exact_95": [ + 0.019154942987404933, + 0.24331635075915498 + ], + "sol_only_recall_exact_95": [ + 0.06021773417290667, + 0.6097425595724212 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 13, + "cost": 0.47988020000000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.6863413301401882, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 3, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.2, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.06521872845, + -0.022116862650000006 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0.017123199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.7075982261787133 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.7075982261787133 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.03958005, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8082138911501338, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004503112775000001, + -0.002399418000000001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.016453449999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 7.418084090000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.30757800794859325, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.20128575522, + -0.07104596217000003 + ], + "strong_routes": 13, + "classifier_reviewer_cost": 0.012589399999999999, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.07692307692307693, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0019456284973467497, + 0.3602974352678774 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 6.71118974, + "unknown_billing_requests": 1, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.2054796534431248, + "baseline_solved": 18, + "gained_sol_failures": 1, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.08 + ], + "mean_cost_delta_bootstrap_95": [ + -0.14393527747999998, + -0.017818168519999997 + ], + "strong_routes": 16, + "classifier_reviewer_cost": 0.01273303, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.015513603815413892, + 0.38347623684926346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "cap_review_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 81, + "cost": 7.582970260000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6371160543109662, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 6, + "paired_exact_p": 0.03125, + "quality_delta_bootstrap_95": [ + -0.088, + -0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.15162677911000003, + -0.0657057803420001 + ], + "strong_routes": 95, + "classifier_reviewer_cost": 0.14986411, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 5, + "sol_route_rescue_precision": 0.05263157894736842, + "sol_only_recall": 0.45454545454545453, + "sol_route_rescue_precision_exact_95": [ + 0.01730787584643997, + 0.11856278880488623 + ], + "sol_only_recall_exact_95": [ + 0.16748809406370715, + 0.7662064023406548 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68462508, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1011024455797076, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00456608113, + 0.007829581510000003 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.03820068, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.25031600000000004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.21291235414970755, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006985228249999999, + 0.001141979325 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.026543999999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 2.47515794, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7689627441987307, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.4611561338500001, + -0.21285575223000017 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.046706899999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3694166475528192 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 14, + "cost": 3.17287124, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6243720033483746, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 4, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.32, + -0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.3343958345, + -0.10576620506000002 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03841253, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.2646484693970512 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.6023646356164746 + ] + } + } + } + }, + "cap_review_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 10.18532141, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.5125802298251672, + "baseline_solved": 87, + "gained_sol_failures": 7, + "lost_sol_successes": 7, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.056, + 0.056 + ], + "mean_cost_delta_bootstrap_95": [ + -0.12620953304200003, + -0.049324036836 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.14655205999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.18181818181818182, + "sol_only_recall": 0.36363636363636365, + "sol_route_rescue_precision_exact_95": [ + 0.05186729931243693, + 0.4028457830778714 + ], + "sol_only_recall_exact_95": [ + 0.10926344381909812, + 0.6920952849883228 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3256502000000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.1335302468587085, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.015286975900000005, + -0.0016845305000000009 + ], + "strong_routes": 9, + "classifier_reviewer_cost": 0.037813799999999995, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1111111111111111, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.0028091367465992167, + 0.482496514917337 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.0433862, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7897710974144281, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004362987525000001, + -0.0023665260000000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.025990199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.9752569899999999, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9089671430540687, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5038894888000001, + -0.2904808679400001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.04584315, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 7.84102802, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.07172102992686824, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.11185988509999999, + 0.055466048719999966 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03690491, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.25, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.054860644527992754, + 0.5718584618781891 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "cap_either_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 81, + "cost": 7.582970260000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6371160543109662, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 6, + "paired_exact_p": 0.03125, + "quality_delta_bootstrap_95": [ + -0.088, + -0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.15162677911000003, + -0.0657057803420001 + ], + "strong_routes": 95, + "classifier_reviewer_cost": 0.14986411, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 5, + "sol_route_rescue_precision": 0.05263157894736842, + "sol_only_recall": 0.45454545454545453, + "sol_route_rescue_precision_exact_95": [ + 0.01730787584643997, + 0.11856278880488623 + ], + "sol_only_recall_exact_95": [ + 0.16748809406370715, + 0.7662064023406548 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68462508, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1011024455797076, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00456608113, + 0.007829581510000003 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.03820068, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.25031600000000004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.21291235414970755, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006985228249999999, + 0.001141979325 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.026543999999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 2.47515794, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7689627441987307, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.4611561338500001, + -0.21285575223000017 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.046706899999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3694166475528192 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 14, + "cost": 3.17287124, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6243720033483746, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 4, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.32, + -0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.3343958345, + -0.10576620506000002 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03841253, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.2646484693970512 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.6023646356164746 + ] + } + } + } + }, + "cap_either_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 87, + "cost": 10.18532141, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.5125802298251672, + "baseline_solved": 87, + "gained_sol_failures": 7, + "lost_sol_successes": 7, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.056, + 0.056 + ], + "mean_cost_delta_bootstrap_95": [ + -0.12620953304200003, + -0.049324036836 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.14655205999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.18181818181818182, + "sol_only_recall": 0.36363636363636365, + "sol_route_rescue_precision_exact_95": [ + 0.05186729931243693, + 0.4028457830778714 + ], + "sol_only_recall_exact_95": [ + 0.10926344381909812, + 0.6920952849883228 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.3256502000000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.1335302468587085, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.015286975900000005, + -0.0016845305000000009 + ], + "strong_routes": 9, + "classifier_reviewer_cost": 0.037813799999999995, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.1111111111111111, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.0028091367465992167, + 0.482496514917337 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.0433862, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7897710974144281, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.004362987525000001, + -0.0023665260000000003 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.025990199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.9752569899999999, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9089671430540687, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5038894888000001, + -0.2904808679400001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.04584315, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 19, + "cost": 7.84102802, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.07172102992686824, + "baseline_solved": 18, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.11185988509999999, + 0.055466048719999966 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03690491, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.25, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.054860644527992754, + 0.5718584618781891 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "v2_review_strong_success_retention": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.9753999, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.0037802625006422286, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.026018115068, + 0.02134152558 + ], + "strong_routes": 115, + "classifier_reviewer_cost": 0.17438415, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.09565217391304348, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.048722767492542195, + 0.1647044453125498 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.60758355, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.050746661315708375, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0014359300500000006, + 0.006706794349999999 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.04041915, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.13636363636363635, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.029055851128746705, + 0.349122097257408 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.24409760000000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1827809435205645, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00032057892499999985, + 0.0010412571749999998 + ], + "strong_routes": 48, + "classifier_reviewer_cost": 0.0383536, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.041666666666666664, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.005086467930032977, + 0.14254097320137743 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.12625244, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.14813342396758788, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18424675188000003, + 0.01805392697 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.0530524, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09523809523809523, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0117493178844458, + 0.30377440691392416 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 9.99746631, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.18357410608865954, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.01264921694, + 0.10895171352 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.042559, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.047353626607257136, + 0.37384171348868267 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "v2_review_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 8.649280130000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.5860876683770405, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 7, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.048, + 0.072 + ], + "mean_cost_delta_bootstrap_95": [ + -0.137710579418, + -0.06163882030400005 + ], + "strong_routes": 15, + "classifier_reviewer_cost": 0.15609798, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.26666666666666666, + "sol_only_recall": 0.36363636363636365, + "sol_route_rescue_precision_exact_95": [ + 0.0778715462910436, + 0.551003241036971 + ], + "sol_only_recall_exact_95": [ + 0.10926344381909812, + 0.6920952849883228 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.0584044000000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.30820709777612765, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0339924741, + -0.006823776650000001 + ], + "strong_routes": 6, + "classifier_reviewer_cost": 0.037696, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.004210744514489473, + 0.6412345789976748 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.04878945, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7635895162228166, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0042552748250000005, + -0.0022587952250000005 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.031393449999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.97752454, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9087554844277945, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5038272815000002, + -0.2903748879100001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.0481107, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 20, + "cost": 6.56456174, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.2228385620551977, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 1, + "paired_exact_p": 0.625, + "quality_delta_bootstrap_95": [ + -0.08, + 0.24 + ], + "mean_cost_delta_bootstrap_95": [ + -0.16652745669, + 0.009938520709999963 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.03889783, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.375, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.08523341413725358, + 0.7551367836334484 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "v2_either_quality_first_under_sol_budget": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 8.649280130000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.5860876683770405, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 7, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.048, + 0.072 + ], + "mean_cost_delta_bootstrap_95": [ + -0.137710579418, + -0.06163882030400005 + ], + "strong_routes": 15, + "classifier_reviewer_cost": 0.15609798, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.26666666666666666, + "sol_only_recall": 0.36363636363636365, + "sol_route_rescue_precision_exact_95": [ + 0.0778715462910436, + 0.551003241036971 + ], + "sol_only_recall_exact_95": [ + 0.10926344381909812, + 0.6920952849883228 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.0584044000000001, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.30820709777612765, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.16, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0339924741, + -0.006823776650000001 + ], + "strong_routes": 6, + "classifier_reviewer_cost": 0.037696, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 0.3333333333333333, + "sol_route_rescue_precision_exact_95": [ + 0.004210744514489473, + 0.6412345789976748 + ], + "sol_only_recall_exact_95": [ + 0.008403758659612645, + 0.9057006759497537 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.04878945, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7635895162228166, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 2, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.06, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0042552748250000005, + -0.0022587952250000005 + ], + "strong_routes": 0, + "classifier_reviewer_cost": 0.031393449999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": null, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": null, + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 0.97752454, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.9087554844277945, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.5038272815000002, + -0.2903748879100001 + ], + "strong_routes": 1, + "classifier_reviewer_cost": 0.0481107, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.975 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 20, + "cost": 6.56456174, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.2228385620551977, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 1, + "paired_exact_p": 0.625, + "quality_delta_bootstrap_95": [ + -0.08, + 0.24 + ], + "mean_cost_delta_bootstrap_95": [ + -0.16652745669, + 0.009938520709999963 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.03889783, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.375, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.08523341413725358, + 0.7551367836334484 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "ablation_cap_empirical_review_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 82, + "cost": 16.807459010000002, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.19567704561253263, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 5, + "paired_exact_p": 0.0625, + "quality_delta_bootstrap_95": [ + -0.08, + -0.008 + ], + "mean_cost_delta_bootstrap_95": [ + -0.06854206789400004, + -0.0038170915060000052 + ], + "strong_routes": 115, + "classifier_reviewer_cost": 0.14655205999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 6, + "sol_route_rescue_precision": 0.05217391304347826, + "sol_only_recall": 0.5454545454545454, + "sol_route_rescue_precision_exact_95": [ + 0.019385005671561636, + 0.11010465960377613 + ], + "sol_only_recall_exact_95": [ + 0.23379359765934518, + 0.8325119059362929 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.6842382, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.10084957357916369, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0045572192499999985, + 0.007806525299999999 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.037813799999999995, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.24976220000000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.21022890258557192, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006922497249999998, + 0.0011280457999999997 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.025990199999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 7.90735119, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.2619086281037478, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.24478701523000004, + -0.007153033600000035 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.04584315, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.1684334709830853 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 15, + "cost": 6.96610742, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.17530061048597034, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 3, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + -0.24, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.17648076557000003, + 0.03294136301999995 + ], + "strong_routes": 20, + "classifier_reviewer_cost": 0.03690491, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.05, + "sol_only_recall": 0.25, + "sol_route_rescue_precision_exact_95": [ + 0.001265089497949805, + 0.2487327627720277 + ], + "sol_only_recall_exact_95": [ + 0.006309463209709872, + 0.8058795503167566 + ] + } + } + } + }, + "ablation_cap_original_review_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 81, + "cost": 7.582970260000001, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6371160543109662, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 6, + "paired_exact_p": 0.03125, + "quality_delta_bootstrap_95": [ + -0.088, + -0.016 + ], + "mean_cost_delta_bootstrap_95": [ + -0.15162677911000003, + -0.0657057803420001 + ], + "strong_routes": 95, + "classifier_reviewer_cost": 0.14986411, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 5, + "sol_route_rescue_precision": 0.05263157894736842, + "sol_only_recall": 0.45454545454545453, + "sol_route_rescue_precision_exact_95": [ + 0.01730787584643997, + 0.11856278880488623 + ], + "sol_only_recall_exact_95": [ + 0.16748809406370715, + 0.7662064023406548 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68462508, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1011024455797076, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00456608113, + 0.007829581510000003 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.03820068, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.25031600000000004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.21291235414970755, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0006985228249999999, + 0.001141979325 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.026543999999999998, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 16, + "cost": 2.47515794, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7689627441987307, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.4611561338500001, + -0.21285575223000017 + ], + "strong_routes": 8, + "classifier_reviewer_cost": 0.046706899999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.3694166475528192 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.841886116991581 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 14, + "cost": 3.17287124, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6243720033483746, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 4, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.32, + -0.04 + ], + "mean_cost_delta_bootstrap_95": [ + -0.3343958345, + -0.10576620506000002 + ], + "strong_routes": 12, + "classifier_reviewer_cost": 0.03841253, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 0, + "sol_route_rescue_precision": 0.0, + "sol_only_recall": 0.0, + "sol_route_rescue_precision_exact_95": [ + 0.0, + 0.2646484693970512 + ], + "sol_only_recall_exact_95": [ + 0.0, + 0.6023646356164746 + ] + } + } + } + }, + "ablation_v2_empirical_review_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 83, + "cost": 16.32433933, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.21879679548722408, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 4, + "paired_exact_p": 0.125, + "quality_delta_bootstrap_95": [ + -0.064, + -0.008 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07417190475199999, + -0.006337188472000025 + ], + "strong_routes": 114, + "classifier_reviewer_cost": 0.15609798, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 7, + "sol_route_rescue_precision": 0.06140350877192982, + "sol_only_recall": 0.6363636363636364, + "sol_route_rescue_precision_exact_95": [ + 0.025041319674763633, + 0.12242691621974269 + ], + "sol_only_recall_exact_95": [ + 0.3079047150116772, + 0.8907365561809019 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.6841204, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.10077257729694677, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0045775936500000005, + 0.0077651657 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.037696, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.25516545, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.23641048377718343, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.000798436425, + 0.0012493406249999995 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.031393449999999996, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 17, + "cost": 8.98205814, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.16159287026626945, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.19684950881, + 0.02339831545999997 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.0481107, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.045454545454545456, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.0011501475265735762, + 0.2284443976676335 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 15, + "cost": 5.40299534, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.360353395405271, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 3, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + -0.24, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.24350404569, + -0.014465777570000041 + ], + "strong_routes": 17, + "classifier_reviewer_cost": 0.03889783, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.058823529411764705, + "sol_only_recall": 0.25, + "sol_route_rescue_precision_exact_95": [ + 0.001488174391266636, + 0.28688939666722063 + ], + "sol_only_recall_exact_95": [ + 0.006309463209709872, + 0.8058795503167566 + ] + } + } + } + }, + "ablation_v2_original_review_benefit_per_dollar": { + "total": { + "tasks": 125, + "solved": 85, + "cost": 19.8241285, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.05131396757892248, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.04, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.03935038515, + 0.014239168169999995 + ], + "strong_routes": 120, + "classifier_reviewer_cost": 0.17438415, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 9, + "sol_route_rescue_precision": 0.075, + "sol_only_recall": 0.8181818181818182, + "sol_route_rescue_precision_exact_95": [ + 0.03486739061196212, + 0.13760105789036722 + ], + "sol_only_recall_exact_95": [ + 0.4822441476398273, + 0.9771688017000403 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68684355, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.10255247904498477, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.004706471149999998, + 0.007860030849999998 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.04041915, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2621256, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.2701360623328295, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0009425592999999999, + 0.0013770193499999996 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0383536, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.30107364, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.13181518841132078, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18580700010999998, + 0.02750766961 + ], + "strong_routes": 23, + "classifier_reviewer_cost": 0.0530524, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08695652173913043, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.010709966037875893, + 0.2803793359094109 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 16, + "cost": 8.57408571, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.015063768666083277, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 2, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + -0.2, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07362103302000002, + 0.06466245526999996 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.042559, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09090909090909091, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.011205586024150999, + 0.29161274155393346 + ], + "sol_only_recall_exact_95": [ + 0.06758598648854298, + 0.932414013511457 + ] + } + } + } + }, + "ablation_v2_original_review_per_model": { + "total": { + "tasks": 125, + "solved": 88, + "cost": 20.9753999, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.0037802625006422286, + "baseline_solved": 87, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.024 + ], + "mean_cost_delta_bootstrap_95": [ + -0.026018115068, + 0.02134152558 + ], + "strong_routes": 115, + "classifier_reviewer_cost": 0.17438415, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 0.09565217391304348, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.048722767492542195, + 0.1647044453125498 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.60758355, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.050746661315708375, + "baseline_solved": 14, + "gained_sol_failures": 1, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.12 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0014359300500000006, + 0.006706794349999999 + ], + "strong_routes": 22, + "classifier_reviewer_cost": 0.04041915, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.13636363636363635, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.029055851128746705, + 0.349122097257408 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.24409760000000003, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.1827809435205645, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.00032057892499999985, + 0.0010412571749999998 + ], + "strong_routes": 48, + "classifier_reviewer_cost": 0.0383536, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.041666666666666664, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.005086467930032977, + 0.14254097320137743 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 9.12625244, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.14813342396758788, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.18424675188000003, + 0.01805392697 + ], + "strong_routes": 21, + "classifier_reviewer_cost": 0.0530524, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.09523809523809523, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.0117493178844458, + 0.30377440691392416 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 18, + "cost": 9.99746631, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.18357410608865954, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.01264921694, + 0.10895171352 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.042559, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 0.16666666666666666, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.047353626607257136, + 0.37384171348868267 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "ablation_v2_original_review_rescue": { + "total": { + "tasks": 125, + "solved": 86, + "cost": 23.408775900000002, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.12022976104120486, + "baseline_solved": 87, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.024, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.012625334294, + 0.02913148761 + ], + "strong_routes": 124, + "classifier_reviewer_cost": 0.17438415, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 10, + "sol_route_rescue_precision": 0.08064516129032258, + "sol_only_recall": 0.9090909090909091, + "sol_route_rescue_precision_exact_95": [ + 0.03935028219135281, + 0.14331003914481275 + ], + "sol_only_recall_exact_95": [ + 0.5872200830116174, + 0.9977010277861857 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 14, + "cost": 1.68684355, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.10255247904498477, + "baseline_solved": 14, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.004706471149999998, + 0.007860030849999998 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.04041915, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.12, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.025465396647733205, + 0.31219030728623537 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 37, + "cost": 0.2621256, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.2701360623328295, + "baseline_solved": 37, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.0009425592999999999, + 0.0013770193499999996 + ], + "strong_routes": 50, + "classifier_reviewer_cost": 0.0383536, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.04, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.004881433426164381, + 0.1371376256039674 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 11.40340044, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": -0.06442110294609615, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.021655114940000005, + 0.034285069830000015 + ], + "strong_routes": 25, + "classifier_reviewer_cost": 0.0530524, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.08, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.009839590018797513, + 0.26030584210521424 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 17, + "cost": 10.05640631, + "unknown_billing_requests": 3, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": -0.1905518598164304, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.12, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + 0.03293031734, + 0.10278958102999998 + ], + "strong_routes": 24, + "classifier_reviewer_cost": 0.042559, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.125, + "sol_only_recall": 0.75, + "sol_route_rescue_precision_exact_95": [ + 0.026559314986248944, + 0.32361135818883285 + ], + "sol_only_recall_exact_95": [ + 0.19412044968324343, + 0.9936905367902901 + ] + } + } + } + }, + "public_examples_cascade": { + "total": { + "tasks": 75, + "solved": 53, + "cost": 1.3885004, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.20031998709915222, + "baseline_solved": 51, + "gained_sol_failures": 4, + "lost_sol_successes": 2, + "paired_exact_p": 0.6875, + "quality_delta_bootstrap_95": [ + -0.04, + 0.09333333333333334 + ], + "mean_cost_delta_bootstrap_95": [ + -0.006993307266666667, + -0.0024630961333333354 + ], + "strong_routes": 13, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 5, + "sol_only_captured": 3, + "sol_route_rescue_precision": 0.23076923076923078, + "sol_only_recall": 0.6, + "sol_route_rescue_precision_exact_95": [ + 0.05038107349115152, + 0.538131539234041 + ], + "sol_only_recall_exact_95": [ + 0.14663279963467318, + 0.9472550494736831 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 15, + "cost": 1.3645444, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.10810827062951323, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.08, + 0.16 + ], + "mean_cost_delta_bootstrap_95": [ + -0.013030371000000002, + -0.0006957486000000024 + ], + "strong_routes": 11, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 2, + "sol_route_rescue_precision": 0.18181818181818182, + "sol_only_recall": 0.6666666666666666, + "sol_route_rescue_precision_exact_95": [ + 0.022831198299959672, + 0.5177558523601726 + ], + "sol_only_recall_exact_95": [ + 0.09429932405024613, + 0.9915962413403874 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 38, + "cost": 0.023955999999999998, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8839206109237508, + "baseline_solved": 37, + "gained_sol_failures": 2, + "lost_sol_successes": 1, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + -0.04, + 0.1 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0048078653, + -0.0026865571000000005 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0.0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 1, + "sol_route_rescue_precision": 0.5, + "sol_only_recall": 0.5, + "sol_route_rescue_precision_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ], + "sol_only_recall_exact_95": [ + 0.01257911709342506, + 0.9874208829065749 + ] + } + } + } + }, + "hindsight_oracle_unattainable": { + "total": { + "tasks": 125, + "solved": 95, + "cost": 4.51896464, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.7837444084882348, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 0, + "paired_exact_p": 0.0078125, + "quality_delta_bootstrap_95": [ + 0.024, + 0.112 + ], + "mean_cost_delta_bootstrap_95": [ + -0.173399360712, + -0.092329002966 + ], + "strong_routes": 11, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.7150858470818456, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 16, + "cost": 0.3436374, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7753921712167242, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 0, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + 0.0, + 0.2 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07116694220000001, + -0.025871578400000025 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.2924017738212867, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 40, + "cost": 0.0352694, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8291012520835757, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 0, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + 0.0, + 0.14 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0045176579, + -0.0025793627 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.15811388300841905, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 1.4341749300000002, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8661306275404075, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.48791458023, + -0.2698511783400001 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.15811388300841905, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 21, + "cost": 2.70588291, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6796575405129992, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 0, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + 0.0, + 0.24 + ], + "mean_cost_delta_bootstrap_95": [ + -0.34443187702, + -0.13137291555000002 + ], + "strong_routes": 4, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.3976353643835254, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + }, + "hindsight_cascade_oracle_unattainable": { + "total": { + "tasks": 125, + "solved": 95, + "cost": 5.26530215, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.7480283379830963, + "baseline_solved": 87, + "gained_sol_failures": 8, + "lost_sol_successes": 0, + "paired_exact_p": 0.0078125, + "quality_delta_bootstrap_95": [ + 0.024, + 0.112 + ], + "mean_cost_delta_bootstrap_95": [ + -0.16898772801800002, + -0.085152777052 + ], + "strong_routes": 11, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 11, + "sol_only_captured": 11, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.7150858470818456, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.7150858470818456, + 1.0 + ] + } + }, + "benchmarks": { + "lcb": { + "tasks": 25, + "solved": 16, + "cost": 0.3657364, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.7609478516860748, + "baseline_solved": 14, + "gained_sol_failures": 2, + "lost_sol_successes": 0, + "paired_exact_p": 0.5, + "quality_delta_bootstrap_95": [ + 0.0, + 0.2 + ], + "mean_cost_delta_bootstrap_95": [ + -0.07068738919999999, + -0.024637301600000005 + ], + "strong_routes": 3, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 3, + "sol_only_captured": 3, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.2924017738212867, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.2924017738212867, + 1.0 + ] + } + }, + "mbpp": { + "tasks": 50, + "solved": 40, + "cost": 0.036092, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8251153234872273, + "baseline_solved": 37, + "gained_sol_failures": 3, + "lost_sol_successes": 0, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + 0.0, + 0.14 + ], + "mean_cost_delta_bootstrap_95": [ + -0.0045073966000000005, + -0.0025570753 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.15811388300841905, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "swe": { + "tasks": 25, + "solved": 18, + "cost": 1.54781684, + "unknown_billing_requests": 0, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": true, + "savings_vs_sol": 0.8555230155548812, + "baseline_solved": 18, + "gained_sol_failures": 0, + "lost_sol_successes": 0, + "paired_exact_p": 1.0, + "quality_delta_bootstrap_95": [ + 0.0, + 0.0 + ], + "mean_cost_delta_bootstrap_95": [ + -0.48565299367000003, + -0.26237691106 + ], + "strong_routes": 2, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 2, + "sol_only_captured": 2, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.15811388300841905, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.15811388300841905, + 1.0 + ] + } + }, + "terminal": { + "tasks": 25, + "solved": 21, + "cost": 3.31565691, + "unknown_billing_requests": 2, + "baseline_unknown_billing_requests": 0, + "cost_comparison_complete": false, + "savings_vs_sol": 0.6074679782191796, + "baseline_solved": 18, + "gained_sol_failures": 3, + "lost_sol_successes": 0, + "paired_exact_p": 0.25, + "quality_delta_bootstrap_95": [ + 0.0, + 0.24 + ], + "mean_cost_delta_bootstrap_95": [ + -0.33003004255, + -0.09622135258999999 + ], + "strong_routes": 4, + "classifier_reviewer_cost": 0, + "rescue_diagnostics": { + "sol_only_opportunities": 4, + "sol_only_captured": 4, + "sol_route_rescue_precision": 1.0, + "sol_only_recall": 1.0, + "sol_route_rescue_precision_exact_95": [ + 0.3976353643835254, + 1.0 + ], + "sol_only_recall_exact_95": [ + 0.3976353643835254, + 1.0 + ] + } + } + } + } + } +} diff --git a/cookbook/auto_router_selective_training/calibration_results.json b/cookbook/auto_router_selective_training/calibration_results.json new file mode 100644 index 00000000000..882ad358d4a --- /dev/null +++ b/cookbook/auto_router_selective_training/calibration_results.json @@ -0,0 +1,5849 @@ +{ + "scope": "Held-out diagnostics only; no fitting or policy selection from these results. AUC for the signed success-probability gap measures rescue ranking, not calibration of a Sol-only probability.", + "results": { + "cap_original_raw": { + "efficient": { + "n": 125, + "brier": 0.238939608, + "auc": 0.6713124274099884, + "bins": [ + { + "n": 2, + "mean_prediction": 0.105, + "observed_frequency": 0.5 + }, + { + "n": 2, + "mean_prediction": 0.295, + "observed_frequency": 0.5 + }, + { + "n": 6, + "mean_prediction": 0.5133333333333333, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 33, + "mean_prediction": 0.7396969696969697, + "observed_frequency": 0.5757575757575758 + }, + { + "n": 82, + "mean_prediction": 0.943890243902439, + "observed_frequency": 0.7439024390243902 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.312881, + "auc": 0.8205128205128206, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.73, + "observed_frequency": 0.25 + }, + { + "n": 19, + "mean_prediction": 0.9302631578947367, + "observed_frequency": 0.631578947368421 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.22063651999999997, + "auc": 0.6633771929824561, + "bins": [ + { + "n": 2, + "mean_prediction": 0.7, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.9744583333333333, + "observed_frequency": 0.75 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.22333199999999997, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5700000000000001, + "observed_frequency": 0.5 + }, + { + "n": 14, + "mean_prediction": 0.7378571428571429, + "observed_frequency": 0.5 + }, + { + "n": 9, + "mean_prediction": 0.8444444444444446, + "observed_frequency": 0.8888888888888888 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21721200000000004, + "auc": 0.625, + "bins": [ + { + "n": 2, + "mean_prediction": 0.105, + "observed_frequency": 0.5 + }, + { + "n": 2, + "mean_prediction": 0.295, + "observed_frequency": 0.5 + }, + { + "n": 2, + "mean_prediction": 0.47, + "observed_frequency": 0.5 + }, + { + "n": 13, + "mean_prediction": 0.7507692307692307, + "observed_frequency": 0.6923076923076923 + }, + { + "n": 6, + "mean_prediction": 0.8916666666666666, + "observed_frequency": 0.8333333333333334 + } + ] + } + } + } + }, + "v2_original_raw": { + "efficient": { + "n": 125, + "brier": 0.222738496, + "auc": 0.6702961672473867, + "bins": [ + { + "n": 3, + "mean_prediction": 0.06999999999999999, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.3133333333333334, + "observed_frequency": 0.5 + }, + { + "n": 14, + "mean_prediction": 0.49714285714285716, + "observed_frequency": 0.5 + }, + { + "n": 32, + "mean_prediction": 0.6912499999999999, + "observed_frequency": 0.6875 + }, + { + "n": 70, + "mean_prediction": 0.9539142857142855, + "observed_frequency": 0.7428571428571429 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.236602904, + "auc": 0.617816091954023, + "bins": [ + { + "n": 2, + "mean_prediction": 0.035, + "observed_frequency": 0.5 + }, + { + "n": 2, + "mean_prediction": 0.32999999999999996, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.5191666666666667, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 26, + "mean_prediction": 0.7296153846153847, + "observed_frequency": 0.5769230769230769 + }, + { + "n": 83, + "mean_prediction": 0.9463253012048195, + "observed_frequency": 0.7590361445783133 + } + ] + }, + "gap_mse": 0.15295239200000002, + "predicted_mean_gap": 0.052247999999999996, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6176236044657097, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.249792, + "auc": 0.7532051282051282, + "bins": [ + { + "n": 1, + "mean_prediction": 0.18, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.32, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.5066666666666666, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 5, + "mean_prediction": 0.6779999999999999, + "observed_frequency": 0.6 + }, + { + "n": 14, + "mean_prediction": 0.9107142857142858, + "observed_frequency": 0.6428571428571429 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22572799999999998, + "auc": 0.8149350649350648, + "bins": [ + { + "n": 1, + "mean_prediction": 0.3, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.52, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.72, + "observed_frequency": 0.4 + }, + { + "n": 15, + "mean_prediction": 0.9253333333333332, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.20014400000000002, + "predicted_mean_gap": 0.05520000000000001, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5909090909090908 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.22922823999999997, + "auc": 0.6622807017543859, + "bins": [ + { + "n": 2, + "mean_prediction": 0.74, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.9827916666666666, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.25340925999999997, + "auc": 0.6330561330561331, + "bins": [ + { + "n": 50, + "mean_prediction": 0.9825, + "observed_frequency": 0.74 + } + ] + }, + "gap_mse": 0.10050498, + "predicted_mean_gap": 0.009420000000000006, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.75 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.19096400000000002, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 1, + "mean_prediction": 0.28, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5, + "observed_frequency": 0.0 + }, + { + "n": 19, + "mean_prediction": 0.69, + "observed_frequency": 0.7368421052631579 + }, + { + "n": 3, + "mean_prediction": 0.8333333333333334, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.17622800000000002, + "auc": 0.8174603174603174, + "bins": [ + { + "n": 1, + "mean_prediction": 0.36, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.5, + "observed_frequency": 1.0 + }, + { + "n": 13, + "mean_prediction": 0.7569230769230769, + "observed_frequency": 0.6153846153846154 + }, + { + "n": 10, + "mean_prediction": 0.8450000000000001, + "observed_frequency": 0.9 + } + ] + }, + "gap_mse": 0.07872799999999999, + "predicted_mean_gap": 0.0904, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.2282608695652174 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21448, + "auc": 0.6544117647058824, + "bins": [ + { + "n": 2, + "mean_prediction": 0.015, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.32000000000000006, + "observed_frequency": 1.0 + }, + { + "n": 9, + "mean_prediction": 0.4933333333333333, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 6, + "mean_prediction": 0.69, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.8699999999999999, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.27424, + "auc": 0.36904761904761907, + "bins": [ + { + "n": 2, + "mean_prediction": 0.035, + "observed_frequency": 0.5 + }, + { + "n": 7, + "mean_prediction": 0.5214285714285715, + "observed_frequency": 1.0 + }, + { + "n": 8, + "mean_prediction": 0.6912499999999999, + "observed_frequency": 0.625 + }, + { + "n": 8, + "mean_prediction": 0.88625, + "observed_frequency": 0.625 + } + ] + }, + "gap_mse": 0.28488, + "predicted_mean_gap": 0.09680000000000001, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.494047619047619 + } + } + }, + "cap_empirical_raw": { + "efficient": { + "n": 125, + "brier": 0.235516, + "auc": 0.6782810685249709, + "bins": [ + { + "n": 3, + "mean_prediction": 0.11333333333333333, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 2, + "mean_prediction": 0.365, + "observed_frequency": 0.5 + }, + { + "n": 7, + "mean_prediction": 0.49571428571428566, + "observed_frequency": 0.42857142857142855 + }, + { + "n": 37, + "mean_prediction": 0.7337837837837837, + "observed_frequency": 0.5675675675675675 + }, + { + "n": 76, + "mean_prediction": 0.9486842105263157, + "observed_frequency": 0.7631578947368421 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.318396, + "auc": 0.7564102564102565, + "bins": [ + { + "n": 3, + "mean_prediction": 0.4433333333333333, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 4, + "mean_prediction": 0.74, + "observed_frequency": 0.0 + }, + { + "n": 18, + "mean_prediction": 0.9255555555555556, + "observed_frequency": 0.6666666666666666 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.21883999999999998, + "auc": 0.7050438596491229, + "bins": [ + { + "n": 3, + "mean_prediction": 0.6933333333333334, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.9765957446808511, + "observed_frequency": 0.7659574468085106 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.21998799999999996, + "auc": 0.736111111111111, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.49, + "observed_frequency": 0.5 + }, + { + "n": 17, + "mean_prediction": 0.7570588235294117, + "observed_frequency": 0.6470588235294118 + }, + { + "n": 5, + "mean_prediction": 0.844, + "observed_frequency": 0.8 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.20151599999999997, + "auc": 0.6948529411764706, + "bins": [ + { + "n": 3, + "mean_prediction": 0.11333333333333333, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 1, + "mean_prediction": 0.35, + "observed_frequency": 1.0 + }, + { + "n": 2, + "mean_prediction": 0.58, + "observed_frequency": 0.5 + }, + { + "n": 13, + "mean_prediction": 0.7107692307692308, + "observed_frequency": 0.6153846153846154 + }, + { + "n": 6, + "mean_prediction": 0.8866666666666666, + "observed_frequency": 1.0 + } + ] + } + } + } + }, + "v2_empirical_raw": { + "efficient": { + "n": 125, + "brier": 0.230588872, + "auc": 0.6743612078977933, + "bins": [ + { + "n": 3, + "mean_prediction": 0.07, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.2833333333333334, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 11, + "mean_prediction": 0.49545454545454537, + "observed_frequency": 0.45454545454545453 + }, + { + "n": 35, + "mean_prediction": 0.712, + "observed_frequency": 0.6285714285714286 + }, + { + "n": 70, + "mean_prediction": 0.9573571428571429, + "observed_frequency": 0.7571428571428571 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.240495128, + "auc": 0.6199334543254689, + "bins": [ + { + "n": 2, + "mean_prediction": 0.060000000000000005, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.28600000000000003, + "observed_frequency": 0.6 + }, + { + "n": 7, + "mean_prediction": 0.5157142857142858, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 31, + "mean_prediction": 0.7187096774193551, + "observed_frequency": 0.5806451612903226 + }, + { + "n": 80, + "mean_prediction": 0.9424125, + "observed_frequency": 0.7625 + } + ] + }, + "gap_mse": 0.15314735999999995, + "predicted_mean_gap": 0.028304, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.5283094098883573, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.26556599999999997, + "auc": 0.7660256410256411, + "bins": [ + { + "n": 1, + "mean_prediction": 0.14, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.34, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.495, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.7183333333333333, + "observed_frequency": 0.5 + }, + { + "n": 15, + "mean_prediction": 0.9146666666666665, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23513432, + "auc": 0.7922077922077921, + "bins": [ + { + "n": 2, + "mean_prediction": 0.29500000000000004, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.5, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.7183333333333334, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 16, + "mean_prediction": 0.90625, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.20596031999999997, + "predicted_mean_gap": 0.015999999999999997, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.3181818181818182 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.22757317999999999, + "auc": 0.6984649122807017, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7600000000000001, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.9862765957446806, + "observed_frequency": 0.7659574468085106 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.25562665999999995, + "auc": 0.6642411642411643, + "bins": [ + { + "n": 2, + "mean_prediction": 0.77, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.9852708333333334, + "observed_frequency": 0.7291666666666666 + } + ] + }, + "gap_mse": 0.09847623999999999, + "predicted_mean_gap": 0.003960000000000002, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.8385416666666667 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.200724, + "auc": 0.7569444444444444, + "bins": [ + { + "n": 2, + "mean_prediction": 0.495, + "observed_frequency": 0.0 + }, + { + "n": 20, + "mean_prediction": 0.71, + "observed_frequency": 0.7 + }, + { + "n": 3, + "mean_prediction": 0.8333333333333334, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18122800000000003, + "auc": 0.7261904761904762, + "bins": [ + { + "n": 1, + "mean_prediction": 0.49, + "observed_frequency": 0.0 + }, + { + "n": 14, + "mean_prediction": 0.7299999999999999, + "observed_frequency": 0.6428571428571429 + }, + { + "n": 10, + "mean_prediction": 0.8220000000000001, + "observed_frequency": 0.9 + } + ] + }, + "gap_mse": 0.07663199999999999, + "predicted_mean_gap": 0.04959999999999999, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.28260869565217395 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.231508, + "auc": 0.6580882352941176, + "bins": [ + { + "n": 2, + "mean_prediction": 0.034999999999999996, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.27199999999999996, + "observed_frequency": 0.8 + }, + { + "n": 7, + "mean_prediction": 0.49571428571428566, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 6, + "mean_prediction": 0.6883333333333334, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.8879999999999999, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.27486, + "auc": 0.5, + "bins": [ + { + "n": 2, + "mean_prediction": 0.060000000000000005, + "observed_frequency": 0.5 + }, + { + "n": 3, + "mean_prediction": 0.28, + "observed_frequency": 1.0 + }, + { + "n": 5, + "mean_prediction": 0.524, + "observed_frequency": 0.8 + }, + { + "n": 9, + "mean_prediction": 0.69, + "observed_frequency": 0.5555555555555556 + }, + { + "n": 6, + "mean_prediction": 0.8966666666666666, + "observed_frequency": 0.8333333333333334 + } + ] + }, + "gap_mse": 0.286192, + "predicted_mean_gap": 0.068, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.3035714285714286 + } + } + }, + "cap_original_upfront_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.21578879236003057, + "auc": 0.6536004645760743, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5055888529225132, + "observed_frequency": 0.0 + }, + { + "n": 90, + "mean_prediction": 0.7176629375856297, + "observed_frequency": 0.6444444444444445 + }, + { + "n": 33, + "mean_prediction": 0.8345212917151, + "observed_frequency": 0.7878787878787878 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2131703163517031, + "auc": 0.5931639443436176, + "bins": [ + { + "n": 1, + "mean_prediction": 0.555770699456167, + "observed_frequency": 0.0 + }, + { + "n": 84, + "mean_prediction": 0.7661084084960381, + "observed_frequency": 0.6904761904761905 + }, + { + "n": 40, + "mean_prediction": 0.8504392225830284, + "observed_frequency": 0.725 + } + ] + }, + "gap_mse": 0.15128313352119796, + "predicted_mean_gap": 0.04629120961035594, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6523125996810207, + "sol_only_rescue": { + "n": 125, + "brier": 0.07955130967379873, + "auc": 0.6523125996810207, + "bins": [ + { + "n": 125, + "mean_prediction": 0.08377164419969689, + "observed_frequency": 0.088 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.26436027656729394, + "auc": 0.6858974358974359, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5055888529225132, + "observed_frequency": 0.0 + }, + { + "n": 22, + "mean_prediction": 0.6976614387317333, + "observed_frequency": 0.5454545454545454 + }, + { + "n": 1, + "mean_prediction": 0.8341471865750809, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2672982373384959, + "auc": 0.5714285714285714, + "bins": [ + { + "n": 1, + "mean_prediction": 0.555770699456167, + "observed_frequency": 0.0 + }, + { + "n": 22, + "mean_prediction": 0.7529751123092933, + "observed_frequency": 0.5909090909090909 + }, + { + "n": 2, + "mean_prediction": 0.8354998573964358, + "observed_frequency": 0.5 + } + ] + }, + "gap_mse": 0.19613405616550308, + "predicted_mean_gap": 0.06393385362141006, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5909090909090908, + "sol_only_rescue": { + "n": 25, + "brier": 0.10598046514143906, + "auc": 0.6060606060606061, + "bins": [ + { + "n": 25, + "mean_prediction": 0.1089674247207968, + "observed_frequency": 0.12 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1841850742046408, + "auc": 0.5866228070175439, + "bins": [ + { + "n": 19, + "mean_prediction": 0.7211839376102072, + "observed_frequency": 0.7368421052631579 + }, + { + "n": 31, + "mean_prediction": 0.8346594628532378, + "observed_frequency": 0.7741935483870968 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.19598177198078268, + "auc": 0.5769230769230771, + "bins": [ + { + "n": 17, + "mean_prediction": 0.7699827029241958, + "observed_frequency": 0.6470588235294118 + }, + { + "n": 33, + "mean_prediction": 0.8569082359937822, + "observed_frequency": 0.7878787878787878 + } + ] + }, + "gap_mse": 0.10527414754717614, + "predicted_mean_gap": 0.035814791489236844, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.5104166666666666, + "sol_only_rescue": { + "n": 50, + "brier": 0.04088844390444613, + "auc": 0.53125, + "bins": [ + { + "n": 50, + "mean_prediction": 0.06893827316716425, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.2341796766580581, + "auc": 0.6805555555555556, + "bins": [ + { + "n": 24, + "mean_prediction": 0.7303836916583736, + "observed_frequency": 0.625 + }, + { + "n": 1, + "mean_prediction": 0.8306120915728509, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20078602119445169, + "auc": 0.7142857142857143, + "bins": [ + { + "n": 23, + "mean_prediction": 0.775783039816873, + "observed_frequency": 0.7391304347826086 + }, + { + "n": 2, + "mean_prediction": 0.8285393863546553, + "observed_frequency": 0.5 + } + ] + }, + "gap_mse": 0.07219176419405957, + "predicted_mean_gap": 0.045610719884943, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.9347826086956521, + "sol_only_rescue": { + "n": 25, + "brier": 0.07067574794325598, + "auc": 0.9347826086956521, + "bins": [ + { + "n": 25, + "mean_prediction": 0.08383557123821561, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21203386016551926, + "auc": 0.6102941176470589, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7203763726485458, + "observed_frequency": 0.68 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20580377926400273, + "auc": 0.4920634920634921, + "bins": [ + { + "n": 22, + "mean_prediction": 0.7661335444256061, + "observed_frequency": 0.7727272727272727 + }, + { + "n": 3, + "mean_prediction": 0.8038395426747105, + "observed_frequency": 0.3333333333333333 + } + ] + }, + "gap_mse": 0.27754155215207504, + "predicted_mean_gap": 0.05028189156695295, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5238095238095238, + "sol_only_rescue": { + "n": 25, + "brier": 0.13932344747540631, + "auc": 0.5119047619047619, + "bins": [ + { + "n": 25, + "mean_prediction": 0.08817867870514366, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "cap_original_upfront_scalar_calibration_C1.0": { + "efficient": { + "n": 125, + "brier": 0.204343872017707, + "auc": 0.6713124274099884, + "bins": [ + { + "n": 2, + "mean_prediction": 0.12415807062382743, + "observed_frequency": 0.5 + }, + { + "n": 6, + "mean_prediction": 0.31665388316333204, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 42, + "mean_prediction": 0.5225772167682418, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 23, + "mean_prediction": 0.6771396736131469, + "observed_frequency": 0.5652173913043478 + }, + { + "n": 52, + "mean_prediction": 0.8755975141269776, + "observed_frequency": 0.8461538461538461 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21018152489753747, + "auc": 0.6300665456745311, + "bins": [ + { + "n": 2, + "mean_prediction": 0.2932246265086425, + "observed_frequency": 1.0 + }, + { + "n": 9, + "mean_prediction": 0.528432999443439, + "observed_frequency": 0.4444444444444444 + }, + { + "n": 61, + "mean_prediction": 0.6900385181760702, + "observed_frequency": 0.6229508196721312 + }, + { + "n": 53, + "mean_prediction": 0.876520093194308, + "observed_frequency": 0.8113207547169812 + } + ] + }, + "gap_mse": 0.15308437289594812, + "predicted_mean_gap": 0.06950796019078283, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6535087719298246, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.21194578519747873, + "auc": 0.8205128205128206, + "bins": [ + { + "n": 1, + "mean_prediction": 0.31702625485635816, + "observed_frequency": 0.0 + }, + { + "n": 7, + "mean_prediction": 0.5170020859114358, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 6, + "mean_prediction": 0.650597617214943, + "observed_frequency": 0.5 + }, + { + "n": 11, + "mean_prediction": 0.8600026540064601, + "observed_frequency": 0.8181818181818182 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23582117078528128, + "auc": 0.8246753246753247, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5398896593836647, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.6971397131389456, + "observed_frequency": 0.4166666666666667 + }, + { + "n": 11, + "mean_prediction": 0.8667288110681942, + "observed_frequency": 0.8181818181818182 + } + ] + }, + "gap_mse": 0.20285508657256632, + "predicted_mean_gap": 0.06719268178350733, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6590909090909092 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.17696213686411177, + "auc": 0.6633771929824561, + "bins": [ + { + "n": 2, + "mean_prediction": 0.48992704352365574, + "observed_frequency": 1.0 + }, + { + "n": 7, + "mean_prediction": 0.7072015801321168, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 41, + "mean_prediction": 0.8797815009885797, + "observed_frequency": 0.8536585365853658 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.19874376048749023, + "auc": 0.6361746361746362, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5919496252045799, + "observed_frequency": 1.0 + }, + { + "n": 8, + "mean_prediction": 0.7558572229972363, + "observed_frequency": 0.375 + }, + { + "n": 41, + "mean_prediction": 0.8804667827201775, + "observed_frequency": 0.8048780487804879 + } + ] + }, + "gap_mse": 0.10028427296896268, + "predicted_mean_gap": 0.014732776244116974, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.7447916666666666 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.21914075548246842, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 1, + "mean_prediction": 0.39505570458862127, + "observed_frequency": 0.0 + }, + { + "n": 19, + "mean_prediction": 0.5231216435409408, + "observed_frequency": 0.5789473684210527 + }, + { + "n": 5, + "mean_prediction": 0.642393779838937, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19146512742697916, + "auc": 0.753968253968254, + "bins": [ + { + "n": 2, + "mean_prediction": 0.570131757785125, + "observed_frequency": 0.5 + }, + { + "n": 23, + "mean_prediction": 0.6728242556554616, + "observed_frequency": 0.7391304347826086 + } + ] + }, + "gap_mse": 0.07526325810684778, + "predicted_mean_gap": 0.12275542258338723, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.5652173913043479 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2367085456803643, + "auc": 0.625, + "bins": [ + { + "n": 2, + "mean_prediction": 0.12415807062382743, + "observed_frequency": 0.5 + }, + { + "n": 4, + "mean_prediction": 0.2969603348837531, + "observed_frequency": 0.5 + }, + { + "n": 14, + "mean_prediction": 0.5292902277543512, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 5, + "mean_prediction": 0.7016493659386438, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2261338053004462, + "auc": 0.5277777777777778, + "bins": [ + { + "n": 2, + "mean_prediction": 0.2932246265086425, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.4859761338621977, + "observed_frequency": 0.5 + }, + { + "n": 18, + "mean_prediction": 0.6780476326121916, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 1, + "mean_prediction": 0.8224099260209196, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.2867349738624011, + "predicted_mean_gap": 0.12812614409878564, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5119047619047619 + } + } + }, + "cap_original_review_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.1914107556813511, + "auc": 0.6878629500580721, + "bins": [ + { + "n": 7, + "mean_prediction": 0.32213130893111247, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5420956711656675, + "observed_frequency": 0.5 + }, + { + "n": 55, + "mean_prediction": 0.7377653233839958, + "observed_frequency": 0.6727272727272727 + }, + { + "n": 55, + "mean_prediction": 0.8377278940115311, + "observed_frequency": 0.7818181818181819 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2038560003807696, + "auc": 0.647005444646098, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38350868842862773, + "observed_frequency": 0.0 + }, + { + "n": 7, + "mean_prediction": 0.5451755825208928, + "observed_frequency": 0.2857142857142857 + }, + { + "n": 34, + "mean_prediction": 0.7359347238019713, + "observed_frequency": 0.5882352941176471 + }, + { + "n": 83, + "mean_prediction": 0.8459653650194651, + "observed_frequency": 0.7831325301204819 + } + ] + }, + "gap_mse": 0.1481425974333536, + "predicted_mean_gap": 0.04954265746688325, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7615629984051037, + "sol_only_rescue": { + "n": 125, + "brier": 0.07799011548857751, + "auc": 0.7248803827751196, + "bins": [ + { + "n": 119, + "mean_prediction": 0.07480052526589055, + "observed_frequency": 0.08403361344537816 + }, + { + "n": 6, + "mean_prediction": 0.24507485827055805, + "observed_frequency": 0.16666666666666666 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.12957033814377258, + "auc": 0.9423076923076923, + "bins": [ + { + "n": 7, + "mean_prediction": 0.32213130893111247, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5220924689126838, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.7162147302860672, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 10, + "mean_prediction": 0.8146838920339935, + "observed_frequency": 0.9 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19524390639329517, + "auc": 0.8961038961038961, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38350868842862773, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.5408249033452184, + "observed_frequency": 0.16666666666666666 + }, + { + "n": 5, + "mean_prediction": 0.6779739491315399, + "observed_frequency": 0.2 + }, + { + "n": 13, + "mean_prediction": 0.8314592152124622, + "observed_frequency": 0.9230769230769231 + } + ] + }, + "gap_mse": 0.18975451296155385, + "predicted_mean_gap": 0.08336264998080616, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6212121212121211, + "sol_only_rescue": { + "n": 25, + "brier": 0.10591855099378641, + "auc": 0.606060606060606, + "bins": [ + { + "n": 19, + "mean_prediction": 0.09522965045841934, + "observed_frequency": 0.10526315789473684 + }, + { + "n": 6, + "mean_prediction": 0.24507485827055805, + "observed_frequency": 0.16666666666666666 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.181247091176039, + "auc": 0.6030701754385965, + "bins": [ + { + "n": 10, + "mean_prediction": 0.7679947868944556, + "observed_frequency": 0.6 + }, + { + "n": 40, + "mean_prediction": 0.8463055694722218, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20037523275837466, + "auc": 0.5966735966735968, + "bins": [ + { + "n": 5, + "mean_prediction": 0.7786485718232278, + "observed_frequency": 0.6 + }, + { + "n": 45, + "mean_prediction": 0.8590560269296759, + "observed_frequency": 0.7555555555555555 + } + ] + }, + "gap_mse": 0.10152808797950025, + "predicted_mean_gap": 0.020371868462362528, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9791666666666667, + "sol_only_rescue": { + "n": 50, + "brier": 0.03698189800261555, + "auc": 0.8541666666666666, + "bins": [ + { + "n": 50, + "mean_prediction": 0.05711060801812077, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.264293879624284, + "auc": 0.36111111111111116, + "bins": [ + { + "n": 1, + "mean_prediction": 0.46929269429144527, + "observed_frequency": 1.0 + }, + { + "n": 19, + "mean_prediction": 0.7579112042425926, + "observed_frequency": 0.6842105263157895 + }, + { + "n": 5, + "mean_prediction": 0.8151944942810779, + "observed_frequency": 0.4 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22245102108155135, + "auc": 0.3650793650793651, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5712796575749394, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.7561514741196076, + "observed_frequency": 0.5 + }, + { + "n": 20, + "mean_prediction": 0.8306263911696631, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.07197077496742654, + "predicted_mean_gap": 0.050513413245621426, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.8260869565217391, + "sol_only_rescue": { + "n": 25, + "brier": 0.07156211035442267, + "auc": 0.7391304347826086, + "bins": [ + { + "n": 25, + "mean_prediction": 0.07386940067886485, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.20069537828662096, + "auc": 0.6985294117647058, + "bins": [ + { + "n": 5, + "mean_prediction": 0.5646575474417054, + "observed_frequency": 0.6 + }, + { + "n": 20, + "mean_prediction": 0.7099771827424776, + "observed_frequency": 0.7 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20083460891225208, + "auc": 0.5634920634920635, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7357031054007379, + "observed_frequency": 0.7 + }, + { + "n": 5, + "mean_prediction": 0.8272212927249818, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.2759315232787871, + "predicted_mean_gap": 0.0730934871832636, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6071428571428571, + "sol_only_rescue": { + "n": 25, + "brier": 0.13850612008944743, + "auc": 0.5238095238095237, + "bins": [ + { + "n": 25, + "mean_prediction": 0.09558534920213385, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "cap_empirical_review_paired_C0.1": { + "efficient": { + "n": 125, + "brier": 0.1884210186201437, + "auc": 0.7110917537746807, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3016674163230186, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5483539094554382, + "observed_frequency": 0.375 + }, + { + "n": 55, + "mean_prediction": 0.7388728916157172, + "observed_frequency": 0.6727272727272727 + }, + { + "n": 55, + "mean_prediction": 0.8411380031183535, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.20574455245647424, + "auc": 0.6013309134906231, + "bins": [ + { + "n": 8, + "mean_prediction": 0.49402970509838184, + "observed_frequency": 0.25 + }, + { + "n": 34, + "mean_prediction": 0.7461213580731263, + "observed_frequency": 0.6176470588235294 + }, + { + "n": 83, + "mean_prediction": 0.8467219090870934, + "observed_frequency": 0.7710843373493976 + } + ] + }, + "gap_mse": 0.1479538572232481, + "predicted_mean_gap": 0.04959343895378859, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7687400318979267, + "sol_only_rescue": { + "n": 125, + "brier": 0.0779799699311674, + "auc": 0.7256778309409888, + "bins": [ + { + "n": 118, + "mean_prediction": 0.07402949252697953, + "observed_frequency": 0.0847457627118644 + }, + { + "n": 7, + "mean_prediction": 0.2280473214835302, + "observed_frequency": 0.14285714285714285 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.13420378327392754, + "auc": 0.9935897435897436, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3016674163230186, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5844449187014562, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.7569869779976703, + "observed_frequency": 0.4 + }, + { + "n": 11, + "mean_prediction": 0.8356245061354722, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19903158584621417, + "auc": 0.8636363636363636, + "bins": [ + { + "n": 7, + "mean_prediction": 0.48630229151021614, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 4, + "mean_prediction": 0.7254938389078004, + "observed_frequency": 0.25 + }, + { + "n": 14, + "mean_prediction": 0.8456425485269531, + "observed_frequency": 0.8571428571428571 + } + ] + }, + "gap_mse": 0.1931233530095146, + "predicted_mean_gap": 0.07550883465749872, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6212121212121211, + "sol_only_rescue": { + "n": 25, + "brier": 0.1082483777519406, + "auc": 0.6212121212121211, + "bins": [ + { + "n": 18, + "mean_prediction": 0.07825027382251246, + "observed_frequency": 0.1111111111111111 + }, + { + "n": 7, + "mean_prediction": 0.2280473214835302, + "observed_frequency": 0.14285714285714285 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1772992624031761, + "auc": 0.5328947368421053, + "bins": [ + { + "n": 12, + "mean_prediction": 0.7500658239208748, + "observed_frequency": 0.5833333333333334 + }, + { + "n": 38, + "mean_prediction": 0.8467148452242442, + "observed_frequency": 0.8157894736842105 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20304421268401235, + "auc": 0.4303534303534303, + "bins": [ + { + "n": 6, + "mean_prediction": 0.7627406484622891, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 44, + "mean_prediction": 0.8584286521195373, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.09983372322843556, + "predicted_mean_gap": 0.023427011569231916, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9895833333333333, + "sol_only_rescue": { + "n": 50, + "brier": 0.03569536629394268, + "auc": 0.9791666666666667, + "bins": [ + { + "n": 50, + "mean_prediction": 0.061181062669169686, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.26710729772582764, + "auc": 0.3125, + "bins": [ + { + "n": 1, + "mean_prediction": 0.439851863525283, + "observed_frequency": 1.0 + }, + { + "n": 18, + "mean_prediction": 0.7649098051223528, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 6, + "mean_prediction": 0.8159260809163279, + "observed_frequency": 0.3333333333333333 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22251407086765682, + "auc": 0.3492063492063492, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5481216002155417, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.7711520757599395, + "observed_frequency": 0.5 + }, + { + "n": 20, + "mean_prediction": 0.8284150887312146, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.07541550609767562, + "predicted_mean_gap": 0.043889873466159655, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.5, + "sol_only_rescue": { + "n": 25, + "brier": 0.07507170965033697, + "auc": 0.4347826086956522, + "bins": [ + { + "n": 25, + "mean_prediction": 0.06587682979248362, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.18619548729461116, + "auc": 0.8235294117647058, + "bins": [ + { + "n": 5, + "mean_prediction": 0.5556179149430621, + "observed_frequency": 0.4 + }, + { + "n": 20, + "mean_prediction": 0.7041953884811629, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20108868020047574, + "auc": 0.5555555555555556, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7402549312520803, + "observed_frequency": 0.7 + }, + { + "n": 5, + "mean_prediction": 0.8199520613934961, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.27156298055217926, + "predicted_mean_gap": 0.08171446350682078, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.619047619047619, + "sol_only_rescue": { + "n": 25, + "brier": 0.13518902966567417, + "auc": 0.6071428571428571, + "bins": [ + { + "n": 25, + "mean_prediction": 0.10484005244431145, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_original_upfront_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.20955101845143284, + "auc": 0.6591173054587689, + "bins": [ + { + "n": 16, + "mean_prediction": 0.5667534503435131, + "observed_frequency": 0.4375 + }, + { + "n": 54, + "mean_prediction": 0.6789223136887862, + "observed_frequency": 0.6481481481481481 + }, + { + "n": 55, + "mean_prediction": 0.853313330691114, + "observed_frequency": 0.7636363636363637 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2120795262896164, + "auc": 0.6321839080459771, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5932803491724423, + "observed_frequency": 0.0 + }, + { + "n": 53, + "mean_prediction": 0.7131101509259069, + "observed_frequency": 0.6226415094339622 + }, + { + "n": 70, + "mean_prediction": 0.8599280022004958, + "observed_frequency": 0.7714285714285715 + } + ] + }, + "gap_mse": 0.1518496395206646, + "predicted_mean_gap": 0.052114124150005804, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6339712918660287, + "sol_only_rescue": { + "n": 125, + "brier": 0.07999504149700941, + "auc": 0.6228070175438597, + "bins": [ + { + "n": 125, + "mean_prediction": 0.08489086207563169, + "observed_frequency": 0.088 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.23665019595851228, + "auc": 0.782051282051282, + "bins": [ + { + "n": 9, + "mean_prediction": 0.5484291815802487, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 12, + "mean_prediction": 0.7395154157522984, + "observed_frequency": 0.5833333333333334 + }, + { + "n": 4, + "mean_prediction": 0.8570040691886698, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23604831208369462, + "auc": 0.8506493506493508, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5932803491724423, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.6964671564970546, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 11, + "mean_prediction": 0.8374086915518695, + "observed_frequency": 0.9090909090909091 + } + ] + }, + "gap_mse": 0.20840532574641726, + "predicted_mean_gap": 0.060703931335024296, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.48484848484848486, + "sol_only_rescue": { + "n": 25, + "brier": 0.1119211071362905, + "auc": 0.5, + "bins": [ + { + "n": 25, + "mean_prediction": 0.10122056872175196, + "observed_frequency": 0.12 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19295227968276996, + "auc": 0.5877192982456141, + "bins": [ + { + "n": 1, + "mean_prediction": 0.6813398775411287, + "observed_frequency": 1.0 + }, + { + "n": 49, + "mean_prediction": 0.8531639371020696, + "observed_frequency": 0.7551020408163265 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.21156910489292113, + "auc": 0.5446985446985447, + "bins": [ + { + "n": 1, + "mean_prediction": 0.735504011160592, + "observed_frequency": 1.0 + }, + { + "n": 49, + "mean_prediction": 0.8731227105981716, + "observed_frequency": 0.7346938775510204 + } + ] + }, + "gap_mse": 0.10130703391819111, + "predicted_mean_gap": 0.020642880698569174, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.8229166666666667, + "sol_only_rescue": { + "n": 50, + "brier": 0.03845277659040781, + "auc": 0.8229166666666667, + "bins": [ + { + "n": 50, + "mean_prediction": 0.046373446628495525, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.20882354664311553, + "auc": 0.8402777777777778, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5941141668655936, + "observed_frequency": 0.0 + }, + { + "n": 24, + "mean_prediction": 0.6667613159417245, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19101930004530196, + "auc": 0.8492063492063493, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7274520936924024, + "observed_frequency": 0.65 + }, + { + "n": 5, + "mean_prediction": 0.810470069447004, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.07101647541319869, + "predicted_mean_gap": 0.0802002588646435, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6956521739130435, + "sol_only_rescue": { + "n": 25, + "brier": 0.0721550946043794, + "auc": 0.7173913043478262, + "bins": [ + { + "n": 25, + "mean_prediction": 0.11549946293873621, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21637679028999643, + "auc": 0.4779411764705882, + "bins": [ + { + "n": 6, + "mean_prediction": 0.5896797340680631, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 17, + "mean_prediction": 0.6531769700014324, + "observed_frequency": 0.6470588235294118 + }, + { + "n": 2, + "mean_prediction": 0.8495919966275949, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21019180953324298, + "auc": 0.3412698412698413, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7076343118049885, + "observed_frequency": 0.75 + }, + { + "n": 5, + "mean_prediction": 0.8296202760837431, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.27721232860732486, + "predicted_mean_gap": 0.07838066915322284, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.40476190476190477, + "sol_only_rescue": { + "n": 25, + "brier": 0.13899345256356152, + "auc": 0.36904761904761907, + "bins": [ + { + "n": 25, + "mean_prediction": 0.11498738546067915, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_original_upfront_scalar_calibration_C0.1": { + "efficient": { + "n": 125, + "brier": 0.20454289048147975, + "auc": 0.6702961672473867, + "bins": [ + { + "n": 2, + "mean_prediction": 0.11733927578309869, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.3676547804526239, + "observed_frequency": 0.25 + }, + { + "n": 42, + "mean_prediction": 0.5306628758263982, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 25, + "mean_prediction": 0.6766111515699391, + "observed_frequency": 0.76 + }, + { + "n": 52, + "mean_prediction": 0.8801589054214638, + "observed_frequency": 0.7692307692307693 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2108054170799802, + "auc": 0.617816091954023, + "bins": [ + { + "n": 2, + "mean_prediction": 0.2943642691374838, + "observed_frequency": 0.5 + }, + { + "n": 9, + "mean_prediction": 0.5533227554610396, + "observed_frequency": 0.4444444444444444 + }, + { + "n": 60, + "mean_prediction": 0.6830970782840848, + "observed_frequency": 0.6833333333333333 + }, + { + "n": 54, + "mean_prediction": 0.8769448478815192, + "observed_frequency": 0.7592592592592593 + } + ] + }, + "gap_mse": 0.15120245748014532, + "predicted_mean_gap": 0.0578623959265715, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6885964912280702, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.22091798281245, + "auc": 0.7532051282051282, + "bins": [ + { + "n": 2, + "mean_prediction": 0.35519679173027074, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5162717553668691, + "observed_frequency": 0.375 + }, + { + "n": 10, + "mean_prediction": 0.6957773174493485, + "observed_frequency": 0.6 + }, + { + "n": 5, + "mean_prediction": 0.8707386746370119, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2246470342419736, + "auc": 0.8149350649350648, + "bins": [ + { + "n": 4, + "mean_prediction": 0.5541291747850261, + "observed_frequency": 0.0 + }, + { + "n": 15, + "mean_prediction": 0.6992917836050431, + "observed_frequency": 0.6 + }, + { + "n": 6, + "mean_prediction": 0.8467414195646222, + "observed_frequency": 0.8333333333333334 + } + ] + }, + "gap_mse": 0.20532577111426073, + "predicted_mean_gap": 0.06537231186117777, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6060606060606061 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1898629366301952, + "auc": 0.6622807017543859, + "bins": [ + { + "n": 1, + "mean_prediction": 0.596804463889429, + "observed_frequency": 1.0 + }, + { + "n": 3, + "mean_prediction": 0.6947712497458647, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 46, + "mean_prediction": 0.8823113755502815, + "observed_frequency": 0.7608695652173914 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20786062055512197, + "auc": 0.6330561330561331, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7097151707629702, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.8816373441177446, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "gap_mse": 0.10079426640602042, + "predicted_mean_gap": 0.005973183947658731, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.7083333333333334 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.20982837369936858, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 1, + "mean_prediction": 0.3801127691749771, + "observed_frequency": 0.0 + }, + { + "n": 17, + "mean_prediction": 0.5597635631691373, + "observed_frequency": 0.5882352941176471 + }, + { + "n": 7, + "mean_prediction": 0.6456432545484561, + "observed_frequency": 0.8571428571428571 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18885350649032856, + "auc": 0.8174603174603174, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5496063311842023, + "observed_frequency": 0.5 + }, + { + "n": 23, + "mean_prediction": 0.6848065109012914, + "observed_frequency": 0.7391304347826086 + } + ] + }, + "gap_mse": 0.07488918328015506, + "predicted_mean_gap": 0.09736665152834414, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.4782608695652174 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21224222263518988, + "auc": 0.6544117647058824, + "bins": [ + { + "n": 2, + "mean_prediction": 0.11733927578309869, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.3801127691749771, + "observed_frequency": 1.0 + }, + { + "n": 16, + "mean_prediction": 0.5028051065005626, + "observed_frequency": 0.625 + }, + { + "n": 5, + "mean_prediction": 0.6707378167356419, + "observed_frequency": 1.0 + }, + { + "n": 1, + "mean_prediction": 0.8282464334181043, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22480530355735504, + "auc": 0.36904761904761907, + "bins": [ + { + "n": 2, + "mean_prediction": 0.2943642691374838, + "observed_frequency": 0.5 + }, + { + "n": 3, + "mean_prediction": 0.5547251458802822, + "observed_frequency": 1.0 + }, + { + "n": 19, + "mean_prediction": 0.6640396147342544, + "observed_frequency": 0.6842105263157895 + }, + { + "n": 1, + "mean_prediction": 0.8376180946802971, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.27420880019426996, + "predicted_mean_gap": 0.11462664834801807, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6726190476190477 + } + } + }, + "v2_original_review_per_model_C1.0": { + "efficient": { + "n": 125, + "brier": 0.19932787533774815, + "auc": 0.6492450638792102, + "bins": [ + { + "n": 8, + "mean_prediction": 0.09808319476498092, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.28536439122575186, + "observed_frequency": 0.4 + }, + { + "n": 8, + "mean_prediction": 0.48886282725063324, + "observed_frequency": 0.75 + }, + { + "n": 34, + "mean_prediction": 0.733993887146487, + "observed_frequency": 0.6764705882352942 + }, + { + "n": 70, + "mean_prediction": 0.8715403372888845, + "observed_frequency": 0.7571428571428571 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.20931594252899682, + "auc": 0.6091954022988506, + "bins": [ + { + "n": 6, + "mean_prediction": 0.3403154042414572, + "observed_frequency": 0.16666666666666666 + }, + { + "n": 2, + "mean_prediction": 0.5074102884175007, + "observed_frequency": 0.5 + }, + { + "n": 23, + "mean_prediction": 0.7316869758668929, + "observed_frequency": 0.6521739130434783 + }, + { + "n": 94, + "mean_prediction": 0.867781103974221, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "gap_mse": 0.16328838238192742, + "predicted_mean_gap": 0.07496745052274353, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6443381180223285, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.05652455210980535, + "auc": 0.9743589743589743, + "bins": [ + { + "n": 8, + "mean_prediction": 0.09808319476498092, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.25617641805972485, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.41798011212615066, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.7562765355320514, + "observed_frequency": 1.0 + }, + { + "n": 13, + "mean_prediction": 0.8763633997087588, + "observed_frequency": 0.9230769230769231 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18002485102375843, + "auc": 0.8701298701298701, + "bins": [ + { + "n": 6, + "mean_prediction": 0.3403154042414572, + "observed_frequency": 0.16666666666666666 + }, + { + "n": 2, + "mean_prediction": 0.5074102884175007, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.7353023838701571, + "observed_frequency": 0.2 + }, + { + "n": 12, + "mean_prediction": 0.874839672100801, + "observed_frequency": 0.9166666666666666 + } + ] + }, + "gap_mse": 0.20595801309115422, + "predicted_mean_gap": 0.13469206994931118, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6515151515151515 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19091964059448877, + "auc": 0.5021929824561403, + "bins": [ + { + "n": 2, + "mean_prediction": 0.29503917005045877, + "observed_frequency": 0.5 + }, + { + "n": 5, + "mean_prediction": 0.7805277780914377, + "observed_frequency": 0.8 + }, + { + "n": 43, + "mean_prediction": 0.8735118156150351, + "observed_frequency": 0.7674418604651163 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.21240803577351028, + "auc": 0.40540540540540543, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7392149518637621, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.884324929859315, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "gap_mse": 0.10885331642033774, + "predicted_mean_gap": 0.03454382513948935, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9791666666666666 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.27579565498158387, + "auc": 0.3958333333333333, + "bins": [ + { + "n": 3, + "mean_prediction": 0.5323266144146128, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 11, + "mean_prediction": 0.7354168760459815, + "observed_frequency": 0.7272727272727273 + }, + { + "n": 11, + "mean_prediction": 0.8569469095234605, + "observed_frequency": 0.5454545454545454 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21960534537584622, + "auc": 0.39682539682539686, + "bins": [ + { + "n": 5, + "mean_prediction": 0.7302710591423467, + "observed_frequency": 0.6 + }, + { + "n": 20, + "mean_prediction": 0.846885721543498, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.08086696652085992, + "predicted_mean_gap": 0.05904352968295964, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.5652173913043479 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.28247988840837396, + "auc": 0.3014705882352941, + "bins": [ + { + "n": 1, + "mean_prediction": 0.32439077990839177, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.4739856656587692, + "observed_frequency": 1.0 + }, + { + "n": 17, + "mean_prediction": 0.7180759470873835, + "observed_frequency": 0.5882352941176471 + }, + { + "n": 3, + "mean_prediction": 0.8758917792678234, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22213344469835924, + "auc": 0.3650793650793651, + "bins": [ + { + "n": 10, + "mean_prediction": 0.7283288374284733, + "observed_frequency": 0.9 + }, + { + "n": 15, + "mean_prediction": 0.8381574382739623, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.31191029945694765, + "predicted_mean_gap": 0.11201400270246818, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.1547619047619048 + } + } + }, + "v2_empirical_review_per_model_C0.1": { + "efficient": { + "n": 125, + "brier": 0.19010512226877627, + "auc": 0.7047038327526133, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3334240172369359, + "observed_frequency": 0.0 + }, + { + "n": 9, + "mean_prediction": 0.5417594715537408, + "observed_frequency": 0.4444444444444444 + }, + { + "n": 60, + "mean_prediction": 0.7254281408821367, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 49, + "mean_prediction": 0.8410865458465858, + "observed_frequency": 0.8163265306122449 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.20735573624589884, + "auc": 0.6007259528130672, + "bins": [ + { + "n": 7, + "mean_prediction": 0.5373237575440344, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 33, + "mean_prediction": 0.7638175292576284, + "observed_frequency": 0.6363636363636364 + }, + { + "n": 85, + "mean_prediction": 0.8353217086986139, + "observed_frequency": 0.7647058823529411 + } + ] + }, + "gap_mse": 0.14802323408721804, + "predicted_mean_gap": 0.06416685954911215, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7416267942583731, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.13994229066694372, + "auc": 1.0, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3334240172369359, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.5696121158980051, + "observed_frequency": 0.0 + }, + { + "n": 14, + "mean_prediction": 0.7574108461100196, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 3, + "mean_prediction": 0.8465217062183243, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21306367488670708, + "auc": 0.8701298701298701, + "bins": [ + { + "n": 7, + "mean_prediction": 0.5373237575440344, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 10, + "mean_prediction": 0.7609076761326313, + "observed_frequency": 0.5 + }, + { + "n": 8, + "mean_prediction": 0.8234084087310922, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.19308006491920632, + "predicted_mean_gap": 0.07642852532925948, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.18284109550990832, + "auc": 0.5021929824561404, + "bins": [ + { + "n": 4, + "mean_prediction": 0.7287123247106313, + "observed_frequency": 0.25 + }, + { + "n": 46, + "mean_prediction": 0.8407320788658201, + "observed_frequency": 0.8043478260869565 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.2031037287625894, + "auc": 0.41372141372141374, + "bins": [ + { + "n": 2, + "mean_prediction": 0.789903402752905, + "observed_frequency": 0.5 + }, + { + "n": 48, + "mean_prediction": 0.8484833906880502, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.10064951069561175, + "predicted_mean_gap": 0.014369692637239322, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9791666666666666 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.2466896387796678, + "auc": 0.5555555555555556, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5572098889724613, + "observed_frequency": 1.0 + }, + { + "n": 23, + "mean_prediction": 0.7432932681777755, + "observed_frequency": 0.6086956521739131 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2167319530762399, + "auc": 0.26190476190476186, + "bins": [ + { + "n": 7, + "mean_prediction": 0.7745524256235462, + "observed_frequency": 0.8571428571428571 + }, + { + "n": 18, + "mean_prediction": 0.8161483637753537, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "gap_mse": 0.07097323173566336, + "predicted_mean_gap": 0.07609490325149727, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.8695652173913043 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.1982114908774532, + "auc": 0.7058823529411764, + "bins": [ + { + "n": 6, + "mean_prediction": 0.5319672250234567, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 19, + "mean_prediction": 0.6795443231819248, + "observed_frequency": 0.7894736842105263 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2007755957413684, + "auc": 0.626984126984127, + "bins": [ + { + "n": 14, + "mean_prediction": 0.7568019942360563, + "observed_frequency": 0.6428571428571429 + }, + { + "n": 11, + "mean_prediction": 0.8179276971409692, + "observed_frequency": 0.8181818181818182 + } + ] + }, + "gap_mse": 0.274763852389997, + "predicted_mean_gap": 0.13957148389032537, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.7261904761904762 + } + } + }, + "cap_empirical_review_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.1884210186201437, + "auc": 0.7110917537746807, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3016674163230186, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5483539094554382, + "observed_frequency": 0.375 + }, + { + "n": 55, + "mean_prediction": 0.7388728916157172, + "observed_frequency": 0.6727272727272727 + }, + { + "n": 55, + "mean_prediction": 0.8411380031183535, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.20574455245647424, + "auc": 0.6013309134906231, + "bins": [ + { + "n": 8, + "mean_prediction": 0.49402970509838184, + "observed_frequency": 0.25 + }, + { + "n": 34, + "mean_prediction": 0.7461213580731263, + "observed_frequency": 0.6176470588235294 + }, + { + "n": 83, + "mean_prediction": 0.8467219090870934, + "observed_frequency": 0.7710843373493976 + } + ] + }, + "gap_mse": 0.1479538572232481, + "predicted_mean_gap": 0.04959343895378859, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7687400318979267, + "sol_only_rescue": { + "n": 125, + "brier": 0.0779799699311674, + "auc": 0.7256778309409888, + "bins": [ + { + "n": 118, + "mean_prediction": 0.07402949252697953, + "observed_frequency": 0.0847457627118644 + }, + { + "n": 7, + "mean_prediction": 0.2280473214835302, + "observed_frequency": 0.14285714285714285 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.13420378327392754, + "auc": 0.9935897435897436, + "bins": [ + { + "n": 7, + "mean_prediction": 0.3016674163230186, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5844449187014562, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.7569869779976703, + "observed_frequency": 0.4 + }, + { + "n": 11, + "mean_prediction": 0.8356245061354722, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19903158584621417, + "auc": 0.8636363636363636, + "bins": [ + { + "n": 7, + "mean_prediction": 0.48630229151021614, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 4, + "mean_prediction": 0.7254938389078004, + "observed_frequency": 0.25 + }, + { + "n": 14, + "mean_prediction": 0.8456425485269531, + "observed_frequency": 0.8571428571428571 + } + ] + }, + "gap_mse": 0.1931233530095146, + "predicted_mean_gap": 0.07550883465749872, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6212121212121211, + "sol_only_rescue": { + "n": 25, + "brier": 0.1082483777519406, + "auc": 0.6212121212121211, + "bins": [ + { + "n": 18, + "mean_prediction": 0.07825027382251246, + "observed_frequency": 0.1111111111111111 + }, + { + "n": 7, + "mean_prediction": 0.2280473214835302, + "observed_frequency": 0.14285714285714285 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1772992624031761, + "auc": 0.5328947368421053, + "bins": [ + { + "n": 12, + "mean_prediction": 0.7500658239208748, + "observed_frequency": 0.5833333333333334 + }, + { + "n": 38, + "mean_prediction": 0.8467148452242442, + "observed_frequency": 0.8157894736842105 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20304421268401235, + "auc": 0.4303534303534303, + "bins": [ + { + "n": 6, + "mean_prediction": 0.7627406484622891, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 44, + "mean_prediction": 0.8584286521195373, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.09983372322843556, + "predicted_mean_gap": 0.023427011569231916, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9895833333333333, + "sol_only_rescue": { + "n": 50, + "brier": 0.03569536629394268, + "auc": 0.9791666666666667, + "bins": [ + { + "n": 50, + "mean_prediction": 0.061181062669169686, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.26710729772582764, + "auc": 0.3125, + "bins": [ + { + "n": 1, + "mean_prediction": 0.439851863525283, + "observed_frequency": 1.0 + }, + { + "n": 18, + "mean_prediction": 0.7649098051223528, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 6, + "mean_prediction": 0.8159260809163279, + "observed_frequency": 0.3333333333333333 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22251407086765682, + "auc": 0.3492063492063492, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5481216002155417, + "observed_frequency": 1.0 + }, + { + "n": 4, + "mean_prediction": 0.7711520757599395, + "observed_frequency": 0.5 + }, + { + "n": 20, + "mean_prediction": 0.8284150887312146, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.07541550609767562, + "predicted_mean_gap": 0.043889873466159655, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.5, + "sol_only_rescue": { + "n": 25, + "brier": 0.07507170965033697, + "auc": 0.4347826086956522, + "bins": [ + { + "n": 25, + "mean_prediction": 0.06587682979248362, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.18619548729461116, + "auc": 0.8235294117647058, + "bins": [ + { + "n": 5, + "mean_prediction": 0.5556179149430621, + "observed_frequency": 0.4 + }, + { + "n": 20, + "mean_prediction": 0.7041953884811629, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20108868020047574, + "auc": 0.5555555555555556, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7402549312520803, + "observed_frequency": 0.7 + }, + { + "n": 5, + "mean_prediction": 0.8199520613934961, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.27156298055217926, + "predicted_mean_gap": 0.08171446350682078, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.619047619047619, + "sol_only_rescue": { + "n": 25, + "brier": 0.13518902966567417, + "auc": 0.6071428571428571, + "bins": [ + { + "n": 25, + "mean_prediction": 0.10484005244431145, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "cap_original_upfront_paired_C1.0": { + "efficient": { + "n": 125, + "brier": 0.2162848443236792, + "auc": 0.6445993031358885, + "bins": [ + { + "n": 2, + "mean_prediction": 0.1765799121631378, + "observed_frequency": 0.0 + }, + { + "n": 13, + "mean_prediction": 0.48784999388299466, + "observed_frequency": 0.6153846153846154 + }, + { + "n": 72, + "mean_prediction": 0.7327722309375719, + "observed_frequency": 0.6527777777777778 + }, + { + "n": 38, + "mean_prediction": 0.8700950590095132, + "observed_frequency": 0.7631578947368421 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21684750902033487, + "auc": 0.5486993345432547, + "bins": [ + { + "n": 1, + "mean_prediction": 0.2814455913620738, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.5368912598347659, + "observed_frequency": 0.5 + }, + { + "n": 51, + "mean_prediction": 0.7404760777754543, + "observed_frequency": 0.6862745098039216 + }, + { + "n": 67, + "mean_prediction": 0.8591020916406562, + "observed_frequency": 0.7313432835820896 + } + ] + }, + "gap_mse": 0.15175632352505936, + "predicted_mean_gap": 0.0504679251373675, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6459330143540671, + "sol_only_rescue": { + "n": 125, + "brier": 0.08154577784631287, + "auc": 0.616427432216906, + "bins": [ + { + "n": 116, + "mean_prediction": 0.06741066524163526, + "observed_frequency": 0.08620689655172414 + }, + { + "n": 9, + "mean_prediction": 0.2884753665190133, + "observed_frequency": 0.1111111111111111 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.253659251872235, + "auc": 0.6858974358974358, + "bins": [ + { + "n": 2, + "mean_prediction": 0.1765799121631378, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.4682494528044025, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 16, + "mean_prediction": 0.7354111933302958, + "observed_frequency": 0.5 + }, + { + "n": 1, + "mean_prediction": 0.8773105057960244, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.24908676767101326, + "auc": 0.6103896103896105, + "bins": [ + { + "n": 1, + "mean_prediction": 0.2814455913620738, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.5178774123086126, + "observed_frequency": 0.25 + }, + { + "n": 10, + "mean_prediction": 0.7417186180949691, + "observed_frequency": 0.6 + }, + { + "n": 10, + "mean_prediction": 0.8319429948483291, + "observed_frequency": 0.7 + } + ] + }, + "gap_mse": 0.20363943995631115, + "predicted_mean_gap": 0.09132100919184234, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5303030303030303, + "sol_only_rescue": { + "n": 25, + "brier": 0.11789359780682673, + "auc": 0.5454545454545455, + "bins": [ + { + "n": 20, + "mean_prediction": 0.10607970040615036, + "observed_frequency": 0.15 + }, + { + "n": 5, + "mean_prediction": 0.2855816159271335, + "observed_frequency": 0.0 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19009889910930744, + "auc": 0.5603070175438597, + "bins": [ + { + "n": 19, + "mean_prediction": 0.7379037794053194, + "observed_frequency": 0.7368421052631579 + }, + { + "n": 31, + "mean_prediction": 0.8762443664853552, + "observed_frequency": 0.7741935483870968 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20380262054017337, + "auc": 0.5353430353430353, + "bins": [ + { + "n": 7, + "mean_prediction": 0.7107600299026597, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 43, + "mean_prediction": 0.8724268734854672, + "observed_frequency": 0.7441860465116279 + } + ] + }, + "gap_mse": 0.10508667079941304, + "predicted_mean_gap": 0.026118571988932623, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.5625, + "sol_only_rescue": { + "n": 50, + "brier": 0.040928686216825626, + "auc": 0.46875, + "bins": [ + { + "n": 50, + "mean_prediction": 0.05367778464962154, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.23708523366794856, + "auc": 0.6736111111111112, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5866178031538105, + "observed_frequency": 0.5 + }, + { + "n": 17, + "mean_prediction": 0.7264681255964354, + "observed_frequency": 0.6470588235294118 + }, + { + "n": 6, + "mean_prediction": 0.8371210625865765, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20270167191546734, + "auc": 0.5634920634920635, + "bins": [ + { + "n": 16, + "mean_prediction": 0.7384115132483009, + "observed_frequency": 0.6875 + }, + { + "n": 9, + "mean_prediction": 0.8294199795518544, + "observed_frequency": 0.7777777777777778 + } + ] + }, + "gap_mse": 0.07052358338676709, + "predicted_mean_gap": 0.02933775643892083, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.9347826086956521, + "sol_only_rescue": { + "n": 25, + "brier": 0.06743215967349744, + "auc": 0.9347826086956521, + "bins": [ + { + "n": 25, + "mean_prediction": 0.06216253801825453, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.21048193785959757, + "auc": 0.6176470588235294, + "bins": [ + { + "n": 5, + "mean_prediction": 0.471863519468979, + "observed_frequency": 0.6 + }, + { + "n": 20, + "mean_prediction": 0.7311445795189984, + "observed_frequency": 0.7 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.224843864434847, + "auc": 0.3015873015873016, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5749189548870726, + "observed_frequency": 1.0 + }, + { + "n": 18, + "mean_prediction": 0.7531771869059471, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 5, + "mean_prediction": 0.8522549631197791, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.27444525268339237, + "predicted_mean_gap": 0.07944371607820912, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5238095238095238, + "sol_only_rescue": { + "n": 25, + "brier": 0.14054575931758895, + "auc": 0.5238095238095238, + "bins": [ + { + "n": 21, + "mean_prediction": 0.0695281179509163, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 4, + "mean_prediction": 0.2920925547588631, + "observed_frequency": 0.25 + } + ] + } + } + } + }, + "cap_original_upfront_per_model_C1.0": { + "efficient": { + "n": 125, + "brier": 0.21487591950282664, + "auc": 0.643437862950058, + "bins": [ + { + "n": 1, + "mean_prediction": 0.16121382253171718, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.29434527988269404, + "observed_frequency": 0.0 + }, + { + "n": 13, + "mean_prediction": 0.4991731160013399, + "observed_frequency": 0.6923076923076923 + }, + { + "n": 72, + "mean_prediction": 0.7299767055000576, + "observed_frequency": 0.6388888888888888 + }, + { + "n": 37, + "mean_prediction": 0.8651218558885555, + "observed_frequency": 0.7837837837837838 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21545835040915215, + "auc": 0.5565638233514821, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38229007133429044, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.48540807315296636, + "observed_frequency": 0.0 + }, + { + "n": 62, + "mean_prediction": 0.7449979650518022, + "observed_frequency": 0.6935483870967742 + }, + { + "n": 60, + "mean_prediction": 0.8619209259445029, + "observed_frequency": 0.7333333333333333 + } + ] + }, + "gap_mse": 0.1549157397759275, + "predicted_mean_gap": 0.05960999402661531, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6299840510366825, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.251852547495908, + "auc": 0.6666666666666666, + "bins": [ + { + "n": 1, + "mean_prediction": 0.16121382253171718, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.21639030958729122, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.4830239547920591, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 16, + "mean_prediction": 0.7335550902460288, + "observed_frequency": 0.5 + }, + { + "n": 1, + "mean_prediction": 0.8437907198728529, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2549590803118468, + "auc": 0.5909090909090909, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38229007133429044, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.48540807315296636, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.7217046395253081, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 10, + "mean_prediction": 0.8343312541280412, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.20977700883433306, + "predicted_mean_gap": 0.10001817634174617, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19049473966512714, + "auc": 0.5603070175438597, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5488948385357809, + "observed_frequency": 1.0 + }, + { + "n": 18, + "mean_prediction": 0.7413171701165587, + "observed_frequency": 0.7222222222222222 + }, + { + "n": 31, + "mean_prediction": 0.8684711986080795, + "observed_frequency": 0.7741935483870968 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20024743170144074, + "auc": 0.5374220374220374, + "bins": [ + { + "n": 8, + "mean_prediction": 0.7115201910282071, + "observed_frequency": 0.625 + }, + { + "n": 42, + "mean_prediction": 0.8691817574451719, + "observed_frequency": 0.7619047619047619 + } + ] + }, + "gap_mse": 0.10582504813726028, + "predicted_mean_gap": 0.027651685668771468, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.5729166666666666 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.23444262481708145, + "auc": 0.6805555555555556, + "bins": [ + { + "n": 2, + "mean_prediction": 0.5843762128843636, + "observed_frequency": 0.5 + }, + { + "n": 18, + "mean_prediction": 0.7276852530794122, + "observed_frequency": 0.6111111111111112 + }, + { + "n": 5, + "mean_prediction": 0.8486221582306458, + "observed_frequency": 0.8 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19940750075049063, + "auc": 0.6666666666666666, + "bins": [ + { + "n": 22, + "mean_prediction": 0.7599564854719414, + "observed_frequency": 0.6818181818181818 + }, + { + "n": 3, + "mean_prediction": 0.8330987285102216, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.07079617907245593, + "predicted_mean_gap": 0.02832564374247998, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.9347826086956521 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2070949458708894, + "auc": 0.6764705882352942, + "bins": [ + { + "n": 1, + "mean_prediction": 0.37230025017809687, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.4683648787401388, + "observed_frequency": 0.75 + }, + { + "n": 20, + "mean_prediction": 0.7189698867270101, + "observed_frequency": 0.7 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.22243030758054202, + "auc": 0.23015873015873017, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7559106975149834, + "observed_frequency": 0.75 + }, + { + "n": 5, + "mean_prediction": 0.8734026034323744, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.28235541469832803, + "predicted_mean_gap": 0.11440277871130744, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.48809523809523814 + } + } + }, + "cap_original_upfront_rescue_C1.0": { + "sol_only_rescue": { + "n": 125, + "brier": 0.08098389746060136, + "auc": 0.6116427432216905, + "bins": [ + { + "n": 117, + "mean_prediction": 0.07044439549245314, + "observed_frequency": 0.08547008547008547 + }, + { + "n": 8, + "mean_prediction": 0.24371366434908592, + "observed_frequency": 0.125 + } + ] + } + }, + "v2_empirical_upfront_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.20614602966847304, + "auc": 0.679732868757259, + "bins": [ + { + "n": 14, + "mean_prediction": 0.5592501035333568, + "observed_frequency": 0.35714285714285715 + }, + { + "n": 58, + "mean_prediction": 0.6849112164846393, + "observed_frequency": 0.6724137931034483 + }, + { + "n": 53, + "mean_prediction": 0.8262493609452123, + "observed_frequency": 0.7547169811320755 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21078160725794215, + "auc": 0.6358136721113127, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5469473101052138, + "observed_frequency": 0.0 + }, + { + "n": 68, + "mean_prediction": 0.7315550309308635, + "observed_frequency": 0.6617647058823529 + }, + { + "n": 56, + "mean_prediction": 0.8500830125525453, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.15106011643393696, + "predicted_mean_gap": 0.052414159845393084, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6467304625199362, + "sol_only_rescue": { + "n": 125, + "brier": 0.07915810297684357, + "auc": 0.6539074960127592, + "bins": [ + { + "n": 125, + "mean_prediction": 0.09147465916930164, + "observed_frequency": 0.088 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.23267478328247318, + "auc": 0.8076923076923077, + "bins": [ + { + "n": 7, + "mean_prediction": 0.548700819886402, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 15, + "mean_prediction": 0.6814580100271866, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 3, + "mean_prediction": 0.824160379060679, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.24226287696249313, + "auc": 0.8246753246753247, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5469473101052138, + "observed_frequency": 0.0 + }, + { + "n": 20, + "mean_prediction": 0.7131054223387527, + "observed_frequency": 0.55 + }, + { + "n": 4, + "mean_prediction": 0.8389054225050308, + "observed_frequency": 0.75 + } + ] + }, + "gap_mse": 0.20305957306776407, + "predicted_mean_gap": 0.0651768168042296, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.4545454545454545, + "sol_only_rescue": { + "n": 25, + "brier": 0.1080654158532295, + "auc": 0.4545454545454545, + "bins": [ + { + "n": 25, + "mean_prediction": 0.11380770464791952, + "observed_frequency": 0.12 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.18690428069273377, + "auc": 0.5701754385964912, + "bins": [ + { + "n": 2, + "mean_prediction": 0.7958290172277925, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.8257839939314301, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20471339507554998, + "auc": 0.6133056133056133, + "bins": [ + { + "n": 50, + "mean_prediction": 0.8507768113197909, + "observed_frequency": 0.74 + } + ] + }, + "gap_mse": 0.10223690699459223, + "predicted_mean_gap": 0.026191016456506268, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.53125, + "sol_only_rescue": { + "n": 50, + "brier": 0.039114126318372536, + "auc": 0.5, + "bins": [ + { + "n": 50, + "mean_prediction": 0.05598018946360002, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.21704544995216324, + "auc": 0.7430555555555556, + "bins": [ + { + "n": 25, + "mean_prediction": 0.685798204703094, + "observed_frequency": 0.64 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19692586930075023, + "auc": 0.7222222222222222, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7493935961465694, + "observed_frequency": 0.72 + } + ] + }, + "gap_mse": 0.0727382215111854, + "predicted_mean_gap": 0.0635953914434754, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6521739130434783, + "sol_only_rescue": { + "n": 25, + "brier": 0.07459963628267795, + "auc": 0.6304347826086957, + "bins": [ + { + "n": 25, + "mean_prediction": 0.10674067594073243, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.207201353722261, + "auc": 0.6323529411764706, + "bins": [ + { + "n": 7, + "mean_prediction": 0.5697993871803118, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 16, + "mean_prediction": 0.6728979533542716, + "observed_frequency": 0.6875 + }, + { + "n": 2, + "mean_prediction": 0.8405516421027848, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20529249987536755, + "auc": 0.5079365079365079, + "bins": [ + { + "n": 23, + "mean_prediction": 0.7282084240373664, + "observed_frequency": 0.6956521739130435 + }, + { + "n": 2, + "mean_prediction": 0.8550932234664312, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.275028973601551, + "predicted_mean_gap": 0.08091655806624784, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5595238095238095, + "sol_only_rescue": { + "n": 25, + "brier": 0.13489721011156525, + "auc": 0.5357142857142857, + "bins": [ + { + "n": 25, + "mean_prediction": 0.12486453633065611, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_empirical_upfront_paired_C1.0": { + "efficient": { + "n": 125, + "brier": 0.2080943989531877, + "auc": 0.6736353077816493, + "bins": [ + { + "n": 4, + "mean_prediction": 0.15644375818272682, + "observed_frequency": 0.5 + }, + { + "n": 10, + "mean_prediction": 0.31150298086087785, + "observed_frequency": 0.4 + }, + { + "n": 8, + "mean_prediction": 0.5588278855410702, + "observed_frequency": 0.25 + }, + { + "n": 55, + "mean_prediction": 0.7094333493962423, + "observed_frequency": 0.7090909090909091 + }, + { + "n": 48, + "mean_prediction": 0.8537685801458172, + "observed_frequency": 0.7708333333333334 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21360032306533672, + "auc": 0.6137326073805203, + "bins": [ + { + "n": 1, + "mean_prediction": 0.17075662235704414, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.33976772838680547, + "observed_frequency": 0.75 + }, + { + "n": 5, + "mean_prediction": 0.5151525354022158, + "observed_frequency": 0.2 + }, + { + "n": 55, + "mean_prediction": 0.7426988360322682, + "observed_frequency": 0.7090909090909091 + }, + { + "n": 60, + "mean_prediction": 0.8613185302743246, + "observed_frequency": 0.7333333333333333 + } + ] + }, + "gap_mse": 0.15682659187926692, + "predicted_mean_gap": 0.06737587217351008, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.5454545454545454, + "sol_only_rescue": { + "n": 125, + "brier": 0.09036826846799315, + "auc": 0.5574162679425837, + "bins": [ + { + "n": 112, + "mean_prediction": 0.07325571273365372, + "observed_frequency": 0.08928571428571429 + }, + { + "n": 9, + "mean_prediction": 0.30635966893978644, + "observed_frequency": 0.1111111111111111 + }, + { + "n": 3, + "mean_prediction": 0.428030746619577, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.6652938490985272, + "observed_frequency": 0.0 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.18543689404289726, + "auc": 0.8269230769230769, + "bins": [ + { + "n": 2, + "mean_prediction": 0.1487132700934445, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.31977350532684157, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.5688527980972494, + "observed_frequency": 0.3333333333333333 + }, + { + "n": 14, + "mean_prediction": 0.696359329420184, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 3, + "mean_prediction": 0.851002535914096, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20632419886038997, + "auc": 0.7662337662337662, + "bins": [ + { + "n": 1, + "mean_prediction": 0.17075662235704414, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.3523191492394685, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.5151525354022158, + "observed_frequency": 0.2 + }, + { + "n": 12, + "mean_prediction": 0.7536107186692327, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 6, + "mean_prediction": 0.834100926820931, + "observed_frequency": 0.8333333333333334 + } + ] + }, + "gap_mse": 0.21189705150266153, + "predicted_mean_gap": 0.07525715853919776, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.303030303030303, + "sol_only_rescue": { + "n": 25, + "brier": 0.12866363917485774, + "auc": 0.3333333333333333, + "bins": [ + { + "n": 21, + "mean_prediction": 0.09084986563564264, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 2, + "mean_prediction": 0.26677127148198065, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.436230798411804, + "observed_frequency": 0.0 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19168814317670066, + "auc": 0.5745614035087719, + "bins": [ + { + "n": 8, + "mean_prediction": 0.7490222205011471, + "observed_frequency": 0.75 + }, + { + "n": 42, + "mean_prediction": 0.8545431022757461, + "observed_frequency": 0.7619047619047619 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.20938010339620747, + "auc": 0.604989604989605, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7604430685421785, + "observed_frequency": 1.0 + }, + { + "n": 47, + "mean_prediction": 0.8648773647651561, + "observed_frequency": 0.723404255319149 + } + ] + }, + "gap_mse": 0.1015502040973808, + "predicted_mean_gap": 0.020951545799966984, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.5104166666666667, + "sol_only_rescue": { + "n": 50, + "brier": 0.03981500899372741, + "auc": 0.5, + "bins": [ + { + "n": 50, + "mean_prediction": 0.04846643458664766, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.20137012683964148, + "auc": 0.7847222222222222, + "bins": [ + { + "n": 3, + "mean_prediction": 0.5540030399384587, + "observed_frequency": 0.0 + }, + { + "n": 22, + "mean_prediction": 0.6992186345662055, + "observed_frequency": 0.7272727272727273 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.190760341718989, + "auc": 0.746031746031746, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7444086986569519, + "observed_frequency": 0.72 + } + ] + }, + "gap_mse": 0.07131746482173262, + "predicted_mean_gap": 0.06261593544607624, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6739130434782609, + "sol_only_rescue": { + "n": 25, + "brier": 0.07580317490467339, + "auc": 0.5217391304347826, + "bins": [ + { + "n": 25, + "mean_prediction": 0.09726600462104401, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.27028868752999846, + "auc": 0.5514705882352942, + "bins": [ + { + "n": 2, + "mean_prediction": 0.1641742462720091, + "observed_frequency": 1.0 + }, + { + "n": 7, + "mean_prediction": 0.3079584703754648, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 2, + "mean_prediction": 0.5510277851107188, + "observed_frequency": 0.5 + }, + { + "n": 11, + "mean_prediction": 0.7177105345859156, + "observed_frequency": 0.6363636363636364 + }, + { + "n": 3, + "mean_prediction": 0.8456913145585326, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.25215686795488956, + "auc": 0.48412698412698413, + "bins": [ + { + "n": 3, + "mean_prediction": 0.3355839214359178, + "observed_frequency": 1.0 + }, + { + "n": 15, + "mean_prediction": 0.727570712379575, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 7, + "mean_prediction": 0.8607528730816514, + "observed_frequency": 0.7142857142857143 + } + ] + }, + "gap_mse": 0.2978180348771788, + "predicted_mean_gap": 0.15710317528234244, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.48809523809523814, + "sol_only_rescue": { + "n": 25, + "brier": 0.16774451027297987, + "auc": 0.47619047619047616, + "bins": [ + { + "n": 16, + "mean_prediction": 0.09011380018513991, + "observed_frequency": 0.1875 + }, + { + "n": 7, + "mean_prediction": 0.31767063964201664, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 1, + "mean_prediction": 0.4116306430351231, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.6652938490985272, + "observed_frequency": 0.0 + } + ] + } + } + } + }, + "v2_empirical_upfront_per_model_C1.0": { + "efficient": { + "n": 125, + "brier": 0.20657867476950909, + "auc": 0.6750871080139373, + "bins": [ + { + "n": 2, + "mean_prediction": 0.17533662530467625, + "observed_frequency": 0.5 + }, + { + "n": 10, + "mean_prediction": 0.30144249797903244, + "observed_frequency": 0.4 + }, + { + "n": 9, + "mean_prediction": 0.5314527188685427, + "observed_frequency": 0.2222222222222222 + }, + { + "n": 55, + "mean_prediction": 0.700527687016309, + "observed_frequency": 0.7090909090909091 + }, + { + "n": 49, + "mean_prediction": 0.8497572027666681, + "observed_frequency": 0.7755102040816326 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21170684744219537, + "auc": 0.6110102843315185, + "bins": [ + { + "n": 2, + "mean_prediction": 0.3286678524251529, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.5251891302489499, + "observed_frequency": 0.625 + }, + { + "n": 57, + "mean_prediction": 0.753289270092932, + "observed_frequency": 0.7017543859649122 + }, + { + "n": 58, + "mean_prediction": 0.8627523548403426, + "observed_frequency": 0.7241379310344828 + } + ] + }, + "gap_mse": 0.15730952554883526, + "predicted_mean_gap": 0.07616540240958886, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.6204146730462521, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.19446320585991278, + "auc": 0.8012820512820513, + "bins": [ + { + "n": 1, + "mean_prediction": 0.17030812553329366, + "observed_frequency": 0.0 + }, + { + "n": 3, + "mean_prediction": 0.2804262563941234, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.4982774513801156, + "observed_frequency": 0.0 + }, + { + "n": 15, + "mean_prediction": 0.6833517333570753, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 4, + "mean_prediction": 0.8454265758755718, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2157219226480885, + "auc": 0.7272727272727273, + "bins": [ + { + "n": 2, + "mean_prediction": 0.3286678524251529, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.5244952757091477, + "observed_frequency": 0.4 + }, + { + "n": 15, + "mean_prediction": 0.7548579011867973, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 3, + "mean_prediction": 0.8637521691101347, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "gap_mse": 0.2154156477616112, + "predicted_mean_gap": 0.06215252028776371, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.3181818181818181 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19024135884503435, + "auc": 0.5986842105263158, + "bins": [ + { + "n": 7, + "mean_prediction": 0.751118023184539, + "observed_frequency": 0.7142857142857143 + }, + { + "n": 43, + "mean_prediction": 0.8496762066956742, + "observed_frequency": 0.7674418604651163 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.208579449311202, + "auc": 0.6133056133056133, + "bins": [ + { + "n": 2, + "mean_prediction": 0.7631408772085947, + "observed_frequency": 1.0 + }, + { + "n": 48, + "mean_prediction": 0.8634623986342778, + "observed_frequency": 0.7291666666666666 + } + ] + }, + "gap_mse": 0.10214024588690591, + "predicted_mean_gap": 0.023571476773134994, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.53125 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.20418640315421943, + "auc": 0.7847222222222222, + "bins": [ + { + "n": 3, + "mean_prediction": 0.5549689153865006, + "observed_frequency": 0.0 + }, + { + "n": 22, + "mean_prediction": 0.6907296381938236, + "observed_frequency": 0.7272727272727273 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.1935848290660298, + "auc": 0.7380952380952381, + "bins": [ + { + "n": 24, + "mean_prediction": 0.7488403567480294, + "observed_frequency": 0.7083333333333334 + }, + { + "n": 1, + "mean_prediction": 0.8011884587899738, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.06779457375523315, + "predicted_mean_gap": 0.0764959293727624, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.8260869565217391 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2537610471433447, + "auc": 0.5588235294117647, + "bins": [ + { + "n": 1, + "mean_prediction": 0.1803651250760588, + "observed_frequency": 1.0 + }, + { + "n": 7, + "mean_prediction": 0.3104494586582792, + "observed_frequency": 0.5714285714285714 + }, + { + "n": 4, + "mean_prediction": 0.5304032052242879, + "observed_frequency": 0.5 + }, + { + "n": 11, + "mean_prediction": 0.7113516893622699, + "observed_frequency": 0.7272727272727273 + }, + { + "n": 2, + "mean_prediction": 0.8601598720752253, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23206858687445464, + "auc": 0.4444444444444444, + "bins": [ + { + "n": 3, + "mean_prediction": 0.5263455544819534, + "observed_frequency": 1.0 + }, + { + "n": 16, + "mean_prediction": 0.7572605975703292, + "observed_frequency": 0.6875 + }, + { + "n": 6, + "mean_prediction": 0.8668327466956933, + "observed_frequency": 0.6666666666666666 + } + ] + }, + "gap_mse": 0.29905691445352006, + "predicted_mean_gap": 0.195035608841148, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5833333333333334 + } + } + }, + "v2_empirical_upfront_rescue_C1.0": { + "sol_only_rescue": { + "n": 125, + "brier": 0.08525474469007004, + "auc": 0.5789473684210527, + "bins": [ + { + "n": 112, + "mean_prediction": 0.07622955214302686, + "observed_frequency": 0.08928571428571429 + }, + { + "n": 12, + "mean_prediction": 0.28427142383378873, + "observed_frequency": 0.08333333333333333 + }, + { + "n": 1, + "mean_prediction": 0.47459195550358957, + "observed_frequency": 0.0 + } + ] + } + }, + "v2_empirical_review_benefit_per_dollar_C0.01": { + "efficient": { + "n": 125, + "brier": 0.2112880942819632, + "auc": 0.6933797909407665, + "bins": [ + { + "n": 125, + "mean_prediction": 0.748423428112692, + "observed_frequency": 0.672 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.2135266089139855, + "auc": 0.6237144585601936, + "bins": [ + { + "n": 63, + "mean_prediction": 0.7682466772610972, + "observed_frequency": 0.6190476190476191 + }, + { + "n": 62, + "mean_prediction": 0.8215671846480648, + "observed_frequency": 0.7741935483870968 + } + ] + }, + "gap_mse": 0.1507635135305867, + "predicted_mean_gap": 0.04627022081234104, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7519936204146731, + "sol_only_rescue": { + "n": 125, + "brier": 0.07894204770048503, + "auc": 0.7432216905901117, + "bins": [ + { + "n": 125, + "mean_prediction": 0.08353897476842373, + "observed_frequency": 0.088 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.24649849180878186, + "auc": 0.9551282051282051, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7141349171022814, + "observed_frequency": 0.52 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.26302867995468027, + "auc": 0.8701298701298701, + "bins": [ + { + "n": 16, + "mean_prediction": 0.7425919983569584, + "observed_frequency": 0.375 + }, + { + "n": 9, + "mean_prediction": 0.8102934385002505, + "observed_frequency": 0.8888888888888888 + } + ] + }, + "gap_mse": 0.1973426180626784, + "predicted_mean_gap": 0.05282959970626206, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6363636363636364, + "sol_only_rescue": { + "n": 25, + "brier": 0.10542548757799036, + "auc": 0.6363636363636364, + "bins": [ + { + "n": 25, + "mean_prediction": 0.09453413502816466, + "observed_frequency": 0.12 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.1813557390507603, + "auc": 0.49122807017543857, + "bins": [ + { + "n": 50, + "mean_prediction": 0.7880444684447534, + "observed_frequency": 0.76 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.19896197510709127, + "auc": 0.4490644490644491, + "bins": [ + { + "n": 50, + "mean_prediction": 0.8243777135690532, + "observed_frequency": 0.74 + } + ] + }, + "gap_mse": 0.10266858123475395, + "predicted_mean_gap": 0.03633324512429957, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.96875, + "sol_only_rescue": { + "n": 50, + "brier": 0.03887840062044573, + "auc": 0.96875, + "bins": [ + { + "n": 50, + "mean_prediction": 0.07164732277157736, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.23898826861324188, + "auc": 0.625, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7360993887082864, + "observed_frequency": 0.64 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.2068442735792015, + "auc": 0.507936507936508, + "bins": [ + { + "n": 24, + "mean_prediction": 0.7855037419478069, + "observed_frequency": 0.7083333333333334 + }, + { + "n": 1, + "mean_prediction": 0.8024856236167205, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.07418679787593897, + "predicted_mean_gap": 0.05008362850627704, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6739130434782609, + "sol_only_rescue": { + "n": 25, + "brier": 0.07351623992325741, + "auc": 0.5652173913043479, + "bins": [ + { + "n": 25, + "mean_prediction": 0.08684112692043212, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2082422328862718, + "auc": 0.7647058823529411, + "bins": [ + { + "n": 25, + "mean_prediction": 0.7157938978633855, + "observed_frequency": 0.68 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.19983614082186324, + "auc": 0.6587301587301587, + "bins": [ + { + "n": 23, + "mean_prediction": 0.7680860385648011, + "observed_frequency": 0.6956521739130435 + }, + { + "n": 2, + "mean_prediction": 0.811576599804194, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.27695098924480827, + "predicted_mean_gap": 0.05577138560056696, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6190476190476191, + "sol_only_rescue": { + "n": 25, + "brier": 0.13801170976028596, + "auc": 0.5952380952380952, + "bins": [ + { + "n": 25, + "mean_prediction": 0.0930249663503671, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_original_upfront_per_model_C1.0": { + "efficient": { + "n": 125, + "brier": 0.20586976539836235, + "auc": 0.6689895470383276, + "bins": [ + { + "n": 2, + "mean_prediction": 0.16734055590238106, + "observed_frequency": 0.0 + }, + { + "n": 10, + "mean_prediction": 0.31257471880364485, + "observed_frequency": 0.5 + }, + { + "n": 17, + "mean_prediction": 0.525050269638746, + "observed_frequency": 0.5882352941176471 + }, + { + "n": 42, + "mean_prediction": 0.685490493237321, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 54, + "mean_prediction": 0.867196084861678, + "observed_frequency": 0.7592592592592593 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.21463596402764343, + "auc": 0.6046581972171808, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38349530742347737, + "observed_frequency": 0.0 + }, + { + "n": 12, + "mean_prediction": 0.552574380463959, + "observed_frequency": 0.5833333333333334 + }, + { + "n": 41, + "mean_prediction": 0.7310730169579913, + "observed_frequency": 0.6341463414634146 + }, + { + "n": 71, + "mean_prediction": 0.8737483687005557, + "observed_frequency": 0.7605633802816901 + } + ] + }, + "gap_mse": 0.15835986324432882, + "predicted_mean_gap": 0.08815234851048069, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.665071770334928, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.19488912439376446, + "auc": 0.8076923076923077, + "bins": [ + { + "n": 2, + "mean_prediction": 0.16734055590238106, + "observed_frequency": 0.0 + }, + { + "n": 1, + "mean_prediction": 0.2802147089293126, + "observed_frequency": 0.0 + }, + { + "n": 7, + "mean_prediction": 0.5046435849841482, + "observed_frequency": 0.42857142857142855 + }, + { + "n": 11, + "mean_prediction": 0.7439326104067401, + "observed_frequency": 0.6363636363636364 + }, + { + "n": 4, + "mean_prediction": 0.8810929519268931, + "observed_frequency": 0.75 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21868500464071708, + "auc": 0.7922077922077922, + "bins": [ + { + "n": 1, + "mean_prediction": 0.38349530742347737, + "observed_frequency": 0.0 + }, + { + "n": 5, + "mean_prediction": 0.5529075674542637, + "observed_frequency": 0.2 + }, + { + "n": 7, + "mean_prediction": 0.6993408522286815, + "observed_frequency": 0.42857142857142855 + }, + { + "n": 12, + "mean_prediction": 0.8521454465974614, + "observed_frequency": 0.8333333333333334 + } + ] + }, + "gap_mse": 0.23149067001288734, + "predicted_mean_gap": 0.09656532126641114, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.4090909090909091 + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.19701885570569297, + "auc": 0.5877192982456141, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7361446911347711, + "observed_frequency": 1.0 + }, + { + "n": 47, + "mean_prediction": 0.866992066203904, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.2168710265280743, + "auc": 0.49896049896049893, + "bins": [ + { + "n": 1, + "mean_prediction": 0.7152948899908204, + "observed_frequency": 1.0 + }, + { + "n": 49, + "mean_prediction": 0.8868058951771391, + "observed_frequency": 0.7346938775510204 + } + ] + }, + "gap_mse": 0.10274702190273528, + "predicted_mean_gap": 0.02423445137365687, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.8229166666666667 + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.1825976903237712, + "auc": 0.861111111111111, + "bins": [ + { + "n": 1, + "mean_prediction": 0.3066542216250509, + "observed_frequency": 0.0 + }, + { + "n": 4, + "mean_prediction": 0.547373057749475, + "observed_frequency": 0.5 + }, + { + "n": 19, + "mean_prediction": 0.6659394144995158, + "observed_frequency": 0.6842105263157895 + }, + { + "n": 1, + "mean_prediction": 0.8515193573703993, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18390186166529454, + "auc": 0.880952380952381, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7422146865505383, + "observed_frequency": 0.65 + }, + { + "n": 5, + "mean_prediction": 0.8203560846538114, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.06890436328025466, + "predicted_mean_gap": 0.11782237875182698, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.6956521739130435 + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2578243008628901, + "auc": 0.49264705882352944, + "bins": [ + { + "n": 8, + "mean_prediction": 0.3173597821852606, + "observed_frequency": 0.625 + }, + { + "n": 6, + "mean_prediction": 0.5339762096619575, + "observed_frequency": 0.8333333333333334 + }, + { + "n": 9, + "mean_prediction": 0.6384510058442484, + "observed_frequency": 0.5555555555555556 + }, + { + "n": 2, + "mean_prediction": 0.8520351529345824, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.23685090077605683, + "auc": 0.380952380952381, + "bins": [ + { + "n": 7, + "mean_prediction": 0.5523363897565986, + "observed_frequency": 0.8571428571428571 + }, + { + "n": 13, + "mean_prediction": 0.7322322391288681, + "observed_frequency": 0.6923076923076923 + }, + { + "n": 5, + "mean_prediction": 0.8510239063242085, + "observed_frequency": 0.6 + } + ] + }, + "gap_mse": 0.2859102391230315, + "predicted_mean_gap": 0.17790513978685155, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.6309523809523809 + } + } + }, + "v2_original_review_benefit_per_dollar_C0.1": { + "efficient": { + "n": 125, + "brier": 0.19475731872208132, + "auc": 0.6823461091753774, + "bins": [ + { + "n": 7, + "mean_prediction": 0.2730479048664671, + "observed_frequency": 0.0 + }, + { + "n": 8, + "mean_prediction": 0.572792908294837, + "observed_frequency": 0.5 + }, + { + "n": 48, + "mean_prediction": 0.7167708857902779, + "observed_frequency": 0.6041666666666666 + }, + { + "n": 62, + "mean_prediction": 0.8532704382769977, + "observed_frequency": 0.8225806451612904 + } + ] + }, + "capable": { + "n": 125, + "brier": 0.205236316674579, + "auc": 0.6167574107683, + "bins": [ + { + "n": 7, + "mean_prediction": 0.45844405121046133, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 42, + "mean_prediction": 0.7462895669734516, + "observed_frequency": 0.6428571428571429 + }, + { + "n": 76, + "mean_prediction": 0.8621683001695367, + "observed_frequency": 0.7763157894736842 + } + ] + }, + "gap_mse": 0.15083555656813358, + "predicted_mean_gap": 0.050212901541694555, + "observed_mean_gap": 0.024, + "sol_only_gap_auc": 0.7073365231259968, + "sol_only_rescue": { + "n": 125, + "brier": 0.08031896724608772, + "auc": 0.6842105263157895, + "bins": [ + { + "n": 120, + "mean_prediction": 0.07198762815203195, + "observed_frequency": 0.09166666666666666 + }, + { + "n": 5, + "mean_prediction": 0.2500931349617063, + "observed_frequency": 0.0 + } + ] + }, + "by_benchmark": { + "lcb": { + "efficient": { + "n": 25, + "brier": 0.13181318837313602, + "auc": 0.9487179487179488, + "bins": [ + { + "n": 7, + "mean_prediction": 0.2730479048664671, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.5547376465205437, + "observed_frequency": 0.0 + }, + { + "n": 6, + "mean_prediction": 0.7248448731838749, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 10, + "mean_prediction": 0.8399584582796091, + "observed_frequency": 0.9 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.18787617751354888, + "auc": 0.9090909090909091, + "bins": [ + { + "n": 7, + "mean_prediction": 0.45844405121046133, + "observed_frequency": 0.14285714285714285 + }, + { + "n": 7, + "mean_prediction": 0.7329031940881119, + "observed_frequency": 0.2857142857142857 + }, + { + "n": 11, + "mean_prediction": 0.8544242032361381, + "observed_frequency": 1.0 + } + ] + }, + "gap_mse": 0.20039222751558342, + "predicted_mean_gap": 0.07874530014727336, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.5909090909090908, + "sol_only_rescue": { + "n": 25, + "brier": 0.11162071470745126, + "auc": 0.5909090909090908, + "bins": [ + { + "n": 20, + "mean_prediction": 0.08852120639588114, + "observed_frequency": 0.15 + }, + { + "n": 5, + "mean_prediction": 0.2500931349617063, + "observed_frequency": 0.0 + } + ] + } + }, + "mbpp": { + "efficient": { + "n": 50, + "brier": 0.18695068562083517, + "auc": 0.5043859649122807, + "bins": [ + { + "n": 4, + "mean_prediction": 0.7378031217945231, + "observed_frequency": 0.5 + }, + { + "n": 46, + "mean_prediction": 0.8613292739755435, + "observed_frequency": 0.782608695652174 + } + ] + }, + "capable": { + "n": 50, + "brier": 0.208890426637236, + "auc": 0.3866943866943867, + "bins": [ + { + "n": 3, + "mean_prediction": 0.7717796010734302, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 47, + "mean_prediction": 0.8742327268387793, + "observed_frequency": 0.7446808510638298 + } + ] + }, + "gap_mse": 0.10035736842153892, + "predicted_mean_gap": 0.016638357491796322, + "observed_mean_gap": -0.02, + "sol_only_gap_auc": 0.9791666666666666, + "sol_only_rescue": { + "n": 50, + "brier": 0.0361568271447332, + "auc": 0.9583333333333334, + "bins": [ + { + "n": 50, + "mean_prediction": 0.04666064024782396, + "observed_frequency": 0.04 + } + ] + } + }, + "swe": { + "efficient": { + "n": 25, + "brier": 0.2505947865898087, + "auc": 0.4722222222222222, + "bins": [ + { + "n": 1, + "mean_prediction": 0.5984650151506499, + "observed_frequency": 1.0 + }, + { + "n": 20, + "mean_prediction": 0.7368278035525606, + "observed_frequency": 0.55 + }, + { + "n": 4, + "mean_prediction": 0.8139818632578771, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.21767436620765138, + "auc": 0.40476190476190477, + "bins": [ + { + "n": 12, + "mean_prediction": 0.7745779866915775, + "observed_frequency": 0.75 + }, + { + "n": 13, + "mean_prediction": 0.8351767748528173, + "observed_frequency": 0.6923076923076923 + } + ] + }, + "gap_mse": 0.07268589804642063, + "predicted_mean_gap": 0.062451414966087394, + "observed_mean_gap": 0.08, + "sol_only_gap_auc": 0.7391304347826086, + "sol_only_rescue": { + "n": 25, + "brier": 0.07280073531343963, + "auc": 0.6086956521739131, + "bins": [ + { + "n": 25, + "mean_prediction": 0.08365296583491949, + "observed_frequency": 0.08 + } + ] + } + }, + "terminal": { + "efficient": { + "n": 25, + "brier": 0.2174772474057914, + "auc": 0.5808823529411765, + "bins": [ + { + "n": 5, + "mean_prediction": 0.5748805916333917, + "observed_frequency": 0.6 + }, + { + "n": 18, + "mean_prediction": 0.6871202622555995, + "observed_frequency": 0.6666666666666666 + }, + { + "n": 2, + "mean_prediction": 0.8130542672356273, + "observed_frequency": 1.0 + } + ] + }, + "capable": { + "n": 25, + "brier": 0.20285018637722257, + "auc": 0.5317460317460317, + "bins": [ + { + "n": 20, + "mean_prediction": 0.7301782405374482, + "observed_frequency": 0.7 + }, + { + "n": 5, + "mean_prediction": 0.835977668555605, + "observed_frequency": 0.8 + } + ] + }, + "gap_mse": 0.2803849204355858, + "predicted_mean_gap": 0.07659107761151943, + "observed_mean_gap": 0.04, + "sol_only_gap_auc": 0.33333333333333337, + "sol_only_rescue": { + "n": 25, + "brier": 0.1448597319200813, + "auc": 0.2619047619047619, + "bins": [ + { + "n": 25, + "mean_prediction": 0.09774940368248107, + "observed_frequency": 0.16 + } + ] + } + } + } + }, + "v2_original_review_rescue_C1.0": { + "sol_only_rescue": { + "n": 125, + "brier": 0.08576811223362082, + "auc": 0.5773524720893142, + "bins": [ + { + "n": 117, + "mean_prediction": 0.05753457988877527, + "observed_frequency": 0.09401709401709402 + }, + { + "n": 6, + "mean_prediction": 0.29215656863558076, + "observed_frequency": 0.0 + }, + { + "n": 2, + "mean_prediction": 0.4430515794724206, + "observed_frequency": 0.0 + } + ] + } + } + } +} \ No newline at end of file diff --git a/cookbook/auto_router_selective_training/development_pair_counts.json b/cookbook/auto_router_selective_training/development_pair_counts.json new file mode 100644 index 00000000000..07a56eb2715 --- /dev/null +++ b/cookbook/auto_router_selective_training/development_pair_counts.json @@ -0,0 +1,62 @@ +{ + "train": { + "lcb": { + "tasks": 50, + "both_fail": 14, + "sol_only": 9, + "luna_only": 3, + "both_solve": 24 + }, + "mbpp": { + "tasks": 80, + "both_fail": 7, + "sol_only": 2, + "luna_only": 3, + "both_solve": 68 + }, + "swe": { + "tasks": 25, + "both_fail": 5, + "sol_only": 1, + "luna_only": 0, + "both_solve": 19 + }, + "terminal": { + "tasks": 5, + "both_fail": 0, + "sol_only": 1, + "luna_only": 0, + "both_solve": 4 + } + }, + "validation": { + "lcb": { + "tasks": 20, + "both_fail": 2, + "sol_only": 6, + "luna_only": 2, + "both_solve": 10 + }, + "mbpp": { + "tasks": 29, + "both_fail": 4, + "sol_only": 2, + "luna_only": 2, + "both_solve": 21 + }, + "swe": { + "tasks": 10, + "both_fail": 1, + "sol_only": 1, + "luna_only": 1, + "both_solve": 7 + }, + "terminal": { + "tasks": 5, + "both_fail": 2, + "sol_only": 0, + "luna_only": 1, + "both_solve": 2 + } + } +} \ No newline at end of file diff --git a/cookbook/auto_router_selective_training/execution_diagnostics.json b/cookbook/auto_router_selective_training/execution_diagnostics.json new file mode 100644 index 00000000000..fc9a782f93a --- /dev/null +++ b/cookbook/auto_router_selective_training/execution_diagnostics.json @@ -0,0 +1,315 @@ +{ + "scope": "Execution diagnostics for completed attempts, without reading grades. A length stop is a model output-budget event, not an infrastructure failure. All recorded responses remain billed.", + "groups": [ + { + "split": "test", + "benchmark": "lcb", + "model": "anthropic/claude-opus-5", + "attempts": 25, + "responses": 25, + "response_cost": 2.384855, + "token_limit_responses": 7, + "token_limit_response_cost": 1.4849325, + "token_limit_without_action_or_text": 7, + "token_limit_without_action_or_text_cost": 1.4849325 + }, + { + "split": "test", + "benchmark": "lcb", + "model": "anthropic/claude-sonnet-5", + "attempts": 25, + "responses": 25, + "response_cost": 1.3789665000000002, + "token_limit_responses": 14, + "token_limit_response_cost": 1.1779045000000001, + "token_limit_without_action_or_text": 14, + "token_limit_without_action_or_text_cost": 1.1779045000000001 + }, + { + "split": "test", + "benchmark": "lcb", + "model": "openai/gpt-5.6-luna", + "attempts": 25, + "responses": 25, + "response_cost": 0.1164804, + "token_limit_responses": 7, + "token_limit_response_cost": 0.06988079999999999, + "token_limit_without_action_or_text": 7, + "token_limit_without_action_or_text_cost": 0.06988079999999999 + }, + { + "split": "test", + "benchmark": "lcb", + "model": "openai/gpt-5.6-sol", + "attempts": 25, + "responses": 25, + "response_cost": 1.529944, + "token_limit_responses": 4, + "token_limit_response_cost": 0.667568, + "token_limit_without_action_or_text": 4, + "token_limit_without_action_or_text_cost": 0.667568 + }, + { + "split": "test", + "benchmark": "mbpp", + "model": "anthropic/claude-opus-5", + "attempts": 50, + "responses": 50, + "response_cost": 0.224865 + }, + { + "split": "test", + "benchmark": "mbpp", + "model": "anthropic/claude-sonnet-5", + "attempts": 50, + "responses": 50, + "response_cost": 0.09884600000000003 + }, + { + "split": "test", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-luna", + "attempts": 50, + "responses": 50, + "response_cost": 0.017396 + }, + { + "split": "test", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-sol", + "attempts": 50, + "responses": 50, + "response_cost": 0.20637600000000006 + }, + { + "split": "test", + "benchmark": "swe", + "model": "anthropic/claude-opus-5", + "attempts": 25, + "responses": 381, + "response_cost": 10.529565750000002 + }, + { + "split": "test", + "benchmark": "swe", + "model": "anthropic/claude-sonnet-5", + "attempts": 25, + "responses": 833, + "response_cost": 11.053656000000016, + "token_limit_responses": 4, + "token_limit_response_cost": 0.4144736, + "token_limit_without_action_or_text": 4, + "token_limit_without_action_or_text_cost": 0.4144736 + }, + { + "split": "test", + "benchmark": "swe", + "model": "openai/gpt-5.6-luna", + "attempts": 25, + "responses": 427, + "response_cost": 0.6371064399999993 + }, + { + "split": "test", + "benchmark": "swe", + "model": "openai/gpt-5.6-sol", + "attempts": 25, + "responses": 403, + "response_cost": 10.713241600000002 + }, + { + "split": "test", + "benchmark": "terminal", + "model": "anthropic/claude-opus-5", + "attempts": 25, + "responses": 414, + "response_cost": 32.915727249999996, + "token_limit_responses": 80, + "token_limit_response_cost": 17.381599, + "token_limit_without_action_or_text": 76, + "token_limit_without_action_or_text_cost": 16.427720750000002 + }, + { + "split": "test", + "benchmark": "terminal", + "model": "anthropic/claude-sonnet-5", + "attempts": 25, + "responses": 795, + "response_cost": 25.464663800000025, + "token_limit_responses": 106, + "token_limit_response_cost": 9.517311799999998, + "token_limit_without_action_or_text": 98, + "token_limit_without_action_or_text_cost": 8.773203099999998 + }, + { + "split": "test", + "benchmark": "terminal", + "model": "openai/gpt-5.6-luna", + "attempts": 25, + "responses": 839, + "response_cost": 1.8821569099999986, + "token_limit_responses": 43, + "token_limit_response_cost": 0.43733629999999996, + "token_limit_without_action_or_text": 35, + "token_limit_without_action_or_text_cost": 0.35178958 + }, + { + "split": "test", + "benchmark": "terminal", + "model": "openai/gpt-5.6-sol", + "attempts": 25, + "responses": 341, + "response_cost": 8.446844399999998, + "token_limit_responses": 3, + "token_limit_response_cost": 0.5060674000000001 + }, + { + "split": "train", + "benchmark": "lcb", + "model": "openai/gpt-5.6-luna", + "attempts": 50, + "responses": 50, + "response_cost": 0.2573764, + "token_limit_responses": 16, + "token_limit_response_cost": 0.1596568, + "token_limit_without_action_or_text": 16, + "token_limit_without_action_or_text_cost": 0.1596568 + }, + { + "split": "train", + "benchmark": "lcb", + "model": "openai/gpt-5.6-sol", + "attempts": 50, + "responses": 50, + "response_cost": 2.630148, + "token_limit_responses": 5, + "token_limit_response_cost": 0.836052, + "token_limit_without_action_or_text": 5, + "token_limit_without_action_or_text_cost": 0.836052 + }, + { + "split": "train", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-luna", + "attempts": 80, + "responses": 80, + "response_cost": 0.0218284 + }, + { + "split": "train", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-sol", + "attempts": 80, + "responses": 80, + "response_cost": 0.289156 + }, + { + "split": "train", + "benchmark": "swe", + "model": "openai/gpt-5.6-luna", + "attempts": 25, + "responses": 398, + "response_cost": 0.5176403399999997 + }, + { + "split": "train", + "benchmark": "swe", + "model": "openai/gpt-5.6-sol", + "attempts": 25, + "responses": 346, + "response_cost": 7.403714199999999 + }, + { + "split": "train", + "benchmark": "terminal", + "model": "openai/gpt-5.6-luna", + "attempts": 5, + "responses": 87, + "response_cost": 0.12343140999999999 + }, + { + "split": "train", + "benchmark": "terminal", + "model": "openai/gpt-5.6-sol", + "attempts": 5, + "responses": 34, + "response_cost": 0.7672414000000001 + }, + { + "split": "validation", + "benchmark": "lcb", + "model": "openai/gpt-5.6-luna", + "attempts": 20, + "responses": 20, + "response_cost": 0.0921004, + "token_limit_responses": 6, + "token_limit_response_cost": 0.0598586, + "token_limit_without_action_or_text": 6, + "token_limit_without_action_or_text_cost": 0.0598586 + }, + { + "split": "validation", + "benchmark": "lcb", + "model": "openai/gpt-5.6-sol", + "attempts": 20, + "responses": 20, + "response_cost": 1.0552880000000002, + "token_limit_responses": 1, + "token_limit_response_cost": 0.166696, + "token_limit_without_action_or_text": 1, + "token_limit_without_action_or_text_cost": 0.166696 + }, + { + "split": "validation", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-luna", + "attempts": 30, + "responses": 30, + "response_cost": 0.010364400000000001 + }, + { + "split": "validation", + "benchmark": "mbpp", + "model": "openai/gpt-5.6-sol", + "attempts": 30, + "responses": 30, + "response_cost": 0.14699600000000002 + }, + { + "split": "validation", + "benchmark": "swe", + "model": "openai/gpt-5.6-luna", + "attempts": 10, + "responses": 149, + "response_cost": 0.2096464700000001 + }, + { + "split": "validation", + "benchmark": "swe", + "model": "openai/gpt-5.6-sol", + "attempts": 10, + "responses": 141, + "response_cost": 3.702219799999998 + }, + { + "split": "validation", + "benchmark": "terminal", + "model": "openai/gpt-5.6-luna", + "attempts": 5, + "responses": 66, + "response_cost": 0.10991243999999999, + "token_limit_responses": 1, + "token_limit_response_cost": 0.009907 + }, + { + "split": "validation", + "benchmark": "terminal", + "model": "openai/gpt-5.6-sol", + "attempts": 5, + "responses": 63, + "response_cost": 2.5312083999999997, + "token_limit_responses": 6, + "token_limit_response_cost": 0.9978314000000001 + } + ] +} \ No newline at end of file diff --git a/cookbook/auto_router_selective_training/manifest.json b/cookbook/auto_router_selective_training/manifest.json index d1eca8caed0..8de8767b0f1 100644 --- a/cookbook/auto_router_selective_training/manifest.json +++ b/cookbook/auto_router_selective_training/manifest.json @@ -4,7 +4,7 @@ "training_tasks": 160, "validation_tasks": 64, "held_out_tasks": 125, - "held_out_status": "primary_pair_complete_anthropic_controls_in_progress", + "held_out_status": "complete", "current_model_pair": [ "openai/gpt-5.6-luna", "openai/gpt-5.6-sol" @@ -13,5 +13,6 @@ "solver_effort": "high", "diagnostic_upfront_profiles": 12, "diagnostic_profiles_sha256": "468de3b22e309ca3ef800ef90bd9b03a8aca3eebef32bcc2155377b4cf7e1f88", - "primary_results_sha256": "e91ec5578d496ad37c1e8cf5760882042f4b7768bd3c2ff5d65d92f19df7a12c" + "primary_results_sha256": "e91ec5578d496ad37c1e8cf5760882042f4b7768bd3c2ff5d65d92f19df7a12c", + "final_results_sha256": "b724f5607b532e1d8b354c95129f5f6d95514926ca5000e9a93a9104b13a844d" } diff --git a/cookbook/auto_router_selective_training/quality_cost.png b/cookbook/auto_router_selective_training/quality_cost.png new file mode 100644 index 0000000000000000000000000000000000000000..aa381d853bc6f6bf777b3dfea96701c11c4088cc GIT binary patch literal 230338 zcmcHhbzIZy`v;CAh#XPGqarG0ASodTNaH~zq(h`pItJ1bBUD5b0YzF`x@+_Ri|!nw z1YyWv1BnSEzSlhOPyK%1f4@8)=ZJ%CuidZvzOL(8hG}UiG0mD{)U zo_tuIJUXGbesFN$P1dlV)*s;dct+rrmBDSDce9q$8BtYH%j9>aZD!@Rac9!sz0w-U z*T337wd>MLB6%+aE|^q&NO0W3yEaWOjEEVC+V*^yM@|v(dr~C^WeOKh3d5WK@2@uF zucD3A|N9a6URi+pE?mz4{`$pwDD!{+9lrO82-5z)4+Y66HS_%6hZIt-Se*I44_)m2 zTbQ!M|LZFX_t!bvKYv%weDw7dEyuGX46k;^8%a~rzsH>?>k`!Za8mfR&0AXoYd_@B zL00iHK5lKUVqVJ+#GPJTGi$hMWggBdDPlj$@`E`!hp<<&eluZ_<PNQ1#i*b9 z|Ngo{Y&xgx|2&iO-T&*7X}@Eq{73_fLL?neYa6eUS6raili zwj~iJ7s$jWiKhLX0>cWMHv*%EnRgi~h-h-+2{x(t*{>C(DXF6MWlY72ugg>eMz7$0 z_YJ49K98RRU*Gc^2->HvPX`_(@aksAXWffSgqP{mD>53cwCn%G>bsbpZPnhzaP{^0 ze$VinQs+p3QQ4!Ay~1Dfok^CFvLnvxdh9Ib$junTa;q@Idaoi`n@yi&X>akXs_%J* z%H8+Q4M=IG%1suUHI5e=SNEiR9w$lVFv_ed=%So3?@OM^k&|4|Ww@1{O?-EXAHm*X zv=|ZDxHMeA-7q14ushh5DAq|JsV9kJrhh&eEN78$b~37VMs&RAwJcAF7_}eFPKe<* zNMv9UkMD@#Pco`eV}c6)(4 zKe;RH`Y%_w&fO)~H*Sr0-mvRy-pUg}`ZHY}r-RSvURpFzrA5xr&b+%2rFf!j<-SN( zzEBa0jCEyy`DF#X`9@=0?Oe6@m+5Kv!1{~UjtLs6KT`wu z%diHc$_R7Q)b;N0^P*isO}meLu#5Ti3K=~ z6dLM$yT3H;u$#F#?33$W4oiw^%5XeK*_T$yl>bw4_HzX8{@thb*#|aGsa7X>_O;MI zv7`z2Ud*)1Sm~w5-$R&>@EH`d{yOkA3D};07?(vl&LVzItH`K|n&`@|UvQt{_<0e| z)lu`nuO`)hU%7kRBx1VN&^DQA-F+W6Z}Vb*Ce51*Vy|jGlv{Tm*&z{Ks+%~*BM2l% z=}y?4Y&E}UU(@(4mnn_)VjN2vsY#RG<`%e~us ze)I43o<=g?-2XjCW8LvStcD>`9N{qP0@vwUG6-9X&8|P=&KKlJwaaw#r;uZrW=Xma zry6{MuSCj-JW5KpRB3f@Sh43TnRXs$j^@@pVe+_zHu&2c7A08;kFKBf0sGsy5cJu* zryR+g<8hul$3{)PP1}g_zh|B)G_d=xwnSMstbRHcY&C{P`prODKX#n;rm$9yMsn_S zj$WQl@KmE8olVb&pMKj3i}&RBHU=;4QIjaGAlSWER-)8lI6oI3cP(s#!DHfv%an{& z<36eB03-K0)twM$9&mJH86&DyZq;6jRp=a%)BFOv!iKE&HsWt&cF(8_dYfyZd`QTn zNod4#X)aP;aMh=5=MH|P=$Jis2^_+jajIn7x5bAXk~HA0n+LM)(fo|+k}JTLH0@tl z9L$MCnt!y;Po^yGG1g~g>_1X{f}-*No==|C3(ny~ znUu;A9CTcsk+zMCpT}yxBL@BS=p-ZUgD><{T0w7~2F~s|CzeKj^V?_I8fWT8A+Sys zo-+87pwJ@YYObN43D;IOw4CCtrvcPeI#rCUmfym|E=+t!4iy*(3BwHW9Mx^=J;}tQ zopJk%HGXHkSN`BIpI+|C-N{>YY-e7MCfKH&FE=rD$)wHTS>#Va6u93Q*v4nMt4Ub= zHgwKXjd7}vEe`6(r3Oy1hX#zOw^mshH|IW0nU$5;sQJ~wkEA_m0}t$b(i!s$^O->v z2wdH(yRIxz$NBk<#MC(wNyjfuL0EnO6F1_i82m0E*0|t)QFEmHe!Sf ztmEv2QKhA4zE~=@dsB`&AC*?dtLD%NoM!EW3aJ|et&26&^DEWU0l`=A#eOlF7IPdq zJYH#knzXY8ZK=6@*7QXMBV``Ac%Davk@9CwZn#NriY%g5L3>*-=As_&KQX_cvp5Hn z{C~A`Z^`l*E@lsSX(Zf?iPmk`3XS!IrB9HU_8(*o++E{3J%uEhx{urEFgExcfH7S7XqzvV5+Ac$j0a5({;TyUI_jBvW;Y?iQEf4xAiZlHE#qS?0L5l`So%wNGjclVhBm& zm|eZ2>O}mz?@&`EV5O_A11Bq8^fW3}WB9fnBuKhv3-IHg%mmXpNw04VYATs*z5M%1 z*D;|6t(p2fy{cIAIuHB$<1s1*wPrDgd$^>kJ)Dpm{YtdxXB}q>5W4j_bHkGh=(TSw zTO?oeJe@56z^QvejaDxhIoSQDKB}Cqm<}LgR(`zC@phUQqJh1 zUb}+A{u7WxEU+$UC{S%&s?j-Win!PBc`xg!KH6luyJXbNuiFHj!3Dizx+(eAJz?4Dm&V`|Lo6HDpl8=RvblESqCgW7(_kZAWly^SLiJtOIDI1>Af<- zRgH61J#)zMS{b)C&kP(+)lTwip=JKE>65o~51%+$?P9*(BTan1Yc_prlpzN1)$!y2 z&Yzd;5J&CquTFJT-P-QAPgl^Y$O6smy^rfE zNJHF#)nZ>ds(YyF(uEbqs^lfc+Pu_^wHN#P@^`TJ5m9q3bR3nVKV0X&g_sf^hO*hf zVEDI8u^f51A{VK+7g#g>(-)7Eb@(g|?QoaU=bw#=R!P!v_?PAxl%!zFEJG2No*Fkd zCdnH=Ey?tCD4c4l=TNTp!Z#L|*qBy^NdkYOWf`2E2f8|>Iv!_$UQl|vn8JR zkxQMCdUC5!Hu$6N;9&SWam}bQx7jhqQv(w)gQrc-PF0UHs>EJ@^{|H z9Ep30TUZ$O&MVtsGp)l1Dg6%F$aqtEo}_s^LD8u^izRP6Dy-E<}9D-6rdZFqCv zpIN0Pd-#&3es5hRv zFJ!H?Hf)T;J$HSv=W#OH-|Bni$c4>#{i#vod^3z&^1E+8C+MU!Ch_UlGBcGTPb8gx zfX@-{-t#pzACB0W6xXO9hFQEB7Lo-uQKEbI0k&sv>fz@Xw7VCiy`0+oH~^EpERltg zNYTu=DFB(YDQR-{AM2CLLwUMw_?Uzni7L@tAq{dy(4qjmS6h3sH-WKp)!RI?Qa)QVHD%;@ zCIMYoFF@!^R~$lzlH>K7`u8MZvhr>gauGMR-c*l;C7NRAub%QPkTt(-T{^Osb3BqD zf?JyV)*&8WjIP{iB9M3-XA#wt_b;9ZP)9!&Y-ixj5ONW|TTLOHVf zNk7esQMW##yrE&rRsW$CZvTt^gFgP!uxcf*9=Y_Kf3Qd;<8h1e_YY7tE)n&^tXwQ2 zlDN#7CVm8H9_;TOYc##__G3FA$xW=5nYZnmYVpaOJ-w--T;S!ysu%J%t1&yPwNB?Rc^`ejNOc!&}Pc^I*#ltySg0R)2x&7Fc?s}fUt24`M z-$OZG6edxZFr~4)*_zZ=Vw*3RXnZ>AaqY`sz)}N`Znjz%+UTLrGnAgM^!S#(7MeP$ z5g)?*poX+S(V@&RGnWaEmB)0Zc@32u^w~N=DddH4DC8}?FTzTgmekw23bJBA(WLd%#_er}iLz=38 zkL}D4j-O**C819@XNbNN#fIU zcW6V0%`^8+Wad)rkZGSxQYi)aqV`wO}2cZ)5=7?>TEGf?RYJI(zM(M_pQ z{>AeJYMJJInz`@t`k&mpl7HMee|~nq=c5{%e5A~;m-a)sZ&h-D)xivu^|N{oX)ly_ z3LcX)uy5l|d2*Q;kq+Ijzu{n?+=An)EkNeFk=U04$72!#m&;n8_Ufms=jEi7 zTPtMdQR=3gw&Y`kq7ZQ{R|gXiBVasYm56#%f8(bQE}6M%qkdy)xHIK5;Z=a}w;*sv(8ObUhat1lq$V~y%< z>B{NJtP=O&Vm2sLUaQ*_LLnP)%sLW4l#z1S*1;YzP{mXz26NxXu4#z9V4~7*Z#HHb zU8$N<-}$(JO=_ibFhj5@@L+!-T(n0x;iJwnyqj~I_awYv7tjH012*I=x$dztOT^Jl zhIwzSFIKh4v9O#FXW|xyXk~--FKPVtVzAMVG}HSEySuyIYZ)6m>Ni0=W{@v8 zL7XIp-63Bcx$XgTBo@_v-F-ogY+BLSn~$G(N_*^|Z&=clpUaT-&l-RD1;qg7pHfQ% z8a9|wUElbXr*r%vCY}G~{n~h$`0w8U%z`c%)qttZX|hi2w|`&SEgNDZDPkLagyF)N z0z0q+(x(z{c-fs=g7b4Cp?T!_2%y6ATblr)j_wz>3lqP%-H0>uwQsXp3FQb_#3KR! zA5AzH{t+0uY-muIP6y<@e@&Ji6mi|X;XHA(xVg-#_WKJK|3<>NV+q4ik;il)3!T@0 zFr?H$Q(8IfFU4E>m2D%(ve)J0(poErOXQVS=)cVCvcOk}wP!Tf92kVP%AFN5b9&c} zoZvl2$WC(pwJC>CBCe4GRX#t$Pt;cIIhXM?x3d*ytt;RZrM806uca*>1b-`gYB&-6 zf#7v7_;S|>L1j!FQic})6O=% zDG0YWPx~fX5;W!aJi16VG%IH|3Kd$N#+mQMUb|jkP_nf*rMHpQh+6>|xaerQYi6!< zD>y{fZ|k_^k@0F5muGh-r={619J_r%qLvK?<4e6)5XMEBPb=8y^;lsBFY!Dt=Aajh z*1X9k?RDH<`U-7e9gWJ~k?^|d%p}>_qo4GX2%&z)OExjJ?zluJV#Nz-^W6yJ>M+n1kFXkZ1dlYe;Yi&^g_3DAI7_ik1GNNfq4;T;?rB*#3aUh;73bR ziJMy*Mt*VgaphSx6)X(3bFN^fPpt*Lm+JH~BfQ}a035lG78a-`_*DiM+k=cZHp z;jJpwW9NZdJ(6%U;5|Ym*!ZVjqpznsuny#&+doo*432a@#^}0lpVP514CQLQLYtlC z$aaP^{bJt}o-wZRvYd-7)!#pdVw)FC(x~*jM=-^v_20Vw!D-#?V3+khoa!=B)8CaU zKRxQVcf?E;I=NyBW&E(m(9y?w&{==&sJpOI6P7cnqo6DEKBOxdNdVw`qJ5V+LZrG5 zd(A~yx5i!R(Pl^nf~Lx4Nz|^#UC|CUEI z(eEFP)2nMXb-J;v>W{*vNhqMyNxvq~2TWiN>t2f6@)mC!T8j=#J4~zbMc3gzCT`-q zZdD}H#U#eos2)Xcs_#IJ68TRXl-JU1dMxvvyv$A*nCXlas6gcSaqur1v_u1;RnZ&s zRlns-hm03=@T;3f(N0ryK>wX^Z9D)n`RLK+{$CS^_}_ks_Ef`_6=O6`s@GWFWFj}S z%;a4pJsB5p*(T_rjOiU*M+^)fm(Fn0>*o_znM!c-Z~t-PrEZWWto5u>iBp!s5*+8DBN}`+)q_LnsWvPDlyNcVu{|!X z=$QH4!C&&+YT9ev=Aycnq^}E0dRKi%V&R;6!pprQb*&Xb?foBSSKe5u`N1m)td&W0vV!KOer0l_gwV^U^2swjz!Fbx zl}t+5)Y8Z^z0AiK=&G6q=Nh4&@=r?WEY@XM*ph5N?wnI!h!@gFj=-qTe>1=pWh;Xz ztJ0kOv>E)qn+(g%DA)Ak-M!^1*VUPp1;7tgmVd;pHm3h-4{KIAu5>-TgL%^Dc+b*y z=gO3erZc(RXHs8Celi9^-zNzx;juVUBPh*5yqo5!wn#vqH)SfZJ;vKX|;UZ6!o)toJ&@S;v z7b+Tle&4NfdaH+|e{uw-n};0q_x0|Z(js_1Cx{eUA0w`%T{B->wb|P6^Z)9e!ZTxd zbv$|b{r8@?ddoB}^`1b|Cw@}4uFndO1p?J)b-Z{>Os6?lx<2>#g&S=oaV;uek?LS? z^RuWuH7;*2LpJ%f!{ym-jr58wpXgK6DxbUOyL?k&vVK>nn-kHUv2-}(#j$hxGs~}q z(C6H`-E4{KT(qzFa`~r{Y)@gB+X?sTd*3)oyL_&a?8yE(p>~l1d24@WC%r4*y*@bx~jr-x{@!-^R16#g8` zS;1CIFq?Y3Wq9hf_>D@|V`V@+-J$dSroQF=1u=&&dw+Dbi>CiY%;^KGZ;1(Ax+@XS}GO}77RduFgA*xJ=&+7V=^|4;e@**L*_M{ zoot;_aU-q0!?#A&7rmt@Odh(AbE*s7_#)(E1n%=bj!E9sQRu62opVZy<=T(QTLt(f z3Scg^{RVOE{@C28%Sinve~lV2n$h`kxVWRU8D1LOp9{wK2Nu12GFou*N5i`zSXEt#Pt#s-rg4zd0IxsM^U zliaO2hc~~LIGrPA<@Y8QBX?lY)#TGh~db;&q2tR>y@&Wz>E3~`^-`|Cm*q4T|>DvOckRP?TDp% zOA|MyC#OZE)8pUMePSTlY9(ssZ&%ha+}XhL*#n!Q~AkT_EQ7vQ`0emrsEWG z;U_+QO9k_468DsPOevBbK|?sRShSozZa_e$|C)Z8Dk5Huz>0qy_|&h&kz9dsi_W?o zc1j#;RYRryhAG*?#(Lc&u966R9z*k1Y})8`DW6|IX7FdC;=|<3MdrtM9FZmU1gZK1 z0$i*+%+p6mF%m&~i$Q+N#+=9%~G| zVtu1y%d(9a5c&SRNHRXHJUK*iYeOqdpA))Y)vddH5{1n&*5!T_+BFH!U}geRvE)L$ zrBmts>?e`cpV*u9^Yzv?(XbN>3(ecd>831kFre@Zxr&0G9<9p9Vp8oqIpZEPvUS9+ z?wRuN37(h27l`6ocSk-s(YR_MXjKs>rTS)vvHom+l#GEw(fv+kr@M zsRbQTg5K#sy(XYeQtpw+n}3}QKFKTJK(nQ= zdp>TC@4z7Q?wcqa;a5c4?*5hr)i+%?8Ma&gcYglfhH+%hx3Su((%aZNqg(l`x>3F8 zjcypZJ|56SRR)~ZO$1S{iGQ!U`>O!k2edb)gTH;jDm&a#r$7%x2+YnV);BcC?SC(YbCa0k< z=}SUur4>^v?^%g-Wh1}KNtCZNFetU|wsHirzSGu~=;uAQFka=TF0O(TGd$GctO1K+ zp(kAA6&pt63dnC{-Sesz&?<|}u@OAYVDsCxkamu*t()-u`!w6zSdyRJp2 zDGUAa)W0s0U$ON!W??+-f}bOs49=VAiQJed|D?Ue;~l21u2H~Qgw+FI6~C(WEJcDkWeu>rTR@pCX!u$Sn{fj{R(*Uz+YF2a_h zCNjyyzPjPO3=pc@GrdBM8#--X8))$ldM-599`3fw0#4)vH99sAEoB>hZn9bVC}ows zgLFWARxEB^`!HQ=c3}sg_h0Rw@EehyUAz`U>6YbCez-9EusNcdHIXl-ld8+D&6QzP z`lr1An;(1juZA~gZ%Wt?e2m)Lfl8VP41~45u(L=>yAqa~34BLXfp5Og=S&lHmdyH_SUsTI*->sBRZ+Gsi*8afw&cnL+!>J54o6ethpJ#Vec-+C&SkL!rud z6Pm}c6u)Cq{j$UBvjYB&q|H%tQe(u_ob375(o|$xc|E}b4LbtsofO9l9H#^n5k|RG z;fGHdyf$Jp4bm3ExKqx#`%Epntu0G5O^`38Z0TD2PM+7*tu+ggwaHXz-Nc#>jIpF$ z2pd-=ZeG!yKt_K0@C+;}!hdLu+NTp9r-R4K!2WV1uw?Y(Wvvclqj4g-Pjb2EMWTI& z2L42D3J62Q!my6m+$1WITM|ckgbBar0Oi)*KkxvH2^Y&^-CU1U;U0k$ZxEDO?^2Xh zrT35ADYiL+Tw?J7y^Wr@19#g9Qr4BElFY&wvl#$VJF3!0(3V-1T5tzJQBqDP?@HML`NruLn2L0F<0A?vpQT(zEMs8W;`+|kW#1yXGyzON#40> z&rAB!3dJiS;&nXhRXy7~m;?k?vKr}5u%3ITZ-_OWe<|9|!t*0X#v>nuCi&kL*q|%Gj+ZilA;WG7U-GlEaFbW7|#Cn`!S3nD!9t=+nWUww~* z;RTtU#Z2^+-=Mlgh>U^w%7Y2aKTLi8U~iyJpsF}4&KWR|TM@5inOK(_M_`I5ly!1?(0plYP9o|wHYHzEAnZv47>(+R`EDm)^Ne%X4j^=!dsD*9tMRo=g`2cb zZc{8L@uBR#Ja9MiyfdVdTeb3e&Z$}lv2~c`DKNNXS1G^WEGs3~+jJ)`JUu4(*U??+ z?H|!|CoUw5SgVdvrLN7jk_8!7oM=efd7JA;PSr+UrF|&H=)Fy_ss*OUDeApq3-8;G zXsL|@pjMSvYpF{GYk$0}kuzwC?=Mf_JGWU_(NlgaG6I_Tq)n^Zs9X(5Ns6kYP>iZv zts^~D!}ZqJNj7n;azME=%xdPA~Iz(zLTAt<-`yJ1GNJX3pUp&F3 zdt~kbrrvV`4VoF(;2G8&uZhVLqe{EC3{e|Nykaqe^~>c^IPf84ZUc(UMB8@7Ul%`y zQfvt#u&`fVv!Maa*#Jo!P&bVW(5$guc&E2%7%5G-IS7`LsInc3UMG03^+yP^*|}f! z$}8nt2hf4vnCIV|CD8b_dbgh=Q5**f8UHaEjsmJPeW4eu3|-=waklo=Bb$MQqx*dN z`L~b9_I-RWt*n;7YK(F^|G|p1{_?&0ozsnkLZ=SY2t{4`n_uT-2v-lNTh61IU^)|j zDS)Nfce4+ap0K{3MHy!Q>APo`e?!G*A;ESW_m+D2Q$O0P9zX@6FOyp9M%hof7Za@( z+EM`@R`L`?X}|b#C+PQRAM)W&_RvxG`UYZM0>BgQOxR&EYhO-Z!WR!@A+V^b;AH}0 zMr%aE`nq1*{l@mmnoR+)O znDh4Z{L-7_PixBbJ7G}*#dw3u;1r99>eaGH-+Ay|wS~VTH^SERVAixm&%DXAA}FbR zqwX0eo76P;3j#-QmbX>IABj zwQ*_X@9JDCT?f}PXPmuDJu00lmpJNZ7B{B~it9R@PtzKZx3Pb$v|j`$t&K`$1BOmJ zasJ_`J|0tFy7RKvs;oEa%M?##i~wBs|4Br%*3GL!k? z$tLBo+DK?$o$@2zaY_Ot-%}Hpv*-8eDED(yv-O4Ul>>X00tTtsmshC4fy=upRrRLd z#0z6hJ2i)pCxP4%W$6y(5?w4ELKzj$jHODQh3aQZsvUm~e#6k+0}8`r;Fq|BCyf*t zuatjcBU8LJlj!jtirrz~yw+`g+1tA*;LFRu(Ip}MX8dbTsc@@l^9@Sa*W$h8E7{AS z7Kl0-YZcrVz@E(>NECCVrkGOw*zHpt%wZCdv@$nhYgmif;B&^Th@uIdG16cf%9d(Ld$^z&gp;CnB z@}kkg2f&rF0;`G8^JI%RFHn?x>qY-TRG8O^5d0O>C*MQR@)9ZH7crLQY8!@kRvy*t zzZv&lsfJ2&T?=kGbxFV2Ch!L&PbbOITxiW0e0BtrY~|GVt%M!3{y!eR@u??Umnf&{ zw&EWveozDBxfoCiWjLuWk7T;dcLalSadSyzPq08ZS`Kc=G4OU@PPcZX7SwsHEk_rg z?GRWGkX7C(O`|51Yn$ozDWaEg*`qraAP2p!-L=E=utcWb4mtKN>`eSQLb^r2U3BM9 znHM_eh56iZJ)4cDI)4lE8Qlj9+-a#T)!1w2e}nk!Ps_jK!&Bqk{_d*wsD$aE%#1CC zTPN52>1g@LN4-vr_VD~Q*lYV<&{S#_^t{%pT`K}jDD{0$xvRe<>BnE%V$Z}r^*OV9 zR#12zK8u=NLcP}ty$RS~w`7#9SSA99LjN$;#K(-RrG=aRKJIs9$KK%gAS7*=$g-SOD-UdRBO8!{E%B3!2h?YhRhM zC@J1)oloEpa6WtOfyF&PgCZlbS-jqTtt`TZQOwn@%KJq|2^6)8 z;!=x6>E|`t=>_Omi|Kc0Ty+A&G2&H*9+9}vorEoaxkn~R0JiyyoCtc#s0eeFq{|p5*vCV=kA*es1F4lQr~n_ z#0;>rk1bIasq7W{N;v%;pEo|iVs-ht5!xv6hVZ=9t$nWa0qLEc)y zVv(W`KOLsz1D?wffdyg=xPQ}uS?`nq)_$frSUWBz0-3hu4EqXIS>93kEn_|fK*j>V zXmMz!Xcg|N13q$A&ScGO}XMtGPl-W*vgoRj)H4;|pRy>o(3w$N?jJB3nCadcnk zD#PiBYfH-n?)$sLEP@ds@I{-T{kLC|ubhqX@^Y~i;2xe|WaASnl6zWw>+Oj7=qi-4 zu@q^((BvWyRx}v!7C`Zc7>jj~X8;#tSBfl_6*Ors7TMKi8bEHvN97K{TC}`f>@K1^ zWWEiavM$(#PS^ou4^~H0uoK&m4c0bV=Ru2nY2+q%is=hWgCrnXoh(aqjvjR=2iGTT1X9B9u~O$BC+bqpuL1E~0g>Zw!d z!_P<70l;G|Ur#xT0#=7bhRFemjucYh@1G)r1ofT?8RzuxQ!wE;oQ|PJ4JW4 z_RfBNz4EbG@igj+Qp9n&X%r%fK|?PsVXNY4;RX^KR_Q;1IVR4cx3NFK(ktpXQg|KQ z(3PigMuD{Ayt$lg!7bzRgE^YjWLKCEY(N+jx9c!uWD6{Ns@h2Y0fI+aaGewha4`p& z3|7?yFj`=VSwXWVX?r37UMpb4s1H1ISR1;)V4f~F*+s<>%FEaTG?A$kg zhCj#Yqym}__Wk?_c?`O%&&LI5y(*nB0|EEt1_O3Ln@1?T=}Ct6gJ~9~eV6-YP+Y6e zIO7awW@v=`iZtzCOJl$TCAM?2WzugPKo3C#y&oxnU*D3hucbM}^jOMkd91KD0563% ziL2a&PBjk3WHsk9FFBugW(`OWtxKSVhmGITiY z0Y^h zkV+){-$F?Fji?(?XlfInp{{PH-FoKaV^4V$k|<~YCyaKHmtrI+u3O5VS%+aI{~|s*Ufzu`+m)f-{&|VlZ`{1sK>*+b}M60p7ERHp;YnaERteJPbZo9&GkTbw8~3F;c z{{KbxE|n5(@dx>jrvQb8UG;D@@6PM9+3=aNVGeEaq?962lo@b_3>aq!frw)$=FpbS zN#LqP8>M(Yd7x{0Ig&NExstHM{&rWY)F(S5Z76(uiD|o{p6^IBMO{0qo%3 z^HOSM0yvnHz~I~00B&DOo(^t^7+2EbuD6Rp72`_{r=FE&myO`2|`c z>RkCFO5I$$@}HN=g+Vn5Ac)Lwe{fEfV_c=Z2~JxOpt9Kh4&f8q#{#ekUqyh`FbO#F z#45+p7rSPAWq?%1=?}wmC6dGuB4CXXr!=1PbN}wxAdi28Csb}j!17x{zhm;>hc=N2 zdW*FOCA#%0A23WjxZ6VyzlI57c&bm-qM6E1QL~0b3D&8B!>Ive81cdXL^z}P{?)k^ zA`&ItZB}tQS*i&j-t|}|foB;*M|H9O4XfmU92F-4(jR5e-YD@mO8gH48{!ID30VUQ z-&iXgG-oc24|bxF)NPSE7|N=`P5U=sdQqv!u!8HiJ3JH7O!*ulbHfJa3_bw>DDE(v z-Q?5GcrFPh8{JZyKhL9TKH%Ly*#C2;f5Zd*OBcxX0IY%+-Km3vwxMG^>!r*;zMD&5 z1^-@g17Skx>z@)CJ{Djw9@zgZ)mcI^L4B&bPY0c!;o@!fS)YH0Nt*hebt?X zNBB~zBo^{Z`d-E{*9-?J6T?u6`ScCM_#g4}4FCOWD|j()f(b>>!mQs8uH%y^a4&;r zs7Q`dI!lE`3w4Fpictk@TU`iql1;H-dwRv(yF!-p?1F(VaslQy6fS7|JhBODSUia7 zTw<#}dpnyp5LH2SgPNC&fqbSKajFqfAIku9t7W81kvRa=Wmhmt=X_*U`UTTcqR;vQ zW?gwY5I~9z2t<{bm*8x%t@e57G+x|a@jkyS0dcVRz@=A4!CB-yY-l=C? z;&p8rF;?1Q78)mDY!k`E#!SclB+VXjuIbnny{an|?1c*-eK>)rC}?mJ44Is)86uS+&zzO;3Z4C z;8?B%jFuG>-2fJ2+^;79u_q7iw=-lzKp}CB=tb7SDg|t5?f?p+$>p-6o=$nGkj&=y zL-t;kVje;jxD?^hJL5o9lp$VjLHtaaadZC03muj$b!n?M|nvzmoW{Iw;wcaWlG*8 zWqbe?*ywdwDMCLLb6{qi_r**}j{^3`Z#FfSYoVE&@2dawde?f6?bMWd}vg zLzYu`2|rmPi!ZELCIb6$_8i#e(x6pcIv~nWic>S3Ot#;!PfqPP8!06~3StIWtz++b zv@b8?JQn*d2?Kz5j(RqN0DdhP=FAXN6gvQ328vElBUdH^gzP2a0dVxo#JK;Y@c(0lS{K+e2^NQfvgncHPgBqQ7}mwb%Xow-l-(5)84fi~`wuxY52p^oWIYW41-&#V z1rJA|K=1`qRD$JSlvtYIeu$qr2>aBZLm3%CdipwAm-prg^f_we&c;!&LW?bJ@fsAH zSb$wB7=l^dByY{T#_V})VZ$dDpSVsLlv}Cz5zc-Cm}1Do%B@16>&Wcinn|Y}$!Ac+J-CN2 zcK2aaIQHe<9vex-xpqEQK_quC3I0uj84ycKafnJ_{U_flv7i49bOusLe&BjmQ~Dx= zk-Tbpb+Cm8k=2YJmIC}A+G95t`>!}~m|_iE8^*OZ10P{ZA1Yb^|LMhg!9HP$<-dc{ zRp`1{O4{Xmq$e`<^9ku|%VjI<)nw$CoqAR{&{}%`+DxqMAw+qTh+|yzT?S}`rndd0 zj(Wo)A!uXqRu%(w~ zMH*M~xwd;{*44~A^=D?_2|xTec*cwl*{v1UQ^!GGYsqoj$h)btM86=G0o|Pwq)i(F zF#hu>sR@YGxgGXA2PGg%PEs;Rb8^{prUwri4&9-DuG<%u>ZyG(BI&8aUIQ`SGNP>2 z=~_Uq5=n9NhWuWwRn`2I7iVmlW$TL#B$LbrRCGG}dlHSgP0MB+Rr7fpFit2<-k(x} zMvA~jzd4k-R6sr-eUp*)j*wwF=YrekDrm_)o1X6P*y}E}X+F+RZEmOJbU6QV0MzM< z$b{>Wu$tUWu?@&$#Gm*Wv86SAlZEE4JOHQ~>@``aIymVd;Uwz2OdGf9q&m5J{2)-5ebRakl4`hKJl4h&i@2i{fN<;y)=d9<*HWvm~^4)ROg zTLRDl;Xau!Nq5i6kIJlIgK6l#HKLcWKK}}goW+YHdX$lX?fD^IOj@oI47&Dp<O>kWFmVUf>Sm?otJ?&R)Q%Y)sU>Qi z?~_CBjCozTkFiDCA~lhL5~f0WvkwsAs!@1RN<*JHXriQM<3HT6*BNq4>hUB~w9V~C zR9;7+!#3Y{N#u?PS+ed02RP;zesu~-Y%9!vCvW1J8%{W;=E3yhi(-8G`0RliD2|oLzhYd6%ZGZkj+&ArV^yUD&7jR%NTkf)78bh->ctK ztF=<~TYjTXn9=5)qBsoFxFw6mJeFd<1VD@g~aSreSD@RsAi$6rX|Y)`KuSCs~}^|j+Lsukuu=A0yfA7 z*&~y!>KWjtl353tYAhA{gmZ~^z&HH}%0CqGyJExuiL>e_5rzxmWCJHJZ~6^OjBF<$ z8gS2c=md-^OGmH9$c%c8iYc~6?+8Q~nE8ASP})nnH`Kd0-PF{0|J9EBj&t1+a=PON zK;x7C0hBNcBIXq`z3Iy06P+dlL|sG`fX(q=gmIw=N+=kW19wu%BsFJ$_$c=W3oOjmhC^N>A+(<_*5uTyXf1k^yxE1Np%8q)4PX?{ICsN1}*q9;Y?ul-ji(TVCvk zj%;7icRbHxJARzKFN3ktDPwp)soWZ~PGKHbDJmtywmdHV8h^)qyv!*618H&Z`_!Dw zZtPC@H|j@|ca1huQ*WM7$NRyf z1vgv)<|vMs8x}mW=JlndX05Dr&rUy{uh&N#wMLRRchp=|9+IwS9DhO>4FqxjZ&Fpn z-`w}KWm@Cb|1ubVkc#WMo0nPwB$WlPvqVe?1R=fuAKt$Duj)1X7xAEi*r1X&4N7+? z64D)#3Y+dmN&_2z&NwQ2f8tqd zy=(c&B=j)UE|m+~cXMO3&&K>zoo1fc?0@sb0eo3ElHRXpAIwgUn0-*OQfy6#$ZoYZ zy1D8W6*{z=>~v3w%%yY-UY#f2c0+O*A0=x><022}9~+ebW^fY7=CfKpgVuhE^Klu( z9^i7f{8jP0jC4vSqOiRnhf0x44D0s$AWHF`HbrWX)D&A;`J|Ip=gNmE^K@Y0PO6-J zjCr4+R6phM?gN>4f=8wky|$5j6us&$2J05-!LN3rP^ApUt4+x-dw*$3NEc%HUbWz& zQtbg86hVH)_tzq9mF&H|t>txK$gXgDFg;MaW|mr-_iiPc;q>|{|B{9ms8b)5$-}H> z5FI86f}<5rRijTP>2pK%ikc4CXW5h=^GEt_r!r#)ga-hK)$u{>DI^q(x8&&O&0ODypahbu z@{H!zOrB$u^{rxTdsAcbYmQYYv&J*K;G+C+n~MNI8r}%ZH6KQ_Sm6oz(tvq z@#g9>;8s_@Tm`UvyaDolh-#d6Rz}N?et71U9RacjB)7*|?6iPGmlxzv+%ggvaaK7J zKVZSUz$)oU%=_xjT3)-Az~9!MjBv{ze18F&=S;*`xA{j?g?rbb-@UQkmu7SOn^c7P zQ?ZEdK75M;Pj5(-^fn;swl8Sqoqu;}1RJWh zGlrw$BrJG7VnXt3$upEDC)E{(Oi^7gkXd{tUf4+-AiDgT#3gHFi85rKAA#FX3UHO= z_j3G0GELHT9GU1ob{r&{U>cMsLn?e<4PALnqWvahIRxj%caGqO#O#>{M|@eZaqMNS zQ}8*MXdfHdC}I32Wz_oElWlBMN0Ko*sw`@_fT{eaZyOe-$;|W z+bM3?wHqCRAjC>NA)9kxhGboC`PFMxK_`Q9bW8y3`C?43!5&9&`tsN799TCJ8Wnj`i~(h>(@KrsX}e_R|xN^JBs!r`A06aw-0Ne=lS%aG+bdO>}+BYQ*By zRaHIl`)@rLPY62qX4Sbp8GrY%RrlqhTea_wgWxaw1f}V|NnL=n80oo*t9*AI4APJx z8SCYk5ZK}-!`t|}@UZNCeE7tAA8+iUr#qXKLhs?gwEw=sb;q>pF56p^9fH4mMOPg; zE^cQ#>e$z3f+h5c9u9yOznV3ur!rF0j0xPl@Eqdp-n3)>D-R({5C6%~dO5iL*0T;x zVNm7SdUOk7Ck#4Xi=|?69)2*`{Lx0xp2(|Wr1*1+lHrLv#8fnEhawm#`h~p=INPJM zww7TgEBqN#d_SY?K8wAVe3U|vv0e_PEkia_qLg2bitcR&+wA^l3`g@W@RIo6ar{J8 zAh&2VU&=snHBjnLWyf4ju00ooA;ffs-;sVG)kO_FChIh}N$+)gggPXBc$hZL_JSce zYqN%dHBnZjT{}IO3SJR*Xn6wnsD%p%u?Lf@4orNB37hFDMaffkOJ}Y5)7auYvmIP= z?KZ4{BhKeVKG{0)pGvB8lkYcs7(~fKq+avisY(ZEysWdP-CjJ^{uxL+ovL3;*)0j2 zktk3ewQFtgK5pQ7GAdNHZO#3CQ7*%*7SNbktxdArlO&tfD1yjf^%br{2!hZocYj+{ zuDt_~Qdy)c2Xr5pFIzAqjunlSPuNL!VtHFj2QmX`5}iJ4*5RH3CS34&`_QobC!+Xp z>X>;d2F={MxONhFO4Ul!-$?O&U{)kE+^$EV9OJj6OsJum`?5hDEQRL<&XmUa$1MQg zTVo{ka< z8Jgc5*ZnLy!i(iyU}3)&(9Vvc85-w-*8DimKsKoEh&{Rg+ihrgnUjGc(GouGU9Akk zMAmXQy$`v6fJv0i=LH?ggZz__)%9wZy3`k;s=sz2a0~VPtIhRlkh#vhf zx2XIPJIW=v$eSG*?z@{8V$w$z1lLBg3TlBqNeiJ@n#7`(=r#F&j#Wo!5^zLCiX!)c ztaw19c9Wp{#cBNBW(B6b*(2y-L_OydC2c4mKjxRQERQR#R%#`jH=;i7aQj5{*3IGW zYe$^Pjt;O4lP8&YBTMopz3qnv$NiCF? z<#9%GyLkbf%NO4r0G@BOPdS9$nMLs5VT~Ks3dwgr7T{0-1I-?EISj>sf&bYfPKYPs zu&+4Yow7PhVO`TVvRT_0Q7L2jHpzY-;V^{MyU% z9P|DL&@}Dsw=`oyk}4p6@e%HM%lc`@cwef7`JC~$rk7lJkB;fD=?iX0r4rPWs6lVv z1(==1FPGed8&!nM0NRv!hw*ijQeF)%?@|oQ_n9#0J43tX*$$9D+f)zni02-m20v(% zSI&rs0Nxb~z7gDBCyP$iiwTGGk3f_l1EWK@XFfpn96!cVl9x0L%7NVB{K0vIn?8~$ zatHqShPRFE`ce)OPHB4fwF1p}Kh_jih)= zf|ksnPHq(g2%Jw33J+|>s~imVf^o46oAtB zTmh%9v%1<=c8G9Xg~9~lip{oMxBD_>~pHmC}q^Ru*oo(NAnv*O|Q?N^=YG7GY}bFnYe$m&!m4^ zo=Z%aV*^CP?Q%%hKSAKf8>!j1?yMjZ!aqYFvn!}JWZbtwVqTw~N@c6(Ork(e|H-O~ z{?s7Vmx}>=f>lpi^1xqFC-kXD=#|+0WF+f`M}e_7zrxpt^hcTduHAHpteRH-^8(;S z{0w`P6*7v<#5cw*2OjH2N1wa@pZ#WpY(A zUlel!E__W&5Tw^l&Bypqg9+XG`0;l=zmzgWv4%PL*Jha)Gy6^C9?8|OcV#?}m$CHK z&J0)wfJ6Atez-JCAz1?e92Oyfz<#{5)qf{(s~I zTO=f)oXzS=U=9`MinT3IQo_$Vr|Cs-X)oc6QwVLs$ql7B&8W~p!y-0oGr@g@Um`qw zM=B?DF}yz!hdwHB@RS zoDOeTv#{IK{OL6dxqTip>H6fBcS|3&mt=L;zF{kRX|6TF{VA=AS$&Rk6~HZAkI=}D ziqSe)PR9*Xhg6lugH54xUtSQqxg(s6Fdk_j%x2yA$B^p1-W|&61+-I{=aLM33KU zq9V+mJ66-Y)*%>#2=bP(cBNiK_^L?^qqvlh2VVfC$A}3}opZjdz1?^)Ve;L-nrf(v z6s!VU2UA>@V_(gq2kxF7p#jIpB;>s^e-&2#ShGjM=LAooQ~F8tEWP;`Bs9EHKibxW zd3_NREc*V%|)dlp8qR*-)vtkv2)ps!^*5h)I!}>@wC!V2;Emrvi51W zbwQl=d{9x|;(I*NC^6@Je!l$K$~Uw+vY?;!+>Tt(-8u9&Y0Hb6$Hxnk>Rmc@TmXF-gmN zImEvKR)aVyYts$GlRK3K1{*tWjxyDunHH>|E2DUhXs$cI4-HS8%#6Ta68=A^>Cys#gV zY5l%)p6{rp!)_ri*E~L-s#`bQ=S4D%(2t7%H!9sA!Q)fb;*dKDSL%j z2@;u{wmX;srQU@<8?%>)UI$9l%)jck$GSBo1nznU^`3qE3RnNp(kho&MuO7z#Th)w z-!-qCY6o8A(HxMzI&YQ4_;nEkaD`;k9F4ixGa-_td3$%m(6T*xmNfy#6Mx5T8cIVS zpFHRnfG$ZB5tk?Mqjwtty`wR^?jp@gB*dR;4$B*7Bb^To0=?UbvODBVrkk!iYE3v7Jcgl%2B9CC7VzEyW?8r zykF3{$z$$kC*Nh_nfO~JFG0)Jz9aes?p3nBIRTEAoayn~4KFSpXgqy{89!_Qil;Ec z*}j=w7-i7#{Fp5UyS)rtL5V>b*t%yYue1r37gvU65_g3dPW6U&?#zJjiIBpiAf5*3 zMoVQR{yoYJ$~W60F@)485HewKPvu%{(7W3fLFM&VwN&&9oIHK{OzXIdyS{Gu0g?1f zirHsR$Pkp>|IT6R34u+cY~_$~o5>5q*Qe;8L(T7tRJIofT8(laSK6hw-GRID1BdNU zBkJ7;@u^P_=Z3J&jK~&%&|}k#zF(njHb0(SBAejfeWPB*Pxrn91P&E4kLS&-+Wgi8 z486YV@_biNv=wD&CA|(#_Lv)=8gTvG_OrpUkN}P*-DSQ9Cm^Yim zlwAk&je2E;3LkEeq&@}Y`dl@0B>li-K4Oc>w2TQ#k@)q`J~(#RP=wB~o^<4@+W3&{ z&#@~k+HVUTADGeOlC6H*EIeCf9>@!7UBlZ`i9XYFatuWH+ouVs)f!McZTUQ|z2Dr8 z;Z{y{SNqP+4eF}O?gM9kn}d%k#(AWc#sn#RKikAcZ#m{AaF~Yg0!eIXlTt)ejrGlE zd?hcyO=Em|@&yuN#1mkl>kV`%e#Lfod4K5YL+*U^mPk_nt7kbEWoVCssu~gg=4N8& zxsB+?t^CyK9;u!AET+P7cz<>D*;o?&*<^l@m)-w1l&d*F%@2;0(u7~CmBWB|j5Tms z<7F3}MD~Ik5pTc`%$qVhe2{1}pMt%rIG5D)j+TM+oovmteC(6cTL(3FEbIfEkFJz; z1sejA=F~K#6K4mg&yI6kuKM|R_%xBVBNW6jk~8r>bGk0y2;wWHyq)+?M0qPDEoVF@ z#wfS}mMOzU89H|2CGX{J1=miQmDv=U$yG_PI9uqa`Fx0aq`bNbhNLdw0kp557d@t3 z_2fOB0=tczsEc;SkGD{b1O(ryIkG2;Hn$L_HCs}rE zSJ5Za#9D@wPi};`eNVS%?6bc&x@!ESUYcg1!XeAO*oXWU!A=yta*OUjL|5%KA~%8b z*VFdM`dNN((LaBb&mUen>VaZo`GF#gwA(wEV$-<+Sv)h*iPnI8C7 zJONWm((mnF$MFOLW$#>~i$kSq2Hco@buKBYkuv$)IR@Vtx2~ zYGMce)HrKB8<4#X$GO-mz{zhwMa9kMhhDJTC;zGqMc(Rja%0n!1!~qmz`(eUJ%m4G z>k-Y33v&w>P)$nhyfZ~ey*__A!`(FhDaGv=n!repM|W&VEs_a)z*+S&(lUYFIK&mo zhAczY{v>q!?)AEwwi6L!-?~4$3yW_a;7E|Br%mqhE*B@`Vl#ThuqMDROtuPXfCkurElr(~HD>C!IeT5b+ zjAEdgkWuuK)I(bP+)kX-#~wLXDa9LgNz9O?t9 z4u0eBwz`ndBt^whS=!dUH#6Nf6RQgVVY(+xcNflep9g15 z?#_GQLjN*XmuTfm4umi|`Fi!9zc}|okdb~JgQo3Q|1KzK5Y^Nr#+g z!Y);jCenpXjE=%r3;4-P&-8NJC@@a1cE|vT(33Q}9mT?8R^&nP<7pdTIS8?0QK;hy z&x%?gTIg||F5Q=e=Tq8utJ0UQ(y!FX{XPIF;*@Km<-7<4bBU7up#(-Mh*7_9y>LDu zObDPVk6L-lDw@~`vDTm>R}Vmzxb!>T_w+jV{Da-iFyDo-HN0vcBzZ3<1TM(Cz4@{n z_}aouf+RP2#Q}0s?8ox?M@77g_?=c>RqMQ29LS89QJzW1*QMYvIlrIaBM%AbNecBs zIyIWfp9rceahJ{XNdJjjsN3K-l6(n|S)AV)iW5mkKnvPpf_gI6!|5>ds(x1f`AHy_ zJ98qm%KC(2%bE%gXnMnk*MnQ!m|EF~RW+AfaH8#8D?&%bwOvwkJgcAqKUEo0Wu0qp zoHz8g6fQRn=>#qX*l2oM zjJk5-KT$N0nxxJ2?JH|7q1s0jr~?fn{8wxEgzXUrt}VZFL%QCtrCD zVz&+nFVXyYmPb=NSuoU`IxXBC3c-wqH`&wz>NTb%lJbhag(P|IwF$irx0pV$){p_t z@ObSZVCKTsi$K5{=@E_n8ah`xq8sR%Nar#kXqvy7n*H_46mi3br+7;1YqqNFADN<~ zg|F?R6|t4fjrXVVQ zu0L_meYQ@8y&;-fReS4c%`s)+3)oX&0ha#53_LQIoU4D9cS&Kq=ptr0oTC*dgK}~^ z#^%-<9t3Bqa0!q?B+p*;3v3Mw5xi!U;90xTGwI#%@{Q$442s<{n!TIn)`~gsvio84;FSwrPabrC(={$~=U8#!54{?dvTl!eIC|T+bh7W< zIuiX-J2jwll9OzRaAxb##3JAL?TR_5K^)l=BLOScSRChla(Oa6EN}8=2I3>dqRR6q zfnqMePQn+sq4_K&8}cW<*bBFQ$frQ9$ski5z;Fw;i4i5I;!w=|xmIpl1 zpBfmvxRjG)iqfjGnbptBG7*`K$TF-ECm{i7+wFB{6hQ5c`87~xr*5xMHV>oNTkW)l zIS^T$Ok&hm`B|fScFY#aK_-*$6l9-VuuJ$o4yhyWt%QC{Cw*htH~_$Ym;9O_AhtW? z=dG*LBDsX%bj|12zNLYbMOhYE*Kw|Bwp z4fTo)D$SR5WNF5~Lu1g_+zwHjWLw25iQ}u#7=|p{m*fK;Yd$#gS52|`DutMPty9km z1@yI*Wf1l-=2zUzC%j8QPA4kHML^knE56*DU;n{qU(p@WG00MWfS+U#3n~Rkg_k(D zJMsACq!05u)%%LRpb#I!YmCLAG#yQq$Q<6jtn=3#ZeuIuD&FyMWs=5|{rd~I`u2@s;`z%rOCLC4K=Kyj6cl-vh>F4caw?tz_z{Y)H=SYQZ zKu|F+UK}Y0vN{F|6PU@mid1vPFC@@{)G%}%df{HL9W>=$=4N}oz8?{e;5yk2kE79M7Bzx8>w zg-4O;?wUoU=Uy$5!th4g;S+%gO(oJ7mp|0@x>EFXk**zvwUC02H*oJ6LDZ6)VMnkB zusd$q2gA{NiYoVqvpd_Dr>-!V=X;QKftWcaYTlM_ZzDjV_XqUTzCA;rXkEvS zrOu+fZ30g{=+XxY^y+8qOOF%91z+CJIT|pZ;AVLT4v!liJpMkNpg=U99YMyE>F<;f zqt9d`8mK&(3~f3D_9ZgQG;qRms6g^Btoh>o3hfl0A|_z)TGK5|twOK2&pBlK2_tlr zZFZsMG_XB9bqH|tSkG@GNgE&A$P40_J<7P-HZh{uE#AlKwGt7MQvpV@BFmu%YvHdL zD)VLT7Wyzu#9ogSEQF$0#-x=dJi&NR>Hed!Zb0CRruo&6ix<06Ec|gWu`5w8(B*n7 z9w9AS3z6UEO;YMru}J-fo{ZGKKY5!T*^IR zOLYPpevZPd(qXkmKc1dZ{0^hunoKz6Nma~{(vmJNF)xi;&-;G!WUH`V()L>2ZFZ~s-qQc zkfT5|QCtr}m8OVgWqimPf;Vo`Lgt%ljRQna`^!#o-yk<-8>jl3jz?@+Z8P5o*fr+w znvr6B!>A`LmvyiqLVG3@X&ldfo3MKw?!Uq*cnc8O*~Nsa;(36;RVIV(S4)DJ{G}nH zC8b|aD%$M)eWXg)9oCk&-+*R~=~*k4^FK-E<5|oDWlDZl+yt4EP^wW2xl=jqp-qxr zMWvYh(&sb@UHWsA^*Xx+EUfx@E(m@!=bG3HTfyzs3>9Nk?n4m#{jrgbXhA`5WO1Eg z=Jq(f@_xnJ(6R9fO>kkP36EEit<;c#o1Vzh+#GOlc^| z$(qq}Ud2dC=Hzy(V-O!mhA;2$!r5AW(4wML-%LevC2#tJ`TM5!~QHFckMPIkt$UB;vPTS7G;MiYWX{9y$5_aQ8YY@Z4!rS4uJ=Iocb#5V( z5k=ND`d8{ldXPpDX8y{j){*j==|s+v{axwN!dnELW|SW`ngRJVZEaf15%|`^G*#~` zmScBo&GP+ne#rev%9Qxj$@!3^plo=xMBe<3x;oznlSKOH2mDY_XcHj7_pc!yr8Ojb z;C8%$*1YVZ0}nesEUeloeL1;K!DFL8gOd1!?!xwuWXL-4Mh0eJs%ig$1x-3L#ZbbM z8i1)!@oWKE`Gct~u$_!y_m|O56R+j$P7{FmKbaoJRxLE}%xSL^T*LnsE-?Uj?f9$ta1x@YV-dXtFL@<{K zuGp@6KnHToyJ@q!pyY6E1aKmzUO3$wSX(JCEkG!%2RQASKLWpcSwlX-x9!JdJ|a86 z%mbh0>5Xm-3Kc-YZlYXy2=O^SkRiLZ4coJVi>8OFi4$feu6dLkOd+HG0rI;u+sHYA z=zVGPZp93~9`3QF?+8*)!)N}zeJF`@1i!sE%$KAt9zc{Ihus+Gey@&?lyRHMm+I86 znt;j4que-vx*(fp?NjASQ3sKGboVYmLKADoRLe-P{_AMa33MDZKeqXjZdQVt?=+KM zM{ns5h)zC*sN0>mABRVvrQ9_GK&_CP?+UnxD0x5}aQyfQ;*axBDrVEcZ03!`Ii{;Q zqOI?yEpNft_;JK+eQ)s{-^}yl_)w%vrbVac(JmrDhVvTCONDRt?e>GW=;P`E3JAd3 zhEfYY$S36MS+81V(VJH)+GFi&wwEKF+YiK!Mddg(R}0jljR)Sy8I;KWI3r{O7*A4{ z)vz0)_-#MxPMG2TL`e!?Pn`V@+B2d^xIPV7VVJU4oz#WJ=c90AX@?7d^v>WaTlFgFk~H3$%je)W0?>+`PazURnSG^l^LfMa z29FN%S`XqN-&|aK4RT_d4lPq4w7@CW?9!m|=3)JwIT%))w7v8Oo@EPXE=T8wF7>#M z4DmXp=zKtY`N!EMn+^n3oj>(&iET~}_-NCZTu2?4`?&U5>H6w&g3DuAq40Gk+UA^W z3vwU454b?^^aae?Lcd$FVAE@Ltm%yBzPs{|Uy?sfF~lcE4;)L)o;dN6kfd+NPTmA# z(2z9^wtbEpu`*MuIc?)`C3;5o=y+DZj~%kS1vwJe*lekKL?a^lK)faVgtogjR5Akm zc-_qB4X>_USfzZ2;)Naw3x|SxJ(J^$lwbSEi$6ZL#|}< z{NVm#7+=FdUcvagLs5ZMNr8f3XWZ@7_0@11ID=Jqp6)|3YwFfd zb}tSY3Kqmq1ryHJPCY#RH}zp+WSaP$e@k&;OXC}>FCp6sHcN28KeY-(()79MvVTpk zhv)7r!lUv0s#V3e6A0zhG8)i{V|)kpmhmD@hSKyRZ3fFEx~R>5Ht`Tjw3Uylka$g1 zT?Ye0pM;hg=LHb|09Y5F8p78et$kVo!=`$11ym!!=A$6UVuSn33FmhTb10SR?%60y z6eVRDAQDA8u+!-{T5(2&OAhl0oiEkXs$<$eb0>d_49MOQmk7H0MZRg2(#%#nuHzB$Hq~$IPk?-v`x8WJ)DKFXc24X%-bue@^lwA zPZZvXEIpHwtx=f5+=v5Rz}RLR9783m;$V~V$cvXcvGhlkUOcJ_8Ch4^W_i_A_?$ps zFb5!bSx2DQYR0hYaxR;H_HQ^?`~=ivKdD4D4RR@)eup(5j89#is3iEP(LKl7s-!$5 z)H<;XgKE&@Dz$VsTA8={p^F-;Z$y`WgUWjRKnWRd3Pw1kM`(2T9rKM&my$AgWrxQB zSj~<$+I3ovy%#;Hb@_2R^LC;}hkE4@oSI{ancuSVs03U->iEM^`B7mylkSliKXR(O zy)zRw(UV9uw*}p97l1Ani`w<`azbcp*`(Uo$v0U#6>C+?76D68?qe-7?T--a;Nj#y zj8ymBAn0KpEK=Q5Ykil$Xm+$8xeVh1Ta#IgW0$IMR{_-d$+FU=D=MIIF{{q?j2Zn*x5q)b>edE<6^f61qL=l1{daYhMbzJy zVf`d1IEsFbs74Duj(>kNussJ;e962UM-|P|o*iJ{ae$7AyC-z_feNOxj%yfO2P38LKm;AlPV9IWT>Nm|3bPX1k+!S z)5Uo>I@tb(5h(86a)sH*Qa`}w{>5*J|0tcGhs6fIJ+LqZL%aOfhgzU(c~p`ANbKN( zaH8;Y#y=|FcTo#;%P_?eIuNesQb$(fWnFSCjL!=o>xu?O@vr*6%A}{VSIa?lgVP}~ zNe;N;A;(86#_nx-ejMfD7?%3@0+NW~vYxV_o#7a*gpE6Ieb4gIt+fF={=5udR6SSY z$%yo^a=t66vT2`18&l&N)OufsT=DShzLfBR=#LBskARj*k{nvp3Bz&OaiYjR>;SeO zt9)Iu$|uIU^5cMg-~SBV#nmjf7RLUv*C+wLI-UY1;b$oz+(DQ(5Dc_L*QP^h=l-E<_H)u^E=q zyU-yOPv3fsrr4^xl2ADMWLiFCWbT$$E!bQ747%SP4Ip#^SJ}<<7K~mz zJn{^Gl)ch=C38Xfs>DX&_`gQTA^#r5I(o=ihXY z=Qkx{zy5U>6|>Dw+I%__t9u6~`-imvEj8-*IYgh4@N@&QB=Dlk;1u*_uM&bc2=Dod z$lt&s__=m<(41*)s-e8c8WvlCAd6z8>Y!YIg3T)<2tPM=NC?BJENPL6Q=uMK(*k`j zy#E;B@kn@)bLpU5qwp!(3)cC23I6(!6nkUNU1an`=VJj)l}wq*U}ds)w;6 z?uMw}N73_F;7Kiu%u{P@<;$2D6$4YL83@i=)c$(bHU0%s(TBxL`wDz`7hHf_hexou z;t4p?J3U3TVXQGic1_p?XHg1~L-(})uEIJ0@ZVe@^e3Ma^8q_*$@}cOChSjpf|3*l zsvOO^7|{QD{S|R4`sd&On1bD$RT-v`fqWf)dl1pb!l3hVHS__aW>H%9Uz?Q|ItHPI zE>ivb&2PLW!1oqeDEObh;XnTjlm9nG{eS)P|J~XA-(UP<|G&Q9zrXr}_g`l8zrP0$ z@Bh~->Hq%7|K}g;x&ex}1xCu~8jQ6pzmE12wvL`Lzs}caB|<;-b`A>0*f>#SF9R73 zGRn{u;C3Ny;$G!b2)J-ckOsvL0yks|&cg2wBc+i~q@+X(towhw1|s0Zh?elJ?W=(_ zeuZ?;HBgU6(vC#xl$4}*btll2t63?A$_O;c#wTrn7tcmBwk*_)OXOiz;LNQ-hP>B| z`S)Ot8yBb9;5|5hxuC!BHwA{43>pa(Amq+0owS_mn>t2`8g?-8NP!4<{VMB`*!WSL z_0T&5kj-^HKX$wmnz)$7A?Bj1f%sq4@1W)8&nLD~ya5mzf`0_gpZa=#rHgyoqNvL> zGrE;p~)vOSJaH=jJS}iBaThDYL zelRuU?(f*MwL+J%<|L6Pn^OvFe3I|GC^1mdU9@C?El#y zz^8>jyQ@FkY4JUQ0~+(%m1HweUfVKDkf|iBqw#70$uhDZYKoJJ;6SG^DkBJwFel=^ zc+vhW;FJuCcbn}X&o&Olaxap}&~E7L#ThX?lVJUjl7Qx~?A#daBzhX_d<(Dc3C001JeVg;ce&f+a_!s4D$ z*$n`ev{GM1zm@EJmIxOIe+Mtmn)OglRM%4W>;p~lDA(U${N_d-o~G1#EmoIeVIRU7 zclY+ukbH2Y0W`Hv)z>YBJIA(l{th}{kKJcz`aT3~b<(M{yt)-a)q?Pl3G6B6$Ja?_J=ZwwSvptf|iF z{m_pJo+G>f7r3pyjVU=a$um4r1*au0*&|&DGN4#y^K)aEH{w|bcfDkt%v^!7F0NN@ ztFb|EKTP&_aa%A2SzG@IOoLpOxwHtm*L%BPyxPBH&x)Ub*yasK^7tg%7BViktd@OU z$n`0#T}0$9*Z3Or_m*C2`ld{6JZDSSh^`Lt=Gdo0zRyct8u&~_fmXL)4?6>6Y4FRq zh2JL7Vd5Rghvr~?OUdj>3}phBWt^;9>h7bo%La;gyDY!&HsRs@L2-g;EtG!rxicCy z-f5#hs+RNF0b{F1K`xdbO;R}8(jFyVlH`{SQHPson_m|Uh)pK~`F>-c!>J{Z_CTs?p%_yB47@zyHS6E0L_g3jp5fBry4TT` zhXHN4FPyjTXo=zV8$>=1?P7Gwe(eTXEAcg;+BE_^(6titK_&b9vSU?s0pu6#%)_y2 z>;3&Q14u$vK*CceOA}Aekmj zUSQ*)R!R|sD4;oK$CygE{XrUGH5?|~-`2C4SYcH{$#gnc@aJIsWB3(^0?&<_9qR&z zQ&D&nF!A>wbD?m$we1-Si~2#dJ^l$pDElIbDK#epB^AvqCv0x^(G*M^#dIA@Z5PRI zyP@ixrCVD?RgQ82xW3vzds!Egy79^?A0p(XQ4R>2 zqh-pJYRF8U2~b*BzB=6Et~)W>Ptn47$9NMVWp07LJJ#Eu*TuBlBR@yq)(I$s6$-Axx!Bb>d0OHFFSaaVqUKu3Fp;3HN_^ zz{|P4T?auZ|DcuYuafFPJ6AmVgp=U z>J9q|4=hXpcWhu<@Yj*67`Y1p0yZsV#t#!jAJ@1PdL%_tS-SUu5TgNL`%n3)bDy4e z^(+>@>lvg}F0p+KUk4vLwpSIha6f;TTP;8zK)4SDr*7xHCrgkG?&d}8{>VaK>aDtv zLOO#!^bnlNG|V`zb_}m_9>C3VP4$6Hmk@Nx#z+1PZo zD?3DC7Jc^2UpuSXk_@+bwEAHbt0N#;NZG=U>^*1=$?9YVIDyKXtDYqQ1~~^;y<8Yc z+%wR@FB-arH&mpuWCwF5Att4-;u+9SbD(!E>T$&86!k8gRpzFOhnA@h=0Qcr8V1Jv zH5XN{`vSakfOa5i|jdvhO`wxiC{sb&*&A}FUU9?ZIN z@PNOhN-`nN+=yY{Q79u<<>+;ho~4VDPlD~@I=YZPxvD|`ct-}}Pj`Ayq9jO7J|Rhq z6J@F;Xs<%YR%posO|{cj-hVUk2<3z2SuOSf>9sLdY?lEI4&y*Kb~dsD9h=l zkI%@_Gnq(F;;f%bU(H+v0UNoNi5p#YR!dggY`+tLZ3rHmUI*HvJ?^+N>7mN|ruUXq zXAKlvQr||^?p4#&##OUrvfBuZUgf)HslYV zvtOS^tns^QuG6=ss;@ziZnZ(+tF7gP;w*vR{vu7s6qegugxwHYR;IE{Ugta~xkA!W+XH3Xy(z*T*Z^Ll*kr z@N9tMMuGC39(2I}(zVfCx>ZdtEdvzX>d_=h37q~=UoZ_CZ&!k7EIb@1Vw|DT_dn>g z_yRuOmKt(%m-;QJA*AlI&ab-iSYic$u>npdtr^Z|(;&;d;y#HifsO(zD%iI| z9?!f-OlvYg?|_z>b5$=xO!aZd>^sYX*6x5)Z|oidR)H) z*@OCO>cN5$jR)L;Ae{SxWxcS->heoptE(v#MlS+*KMkfwfG|pPzlM&w5R+M|;BnZ; zd2IzJpD4h)!iI8ha`80YKOl&AWmlWli4_{U65NbK>`}eQk(S34oCU&v(0o6}a0L(^ zP4Ay*i%}!*`^(IaN%%gQl(hS4m01*S8F!nb{Em+ualpXn18#Yt+0v7cs_sfe}j*AehMO5s5&G2vlMT|vo^~*Pw_@# zO6br^gi4qeR(q;Wt2i5qLyE6ve8Lu56XndxV2+kfnG_GxiH6b*1hVZcgtR#*ose6d zmwbKYt`&koibG!y3s3CWunnS}_EV)~0c&d8@ju0!vlzB=*v4P2J@tUT-y(zR?&O^u z19c$xFerfAwzB<%)>|eXxMCtSboMe(e|&nPHhzLLzl0Pol3CfH6ngSbqx9dRvFeml zE9~t|sTP6Xg9s*s{X5iV4-@!(O>d(J5i0pz*V_>-!tg={b;T`Z^i%?&$Yu|cjRZpuVOv9Z=`a`|s8bW+shHk`hUzsNCv z5OQ>&8LPJ~C>2et=OVTIK}NjKq$Z76^jkEm*&vhMo5W94PAr5 zdNs^3#UVVbOQ-DUK|&>W*Bb%A^Ax%DL07P{{0TEjYz%dS9VI$=AC~X^S*v7GmAMKY zUT+4c?uybGAubGcZg!7xjg>Nz%ucry^&@mAMTg2fY z8E;UOjpHSW%y`5X+%srB{4ku3IZjjUq|FF4ElC3 z8#-+oKwb&gX+y>284cO*U9j*=NEnpFonX$tv}M(0!V!xDB5iEMG-Va$&C=sTHnA9` zM@b*_zd?sF1w8X0;({;udXwUzgubx#WMN&p8C7Q!GZAQA=HsB&(nj79Pbvryl6}yD zK!JqZJ~WB%Y=IDfAE)CGqzjPsPQH@$+;w!9)lKmqX-a@9@fo}4zRNiF04^Gl3PbN( zG434t<*cHv7t1j?!mC4Fwrev|+XbX5*+TWYsXgbnC&6oi@tT6c!Gi#g8qVw;@hXR9 zXha5&oKG5?x&J^8ECMeW*Y5*uK4x2B9&X5!wAnXA<-HnehgwMA=*vxNY6kD z8oOjI4O<(i)IpqxBj4n=vsw-AqiDqA8_LMC{o+;&pVCA8P0uy`r z$J`QNSYD|KOd5ICN*54hpCHmL*%%oYp;RwCW}N-H5juCP54LogfmU`CfN_1j-U(iU zc!$lYAiBrph?(i`LX03<_Vbb369tkJFlgSpd>|YocFTZ?r~$>(KevtpgJlBy{j?&R zZzlbU5aZw03Uql+u3-CNws#!6hT2Dqv`&r8nH67&zJ} zmRVG}F|D=(%^AZwqsCEUV)t90W$pXZd~?XmmZF9_H9WHEwzFb7B7LckAunr%%X6VL zZhvTR@f^!Dx(3a0rRC&{^i`rUs5EPSPyB#9hdNxrg*m52p2| z2eS9h%a#1(Bc;CC$JMJL?j2S|-))Je;iLq%4NtN7uMX2;S3>wD`{Mfss3ONe98fV# zPr5uCLt{S~>v=J~VE~cfOpi4nOd_bJEy&MVF5~N$UyxtqC-GslCeS z%S@6ci5=FM8t9J2sb(4aWNHis(I7z#!SQON%8th;H4y6NU*lJ}aP)+UoD99b4d_tffhoEMh=X=7+m_AP@%+}o?{6A?L8srPg^V(0z| zGz*uz^b|H-#4v}l!QtyRl1dE+&Y3z|?Kwq)GmKv~e06D4GifwPRX(gpw7*T|e;6gp z84a-WiENX&gc?Z1OrmpkOYq@D_NdO;S_G7q6cib>df@4aC`5Vt2 zs;G_6viE%h`nw$8kj!OpVIO8jyv+L8ZyS-|mU8z^+hb!dIt`VGPbo~PafZRsd7MKA zj#$?3LDKN}D3p*IQ3Bu|l67>=I%&XKPdFifn+T^_6(B<>`cA(OLEaBY(oks7WLEYeH% zCNPUR|9`UmPRgvBHUO*uS5~s`q-AT z05KH0z7Fw-x3=I|9Tz@tA`DuWcC>GyQ8;daMaH8Xlsve<0AH9iN0|$9fWh?Wdp}g} z4&hzr6N%dhvKUZ|t&)%XAzVeA6)+oXx_7PQgu=y084A!1tj-hvW-2i|g z`afLnH6PX-%OCC3@te{~oVpv8q<$w5LfgmY!Gukbuwk6hvb47T{V)U_VO0~er|#6e zGN!zc|Bmm0AYjWd!$JX7-5Su4(Pu(+4WJ`$D3?P#i<`PxIS|8kdgCF^Bb zYG@!{6CpmSj;_HPiu22EnK)}dVzQ?vqm@7d$SI4*r#lqYR+|mMgI%|kGHrCWaRUSh~U{*?%z8d z7c*KRG5Na7mf@56s;1zi)L_vlG7e4d53a4hMqP+e1t+>*jSSwe_!pF&XM(S?-keK? z>3e4zv<9iWP@Ffdq!|?y?=U{RBuI4G5{kN%P=`)apJ1WZ+kUD-8mY-+O$e)WY|+Xme5gm7z?2J2ANrN)`V%$KHonsE(F$ygx~B?toDv%t7A86}{wT$__pJaJ_&b*XF>0uIQgIVcx-OWRHm z^$z`M%U=2gL4t3!d4&?{>FHjLSU-})yb}EO;!-CFUVK^?;RBdklXw2s(3zFNu!kdl z;_4S*ji0ALKVjA=1hF&&UC^1a)k?X3j$Q88Rw;?E;Rsz+Y5O9+7C*$Gl68L5D@ZKD zAK>2^zm zD4OIW-EMxRPrdpGC^*sK8xUCw(J7$Ee8ewl zA{gbmRsvHPU3{)Opvb&+LLVBCz z>yaOiqf}j7!QD$Krja$OcthfhUSMTYI*j&qLe~^z-@IKb;!wf^YwT*ZjX!MDw z?GYmZ52fdquUxSpS0mkm_U|7?QwP|1w6tsthqQHtgBLe^sD?*_-zV@KLErsRG5ibJ zIXcaNmdkIlHnq$8pLv1W`o@-l6}v%zZ2j1ISns3!qVJEhUDR4E9M6ZY~QS{ElFrt^)EfJ&!yNJxi-q=3{p1|8BVN_TfEl2QU94NAJDBt)7!<_3?x z=RM!=yZ3&7ed`Z$@4eQVYtAvpc*ZlH@dZlh@t0fQ-!-pb|4Lc(EmbHDsOu-E;&FE@FmnsO!7vxj&o+_O73ST$VgbdQXYi^WMn-s1@r;ejfDO5sG%|+6c05r$LjTVJ9@2UNNp45 zOqQ}G%CQhJZEm4x$!WtRepST>8`Ydg(r<76NtZrC&0BEK*#p~zwfivqXJ@Bhf=XrL zoT&oH$y=c&a-t-`Vs-#ePS@yCCr=Ln{EoM9Em+?A^6lh@)e?b{kfA-iItWt1Mu&WT zfxT4&ZI2`Nf6z@0G^&d(SVYLx&7=qsTqfa@4(Lvq+=~L=6eUl`)DJ!)6j>Z za9!}BtS>2NtMf7doo9~i{aDq?zX(wu2e4z=A&HzFX!VZA8ic#T07HpHid37at}@33 z3NolA!cHx-1}hibO`*-jZ|bvxoOY|vp=l83L8>rP8$>*mWkOWjMR9*Q#F@8I_? zZoR-7D?5y53&4+*YF!<4YHLIdEtV5m^;X-9`TM2*43QEFWzl=rGjlF8zPQevWEADU zZJSdpKIVS0^ceS(Co!B<9b^txoSanziF%Ok8)sHSs? z(Z}>~>`+CS5oA#rG*d=ZdhOXy*zR=`)MUPWD~&XRq-WV{i# zN_^m;vGXXDP-I&%k;oZNCk{^?Y%Rj{jU&8Nn(i_!xD_pjJQaGed4acKVvsf;N4`b=>?!?2%Ym z0x_EGT*!S{gr1UjO<^^Pd)+B*-T}78ug8gOq>p$6-l%Jp zl!*<$AMzC06W2VW9(CdGgdRcq$1OVBrc*K$66HsRqEcMp(lM_kby!;pk{x395_K8)+3JLw z=`K>!-dvekgO%0qDyC<8oKrZS6dUxqR$Y)HnoZ~%D=!I%k$vc9A&@+Q+_1ekH!RP~ z*q>ys<~j8&=p~}L z{$=40C_2^^x8T9^T|2Rs`Hmv!s2&T#=9z*|E0HU0OF;o)$m8yod+!RfUUD7aF+TSI z^?9)9Lk(ffFE8HSG}249Ucs_XC35QBvY@n*2ox*5BK2n~h~)=Wa7*oZjG80ej0Uxr zQz><~+b{dyNjRbK>NC>n4;Lsn2T6-C)xgr!pa(z~>(eu)4><9zdad8Qs`nj!#IY{Q z{JNK_ng5G7b`5RwN_VZAbZ%U)E{eJ$H zF+R!L6A<%lP1i&D!0B@j>DI5nEuAuYahsSixJ!dG+KBfy=bX_{194}{#n(__szpZ( zn9}_~Wf&*X+fc|(Mm9NI1IK=FJqgQyjP(~&Oa+T}1+*_WJW*$~_^6_H8yX>e z+2KoJULS%PxIl~}t)tJzs6amul?m8JIS|VTu;>cN#6~v;Psf7ugp%oOkPPDhXYkz; zEqdx0M{V$lHFrdh(C@p4|K$IlSZfA!#d%=yRXfquu z@8xxMyvFP1=iI^TD*tjufMaxJzw3a|wG+OC?<*-fiwRyOlQC_Oo#vM@-DY9v@6%j1 za@MYzAYV4x(uzUfX6eGQ(4UIpw36lrzvH;Gfh$sVX)iy4_Idb~;u;#p2GK_fnL$_u zbZ$8O$5*ah#Jt~vr1+<8VPG!uORfE=%2>t?KKvg}h{-*=h#>l259B~fm45$|a2jpG z=s<7QC7iH;A9L}3wfe&S4|Jt>WF2;YL*HDlBhtoy9Q-eSb|w4!&lGL{^1ydGgo)HN z*4t{m-Mzd|bM6@br(yN*Yl9TplPqQ3I2c$eaNIulxZHCt!uv6On(*Vs_W#3-8H;y& z;nay^w!+KMc$FAb&*e!B`@P8BzPNqW5AM}J8ojwe)Fo3j^goBrKiauB``xjXF_k>J z?Edny72!_fES^OBVAXI92FlYj0Qd6SUfgLXwR3+IrW$vzzCGymu63}iW$LIwHX^dR zvQg~3Mj-qCI!E4YH}~>zHLXmkcgjZ-mvw|0V<>LuT*dTaTLk%@@NRD&vg&+8qif=X zE;*J(T4ZNbpS*fAeD|?K*!QE(VtvcKX-0Hb9X-;y+E;g`oYpaY)hIKG!-;QQP3VOA zT;9I(AKon2N?5gdYuuSG*5aLkO3d$^oj802ZWj)%FTQVdPwW+J>a@g-IF|Tv@anoW z2fM!>wL`8~-P#R!k-yp0g53*$+~hFsN`DU5O*hz_k@< zW#=%ln6xL_=r5P*X>HHB*)%niM}7pscLwSLjSr5oW0J?9)!oo_`1}c= za&SW4jv2Nc4%jo@fF5c!eLskK0(M{Nua9uU8m`o1p$oX={%; z0V58Ys}bVgu5@Ve4(F`hFG2BH&pTrNPbUG{=CiDXvk$}UUy2Vu;oFCOa~;^RJJTFw z=eG0RnwpR#TYs~Qw5JW3q*Hvm zJCi*U8GaK35p{GS?$j6-KMVDVpvxKqTRCB3H!D!f3>!yN&Qen{e+92+2-flc{&dl zQXdAE3hfwOBfPI>gE8E|Id7%Lwvdl(6^U?;@%9U&BsW6y=*g3-VDCzvyv&ewY3bB5+m z8}sr^Xm^6lPdt5|Xo9h=Jz>%HW=52-<{v@54;w^*CI>q&j4-_=B%dPz6=~brfb4q& z%1$%P+3erDN3&PoQ#*wGNqvs19o+foffWA!zf7)aa0JtZ28qqa1Rhpn!b>XTBrVb^%s3hF zcr(JKCu!BqDZ*vJDaK5!ou~j{3gTf~oOSPDcze*m0X8}Q5F${!xJvg0uFJd?ct`o= z$aL{9lMeh)mP<07l*l?L>u#aGoGM?MZa?AvXX5RaiWVw44WIQ*zjL;w+*w^(S5!FT z&+pE(s7*9ju2D$sHxkxO_m(5D7s8-WAb$MUlp}o?B$BSJ+`E4m2MDpC$;OTSLFLj1 zzh5qeD1jDr6X}mZC339LX>0eYW>>7bG&jhYrQunS;Sp~7zh57c_~z7bqInd~s!Ox; zIVqt-d0=(#Sc{*arAx;9mf}%uy>nM)_H&))E7!EoYpaD`+ZrmYb<8z!zBRXL-OHrk zxy&G{NfUqQKbK6a- zcY=2J+XiBY_-w9zLiswcv>->VA*3pUX(+GuokAr|kaZ+*)9cmd^K)z zww2j)IGd@f!BS@f-~}_VOiuCw+eqIfYrNFOrI9K$esg_1&gG#AD>>6T_w+XVVBWLU z)E^2H*h>c^QYx}?+up7k_Db90Sk3`)x&a|L=fpli0L#K%FcAB?absko94H6LSq}}K zTxrIQN4!RAo!FmeB%Y2Rt+Oeae4D@*&T#GF*_F+g6K)afJU^K6#e)ml`sK~XQ3k=Err3LF?BlCA62FJWi%%zaU9rjN{4l_46|h-yyG3Z? z>7_%Fu#b7Iz37ZQZ=!kaTjykRQQG#^S_$UPjMhq%t#dWVyf^4k!7iVz?0S|C+xs5e zS(n}c2ttDNp($AJN(0ZimuWP+NF=)Z2@9bS=tU=+$D*?S58!XL0&<#QAnt<7lZ{$0-(^X;Jkrh##HiLoH( zumhTOvi`es_*?5&+(|J1_eC>mjS^Y#86I4@;&(Yv%P(|n`Tl`XakqzGU8e=WS)-tt zcSExV9XKBwZ)X8tF%a)?0V9-XK3tgy&@#5oj@HE03L8}l3%?>F=5DRN{0 zamVk2o)mSI$Z5ob2G_mUg%`Ev&DCpa2EtG?$V=(e`I6nwdnyHb|7uR(yABLpb67H5 z1YSg63{&Zsi+P|2y6EW7^mCeZ-*KHVAM zAUR-C@8{93`D1g&U>#814bTX>vXq|J%7PMaPFnz;-&9fI$Nz#wTB8w= zt?K{O0i>R%8!=+<#eq9g&FBIf2^-kHtWBgBeP!k6Q~&vi*ig7y@eumMTLUmU*;!XW zyW{E?#GN^@S_K|kzssfI$Zg2cw1Ce_z@CgNGW+>v2n)K$D#7c+WyU z70wXWn`B z*QUy!v8X12!2s=ZvJo;tzvpZ9rBd!jHz2IW?ig0j-kLabb&o8S58WgNa4X3XuY88@`BH$ zlHo#@?7}GHs3wzZAzrWdC|-l4_uA&nPcd*unR}!)BPNV`d`VHcNN+W1dg#s~@gp}|&Lp41GIdP(61PVg3 z2fW@v7VB+@f0j(#Gh%@LHdTm)56|v*o&G*cqxrAqI4OWy*yihpM&4>vOuUJB=x>Hm zo1ix_k!(h1fERd&7KMK;T&m!EQ)aUN=V0=$jq5&ZH&leGY_KYF>v@3;@}7j-+f_(2 zA+~b(rUx#p`t>INlMA%>tby~7STk3lO=%I;R|Hdi+9tDpfAdWYYM8jZ3GA0FdssZv zZ7|)dQtrIEMe-4G@P9if3M?7#S+~auP5^St3D?E~Ax1$}f5oGsa453j{+b-u3^^@5 z)U+7cH^jwv)C7`d3t;kwF3vEeT0jlO$QhDi;1)DP1vV}}g_;|OGQq@YdC&Qers5)* z0o3)Vudny{8-2X-5W(t$9pU&QN5W=!54CFs@}|tE3WVx&PlV|-H;1#LdYb7tucKf1WlO(V!cor7&)EYoCl-|ii{ zPS*~f-s8HAVuJ0ek<6Gqm_%}t_F1CY8&c@;&#n=s^z#1Jm)XY}v z7izZi=iRUW@2=*H_Tev{s>yH%q-Dwd1ZMH0s3H&z<<^kDhilZd}XZc4{ zdhhiQW+kHZe2KT4#(|dBg)&UIc zZS-G)F>91d;(UlA2=Kioc`mv_#|ziFl>fL4IMwc_!6V{69QXGx|GauSquGNP{@W(Y zqc%(T#C^0bH;mj;6~NhTFm8VzxaY@>eZ1`}^3U-Vg(u66 zoppO5{%`;J`3e7g=KrUI9W|e}&+*>}d%-jMUk96CD&mHL*}o6(q^9$ufBVnR%f;X( z)X(2suhRg6_4A-;8#k%`LYGy31~@YJkOCN}dXPa@zytU^+_2-yC(VGO1G&puXH2Zi z_Et{y5jp`#!o>m|V!ktGi*0M`M^!yIKqQ;2@R@g9Kt{rJz_#DiqDRY35PG@90e)np z7FGk|{&~{ZfiEOJXbaiSd8kCK1s9MATr_i%Lj?Hv37x<^o_nx!Vg5y87Y{{4p@_wN zEuj+MESNj+$f;hy4sKlv8^$P#lmi0#_=fnlSKGK?>(+smE!feHjt)bpWk9lE(WoOS zP1J*1rJC@Wx#F-5=|EDf%P%l26A&Y2+vy%_2Pi7lf#!Z*h#=HgXj;}#2=zPqs7G?k zd*hu;G1TBz(pm!AK)Mvc4dCsL04c;9iYn`YG&2f)=cMVHD7$nK7@}Gv9UyuV1bNt0 z*hEP-0l(3Pv&1}CehMHeFNR?EW6r*yRI$!$;qZdCpWri&c(EnY zF){gp%y}K8)gm(kML%HcDpdi_R;&ydap(R{#Bg?A z?;qw+=@0yT9FKd+CXgn8bkZX?H+RD92>qK?(LzhJfR?D3Q}ZBaS_d~=^&#sbG4sXM z9IcDQf~e(MR~CbEHU;d5ISp%oFMA>-teEYMNg>h@4G9taO|I{s9p9|p(s^I>$??xb z8tUd7TUE_`zDCZLYKwk45iaLCH%7xIDx*@buEEDAHO6{CiIR?oZhMVIJz3}psu;@9 zA*d?xDz^nEE}=0Nl6e-Yf-r|&2NrL`t5a#?JRm}0T<9BVjNN%?cP?AfYO$rP$RhDq^Bj z8{WV)-EZzW49_{c7%31q*uOo_xpZI;hJVkl73KG(_Fe|TPlpnH7DGwc#C^b?QRrNCc;Kiw4}{?9~k>~AlZ zDB)4HIb)8sJTr`%7IY)20ys{~1c(pcf+?XLtf1%t)1wA8@j|J?pdk9S)0vfPU&A;| zRlkqwmcUYFwqF?gI=Qu!#3u;Cb|d85E5g%Id@iG=b`A(<7Ez81jYd=`GtKl7>aTl% zBs(=wSbD-jQ$s^z(&-j;a};+cp{T}-n1l?~c&klXpPR#?99GV>1)CuIqG^n?P>(28+Buphcxdh1#Jso|F3BSXPGS zUF=SdZw?T$%_o7oa?gHag0sp4Xmi)qD?z_RC(egs;pxtv$X0L%v{fgRidPh zuUs1C(ln%BBaEW^a+d(6B0R-0!>8@EqEtwul--x-YqIs16)4NB*N-$MKrLgAk#Yc~ zTA8a{1h_JB0PMi#|Bf1Pi5sA6qKJcL;O5Xjb(`$6?GTP?qpTtAF!i z06F0c7_CZJQm<_UT=mCzPe5MbUCEI+g~C4LH0#tBh%`D^p)RRJ8ni2J3~7$yb=A84K&-vo7ojj}~ZgFP)!@DXrVzsvwa`SSDI_uUc%k^#ay z?vd+W5hBS{H&g)13=hECeGM3GHqX0JRAIdZj&l9Ax$?Z4A_c`8rjiG$ciXdHEnv@cxP0Oe%1{$v)ujwA==t#y{GUQOml<{j>!@3WClrPx9Zqbt0@AK_zDX|&sMkG%U70Lmd=W-7+Zw2w9$M8WLxO2Pd-31}hu`s~MYV9YZ34zxr zi~777b%mZFq4Y2RX*_(=d0y%q z*9tlIClm>DEhfMr^Q7c#RbYL+R@)HCQChl@9GSV=HM(?yDcjkvB7dKjpDg97*vIur zgWH|33vPIN7SAod!Y!uVmo!{CBh=_{(ub(<%P13p)2^w9e{Nho9H+baU@1o8d{KiH zx%7F$w=~j*g&aG}2VTIRc?CTF*C*}ma=ymGgH76&cCx?o+V`4u+k_cz9R0`Dg!jMS z-i!Hm13s(3O}M-)Xp+kN)~maDlg2HBxHy@jsbptOe|q#Rp60$!7{`^N=q0_3tmBbO zdfrhSDUJQV;^`8Rr!o;OW8AE9jziN4mCoJYwl_X4KRmH5eXe3CMBJ$&UC+qcY|yr( z?+@LqMaD9Q=*}B8)WYx-RE8BC61%-Y@PtOG~{O}U~!F9#y*Q+mEQ?(3Yjwo5k zSVV1&Mr-$Xb$h&hQ9;EUv({ocXO-crK9U*jC+Ze^XrbTs?#6#dHqzCXTNp)W9d)*c z>y5I82+^=U{$XVo6H~KoYQ-N^K;L4x?mE5HIm7w;m{4d+M9Z8uci~`K%Nk_iDQw#M zQ{a0QM3@i(sl0MZ>nPT9yQICS;81(z_H4$FxF`~~7>bxc z!IM?iTDgW7!)g2iC-cH_(Jean_~+AaN%9A8A0If;piX|AJX{Jl`rzxXM$z+>IKq2^ z$1DnId`fqt$N=`l)^Pkb=T*_ig-)|g+jaRRc`QRaiyGrn%HOVJgf4x$SAmgIZS&Zviz*RA+W5UC!#gx8Z76T1c24Zja?;srh z{fM$#3>t_dNRbDOtq_3CvB1u42l_7J#f~5jzyxoibj*2>Aa^zalPjaBNI-VY)*HZ1 zHb!Vd5q9mNS)k*I27z2dd{GF@KM0Ffe0(OvqB9g_H__yG-^p}l=c)VnD9xUyF)vm> zw>+IO-}}KsJe_l;$**HG|9)@ZuFY#;1*gqtb+L-vxhy}Q4T`sj0o9XC0#Y;^L}5jC zAXpI#II*}1pTp9Z9<#C;ZK!?0j_O8pTTRrUZ5<2Pzb{-XFO)nE#K#;ZiJ&jF>-Z9F zVB>A07z!5Qbq5$DMLUOPVdnc=*lQ@N%eb1+9Zl-=T z{Q)i(n~F8K`93g~9EgUFTE#$Sn6t0QoE!i>@wCuej8tCw_Y-7rF~1y&eco=3#5-$R z%gFHju<4#{Gv_66d9xBJPFw-QD%u^n_e+Dd5I3tA&Y5dLh+YVtvC z#R!Ci5}^pf94up^7YQgH54k%MTD759is+^F1b69GqjW**-J;=wTYM8_Bh;p$IRWwg zqOhw!*uHOvHbv5_%1!283>Lco+5F|kSn&jTJ=-={;%Yr}ixaJP0Dhsm>CD&h7A2KS zLu<9<&R12~EZ+A;CQeaJ(RjvQF1%|REo4r&W4a{g+yVsg6cmK;8c!{<4?@_~wXS>} zz2WAeT9W(p<&$g=KD|CxKor5D!bVExvlrv9j({rvo>%>)^KWjt_1df^Bkz z0;YTYpR`eySVxlJi~U*s?VDUyywwhD1%MPKDq#J}qzw^Ny2chzA7f)Gy` zEKt_&F^uzxC=<<=e=8(kQ)JU7-0D8I8Sn$Vj<^^4fX&UFD zCzI?CrNx4Xetv2j6Qy%#li&$uZ~6AEah-B5v>Y6+$9 zy$0ll&KN)}A=|BPUhzG)xR{HUmUaYGIJ^gK`!ha%59mIFEFiw3@~h{nCr(eLYn*xb zqKo-Nm9?=1W0uJW-S#o3fFJjb*E9jp95WCA_|(=^{O3+oQ%(A;Qa{%);QHEaz<)^Z znieROZh)7{WTU+L=JBDVCb%K-y8dJ0;v}Z&VsS0u^brrp5B+o?ojCOzetLh(%8jw? zZm%>IVS|EJ3oOx9;8cSuy*d5-8mdP^`bD9BT-ARljoEe>TG4n>?;xbykyqD1F{}ne z^N2u8)eEv+!oYsghA7t;^vb#(X(ABoEC6Jv%OOcZ1xEKHRrm2<2KFtK45>%cJJjMU zZG5`AR6ym89DO6Ic2wtulrq7E4_l*DSpxGTPEPZ-3j~*ZeU=WtoDUSryP(4Mc_-Ry zeA15E2pv2|bPCcXny8BsnUFRpY|KNF6Jy~*D1-#?hv`VM4k}? z`5nm;V=XN#EYu)T)((Xp2~rBTKPp|ha>c`Cb4t@_K5T4jMC#hiS&&8)Puh2aivmdg z1PILmEuz5_r@159A{Y>Yqni~aPE-M7*NM4`GhtAhK@5tqzR=pM9yF$39JGOS<%6}O zZ}q~tt@PLKzoFzG&<&J~c)6SY<9DJ8=D*Jf9WOs0N?KT0pjv)T=>?*x>lPsKS$mMw z#NU^<2hzS@K&B<)-Wq^aG_dH$6NiM7G*k{*L(SRF) zpT|O#0FD@iWS9s|!^imcw@lF_flX7+mEr^t9W&8+Bkuysv=W-dtlxt zG$ufh+b#Pec;nW$)U-@bn~y3qTPyW^9LA7qz#SBw!9rACVAhKc$3aw>sWA)Lwu~vH zH7B`O@;aal1(~puMsY!^szdX_kaee|jH+s%2*Yk#+_jm;M+{v=mRo$55hx!vInKZu zOaK0Tdo^Yo2fa80)>H>}z2DGu|I7JvOL=6DI~1)ccmA6vq+@0pV`oR<*(Cvv*9Y1% zU&j-u9zz<86UkFq+k>trGX$K18sxwrL)w`+?`YF%KvuG^f#KlGoDI9j1prFG0q)^T ze_9{?N4bX8xGY3ah0D#4v7pOj6gvr0W+SMp((RKs*u=yLHvd4xDTzpqj5Ukg0Z7JM z+XKXly|dJ~fyN2OPA}b5P>}EXIsRan2Hq7kA8H{{=Om-cFiF$%y;kJ@hDIzjJxfEO%}rvM&AnWD77->hA-m59fp{8Jd|^i z)6Cll z)j)r)LV6BmyI*J2BLBGlvHjq-1|tiOJ^0;JUC%e1t{vlDyYi>#R!`_2)d zlmC|6L*NmlAS~Ji2@~W{&p&Vl&O^pnocza^_De`CLmBq3h>`>hTcYN4K{ON5e}4e# zs|0o%(BdmdEjo})gWIUUQ8zYr>Kfw*U%$#Ew?$Kd$m!Q7`D_z3B(`UF>!A>TxvPo8IzI7S+H1hXl{Y(qn(vKP zphopT@p4>xIz1?}5kXO;ks1`|B9xJObr-{VA0!sBGzgb{kJ|>|8W01HEib(7tn9bUC|_KjRI<g{zv zwU4b49GP*nAbV~?0ePU+_uK46CQ~kCo&Od8K&Cw4=&@eQ=X)@Qs(eB85y`16U@yY5 zu)f!3lZWad)#HI=`#&n{_*mP2di^J{F*649UhOyR3S7O_MZ$PI%QZ-1SPTLWjrpkMemecw>Ehp-u2czC#^mk@v0p)B!mo&wB z2IOWkH8jB7hL-w8WKs1aXt&96IG)8wJt#D^+vvuEk}Q@Wis_4{q0|;r?EBWx4n>9| z3+D5H&D5+AJE?t-ganOwXl1Y1A}ns>p0j7MjJj{SOYJFV~hnC zAv7HQmw;lrf{u=EO8!szVDeNh!sL^a5b}~du6%lnTT{PH-G33d92g9XAnu%&LqMic zQI}Z)ztvmEms{V%2EudAq*v_sD;2FsapwZ&D$Yza7@>+=w3M(5Nb?eaY1NFW49F{V zH33qg{&mav_xC)DKSV(@Q#mS)ECxRT*Z)-%fWfdG_yzR>F@j%;Bzsi|rCgVw{IyxPc=UISOjsf!=0Qg8YK{90G6>eAFxtB)W_K3Id zndnsV-$7fk@K*5lYN5>`XNcV4%MPI+vFtn$3SNGT`a1&AZiKo{^e`DV`YwRAB_t-_kMC&r+lV1pQ#)ELpIqZNgmd5s85THKc`U#dZ<^ zYJOYV9m<9`5o;^qAemnU_eRJbUaNJ~XUHJ|0ONrzf060sZ0 zPs7%Vz4KkP`pa~0UZ)KQXdxDZPapZI6HegM^3R(k7S%Rr*#+bwpCW#}8@xI@;D>d- znkEN1Cgh)>v;s;5a;@caQ4iAvhhXZgN3FvFh~}#QNfDD z9cTXC{On2+w8x=(LXt{P#Z9IByQiK z6U{xOWEM52iTepL&n(ELSY{ge5n3Z z(_R~#b0!3z!3&c2Jczyk(pR{3EoX0Uzcv8YBQP^CJ<S2=h?6q_u^_Q+&kgEA>%Ft&3X_4ph>L=c+%KX zaBviFBjzDL!S7dRMwc>o5Zwx}T5CYnxbyR*i%4!OkTgIYir~3XR{4S$U@GkMaN)J6 zIWIy>$v3|Uqm*b3i1(^Zae%4tL6|1uV4@n7H7*ojt~5$Jp8*vPHI$iaX`23Wn=H%vwQ*2_#p1G1OZvi4#B=nhhx_0S6^w8Sj86`a)$Qx+_Ul6pWM zvIYXtP--VI4IBtALZ}_u&2pd;IMFm63+YmcwAdHm`_6-ee+}T7r(w;>$jhIF8f#Ak zQ`$n&8W-?%J0sDm)ipp>r~HejaL-D)+EBJ44~E$pqG7n|f%(Ur2Lo*ao-`*YY?!!1 zAk2(tAl9oO=q`qL6R5;dTe0&%iSPrzr51dZb6`rBFZNT3z7#DBjDH_LC>m=di%kUA z05#F>f%A8MH-mH_7dVfsA2l!}^-80aqF{K8*dL+`xkH|LNk1g)6iV)l6~?|BlhV8a zq{3^31C~v8uzz)ux`=kQ0=mtp2$8_jcgoX?s3jQ0ZSkC^T{cF^>tWkEpOBEyG}sgG zDOqGTs13UQ?GPT@L7KUP_7yv7XJP@|b0Js-$Dl#1!;jf$UOr&JB2>x@~2Z=-u6(3Y(y*XWs=;0jHWpGp|vy62aSKrJG>0Ma}o3 z$jvKVF8J(fUd$$aMo)MO>0_|Jwh{CZTp%qMB0?Z}4kV-I*N3-x$zr!4mheP;2d7Gy zgW{3|8Am?&0^^fSi;X*n4`d>HU;|+706XUD*dB%E9HvDaH`wvRU_!NAU5vxW)T&)#ETM%Omt!`R@`V%_5|r`tS@-n#MZ#0#gNTD29lJ-t4rJ zJjkWIhO2sx2t||%WaJ`M0ZU2^4b*)PJQk*$R#=szz#tb7g%u4t2F>m;;~1=l!76!! z=&V;L9 z0;3^OZ73)xSdL(gJM)b`QY?Kf@0xI<2)B<7w zBmDt4(!ICQNc%tXgw30dG*>Gi3=xUzp#7gQ8MQOiBIdMk^w(^xE{?)HmcqNi`WX_` zULe!Lt&SK)qoB*-d7Pt%0w!fyF+Ai^fVw0#p=MNrhn&xO<7DN-j!FQ|VBUB@ZB!i+ z&PI5kuByU;j)TsF@9!ye5h{dT1YU=Ekx zWJ6#U`gV>IP5&M)$9FY@KUvf2Jrdeb$T=Wo*?Y7+($WVNt8eLc$tKo+~A zs!>8i5CSX^&WM_xAk*!{ShBi;_D%3w&me)?pJ;`3SWQG@tQ(oFGS4xm0D1@5%7~tw zhY_yd8E(Iyrhs$c!Q15Y^z`$VBW}k?*e=WHG%2!Ju4Fwu10v!;WvO@_)>4z(#^vLv-{o{NOcz6x5hD|s}za=8bJqip|$&U5HQf9l~nLU6V8JB!SG??k~#a0Wb%}#n-L4$BffQeec!`#Z4=N8g!*kcc&G;5FzSAnIy+@OiNah?eJNZ zX)i^sApMJZF^)I*4&7LAH;FM}YFXj2iOTiTvK-drr!K8;c_P}r}H5pv&YlZPQN=q-l1^AYvH2R#eBVak; zCzJxURKhWMscqSeI^q%0a{x@_{KvcI3rvJOq;%xZ=8lz#=zf`c#sf@Y z=}E|aSF-2&E(V&64^T#O^EE1JSCdmy$9p}I6}VVFz?@^1JWqDR!J1@0BJ>^!=l7q8jibhWZmT3f$bo7U)M^3$oTayP z@bDN#|NY|`-e16m>OroR8H$XBAi2T;2!`aGzDIA%^m03{q3z~ zK|G0iKF5lOGWv90gwyAO7Z0A?1Dp?Nc~?e{CG)z^wjd#Xf|VwxWClb8(V@Y?s|$^3 zi&lM0#MtuN>$ZR6dJU4=w^F~8-8UU8#&VdrsFfOjZ z9sezC6(h<>PBttJf$Q;C_-%CkT|6Uy=Ecy(vN7)P2++@93a?W-sr#J2!Gjb~7$0Ig zsKAJZdkmsywC}f1xq~?z2P|bTM5)FZ1yl{$a33^LwNXSkUpBnSs0gU5L_QvdVBYIS z(RUvVKP%4DD0yvDQ+I)SKMFZZQj!6&%dI3uEwOQ!GXVL7;{D~x$(JS1IRT|q8@3+r zTcM8R9tRx6=KdL$z4c#=l#o!|UrXw@Ls4E{+V?)> z<#!qNd($B;aAO-qIH-9Z8IQF)>L_Ik!Os`K1lHCdafXk=+W`n)n8EB=5Bl+Kagn7W zGdV#0`#A5w13*Lsqr{2i>XpIkBebso21E`tHLn9HPLgXg`EqE_%Y>P_0)f+uM%#UN zK4osy@562^jV_x*9)kZSZci?$v8ic>VM2MQ0Y$apPhf0ZNax4H$_!LJ4Wda>=SqF^d-~Ww<~!#qoMq3-II2T z`$P?-Ftx#H@Ir&ofh4kd1xjdb^{sU`L$WBjCqqz8j!qVsmx5WdKM}dWZ>s2%pF8a^b zs(DgEL%voe5q4GEd0;CNE3>7p7nE`5tnjAhx^lupQK{vse-vjB+^aSmK~MlH>tKfF zWnVD{7L8_sy6wW}?I{su;Heaah*i5xAVh%wyDp8TR_>RKBq_gexk_mo!~T@Zz9prz zqH(=7|D4L_0(f%(l&FL9P;!*@=m3(vrD01->cSV``BNYoH$~guCi4iZ3fuGRQZJ-6 z4q;xp(t0zR0n-dc-PR>wK6C0w?(4>y+Sf82GGb&|Z|q)NoS)o^Zv8P}AfR zRBT>>n;DL5{N*Q8?xM#-dgHdqMVw9Ihx@2JZ1+_Sh+fsn$pyjw3!{rxvcM`=RzI6q!bNseAh6u2mBJ;m3CoR$Z~ zmTV+%|A%_B{AN>V3VsSPAfUjC22!eWgC7B0<=(Wp42!Fm6t6Xkn-v=Uq!a@MUI$j4 z4`?PNDm)6n0{uU$lNU($F14^v<0D!S@*IE(KmtzO!%|JS(TE9&70SwTTYgpbeBOZX zRpn;=`_UulS+fgMJNx1QrM_x`_i+|0888-Tn0Le zFSz7<$a1oYFaTEukzCTUDoJp-CJ-)A3evlf5Jfk-wcGA`2LZN`7bW(P=TvB+*3_^29(!D0bHq=&ecUsuh`Y5QKz%aJh#j_%!$yxA#QRFkiZ^D4B{*s z+D?XGP_4^=08oP(27&B^H;}ICq3>u&&NA%CV}TRChtrV$G7eWo$)M-d0b;Lga3LeY z60KJ`t5zi=F0VnapoIAm$S{R$C4U3{jc;0JDeb5!phcGA0ZUx7FW!~+!NbSzYMh`DW-eTnrN0j+UKIw$HqT4y$> z6;xR+q>!wEr_ZfE+nuH95Xf|x15tv&2WRmdMy6=^Z|53GHe< zP+^l*gUTT60)(n~+t4n66uBR3T&we+LAT+#ln3Qd*`J^z^iYRolt`0P;Fn z&d!K*HZn3o#Or7$L=t(7hEJ>di8XhLN9gn;i^l8)5Xcq_6c5y7`iS_X)-MJm)O38< zLwc4W^&{zaQR3O<^i-~W{I+Y0=6(`5+}S`vIRQZa<9)Lb@cRMhIKD|#ucJ@iM-go! zsH7$&E*Q4hOikgag&0!SzK|>1t;u0=w9QA+&=~uP@UN4B_pD);$1cDg zt~gaFnrpbwR8w075XH+?o$5sQD}kT!%B-;gj9y3gfM?yp6hqX>XLmJjogPfCbyW$y zlFqGp3N~jju<`8~!UVO)AF^*(IYCm?|0>k~r6vCPsp0vhv>4Lsb`N-lEt?oZm8-xH zulW3tXb(0Qhs&nLq?MGvIY>%x)1TUn^d^oKoOP}HhIMBGKBA#2;8Yxf65#_&;N%`z z)4( zXMJM?*rFIv8QdZRz@C>(U%OlVp}OuMrLx3db(D6^PoR2-App|{WkpmU!-0uQ3{wvf>#8fc`INg#HbM892YUez(_`dpR zn!$?)_YU6Oe-w*j4SY?_%#mQMeG=|_1wkiNACZdrTKF0E(FVQKc(O{rAsnwDt^G_x zIBI^TT4svHbmF0=kOkfV9Sno|^{`WT6ETiowh1^NIp_^6PuVkDiFKB1P8|@OUr1ok z>2&Ul%Xb=_P2fy)yq*Ze)P}0AT?*|w1Ks)bhfmvwp~4Tl(Ryvp8+<51(D>_&e0o1U zM+mjX@k$5qr!8^Od2>yWnsQx5FIb^t9QfT_!0D)};X$mjh-sAsV|xf)yV?p&`)yO^ zWDtuLwxIXr%%2zf0F6Mrf9!%LP_k6JQ&t^YdQeYz`d}+kgUjQd+$HMDl(;D);P_oe zuQ_ZWe^{W&o@#!J6u;6L{MgK^h!&rqpo_9WN;LA2XDCI1hLC7uY#AxngL^!Xww+<^ z>=gfG=NE0(ty8==8)|6Rko{it)G3p&K;5@sp^F6kr+hbrFos!wV+hXcZ=+^zZ_`qqie(3@7>lKRnA zLW0Mkk9%xcYFj@_i+Ljz1bUCi#ch<;k@;38sxZ)wCWzMN+$M1HTeX&0eKGp8@E2Im z57{qALZ=Hq2{%31M>w@X&<6YdIH_rV` z?GOTKQP9K%ZD~8jxJ0LmQIZ3YfBf6ej-_l}_zcwP&Vd#%Y!RaUyk!FUhT-i_1Ol`~LLj}Pj1s(UAOM34AL<}q_o2hx{^&(BN6U~ba%d3c7T-qj--JEl!b-6P z@%ngA2~%KQLo&bRf!F`y>n*^dOxOQ$Tx(5MQBV*t2qh&1X%zu!K{{1HKuTa}80;dY z8zrTemK=-DVdz}xMi`m_{`WKN?sw1G^Z(6t?REC7ILy57^FGggf9eJbs+t<%Uczlx zlF+r2m(>nMg(=TwpBrC6Q_hhU`4xHMaZ1(k)BtH4F$dTBDAlfyHF4?;P+pN9t_Wmz zzddwFVi$$~vy;-fZAjn=&!>E7pdpurEZet55YOqOQScUhj-YLt2gOMOXtDzsaXEU0Ewr!BZ^F>3* z{YvFdp^_=;KWwD=>-k_Sn2huQNZQ65CjGpbL2~uAb!u@5be_o)7ty`bwTm@xlbrPp@a+q+kf>QsW3cNuzLH^f zYW}RTVas^VU)S#ySw~iV4t5}G%hLJEHnJeKpCSr~v-PW|9k&2#^M$Rot=J=gDMF0_ z=sz1m6>uWTRLW@u$^l9U@CzV6vnFAAY$hrs=**8YH4fwAOl{MeWfO9`<>*f`b)t+0USDsCSI>ixhK!0Wptj8(D^`5xbfpZS~y6F;6Km;$4x21h0}qi zLV9N)O(iYidA5aI&noESbHI>r6Gq$*q3#GdU=gIquaXDNiaB#Q0zptRhKVR@Y9CMg ze}4|K1say+YomC`2t~6rk+vk8pd?ZOcCr&-h_VQZdF%U64 zY<7e{MgHwdyRYPOqL9lgAV)pUDpo%(A>HfQ?qP=(GnL5KT>JChKGW23T^RGqqLd%&u_>|@ z=H~KUttsXlVx1~@_#AR4Z)^UgNPW)`kkEW|lad&{Yi}rj#9`go(wOuWNtUgZAAvrrl;2z${Iw+79Muxjb;Re*WajhL}VC(uGD6DI3g(E*pax~ZO4Ku27&{-?qV za7SzC_$Ej&1LoG4?85jF!CwEPiN6DK%9S$I zw)Saozqq8s+~A4Ji@-~$ATAyWlHjjH#TwR#2;BO7kdr@taN!@oCdj(LG@?HQsE9KA z+oJ%^qN<)Oq(B9bfK@j~&McfhfsKG{Z~Tup=+$WP_d)v^V)zF^`}-PUktlHj^jUhw z(LiTcQtMhCLlFzWwlN@1CBbz?s|_g9h8-;bzc?=5^8Zaz2m`_$Y){_idm;bgKiR8m z1S+aghHXRSj{8qe0@wjii{H4)E}Kql~u3sqGmXPcIZg zu;f8Uv3gyy-&q1R-(*J6?#+%i03r6k>?W1krqRJ&?EuF#Pygei`hMg2pOn}Ln%`&&sC^5;=JCr ztl44vZrl95E6BjEKpf+~HO9=$$@@1!O~-Su@Ad!G*t|I~CH?;(sK-u9gLgN#o_=w< zi;oLBs#RDYbgvk(&S%`Fv-7Pxpw<|%xJ1$F&q;F%#@X$V4`agjzrnvD#{Y@>DJ!Vi zE3sDuVfybyW313E`*&T+0Z)5R@lHrGAfLDZhxCE*P}3FLRa%_N0D#mqx(C`9nhPB zUd!^|gxGEP6XY4e~DnjK}a0+0U)RoJmm3ET7_egqAMrpA8ThJ{VOVN zki8V<0^;1?bRZIQ9JrKekSC!H3Nr7-kRQHLO&9LJ*@J)Ff3&^9-pM)(R>ZugW{Myo zc0kvi2b3HjMo~bfJ;Zo|*Q>l59h$zKo*n->G?9)Gu`>SWBwN;+u*qQAUW}q#ILy==pa)5WL?fevL6OdI@DC9B$Pa^8X`V_k@H0N+71y$_y1VrI*(6EgUIMUfaqSLE1)N+ zc)It{eUSHPGim2d`i+AQmfKVQT$3V1L2#Ffutx8$6DV_kfHVqZ!N~BRr32C^AfX|$ zC)t?=FzJi{=%Kk@_X)}P-#G*>C4p)XGUI9#EY2x=KrF=ca&nFB*DC~q%L1HBIq&|l zY|y3*{f^QrbtImIX21V6;*ii8xtNbg(svFLfb8*`=YdHl+=6E~oJnZ$r+ji{f_X5+ z7Q~2#@E`nVK}G>jBC~f4)G2si)KFt&ga}pSeMKn)@c)$=I~IXtO9+-J1dED7!RZAq zZH(R8S7SU|jgf3T(hvL>%Lni=15yh`oYeuz6XFrjY0cpN^NUowe5!vxJQNe6+B6?^ z$q5epg$FK-R#`lIkVs6r%~Ndyr^s=}`)PiGRF1D3V@UF2*x%UPPq> zqh2zY%@Ex;`Ny%3ZaL$Fvhm2g+^$`AeOvFt^(1e@-vnc?iaiL`l%AN$9U2x9zL}iK zNjuY{5O95<%kyUaDRH-ysl2H>w3`g$^c5M@TkZF}wx$Gi+$bJ{4g~55JEVF8a)-=C zr{Q^bBB>P#tG~V&c*`MNlgv776MkQyX=p{tb^yAA6*Q#3-j{FD0eQ+=%L08pDO}%&7Rv(H2q|!d}%iYnS z(ZB489}B4R(KpE}pY@zh#B6z$6THCsSNHMDpl(Phe^4G|c{Vt(tq>DNNsJXWZ zDiOuSW=Uz@8ub~k7yL|1V z28jkdL_~6nGo0ya+Sv{^3hO274rUI8Hw8K(jdOF4s^E>Sidxfw)3WR7f5 z+57yUcZ%K~myyt6`jIOKQEWpX=(8=>GcbtBwmL@_AVs50tGG<5sG^_uAQ*34_EIiD z1g+k~B9CsCI(`4Rx~NIjGU0{u$s#P*xYjt< zs}H2SIy@C~m~|DxHkg-z_kn1vh7}QZF5fl{T~1%og=z$-AYTGmZvooxnCyK}r-(%3 zpw!AgYJSzGzmzO@A3^k_QO}J6Xp!MTK#Qid)=ZK)SjzOU&FXT#?DGHq{lb9PVXz=b z<6IyOo-_i9${-vNWa19i&Y5sKSf@$E;g>_;T_~;gse%mns$Kif_#eH58;!82L^+Uf zVIpCv-l9#sl;-syuGWnAu!oUZ`sObbT{QprvPD)kwq2q3z=e;8uo(b^2G>=GX z5m4m~=@UMPP~sh#^EaYpXABsXM*VP1b%uqxq@eGy%J zxz%GU@~>bDE}MuI`2cCZQcF3})s1%ihnaw}AK;LVOgcB5H)a!#tDlTh4_J!9Ik--* zPq}W>1T5$Koa?fYrP_>)zr3T*pJbEmv^>0Xk6Nqb*fEPQO#J^;V`;s>!-Hx2X1ID- z`kvj8@PXqNFmryjIsp(2P6jX$jkhGqJH*{qKi&e3A|at3_V5Spx*xsnU0@sRnNZx& zXGS3r{NGy-1kZ2rEs1#cGwK~9un1W1)dy9(30ePrTN5!5!tYM+Jj?z)lllzS5a#a( z6E9d5)6-M6aslY7K%_I;)k&l=i7W#_ZSErTPN;pouUzdKK(sDOJ+REFVX=jJv;uM) z^u!KVGH(5RMHYsm8W*_;1iH=Qy)8>Pf`cWZuBtmo=4Ypp{KX(GvDZ)x6(p$ka4Ht;3N>L+#M`DM2UCKC1GM z((~w`=lYM84#gh^jE!hRfCy9|9EClBf=-R?f%(TR&ufr6(F5CG>cRuk{Hp+;6K>WZ znXJ(faYU3pZvq!!f*=!IgfZ0hwOrC>0K5Yd`h*cUcoj}xz5>ox^(w~-+l|xo@%CK- z-@XaHL}?lF{WOynwX1vQ8>#GR7c0<4jd7zmi!;$Bi{2RLu&XY@_{(6c`GC%qF7xoy z4`k>a)P%h^1_ghy=)WwAfKkH}zA!NgyAT+A6HWw#VUf`baE73f38&4cVJjN~>w;Wp z;Pb{|kB4czPPLbk6Y4+$^H3q`s%=r>i*+5-rGQVw^h&gW#Fi>AV4HuF+}G&4cPX7q zhq=qTIy!eMU(t4|Qr6O%{Op`W&rWMc*|zJsXfsg2G~eEVQuGFXv;(ZMf(iR#Y_`Hy zbNM+u6*tI)S+V*ZRHq`&*XG27ub>sBg|evLQU@GKp~d|O$uEoQ{avpgwz5D7S!juB zlcLix$xF8}&zs6M{<^hrYiFAf-O-shFOs*ZwfKIM1A&v5VZ^po#+wkN4seGxIS(ot zX;7OZ-L(8G#`>Va!8xKXhu6l+S=4^7fxohJ1ogDBL zY2I7Y>5dLMxKHY~i*E0of;>e<*hbZmL6~~hR87TwzHq*-7uzu4GugxgXyTFxfiE#JZNxl(zi-hYq(jHZ$DfC z8R&c0@ij0W$Urk2;-*`jBnJT*H6fF6M>2RLn4BO(pl3YKMLTaGK~Y$p`3aYNjzNw- zuSbW2j)HcPUJi%`!z;a!nz7oNeBe!o++pqtc-~~VwGcY70N?@@Fie_IS?xz>i4vN= zEG#+*uy#zQJ=h73ZmKPO$Nb4|&bQ*7{Sd-@PFtDQ&4*NmK_w^N9?z|A+bAu&N8sf}Q;7mHv z5w1@{(-t4MQJYj6)5gZMk4yGM|B($`+$RUyvJR1rglBY%1{BveP44VGO_>+CphRr` z4XspAqbaIsw2KvGWHSrWp^5|Zu-58>Pyj5v0CRRn8~oLcnf9NXAxoS-jvjJGl|K_H ztq{hkY4Sp8hK_{&zxUMyU=c*IAGuMpdX&{0@w$y*aH_O9a^iI{3&edS@+{5{@cNdg zdU!j#85p0Ri~nf$Lp(6$!&}{xGPwP&GEfA@-&d*#$SUMg0@LhLYJFJI<)S?Op1HtV zyB;HnBNUjyQ?Kj()G;r}sEqJpk843#EGn9_a9hO_I>E03YLgD&J|`q8S2Y?wY%xm; zkgEGriJXpozwa+B1*dm2VZHvawo7tgSH-+1!BC3S^d@J1I!Q6yyV%36=@ zXPr$W+?9x$iF4_k0yH2GLhQiXAeF93b zv~u{=y- zC$8m;zt0`hgo0m|0aRPEBP!KWkv63U(CS_DAQ5vvnCFs$arHT+!xP8U*BnbG+h4V; zW%+d#7DfI-aqY;tagbhydgG%^aLDY{9dUpjknYceV&u00k2r zbYtGoY+!_;pq|R}Ls8_&!m9;});LJ=#Y3#kIJjrYo>LHGx4w zII@>N7VElCSFym^`Fn}H;wdr65{)bsqflg~`qnHsf_*ZQ#kCG41hva-bk*4R$KT{5 z6Kc(@z(C*RDnCSyHExYP81F9wk%Kx?iNL7XT?eGb2!wCmP^GY=!Q-%xWDp!_^f4+9 zIpc`=2Th7P3ha2$5OESjOg_jejS|7U`5Ip%iS|2*y^2=f^JM2|6SyHtu`_|&XqaDJ z8N44UT*Y^{;+mNKB6Dhyl+Zs~?bHj=>{8&O)nK`tjkH1?9P|^M3%xKd62{r>M>*k0 z#PSJlNLnm6Wcs4mZzQop_7K3Rz-KosWLLG5ZQ2rec9wQ4ma%uM@yGU?_>LaCl&sKv z4xh%19xhF~T@(@V#b)+MR|k*?gx*!}4*x51(y}d1>YI%_CU);Ym-N!@FC1Nj!U1bG7%;>jzDKA z&h?iKfjLrp@(~d*KOfL)F76MMs}U!RHTq78yBFaHSY@RZ`^UH==jdm2g9X+%c2OAT z6q^~Tore8k2axFh{lhG7wP%$%#-byce_gCux8>qB1}$6@E!nI8@P}!H|2JB8ZOnWE zB(ot1Ga9rE6C_B2dW(#iFuo;HJQER=Pc+ve-y&nA6Robv=t<%Tb8P> zu&jMu&*;72Apt^~G_GRjV7$UDHZK9ljx-P0WZu|&c7?utppm@L-(5Z^lnRn5=(XS` z8#2kpp+E{`ssY5D2Hi)9#*={xbrEJsqu8CRFe(&4)m6jw*!0NKxXa z6+`aUb*yPD=23ccibs8zkjT~oZnuU0*jR}b725HO-P{Z)+k`Ezw`DcQVCNg0 zCkE!YvX#;_IEOqNMTPaZ?k{%C`;{d0n%9_Sk?Nj(UjLA zL~0kSzI@m3D*H+N`qLux?ppSShVPGepF*OZ|Cig#&(k0dj_AO~a za)d0-vZ48VGUgSZ!E0cKD9ge3YisZC9iyat4O0~ zD^nj)HvD2NL$K%aGRW02F3lI3s@xbov%d=?YgrVC`6m{7<|zF(7(IpLbU78ihl#Ch z9BM7GDf~I0pTs=Swp+QppxWL_PPZ!7)6KY;1j@sqcDLA6hn_pWTP_$~hmpkPp#vT+ z*rk#D=*aNdIx0#rA*O!!6`XuJp~z@swXX&u5gAENH+wgm?eohAPUD2nY8YhSsgL0k zY{Ze4bvn8%#yE}A6a{aEX2Dp6Q+$qJk57%)*{5+El1P@b<)s3iSC^W{o6GeTBE>Io z6|c|gbgWgx^hj^}tS8iIZVf#N``O?mpH7S~x2XKGF&)MwvDJ$kY7>;ux=-)1*0Rl& zB`mmV5O8rF7;w&Y2=`BpvXj$M`qZ|j`erL@tgkPt!ACKr&5?6%(#c0> z?vJJm+4j-9Aiod8Go6V*~zYx1J)ZiqcJ-oD~LK~dn_FX=s zva@Eg8{R? zR1$6P_3@5jbuwy2Fvmb5rO0)6!*zVxi`EY2w@Q3ikf*7R)qoK0a z!dywen5K`1lkRF^!<3|icgHKPHBYkuW4HBr+^x_m-))wCVyYWU&vq|&i@T=xGwW~R zbQ!!|h#?&B3cCC{ymm=+eO%w3?#}#x(N3{HlfBI~VYjf|l`HwOeI3h6U! zh&(1ZaA#v6%5C~*pw`^Bp`>hPSL^ah-}=cf&fmY{ zgd`Y+1QM7ptcQlcyV3=C<_I z_hnscy-iO?D^DlZZ9{c^iPcA|)%*+}8xFjhJ83f0ExqO)Zu>3TudP_gPXF$+ZHZEo z1t+Xkw(EkRb=!adceyx6Ki0FxLbqmMMYc5b)Y`}==bvxR!Si!(V-{Ma^xGa*?*+P= zL%Ff#tttmIg(x#)? zt{pG3*MkN77!R?9cpoV2HQwyB3w$x(EUA3t&9V4UQc1}%$UB+2=c)`pH%_t#`x1OYa-i?KbvARJI(3-5tTAuNsOSB1Js9iK!9tkYh z=k=|B4wzzv)r$B#qGxEV=VYPV>OyDz<5^k5^nWpkz_$*I^Aa=rN-ZkV`l*~yj_q?~DEXESY|lTn3$ z6WK4XyWdJ9kuJ)FpmP$npiU7Onqr{}I2f-mDoNa?5RzlZL&VcS= z9+bH#7?Tdb#lhq1*IAXKPQh2fMrgSLL|g#{TzGbtu0m&CQd2aPaju5FLZ!>8$qu&v zkGxS5E73b}4YS8OC8F11rw1Y>p(C^Cnl^MSXZR{3zt;BY*v2^K@S>Ma}EOUZ~ zP8F1T9e)r_YUxJi`;W3E3HGuQ z582%g$aDEq*_!lKaY4LhLly;rYI;D-8FYoHu>R=q>l#WdIVN-F9T)3Knaw}Ao$_te zX$z+x-PGgzI=zXnc1#Tao^z5P-mUF->sGa7V(xshQH!e8uMsCc20vWo^``ZGO`Qo3V zztcTgWR;gJ7`anJ0nAwMM3He?rYK_%*&keBX3Sd%O1WZASCgJ%|DYn*5Zm(JDt2Q5 z$73~{KHi+_^9_P_O#TAy*0*PLdsDAMO)OF?5kr5{o(<-oY#TfN2Wlrr1MMP{7YnLk zVe;hSQCRCtlP^X6ClK-mVWbaG5Uy+>i@GwXMcckrcet0AmX^9kf(xn=w1mN2XiTW> z>W^1dD4qxwHV>jZz~o^B23#)`Q8q;$&9XRA9mFc_`5TdqR)f>Rx`O08mvIaKS z7l!2)xvnGHeGK`eq+-zM5bCa3V4r=Uc}lK>P(#ghV6-NFCnEj=A=WRC(npj#wriVK zzzV}3j`dEW^(|Mol|6$WV@1Ogo)zq;sD3pW=cP$*o)%)EdAd*+2~Zxy^}82A~5OAbzLak1J(q%a&Rvz>Q!M zT^PtASAY&;r5T{9T#Gyx(i1JyV4Kyn< zy!eXyThN%jzAf98A+m+Kdw>Iuzi?LQ%+F1Ymjn4&XDQ=8GYBk}A?u&l3U23 z_lo=UA%5pM|vWKTKyujCNTsHFKPc#A+ z7BK<;jQi4?>!wXvSJ1-lVk_z_xi`RAF0}nQFe{O$h%LoZ3a(+Td^j zmEWrNzo$>a+noh)vsf1rV;@|TUYu@k6C{T*qb2rYC#PKQ@ZZ4u&b-Fq1eh!&1ls_- z;Oj6PC0lk2S#0&zo?4lBiYYIZ(>Ym@I+b^8?8W$hI$`aHa4+{BV*7V8tlb6n zU+?n3$2n4^=yf2LySmdUw^oO9@VO`E;->_B7Eb@S{hJ;S6=Ky-cZi9zGl<~sq!--x z8nq@z@}f`Ue;r~Z?Nn8@`2!Ic=2?S1)Nn)@CQ|_tMyg~rz2X#zehj@HAFk_qjz_~L zj$Zxa@i=Hhk$49UID|}ZTc}JbLEBVXKa&he<%nspEE);PhL~bBjEEc~3)+Ax01Ftw z+-V-rw2NPX<>d#P+RX=P*#lg!_Zvo+RohdR3X5b2|2aJgb{4-Vx`1+sfl~sS4Eem% zV&ws&1MQYVGSWmT0qg=IZ}x|MRiJ)41s3N=VAVe7(F;LJW)J|3lG&rtKm_u<5k$39 zd9XbmeWykhU8rSkws6&D>*=?;Gk;%=YAQ6q5G43nO4ML(-yA|=A84940Xf~tsc{`B z_|F;6jL^<4IE1|rt)WN_P?*OMnj;8TrE0h)5QWRmgE*`b1rUJ-)N-L|UvqDplzmF= zxKhue@_&!ayfnt|Y0ux$L$_9e=l4D#SA~A;7 zv`7jsXoDh?sIHyUplL{CBcFnB=OBS1Cy+cg2K^r2|I;9VMH!8QdlkUV%?%9{Io5;; zT&NZ(bzHIa{^Q4jg+N04WCCurf;%oR zFHg4nkiSKn6ipUyM20nPZUW8)c>QClDJb2M*hm4m;w_gmKpst$*8cqQZRqjtqqovt zXBLe{AhiliV1zLCQ^FXt;$AS}ArtQza5*7r359E?=hU)gLr84}2+-g-Jb1Pw)ob)d zFn${=&hp>STQYMMrpZoLbqo&P)3}YkIkILW8RWeQSq)%O7Iv5nVhKhrBb3-UIRiuO zPXc-mQBBXuQB5$ZCWgfekwXxOaSEAegf|yLao$K$4(`y&-pt}mU6^Vy*L1kf2!zir zYkM0b)bY1AqEC*D*q@KpCytkzXwYN?q&qdl69y^w=hYVzb78avCetvHXV_vu7W-&t z{2w`E9j?|{Cw6ZZ2(-yVg+GckX**z06Lc%!M&R%Vp#QmY8epxZK{j0r%3MS9?lU7x z1z>s&=HvQzG=G>S@t5JqEL@tm%)m#(ie{O9t>1=NGzD}bckU-V)Rme6UMGWnArv8$0_e^UaiEIoMmu#aD=%? zf3$VYFBLA;^Y0uWaG`Rfm6*Mi07ys5JB*O*K28|pk}v$nrR1mpE*z-!$&m>z(5Gb? zm?9zhq#FdJ7rF|aPGUG~*t!&-c*C;AXi5F=@czfsgF6EyQqD~T{umr< zijC#Uf^{#i&z*8s#{|L{kd*}m0<iXq}N%+H^ z?eJQz+`)sK%I7|e^p%&7Hl}&|Rn_|m5zD!sx8BasrIs3CV8*AK%*@0>G=}?DWvP}0 zF(g}VZtl*Z(Z*<+ZxW2B0VvTm-mG;AR^j0-N>Qt8)R8`sbxb9%(=Fe+jdwjgUpnt$ zn_ZCvG&e3tsl*(6SV#g?t*AI10<|klt(B7a3OdZxR~d7lc>xhb@_Igq$b=P7p;7EZ zC~Oeq;->Gc*n!r7uLoYZhY6eOOBMl5d;MqHq;j)s$1Q9;fSKsVJcAjnXw~~C!U>G6 zPT)B-_zu_fdb7!bCc3zNyT2-E{?Oo)?eJq8Q3NiHNqn4afYi#yRM4rdHam?2DIb&z zsptPFiVnF&t0>;nz^dfeH4?1ix-orq&G+bpQxKCEgG&qJVD2$5YUhV4UFVe|oh{P^ za8s%SQ&R!+^EiF$H+!YL3Or}}#fE&RxB@_$*}SKmyDNNmfHD5x6u)_#DQ_7BBV~hZ z4MmCm&G8Rx-z!HZMu=^S z0!fsqn5HghOLSTPSSGY=UMT6obE&fd3USIEM=&EQ1bTh*hJG7B|0lV|X}=y=Az`DAU?7 zSuS3^Ca%Wujj=?{mTy^sM?8%2cyn)jDraDrSVsK5veHj(;rE>}Cip(+`ON&W$5Wt}`eqCrO8HzF!N)@fPvyUTV(R#Mo8)Am=I%KY_vNkE1#FIW9k}N*c=3n> zgz-zc-O@DaX(&FP2ztnb_70fYbP5z?N~Of@&JxN;La7V3pIK`ZmCrueb3^u*J-^;` zQYuD>PS+;ifluw2#cgkM$F{n@07Z_SDQ_I%H;Oy|n_w~6#yCB00A zVY!BSnxAzSACcol{e5Gu zw}agwbvh^{jOVWJ>uW0m&}Z~tg~!Y<^~#{5)d zuQT}(X^!E{t>>Kpsqn|_lC}Avs$@hBx4&2IR&E~uq;jdxndFW6Ln?P+arrohj;g>@x|~p7b666@n7PuXuFR-=6AapNDWCkNA`IC(#+y8)y)iN2)t%5sD}P}TntXO zJ0Kh*z(A~k5GZ9^Arr9`GGFrmJ4=E}T`(Hl%oCtERfrZ2*8J_P4<)V7S^SSz`F?lt z>&7%kqS+!ZQH^_#UZSRWEHqJf`mw+Sk-KuFW9!h0`HVXGJIMuYygqUB{WuC2d3l5` z#P{mp>nKJ3mrtQA8?~NbRg4IN$q)VSk90sO>Du5_E2P;->@V!l76L)Q@1oc`{cR=2 zEt^GA;$gjX&-EX^12se29upahbIAtj1*G_+-B;ArFmkaTJY4#2bslYvxrFusr8=&% zn+&+cZd|f#U~(g0)8v2-msL5Q)`b-FYAevdZR45C=tRlgIm{ok0}VqqrJMKG$0U_C zH(d75WP42JUTflasm;UG#BOWMi0WFGYUnt0`io*`GFFnCD!rpkV@5&@cxOH>y>ubr zlMR)cN>?jIiI`cvcUZ?ixLn+at6~!)=iEPaH}!ZV&m}PLTWlH{GqjbDxUvy>xA}IF z-Ab!fty4>Db{kX1xDbzz;o3ar%hH{eL9bkq#sxQuF>RG75f@bWA(jOf1NSthdN>2^0Y zstEItGZwonH7~gKirq#YLYN~bG0|-r3Upb8atzx*w z1M-xj>y`|2-D^|V{xIh&?;nupa!PKEckAGvO_McWHJnX#j4s${bTzLS**`YVZc{4a zLoabJ#-%);q-x$XmmXQg=svx+_|;;ro=>@np|9m0aBfut^z*jsJsy5=kvB+pr}C=5 zdbHGs=eF$qW|QF`EV|W!lL!*0U0Q=1ZeVw9{mXLX@O*NtrKy5k6Zdn4RB;!t`|>+rpi7Ubnp^LY@+{FY|1k6XrJ4w)-+Q;e%qE|OqYX`hDeh@WJaI{?fO!- z>t@Ev=%#9C)?RiFk&9Niigg##4_3olO=YBI(UF_?BQ7j&V;LCd@gFfg5{wzk#N}?6 zI{v%wUM#E%g>L3|yF}GR&g%8&x`pP=Q5M9^*}HV!&9F&4(ynvKZvfjAKe0(E)4-=} z5mi^%Z9F52-NtQ48eoSu;p4w|YQp@C=^2|Z?BBlF=dvBxhV9=a(JjE)YWjrVYTX<2 zL%mQ_q{&G@Pb?#QVZnvO?Iao|zJ{q6C35gNXXgj^1FRhY#u z64gVDep{k_tA(#pH%o#1;y9vn|D90dh<&W**qENU;MhXDP^fZ@s8HGFmdu2qAFh*W zY{A01T+nCDjwDLF8k#I^7#eFg{Uk+z7wsm|ZfgxLEIBNkq!gVojf5yI^r_J#kuKK)J0@UZMelnFRvUj^}ejittQTiyL#qYgo zyxc*MX_6Zmj!rhsTV?|m>IUrPJMUvM1u0+LzLckV6DrA*)JejSz!qg2w0xR__WB2P zzPAh*^Er}ElAo4IO3!Djg>x zUcG&Lr6ugG?2?YaQ>lTe^?SCbx<-ch(#u@n>;vY#)SaI-)ByVo^)_PnwM=zqXmYpMTXUfUSu^vCLYKkq*4dAZ2!vy%XO~5VOd{_M7=oSho zfn2H+mn4v3=B1b7b~&ZsBuy?H-O*OM=$`!4>5;Nq2(OJ@AF z7EK3C*P9Gd@--pj)Ai%WCWYqqlWf-N&m<0AkH4Z$dgzw=d*<8o&7?*+ze(#}-<%4I z$ZH+0z#B^O)%W;1=||&WR|?zTU3lwoF5!R}L0twL>_%g0M)yarm0#J0hrWS5Yt%${ zkxT8|s-nGFyt?B@!|^ZPNBPn`a2BDBL>z?2w93AO_5Kob?3~sKxq;4kDtHFq^@sTR z=G&Y5qtRGJ7#XMr1sjmG+1@Mf6I@lN!pCso1E0@)J+QI`w(Zpo_F6e zgXpfxBka-ZvM4&oU0DAIKc*{j%##_Os=&kXdF)Fy{A^l(qV~0hdt4ZtZRey-_xTi5 ztZA2jD8|hCTZ?2XyLt@Q^IJ)exCbxQ^yIdLIhiIg7dt6L3qI^z2XmK*hmfOQgwWK~ za{EzwwmY~Fggdyzjijy%=Y7oNoWy^s#T0xCner^VJr)$A?|QBH6ESPFhJ*G?oW3y( z=P^6>i?q4Z!$Y)vBMC7R1p`VgM}Wvjue{wvy6G;}FLCP6q&4YU_Vg8IUB>NQOx>Ev z$6q5c-fgVom+SGji>+}Re>Ov({C>TylQ%>`5yS!?TI9-J zFX{Ndaewlr+m|H?KV$(vWEWR@F|?&Q-~N?=I+tSn+|#mueL!I~xxh|dy7le- zG~K-!ruEg@6E?WmAjeIICfEI5jeQv&#?(G{@z`gybdfw*j%1XHyneI>@w z{`G-$7Iypbfan%0)@51!Jw5_y%a7b!1cU@GnfNN3MYbNe^oSw(+!^kx24+VaIjRJZ!_DqyZ`#LL@Dy09S8g6U$5Jvf1l6r zpMr#T{oj3}Fj4x9RKh#2YvEM?@z>aN-1eGj2PP}YUywK*$f6t`^PoN|Tv$^rfuXM? z;kws~yjZbAzR3IMY8o0zN9l$0x)%~>!z3LoX4KIezCYqTHWWUA*Gx|K(VPqj(T!Vv zaBO9`)(^fyBY-RJlAp~)eiQs8?J3Sxwy-QSQT8sLFx?dl1!A_*D}LkV$9W42y9tlC zJZgoj4|hL322Tgow&=NN``dGQt6!sOOG&JT^>S55WBru|3Y~OZuYC)(Ike8*n~e5X z^vKYx)L<7k=FPtb)naP&@3cfGFQ&|6Bi-yqYCLXIj2Ti;lz)(4=M}VVX~k@8HVyxX zrj%BvLuV{sOrApB{YV|w3XxV$Y7l<6(M6qEIQXq;AXvNPuX@&-EPjdSetT&&ueSwgDD}Q zhCo|L8cvyLd=(RDenLkpsxFRh{mCXyNUTH93^RKd|J%q-myr{PDJg!!WWt0w@O1*b zr6dP)iDWWq+GSu&PYb|aI#a*@@$?Y&m?4zHR+bcx1vqlvG=@!u=#V^TIpSw)(7m3K ziad0x3_yYwu%5+KK}p|%$iT5_UY=6Gq)`D-Pe9`@dmm{<0@Ftx%&ZtKd?f=)h7`S< zw`V;3aINEa@u}D2S0t8IwI8rNS1o`=*5Xjb)%pHCKH{lVeAGosiub)}f-34OCIF<9 z7wmU7qP=|gT67&UF+<{dar-?W*cyq_+|-ctAalr_ih@Fit;EbqTe>e0J*C~x1ZXb1 z0v0&h0qz0;!Gl~t#8jAr%D+KdCf;KG5#$AR=(I!jLKuou?e+06=HR5kC%Q|Pe8{T# zc+oHW065yw+7*iQ7nMLpiG`tpbgcy zSd^nf`ssv!;loH$QZzqDtS*UGOdGjc3pnACgI~!0_t(>e zU$_X94eGKG4&Duu&xjUZpnMC4c!R=4fZ~%8CJoF)aW^kHV-A?;%T^FnV-AS6?1=F& zt5FN=Ns2lT;8FV}BR+|)wIo(Z97p|qF^qfEhSIUq?ioOg{4YMrh?9PPb0I-`RG!K% zirT9sX8PLF&tR|l_Sas+V-q{eYz+~V)%w-j;5xYlkBkKafK4&Uis8J>OJI869~|S@ zZEpT#FLjvd8t{}?AvE3AB^=T7HSKAzNR}dO2EV^a5%Qsv_|R2rHS3V*;6%qDGF2_T}>#5 zv^f)_E1w}%F&K1&cR)(q4V`F@Wwd|Gn5gg2-Z2^--J)&I;_*4Y$$BoG--!B)D_Jev zPE);^vo-mJLy2YmGHR@Qbb4M|WVLpuK^G)seCh}?$)Yp?!g((a)x#%z&LJm@=6aPM zqv1@>t21`DVjRH&d!nR}o0acDz~$@(Y`XZ; z0Yv7%`M}*yLTOlo@KxZ8WEcGW$oeVdd475rr ziiH7^@iMu=Q|7G&4OHtFK0wub3U3npgA>FnJN7RGR69kJ6TMtteV+#ASO z^?8y#YyhME7r&xMAZhA(9=-2gs2Xt1-L>;J_$4O=g(D3fzYz_$Z@t1y_Lp_{i4`r| zl|oHMJ}~GSFVptb$o!5{X9z%fgrKo?teI=4}0J8YO6G-+tkpVw6d(x7(JIqjK@B&(ZH5Vds!HTBv+i& z9G?Gm-%NW(55S8+fS{3+ECt&Zu~v=Fn)z@)@WUFXIM(d_^R--yDF;Yv&|k zzl^DyiwStZzfhpXK~twW5_1N2t=qocYM;k}e2IS07!_UxBT!U_(7s#zkZ1a-}cKH#4(cFVh~2s@i5} zP$XDono5Vx??3bCvEl)W=EfW^jZl+TF*J<(ym%g7ka)VV=FmR=fJs zOG%$uIBgO?DC#oYmdNi5GjRWr#b;QN_8zpNl_j90Ax=T0SV)NWgVT}6Dc(n$&_s*W zeRQ*fx4b4~ac5UNKG+>2#Q2hu(V$*HrwtbW+rIV{IB1U03-kL)fbBg6ylIs`K!nQ` zqT)RF^bprAD?C(23(Y5}y>iyCC~9BZY#Xb@JR$!^bVmqildA&a=`4D3q94$3ej1ri zF<;@UDq=-#bx?-D$;;3oy^t)_a2G3-VX%7){=jzi}d?t#YDt72(p6`1^ zddhs7G_#tZwo~@6cB{&B2BVE#Sg;CgjFCv&;?lnjxh*aCy*NX5E1)p1Bni?F%7$Wd z-FT&2=5`EK^G}I~GBTpJsHGh78Cao$nqQ>T*v8!KW|o|vv?nNaE??aa{&&+L5PUDs z-((R;G$ka>=ms4*AwZr>+wu|0N$!SXp#?^y2ZN;?msd9(r!MvT*#Ae|dq72*bzQ=U zSp=n3kRYhEf|3NuSxJhLMF|2b0#!hXNY1H^Bxw;OBRMAvkSr=eG893wiiDyFk~1~u zmUh4Co0)%R*0=uZwOncoR6Wmg@7-tbefHt&%k29yO-x`}(^bpm!Z)iX5SFK+8Hj|u zD$(SO1|ep^SGdg@A@CNLg1hKJE~fREG?iT6!siS^~3(yoK{5Em)0?*MSTz115lS1IGuQ2(TEs}TEfT#xusR{hc# zYe(r|RlKdau$aTjAFMnj3tnob&Tdxuny9V`Sw#h6Fu_mLRS`#JaliGlj1n47MWM*Yf(hf|@#8o9q^iA!?WDo}5D zv59EQIg?mab4_JnrjEp!o_+1NI1~ApHib_0e_f?0e54&&CFY;?x!Ul$A*0YPfl?!c z7HmB;Y%i$gdmiIqs}!-RYW*XiE6+rQL^l=Hfb(Cvub$2O;MXo_z2_*-{3I7V=AoFW zPO6F99@adi0uWkEC0A@<{gBU@>&VZ`nUrZ_nFD|Q*Q@Eu|E9{KiW88qpKGL4aj5D( z)BdaKMy&RhpRfLZkO0qP(9iw(ifi0^bZ4CE+7Hi!|EP;8GMk~xl^4fPJ! zg`Yo-@;-1P2EXb@DlVI84r(!oU;9C8*iQM!KR*Ympz)Wx2bSdi>o;uVYN=qEL%F&d z-+t9oUsyn*pnN&yQv9D4;qZ7cu(!7S*KatME!Xxxe_Bc6Pwo``G%j0eCAf}T^Y2;y zl1J#{@R(YVHB{l>#i^DI$JEY|N-}aj_h$U$rMUjSRsWaQS%l@>8bIMh=qu)>Ws?$v z6nS2I01Qa7#zKvH74jM@GYm%o`d>mp^w0}l-ky8=gBDC8(}t0DXzL3HD4b#kr5k=_ z`R_|dg}Db(44_q(-!fc5nodI4c*SUk-=j0y$%ls|VMZ_kX5Wg|=0{69szHsxvn5Nd z3NtxEpsKj$W(}}k72)oQwglxD*^ZF^^o(?X7Ym?zwE$dTB2uRIZfX%y!j(&tZCRhX zX`&?mOa2F6G+gbjpyDlYyHA^Ltg`F6#hCGb@IWGAw-qBY^yil>6CYT+5+0YA4gwK2#z@rh}xd2}$05U|lf{pVqu2q`&T%Hd5(0Z4g z{72eHTE(9zmm}o#ue*f#uq^@M77s&*9#aAs0$xG6Ui=LY0(LU|=9Hr<*}medZuO+WAWZeve(xN<@I<1E`+j~WL?KL3k)-=^Z{z)I!S-Z4t8 z1T(x}8bicj=QT}DftH1F2M?eIA~|;6s{(X$-LmH)PXdal$HfcpiW85r{}k{aj~k`@ zV?yO$ye&lqV5Xm3hn$r#MlAfk%Vm3(vT{Mg~BH7 zO=Illq&X4NO6!M_rMo;d+yDk9lnXwupOd$=nbf>jJJvyb!(7hO5jN}x)gUxBpjTYx zcBLT6>9?#r%!W0n$wr(5Qw2{`cm<4zOF`LVg3PrT`L9jP|Vg&-F^hsk7XqDGf@sE>I?uv!&wuBQt@`#U<>gYyb8uso; z`*O-PrTn6z8M*)HPA}mbMuX7N2rr8fcy36DgXB`BoLe5bACj}mDeqG5sqJ4_%q~+9 z@mD`za!f#Y%SA3MugBC;emtVd|3QVr|LL&J`P4jciU7WX&-i67()rKX@>f(=Y@ne1pU~!8&_$3JTG+CJL{P1YI{>R3ru}Pn-2V%|C zSfkC$bnoR^7v4MkRJi)@X@vGV^_s*_+Q}(Q(9qC)4aHB8Hg7|GStP46 z69AvBC8mfz!=#oDTNk$tJ2J96dE4f?(0J7K8@U+modJPN;@q24%P-oqE2qQP_NNYi zNEf)bkZDJJEmavKtnW*U4bhb)_%Ym8*;YR zY+b};gWR54Ir(zC5{a8yiLr0^QHOKYDJ@%Dw~Kup`u6XizDQ19XY)KGO4oHmi0)o_ zTYB1!pnIweJo-A_dS?u6F3s%P8K8u06&^27`=?!XXPkLO_cqtj7tC{iYkO+t>-Lv+ z-+U{Vu`@s-XxIsB$X_AKG38LRa+-g6lx%P=OC-_OLbaJ+?E3wC-t9Ws_U|8DkG0GQ zGKvn*7WgVO|vhqHRLDZrU~h2SAeq z6FJ86k#BSXi62d`G*!*Mij4k}KL+9+RUMoka+vX)n^Qf;qc2Rr@EEhV#W9B~a_9;v zw3_emR62-}2hk%~q283_@kbS>8g|~=Dh|>$Vm9DJ^8}vFcbX%;L))Oc-~M7n z^SR!r5v6irK|iHj%#*m5JzM_ATOaN6d&I?AZ<)513+ifT8^i7_^{@qm?+0Csf7*J$ zI9RVB)wDAEvM(vz*u6h>#Jpzi1}=8j)^|;H(TnF@U;}PPyY}kY;T+FRO9?oH2>@!N zcP2u-ZyGdzV4)$v8Fgv2Ljqp}9RDwR)9X&o2R)8a1zpXuf9^#}vLw3;-&B56NvGLM ztW_??q$DM%rWZbxMXUJ<3_km6g6(dCGPaA=eW1~ebzVLmC34e8ceybbc#XfiLIiZ4R= zc-Pr$?so2NpD7B6W?-g$YZ>HDM50c0J3y*QEY@WLv`5dy4K@tI)Gut1^im!8D1_F~ zAQu4M-<7JIW*}cu72z-#hhSfNB!=CF?it^1mcDbk1>QflYKwzBFwQ-edV$gBiPf);p0LCYJEP#zGe(>{kVaMp_p-29UFOk!3 zXV{%2k80~FBfB$0t@p+)1s6YX5;}s_i`H@bnoOa0Sp^7FQwVt9df5!>2qiGjeoEi< zQN535f>M$^o%ruf5UCLj;E4$Ryy`G)gj;O}un8X-4%;gzMEH%QK*<3sOqj1Q(-G4E zrta@m@lD5mni}c}8y$N|^?WWLXhpxz0{Tu;T^4JLvcISzi@BD|MU|vtz#2k_1XsN4 za7;zn<)W&$DG`NIx0>(mU?By0ZkRAbhnYT%nzdzUd$3sF$1aR@&Z$_`e6zz(8=f^QBY4>DhMfJ+m37SrtzeB8aomkU%wVYj#uAieAKzdYr2M z=*JuP8b+v-M&R}|XdQE;e(4(1Ug~)C#g(^ZMPtqPosww2ddx0z;+M(-=Wc2<%tehwvmm8BzIg90BnEV_ z+P~-GuTR4JDmlv9x_R(EAqp5b9-zv?^|~IgfVZdBMv=#7-}pM@ef2smVEb*{?CdhZ z>;c=tBXy#D6F|7swN<| zHE~2WrsLG+W|L`zG#u?RW33IGiINQAp$vM7OXfGZU6K|O&Pyh-1jR ztkrM4~qw2NU(vhZz9461c*d6j`&zs7&E3+3u-Uz}S4iW7()nJ^m`4vvQ;f<>61Bc; z_?YY*3l%R8x`acWb^XH%V&ieX%;kCY1!}ql{Yx$ZntjXDgVNdDoI->G)5WT|ow05L z+|h2sD;6p6YvZ5effd(UDyJ58`5b@Vwua0Ycde%4H!L{@GY?t0v+8E>Q-9t*3=a1j z=7!xWccxHJd(OQ1hcpHKzDIFPX@u$*^F5*^7Glm+8}*fG(IP{dNUH zZ-Chmsm%LzpTy{Y;tS>28^-K`%O)<3-4Wnx;u`O;3^waI3P954=N`3qv_wZG+QRcr zn40RE;cp8$UJBF+Ud>NHwP$waSibOWwmE{pI~2_Q%=ic_j5v#ql8)CIZUR z?Db<16K|SV6_Ek&L93)HzHM_OqQSm7$46-!mtwTb(m#H_QS2g&DaEy#Z-Suy$`e^c z;Iy@2^Ms@ci>MiJ)VAcb-Fn-hb*12wY9&}=BJpJ}A9)xW?XVS5ZUy}8x5V_rUhaO` zKFEHfns@*k?r1kOn(-j6&GZd=3zvwy{P-$yh-u#deyW5-fb`8eq2fl!D-21LqAmtj=ZxbfWW6^ymNn=L0eL^8GyNdzX&T@QEAuoeVX$ z3L^F0742#bMZMnvOy`ru8e=x?eLq2??3j}*wOj_pFawA9i_+*JkV77P!QAetlUOUC zapy;~?(HW(3q(UYZd%qe+tx0YtO%FIaS`K@k8=h%{DjmPK6xKAF8h~HO(O;%Vx|Sm zMq0Ih{g6Qg?F45zn+!Pj(5vlLsu;{axsP%62Kh` z=8aWbu#p4q)BOPqP!b~df@Z*EoMs7W`KNHIPy}I_G_9pUhuO8$uj>d%Kn2&_Kd&-+ z6=VH;+mGLZ{Q^5|f8!H;o{5t2XV@?5Y`FW&3vIOS@R7DB;of;|E0$bCgPpU$(JC%y z2MQcEEh$t5digh(v$vtyG@F9D?|clnQ@nLF<(E;qk_$4 znbaBwJx)GKV46`6WyQpnI83)D*HhqmT|er^jS9fCmL^z#k~ncj$Bx*V3BY`D!QC}Q zam~&OKCmi$V70$+YI*`Moxbs7l{3QnX7Xe*jdf&X==JdjNjLb9xT*vVzt9@_{Cjp! za-f?;fa}DZ5a4vJkxb&eRR~>lNum+=%%5wSo|n+$%~0AlvGPY@nlR~)+31iTRi9|DzmV{Et=0^+nhw-z;RJ4isqEHS z_qBTjLj%4Ns}#HOScX>|NA3SzUURf$N7auVQK=)P;UQ^c?rtU0!8RueTKD`CVE8#M z)~hYFw-wB;r_?$^NVCq>i;kn+zRSI@DLi(X#puVD_U}d0>gPEPF?^FbDdZWQEA59`>4ZIBxEf^sgXgff`r3l1qk0WGX6ZynIyzh+Lf_~$8svq3 z7kAlLsgz8PYjZjRh9~sWV79xAEPz7T*740N_A1NLQoOiwy>9$CRnbprh)Tu#I_i;( zyR7(pT&A>i1brENp$dMGs`WH)(ZZkq$!zUGVxGkP<(Wd%T>-CC&3`$F+VbkfD1Sxy z#q<)KU%2)U>Sh1I)+b(Z0Wi0|wK=p=((@>z=-*k)iW49H=f2N>QOy2TsmhiI64}%; zN5K{9iN{?=aNxv6YYuQ9oA?8Yhb!8gJU@Sz*0I}^!mj;%w_LV@{ud>kEjnL(3^`fL znN**5`HZa_H6GVI3`(%a>w;B-KZbX{8){gp6V`CfH~!V20!nY8`~RhG_P_WL|LS=q zX+TeLPcQdBSYo;P;J+q0!rP~)tAAf~ik9~GpD5bpUw`rv{7L-%{PX|Ampv3QgW;w` z=qbLO678Vh3vw3WOO>xSXjOCf!SB$K;SU&2=os|sV8JJp217uw=n#WZ!kWn0lb)Gr zwKOBQx(0)#+ToapMoG&;FwO%tp&(}<9zvp$qaAwB41meNK2!j#J#IWX;G;`J!;h+V zGF}BW&H)ke1TYO6e0SCGK=XU-Cel)pXWVC8u@8@2zeK}db{`Es^ccARftG;AeN?~< z9Q}s-(}|}x;ppSvau0ef(i}-O;u&ytz3VxNx;F);9-NJaxO%KFt;i*4z~YZF5$Usf zbF-37^xjR&oRg<0xgO9FWl$0O8QMPGjPHk=hF*r^+_rHC$)?W>eb!R4!=2<*|?lzcQ{~+39q&6j^fV* zNzGlV_-Y)D5+<7cM$lvt9H5oeAg3o@Wk9ARLukw>xfYM6lp^Y;DR(KPbk93()+;BN z*(o_A|1KleN;P=YnZ>H{NWN9KpXswUeq*Bp&cMA8--Ikh@tdqFj9@?S3?0uEEa&aZAOI=F26 z?WUISCCZqtTKcIQQKBe7*bXEF#v$muJqLrhzPTbkMgk^l>zf|Kicj-z?_hnGk34dL z$0SD|-?soRr|A3dJa`#TLVe4|B;ggqQ~7T)SAV~f{s{tfqC@p9(oX9cPbjb3pvyZk zgTuCm4sO+*+lSNG(eNKG*z!;o7=zvm(|m1nF?T<&&v=p}eUrW&SXN{-3?b*DnVUq2 zMzny3)A$;0=z#o^Rq=kjg}Tiy z^tEPEs!q^~lmrF)m{PWr)?VhbfeoNeNJ>N6I2NW!uYUIDk+PxN09Rj>HB}%q0m?2i zjMb+{Nn(+=Ne$r`x_xPUPQ_=(RK4aZm6F{p>+H8p5zrhoP!vl+`RZ}}CZo>EuQ~Q~ zZ2=-jUZ4!4e%sws210e<*#FyHyw3Zt=3=+-oW8xwVcRT3fW{JnEEne8Rq;vHL-;wu zXLtRB()=AuEYL$Shi4CqO@2E>gsa&WI!LGZNHcEg70d>dnn6l=yWcfr{4L;H&oJIR z+tLl%^03*=5Sa zrap1TMw$0iZjA6(6VV!pa2vo5fw!pQ+pIn4hb;o&Rt&X+rT%^RKFINnp4~LHruI>y z+v>^8;Y(R3x_9DbA&4dYrdhb@oB$d|T(~r$@l9mFj^LKbQ#zUd5_)%EnIluNUo8Py zj<4$}>j$nkP<56&$@i^*)T*skw;&R_Fum8l#`!hdk3;CpI`5&c0LrhkrRxqWOgx> zTen4GDO^!m^FDFVvh-c;xonX7nv;=TVv(Mj`pVp80eTrMGDc?W_Q^ZjdNO%E__e>N z5ZgU&di2j&dE%48*taFLs+GZ=ps>4cpqD}0@E!(UyCf79Q8d2RCSwO-7-DSRpLVwG z)VE8z(Kp)zrzRe1v+p*woT?~Slbg~kDK-snc@0^gOX;LxwQ9leWXtI8UvGPe`2oD` zYfiaSyJc6ZSjXs7n*|$qUKksL%dy}7OF!I*U?aELYN-m4hkKq}Ymokps{Uam7Mgt6 z!`PQ48-fZK<~VL0?tKcRbb(j}LbOcnMUc^kGdHU0WnFi$T4<(`a! z%%@;M+BeUX?bX=ouEQJCmh;8AcbV>EdWyE~4KMgU>`}WQcD}@p_$Py*UE9r^RmcJ9 z>CI4iDmIz>VU7gPX`};0W^L#T-?{_66Aet3!`teSOZ!N7h}J0E-c}>Gt<|wrX6`>T zkquSW2_;E2X*wq0=0&s0pmMP+md=BLGpmkLFgIvJV-!xS*4_*yT0DDE%X;+IV5kg^ zIHLNF3x6oqZSJy-+tX8_sWuhw@o++^FkTZ%LrD?mo5eVSlu!4i#tAR2M7c6y#UZ=+HK`# zA)YJ(IXe2i*govz&V0@?e=>C;+&tzo4NtC|We(q|dggr@%(Rw{xEeN zoy*1R1Ka7u;(DzH@Vr~eFN*|BYQFzLqnQ(U3x!2U!d@Y@+lM@Hqtark(}b@#sBdOd z&C~iySmDCb1$1#hwC4(}ZTb>%E&94W(<^ox?Nfqcfy}~Zjc+L^Dc5plnL@!RKCC8` z$ndeLf`;$yaav}fy1dJNlCXKVugne;AZkf#@@zieR`&J~vasNkwn;V(#`BwZkq?@V zg^J(-ID}TKhF&ef6wT4>33(xZrMTm)_8mXlwEcMUFAW}r;JHuoPpH0&d3@X0F06TA zXZ-+xn;10+W`Xh0{7ERg!KnG&2j9NUm=_2 zq-uoF85H^k-Og-@l8_@qfTXv+-i&&3%H~`5!I_2S!UgOVYjyyTt&=crGj?S1^jG*{ zY8jdN&$84q%!Zj@{Oi5_$T~nkHU+fw@E|TELI+0o(th3vILliMA7;ZDPK3w0IO{-b z;9DiLd+@n0ZNJ(%i$<7DfgecxzR*YC`=SGO+WOFYF;Dh{v1XtC=xkQZZ8+e$@*&Dz zBrphAZ&^n&qrCqC77-q>f7%xt!9&Ggv@`*H$Y;pja9AzOqAB_j?nb zFu|9BTM*g`Mh|am2p-|QC#B3754|LdMqPP7Hv#^6TiC<27xmBOk5A$WYLq6gTDR=I z!FpqG{Vm(<6Y&ZkR*}c0l-dn`T01hGtSyql9v}6DmS5WsE_OrC0JGquYsGiSdoS^1 zXPumt`|2GH+|8SU8&6liSnIDJ3a1;sP;&5-0vc)5Wt8Ng4(AQ_8D+(@!HNeXCjL-x zZ|N7f0NH50?_|(D4xRiNid_M7_FzO|EOmGid8Uo3_i*i4f5W?tXFO+zww8+KSAd~x z8#f^ozj!1DCie%!n`jfe4hdgw3F#9D2QD%+ZpQpt=@pGF04Q33CDmjN8csSLES!J5 zBrRprp;$k4Oz9j(D6}ODFgx)HM_}D>usbqWSk5jcE)*+M;~g9LO07ITpDY8AmQ^m% zxZ9J^n*tPIrP#D+e+dOSlu?jFd#I7e4k)VfU^hQf3j;$sna8-VX7r{~1vDAiK(yA_ zEE+FjZj<>5hH}4(J(@lD=O@9!y z1D%)5cs9V_wVTaL@&+!>^yrbY;}(Vik5X)g{5QHQhrPnjPp$Mi`pMPbV63h>8)eQeY6xnPLLRE=u57f*U^Q7i zEJ(b`w?88}Dre^5S7Zz~W(RF*A(p&h9z#U-?b(xw0888(VK8N2qeLeP`ywDHSqtl( z@NK<0S||lY>aa%&)1oX0o5A<4Xp;1-^e<4l`DzFXrdY7IReqL#j~+qiWUxk-e#RVP zIu6Ni0Sv~1USkj(JEO5{fN5~1<2KdHL~~tv_I7mkicN3 z?^@sagr4IpG9A{#(#V_xH6Ia1_5fOh&Z@~mm0{p&kKG2PRHN$wXXxXL8Do2}k>v)p@mVO{ zNZ*E3aG0N<|CRe1AggQ0difFw10zt3K|&Q^=HhYzkD-S{Ki~}SQ5nkIM5vOvRDH9Y zSGBjjFNJPa-EwypHSHVV{5yLeUh8Z$p%lC`$cu%A!9;4tZnKJcerc#)Wjb$mxKwBz zB2t4(9HuOaT92bXO=vddleygo4RjK*63cG*_x!Q<(CCT*linZ)x1*xsLsi|A8>-y00X->gmdbd328BnP zb-zo%WX(G)PW@rxsMiQ+he3bKmVUG`uJ)t=t~-q*PQA#`!dq!JVd<8f&Y z;fMo@9l!>kS`bD{5J7ud<;{M_Zdaa}4UmNLd3{9w6lnHZsYW9PuU~5Ja>CLu*|{5gYSdmCtVNa1hbCKL%&o7zJa4 z-(K$&OQUw;C4d!Cf!tuM3#`j|GHNVWrKiw606k-A8k{?*FzX$A``AD!eF%3^3t~$! z7Xe#cMYT2QneXqc4xb@0lt--C2rey?Cso3 zqAYNnHs>|Y%s}+F?rHGwCjaK8Bo3iW+`~fpVlUOg1(O~i=72l{qRa?Nebx^{jI>{a z>_V&jOPJ_VPqDaXrlxYK8Nj2?O@@V}V^9N+zE%$;F@GxqpIf8+SF-#1QWh${Ms>}) zGWDuWw}8iQ(H@0-UW1!k&>j+Fz6M>*At{L9fd>T)jibxMi>Mi37=j_9Og7UEvMbU< zWsA_varO1i91oph;6g@cKR*Jh;vH)&B;1l+0!NxbDX)i1j~6q}wBD#Zheq2*o0L|Q zGeQ~HGXZ2*lMg_i$Q)eiRGWd}QWSt(-STYnP!8%9vKfRYtRmmCJG)xYqzo#lYI5x? zN?322XVO%<$}Z(}e*|b5jhWuckBULWz1s+ zu&vEW%}Qphv80w5E!qG$InvEU<@E^N=4+fA5pbhVP|3L_s-?nzE5h~{?JvenK?&8m z<46@OTt;684vkrWHd(f->Dv=weid#e`!E4#*wCC*Rj*Oh65PnqFV(LYwt3p_j;b-&VZIgVA{D!bV)N}2Us{sE^ z!7hWY-+eSqVids@HwWBF_6Gbiw+;h7Th<={=xHHzLkYurQ-7dDxmqAF2>b>qZ*eD` z-W+^;wCY)2pg;?3W5(i8m$E7xMeS00tG?_Tac@0QE3QSQ{0=RpcP3UdVoL8JWbG_} z$$0#oci+KXEYz~>=e!PUK1I28Tvc=+B#dTu75UUgP~tbYyG~P!ME9w}Hej4?^cSIS zlo5bH&j_e-q8{3;MA;c!Bqr~6z4!c4bknRTP z>Di33+zNyX=sN@QZ?>JhvB}#K#8s)G=2jGOxbPF?cZeBpOPDp_ax?ncza1tPryWW^ zvG`cP2Vjin*vTu(<)Jw)vu^>jHyyfr zbZeznc9a;LWdjXUwpS%#Rr`6{0cRay-m@o!XNSl2CWWJXu%&Kn)pO@SyEQs!=hu;t zGJ?EX=Y^DV%`fcOIMU@jeKq1?T-}7vWg3K!rKEkBA~_(DBg_6Cth%OfeUmk8N_5v3 zueLot{#WDk*by4C7+aml$T=GZd8{{^A&^3Is-*VdI$k1=OylhZ)uD4h-|OUCL+>3^ zu`CY}TyS=|8yww-=Jcc}K5Doahcz5^j3#OAX%C?K4A7)vuptX(KR6ddb_piNtMy=I z+xxYa{3-L(5Biyv!Dk~qIowUKuzfyaaQjzTz5q(AoPv7 zd=L)NmuOoA%z^%cKl=UCALR_%^Gqcwc5_*Sk;%3A7r&D2#g!k|?3DqL%o~IHpT?P>v$@nYC zu=R~~&$x|AI>)R;y?50~LHU2!A783EJ;D%ThmN(PUhL5N;PM;^HPuIac*Jr~cjKOp zJxo+Wx?{lBB)DWxqag-pGAkAwpP(WD5(VYA7lJvV6zFUwz&T`BcjECi$thTzy1y8@)6!O zES0GR5`dv@x{IVx0x+5|)|?P4f?0(+t*Cby*p}IED~NbtWf!=kFiQsSuwuXna^<3Y zJrz{`8l9$={s51Aly5t%iWk0ej-bFeQ!7DgP((JQ`oIl5kOZJaxRQPNbxB;K&z@T( zI3KfyQYIX+D#Ezh)y+xdL9QHx&AW5g7z1=K2~+}$^H}&^By4&eYfRvBjWwxk!9m4d z;8Amth8NQ19C|ELba+Ln_oIZtT1dU<5rUB+QiIRE#*+2NCO(!ue5NVib*AD=p5NkdLgz8##u z$w-zy@D+GJbuujdc0gY*JwT8~VKp7A;j?;qSsA;aMZER?#2-T%aArhBO5ENATuX&F z{I|d1g{(iX)AtckI~n@bTy|Zr1nK~0yxU!oOtl^5M-Ef3y+_S=ti})rV!JB?((X!p)T>&(t_4o&|T{^wmd#RNy*Z6cQ)jMa}Ln26TXuSP`pUwlJ zmSyvV_}XHrIURL6O2~3_DXKOnsS(a7DJl^E==O9 zD1uj=ds*Ny#axDluL!wo9`S-lFPMIX<53L~DvT~$_Z4ryNR#t$ISgd*sNe-Cd+NQt z+X7Y*k7NE0BmQi9`~$zotlu?okc|C6T#td8{Xjy?CR6aK0-0EPdF`|_99D)Aw$Uu5 zYV($Wu8T>pR-~1azd*Fjd4#0RwNd1??NV3zg$~uUR%ti)(y*DnBI{-!*z$@-7yWN7 z!s#b2`-ydZ6ir+~YC)FiVd`5nv9vJDE%i{SsQvriB0q`S1WFN6+Of9*9)+nsMW}U9 zodZFea)+Mo_oTmD2nnxaWOgu(5Fw%+8#-kmtUZD}ip*mf61eHf(1&p?j)TseEuxO^ zFrChK^lzg9XU3M-2@kG88_Eg8^ssN{9g97*n;WAy$7-^*zlx#R1rfj^Mpp(*6BQr?T@68o%OwuQQ{u4FZcEA`9DGzuYLO)6<-#0sP zuJ{D$@X#rRH1K8z(MqUtr zU1Bj*%WMghoPJeHqvxH0Cik)GvRuDuQ?2C52EmRRoIL6ie-v0VQZNElMY5W1Ovhf! zAwG1p$@PJP8y9}o_u;$x@tO@(3BT}Avg_VNY(fxXDpxPM-D?YQyGjFB#kFAMaMvu- zfB+?^AYohJV8O(~MLBHea_x@RhMxZH^S-7g zZ#1r_025DCmR-|i3o_M`aE0rpQM^WOgq|4=gH(5 z@)kc3kd>D0Eh{7Zp5fXJ$_EGGJ_jQk#Wy@D=R{;-NcX7jG=$?d=Clc~Lo;^-q9n=e zq$7q6j6Mn+a<%{y&hjxSpz!Su8h6(u6 zRb$2!a5Wha$0%#%C#Xw7!!vmX!OYc#l(y@_eaHwMFmhPNBtW>U&JgU1tw8BCAW(&B z5-Qr)YPv>0cTVqo?Y(jdQE-jvkAyysfE2P~2UItF4P;3fl&TQLNjaQtbv3;n} zET)9`5fLA;k30jVzzFcM`X)W_NsVjqRX%YEo`g=P(KzZ%0*@cf%#n9z-td0aPFj6spxlzx#5jM}6qKq$A^N`B zPb7>s4Dfi$oqxucNd{v{A{s$Smd(GWr1UWvcFf4{SDysq?xTa9iJqfIsD?w*+3NvMa6tX(b6^Z9u(aLndxHb;s9+$XlZ3tyyRDwaerm*E(;xGMy?Wd)u&9>hB{-Oj8Mh-d2oy+UqG;Xxha;hR`bO9iT*nS{6P0aNeeA$TE!yI z2XbHXvyd3|3^^KXfVs`!VzKrepfk*`cWb>I4z?U4d2W~Lb zjlc(C_99X~dplMeVEg^=|7@uQ1X?;WZsc^pv4;{K3c`6sVE5|)rmSobe6z-uIgqN*L4?$r zPu)2JAH{ghK`r6Fw+=ll?~?hqA@{6JEYbpbw^orrfm%GbHV}tiaO90ylM2c)C2e|= zmp#F~fG%dpL1=#!dU|oY+HK4w`Fc&|W5O+Solkd_76a_Go9H-$VlT6RN+Y5nk1k;K zAZ<)y?x&}fK@FCOnvDRNouiC>0m?h;M#6NzJOowh!?R8ReD)I6jmc0i3KR&#o?(U7 z!eAjf#R}%FpoS@!j!pjnxf41wEfv6q5s?RN7-Y0kuLDnF1a<+@AD|p1ibA}VL85iSDh3cL(ZdjfkkTmYL1}TnGYq-EE zrsb+5ELKqR8VE_}QXG`3#na~j6L%SnI1s^taS3q{G9H%u%A%k_jM9-j!vf5~&}HOp zoq*Ae9^(({o}Zu~t~EGALw&_aT7BgYCm!ud%~I=UUj;h#Wi8qr^=Hh{rERT1tpa-e zAT42`M73{b+~y|Qe_#HE>DE#3;Zizpp{61rA{>g%A|d;U4yBijBoQzSqJWWPM86{@ zh6oVO^1gT**8-&HQff~_xV%`vKI&jNb}DD~Q@N12jOo68M2{357V%IU)BQk&qCoQ}UBm;7N zPNMklDner+lDi_i66_uXuyDOXbv7QD#1!Dk%L{IlepiqdDKyw|Qfa^(&YzaP&`?U) zF;H)#vlKoaB-#1Kd7xub5Tn0)sm9%Rg$$Vk^PA6|-pzuIqGgc?cI0&|oFLY1$BAh8 zC%b4K9|C)R?tJKSCS{=>6R)}7yq$v$hfy#U9P8#Nc+Vo5jDmD7gzJQ51=VdZh0CNr zGnEB=xB%Z$(lL^4TR8H3@GV0Y&y{FXC5DE&4Nc)&LM1{$U%Kn|F^?#7T)E5e)IUJ%O zoCO~ha%C4mkFN39{)F3Ll&J3}L9L`VedhQChW;w!0WZbl-$8tDmxb@cdP!e9`Qy@Y zm+>c!A&@fo6$;RBQa-)i(Zx6*{{*Dra~B&a%=@{xvgQP5&4aX*;}Ciwzcm zyog6fI!}U)`iY4wWPAX=ETlLi<=lmXbWj5Eyd=-4%01JigDk5#c~cRwDO}o%zg8q+BW3u41O)lT#n>m zA&ZXGC>u^6U(4SH+Lhm7c1+i}1!H0J$4CGV;~?O78wOz6CX@ zOv$#*q|#p zWt*BV)<2u$s$K%}y(GVj_7XS0>^a);p_FQuL^${(9-)lB2f<8&3cg(^1!&lCH<-{!kW)T|FfJX2DN6Bj%dRpqh6>B}-OVQ@ z9}v}lUaa#l53|eCWcc>681TX3ks11!@Dr+!=@$y-wTqIr;zdLDOz%f&2m$jrIM%@N zW~2gGbzKVj*yyd8E1&u}JrO7=wcd+-waBRCM)mzsJ zV73b66hCz4REx1lpoZZukcHrNnz4tKBaUD20qDE5!p)2UN7@t(8d(~I3hIvPp99;< z>RIlRf*}9nJAo-sNoaVsD=C>PrT9qxn@85*pq$@<6A7zRTL(}&I*Up|cTkSA94a6u zB?Sb5+G?^gqicbE^)jX4M~U3USD*yoQf#A<-(gszF&_N-xZqKSOlIY*>z{Xz%(#k=SIGaoI&aH1&h@gTl>3C&K80=0>dhI>x1nKmDAXrBf6=v{?)(qzplCeD*l`&+S{k1*o%lEMWX)d_@fK zsQs?BD6)zHU@QfX7?H9|?+qS6Tib+AGq}CxH}VWm{&))=w9b_^AH0Q+4b$2q?Ti!+ zcz@Y4ieIKwMz1rTwDR5~O3+lr1@Mgal`~QsOtCBn45;PoD_BJZ23&>+;-wh*j~hh; z>#=20aEgkcQ}h!p(fHMif>Mx#|I{0V;8BOt-}~ZrVo8Z_scDd17bfOjSrSgYE^;9wEQXq4irE;Xl~H>F5e1om zY^A?7y?DgC42>R`=G%f2sD*UofV|Q%p)K}s?JziSH}rT7(OnWR8n2h4K0Unh6mHUS zACUw#_P*?FHnBSf%B^>HWTG;I2tYv%NGDUeY5zr(e>3Ny3mq5}`v(JDjCVxUY~+zo z#)u7@kp9I&bWpenqIGJOv8e00J!APE;uRcI)`RsBKKhL@@P ztqt$B=(mObqyhv8%!h+UzUfP{MLlyJ4IV$?vTHlY6R4g0&6BGfCzyTW{i=$~L6a@+ zD@>mkp>Q-%R(duJ+iX9bU)GqDuBRJsP}Ic-Hyl)>>RHT9+!HXsWIG+=7OYj^xFgie zAo#c|*`j+6A>gcerQ%>|A>>0X82sI@rgp+dYa3Nu))UfXUER?M!lj$Ssd$TAY!$eP zLQ69Oz3S_h-U503-K(N*P}Xhb2kUpn8D-HfdZ86iEjsnBpX?6_{)9iH5G}x_umMnQ zj*Ju=JHV$Ym0OEB6pFPMwi!SB%JJ$G)G8T=ssd)?^gp}0ANMp%PY0Lo|PQDnMX;o+Kd^Q{ree%A*s za(iRJ-Vxfcn-fFRq#Yr-4VG(#kDb2J~JD}40%uotKwly!=J}% zEI!(|f=@0shjI9I@QS1i6f;qZvc0_N4OiK_IcyA>x-(#B?>=pF>h&-oPo_C{W&7a7XB7M_0Llyh4JqkiRs9V-iEO z)z0iP+ZrNo4!N113y2icdDE2%q;CKp3DJ=+LJ0lo4!b|MD&@njP%6e2ZgS@d)z@)q zLFO9?-U1~{8O>n8HDt}1+BCaHzJ5QqqPUm;If$dy)nuIwTIQI=ZQ+ErheVnS) z^CR0#=&dNi{P9l_F590x8oMY`1uk@(n8$f{R`=}kce!>qs|@zY(X?1rnEhK!GmDUo zxlKQi7X%cS<4czzV#WxrYV9YFX7=3Yy?cA3r^O@myo$r*O4FN}JfmRV^Ch}L`>)df zan$}P=2FhtlE?fz{1^nE*2v>_BC3WC@4ss~!bgqfq&za88Sroy&h#_Mg88krj5k4z zm3BQZfJ56tzB)hn!ufnB3J#f3T%Hp)y^l+WePmCae~UlPuz<9<5<_3cRF$YXM48qF zn-6q528)v_$(z<}L5*?2XEI87(&*38Z_8|}c2(WiC%JU{ZVc~ATaZV{c?ZRVHXQ>A zgG?mW13}H!hEA~QtE!&xF`<$6|F{_pb9Ot5uih>_q9A9xzYY8=L0q;d0<(LpcKM0b zI~ya)?)79GrA(q~ZW8QH;nZ4)`{3<2+T~)6F$K}_hVeb!C`L=m61Sq!UGKgx$+f2c_j+sfxOwi1I zjyv5YT2gH3kNNJOJt&`PD|9k6{bF)U-7Tb;fKW@DGix^w2Byem*U2$tx2>K$f8yRF z#jeN*!T7_>Z5Jy&HYXOtJ9!PFFg~JD{{y^~SfAIDeP+6@} zw=gOODh3KlNQi*6(jWp#mq>R>cT0ysND2rD2uL?bcPJ7{x6*=yARygv*Mpw#_{D$k z9sl^pzsEQnkeB`Lckd_GnsctX65_e@bMTH*PLB`=_ughQ#d~T-UYldQ7f*VWB=LPU zL+4A9wbwH$+Ac~@x$UT8c!*u0SA)^(+_k7F#Vn~_tpSI$A~q`volr@(PmT6N-9E0DzY0y*s{yCzOni|P zr>eY1IG%xMw&EyTYdr#72)wV)+cz<90nnR)DM^`(7O?DA`fT+wG|rdNGX z0B@&>L9-|#zf_UrNP_3tRnL=tPy>!Arw-H8>aD{4wkRuH+PlhVtJqI8-05xD;!8(u zV-S(viV~%Q#LHfWuU?#Z2mL2#*}HHLiFsBJCW88m@JnBfA8j@5UQ(ZwZ`GB}z!ypc zkyo0uV`ayro;oVKsFbh`x}vN*F7ySZ*MxP~=N_^sHMYLH&+Y&evVPH|-2Bs2o}FCd zS}WUZrD#J#`ZOu~P(a^|ultRolEB8qoQvviim4a4@$C|4d|ej2$H&X>nj-ICReiJT z7Z{&@x!)Um<+S>TD?PYB6*Z;pZ*nY@PE27bWMy1<&ADMsHcQrdM$pH*FXIdgoe!6U z9q=O0_D@#{3f!e~;!2Z*NT;ksnZLex5iF_#)C1Wj_ur`M+GmSmp1N|B1aUf!DZ(3T zo+po!lRg5=6Y&)tB=;FE?!&WGc{se?qu5*i$aSuc^@V?~G&@62E_p2{lV|UZr+gDD zn#k=2a|Cw%rT-;O7vB5|lo#7Q>g$K;u~}_}^!1y>Gt? z`_(Q{Bo*QHqinfIj6(3~dZL(avrnpZqSY<0Qx5~${UvCNUNmG~HO2|>zPP*Dtht}x zXMJcv)%BPw&m^&L*}s#F3nd(Q7mB$(W+s)HjUV z_VDpFOfEes2p%pf-|j^!F`|Q!jp~RgMc2M$>RD$`v#*zt&`kBz=YXFXcDNjLT*fyt+9J+cgIq2z<-IBPIjk`bzV?Syk~Wz z%6MwC^fC1f)tf%v{MVZe)eh=eRn;ows~YHhvX+jWIBV!t1m3J_$aYj&u;UT;Jwj}P zIQMGir(TvskTuyC-2&E)XpCkbpgc9vcX$gyGf7pl{{ zC;2rbBMQ=os{SJ5jl1#@5I7d!$X67qHT2nT8v;oehC11tRxiWPOrA7_3(8_TX}M*` zmNHHx2$DaRy{W6?@3mN-vo!I(8Z{Pz$`R_5+k$WPYXnC@cE=fo`d3(cErNKqXNKx`AuV<;7!Q)YFD zFK6f^%&eZ187&q70G^5$%djO>nQ~jc$l&p18 z(D<(nt$SGsJB0}R+wu}-(S`e9D|&^U^^uCrhz~w!cFj5b+qr?Lw6_0M#JZ07K+X=)jFt@jAk}&IOcd!M2XeoUB@-o67%6HEXtUj^3wS0qy;H0 zBAqe%vOcC_@doX&Yf&|`{04cZWIn9qvGt0BwRV3<|I0Q*(SPU0|TwQ3w0&qCcc*tOwHz2 zBB`9q;NxVc-h*x?AX9KN=RZLcskmo?g&ne20ZESl>zF8c z+Xy^%<^-!G;|qG+*DAn$gKf@0xPPIORJKVUf0)1=2)XoEyMjZ)hjxr|h zG?spqE4ovi@=))DDW^#NJvtpY77xuw zQ33T%u`3fcg$9uOhJ@!SiLeKb$3n9IO8NnV$Q=+Q&IV2 z{F(Yz8QXWsf8{d3J?qT}EojD&&uB<7zu&`BRi&TYuIEC_{bF9 zxo><$odb{QIICT)p*sG(OtvIc6IB$x2CxZU#Q+h=EzVhIOZF}@0my$Mn;i&C=np&q!j zeFvw9JJ~6BjLcS8CfQ78%gs!3RV6ccZV)_aF%#5XrZF^h-2`!mRLy2RYGv$+J7%MM9K90G{wF*col>?(+757;u+hG&QH;*XDxf+pZ7RHW8PIoo3=y zn(3WG0cv?~_fWbxUU7O2LY|oy}tNKin$5yggu9HH!U#C}U)hQ{ch|!p~=Drl|uj(+wO`7%E5L{^(HI zKa}u(71eSjl7(*jx%T!*i3?xhvZ&tD&pR>?B+ELqC`aD$RH>t&`yFahh?zMzBjvs= zyL5o+0<+_;MFt*d!9dnp*0`~Y&$*N1%pY9EnZ zGO|M=-bahadtX?eK2i>tx`W@z00r0A<_qO!vWx9gO-~6M1u-YXP1HGNQ&6Ge?YA2c zDn39J7X?|Gx@V9Y=;7`GG9%`SpmqsL^A=tt!np7seXj_0zTP`#U6yRx^7?ksIcRdw z9NBedBWL2tO9!8H%j~OFkC0Iv=Z>L`k4e?NXD23?fJe{@)F_RG&W?qE-c8*AN})pu z1apQLJ3Tg`27dY*oNDhD7TK!E^P>4@V0+ltl4f9E# z;!afH=EpVg>-XB-CbkP(X}iPNOHr*uhQU;=(@T6FmgH!l9)G%=yd?0>q}R0q*c$e^ zeN&n|DWuuF!ki?eH8EsYQC6TY49K~D1{pl9B0`uoyad&Pn6g++ z&=M%40)Qcm66ky|?bTrTl9pu%oI@V184tOF0(Zs4mV65pH5^d|j)7oWv1VJoYRYeJ z7&wix_mjfvRK&66@TWF)R;&3J4#`5;L3+zgbtQrGjb1LY!c3b75N4~KYoFwvVJEEs zW|Uwq_R^Rprn!(pa%p&O8`5nKovF0|Q5Vq={&{Q-_(+_z2D+O;4I2*b;C)hNenB}3 zNxcaOKauk>5qwOnXh$Uin(e}ThFSJK(x|TY%@1aQ3kE$GGn`53pGpD(+P4ZDQ3{CM zc?=Z&NgL7%1o%uaT}Oa^<@~}_PMn>+aB1fUkVvh~!V0OUhWzMKmdYUzoL=+s=$Vlx z>zfA#y0ux5k+A5+8V#m19?Vm6R{)3(eRr!EE)#q00)-z}3^iK%qj8cH>*|=8AjT)1 z^_TAVbFF~l{z)zu%=;hhU4ZCZxy`*%Y-prj@wm0ojOa?(>qiR z>zDgyRS7+mr#w`M^QS~r4AV0*f!i;# z$b-u$%k}*m+D;7-L3vMhvjDjrnT@F3btPtkS|rgzI!%>k-#xq);Y0(LK*9C5W!U&k zH-g=iNsw|zN?rJwIm;BVH8y@HMnGay-GJ$jSDqZ?(Rm-Q^@H=;#d}5oaU8up z3c4Fq@HiPyd67l2$&jn0^l?yD0@p>{L%pfTMQ$id>Db$awntZsE(#vAQS7J8%|$*Mtf;8vt*_X z>KDlcIxKbCnK54FmJ$y-CCYc3ZR}G?z8TuYXwtY_o}A65!WyCIJI+;f`)*wJyAP*1 zIRzz8pVw=l-*4+*EveRmn!60Et^4=Ao76@+KwIrmj@Xkdff71+LQHi68~&Y=TJf}; z1PtZR4hen>9zh*BTN@?WzSj=3oJ_p}&vb1qI6#5GVl&k;v{??}-892iLh!AIx#(UB z^9I3*E!g$LnggxL{ls*ew%(t2dx^S9I?az78Zy{^y)udgNqv0-p~V}LpS~w6X9usj z$_KwF?8w%$btR4@oj6i;>ZczAM})F|2BQry-r|!_|biV=iVHArAx0 zb#;KZ%QP-c3aC>MbNiWhC6+u_3fji>_ewJnqHJk0Mzqs3Ur*YBz?c@MmXKHf43{mi zFQ56<&o|3cB?i>?%dkqbnx;}e1E&MZ-qYXL*&q$M(#|2H$92LPa$BJ+FO+aA;oc_1lg-$@Ac}YxI)gs}E;fz}t`oqkh#)~}^q|txFFoD* z0H<9TY*ApW-8Ybl$%h4|b34S&>Gy`ah1pkwm5~8HM&UIG6+Bb~OM-&0QN`YV!LT); za^-RGmI3S*MyUN0RQ8FZUC<;K2i}f?f$6P~P#@THry?F}4~T6EGf~~jJw&%Nw+SuM z7TKkq!8mSp-eL8%)sNm19x((1L4{q(;0k!!CFz|6h8p|Y>l3SU-ISNkLR9Tc2|o2{ zvbNZVzH#yj$`pv^H=Cs#L&5{pJ(!4P8$=aN45#7MRci2f4-1K5)s$SUIdD?}+GDUK zyaKqLZUB~IP&p~uKMO$t3=W+~IvOPY_hzX@MT!tJ8QuUL6kiMv$o+{dIUyRc1%ZDG zux`n%dRI`PUN@*@z~B%dTRy_uAZjfH=xaKbJ& z#eHO8{pa9|oI7j?{~tfE{D0B}9sq!U&iH?76F6|6%Xa3_igbVksTI+o0@d(_&rk=I zlQ-=GNNb3Bo-&`=6p8Q-0fhjAl^9;zKQYG;5t!RUxR)EW+yiH4?5Ax$D- z{|9ucLMj-E;8wabf4u*{|2ujGI{x3f8mI{#c6{t8udbOFdJJznA7Y}7PB?H0wSVYn z%dXh={hzKAy#|ionn;hw3qgv*u_rNRr;9gYrFbhBrM7TAid&0rVRWdWVr!(^Lt*$p z6C}20QCC8~I~$*7Zlg1sHAL$Or&bY{(zIs0iCNekOJ2?P#FL_3y#~%_$N9+1@J@KF z$LHrW4I?oHk}5d?$B#O1g(F`}WW-ho;+| z0HB*+OxI!R>I~jc>B#2?`l9@Ry-iArdr5!`3D`m75d($=H45qoA$@=znMXd0$#x;j zRCdJ5-0o~ovdJvFZP@PVLW`1`gAZ>G+!v3dE8~kki#NPMRU@%jd7vE{RyM@+tUHcz zOa0UF094EkZeL8k#K=?(YTiYW%pdeD5~#vHw`Sj|9Oj?RI+*<4wEz8%m+mFF4_Y*@ zPTt9dUPEW-H?j-6S zE=gSOK~?MPJ38TaY)bl9mNeR~wLBA;NQ{fEiQ9t%hz=PIuS5fYX)s{}4H>$AA`t{3F zxy(9wDV45~XYFgR$P&+^Ab&ZqR|s8?h)n++CM&8=^5dC6ALo(rR72diAE8o169asu8m*6l^04!zTCy zH^4&(f5-&>u9kC1HU3p8;qSM*J_+mYd86J8`SJ%}cehrO!6gZYfWX+*B29Ds4TseS z2@u7GgZ}@4^;F9z@n@Lf#9zZcZpSbc>WjDGVU&{ES6+deQXIo&Ck7r0v2ro7P)yI9 zROJ@_*7oWS5vR3?h=@;?Qr-s%cu9zIfqvv-3|giPrk8PrbaZr*!C4pQ;zhExxqhj~ zs@Q&g>9WEp5|P(I7LtoELrO;{0qz?#P`to4XqX%YWg~hri|Fqgom2Ln=^s@S%uy># za$K2W(rvmBdB;9y%Sua(Uaj;YR5O@F0qJnv~Yl`(gJfx>zORa_^Q9(yDog6 zUsth7G}=Jlph=XIfZ&7jF^nr8t8QUnEMz~PAL#3(23dhTr^7F;jCIs+Ktf4wup?N^ zR-m~#f-$-_jVEQb0LvA&Q4GuD44T@u;0u)RxS|hb4t}rbRTbRe*54>muW_paw_BFZ z?GOs_&rr2VU0q$h3>EdwpmhBXs@HQld=aP_rS6Iu-upr5~rlNZfbx)(X5K zxu<#`q}=F_uj5PL3LxZRkD8>1@%JOQcj1#_1fBLb*6V;oTNdnL$N^<~38e*86;;{q z3|&<8BMYA!8>w0iQ16>yTE!d@@RgIqm9BkM zo4YbuP(?`bV7$vkM#$ozQ$lnJhtaXxnmqE$J8~*wj5nt?V2di{z2+(-Ox5{tI)IQ} z7s?9s^PA#$Yp5Trg1|xi$&G4;2E+!@!JbINIKt#!; za8tqM*OGfDL48e7S$co;j?30^a>d=MuO&V&UWnQ|!>>JPzx>)^U_P?20H_h$3QWU+ zETW_Bws&dnr~KTbP}lB!)v>i5fc(qsi7!+Sn z(W?}`1mp*T9&AE3rHD+q^hXhFXq;oT-M(KBWt=Vf=QvaAI=~n5>sj0@7cP(r`r(B^ zVD}k}4Ty&2b@Q{T_hs%{e;EH%ec&(};TQo{F1wlFbnh$)taz-N=YAMo+UBu&|Md9O-dLJ+BGG`N zE(3{$#n!jZ{pE*wV)#yPy#3@u7FDaX0#BlgwB+97=JUI>2O8zBDrftG1?bn6Z`?~G zY>zrQH{Gfpe+q6T2Gu>zFsYKElIVvyBO9AStm70ur}KK2qs}j&W>U%nUY#?ziia2O zMSK*?)7#rdYz(JV;TsM7_oLtVZB5YNaOOy*BXivV09BVSU;aMBz!3(6Q7Hlt12bxk z%F^_kU0KS^OY7_5H3$2u+Z&4(La|rbOkWQHY1#$1?{Z1I^ZovC$;!#WhGE^$`|1hgr!Ee#(@~vkD~{GF@0Y7BqrwTvcLxk#TKLHzW5ZZ zzR5f}#{D@l)UN0$bEkZm*;<)Ko~Gjy6FM~SH@8!2cEC!lSmAQaE$yeP0I5mN&s+(l zPG_rQld{BB%S8YE>M>zmKX4bGdT@aKT>n!D**tuy&u$B)!a8@Zn1S75L=AMm z!G?scW<{UV6iA1woJ$LxPkZtxh$pIowx2uOH23Z zHS3_gH-8dZii85=)Ik?S!aMR@8qH9tmA0|iz=fkPzxiIKexzOvn`Ow*1KI!;B3?J{ z`X|z*5n2a)kt12rJ@gM{G=?}Bdw1LiW{iF~TO^+SL5NXXT?#vBZ55QYNevX6(L#jq z3g)#K)TbuF<1Ys2ecO-s7>@LKkDN=nN5u>17ng0(isH#d`wYHC>)4kzn7Mr_B1hKA1o!5dO0 z*LxKnU4>bM<8tn8m>>vSX+}m$XkV(*_{4HM%HYu{k%He)pg-|l70A-Hz0&$0%YzGn zE@(R{f!D#X#5x$Bce-%Ri^ok#t`0@f9;w?KoA|H>)7`q&A$L)FXJb*yx&r3oAOT*H z9MO|$6(?PTsCCtlhCNu_f}G)^hX(7;^U^^HUI<*15+qt$Lf0T{yM=on%5G>TKZk`v zGc&QN)@Z81VzbZ>o@c1OXx1gKOt(kC3i=6>>FbWm--GA+bEC`lt8_o%Y(bPQc?%a^ z-f;$bZCeoEf8qi8+$nsRPRfA?ChzXlB`kTzr{<#+lY6+(B0PHQ3;Q-#IW3>p?AL0Hr8{7XJPEQL$jQq&14DcS(7sK9LP&wSgA|2#BJE3-l!KN|I0c zq37#yri*Oi{{8z|p?arEfBpJ}Ime1Ypb5_-2@EWzPI_%q1f3A3-!bm(WHTFRf|{n5 z4jV9Muxhe-wxVR^5+u;rkYqHP_C+;+{EJ4#1^`LF57N8bl((G5KVm_a@q zsq^B!SDTk%0SX=7Zxdrdzrg5Eh3M$p4Da5mY$UIHHGrQF%||M!Da6BB$U33wQ4F5^ zwP8t)F99{fgm!PHb+e>HICZ6Z^#v_4ihspPJbn^iu7IKi12z z0qP9aotV>l;-Y0;6yH07Cmx%~4tAq78H0ndZ534VY~^c zsflZeTFxQjaFM&7h;yLDKp!T^N|!N@29Tx3Vc3Roe#51NZQQ&NXJ&%AU!>{DFwP}h zqlk4DIx$b_T4ojowFe`Ed0lk<^bf_V%{yhq^xC6;@<>HwnibC%uhV{Y+D~(M5y3*O z#uMsv7DI&gFgYMiW*JgW##vNFmmxKd+sH8=X_HV@+480v+xgU)Gt4V*-;zGK`G%mM zM(z0T|HjH4q-LDXsc|x1G57kscf8prsKnlP!|z2vSnKqD*sgNExTo6jkDZ(DCrkIo zK5epJP;*CT$H2gn{?nmk>>R{R4?OWeC{KO(Sf^=jZ7m_7PuR^6>DmB{>G~ zOU-{cP=8sHYfL_~gnw|LM7LB#=>_xn=AyM-^R%^V`=j93GapkM8>OCR7Dl{T+ZNS& zv7N$|#0=J3(O7=lo10BgcAo^YtCak-6IU&U%PD{nD#=nQ5(9HHc(T3OxU>oqsDsLk zIlP6(;b8Kp#c+RsrXM#B8ka+OuJf=kEX`BTzTs^lz28Bk*T$39|ap*-_(A2 z_5v?FZP`6LNUvo|%+)`E2Q}u&lj3{ZYkJ^_X78#2<*s2M_z^3!o@}vt_VlTQ$U69% zzIyrcCFYbDfJ+<>rM;)c}yg3)wTbAq_H1``7=L$1bxo# z=r$TS2w&ixxc#C!ym*hf8Jv#03N_{AUU7rLW>>YVvvS@ePb?+lF`4OY`-;c&NsyG` zT)P%ydI#(_5}@b9J-59z#Y_7=#wj35jW~Pt79T1N`&oVa_U&bG(s|th9PPf*{b+4% z?R>pvJZMS;-0!b%q6<4<>@EUHiR716D4x~VoSy@y8H8f6zVD2=W3N)Ov9xqs`-|z< z{K%TU-QAYarPS|Fi{L|9y8XTW)z%qWN_)pknJBTi8OjZIuD=n5XCuicQw7nvHB>&y zm4Hmqq6xq#$vYM`d&O$ywxSH0)uu-iw_isA!sAFs4N{0_Sf*|OQCLrY#%2Do50Pz_ z<1XawK<4*Tl^uZbo*qodlzan2!!$J@EFe0ehRSBPAeb&H5ahoytO2jfX7~&93}3J| z5NeCPOUrIP^b+vAuhp&9$ z-tS7Xerpt^Y37seu4Hj>fDT(%V07|LUH^O8Bo^|1B{hF*RkWl?`T7IXiv}|Jx}qjG zWoeBe(UXS!hYTDZBEYcvb^Ji+nX*_O*MYP6xghAKLGd$+nRa7n6&WslAYw6mxush7 zmY}*&@K^#osIs#}935w=asiOhxt~OkQ26{oXu=ModbwP={i1pHf^wmuG+M+iGpKzz zYgD6k??dOa9L*YYY*z-A!lvh{XCP6`zMjzt>3vb8$yzVOoH9%yUUEfT88tOEKNf*- zngscFFswLY;B@dEunK%97kH?<-hXcu4zDv+c}gstYX(d1I=^Pp^G)AR=g%_Xkn`5; zw*&Al9KyiY7m~er+T(cR3`gO%MU(1iX}!hrA6N!@N0g7x1Aj{WS!A(c?*nY-frQ?> zy^!>Tx|}=D{mTPOVMH`&2jGFhAA52$ZnT)(e4;RgJ^6!1%7iq}{hNiWH`&{LUhb*dt`6U#pH5=&{t6Y5C{!Qx_j^%3oO9`O zFuAww{6O#7V>KrI3}xW63x>u*DkbJ-#OJZFg4gEu!mt4t*Hob=xFT5Yy3vsnno_nzmiI|eNA1`nwXipfrCn4b+2s)nbYBOoN?8y6S%bwnAh#*nmAcXoCngje@jTwY!t}ar-^ebK@KzN6n4EC0 zP^ZeX807&--uXLt!h`{vqXIj#zN^HuS9k0eUcuV^8SL70`+5OzCtxwu8cq=jz7Eo% z+}D#JsQe5|eD)Es7?7w8GMNu?tdw{$ox!EK(RRNs0k8v>=9Q^d=4ma63@C{*Y^&W5 z+_qqXP69$0HrX6O^`!Wv$4YJdV5NH&httYN|Is?$O9)OU`Q2E{j&xQh!P<*WgfZ%+z-uLrSYw<2Ed(SJct501Yc1r5XTg=}|%UpO$}0gDSVj*#1*l zqf&1^{edf(v^xQLlw4mJriDd}srMd&D?HC!35TXD;;?bITv)L85~M%U4S8=fxY$#G zGgJf&sOWb_0Inp#veT6+MdCRAg6g9(lKOV#>xoBlJLaQ>04NjV*cvb1=2z6@upvf3 z)-M95$N4!uH%AN4EEE{3*ITdwwyr3^K2Oc{7*KRw;4qe9(C7`yJk6J$C z)euB^U73oEmqZ}k;+tCX^*iXM)*X7Wf1s&nRjz$Loc+=E8NS*`tLzW*mf0V=b9>_# z-L$84MrTa=A0OHE-gs_#dBJ8(M9tmO)8wVIHplYzF%04EMS7_RW;7QY-KFeRpHM7B zJZjIDl1oQo2M-(3U1sXT$A9?6L$%w|AiizX$09J{hq3SmV~^uG?ZXV|_^lgmy31y z@6jdy`R{0sAG+54(Idj++s_iMu%FJI$B<#z9x&qEF^iDHmddT`Fwv9#kbtgn?(K(v zH3k?Cx3FFDoEHpUa6ql-{(jl>d1X4Q+tM$JSC*F|-qOrd80@*>C{+^Ib)aaR!RybG z8opcF*)mvU{Cy?|ebPu@lr!N31_mb!xH1nr{XB*i{-M)%81sU)$FE(uQGM_D4HHlA zkL)*jQxQ6DDgDp7)R^Pm^5*cYzw^H@De)aYSu3m$Q5yD%`I&R*4gNYvb^fms)&C)L znx6Z8GRrJA&ZmwjY#j?m7$!PAs9<>b`PtY~aX*QOh={a-@c|=DpZN0Sk650v6DppW zV~778SG%%^_3h?v&9T6bYu>BA0ls1HYRGS}F|l>(j2s!kk)wWCxfe<-;Ae6HUhsQC zcysaJ%0wIICnHO%d;)Ok6WBHS-j`6yLpbK&+y?a1(`x1ONMpu|` zXbr4w(A?u6Xowsh|B-3C7U*V2VkllDCT0Lq;1z^DXbvU`yui_KvX4RLazCrLOZ5i7 zr$A2`gMk&;r(S#|@Zz~UNa2Z|89E$rypj$twDc8F*!;lrgypd9?(W{Ohc0$J3v!4Sm|iTjeKgtc?$`uktY+B@Vjg^MwRjwc<&A13oHt#3@}{W;45 z-#@2tad9CUhS{vu_--ania40?qzv8#=p3*}k#j3H`~KS=CAoofRfiV{Dcc#|g5OUU zP^sq%c#;shQxC)zBX|T3QZky~2*g6Kn-pjH8NL)SXp57}QB#JD^!muih)j~mMX2Nt zu7t93nJOncjf)V8iSBC8%nzH3Wl;Quulz}VwwAx!+FtRHxPpUo9X@C{u%%+KFayTd zA%Rvw_7Mkup&;UzjP=UON|{1+B_$;S=d($Y4h|ft=g!rwdB1{(m|9n2? z(7w-`1L3+6z=;C!4ym8R3x9KgA9NysR4`m;`Yho=byG3t4OCR0w1y=gZQ0PJ=M!Qa zfv%J((3gDS=NAgTU`^osGc%KO6d;ES__$^$AcCXz(!)dG2@-6cH%4gfn}aEhp9AN3 z(YLj;)69SVuC7;7zY#=wDE@NQmE7U#KK58PWKZd&gfHF7==!o6vzcW zvd<0YzVAef>ChSS#9tPqhwo)cDfb&<;uAP}?ARn^U`6M|D*$nB`}0KLRYNEKL`3_V zkdbCjkAsZ-4FEt)ukAfb#EbsCu#}01_0L1b;1Gb(@=1ZL18iiR2M(7~n9k2#>LsD= zyaH%z50noA{rtoW;Wn3oIFT_Qq9%lKx9)<2=TzReI`3xaNW@gset417D0!T-PM1k0 z8i4oZeo|6X0;5Xh!+zk)FHK5w+wb-+E-yMyFO<7S9^k%u>wNT*9A59cv|?Im{64e8 zQ()9mlnH9B!n8mlzY6mW@GKO`RLEvZmiqk`3A}Wx%gbWROHEB8jv&fuwJtXOio^hs zUtp!@j_|rupx7Lh;>t=FRbc*dcn3jAaFmiqoc9M310^&!u3x`D+n>t_6oYgA;PmG( zamdEBW1tD+&FQw`y@*+Oy+?)d-q;>uc#IWu)Pi5XevP?^O;0ZV{P}aG{*xz8h&}9> zzwoIcfKU&J4MbrY4HTwdrdR373JMHl3;L0U3lE)n4&;E8km!F8I|3k~)qfz4>P{3u3RT8JEf9+JqkjaMr8`is{+~^TmmYQ0hH|tzdA(LBiGqaa%I54>Hv*j+d@Nw}gyH#cUOM za6T9OT4WLdt*RP2)sJJ;PQX$#Xm_}`!k;o|XcvMLiS1$|V#2`Il z;ck(hmsa?E?;2Sibho7SGggu-5kHS$vsf|ygwZvJhYTbG1|@7Z5fzHWab*qE{E2`Q z1%vPMCqQMg=0J795F8x53|!)C(HeI*6!1c1KP8+RE7a~C_~DTsRV+%d>?cE>bi%sU z40K!z_8j0Z3~jZ|&9s7po}Qk14w_^8AL{Dhsz?R@6QLbxlRK;N^y$;Xh1|oo^OoVd zu`;J028J$1Hl1<%yR@$L!r`Ox#(vC*Ac1UA9tbE3NC<<$mP`zGw;M2;r>wC3X6@U+ zWjsAgtxN`DPpW7(vk=I${HPGxl?-CEzy-AWGbwGmxy_TpY7z>jm7id$Cjl29xbRn8 zQF`_t4HMYYN$N>s`zfT$2tHmIq6P?{*h^5p-WOL7;g+Cmizb^rPr^e7z|mvF8WFvE|AC*7W*C(ePrIvTo`Y)XFbs9?k-wFg^iGF~K6I0?k zCMISR5PTGx)h?pugBkL)1$*zb>vUmOOx^hb$xhGqCZsk=a_KVqTaXd}EJhD3Sa6X^ z|7Qf`@I5)2f;|lCjSnvd;(%&h{rOc^?A1-}f&9rfomX_qOz$UDu-CwxH3Vj76FlM@ z00nYWWxp8!-M|lok=CNgV=k?|3ippms}^HvX(<>CUS8zpHczVmyOm?$|NQy2U>O$f z$VnjRL_JUAE|j;!I!D35@r4fNBj>02V)J1dq#Q$vso>jdJ9D8_Qc>~qMIk=`^`Txb z1|5g2oM;UdFDOgXqoOFp@h%bd?pK0dkI7<$yMhGPT#SYKnji zBXJ}UqitrEjh2F!O7;2me_`yFnkt^Mw3ZowAfy_STzfJNzOzit;2NKar8m%(tH~>i zCHE|7X(vHtHW$fJH>ySr&c4V_UF%xrH+WN*Uo zcUvieXcWsjN2TcVPj?_RBf0D{&f+tDWN4p+5wi?TrHXekqg>j{wLHjr%HTG(z8YCc zHq&bkZ+$MnZZ@Esg6L%=C;~`|uRIQ24pDpZ@6ZQb4;+^4T-M2gi28QCYD9i*Xw zpcYL|>>c|D#ICE-oYvoc)PvBD-T3Qqk@hh_Q$pT&ohLCL?#mCHMY#yya^HS=-M?@# zMv@?8dkfoN07IkT;u>I};s6z4SEZv(!4PDL_#kRA9WL)ZD;}qza<32u2+-2inDJks z>ZA!YLXxb#e`a`>D;=DkBUb#l(EL63i@1P^#5~f`Y27~LnC*PfkgmP->o1n7G8 zFf`+`v7eiwX=cDCNFqYM?}s;{r3XSeJm)Ql9Fk#rr)*DyI0%{4rz{(Ssjlu&snz4V z;es$(lVLpQZ$X%K>v%N0!Po#d!z~qv8&~FTHrp(Tj_y?(1*vM;wXVS~%9F(;kvR44JJQ%We@%#9=v>vm^2P z);$F2hHHZV)QyrI&=)Rle*wmX3WT72Yd}9Rt(>{SAfl_A)QU>PY0~5QyG%rE!5MRE z5p)pE00n)Fg|Oq%6x6#$a62x4DMA65F3%qV`p*r1D-=Kg@><>7M_K_u2U({`2$ep7 zqD(;4nAml}`pWXj44@3-0($IZgf7zo_ILZej(ny9s|~$IA?y|cJ0QX8##AsXQ|*1| z8i*I3kwG1-Mn+i#XbRe-lwp_@0r-+!iY|bI>JM+Yvb4l@n8O+UmdXr+B5N5I2j8He zp!7;*7#fkcb-dK7oO;V!L5-#d{RFyFBndCI(5aUfy1=&o$%{NOF>w+I43;Ni;=il` z`qM9Uz)r;jx(a4!8RiazYjOw$3tw|6cLm&X%NaxqT?v9`WLcQ0o4qim^Y!1(93u4Zq04OVhh!qE|0d8+U02@XAR@vqgc9X zSlRDSJ8VO5<3xmceatP!`qwfXp*nk)Avb`qI_Z4?oJJq>^MyFRjooz)*YUb+2? zb1b#^#Kh*r&5_Kh2rGiQ7?u_Q;$=~g$5ODc^z2@{de!hl79vBA9zXs1R%UYk56Fj; zL5!rYd@M#~t>->q)@0)lVN$|yVwziB1jtVu=oVQQ4Y1`TgS98$?Qy)Bp=TEo*p*Pj zvP!4)u9o021Xdm>DbO#uWs4Jz%xWFsp_#5?FeSKO~4{zO9BL7Rrwq^3rRL zGePEs7VyL)z;~jlP+aLUT%_wmZI*%uEVx42!g9hMe-I4MRL#srsTk%J9KkaKMd}y_ z-|mtTdF~K#;PjKIeIxj;aW$}VAEP#yJNrZ0-H*Aq2x_*e;1iFDr5$2}4yEnY`|tl+ zw9!q8eg2t#BXg7Cf+G0WJrW(jC1A6^dWtV@#DkG)x%T$`^7x)P&44+J=Z;X`@d89f20UG z5-&bLx&(hVcH!TQ<2O|I@3}Dk(H1zAYERTK{X4;^|UEu4@q<*&E?M-9`1cx3wj}8X}x?VVbJ+!j2qGXv$+hZ1o^6~koO(6@ zKClEx9!Vk1455-D#^-i;S7+R}D+RwI`?40j=)d|X&H68bF0*kK!D zMp59pC6x-HP$nN%Xlj*0!>48dDDG(BjL3aOr1 zK^4T6pgap@F!7N0e$>x)^ytwf$X=F*Y}!9Tq!seB8_6-PAvuCS5G!JosP+CSJoE_N%qJ?_)er<;zD8cQ2R`eM0?p!^!{=?g z8hfGlli9vG#GUz#94?#h9Gsm~)7_Vzb#HGY+6@2#^OO_V1WhHQZqeU)5V(Xy35e{G z&yc&_?;k7WYK96~G;+VAfc=eq^W00QN-D_J5C?g12ryY+hZt#SKFifYYw>i%tExTQ zCIFF(fN?N&x*AO#2c3*FR!J9JoSu_=oN$EO7qR zqG%(sb=@T^Qe!%3C?qmhE#h6X70#{EU4n3`tYC96+*PshZrG z^dTYrS{o1Z8(wz}NULGyFyKG%vB;?&hl2ao+ ztHx_G*xeL@b#~Ay>GrofX8g3#DC2_acpD^iI1Fi!pu7f(22>-~O1X>-r3brzAZ-X_ z&dSTv?Q!4pKm|er^+-(d(({SH#d7ivEW%xReaoyloSuT*=$*{7Kr%A2DGT*Jn+pe4 z$wvl&go|EDpVr_nDs};*b;@7qxtSEEQ7TRkR+R|2dtd$ny%zXqN$RUqJkG}Q8o_@G zbyY!nP{yaPypu~No9xRIe%a|6+b;W)<%US_yDaV<3 zZ=8Ail_^PtL#)-@OLnaWQNS(~$mn;OAmEgROa{dS|HK~WQ(9PI$MsqdWMcKQ}I&CY?-bcmtqO%FF3bM6kz>CwY?~ls%${s6q(qQ#S zmsbj_E=K){3*)LsfWSHAQTy+0bX$H0EbKeE)(Iy6f zk)<#W^aBZ0r{=6eCf8mwSyQlSTjeDPxdA=+lms}&a{X}J1e&MSDZ60+xYddc5T)J- z`j6-@6_5a|ifhJ&9W_`1DZF$9Zmi!Ndu#5p0>kBzDlTf+)B7RevrK@re6Ggb*7=!V zOW+!C`@N(%u0%4~-E7SCZ|rOJS2SN^Xj#9P&2CDoi&M=IFxG2;tRa#wR^fF|$7l>o zT1H*m+^-2d6$-gyDL@M|%hRy#rtMG(9`kZ~1J(^4Dz{~K;^kS^I6b|*K10$)1@wQB zF;u7HTnfEaYRXAPu+TFNhQ%8!*ZAxFxLV2|gww=B;=xQNjcrSleZy(mB;#Haq_rZD zD7Vg~tiCb6szmEo4Mb2%I{;b_-i^XvOFmTee%pV`7~tZppdVaji%?Jh61EYFFH*#NJKUfFvI8JyR4aB_YwxOg_e_*Y$@-9_AMRPJ%5 z6v3Uv8&Uxnyo~8`LyEh5IOmp(q@+7Lt!G9rjZZ}XD*Gtlr*>zODz8(+dT(Z)aZr(- zU zl(m9yqWRHCSWppVl&)Hm^TqFku9NJ?JAKaN_me&Pvo&a zFL&9pmNhDYY+fN;lxvBF||_W96AznDMYj7_TOhng4#Gv~=h4Zf4^5Q5`IAg|)zyuQP-AS#la?6~8tSF^I#;hO+kIZN+-){ZTx zKxsyZ)6)nZp5g;lF4yhC|HasQz~#Kh@54D9gm5GylvPNH7Scf4CGA~ETSJSMvg#m8 zd#9-#?V&+v@1@c}I}J_k=lXR0M{dQl!ML1VjG`@`fv#_r9B(wjg z(Fv7*)Y3aY7kmzX_%@8mR_#Oyonn1r5x2uBp-Q^ck*pP+xBh*eV;q&#NbHpwKv7CQ zBO6j9ofI}Un1-&N=ZL;F^-Y zSH+*XVb9eBEX`xk9tCXblL|cbEO9Dz%R9vPT*+8m;doST`+~XP+bfn!YIORIIVwu^ zcdMiA3&TVo2o0Uskjt{O(OB*+YXZyb_fyA%`z?<$X~8O?=vC423BTrJm!)V|H(cp6 zs+mmOyiFd+c9NwvM6~+Gx}61gh%%V^Yy(p`?T%dQ@u-*y>)}tQ+4Um@X}0Tj*xtpw z+qdeN#g7RPAbp>QI(`D7(}+Xse`q-8xil>kG1EY@Rh*R%Sri?B&h z?i)}02%ydS$q1MJ((Z~{6RI~+5mj&`u(0r1K{0yxH{-=3KnMEo2MPvy?XR$nFxo+h&SlPfb+dd^AEBaU!IK7}m$ z>U}@l+Q4L)%Rc|{FoQprkwLvf!s<9gn#mD@bf5WJsv`Z#qMV1C4D2yQ z6YT2}=nO(>jx>-76LgU>uW#L8KQCrSE0bFE%1fcdg3%iXwZ37Uzqdj7iWMs`7oN6A z3;J~W;nJ;n^RS|(Sijk3PX#-pPnoYM$D?{CN5Mf4@3F)D1dO^jUol?tZ&E*6{J}xv zdlIK2ZF*a+iQ)j(!GHSqSovdyj0H3t*FRQosEG2~4lkgU1Olt@S*8EIC%~i>Gwq`! zek7Oz@_6+BYJdebtBK3V7 zv4!`Lt=!J+0Q8j58n=!Z#x%xU;VXOD%x#y+f+ary0JUCZbq=o;BH@&seHCf|PT7;N27sKA=0Np+dmifUZJJsFxwK%MX!0ER8l%flVH@y61kuN50xr(b4S&F`QQqkr}o%y2o#|j1-wyEv1DOccv-+zKyoE zRC?by$>6NDz6N{$n)`z0G*jNX{Oh^Q6*oKqmrey!*7BDfv}~_cEp8T)7euFete^o<^gMOhwG!bjCx zxUGb$$MtXEh(MBnu8uJ6GUp13`KE5Nn?74yCpVxv_v82VXDb>C+o zwSK0_EZi>kW?vMUp3uiP)ZC~?2BM@d$LZMvhPs!n?ZGKDbc&R!1bO8k6BCn%4OaB6 z%Y2Z6!Y3iz9TJW&syFr7P7Y`@DMPEbeans=AkVFE6o`Aw9a|cmi)$p9jyCP4QvH4= ziHH(P*7MHgTJy1i+!3!=f4BHJ zjb&8T`MDx3SDKtSA+5)2=C^NnVoZthv7uBhw*In~m{Ug6=x6k*){5+B)N`xdZa@Wyb|Be z(}RcBMa_qXA>HulYJqJ%~j$wPFZ&5;GOLF1Z_ZbWMU# z@OF{%bA@CAqdz9L?E9b8QuM%$Jn%AouCddGomaglCz(`Iqpf)-to0r!1gTEtd#twP zoQSQ;m+)g_{5!VkiG~Ah4=?EF=tGs@r1tDGzmYk>7a8(?B~ot5lbE)%M0W1qJIan26ka1>A`Jv!XZ+%y!&P|5{RboD?>Yq?K|X zmf27aid%O*<+fLpGllQmvu(1zT2IJXkY(+YKPWn_uSl;~-1x*my9jAZMXL^^u|+ru zbH03WPJ-&C{$6Qk?Hbn6? z$sK6e8@75OH9Jb7wVt92qdEexJ)Ei;c%1FJ*e73jUusm z!7IQD2}5t(M(?=NJr_-fg|rwpS_i#bkI{hp7jc@#tWvn!u_k0!46PFrV;yO???&S(*LR#bk=#nAH)4)XAPdi@mG6efI>+Q|LC(t zA>D*f%h+SK{};CLo$@exm%;7pNso`V^I%rc?NQAfDtps$8pyGgt@l~lUnIq_csD{R)UNiG5qrw(p`?{#Wg|J) z)A>7_54~?{5)jrt;@7>f#*NQ(aLYl2j_JnB>X0WDvao^%ZbM-Fll+` zYfN~kt<7@R3owg(B-ZL|vT0@JjWXjIt)|5btv_aFN{{BY26R)uU@g9@&ELwS@uDa| z^x55+^0E7@+r-M3HXYj0v5+?GHD@T#RVJ=r?0KN5im@fi^v>(rnTI_(GgZ86v=^d+ zxLl`YlB~9LjvVqyaw>JPK5ArKHyb(^dA|Ru&dg&IZ}Gm#T@TZJ*Uz9+Fmov$nJHn_T`RN&EX)vo6J=$Jk;>{nNrxX@W?L0pqmm^HukfYAqJJ%4qK|*uIqc zrjDgj|0k_RbR!RjUBw%jSD0~5IN8k%nFX}+w8RI0lFhrX9LzT@oMkpNb^O8iHr2cc zp4#|CWsT{c)bBOpd{yDLhMalV)Sz-d||-Jgjnwo5w2W?wB|KlJ}F4IgwAVWZY#mBA@%ul%&Iw;GjIOmzVgP=@pKIbYx7<=P+L5! zM^EXWVNy$Mli0ymEb`_~M%U-7{15Y`EbdO;S+O{Fwdzp$SdE9S_V;zInu%6M)6ch- zhqN-gVo&ekdb*gw^Uc8JU82#xEi)aaemjSvqypoasSnlk+Z_qmJ}0^ETF$j;hcba< z>cF{8HciuN4u2ezIcekeG2rGs%}7a~v7KWkGsz)btvrlN-e;!syF){pv>V^bKU|~_ zY;-7dnA38t_)2vHr^YYL?!4#3$x>K6sJlaim+AskfuB~X zaaMyHpa!7FXg zfQT~>R99yP6pP+fpif0m(+IIcZhS>b?(MzbW8pXm1Kg(8g%fS;P7Mj?!rJbu?~N6g zk`*7+BY%MZ%mNqaQJB8%~g&sVj?w|(EI=spl%9$!LzbB-l@<5uZ!ZY@JM|B#- z(k3f=U8{MPzgq7qCGev>a^oVo|;BYK)rL(bK#`N|Yc15h5_vs92Smg4#qJ zveVZyJpt~}CXWAkdtTHzPEm#OeeyAG?$6clDLM@%*3a`RM({DmE~0Op$~Z}7RB6ZQ z+_i4Zz5IDm&DSh*B{}VqVHRG#jo$I6*?V6L{diZx`o4~S8^%UYBR^lHvi6Q12#EUJ z#o{GoG9sn4bqZhkg_{txz_y zFLf*}9(wcU&BG`-n}6WBjk&^LHk3hUGMgye_;%$#b@Oet{vlVlz*YIQ*v;p;-Cvr1 zya&Gugtz3;3A^@EuJg>>Gf~`E9qlA|a;vL{;hmi}7^Av946SMRkfq<2R=r^1!1VZS zFuXU)bvd|YBmn(t|Ef4H!llh|&p%`PPZBnu~AXtNI4&VUl6QUDGd#c zjcpx4Ud+n1E;^V_9ni+WM!&KdWvVt*R)n^75#wCPpr0?r^JzNnq#vcJE?TXW+vl_l z2QZ0jzUX`=B_0W)j^Q~}6dPlX`R|k$VrSzRh`CbZvhm668cHv`*YacgPs@T))BKT- zx{L+#fR^5Gkks+}m)BMJ_AMNxC`_9N!u+JQIS!l;52o8z5nUafE^8vP%_)>n-|$fT zB@J;DCwff6^W_fWv967$hoj-$Nuo9m&sj%tOjVsDCDx^dokL8CL+gBac7D1?ZnRW( z%dd~qdF!9)>&Yusct?5ddb&_(Y<5!9rrj4zTOMTN-4cIc;{}ns;ZL}%D69xP&HmnK zAU)ePor)syTeY`dj{Nqt>Zy%JPrO~c8>SpOntA&N@iLPEQ zS>E;w>^;^Lrn~`H<&5o6bkXN^tD9HW0vhVrS$qh)dRs{E{d0rn2b{ zEI4%T6W0zucR}ieO_SV5LN#7|vx)yh+dksbO!&$TYli(nQ^L!dd8^SqEPd_jLWs z!~dFP1W!}$dbG=an@2@gGFZ&C}HMSmC$H4_@t|-YJXjXtm_8ms0DQ zdk>vX{=Vy$ZMA7%Sx1-K*BlekhjVre;#O5WN&k{xZuPuy9980;pI`zI8W0$^2s&X2|+c2xBeKn7$=6Si8@%hQ)7H!7NK~rGEU%I=sgWSzA#oc-0 z>V0sS1%XN&goH!}^(_%!GN9G3n{_GAm;fvErYd0-{`+t|WBRzD55u)<7)c&!nF6)n z47iQo5ON`of=aPv=guRL?f8H`Cl0267jU_q?YgHv)E>xjSCaZE8~W$fI`h={j??bi zbrjEZ2k9&ZmJrFU?b(*%811nT2ng^E#TrQiWyAqqFle_>M2YL+7lUbuBkP z$Z^5yL1604$4WO3GY*dX$P@bSqtn~&{d-A&4pAbIVQhdDpsP|3toD20#NfN&Uqe@{gB#ccR$GPF|8l$$zWJxfwx4!4JPRtkwx8YowwHz!{pX%9cm{KubxnJx--d(d3jJjIpB_vSntk$}Ae^B}_HtiAC zowVEJmp3)6-kldY4vP_D8?@d3l9_y{TrLK1RczSRL1>Ru6UM23#B4mB!~|7Tmz8%B zy8J_Uw{$JnXVS^Dv*MGl-?MeEYAROzv>TUNLC~jc8ZAb=%X423vinf1JN5(8o0+V}4Z6!8%$gaszK=;d`c%=>(1AMYtkN51H5>ZhKZr!7>20gkP*wKR$;?TC<9+!(k$nP>-&ygV~bOBo-h~^zY!pk zb^QxV2iGv8?9>udxK*NipVFLs@!0s(vnvhs&w5An_qM-kzxbWj$=MlXv$KzWX+j}o zJ?lWve$#1g_eZHqmo~p3B(pCkM@@l9A-0%o1GV4=f3WKFZM*+ zvTRX?`;@DC3o~_KQ)*Mb%z-+--#`5Gzlp`5kHZ2`WA%m|d-?eI2r&A6Qqp?*-o1OW za(+h^wZ2bh1!QJsZk68~St|9xaI>6rI*pC&>iP7uI`kjid#r+EE=lnj|L|Nj^XH?K zFE)37m!K~+;{KA(h)Wso_|H3LvidOjowDe=l!cl7#89gu7zcVzFQ3%saznPdA~{&YceaD%n!mUS5ir?<5V=VMJ|Bf zI^5a5KUZOQ$AOwxC-hrRGEI5MpGx=H{;R6O%NR@MpRw#(+!+c0QRg-k8Z)?4aF>7} zwI5a)7hmSbqHH6sF>Lm;7R>-$E(i*W!jl7poOTxtP{E~0Hgh-TG&?7*^9U2$BZA5w z5Z;KT3-`H^wcO=;7!$#uc8zT5A|PnOmoD*U3GhQG&JjwON}QZ0WQXvaM;qv#ws*8D zX3Iw(y4}vF--0t<5M*~k*Khe7OBF6lw)gOEKcLQ+$*6lQGGXoG;TOnnrCyKC0+!psEqUxpD+Os3T9ExY5DT zs48`F|JVOvNtc{fcqt}0$#*8lChtqn$S}>|p%#T{P5ZZ2ePl2A`7o8AwI2vbNi2I1 z_o`>aW4o3t4XMK|`6Gw)Y%ArijQi;ZpLlQ&-GXEv`)OjM2qBPfH%=47t$2)5qt;5x zeA%R#)yS8HI)T=%VpQzS@)0m__k_h2zbBR`t6r(8#I+L*qDoPOU zh*?q8Wf?Ofej*tEH1WXz|)eLHk<~yTd@!p7DgNuYr*v>2{@TF6Ra+fiu3IS zv^y&6hexoXYLZW4lRg;fjGc4DUgj6+jZh@5|3iB-cuCZ7j*b+v60!fKy>$+MVmiJ~ z>06%XsN&t`g_YV?k2S^w>vRtmvmSq=H9MV(0NN%N8~>p%bA#?fC(r&qb`*|rwViE0 zbnitfQTdZL*>%^;j|H+cB1=fI%P5G}J~d06gzj&a6%g>=cH%^TTyhIk zi`E$bX`_IODvjC3UmF`q+gHCgF5&Dj2wcp>wvqw;SRYV6t$s?KC0q>be$Tw>99UQ1TU)#+jU#sw!uMFX5B~D5( zE-vM9buq7HIG***V^VAykE&vpwP$ zBt#TeX5s)0yA&M$B=hxccYK-UdQW$wJ?(B!yA2p}s_ME80){MXyBRcUBTQ#Dj1YeC z&xc#{y58^4n(vuMFmd?H!C)I+P){Ofd!Zzjq^R{?xn^TAh7;7+LuhGzQ>)j7=ubj1 zS915y^%C+@R4fK-WmfutZ&=4b%=6Iql{{f14OPR{9{8iVF}+0euY9Ls|+ zJ&|}BVmyZc8^6u4wpFh<4S~+_6=GDe2Db3Me+JFop^s8WUqJ~Y1Nob7Y|_AZ(^nWH zHAv9gPdtP5o8L0Y>m*f-0cRbGjetBo%_X>}6F20T%`j8el$D>ln0>n@Oi(76Xn~IMg5gLYZ3!})}IvUUc9UQuINm6 zbD!@=rDnTf&5{s?9Jylovw_d@zDv*QtT)y9<)}MhXxW#e=}SyU|37pG$`Kr^1`J?$ z@aEhe4RvREXhm`=#KPeNlTAeKJssB*H2nM>av+R}#7H68cG?)4IhUBx{@z}nSiUKM z@^Z6s#DELAduJY|FB^e@-k_5?1Fh9!GO6dYgQ3+NP=GXN6g0= z*Ch`(rwAjsjW1;$*7+si1YNe5IQqB2q;#9`y_dL+#p`u*g;KNQoi1^zsV1nTZ4DxQ zj-tQ==}ZwhrE|q$T~~%M4)X{{QaJZpmWdi95E*bE{=B+=^~sV7UXM zhGOXOTABGVfr;5TYxt)W&_mzvVD3s)C$SS1_90pKUxSwqB$?@ z)WpH4P>VCu$hJ80en=AQnMxW*Oz#{z5OK(S8DOw&_3h)nsR*z0BO4y8#O8Q2S=t^H zwU@j;nFSsoS7BJJUQHNT#8O^^LA}gBEAaaT@~DkmOeiPK;ZN(sOj|vRqEI~_ybhTq zuHsHJ>%#R)&694{Ehf8you#nG0rOKr+*Lk$Z{YY)f2fxj=QVwCEVUBPrS%__pXnbR zr{zk3^*iNVIQjYZ#>+xVA5X*aweC^0inMH8gh6BtE>ump-)o;w6|EVFQdv=NmPeeV z&6%L!;S4lvZr4an5yTY^v+)6+ho z-`HUNVmgW8W2G0QI-qUCW`>#|sM?qrQr-=n&8Y3{0{JuK3)dk@|7!`Wxi1$`z-#wf z_MCEr$g>>2#Ly~G`VZ1B=7R5xK2=}+#T(f@=^87;RnASIPh~v5E5u2x7Y4rw%ecRr z*G3M;X$DcC*!r7?3Y4P!X`45?fn{qjSz(jpLoiNXz3A-hJV-x2nY+iKdh>NV@7vB! zPX0$y<^UvivfZh{JYIq>rxD7yRS(A)J@tr=)QjrlYm& zei{A1=>q5RY*=jn{{v-aqL|vgY0!^;eIh>Pl9=jafUQ5noZhh#d;i5 zZ5*VEF;;~ScO@sDN0GUsoqD-l@3gK|d-SYHM=fKm)iJoZFS9eprpo9Uz&22r z4E71rWX2W)QpO9hXdsRdc6=G_B}*D&N9U)?REiIAD;YKo1nUSFc4hP@W#_nI95x!? z+6f|$PSLIM7d5$Rsg5e7f(v$$%kYLN%ktHO>NiDi2S?23J}>g=Pk?SydvxKVEf@+Q zWCd^CBuvlGJd2&GIW0N*J<(4?GtEW63$#~1d@5Bsga{7KLO9|(gOZV9|Ni|ox3Rjq zW3_J730S6vY%&}fTF{Jt4l4#7-~|@cjFOd*)V<${&abH)sL%h!5b3;jM_RQpNz>%^ zi3T)5%`dOq%i2e^cCE;zWR0eNt?Vbpab@HrP&5^h>j)2?L(D47AsOZNyv}|(HhQV3 zFo;WLO8gyZd7@LrRh&F08O+m6yUo7aOR)?24<&Ak`afjnCebDO-IZiIc<|kBbJ*ec zj6ed>Dkq+H3oHx?U)g(pDF0qQ%RHCE00T>6oz7uDoQ*%qFbGLL#rPy-O9Lq@?oBM{Ed0z z=r{CWUBucQ498-yB6ojO#n(Hu-i3s3V8a8e;ouL3njqNCl&8a^O>}X~GCbF*0#AQ( zBpiG13+isF+{alqSmFoR<};*Z4H zpe2+=-MHgD*tAQ1_Zr}^NZyIYS11K(ECTigY;ZMLHK!kd1>PMD#S+LL%Vz);h4iO5 zX;>!i`-Fz!jC^Dzo?eOwAj9BkIJi`pHy%$2?j_6ScdqZ964Bu)UXO);`X8xGLJoq^C; zu-3C_2!y1k>eqev_QLBVzLde+;1&leX#-W2Aqx=xWnNuNGOx|V@jKE9QM6bK?q&gDd*T;rLi_2MgpwGn^iaEq%royB7ZR__a3J z|6Ut-BN*oUk9jSN&!xNu3iP0=mhdVEWwqHeSLzsZY^xp(O&b}*IGS@?PKEh7dUo>e zBVZZ9Ab7lPcbxfAHmD!F)n5({t6HgocxFL`Q-mu_jnH*0Z}D%RqnZLE#~9)_ zsSs~i4GV%5bftYWa$+P2?t16Woi8OjZf~A?XR7i~pu_(xKQj-tQ%6z#98&xrfaO!G ziBCi8!nY@Wh|*V@`9UK#-upV&A?EG|aZ{%1(;<%83d7U! z2ecx*orLOA_{m#uU+bb$pQzqSFh~lDs={B-RnxR73?>6StcRcfZ+EqVhewIUP$?`I zNj@zB!2uHEunAip^djuOi>@v^5l9qz<=42emW>QFUeR^xTF7I`aK zmV@3ztFY`P08D6P&HGa3b<_v@ zkN`CP2T^yW3Tu+@c#VSU(Uxt32Dr4b4X^*#j$3BW4s{rv*}<`+NAVTvS}(Y3ib2SA z-MAGyvaGZ@$h}j2%Uxf=n_p*f`TcK+T(*ETFDk4L@sbD7ze`%Rl68CrB&XVT0rH%k z23^7_<*OoAkrl@A%(A;P`TO8^C)3UlMO-;ZjFwGC=3T}6aOL;_K8Q>pRS*T+r;5Kt zD%oUQ$bl{h`qyL3ePCTUU~H~))u;hI_Y}0M0?MeXJxPAZ(F5ao@`vR|=RPbGJ7{TX znc&ZPWOn~|l)3K6D@mPNK7O4S{*Nz{D(1xm@wDz9r{kybK{#6*;XsqHu)Ob|`s-KW zvi!<)TXxuklPbBEBny;m46pb7C69$oPTBs;_$AX}Cs^@E63*kz5r|987Wa?QCHq*h4gi>Ag z)`f_`Q`GxEpA0`Z+sN{Le9|jkc|P*i=hyC>$Iq%gS$r<<5Iqo$9EQZ@j1BVhx3(mm zTCh6k^^~6f7|AF>dwy`{{Ufe80h&aOTonTB23MK|*P3SEx$yhSO!?=^Bv+^I8|3HM zNJSU%-25KS1>N8(;2QUx&eV z&l(renuXi5{fCJEG-|sGgjC?5iVn^l(ksQ_TVE)l4t2%K_P)D=S|g8^kw3oU*vYpC zWOGd}HJ;UPi;wE6?6(W2?K>UVzGU^WZolFclYY)4rv$x(SgzC@)xDZ9C?+@Q1qrWu zSh9h^jgXaz{I`-l=XVw79^Y;K>lj|+(zN9WqY<V}6Qu86(K;X5YUEl6uh(d6tJs)iKsx_V}nGaSE*eMo+h|tZ~F`ie=D$ zG@8BU(u<7(^!yVs<_mMiq0*{Xq^SZ7Y=?KCTmQ7@-&t%7o%?O9|Ba=@i3Q8<7v>Sl zwEcP*Bx*y4$98jAO}3eX0ixr51pkVWz-?;4#NQv`C1jbaunl{BbPNaWt~t%3I+I?2u{Eee0zx0qeSB>W#I(X+IFgcGwGc?ApK zJZR$rn4n#^I8iv6WiUxYj)ddj>q8UYOEL}IFb_#N7J}S8wVL32hlX0(+gDB*v8E8Z z!atX3?T#J1SQTcjzC;;JP}nSpJjw%3N)cnBBOudmpyN@3_eD}Y7f4j7AXW@729IJn zN;~FrL{>;m5(~w=?Uz9HRvx=Af%;7}0^vnBh1oaok|&eLS(TOH%h)L{=~5JTWWrI4 zbx-!Q;8HM>p~y}WtB}A132Ue_saOiEX^MSX!9dSDR9`)a)XwD`JVj0WhS}YQ4Y3S~ zJ@5QBKYa2e#V`OClY`~%>z>@6nYU!ms4lWR(}oJ_-<*esiio~a&l6n-$7@no!Y1+> z$>_;th%n0v z`||BP>${RqO<~;w&F@^hnGe?XujDTY=GRX0%2^9Mk~jds_ips;@0#$9$M>JNT!Ar- zkP*SLUxT1bd`y6iICPPy094w<++ZG~+$nBg6F7R5SuM8b6aGd%)(ZANSFcuSx(N?y zC_nC@&kYitNZ8?H(A3VcVYY9xCbR=KQtOxVfyj?iywjai^|YvcbR;x1v{BQP>`SH^ zXRPRtkNZ!{n_4J#GOB&%y@H)v!b`eLWHYGwJ(f9PY%*9iD3O16ZJ+BIDa^4QEnSMXHMmKMD)^?`@ zNz3&V1%*$-aU0k;StEcj0tmkh0Jft1}B!l)a$INK| z9SLfozM0^5{zXgJL5}Y@4sxeSMF;Zl*2<`peID=1bCM;e?tu`|2a*OH#-U&Xx{_6p z0ALHR;uPrGaHGfYach@uh1u1^7#<*1UBKc>>K8igi1`B<^ljR>Jh+G7MHf+xXs#um z3!sXyM=e*M6feG%j!=?-l)Su>e5yq_Ty|mo&_VPvp%(o{++sPRT$e1#{rs_12{zd^N z5P^y5?nX^~*}dV`FRYvaPK&^ajD@%;qWQ=!-+qsrKF0}j@9F!*VZzOA^WfH9wr4n^ zY0iF0+O&Rs%J4X_c0ZSQ%#j&Kr2qS!32)=MldD-)+PR+7J)95s?w zvp{d4)R5X+3jYm(@a!OXRN5FV*Tg&zd7VP5%B|`KV94kWD(U#ldnkdZ9|(@Xbl!Ld z$Ub2#HCoR46=quwc45=-1`^=Ml{y=(aFj^lkP<5&79MVtPQ1g$^HdJ0H&)S4jITCA z&Q`&O@rD1@tHb{@_Q$?U!!178V)@_b(wEfG7aK$tlkp)JDNM12il);>M~=W{Mx^G1 zd#jX#Z(LD&wRV{mxAN`GIo#s``KoARzMUXeRA3av42J|evU(XSu=>-b*^gl~ zHj~cX-h=Eob!N!FECW&_snoiSx2nY$EpA1}8LMw@B;alx4b!#Rq$&q^pY^bx1!k=Y zHil3822u)`(wn@le_x~Sxc&w%g4q?>x1Za;ASr<*VYq!M`TCY%dQuTFvGV%D^CgT8 z&7jO1G1NbS4_yXt zRc3f|NR53YI&zGf{i>sV_T15y0d0Kpt?yTt9mNs6a<~muoW3)dbCM6un|nJKYf>&0SOn2Mwd@NLhlA zAVxR8t@aC0T5G*6M61o%6@N~eR~U+%@Ok7s!6W+_$MlKQBvh9aLxEC}z0&-Ka~O!!}J}%G%yUBf{}ws2S!_SiR@Nj=Uc}XhkfjrvSx=RtO_XCNyub;eWKr}#?(B?OO3vV4* zdzAKH2as!i%MKu$!<D97sodWC2{j7GbT4o&A56UiC=!F4umh6Rh3j-p6iH4TT34?Ute1U-5S8-4 zOkHgJ>}mI4;h>@@ z_4VjokY)U;lA&G=?lRX$wkQIFB$mTop0^0+>TvMHHW)6!Gn?TfAE&)UC2j#pnoCrV z$uhed!2VCh^cOv^s*Huz3}D}wm0Iaj%m7RH>)x*7~1q?${cF5 z9?NvWB>IFchB0ojpD*ueSFfsp&hUEd^MS1T^lzf~Qey!LN6Ex>Th@N%M-AxCu6vRC z@3z<0Dg7a)KPLJ&JvxP}4A;v_g(I18|QQ(zA;ak@Z&K!C))=Jiq-_kfnb?h9|IZS{25gRyS5x|M&?bfM#+%-_jSlVSWYBq`}y z#TVB1>wCT8@2vPD#?j>$e5H$%qxAR$p!f&S3uvlPO;moaWIvDjmBiW!+;7QnFNsl1 z3a%JRT^F5j($&zR9&i47ib8&J{aoQ%SQGuyLGf!o*jl|V z0>zowEB%=h6F`_G|7Op66l}AB%Ecg4G0viWXRy3YBJ-7+ADI8&k*10q7=L9+72yQ~ zheuJQDoAc-wQu9_jc1@qW@lk6d8A~BSs-ih z`1mYs^SV7vP;2(&eVoJH8i}|v3>I^H+7dQ%5)gA% zXO+-o52neuLi~PXeQNTsv)XdM+kO2agYZ%3biJDMU=wJmzW`@qQdC@Ckm|-j?_t^j z@kH9bHo7|LGiga=BXTg5e_uprD}k6D-NAHPvwF862D#W)!(xuyqI4cb-L)=E5bOfc zJZ>{Pp>N2aKR+gJo@l9bYtNoT3D^H*Wg*TLJvJJaO4B(t>w;snPdoAXcv}!uDs%LXZ8tKUk>~YetCtWS5jwIAJtW#4&q*%SR zmdWp?t;A6#gUL(l@f5-lB%lr&R6_1vHD{n#Pgj(u+8`+`zf&F-4`P9^cu=VUgJyWA zm0Fo$pm%7GtnG$eEBCmKgYnEwkEZz9*w{*jP@Q;Pw#B(-yd#|}jjZPYKen+bLQcT~ z@%!`#K2b2nz#>Na7dROm_poA-u`vEL->6n247T~A`3 zK>MK$xq<#6x@J{1W*ew>Cjm5JQx%)_4a73;Hv?vB7iwTpvmDG3{jKqnbr!qvkuThl zW=UJZR3bq0E&<9=-|8}|qqbjvsBO7_Hy<2T(qdl`1K1Z&-}uwJ_sK6ih0SBMY!ZpJ zQQ6=EQu=9tPd1-$MNnL@3SQ$!Yo=(uZ4u}oTkiu1q9y)<4GJ$r**#MpM(*C-A~=aZ z>w47y2Nqe2Z=5qARZ~_EuY_LxN zPtykx`g?WZI4wh}65G#!G7IOo)+?^24LTeg;2}YtakNP8W z^)6Yq*9&BE5gL-4&COQOT((efYezfskA=csssCxA0A6ci4&51~Yo|sQrO0{K-zE`x zb{X{lx+ut!7<2FZ*@2^jIgavMV$2chh_15l1h@vkPgf52R&lA!raJkD~jc zhA^&yl!Hy4jBX7mzk3w&z)NM8i{fpkE3A5P)=JldUFMd=Rp-GT=dHEcH56jbHdoQ?#HYtNY z5UHAB+%ub&;W>5Uv@7qFuN~Y4duAM_+ivz`*HfhMITN3|E4>*D5m>&qu@2U~?>98H)9e>Q!CX~?-u)KFw>c34D)cHf^?!U7( zN8M_sP}<_U%hwP>E}?W$z=8H%Ie5O52M1nus_g9t;6MYwXfTgN9FVXi!WgISs#_OM zyT3j^Z6H#+cp`^DaFp8(C@wu_)wh&PNRL@jjNV!Vbb$*p%)i{5XaL5%( z3sGS>2|t8Y0clrbS9_(oCv=l2i^?}gD`m|29!w}6vk8BF`mqnRr@SNa< zeo9y5ObpWMXWQVFol#>nsrYkT@edD(;+i)}B}PDkp10-NTlkS{W|uiwqGOl$wqS&+ zkv$F;ZV7;dsJA?M^27`1kAICMt~9q2Y$F~HNQ(i7tV}&)fpom-o{h_;#B{;$X?qCF z6%bog#RmQF5$Uk|kef=8S=sl=NiPUExU&giCwS)$dxWe=So*?~hD3UJd>TgGrn8Hc zm6ec@Z043fkm*kfAC~Y*8m5s0f0oXEOp`Q{YW=hJC^(y7KWVmjmC|b-MdD?WHGDaoi4^+Nb*ynR@tGUCe)Fh@4(JV2&#p9)CVxAC@Lx<9s4=7 zrwx?f@P6(8l6MaASI}G=H~YxME%Ti8_0#Wc-=_ZWPU1Djg6~V3@14*wcKxNx#CPu0 zxg$M^?xmPoif12&JZZQ4Ee{zf6owJZo&%@}&mPTu^Jbsl z{+R;#$Zg_}%RAen5Y4#KHjMj^-?8ras_J=DH`yKH1?}-wtKQhY&H4JjnWmFhrC5p?^s)!Dg$VaJPmZ>iC|7t5E^hIWKh^Bx9{jw3)wj;P4J+WJeU@K$* zl=P*ySGR|7_mEj6grQyF^J5oL?V5eE5v4zovbF2)fC|D&h-!K>yA97^4z=G~G4?cL_zo zo>Tul zMU7fyeucUXF7>(+5n=Rqf5}=)A z48bfiHL*!7bk>(nRe$nQ!hVC$eYK?c*QW9y#zY&mlCT;uqAQE_t49>+jLb44B5?FS z{0v8lwJvlBl`Y2><$HMnVBb5S)WpxSzbz-jV)#?DooKYYwng-GdBi!H$GAI{ebA&G zf>*YVHHOGs)AmH8HjmFkqIeLrL=LoY*_Iub84J{xtF?%&|4yXyV%pBdS>TYX!%h3D z`gX`)QRH!HZX{NljguGwKZk6h44uwF_61B-8@sVzEBUz6^Bb!t0=uuUPouqv>8PY- zEz*`sA>KTpPULGdC+PHvn!j|GBKWkJD09YvpZlAz3E1syT<&kdZ{4zan)Zts(8jG#nO-0`W z==PrN3!P>X@nT&2@zN^kN6#&`Zk7J)&7(gT?}l9!xOC2cuZaBSRnnItMAt-$jO2RU zrw|osH>Yn`mgOp8Y`&J#Jkks4{TXN#n)j6t&rki9(4NW=pG8yXZ`y;Q#r1LTW9kOz zfKOgbwSymfOpBj1CM1$HyY^uEq7dw`XibEOpR9X>qSntkIxb~5GDCl0WCKpERmM#; zsad9<@A~mUc&wA$~oLA$sa(TO<64hmie7!9D)96HV(|@HlqftUjMVE?$smIuj|oPAXNYAgvELwc zcMaPi&Ycj9!5gSKTWYx^yVh3qYtJP|h%NdpUNY^G%W775JCwgB*=|rozHd6Y*b{}o zmbf2&JJ)SL#!T7_%$t#L z?n?mpXF*q!YB{M)>OP}oDp%&c_(i{eb4DDgnOs!RHSW9l>4va-)!f{iICi)9@`NGu zVQzXPW6loiZTaOYGP(r2<-^X8=vgs2(Nta_C-wK^klYT-HCW_1z7jfsK`uJNQY_!) z94Akb#pBvi*juL+|w`5@u-e zaPZ({bX)*=k<3;9UfsruS0=E!63BJuN#nqn{ZQWT8p$FdZeTl9%%dVH0}bqJ0u<3U zWT-AdQ-irCyHpV{p2wuA%e(>#3aKwvG(rt< zePgNO8F%rULSu@xoRm3TxL4A(AyYj{Wc+S!Znp-5!8#$1Y^xJw{NboZ_TCt^*jUA+ zV_*z=Vi(gLi^Dzo-$(Hd8x@7f)!EmMWEXGGB{}EOcqj+l&1DwCLa!T z^p#2D4saCytU~0zcm%&2a(|{GO=7NeMAi=(laz7`o1CBj_|NajOeTuU7tTL_gFozi z{Er{^ufIOa_0v7&KhN45AZ~ei0w*^aQk-S=a4#SN1~OfKZ9PtykP*1c&9F-ArY$pg zIF5jdSrkjXH8GgqP>YAhm%|&vsZ2Es~!H>!DqW*PDEa=0yFA^#WHYnU~jgt%` z%=Ufg9p?YtT(pg)66UgX4i85^E_alBp8`8jH01-K$CN znKRV+HjEUoN9MYK+0b-)DPWsZmayXX**1VBiBcsRoRxa z-x=U#zgk#G6KhXE?!LOK8*fRz#U`zJ`36Z01j%31!HA|ApQth$4MgVc`Jo-u5A}OU z0ZgS>|8{Iw(D-AoEORUAuC>2j-Q3)M-%yjcL0MY0)Hw;t&y;0pJD`b%)kdSUPy)OO z2rU=VhPTS^i;G3@IP16o&{MRX9UV!VA$Mm;+g}z z&)oU2L!s_`3{O$K>hmFz)FD3mi7ZUPr*CtXp2V_ym1sTmka_{=u{D3^NSSKe1oFK| z^2^2I&k&uo{59-q6GPAI-XHGE8?*#5nQdeU@yO=ZvM%4B(`O;9U6v5?U{wexgMS) z1zXhbjNgL-Sq{jbepLq1<#Z~C+78BX@A{I~tS!jFe=RRUp5N21BI#yKpVRa$Yy+jE zF<2Hv_u={QmRJ=*thkxJO<#DQhBOB**dv;S_aITFr$C2k2ky??orTRn`tlKc6Cr2TZ`2`v8OS<2G5hXu z1ZqU(X6G5pJxVQPYvF{5J`V1nhAnCDMs4vxaGrmUJd2@(_ES6jKTCCX2D`#`AxDTtW_&3D|C=W_UK=LS^F^Nb0a8>UR?$WzO z{q#2N8t(~1i`L+lj3ZCfu1ebjdz*n5RNOObkB=Z{mTLB$4rQySI-QamDSnL==(@v(laSo? zLi#?E)S>!&tQ;=DAM+vZY?23C!JnJh8Lm+)Lw!M?3DE%hLd|HL#5mpLwk-nj(uDAh`AS${14?mCEbYCg>j(A;+A=Cz zHAyOcz2cQr`0~f;l#@J8eI@~YTdUxfMw^fJI#s)plG^~KSY}}Woh^Dp=9uQRA3s+9 z=Q-Ew)rWNzr(g&n3gOu{RcE$>8p@`YefG>xiC#dXP)AakZhLY;hMTsM-t0Ktllrb` z*)`I7yuGbG->^Cx`q1)-`>A}LLiP8UQ2&dSya%ws@iq$bRK&d!I@+Lltm_+bH|6L) z+IlZ%tvGlV#<`iHfEdym`FQ+e7iu{^Bn`&3Zca*#OA*Bk8NxlysIkxnxVm@aDYy2z zFL$}IIr&pr!!7J=l?rV7`h4i$9%bH~hYtr~6iP7gMGzA^f-Dguvr{ZjP_ngKw;A=G zwj*k^Sb23j?YczQ8TQD9rm`0^NH^itPApdO0txo3cabi742k>J1>WsgHs`e5ecb`z z=H~qER-WsP<@4U85~ad2rTy>o5-Vq%{+8O7Um3TI;rTq&`Aj*%aw{5pw$&Y!K=E+A zp6-VD1Hu3$X_c>HP>3<(8|oSs86!uY$3NDmsZYGH>OM(bkiE9+m|C!=spLgZv%H*Z zrO0@?$(pA0N`hLSxdDO!3l&wNRT28&3kT$?19iO(?ZM+=Up2NAeOVbH8w{3jDcq1L zfHebQ@>$fzYE8_JwE4vf%%hB?1h+!~p6O0mh!n5oNC{O^@wt)yf+4R9RhCnO>@KS; z)`+U`*}{zZ7`fGr9Q_%&;X4)5T!;v==h0BpB6_Q<Ty0U*iT*A-|Fn@@X6@L6krBQ4s2Mbqy6EP?+CgC-J zQ|GX#eC}{ByZcKwPY^V ziM8F7eMV-RTm`Bq^WV`3C#fHfdXMf>x_}bc%rHbL#qjAvR8<>P<2^cY!)Fd)V~xHo zfbkeNc=^w+VwiYG)uvYOO&1eK5?H_HV+fzjXl;mjB&@gPn%Y>Ht1Z!kF>E3u-(TcLjA1WL z^HD_obduU*@ZDqI17|II#ofw1>3B$I^{6S6)bb^(-ThLyR(1XMDb**z3Z{)DPtm1y#hZa4Q|0#Le$;*R@Zp`c>CM-QSr+O0sGE6oGvxe{ZJpTWH4q? zNl9*-Z(7K)U$D*_;!sRuY)RF(^~yU`tRe_pF#6FTvg9-ow4tpA+d4vbziNV#55-BW zFTDwJ%5n~4=}$WHCbR}&9|wqVBz@fAiDABdTZe53(GKOnUSr zIR=d{r~BxO$r84SSHZt=T{URK4U-l$C9})Q6~Pmxu$&4Um?0W=3SI-P*k{?E29DQ| zLF?z*AH%piX`^kFMm@B5_2NT|V`!&3c5=2uIW0rm5i~Q#;6J&SqQe$NDcOy)j&>sV zt=$ql1Rfy{+kXHsORIw!X?v(l9g~;DcY}-=f3*L>wxmtO>P07|8B~=B!;a+B8+}P%y9x;YBW|^B&|L44M%NGSk27mwTsunv0Ele)_Y)HtUh5 z2WIqM%6E=R&)|Z4KTUAYz2Z{~pRc!fdYi{qa_^0MRpLg&CWhJd`aV7XbS%je*iZw@ z!sqGFw^PN=M8|ne+ub7(<8bHEV?EgvEBVHGb4HM*NUlipxh_D`kq(RV%+KO9WT?Hd z1=C$`IH%jbAz19?@?ojSzX#@|*+K_rcjqRqRmN$y+wRa1A8@3WA!6-8E5}^*j+dDi z-7gK8)fnLviE-Ygp=vxR)4H?6ggAnM`6lW@kBEgo?ApQ4u^d^h71^I5d+#Zaf$#c3 z{UT{EQ8XSEX0>KgQ+re6L$3NCf0aK=*V=gg;yb!TL*Sv0hLlpIUtX-Ayw+S_7GbNB zKCdVC(Io0#hyYF_Cyk}me*UCl6<5fFTU1#_Pt;fs&YEINySOSs?3fY|Kvq(V15#^Y z{d-U-dFiw8z`Frlh~aihuS%s-AGjYt^V7xbvJ#>3othj$WStUx-%3yE@)jblRG?7Ly+NgB#e3Qw17 zE^ZP1G4jS(k~c-(>xJC6hMHk02_LJ7IDSvyZf}Kwg=N)1`#~pabGFQQov`-E zba6GYFgftK8|ez~^4BIQBpgr^WSQDq@0Nm}xN6;j_XzquZLR(EbpPyix-$--r)!~D zWxf=&^<$4-)OG5F!Ba7>bOn3)b-h3A6Feyw)qv$VFq0Hnuh>q~e=N}tSR{D!#Gvg~ zpnHEGN0&yrAnq%|GZ+=`oxd1nYjFvG+!W4Jso!(7RjoiHl9z2K zyTT1$dsOifvRidNa~)8*H?)^e`E^YUt73_AG<6l*yA9tMZ!bk@3+mX}DP>oD=Ngx& zDGpjwmTG(_>+^u)s+^rJY}IF}Ofy2yM_O!a=&?Cmz5Jm zJ5iZpY$i%BCtR%I6YlKmASrh?*=XjB9cuM;D8%4kf@}KzE(Y~tv8e_F| zue0P+T57D^u7c$USGH-zeDDeNDltKv_t%pQ2FujlO1qO!FH;ryq|^25JWYx*iUb=NEV(ab#Sy>9&<|CXj$ciH7%$>(} zbq|vl(>E;vnqVP&+>xx5p06WG@&Me;9MT z^&W$j_9K5K192G8Jwm0aPM6w{s?+{Eh9~x~fA5dG@_eGgFL}LtyqSAS3rVg_sg~Ai zcwq)y`>V`Clh7$1WLw%TG>Meb8?E7gfbo- zp<=KPgS+w>L1rmW^}nq|6=V|JtCXK+v(0B?Vc@uyRd(J-a?5+5WJS_6hXYPD-&`a1 z`%a;Gy~_l)G{>OpzK6#Sb#?VQbfkRgw1hxHpxFCxCleft1yoZGl2?ftp%%jzb2zDU zfb8|)42E9w*P%e(pYDk`GOGaHq9v0Jpt>p@d#Ox^YWiIZEXvEbtL)k-5tkc#k2Qky zYh7_=jnj|SA}q>ZRZ4#L@ETE5%gT&f<*kOyGG4(C@HrF3%G86As1Irv^BY{=OCJ_s zrPk~E5&3W4*j)X;poc&;XA7B7g#l&A#c*Xjvf6oG3G6)st`0cak)A98{2*g`I{WW; z*~Wd#;o*O%QRt`)E%5pfHz%FX;jIIrO;x@x zIqy_yJFd-%--2`S05b9h%oC8j3*W;M8Iy|%$M_DOK7E?{=<2$w)dma^J?np;n^$~! zsiRt1Y{ys5>b~{a%mLFnrWJq1?nrP~vdlTL@j}mo;#ilS`;MztEt=f(pF2yJ%(v|1 z5h*)I`CXSw1ZqKV#OriyUJ^_rFAygTgtZWx%p(|JBm}LYGLF$bb8Jq1*TSDyaKq{N z!Y5JdmF>Fv@MV*w`5^PbZ9butx4=g+y;qygXEPd{2$4MYnJBIqc6iN?J#co zThQCU!qy!LCF*zV?TIlK(fY*bDIc*qrbEnMx2W8+*g`wfeJ^ODYIZy?EGvJ~-ZI)+ zWVyrZ)3&F%`LT*ic{FOAiF+D9oE|ahnl4dh+33jf$!!v4H(m_o^1G3Y`FLuyL-`DX zX)jg7J6#%ly+dS7fk#Uele2DMwU%+x$cjRFeOH>BPSt6M{Ez}EwD!$t&C7I>mk5<;~$^-!3-cW zK66|~X+L@$B&{oFW$>F9|BkR84Hoqi@EmaMeotOUcqa9}?;6Sa$N>cG zG1{Mi(RGIo9nyjVpvvF``JFNGa&hUjRM)8f|nkX~_?D9UT2m`Gg%c3-j}30IJu!uROQbS;(E{>j^LS-brKkMVFFqUgbc}00w5z<(r3oODVN`_eU?u{hS9}lcEVOIG`8!E>Y#P5c;w2hK zj?ndBJ2T?R&Z0hXBI~sn@QCqVS6A2KjqbSYjWRnMGpqei+%@iB!LSOkU(^ody!R}`I+<2P&ILg=cfHd3X>b+&Fi~V&NA0_ks&3cL zjQ;=kuP=Xc*L{PveGN4#^1q=rE9%L+y?jP9dwN&1J*CZtRy$eQapBW{)KdiGvvM|)fj zG*Sy>4Nj0?CRNZ&eLk9p4$tCl>ZSYJe%?GqT4}F!E{uFuYxAx<4z*|-Kl)1>U3vph z++(EMHc_?wxx{zll2^MF-v9{!Ux84htsM4A5 zU*Fs+*Awz3Waj5CqLd#Q^&%)A_3f*UOeTSBclC9-1&3G~V&h+m(+7(*jd(ut_n$JU zZc~obpgF6j@a`jRCAs&=YnIw09<@#xaJWaR%*@Qfz5s6X!Z47eVldE`nLR1byTaH7 z@9s3^BKVYMXNGR}V<(onzy5Q-ZA(!&y28!lY|ZjA6QApzs^9nRGky>4{@&}Fdcg^% zyMFxVv_L7}V6`L`>}bkUb>8%cB^A}%zNcN3C!BILKmJGy;IrAqsdq&xnCAOKye4-JLBo>1Z7 zdto$*(U75PeQ_h0Ii&$yF>t0~Upz`?_1*_I^j4QYy!Ymp876s)E>V~X9mNP0Jieoy zr>VUF(sj1;dF)}#2!<=UeG>V#-WqGq*58lV=sHg59hxjEaJ(X~qQ{ZBSLO*`4qlYA zt7}^?4E`ED;LqxHtK1kL8zy-$HXz^XwD9E@PDfXW)FuzK)Up5oZ`!HAjDV{%+bqJU z^WqjkMtHrExd7K$EGFO+GGZ$IJIJIt0K~cC7s;E|>9u(-YEupWm~8*JVN8z!$>yPT z#sDHOm#m8|FRsj^^${a%Z0tjsru~F$cOjBI*OORkSkO@U67(Cx9HSxqzZ_Byxk={_|$j-1hU{_g;J*J|d#xD_Cs8ZxrNn>>763D;9@8|$McN?jIgkA0w9BTi(D*=S~39RhL>9P(XN_*|7J#< zle#N|=osPN?Z5yD7%)@(_*>afcYHLmL%JGO+~`)U7}o4AYnN$Uke$&enZAj znixbD7%Rb;x*`*8x|SQ;KsrfXCbn3_w_5skI+k)~%d*9Bk7;Pw1ITsN`!bdLjYKL_ zeqqrh<;vouCRWpS)adkD*I!6*-i&-i*yG8cf=$qlV``^MwZP!g1Rp}1Sn7}G`h}u) zJ`(%n_-}OEwjCtK3aYWF!H3fH8(F1q#DTm?`z;>7%V}!-YTrEm^3O~22#WmemF$|vU4UTNdCWFYAo5)Hg48?`&gX|>L* z%kx9WCO@_>>~^1i3%$w|`2lLr_TQIK1_8O;%?OB3+?iMFzv=No+% zR-a9Xz&J6F;5<@-2Z_X1M1LN{9G!3#iTz0czm&6tgSD(LpdE9%zxRhdOqFByWea3h#!x73#-a>lczltgBQvD+{;Wweab90G131W}6jj^P3b1XgPu4#y zI~%q_H0e9(gT*v8BTpfY#7Jx+Bi-bq+Gyc|@f_S8n7Fwc(j^nJk@c*66&?YwfU(W* zk*LZSa(5ZsERWVs-A8 zp`$u1rQW7|^uJ7b3@4aLla2o!blqPAs)XMtu{bC6>TW?l2 zv--BKxvfux;K^;Wpi@g+=1k&l_y)Fg$+Po!4I58pNVGEL%JpN(eM##@FCa$}n(M7{ z8@22(E7F1@k9jz5g+#|yC0+Shy5Aem^`S0_g0V;RxhJ4#K53Z+N9G2rG0OVWr6a_+ z(Bb>Ih;oJ;7C#Ha4E;DD>&Z#{@mzY|a(Xm1z(7SYfiYSb1JOFJCZ3xiapVQd!Gjmb zQciLh)~Ced5e@tT-?8eTV#1%QZ$l&Rt+jz^=`-MBEO!}IjPxs06l>;PR4xA*3>7^~O+W_XAHz3Fj8F8)`><@#Iho-+N z{#lpZdFu^}UF>FVIc>J#SWBHR;-)E6J=NuNd7jfFBCms~;wf{ijeR!)9r0CpC zbAtRa>6W^|or1<$lTdLz1YJBTPGBC7q(UL2a9oQ(kBJ|-cd~V-SuuU|wrD{J&`#VO z2`>bJPt^Egp1XXj{T+S9g&NjqXict|pGsh}5ey-^H88|jl%mhyU2EO^plt?&3O5ks z(@Q;gzP2VRzF_ntB2Mue4N7OizyiAh*jvhQ8T%KpZ+05qL-BPWcO+g#9Z!(xi}()? z5*j<;aK`v}!=R^H-c=K`F7DT0)DWLu)c&Gi*Does7Z(>b^p_?d&eE6mNx5@=tXFQD0;>>A_ zs;na<)t@gX!`B<`gZL^(qNbxU^JD>+nzJ5+J?mS_t<(9|3CZ9ZotlGXFJQ9z!))5KM-g1q!+d!aDJjMy$LgeXF^nMg?Zmoo4|4G4q8}s zq<_gQBhGm37 z7*i-L(&itGwx-BKK1qo}+r?%PP;47aJViF%KWW^qKuDu8B#E%->2-~UHW}*?{L~n&twICUvamGgz$H0FZGh`%GBcQ1#nl+mH8k1G2eadm!abi9?-c zJDr5l>{QE$KPtXH-RpVr6iDdW8^WLrQPh%gWJn;%MVOwKCP&Z+QGDlqz#~tzcTnq9 zfjVv8S7j8h3H$5<5>W%$`ZJQ9f@>Lt5Vq)YTPW0T4e47+sj9=_wp3=4XewyWEDXQ6 z6;=s@-*@@GP!0rLkAjAEKgdH*KnuN$@n{eBR-5@Di}VMjdT?jC142gcbT#lu)MqSs zcj$b91u}amMF>+{?3K3th$l0RZDqv1a~BgXb_`3Z7yZ-%Lj)}!R@>~(jQ@6x_X#$_%zJh|*hAOYvzPk8K7M=Y+&j144IevdDX6NRMX8u6k3e=12DbjD zQSTm(_J8RV5Cn@N?V?ymc8uRzJ|wOm!|LzZkfQ!^^?{6yv*>~d%bbL5Pk@nAh8rhX z7k*!TfuqB0vhkX%#w6o?)A~H8m+0_7$w9>vAQla~9&K7`1>C8PtB++U#Y7yaYB0r_ zs-5QHP8cE9ftM~_BBXWkcMaIuWN^81g5eU7T9R)4$S{w-zp?;SlG zCm!h?>+=zNXa875_+m(hu!^~`9c&^k z_s|SZja=OmEJ7zUZZmI9R&9cN!zPwkxO4-K^+iELnaN}F=`p?Q8Yo2G*a)M)4Mn1_ z_E+CBx3Caoz6hgsUl?GKDw$Cr`_?IyUJg}19A{lm-(p&4;+UbWh2>I70yBF&_f(w5 z6%n=v?=kb~@HDQLnaC7g(wboBMa@QNf99jS$Rxslk=H8kK*y&06%=5Zyl&)1Iyb?d)< zke;;POm*i{VT;F8RatoCS&?4)CRp}yHKSib(FZ})`= zTSH+%pEjAD8GLkf5=by1P$HY>DatcI{-`@vR9&8wiFKK(=MS1pkF! z;A2Q{34jB>oE&gj0tx$(C^4-0+e|u;EK$8$q7i2BY;Z2)9U3@E&<>axJcLgnEw`?> zA*74#59J&hjQi{90M{u@HOo2*6g-NwKAX}}6T?gbe7G|@T6BC;V=Jv5^2Z3}|G{;V z8vwci??l}C2h21(C5&uHAM)c>q7?5l;Zw#GZ1++GjWq=pvdZb9!Ghf}QndvXpXz(g z&X+L^?SVun+D#Waa>H`C!!>#Y!H^XvJEL-TX6B@ikkuSn!a(~T$W-gg*)t+_CZnT< zrRRz9(08pPN{2%=FX?g{|JJbK0rRFmwcAo-3p1S*6qPS9Hi@3iOurCOuU5t8_29|b{eniBL0caUg)Ehvm2?*3%CVcTTnO)XLD{y&Qn@EhoBWQ` z)c*>L)PQac9P{yC1Gf1anJtYRJ?P+!h$v4c_WF6E%0eZW*;DxK^Tyjqda9Trw}(mo zxC?*?m8s+K-`Ycm>opGOuVx!_C)23&$pbe z@Pl5C{VcaP2v`R{9k5kM6z|{8B!4W*QTbu#^R&o2i7Gbcyi=t|yp=UOhgLjd(fNGS zW>6~eG-U>S`Mb`uODm4i&IZ5SQRz2VythT<(j{lwzKVeQMu`=V-sNntd6nbV(6w%l z;>JHEvo~lQ$==qtU*&7j=c}r``!FY|DWvlTQ3&B+^2z7MQQOFtHPw9L^di~Cx^e2lrUSC%S>{YcW zL>1T;4H+sMDV~61T!tvIhj`Qy$Axj{Rc5 z)a6|qtNSP(JyzKg#GrgsK0{x!On^=OcMItzRy%Tt@ENB`xe-&o9+4* zbaNwI>#u*>^izbG+@AWzL%E~vLmXv|Nz0{iG~)WQPN~7hO8UnKblTK6LnHi!6{$v> z2-r5)vuk_J9efBapta+^1T|mpe$t?+!udkR&^#0@fnrf40gB9Xi6oPtrl1r`v`bX& zT{6z-$kwP8t#&PqdOX9XF-Vz@8lIZCb)~x6;!0K5lsw0u39?LJU5@u@r#iXj?&bmh4r|8?O-?-+_vD!JT9=%Y|0En&!-j#(ZO2qxK=3QG`j%wD>jAdQ`?aoU13gwV_)+}0`do{YK z80r=__BjjRHMu1^DgN67eFe+b^4n2TX6ml5!5<+-^r>_1Brns!levK8|e87qS`Q~bAMmJ>gp2(Aqv~qH0X{-me7-Ojwl(090HgDLl%|<#x zN;Jke#a64&xcXtLuIb-iw-b>l94@fhoa_~wa#LWIAX>Sa-qPk2YWiC7= z--+a-Z4%wRFR6P5P*{*;nr;X^CB`PUA!nhZ&X{9<-MhS}z5IV265Gir^QVdxu!V}kj+9vp zvjdWYNS@5N&HA-kKzemvN0~iE4(+~MT`O9Eof@fOaLUBVDMg%EC)FJ0v`D`6(p@OSDL4MSc-eC zZBrY!>TDJcYPLAaeNsZBmL}?t9B_ca4}Ta7U%;6}T00F8bti zoa!|_MoQ7WJG$FnGC934OMb0tn!IwfoCb5McgysqOjT#I>G~M>mVZ3M?fI{D5dx!$ z!zK}j#0+l-yTWz&A0Qk-7@)Kk>)&7=o(Bsm;4t_=b6^t&Xn}rDNp@-!vbVR7g5s?% z!$6*Ren`IwPY&0pp4q~E*=lyGmAW9g59AfA$5RiR%|%E5Qj6Q{-kQK2k76x{H9OWv zcu)Gqk~CCr#o9loy^R`@c5tE>$Q2~Dgy#Nrp26ItW$DuUcc5sNw;^J*%%@S)GBl=I z-52#i=~6@tj@N7tXd;mp;(m`)%ZYT}v8n3o@vgZDUM~zgx^}JdjEpiUVwg31|`qKw0oQ2AxKMCH|34gTiCx^^_ zq+px^gkA06a9X~uu_O7l)n=9ZlUqhjH=7^bBp>o~Cki;)^Mp3--GkEbdlLbls)l@e z`b#IbnH`l1507|P-7Ih3`_)hC-3GC$q6YagaW?AP(J^~&QwQY6@0oZ0hh{XR#oIj= z^>CZjp#iO%-!C0Bu*3G$-8Moq%*3|&Qt5O_q~H1~CLFDg%jx?12l+1I1EgsA<<-9Z zPsuG2VKuBhZdc^IJ5zP!esg~hu0l=f&2(|wGvWt1Sdu>na(%dB>-ezE^5)pz1uWrF zZ6fmBQnwO2Qv1L4eW%~-u3dbSB7OMJ`l(=s>oXko zi5j&Vgy#B4J^k;?bklt-HxAV6Mk32;DOr`P{U*6N}+J{(&=4>FT4 zalHD^KZ>?4loI@ts%Qy8@jxq<-80HeVt``Z*}M~RKlMx zKN?SahJhEble9`&* zpS9KbhZMIciQA}E29C7KGTQ_kFvf2KN@Sqs6`b~$!Tp-ENzX_|y&H`#$vWHkBu z%^NL{S0);yuZd!C$OOBU&tqf9U@9?Dv}#KU&J!a)KmT1d(HU-d3vWN%cmnUrx$D2) z|KP&aOXY8)wrUSw4Y@NR%2fzaJ5G5~m=zGgYb{v3@~35S4x&J;=7c91piM_;yQQEt z>cR9T&2m6$pfSOZ8>kd!W`5^W-V4%Tr9*}>J8EFDso=Fjn0^Z10H>sA%^lU}lp;s5 zxIkhkH+Q{LTm9f6&zg9~bJ~bL?Q(jfMd{_cIv$wt-dyr>VXHdJaA?h%HD*o^jaiPh zKdCyDa`{7s!kl?6k0Kwpf*Z|OGxtMn$RmpTcmBx^{<4DGP``ORJdh^9Hg4$^!9>JJ z?|3>o7i}fvi7Nqq;vz*k%=IOS&H9BHPX`h24nfd?NKB7Aqcdy=!CuDD>IxYFd zwy_d_*|uOi&a)kPo%^!B1uD}sCy4c<=VoQO8hqKhB{lVU045*-qewRWHoqiqBbpcd z5dsDe8h{b{&{`#_#KD6ybAKk7 zj=m!3hJNtfuJ5ay8U)NfMOP@>3wGv%)>dAhWS%YIQwwH9DegOMR%iWw3%JOLatIhS zvFAYd8Gs7RCch8he8?Zir&09s)+_0yV*;zEKL+U!NxwVM_P3{}FX4uvoh7OePUuSN z@7ZA*P4ZO6n+PQm^5M5Kmq$P%>M$DmPWp9Vf*PJeDoRJJu)z_5a0@g)P(n))zYKWE z^=1q$VF;Tk^D+cOMP~gMo|t`VC=0hmUQ{7-g7serbah${ej9+o&bG|XVdC(@&Jbt7 zdp8Y*0=0!#Yv$N59pZv+jKC|$KBxQL#k9|+<0RmmYXuz;oyqJtm%lqw~t+>wP@ckQOmJLj%m9;)VjmjVO z00-Ve|8vC~>2z%PrRXV+cYmM6X*+SD^}@qsh#c>F6K{|_gb3IXxd}#@53*&C!u{{z z;51KcoDZKY3br2TK$(A|Mbs031s|+FrKDG(jRJN)0X;)0fb|~Kbj05($Lx#C@Q3Gb zWcV8OlEe{lTYqI$j#6zp=0&~FJ*Umu&-+QDH{$vN9W7CNb&Tj~)dD!b0zpb7Mi>Le z`>{T;Fu?xxKd;Maxv_b2v_>syQBMv_z81e0$Xz%|S4dr+K>hct6^lSsX2Fs#FB8+ISR;Y9e4gba!N!4s;bF)qmos@2M`B#cZh z*w0Tr1Ubh&LHCACl3D@3^lU02z%iST6;bNAf@4ny9I;czeq)?Y=p=Pih|7==<+}G z$rIWRz*j0sMY#X)oeuRle>=WjpG>flodvLn$AWQ=3%)GTNPxR6MDjr zv79*X=-|6u>QBgEF3t#3RGu#wN-(!Y*&?HZF+x|X0g_4ouL*L)4Pzn`y_IZ)QgV0g zWuLXiT*^dggAx1vcn=pBm)6%W@f2A`XQ4JJdb*V|9z3lQ_gScdDw}j3SManNzJa4( z#dq|nhdaN+9)zPd*DSR8)dVh3aS+j@h@90RylbjbXhphey|6Z?W?eWLGpc(@@?979}kGo<%TEjW=s^92Rru!bUOss{D%S zx4#zVe5!Tt3D15UY8S&WS%X}7WY!@%#V?&4wd&8KZhs-z8c0syS8l>*@}9>9SI{C< znSUpJERvFv(E{IojQV68P2=Zh$4vIchqfJ`Y=+sYSC)s1uxcE3CMj6r01xFd@@S*C2e9!tsRB==)7u}J zxm4{{#F1Ld{fP@gu^E+`A5L5D=C_Q!qDK5oV8)R{@d}^@1qlpsJh1>Kof$#2GTcWr zIhF+>+o)aNC$S@Vwmw0vz*c)N(kxLOrR90ZErX~p^r)=_6LMPRf|L)+j=)r`LtiwJ z(ZnsgHm}W~%83V8w)!2Cy<&goP9z)+GK~ka3|szKt5peP!0F46gWP9a`JVolH9Ini za!?t$QD*@{=P!)rrGc~Rnit`}PXcw;x1qfB?)c}bzW&(s z>s8JCaaGZC6DcBB64GG_HK*sc`Fl@#gN^Q+5kM&HwQr=sTJO5dBEk@&H^f5%$k9v0 z1k#n8ucdeN85OxS364*KbYQHE>m_!j5N<(6u z7^nWSkd~zfqwt5Q8v+L4aCaG$HVNKoeiHV?tk8m$im{n6&I3qDj5wbR5y8=MHx9<& zm}ztj-~_#cRpv5QAQjLKFVT53g!aeKL&_!0`cTw{4$n(R zi$8=OvR&qSCAOFKZ#zM^h8AFW2U|ES{ZvU!NA4z&w9XMrAy{;Kr#(?*{zCjzKY3JwC}p00TL>qBs!a`;?aE$0eug z(xAxr5N%?nSp+-hDi1s?*v z0ATk)oKz{e&CR^Q2S5<>xck>%e9ubi{D-nqif9NULM?@csLu%bEEFLUeNDk6mQHX>j7OynQ4!$Oih=biY@S z5z#0O4MvmW4IoS+Czg6smKE3VzeO)cJ)7U1o9o^m3Hda2b=Px zmf(~G$|h8lG1wgcTA6(D{{R9q46OYA_hA5%puEX{{`luSo^IdbF8|A-O}?S%@$Wy1 zi*^^IJkpI(-J<)S=Z<{!=S$P#|JUK~{qKIb`NB=j*}=s|uIoU21xi`y?oN^oCpY3Bo>$2I835E-)YaVTOziPWA zhdHmp{-HbBTR`0AN>q6MdSEDK`V^xH2@e0jW6~LX5z|M`)&F%^`Kh{Zdz3XK-OpI} z)%2r`By7Dp>>6aZTfJ_b`^%RvcW|gE1aj*gL_?ZOI)cW^iY9o=8m@fE-_MQ1;@odf zJ`~kH z(Nw$qXmYzgMK7mT{BpAvzd=#0XNmGD+RlX99aP?PFk83hqNk+=!J40N-}7f$4G2a& zheDANuzq7@)i&#~P9ql#nsmx+f4>c3e#P@3&`w=Zv}^wM>phBPkJ>#~S(gNJ6Fiml zOC(Mj6MgN!!ocuNKlBja zz*k@nwLugFIHerK0Vv?CVZ6$$G^-&gc`YvCX7Hn`3_Y`0bFk)1WF)iXFnKXtni(Z`d- zRIGILCM=*o!&hu@W?;P8%5pD!V@b#plDGY5rTeZ0a58mH%oi4X`|NJsW|Sog{ya$m zgfRBts<%BzETs|{a(;@r!_(vYTgMOhhB-G}|RsilW3Y&T=UoBc04!_(K5aC7`l z2NI13%VV1K^^`@FH6fEPBOB4hks{&})5AO#!K90(Vv!im@IL9<_nZMz)z z37yGPQlpKd<}s{#7)f^V%&x?>a`YJhTqXlWJa%ia_72|e8yYg`{DK`?6y&vSRyrxK zv;||Pzaz8PF29d^II+){P^qq^Uo+@Lr#@mT0E?HVpWP%`4FGP5+r9}`6y%;q`|3aV zY!Kz6#H2O1N6eq~Fa{PyQaJ-=)aG*&T3m3*OxwXGS0VHFUQW%+X>BAYK!Wf3-H_Z- z__IkImP+u@Prcq2JGot-7G{5p)kfkI0b@JoY2cuNpK;8`SjqxSU0=qTiVZr?l1%Uzfc}V$aZ%Nm$rH zI?q6CX@Oq_e2G7=0#km6g%Mt~iFxh$x24sHdvr;IY4pCcNoDb%vU~D-?lEkfWh4H9 zFeQK=)YTKm<5$#YujuL4{V+)|rM}R$s?!C{C-r<uk^TpmiUe%q>7*fd2seefS!#q6*0P?gt{_C)c>vTT$cw>nEZ_m-6_C9x6jc(RKx*I>cW0@u6sM-&m3is8$%$ikc2{pQ<3nY)Cl-rFx=CyG@9&wiBw`s+Y()}TUt5KUE#e4AjO;n99B1ySX2dbn3w{Ua z-GfKp`qoDxTqK%8VyJ_~keOT8OYRlL4z1Q2j$C%pZgFcjzgS8AXW{t!J<)>$fSV)~ zc#};E=iYmuQ`(ghw9rt)`b_RPPZ%)DG9ZZ+nZ%@#F)NJbGEaWI&mUjve(y)SX!xoI z45_zv4q%vTC$6Y)u4W>>%*^d^ufBkC%F-51Sn9`9U5t{>#mTK(1T&+;z+FI{Z9E=g z{UEWNR>4I`J}Pe0&)zmlaM2*A-(t~+A7|fb(?<81TQk%xHeWBtH&5T6ez>dp%#nBD z%Lyu`s^MCGdtV4v$)37CSaH;^w~4O~UJ& zQq_7lrl6=g(UD{jCE_k;*T4|BneMV6Wa=Ga(E}}6NXX*R%NoLNMbcrNpF+ zkJ}hSdbMV;q=5lXwGKjrPfHMl9Z;)~JqECk+Fh(R>jr(!1^pum1Y>g`ywXpRO&>Pi zRBM_{Giv#yky&s*(0Jc&b2*_V^yh)I4I7T|5_-j@X&$4_{1zGi#>vYUlav%?_Z!5@ zAjV5P1InB=Qn?Z9K-4oW=NOpYb(EMMGIb_klO)+!yDIsc+*63U2KUIw(%^g6?PDc1 zVD$UtpTT)4qD@cJReBvBe@o_aerl|s{H%)KS5;A)7e3pP*as`XE`%KT>3KGHelFq_ zk*i7`6Y6G@OQasY=3T2%o7f~+&?%7lAroOXQJ-&b*Qjah?I_e(!P<)(dVBu(%@Lc(ookM~h2 zA|G#3C+Ht;l9UUY%3x`Zp6shG7I0*T3d9E>VDZL3^<`Vp)?Ws=B+*mn+|e{NRL3zu zL(1H=Em@m922--YMzNKIqGO*$*yBSYn=Le=cJbhGiOxCKp#J7<)Jy;89Knz2rY;;A zO>x6Hp|bH&;hj1%?rxR1@>0bEf9|n2M5xwn{j`?aUYkOSi{qX8asjye!&0 zlf=>TIH4}CSk5cq&_yX^opI{Ru6xi%Utql=6k0LYHiHaNE^5!rI5^sjXFfdsyZ(cu zxz>?8H*P$OZYibCQEkU}2-oE|hK#s2pPfNFg?1dx--Pc92`$$Gz&`65<%g&iQkC`P5eGDnf zRe4dY+G%Q}HP9i|0{rL9Ie(jdplMM%@;WI}9;P3mmU*j;vH`>%@Y?oguoa48k5!X}iR(n3=9zGDbtu zbNeGw=e4pn#90~F(4*mRaR=>JI{ds;Igl87wQG*_(Ju!juL(}exTI6#6PD=T`NKX^ zM$_zwV??^q;hi)ur7lRlUM}G9Yj$mOsWIBRZTmoO^825cUcvKpC+Srmaq(F|{=c7` z4 z$(up6fYsoRu^h0_m~%{~TMbf9rH_Hd?@vrYi8*|Sz0PXE5MR(mW3umn+H3g#k@gi} zRc>3mOBCIpfP#pCAO=cENGeDujdV8#-5?zTDxiWW(jeU((k+M}p>!=uq!Cb3`i_Zu z&ffoj?tT7y?|PoSRbs8h$Nc6T;~j7CJ(+b$BNbJKZj3!QxESX(DT5Mg=f1(?=W(qH z$M%J7NDXXJn(taRfpZwJE(q+Cv8(@t498$AYAuL$D2BgXEtFZ&|jO6TK`5J;D_0QXk?!j##=EA=H=#On$C9-=H+-xZKeLLY?H?W z*`~KsE`8o&*#)*nI|mnILaNKp77=LjQUx(l5fbm`cEe@9_dIAhcvx4@)kvv02X>HB z4OskA;-A6h0$4o84D=`&^+&pmN&N>ft&Ff3DWtT{IO@!J$57ecbtWlPs#|29LL2&k zI;Ce_a#4pVsb+?E!Oo6dnJNoEQez?y&bH)P#Dy@|&ecTBCrvd4Y|;S%yXm%bif)RG zDwA*(>)nnVJ{(~_vzyrzJ>U)@c9fSM&t}QCSY}BZ8GoY12KGA zM>$)o|J0inx~i(GOQRY0vuIp?YgSrajIKL=k5v~v7zRQN{nt6guvQH=k)@^fJ4VJs zks8QG`ugQUjqFqH8lYOPm@T4s+Oxca;A4~tyug+DJ;na|+2Z)rBshQ!&H~ezYw=|? zCA3@8iY}nKvOyrx5UM~&MHv_uD z+LvV`5U&!J9=WEJqcdciWC6n`+2jJn5!0-e^ajbgV2bSSWx)|sf>Dz%>FBhs$yN5^ zDz%v5^hkr=`kud=#&q6K)Kfy_6T&t=ne%xMT;gK^uhqM2DN`;A#D|4R__8H{M3XY) zx(9#{*!%lA4tbf{)dRgIY@wbrEQM5*Q#kb{M5HBjpQq$7x>M^#G2NLxS?HXgg6+bL z!>Bq522Rg<=(WoV)Wwcc1LV@u(uu6%_TBOjS|zmAvo)I)j7sopDpLY0 zZ9=crk?wK>CWWnY-Ctcn9ZwPg!d=_fb3pKI_gfuyHyCd26--u1a0A7gSui&XU|s|A zZw;iKX46uTiBJ%j%U}O!e&grpnyE4pc#3XVaDXE7f?0fOBE z7kTrSA=$mY<~z!2fFU8bmNW}7n`(;Zb5SoZU5w~Wd`MudHr39q7i#b9%!Qfz!`q_=-)L{YXNWb{Vq#jyc` zjU~;khPA5lS&3GYiPn{_((0!Eb|1H(r?#1^-R}JHV0?1Vf$D_nSLb30M2cACA6&vVN!zclo4pjfXy}hAJ>#r=sYOlCA>3!df=-z+n5&bdTK7t5_7hr- z$s$Sp@pt8vVKWmMMU6dCTf|I(jE1o*bHhDSivqT{CRw!29o&a_i-Kx#AJOt`CcARo z%^uQ^;7q4==*l!pHe~v#?H|PxY82i6QEbDe!TmSS7#GCRN9TD>VIE|lRj0zDSNt$W|33#9INRNv^GlV^9E?xPQ1UzRN2nj2W)$D@BUBK309 zIb=xxr{==2d-GXir#Y>Dw^7IB9VXT4-BS78jrBY6@X1djONqN74LjwtjU!9TWXw%v ztHos{l9C&gNz!C1pHz&sPW(uxi2OFPbffn+kiQcL=ao#HxYcQOygsFDS#Egq$>(qP zxmPDUgt$m%hnz2E*86F<(e`fT&8uLw81kX*3)co;J=Bd~Nj_DxaYVLmL=cR41eZRY z@e2Viyu|y7J~OIQcC_>76zd{XxR;zQDbrFapEHAE|Bx1E8{$|JWrI|^f7A5B&e9^Z z{F9*^a8I*We|OrNFQ$SRZS!7e({v9Yxc}Ef(0(!#4dwy@2&r$-fs~A^a{%`SeC|d? zZ7SaYtsuZuQVtjx6?4#T1TKRQenfh_#*%LHkZGyCVM4^QH_vAO`r{+im_0R{Y|c9o z9_n41SZmaprhl>I=FyL77qDrQ$OGGXf7?^P5eSfghOzwj1H+?jmX&dIMKB9QO=PyJ77aVL-8GtIDY}gLPiU@Q8jW+?( zv3g|YLbeVsYXD9j3woiEK;I%ShNw9J8bj&L5yF)JoaAB3MB<`xiUqssgdl)N(_Q8` z)*`~fjlhD_13Dh!fhbzvazB(>2MNAZJZb~j_h6WSVi8_L8~QOupc-*JFG+>bs4y+-$6&&8h8;Y&9<)FX4nl)$AZ*5J8eaa*d)2`f7}$XN0Xz7Y zR1=Wx{*}5U!CdUTY61pOFT)?gS;ZDx2;U&sW8XQK?5g4zfAS^%iR!*Re(ICH_yJI! zN?GqL<@kupb=Nosi*R0u^Y|2;muj=>rZPSTUbr;8M#?C4c@5d(&Mq-ER`M`6g3ry1B7=Pw7&v=eDrL$#lVE#2OA=jIM-!u!)IGfG$F1ITy;*sXIh*P&@|f~azxRzo z8{$(U$%s#q#{2{Fn41BBMyyoAEC)d85s??qS0znB1Z1g3wed*r0L~Xr0a?GIU8bbt z`lJ4sJL+s;V5x|fD*OfTDqyG2LN5|x813{y%C!DZ7*q=Z|81y!P^gW>Fvl^N%b@mS z22OIYCl7_-%|o_p9) z2%>^4ZzX^~%2+E7hK4djAub3FE(JpfU&xRj2O<1@TDt1cmF*kF;b%r?4%AbeG`?^6cnHqQM|kI>~b_KHztq`6v4qRzU$6c2h+(b zp9U;EbUzfZBoz4u0mD-a?lP0)v##H7mN$d%W;lK_vYr6rq~PCV=LBQ%0!b2gRc^Eb zJOaP7Bfd9HWV%*drqUHv9_hL0@4QTx-DEG9OJz5C5-TyWzQ`h>*Z3&*TuIy|N!$gY zS%#@#3dIm=WwC|+RRWwFUb9XZAIREVpDP86Cw<9sc<#9?nuY!7&BMdv@~8|g{P{{6 z2$>&?b*nDBtet&L86Ypg%)MLgWiVK58tJ%vKj9wj=_6{Ylo9le?EzWN)vwNl7Pq&@sb0r%465fl z_w1;}fBB>_v73~_2x4)<>Cc~kbdWIg%l2d5 zH;+Aciy4}h-8ruU`6_g59#{ym##bVEg3v-#*o|ncBj+hVgPFoc-*)J>)5Ol(V5|w zah)K@<}t8Ce?R(h_eat-bsIk&BM&q#AT#sG;k^4|ubb6BL;1|4h(d$4e9)*HJsgS4 z5@CzR+!%bS8e?Ne;vprUra+6LSW!qI(zStm>iu|qaCu^Y@eW%jY!d>D@GI{>^KVUP z(24yw=6DE(ID$$*C+w^Z#3G^)gX12J_ z+}q#4{`brlngB-C`{V28+%3qn&|@oMpBq2=Wtf_?-*1n*1Xv_!N$4@`2gwZ#leBj% ze`U1c=TO0){PCLp@secH*q{#jkp{_sbIHlb$@S3(e#Hw4cHC{ZI6H*z%Rn`EKkC;O z{*STU`=h=(_|;$nISp#QXhWD5APzB{;lDJRqlf;%E0ORP8DioXKr#)R|96>&>Dm1^ za_b*lKP}0-@(%bfP@t=UE2L(r=ic!Yn%jmxKOb6U$GHr|u7wq(52+?gaB7HyNfo$n zGyxk>5Bz&OVB&xXzsTe)KLOail3??c1P2o6W8iAM@%THr*^yNYX6?ij<|y4eL#0&; z7?%BUc9|2;Ptu~t|NC!acK>Ay)1Sa>JjVup61`9Y@C8677&Gt$zBxHL z^VjEX9>Tl=Ss!ZtrEDC!0+ZPwAjC#Cz{c?!MzDHq8E6B1^LYP98vO!o%m_L>eu*JR zXF4cO+4-~?0qB6V0FWvtHO+n34KCR*ses>e@OYVZc}}lnLR$A*L)o`( zRHj}i2v{6$t3==o8$7HnI?L~>24#v!>3k9PA|0JLgdD)GrW64A42>{}BQdWA&o6_2 z6C_xhXJWsoHbZUMdV`Np;yO&VP{0!k(EJEl&B!kWu<@3>_Vi3VSY^K$680JSut{|E zoFbUJv;qc(ky8ze*N6M=Q_%j~?bxMTs+_SQ`mOYu_c&=cYsZ(S* zLcLgu8U{O{$K3jCmKDqPucPCiR4=3{x@UFBgIBVjU&THG;1~n zL>i$mCIuROT?jb8yeM`xLEI@WZRo2widPZ20PIK`Q{@xJVKjAV&Xg9;YsMEv#+{-_HVpFoeLg9S_*kX0<4v?E*1F^t zz~5B5O<5>&AQ{dYurGazgbfDO&RReS(9_I8?2r4wtjEis?1QTA6=an5x#<)aqh!b- zkjeWDfE~HV-XG%HCqADRSeCzuYtfDUFv~kb(VeRgr2CcvmGXjyh@7EvfwjK-X#-U% zbzRwg$jGfz`GkS;D#2(pG%oQP)kHz4?K1M*Z7;czLkz=FKQtTvWn z7;0foY~GYb9e&Wf`T%4U~{dQ2juOM;_CV8_o>U?NFFSpxf68GNBccvk+$~ z?Fq!%e0QAFXkhp?#P5ZB2Fbye%b4bRve>t}nzE7UZwb_-m=dpl>>?h1_`U|+ZK?J% z!*Khs!vt50q{;7gh8=k4wMv<$DjbYRyLzVv1ACael+gn)3A{8KrlNtn0@UfEtv4!S zC@3?)}(eU+({bTtPzV3g%Uj_^eafFsjG!EBYbeR)8Ipbl0Vjosf#nSxxQXqrkvF%KFx zJX@3AdDs)4uMZ-56l`tAv;Mna%M*fx*;3W3J(@^YmzWU+p}jsox3;iupJpFvm#4CT z3Hy^LPx`0|G#!(M-Zj7&$sqzi)w&!ENbAe(v@)$yE@W~UuYLwAJ=*?Db0yP1sv%0+ zdP?f^lCh zR}``mffBvq_%s|n6Yx46fJG~oa@;Lm-7G7(?YvJcqA~yHH7;_}4SGY=`X)D3WIbi8 z6+d$}j$%J1Dx4p-LgbSu1ve!2i@9;{GFpG(4}s4&4s#qkNy@?j1BFaxY@zQOn0@+q zC*mX^#@T~=;gH4&PAIr1)nLKWto~uys)^tH`(Bf(*)9tu zL{pdBEL>Ifo8S^i3sip{W%;avv~$8)dNW3<-!%T=?)KPz-XbSsqf14jJA3AZ;b4F~j zr9%m%a@>{AVvSG_7Ga`zh3?mIY8bHNxm=opcH4dmZEwjYDJrM5ORVCuI*g!#QPj?R zd$0a^M}^Y^c0Rqv3|@1B@|maT&6S zk;`K;%!S-KNh`ohNi$*~>76=(8&oQ)Xf%EF)RkADuERzPfI%)^((r@Ne5gPg81g3F z?P8w#s&cXrxqp4T53|fmawv)ls(6ct{ba3QXJEsY^+$%kt;@?zsW1T}^u3hHMnOVy z{{ymoK6^GsulK*`KG>goGlQkeeVjmwY=j=rg6=4pcLQju^tx3QD|5Y{mAkwg8y8^sC@@f(P3D`+L0n`$CWYpfY_~ z76QEmaO+SyuAQ&<>>IXN7BQ3`B~Wa4&BjBODVMLTh~Ni$rPNnYwlZr&K!?MyQ%V{u z4>;d zaqp|nm-BUM)nj8;b>yI%ZdG%SJEEum%eDRc%7fLy^$(Fh8o4)l?8-5_b%wV}@O!WC zyE@ceZ!Bchasv2RC3juewoE8t;TLEo7`pPeOAye3<>T9jMr$e^%ggq=@!y|b#|NMg zjX4L2_8aKmABJ>Kf%*t;a2kJOO;>5fr0^gelj2!9r-7=Ta4_(5Njn1q;j~_#rNQ>PG#P*D1bl9E;qNB|hPZv-PT(3+Y6x<@`&{$q+kgN0pHv6`pzWit z^Dz6=`%7a#su6XzDzX@Tkowub(TLwAW`=U*qyH-*LTZlTzCHFYO(1&Kn!sOLjL|PJ zA^Gq226(^PPV|+(AOFK2;2(k>^sRziQ**PZbH!L?m2oosI0b*JxquvE_aCyq+Qsiy zR9FrNI}-G<(0^O~{}zD!`qKUX|9%@DKAO33ua$Fx_xYJn%n8#nY8`L>TPay600Gp5T$R*aPu$7k$m+d5YflMJ`RbgfWxE!aF{V9h9gEEfN2Y} z$p)GbC>#PqlNPCJAmC5-9aNVFC%FnJJujWS2IrzG_YsLO&>5Ivi=xkhosDMk1g7{a7{WoL)(C#T`tTNzoC7tVF!qrIfG`o{W>5|=F5i9$ ztoO_1QAklBFbboq5Y)#(M~4*I3WX-h%Z!hK>h>%qMmM>V8Dm8zuj=2NMy}l|9j{j} z{IM|V!9?Gu(#cEm?U$2%b%)2oyQScEya*Aal8_v90ofAApi96Ajd~FD+Rac?e+oE= zp~)J-9rSyhYtG(U;{cmRiJ_Y9ZJ)S^m_qK5;W)Q`OeVQc@_mkX9(HErH#{c=%nlTTp-`Cc$$2{MvZngM|R} z8~!F6uVZZo(K?O9`CASLkf!N)OeNSuzI~H14}}KUWSZ2*BMadn zvNk{_#U>Kt5trz8sHsA##a}2szrdMuLC>9 zbp_1eb65rC4vQAVo4|naQ$4^zN7r}*v>7Ae4Y1gP;V!S&Tt~1NSWQDPik~a+Dklx% z9$tf(s)JCI2g~wrydR`Hgc*z{D;rqV(y%;UUn!r6{Zb2?u3ix&HG{6zp~V(DFg-wP ziY$_FaB!s7MF8etdkLGuQQ&ORSQ2Au5`G|Fu$2QM$kLxqtdyQ55>}km;Q_i%a^%+% zZIL-Q{j9%!B3AxZXBj~je|onhvZ{n>P9b>2_YVyNfL7n!+wwDqbM4}|5fP(c zp7~YK9|cMj!JN(xm97ykeV}OJ$`M%%Fx6ZU%lm@VM@Q%HZIw<%2ZGUlggNxS;s{{} z4t=2;jKB!zFO;r=?TE$NANkSTk*ZC#jCESWe`=31d&@U>jo+mMXc86&z%^nQoh(jR zoC0pC(JikC#YP~%0xTf{wvKn4ISR$xGc&iCL8IWbjkGG~)!=G#zSB1kya17>6pLZT zpEHo^=lW&v!L=Cdd(&VI7hHM){MJ9hFw%HY+hHj9UQ#pE=aJFcb}nVn(cI32CYOZj zdOx-|Rc`J?JUo(8dUU5aX&x3?_BE8oaVf|!YJGE2VI-sL&3f7@m|leCouX+@?M+#% zu5x$gznB2hMOIlP~MER!v_uh50&D*Xi|b z*=%jVnUbGs!gEVg+VTRXR*aepdTO%yt7@U$%%iJ|S}=B!4xS3yjWSVNe|O|;5K8VI zv)FWysoNnE*v_5tRQoIj)~xm>SFb01Epfm7ZD+pSX>;)sJs2_OtSpX`qzT^L&EZmXPl1xwIWhJWO^7(^hY%?|nom{mPdzQCH|J1v6`DQhCtw4>it= z9_xLX89XP}E8*Yq;@UQx#hu7+70h1lCxZ{1o(022l;iFMTu^D+13Op8fGOL`dkFJg z%@tI^n8EOeNA7qPbDU25!YF23(indr3o#Yf`tXnF0K;=dEA8>&W3XTt08SIPi9{~m zBfLf|m&*tU{xbP|Sz}b(rEwelzNLO%IY;nN-L{%%ddFfYns8t!|1c2{DBg@$aXwi? zDZFi|Me=oO&aMSR0^8po<8_({GF>o^l{u(4pqScW%y-y2MwKGAdo+w%mT!$By-=*& zvj_90Do4eNt#Bo5x@v{@Ji+^!^_vt5U3X!K4E{#%9j6|NxIfuOi7!osl0w#HFZf*6 z!fqaPab6gLWHX6{EtH_53%UO{;6}r~7iHIGnnQJ8j0KSu? zFR!k0F7Cl_Qy(^-4p6*xCann(@dX)`!;Fn}>OIUpTki82t_*gr3)iJMALEeP1fttz z4b_*vDzn||a>iH=y{tVI1iq1Y$oP9Nuk@*D*D>yJC(I*ZB>BqexiwCy*((DVvJMvn zA1(-dJ=`05#x2K|QnEO0F>TkSS1^H)VzMjVn8K`xtf#Bm&RK?$&wb0cJKBdgmzqm{c+?dH?fm2gsH`5sbu6x9Yg%b0Ezrbh0%wB0;fXMCLV z+)#8?iuH&k!I&wlmJgiA5z9(KS(_9bO6gmj$J0nJ`8Nmj8n=5aP+C^pJ?b^PwxZR$ z11)mz8Cf@?u~TmqowYb>GT)gJbQBr2*M4!s@yICE_i7KCu781K8b6{tj~XOZ3+Fjl zAHn1ZdVFJoO6*p# zNaMunm!fmqBbuax3h~YL+Oow7*k(zNK|Nd#jvJvW6#1RULsYCQ0EYuIXs%oXg%vn! zg_Q~b`tVAxIHB^6iLx}o-I+3qhn+RyhtG4mrki(dtfluE-;oefE%Rrlgxuvv(HaMk z?c^oB@G}G>#vYQ@b8Y6+4-2S3Tgd8}0Vp^v!~1cw0Ud%(bz6;tnLS7#=#IRPDjX>K z3dA1kso04U2q$}~jK1M>DMG6FOUIHMl!Xd3yp(5LGDt6yblk~N>Fcr@MiC~pCGcA8 z4aymXCbS}aRXj536p0IB?k$;>L(^=|{4-p`&U0Bxm4~yIu7iyeyU~^%?4)1_HnSdP)m#qy#H=LqCA zS;9^ccXUX!oK`Am@i z*IlOeo1xb4k)={jRXv5YZ9KG>u9oAnNopVdiW!1`Sn?UaIR$zMJJ$-JpY=E%h2f{%Hf=ejI!K@~HZ z10Q&ZvoLyzf^mm&qzvM76kURy~>4$V{HYXW{Js4 zeO)A!VKi&naiG!HrSV8 zB&X_Z#yO2B3wh$fLB4#ihs%h2jKZ;VHesN(j9M`(BDEmWl7NVT&k~{YRJsHn4>#QBhwv(tJNNc}bl)&( zdukAbPcMHiD6VBWGhQm(N6(KfSwZ{*oiLle3Y}Mro@8>+v1bYrB1y~c6}{`#YnCGq z*7=u9r{1{ees1{6zfijV=50eESm@4w-pwpu$Pd?kj6*1L^(=!g!&m$sMZpZ-g_DhG z9`|skW=haOL5%Mq6{Kl6FBY6dGn1`*qiG=Bu7}^|=JRx3+X{xa)1Xo?3cXpExJ6&m zK~;x8h`Kpxef#6PZT_3)ChW{FuYg|p6TDvmuBQ}wJA z6{E#ju?JJ9UYHdE+1!u$EJjc9-4VapCy5_IAI?3y66L6r2eQ!xorC@Dil_Y_-CFHx zfvRu*qsEA;h8?G4kdBrt@d3=P;XHQ^#qx-^pM`&;``jKkl$;*K@r16Iubft*av5=3 zk8;a06|sW|ZMjqJXhqMgt*h>ZagJOJk>-$(h)#}V2|D-b#}OL+h=uEna#C)8<>liu z95rgo{n3={1YH@nm-#7+JU-z!a0MY4^d;O}RxG;RdO0GbLI1K{INq}7BSF&4q$@rw z)!R0C>!cX^Cf2mLdBr#nVS<-b!k%{V-~8~G>^R0tHeYaj?v&wWvK?p=WUy#lqQM`% z9eVE!bH(@CdwZ%@aOVzvGu6*kCeAX#mr=%HMuc+Tim9eY$#HiojqvX%os#QqNEX)K zz0(Q)`5y+Wnw%*_Jw5S>Rk);s=V>S=wS;0Cupd6!d~7&*?aZa2)!ROgo)>CssmKW6 z_A-d8*XC0(W@H^5^AlNZx5t{Re%l*h&erk@CdXa!rlMp%ET%5B(-2HKt4%I{pKm{Twk{U1Bu2%4BbgmY0u8iznnJ(Wv`Y{Isw$n3Io<$Xg~HkiJeVu zE@oT>wxH~mt#`-j+%uY_k17{(Ve>8u2!ARpORXV8qwSo2gigXV#77`F!oSdcSFcHt zLZ!*`8K>_WR`gIX0l6F21r(;vNl#1@{+Zb_h;W!LdEO0px!6rZNEF}A5D zZEIs!UIZp~Vk~`Ey1z_o$6BiO2xLox{u#|eO_CAp=JufuyeD)xGPu+n2{fIj^*2QR4r1L zKPJ_8HB(UY#ruqzhB0q-qOqpRM`@3Xe49l2zR%-pIAH7DwdO$~&h=Bh?j2)kBwJ#- zPLsX%_@i%y(i{xd4BPr56Jm2mFkZ2v?0yWi8-O$zP#!9q;f4r4telFZ#rpWm>Z|k9 z^o&XGRW39{<#xq7gR^~ww@(K|iugUt{48UTCA~*1OoZ8py<5q%c7$&Jy{*;o(eZBpG*I>WqlwFro=~? zB5%6MkZD~Tqsm&bby(pXSt+-Mm1 zkX>Q8bdHMbDHYclw@W=N)C2)lU=|nQL+#v%#C7)JFmaiqT4WZyw(02bF8hbKx$MQ* ztD5>BsK|!mL-JJa;QHpkNTj?7b8M`v*GW6}fe%K>e26)BGPll{f%rM5S zs5K=(9@e(Ea5dn8C8XrjtPZ8?F-Pb)+KxSGZs z5@O+VW4H=lrPJ}g-(F$ympbc`kDRd*%)?Y6CqX~=a6%J>z4xfNe*8KJS1IAXiO#NH zKpA)Tm$bJhKec*1H(#A$F7*u{RJ2u67A0n2nZDMbM9*$`T%TFl z^_Kc%Q6A1^kT*sL(kxH=WzdQon4ys9UA@a7uC0E3XF|mK3iqIu(DaY!rrg;L!J9wQ zHGzmAp*cmrm4Y)K<2Z)5mEgPpPJ+i;XCt0VnK8P1RPUX`#WXr3u_-^Ncr&u^L4AD4P_t^F|0;hmt1#h=>iqPOTH$AB{8 z>pHy0(>_nhgvDM9f&Ak4t>SMMHtWay*Fo_VZvFK`ICQa9YF5^LKNdg~VbnTqqo3aL zbA%|}RB1g5FhP6XD>@Z@u8^%h2^c`J-LR%*r9NO85$*b)gw>ZrNsEavvB!g&JmlT2 zQf1EX3by5@*llkUWZ!bbIf!9PbB^43k@nqP=&ix6QvBdw$Mcy!uuwA>eGVsPhrM7G zKmwEL8QJ-DJO~6`V+UJS%B{l~IRkWDWALLy18&TwCVZPi`?f2H8}QtFhgf=8KU;)- zf#&q^LlvT=OKd0e@+a$EEE}jCR=mT=^!t1{7Xf-9Ej=6kse&}K8~WNn>D{}1BzTW* zfZEtx^d_#YaZngklq0*k;VMH`(`gEOpGVw}Om+2g*?F1+y29KMGyZ5zYfs*IyvGSp z;<%V_PSX%1fOV^EW-dRuWsy0*1*n?W;ZtSi+rSJ9{ds(ZIWJ7~GJYUjV&X-|-mB~o zEm^ZZI9tq?d9F#M;B-i&PA3x%ffz_Lum(a8?(6sp`M(n{1Tk8 z1FJhGy3FeN7n6)j^7HFYX@-G_Fp+}*)99r@$%+CcPf~hU0lsbkEdGKV*lf>k4j~fA z(NpH&nb8wUidoXGk059@;N8wqq*XLounLNT=2MWrJy&=3RR|DAC!MW9G)le+!!x>} zCRKV}xlZX|Q4(aSs18OL1Z;kx+Ok6OjEzb{k9{AJYCnpDZY;BJij+0wh#ZMEyzU`` z^E;D~mMsucsIPk##^55u89=?5t`O;8Cyu)t%^72)5e>Ksx7Bd}=vf(O;PzPyN*VBtrh0qnQdtQk#JEbuPA5B;uRPfVx`I%BB3}oZ<**? zBRV=D<>gDe0jG=5W9ClnYPB({#dIO9Gg1Zcp%LCHi?W&In$O_5`5{xgyhrbCODd29 zAG(ej2GOgZ{8kAisPi6*j<_#V4$OmkoKN*xmgnsxUfuV<{y|1)LcQSugJ+85tx*vYi-lH9dK;uGk=OLc-Z!>PVflW_OTzKxPy0!l1!Kb{OgyI~w zH4yxk8bCCxj+LIAiiEiXw>e*1*8NubaFnNLE|>hsVU?F23b4mSf#TqMZ>AZWu;wUvHMjLD>d;Ak;LAW?eH}ig+cM#7^=GW#pAcyZ4B1Nx)nW6YvJcw;wi`~$m!^-<)djC?AD{`vXTGCk zToU|9Q5)OK{@ z!DQ(Evz)U~K%}>p_s6tInQ|)@s`vUyL61*EUgobfNyH~O?OoWx`2{jAzXkq8cXp-I zRd7#ddHfd!qfhDv!>%Bm3C2+FsAv;?9vNeOo*RR;Tm75QKj69P#w@$bcDAo@47oml z)q@q3#SNvWk@F0|T9jvj%k^`DDh1Xw?*}T!1KE6|t!nW^a0F4}@m4Y06skp&VWh#u zI7G<%?T(MGij~Uwx`esiHDLdKt2DtiVmPgQAd%vhhT94Jzg7)LZ9U_G$(MgXx2!lz z%4IVXM-wrM77a~LZ#f33%k9ua#7t&vm|hS1D2Kh-X##{!dOT(f2ygT3)j4!uY|gNq z6jhY#1DxT@PFBrZsoeS#aKwzfy-TpML->eu`_P_{FkL_Tod8XqD{jpE%{_gEO z-VzQwwW%$3i^D@zDwy)7u;dJ9$(?X~I^}-+1e*4Qm;3`&kn55t?Vr53zYqneZ5q$5 zxw+REB;WX6bCAnQf#@rvkq^Z58$dUOIG8|P*A&Ava+i$`jJ+3=t{;%@i14qfu74JY zzjhu5OeeYc2(o2rq8x_%hxO!BlCl}QXHp}@%Cfs2d(G-W>yVki0xf=r^t-4+`*e$0 z+q9XTF0)9r;{p6rd(p{(FW<%c=RaXM7=?3zlp|?We)S05aYeU-%%>&ryo03W*L|s0 z&l%a?)~`{liNNu(bys(1oJ*o)yDRQ3!niN&D z->Jb++R|!R;9f;89>bKfPZpvWebd=4%np~~4 ze2(*%$W>bAI>Y2%7C(O<r#p-k%v0z-a)kL`L24e4`p4ZFMm~!AAVU*J(=P$-u@Rn0=M$gjW zmV$rRV*@Z~tY^EPaRu?qx%#vwlqZDjo;}ox_Yh8{RzCK6_FfdFuw5ec6*(&MlPxSL zHfkqpLZuQPL8;pGTg)B4yY`>rg%&V6wReQx7nTzoyF^VekxX;gX(XuM!CQ~+P zhA+Olp)Ud|LNfF3Avn&R)=l&&eRt$=qfMTW(?r}8Jvo?a6OUQ!xRuA?tfq5w0H0c# zqmSwxR^dQW69o)=%-%xBm>-Tg8T9DrX(pw?Gs2hJ`=4rIb7#rkj^_oo2nbX~cCoHB zUC~HeQ&HFEq9KLe?_`j2tQEi`T3-e|wc@vfnZ5eR1j&V&<##bB8E{0|eLdP&c3{fR z#d;^!QY3J|A)0}{bFFRl^ z1>q1G;RK1RM@3_7dbBbSs-BA#^SWJEboq}A~B>iv>1mCpMPrMePcH(R}^T5v-!VdcH`cUm(VPYJxk1_6i5v#!)bB1T-6o#gH zC)ZITn+#Fgk`h^B+xFYf)TQ9&NEz8eHh1^cm;Eb?$u%wU_@}z3Jj~?ng$cs7LHGF< z>Yhj_K89`Q2Z9Owi910P0GTgskFaBi*-AixCe5gRp06B$4l*pte&8cv+%Qa6DTSYR zaT<3d(lj5&{5q9x5X?3%aoWLPjvsyb z@NX+9{=$}qz@|hKWk%(4Uz6cv_-l=^eaG**emDu^+|MJ55KXx z^XqCf;tka9e-cA?`}FI5x$Uoi?teqjh2P&W9LC`ze1t$NvBSrypV@dnSCoROe}aEa>D)b&v}X zM*mo-%f@5VTj=k!FMjNXiCuEOY7?D?zr=#1y=OMa<@Fej7b0cRH z4c+gADv<#6mffEDctY9`zJ%~iCG^~X9{zfcfyi8vew*{;*jE4Rw4ELh=kP{B z3pf%a0^*CMCS(4Op(~fYdPf2M#3s>)l~I&L_LB2W+t9 zf23q=2*&7DK@^F9v!jVHP(ymcC47MAz9gXmI4}T2a zULUkTx}OYWZ_j3u#jQ^u@oL2-)4Bt82gd1O4NJjkqUR4m@XHW$bc9(Ek`XGZIor|Yo&PU3h+VRD zL0$)zC=mqIdM3!&2^_ia$v`{`3Wb0FrWZ{Bm@8KlI>DdU<;=8L4)nlg&BARzlGw(k z3*ATNcJ$KxqHH@P*3_)FCZW+k`o{egoL!Mf?FfFr{js;Qs*4^M4MF8V`7;XcDW~Sl zUNnefDju-dd-7FmK zw$=~%Rzl~GP}#~Q0=TNP5y5)*R&6I5{#-D8V=bBpS0ey~4Z^d%IcNLfB`+YGJjrIS z+K`v}!Oo<#$|~;Ah&FUk)0Z1q6pT{Pvsllt%mIEP&TL%>E8N|V8+$4am$||wZYdD2K>Ae%n>f`|N1$e&asmsu3zws8 z6%t{g27sNk7yw9T`b{#j&I1UcHV%#8$GP4681aTj%I?ApL!7KDQ$EV>CtD7$#VmhS zFR{AYQuU>BZGf?W3MpZ>hCv^H0pt{UIS>Ya5@CnjtpI|HfIFOLjT87>-RJ=M{o-4= zJbuu}G;9q6!)>DVf^$EL4>>=d2pBb%zZ%SA9rrr(eV87!l>s-C1NogSZfTJ1Eumj$ z+{n6R1DUSt)eZdbeiiHkvgo~z}D~B!0=g$Mj+J9pQkik~$B@4BexOYwwxc4chOzP3>mHrR4=YRWCMHW;PVsGb8)*?mBaUZnap@uREAiAGJ({( z6`!XaJGj1|{meaNtC+XA(YyzD`GT9+at?&Wtq;RqN4)`+;r)lYISFnX6QLp@x97%n znX?|O@F}fsLAp&xRgP%}E7%6CC$ocq#V=TWo!tC^1Cb22R#(7;ui*1NH(h=90oc;! z`AGKiQ!Uzet=%2w2RN6YHTBM32LN%bn;Rpia-0=k?_<1(uNlUt`pdXw&}=-5O5Mc} z6o#Y#b^Un)E%$?!V1p7z^_)3Z;zjnf@0GpJum^Sy;?T|McWXaAvUAb-#8$^`mi~z! z+~0S|7E_{~8Qw|ji&upx&ngEAt7}&{S_cpka%;OE0V?wP;@Rah(K7o>dEn;4hcpY5 z`ZE4wWj8(Z-?=BSZeAuikO1qpd4Jt*_ey39bf2A$jRAirBg!!u-NWkxU zFtrI*%h5Fjr~fr&2#7{6Nj`TB0iZgw|g!9?wmbh+RLwWwY3)XQNwHcQxcD)&O>y z?+XV_M`%e{o#AB1uEU8+if9%WpzX=uQ>EGjS$_ZC+zwpQ8^n`9+gzRe(!_%avWMnC zJ=T}1H@6-4C9|tA0oPujFl3nB>FDecNpx8fl`SsyJ;vF#2m>~kcd6@OIdPgjSEAa} zEJRs5H9a%^>oyuPksbSZ=nLv|buL%%kW_6YQ~8pW z&tKDJWCt37S7V-R(jE-IhHW9_$?z?<#Kr|u4-AtCV0!H~Pvh{FgRb)!O&NItWupdur6l0BekQFW7C zob+kxiAS{|(&4$QdoUWil4eeF$}zR&rG-vY1)20z1~s4a6Xur!qtZLC@ELs9fK1{N z^NjE~<1-KWA)+=H3NL!24VcwVG6}Vd5}-cn$B+_ zE%cG(QNGSYKsa>V9p$bo-aEXr_nt)dSd>q*AQl1_+wA^|tj6)v=E8yn^~DMTt+td9 z?YZO92a*>47kTd))@0hXjbcGZ6me{b2x3P-dQm!3r3gsxBE2IJ5Rf8Plp-P^AVpAm z?;RBtr1z2l!9iL=h_oOC$hR)%dEVpw_OJbSAA5exs3VZ%zVBc!M8C$Rb?sSeVl6%|uh`PG zLdQmqKDZ?H{`Uj-nOj1~)~Nn6 z6t#$Q`E~M*MW|cQvf^kU=^6os;hHV@G$>@~CA!d4#bPe>T6Bab7xI$@;Mpw3*Z+9J z+Wxolsi^+7Lb!>b?w;(cx@)zJJnW#^!_i>nS6}Qha9vH?0h8fk2~Z2qi?|$3zR#O0 zd*k&0YFwbX8?J+Ni40g=W(zFNN(K%ppb&O&_`|EO+Cq_^e~4|WE0P~V8+3if2$t1SO}0kke}M?iZ>BE# zrg-)+wT8WZgOGb9Vemqn=^e6zlEg&tuK9Y6yF+ysY=9!Pg|m&a3V`Rys{5cpTcQ}0 zp}EZaI$D_yp=Z};(gRJkO(>1DZMA#r6<9~^KFGaS6i5IUX0!G4jG5MFtyX6X>XuY% z8`Igmhb<{+f(9a#GwnGc( z7CFE{FbRUbZ=Q&%z`{0)task}`@HiuG)}A2Dea=}#1CXgwlo*$2y9_>6=f9(VBuq) zs?Q7vguc}G?(QD1NI6Vu#NIJ6HKjBg${VYQVj{Z+t|ew)Wke`+NPFkq26S>{9%j|8 zf_J;fh0FtrS#!PGH5Vwa-fYtDIK z-B|y@E3RBh%M%Y%FfWG^4pZM?fzMkua4UtI^oAy z{nM4_DGThZT-yLrLe&xb7Qf@C>gw|tjpmI{aw74A*>XY>jrW&m%J>6Ys8z%+&%zWm z>%VZuUFy)}Oy@Z?dXr}MmjHZ;rTfYOn_`N#P-BC$W(D7F>DtX(tFR35-tqyHWI&#! zynGLjux}LXHx3(TAIVrnt11J=U?E0#*W-FIyUtTolfLezO@`+OHtsV0CMFLdy@R?u zUQg>E*xTs{iw|rcu9W4Eb-~ZMEwcuXkDaiVH~4TVtI%2{cl&&%q;R+OP&J!hQgKqh zPFllUP}*2-CU|#reLe0>%vSCfM&2_`@C2Q%2?E=>6X+RG7Q*= z@|4ZyFR-95s=IDSYBykK*2B=H)qe|3j@X~36kV%w4@%6DWD43cdYsSvlqZ(qxE|%P zb!K?Ur9>MMGZ@k;0=NVQOXj}ouN^&UQ}5L_Q^N35#|TEMoHg-{Pz?4Eoz@$V0M{KQ zk1O^C_i{jrFb}$5w;sa@xV$&XisFU^U(?(`S(5L<$5PT}>q(p6`zLquq(5v5q;2`| zsafPLN3;bqYZwV0`}tvxYgTP2lHoTN0LXcN$Dm71^y=#mccxz^`opkVS|U}_;X=NL zGmYQ(ucQV_NM*)9W(C z?o_6?d+tqgnR)o?o$M9H$xRq=5}XREzu6M{R+F$}V6ZBQKjc=wI`}@5HP);w&r{By z;nj<{^1&_6ea}6iwqPeD>9fK?%U~;|`!Ye0n^0GHI&MS<-*GhP<09mf3yO~Qexq@` ze1$t-5){K$rB%g~4hbu3t#SVO6ua(O(Q2;J(1l#6loIc@gu!gm?NCkeHoPrO_gWPC zFP7;b0%uRA^(OxN&$3nF@FyP_)4YBmV@v(*{yyOv`#K1-ago;w=XxCoC34PVIU54N z))Wl#Y`C+huDky9G+IqJSLAKWd14R4y#G9LGD4zL`oOFIeuS@A6+Ur0`lIWMTEB9! z{odogcAoGoWETCDb%w&GnYAm5@Uj4&i_(Dnp5&DxMF-*xomNc@pTW3kSzp6+{y9Do zT>mz{XkreV;Vmywzh7eBDl_tn2&}o<-iG+<>oVz1<+GL@=@47~=>V%}`$e(tYSkuw z?)21v`- z7ICmskIp>fa!)W(taGr5bK8a0$=-wJzfJzxw;e9_$oK5w_{Ng*njhU(J$77>89=E3 z2Dvbt$qt{ry?)5i z`SD`o4DDt`=2*7QB~KLKg398HYa)S1k0L8yA}saAf?Q_&Ex-^Ix7V)!UV+eZ)9p1i z$A9l#pth=cbNyP^Q7K{ywm*>tpbhir-Q9G5#gaDsVVjl+Qu8()scAn{Ga>>R)~q}^ z^ZQ%RMyXyr$#EQev1M=n1C^MPhm^U#W*(!GFA?MK8wRbl_U>%Clyp*9h}~1)BEh(K)0AjJx`N zh8WvR*;6-^)|Y6^pI?SMKQ@*TV-h>ybBMl=eguF3_|*St0Pg-qmqnAW8pLhw!VgKA zGX9;+crwEDwJs+sfNgX1GFtHC7{qfRau>I&!n9kX%cau8lZZVaD_~=m@_XT`3K9W+F)h#XtMX z2GG#&v>Q*cGrV*S&MQXg& z?-g{u?UH?$26un94h4E)VL1JZNxFV8`g8zT_x%!6u}IU z2hCZ~y*>@Gc<05) z=_5|no8-8Dk27LR)p_6K>y0a3dl?NJnXmbtf0GSkowYoO@tXDUXLfhl5{&Ao~dIvx?L9f-EW2XWek)A;{9L=cqpZ&9V@ptFh?+TLv&R0T2;q z1xUG1$a=zD?k|_hpszzWfKXKNrAM#6nBM(*Y#L^I?!%HfVL=3$ev$MLMgLVoOt&o= zVkbwSC735dB+=MG15!rrj{=bnP>gdgaNf=Cl@I4_n*6;uS^8>X8t{mkZ&b|GtNLHiz!oDxclM1raCQ<(D3SPhL5EFSosHBPB#G=7ykn zw-z1;VB-# z*xvw-G(`;he)(JZ=zhP|K6i2g4dT>hJ1jo6h#$wOi@0F>_m8VTBJ8~o)azA#u+=?R zR{785=cN~Ml#`vi#ndQVq$F%7XN(rtp!-TZPiakOZ&?}R=-qsmbisEn`2n%&A6Gzm zWpfkoSgP^9Qr?=`w6i0g<>7jKkd{3>7!Biw|Ag15sFspG+Yoq35M=g7CO3|E&*$HG zHQ^(MC4GZBhTlaU`)4sFBByq!1{rC@C3Zp4o!4qxj1718`(Pp+H=v+Z7|V9SNPi$c z^M_^s-{}z@Fx2~MZqMZW2Ts5Ww7@ybxf@`E3|4drbK}D2!6MDmvMfROD1JlU@rFY3 zjYWS>0{Cu)Aw{=H?rCd6=EuwQw=cKcc;HYu=A8&x*>V%MeeS(TFh*hq71JTjCN=IO z8G;*^C(}ADU9T=FNCMAb3!r=HrwoP`kG6EWN?5Hnycc5T3(52kY+en%gY`n=>jqn| zu?MdhmlZL&S%>VLUZ^4R*18A_R}boadY%xs(~A{o4Zuzky2)_YP;9}(n*qSL=`6c7u)!evN6p*NTW3=4Xmb&?T&{JC-|}qu4d`-Pqe`wk zVf>!Vn_v6t6esr-bY0Hhlw4T15CCrr>k7OBR#})&5oUIBnrXQrBB8-&MC8ajvb_nH z`x4yy<>#`px=cpokLhLyw?RC>)4-q?S1O9+!WqoN0c44*Q9q@26#`=8MIz9wmpN>7LImx4d3zNL` zmTnpand&^ho${o6;;duZ#M+4jIdGP?hGxS0BWm%r#Z)yp1g8TW|p(6UDa zj(@t5u=5NI*{_z$t~Ty!QO?~qvoohU1ghc#wlWJlEQn1T7d0I;&(NbEp=Tr~anVn6O-`M8dg>0rqx$R2=@#3Zl4ZV(PsTJo$1RtITgHPSKZU=N4@-kx=4hb zCAf9F7lM^FVPR10wWpQopX7ABcX^kp3IToPA%UyOtxkqddZBq7+q0?R?9Y|jddgz- z3_jtpadJCaRZ~A4=wt{P2;3Pgcy1@hWQ`BG;1nSWc*;Eujle#s08Bb>2k{+=zr0rK zA)ar9yh_H){1O5($>%-%o+314>Z#G&CJkv+RI{!7&|lmLI{o;9iH%4Yw+SR-JX^mO zGYQn!NS8TKZx2$8U^kUmJ9o>X;1rf?4BVP0gXXmff!bfkhkh}+ zR;^hl>-OkKFD$omfq^^i}f$RppOAvR`hLEMKd?icHvIn_tF za_4$~?K{~kyU;*&C12KSG;yu|x;0B?s76o%%`uC~T(CR7VOviZe3)c%p3&qN*seV+$6le6J*ot<%5|uWc8;y9|V; zQ6Ff&3(wt>=c%^DX#3fe<~@J1XOVW5L1l?%F?9`#88WkTQ|E(G;NadR0BwB3ed>L{ z8!+4mv<9K7DJ1pRl#|g1bHy7mdi3DKPDfSdb`Juq7x-7=o#pB1RPohQ;C>K>hpiCp zhnxT9+6hS9yri)3+x@qDy3K4>r+0)IK_pC+XmM+Dmc|cSBewIxK;wgw3#>LT#Sev^ zki1vK%p(rV+}=NVmjVkG47TA(i0OffX^l-_q$L;xZE*dncX1LEc4PC+X2F*h`chle zNO-vnnsC?6y&sz4p?A3{K@!Hf-n#vcMmyPi|MR1L6f)q$l26q_b?2l}8U|^XKn?jx z$rRd&1u#@aF&&VS#g}uDr^o+_FPZ@0L6||;#yZCNx=!l6*7J2(yn65Q4A{)svk^4o z15?X6KyB@c$qrZZ?>P`ebP>u~*X7U>jDV3KfR47K5Ue}ACR|oR4GVPa&$7%+x7<5Q z1G4@wi9={BxaF4deo zSPzWz*PPh|(|sQ4B?>$Do~ueX38-YCTB`F-|n ziE&MMtW@)YcG;+jwsK%N{y_;gFWxdunmvRfCRo-eJ9Q;`#)qz@EVF@j)&0$SpPlq6 zqxwLzs4fDz&sO@>@Ab?^+M^_$=dc&KQpH4V&2DqeX$8$mGkyY5Phb-yR&*IanD5>^ zyX1Qt;p$VcIiRr^UrGWoe^iMxNVubbD^P;yop_=mXgv&yA5H2!5KM=;cN(7GN=;iS z0J3eN1I>VVsSMDQh?Y}9m~RK1_S@bvz((za+{Ps(&ac7PW)D=`rNmGq)vnacT!S-{ zU&0<#813O=l07PpS#W`ESHW7zT$z}CE@M#@z-f{G~bL@Z@ z8vwW7Bf5hY{v?q^7u^vB3GaFPwj9<9b{l!=KWwM25yt4X>2xK8(tO*jD|u>H@~5|( zIc_7;x8fD>8?0!xX5eI5D}*~@Xi&)q@4`d|NWs{Fr` z5dMGvZ!vsE!Xc0%*#+4(0d(8}go`>X0MdX(EAw?L%K#nP+62n-fMld-iX6P_po+ab z38qI|$co<=Pyvj#Hx@Jl)|cSZOT7(7&yq*~N(&+4X+KEVUY@15t4{rF2zcvl(Bat2 zR03X?0Fjo(9n0N_Ky$77u?K0HZ~+5Z+UKY`;QKJ;JGnH9R0IJp?LXQ|0%<51XX+6e zAmu#FAMN0J4F&VefP6BzAE#$}B$nOu$XWO=$Z64QcnJd)osqC@`w1kpUj`;39yt)y zL23^>UlLuCL#|W|e+Gf^x9v`0`p^alpQEe-LGE19gAb&3Y?r}O&BIT?7$x3Qw5e>3rzq;1&8}dpwUxD zz`oo816MmZg4)4kmprfvFN#yG>LQOMTqp~7jepgd!OrJEu4mm1>JP9%1k~N`K?rpi zn}jI8p|04D6+qYix4`USD<-&PHvZD4a_Zsz?co+cB*ZFGBu?{e~_uRq^RS2AvKl zQB01nt^rISZg4XwO%NJ+Ow+b|$ zvWo2Cz{&uPl$tfD*jyZ|(Of4O83@q&G>GC>TdyA?4W&njw>C6@N4TN82E#z@(Db112+EV_OV2b2WgKR)I7qslOUz zKoP5gx+Y?WD2@ju`jP9eRE`QRLFwLm(+ZSRQQ&F4my!?5FSqUn4MCr|1bWqD>zy#l z%mmB;LBK8A0T|sk>b4@YneO}+8mJqFV0-Ra8hNNNWvekT1rXOioc;?Qi|)0}eBf;ud@@p+!+p;rF^C@#a;OLryN;K(a1#Yh)ZrBr5uheAxG zcDj-;PU%x3ue>^RrC5;#p; zfLx!WS3k%i_V4WB0PO+>H0_S)4 zny+Pe?e!pKTm6v-ysQx#8E_lLZCbZH(+G{}1V2Z~fWFDdfp7+j*U?`yvDd_-Gxk3} zz;F?RxH)P=2utbT{DhKnB`Wlh&KdgkK~0Qe=h^skI8UGZaa2I7A9!{_7rSo}airExK({dC(H4!rSos4z;1AMM(n8(j@;@_qm-KlQ(?g zC-2JDFa)E}ro-Fj(B_cV02ndn;n4L8t8rOH>Jpj-RJAo)>U_GTAjb0-Xy|_lmV84C zW~k^?>e@4%agv{lw{#J@iZ@Fh@FBj2=EFl{wi$hd|I>yXdfm013WUlA(w$S*0DNi( z%<9$38ai0PNwmnd4xdhGBOF(DFb7JIX$-%EAS`sASlTJx2APgk!#)t5e+M58&>U!E zARwoxv2k-rZ5e)lj}`#g>iO#F@3jQ{`Qw5Lwj3-+Z$%X8wtl|}0? zS8R(Z*ht_e9BY>#*IB^>Vuy-bCwv(18}KkseWXEojvXh$AN2XxyZ^XOCIc@h=2Zh@ zT?&i9okvCu_p7%ec1qcR(6C6(p)K}Gsaz_c4bH{EJ*X#BPZ0hNEl%Gl$eN(g{I1B! z#YgNMh6k3}GhmbW>V_pwV9d8GUZ{B+mBw8UJ(FM_$8B%JV=rH+0rDiPPAiawD^mQ+ z;6!MC+YHCKEA)4XAJ;bf^nxwn>?ILFjnm>hT=c2o=EaF?I&G&EQZQm|uZ%9{ztGJy z?3st;E`i8!6c0eHLyL?6YLyl}r1R+$LzOgeGR{Sg!+~w*nd|ZcPQ0jS22r~e_^zyB zceEAtoJGKD40^CT;Mj0Os{K+pVw#gOr$4xmm`*`Ii;nh98FXi#2zUDY<$v0?SF%r_ zsSO4Qab!N$19dHMU|q>3bC+5kqOZTko2rRaALLbi6PIJ+W;QI08xcFj!5vIKE0>BG ze#TQy5I@WKT(>)m77!%skQdxjgjp<%3^EhEY_jbA;6>|~YKV#MK3&hZJ)abaqGu(T zu2|&xR;j;g6;C9Hx-Z{F0y{FsaC+F8*Yl-r-1itydIj!M=Q^vVAk!9TwOo!Hjb+t> z4h}h7Nupx_KDI-}EiA)$4M$d(!*PTMNr$U`)$=87>gNxKQiY%={W!Oq5e`=- zn;RO|>>J--7ljoqT6@Opm}PiD%i4bMQqH7NdFGYL-a9nw^1sVl2tYGRsTFJ1ej5tvqexwD>#s^0p_xnR%%j{+6A3m zXCSZm?S2|d{R~78)?nK6GCoAHK9P<;Gq8M@eQ`QzezUZ?J)1Y0dj=jIAyn%}tQ8q8Iyl;pEMTQHJ7XTSh2c|s+%O3|7cL%+gbn)qStWP4v^p8}PA5GUj`}gHmJ2t? z7%q9_WDTxEUcQ)HrT9Ds<}h6lbT$M+End14f9&nt&gf|LNZQ|{Kw??5lqj&9agRzA zFRQFT@hA#%y7v`Lc>Wg6;n6pzhfrW79-z~w>(F}ZBuB)~pgK@!ra!>};9kXp^4u1aZ94>EsYB0HC#y@N@}~CF8cWbZr3_d^B3iY8%q;6hHp%K2 zhBa||O-XIgCLa}hXp>{a`4H`RN0LxVd8=a{N7F0QX)p`jGZIA1cDIi=%#VD(mY%S> z{(7Ra(zl2VD$Ek1|6B=zS!w4(1*d_Wse1Ejj}JFup1dlA@e!sgop!oWh3Q2g?MLv? zLNjcIO~3QacCKYbZcLTaLytZ%4L$kM}T|;U_ z6uA=CIoZlxSk@GN`7bwMiA6xR9kDi!(kN(04KwJv5%TJOlJ|EXeb@Ta^f(wCpnvZB zc93r-#))5`F%2oIEouZgsDBsr(KTon%dI_Jsj+P zna}6Ybs(UPnjCS4{%r(kv)WT+@OyM>R)_p|?LAl)g!s|GPPB72?w#JJ{A^nmy^DQN zHLF_$SQMx;uhe)KL#}U?<-69e>euG~gx7N*DI!rh^JdcguWb0}TTT`1Q)4dJR}CHg zf1aohAW5Nnd7)J)>s{Fz&P|58re|PgBgAg|JY*1I`Z}=J4I!hf3K^+*{N62iqFOB4q$dry| zpjSl6HlprYpYAhk=y{(NV;Xy}+npvTpCKM;IOTvM!t?CeUVqYWd*|yURo=kRt}E+s z<4S49H5aU~$pSP7mYr8859q*As8eVzCBXzNYa%##x)d9H5a$OxE^a-0v(j;(HBhtp z8z@AbAKomNHg>gHS?SYSMi_ijW~>GA-EA#43qQ?xD;U^|&*J+)qm&sLfB*)G6g1)k z5xmrahoBECm%G#I znl;35vdQ`Z`-MXgJQqIJ6393#MDnLF}O8 zE>U;~E@-uwP#c$u0B03{Vq2bI&X3ucB;B@DmwcOBx2-|G-45uT8`3%4!S~UQzR#iS zf#ER$#x3lmHu1xOG-Sl7fl=oV##s2#gngBZ`--9=W8u*?DBiK2(} zK%?p`w9IQ#!IQm8g4sMNr8TN8?4(9p!Gx9#(|~j@MO&5xVqlU_`TbTx4o@?@1BSOx z1RJ4;uJumjw1R9Zx{`-ZpL@aSjPB+Cls5-0-dBXGKeu@mL^~ydY~&3nZ{q=Ty7-RF z`9@!c!C2!_NXoGZf8Zh$5R2Jy0R~(^F&X|l7`0sdW0wB;H+?9V1QGS4fIUPn&N(H( zxLCfd)&0kP$`SNGt&;OuWUu9jEQ93dw@|zhUQ{GMW|y#2I}X+xKsS2750J7j%|DxG zUyA?tynCr}tW+EWKl?lhtR{lc0SE)gr82vm`PSokefFEbG<4*LEI?X1|7Uwii~N75 z4i1x6#j{Z=gBky8ixqkg!Edcp{-3LKZ$%RoyR0h2SVU06lIg}Mi={60S%OmMr0va04XB>0yBIy+iE=?o0Y z0uTjgIXkn$65x$+PS8Bviy!S~Q1KEwvs|2xy%H@h^_S!x)lOKS;xfh$evi`IpLU`; zhA>9tLg_y4o1Hqz`bo@7~~p$nxJZhKdLIWaZKxE7Na`=+SnmNV_$hW*nkJ9 z?TAxAD@#FWk-i{Qv#0A=Ii`QQr=DdjU2gj}zGw?#22sWY%TL~uQ9%{MclHBR$4X`W z>h^|n)<`V-9|&2p)d$Y9)$3!{{zF~OH+?IeYaCu_iUkDsIzraQG-@e9(jdet=DM<&63Rj&rwG#hvDj#Jhk zO~(y-RlWI(S`FnFFq5iA+K>QX#{Vg;f0yjEGpMsEPb#pd=(d^p2?b9FI>V3gC#cYr z9hC8MmThYdmtVV9&l}JQqDgTY#;fu|<**Z$r%w$ur3L;81mqF7X+7!*il=D}t$z?L zN8Q?YLc?5?iea$C|MW~zKjSPz#~F@Vs{a zLiV>qFxs5WVKczid_gzRjwUC_V822YkK2)6?FG#d_B6-owp2kcu-*1INCW%7OX$f@ z6O$!?UrQUA07BK_%-S2=9xq_EClT#$Qh603xB9XYa#@NDi?UFN4aEo9bS@emoolNM zOnRl1a)6G*c>NCT4_^G`{e;#ZYTkYs`n};)ZCcT?p9d9$w)|);+};)n!naelfkNfuOnn5(t8cE*0^R!JZf|>~O1ll0R1MQC;;2&7gI1QloGR7 z2mGA2R_hTq@m@aZ1Q!!vJLU4CSz%d{E&z~t2QpYQq^(MTPKM;fxIWRELG2Ok*&vob zNoU@Q*fw1eRmDg+1STX?LM5TovPC9G8-po#7^;L&=6)1e5;Bf#A%MN|fnmba66#}{ z(NO7a1bCTTL@-`u+>^C0W?I z549PApm>PBTyS5)_$IAHafqAiX`GpNJX)7-cJWF|t3R6e1&-%R0-ex#a3FU>7H5ii z(V7_11WBjyFnx>9O*JllVZCO7-onrwa~jsPM2ZY=RCPG<^Hc! zfLH++p%`tOx4KVS)wx4Acs8yMxD&aCV63mcG6MipX*I~1Nm}18=*6+YtmGYE7HQS< z>lbVjV#=>P4ET?QZ<7JYtMH~?J&Jc5(>?fM^}@%sWZ!8PPv}hjr*s-u(Iy?Mo1W0+ zPD5*6hI!MF=~n_gX@xWK>K5F{FZu&qhd;xF$(-Bdf#&in1j*bp{xJ^_Fn{x2 ze0tZh&@!hGv}73BPRENrom?V-vR?oW?Yr08q8wTIwZR-N2ENRQ{KkRKzQscC(1=R| zQ;QWoUS??lqY%_x+kmBTMWX5j;S!j3igu36pmm$Q7K@k5DVp!C8B*%9U0%ie^rbXW z2Us^sqUf%VftTJn*w;`(Z>m|Q5b3T4eYp6<`Sx^ZjywmmqZMurNX~<%N->eJy(LkK zGTh|am0FT8zeF(X_0lDONh7jWGRBx)i}L%JLejHQvQcg>j$j*b+O_8ZKU%^?@r}{b z&}mlc7I>%L?P{cZGyQVyvvHk7&N!+tdpY-jEgr>#3eNuSEzb3P*-78i>p=X##pds6a$l({WpTLD%ks$=(-ORpUm)6NtW!)o9DI z3BNRWjWa1AYQFkR(gwcqAXhYq=)zIh()|7Ou2069tgcm`c;lvfphWrys$cAL6ele? z$3C>aa4W86*+lDG-YnUPm1|c+>oWoao0Fr^lk&t{E`^t_PpP#b#S3nuwfruIy>InN z!7(!>ewq9iop`w!4_eK|URud26d2e<9Kfxy6Qe}L4ZU81D6|m>TzHOpc*VDz{Ym=Z$a$QT`cs+3ZubgwGn1W}RD}U#@iDVUC3L`{CYf6te zQ5{%-ZW^j`X}nG_aQ9M26s9+8Zk>IA-o^5EtO3k5t+k>yXQ}!#j|Hs04R6T0@R$4O zrUU^ppxD{ucw|oU-K64-ikTSkM7SbOcI!xi^iZUEF4NcWiXWL%-5w73^)7=R-3GIO z-0cN>aaSZpHgTKVd@q>cWJr49g*}ixddXwRVOvz!%l}h(^yoHhsMyWcl+Et{{k2V{ zsWH2@|HX~p+^;EaHEVy3`A&j@x@^WHK$)0<(M>VQs}b(;fAoEnRl^&fRj4&2B|hVT zU3jAvRJWk^LGYDA+Dwf?!8zfC@QPf56Pu2sL^#-PD?S}rZ?N{H?w}Lo*-59;2H0*c z_jzjcGvJu~Qi71N9p-?t!ao8qAfw(x!uPlx?9gK$fm5u$i-^*B6t|ID;T%HSqI{i4 zs{~4b=sHF~?pqoQmJ1ducofSEID z4t{_%S_Qp35Pv0w3W+|e^D|q--@vcZ2Uc1%22zAJ*hI&WwY3CbU|*scw7x13mva`m z>efa;_xPbjvK(K=vp@!n?JRI1DZ6-$EI;pdt1-6KFln2{ zyMfr_Mll(xT?=m8Sagd-W4Y#GZ4En_C>UsgNi!=F8=f0s7CE@@Y?|0hTz0P2X>D5o zZy!P2eWgWtBcd|MTM!1K0G;61H_iBe^Dvq%0hOv-69o}d3$Q4rKWh=^Y4bVYq95y9 z0QJZ1U{#U*+CyqyME&ff3Y901=c36TVYe#}6Dwl-8l8>3WoRDu_}@cnGEjtH@R-$< z4c`Zpxljesk*~{vh8`x7x0%c47ltFvXW%mMArj->6ajdY(DQALnycc*Iji>Q{PPEx zlOYKZAesx&CEzA?)D6C66o8>ME%$ZzY)+*?;~xd=gi!VdSMsZ@uFd8-*lAZ^lIr_A za|mvN7WjWhmZ|YMZDr;K4Zr`;3Xv5rq=!)6jCd#1-ogS<{+t{7rRQqTmG&u2-xQO<=2f#6NM?$fCXxe9W68uT?-InTx&34_hIiQ z0*YKP90=1-%pY^uWjn05_}V(H67IUQ_|WpYZ49PwX~@UKrCjePfMC3odR<&z_UoQh z!CCB8;6&CMGKb`5y6$0?-`|LS&Qobl#T|F_1Ral5*?6C59ZXhrqzqE3m56;H9QZ{k zJ%KBpZCO@RS?Lj6`h~yS9J$)4h|se!v=MFxhKn?VB9w4W#gqqlST?BtvubFg)~~H& z4NSgVOG(HYfez9G)U3Ni8Sq_BUPC-zd=lxt>}bnlcJxf;c~t_X48f4 zN+Pb%v9!uNDI)3BFmv`>)?4|sEj;W|dhL=}JvmA`XAqg&0O3qlQ6&@7>Ar9fnhCVI zpfVA+Yi-OrBJv(4yV2#8*TL1GpfAt<_D(Xrw&)fOAq~Ugz-p{2{Knm0*km8uMm^D3 zW}xz^y9%?_rZRpqJ=>99U1~iz8i3;zY!-y#i0ZGSG$OpYK^1Av!eW3ve%?YYgFFES zh$|l3-GUokGTE_I!%ZA-RtHPR0OTh<#3=#R5f1#!SKQNWr;GV&hNs_C!zg!A3XNwG zEuk9b)C%*>iHA({h# zX}UM+g?U?DQfSrd^6>sugY&nhtz34k<||e*mp7Ibj^T4Ig>3LL4 zbnqRe0bZ3L=F)m^w4olXeCMe1rI9}KRN_vTT1XJ%FL53bEYx`KG2bZa03)$|&@e=6 zJ?AORAf+@^*sVy#Ah8#>yj;NiwnMGQ>3SKxX;WGbRo!dM9bEk%%Ew++YrbE~_Bg5k zG^akbLN<>Q(iovJGpts@Q#<8B%$j;3z0L4pD{(>Ua}2kE5(N=lGv%}1&W1sOXV?CU zt`|#aZ&-&}{Vae3+$G7_ifzFox#1}b)x6`FN7!H6m{kj9f@Gl|*F5z?F$5XLE2Bw- z|F$mkOF~&ceIv-j@WXh5x>xu-Bqd&60_%L2&$Qvg-jZPBrGRp3zOOsI(W3^s?0v_srT)LDcW~<8}}@B`$qU&B*sf>Ge@S zt_`n&yqeo>yXNJygM-l!sLdhhI(k*Y&Ftk16sS*DMyYg-wNtSs$7_9G&VEg$JPq{P z^+iLTO<>_kRmBJDD2Q}U7{Q9`%0g){0IGB)OOdFAXBgaQ&qmb?ZnN%AfT?Svq$W#>4ONIxSk#r$o?Nu&~X zzeMS`M1{ex1-!fi0vQw9m7QFy1j!*guSHVpjqb%v-+2hqso1QVlxGuJNHc51G>P=E zx<$l(H48(&%P8nLV8K>%LB%H&2H`vRAx`i(KjFNem16S}2>ovAOFUNaaCzQzjZ&*+L{;#~*nF_PDE+{`lLcOL2e5fMFmd)~S_;;Ks_cxCR% zogfrplHBFQo)3)7i}xa}ga78W9HD{B1*!U^HtSsnR?K_~hVD)w? zsWc#}b*Wf9d`uwSHyL8y$15NhRH>LuoPNyhS#u^A3&hAe^r59^DZ)Ps2KHBNDDdxKMpe&&a+3a@3k4dZiJe8hk$ZHQA%6d4danS0=3Nte|?2iDe@y?phkN}+>2+e*4@Hd4(He@8Tg&R{LE zdmO26!OcRk&Ewv`?fodrxps?;jqYj7sxfOztLY>uyO<1onBytz99wuY2nYn*zEY5c z?H+6{rhGyfdm^bPs=4^#`_l&vMbDUcBE%l{5{#@h1oR+?d~svwg!r`3wsMWb>dg5SvbS8DpTc|(weh>y1P6D+gBjV>dRb4A0}?>tIA z0^~S(4;D{G{Jkn*pIEyFz=&;aN2df?lJBL(&`xomDoRA-LYFSh_-w#lUiTZ#zo0cI zO6^XwG7aE!wBww;iie*ZaI~(Vgh)7%)OL(>lE5)U{yBqx>KNS!9KPblqG^44jofmt zNLi<5L7;YO&VD*=I^~L1LmqKnv)V;8xeh&bWzDLaJK4Xgr?huCMuCR8*a@Bgkemxg z((+h&8g00Bs`~60BofWcO1aprCf07~P>uV%c03V}-P$sFJY1OiG{Njp!_L2ty(YJo z5xZ{5yuhAW@68BnCYi+iO~0RTMk+s=U`n4}Tax+o^T2`wq|uB88V7FKn<3e%Ogbhs zgLuebd!_qLCq821&RF>aYy`o7?N2_8zuXXO50enrn|96S#>3WC{%06?Fr-^Cd@jn& zZ}Xnczl!M$NK+tmD`59SyFd4O%5cc>aqqT9CFA76rP+IR0HM@*Mic~KI_6`gvbxSj z`9I1YXVtLAS51N#l9De$CRP%xuXqkOU5~#y^fjk(pyF=dH8^bRH)L(50r)7L8d%-? zmElCkG-Vf^H9I2MJ_Yy8UP{@NDd_9Q|Hx2s_u`01XT1iy%<>+J^C~h&)X4UIjgpZM zBWg)Zh6&vY6a+xrobm&{8xN0ig}ycL-bA?a8E4cc{eM{G@&huDi__g}Vt zO%3$2I{Qp~b?osk=T8d2#L6lXBv1?djL45eyoA7QBKQNp}(|#5YS>_!Abq zmriTS*{CP6ZeL1`S-+g%`nh&3J>@**bP(au{?@$r5&Pl<&TdRApJh`eyzyWQFDN~y zec>4Y$rs|gyoB^IYF>1YxSQohF5L-jm@r719@GqlMd3#n z8kCAh%Q{bUiTqwW&i3?c|I_N>HQ1#7-(wQOkwQltaysj+Ln?Z++xkDvhnPMTtl~6z zG;Xl&os^W_U2*qQX!h>L-)(PduVp5Flk@6RA`>r`>|Z%dzoXhyRq*f#h1bw{N+SC3 zz7ydC0<*2@uHO{MOf~}j{=n`j-(nFfJ;qOHD4*r)^JRo|f=Aig2hlBs~C*G-XV9(9>1gy`$PqU&Uqaw;_7pLK+XM$n1bC-_rC$u)G%#6Nt+#*}-rPqBN z-ut5<|8ofQz;xfbjI%{%A8(iUwHC}rd&7~#17>~}MsI{h%r)ae21d&(_Mht1|7ic` zWHGq7nrfxWbf~&ZVMwmPxmMIwi!II-juMhMb93-noi8A+gW zR7)GYKf<50^?o@k!oY_9AaR$6k?q!lPjE6!bgt*%Kew7lmtT%1HSc%iyZo!IzHsYQ zK==j6bJc6<%30cbG?~9l1-)svs2JPj-I{vxsBA`!;KsW%2HR$`mF#q@wJ`11c^npY zjF`0)jbo7u9DUG=9abMCgj$$->v)JJMcGGJhBMq1eI{HWkF;^&RJzxtmC=;!#CpDX=AYmzbV*xtyejOXcu6fwx=2r+J6MEYs)T`YEI1sGd zO5R3VKoDrU+IA(oAeFpCBur)F*7DGGRmh`GlL-Cq`{A`RMwAz90W#uC?hO!2`pN+e z5!=sb#aAc!^$$V(oN-(@q+hPy?tUc0ESH41yhHbn5*u%kXI?f`be;G(fKi zflE-vgkb{J^kqToi2d)k056ifTw~G$^qx z`i3hO5MX8fE@nBeI(wNt*8q^&MZ~aGf(^B-usMvy3T@i)=tz?IjFsTZe7KjXQx%Sz z)~HquM3bSd3;>^~S7hhcLcl0}Mb9mV(&>`0J-#pxbijqMBgO!2er3fIi7-tv&4D(! zI|Kfzt`R&a6%A+YU~ilgc+J}LcnT$Eve^-l@FJiqNoc^_lT>`Ke*tC1NUa?>Da^Sb zSP#vbBBfp;#c^zt=F|KQa4csO7~K_sLNJQg%JXuZJj*qk(2gC*jMm5i00^bnWt_+Q zWw$D@3M6^U`0s)&G)>=rOTYlm|AhmF#a@zV~3JnI47~*8K4E$ zL>SZx4_?!wF81qaGS2TkA!w~KTh`|gXIFdbX9A(VXlbTXAbW*sT}pn3;$BXglkkvc z-EXZ{GjpL4p-jKtELl9IPvX|CFq04*3~W)N85a;K>N>QD4fZq<`|giKi|6Ix@HJlk zybRyV~bZNClVFHv(3D{fE2n8JX3ZEh* zfZk_>^fRj-C1Kp#nn3V156+J->$E9p$LiNyX|!tKMF67yDrcBu+=k-V15C36by5-(`;ymHfWhyd#6`mKDARZwhA4b=BI zSLWYIBHq!i#lk9eFabSqF&GG{IUPszZ=(xt&w2A>Zij6A^iGw-AHI1WGIMT#7S6o+ z4ezO@s{K`f?e|deC?XuKI^e@HU7euAt!H*VM&62dQ@xA>{H`Zm8?xQy42#G6I<&{S z!yBsZ0lze_SXdDm5)vX_wnEc=L}Ee+^OjW5_U6aVW6EqoJydz(yVV=kV;G$adW7M6 zP@l~|-hxBOi7z8vVhUM@k&IWrNS9~M`BEdCfANr^yt`ot++So{?comf043&X<01Y< z%46|C%wmpgeo;g|2ZP~2$1^w%uEb(9(+}>zOyn%{jTgTU zegXdSMBrrl0%uUZYEpb?>?M~Y0}fbnjR4A0%sA%GtX8v{m9@|x9*@kUHQy{Q>PZVT z4TK`BXX0*ZP3u_dzJJ8)K1V$Cvd)!W*2%Oos48y`9n+F-o8sb_Y~sT%W8a`6B3cINL;?(zRuot9InPFWIC zDLd8JGBu>^J7eFakuVf9WMqjFi6l#CEMpyOCI(ZeqrzARLo>FcEMpidhS2bN-{*X; z@Aduu17APz%UovWy6*dZzn16o@!aY+eGVQTsI%_YZC3W*F_g~o|Ug7E)E)Cpkn+N*aj1~Sy>k%B$s6O#{!cKb#gZXw_z#TF?3BO%Tsyj)1 z<>ZV2Wl4eP+cIO9P{Y{lX6g-l(L8XAs5PuXe%*~`gSF9QBVMleTROTN6jPTmTA(Y= zQDYft>w{g)f2Trp{ogy^f8|1Ff;M&@i0-|iwb?QlVlg+y>Gh1#7SdhUp92~)wk@$$ zj{yrra9Me7#rGrI&A3pLhE3b~N3E&Aks`RwVJ*QF%;u$Uoa?k;LoW$0($hzl+3omE zXa@VRxd7hR;>uWT>6ux-K*P$c1U`2IX-)!qq zxv8(?rA%9PiA7)K;tLt#2e32~!X4_GYE%+hbFW-{a62@*bPG9QY;hwoAwCi!obO1cRLu|Lw0X#~!T$|xF!j3l8hj0I*HXbhU_i$BvApYgC( z!Uk={NhZC<`?2}@o~=GoYM}r15A5_zG`SHj{o6qtb8t>C?Z8ty+;>D*!kEh10FR}Y zfGLF2-bXleb}WiD(bB@Xu)hFjSc{u$0k6&yBU zX0G=v8?L`n8;seTcKHx^-kslkv2b7cv=i5X(3ZT-V^OYTgL%sx8FyKk^H_0?=T7)6h0MlzKL1z{HMMLWt)=q)( z#T8CA(_FvlB*ok8oX{Ic+(%cZ&8As1D;gCFYS`9k-LET5v_Jf!r8def#6B(%-47YC zE5CW7ip{6Ql6!u#@0aF9v7#*LO>(WwrLvf#)9HpN+2^PyN^51Csdx+Z7=b^^vMVWn zmUtSfXIY=*I%VpuG{b)TG&x6I$B4*xgl8pwQ^kY9M?YFJ*0QgF8M8QfS$vo|q12f| zZelvt3`wR18IxvNH_s{aZtl8i#SkW@*J0S(I`b7eK}AA=q_ zt?F#H@;Y6;dHbuH2tn8EQy$#+MM_EHo9{%1Pr1cfFB-BgZ@qVQc_Hs^U2s9>bzhw@ zRx@~biFWwF#=OL^R69mIgR`l_xU$+W+u5Pu)I5BfSnH932q^5|>X@mi>LTuUwVQE6 zq!vif^>N9@44*VroA6h7;oxv`Df}1Y3aBq~@{xC{^ZssHu z2dFLkBZBalMf>zPnyzxCJhr*-Q*=)0;2jZ)2M!Qn1gu+~Gbf#;qjol6c4^~N*slh4 z$q|r#{_nv?|s{0MztL@H zeR-sRJzDO9sH8q*AKQJ%^epP7rET1S^Y1QT#>+wm=kfxHUF9{gMg6P&v(XXJu&$d0 zCv8gb(R$NBHL$u>9bVFd&zofNmZ6^ZSXMQA+C={2hQcvyX7t0Itb4!j?&p-r zKL(0=6j2&~p*q_I$4rJpT~SDF7Bs5hrveS5uwSF~ck42BI=DtClR#!vGp&=kdoZ;0 zkZGOlicaVme9yW3!swmGMpEpmlyO&(eOF+~O^Q_*`0hhPtEgXb>ARZ^`tn?A>Bud3 zh!m;iZ>C55&C?+a-L%t^wx@2KnT^7v592peQG>ZQ9}y<`?z&?Y#;HzH2#XXJtj?Qu z=EAP{&J*m2x7+gXZMF5OGKD`2r)|To?J%Q>TrdjE_4;GkH>s))^TzRmEEQKW ze+~c5B{MKH-iY8PiC?z5okzJoi$vWSXceM57Wtk=RlNhP=>qq$miv`sn8hb+SM@BC z6Sl)1<(;Lz+@@R`fjtj>h;YvA-kMKIG1zQ4(ktlC5WJGnPU_84EqC7(1VVwnVv}dl zj|I<`{VdqKM;wt_82o@;!&3OXMbRG7dQJ)j0bx`0|7y#{Hn}NyWi67gIpwCez!4WM zV#~Ggqtr4j5ZEnx2@(<&U*S1(wLx;OFf^okJgBjlLpHiqdIMWH@0A>jw4H5=9UemZ z`~OvXR4LWZcwU&E3#ta)rsKM=Hs zd0)CZCh#;O<$U&t6mo=Ec#d=)BUTK*Nda+tex;L1gLjL}EU_s*IH1FS!HMhM9il)) zUgLuN-|i>aJs;)IQ!Ox%VhZ?fG%ZA|Z&iDBwY%jC*rW72c^efZRXG`I?&-6FW#tgr!|4_jZ@ec< z#)-r}Fs9N&Pi#)PH2#~RaRW7pZ4J>4+LTKRiYNwG@F<9GCaBFo{2`4q@Rt|IYoM#w zjp}VUv>ty@-|c8;d4ABC#jV9xrKg%RTye5<;2!HNTu>%@Mcs9{=5op#=n{K&P|d35 zfNE=Ve3Kd|idU6&>-k1h(w72#nNeNhahBvj2g7zym7Nr`UVkTj#rr zOZTNP?CIil49}tHB*vu$o2?Gii)1orf(cZv_@O1rIu3z2l z{-)rL44m5V?Ec{vyirq5e6Vl+ihLn%bc$q}>AOYIvYmLb-Nagcw3-H^CPd2ktT`S| z`L$_bMA+)!h7&N)n*4Hi1-g`+lS@-Gk=n3YKF5fw%7xne5nCe|;U7P`jnaYJhJhTJ zv<~}txX}h>FJE#v?#{)%mAoN7lkID%-$Lng{^854=3o)sWNCJ8mdCz4J6=oTWd=mO;M&8A!}S z_Pfu1EO!!6DHAne-wVF(wwoXO@;tY6r>V-*#GQvFKrgx!QW14LM2scnqe&gSTSh8e z2-HqR84L)cNJbdk+#o863I*KwwXs&_1~{SS0`Ku%$uDbUVjhSFA{p4D)3I910mA(G zV!_M$_*k491}S+Gt&ZAmoJuj?`RtR<{O0S^)BeyZAUviRFTbBJNPX8R*(gIxnINA* z!$@{(36$rK)DvGVzoGAWI_}D}&4in`U9R?)xwM;wdfS~<4r8ZZe1WjF)yG60mgW8n*yovoxD@1uvxLju;+=fB4n%t?CcH3n33D|b^jpq%uifcOp>W1cdYmp2Dwsax9W5@4@I-!uB`@d|8eCU=46=AuFFIfg|k|fJb;i{_0P6m0JF%C<-_zAfUN z>GjUHI{WsU9QZcm7CyW+$ttMkrG9W7c37<6MrfgQf;kWU2DKWS%~Ka^GJG7&$l@=S zI1|>)3tL-ikrL&(df`D26605jyo}1Q1v{(KUS2)?4_n(Vyrppd>{tut@Er)0`XHAU zI7g9p>er|Tiq(Jj#IpRJiqw6CTzi$#*j)di&tGGGo~Wjx)QAJ>xP4-@Pj#Y!+U9XS zzvLlV&}zt+0W5X&7qAcAL}#iVB&6#Ln*zJb-L3<|E)^Rw1}g}4m4w*zeQcdRUH31a z?63HbG)z(oh@M;^W8DYbCgo=pzMsqNEqa4D%A%TB1RurX=DffKl}>r!l4 zSv~*Hq3r6|7x5V#4(2IsxIlGLSm-O@&Er1UwsWa^s?1)21A2l{p;AIm(rNY4J2CZ3 zV-X#H@c%UxTW=?Y`4R5l13=rLFX*!bHzx zMK@tQ9aN_5rNCbK^jLceBn8E^&t6y@{cC8PR&x==I#vw1fE?YlUAu?8!%zfCwee70 z?nD1Tud;X3P9ioU*0xPLo$jQ~E?OuOOX|s=JNPz>I1NOJo}BK3C?UG()eH~4*%j7P z{10!IaRvfP33-w&a^5==$lmA14qBaRp*F-eu4L>I7Ar%~?bMFhsb}ny`?XmS3S%=SAS+8DFpVA>is z{P@|-y;l8W!o(a=aHJob?jcf*R)OtXFwBihXbDg2_;;UJCh+~t{?r5e5AuNhr)h67 zOc4l0PRYe6#a%2lA>L<;{WZU7RzdbZ!iaeX2Q}^XXf^`X#<7zK`7_OW= z$9L+@&u>Z+e0j(Dx%mf9sP@o*5VKD$c7%ppERmiA`Yo~Z7Jt($f+*fLj~}-@s7wej zZ9`b%Ac_CY&Uu0>K6*bjDD&#dRBpO{3)Vwm?DUs^$+q)_2{!7^kB7{6sGc~fZ~nn6 zLCM7BYH}=sMqc9zJ{A!-*X#5s=*tjL{AU?`86jl*gT+2@|y4x zP+f9k$*UpBjoB6K)P0nDL%4ZES1ft`a>?k{9rZZ?u(3UxfNuydlCD8cu0qMsKFWT( z@3y6|ccQ?bJW>$LE1G(BB~XJc2Fbe7w~`{;#ptH+Juc)j9asVoQEkg3v*4d>f|bQD zWtt(_v(bm?jTQL_UG0A$OXu%*JFQh7610C-_=kL*)@~ij5TfmnX=hx70%Pbb=ZR^Z=-CJ86 zS_Z6VUB+0}wwKeiBc&TiiryoQJ(sW--Ug8)4g)La0Xb{;B`_|V47XGN zt@1}He2&MkwI*W@zFpGIoplk<_n7|sq}3%bHRr4j-<><3%^tNadFy}Z57Wzb7=QZf zB4!5LZwR@*!J)ok3(_6R4g-x$`NNWV|5TY%zd#i1&((60f;VdnVi0b~$1yK0&fRYs zB3o++h&wSusDv~QMSh{dX13W|T!jY>u*2MD51~~1l4s+`Ywyeo?k*VxqkLDFs%%TR zno7%SHAgx0(0%y=iece8yPlfk|J%Efcla7hEU#B4&vIJUu9bExFchl!R_XGwc%;xjN!99PG1Hh~pKjMGYCMD2iWKdTlkkmkuN(v;MKS!v0k}lx&%M zg{O8~fbb;m*=he;xnG9bM@~o~7GAwFdwXSM`n@?vsIJ75*8n3_SUcLz0jD2-lwG#= zKwpI2h?;Gl8yv=PeOA1|Kz+0tu{+FeAz)IBKCv2(V6kSZm#avKvK|a(f&HoqcNJ~Cz20_G{KjgTg z0JZ6JfXWK%EaFmtedbAc2tbTHUmSZYyzUIFJd!|iULPre>C=7=ltyq211v{xO$;n~ z$5+$`0Y?dxIJltrIK2fdrJh5|RYo&3;9i9iJL}sc25|G6xF%rkXK)C^OtX&u!1Y-- z&^dHv7@q)ZOW*{!1<2~-x4_b`*8u2q0Zfy!7z93=zaANR zW?Fuz`Dwj5?_V{s|1ce04e}JimgbIc0z~z$5`_DEzNMfIGw>D*Y7Lx(Cy84N3`4G*irTq zZm-?73juvZH@%Hs2VVpOBaNjdU>=e5#o8UGbGHI#sL9)r^0pN+Kk=`p>9d1L@#D7s z8UO^>)ffJsfCEya^=m$}GXvgtcAdPlhNZ|#hB0sSPAk7*8k$BaRk9X;>&GDV^_~mb zGQfuNeq%`y)`xDy5BG!tGmRuz>#Y`-aOp7O{f*gzW#43gni5za1yoS2e%W1^y?vk$YRF>21#_Oe9 zq^6S`Py)zVx!O!naQ0XRXcNSO+<=FWtJ_>oeyJHbdbjP|*!J_DP}!kX;~D3_eBFu$ zTEMyapanfOK!V%LIw5Uv25VCBP2JWBL0k2F zM@lvuk#?O(T?VEoUBKjV@pID46~&kt1@l19m9n^gA?M!aMTs6kpCyoGRnT5DPks=g zmn0?I?*^w-}71eF9>pffWBZPJ9nlAq|@JPbIY%d96+~L0RNwXTS|}O)^cnFpy>$w8pAC8_1P!; z94X-U$QasgI4O8ueC%X5*w2h(-vh><{X#?Fbj_{aoagdY4SzQ;0(g-b8E^wKSphrv zbk2aV?WiloD!)&BwY(~DR&m0G*8wu2prTY*~Wfgpo4Z>Wl84w2^n9ulLI#QWCgq$zu2Ad1| zgcUA7%t<$Pa9eqZPuDd^l|A(iRk@Oo!%EaOU!1#*L^~n6%O=mQeCu5;nhpaZ1x}wt z7CE*nFn9Tuwn50{Gyzn(W$aPJjtXhVF*W2mQp!;ARx&{QX9DOMZp1nhUCZwrt1Iq_ zvvM{6=`*H^M$4mWtU~DnmAJBSU|f_}L!*+9pfU_p-^9Rn$t)+37NEcwrED!^6!q0m z*!!p*t~2@wG=y~@kv4GI(|ro}#MwMsu2mP1XV5~Y(}Yct*i=$LB6#qCIvd%Gd*V2v zpQ(_BRvPcKIcNDy52^9PT-h~N9kpq(=PaVJ`ZS6=)tGa{PVEuEM>9Qbu20EaPUw7y z?1$Zb#da6-xs3c^msgWTa#KTx z5MxyeJIj>o5?5;PtsMKh(F_YE9jhiti2~eN_V*R*tyBjKk*eBC6$EE53T7dt ztbWk;II7oj&!21I;u7$asZBcE=+5+XP{TfDyq-Z(!e6ptOOe-gi7y& z-yWI&GqSF&wh1Fs|?jTlSguRm{4^9 zm4>lyk$pj^I*swW?ZkdnKye-JrjVbe-Br)QRJo9_nkKeNy@Je%GP|)*vrfVSP>jKQ ziM>)n;kFLHYW?8SKDoYG>;!A^{BMRFo679YDd%?Hex?$YTI%=l#37dvr_MYcVYz^c zTQS(x=!cxy#clGbJ%-|0ulKFnv}L-M-=Ax`=b&kGcvE?>yiz1U=X6u`{PDU2*zoG0 zc=qnP^Yq}sMx#yLHPMvk=hxz`5YVy>ZdclD;bo6kPFHeai$^{h+A^)BYid(Ldx?4}K^;b#v@@JH7ipU%U0y+Gscg%@KWg zkdKe5wlN&_HzDE<|5C+c2jCZb3B||1%I8Nm2>hxzb#u4T{09I}n}p!Dd{K&lyX?@Z9H(iCwKpu=Pr97j-=4=Gy_6!fR`F(BErc0LV}4afR-!1Kja-S zt4CSsv=({WpXD#vbqagpnyEJ7#Eo*}45hn|={uwDlL4RMa#wZiE(kX6naZvv-{lpz`bs2&)-D+`Q)h&>>G0U{;yCGUmM%#nPBULfpc z2Ke(-K>kWLFo`u00Y$eF*sQ+1;@j^Czz5yHL>1I@{J*!6DFPZ~PslmB=@@TX2IA^= z=}A71;{hH5rwddNuf7%i!FTiM^?nHUEprbX#mWi7=hc8$Mh(>V%0LFA?D96an|tG5 zcRzSday@9Q;ZX7?^L-zH+hlwbIv<2mbpY s|L6Brq7ER_{XaiZ|KEuwd`JpUtFqo=W}1wefIf?%u9;5NHTUTM0X4Qd(*OVf literal 0 HcmV?d00001 diff --git a/cookbook/auto_router_selective_training/quality_cost.svg b/cookbook/auto_router_selective_training/quality_cost.svg new file mode 100644 index 00000000000..c703fe285a4 --- /dev/null +++ b/cookbook/auto_router_selective_training/quality_cost.svg @@ -0,0 +1,4023 @@ + + + + + + + + 2026-09-16T21:07:55.079486 + image/svg+xml + + + Matplotlib v3.10.6, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/cookbook/auto_router_selective_training/training_reproduction.json b/cookbook/auto_router_selective_training/training_reproduction.json new file mode 100644 index 00000000000..c2fab082498 --- /dev/null +++ b/cookbook/auto_router_selective_training/training_reproduction.json @@ -0,0 +1,10 @@ +{ + "status": "passed", + "models": 118, + "candidates": 1892, + "selected_profiles": 12, + "ablations": 40, + "exact_coefficient_and_selection_match": true, + "data_scope": "Frozen training and validation only; no final tasks loaded", + "selection_sha256": "1a7c9241c72c299c8b27fd8171881f94fda069fb7191ac6bc3a2eab9de811853" +} \ No newline at end of file