diff --git a/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md b/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md index 70cb0b329dc..3365efae948 100644 --- a/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md +++ b/cookbook/auto_router_selective_training/PRIMARY_FINDINGS.md @@ -1,6 +1,6 @@ # Primary findings: quality retained versus money saved -Training improved some comparisons, but it did not approach the ideal Luna-first policy across all 125 held-out tasks. The profiles selected on validation do not combine preservation of every Sol success with lower total cost across all four benchmarks +Training improved some comparisons, but it did not approach the ideal Luna-first policy across all 125 held-out tasks. The main profiles selected on validation do not combine preservation of every Sol success with lower total cost across all four benchmarks Sol solved 87/125 at $20.8964. Luna solved 84/125 at at least $2.6531. There were 11 Sol-only successes, eight Luna-only successes, 76 successes shared by both, and 30 tasks both attempts failed. A perfect Luna-first decision would therefore reach 95/125 at at least $5.2653. That is an unattainable hindsight reference, with at most 74.8% savings using the recorded bills diff --git a/cookbook/auto_router_selective_training/README.md b/cookbook/auto_router_selective_training/README.md index 5e28c3a73ae..cdac37db986 100644 --- a/cookbook/auto_router_selective_training/README.md +++ b/cookbook/auto_router_selective_training/README.md @@ -34,6 +34,6 @@ Costs use recorded gateway bills. A ≥ cost and ≤ savings mark unmetered tran Costs include the classifier. These are task-pinned replays of fresh independent attempts, with one attempt per model per task. No guarantee of quality retention follows from a 125-task sample. The aggregate combines different benchmarks; inspect each benchmark before using a profile -The per-model control is an exploratory comparison among forty prespecified variants. No validation-selected profile combines preservation of all Sol successes and lower cost across all 125 tasks. See PRIMARY_FINDINGS.md for the interpretation +The per-model control is an exploratory comparison among forty prespecified variants. None of the main selected profiles combines preservation of all Sol successes and lower cost across all 125 tasks. See PRIMARY_FINDINGS.md for the interpretation ![Observed quality and known inference cost](quality_cost.png)