scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-05 02:41:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 48

feat(router): add trained capability cards and rule calibration

Browse source
This commit is contained in:
Tin Chi Lo 2026-09-14 16:27:29 -07:00
parent 9a664bd4be
commit bbf196bea0
11 changed files with 2098 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

47
cookbook/auto_router_roi_training/README.md Normal file
View file

@ -0,0 +1,47 @@
# Experimental Auto Router training snapshots
These opt-in profiles compare the original cards, a research-informed card rewrite, and cards with training-derived capability priors. They also compare raw probabilities, per-model logit calibration, and regularized task-dependent logit calibration. The router makes one judge call; all learned probability adjustments and threshold comparisons run locally
The sample proxy configuration requires a build containing this PR. It does not change the default classifier or its configuration
## Run a profile
Set `ROI_GATEWAY_BASE_URL` to your gateway's OpenAI-compatible URL ending in `/v1`, `ROI_GATEWAY_API_KEY` to your gateway key, and `ROI_LOCAL_MASTER_KEY` to a separate local proxy key. Start the proxy from the repository root
```sh
uv run litellm --config cookbook/auto_router_roi_training/proxy.yaml --port 4000
```
Send your benchmark's first task request using one of these model aliases. Pin that selected solver for the whole attempt when comparing against the task-level experiment
| Model pair | Zero observed validation loss | Two-point validation allowance | Five-point validation allowance |
|---|---|---|---|
| Sonnet 5 / Opus 5, high effort | `roi-sonnet-opus-validation-regret-0` | `roi-sonnet-opus-validation-regret-0p02` | `roi-sonnet-opus-validation-regret-0p05` |
| GPT-5.6 Luna / Sol, high effort | `roi-luna-sol-validation-regret-0` | `roi-luna-sol-validation-regret-0p02` | `roi-luna-sol-validation-regret-0p05` |
```sh
curl http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $ROI_LOCAL_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"roi-luna-sol-validation-regret-0","messages":[{"role":"user","content":"Describe your complete coding task here"}]}'
```
`profiles.json` contains the exact cards, coefficients, thresholds and validation results for each alias. `manifest.json` records model versions, effort, sources and limitations. `training_records.jsonl` retains the numerical forecasts and paired outcome evidence used for fitting and selection
## What was fitted
DeepSWE v1.1 supplies 113 tasks, with repeated attempts for each solver. We split repositories into 83 training tasks and 30 validation tasks. Within each task, repeated outcomes are averaged; each task receives equal fitting weight. Missing bills remain missing and are excluded as matched pairs only from cost calculations
The trained cards add overall and language-level success priors computed exclusively on the training split. Language means receive eight pseudo-tasks of shrinkage toward the overall mean. A separate research card tests qualitative changes to the capability rules and model profiles
Post-processing uses `sigmoid(intercept + slope * logit(raw_p) + matching_task_offsets)`. The slope is positive. Task offsets use the capability rule for the capability classifier, or the reasoning, scope, specification and verification labels for V2. Training minimizes logistic cross-entropy against task success rates, with regularization strengths 1, 10 and 100. The intercept is unpenalized, the slope receives one tenth of the task-offset penalty, and offsets are bounded to [-5, 5]
Cards, calibration variants and boundaries are selected on validation. Boundary candidates are rounded to four decimal places. The objective minimizes mean solver cost plus judge cost within each declared validation quality allowance. The values in an alias name are selection constraints, not guaranteed production quality losses. A tuned V2 gap such as 0.1649 is the fitted policy boundary; it does not establish a sixteen-point real-world loss
## Interpreting the result
The raw model probabilities and calibrated probabilities are separate outputs. Better average calibration does not establish better task ranking or lower routing cost. The selected configuration can retain an original card or omit calibration when those alternatives performed better
These are small-sample experimental snapshots. Model, judge, prompt, effort, task distribution and harness changes can invalidate the coefficients. Published DeepSWE outcomes use different budgets and serving configurations from the live SWE-bench pilot, so that pilot measures transfer. Zero observed validation loss is a point estimate, not statistical noninferiority
The capability classifier uses an absolute efficient-model probability threshold plus its boundary step. V2 compares the two models' predicted probabilities. Their threshold values have different meanings and should not be copied between classifiers

31
cookbook/auto_router_roi_training/manifest.json Normal file
View file

@ -0,0 +1,31 @@
{
"version": "roi-20260914",
"scope": "Experimental opt-in snapshots, not production defaults",
"training": {
"dataset": "DeepSWE v1.1",
"train_tasks": 83,
"validation_tasks": 30,
"split": "repository-disjoint",
"task_revision": "0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea",
"task_source": "https://github.com/datacurve-ai/deep-swe/tree/0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea",
"outcomes_source": "https://deepswe.datacurve.ai/artifacts/v1.1/trials.json",
"repeats": "Average success across recorded attempts per task; tasks have equal fitting weight"
},
"judge": {
"model": "openai/gpt-5.6-luna",
"reasoning_effort": "low",
"max_completion_tokens": 2048
},
"solver_effort": "high",
"frozen_selection_sha256": "ba2b00700aa338080077936a1fd794fd3bfa85fde28baef1fc5560f319ab1f32",
"training_forecasts_sha256": "67ef5c8fcd76ff6257468a3bd4c27cb7e27e0514ad0a037bd81e5b7d2e0b1508",
"selection": "Lowest validation inference cost plus measured judge cost at each declared quality regret budget",
"interpretation": "Validation regret budgets are policy choices, not fitted probabilities or live noninferiority guarantees. Coefficients and thresholds are specific to the named model pair, judge, cards and task protocol. DeepSWE to SWE-bench transfer is evaluated separately.",
"limits": [
"83 fitting tasks and 30 validation tasks support only exploratory estimates",
"Published training outcomes use different task budgets and serving configurations from the live pilot",
"Missing training cost records remain missing and are excluded as pairs only from cost metrics",
"Luna/Sol profiles have public-data validation only unless a corresponding fresh live evaluation is listed",
"Training retains the source included_in_score filter; excluded provider/infrastructure failures are not modeled as solver failures"
]
}

785
cookbook/auto_router_roi_training/profiles.json Normal file
View file

@ -0,0 +1,785 @@
{
"sonnet_opus_validation_regret_0": {
"selection": {
"key": "cap_sonnet_opus_trained_card_none_0",
"definition_id": "cap_sonnet_opus_trained_card",
"classifier": "cap",
"pair": "sonnet_opus",
"variant": "trained_card",
"adjustment": "none",
"alpha": 0.0,
"boundary": {
"base_threshold": 0.5,
"threshold_step": 0.0
},
"validation": {
"quality": 0.6666666666666666,
"capable_quality": 0.6666666666666666,
"quality_difference": 0.0,
"cost": 6.12658085375,
"capable_cost": 6.126236435416667,
"efficient_fraction": 0.0,
"savings": -5.622021561912938e-05,
"solver_cost": 6.126236435416667,
"judge_cost": 0.00034441833333333325
},
"calibration_metrics": {
"efficient": {
"brier_against_task_rate": 0.16733333333333336,
"log_loss": 0.761216678402628,
"mean_prediction": 0.35500000000000004,
"observed_mean": 0.5416666666666666
}
},
"validation_ids": [
"adaptix-name-mapping-aliases",
"arcane-drift-detection-baselines",
"arktype-json-schema-refs-dependencies",
"claude-code-by-agents-recursive-delegation",
"effect-sse-httpapi-streaming",
"goreleaser-retry-publish-auditing",
"gql-incremental-graphql-delivery",
"helm-array-merge-strategies",
"helm-unified-manifest-stream",
"ink-grid-box-layout",
"kgateway-consistent-hash-policy",
"narwhals-rolling-window-suite",
"obsidian-linter-auto-table-of-contents",
"obsidian-linter-link-format-conversion",
"obsidian-linter-scoped-ignore-markers",
"opa-rego-rule-profiling",
"opa-template-string-reconstruction",
"prometheus-transactional-reload-status",
"prometheus-typed-label-sorting",
"returns-validated-error-accumulation",
"sql-formatter-bigquery-pipe-formatting",
"sqlfmt-create-table-ddl-formatting",
"tengo-callable-instance-isolation",
"tengo-destructuring-bindings",
"testem-bail-on-test-failure",
"testem-per-launcher-reports",
"textual-kitty-key-phases",
"textual-richlog-follow-state",
"vitest-duration-sharding",
"ytt-jsonpath-query-api"
],
"cost_complete_tasks": 30,
"allowed_loss": 0.0,
"fitted": {
"efficient": {
"adjustment": "none",
"categories": [],
"coefficients": [
0.0,
1.0
],
"alpha": 0.0
}
}
},
"complexity_router_config": {
"classifier_type": "capability",
"classifier_llm_config": {
"model": "router-judge",
"timeout_ms": 30000,
"reasoning_effort": "low"
},
"tiers": {
"SIMPLE": [
"claude-sonnet-5"
],
"REASONING": [
"claude-opus-5"
]
},
"adaptive": false,
"route_housekeeping_to_cheapest_tier": false,
"escalation_keywords": [],
"plan_mode_min_tier": null,
"enable_context_window_escalation": false,
"return_raw_model_name": true,
"max_tokens_from_tier_model": false,
"capability_classifier_config": {
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"max_output_tokens": 2048,
"base_threshold": 0.5,
"threshold_step": 0.0,
"card": {
"version": "roi-20260914-cap_sonnet_opus_trained_card",
"text": "- SUP-1 [supported]: A bounded transformation has explicit inputs, outputs and accessible checks covering its material requirements; merely mentioning tests does not establish coverage.\n- SUP-2 [supported]: Relevant repository state can be inspected and a localized repair mechanism can be verified; repository size, familiar terminology and file count are not difficulty proxies.\n- SUP-3 [supported]: Numerical or typed behavior has explicit interfaces, edge cases and tolerances, and the solver can validate the core computation; a test harness does not solve an unknown algorithm.\n- SUP-4 [supported]: The failure mechanism and search boundary are supplied or readily inspectable, and the requested change fits the configured solver; multiple mechanical edits need not require stronger reasoning.\n- SUP-5 [supported]: Behavioral reproduction has an accessible executable reference or complete specification covering edge cases; hidden grading alone is not such a reference.\n- UNC-1 [uncertain]: Material semantic choices or interacting invariants remain unresolved, so several plausible implementations could differ in correctness.\n- UNC-2 [uncertain]: The task requires completeness across coupled paths, asynchronous state, schemas or compatibility surfaces without a clear search boundary or sufficient visible coverage.\n- LIM-1 [unsupported]: Exact information must be recovered from noisy visual or temporal media without an available reliable extraction mechanism.\n- LIM-2 [unsupported]: Success requires inaccessible reference behavior, undocumented state or unknown configuration. Such obstacles may defeat either solver; unsupported does not imply capable success.\n\nConfigured solver: Claude Sonnet 5 at high effort is a general coding solver. Evaluate its fit to the actual mechanism and scope. Clear instructions and runnable tests help, but do not establish complete coverage or guarantee success. Large repository size alone is not difficult scope. Public research does not establish a universal Sonnet advantage for a task family.\nTraining evidence: 83 distinct DeepSWE tasks, repeated attempts averaged within task; mean success 0.461. go: 23 tasks, success estimate shrunk toward the overall mean 0.425. python: 27 tasks, success estimate shrunk toward the overall mean 0.477. rust: 5 tasks, success estimate shrunk toward the overall mean 0.437. typescript: 25 tasks, success estimate shrunk toward the overall mean 0.483. These are task-family priors from a different budget/configuration, not the probability for this request. Use visible demands to update them. Do not identify or recall a benchmark task or its published answer.\nHarness: mini-SWE-agent 2.0.0, Linux shell, repository inspection and edits, existing tests and self-written reproductions; one fresh attempt, high native reasoning effort, 150 model calls and USD 5 inference limit. Final hidden tests are unavailable to the solver. No retry or inherited patch from another model.",
"empirical": true
}
}
}
},
"sonnet_opus_validation_regret_0.02": {
"selection": {
"key": "cap_sonnet_opus_research_none_0",
"definition_id": "cap_sonnet_opus_research",
"classifier": "cap",
"pair": "sonnet_opus",
"variant": "research",
"adjustment": "none",
"alpha": 0.0,
"boundary": {
"base_threshold": 0.72,
"threshold_step": 0.0
},
"validation": {
"quality": 0.65,
"capable_quality": 0.6666666666666666,
"quality_difference": -0.016666666666666666,
"cost": 6.104486446666665,
"capable_cost": 6.126236435416667,
"efficient_fraction": 0.03333333333333333,
"savings": 0.003550301882614515,
"solver_cost": 6.104119028333332,
"judge_cost": 0.00036741833333333326
},
"calibration_metrics": {
"efficient": {
"brier_against_task_rate": 0.12371333333333336,
"log_loss": 0.6694236897841436,
"mean_prediction": 0.5456666666666667,
"observed_mean": 0.5416666666666666
}
},
"validation_ids": [
"adaptix-name-mapping-aliases",
"arcane-drift-detection-baselines",
"arktype-json-schema-refs-dependencies",
"claude-code-by-agents-recursive-delegation",
"effect-sse-httpapi-streaming",
"goreleaser-retry-publish-auditing",
"gql-incremental-graphql-delivery",
"helm-array-merge-strategies",
"helm-unified-manifest-stream",
"ink-grid-box-layout",
"kgateway-consistent-hash-policy",
"narwhals-rolling-window-suite",
"obsidian-linter-auto-table-of-contents",
"obsidian-linter-link-format-conversion",
"obsidian-linter-scoped-ignore-markers",
"opa-rego-rule-profiling",
"opa-template-string-reconstruction",
"prometheus-transactional-reload-status",
"prometheus-typed-label-sorting",
"returns-validated-error-accumulation",
"sql-formatter-bigquery-pipe-formatting",
"sqlfmt-create-table-ddl-formatting",
"tengo-callable-instance-isolation",
"tengo-destructuring-bindings",
"testem-bail-on-test-failure",
"testem-per-launcher-reports",
"textual-kitty-key-phases",
"textual-richlog-follow-state",
"vitest-duration-sharding",
"ytt-jsonpath-query-api"
],
"cost_complete_tasks": 30,
"allowed_loss": 0.02,
"fitted": {
"efficient": {
"adjustment": "none",
"categories": [],
"coefficients": [
0.0,
1.0
],
"alpha": 0.0
}
}
},
"complexity_router_config": {
"classifier_type": "capability",
"classifier_llm_config": {
"model": "router-judge",
"timeout_ms": 30000,
"reasoning_effort": "low"
},
"tiers": {
"SIMPLE": [
"claude-sonnet-5"
],
"REASONING": [
"claude-opus-5"
]
},
"adaptive": false,
"route_housekeeping_to_cheapest_tier": false,
"escalation_keywords": [],
"plan_mode_min_tier": null,
"enable_context_window_escalation": false,
"return_raw_model_name": true,
"max_tokens_from_tier_model": false,
"capability_classifier_config": {
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"max_output_tokens": 2048,
"base_threshold": 0.72,
"threshold_step": 0.0,
"card": {
"version": "roi-20260914-cap_sonnet_opus_research",
"text": "- SUP-1 [supported]: A bounded transformation has explicit inputs, outputs and accessible checks covering its material requirements; merely mentioning tests does not establish coverage.\n- SUP-2 [supported]: Relevant repository state can be inspected and a localized repair mechanism can be verified; repository size, familiar terminology and file count are not difficulty proxies.\n- SUP-3 [supported]: Numerical or typed behavior has explicit interfaces, edge cases and tolerances, and the solver can validate the core computation; a test harness does not solve an unknown algorithm.\n- SUP-4 [supported]: The failure mechanism and search boundary are supplied or readily inspectable, and the requested change fits the configured solver; multiple mechanical edits need not require stronger reasoning.\n- SUP-5 [supported]: Behavioral reproduction has an accessible executable reference or complete specification covering edge cases; hidden grading alone is not such a reference.\n- UNC-1 [uncertain]: Material semantic choices or interacting invariants remain unresolved, so several plausible implementations could differ in correctness.\n- UNC-2 [uncertain]: The task requires completeness across coupled paths, asynchronous state, schemas or compatibility surfaces without a clear search boundary or sufficient visible coverage.\n- LIM-1 [unsupported]: Exact information must be recovered from noisy visual or temporal media without an available reliable extraction mechanism.\n- LIM-2 [unsupported]: Success requires inaccessible reference behavior, undocumented state or unknown configuration. Such obstacles may defeat either solver; unsupported does not imply capable success.\n\nConfigured solver: Claude Sonnet 5 at high effort is a general coding solver. Evaluate its fit to the actual mechanism and scope. Clear instructions and runnable tests help, but do not establish complete coverage or guarantee success. Large repository size alone is not difficult scope. Public research does not establish a universal Sonnet advantage for a task family.\nHarness: mini-SWE-agent 2.0.0, Linux shell, repository inspection and edits, existing tests and self-written reproductions; one fresh attempt, high native reasoning effort, 150 model calls and USD 5 inference limit. Final hidden tests are unavailable to the solver. No retry or inherited patch from another model.",
"empirical": false
}
}
}
},
"sonnet_opus_validation_regret_0.05": {
"selection": {
"key": "cap_sonnet_opus_research_none_0",
"definition_id": "cap_sonnet_opus_research",
"classifier": "cap",
"pair": "sonnet_opus",
"variant": "research",
"adjustment": "none",
"alpha": 0.0,
"boundary": {
"base_threshold": 0.72,
"threshold_step": 0.0
},
"validation": {
"quality": 0.65,
"capable_quality": 0.6666666666666666,
"quality_difference": -0.016666666666666666,
"cost": 6.104486446666665,
"capable_cost": 6.126236435416667,
"efficient_fraction": 0.03333333333333333,
"savings": 0.003550301882614515,
"solver_cost": 6.104119028333332,
"judge_cost": 0.00036741833333333326
},
"calibration_metrics": {
"efficient": {
"brier_against_task_rate": 0.12371333333333336,
"log_loss": 0.6694236897841436,
"mean_prediction": 0.5456666666666667,
"observed_mean": 0.5416666666666666
}
},
"validation_ids": [
"adaptix-name-mapping-aliases",
"arcane-drift-detection-baselines",
"arktype-json-schema-refs-dependencies",
"claude-code-by-agents-recursive-delegation",
"effect-sse-httpapi-streaming",
"goreleaser-retry-publish-auditing",
"gql-incremental-graphql-delivery",
"helm-array-merge-strategies",
"helm-unified-manifest-stream",
"ink-grid-box-layout",
"kgateway-consistent-hash-policy",
"narwhals-rolling-window-suite",
"obsidian-linter-auto-table-of-contents",
"obsidian-linter-link-format-conversion",
"obsidian-linter-scoped-ignore-markers",
"opa-rego-rule-profiling",
"opa-template-string-reconstruction",
"prometheus-transactional-reload-status",
"prometheus-typed-label-sorting",
"returns-validated-error-accumulation",
"sql-formatter-bigquery-pipe-formatting",
"sqlfmt-create-table-ddl-formatting",
"tengo-callable-instance-isolation",
"tengo-destructuring-bindings",
"testem-bail-on-test-failure",
"testem-per-launcher-reports",
"textual-kitty-key-phases",
"textual-richlog-follow-state",
"vitest-duration-sharding",
"ytt-jsonpath-query-api"
],
"cost_complete_tasks": 30,
"allowed_loss": 0.05,
"fitted": {
"efficient": {
"adjustment": "none",
"categories": [],
"coefficients": [
0.0,
1.0
],
"alpha": 0.0
}
}
},
"complexity_router_config": {
"classifier_type": "capability",
"classifier_llm_config": {
"model": "router-judge",
"timeout_ms": 30000,
"reasoning_effort": "low"
},
"tiers": {
"SIMPLE": [
"claude-sonnet-5"
],
"REASONING": [
"claude-opus-5"
]
},
"adaptive": false,
"route_housekeeping_to_cheapest_tier": false,
"escalation_keywords": [],
"plan_mode_min_tier": null,
"enable_context_window_escalation": false,
"return_raw_model_name": true,
"max_tokens_from_tier_model": false,
"capability_classifier_config": {
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"max_output_tokens": 2048,
"base_threshold": 0.72,
"threshold_step": 0.0,
"card": {
"version": "roi-20260914-cap_sonnet_opus_research",
"text": "- SUP-1 [supported]: A bounded transformation has explicit inputs, outputs and accessible checks covering its material requirements; merely mentioning tests does not establish coverage.\n- SUP-2 [supported]: Relevant repository state can be inspected and a localized repair mechanism can be verified; repository size, familiar terminology and file count are not difficulty proxies.\n- SUP-3 [supported]: Numerical or typed behavior has explicit interfaces, edge cases and tolerances, and the solver can validate the core computation; a test harness does not solve an unknown algorithm.\n- SUP-4 [supported]: The failure mechanism and search boundary are supplied or readily inspectable, and the requested change fits the configured solver; multiple mechanical edits need not require stronger reasoning.\n- SUP-5 [supported]: Behavioral reproduction has an accessible executable reference or complete specification covering edge cases; hidden grading alone is not such a reference.\n- UNC-1 [uncertain]: Material semantic choices or interacting invariants remain unresolved, so several plausible implementations could differ in correctness.\n- UNC-2 [uncertain]: The task requires completeness across coupled paths, asynchronous state, schemas or compatibility surfaces without a clear search boundary or sufficient visible coverage.\n- LIM-1 [unsupported]: Exact information must be recovered from noisy visual or temporal media without an available reliable extraction mechanism.\n- LIM-2 [unsupported]: Success requires inaccessible reference behavior, undocumented state or unknown configuration. Such obstacles may defeat either solver; unsupported does not imply capable success.\n\nConfigured solver: Claude Sonnet 5 at high effort is a general coding solver. Evaluate its fit to the actual mechanism and scope. Clear instructions and runnable tests help, but do not establish complete coverage or guarantee success. Large repository size alone is not difficult scope. Public research does not establish a universal Sonnet advantage for a task family.\nHarness: mini-SWE-agent 2.0.0, Linux shell, repository inspection and edits, existing tests and self-written reproductions; one fresh attempt, high native reasoning effort, 150 model calls and USD 5 inference limit. Final hidden tests are unavailable to the solver. No retry or inherited patch from another model.",
"empirical": false
}
}
}
},
"luna_sol_validation_regret_0": {
"selection": {
"key": "cap_luna_sol_original_task_conditioned_100",
"definition_id": "cap_luna_sol_original",
"classifier": "cap",
"pair": "luna_sol",
"variant": "original",
"adjustment": "task_conditioned",
"alpha": 100.0,
"boundary": {
"base_threshold": 0.4565,
"threshold_step": 0.0
},
"validation": {
"quality": 0.6583333333333333,
"capable_quality": 0.6583333333333333,
"quality_difference": 0.0,
"cost": 2.7528806413333333,
"capable_cost": 3.619072949999999,
"efficient_fraction": 0.26666666666666666,
"savings": 0.2393409363761695,
"solver_cost": 2.7524850783333332,
"judge_cost": 0.000395563
},
"calibration_metrics": {
"efficient": {
"brier_against_task_rate": 0.10172270905056473,
"log_loss": 0.679881801698954,
"mean_prediction": 0.4539422735334361,
"observed_mean": 0.4166666666666667
}
},
"validation_ids": [
"adaptix-name-mapping-aliases",
"arcane-drift-detection-baselines",
"arktype-json-schema-refs-dependencies",
"claude-code-by-agents-recursive-delegation",
"effect-sse-httpapi-streaming",
"goreleaser-retry-publish-auditing",
"gql-incremental-graphql-delivery",
"helm-array-merge-strategies",
"helm-unified-manifest-stream",
"ink-grid-box-layout",
"kgateway-consistent-hash-policy",
"narwhals-rolling-window-suite",
"obsidian-linter-auto-table-of-contents",
"obsidian-linter-link-format-conversion",
"obsidian-linter-scoped-ignore-markers",
"opa-rego-rule-profiling",
"opa-template-string-reconstruction",
"prometheus-transactional-reload-status",
"prometheus-typed-label-sorting",
"returns-validated-error-accumulation",
"sql-formatter-bigquery-pipe-formatting",
"sqlfmt-create-table-ddl-formatting",
"tengo-callable-instance-isolation",
"tengo-destructuring-bindings",
"testem-bail-on-test-failure",
"testem-per-launcher-reports",
"textual-kitty-key-phases",
"textual-richlog-follow-state",
"vitest-duration-sharding",
"ytt-jsonpath-query-api"
],
"cost_complete_tasks": 30,
"allowed_loss": 0.0,
"fitted": {
"efficient": {
"adjustment": "task_conditioned",
"categories": [
"rule:SUP-1",
"rule:SUP-2",
"rule:SUP-3",
"rule:SUP-4",
"rule:UNC-1"
],
"coefficients": [
-0.21698217484765317,
0.05369117856329966,
0.005426200516531461,
0.016323471276039423,
-0.0055216963764245265,
-0.0035599589554351978,
-0.012683156629998152
],
"alpha": 100.0
}
}
},
"complexity_router_config": {
"classifier_type": "capability",
"classifier_llm_config": {
"model": "router-judge",
"timeout_ms": 30000,
"reasoning_effort": "low"
},
"tiers": {
"SIMPLE": [
"gpt-5.6-luna"
],
"REASONING": [
"gpt-5.6-sol"
]
},
"adaptive": false,
"route_housekeeping_to_cheapest_tier": false,
"escalation_keywords": [],
"plan_mode_min_tier": null,
"enable_context_window_escalation": false,
"return_raw_model_name": true,
"max_tokens_from_tier_model": false,
"capability_classifier_config": {
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"max_output_tokens": 2048,
"base_threshold": 0.4565,
"threshold_step": 0.0,
"calibration": {
"version": "roi-20260914-cap_luna_sol_original_task_conditioned_100",
"intercept": -0.21698217484765317,
"slope": 0.05369117856329966,
"rule_intercepts": [
{
"rule": "SUP-1",
"intercept": 0.005426200516531461
},
{
"rule": "SUP-2",
"intercept": 0.016323471276039423
},
{
"rule": "SUP-3",
"intercept": -0.0055216963764245265
},
{
"rule": "SUP-4",
"intercept": -0.0035599589554351978
},
{
"rule": "UNC-1",
"intercept": -0.012683156629998152
}
]
}
}
}
},
"luna_sol_validation_regret_0.02": {
"selection": {
"key": "cap_luna_sol_original_task_conditioned_1",
"definition_id": "cap_luna_sol_original",
"classifier": "cap",
"pair": "luna_sol",
"variant": "original",
"adjustment": "task_conditioned",
"alpha": 1.0,
"boundary": {
"base_threshold": 0.5,
"threshold_step": 0.0
},
"validation": {
"quality": 0.65,
"capable_quality": 0.6583333333333333,
"quality_difference": -0.008333333333333333,
"cost": 2.6019129496666666,
"capable_cost": 3.619072949999999,
"efficient_fraction": 0.3,
"savings": 0.2810554013102534,
"solver_cost": 2.6015173866666665,
"judge_cost": 0.000395563
},
"calibration_metrics": {
"efficient": {
"brier_against_task_rate": 0.09294567762840009,
"log_loss": 0.6617986190278198,
"mean_prediction": 0.4511365170033098,
"observed_mean": 0.4166666666666667
}
},
"validation_ids": [
"adaptix-name-mapping-aliases",
"arcane-drift-detection-baselines",
"arktype-json-schema-refs-dependencies",
"claude-code-by-agents-recursive-delegation",
"effect-sse-httpapi-streaming",
"goreleaser-retry-publish-auditing",
"gql-incremental-graphql-delivery",
"helm-array-merge-strategies",
"helm-unified-manifest-stream",
"ink-grid-box-layout",
"kgateway-consistent-hash-policy",
"narwhals-rolling-window-suite",
"obsidian-linter-auto-table-of-contents",
"obsidian-linter-link-format-conversion",
"obsidian-linter-scoped-ignore-markers",
"opa-rego-rule-profiling",
"opa-template-string-reconstruction",
"prometheus-transactional-reload-status",
"prometheus-typed-label-sorting",
"returns-validated-error-accumulation",
"sql-formatter-bigquery-pipe-formatting",
"sqlfmt-create-table-ddl-formatting",
"tengo-callable-instance-isolation",
"tengo-destructuring-bindings",
"testem-bail-on-test-failure",
"testem-per-launcher-reports",
"textual-kitty-key-phases",
"textual-richlog-follow-state",
"vitest-duration-sharding",
"ytt-jsonpath-query-api"
],
"cost_complete_tasks": 30,
"allowed_loss": 0.02,
"fitted": {
"efficient": {
"adjustment": "task_conditioned",
"categories": [
"rule:SUP-1",
"rule:SUP-2",
"rule:SUP-3",
"rule:SUP-4",
"rule:UNC-1"
],
"coefficients": [
-0.19689766507927828,
1e-06,
0.4396371165678798,
0.2359832544627836,
-0.28112556928354754,
-0.03160395112846778,
-0.36291552915127334
],
"alpha": 1.0
}
}
},
"complexity_router_config": {
"classifier_type": "capability",
"classifier_llm_config": {
"model": "router-judge",
"timeout_ms": 30000,
"reasoning_effort": "low"
},
"tiers": {
"SIMPLE": [
"gpt-5.6-luna"
],
"REASONING": [
"gpt-5.6-sol"
]
},
"adaptive": false,
"route_housekeeping_to_cheapest_tier": false,
"escalation_keywords": [],
"plan_mode_min_tier": null,
"enable_context_window_escalation": false,
"return_raw_model_name": true,
"max_tokens_from_tier_model": false,
"capability_classifier_config": {
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"max_output_tokens": 2048,
"base_threshold": 0.5,
"threshold_step": 0.0,
"calibration": {
"version": "roi-20260914-cap_luna_sol_original_task_conditioned_1",
"intercept": -0.19689766507927828,
"slope": 1e-06,
"rule_intercepts": [
{
"rule": "SUP-1",
"intercept": 0.4396371165678798
},
{
"rule": "SUP-2",
"intercept": 0.2359832544627836
},
{
"rule": "SUP-3",
"intercept": -0.28112556928354754
},
{
"rule": "SUP-4",
"intercept": -0.03160395112846778
},
{
"rule": "UNC-1",
"intercept": -0.36291552915127334
}
]
}
}
}
},
"luna_sol_validation_regret_0.05": {
"selection": {
"key": "cap_luna_sol_original_task_conditioned_10",
"definition_id": "cap_luna_sol_original",
"classifier": "cap",
"pair": "luna_sol",
"variant": "original",
"adjustment": "task_conditioned",
"alpha": 10.0,
"boundary": {
"base_threshold": 0.4544,
"threshold_step": 0.0
},
"validation": {
"quality": 0.6083333333333333,
"capable_quality": 0.6583333333333333,
"quality_difference": -0.05,
"cost": 2.4857286246666668,
"capable_cost": 3.619072949999999,
"efficient_fraction": 0.36666666666666664,
"savings": 0.3131587400948447,
"solver_cost": 2.4853330616666667,
"judge_cost": 0.000395563
},
"calibration_metrics": {
"efficient": {
"brier_against_task_rate": 0.09879300726739328,
"log_loss": 0.6739552397112222,
"mean_prediction": 0.4550966860883911,
"observed_mean": 0.4166666666666667
}
},
"validation_ids": [
"adaptix-name-mapping-aliases",
"arcane-drift-detection-baselines",
"arktype-json-schema-refs-dependencies",
"claude-code-by-agents-recursive-delegation",
"effect-sse-httpapi-streaming",
"goreleaser-retry-publish-auditing",
"gql-incremental-graphql-delivery",
"helm-array-merge-strategies",
"helm-unified-manifest-stream",
"ink-grid-box-layout",
"kgateway-consistent-hash-policy",
"narwhals-rolling-window-suite",
"obsidian-linter-auto-table-of-contents",
"obsidian-linter-link-format-conversion",
"obsidian-linter-scoped-ignore-markers",
"opa-rego-rule-profiling",
"opa-template-string-reconstruction",
"prometheus-transactional-reload-status",
"prometheus-typed-label-sorting",
"returns-validated-error-accumulation",
"sql-formatter-bigquery-pipe-formatting",
"sqlfmt-create-table-ddl-formatting",
"tengo-callable-instance-isolation",
"tengo-destructuring-bindings",
"testem-bail-on-test-failure",
"testem-per-launcher-reports",
"textual-kitty-key-phases",
"textual-richlog-follow-state",
"vitest-duration-sharding",
"ytt-jsonpath-query-api"
],
"cost_complete_tasks": 30,
"allowed_loss": 0.05,
"fitted": {
"efficient": {
"adjustment": "task_conditioned",
"categories": [
"rule:SUP-1",
"rule:SUP-2",
"rule:SUP-3",
"rule:SUP-4",
"rule:UNC-1"
],
"coefficients": [
-0.24175211398815877,
0.08835222230098246,
0.05304867006758595,
0.10570907649489589,
-0.04989630727579947,
-0.015835812492558256,
-0.09302599921746896
],
"alpha": 10.0
}
}
},
"complexity_router_config": {
"classifier_type": "capability",
"classifier_llm_config": {
"model": "router-judge",
"timeout_ms": 30000,
"reasoning_effort": "low"
},
"tiers": {
"SIMPLE": [
"gpt-5.6-luna"
],
"REASONING": [
"gpt-5.6-sol"
]
},
"adaptive": false,
"route_housekeeping_to_cheapest_tier": false,
"escalation_keywords": [],
"plan_mode_min_tier": null,
"enable_context_window_escalation": false,
"return_raw_model_name": true,
"max_tokens_from_tier_model": false,
"capability_classifier_config": {
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"max_output_tokens": 2048,
"base_threshold": 0.4544,
"threshold_step": 0.0,
"calibration": {
"version": "roi-20260914-cap_luna_sol_original_task_conditioned_10",
"intercept": -0.24175211398815877,
"slope": 0.08835222230098246,
"rule_intercepts": [
{
"rule": "SUP-1",
"intercept": 0.05304867006758595
},
{
"rule": "SUP-2",
"intercept": 0.10570907649489589
},
{
"rule": "SUP-3",
"intercept": -0.04989630727579947
},
{
"rule": "SUP-4",
"intercept": -0.015835812492558256
},
{
"rule": "UNC-1",
"intercept": -0.09302599921746896
}
]
}
}
}
}
}

402
cookbook/auto_router_roi_training/proxy.yaml Normal file
View file

@ -0,0 +1,402 @@
model_list:
- model_name: router-judge
litellm_params:
model: openai/openai/gpt-5.6-luna
api_base: os.environ/ROI_GATEWAY_BASE_URL
api_key: os.environ/ROI_GATEWAY_API_KEY
reasoning_effort: low
- model_name: claude-sonnet-5
litellm_params:
model: openai/anthropic/claude-sonnet-5
api_base: os.environ/ROI_GATEWAY_BASE_URL
api_key: os.environ/ROI_GATEWAY_API_KEY
reasoning_effort: high
- model_name: claude-opus-5
litellm_params:
model: openai/anthropic/claude-opus-5
api_base: os.environ/ROI_GATEWAY_BASE_URL
api_key: os.environ/ROI_GATEWAY_API_KEY
reasoning_effort: high
- model_name: gpt-5.6-luna
litellm_params:
model: openai/openai/gpt-5.6-luna
api_base: os.environ/ROI_GATEWAY_BASE_URL
api_key: os.environ/ROI_GATEWAY_API_KEY
reasoning_effort: high
- model_name: gpt-5.6-sol
litellm_params:
model: openai/openai/gpt-5.6-sol
api_base: os.environ/ROI_GATEWAY_BASE_URL
api_key: os.environ/ROI_GATEWAY_API_KEY
reasoning_effort: high
- model_name: roi-sonnet-opus-validation-regret-0
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
classifier_type: capability
classifier_llm_config:
model: router-judge
timeout_ms: 30000
reasoning_effort: low
tiers:
SIMPLE:
- claude-sonnet-5
REASONING:
- claude-opus-5
adaptive: false
route_housekeeping_to_cheapest_tier: false
escalation_keywords: []
plan_mode_min_tier: null
enable_context_window_escalation: false
return_raw_model_name: true
max_tokens_from_tier_model: false
capability_classifier_config:
efficient_tier: SIMPLE
capable_tier: REASONING
max_output_tokens: 2048
base_threshold: 0.5
threshold_step: 0.0
card:
version: roi-20260914-cap_sonnet_opus_trained_card
text: '- SUP-1 [supported]: A bounded transformation has explicit inputs,
outputs and accessible checks covering its material requirements; merely
mentioning tests does not establish coverage.
- SUP-2 [supported]: Relevant repository state can be inspected and a
localized repair mechanism can be verified; repository size, familiar
terminology and file count are not difficulty proxies.
- SUP-3 [supported]: Numerical or typed behavior has explicit interfaces,
edge cases and tolerances, and the solver can validate the core computation;
a test harness does not solve an unknown algorithm.
- SUP-4 [supported]: The failure mechanism and search boundary are supplied
or readily inspectable, and the requested change fits the configured solver;
multiple mechanical edits need not require stronger reasoning.
- SUP-5 [supported]: Behavioral reproduction has an accessible executable
reference or complete specification covering edge cases; hidden grading
alone is not such a reference.
- UNC-1 [uncertain]: Material semantic choices or interacting invariants
remain unresolved, so several plausible implementations could differ in
correctness.
- UNC-2 [uncertain]: The task requires completeness across coupled paths,
asynchronous state, schemas or compatibility surfaces without a clear
search boundary or sufficient visible coverage.
- LIM-1 [unsupported]: Exact information must be recovered from noisy
visual or temporal media without an available reliable extraction mechanism.
- LIM-2 [unsupported]: Success requires inaccessible reference behavior,
undocumented state or unknown configuration. Such obstacles may defeat
either solver; unsupported does not imply capable success.
Configured solver: Claude Sonnet 5 at high effort is a general coding
solver. Evaluate its fit to the actual mechanism and scope. Clear instructions
and runnable tests help, but do not establish complete coverage or guarantee
success. Large repository size alone is not difficult scope. Public research
does not establish a universal Sonnet advantage for a task family.
Training evidence: 83 distinct DeepSWE tasks, repeated attempts averaged
within task; mean success 0.461. go: 23 tasks, success estimate shrunk
toward the overall mean 0.425. python: 27 tasks, success estimate shrunk
toward the overall mean 0.477. rust: 5 tasks, success estimate shrunk
toward the overall mean 0.437. typescript: 25 tasks, success estimate
shrunk toward the overall mean 0.483. These are task-family priors from
a different budget/configuration, not the probability for this request.
Use visible demands to update them. Do not identify or recall a benchmark
task or its published answer.
Harness: mini-SWE-agent 2.0.0, Linux shell, repository inspection and
edits, existing tests and self-written reproductions; one fresh attempt,
high native reasoning effort, 150 model calls and USD 5 inference limit.
Final hidden tests are unavailable to the solver. No retry or inherited
patch from another model.'
empirical: true
- model_name: roi-sonnet-opus-validation-regret-0p02
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
classifier_type: capability
classifier_llm_config:
model: router-judge
timeout_ms: 30000
reasoning_effort: low
tiers:
SIMPLE:
- claude-sonnet-5
REASONING:
- claude-opus-5
adaptive: false
route_housekeeping_to_cheapest_tier: false
escalation_keywords: []
plan_mode_min_tier: null
enable_context_window_escalation: false
return_raw_model_name: true
max_tokens_from_tier_model: false
capability_classifier_config:
efficient_tier: SIMPLE
capable_tier: REASONING
max_output_tokens: 2048
base_threshold: 0.72
threshold_step: 0.0
card:
version: roi-20260914-cap_sonnet_opus_research
text: '- SUP-1 [supported]: A bounded transformation has explicit inputs,
outputs and accessible checks covering its material requirements; merely
mentioning tests does not establish coverage.
- SUP-2 [supported]: Relevant repository state can be inspected and a
localized repair mechanism can be verified; repository size, familiar
terminology and file count are not difficulty proxies.
- SUP-3 [supported]: Numerical or typed behavior has explicit interfaces,
edge cases and tolerances, and the solver can validate the core computation;
a test harness does not solve an unknown algorithm.
- SUP-4 [supported]: The failure mechanism and search boundary are supplied
or readily inspectable, and the requested change fits the configured solver;
multiple mechanical edits need not require stronger reasoning.
- SUP-5 [supported]: Behavioral reproduction has an accessible executable
reference or complete specification covering edge cases; hidden grading
alone is not such a reference.
- UNC-1 [uncertain]: Material semantic choices or interacting invariants
remain unresolved, so several plausible implementations could differ in
correctness.
- UNC-2 [uncertain]: The task requires completeness across coupled paths,
asynchronous state, schemas or compatibility surfaces without a clear
search boundary or sufficient visible coverage.
- LIM-1 [unsupported]: Exact information must be recovered from noisy
visual or temporal media without an available reliable extraction mechanism.
- LIM-2 [unsupported]: Success requires inaccessible reference behavior,
undocumented state or unknown configuration. Such obstacles may defeat
either solver; unsupported does not imply capable success.
Configured solver: Claude Sonnet 5 at high effort is a general coding
solver. Evaluate its fit to the actual mechanism and scope. Clear instructions
and runnable tests help, but do not establish complete coverage or guarantee
success. Large repository size alone is not difficult scope. Public research
does not establish a universal Sonnet advantage for a task family.
Harness: mini-SWE-agent 2.0.0, Linux shell, repository inspection and
edits, existing tests and self-written reproductions; one fresh attempt,
high native reasoning effort, 150 model calls and USD 5 inference limit.
Final hidden tests are unavailable to the solver. No retry or inherited
patch from another model.'
empirical: false
- model_name: roi-sonnet-opus-validation-regret-0p05
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
classifier_type: capability
classifier_llm_config:
model: router-judge
timeout_ms: 30000
reasoning_effort: low
tiers:
SIMPLE:
- claude-sonnet-5
REASONING:
- claude-opus-5
adaptive: false
route_housekeeping_to_cheapest_tier: false
escalation_keywords: []
plan_mode_min_tier: null
enable_context_window_escalation: false
return_raw_model_name: true
max_tokens_from_tier_model: false
capability_classifier_config:
efficient_tier: SIMPLE
capable_tier: REASONING
max_output_tokens: 2048
base_threshold: 0.72
threshold_step: 0.0
card:
version: roi-20260914-cap_sonnet_opus_research
text: '- SUP-1 [supported]: A bounded transformation has explicit inputs,
outputs and accessible checks covering its material requirements; merely
mentioning tests does not establish coverage.
- SUP-2 [supported]: Relevant repository state can be inspected and a
localized repair mechanism can be verified; repository size, familiar
terminology and file count are not difficulty proxies.
- SUP-3 [supported]: Numerical or typed behavior has explicit interfaces,
edge cases and tolerances, and the solver can validate the core computation;
a test harness does not solve an unknown algorithm.
- SUP-4 [supported]: The failure mechanism and search boundary are supplied
or readily inspectable, and the requested change fits the configured solver;
multiple mechanical edits need not require stronger reasoning.
- SUP-5 [supported]: Behavioral reproduction has an accessible executable
reference or complete specification covering edge cases; hidden grading
alone is not such a reference.
- UNC-1 [uncertain]: Material semantic choices or interacting invariants
remain unresolved, so several plausible implementations could differ in
correctness.
- UNC-2 [uncertain]: The task requires completeness across coupled paths,
asynchronous state, schemas or compatibility surfaces without a clear
search boundary or sufficient visible coverage.
- LIM-1 [unsupported]: Exact information must be recovered from noisy
visual or temporal media without an available reliable extraction mechanism.
- LIM-2 [unsupported]: Success requires inaccessible reference behavior,
undocumented state or unknown configuration. Such obstacles may defeat
either solver; unsupported does not imply capable success.
Configured solver: Claude Sonnet 5 at high effort is a general coding
solver. Evaluate its fit to the actual mechanism and scope. Clear instructions
and runnable tests help, but do not establish complete coverage or guarantee
success. Large repository size alone is not difficult scope. Public research
does not establish a universal Sonnet advantage for a task family.
Harness: mini-SWE-agent 2.0.0, Linux shell, repository inspection and
edits, existing tests and self-written reproductions; one fresh attempt,
high native reasoning effort, 150 model calls and USD 5 inference limit.
Final hidden tests are unavailable to the solver. No retry or inherited
patch from another model.'
empirical: false
- model_name: roi-luna-sol-validation-regret-0
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
classifier_type: capability
classifier_llm_config:
model: router-judge
timeout_ms: 30000
reasoning_effort: low
tiers:
SIMPLE:
- gpt-5.6-luna
REASONING:
- gpt-5.6-sol
adaptive: false
route_housekeeping_to_cheapest_tier: false
escalation_keywords: []
plan_mode_min_tier: null
enable_context_window_escalation: false
return_raw_model_name: true
max_tokens_from_tier_model: false
capability_classifier_config:
efficient_tier: SIMPLE
capable_tier: REASONING
max_output_tokens: 2048
base_threshold: 0.4565
threshold_step: 0.0
calibration:
version: roi-20260914-cap_luna_sol_original_task_conditioned_100
intercept: -0.21698217484765317
slope: 0.05369117856329966
rule_intercepts:
- rule: SUP-1
intercept: 0.005426200516531461
- rule: SUP-2
intercept: 0.016323471276039423
- rule: SUP-3
intercept: -0.0055216963764245265
- rule: SUP-4
intercept: -0.0035599589554351978
- rule: UNC-1
intercept: -0.012683156629998152
- model_name: roi-luna-sol-validation-regret-0p02
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
classifier_type: capability
classifier_llm_config:
model: router-judge
timeout_ms: 30000
reasoning_effort: low
tiers:
SIMPLE:
- gpt-5.6-luna
REASONING:
- gpt-5.6-sol
adaptive: false
route_housekeeping_to_cheapest_tier: false
escalation_keywords: []
plan_mode_min_tier: null
enable_context_window_escalation: false
return_raw_model_name: true
max_tokens_from_tier_model: false
capability_classifier_config:
efficient_tier: SIMPLE
capable_tier: REASONING
max_output_tokens: 2048
base_threshold: 0.5
threshold_step: 0.0
calibration:
version: roi-20260914-cap_luna_sol_original_task_conditioned_1
intercept: -0.19689766507927828
slope: 1.0e-06
rule_intercepts:
- rule: SUP-1
intercept: 0.4396371165678798
- rule: SUP-2
intercept: 0.2359832544627836
- rule: SUP-3
intercept: -0.28112556928354754
- rule: SUP-4
intercept: -0.03160395112846778
- rule: UNC-1
intercept: -0.36291552915127334
- model_name: roi-luna-sol-validation-regret-0p05
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
classifier_type: capability
classifier_llm_config:
model: router-judge
timeout_ms: 30000
reasoning_effort: low
tiers:
SIMPLE:
- gpt-5.6-luna
REASONING:
- gpt-5.6-sol
adaptive: false
route_housekeeping_to_cheapest_tier: false
escalation_keywords: []
plan_mode_min_tier: null
enable_context_window_escalation: false
return_raw_model_name: true
max_tokens_from_tier_model: false
capability_classifier_config:
efficient_tier: SIMPLE
capable_tier: REASONING
max_output_tokens: 2048
base_threshold: 0.4544
threshold_step: 0.0
calibration:
version: roi-20260914-cap_luna_sol_original_task_conditioned_10
intercept: -0.24175211398815877
slope: 0.08835222230098246
rule_intercepts:
- rule: SUP-1
intercept: 0.05304867006758595
- rule: SUP-2
intercept: 0.10570907649489589
- rule: SUP-3
intercept: -0.04989630727579947
- rule: SUP-4
intercept: -0.015835812492558256
- rule: UNC-1
intercept: -0.09302599921746896
general_settings:
master_key: os.environ/ROI_LOCAL_MASTER_KEY
litellm_settings:
turn_off_message_logging: true

678
cookbook/auto_router_roi_training/training_records.jsonl Normal file
View file

@ -0,0 +1,678 @@
{"task":"abs-module-cache-flags","repository":"abs-lang/abs","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.37},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.6259213875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.7221903125000004},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5457342000000001},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.41702475}},"judge_cost":0.0005955,"prompt_sha256":"16344adf6ad2b0df2cc23da9e3f6074ce299d3f7871c2666c12d0b7802259a3b","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"abs-module-cache-flags","repository":"abs-lang/abs","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.6259213875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.7221903125000004},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5457342000000001},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.41702475}},"judge_cost":0.0006111000000000001,"prompt_sha256":"16344adf6ad2b0df2cc23da9e3f6074ce299d3f7871c2666c12d0b7802259a3b","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"abs-module-cache-flags","repository":"abs-lang/abs","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.6259213875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.7221903125000004},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5457342000000001},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.41702475}},"judge_cost":0.00033873999999999997,"prompt_sha256":"16344adf6ad2b0df2cc23da9e3f6074ce299d3f7871c2666c12d0b7802259a3b","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"abs-module-cache-flags","repository":"abs-lang/abs","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.6259213875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.7221903125000004},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5457342000000001},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.41702475}},"judge_cost":0.00068975,"prompt_sha256":"16344adf6ad2b0df2cc23da9e3f6074ce299d3f7871c2666c12d0b7802259a3b","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"abs-module-cache-flags","repository":"abs-lang/abs","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.6259213875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.7221903125000004},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5457342000000001},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.41702475}},"judge_cost":0.0005810500000000001,"prompt_sha256":"16344adf6ad2b0df2cc23da9e3f6074ce299d3f7871c2666c12d0b7802259a3b","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"abs-module-cache-flags","repository":"abs-lang/abs","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.6259213875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.7221903125000004},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5457342000000001},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.41702475}},"judge_cost":0.00061465,"prompt_sha256":"16344adf6ad2b0df2cc23da9e3f6074ce299d3f7871c2666c12d0b7802259a3b","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"abs-stepped-slices","repository":"abs-lang/abs","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.67},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.5064902249999994},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9172978750000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.4015075},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.96924525}},"judge_cost":0.0008258,"prompt_sha256":"a6365026b9dc3f34dc63130f67dcf8b205660717a8e7846dc9eaf76bcad69ad2","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"abs-stepped-slices","repository":"abs-lang/abs","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.67},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.5064902249999994},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9172978750000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.4015075},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.96924525}},"judge_cost":0.0008258,"prompt_sha256":"a6365026b9dc3f34dc63130f67dcf8b205660717a8e7846dc9eaf76bcad69ad2","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"abs-stepped-slices","repository":"abs-lang/abs","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.5064902249999994},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9172978750000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.4015075},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.96924525}},"judge_cost":0.00044919999999999997,"prompt_sha256":"a6365026b9dc3f34dc63130f67dcf8b205660717a8e7846dc9eaf76bcad69ad2","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"abs-stepped-slices","repository":"abs-lang/abs","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.5064902249999994},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9172978750000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.4015075},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.96924525}},"judge_cost":0.00041967999999999997,"prompt_sha256":"a6365026b9dc3f34dc63130f67dcf8b205660717a8e7846dc9eaf76bcad69ad2","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"abs-stepped-slices","repository":"abs-lang/abs","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.5064902249999994},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9172978750000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.4015075},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.96924525}},"judge_cost":0.00043230000000000005,"prompt_sha256":"a6365026b9dc3f34dc63130f67dcf8b205660717a8e7846dc9eaf76bcad69ad2","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"actionlint-action-pinning-lint","repository":"rhysd/actionlint","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.224540337499997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.2481448125000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.61037355},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9377335}},"judge_cost":0.0005661500000000001,"prompt_sha256":"c0963dc084d61da5ff7b9235e33cbd47a18f8452834639af16e8cbb1b401e422","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"abs-stepped-slices","repository":"abs-lang/abs","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.5064902249999994},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9172978750000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.4015075},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.96924525}},"judge_cost":0.00043997999999999997,"prompt_sha256":"a6365026b9dc3f34dc63130f67dcf8b205660717a8e7846dc9eaf76bcad69ad2","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"actionlint-action-pinning-lint","repository":"rhysd/actionlint","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.224540337499997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.2481448125000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.61037355},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9377335}},"judge_cost":0.0005661500000000001,"prompt_sha256":"c0963dc084d61da5ff7b9235e33cbd47a18f8452834639af16e8cbb1b401e422","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"actionlint-action-pinning-lint","repository":"rhysd/actionlint","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.47},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.224540337499997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.2481448125000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.61037355},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9377335}},"judge_cost":0.00032514999999999996,"prompt_sha256":"c0963dc084d61da5ff7b9235e33cbd47a18f8452834639af16e8cbb1b401e422","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"actionlint-action-pinning-lint","repository":"rhysd/actionlint","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.224540337499997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.2481448125000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.61037355},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9377335}},"judge_cost":0.00032323,"prompt_sha256":"c0963dc084d61da5ff7b9235e33cbd47a18f8452834639af16e8cbb1b401e422","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"actionlint-action-pinning-lint","repository":"rhysd/actionlint","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.224540337499997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.2481448125000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.61037355},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9377335}},"judge_cost":0.00035384999999999995,"prompt_sha256":"c0963dc084d61da5ff7b9235e33cbd47a18f8452834639af16e8cbb1b401e422","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"actionlint-action-pinning-lint","repository":"rhysd/actionlint","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.224540337499997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.2481448125000005},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.61037355},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9377335}},"judge_cost":0.00030393,"prompt_sha256":"c0963dc084d61da5ff7b9235e33cbd47a18f8452834639af16e8cbb1b401e422","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"adaptix-name-mapping-aliases","repository":"reagento/adaptix","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.9246004875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.472915875000002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.9529598},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.20091225}},"judge_cost":0.00060275,"prompt_sha256":"cb03d3f123cc5aed6f726a85ccf18346cec51bfd8232e469e8f58f6dca546b5f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"adaptix-name-mapping-aliases","repository":"reagento/adaptix","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.55},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.9246004875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.472915875000002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.9529598},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.20091225}},"judge_cost":0.00031855,"prompt_sha256":"cb03d3f123cc5aed6f726a85ccf18346cec51bfd8232e469e8f58f6dca546b5f","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"adaptix-name-mapping-aliases","repository":"reagento/adaptix","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.9246004875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.472915875000002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.9529598},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.20091225}},"judge_cost":0.00029623,"prompt_sha256":"cb03d3f123cc5aed6f726a85ccf18346cec51bfd8232e469e8f58f6dca546b5f","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"adaptix-name-mapping-aliases","repository":"reagento/adaptix","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.9246004875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.472915875000002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.9529598},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.20091225}},"judge_cost":0.00039274,"prompt_sha256":"cb03d3f123cc5aed6f726a85ccf18346cec51bfd8232e469e8f58f6dca546b5f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"adaptix-name-mapping-aliases","repository":"reagento/adaptix","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.9246004875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.472915875000002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.9529598},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.20091225}},"judge_cost":0.00027285,"prompt_sha256":"cb03d3f123cc5aed6f726a85ccf18346cec51bfd8232e469e8f58f6dca546b5f","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"adaptix-name-mapping-aliases","repository":"reagento/adaptix","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.9246004875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.472915875000002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.9529598},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.20091225}},"judge_cost":0.00029853,"prompt_sha256":"cb03d3f123cc5aed6f726a85ccf18346cec51bfd8232e469e8f58f6dca546b5f","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"aiomonitor-task-snapshots-diff","repository":"aio-libs/aiomonitor","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.0100249875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.551020499999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5527869000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.2105815}},"judge_cost":0.0006363499999999999,"prompt_sha256":"280a8b860a1490a0a52beded6afd153a9214a0509d10001d1b5866d9c6c6c5ee","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"aiomonitor-task-snapshots-diff","repository":"aio-libs/aiomonitor","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.0100249875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.551020499999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5527869000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.2105815}},"judge_cost":0.00032023,"prompt_sha256":"280a8b860a1490a0a52beded6afd153a9214a0509d10001d1b5866d9c6c6c5ee","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"aiomonitor-task-snapshots-diff","repository":"aio-libs/aiomonitor","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.0100249875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.551020499999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5527869000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.2105815}},"judge_cost":0.00033535,"prompt_sha256":"280a8b860a1490a0a52beded6afd153a9214a0509d10001d1b5866d9c6c6c5ee","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"aiomonitor-task-snapshots-diff","repository":"aio-libs/aiomonitor","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.0100249875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.551020499999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5527869000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.2105815}},"judge_cost":0.00028605,"prompt_sha256":"280a8b860a1490a0a52beded6afd153a9214a0509d10001d1b5866d9c6c6c5ee","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"aiomonitor-task-snapshots-diff","repository":"aio-libs/aiomonitor","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.0100249875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.551020499999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5527869000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.2105815}},"judge_cost":0.0003805,"prompt_sha256":"280a8b860a1490a0a52beded6afd153a9214a0509d10001d1b5866d9c6c6c5ee","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"aiomonitor-task-snapshots-diff","repository":"aio-libs/aiomonitor","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.0100249875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.551020499999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5527869000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.2105815}},"judge_cost":0.00034413,"prompt_sha256":"280a8b860a1490a0a52beded6afd153a9214a0509d10001d1b5866d9c6c6c5ee","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"anko-default-function-arguments","repository":"mattn/anko","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.8860690625},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.9905546250000015},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.9174470499999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8586795}},"judge_cost":0.00064305,"prompt_sha256":"f2e31fb97ff68c979d8708148c3cb38cf2c8a89deefff1c162f98ec79bfce089","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"anko-default-function-arguments","repository":"mattn/anko","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.8860690625},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.9905546250000015},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.9174470499999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8586795}},"judge_cost":0.00026645,"prompt_sha256":"f2e31fb97ff68c979d8708148c3cb38cf2c8a89deefff1c162f98ec79bfce089","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"anko-default-function-arguments","repository":"mattn/anko","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.8860690625},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.9905546250000015},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.9174470499999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8586795}},"judge_cost":0.00020334000000000001,"prompt_sha256":"f2e31fb97ff68c979d8708148c3cb38cf2c8a89deefff1c162f98ec79bfce089","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"anko-default-function-arguments","repository":"mattn/anko","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.8860690625},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.9905546250000015},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.9174470499999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8586795}},"judge_cost":0.00037373,"prompt_sha256":"f2e31fb97ff68c979d8708148c3cb38cf2c8a89deefff1c162f98ec79bfce089","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"anko-default-function-arguments","repository":"mattn/anko","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.8860690625},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.9905546250000015},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.9174470499999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8586795}},"judge_cost":0.00033595,"prompt_sha256":"f2e31fb97ff68c979d8708148c3cb38cf2c8a89deefff1c162f98ec79bfce089","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"anko-default-function-arguments","repository":"mattn/anko","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.8860690625},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.9905546250000015},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.9174470499999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8586795}},"judge_cost":0.00028963000000000003,"prompt_sha256":"f2e31fb97ff68c979d8708148c3cb38cf2c8a89deefff1c162f98ec79bfce089","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"anko-typed-variable-bindings","repository":"mattn/anko","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.641391149999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9788003125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4833519},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.019167}},"judge_cost":0.0006451499999999999,"prompt_sha256":"e3ae8a1faf7c88cd878abf755534582272c6a23f9fb9395fdb2b907d648ac9f2","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"anko-typed-variable-bindings","repository":"mattn/anko","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.641391149999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9788003125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4833519},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.019167}},"judge_cost":0.00034822999999999994,"prompt_sha256":"e3ae8a1faf7c88cd878abf755534582272c6a23f9fb9395fdb2b907d648ac9f2","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"anko-typed-variable-bindings","repository":"mattn/anko","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.641391149999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9788003125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4833519},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.019167}},"judge_cost":0.00040054999999999995,"prompt_sha256":"e3ae8a1faf7c88cd878abf755534582272c6a23f9fb9395fdb2b907d648ac9f2","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"anko-typed-variable-bindings","repository":"mattn/anko","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.76},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.641391149999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9788003125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4833519},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.019167}},"judge_cost":0.00033354,"prompt_sha256":"e3ae8a1faf7c88cd878abf755534582272c6a23f9fb9395fdb2b907d648ac9f2","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"anko-typed-variable-bindings","repository":"mattn/anko","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.641391149999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9788003125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4833519},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.019167}},"judge_cost":0.00032845,"prompt_sha256":"e3ae8a1faf7c88cd878abf755534582272c6a23f9fb9395fdb2b907d648ac9f2","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"anko-typed-variable-bindings","repository":"mattn/anko","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.641391149999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9788003125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4833519},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.019167}},"judge_cost":0.00032533,"prompt_sha256":"e3ae8a1faf7c88cd878abf755534582272c6a23f9fb9395fdb2b907d648ac9f2","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"arcane-drift-detection-baselines","repository":"getarcaneapp/arcane","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.26411965},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.9051124375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.81623975},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.02974025}},"judge_cost":0.0005618999999999999,"prompt_sha256":"e998d0726284d38dd881c2f90fd57af7093d00f7954587469803db140fe70d61","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"arcane-drift-detection-baselines","repository":"getarcaneapp/arcane","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.26411965},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.9051124375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.81623975},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.02974025}},"judge_cost":0.0009217,"prompt_sha256":"e998d0726284d38dd881c2f90fd57af7093d00f7954587469803db140fe70d61","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"arcane-drift-detection-baselines","repository":"getarcaneapp/arcane","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.3},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.26411965},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.9051124375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.81623975},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.02974025}},"judge_cost":0.00053118,"prompt_sha256":"e998d0726284d38dd881c2f90fd57af7093d00f7954587469803db140fe70d61","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"arcane-drift-detection-baselines","repository":"getarcaneapp/arcane","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.26411965},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.9051124375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.81623975},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.02974025}},"judge_cost":0.000533,"prompt_sha256":"e998d0726284d38dd881c2f90fd57af7093d00f7954587469803db140fe70d61","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"arcane-drift-detection-baselines","repository":"getarcaneapp/arcane","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.26411965},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.9051124375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.81623975},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.02974025}},"judge_cost":0.00036631999999999995,"prompt_sha256":"e998d0726284d38dd881c2f90fd57af7093d00f7954587469803db140fe70d61","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"arcane-drift-detection-baselines","repository":"getarcaneapp/arcane","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.26411965},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.9051124375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.81623975},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.02974025}},"judge_cost":0.0006102799999999999,"prompt_sha256":"e998d0726284d38dd881c2f90fd57af7093d00f7954587469803db140fe70d61","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"arktype-json-schema-refs-dependencies","repository":"arktypeio/arktype","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.674665425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.017117875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.78719085},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.35641575}},"judge_cost":0.0006471999999999999,"prompt_sha256":"d14811f4ec11299720939be6006be922acd8be9ddbbf400d00c52853a9908608","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"arktype-json-schema-refs-dependencies","repository":"arktypeio/arktype","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.55},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.674665425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.017117875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.78719085},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.35641575}},"judge_cost":0.0003618,"prompt_sha256":"d14811f4ec11299720939be6006be922acd8be9ddbbf400d00c52853a9908608","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"arktype-json-schema-refs-dependencies","repository":"arktypeio/arktype","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.674665425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.017117875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.78719085},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.35641575}},"judge_cost":0.00036348,"prompt_sha256":"d14811f4ec11299720939be6006be922acd8be9ddbbf400d00c52853a9908608","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"arktype-json-schema-refs-dependencies","repository":"arktypeio/arktype","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.57},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.674665425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.017117875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.78719085},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.35641575}},"judge_cost":0.0003509,"prompt_sha256":"d14811f4ec11299720939be6006be922acd8be9ddbbf400d00c52853a9908608","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"arktype-json-schema-refs-dependencies","repository":"arktypeio/arktype","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.674665425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.017117875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.78719085},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.35641575}},"judge_cost":0.00035636,"prompt_sha256":"d14811f4ec11299720939be6006be922acd8be9ddbbf400d00c52853a9908608","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"arktype-json-schema-refs-dependencies","repository":"arktypeio/arktype","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.674665425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.017117875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.78719085},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.35641575}},"judge_cost":0.00036338,"prompt_sha256":"d14811f4ec11299720939be6006be922acd8be9ddbbf400d00c52853a9908608","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"awilix-async-container-initialization","repository":"jeffijoe/awilix","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.1913638625},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.6648180000000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.50705505},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.7524545}},"judge_cost":0.00066095,"prompt_sha256":"f0fbe06b7b2932815cd9f4c1011643755ea1ea18255702807b430111370e71d8","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"awilix-async-container-initialization","repository":"jeffijoe/awilix","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.1913638625},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.6648180000000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.50705505},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.7524545}},"judge_cost":0.00033115,"prompt_sha256":"f0fbe06b7b2932815cd9f4c1011643755ea1ea18255702807b430111370e71d8","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"awilix-async-container-initialization","repository":"jeffijoe/awilix","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.3},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.1913638625},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.6648180000000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.50705505},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.7524545}},"judge_cost":0.00034243,"prompt_sha256":"f0fbe06b7b2932815cd9f4c1011643755ea1ea18255702807b430111370e71d8","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"awilix-async-container-initialization","repository":"jeffijoe/awilix","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.1913638625},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.6648180000000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.50705505},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.7524545}},"judge_cost":0.00032842,"prompt_sha256":"f0fbe06b7b2932815cd9f4c1011643755ea1ea18255702807b430111370e71d8","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"awilix-async-container-initialization","repository":"jeffijoe/awilix","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.55},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.1913638625},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.6648180000000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.50705505},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.7524545}},"judge_cost":0.00040545,"prompt_sha256":"f0fbe06b7b2932815cd9f4c1011643755ea1ea18255702807b430111370e71d8","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"awilix-async-container-initialization","repository":"jeffijoe/awilix","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.1913638625},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.6648180000000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.50705505},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.7524545}},"judge_cost":0.00032313000000000003,"prompt_sha256":"f0fbe06b7b2932815cd9f4c1011643755ea1ea18255702807b430111370e71d8","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"bandit-incremental-cache-control","repository":"PyCQA/bandit","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.57},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.343033612499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.8282015000000005},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6697089},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.033694}},"judge_cost":0.0006085,"prompt_sha256":"e0e6dec01a63f9d9262642c4923c078e46ccde036112f5cd5c2d43d93f685229","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"bandit-incremental-cache-control","repository":"PyCQA/bandit","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.343033612499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.8282015000000005},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6697089},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.033694}},"judge_cost":0.0003243,"prompt_sha256":"e0e6dec01a63f9d9262642c4923c078e46ccde036112f5cd5c2d43d93f685229","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"bandit-incremental-cache-control","repository":"PyCQA/bandit","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.343033612499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.8282015000000005},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6697089},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.033694}},"judge_cost":0.00033678,"prompt_sha256":"e0e6dec01a63f9d9262642c4923c078e46ccde036112f5cd5c2d43d93f685229","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"bandit-incremental-cache-control","repository":"PyCQA/bandit","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.343033612499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.8282015000000005},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6697089},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.033694}},"judge_cost":0.0002834,"prompt_sha256":"e0e6dec01a63f9d9262642c4923c078e46ccde036112f5cd5c2d43d93f685229","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"bandit-incremental-cache-control","repository":"PyCQA/bandit","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.343033612499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.8282015000000005},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6697089},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.033694}},"judge_cost":0.00036368,"prompt_sha256":"e0e6dec01a63f9d9262642c4923c078e46ccde036112f5cd5c2d43d93f685229","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"bandit-incremental-cache-control","repository":"PyCQA/bandit","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.343033612499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.8282015000000005},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6697089},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.033694}},"judge_cost":0.00031628,"prompt_sha256":"e0e6dec01a63f9d9262642c4923c078e46ccde036112f5cd5c2d43d93f685229","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"bandit-interprocedural-taint-checks","repository":"PyCQA/bandit","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.121889524999999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.4466654375},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.5201549},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.04376925}},"judge_cost":0.00028875,"prompt_sha256":"b61f1cc14f629fb1e730907d1b0c94794b4711638c52dead6770a6b1056f052c","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"bandit-interprocedural-taint-checks","repository":"PyCQA/bandit","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.121889524999999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.4466654375},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.5201549},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.04376925}},"judge_cost":0.00061975,"prompt_sha256":"b61f1cc14f629fb1e730907d1b0c94794b4711638c52dead6770a6b1056f052c","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"bandit-interprocedural-taint-checks","repository":"PyCQA/bandit","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.121889524999999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.4466654375},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.5201549},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.04376925}},"judge_cost":0.00031563,"prompt_sha256":"b61f1cc14f629fb1e730907d1b0c94794b4711638c52dead6770a6b1056f052c","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"bandit-interprocedural-taint-checks","repository":"PyCQA/bandit","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.121889524999999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.4466654375},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.5201549},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.04376925}},"judge_cost":0.00061975,"prompt_sha256":"b61f1cc14f629fb1e730907d1b0c94794b4711638c52dead6770a6b1056f052c","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"bandit-interprocedural-taint-checks","repository":"PyCQA/bandit","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.121889524999999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.4466654375},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.5201549},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.04376925}},"judge_cost":0.00031145,"prompt_sha256":"b61f1cc14f629fb1e730907d1b0c94794b4711638c52dead6770a6b1056f052c","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"bandit-interprocedural-taint-checks","repository":"PyCQA/bandit","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.121889524999999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.4466654375},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.5201549},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.04376925}},"judge_cost":0.00031553,"prompt_sha256":"b61f1cc14f629fb1e730907d1b0c94794b4711638c52dead6770a6b1056f052c","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"bandit-structured-nosec-directives","repository":"PyCQA/bandit","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.857585125},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.7395035625000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6234286499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.12559675}},"judge_cost":0.0006787,"prompt_sha256":"25ad36a50a1b13378147f91870f5545ba2b66b4cbaa4cf5a776a73d34b746c2f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"bandit-structured-nosec-directives","repository":"PyCQA/bandit","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.857585125},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.7395035625000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6234286499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.12559675}},"judge_cost":0.0006787,"prompt_sha256":"25ad36a50a1b13378147f91870f5545ba2b66b4cbaa4cf5a776a73d34b746c2f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"bandit-structured-nosec-directives","repository":"PyCQA/bandit","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.857585125},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.7395035625000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6234286499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.12559675}},"judge_cost":0.0003657,"prompt_sha256":"25ad36a50a1b13378147f91870f5545ba2b66b4cbaa4cf5a776a73d34b746c2f","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"bandit-structured-nosec-directives","repository":"PyCQA/bandit","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.857585125},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.7395035625000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6234286499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.12559675}},"judge_cost":0.00035298,"prompt_sha256":"25ad36a50a1b13378147f91870f5545ba2b66b4cbaa4cf5a776a73d34b746c2f","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"bandit-structured-nosec-directives","repository":"PyCQA/bandit","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.857585125},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.7395035625000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6234286499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.12559675}},"judge_cost":0.000368,"prompt_sha256":"25ad36a50a1b13378147f91870f5545ba2b66b4cbaa4cf5a776a73d34b746c2f","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"bandit-structured-nosec-directives","repository":"PyCQA/bandit","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.857585125},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.7395035625000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6234286499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.12559675}},"judge_cost":0.00040688,"prompt_sha256":"25ad36a50a1b13378147f91870f5545ba2b66b4cbaa4cf5a776a73d34b746c2f","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"boa-hierarchical-evaluation-cancellation","repository":"boa-dev/boa","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.614165975000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.0350156875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.2153201},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.04543025}},"judge_cost":0.00044585,"prompt_sha256":"043fb4500adc749f00a6c04c190767354c8df420e9c59826703c51be82d7c875","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"boa-hierarchical-evaluation-cancellation","repository":"boa-dev/boa","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.614165975000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.0350156875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.2153201},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.04543025}},"judge_cost":0.0008248499999999999,"prompt_sha256":"043fb4500adc749f00a6c04c190767354c8df420e9c59826703c51be82d7c875","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"boa-hierarchical-evaluation-cancellation","repository":"boa-dev/boa","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.614165975000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.0350156875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.2153201},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.04543025}},"judge_cost":0.00042833,"prompt_sha256":"043fb4500adc749f00a6c04c190767354c8df420e9c59826703c51be82d7c875","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"boa-hierarchical-evaluation-cancellation","repository":"boa-dev/boa","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.614165975000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.0350156875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.2153201},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.04543025}},"judge_cost":0.00030822,"prompt_sha256":"043fb4500adc749f00a6c04c190767354c8df420e9c59826703c51be82d7c875","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"boa-hierarchical-evaluation-cancellation","repository":"boa-dev/boa","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.614165975000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.0350156875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.2153201},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.04543025}},"judge_cost":0.00044574999999999997,"prompt_sha256":"043fb4500adc749f00a6c04c190767354c8df420e9c59826703c51be82d7c875","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"boa-hierarchical-evaluation-cancellation","repository":"boa-dev/boa","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.19},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.614165975000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.0350156875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.2153201},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.04543025}},"judge_cost":0.00042703,"prompt_sha256":"043fb4500adc749f00a6c04c190767354c8df420e9c59826703c51be82d7c875","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"cattrs-partial-structuring-recovery","repository":"python-attrs/cattrs","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.29},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.869498975},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5660280000000006},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6950657},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4852485}},"judge_cost":0.00026137999999999997,"prompt_sha256":"3c61df92854a45dbbbdc745cb6b83ab652f805c42aadbc6db45a910a19084834","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"cattrs-partial-structuring-recovery","repository":"python-attrs/cattrs","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.7},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.869498975},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5660280000000006},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6950657},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4852485}},"judge_cost":0.0007347,"prompt_sha256":"3c61df92854a45dbbbdc745cb6b83ab652f805c42aadbc6db45a910a19084834","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"cattrs-partial-structuring-recovery","repository":"python-attrs/cattrs","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.869498975},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5660280000000006},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6950657},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4852485}},"judge_cost":0.0003497,"prompt_sha256":"3c61df92854a45dbbbdc745cb6b83ab652f805c42aadbc6db45a910a19084834","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"cattrs-partial-structuring-recovery","repository":"python-attrs/cattrs","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.869498975},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5660280000000006},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6950657},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4852485}},"judge_cost":0.00043967999999999996,"prompt_sha256":"3c61df92854a45dbbbdc745cb6b83ab652f805c42aadbc6db45a910a19084834","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"cattrs-partial-structuring-recovery","repository":"python-attrs/cattrs","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.35},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.869498975},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5660280000000006},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6950657},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4852485}},"judge_cost":0.00027568,"prompt_sha256":"3c61df92854a45dbbbdc745cb6b83ab652f805c42aadbc6db45a910a19084834","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"cattrs-partial-structuring-recovery","repository":"python-attrs/cattrs","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.48},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.869498975},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5660280000000006},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6950657},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4852485}},"judge_cost":0.0003472,"prompt_sha256":"3c61df92854a45dbbbdc745cb6b83ab652f805c42aadbc6db45a910a19084834","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"clack-async-autocomplete-options","repository":"bombshell-dev/clack","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.863385012499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1666555625000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.74072235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.0162765}},"judge_cost":0.0007348999999999999,"prompt_sha256":"af638e32e4f9b011f829aca1a4f1ad8debe84df75fb42892c11d0b9f4e56560a","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"clack-async-autocomplete-options","repository":"bombshell-dev/clack","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.22},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.863385012499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1666555625000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.74072235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.0162765}},"judge_cost":0.00036118,"prompt_sha256":"af638e32e4f9b011f829aca1a4f1ad8debe84df75fb42892c11d0b9f4e56560a","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"clack-async-autocomplete-options","repository":"bombshell-dev/clack","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.863385012499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1666555625000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.74072235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.0162765}},"judge_cost":0.00038710000000000003,"prompt_sha256":"af638e32e4f9b011f829aca1a4f1ad8debe84df75fb42892c11d0b9f4e56560a","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"clack-async-autocomplete-options","repository":"bombshell-dev/clack","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.863385012499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1666555625000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.74072235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.0162765}},"judge_cost":0.000381,"prompt_sha256":"af638e32e4f9b011f829aca1a4f1ad8debe84df75fb42892c11d0b9f4e56560a","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"clack-async-autocomplete-options","repository":"bombshell-dev/clack","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.863385012499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1666555625000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.74072235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.0162765}},"judge_cost":0.00042352,"prompt_sha256":"af638e32e4f9b011f829aca1a4f1ad8debe84df75fb42892c11d0b9f4e56560a","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"clack-async-autocomplete-options","repository":"bombshell-dev/clack","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.863385012499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1666555625000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.74072235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.0162765}},"judge_cost":0.00037908,"prompt_sha256":"af638e32e4f9b011f829aca1a4f1ad8debe84df75fb42892c11d0b9f4e56560a","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"claude-code-by-agents-recursive-delegation","repository":"baryhuang/claude-code-by-agents","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.2496796},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9132018125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5435619},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.107036}},"judge_cost":0.0005053,"prompt_sha256":"e40abd0bd02e1a4bf12ea4f879d6041e460b992122002643dffa5afd8f83926f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"claude-code-by-agents-recursive-delegation","repository":"baryhuang/claude-code-by-agents","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.2496796},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9132018125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5435619},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.107036}},"judge_cost":0.0005053,"prompt_sha256":"e40abd0bd02e1a4bf12ea4f879d6041e460b992122002643dffa5afd8f83926f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"claude-code-by-agents-recursive-delegation","repository":"baryhuang/claude-code-by-agents","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.2496796},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9132018125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5435619},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.107036}},"judge_cost":0.0002835,"prompt_sha256":"e40abd0bd02e1a4bf12ea4f879d6041e460b992122002643dffa5afd8f83926f","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"claude-code-by-agents-recursive-delegation","repository":"baryhuang/claude-code-by-agents","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.2496796},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9132018125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5435619},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.107036}},"judge_cost":0.00029838,"prompt_sha256":"e40abd0bd02e1a4bf12ea4f879d6041e460b992122002643dffa5afd8f83926f","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"claude-code-by-agents-recursive-delegation","repository":"baryhuang/claude-code-by-agents","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.65},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.2496796},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9132018125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5435619},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.107036}},"judge_cost":0.0003074,"prompt_sha256":"e40abd0bd02e1a4bf12ea4f879d6041e460b992122002643dffa5afd8f83926f","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"claude-code-by-agents-recursive-delegation","repository":"baryhuang/claude-code-by-agents","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.47},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.2496796},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9132018125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5435619},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.107036}},"judge_cost":0.00031267999999999997,"prompt_sha256":"e40abd0bd02e1a4bf12ea4f879d6041e460b992122002643dffa5afd8f83926f","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"cliffy-config-file-parsing","repository":"c4spar/cliffy","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.5184172375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.407408125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.64849275},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1712765}},"judge_cost":0.0003049,"prompt_sha256":"927c4093f17b5bc967407b2b3bb0dc12c83ed4453c9817d272bf2f0b82ffbcf3","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"cliffy-config-file-parsing","repository":"c4spar/cliffy","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.5184172375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.407408125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.64849275},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1712765}},"judge_cost":0.00032337999999999996,"prompt_sha256":"927c4093f17b5bc967407b2b3bb0dc12c83ed4453c9817d272bf2f0b82ffbcf3","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"cliffy-config-file-parsing","repository":"c4spar/cliffy","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.57},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.5184172375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.407408125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.64849275},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1712765}},"judge_cost":0.0008303,"prompt_sha256":"927c4093f17b5bc967407b2b3bb0dc12c83ed4453c9817d272bf2f0b82ffbcf3","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"cliffy-config-file-parsing","repository":"c4spar/cliffy","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.5184172375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.407408125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.64849275},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1712765}},"judge_cost":0.00030616000000000004,"prompt_sha256":"927c4093f17b5bc967407b2b3bb0dc12c83ed4453c9817d272bf2f0b82ffbcf3","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"cliffy-config-file-parsing","repository":"c4spar/cliffy","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.5184172375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.407408125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.64849275},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1712765}},"judge_cost":0.0003072,"prompt_sha256":"927c4093f17b5bc967407b2b3bb0dc12c83ed4453c9817d272bf2f0b82ffbcf3","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"cliffy-config-file-parsing","repository":"c4spar/cliffy","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.5184172375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.407408125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.64849275},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1712765}},"judge_cost":0.00030528,"prompt_sha256":"927c4093f17b5bc967407b2b3bb0dc12c83ed4453c9817d272bf2f0b82ffbcf3","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"csstree-shorthand-expansion-compression","repository":"csstree/csstree","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.967813425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.433430625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.533532},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.824266}},"judge_cost":0.00038375,"prompt_sha256":"6414efad50f6524384e152ed5274d4467e810ba1bd3cc08b82e28c7fbe6b71e6","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"csstree-shorthand-expansion-compression","repository":"csstree/csstree","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.967813425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.433430625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.533532},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.824266}},"judge_cost":0.00042383,"prompt_sha256":"6414efad50f6524384e152ed5274d4467e810ba1bd3cc08b82e28c7fbe6b71e6","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"csstree-shorthand-expansion-compression","repository":"csstree/csstree","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.48},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.967813425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.433430625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.533532},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.824266}},"judge_cost":0.00084315,"prompt_sha256":"6414efad50f6524384e152ed5274d4467e810ba1bd3cc08b82e28c7fbe6b71e6","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"csstree-shorthand-expansion-compression","repository":"csstree/csstree","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.3},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.967813425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.433430625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.533532},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.824266}},"judge_cost":0.00038484999999999995,"prompt_sha256":"6414efad50f6524384e152ed5274d4467e810ba1bd3cc08b82e28c7fbe6b71e6","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"csstree-shorthand-expansion-compression","repository":"csstree/csstree","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.967813425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.433430625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.533532},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.824266}},"judge_cost":0.00037812999999999996,"prompt_sha256":"6414efad50f6524384e152ed5274d4467e810ba1bd3cc08b82e28c7fbe6b71e6","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"csstree-shorthand-expansion-compression","repository":"csstree/csstree","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.44},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.967813425},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.433430625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.533532},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.824266}},"judge_cost":0.0005202599999999999,"prompt_sha256":"6414efad50f6524384e152ed5274d4467e810ba1bd3cc08b82e28c7fbe6b71e6","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dasel-html-document-format","repository":"TomWright/dasel","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9698815},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.78713675},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.453849},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.086567}},"judge_cost":0.0006173999999999999,"prompt_sha256":"d826c584d471a1ea75bfc520a6c44bf75f726abf2516a7fe62a258efd6ce189a","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dasel-html-document-format","repository":"TomWright/dasel","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9698815},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.78713675},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.453849},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.086567}},"judge_cost":0.0006173999999999999,"prompt_sha256":"d826c584d471a1ea75bfc520a6c44bf75f726abf2516a7fe62a258efd6ce189a","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dasel-html-document-format","repository":"TomWright/dasel","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9698815},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.78713675},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.453849},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.086567}},"judge_cost":0.00029408,"prompt_sha256":"d826c584d471a1ea75bfc520a6c44bf75f726abf2516a7fe62a258efd6ce189a","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"dasel-html-document-format","repository":"TomWright/dasel","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9698815},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.78713675},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.453849},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.086567}},"judge_cost":0.00031039999999999996,"prompt_sha256":"d826c584d471a1ea75bfc520a6c44bf75f726abf2516a7fe62a258efd6ce189a","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"dasel-html-document-format","repository":"TomWright/dasel","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9698815},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.78713675},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.453849},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.086567}},"judge_cost":0.00031150000000000004,"prompt_sha256":"d826c584d471a1ea75bfc520a6c44bf75f726abf2516a7fe62a258efd6ce189a","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"dasel-html-document-format","repository":"TomWright/dasel","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9698815},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.78713675},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.453849},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.086567}},"judge_cost":0.00030358,"prompt_sha256":"d826c584d471a1ea75bfc520a6c44bf75f726abf2516a7fe62a258efd6ce189a","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"dateutil-rfc5545-timezone-interop","repository":"dateutil/dateutil","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.702252424999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.426980875000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6172397000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.16898125}},"judge_cost":0.00068545,"prompt_sha256":"164da16b5ce058d2ee58786450b6b0d667f0826bf241e84e7218586ddb2a9cdb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dateutil-rfc5545-timezone-interop","repository":"dateutil/dateutil","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.702252424999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.426980875000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6172397000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.16898125}},"judge_cost":0.00068545,"prompt_sha256":"164da16b5ce058d2ee58786450b6b0d667f0826bf241e84e7218586ddb2a9cdb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dateutil-rfc5545-timezone-interop","repository":"dateutil/dateutil","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.702252424999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.426980875000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6172397000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.16898125}},"judge_cost":0.00045525000000000003,"prompt_sha256":"164da16b5ce058d2ee58786450b6b0d667f0826bf241e84e7218586ddb2a9cdb","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"dateutil-rfc5545-timezone-interop","repository":"dateutil/dateutil","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.702252424999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.426980875000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6172397000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.16898125}},"judge_cost":0.00043172999999999996,"prompt_sha256":"164da16b5ce058d2ee58786450b6b0d667f0826bf241e84e7218586ddb2a9cdb","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"dateutil-rfc5545-timezone-interop","repository":"dateutil/dateutil","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.702252424999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.426980875000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6172397000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.16898125}},"judge_cost":0.00042155000000000003,"prompt_sha256":"164da16b5ce058d2ee58786450b6b0d667f0826bf241e84e7218586ddb2a9cdb","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"dateutil-rfc5545-timezone-interop","repository":"dateutil/dateutil","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.702252424999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.426980875000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6172397000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.16898125}},"judge_cost":0.00043043,"prompt_sha256":"164da16b5ce058d2ee58786450b6b0d667f0826bf241e84e7218586ddb2a9cdb","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"drizzle-orm-window-function-builders","repository":"drizzle-team/drizzle-orm","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.627429175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8505669374999996},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7048063},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1856935}},"judge_cost":0.00030179999999999996,"prompt_sha256":"c565060a34853526dd3cbec04c327b068f894bb3bc4098b9839dc146e43006d4","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"drizzle-orm-window-function-builders","repository":"drizzle-team/drizzle-orm","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.627429175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8505669374999996},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7048063},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1856935}},"judge_cost":0.00032028,"prompt_sha256":"c565060a34853526dd3cbec04c327b068f894bb3bc4098b9839dc146e43006d4","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"drizzle-orm-window-function-builders","repository":"drizzle-team/drizzle-orm","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.627429175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8505669374999996},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7048063},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1856935}},"judge_cost":0.000682,"prompt_sha256":"c565060a34853526dd3cbec04c327b068f894bb3bc4098b9839dc146e43006d4","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"drizzle-orm-window-function-builders","repository":"drizzle-team/drizzle-orm","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.627429175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8505669374999996},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7048063},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1856935}},"judge_cost":0.0003389,"prompt_sha256":"c565060a34853526dd3cbec04c327b068f894bb3bc4098b9839dc146e43006d4","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"drizzle-orm-window-function-builders","repository":"drizzle-team/drizzle-orm","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.627429175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8505669374999996},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7048063},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1856935}},"judge_cost":0.00032498,"prompt_sha256":"c565060a34853526dd3cbec04c327b068f894bb3bc4098b9839dc146e43006d4","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"drizzle-orm-window-function-builders","repository":"drizzle-team/drizzle-orm","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.627429175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.8505669374999996},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7048063},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1856935}},"judge_cost":0.00041251999999999993,"prompt_sha256":"c565060a34853526dd3cbec04c327b068f894bb3bc4098b9839dc146e43006d4","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dynamodb-toolbox-conditional-attribute-requirements","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":16.1276764875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.4202104375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.7856090500000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.710677}},"judge_cost":0.00027992999999999996,"prompt_sha256":"e4fa6a24fa49fd1fe577073c7d9f2b93abb8ed4dcd9d86f22f2a01baf893ea81","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"dynamodb-toolbox-conditional-attribute-requirements","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.4},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":16.1276764875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.4202104375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.7856090500000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.710677}},"judge_cost":0.00033105,"prompt_sha256":"e4fa6a24fa49fd1fe577073c7d9f2b93abb8ed4dcd9d86f22f2a01baf893ea81","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"dynamodb-toolbox-conditional-attribute-requirements","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.4},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":16.1276764875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.4202104375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.7856090500000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.710677}},"judge_cost":0.00069925,"prompt_sha256":"e4fa6a24fa49fd1fe577073c7d9f2b93abb8ed4dcd9d86f22f2a01baf893ea81","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dynamodb-toolbox-conditional-attribute-requirements","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":16.1276764875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.4202104375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.7856090500000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.710677}},"judge_cost":0.00033086,"prompt_sha256":"e4fa6a24fa49fd1fe577073c7d9f2b93abb8ed4dcd9d86f22f2a01baf893ea81","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dynamodb-toolbox-conditional-attribute-requirements","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":16.1276764875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.4202104375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.7856090500000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.710677}},"judge_cost":0.00027695,"prompt_sha256":"e4fa6a24fa49fd1fe577073c7d9f2b93abb8ed4dcd9d86f22f2a01baf893ea81","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"dynamodb-toolbox-conditional-attribute-requirements","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":16.1276764875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":12.4202104375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.7856090500000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.710677}},"judge_cost":0.00028463,"prompt_sha256":"e4fa6a24fa49fd1fe577073c7d9f2b93abb8ed4dcd9d86f22f2a01baf893ea81","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"dynamodb-toolbox-lazy-recursive-schemas","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":26.929387162499985},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":20.663338999999997},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.73068325},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.64882675}},"judge_cost":0.00061075,"prompt_sha256":"56a694583edadfe3ae777e11b109a3392368afb843000a946f9635caac781641","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dynamodb-toolbox-lazy-recursive-schemas","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":26.929387162499985},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":20.663338999999997},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.73068325},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.64882675}},"judge_cost":0.00031935,"prompt_sha256":"56a694583edadfe3ae777e11b109a3392368afb843000a946f9635caac781641","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"dynamodb-toolbox-lazy-recursive-schemas","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":26.929387162499985},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":20.663338999999997},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.73068325},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.64882675}},"judge_cost":0.00029463,"prompt_sha256":"56a694583edadfe3ae777e11b109a3392368afb843000a946f9635caac781641","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"dynamodb-toolbox-lazy-recursive-schemas","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.45},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":26.929387162499985},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":20.663338999999997},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.73068325},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.64882675}},"judge_cost":0.00027245000000000004,"prompt_sha256":"56a694583edadfe3ae777e11b109a3392368afb843000a946f9635caac781641","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"dynamodb-toolbox-lazy-recursive-schemas","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":26.929387162499985},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":20.663338999999997},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.73068325},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.64882675}},"judge_cost":0.00030506,"prompt_sha256":"56a694583edadfe3ae777e11b109a3392368afb843000a946f9635caac781641","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"dynamodb-toolbox-lazy-recursive-schemas","repository":"dynamodb-toolbox/dynamodb-toolbox","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":26.929387162499985},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":20.663338999999997},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.73068325},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.64882675}},"judge_cost":0.00027773,"prompt_sha256":"56a694583edadfe3ae777e11b109a3392368afb843000a946f9635caac781641","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"effect-sse-httpapi-streaming","repository":"Effect-TS/effect","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.54},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":15.4918653},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.787366874999998},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.8410906},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.7687785}},"judge_cost":0.0006991,"prompt_sha256":"f227c4a99ad3ec2d6859e98b8fb0ae9a9a5b7930335ff197de7e4ac420261509","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"effect-sse-httpapi-streaming","repository":"Effect-TS/effect","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":15.4918653},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.787366874999998},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.8410906},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.7687785}},"judge_cost":0.00033858,"prompt_sha256":"f227c4a99ad3ec2d6859e98b8fb0ae9a9a5b7930335ff197de7e4ac420261509","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"effect-sse-httpapi-streaming","repository":"Effect-TS/effect","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.48},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":15.4918653},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.787366874999998},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.8410906},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.7687785}},"judge_cost":0.0003765,"prompt_sha256":"f227c4a99ad3ec2d6859e98b8fb0ae9a9a5b7930335ff197de7e4ac420261509","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"effect-sse-httpapi-streaming","repository":"Effect-TS/effect","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":15.4918653},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.787366874999998},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.8410906},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.7687785}},"judge_cost":0.00028976,"prompt_sha256":"f227c4a99ad3ec2d6859e98b8fb0ae9a9a5b7930335ff197de7e4ac420261509","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"effect-sse-httpapi-streaming","repository":"Effect-TS/effect","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":15.4918653},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.787366874999998},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.8410906},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.7687785}},"judge_cost":0.0003536,"prompt_sha256":"f227c4a99ad3ec2d6859e98b8fb0ae9a9a5b7930335ff197de7e4ac420261509","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"effect-sse-httpapi-streaming","repository":"Effect-TS/effect","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":15.4918653},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.787366874999998},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.8410906},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.7687785}},"judge_cost":0.00031808,"prompt_sha256":"f227c4a99ad3ec2d6859e98b8fb0ae9a9a5b7930335ff197de7e4ac420261509","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"eicrud-keyset-pagination-cursor","repository":"eicrud/eicrud","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.7034944125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.3678230625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5771226500000001},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.5703915}},"judge_cost":0.0006081999999999999,"prompt_sha256":"790fa1385f54561dd5ab3b61e86d5cfc25579d67ccc621cd59aa9e09a233ed87","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"eicrud-keyset-pagination-cursor","repository":"eicrud/eicrud","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.63},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.7034944125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.3678230625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5771226500000001},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.5703915}},"judge_cost":0.0003504,"prompt_sha256":"790fa1385f54561dd5ab3b61e86d5cfc25579d67ccc621cd59aa9e09a233ed87","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"eicrud-keyset-pagination-cursor","repository":"eicrud/eicrud","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.7034944125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.3678230625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5771226500000001},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.5703915}},"judge_cost":0.00029208,"prompt_sha256":"790fa1385f54561dd5ab3b61e86d5cfc25579d67ccc621cd59aa9e09a233ed87","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"eicrud-keyset-pagination-cursor","repository":"eicrud/eicrud","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.7034944125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.3678230625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5771226500000001},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.5703915}},"judge_cost":0.0003167,"prompt_sha256":"790fa1385f54561dd5ab3b61e86d5cfc25579d67ccc621cd59aa9e09a233ed87","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"eicrud-keyset-pagination-cursor","repository":"eicrud/eicrud","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.71},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.7034944125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.3678230625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5771226500000001},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.5703915}},"judge_cost":0.00040963999999999996,"prompt_sha256":"790fa1385f54561dd5ab3b61e86d5cfc25579d67ccc621cd59aa9e09a233ed87","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"eicrud-keyset-pagination-cursor","repository":"eicrud/eicrud","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.7034944125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.3678230625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5771226500000001},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.5703915}},"judge_cost":0.00028478,"prompt_sha256":"790fa1385f54561dd5ab3b61e86d5cfc25579d67ccc621cd59aa9e09a233ed87","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"etree-xml-diff-patch","repository":"beevik/etree","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9460424124999998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5187945625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45262095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4038785000000003}},"judge_cost":0.0007945,"prompt_sha256":"443bb2f2eb3d87937386b9010d8ca8d1da0102aecfcb225df8adafb14d68dac6","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"etree-xml-diff-patch","repository":"beevik/etree","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9460424124999998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5187945625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45262095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4038785000000003}},"judge_cost":0.0007945,"prompt_sha256":"443bb2f2eb3d87937386b9010d8ca8d1da0102aecfcb225df8adafb14d68dac6","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"etree-xml-diff-patch","repository":"beevik/etree","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9460424124999998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5187945625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45262095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4038785000000003}},"judge_cost":0.0005367,"prompt_sha256":"443bb2f2eb3d87937386b9010d8ca8d1da0102aecfcb225df8adafb14d68dac6","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"etree-xml-diff-patch","repository":"beevik/etree","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.2},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9460424124999998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5187945625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45262095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4038785000000003}},"judge_cost":0.00053718,"prompt_sha256":"443bb2f2eb3d87937386b9010d8ca8d1da0102aecfcb225df8adafb14d68dac6","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"etree-xml-diff-patch","repository":"beevik/etree","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9460424124999998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5187945625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45262095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4038785000000003}},"judge_cost":0.0005054,"prompt_sha256":"443bb2f2eb3d87937386b9010d8ca8d1da0102aecfcb225df8adafb14d68dac6","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"etree-xml-diff-patch","repository":"beevik/etree","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.9460424124999998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5187945625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45262095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.4038785000000003}},"judge_cost":0.00051908,"prompt_sha256":"443bb2f2eb3d87937386b9010d8ca8d1da0102aecfcb225df8adafb14d68dac6","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"expr-try-catch-errors","repository":"expr-lang/expr","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":11.331634424999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.3351565625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.22852245},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6647985}},"judge_cost":0.0005817,"prompt_sha256":"635a8723abb17dca898b21b303e1f475684bd46c392ecc15e264ffd0e29c4b6c","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"expr-try-catch-errors","repository":"expr-lang/expr","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.2},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":11.331634424999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.3351565625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.22852245},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6647985}},"judge_cost":0.00028717999999999994,"prompt_sha256":"635a8723abb17dca898b21b303e1f475684bd46c392ecc15e264ffd0e29c4b6c","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"expr-try-catch-errors","repository":"expr-lang/expr","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":11.331634424999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.3351565625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.22852245},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6647985}},"judge_cost":0.0003299,"prompt_sha256":"635a8723abb17dca898b21b303e1f475684bd46c392ecc15e264ffd0e29c4b6c","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"expr-try-catch-errors","repository":"expr-lang/expr","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.22},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":11.331634424999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.3351565625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.22852245},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6647985}},"judge_cost":0.0003238,"prompt_sha256":"635a8723abb17dca898b21b303e1f475684bd46c392ecc15e264ffd0e29c4b6c","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"expr-try-catch-errors","repository":"expr-lang/expr","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":11.331634424999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.3351565625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.22852245},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6647985}},"judge_cost":0.0004692,"prompt_sha256":"635a8723abb17dca898b21b303e1f475684bd46c392ecc15e264ffd0e29c4b6c","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"expr-try-catch-errors","repository":"expr-lang/expr","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":11.331634424999999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.3351565625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.22852245},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6647985}},"judge_cost":0.00028468,"prompt_sha256":"635a8723abb17dca898b21b303e1f475684bd46c392ecc15e264ffd0e29c4b6c","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"fastapi-deprecation-response-headers","repository":"fastapi/fastapi","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.991325324999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.2602995000000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7267831},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1255335}},"judge_cost":0.00041664999999999996,"prompt_sha256":"bc839003d1d4dac79d0a687149617138a94c52faade31343ae09af04eb46cab5","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"fastapi-deprecation-response-headers","repository":"fastapi/fastapi","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.991325324999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.2602995000000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7267831},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1255335}},"judge_cost":0.00082805,"prompt_sha256":"bc839003d1d4dac79d0a687149617138a94c52faade31343ae09af04eb46cab5","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"fastapi-deprecation-response-headers","repository":"fastapi/fastapi","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.991325324999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.2602995000000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7267831},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1255335}},"judge_cost":0.00040993,"prompt_sha256":"bc839003d1d4dac79d0a687149617138a94c52faade31343ae09af04eb46cab5","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"fastapi-deprecation-response-headers","repository":"fastapi/fastapi","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.991325324999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.2602995000000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7267831},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1255335}},"judge_cost":0.00031078,"prompt_sha256":"bc839003d1d4dac79d0a687149617138a94c52faade31343ae09af04eb46cab5","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"fastapi-deprecation-response-headers","repository":"fastapi/fastapi","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.991325324999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.2602995000000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7267831},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1255335}},"judge_cost":0.00042734999999999995,"prompt_sha256":"bc839003d1d4dac79d0a687149617138a94c52faade31343ae09af04eb46cab5","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"fastapi-deprecation-response-headers","repository":"fastapi/fastapi","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.991325324999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.2602995000000004},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7267831},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.1255335}},"judge_cost":0.00039663,"prompt_sha256":"bc839003d1d4dac79d0a687149617138a94c52faade31343ae09af04eb46cab5","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"fastapi-implicit-head-options","repository":"fastapi/fastapi","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.303303287499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.93086},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.0156014500000001},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.133425}},"judge_cost":0.0003008,"prompt_sha256":"72d9ef698118e2b40a56a52a4a71a7295259421e762c4a1f16281b79d9020763","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"fastapi-implicit-head-options","repository":"fastapi/fastapi","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.2},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.303303287499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.93086},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.0156014500000001},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.133425}},"judge_cost":0.00029168,"prompt_sha256":"72d9ef698118e2b40a56a52a4a71a7295259421e762c4a1f16281b79d9020763","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"fastapi-implicit-head-options","repository":"fastapi/fastapi","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.303303287499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.93086},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.0156014500000001},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.133425}},"judge_cost":0.0008981999999999999,"prompt_sha256":"72d9ef698118e2b40a56a52a4a71a7295259421e762c4a1f16281b79d9020763","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"fastapi-implicit-head-options","repository":"fastapi/fastapi","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.303303287499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.93086},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.0156014500000001},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.133425}},"judge_cost":0.00026244,"prompt_sha256":"72d9ef698118e2b40a56a52a4a71a7295259421e762c4a1f16281b79d9020763","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"fastapi-implicit-head-options","repository":"fastapi/fastapi","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.303303287499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.93086},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.0156014500000001},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.133425}},"judge_cost":0.0003331,"prompt_sha256":"72d9ef698118e2b40a56a52a4a71a7295259421e762c4a1f16281b79d9020763","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"fastapi-implicit-head-options","repository":"fastapi/fastapi","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.303303287499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.93086},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.0156014500000001},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.133425}},"judge_cost":0.00029637999999999995,"prompt_sha256":"72d9ef698118e2b40a56a52a4a71a7295259421e762c4a1f16281b79d9020763","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"fd-deterministic-multi-key-sorting","repository":"sharkdp/fd","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.125094562499998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.880816437500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5505151},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6243935}},"judge_cost":0.0007394999999999999,"prompt_sha256":"e2a9331dc92448ad42b3b02a65c9eb9d2ed605e60300ee744eb5356a8ad7c8eb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"fd-deterministic-multi-key-sorting","repository":"sharkdp/fd","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.125094562499998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.880816437500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5505151},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6243935}},"judge_cost":0.00046489999999999997,"prompt_sha256":"e2a9331dc92448ad42b3b02a65c9eb9d2ed605e60300ee744eb5356a8ad7c8eb","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"fd-deterministic-multi-key-sorting","repository":"sharkdp/fd","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.125094562499998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.880816437500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5505151},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6243935}},"judge_cost":0.00044018,"prompt_sha256":"e2a9331dc92448ad42b3b02a65c9eb9d2ed605e60300ee744eb5356a8ad7c8eb","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"fd-deterministic-multi-key-sorting","repository":"sharkdp/fd","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.45},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.125094562499998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.880816437500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5505151},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6243935}},"judge_cost":0.00044799999999999994,"prompt_sha256":"e2a9331dc92448ad42b3b02a65c9eb9d2ed605e60300ee744eb5356a8ad7c8eb","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"fd-deterministic-multi-key-sorting","repository":"sharkdp/fd","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.125094562499998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.880816437500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5505151},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6243935}},"judge_cost":0.00032904,"prompt_sha256":"e2a9331dc92448ad42b3b02a65c9eb9d2ed605e60300ee744eb5356a8ad7c8eb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"fd-deterministic-multi-key-sorting","repository":"sharkdp/fd","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.125094562499998},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.880816437500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5505151},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.6243935}},"judge_cost":0.00046048,"prompt_sha256":"e2a9331dc92448ad42b3b02a65c9eb9d2ed605e60300ee744eb5356a8ad7c8eb","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"geo-shapeindex-serialization","repository":"golang/geo","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.209905062500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.19009275},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.42618525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.991511}},"judge_cost":0.00028337999999999996,"prompt_sha256":"998f50e0f0558f89b15673b9bcefa09aa456bd4abd4301367603728c1a06ef93","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"geo-shapeindex-serialization","repository":"golang/geo","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.209905062500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.19009275},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.42618525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.991511}},"judge_cost":0.00031769999999999997,"prompt_sha256":"998f50e0f0558f89b15673b9bcefa09aa456bd4abd4301367603728c1a06ef93","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"geo-shapeindex-serialization","repository":"golang/geo","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.209905062500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.19009275},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.42618525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.991511}},"judge_cost":0.00028471999999999997,"prompt_sha256":"998f50e0f0558f89b15673b9bcefa09aa456bd4abd4301367603728c1a06ef93","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"geo-shapeindex-serialization","repository":"golang/geo","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.209905062500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.19009275},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.42618525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.991511}},"judge_cost":0.0008443,"prompt_sha256":"998f50e0f0558f89b15673b9bcefa09aa456bd4abd4301367603728c1a06ef93","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"geo-shapeindex-serialization","repository":"golang/geo","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.209905062500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.19009275},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.42618525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.991511}},"judge_cost":0.0003872,"prompt_sha256":"998f50e0f0558f89b15673b9bcefa09aa456bd4abd4301367603728c1a06ef93","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"geo-shapeindex-serialization","repository":"golang/geo","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.209905062500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.19009275},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.42618525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.991511}},"judge_cost":0.00027488,"prompt_sha256":"998f50e0f0558f89b15673b9bcefa09aa456bd4abd4301367603728c1a06ef93","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"go-critic-doc-link-checker","repository":"go-critic/go-critic","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.110033775},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.782596624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.61364045},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.7402715}},"judge_cost":0.0007033,"prompt_sha256":"07a8c035a07c2dd59fe26eb1c2d37d64d3014e8796079116157bfc04799431a4","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"go-critic-doc-link-checker","repository":"go-critic/go-critic","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.110033775},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.782596624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.61364045},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.7402715}},"judge_cost":0.0003279,"prompt_sha256":"07a8c035a07c2dd59fe26eb1c2d37d64d3014e8796079116157bfc04799431a4","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"go-critic-doc-link-checker","repository":"go-critic/go-critic","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.110033775},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.782596624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.61364045},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.7402715}},"judge_cost":0.00032837999999999997,"prompt_sha256":"07a8c035a07c2dd59fe26eb1c2d37d64d3014e8796079116157bfc04799431a4","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"go-critic-doc-link-checker","repository":"go-critic/go-critic","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.110033775},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.782596624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.61364045},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.7402715}},"judge_cost":0.0003362,"prompt_sha256":"07a8c035a07c2dd59fe26eb1c2d37d64d3014e8796079116157bfc04799431a4","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"go-critic-doc-link-checker","repository":"go-critic/go-critic","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.110033775},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.782596624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.61364045},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.7402715}},"judge_cost":0.00045176,"prompt_sha256":"07a8c035a07c2dd59fe26eb1c2d37d64d3014e8796079116157bfc04799431a4","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"go-critic-doc-link-checker","repository":"go-critic/go-critic","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.110033775},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.782596624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.61364045},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.7402715}},"judge_cost":0.00034988,"prompt_sha256":"07a8c035a07c2dd59fe26eb1c2d37d64d3014e8796079116157bfc04799431a4","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"go-genai-streamed-function-args","repository":"googleapis/go-genai","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7089415874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9573174375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.71195095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.90167225}},"judge_cost":0.00061775,"prompt_sha256":"195153fdc78766d3b7ca2e524522875fe7c2e91c1d16e1d77f846ba9a48a98e3","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"go-genai-streamed-function-args","repository":"googleapis/go-genai","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7089415874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9573174375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.71195095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.90167225}},"judge_cost":0.00032875,"prompt_sha256":"195153fdc78766d3b7ca2e524522875fe7c2e91c1d16e1d77f846ba9a48a98e3","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"go-genai-streamed-function-args","repository":"googleapis/go-genai","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7089415874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9573174375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.71195095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.90167225}},"judge_cost":0.00031723,"prompt_sha256":"195153fdc78766d3b7ca2e524522875fe7c2e91c1d16e1d77f846ba9a48a98e3","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"go-genai-streamed-function-args","repository":"googleapis/go-genai","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7089415874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9573174375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.71195095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.90167225}},"judge_cost":0.00029745,"prompt_sha256":"195153fdc78766d3b7ca2e524522875fe7c2e91c1d16e1d77f846ba9a48a98e3","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"go-genai-streamed-function-args","repository":"googleapis/go-genai","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7089415874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9573174375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.71195095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.90167225}},"judge_cost":0.00034737999999999994,"prompt_sha256":"195153fdc78766d3b7ca2e524522875fe7c2e91c1d16e1d77f846ba9a48a98e3","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"go-genai-streamed-function-args","repository":"googleapis/go-genai","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7089415874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9573174375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.71195095},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.90167225}},"judge_cost":0.00033392999999999997,"prompt_sha256":"195153fdc78766d3b7ca2e524522875fe7c2e91c1d16e1d77f846ba9a48a98e3","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"go-git-worktree-merge-conflicts","repository":"go-git/go-git","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.375930075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.822291375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.87410295},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.85314975}},"judge_cost":0.00033315,"prompt_sha256":"f115115e810e2ad5bdd29e292089d55d0f3a39655c6e2d56ff1e58b8271871c9","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"go-git-worktree-merge-conflicts","repository":"go-git/go-git","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.375930075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.822291375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.87410295},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.85314975}},"judge_cost":0.00075775,"prompt_sha256":"f115115e810e2ad5bdd29e292089d55d0f3a39655c6e2d56ff1e58b8271871c9","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"go-git-worktree-merge-conflicts","repository":"go-git/go-git","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.29},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.375930075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.822291375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.87410295},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.85314975}},"judge_cost":0.00031083,"prompt_sha256":"f115115e810e2ad5bdd29e292089d55d0f3a39655c6e2d56ff1e58b8271871c9","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"go-git-worktree-merge-conflicts","repository":"go-git/go-git","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.375930075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.822291375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.87410295},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.85314975}},"judge_cost":0.00029858,"prompt_sha256":"f115115e810e2ad5bdd29e292089d55d0f3a39655c6e2d56ff1e58b8271871c9","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"go-git-worktree-merge-conflicts","repository":"go-git/go-git","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.375930075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.822291375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.87410295},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.85314975}},"judge_cost":0.00037864999999999996,"prompt_sha256":"f115115e810e2ad5bdd29e292089d55d0f3a39655c6e2d56ff1e58b8271871c9","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"go-git-worktree-merge-conflicts","repository":"go-git/go-git","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.375930075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.822291375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.87410295},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.85314975}},"judge_cost":0.00033713,"prompt_sha256":"f115115e810e2ad5bdd29e292089d55d0f3a39655c6e2d56ff1e58b8271871c9","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"goreleaser-retry-publish-auditing","repository":"goreleaser/goreleaser","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.486067074999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.678626625000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8697235000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.3663695}},"judge_cost":0.00035517999999999997,"prompt_sha256":"752c0daeddc35e06540b18b95842e8d7599e69dba91a13064ea2ec388e409819","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"goreleaser-retry-publish-auditing","repository":"goreleaser/goreleaser","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.486067074999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.678626625000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8697235000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.3663695}},"judge_cost":0.0006941,"prompt_sha256":"752c0daeddc35e06540b18b95842e8d7599e69dba91a13064ea2ec388e409819","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"goreleaser-retry-publish-auditing","repository":"goreleaser/goreleaser","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.486067074999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.678626625000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8697235000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.3663695}},"judge_cost":0.0003787,"prompt_sha256":"752c0daeddc35e06540b18b95842e8d7599e69dba91a13064ea2ec388e409819","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"goreleaser-retry-publish-auditing","repository":"goreleaser/goreleaser","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.486067074999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.678626625000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8697235000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.3663695}},"judge_cost":0.00031408,"prompt_sha256":"752c0daeddc35e06540b18b95842e8d7599e69dba91a13064ea2ec388e409819","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"goreleaser-retry-publish-auditing","repository":"goreleaser/goreleaser","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.48},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.486067074999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.678626625000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8697235000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.3663695}},"judge_cost":0.0003486,"prompt_sha256":"752c0daeddc35e06540b18b95842e8d7599e69dba91a13064ea2ec388e409819","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"gql-incremental-graphql-delivery","repository":"graphql-python/gql","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.603259649999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.798948687499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.7647185000000001},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.867573}},"judge_cost":0.00068255,"prompt_sha256":"971677d9a67787c0e40c8621e8126c01d75e4c922c8344cf0b57badad2054aae","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"gql-incremental-graphql-delivery","repository":"graphql-python/gql","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.603259649999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.798948687499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.7647185000000001},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.867573}},"judge_cost":0.00028483,"prompt_sha256":"971677d9a67787c0e40c8621e8126c01d75e4c922c8344cf0b57badad2054aae","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"gql-incremental-graphql-delivery","repository":"graphql-python/gql","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.4},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.603259649999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.798948687499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.7647185000000001},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.867573}},"judge_cost":0.00030595000000000004,"prompt_sha256":"971677d9a67787c0e40c8621e8126c01d75e4c922c8344cf0b57badad2054aae","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"gql-incremental-graphql-delivery","repository":"graphql-python/gql","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.603259649999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.798948687499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.7647185000000001},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.867573}},"judge_cost":0.00068255,"prompt_sha256":"971677d9a67787c0e40c8621e8126c01d75e4c922c8344cf0b57badad2054aae","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"goreleaser-retry-publish-auditing","repository":"goreleaser/goreleaser","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.37},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.486067074999998},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.678626625000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8697235000000001},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.3663695}},"judge_cost":0.00045947999999999996,"prompt_sha256":"752c0daeddc35e06540b18b95842e8d7599e69dba91a13064ea2ec388e409819","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"gql-incremental-graphql-delivery","repository":"graphql-python/gql","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.603259649999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.798948687499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.7647185000000001},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.867573}},"judge_cost":0.00034233,"prompt_sha256":"971677d9a67787c0e40c8621e8126c01d75e4c922c8344cf0b57badad2054aae","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"happy-dom-abort-pending-body-reads","repository":"capricorn86/happy-dom","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.4814272375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.353841062499999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.1813355},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.84721125}},"judge_cost":0.0005953,"prompt_sha256":"620270ee5bff3586c9d64678f297a5d9bfa37c2dec8c4ac7e5ed7ccfece303cb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"happy-dom-abort-pending-body-reads","repository":"capricorn86/happy-dom","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.4814272375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.353841062499999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.1813355},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.84721125}},"judge_cost":0.0005953,"prompt_sha256":"620270ee5bff3586c9d64678f297a5d9bfa37c2dec8c4ac7e5ed7ccfece303cb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"gql-incremental-graphql-delivery","repository":"graphql-python/gql","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.55},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.603259649999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.798948687499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.7647185000000001},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.867573}},"judge_cost":0.00029745,"prompt_sha256":"971677d9a67787c0e40c8621e8126c01d75e4c922c8344cf0b57badad2054aae","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"happy-dom-abort-pending-body-reads","repository":"capricorn86/happy-dom","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.4814272375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.353841062499999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.1813355},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.84721125}},"judge_cost":0.0002343,"prompt_sha256":"620270ee5bff3586c9d64678f297a5d9bfa37c2dec8c4ac7e5ed7ccfece303cb","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"happy-dom-abort-pending-body-reads","repository":"capricorn86/happy-dom","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.4814272375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.353841062499999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.1813355},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.84721125}},"judge_cost":0.00025158,"prompt_sha256":"620270ee5bff3586c9d64678f297a5d9bfa37c2dec8c4ac7e5ed7ccfece303cb","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"happy-dom-abort-pending-body-reads","repository":"capricorn86/happy-dom","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.4814272375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.353841062499999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.1813355},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.84721125}},"judge_cost":0.0003038,"prompt_sha256":"620270ee5bff3586c9d64678f297a5d9bfa37c2dec8c4ac7e5ed7ccfece303cb","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"happy-dom-abort-pending-body-reads","repository":"capricorn86/happy-dom","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.4814272375},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.353841062499999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.1813355},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.84721125}},"judge_cost":0.00023228,"prompt_sha256":"620270ee5bff3586c9d64678f297a5d9bfa37c2dec8c4ac7e5ed7ccfece303cb","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"happy-dom-deterministic-intersectionobserver","repository":"capricorn86/happy-dom","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.3570851},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.6087885},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7319916},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.39679}},"judge_cost":0.0002838,"prompt_sha256":"5b59269e68c8a215aa9e70d42499235841acea346c3d92ba5b10396e30523d16","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"happy-dom-deterministic-intersectionobserver","repository":"capricorn86/happy-dom","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.3570851},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.6087885},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7319916},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.39679}},"judge_cost":0.0007312,"prompt_sha256":"5b59269e68c8a215aa9e70d42499235841acea346c3d92ba5b10396e30523d16","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"happy-dom-deterministic-intersectionobserver","repository":"capricorn86/happy-dom","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.3570851},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.6087885},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7319916},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.39679}},"judge_cost":0.00036467999999999993,"prompt_sha256":"5b59269e68c8a215aa9e70d42499235841acea346c3d92ba5b10396e30523d16","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"happy-dom-deterministic-intersectionobserver","repository":"capricorn86/happy-dom","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.3570851},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.6087885},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7319916},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.39679}},"judge_cost":0.00030577999999999996,"prompt_sha256":"5b59269e68c8a215aa9e70d42499235841acea346c3d92ba5b10396e30523d16","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"happy-dom-deterministic-intersectionobserver","repository":"capricorn86/happy-dom","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.3570851},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.6087885},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7319916},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.39679}},"judge_cost":0.0004265,"prompt_sha256":"5b59269e68c8a215aa9e70d42499235841acea346c3d92ba5b10396e30523d16","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"happy-dom-deterministic-intersectionobserver","repository":"capricorn86/happy-dom","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.3570851},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.6087885},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7319916},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.39679}},"judge_cost":0.00052076,"prompt_sha256":"5b59269e68c8a215aa9e70d42499235841acea346c3d92ba5b10396e30523d16","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"helm-array-merge-strategies","repository":"helm/helm","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.685666075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.430369625000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.71195135},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.62249775}},"judge_cost":0.0007264,"prompt_sha256":"1deef585895b83941afb9e5dfd1c20ec04794f928b32671fdff39619cf63f882","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"helm-array-merge-strategies","repository":"helm/helm","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.685666075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.430369625000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.71195135},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.62249775}},"judge_cost":0.0003522,"prompt_sha256":"1deef585895b83941afb9e5dfd1c20ec04794f928b32671fdff39619cf63f882","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"helm-array-merge-strategies","repository":"helm/helm","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.685666075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.430369625000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.71195135},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.62249775}},"judge_cost":0.00036828,"prompt_sha256":"1deef585895b83941afb9e5dfd1c20ec04794f928b32671fdff39619cf63f882","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"helm-array-merge-strategies","repository":"helm/helm","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.685666075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.430369625000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.71195135},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.62249775}},"judge_cost":0.00030356,"prompt_sha256":"1deef585895b83941afb9e5dfd1c20ec04794f928b32671fdff39619cf63f882","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"helm-array-merge-strategies","repository":"helm/helm","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.685666075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.430369625000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.71195135},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.62249775}},"judge_cost":0.00038617999999999997,"prompt_sha256":"1deef585895b83941afb9e5dfd1c20ec04794f928b32671fdff39619cf63f882","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"helm-unified-manifest-stream","repository":"helm/helm","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.390764012500002},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.095946937499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1268288},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.78430375}},"judge_cost":0.0002517,"prompt_sha256":"240190c778565ea2583a7cf9204f0c0e184571ec12c9583bf233df1356df7890","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"helm-unified-manifest-stream","repository":"helm/helm","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.390764012500002},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.095946937499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1268288},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.78430375}},"judge_cost":0.0006103,"prompt_sha256":"240190c778565ea2583a7cf9204f0c0e184571ec12c9583bf233df1356df7890","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"helm-array-merge-strategies","repository":"helm/helm","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.685666075},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.430369625000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.71195135},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.62249775}},"judge_cost":0.0003461,"prompt_sha256":"1deef585895b83941afb9e5dfd1c20ec04794f928b32671fdff39619cf63f882","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"helm-unified-manifest-stream","repository":"helm/helm","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.390764012500002},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.095946937499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1268288},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.78430375}},"judge_cost":0.00026898,"prompt_sha256":"240190c778565ea2583a7cf9204f0c0e184571ec12c9583bf233df1356df7890","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"helm-unified-manifest-stream","repository":"helm/helm","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.390764012500002},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.095946937499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1268288},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.78430375}},"judge_cost":0.0003032,"prompt_sha256":"240190c778565ea2583a7cf9204f0c0e184571ec12c9583bf233df1356df7890","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"helm-unified-manifest-stream","repository":"helm/helm","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.390764012500002},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.095946937499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1268288},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.78430375}},"judge_cost":0.00032408000000000003,"prompt_sha256":"240190c778565ea2583a7cf9204f0c0e184571ec12c9583bf233df1356df7890","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"helm-unified-manifest-stream","repository":"helm/helm","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.75},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.390764012500002},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.095946937499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1268288},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.78430375}},"judge_cost":0.00039896,"prompt_sha256":"240190c778565ea2583a7cf9204f0c0e184571ec12c9583bf233df1356df7890","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"httpx-deterministic-cookie-store","repository":"encode/httpx","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9573133124999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0656293125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.44728625},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6898910000000003}},"judge_cost":0.00036635,"prompt_sha256":"d61a977d46624fb4bd2b499d1a00959c0cc4ea9b64ff1bfeee3a8db946a0bc87","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"httpx-deterministic-cookie-store","repository":"encode/httpx","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9573133124999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0656293125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.44728625},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6898910000000003}},"judge_cost":0.00086895,"prompt_sha256":"d61a977d46624fb4bd2b499d1a00959c0cc4ea9b64ff1bfeee3a8db946a0bc87","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"httpx-deterministic-cookie-store","repository":"encode/httpx","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9573133124999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0656293125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.44728625},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6898910000000003}},"judge_cost":0.00048803,"prompt_sha256":"d61a977d46624fb4bd2b499d1a00959c0cc4ea9b64ff1bfeee3a8db946a0bc87","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"httpx-deterministic-cookie-store","repository":"encode/httpx","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9573133124999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0656293125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.44728625},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6898910000000003}},"judge_cost":0.00045865,"prompt_sha256":"d61a977d46624fb4bd2b499d1a00959c0cc4ea9b64ff1bfeee3a8db946a0bc87","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"httpx-deterministic-cookie-store","repository":"encode/httpx","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9573133124999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0656293125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.44728625},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6898910000000003}},"judge_cost":0.0005793,"prompt_sha256":"d61a977d46624fb4bd2b499d1a00959c0cc4ea9b64ff1bfeee3a8db946a0bc87","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"httpx-multipart-response-parsing","repository":"encode/httpx","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.3742994250000007},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.7481950000000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52200785},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.4842565}},"judge_cost":0.0005613,"prompt_sha256":"bbc79873e0afe00f26bba554f2757ecc9ad117835e19fbd8280270b42e628569","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"httpx-deterministic-cookie-store","repository":"encode/httpx","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.3},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9573133124999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0656293125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.44728625},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6898910000000003}},"judge_cost":0.00042433,"prompt_sha256":"d61a977d46624fb4bd2b499d1a00959c0cc4ea9b64ff1bfeee3a8db946a0bc87","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"httpx-multipart-response-parsing","repository":"encode/httpx","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.3742994250000007},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.7481950000000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52200785},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.4842565}},"judge_cost":0.0005613,"prompt_sha256":"bbc79873e0afe00f26bba554f2757ecc9ad117835e19fbd8280270b42e628569","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"httpx-multipart-response-parsing","repository":"encode/httpx","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.3742994250000007},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.7481950000000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52200785},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.4842565}},"judge_cost":0.00035269999999999995,"prompt_sha256":"bbc79873e0afe00f26bba554f2757ecc9ad117835e19fbd8280270b42e628569","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"httpx-multipart-response-parsing","repository":"encode/httpx","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.3742994250000007},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.7481950000000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52200785},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.4842565}},"judge_cost":0.00033038,"prompt_sha256":"bbc79873e0afe00f26bba554f2757ecc9ad117835e19fbd8280270b42e628569","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"httpx-multipart-response-parsing","repository":"encode/httpx","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.3742994250000007},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.7481950000000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52200785},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.4842565}},"judge_cost":0.0004054,"prompt_sha256":"bbc79873e0afe00f26bba554f2757ecc9ad117835e19fbd8280270b42e628569","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"httpx-multipart-response-parsing","repository":"encode/httpx","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.3742994250000007},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.7481950000000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52200785},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.4842565}},"judge_cost":0.00038787999999999995,"prompt_sha256":"bbc79873e0afe00f26bba554f2757ecc9ad117835e19fbd8280270b42e628569","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"httpx-streaming-json-iteration","repository":"encode/httpx","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.74},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1455781},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2802523125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.44592395},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.3155135}},"judge_cost":0.00061585,"prompt_sha256":"d5a09f7decaf6aa43cbd38d403241f40123af25d5ed83f0ccb1746b9ac3fdb14","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"httpx-streaming-json-iteration","repository":"encode/httpx","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1455781},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2802523125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.44592395},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.3155135}},"judge_cost":0.00039645,"prompt_sha256":"d5a09f7decaf6aa43cbd38d403241f40123af25d5ed83f0ccb1746b9ac3fdb14","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"httpx-streaming-json-iteration","repository":"encode/httpx","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.47},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1455781},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2802523125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.44592395},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.3155135}},"judge_cost":0.00051453,"prompt_sha256":"d5a09f7decaf6aa43cbd38d403241f40123af25d5ed83f0ccb1746b9ac3fdb14","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"httpx-streaming-json-iteration","repository":"encode/httpx","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1455781},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2802523125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.44592395},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.3155135}},"judge_cost":0.00027734,"prompt_sha256":"d5a09f7decaf6aa43cbd38d403241f40123af25d5ed83f0ccb1746b9ac3fdb14","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"httpx-streaming-json-iteration","repository":"encode/httpx","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1455781},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2802523125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.44592395},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.3155135}},"judge_cost":0.00035435,"prompt_sha256":"d5a09f7decaf6aa43cbd38d403241f40123af25d5ed83f0ccb1746b9ac3fdb14","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"httpx-streaming-json-iteration","repository":"encode/httpx","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1455781},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2802523125},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.44592395},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.3155135}},"judge_cost":0.00038482999999999996,"prompt_sha256":"d5a09f7decaf6aa43cbd38d403241f40123af25d5ed83f0ccb1746b9ac3fdb14","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"igel-persist-feature-schema","repository":"nidhaloff/igel","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.05591855},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6723145624999995},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45297085000000004},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.8563247500000002}},"judge_cost":0.0003141,"prompt_sha256":"d181b0b5c9ba8c44e655e6b3d7d300f5912a8b8e0b94330fecc574801112a020","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"igel-persist-feature-schema","repository":"nidhaloff/igel","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.05591855},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6723145624999995},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45297085000000004},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.8563247500000002}},"judge_cost":0.0006103,"prompt_sha256":"d181b0b5c9ba8c44e655e6b3d7d300f5912a8b8e0b94330fecc574801112a020","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"igel-persist-feature-schema","repository":"nidhaloff/igel","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.05591855},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6723145624999995},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45297085000000004},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.8563247500000002}},"judge_cost":0.00028817999999999997,"prompt_sha256":"d181b0b5c9ba8c44e655e6b3d7d300f5912a8b8e0b94330fecc574801112a020","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"igel-persist-feature-schema","repository":"nidhaloff/igel","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.05591855},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6723145624999995},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45297085000000004},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.8563247500000002}},"judge_cost":0.00028399999999999996,"prompt_sha256":"d181b0b5c9ba8c44e655e6b3d7d300f5912a8b8e0b94330fecc574801112a020","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"igel-persist-feature-schema","repository":"nidhaloff/igel","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.05591855},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6723145624999995},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45297085000000004},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.8563247500000002}},"judge_cost":0.00029288000000000003,"prompt_sha256":"d181b0b5c9ba8c44e655e6b3d7d300f5912a8b8e0b94330fecc574801112a020","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"ink-grid-box-layout","repository":"vadimdemedes/ink","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.0754557375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.701998687499998},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.83048595},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.67271925}},"judge_cost":0.00026504999999999996,"prompt_sha256":"64da7500a38d00b6e5edaa4480809224568a891c3d883af4f0f56429f22e86fb","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"igel-persist-feature-schema","repository":"nidhaloff/igel","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.05591855},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6723145624999995},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.45297085000000004},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.8563247500000002}},"judge_cost":0.0005876,"prompt_sha256":"d181b0b5c9ba8c44e655e6b3d7d300f5912a8b8e0b94330fecc574801112a020","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ink-grid-box-layout","repository":"vadimdemedes/ink","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.48},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.0754557375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.701998687499998},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.83048595},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.67271925}},"judge_cost":0.0007760499999999999,"prompt_sha256":"64da7500a38d00b6e5edaa4480809224568a891c3d883af4f0f56429f22e86fb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ink-grid-box-layout","repository":"vadimdemedes/ink","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.0754557375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.701998687499998},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.83048595},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.67271925}},"judge_cost":0.00025713,"prompt_sha256":"64da7500a38d00b6e5edaa4480809224568a891c3d883af4f0f56429f22e86fb","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"ink-grid-box-layout","repository":"vadimdemedes/ink","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.48},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.0754557375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.701998687499998},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.83048595},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.67271925}},"judge_cost":0.00037535,"prompt_sha256":"64da7500a38d00b6e5edaa4480809224568a891c3d883af4f0f56429f22e86fb","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"ink-grid-box-layout","repository":"vadimdemedes/ink","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.0754557375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.701998687499998},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.83048595},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.67271925}},"judge_cost":0.00036878,"prompt_sha256":"64da7500a38d00b6e5edaa4480809224568a891c3d883af4f0f56429f22e86fb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ink-grid-box-layout","repository":"vadimdemedes/ink","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.0754557375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.701998687499998},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.83048595},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.67271925}},"judge_cost":0.00023538,"prompt_sha256":"64da7500a38d00b6e5edaa4480809224568a891c3d883af4f0f56429f22e86fb","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"ipython-session-bundle-replay","repository":"ipython/ipython","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":null},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.830068687499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4604987},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.00310125}},"judge_cost":0.0004514799999999999,"prompt_sha256":"c20b989d562225f3389a028007a5b983ad951b8ca9d27f4c44af8dabad3f1919","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"ipython-session-bundle-replay","repository":"ipython/ipython","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":null},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.830068687499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4604987},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.00310125}},"judge_cost":0.00030834,"prompt_sha256":"c20b989d562225f3389a028007a5b983ad951b8ca9d27f4c44af8dabad3f1919","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"ipython-session-bundle-replay","repository":"ipython/ipython","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":null},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.830068687499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4604987},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.00310125}},"judge_cost":0.0006341200000000001,"prompt_sha256":"c20b989d562225f3389a028007a5b983ad951b8ca9d27f4c44af8dabad3f1919","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ipython-session-bundle-replay","repository":"ipython/ipython","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":null},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.830068687499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4604987},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.00310125}},"judge_cost":0.0005585200000000001,"prompt_sha256":"c20b989d562225f3389a028007a5b983ad951b8ca9d27f4c44af8dabad3f1919","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ipython-session-bundle-replay","repository":"ipython/ipython","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":null},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.830068687499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4604987},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.00310125}},"judge_cost":0.00042737999999999994,"prompt_sha256":"c20b989d562225f3389a028007a5b983ad951b8ca9d27f4c44af8dabad3f1919","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"ipython-session-bundle-replay","repository":"ipython/ipython","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":null},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.830068687499999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4604987},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.00310125}},"judge_cost":0.0004545,"prompt_sha256":"c20b989d562225f3389a028007a5b983ad951b8ca9d27f4c44af8dabad3f1919","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"katex-multicolumn-array-spans","repository":"KaTeX/KaTeX","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.068496987500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.92208775},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.92048945},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.60055825}},"judge_cost":0.0005885999999999999,"prompt_sha256":"89d392a8504c791401a3589580bbf67a82a1b80ca843eb2e58d4f1bf73654089","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"katex-multicolumn-array-spans","repository":"KaTeX/KaTeX","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.068496987500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.92208775},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.92048945},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.60055825}},"judge_cost":0.0002924,"prompt_sha256":"89d392a8504c791401a3589580bbf67a82a1b80ca843eb2e58d4f1bf73654089","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"katex-multicolumn-array-spans","repository":"KaTeX/KaTeX","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.068496987500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.92208775},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.92048945},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.60055825}},"judge_cost":0.00028448,"prompt_sha256":"89d392a8504c791401a3589580bbf67a82a1b80ca843eb2e58d4f1bf73654089","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"katex-multicolumn-array-spans","repository":"KaTeX/KaTeX","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.068496987500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.92208775},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.92048945},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.60055825}},"judge_cost":0.0003078,"prompt_sha256":"89d392a8504c791401a3589580bbf67a82a1b80ca843eb2e58d4f1bf73654089","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"katex-multicolumn-array-spans","repository":"KaTeX/KaTeX","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.068496987500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.92208775},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.92048945},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.60055825}},"judge_cost":0.0002611,"prompt_sha256":"89d392a8504c791401a3589580bbf67a82a1b80ca843eb2e58d4f1bf73654089","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"katex-multicolumn-array-spans","repository":"KaTeX/KaTeX","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.41},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.068496987500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.92208775},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.92048945},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.60055825}},"judge_cost":0.00033717999999999997,"prompt_sha256":"89d392a8504c791401a3589580bbf67a82a1b80ca843eb2e58d4f1bf73654089","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"kcp-go-multiplexed-kcp-streams","repository":"xtaci/kcp-go","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5117456625},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":2.8398746666666668},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.40239020000000003},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.05411}},"judge_cost":0.00034885,"prompt_sha256":"18685b6f0107785823d8a305897d5f4b08a83f3f7f6dd7fb063cae36462b942f","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"kcp-go-multiplexed-kcp-streams","repository":"xtaci/kcp-go","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5117456625},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":2.8398746666666668},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.40239020000000003},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.05411}},"judge_cost":0.00037812999999999996,"prompt_sha256":"18685b6f0107785823d8a305897d5f4b08a83f3f7f6dd7fb063cae36462b942f","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"kcp-go-multiplexed-kcp-streams","repository":"xtaci/kcp-go","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5117456625},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":2.8398746666666668},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.40239020000000003},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.05411}},"judge_cost":0.00033674999999999997,"prompt_sha256":"18685b6f0107785823d8a305897d5f4b08a83f3f7f6dd7fb063cae36462b942f","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"kcp-go-multiplexed-kcp-streams","repository":"xtaci/kcp-go","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5117456625},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":2.8398746666666668},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.40239020000000003},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.05411}},"judge_cost":0.00092465,"prompt_sha256":"18685b6f0107785823d8a305897d5f4b08a83f3f7f6dd7fb063cae36462b942f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kcp-go-multiplexed-kcp-streams","repository":"xtaci/kcp-go","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5117456625},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":2.8398746666666668},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.40239020000000003},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.05411}},"judge_cost":0.0003787,"prompt_sha256":"18685b6f0107785823d8a305897d5f4b08a83f3f7f6dd7fb063cae36462b942f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kcp-go-multiplexed-kcp-streams","repository":"xtaci/kcp-go","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.5117456625},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":2.8398746666666668},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.40239020000000003},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.05411}},"judge_cost":0.00033602999999999997,"prompt_sha256":"18685b6f0107785823d8a305897d5f4b08a83f3f7f6dd7fb063cae36462b942f","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"kea-atomic-signal-selectors","repository":"keajs/kea","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6123379874999997},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.367762250000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.810974},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.94137325}},"judge_cost":0.00039485,"prompt_sha256":"418b03c437e93ec3ad1b207be324ff778d968c4f29b63f44e104a2fc7cceab33","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"kea-atomic-signal-selectors","repository":"keajs/kea","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6123379874999997},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.367762250000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.810974},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.94137325}},"judge_cost":0.00043132999999999995,"prompt_sha256":"418b03c437e93ec3ad1b207be324ff778d968c4f29b63f44e104a2fc7cceab33","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"kea-atomic-signal-selectors","repository":"keajs/kea","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.35},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6123379874999997},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.367762250000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.810974},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.94137325}},"judge_cost":0.00082785,"prompt_sha256":"418b03c437e93ec3ad1b207be324ff778d968c4f29b63f44e104a2fc7cceab33","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kea-atomic-signal-selectors","repository":"keajs/kea","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6123379874999997},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.367762250000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.810974},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.94137325}},"judge_cost":0.00043194999999999996,"prompt_sha256":"418b03c437e93ec3ad1b207be324ff778d968c4f29b63f44e104a2fc7cceab33","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"kea-atomic-signal-selectors","repository":"keajs/kea","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6123379874999997},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.367762250000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.810974},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.94137325}},"judge_cost":0.00040483,"prompt_sha256":"418b03c437e93ec3ad1b207be324ff778d968c4f29b63f44e104a2fc7cceab33","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"kgateway-consistent-hash-policy","repository":"kgateway-dev/kgateway","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.082302437500003},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.757748875000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1016709},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.2217389999999995}},"judge_cost":0.000645,"prompt_sha256":"f257c08b0472cc30edb587af1d3d9c94b90861581abc93abc5048f7a47a5308e","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kgateway-consistent-hash-policy","repository":"kgateway-dev/kgateway","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.082302437500003},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.757748875000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1016709},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.2217389999999995}},"judge_cost":0.000645,"prompt_sha256":"f257c08b0472cc30edb587af1d3d9c94b90861581abc93abc5048f7a47a5308e","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kgateway-consistent-hash-policy","repository":"kgateway-dev/kgateway","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.082302437500003},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.757748875000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1016709},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.2217389999999995}},"judge_cost":0.0003752,"prompt_sha256":"f257c08b0472cc30edb587af1d3d9c94b90861581abc93abc5048f7a47a5308e","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"kea-atomic-signal-selectors","repository":"keajs/kea","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.6123379874999997},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.367762250000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.810974},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.94137325}},"judge_cost":0.00063294,"prompt_sha256":"418b03c437e93ec3ad1b207be324ff778d968c4f29b63f44e104a2fc7cceab33","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kgateway-consistent-hash-policy","repository":"kgateway-dev/kgateway","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.082302437500003},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.757748875000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1016709},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.2217389999999995}},"judge_cost":0.00035768,"prompt_sha256":"f257c08b0472cc30edb587af1d3d9c94b90861581abc93abc5048f7a47a5308e","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"kgateway-consistent-hash-policy","repository":"kgateway-dev/kgateway","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.44},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.082302437500003},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.757748875000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1016709},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.2217389999999995}},"judge_cost":0.00034389999999999996,"prompt_sha256":"f257c08b0472cc30edb587af1d3d9c94b90861581abc93abc5048f7a47a5308e","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"kgateway-consistent-hash-policy","repository":"kgateway-dev/kgateway","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.082302437500003},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.757748875000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.1016709},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.2217389999999995}},"judge_cost":0.00036717999999999994,"prompt_sha256":"f257c08b0472cc30edb587af1d3d9c94b90861581abc93abc5048f7a47a5308e","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"kombu-single-active-consumer-priority","repository":"celery/kombu","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.889836462500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0968265625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52554335},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.34415425}},"judge_cost":0.00043335,"prompt_sha256":"c41b443a124cf49e649bf90330ce684fb07a525c85a01786dab976fdd3fd4ffb","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"kombu-single-active-consumer-priority","repository":"celery/kombu","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.889836462500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0968265625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52554335},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.34415425}},"judge_cost":0.00083395,"prompt_sha256":"c41b443a124cf49e649bf90330ce684fb07a525c85a01786dab976fdd3fd4ffb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kombu-single-active-consumer-priority","repository":"celery/kombu","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.889836462500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0968265625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52554335},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.34415425}},"judge_cost":0.00047103,"prompt_sha256":"c41b443a124cf49e649bf90330ce684fb07a525c85a01786dab976fdd3fd4ffb","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"kombu-single-active-consumer-priority","repository":"celery/kombu","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.889836462500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0968265625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52554335},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.34415425}},"judge_cost":0.00044885,"prompt_sha256":"c41b443a124cf49e649bf90330ce684fb07a525c85a01786dab976fdd3fd4ffb","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"kombu-single-active-consumer-priority","repository":"celery/kombu","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.889836462500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0968265625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52554335},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.34415425}},"judge_cost":0.00037562,"prompt_sha256":"c41b443a124cf49e649bf90330ce684fb07a525c85a01786dab976fdd3fd4ffb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kombu-single-active-consumer-priority","repository":"celery/kombu","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.889836462500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0968265625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.52554335},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.34415425}},"judge_cost":0.00045772999999999994,"prompt_sha256":"c41b443a124cf49e649bf90330ce684fb07a525c85a01786dab976fdd3fd4ffb","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"kombu-virtual-queue-dead-lettering","repository":"celery/kombu","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.060389862499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.344722687500002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.64456235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.95201}},"judge_cost":0.00087965,"prompt_sha256":"3d29546f0f1f6614a943aa519a7f24ad90a4be99c9a0104f4634c1249379a983","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kombu-virtual-queue-dead-lettering","repository":"celery/kombu","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.35},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.060389862499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.344722687500002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.64456235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.95201}},"judge_cost":0.00048985,"prompt_sha256":"3d29546f0f1f6614a943aa519a7f24ad90a4be99c9a0104f4634c1249379a983","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"kombu-virtual-queue-dead-lettering","repository":"celery/kombu","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.060389862499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.344722687500002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.64456235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.95201}},"judge_cost":0.00053833,"prompt_sha256":"3d29546f0f1f6614a943aa519a7f24ad90a4be99c9a0104f4634c1249379a983","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"kombu-virtual-queue-dead-lettering","repository":"celery/kombu","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.55},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.060389862499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.344722687500002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.64456235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.95201}},"judge_cost":0.00037245999999999997,"prompt_sha256":"3d29546f0f1f6614a943aa519a7f24ad90a4be99c9a0104f4634c1249379a983","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kombu-virtual-queue-dead-lettering","repository":"celery/kombu","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.060389862499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.344722687500002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.64456235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.95201}},"judge_cost":0.00045614999999999995,"prompt_sha256":"3d29546f0f1f6614a943aa519a7f24ad90a4be99c9a0104f4634c1249379a983","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"kombu-virtual-queue-dead-lettering","repository":"celery/kombu","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.060389862499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.344722687500002},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.64456235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.95201}},"judge_cost":0.00049143,"prompt_sha256":"3d29546f0f1f6614a943aa519a7f24ad90a4be99c9a0104f4634c1249379a983","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"koota-composite-trait-aspects","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.201745037499999},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":12.732614916666662},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.27947105},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.63375475}},"judge_cost":0.0005786999999999999,"prompt_sha256":"69097162c0044bec1ecb29c56fffbeeb0482bc37e1a9495646ec546d6134e54f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-composite-trait-aspects","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.201745037499999},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":12.732614916666662},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.27947105},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.63375475}},"judge_cost":0.0003293,"prompt_sha256":"69097162c0044bec1ecb29c56fffbeeb0482bc37e1a9495646ec546d6134e54f","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"koota-composite-trait-aspects","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.201745037499999},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":12.732614916666662},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.27947105},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.63375475}},"judge_cost":0.00029378,"prompt_sha256":"69097162c0044bec1ecb29c56fffbeeb0482bc37e1a9495646ec546d6134e54f","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"koota-composite-trait-aspects","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.201745037499999},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":12.732614916666662},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.27947105},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.63375475}},"judge_cost":0.000298,"prompt_sha256":"69097162c0044bec1ecb29c56fffbeeb0482bc37e1a9495646ec546d6134e54f","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"koota-composite-trait-aspects","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.201745037499999},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":12.732614916666662},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.27947105},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.63375475}},"judge_cost":0.00034535999999999996,"prompt_sha256":"69097162c0044bec1ecb29c56fffbeeb0482bc37e1a9495646ec546d6134e54f","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-composite-trait-aspects","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":11.201745037499999},"claude-opus-5":{"p":0.6666666666666666,"successes":2.0,"attempts":3,"published_mean_cost":12.732614916666662},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.27947105},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.63375475}},"judge_cost":0.00028888,"prompt_sha256":"69097162c0044bec1ecb29c56fffbeeb0482bc37e1a9495646ec546d6134e54f","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"koota-deferred-mutation-buffer","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.0729480375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.23873725},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.93707645},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.2453822500000005}},"judge_cost":0.00026725,"prompt_sha256":"ae5d96bc0da4f1b269c332af919b23629667d2906656944ce5bf7c067c833df4","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"koota-deferred-mutation-buffer","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.0729480375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.23873725},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.93707645},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.2453822500000005}},"judge_cost":0.00026412999999999995,"prompt_sha256":"ae5d96bc0da4f1b269c332af919b23629667d2906656944ce5bf7c067c833df4","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"koota-deferred-mutation-buffer","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.0729480375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.23873725},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.93707645},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.2453822500000005}},"judge_cost":0.00078185,"prompt_sha256":"ae5d96bc0da4f1b269c332af919b23629667d2906656944ce5bf7c067c833df4","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-deferred-mutation-buffer","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.0729480375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.23873725},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.93707645},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.2453822500000005}},"judge_cost":0.00038758000000000005,"prompt_sha256":"ae5d96bc0da4f1b269c332af919b23629667d2906656944ce5bf7c067c833df4","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-deferred-mutation-buffer","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.4},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.0729480375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.23873725},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.93707645},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.2453822500000005}},"judge_cost":0.00044115,"prompt_sha256":"ae5d96bc0da4f1b269c332af919b23629667d2906656944ce5bf7c067c833df4","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"koota-deferred-mutation-buffer","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.0729480375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.23873725},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.93707645},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.2453822500000005}},"judge_cost":0.00030002999999999996,"prompt_sha256":"ae5d96bc0da4f1b269c332af919b23629667d2906656944ce5bf7c067c833df4","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"koota-entity-snapshot-rollback","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.4897806125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.837099312500001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.51870835},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.48850975}},"judge_cost":0.00064825,"prompt_sha256":"e57edf84d0e7e89f98f1ac0fcab7099f98b2420141be165aed453f8ac22191e5","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-entity-snapshot-rollback","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.4897806125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.837099312500001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.51870835},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.48850975}},"judge_cost":0.00039645,"prompt_sha256":"e57edf84d0e7e89f98f1ac0fcab7099f98b2420141be165aed453f8ac22191e5","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"koota-entity-snapshot-rollback","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.4897806125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.837099312500001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.51870835},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.48850975}},"judge_cost":0.00033573,"prompt_sha256":"e57edf84d0e7e89f98f1ac0fcab7099f98b2420141be165aed453f8ac22191e5","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"koota-entity-snapshot-rollback","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.4897806125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.837099312500001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.51870835},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.48850975}},"judge_cost":0.00038195,"prompt_sha256":"e57edf84d0e7e89f98f1ac0fcab7099f98b2420141be165aed453f8ac22191e5","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"koota-entity-snapshot-rollback","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.4897806125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.837099312500001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.51870835},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.48850975}},"judge_cost":0.00033758,"prompt_sha256":"e57edf84d0e7e89f98f1ac0fcab7099f98b2420141be165aed453f8ac22191e5","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-entity-snapshot-rollback","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.4897806125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.837099312500001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.51870835},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.48850975}},"judge_cost":0.00034403,"prompt_sha256":"e57edf84d0e7e89f98f1ac0fcab7099f98b2420141be165aed453f8ac22191e5","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"koota-pair-relation-tracking","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":18.410053312500004},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":14.705961187499996},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3118224},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.8106842499999996}},"judge_cost":0.0002554,"prompt_sha256":"1eb7ea611c1e85b0e3a9724c431aed441da4c3993ee76458d14a3a2c5a4c1f82","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"koota-pair-relation-tracking","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":18.410053312500004},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":14.705961187499996},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3118224},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.8106842499999996}},"judge_cost":0.00024028,"prompt_sha256":"1eb7ea611c1e85b0e3a9724c431aed441da4c3993ee76458d14a3a2c5a4c1f82","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"koota-pair-relation-tracking","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":18.410053312500004},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":14.705961187499996},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3118224},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.8106842499999996}},"judge_cost":0.0007543999999999999,"prompt_sha256":"1eb7ea611c1e85b0e3a9724c431aed441da4c3993ee76458d14a3a2c5a4c1f82","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-pair-relation-tracking","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":18.410053312500004},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":14.705961187499996},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3118224},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.8106842499999996}},"judge_cost":0.0002998,"prompt_sha256":"1eb7ea611c1e85b0e3a9724c431aed441da4c3993ee76458d14a3a2c5a4c1f82","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-pair-relation-tracking","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.6},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":18.410053312500004},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":14.705961187499996},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3118224},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.8106842499999996}},"judge_cost":0.0002697,"prompt_sha256":"1eb7ea611c1e85b0e3a9724c431aed441da4c3993ee76458d14a3a2c5a4c1f82","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"koota-pair-relation-tracking","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":18.410053312500004},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":14.705961187499996},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3118224},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.8106842499999996}},"judge_cost":0.00024858,"prompt_sha256":"1eb7ea611c1e85b0e3a9724c431aed441da4c3993ee76458d14a3a2c5a4c1f82","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"koota-query-predicates","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.2305820375},"claude-opus-5":{"p":1.0,"successes":3.0,"attempts":3,"published_mean_cost":7.895168000000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3549556},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.01295925}},"judge_cost":0.00026365000000000004,"prompt_sha256":"d68f65f15159ed962daa9d60036a7fa9ebf7a116074570d24a2dae01db9b8718","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"koota-query-predicates","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.2305820375},"claude-opus-5":{"p":1.0,"successes":3.0,"attempts":3,"published_mean_cost":7.895168000000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3549556},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.01295925}},"judge_cost":0.00027613,"prompt_sha256":"d68f65f15159ed962daa9d60036a7fa9ebf7a116074570d24a2dae01db9b8718","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"koota-query-predicates","repository":"pmndrs/koota","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.2305820375},"claude-opus-5":{"p":1.0,"successes":3.0,"attempts":3,"published_mean_cost":7.895168000000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3549556},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.01295925}},"judge_cost":0.0008214499999999999,"prompt_sha256":"d68f65f15159ed962daa9d60036a7fa9ebf7a116074570d24a2dae01db9b8718","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-query-predicates","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.2305820375},"claude-opus-5":{"p":1.0,"successes":3.0,"attempts":3,"published_mean_cost":7.895168000000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3549556},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.01295925}},"judge_cost":0.0003067,"prompt_sha256":"d68f65f15159ed962daa9d60036a7fa9ebf7a116074570d24a2dae01db9b8718","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"koota-query-predicates","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.2305820375},"claude-opus-5":{"p":1.0,"successes":3.0,"attempts":3,"published_mean_cost":7.895168000000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3549556},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.01295925}},"judge_cost":0.00025275000000000004,"prompt_sha256":"d68f65f15159ed962daa9d60036a7fa9ebf7a116074570d24a2dae01db9b8718","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"koota-query-predicates","repository":"pmndrs/koota","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":10.2305820375},"claude-opus-5":{"p":1.0,"successes":3.0,"attempts":3,"published_mean_cost":7.895168000000001},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.3549556},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.01295925}},"judge_cost":0.00025083,"prompt_sha256":"d68f65f15159ed962daa9d60036a7fa9ebf7a116074570d24a2dae01db9b8718","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"kysely-window-grouping-helpers","repository":"kysely-org/kysely","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.896742387500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.151689999999999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7998569},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.305157}},"judge_cost":0.00037909999999999994,"prompt_sha256":"8daed6ff131069ff5351de42a4a687f73b2855a49dd8965e6b6ff6f74155c94a","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"kysely-window-grouping-helpers","repository":"kysely-org/kysely","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.896742387500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.151689999999999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7998569},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.305157}},"judge_cost":0.00037717999999999996,"prompt_sha256":"8daed6ff131069ff5351de42a4a687f73b2855a49dd8965e6b6ff6f74155c94a","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"kysely-window-grouping-helpers","repository":"kysely-org/kysely","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.896742387500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.151689999999999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7998569},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.305157}},"judge_cost":0.0008408999999999999,"prompt_sha256":"8daed6ff131069ff5351de42a4a687f73b2855a49dd8965e6b6ff6f74155c94a","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kysely-window-grouping-helpers","repository":"kysely-org/kysely","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.57},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.896742387500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.151689999999999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7998569},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.305157}},"judge_cost":0.00022703999999999997,"prompt_sha256":"8daed6ff131069ff5351de42a4a687f73b2855a49dd8965e6b6ff6f74155c94a","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"kysely-window-grouping-helpers","repository":"kysely-org/kysely","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.896742387500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.151689999999999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7998569},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.305157}},"judge_cost":0.00037299999999999996,"prompt_sha256":"8daed6ff131069ff5351de42a4a687f73b2855a49dd8965e6b6ff6f74155c94a","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"kysely-window-grouping-helpers","repository":"kysely-org/kysely","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.896742387500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.151689999999999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7998569},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.305157}},"judge_cost":0.00042867999999999997,"prompt_sha256":"8daed6ff131069ff5351de42a4a687f73b2855a49dd8965e6b6ff6f74155c94a","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"langchain-request-coalescing","repository":"langchain-ai/langchain","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.1976026625000005},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.644780375000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6061362},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.22481525}},"judge_cost":0.00067595,"prompt_sha256":"ac04265522f06c0e61dd310a5a535e1c9052d0c186cdec8d32d67d3b284fdd5e","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"langchain-request-coalescing","repository":"langchain-ai/langchain","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.1976026625000005},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.644780375000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6061362},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.22481525}},"judge_cost":0.00031975,"prompt_sha256":"ac04265522f06c0e61dd310a5a535e1c9052d0c186cdec8d32d67d3b284fdd5e","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"langchain-request-coalescing","repository":"langchain-ai/langchain","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.1976026625000005},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.644780375000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6061362},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.22481525}},"judge_cost":0.00032863,"prompt_sha256":"ac04265522f06c0e61dd310a5a535e1c9052d0c186cdec8d32d67d3b284fdd5e","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"langchain-request-coalescing","repository":"langchain-ai/langchain","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.1976026625000005},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.644780375000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6061362},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.22481525}},"judge_cost":0.00031845,"prompt_sha256":"ac04265522f06c0e61dd310a5a535e1c9052d0c186cdec8d32d67d3b284fdd5e","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"langchain-request-coalescing","repository":"langchain-ai/langchain","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.1976026625000005},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.644780375000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6061362},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.22481525}},"judge_cost":0.00024009999999999998,"prompt_sha256":"ac04265522f06c0e61dd310a5a535e1c9052d0c186cdec8d32d67d3b284fdd5e","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"langchain-request-coalescing","repository":"langchain-ai/langchain","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.2},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.1976026625000005},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.644780375000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6061362},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.22481525}},"judge_cost":0.00033573,"prompt_sha256":"ac04265522f06c0e61dd310a5a535e1c9052d0c186cdec8d32d67d3b284fdd5e","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"mashumaro-flattened-dataclass-fields","repository":"Fatal1ty/mashumaro","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.921612687499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.183088750000001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.89050645},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8877224999999997}},"judge_cost":0.0002218,"prompt_sha256":"5f66ff4adc6b58764571fef3a7f7f4d801a2df4a22874233ebcba0a750533157","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"mashumaro-flattened-dataclass-fields","repository":"Fatal1ty/mashumaro","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.921612687499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.183088750000001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.89050645},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8877224999999997}},"judge_cost":0.0005876,"prompt_sha256":"5f66ff4adc6b58764571fef3a7f7f4d801a2df4a22874233ebcba0a750533157","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"mashumaro-flattened-dataclass-fields","repository":"Fatal1ty/mashumaro","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.921612687499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.183088750000001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.89050645},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8877224999999997}},"judge_cost":0.00027267999999999997,"prompt_sha256":"5f66ff4adc6b58764571fef3a7f7f4d801a2df4a22874233ebcba0a750533157","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"mashumaro-flattened-dataclass-fields","repository":"Fatal1ty/mashumaro","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.921612687499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.183088750000001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.89050645},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8877224999999997}},"judge_cost":0.0002409,"prompt_sha256":"5f66ff4adc6b58764571fef3a7f7f4d801a2df4a22874233ebcba0a750533157","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"mashumaro-flattened-dataclass-fields","repository":"Fatal1ty/mashumaro","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.921612687499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.183088750000001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.89050645},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8877224999999997}},"judge_cost":0.00028636,"prompt_sha256":"5f66ff4adc6b58764571fef3a7f7f4d801a2df4a22874233ebcba0a750533157","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"mashumaro-flattened-dataclass-fields","repository":"Fatal1ty/mashumaro","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.921612687499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":9.183088750000001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.89050645},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8877224999999997}},"judge_cost":0.00025217999999999996,"prompt_sha256":"5f66ff4adc6b58764571fef3a7f7f4d801a2df4a22874233ebcba0a750533157","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"meriyah-explicit-resource-declarations","repository":"meriyah/meriyah","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":15.014096100000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.062850500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.33523625},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.977781}},"judge_cost":0.00054185,"prompt_sha256":"9d4920e4edf646036b41614a0166007b3c7d4eedc03efecd0eab22f69a26c96d","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"meriyah-explicit-resource-declarations","repository":"meriyah/meriyah","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":15.014096100000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.062850500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.33523625},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.977781}},"judge_cost":0.00032485,"prompt_sha256":"9d4920e4edf646036b41614a0166007b3c7d4eedc03efecd0eab22f69a26c96d","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"meriyah-explicit-resource-declarations","repository":"meriyah/meriyah","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":15.014096100000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.062850500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.33523625},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.977781}},"judge_cost":0.0002479,"prompt_sha256":"9d4920e4edf646036b41614a0166007b3c7d4eedc03efecd0eab22f69a26c96d","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"meriyah-explicit-resource-declarations","repository":"meriyah/meriyah","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.48},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":15.014096100000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.062850500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.33523625},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.977781}},"judge_cost":0.00037573,"prompt_sha256":"9d4920e4edf646036b41614a0166007b3c7d4eedc03efecd0eab22f69a26c96d","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"meriyah-explicit-resource-declarations","repository":"meriyah/meriyah","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":15.014096100000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.062850500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.33523625},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.977781}},"judge_cost":0.00038114999999999997,"prompt_sha256":"9d4920e4edf646036b41614a0166007b3c7d4eedc03efecd0eab22f69a26c96d","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"meriyah-explicit-resource-declarations","repository":"meriyah/meriyah","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.44},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":15.014096100000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":7.062850500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.33523625},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.977781}},"judge_cost":0.00034083,"prompt_sha256":"9d4920e4edf646036b41614a0166007b3c7d4eedc03efecd0eab22f69a26c96d","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"mnamer-daemon-watch-lifecycle","repository":"jkwill87/mnamer","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7608568124999993},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.330700125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.36348525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.932641}},"judge_cost":0.0004211,"prompt_sha256":"0dbe6a59913fc69e9eaee625725b67137097d02e78a570f947eeaaf795ba2fdd","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"mnamer-daemon-watch-lifecycle","repository":"jkwill87/mnamer","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7608568124999993},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.330700125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.36348525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.932641}},"judge_cost":0.00040837999999999996,"prompt_sha256":"0dbe6a59913fc69e9eaee625725b67137097d02e78a570f947eeaaf795ba2fdd","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"mnamer-daemon-watch-lifecycle","repository":"jkwill87/mnamer","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7608568124999993},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.330700125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.36348525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.932641}},"judge_cost":0.0008253,"prompt_sha256":"0dbe6a59913fc69e9eaee625725b67137097d02e78a570f947eeaaf795ba2fdd","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"mnamer-daemon-watch-lifecycle","repository":"jkwill87/mnamer","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7608568124999993},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.330700125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.36348525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.932641}},"judge_cost":0.00037659999999999994,"prompt_sha256":"0dbe6a59913fc69e9eaee625725b67137097d02e78a570f947eeaaf795ba2fdd","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"mnamer-daemon-watch-lifecycle","repository":"jkwill87/mnamer","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7608568124999993},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.330700125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.36348525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.932641}},"judge_cost":0.00036528,"prompt_sha256":"0dbe6a59913fc69e9eaee625725b67137097d02e78a570f947eeaaf795ba2fdd","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"mnamer-daemon-watch-lifecycle","repository":"jkwill87/mnamer","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.44},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.7608568124999993},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.330700125},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.36348525},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.932641}},"judge_cost":0.00044068,"prompt_sha256":"0dbe6a59913fc69e9eaee625725b67137097d02e78a570f947eeaaf795ba2fdd","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"mobly-grouped-test-barriers","repository":"google/mobly","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.154647712499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9665081250000003},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5491566999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.24949125}},"judge_cost":0.0006272,"prompt_sha256":"97ffb321d232dba7f0165f5932071affb0f131b0882555a9678e370d86a21d32","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"mobly-grouped-test-barriers","repository":"google/mobly","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.154647712499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9665081250000003},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5491566999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.24949125}},"judge_cost":0.00035860000000000004,"prompt_sha256":"97ffb321d232dba7f0165f5932071affb0f131b0882555a9678e370d86a21d32","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"mobly-grouped-test-barriers","repository":"google/mobly","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.4},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.154647712499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9665081250000003},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5491566999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.24949125}},"judge_cost":0.0003501,"prompt_sha256":"97ffb321d232dba7f0165f5932071affb0f131b0882555a9678e370d86a21d32","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"mobly-grouped-test-barriers","repository":"google/mobly","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.154647712499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9665081250000003},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5491566999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.24949125}},"judge_cost":0.00037108,"prompt_sha256":"97ffb321d232dba7f0165f5932071affb0f131b0882555a9678e370d86a21d32","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"mobly-grouped-test-barriers","repository":"google/mobly","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.154647712499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9665081250000003},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5491566999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.24949125}},"judge_cost":0.00031516,"prompt_sha256":"97ffb321d232dba7f0165f5932071affb0f131b0882555a9678e370d86a21d32","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"mobly-grouped-test-barriers","repository":"google/mobly","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.154647712499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9665081250000003},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.5491566999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.24949125}},"judge_cost":0.00035057999999999997,"prompt_sha256":"97ffb321d232dba7f0165f5932071affb0f131b0882555a9678e370d86a21d32","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"narwhals-rolling-window-suite","repository":"narwhals-dev/narwhals","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.79},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.9603392125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.623060312499998},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.95645275},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.861325}},"judge_cost":0.00067,"prompt_sha256":"943615327d267cdd03f71830a99f329007adf1a6ac9f18aa9a84f2778e89cf8d","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"narwhals-rolling-window-suite","repository":"narwhals-dev/narwhals","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.7},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.9603392125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.623060312499998},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.95645275},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.861325}},"judge_cost":0.000441,"prompt_sha256":"943615327d267cdd03f71830a99f329007adf1a6ac9f18aa9a84f2778e89cf8d","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"narwhals-rolling-window-suite","repository":"narwhals-dev/narwhals","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.78},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.9603392125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.623060312499998},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.95645275},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.861325}},"judge_cost":0.00025699999999999996,"prompt_sha256":"943615327d267cdd03f71830a99f329007adf1a6ac9f18aa9a84f2778e89cf8d","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"narwhals-rolling-window-suite","repository":"narwhals-dev/narwhals","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.48},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.9603392125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.623060312499998},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.95645275},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.861325}},"judge_cost":0.00040188,"prompt_sha256":"943615327d267cdd03f71830a99f329007adf1a6ac9f18aa9a84f2778e89cf8d","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"narwhals-rolling-window-suite","repository":"narwhals-dev/narwhals","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.9603392125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.623060312499998},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.95645275},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.861325}},"judge_cost":0.0005741,"prompt_sha256":"943615327d267cdd03f71830a99f329007adf1a6ac9f18aa9a84f2778e89cf8d","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"numba-stencil-boundary-modes","repository":"numba/numba","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":7.704347887499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.381939375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.0827757},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.0178709999999995}},"judge_cost":0.00062635,"prompt_sha256":"1564cbb67b3c2c7a310a16507bd5569e86adeb4a68e9db5f7c38f59c18bcbad5","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"narwhals-rolling-window-suite","repository":"narwhals-dev/narwhals","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.5},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.9603392125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":10.623060312499998},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.95645275},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.861325}},"judge_cost":0.00048338,"prompt_sha256":"943615327d267cdd03f71830a99f329007adf1a6ac9f18aa9a84f2778e89cf8d","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"numba-stencil-boundary-modes","repository":"numba/numba","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":7.704347887499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.381939375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.0827757},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.0178709999999995}},"judge_cost":0.00027975,"prompt_sha256":"1564cbb67b3c2c7a310a16507bd5569e86adeb4a68e9db5f7c38f59c18bcbad5","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"numba-stencil-boundary-modes","repository":"numba/numba","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":7.704347887499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.381939375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.0827757},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.0178709999999995}},"judge_cost":0.00030845,"prompt_sha256":"1564cbb67b3c2c7a310a16507bd5569e86adeb4a68e9db5f7c38f59c18bcbad5","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"numba-stencil-boundary-modes","repository":"numba/numba","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":7.704347887499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.381939375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.0827757},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.0178709999999995}},"judge_cost":0.0003329,"prompt_sha256":"1564cbb67b3c2c7a310a16507bd5569e86adeb4a68e9db5f7c38f59c18bcbad5","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"numba-stencil-boundary-modes","repository":"numba/numba","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":7.704347887499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.381939375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.0827757},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.0178709999999995}},"judge_cost":0.00033663,"prompt_sha256":"1564cbb67b3c2c7a310a16507bd5569e86adeb4a68e9db5f7c38f59c18bcbad5","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"numba-stencil-boundary-modes","repository":"numba/numba","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":7.704347887499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.381939375},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.0827757},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.0178709999999995}},"judge_cost":0.00029092999999999995,"prompt_sha256":"1564cbb67b3c2c7a310a16507bd5569e86adeb4a68e9db5f7c38f59c18bcbad5","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"obsidian-linter-auto-table-of-contents","repository":"platers/obsidian-linter","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.8624497499999997},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2348280625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6677165},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.10461575}},"judge_cost":0.0005933,"prompt_sha256":"7b5f88a8f9d9975b2772451566379dde74a79865aba224a51cf2c47690579917","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"obsidian-linter-auto-table-of-contents","repository":"platers/obsidian-linter","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.8624497499999997},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2348280625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6677165},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.10461575}},"judge_cost":0.00034389999999999996,"prompt_sha256":"7b5f88a8f9d9975b2772451566379dde74a79865aba224a51cf2c47690579917","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"obsidian-linter-auto-table-of-contents","repository":"platers/obsidian-linter","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.8624497499999997},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2348280625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6677165},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.10461575}},"judge_cost":0.00031318,"prompt_sha256":"7b5f88a8f9d9975b2772451566379dde74a79865aba224a51cf2c47690579917","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"obsidian-linter-auto-table-of-contents","repository":"platers/obsidian-linter","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.8624497499999997},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2348280625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6677165},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.10461575}},"judge_cost":0.0003606,"prompt_sha256":"7b5f88a8f9d9975b2772451566379dde74a79865aba224a51cf2c47690579917","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"obsidian-linter-auto-table-of-contents","repository":"platers/obsidian-linter","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.8624497499999997},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2348280625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6677165},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.10461575}},"judge_cost":0.0003184,"prompt_sha256":"7b5f88a8f9d9975b2772451566379dde74a79865aba224a51cf2c47690579917","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"obsidian-linter-auto-table-of-contents","repository":"platers/obsidian-linter","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.8624497499999997},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.2348280625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6677165},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.10461575}},"judge_cost":0.00035747999999999997,"prompt_sha256":"7b5f88a8f9d9975b2772451566379dde74a79865aba224a51cf2c47690579917","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"obsidian-linter-link-format-conversion","repository":"platers/obsidian-linter","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.71383965},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.379831375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.55286475},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.65233525}},"judge_cost":0.00034215,"prompt_sha256":"1e7a1672d255c0a513c381224040c1819a5469fe73f7de0f6bf398d41a703d4c","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"obsidian-linter-link-format-conversion","repository":"platers/obsidian-linter","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.71383965},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.379831375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.55286475},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.65233525}},"judge_cost":0.00081715,"prompt_sha256":"1e7a1672d255c0a513c381224040c1819a5469fe73f7de0f6bf398d41a703d4c","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"obsidian-linter-link-format-conversion","repository":"platers/obsidian-linter","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.71383965},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.379831375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.55286475},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.65233525}},"judge_cost":0.00037983,"prompt_sha256":"1e7a1672d255c0a513c381224040c1819a5469fe73f7de0f6bf398d41a703d4c","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"obsidian-linter-link-format-conversion","repository":"platers/obsidian-linter","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.71383965},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.379831375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.55286475},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.65233525}},"judge_cost":0.0004049,"prompt_sha256":"1e7a1672d255c0a513c381224040c1819a5469fe73f7de0f6bf398d41a703d4c","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"obsidian-linter-link-format-conversion","repository":"platers/obsidian-linter","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.71383965},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.379831375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.55286475},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.65233525}},"judge_cost":0.00033125,"prompt_sha256":"1e7a1672d255c0a513c381224040c1819a5469fe73f7de0f6bf398d41a703d4c","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"obsidian-linter-link-format-conversion","repository":"platers/obsidian-linter","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.71383965},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.379831375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.55286475},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.65233525}},"judge_cost":0.00036052999999999997,"prompt_sha256":"1e7a1672d255c0a513c381224040c1819a5469fe73f7de0f6bf398d41a703d4c","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"obsidian-linter-scoped-ignore-markers","repository":"platers/obsidian-linter","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.5924990125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.97511725},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7014876},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.62354325}},"judge_cost":0.00034218,"prompt_sha256":"0423fa145af5afbb5292279e2ec9575e45d02ffca93693b9ba1364f2b7582b05","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"obsidian-linter-scoped-ignore-markers","repository":"platers/obsidian-linter","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.5924990125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.97511725},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7014876},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.62354325}},"judge_cost":0.00034889999999999997,"prompt_sha256":"0423fa145af5afbb5292279e2ec9575e45d02ffca93693b9ba1364f2b7582b05","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"obsidian-linter-scoped-ignore-markers","repository":"platers/obsidian-linter","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.65},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.5924990125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.97511725},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7014876},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.62354325}},"judge_cost":0.0008239,"prompt_sha256":"0423fa145af5afbb5292279e2ec9575e45d02ffca93693b9ba1364f2b7582b05","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"obsidian-linter-scoped-ignore-markers","repository":"platers/obsidian-linter","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.5924990125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.97511725},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7014876},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.62354325}},"judge_cost":0.0003308,"prompt_sha256":"0423fa145af5afbb5292279e2ec9575e45d02ffca93693b9ba1364f2b7582b05","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"obsidian-linter-scoped-ignore-markers","repository":"platers/obsidian-linter","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.7},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.5924990125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.97511725},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7014876},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.62354325}},"judge_cost":0.00037904,"prompt_sha256":"0423fa145af5afbb5292279e2ec9575e45d02ffca93693b9ba1364f2b7582b05","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"obsidian-linter-scoped-ignore-markers","repository":"platers/obsidian-linter","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.5924990125},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.97511725},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.7014876},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.62354325}},"judge_cost":0.00033608,"prompt_sha256":"0423fa145af5afbb5292279e2ec9575e45d02ffca93693b9ba1364f2b7582b05","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"ofetch-per-origin-circuit-breaker","repository":"unjs/ofetch","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.5496932999999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.063002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.50274465},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.97394075}},"judge_cost":0.00072445,"prompt_sha256":"06e87b9de6da67677b73ba3c6cfcfddde6237bcb0942b5de436c1a4d9fa815bb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ofetch-per-origin-circuit-breaker","repository":"unjs/ofetch","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.5496932999999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.063002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.50274465},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.97394075}},"judge_cost":0.00041144999999999995,"prompt_sha256":"06e87b9de6da67677b73ba3c6cfcfddde6237bcb0942b5de436c1a4d9fa815bb","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"ofetch-per-origin-circuit-breaker","repository":"unjs/ofetch","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.5496932999999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.063002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.50274465},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.97394075}},"judge_cost":0.00042992999999999997,"prompt_sha256":"06e87b9de6da67677b73ba3c6cfcfddde6237bcb0942b5de436c1a4d9fa815bb","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"ofetch-per-origin-circuit-breaker","repository":"unjs/ofetch","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.67},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.5496932999999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.063002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.50274465},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.97394075}},"judge_cost":0.00035702,"prompt_sha256":"06e87b9de6da67677b73ba3c6cfcfddde6237bcb0942b5de436c1a4d9fa815bb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ofetch-per-origin-circuit-breaker","repository":"unjs/ofetch","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.5496932999999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.063002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.50274465},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.97394075}},"judge_cost":0.00045095,"prompt_sha256":"06e87b9de6da67677b73ba3c6cfcfddde6237bcb0942b5de436c1a4d9fa815bb","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"ofetch-per-origin-circuit-breaker","repository":"unjs/ofetch","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.5496932999999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.063002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.50274465},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.97394075}},"judge_cost":0.00043103,"prompt_sha256":"06e87b9de6da67677b73ba3c6cfcfddde6237bcb0942b5de436c1a4d9fa815bb","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"onedump-dump-encryption-pipeline","repository":"liweiyi88/onedump","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.9186128499999993},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.0771030625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.36157225},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.1492115}},"judge_cost":0.00036748,"prompt_sha256":"4ac8baac679e4e62ba819d687680411cfeb1338fba502bfce16f21452928d543","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"onedump-dump-encryption-pipeline","repository":"liweiyi88/onedump","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.9186128499999993},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.0771030625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.36157225},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.1492115}},"judge_cost":0.00033219999999999994,"prompt_sha256":"4ac8baac679e4e62ba819d687680411cfeb1338fba502bfce16f21452928d543","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"onedump-dump-encryption-pipeline","repository":"liweiyi88/onedump","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.9186128499999993},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.0771030625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.36157225},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.1492115}},"judge_cost":0.0008012,"prompt_sha256":"4ac8baac679e4e62ba819d687680411cfeb1338fba502bfce16f21452928d543","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"onedump-dump-encryption-pipeline","repository":"liweiyi88/onedump","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.9186128499999993},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.0771030625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.36157225},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.1492115}},"judge_cost":0.00024868,"prompt_sha256":"4ac8baac679e4e62ba819d687680411cfeb1338fba502bfce16f21452928d543","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"onedump-dump-encryption-pipeline","repository":"liweiyi88/onedump","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.9186128499999993},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.0771030625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.36157225},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.1492115}},"judge_cost":0.0003357,"prompt_sha256":"4ac8baac679e4e62ba819d687680411cfeb1338fba502bfce16f21452928d543","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"opa-rego-rule-profiling","repository":"open-policy-agent/opa","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.169484875000001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.852623875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.60134115},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5559849999999997}},"judge_cost":0.00063225,"prompt_sha256":"e4697907ba3ec7af3bb41ac354e4358153def71a58ef75af799ae4484a8bacc9","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"opa-rego-rule-profiling","repository":"open-policy-agent/opa","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.169484875000001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.852623875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.60134115},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5559849999999997}},"judge_cost":0.00063225,"prompt_sha256":"e4697907ba3ec7af3bb41ac354e4358153def71a58ef75af799ae4484a8bacc9","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"onedump-dump-encryption-pipeline","repository":"liweiyi88/onedump","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.9186128499999993},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.0771030625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.36157225},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.1492115}},"judge_cost":0.00037338,"prompt_sha256":"4ac8baac679e4e62ba819d687680411cfeb1338fba502bfce16f21452928d543","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"opa-rego-rule-profiling","repository":"open-policy-agent/opa","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.6},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.169484875000001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.852623875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.60134115},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5559849999999997}},"judge_cost":0.00049565,"prompt_sha256":"e4697907ba3ec7af3bb41ac354e4358153def71a58ef75af799ae4484a8bacc9","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"opa-rego-rule-profiling","repository":"open-policy-agent/opa","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.169484875000001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.852623875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.60134115},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5559849999999997}},"judge_cost":0.00040373000000000004,"prompt_sha256":"e4697907ba3ec7af3bb41ac354e4358153def71a58ef75af799ae4484a8bacc9","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"opa-rego-rule-profiling","repository":"open-policy-agent/opa","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.169484875000001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.852623875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.60134115},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5559849999999997}},"judge_cost":0.00034675,"prompt_sha256":"e4697907ba3ec7af3bb41ac354e4358153def71a58ef75af799ae4484a8bacc9","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"opa-rego-rule-profiling","repository":"open-policy-agent/opa","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.3},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.169484875000001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.852623875},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.60134115},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":3.5559849999999997}},"judge_cost":0.00039883,"prompt_sha256":"e4697907ba3ec7af3bb41ac354e4358153def71a58ef75af799ae4484a8bacc9","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"opa-template-string-reconstruction","repository":"open-policy-agent/opa","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.004536624999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.541801499999998},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.91249975},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.108401}},"judge_cost":0.0006003,"prompt_sha256":"f6201d97b5c37d5f2fe3f7496d8139a2f17a27a2565bd5cae858d189402b14d4","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"opa-template-string-reconstruction","repository":"open-policy-agent/opa","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.004536624999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.541801499999998},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.91249975},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.108401}},"judge_cost":0.0002957,"prompt_sha256":"f6201d97b5c37d5f2fe3f7496d8139a2f17a27a2565bd5cae858d189402b14d4","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"opa-template-string-reconstruction","repository":"open-policy-agent/opa","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.3},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.004536624999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.541801499999998},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.91249975},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.108401}},"judge_cost":0.00026618,"prompt_sha256":"f6201d97b5c37d5f2fe3f7496d8139a2f17a27a2565bd5cae858d189402b14d4","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"opa-template-string-reconstruction","repository":"open-policy-agent/opa","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.004536624999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.541801499999998},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.91249975},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.108401}},"judge_cost":0.00035136,"prompt_sha256":"f6201d97b5c37d5f2fe3f7496d8139a2f17a27a2565bd5cae858d189402b14d4","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"opa-template-string-reconstruction","repository":"open-policy-agent/opa","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.44},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.004536624999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.541801499999998},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.91249975},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.108401}},"judge_cost":0.0002944,"prompt_sha256":"f6201d97b5c37d5f2fe3f7496d8139a2f17a27a2565bd5cae858d189402b14d4","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"opa-template-string-reconstruction","repository":"open-policy-agent/opa","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.004536624999997},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.541801499999998},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.91249975},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.108401}},"judge_cost":0.00029727999999999997,"prompt_sha256":"f6201d97b5c37d5f2fe3f7496d8139a2f17a27a2565bd5cae858d189402b14d4","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"optique-conditional-option-dependencies","repository":"dahlia/optique","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.931402237499999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.262896062500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.18439865},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.995551}},"judge_cost":0.00038875,"prompt_sha256":"6172841f3a7c523111776cfde29dc22d4306debbc7cca8074eec3c668fb67dc0","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"optique-conditional-option-dependencies","repository":"dahlia/optique","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.67},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.931402237499999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.262896062500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.18439865},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.995551}},"judge_cost":0.0007389499999999999,"prompt_sha256":"6172841f3a7c523111776cfde29dc22d4306debbc7cca8074eec3c668fb67dc0","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"optique-conditional-option-dependencies","repository":"dahlia/optique","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.931402237499999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.262896062500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.18439865},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.995551}},"judge_cost":0.00038203,"prompt_sha256":"6172841f3a7c523111776cfde29dc22d4306debbc7cca8074eec3c668fb67dc0","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"optique-conditional-option-dependencies","repository":"dahlia/optique","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.931402237499999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.262896062500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.18439865},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.995551}},"judge_cost":0.00035265,"prompt_sha256":"6172841f3a7c523111776cfde29dc22d4306debbc7cca8074eec3c668fb67dc0","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"optique-conditional-option-dependencies","repository":"dahlia/optique","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.931402237499999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.262896062500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.18439865},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.995551}},"judge_cost":0.00037113,"prompt_sha256":"6172841f3a7c523111776cfde29dc22d4306debbc7cca8074eec3c668fb67dc0","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"optique-conditional-option-dependencies","repository":"dahlia/optique","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":9.931402237499999},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.262896062500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.18439865},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.995551}},"judge_cost":0.00040281999999999997,"prompt_sha256":"6172841f3a7c523111776cfde29dc22d4306debbc7cca8074eec3c668fb67dc0","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"oxvg-structural-selector-preservation","repository":"noahbald/oxvg","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":14.249077987500003},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":10.27719725},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.4684042},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.17807}},"judge_cost":0.0005921500000000001,"prompt_sha256":"ec27db0352007346c7d0b3bf8c83efe91d9c980d37485e96bec5631192361192","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"oxvg-structural-selector-preservation","repository":"noahbald/oxvg","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":14.249077987500003},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":10.27719725},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.4684042},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.17807}},"judge_cost":0.00028203,"prompt_sha256":"ec27db0352007346c7d0b3bf8c83efe91d9c980d37485e96bec5631192361192","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"oxvg-structural-selector-preservation","repository":"noahbald/oxvg","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.55},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":14.249077987500003},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":10.27719725},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.4684042},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.17807}},"judge_cost":0.00023882,"prompt_sha256":"ec27db0352007346c7d0b3bf8c83efe91d9c980d37485e96bec5631192361192","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"oxvg-structural-selector-preservation","repository":"noahbald/oxvg","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":14.249077987500003},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":10.27719725},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.4684042},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.17807}},"judge_cost":0.00033315,"prompt_sha256":"ec27db0352007346c7d0b3bf8c83efe91d9c980d37485e96bec5631192361192","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"oxvg-structural-selector-preservation","repository":"noahbald/oxvg","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":14.249077987500003},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":10.27719725},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.4684042},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.17807}},"judge_cost":0.00029945,"prompt_sha256":"ec27db0352007346c7d0b3bf8c83efe91d9c980d37485e96bec5631192361192","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"oxvg-structural-selector-preservation","repository":"noahbald/oxvg","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":14.249077987500003},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":10.27719725},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":1.4684042},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":6.17807}},"judge_cost":0.00029033,"prompt_sha256":"ec27db0352007346c7d0b3bf8c83efe91d9c980d37485e96bec5631192361192","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"participle-grammar-conflict-analysis","repository":"alecthomas/participle","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.0819728625000007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9560529375000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4253161},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.80140525}},"judge_cost":0.00074685,"prompt_sha256":"cd798074e70b01a049c8544078ecd70e5c515f43d497a834bb07f180ff64f106","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"participle-grammar-conflict-analysis","repository":"alecthomas/participle","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.0819728625000007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9560529375000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4253161},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.80140525}},"judge_cost":0.00044345,"prompt_sha256":"cd798074e70b01a049c8544078ecd70e5c515f43d497a834bb07f180ff64f106","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"participle-grammar-conflict-analysis","repository":"alecthomas/participle","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.0819728625000007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9560529375000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4253161},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.80140525}},"judge_cost":0.00039233,"prompt_sha256":"cd798074e70b01a049c8544078ecd70e5c515f43d497a834bb07f180ff64f106","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"participle-grammar-conflict-analysis","repository":"alecthomas/participle","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.44},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.0819728625000007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9560529375000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4253161},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.80140525}},"judge_cost":0.00043135,"prompt_sha256":"cd798074e70b01a049c8544078ecd70e5c515f43d497a834bb07f180ff64f106","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"participle-grammar-conflict-analysis","repository":"alecthomas/participle","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.0819728625000007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9560529375000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4253161},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.80140525}},"judge_cost":0.00030414,"prompt_sha256":"cd798074e70b01a049c8544078ecd70e5c515f43d497a834bb07f180ff64f106","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"participle-grammar-conflict-analysis","repository":"alecthomas/participle","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.0819728625000007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.9560529375000004},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.4253161},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.80140525}},"judge_cost":0.00040783,"prompt_sha256":"cd798074e70b01a049c8544078ecd70e5c515f43d497a834bb07f180ff64f106","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"pebble-durability-wait-apis","repository":"cockroachdb/pebble","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":12.1537930875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.302153624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.564021},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":7.2811725}},"judge_cost":0.00070545,"prompt_sha256":"6dbc4cf690c7a41a156cba0ef1bae086429bb7523bbc164792065afc13eacb6d","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"pebble-durability-wait-apis","repository":"cockroachdb/pebble","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":12.1537930875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.302153624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.564021},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":7.2811725}},"judge_cost":0.00039245,"prompt_sha256":"6dbc4cf690c7a41a156cba0ef1bae086429bb7523bbc164792065afc13eacb6d","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"pebble-durability-wait-apis","repository":"cockroachdb/pebble","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":12.1537930875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.302153624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.564021},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":7.2811725}},"judge_cost":0.00038213,"prompt_sha256":"6dbc4cf690c7a41a156cba0ef1bae086429bb7523bbc164792065afc13eacb6d","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"pebble-durability-wait-apis","repository":"cockroachdb/pebble","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":12.1537930875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.302153624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.564021},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":7.2811725}},"judge_cost":0.00038515,"prompt_sha256":"6dbc4cf690c7a41a156cba0ef1bae086429bb7523bbc164792065afc13eacb6d","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"pebble-durability-wait-apis","repository":"cockroachdb/pebble","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":12.1537930875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.302153624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.564021},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":7.2811725}},"judge_cost":0.0003759,"prompt_sha256":"6dbc4cf690c7a41a156cba0ef1bae086429bb7523bbc164792065afc13eacb6d","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"pebble-durability-wait-apis","repository":"cockroachdb/pebble","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":12.1537930875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.302153624999999},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.564021},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":7.2811725}},"judge_cost":0.00037483,"prompt_sha256":"6dbc4cf690c7a41a156cba0ef1bae086429bb7523bbc164792065afc13eacb6d","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"pest-character-class-coalescing","repository":"pest-parser/pest","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.9866765375},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.422536749999999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.65985095},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.053428}},"judge_cost":0.0006118,"prompt_sha256":"8a9974e9e5809d6d8f89a29137d16dc6f763c7e45b0251f9074d6b75b7127edc","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"pest-character-class-coalescing","repository":"pest-parser/pest","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.9866765375},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.422536749999999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.65985095},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.053428}},"judge_cost":0.00031919999999999995,"prompt_sha256":"8a9974e9e5809d6d8f89a29137d16dc6f763c7e45b0251f9074d6b75b7127edc","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"pest-character-class-coalescing","repository":"pest-parser/pest","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.9866765375},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.422536749999999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.65985095},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.053428}},"judge_cost":0.00027528,"prompt_sha256":"8a9974e9e5809d6d8f89a29137d16dc6f763c7e45b0251f9074d6b75b7127edc","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"pest-character-class-coalescing","repository":"pest-parser/pest","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.9866765375},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.422536749999999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.65985095},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.053428}},"judge_cost":0.00022268,"prompt_sha256":"8a9974e9e5809d6d8f89a29137d16dc6f763c7e45b0251f9074d6b75b7127edc","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"pest-character-class-coalescing","repository":"pest-parser/pest","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.9866765375},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.422536749999999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.65985095},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.053428}},"judge_cost":0.00035270000000000006,"prompt_sha256":"8a9974e9e5809d6d8f89a29137d16dc6f763c7e45b0251f9074d6b75b7127edc","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"pest-character-class-coalescing","repository":"pest-parser/pest","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.9866765375},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.422536749999999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.65985095},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.053428}},"judge_cost":0.00036278,"prompt_sha256":"8a9974e9e5809d6d8f89a29137d16dc6f763c7e45b0251f9074d6b75b7127edc","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"prometheus-transactional-reload-status","repository":"prometheus/prometheus","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":10.3697440125},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.7880458125},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.9606196499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.6720645}},"judge_cost":0.00061515,"prompt_sha256":"c0222692d795fe759dc03746e896f6e4a214652c08d2b9125cf1f9745c26af86","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"prometheus-transactional-reload-status","repository":"prometheus/prometheus","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":10.3697440125},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.7880458125},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.9606196499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.6720645}},"judge_cost":0.00036215,"prompt_sha256":"c0222692d795fe759dc03746e896f6e4a214652c08d2b9125cf1f9745c26af86","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"prometheus-transactional-reload-status","repository":"prometheus/prometheus","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":10.3697440125},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.7880458125},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.9606196499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.6720645}},"judge_cost":0.00029183,"prompt_sha256":"c0222692d795fe759dc03746e896f6e4a214652c08d2b9125cf1f9745c26af86","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"prometheus-transactional-reload-status","repository":"prometheus/prometheus","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":10.3697440125},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.7880458125},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.9606196499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.6720645}},"judge_cost":0.00035005,"prompt_sha256":"c0222692d795fe759dc03746e896f6e4a214652c08d2b9125cf1f9745c26af86","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"prometheus-transactional-reload-status","repository":"prometheus/prometheus","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":10.3697440125},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.7880458125},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.9606196499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.6720645}},"judge_cost":0.00028073999999999997,"prompt_sha256":"c0222692d795fe759dc03746e896f6e4a214652c08d2b9125cf1f9745c26af86","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"prometheus-transactional-reload-status","repository":"prometheus/prometheus","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":10.3697440125},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":6.7880458125},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.9606196499999999},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.6720645}},"judge_cost":0.00031092999999999995,"prompt_sha256":"c0222692d795fe759dc03746e896f6e4a214652c08d2b9125cf1f9745c26af86","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"prometheus-typed-label-sorting","repository":"prometheus/prometheus","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.548145412499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.337477750000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.4373964},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.31254825}},"judge_cost":0.000629,"prompt_sha256":"78e29fc339cd1d1388cc276482f64a94a6b79e25fa6413157ce30803bbfb6a3b","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"prometheus-typed-label-sorting","repository":"prometheus/prometheus","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.548145412499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.337477750000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.4373964},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.31254825}},"judge_cost":0.00030688,"prompt_sha256":"78e29fc339cd1d1388cc276482f64a94a6b79e25fa6413157ce30803bbfb6a3b","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"prometheus-typed-label-sorting","repository":"prometheus/prometheus","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.548145412499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.337477750000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.4373964},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.31254825}},"judge_cost":0.0003748,"prompt_sha256":"78e29fc339cd1d1388cc276482f64a94a6b79e25fa6413157ce30803bbfb6a3b","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"prometheus-typed-label-sorting","repository":"prometheus/prometheus","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.548145412499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.337477750000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.4373964},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.31254825}},"judge_cost":0.00032355999999999997,"prompt_sha256":"78e29fc339cd1d1388cc276482f64a94a6b79e25fa6413157ce30803bbfb6a3b","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"prometheus-typed-label-sorting","repository":"prometheus/prometheus","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.29},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.548145412499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.337477750000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.4373964},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.31254825}},"judge_cost":0.00026958,"prompt_sha256":"78e29fc339cd1d1388cc276482f64a94a6b79e25fa6413157ce30803bbfb6a3b","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"prometheus-typed-label-sorting","repository":"prometheus/prometheus","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.48},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.548145412499999},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.337477750000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.4373964},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.31254825}},"judge_cost":0.00038789999999999994,"prompt_sha256":"78e29fc339cd1d1388cc276482f64a94a6b79e25fa6413157ce30803bbfb6a3b","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"psd-tools-blend-range-api","repository":"psd-tools/psd-tools","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9871412874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0132506875000002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.54440515},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.6487775}},"judge_cost":0.00072605,"prompt_sha256":"478c045993c22418064ba83e02229b061a41de5aa4aff2afe7582faeb40e7928","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"psd-tools-blend-range-api","repository":"psd-tools/psd-tools","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9871412874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0132506875000002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.54440515},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.6487775}},"judge_cost":0.00043585,"prompt_sha256":"478c045993c22418064ba83e02229b061a41de5aa4aff2afe7582faeb40e7928","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"psd-tools-blend-range-api","repository":"psd-tools/psd-tools","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9871412874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0132506875000002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.54440515},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.6487775}},"judge_cost":0.00034873,"prompt_sha256":"478c045993c22418064ba83e02229b061a41de5aa4aff2afe7582faeb40e7928","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"psd-tools-blend-range-api","repository":"psd-tools/psd-tools","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9871412874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0132506875000002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.54440515},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.6487775}},"judge_cost":0.00038175000000000004,"prompt_sha256":"478c045993c22418064ba83e02229b061a41de5aa4aff2afe7582faeb40e7928","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"psd-tools-blend-range-api","repository":"psd-tools/psd-tools","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9871412874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0132506875000002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.54440515},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.6487775}},"judge_cost":0.00034983,"prompt_sha256":"478c045993c22418064ba83e02229b061a41de5aa4aff2afe7582faeb40e7928","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"psd-tools-blend-range-api","repository":"psd-tools/psd-tools","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.9871412874999996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0132506875000002},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.54440515},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.6487775}},"judge_cost":0.0004051,"prompt_sha256":"478c045993c22418064ba83e02229b061a41de5aa4aff2afe7582faeb40e7928","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"pwntools-tube-multiplexing","repository":"Gallopsled/pwntools","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.310633475},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.868138375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.38497804999999996},"gpt-5-6-sol":{"p":0.3333333333333333,"successes":1.0,"attempts":3,"published_mean_cost":2.24598}},"judge_cost":0.0004437,"prompt_sha256":"0edcfd810216f499fed9d9212d2bce997be7268f50203e0ccca6e82251eead2b","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"pwntools-tube-multiplexing","repository":"Gallopsled/pwntools","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.310633475},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.868138375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.38497804999999996},"gpt-5-6-sol":{"p":0.3333333333333333,"successes":1.0,"attempts":3,"published_mean_cost":2.24598}},"judge_cost":0.0008947,"prompt_sha256":"0edcfd810216f499fed9d9212d2bce997be7268f50203e0ccca6e82251eead2b","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"pwntools-tube-multiplexing","repository":"Gallopsled/pwntools","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.22},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.310633475},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.868138375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.38497804999999996},"gpt-5-6-sol":{"p":0.3333333333333333,"successes":1.0,"attempts":3,"published_mean_cost":2.24598}},"judge_cost":0.00046098,"prompt_sha256":"0edcfd810216f499fed9d9212d2bce997be7268f50203e0ccca6e82251eead2b","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"pwntools-tube-multiplexing","repository":"Gallopsled/pwntools","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.310633475},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.868138375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.38497804999999996},"gpt-5-6-sol":{"p":0.3333333333333333,"successes":1.0,"attempts":3,"published_mean_cost":2.24598}},"judge_cost":0.00028327999999999996,"prompt_sha256":"0edcfd810216f499fed9d9212d2bce997be7268f50203e0ccca6e82251eead2b","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"pwntools-tube-multiplexing","repository":"Gallopsled/pwntools","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.310633475},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.868138375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.38497804999999996},"gpt-5-6-sol":{"p":0.3333333333333333,"successes":1.0,"attempts":3,"published_mean_cost":2.24598}},"judge_cost":0.00042080000000000004,"prompt_sha256":"0edcfd810216f499fed9d9212d2bce997be7268f50203e0ccca6e82251eead2b","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"python-statemachine-state-data-scoping","repository":"fgmacedo/python-statemachine","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":19.155471337500007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.668228625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.15482235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.94229375}},"judge_cost":0.00030345000000000003,"prompt_sha256":"c15a4c0ca5b4f54ca3102599f986c80734072b58f5c0cc29cd85cc96ff53e269","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"pwntools-tube-multiplexing","repository":"Gallopsled/pwntools","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.310633475},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.868138375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.38497804999999996},"gpt-5-6-sol":{"p":0.3333333333333333,"successes":1.0,"attempts":3,"published_mean_cost":2.24598}},"judge_cost":0.00047888,"prompt_sha256":"0edcfd810216f499fed9d9212d2bce997be7268f50203e0ccca6e82251eead2b","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"python-statemachine-state-data-scoping","repository":"fgmacedo/python-statemachine","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.16},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":19.155471337500007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.668228625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.15482235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.94229375}},"judge_cost":0.00030272999999999997,"prompt_sha256":"c15a4c0ca5b4f54ca3102599f986c80734072b58f5c0cc29cd85cc96ff53e269","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"python-statemachine-state-data-scoping","repository":"fgmacedo/python-statemachine","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":19.155471337500007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.668228625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.15482235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.94229375}},"judge_cost":0.00034406,"prompt_sha256":"c15a4c0ca5b4f54ca3102599f986c80734072b58f5c0cc29cd85cc96ff53e269","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"python-statemachine-state-data-scoping","repository":"fgmacedo/python-statemachine","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":19.155471337500007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.668228625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.15482235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.94229375}},"judge_cost":0.00033334999999999994,"prompt_sha256":"c15a4c0ca5b4f54ca3102599f986c80734072b58f5c0cc29cd85cc96ff53e269","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"python-statemachine-state-data-scoping","repository":"fgmacedo/python-statemachine","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.3},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":19.155471337500007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.668228625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.15482235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.94229375}},"judge_cost":0.0008036499999999999,"prompt_sha256":"c15a4c0ca5b4f54ca3102599f986c80734072b58f5c0cc29cd85cc96ff53e269","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"query-persist-restored-query-state","repository":"TanStack/query","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":12.003167587500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.502348999999999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.72741365},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8881862499999995}},"judge_cost":0.0006039999999999999,"prompt_sha256":"2c296a3db2570c335a933213a22a036e3890ac8e44ba4ccf400f7434c88afc27","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"query-persist-restored-query-state","repository":"TanStack/query","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":12.003167587500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.502348999999999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.72741365},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8881862499999995}},"judge_cost":0.0006039999999999999,"prompt_sha256":"2c296a3db2570c335a933213a22a036e3890ac8e44ba4ccf400f7434c88afc27","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"python-statemachine-state-data-scoping","repository":"fgmacedo/python-statemachine","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":19.155471337500007},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.668228625},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.15482235},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.94229375}},"judge_cost":0.00031703,"prompt_sha256":"c15a4c0ca5b4f54ca3102599f986c80734072b58f5c0cc29cd85cc96ff53e269","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"query-persist-restored-query-state","repository":"TanStack/query","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":12.003167587500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.502348999999999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.72741365},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8881862499999995}},"judge_cost":0.0003618,"prompt_sha256":"2c296a3db2570c335a933213a22a036e3890ac8e44ba4ccf400f7434c88afc27","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"query-persist-restored-query-state","repository":"TanStack/query","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":12.003167587500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.502348999999999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.72741365},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8881862499999995}},"judge_cost":0.00032388,"prompt_sha256":"2c296a3db2570c335a933213a22a036e3890ac8e44ba4ccf400f7434c88afc27","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"query-persist-restored-query-state","repository":"TanStack/query","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":12.003167587500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.502348999999999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.72741365},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8881862499999995}},"judge_cost":0.0003101,"prompt_sha256":"2c296a3db2570c335a933213a22a036e3890ac8e44ba4ccf400f7434c88afc27","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"query-persist-restored-query-state","repository":"TanStack/query","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.29},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":12.003167587500002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.502348999999999},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.72741365},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.8881862499999995}},"judge_cost":0.00033458,"prompt_sha256":"2c296a3db2570c335a933213a22a036e3890ac8e44ba4ccf400f7434c88afc27","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"quill-shared-toolbar-focus","repository":"slab/quill","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.967617837499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.036870500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.81456705},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.8601865}},"judge_cost":0.00030265,"prompt_sha256":"465b0b8c710c882bbe1ef2b0debde0aab665119cd779b8c97e5382e8977710fe","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"quill-shared-toolbar-focus","repository":"slab/quill","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.967617837499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.036870500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.81456705},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.8601865}},"judge_cost":0.00028393,"prompt_sha256":"465b0b8c710c882bbe1ef2b0debde0aab665119cd779b8c97e5382e8977710fe","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"quill-shared-toolbar-focus","repository":"slab/quill","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.967617837499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.036870500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.81456705},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.8601865}},"judge_cost":0.00074885,"prompt_sha256":"465b0b8c710c882bbe1ef2b0debde0aab665119cd779b8c97e5382e8977710fe","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"quill-shared-toolbar-focus","repository":"slab/quill","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.967617837499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.036870500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.81456705},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.8601865}},"judge_cost":0.00040966,"prompt_sha256":"465b0b8c710c882bbe1ef2b0debde0aab665119cd779b8c97e5382e8977710fe","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"quill-shared-toolbar-focus","repository":"slab/quill","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.967617837499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.036870500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.81456705},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.8601865}},"judge_cost":0.00030135000000000003,"prompt_sha256":"465b0b8c710c882bbe1ef2b0debde0aab665119cd779b8c97e5382e8977710fe","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"quill-shared-toolbar-focus","repository":"slab/quill","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.3},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.967617837499999},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":8.036870500000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.81456705},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":3.8601865}},"judge_cost":0.00029823,"prompt_sha256":"465b0b8c710c882bbe1ef2b0debde0aab665119cd779b8c97e5382e8977710fe","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"returns-validated-error-accumulation","repository":"dry-python/returns","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.1034396000000015},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.649151375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5209558999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1329475}},"judge_cost":0.0006945,"prompt_sha256":"c38035d596efaa4a537ae23416cd27f4d60c4ecbfa0b92732412e2c9aeb812ad","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"returns-validated-error-accumulation","repository":"dry-python/returns","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.1034396000000015},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.649151375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5209558999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1329475}},"judge_cost":0.00037598,"prompt_sha256":"c38035d596efaa4a537ae23416cd27f4d60c4ecbfa0b92732412e2c9aeb812ad","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"returns-validated-error-accumulation","repository":"dry-python/returns","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.1034396000000015},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.649151375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5209558999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1329475}},"judge_cost":0.0003815,"prompt_sha256":"c38035d596efaa4a537ae23416cd27f4d60c4ecbfa0b92732412e2c9aeb812ad","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"returns-validated-error-accumulation","repository":"dry-python/returns","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.1034396000000015},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.649151375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5209558999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1329475}},"judge_cost":0.0003934,"prompt_sha256":"c38035d596efaa4a537ae23416cd27f4d60c4ecbfa0b92732412e2c9aeb812ad","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"returns-validated-error-accumulation","repository":"dry-python/returns","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.1034396000000015},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.649151375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5209558999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1329475}},"judge_cost":0.00036864,"prompt_sha256":"c38035d596efaa4a537ae23416cd27f4d60c4ecbfa0b92732412e2c9aeb812ad","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"returns-validated-error-accumulation","repository":"dry-python/returns","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.1034396000000015},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.649151375000001},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.5209558999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.1329475}},"judge_cost":0.00036268,"prompt_sha256":"c38035d596efaa4a537ae23416cd27f4d60c4ecbfa0b92732412e2c9aeb812ad","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"scc-bounded-memory-spilling","repository":"boyter/scc","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.55},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.616863400000001},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.86845525},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8125177},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.66710475}},"judge_cost":0.0003111,"prompt_sha256":"3829847d57131f5dd0d22fc05d9ef4570175a792823352c34435197eff063d69","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"scc-bounded-memory-spilling","repository":"boyter/scc","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.616863400000001},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.86845525},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8125177},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.66710475}},"judge_cost":0.00031278,"prompt_sha256":"3829847d57131f5dd0d22fc05d9ef4570175a792823352c34435197eff063d69","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"scc-bounded-memory-spilling","repository":"boyter/scc","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.616863400000001},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.86845525},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8125177},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.66710475}},"judge_cost":0.0008113,"prompt_sha256":"3829847d57131f5dd0d22fc05d9ef4570175a792823352c34435197eff063d69","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"scc-bounded-memory-spilling","repository":"boyter/scc","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.616863400000001},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.86845525},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8125177},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.66710475}},"judge_cost":0.00026816,"prompt_sha256":"3829847d57131f5dd0d22fc05d9ef4570175a792823352c34435197eff063d69","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"scc-bounded-memory-spilling","repository":"boyter/scc","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.616863400000001},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.86845525},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8125177},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.66710475}},"judge_cost":0.0003074,"prompt_sha256":"3829847d57131f5dd0d22fc05d9ef4570175a792823352c34435197eff063d69","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"scc-bounded-memory-spilling","repository":"boyter/scc","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":5.616863400000001},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.86845525},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.8125177},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.66710475}},"judge_cost":0.00031267999999999997,"prompt_sha256":"3829847d57131f5dd0d22fc05d9ef4570175a792823352c34435197eff063d69","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"scriggo-method-declarations","repository":"open2b/scriggo","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.45},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":21.481656787500008},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":34.793388625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.05822185},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.63043225}},"judge_cost":0.00027079999999999997,"prompt_sha256":"f48b22b3028419292444c88688ae99108ec889922e89208f0346ee9a65f7907c","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"scriggo-method-declarations","repository":"open2b/scriggo","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":21.481656787500008},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":34.793388625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.05822185},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.63043225}},"judge_cost":0.0006594,"prompt_sha256":"f48b22b3028419292444c88688ae99108ec889922e89208f0346ee9a65f7907c","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"scriggo-method-declarations","repository":"open2b/scriggo","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":21.481656787500008},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":34.793388625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.05822185},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.63043225}},"judge_cost":0.00024728,"prompt_sha256":"f48b22b3028419292444c88688ae99108ec889922e89208f0346ee9a65f7907c","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"scriggo-method-declarations","repository":"open2b/scriggo","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":21.481656787500008},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":34.793388625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.05822185},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.63043225}},"judge_cost":0.00031788,"prompt_sha256":"f48b22b3028419292444c88688ae99108ec889922e89208f0346ee9a65f7907c","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"scriggo-method-declarations","repository":"open2b/scriggo","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":21.481656787500008},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":34.793388625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.05822185},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.63043225}},"judge_cost":0.0003067,"prompt_sha256":"f48b22b3028419292444c88688ae99108ec889922e89208f0346ee9a65f7907c","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"scriggo-method-declarations","repository":"open2b/scriggo","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":21.481656787500008},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":34.793388625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.05822185},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":9.63043225}},"judge_cost":0.00029637999999999995,"prompt_sha256":"f48b22b3028419292444c88688ae99108ec889922e89208f0346ee9a65f7907c","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"skrub-duration-encoding","repository":"skrub-data/skrub","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":11.0484173625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.3979016249999985},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6421701},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.78561075}},"judge_cost":0.0008376999999999999,"prompt_sha256":"4c340aa2c24c3cac4940788602a5caac361fe6d962018f3d9f1bbf495f6ae5c6","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"skrub-duration-encoding","repository":"skrub-data/skrub","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":11.0484173625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.3979016249999985},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6421701},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.78561075}},"judge_cost":0.0003927,"prompt_sha256":"4c340aa2c24c3cac4940788602a5caac361fe6d962018f3d9f1bbf495f6ae5c6","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"skrub-duration-encoding","repository":"skrub-data/skrub","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":11.0484173625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.3979016249999985},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6421701},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.78561075}},"judge_cost":0.00041598000000000004,"prompt_sha256":"4c340aa2c24c3cac4940788602a5caac361fe6d962018f3d9f1bbf495f6ae5c6","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"skrub-duration-encoding","repository":"skrub-data/skrub","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":11.0484173625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.3979016249999985},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6421701},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.78561075}},"judge_cost":0.00022784000000000001,"prompt_sha256":"4c340aa2c24c3cac4940788602a5caac361fe6d962018f3d9f1bbf495f6ae5c6","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"skrub-duration-encoding","repository":"skrub-data/skrub","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":11.0484173625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.3979016249999985},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6421701},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.78561075}},"judge_cost":0.0005042,"prompt_sha256":"4c340aa2c24c3cac4940788602a5caac361fe6d962018f3d9f1bbf495f6ae5c6","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"skrub-duration-encoding","repository":"skrub-data/skrub","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":11.0484173625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.3979016249999985},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.6421701},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.78561075}},"judge_cost":0.00040028000000000004,"prompt_sha256":"4c340aa2c24c3cac4940788602a5caac361fe6d962018f3d9f1bbf495f6ae5c6","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"sql-formatter-bigquery-pipe-formatting","repository":"sql-formatter-org/sql-formatter","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.866080175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0219229375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.7214135},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.9532034999999999}},"judge_cost":0.00059915,"prompt_sha256":"89aad598b3dd77ca17aa71d66192a25a296b8455cf35ca2983c12e06d495a339","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"sql-formatter-bigquery-pipe-formatting","repository":"sql-formatter-org/sql-formatter","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.866080175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0219229375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.7214135},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.9532034999999999}},"judge_cost":0.00059915,"prompt_sha256":"89aad598b3dd77ca17aa71d66192a25a296b8455cf35ca2983c12e06d495a339","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"sql-formatter-bigquery-pipe-formatting","repository":"sql-formatter-org/sql-formatter","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.866080175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0219229375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.7214135},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.9532034999999999}},"judge_cost":0.00033895,"prompt_sha256":"89aad598b3dd77ca17aa71d66192a25a296b8455cf35ca2983c12e06d495a339","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"sql-formatter-bigquery-pipe-formatting","repository":"sql-formatter-org/sql-formatter","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.866080175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0219229375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.7214135},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.9532034999999999}},"judge_cost":0.00031063,"prompt_sha256":"89aad598b3dd77ca17aa71d66192a25a296b8455cf35ca2983c12e06d495a339","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"sql-formatter-bigquery-pipe-formatting","repository":"sql-formatter-org/sql-formatter","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.866080175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0219229375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.7214135},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.9532034999999999}},"judge_cost":0.00043005,"prompt_sha256":"89aad598b3dd77ca17aa71d66192a25a296b8455cf35ca2983c12e06d495a339","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"sql-formatter-bigquery-pipe-formatting","repository":"sql-formatter-org/sql-formatter","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.866080175},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0219229375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.7214135},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.9532034999999999}},"judge_cost":0.00034413,"prompt_sha256":"89aad598b3dd77ca17aa71d66192a25a296b8455cf35ca2983c12e06d495a339","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"sqlfmt-create-table-ddl-formatting","repository":"tconbeer/sqlfmt","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":13.139929274999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.664535062499999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6696067},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.5593635}},"judge_cost":0.00061735,"prompt_sha256":"bb73f8298a815f141a434e7da265ed88afa78a7ea2a6117ef22685691e1a4528","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"sqlfmt-create-table-ddl-formatting","repository":"tconbeer/sqlfmt","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.57},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":13.139929274999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.664535062499999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6696067},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.5593635}},"judge_cost":0.00045075000000000003,"prompt_sha256":"bb73f8298a815f141a434e7da265ed88afa78a7ea2a6117ef22685691e1a4528","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"sqlfmt-create-table-ddl-formatting","repository":"tconbeer/sqlfmt","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.39},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":13.139929274999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.664535062499999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6696067},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.5593635}},"judge_cost":0.00040323,"prompt_sha256":"bb73f8298a815f141a434e7da265ed88afa78a7ea2a6117ef22685691e1a4528","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"sqlfmt-create-table-ddl-formatting","repository":"tconbeer/sqlfmt","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":13.139929274999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.664535062499999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6696067},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.5593635}},"judge_cost":0.00036233,"prompt_sha256":"bb73f8298a815f141a434e7da265ed88afa78a7ea2a6117ef22685691e1a4528","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"sqlfmt-create-table-ddl-formatting","repository":"tconbeer/sqlfmt","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":13.139929274999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.664535062499999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6696067},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.5593635}},"judge_cost":0.00037865,"prompt_sha256":"bb73f8298a815f141a434e7da265ed88afa78a7ea2a6117ef22685691e1a4528","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"sqlfmt-create-table-ddl-formatting","repository":"tconbeer/sqlfmt","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":13.139929274999998},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.664535062499999},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.6696067},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.5593635}},"judge_cost":0.0005321,"prompt_sha256":"bb73f8298a815f141a434e7da265ed88afa78a7ea2a6117ef22685691e1a4528","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"sqlite-utils-safe-import-checkpoints","repository":"simonw/sqlite-utils","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.7725545875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.719796687500001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6236709},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.57732825}},"judge_cost":0.00071395,"prompt_sha256":"4a742d96724e03968f276d3ddc1217605b41efd22b55ea4ea0ba8f1986a849e5","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"sqlite-utils-safe-import-checkpoints","repository":"simonw/sqlite-utils","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.7725545875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.719796687500001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6236709},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.57732825}},"judge_cost":0.00037935,"prompt_sha256":"4a742d96724e03968f276d3ddc1217605b41efd22b55ea4ea0ba8f1986a849e5","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"sqlite-utils-safe-import-checkpoints","repository":"simonw/sqlite-utils","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.7725545875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.719796687500001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6236709},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.57732825}},"judge_cost":0.00040622999999999994,"prompt_sha256":"4a742d96724e03968f276d3ddc1217605b41efd22b55ea4ea0ba8f1986a849e5","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"sqlite-utils-safe-import-checkpoints","repository":"simonw/sqlite-utils","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.7725545875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.719796687500001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6236709},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.57732825}},"judge_cost":0.00034205,"prompt_sha256":"4a742d96724e03968f276d3ddc1217605b41efd22b55ea4ea0ba8f1986a849e5","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"sqlite-utils-safe-import-checkpoints","repository":"simonw/sqlite-utils","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.7725545875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.719796687500001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6236709},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.57732825}},"judge_cost":0.00030842,"prompt_sha256":"4a742d96724e03968f276d3ddc1217605b41efd22b55ea4ea0ba8f1986a849e5","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"sqlite-utils-safe-import-checkpoints","repository":"simonw/sqlite-utils","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.22},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":8.7725545875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":5.719796687500001},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.6236709},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.57732825}},"judge_cost":0.00035453,"prompt_sha256":"4a742d96724e03968f276d3ddc1217605b41efd22b55ea4ea0ba8f1986a849e5","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"superjson-error-stack-serialization","repository":"flightcontrolhq/superjson","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.273712375},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1119650625000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4787391},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.17064}},"judge_cost":0.0008539499999999999,"prompt_sha256":"6af1d4d8d1da84d34ad3a4884aa50fdecb5712d490f04e6585fedadc565affbf","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"superjson-error-stack-serialization","repository":"flightcontrolhq/superjson","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.273712375},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1119650625000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4787391},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.17064}},"judge_cost":0.00045215,"prompt_sha256":"6af1d4d8d1da84d34ad3a4884aa50fdecb5712d490f04e6585fedadc565affbf","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"superjson-error-stack-serialization","repository":"flightcontrolhq/superjson","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.273712375},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1119650625000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4787391},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.17064}},"judge_cost":0.00046342999999999997,"prompt_sha256":"6af1d4d8d1da84d34ad3a4884aa50fdecb5712d490f04e6585fedadc565affbf","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"superjson-error-stack-serialization","repository":"flightcontrolhq/superjson","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.273712375},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1119650625000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4787391},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.17064}},"judge_cost":0.00049525,"prompt_sha256":"6af1d4d8d1da84d34ad3a4884aa50fdecb5712d490f04e6585fedadc565affbf","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"superjson-error-stack-serialization","repository":"flightcontrolhq/superjson","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.273712375},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1119650625000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4787391},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.17064}},"judge_cost":0.00046217999999999997,"prompt_sha256":"6af1d4d8d1da84d34ad3a4884aa50fdecb5712d490f04e6585fedadc565affbf","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"superjson-error-stack-serialization","repository":"flightcontrolhq/superjson","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.273712375},"claude-opus-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.1119650625000004},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4787391},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.17064}},"judge_cost":0.00044532999999999997,"prompt_sha256":"6af1d4d8d1da84d34ad3a4884aa50fdecb5712d490f04e6585fedadc565affbf","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"task-task-graph-export","repository":"go-task/task","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.220470062499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.358380125},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7499352},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.43252725}},"judge_cost":0.0003785,"prompt_sha256":"4cf6edbb0b70eee719964a5919075eff3d9283c07f4820e40c31fdecad7a431b","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"task-task-graph-export","repository":"go-task/task","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.220470062499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.358380125},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7499352},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.43252725}},"judge_cost":0.0007419,"prompt_sha256":"4cf6edbb0b70eee719964a5919075eff3d9283c07f4820e40c31fdecad7a431b","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"task-task-graph-export","repository":"go-task/task","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.220470062499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.358380125},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7499352},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.43252725}},"judge_cost":0.00040178,"prompt_sha256":"4cf6edbb0b70eee719964a5919075eff3d9283c07f4820e40c31fdecad7a431b","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"task-task-graph-export","repository":"go-task/task","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.220470062499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.358380125},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7499352},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.43252725}},"judge_cost":0.00036264,"prompt_sha256":"4cf6edbb0b70eee719964a5919075eff3d9283c07f4820e40c31fdecad7a431b","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"task-task-graph-export","repository":"go-task/task","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.220470062499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.358380125},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7499352},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.43252725}},"judge_cost":0.00034648,"prompt_sha256":"4cf6edbb0b70eee719964a5919075eff3d9283c07f4820e40c31fdecad7a431b","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"task-task-graph-export","repository":"go-task/task","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.44},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.220470062499999},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":4.358380125},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.7499352},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.43252725}},"judge_cost":0.0004144,"prompt_sha256":"4cf6edbb0b70eee719964a5919075eff3d9283c07f4820e40c31fdecad7a431b","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"tengo-callable-instance-isolation","repository":"d5/tengo","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.8967287125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.462519187500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.89558915},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.48977825}},"judge_cost":0.0002829,"prompt_sha256":"9a1c6f390c5e44e02595af5e7a7d1b8d9013134d68c8a66329793bf3bd9d0b40","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"tengo-callable-instance-isolation","repository":"d5/tengo","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.8967287125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.462519187500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.89558915},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.48977825}},"judge_cost":0.00032058,"prompt_sha256":"9a1c6f390c5e44e02595af5e7a7d1b8d9013134d68c8a66329793bf3bd9d0b40","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"tengo-callable-instance-isolation","repository":"d5/tengo","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.8967287125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.462519187500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.89558915},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.48977825}},"judge_cost":0.00040135999999999997,"prompt_sha256":"9a1c6f390c5e44e02595af5e7a7d1b8d9013134d68c8a66329793bf3bd9d0b40","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"tengo-callable-instance-isolation","repository":"d5/tengo","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.8967287125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.462519187500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.89558915},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.48977825}},"judge_cost":0.0009055000000000001,"prompt_sha256":"9a1c6f390c5e44e02595af5e7a7d1b8d9013134d68c8a66329793bf3bd9d0b40","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"tengo-callable-instance-isolation","repository":"d5/tengo","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.8967287125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.462519187500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.89558915},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.48977825}},"judge_cost":0.00033079999999999996,"prompt_sha256":"9a1c6f390c5e44e02595af5e7a7d1b8d9013134d68c8a66329793bf3bd9d0b40","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"tengo-callable-instance-isolation","repository":"d5/tengo","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.31},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":5.8967287125},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.462519187500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.89558915},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.48977825}},"judge_cost":0.00027488,"prompt_sha256":"9a1c6f390c5e44e02595af5e7a7d1b8d9013134d68c8a66329793bf3bd9d0b40","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"tengo-destructuring-bindings","repository":"d5/tengo","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.881783537500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.558828250000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.824843},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.25276775}},"judge_cost":0.0005235000000000001,"prompt_sha256":"3c8b5e7c48fab3f62db215df9ec7e978ad3b2ef979338a5ee8291facf138c5ac","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"tengo-destructuring-bindings","repository":"d5/tengo","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.59},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.881783537500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.558828250000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.824843},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.25276775}},"judge_cost":0.0003161,"prompt_sha256":"3c8b5e7c48fab3f62db215df9ec7e978ad3b2ef979338a5ee8291facf138c5ac","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"tengo-destructuring-bindings","repository":"d5/tengo","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.881783537500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.558828250000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.824843},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.25276775}},"judge_cost":0.00030577999999999996,"prompt_sha256":"3c8b5e7c48fab3f62db215df9ec7e978ad3b2ef979338a5ee8291facf138c5ac","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"tengo-destructuring-bindings","repository":"d5/tengo","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.881783537500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.558828250000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.824843},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.25276775}},"judge_cost":0.0003232,"prompt_sha256":"3c8b5e7c48fab3f62db215df9ec7e978ad3b2ef979338a5ee8291facf138c5ac","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"tengo-destructuring-bindings","repository":"d5/tengo","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.881783537500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.558828250000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.824843},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.25276775}},"judge_cost":0.00028584,"prompt_sha256":"3c8b5e7c48fab3f62db215df9ec7e978ad3b2ef979338a5ee8291facf138c5ac","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"tengo-destructuring-bindings","repository":"d5/tengo","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":5.881783537500001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.558828250000001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.824843},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.25276775}},"judge_cost":0.00032248,"prompt_sha256":"3c8b5e7c48fab3f62db215df9ec7e978ad3b2ef979338a5ee8291facf138c5ac","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"termenv-preserve-ansi-resets","repository":"muesli/termenv","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.3629515375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.2071166250000003},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.44982075},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.85725275}},"judge_cost":0.0006137499999999999,"prompt_sha256":"b4de3c1f5c0d6407b4a03234195900a16f071cc119f9f10316e811dde57bcddf","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"termenv-preserve-ansi-resets","repository":"muesli/termenv","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.3629515375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.2071166250000003},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.44982075},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.85725275}},"judge_cost":0.00034275,"prompt_sha256":"b4de3c1f5c0d6407b4a03234195900a16f071cc119f9f10316e811dde57bcddf","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"termenv-preserve-ansi-resets","repository":"muesli/termenv","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.3629515375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.2071166250000003},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.44982075},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.85725275}},"judge_cost":0.00031563,"prompt_sha256":"b4de3c1f5c0d6407b4a03234195900a16f071cc119f9f10316e811dde57bcddf","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"termenv-preserve-ansi-resets","repository":"muesli/termenv","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.57},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.3629515375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.2071166250000003},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.44982075},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.85725275}},"judge_cost":0.00032945,"prompt_sha256":"b4de3c1f5c0d6407b4a03234195900a16f071cc119f9f10316e811dde57bcddf","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"termenv-preserve-ansi-resets","repository":"muesli/termenv","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.3629515375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.2071166250000003},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.44982075},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.85725275}},"judge_cost":0.00030722,"prompt_sha256":"b4de3c1f5c0d6407b4a03234195900a16f071cc119f9f10316e811dde57bcddf","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"termenv-preserve-ansi-resets","repository":"muesli/termenv","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":2.3629515375},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.2071166250000003},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.44982075},"gpt-5-6-sol":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":1.85725275}},"judge_cost":0.00031433000000000003,"prompt_sha256":"b4de3c1f5c0d6407b4a03234195900a16f071cc119f9f10316e811dde57bcddf","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"testem-bail-on-test-failure","repository":"testem/testem","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.116776087499996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.8939395625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.8966757},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.50464425}},"judge_cost":0.0003606,"prompt_sha256":"bc65907216978dc609d7f903651dc2b664cd30b1c2b230833781b8fec8b9a879","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"testem-bail-on-test-failure","repository":"testem/testem","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.22},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.116776087499996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.8939395625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.8966757},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.50464425}},"judge_cost":0.00031668,"prompt_sha256":"bc65907216978dc609d7f903651dc2b664cd30b1c2b230833781b8fec8b9a879","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"testem-bail-on-test-failure","repository":"testem/testem","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.116776087499996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.8939395625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.8966757},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.50464425}},"judge_cost":0.00082,"prompt_sha256":"bc65907216978dc609d7f903651dc2b664cd30b1c2b230833781b8fec8b9a879","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"testem-bail-on-test-failure","repository":"testem/testem","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.116776087499996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.8939395625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.8966757},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.50464425}},"judge_cost":0.0003269,"prompt_sha256":"bc65907216978dc609d7f903651dc2b664cd30b1c2b230833781b8fec8b9a879","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"testem-bail-on-test-failure","repository":"testem/testem","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.116776087499996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.8939395625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.8966757},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.50464425}},"judge_cost":0.00030458,"prompt_sha256":"bc65907216978dc609d7f903651dc2b664cd30b1c2b230833781b8fec8b9a879","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"testem-bail-on-test-failure","repository":"testem/testem","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.116776087499996},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.8939395625},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.8966757},"gpt-5-6-sol":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.50464425}},"judge_cost":0.0003338,"prompt_sha256":"bc65907216978dc609d7f903651dc2b664cd30b1c2b230833781b8fec8b9a879","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"testem-per-launcher-reports","repository":"testem/testem","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.6358686874999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0512608125000007},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.5109294},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6565605}},"judge_cost":0.0006552,"prompt_sha256":"8a198363e797d4ced7998dac19ecfd2fe1c1cbb4ffa5f899486448fde511104a","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"testem-per-launcher-reports","repository":"testem/testem","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.6358686874999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0512608125000007},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.5109294},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6565605}},"judge_cost":0.00033308,"prompt_sha256":"8a198363e797d4ced7998dac19ecfd2fe1c1cbb4ffa5f899486448fde511104a","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"testem-per-launcher-reports","repository":"testem/testem","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.6358686874999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0512608125000007},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.5109294},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6565605}},"judge_cost":0.00039139999999999997,"prompt_sha256":"8a198363e797d4ced7998dac19ecfd2fe1c1cbb4ffa5f899486448fde511104a","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"testem-per-launcher-reports","repository":"testem/testem","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.6358686874999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0512608125000007},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.5109294},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6565605}},"judge_cost":0.00033009999999999995,"prompt_sha256":"8a198363e797d4ced7998dac19ecfd2fe1c1cbb4ffa5f899486448fde511104a","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"testem-per-launcher-reports","repository":"testem/testem","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.66},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.6358686874999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0512608125000007},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.5109294},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6565605}},"judge_cost":0.00033252,"prompt_sha256":"8a198363e797d4ced7998dac19ecfd2fe1c1cbb4ffa5f899486448fde511104a","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"testem-per-launcher-reports","repository":"testem/testem","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":4.6358686874999995},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.0512608125000007},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.5109294},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6565605}},"judge_cost":0.00033418,"prompt_sha256":"8a198363e797d4ced7998dac19ecfd2fe1c1cbb4ffa5f899486448fde511104a","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"textual-kitty-key-phases","repository":"Textualize/textual","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.6755076875000015},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.245598062499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5186022},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.99005375}},"judge_cost":0.0005971,"prompt_sha256":"ceb62e2b6f7424c8b3fd1f29ccf1cc2208114c38aade30c01fbdfbe9519fc27d","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"textual-kitty-key-phases","repository":"Textualize/textual","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.6755076875000015},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.245598062499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5186022},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.99005375}},"judge_cost":0.0003477,"prompt_sha256":"ceb62e2b6f7424c8b3fd1f29ccf1cc2208114c38aade30c01fbdfbe9519fc27d","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"textual-kitty-key-phases","repository":"Textualize/textual","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.6755076875000015},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.245598062499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5186022},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.99005375}},"judge_cost":0.00032538,"prompt_sha256":"ceb62e2b6f7424c8b3fd1f29ccf1cc2208114c38aade30c01fbdfbe9519fc27d","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"textual-kitty-key-phases","repository":"Textualize/textual","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.6755076875000015},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.245598062499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5186022},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.99005375}},"judge_cost":0.0003248,"prompt_sha256":"ceb62e2b6f7424c8b3fd1f29ccf1cc2208114c38aade30c01fbdfbe9519fc27d","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"textual-kitty-key-phases","repository":"Textualize/textual","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.6755076875000015},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.245598062499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5186022},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.99005375}},"judge_cost":0.00039104000000000005,"prompt_sha256":"ceb62e2b6f7424c8b3fd1f29ccf1cc2208114c38aade30c01fbdfbe9519fc27d","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"textual-kitty-key-phases","repository":"Textualize/textual","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.6755076875000015},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.245598062499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.5186022},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":2.99005375}},"judge_cost":0.00030848,"prompt_sha256":"ceb62e2b6f7424c8b3fd1f29ccf1cc2208114c38aade30c01fbdfbe9519fc27d","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"textual-richlog-follow-state","repository":"Textualize/textual","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.6},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.60838055},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":6.791731312500001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.70339635},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.4107765}},"judge_cost":0.0006300500000000001,"prompt_sha256":"f654618c6f8a567ff0f202b59fcc30f9865e8c2b7760183cd872963e423c1484","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"textual-richlog-follow-state","repository":"Textualize/textual","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.60838055},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":6.791731312500001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.70339635},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.4107765}},"judge_cost":0.00029305,"prompt_sha256":"f654618c6f8a567ff0f202b59fcc30f9865e8c2b7760183cd872963e423c1484","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"textual-richlog-follow-state","repository":"Textualize/textual","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.60838055},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":6.791731312500001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.70339635},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.4107765}},"judge_cost":0.00031152999999999997,"prompt_sha256":"f654618c6f8a567ff0f202b59fcc30f9865e8c2b7760183cd872963e423c1484","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"textual-richlog-follow-state","repository":"Textualize/textual","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.54},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.60838055},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":6.791731312500001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.70339635},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.4107765}},"judge_cost":0.00032774999999999997,"prompt_sha256":"f654618c6f8a567ff0f202b59fcc30f9865e8c2b7760183cd872963e423c1484","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"textual-richlog-follow-state","repository":"Textualize/textual","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.60838055},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":6.791731312500001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.70339635},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.4107765}},"judge_cost":0.00030422999999999995,"prompt_sha256":"f654618c6f8a567ff0f202b59fcc30f9865e8c2b7760183cd872963e423c1484","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"textual-richlog-follow-state","repository":"Textualize/textual","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":7.60838055},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":6.791731312500001},"gpt-5-6-luna":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":0.70339635},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.4107765}},"judge_cost":0.00045742,"prompt_sha256":"f654618c6f8a567ff0f202b59fcc30f9865e8c2b7760183cd872963e423c1484","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"tomlkit-toml-table-converters","repository":"python-poetry/tomlkit","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.74},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.585968962500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.578000375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.69358255},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2881795}},"judge_cost":0.0006715499999999999,"prompt_sha256":"3e83480d5c378c12cc4ee21470424c4b387b1fa3fffceb38d7916500bc6d94c3","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"tomlkit-toml-table-converters","repository":"python-poetry/tomlkit","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.585968962500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.578000375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.69358255},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2881795}},"judge_cost":0.00034294999999999996,"prompt_sha256":"3e83480d5c378c12cc4ee21470424c4b387b1fa3fffceb38d7916500bc6d94c3","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"tomlkit-toml-table-converters","repository":"python-poetry/tomlkit","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.585968962500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.578000375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.69358255},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2881795}},"judge_cost":0.00035063,"prompt_sha256":"3e83480d5c378c12cc4ee21470424c4b387b1fa3fffceb38d7916500bc6d94c3","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"tomlkit-toml-table-converters","repository":"python-poetry/tomlkit","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.585968962500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.578000375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.69358255},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2881795}},"judge_cost":0.00034044999999999995,"prompt_sha256":"3e83480d5c378c12cc4ee21470424c4b387b1fa3fffceb38d7916500bc6d94c3","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"tomlkit-toml-table-converters","repository":"python-poetry/tomlkit","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.585968962500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.578000375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.69358255},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2881795}},"judge_cost":0.0003453,"prompt_sha256":"3e83480d5c378c12cc4ee21470424c4b387b1fa3fffceb38d7916500bc6d94c3","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"tomlkit-toml-table-converters","repository":"python-poetry/tomlkit","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.585968962500001},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.578000375},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.69358255},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2881795}},"judge_cost":0.00030853,"prompt_sha256":"3e83480d5c378c12cc4ee21470424c4b387b1fa3fffceb38d7916500bc6d94c3","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"true-myth-iterable-collection-combinators","repository":"true-myth/true-myth","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.4581176625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0883604375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6920486},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.899819}},"judge_cost":0.0006512499999999999,"prompt_sha256":"0c32454cd7d47c7408396fa479084ddce784c7bf12e6dce74876f74a5435faee","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"true-myth-iterable-collection-combinators","repository":"true-myth/true-myth","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.4581176625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0883604375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6920486},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.899819}},"judge_cost":0.00041505000000000003,"prompt_sha256":"0c32454cd7d47c7408396fa479084ddce784c7bf12e6dce74876f74a5435faee","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"true-myth-iterable-collection-combinators","repository":"true-myth/true-myth","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.4581176625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0883604375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6920486},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.899819}},"judge_cost":0.00039632999999999997,"prompt_sha256":"0c32454cd7d47c7408396fa479084ddce784c7bf12e6dce74876f74a5435faee","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"true-myth-iterable-collection-combinators","repository":"true-myth/true-myth","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.65},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.4581176625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0883604375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6920486},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.899819}},"judge_cost":0.00031862,"prompt_sha256":"0c32454cd7d47c7408396fa479084ddce784c7bf12e6dce74876f74a5435faee","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"true-myth-iterable-collection-combinators","repository":"true-myth/true-myth","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.4581176625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0883604375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6920486},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.899819}},"judge_cost":0.00033935,"prompt_sha256":"0c32454cd7d47c7408396fa479084ddce784c7bf12e6dce74876f74a5435faee","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"ts-pattern-match-each","repository":"gvergnaud/ts-pattern","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.4401305},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.3848743125},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.37506724999999996},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.827779}},"judge_cost":0.0007271999999999999,"prompt_sha256":"eb96fb2bdf617e717f1d693397b42469458bf6addb6392a7669e3464fa6cefcd","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"true-myth-iterable-collection-combinators","repository":"true-myth/true-myth","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.4581176625},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":4.0883604375},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.6920486},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.899819}},"judge_cost":0.00041063,"prompt_sha256":"0c32454cd7d47c7408396fa479084ddce784c7bf12e6dce74876f74a5435faee","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"ts-pattern-match-each","repository":"gvergnaud/ts-pattern","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.4401305},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.3848743125},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.37506724999999996},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.827779}},"judge_cost":0.0007271999999999999,"prompt_sha256":"eb96fb2bdf617e717f1d693397b42469458bf6addb6392a7669e3464fa6cefcd","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ts-pattern-match-each","repository":"gvergnaud/ts-pattern","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.4401305},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.3848743125},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.37506724999999996},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.827779}},"judge_cost":0.0003854,"prompt_sha256":"eb96fb2bdf617e717f1d693397b42469458bf6addb6392a7669e3464fa6cefcd","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"ts-pattern-match-each","repository":"gvergnaud/ts-pattern","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.29},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.4401305},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.3848743125},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.37506724999999996},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.827779}},"judge_cost":0.00035108,"prompt_sha256":"eb96fb2bdf617e717f1d693397b42469458bf6addb6392a7669e3464fa6cefcd","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"ts-pattern-match-each","repository":"gvergnaud/ts-pattern","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.4401305},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.3848743125},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.37506724999999996},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.827779}},"judge_cost":0.00037929999999999995,"prompt_sha256":"eb96fb2bdf617e717f1d693397b42469458bf6addb6392a7669e3464fa6cefcd","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"ts-pattern-match-each","repository":"gvergnaud/ts-pattern","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.18},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.4401305},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.3848743125},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.37506724999999996},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":1.827779}},"judge_cost":0.00041578,"prompt_sha256":"eb96fb2bdf617e717f1d693397b42469458bf6addb6392a7669e3464fa6cefcd","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"updo-policy-alerting","repository":"Owloops/updo","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-3","capability_boundary":"supported","p_solve":0.72},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.3020574},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.002597562499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.59361565},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.938537}},"judge_cost":0.0008466,"prompt_sha256":"57bb53e5a59bcb97eac9119a900315da31f8102f3be94585e190f6bae9a92dbb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"updo-policy-alerting","repository":"Owloops/updo","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.56},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.3020574},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.002597562499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.59361565},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.938537}},"judge_cost":0.00047359999999999997,"prompt_sha256":"57bb53e5a59bcb97eac9119a900315da31f8102f3be94585e190f6bae9a92dbb","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"updo-policy-alerting","repository":"Owloops/updo","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.43},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.3020574},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.002597562499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.59361565},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.938537}},"judge_cost":0.00046447999999999997,"prompt_sha256":"57bb53e5a59bcb97eac9119a900315da31f8102f3be94585e190f6bae9a92dbb","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"updo-policy-alerting","repository":"Owloops/updo","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.3020574},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.002597562499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.59361565},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.938537}},"judge_cost":0.00033108,"prompt_sha256":"57bb53e5a59bcb97eac9119a900315da31f8102f3be94585e190f6bae9a92dbb","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"updo-policy-alerting","repository":"Owloops/updo","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.6},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.3020574},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.002597562499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.59361565},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.938537}},"judge_cost":0.0004807,"prompt_sha256":"57bb53e5a59bcb97eac9119a900315da31f8102f3be94585e190f6bae9a92dbb","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"updo-policy-alerting","repository":"Owloops/updo","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.31},"outcomes":{"claude-sonnet-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.3020574},"claude-opus-5":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":4.002597562499999},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":0.59361565},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.938537}},"judge_cost":0.00046677999999999997,"prompt_sha256":"57bb53e5a59bcb97eac9119a900315da31f8102f3be94585e190f6bae9a92dbb","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"valibot-recursive-schema-composition","repository":"open-circle/valibot","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":16.555904962499998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":15.5792064375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.23142205},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.65735275}},"judge_cost":0.0002825,"prompt_sha256":"cbddc83397d2ba8aadf4a806df20ca1d2c4531b437fee0f9633240274d0c123e","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"valibot-recursive-schema-composition","repository":"open-circle/valibot","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":16.555904962499998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":15.5792064375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.23142205},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.65735275}},"judge_cost":0.0005835,"prompt_sha256":"cbddc83397d2ba8aadf4a806df20ca1d2c4531b437fee0f9633240274d0c123e","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"valibot-recursive-schema-composition","repository":"open-circle/valibot","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":16.555904962499998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":15.5792064375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.23142205},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.65735275}},"judge_cost":0.00029378,"prompt_sha256":"cbddc83397d2ba8aadf4a806df20ca1d2c4531b437fee0f9633240274d0c123e","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"valibot-recursive-schema-composition","repository":"open-circle/valibot","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":16.555904962499998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":15.5792064375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.23142205},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.65735275}},"judge_cost":0.0002788,"prompt_sha256":"cbddc83397d2ba8aadf4a806df20ca1d2c4531b437fee0f9633240274d0c123e","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"valibot-recursive-schema-composition","repository":"open-circle/valibot","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":16.555904962499998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":15.5792064375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.23142205},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.65735275}},"judge_cost":0.00030208,"prompt_sha256":"cbddc83397d2ba8aadf4a806df20ca1d2c4531b437fee0f9633240274d0c123e","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"valibot-recursive-schema-composition","repository":"open-circle/valibot","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.68},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":16.555904962499998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":15.5792064375},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.23142205},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.65735275}},"judge_cost":0.00052296,"prompt_sha256":"cbddc83397d2ba8aadf4a806df20ca1d2c4531b437fee0f9633240274d0c123e","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"vitest-duration-sharding","repository":"vitest-dev/vitest","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.638803275000001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.4737846875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8614573999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.7404617499999997}},"judge_cost":0.0008790999999999998,"prompt_sha256":"8ee5fc8e1d4c516abf7d8eccb57a1536fa156c72c3408b253aa6c5185ac32361","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"vitest-duration-sharding","repository":"vitest-dev/vitest","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.638803275000001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.4737846875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8614573999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.7404617499999997}},"judge_cost":0.0005277,"prompt_sha256":"8ee5fc8e1d4c516abf7d8eccb57a1536fa156c72c3408b253aa6c5185ac32361","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"vitest-duration-sharding","repository":"vitest-dev/vitest","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.638803275000001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.4737846875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8614573999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.7404617499999997}},"judge_cost":0.00051018,"prompt_sha256":"8ee5fc8e1d4c516abf7d8eccb57a1536fa156c72c3408b253aa6c5185ac32361","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"vitest-duration-sharding","repository":"vitest-dev/vitest","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.638803275000001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.4737846875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8614573999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.7404617499999997}},"judge_cost":0.00031544,"prompt_sha256":"8ee5fc8e1d4c516abf7d8eccb57a1536fa156c72c3408b253aa6c5185ac32361","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"vitest-duration-sharding","repository":"vitest-dev/vitest","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.39},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.638803275000001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.4737846875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8614573999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.7404617499999997}},"judge_cost":0.00052928,"prompt_sha256":"8ee5fc8e1d4c516abf7d8eccb57a1536fa156c72c3408b253aa6c5185ac32361","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"vitest-duration-sharding","repository":"vitest-dev/vitest","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":6.638803275000001},"claude-opus-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":5.4737846875},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8614573999999999},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":3.7404617499999997}},"judge_cost":0.0006428,"prompt_sha256":"8ee5fc8e1d4c516abf7d8eccb57a1536fa156c72c3408b253aa6c5185ac32361","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"vulture-persistent-analysis-cache","repository":"jendrikseipp/vulture","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.273343025000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.9341403125000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4024446},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.15908625}},"judge_cost":0.0003622,"prompt_sha256":"5d8e3edbe015bf0a1f62ed6c21d6c93f000f05cf59ced99a201972a66e362271","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"vulture-persistent-analysis-cache","repository":"jendrikseipp/vulture","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.273343025000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.9341403125000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4024446},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.15908625}},"judge_cost":0.0006548,"prompt_sha256":"5d8e3edbe015bf0a1f62ed6c21d6c93f000f05cf59ced99a201972a66e362271","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"vulture-persistent-analysis-cache","repository":"jendrikseipp/vulture","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.273343025000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.9341403125000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4024446},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.15908625}},"judge_cost":0.00033988,"prompt_sha256":"5d8e3edbe015bf0a1f62ed6c21d6c93f000f05cf59ced99a201972a66e362271","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"vulture-persistent-analysis-cache","repository":"jendrikseipp/vulture","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.58},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.273343025000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.9341403125000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4024446},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.15908625}},"judge_cost":0.00029163999999999997,"prompt_sha256":"5d8e3edbe015bf0a1f62ed6c21d6c93f000f05cf59ced99a201972a66e362271","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"vulture-persistent-analysis-cache","repository":"jendrikseipp/vulture","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.273343025000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.9341403125000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4024446},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.15908625}},"judge_cost":0.0003393,"prompt_sha256":"5d8e3edbe015bf0a1f62ed6c21d6c93f000f05cf59ced99a201972a66e362271","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"vulture-persistent-analysis-cache","repository":"jendrikseipp/vulture","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":4.273343025000001},"claude-opus-5":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":3.9341403125000003},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.4024446},"gpt-5-6-sol":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":2.15908625}},"judge_cost":0.00035058,"prompt_sha256":"5d8e3edbe015bf0a1f62ed6c21d6c93f000f05cf59ced99a201972a66e362271","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"wasmi-trap-coredumps","repository":"wasmi-labs/wasmi","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":17.2567471875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.239138375000003},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.00660935},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.421698}},"judge_cost":0.0008336999999999999,"prompt_sha256":"f618b3e741cb043c65f608c38c46ce0fd6d78324d613aff624f0a05f799d00ce","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"wasmi-trap-coredumps","repository":"wasmi-labs/wasmi","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":17.2567471875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.239138375000003},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.00660935},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.421698}},"judge_cost":0.00041438,"prompt_sha256":"f618b3e741cb043c65f608c38c46ce0fd6d78324d613aff624f0a05f799d00ce","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"wasmi-trap-coredumps","repository":"wasmi-labs/wasmi","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":17.2567471875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.239138375000003},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.00660935},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.421698}},"judge_cost":0.0004607,"prompt_sha256":"f618b3e741cb043c65f608c38c46ce0fd6d78324d613aff624f0a05f799d00ce","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"wasmi-trap-coredumps","repository":"wasmi-labs/wasmi","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":17.2567471875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.239138375000003},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.00660935},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.421698}},"judge_cost":0.000433,"prompt_sha256":"f618b3e741cb043c65f608c38c46ce0fd6d78324d613aff624f0a05f799d00ce","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"wasmi-trap-coredumps","repository":"wasmi-labs/wasmi","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":17.2567471875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.239138375000003},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.00660935},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.421698}},"judge_cost":0.00041428,"prompt_sha256":"f618b3e741cb043c65f608c38c46ce0fd6d78324d613aff624f0a05f799d00ce","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"wasmi-trap-coredumps","repository":"wasmi-labs/wasmi","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.35},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":17.2567471875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":11.239138375000003},"gpt-5-6-luna":{"p":0.0,"successes":0.0,"attempts":4,"published_mean_cost":2.00660935},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":7.421698}},"judge_cost":0.00044711999999999996,"prompt_sha256":"f618b3e741cb043c65f608c38c46ce0fd6d78324d613aff624f0a05f799d00ce","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"wazero-multi-module-snapshots","repository":"wazero/wazero","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.46},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.6378489124999998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2862240625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.3190076},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.2785565}},"judge_cost":0.00038105,"prompt_sha256":"097ffaf4e37ff039b64f917b645eebe9b4b838b0e546d73312f718e11f38af36","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"wazero-multi-module-snapshots","repository":"wazero/wazero","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.62},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.6378489124999998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2862240625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.3190076},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.2785565}},"judge_cost":0.00073485,"prompt_sha256":"097ffaf4e37ff039b64f917b645eebe9b4b838b0e546d73312f718e11f38af36","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"wazero-multi-module-snapshots","repository":"wazero/wazero","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.6378489124999998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2862240625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.3190076},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.2785565}},"judge_cost":0.00042113000000000003,"prompt_sha256":"097ffaf4e37ff039b64f917b645eebe9b4b838b0e546d73312f718e11f38af36","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"wazero-multi-module-snapshots","repository":"wazero/wazero","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-2","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.6378489124999998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2862240625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.3190076},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.2785565}},"judge_cost":0.00031494,"prompt_sha256":"097ffaf4e37ff039b64f917b645eebe9b4b838b0e546d73312f718e11f38af36","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"wazero-multi-module-snapshots","repository":"wazero/wazero","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.6378489124999998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2862240625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.3190076},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.2785565}},"judge_cost":0.00041815,"prompt_sha256":"097ffaf4e37ff039b64f917b645eebe9b4b838b0e546d73312f718e11f38af36","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"wazero-multi-module-snapshots","repository":"wazero/wazero","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":1.6378489124999998},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.2862240625},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.3190076},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.2785565}},"judge_cost":0.00046903000000000005,"prompt_sha256":"097ffaf4e37ff039b64f917b645eebe9b4b838b0e546d73312f718e11f38af36","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"yaegi-go-embed-directives","repository":"traefik/yaegi","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.64},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.202472025},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.127602812500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8433395},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.75486725}},"judge_cost":0.00061175,"prompt_sha256":"be5b3a5629637d382aa7f0afee62e9fd7985774049a34139ab265482233587a2","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"yaegi-go-embed-directives","repository":"traefik/yaegi","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.32},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.202472025},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.127602812500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8433395},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.75486725}},"judge_cost":0.00032803,"prompt_sha256":"be5b3a5629637d382aa7f0afee62e9fd7985774049a34139ab265482233587a2","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"yaegi-go-embed-directives","repository":"traefik/yaegi","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.202472025},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.127602812500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8433395},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.75486725}},"judge_cost":0.00036235,"prompt_sha256":"be5b3a5629637d382aa7f0afee62e9fd7985774049a34139ab265482233587a2","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"yaegi-go-embed-directives","repository":"traefik/yaegi","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.67},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.202472025},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.127602812500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8433395},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.75486725}},"judge_cost":0.00030201999999999996,"prompt_sha256":"be5b3a5629637d382aa7f0afee62e9fd7985774049a34139ab265482233587a2","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"yaegi-go-embed-directives","repository":"traefik/yaegi","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.55},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.202472025},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.127602812500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8433395},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.75486725}},"judge_cost":0.00032865,"prompt_sha256":"be5b3a5629637d382aa7f0afee62e9fd7985774049a34139ab265482233587a2","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"yaegi-go-embed-directives","repository":"traefik/yaegi","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.34},"outcomes":{"claude-sonnet-5":{"p":0.25,"successes":1.0,"attempts":4,"published_mean_cost":9.202472025},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.127602812500001},"gpt-5-6-luna":{"p":0.5,"successes":2.0,"attempts":4,"published_mean_cost":0.8433395},"gpt-5-6-sol":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":3.75486725}},"judge_cost":0.00027873000000000004,"prompt_sha256":"be5b3a5629637d382aa7f0afee62e9fd7985774049a34139ab265482233587a2","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"yjs-map-conflict-detection","repository":"yjs/yjs","split":"train","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.24},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.4590743875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.841859249999999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.79558235},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.855738}},"judge_cost":0.0002982,"prompt_sha256":"fc7493d6b5211212135adba2fb12872f2aea81c8ea12bfd5cb866af844763f94","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"yjs-map-conflict-detection","repository":"yjs/yjs","split":"train","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.29},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.4590743875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.841859249999999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.79558235},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.855738}},"judge_cost":0.00029628,"prompt_sha256":"fc7493d6b5211212135adba2fb12872f2aea81c8ea12bfd5cb866af844763f94","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"yjs-map-conflict-detection","repository":"yjs/yjs","split":"train","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.38},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.4590743875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.841859249999999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.79558235},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.855738}},"judge_cost":0.0006831999999999999,"prompt_sha256":"fc7493d6b5211212135adba2fb12872f2aea81c8ea12bfd5cb866af844763f94","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"yjs-map-conflict-detection","repository":"yjs/yjs","split":"train","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.4590743875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.841859249999999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.79558235},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.855738}},"judge_cost":0.0003245,"prompt_sha256":"fc7493d6b5211212135adba2fb12872f2aea81c8ea12bfd5cb866af844763f94","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"yjs-map-conflict-detection","repository":"yjs/yjs","split":"train","definition":"cap_luna_sol_original","forecast":{"primary_rule":"UNC-1","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.4590743875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.841859249999999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.79558235},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.855738}},"judge_cost":0.00037964,"prompt_sha256":"fc7493d6b5211212135adba2fb12872f2aea81c8ea12bfd5cb866af844763f94","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"yjs-map-conflict-detection","repository":"yjs/yjs","split":"train","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.28},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":8.4590743875},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":6.841859249999999},"gpt-5-6-luna":{"p":0.75,"successes":3.0,"attempts":4,"published_mean_cost":0.79558235},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.855738}},"judge_cost":0.00032138,"prompt_sha256":"fc7493d6b5211212135adba2fb12872f2aea81c8ea12bfd5cb866af844763f94","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}
{"task":"ytt-jsonpath-query-api","repository":"carvel-dev/ytt","split":"validation","definition":"cap_sonnet_opus_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.52},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.8651187500000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6816815625},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.3731504},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.99272825}},"judge_cost":0.0007160000000000001,"prompt_sha256":"ac82c98d2104e693db7ec0badbfb843dc871a814da14b8a50f9fa0d535d7939e","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ytt-jsonpath-query-api","repository":"carvel-dev/ytt","split":"validation","definition":"cap_sonnet_opus_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.42},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.8651187500000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6816815625},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.3731504},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.99272825}},"judge_cost":0.0004966,"prompt_sha256":"ac82c98d2104e693db7ec0badbfb843dc871a814da14b8a50f9fa0d535d7939e","system_sha256":"0b8cbe6200fec5868faa22723d7cd2515df4a5a61e83c061c5a9f7c174d745b3"}
{"task":"ytt-jsonpath-query-api","repository":"carvel-dev/ytt","split":"validation","definition":"cap_sonnet_opus_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.8651187500000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6816815625},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.3731504},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.99272825}},"judge_cost":0.00039388,"prompt_sha256":"ac82c98d2104e693db7ec0badbfb843dc871a814da14b8a50f9fa0d535d7939e","system_sha256":"5786d54d9d7bbbcff488d2823def9b7e36d08e6334c564be1c1afe6a7c814634"}
{"task":"ytt-jsonpath-query-api","repository":"carvel-dev/ytt","split":"validation","definition":"cap_luna_sol_original","forecast":{"primary_rule":"SUP-4","capability_boundary":"supported","p_solve":0.57},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.8651187500000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6816815625},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.3731504},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.99272825}},"judge_cost":0.00027892,"prompt_sha256":"ac82c98d2104e693db7ec0badbfb843dc871a814da14b8a50f9fa0d535d7939e","system_sha256":"130974d1eac71ba06684bba9a7129084157e88d25e05ab85326bb10be92cccae"}
{"task":"ytt-jsonpath-query-api","repository":"carvel-dev/ytt","split":"validation","definition":"cap_luna_sol_research","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.36},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.8651187500000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6816815625},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.3731504},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.99272825}},"judge_cost":0.00041969999999999996,"prompt_sha256":"ac82c98d2104e693db7ec0badbfb843dc871a814da14b8a50f9fa0d535d7939e","system_sha256":"80bdbae3a602ba8b78c105a98fb7de33bd0ce5ac149244150e24f11459f94c28"}
{"task":"ytt-jsonpath-query-api","repository":"carvel-dev/ytt","split":"validation","definition":"cap_luna_sol_trained_card","forecast":{"primary_rule":"UNC-2","capability_boundary":"uncertain","p_solve":0.27},"outcomes":{"claude-sonnet-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.8651187500000002},"claude-opus-5":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":2.6816815625},"gpt-5-6-luna":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":0.3731504},"gpt-5-6-sol":{"p":1.0,"successes":4.0,"attempts":4,"published_mean_cost":1.99272825}},"judge_cost":0.00040098,"prompt_sha256":"ac82c98d2104e693db7ec0badbfb843dc871a814da14b8a50f9fa0d535d7939e","system_sha256":"9851249bfa3bfd6bf31c1adabff8291a729c4444cc1a346533a7d6358f118323"}

2
litellm/proxy/_lazy_openapi_snapshot.json
View file

@ -18968,7 +18968,7 @@
}
}
},
"description": "\n Unified rate-limit error.\n\n Every rate-limit condition surfaced by litellm \u2014 whether it originated from\n an upstream LLM provider, a vendor batch endpoint, or one of litellm's own\n proxy-side limiters (parallel-requests, dynamic-rate, batch-rate, budget,\n max-iterations, etc.) \u2014 is raised as an instance of this class.\n\n The :attr:`category` attribute lets callers distinguish the source. See\n :class:`RateLimitErrorCategory` for the available values.\n "
"description": "\nUnified rate-limit error.\n\nEvery rate-limit condition surfaced by litellm \u2014 whether it originated from\nan upstream LLM provider, a vendor batch endpoint, or one of litellm's own\nproxy-side limiters (parallel-requests, dynamic-rate, batch-rate, budget,\nmax-iterations, etc.) \u2014 is raised as an instance of this class.\n\nThe :attr:`category` attribute lets callers distinguish the source. See\n:class:`RateLimitErrorCategory` for the available values.\n"
},
"500": {
"content": {

26
litellm/router_strategy/complexity_router/capability_classifier.py
View file

@ -191,12 +191,32 @@ def capability_classifier_response_format(
)
def capability_classifier_system_prompt(mode: Literal["json_schema", "json_object"]) -> str:
def capability_classifier_system_prompt(
mode: Literal["json_schema", "json_object"], *, card: str | None = None, empirical: bool = False
) -> str:
prefix: Final = CAPABILITY_CLASSIFIER_SYSTEM_PROMPT.split("# Efficient-agent capability card")[0]
evidence_prefix: Final = (
prefix.replace(
"The capability card is qualitative evidence, not a measured prior.",
"Only explicitly supplied training counts are measured evidence; preserve their scope and uncertainty.",
)
if empirical
else prefix
)
prompt: Final = (
evidence_prefix
+ "# Efficient-agent capability card\n\n"
+ card
+ "\n\n# Output"
+ CAPABILITY_CLASSIFIER_SYSTEM_PROMPT.split("# Output")[1]
if card is not None
else CAPABILITY_CLASSIFIER_SYSTEM_PROMPT
)
if mode == "json_schema":
return CAPABILITY_CLASSIFIER_SYSTEM_PROMPT
return prompt
wrapper: Final = _RESPONSE_FORMAT_ADAPTER.validate_python(capability_classifier_response_format()["json_schema"])
return (
CAPABILITY_CLASSIFIER_SYSTEM_PROMPT
prompt
+ "\n\nReturn exactly one JSON object matching this JSON Schema:\n"
+ json.dumps(wrapper["schema"], indent=2, sort_keys=True)
)

10
litellm/router_strategy/complexity_router/complexity_router.py
View file

@ -1338,7 +1338,11 @@ class ComplexityRouter(CustomLogger):
if self.config.classifier_type == "capability":
capability: Final = self.config.capability_classifier_config
return capability_classifier_system_prompt(
capability.response_format if capability is not None else "json_schema"
capability.response_format if capability is not None else "json_schema",
card=capability.card.text if capability is not None and capability.card is not None else None,
empirical=capability.card.empirical
if capability is not None and capability.card is not None
else False,
)
definitions: Final = self.config.tier_definitions
if definitions is not None:
@ -2238,7 +2242,9 @@ class ComplexityRouter(CustomLogger):
forecast: Final = CapabilityClassifierForecast(
verdict=verdict,
threshold=threshold,
p_solve=calibration.calibrate(verdict.p_solve) if calibration is not None else verdict.p_solve,
p_solve=calibration.calibrate(verdict.p_solve, verdict.primary_rule)
if calibration is not None
else verdict.p_solve,
calibration_version=calibration.version if calibration is not None else None,
)
selected_tier: Final = (

29
litellm/router_strategy/complexity_router/config.py
View file

@ -32,6 +32,7 @@ with warnings.catch_warnings():
from litellm.types.llms.openai import REASONING_EFFORT
from litellm.types.router import AdaptiveRouterWeights, ClassifierPlugin, RoutingPlugin
from .capability_classifier import CapabilityRule
from .tier_predictor import TrainedTierArtifact
@ -600,16 +601,39 @@ class ClassifierLLMConfig(BaseModel):
return self
class CapabilityRuleCalibration(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
rule: CapabilityRule
intercept: StrictFloat = Field(ge=-5.0, le=5.0, allow_inf_nan=False)
class CapabilityCardConfig(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
version: str = Field(min_length=1, max_length=128, pattern=r"^\S(?:.*\S)?$")
text: str = Field(min_length=1, max_length=8000, pattern=r"\S")
empirical: bool = False
class CapabilityCalibrationConfig(BaseModel):
model_config = ConfigDict(extra="forbid", frozen=True)
version: str = Field(min_length=1, max_length=128, pattern=r"^\S(?:.*\S)?$")
slope: StrictFloat = Field(ge=0.0, le=20.0, allow_inf_nan=False)
intercept: StrictFloat = Field(ge=-20.0, le=20.0, allow_inf_nan=False)
rule_intercepts: tuple[CapabilityRuleCalibration, ...] = Field(default=(), max_length=10)
def calibrate(self, p_solve: float) -> float:
@model_validator(mode="after")
def _validate_unique_rules(self) -> "CapabilityCalibrationConfig":
if len(frozenset(offset.rule for offset in self.rule_intercepts)) != len(self.rule_intercepts):
raise ValueError("rule_intercepts must contain unique capability rules")
return self
def calibrate(self, p_solve: float, rule: CapabilityRule = "none") -> float:
clipped: Final = min(max(p_solve, 1e-6), 1.0 - 1e-6)
log_odds: Final = self.slope * (math.log(clipped) - math.log1p(-clipped)) + self.intercept
adjustment: Final = sum(offset.intercept for offset in self.rule_intercepts if offset.rule == rule)
log_odds: Final = self.slope * (math.log(clipped) - math.log1p(-clipped)) + self.intercept + adjustment
return 1.0 / (1.0 + math.exp(-log_odds))
@ -617,6 +641,7 @@ class CapabilityClassifierConfig(BaseModel):
"""Switchyard-compatible probability threshold policy for two model tiers."""
model_config = ConfigDict(frozen=True)
card: CapabilityCardConfig | None = None
efficient_tier: str = Field(
description="Tier used when the efficient model's forecasted solve probability meets the adjusted threshold",

68
tests/test_litellm/router_strategy/test_complexity_router.py
View file

@ -2407,6 +2407,33 @@ def _capability_reply(
class TestCapabilityClassifierConfig:
@pytest.mark.parametrize(
"offsets",
(
[{"rule": "SUP-1", "intercept": 1.0}] * 2,
[{"rule": "invented", "intercept": 1.0}],
[{"rule": "SUP-1", "intercept": float("nan")}],
),
)
def test_rejects_invalid_rule_offsets(self, offsets: list[dict[str, object]]) -> None:
with pytest.raises(ValidationError):
CapabilityCalibrationConfig.model_validate(
{"version": "rules-v1", "slope": 1.0, "intercept": 0.0, "rule_intercepts": offsets}
)
def test_rule_adjustments_match_only_the_active_rule_after_round_trip(self) -> None:
calibration: Final = CapabilityCalibrationConfig.model_validate(
{
"version": "rules-v1",
"slope": 1.0,
"intercept": 0.0,
"rule_intercepts": [{"rule": "UNC-2", "intercept": -2.0}],
}
)
restored: Final = CapabilityCalibrationConfig.model_validate_json(calibration.model_dump_json())
assert restored.calibrate(0.9, "UNC-2") == pytest.approx(0.5491469396)
assert restored.calibrate(0.9, "SUP-1") == pytest.approx(0.9)
@pytest.mark.parametrize(
"calibration",
(
@ -2575,6 +2602,47 @@ class TestCapabilityClassifier:
complexity_router_config=_capability_router_config(**overrides),
)
@pytest.mark.asyncio
async def test_trained_card_and_rule_adjustment_reach_the_live_routing_policy(
self, mock_router_instance: MagicMock
) -> None:
mock_router_instance.acompletion = AsyncMock(
return_value=_llm_response(
_capability_reply(p_solve=0.9, primary_rule="UNC-2", capability_boundary="uncertain")
)
)
router: Final = self._router(
mock_router_instance,
capability_classifier_config={
"efficient_tier": "SIMPLE",
"capable_tier": "REASONING",
"base_threshold": 0.6,
"card": {
"version": "test-card-v1",
"text": "UNC-2 [uncertain]: Coupled changes need inspection",
"empirical": True,
},
"calibration": {
"version": "rules-v1",
"slope": 1.0,
"intercept": 0.0,
"rule_intercepts": [{"rule": "UNC-2", "intercept": -2.0}],
},
},
)
result: Final = await router.async_pre_routing_hook(
model="capability-router", request_kwargs={}, messages=[{"role": "user", "content": "Fix the issue"}]
)
assert result is not None and result.model == "capable-model"
assert result.routing_decision is not None
assert result.routing_decision["classifier_p_solve"] == 0.9
assert result.routing_decision["classifier_calibrated_p_solve"] == pytest.approx(0.5491469396)
sent: Final = mock_router_instance.acompletion.call_args.kwargs["messages"][0]["content"]
assert "Coupled changes need inspection" in sent
assert "Only explicitly supplied training counts" in sent
assert "Route to the Efficient model" not in sent
mock_router_instance.acompletion.assert_awaited_once()
@pytest.mark.asyncio
@pytest.mark.parametrize("p_solve,expected_model", ((0.95, "capable-model"), (0.98, "efficient-model")))
async def test_fitted_probability_controls_routing_and_preserves_raw_score(

28
ui/litellm-dashboard/src/lib/http/schema.d.ts generated vendored
View file

@ -24907,11 +24907,28 @@ export interface components {
CapabilityCalibrationConfig: {
/** Intercept */
intercept: number;
/**
* Rule Intercepts
* @default []
*/
rule_intercepts: components["schemas"]["CapabilityRuleCalibration"][];
/** Slope */
slope: number;
/** Version */
version: string;
};
/** CapabilityCardConfig */
CapabilityCardConfig: {
/**
* Empirical
* @default false
*/
empirical: boolean;
/** Text */
text: string;
/** Version */
version: string;
};
/**
* CapabilityClassifierConfig
* @description Switchyard-compatible probability threshold policy for two model tiers.
@ -24929,6 +24946,7 @@ export interface components {
* @description Higher, fail-closed tier used below the adjusted threshold or when the classifier verdict is unavailable
*/
capable_tier: string;
card?: components["schemas"]["CapabilityCardConfig"] | null;
/**
* Efficient Tier
* @description Tier used when the efficient model's forecasted solve probability meets the adjusted threshold
@ -24954,6 +24972,16 @@ export interface components {
*/
threshold_step: number;
};
/** CapabilityRuleCalibration */
CapabilityRuleCalibration: {
/** Intercept */
intercept: number;
/**
* Rule
* @enum {string}
*/
rule: "SUP-1" | "SUP-2" | "SUP-3" | "SUP-4" | "SUP-5" | "UNC-1" | "UNC-2" | "LIM-1" | "LIM-2" | "none";
};
/** ChatCompletionAnnotation */
ChatCompletionAnnotation: {
/**