scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-29 06:33:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 267

fix(cookbook): pin reproduction dependencies, digest-verified pulls and frozen-manifest dataset prep

Browse source 
Co-Authored-By: bot_apk <apk@cognition.ai>
This commit is contained in:
Devin AI 2026-10-20 23:53:36 +00:00
parent c0fd58060e
commit 35e36f3d47
5 changed files with 241 additions and 120 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

36
cookbook/capability_routing_pilot/README.md
View file

@ -6,7 +6,7 @@ Five frozen SWE-bench dev tasks (`tasks.json`) are solved by mini-swe-agent 2.4.
The router picks a cheaper model only when, on the shared graded task IDs, a one-sided Hoeffding lower bound on the paired success difference is at least minus the tolerance and the paired mean cost is lower. With five tasks it falls back to the Opus reference, by design
Three evaluator images failed gold validation with an unchanged grader, NumPy 2 in both pvlib images and pytest 8 in the pydicom image. `overlays/` holds the version pins applied locally under the original tags. Gold then resolved 5/5 and a no-op patch 0/5. `images_pinned.txt` lists the digests used and `images_pinned_upstream.txt` the upstream ones
Three evaluator images failed gold validation with an unchanged grader, NumPy 2 in both pvlib images and pytest 8 in the pydicom image. `overlays/` holds the version pins applied locally under the original tags. Gold then resolved 5/5 and a no-op patch 0/5. `images_pinned_upstream.txt` records upstream registry manifest digests, while `images_pinned.txt` separates those from local image IDs. `selection_candidates.json` preserves historical selection provenance
## Reproduction
@ -14,27 +14,41 @@ Run these commands from this directory. `pip_freeze.txt` is authoritative for th
```sh
uv venv
uv pip install mini-swe-agent==2.4.6 swebench==5.0.2 litellm==1.106.1 mcp==2.3.0 scipy pyyaml datasets
uv pip install -r pip_freeze.txt
source .venv/bin/activate
```
Pull the five pinned task images
Pull each upstream image by its registry manifest digest, tag it locally, and inspect the recorded repository digests. Each inspection output must contain the listed `image@sha256:<upstream digest>` entry before building overlays
```sh
docker pull swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395:latest
docker pull swebench/sweb.eval.x86_64.sqlfluff_1776_sqlfluff-884:latest
docker pull swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1614:latest
docker pull swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1165:latest
docker pull swebench/sweb.eval.x86_64.pydicom_1776_pydicom-1031:latest
```
docker pull swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395@sha256:c7b51f4978926d54a788c54ac4d96e6c4c4e2cd2ee62736b30e91c9fd0891f10
docker tag swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395@sha256:c7b51f4978926d54a788c54ac4d96e6c4c4e2cd2ee62736b30e91c9fd0891f10 swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395:latest
docker image inspect --format '{{json .RepoDigests}}' swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395:latest
For each overlay, tag the upstream image and build the pinned image under its original tag
docker pull swebench/sweb.eval.x86_64.sqlfluff_1776_sqlfluff-884@sha256:ca28dab1d331c1c7e4908784576a7e1f3b11387dbaa4a8db466ce42a0eda6bec
docker tag swebench/sweb.eval.x86_64.sqlfluff_1776_sqlfluff-884@sha256:ca28dab1d331c1c7e4908784576a7e1f3b11387dbaa4a8db466ce42a0eda6bec swebench/sweb.eval.x86_64.sqlfluff_1776_sqlfluff-884:latest
docker image inspect --format '{{json .RepoDigests}}' swebench/sweb.eval.x86_64.sqlfluff_1776_sqlfluff-884:latest
docker pull swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1614@sha256:7206ea531b6c8b02f7a8e1760d732e4651f95307781f12f55e4b88e982610159
docker tag swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1614@sha256:7206ea531b6c8b02f7a8e1760d732e4651f95307781f12f55e4b88e982610159 swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1614:latest
docker image inspect --format '{{json .RepoDigests}}' swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1614:latest
docker pull swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1165@sha256:71e033964894077384c0c27a04ae0271e28eaad35602aa7570d40c683f1967d1
docker tag swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1165@sha256:71e033964894077384c0c27a04ae0271e28eaad35602aa7570d40c683f1967d1 swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1165:latest
docker image inspect --format '{{json .RepoDigests}}' swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1165:latest
docker pull swebench/sweb.eval.x86_64.pydicom_1776_pydicom-1031@sha256:0b962e881760f4b8bf3cdbfc695624c90906198ccf1e2410cd639f6adc6e340b
docker tag swebench/sweb.eval.x86_64.pydicom_1776_pydicom-1031@sha256:0b962e881760f4b8bf3cdbfc695624c90906198ccf1e2410cd639f6adc6e340b swebench/sweb.eval.x86_64.pydicom_1776_pydicom-1031:latest
docker image inspect --format '{{json .RepoDigests}}' swebench/sweb.eval.x86_64.pydicom_1776_pydicom-1031:latest
```
```sh
docker tag <img> routing_poc/orig.<name>:latest
docker build -f overlays/Dockerfile.<name> -t <img> .
```
For the three overlays, preserve the upstream tag first, then build the local image under that tag. These local builds have image IDs but no upstream registry manifest digest
```sh
docker tag swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395:latest routing_poc/orig.pvlib_1776_pvlib-python-1395:latest
docker build -f overlays/Dockerfile.pvlib_1776_pvlib-python-1395 -t swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395:latest .
@ -60,6 +74,8 @@ python -m poc.prepare
python -m poc.prepare --check
```
The original selection rule sorted SWE-bench dev IDs by `sha256(seed + instance_id)` using seed `litellm-capability-routing-pilot-v1`, then took the first five with a public evaluation image. `selection_candidates.json` records that historical provenance only. Reproduction uses the frozen IDs and order in `tasks.json`, loads the pinned dev revision, and projects only the six agent fields into `agent_dataset/train.jsonl`. It does not reselect tasks, query Docker Hub, or write `tasks.json`
Create a gateway virtual key scoped to the five models below with `max_budget` 15. Store the key in `.private/pilot_key` with mode `0600`, then export the gateway URL. Never commit or print the key
```sh

20
cookbook/capability_routing_pilot/images_pinned.txt
View file

@ -1,9 +1,11 @@
swebench/sweb.eval.x86_64.pydicom_1776_pydicom-1031:latest sha256:dd35fa4aecc830e588ae5966b3f3cfe91b3b34a6057c483847f82a44e4e77ac5
swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1165:latest sha256:e4134171da28404a3d71ef75f1866b776def32c31b986aab528c6f5716b27423
swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395:latest sha256:2eb9ad59249fc77761ffab8158f8fe2b1f44111f4972f804ca09991e49ae1197
swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1614:latest sha256:7206ea531b6c8b02f7a8e1760d732e4651f95307781f12f55e4b88e982610159
swebench/sweb.eval.x86_64.sqlfluff_1776_sqlfluff-884:latest sha256:ca28dab1d331c1c7e4908784576a7e1f3b11387dbaa4a8db466ce42a0eda6bec
# overlays (local, built from upstream): pvlib-1395,pvlib-1165 +numpy==1.26.4; pydicom-1031 +pytest==7.4.4; originals tagged routing_poc/orig.*
routing_poc/orig.pydicom_1776_pydicom-1031:latest sha256:0b962e881760f4b8bf3cdbfc695624c90906198ccf1e2410cd639f6adc6e340b
routing_poc/orig.pvlib_1776_pvlib-python-1165:latest sha256:71e033964894077384c0c27a04ae0271e28eaad35602aa7570d40c683f1967d1
routing_poc/orig.pvlib_1776_pvlib-python-1395:latest sha256:c7b51f4978926d54a788c54ac4d96e6c4c4e2cd2ee62736b30e91c9fd0891f10
Current tag | Upstream registry manifest digest | Local image ID | State
swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395:latest | sha256:c7b51f4978926d54a788c54ac4d96e6c4c4e2cd2ee62736b30e91c9fd0891f10 | sha256:2eb9ad59249fc77761ffab8158f8fe2b1f44111f4972f804ca09991e49ae1197 | overlay, numpy==1.26.4
swebench/sweb.eval.x86_64.sqlfluff_1776_sqlfluff-884:latest | sha256:ca28dab1d331c1c7e4908784576a7e1f3b11387dbaa4a8db466ce42a0eda6bec | sha256:ca28dab1d331c1c7e4908784576a7e1f3b11387dbaa4a8db466ce42a0eda6bec | untouched
swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1614:latest | sha256:7206ea531b6c8b02f7a8e1760d732e4651f95307781f12f55e4b88e982610159 | sha256:7206ea531b6c8b02f7a8e1760d732e4651f95307781f12f55e4b88e982610159 | untouched
swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1165:latest | sha256:71e033964894077384c0c27a04ae0271e28eaad35602aa7570d40c683f1967d1 | sha256:e4134171da28404a3d71ef75f1866b776def32c31b986aab528c6f5716b27423 | overlay, numpy==1.26.4
swebench/sweb.eval.x86_64.pydicom_1776_pydicom-1031:latest | sha256:0b962e881760f4b8bf3cdbfc695624c90906198ccf1e2410cd639f6adc6e340b | sha256:dd35fa4aecc830e588ae5966b3f3cfe91b3b34a6057c483847f82a44e4e77ac5 | overlay, pytest==7.4.4
Preserved upstream tag | Local image ID
routing_poc/orig.pydicom_1776_pydicom-1031:latest | sha256:0b962e881760f4b8bf3cdbfc695624c90906198ccf1e2410cd639f6adc6e340b
routing_poc/orig.pvlib_1776_pvlib-python-1165:latest | sha256:71e033964894077384c0c27a04ae0271e28eaad35602aa7570d40c683f1967d1
routing_poc/orig.pvlib_1776_pvlib-python-1395:latest | sha256:c7b51f4978926d54a788c54ac4d96e6c4c4e2cd2ee62736b30e91c9fd0891f10

154
cookbook/capability_routing_pilot/poc/prepare.py
View file

@ -1,11 +1,9 @@
import argparse
import hashlib
import json
import tempfile
from collections.abc import Mapping, Sequence
from pathlib import Path
from urllib.error import HTTPError
from urllib.request import Request, urlopen
from typing import Final
from datasets import load_dataset
@ -13,124 +11,78 @@ ROOT = Path(__file__).resolve().parents[1]
DATASET_NAME = "princeton-nlp/SWE-bench"
DATASET_REVISION = "e48e2bd1e9fecd5bbd641e9414ac59da9f2e69f6"
DATASET_SPLIT = "dev"
SEED = "litellm-capability-routing-pilot-v1"
FROZEN_AT = "2026-10-20T23:10Z"
TASK_COUNT = 5
TRAIN_FIELDS = ("instance_id", "repo", "base_commit", "problem_statement", "version", "created_at")
TASK_RULE = "sort dev ids by sha256(seed+id); take first 5 with public swebench eval image"
DOCKER_HUB_TIMEOUT_S = 30
def _image_name(instance_id: str) -> str:
image_id = instance_id.replace("__", "_1776_").lower()
return f"swebench/sweb.eval.x86_64.{image_id}:latest"
def _image_metadata(instance_id: str) -> dict[str, object] | None:
image = _image_name(instance_id)
repository, tag = image.removesuffix(":latest").split("/", 1)
url = f"https://hub.docker.com/v2/repositories/{repository}/{tag}/tags/latest"
request = Request(url, headers={"User-Agent": "capability-routing-pilot-prepare/1.0"})
try:
with urlopen(request, timeout=DOCKER_HUB_TIMEOUT_S) as response:
payload = json.load(response)
except HTTPError as error:
if error.code == 404:
return None
raise
size = payload.get("full_size")
digest = payload.get("digest")
if not isinstance(size, int) or not isinstance(digest, str):
raise RuntimeError(f"Docker Hub returned incomplete metadata for {image}")
return {
"instance_id": instance_id,
"image": image,
"exists": True,
"size": size,
"digest": digest,
}
TRAIN_FIELDS: Final = ("instance_id", "repo", "base_commit", "problem_statement", "version", "created_at")
def _load_rows() -> tuple[Mapping[str, object], ...]:
dataset = load_dataset(DATASET_NAME, revision=DATASET_REVISION, split=DATASET_SPLIT)
dataset: Final = load_dataset(DATASET_NAME, revision=DATASET_REVISION, split=DATASET_SPLIT)
return tuple(dataset)
def _select_tasks(rows: Sequence[Mapping[str, object]]) -> tuple[dict[str, object], ...]:
ordered = sorted(
rows,
key=lambda row: hashlib.sha256(
f"{SEED}{row['instance_id']}".encode("utf-8")
).hexdigest(),
)
selected: list[dict[str, object]] = []
for row in ordered:
instance_id = row.get("instance_id")
if not isinstance(instance_id, str):
raise RuntimeError("SWE-bench dev row is missing an instance_id")
metadata = _image_metadata(instance_id)
if metadata is None:
continue
selected.append(metadata)
if len(selected) == TASK_COUNT:
break
if len(selected) != TASK_COUNT:
raise RuntimeError(f"Found only {len(selected)} public eval images, expected {TASK_COUNT}")
return tuple(selected)
def _train_row(row: Mapping[str, object]) -> dict[str, object]:
result = {field: row[field] for field in TRAIN_FIELDS}
result: Final = {field: row[field] for field in TRAIN_FIELDS}
if not all(isinstance(result[field], str) for field in TRAIN_FIELDS):
raise RuntimeError("SWE-bench dev row contains a non-string training field")
return result
def _write_outputs(
output_dir: Path,
rows: Sequence[Mapping[str, object]],
tasks: Sequence[dict[str, object]],
) -> None:
task_payload = {
"frozen_at": FROZEN_AT,
"dataset": DATASET_NAME,
"split": DATASET_SPLIT,
"seed": SEED,
"rule": TASK_RULE,
"tasks": list(tasks),
}
(output_dir / "tasks.json").write_text(json.dumps(task_payload, indent=2) + "\n")
selected_ids = {str(task["instance_id"]) for task in tasks}
train_rows = tuple(
_train_row(row)
for row in rows
if isinstance(row.get("instance_id"), str) and row["instance_id"] in selected_ids
def agent_rows(
dataset_rows: Sequence[Mapping[str, object]],
task_ids: Sequence[str],
) -> list[dict[str, object]]:
dataset_ids: Final = frozenset(
row["instance_id"]
for row in dataset_rows
if isinstance(row.get("instance_id"), str)
)
train_path = output_dir / "agent_dataset" / "train.jsonl"
train_path.parent.mkdir(parents=True, exist_ok=True)
train_path.write_text("".join(json.dumps(row) + "\n" for row in train_rows))
missing_ids: Final = tuple(task_id for task_id in task_ids if task_id not in dataset_ids)
if missing_ids:
raise ValueError(f"Frozen task IDs are missing from the SWE-bench dev dataset: {', '.join(missing_ids)}")
frozen_ids: Final = frozenset(task_ids)
return [
_train_row(row)
for row in dataset_rows
if isinstance(row.get("instance_id"), str) and row["instance_id"] in frozen_ids
]
def _task_id(row: object) -> str:
if not isinstance(row, Mapping):
raise ValueError("Each tasks.json task must be an object")
task_id: Final = row.get("instance_id")
if not isinstance(task_id, str):
raise ValueError("Each tasks.json task must have a string instance_id")
return task_id
def _frozen_task_ids() -> tuple[str, ...]:
manifest: Final[object] = json.loads((ROOT / "tasks.json").read_text())
if not isinstance(manifest, Mapping):
raise ValueError("tasks.json must contain an object")
tasks: Final[object] = manifest.get("tasks")
if not isinstance(tasks, list):
raise ValueError("tasks.json must contain a tasks list")
return tuple(_task_id(task) for task in tasks)
def _write_train(path: Path, rows: Sequence[Mapping[str, object]]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("".join(json.dumps(row) + "\n" for row in rows))
def prepare(check: bool = False) -> None:
rows = _load_rows()
tasks = _select_tasks(rows)
task_ids: Final = _frozen_task_ids()
rows: Final = agent_rows(_load_rows(), task_ids)
train_path: Final = ROOT / "agent_dataset" / "train.jsonl"
if not check:
_write_outputs(ROOT, rows, tasks)
_write_train(train_path, rows)
return
with tempfile.TemporaryDirectory(prefix="routing-poc-prepare-") as temp_dir:
generated_dir = Path(temp_dir)
_write_outputs(generated_dir, rows, tasks)
generated_tasks = json.loads((generated_dir / "tasks.json").read_text())
existing_tasks = json.loads((ROOT / "tasks.json").read_text())
generated_ids = [task["instance_id"] for task in generated_tasks["tasks"]]
existing_ids = [task["instance_id"] for task in existing_tasks["tasks"]]
if generated_ids != existing_ids:
raise AssertionError("Regenerated task IDs do not match tasks.json")
generated_train = (generated_dir / "agent_dataset" / "train.jsonl").read_bytes()
existing_train = (ROOT / "agent_dataset" / "train.jsonl").read_bytes()
if generated_train != existing_train:
generated_train: Final = Path(temp_dir) / "agent_dataset" / "train.jsonl"
_write_train(generated_train, rows)
if generated_train.read_bytes() != train_path.read_bytes():
raise AssertionError("Regenerated train.jsonl is not byte-identical")
print(f"Dataset preparation check passed for {len(tasks)} task IDs and byte-identical train.jsonl")
print(f"Dataset preparation check passed for {len(task_ids)} task IDs and byte-identical train.jsonl")
def main() -> None:

111
cookbook/capability_routing_pilot/selection_candidates.json Normal file
View file

@ -0,0 +1,111 @@
{
"seed": "litellm-capability-routing-pilot-v1",
"rule": "sort dev ids by sha256(seed+id); take first 5 with public swebench eval image",
"candidates": [
{
"instance_id": "pvlib__pvlib-python-1395",
"image": "swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1395:latest",
"exists": true,
"size": 1692788107,
"digest": "sha256:c7b51f4978926d54a788c54ac4d96e6c4c4e2cd2ee62736b30e91c9fd0891f10"
},
{
"instance_id": "sqlfluff__sqlfluff-884",
"image": "swebench/sweb.eval.x86_64.sqlfluff_1776_sqlfluff-884:latest",
"exists": true,
"size": 1038961212,
"digest": "sha256:ca28dab1d331c1c7e4908784576a7e1f3b11387dbaa4a8db466ce42a0eda6bec"
},
{
"instance_id": "pylint-dev__astroid-1614",
"image": "swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1614:latest",
"exists": true,
"size": 977627664,
"digest": "sha256:7206ea531b6c8b02f7a8e1760d732e4651f95307781f12f55e4b88e982610159"
},
{
"instance_id": "pvlib__pvlib-python-1165",
"image": "swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1165:latest",
"exists": true,
"size": 1720675327,
"digest": "sha256:71e033964894077384c0c27a04ae0271e28eaad35602aa7570d40c683f1967d1"
},
{
"instance_id": "pydicom__pydicom-1031",
"image": "swebench/sweb.eval.x86_64.pydicom_1776_pydicom-1031:latest",
"exists": true,
"size": 1623110388,
"digest": "sha256:0b962e881760f4b8bf3cdbfc695624c90906198ccf1e2410cd639f6adc6e340b"
},
{
"instance_id": "pylint-dev__astroid-1164",
"image": "swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1164:latest",
"exists": true,
"size": 977029021,
"digest": "sha256:2aaa44d8f0abfaa47e2b7b378b204e2341d7a98525888c58f64ec83fced446eb"
},
{
"instance_id": "pylint-dev__astroid-2240",
"image": "swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-2240:latest",
"exists": true,
"size": 978175766,
"digest": "sha256:0bc107e0e3eedcf47322c8ed22ec961cf08b9b394688b0f1c86113378b4d768d"
},
{
"instance_id": "pydicom__pydicom-793",
"image": "swebench/sweb.eval.x86_64.pydicom_1776_pydicom-793:latest",
"exists": true,
"size": 1132897477,
"digest": "sha256:9ad524560667adaa8367dad49c1dbad7aa9778c692c26c8c68edd507fe64f942"
},
{
"instance_id": "pvlib__pvlib-python-1157",
"image": "swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1157:latest",
"exists": false,
"size": null,
"digest": null
},
{
"instance_id": "pvlib__pvlib-python-1191",
"image": "swebench/sweb.eval.x86_64.pvlib_1776_pvlib-python-1191:latest",
"exists": true,
"size": 1725700379,
"digest": "sha256:0678cd6b58ff5b69756945921942444856a5038f82da465259d5a49bf95e7aad"
},
{
"instance_id": "pylint-dev__astroid-927",
"image": "swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-927:latest",
"exists": true,
"size": 977323274,
"digest": "sha256:9ca6c5ea2a9f260e64f7a2fca563fb60a6d2a58bff54f62a3fd3dc75937deba5"
},
{
"instance_id": "pydicom__pydicom-900",
"image": "swebench/sweb.eval.x86_64.pydicom_1776_pydicom-900:latest",
"exists": true,
"size": 1132917193,
"digest": "sha256:e6ca9a40b5a43f0648d6197a9c5ad66c19137812a83337d758a3c166aadd732d"
},
{
"instance_id": "pydicom__pydicom-1000",
"image": "swebench/sweb.eval.x86_64.pydicom_1776_pydicom-1000:latest",
"exists": true,
"size": 1133581628,
"digest": "sha256:e26f22c2bb3e3db1ecb93a467c68260fa97c094676d86f820f43259481e25d90"
},
{
"instance_id": "pylint-dev__astroid-1616",
"image": "swebench/sweb.eval.x86_64.pylint-dev_1776_astroid-1616:latest",
"exists": true,
"size": 977625544,
"digest": "sha256:4685f3bf018042458da5aa31cbb06cca716f526a4ff2c23e095630328402b90e"
},
{
"instance_id": "sqlfluff__sqlfluff-1577",
"image": "swebench/sweb.eval.x86_64.sqlfluff_1776_sqlfluff-1577:latest",
"exists": false,
"size": null,
"digest": null
}
]
}

40
cookbook/capability_routing_pilot/tests/test_poc.py
View file

@ -19,6 +19,7 @@ from poc.guarded_model import (
valid_cost,
)
from poc.label import parse_jev_response, parse_label_response
from poc.prepare import agent_rows
from poc.policy import (
PolicyConfig,
build_capability_map,
@ -614,3 +615,42 @@ def test_input_estimate_covers_tool_overhead_and_tracks_observed_prompt_growth()
assert estimate_input_tokens(9000, 5000, 7000, config) == 11000
assert estimate_input_tokens(5100, 5000, 5200, config) == 6124
assert estimate_input_tokens(4000, 5000, 7000, config) == 7000
def test_agent_rows_follows_dataset_order_and_excludes_gold_and_test_fields() -> None:
shared_fields = {
"repo": "owner/repo",
"base_commit": "commit",
"problem_statement": "Fix behavior",
"version": "1.0",
"created_at": "2020-01-01",
"patch": "gold patch",
"test_patch": "gold test patch",
"FAIL_TO_PASS": ["tests/test_case.py"],
}
dataset_rows = [
{"instance_id": "task-second", **shared_fields},
{"instance_id": "task-first", **shared_fields},
]
rows = agent_rows(dataset_rows, ("task-first", "task-second"))
assert [row["instance_id"] for row in rows] == ["task-second", "task-first"]
assert all(
set(row)
== {
"instance_id",
"repo",
"base_commit",
"problem_statement",
"version",
"created_at",
}
for row in rows
)
assert all("patch" not in row and "test_patch" not in row and "FAIL_TO_PASS" not in row for row in rows)
def test_agent_rows_fails_when_a_frozen_task_is_missing() -> None:
with pytest.raises(ValueError, match="Frozen task IDs are missing"):
agent_rows(({"instance_id": "present"},), ("missing",))