Commit graph

27 commits

Author SHA1 Message Date
Chris Estreich
796ee5c0e3
Migrate evals database when deploying roo-code-website (#6146)
Evals db migrate
2025-07-30 20:12:21 -04:00
Chris Estreich
9956cc1f48
Use SIGKILL for command execution timeouts in the "execa" variant (#6071) 2025-07-23 01:47:10 -07:00
Chris Estreich
984d368f7a
Fix evals; broken by #5865 (#6065) 2025-07-22 09:18:38 -07:00
roomote[bot]
c96b399dd7
feat: add configurable timeout for evals (5-10 min) (#5865)
* feat: add configurable timeout for evals (5-10 min)

- Add timeout field to CreateRun schema with min 5, max 10, default 5
- Add timeout slider UI component to /runs/new page
- Update database schema to include timeout column in runs table
- Create migration to add timeout column with default value of 5
- Update runTask.ts to use configurable timeout from run settings
- Pass timeout parameter through the createRun action

* fix: remove unused EVALS_TIMEOUT import

* fix: add timeout field to createRun calls in copyRun test

- Added timeout: 5 to both createRun calls in copyRun.spec.ts
- This fixes the test failure caused by the new required timeout field in the runs schema
- The timeout field was added in the configurable timeout feature but the test was not updated

* fix: use configurable timeout for Redis key expiration in registerRunner

- Updated registerRunner function to accept timeoutSeconds parameter
- Modified call in runTask.ts to pass configurable timeout instead of hardcoded EVALS_TIMEOUT
- Removed unused EVALS_TIMEOUT import from redis.ts
- Ensures Redis keys remain valid for the entire duration of task execution (up to 10 minutes)

---------

Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: hannesrudolph <hrudolph@gmail.com>
2025-07-19 22:20:23 -04:00
roomote[bot]
5a459d0516
Fix Docker port conflicts for evals services (#5909)
Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
2025-07-19 00:10:10 -04:00
Chris Estreich
99448fc913
Minor fixes for local (non-Docker) evals (#5604)
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-07-11 12:15:21 -07:00
Noritaka Kobayashi
ab55854702
chore: fix typos in comment & document (#5569) 2025-07-10 11:33:26 -04:00
Steven T. Cramer
fca4bea7c3
Update evals Docker setup to work on Windows. (#4656) 2025-06-17 11:23:43 -07:00
Chris Estreich
62c3914034
Farewell jest (#4607)
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-06-16 21:39:45 -07:00
Chris Estreich
fe52c19fde
Evals fixes (#4599) 2025-06-12 13:01:59 -07:00
Chris Estreich
395f55b31f
Convert jest tests to vitest and disable default watch mode for vitest (#4568)
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-06-11 14:48:01 -07:00
Chris Estreich
ab01fb3bdb
Move to Postgres for evals on roocode.com (#4520) 2025-06-11 14:17:53 -07:00
Chris Estreich
083ac9333a
Revert "chore(deps): update postgres docker tag to v17" (#4557) 2025-06-11 10:32:11 -07:00
renovate[bot]
cf7c4a13a0
chore(deps): update postgres docker tag to v17 (#4350)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2025-06-10 13:50:29 -07:00
Chris Estreich
8d5dab3518
GHA evals (#4472) 2025-06-10 11:01:54 -07:00
John Richmond
736c4b1810
Improve eval docker ports (#4489) 2025-06-09 21:41:18 -07:00
Chris Estreich
f5c53f1ed7
Retry tasks that disconnect before completion or timeout (#4461) 2025-06-09 14:26:51 -07:00
Chris Estreich
2bf4347cc3
Document how to add evals (#4470) 2025-06-09 14:26:19 -07:00
Chris Estreich
73ed9f2b26
Docker cleanup script (#4469) 2025-06-09 14:25:13 -07:00
Chris Estreich
52673b3721
Harden evals with retry logic + centralize logs on Docker host (#4440)
* Harden evals with retry logic + centralize logs on Docker host

* Don't delete controller container on exit

* Add randomness

* More tweaks

* More tweaks
2025-06-07 08:58:09 -07:00
Chris Estreich
6a232a99c6
Prevent hanging processes from preventing container cleanup (#4428) 2025-06-06 17:07:31 -07:00
Chris Estreich
39d7f559d2
Update evals README.md (#4420)
* Update README.md

* Update README.md
2025-06-06 09:04:11 -07:00
Chris Estreich
f61b4e46a8
Fix git clone in evals setup script (#4418) 2025-06-06 08:47:43 -07:00
Chris Estreich
24851b96db
Improve the evals "run status" section (#4400) 2025-06-05 22:00:15 -07:00
Chris Estreich
39d6535f7c
Fix rust evals (run cargo with bash) + logging tweaks (#4392) 2025-06-05 16:40:34 -07:00
Chris Estreich
cb5b9c3718
Improve Docker setup for evals (#4327) 2025-06-05 12:16:36 -07:00
Chris Estreich
d87f890556
Move evals into pnpm workspace, switch from SQLite to Postgres (#4278) 2025-06-03 01:38:08 -07:00