Chris Estreich
424bce6078
Add an option to use our cli for evals ( #10456 )
...
Co-authored-by: Roo Code <roomote@roocode.com>
2026-01-05 11:33:19 -08:00
Hannes Rudolph
ca1bc18a21
Fix rate limit wait display ( #10389 )
...
Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
2025-12-30 21:56:32 -08:00
Hannes Rudolph
bd78a63844
feat(evals): add message log deduper utility ( #10286 )
2025-12-22 19:42:38 -07:00
Hannes Rudolph
08eed65aff
fix(evals): add missing packages/core to Dockerfile.runner ( #10272 )
2025-12-22 08:42:49 -08:00
Hannes Rudolph
84c5d2fd61
feat(evals): improve evals UI with tool groups and duration fix ( #10133 )
...
Co-authored-by: Roo Code <roomote@roocode.com>
2025-12-16 12:43:39 -08:00
Hannes Rudolph
c103a4a639
feat: streaming tool stats + token usage throttling ( #9926 )
...
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
2025-12-08 19:40:12 -08:00
Matt Rubens
1f7e1ee630
Make eval runs deleteable ( #9909 )
2025-12-07 23:30:28 -05:00
Hannes Rudolph
8433eafb05
feat(evals-ui): Add filtering, bulk delete, tool consolidation, and run notes ( #9837 )
2025-12-04 14:28:37 -07:00
Hannes Rudolph
d2017c80cf
Add web-evals updates and kill run functionality ( #9681 )
...
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
Co-authored-by: Roo Code <roomote@roocode.com>
2025-11-29 13:08:56 -07:00
Hannes Rudolph
3f0a6971ca
feat(web-evals): add task log viewing, export failed logs, and new run options ( #9637 )
...
Co-authored-by: roomote[bot] <219738659+roomote[bot]@users.noreply.github.com>
2025-11-27 21:15:31 -07:00
Chris Estreich
cad6145241
Switch from asdf to mise-en-place in bare-metal evals setup script ( #9548 )
2025-11-24 15:39:47 -08:00
Chris Estreich
2ca9eac9e0
Enable the Roo Code Cloud provider in evals ( #9492 )
2025-11-22 00:06:03 -08:00
Chris Estreich
045c2d6b3b
Retry eval tasks if API instability detected ( #9365 )
2025-11-18 11:45:54 -08:00
Chris Estreich
25f61943ca
Set port 3446 for web-evals in production mode ( #8288 )
2025-09-24 08:43:05 -07:00
Chris Estreich
0e1b23d09c
Bare metal evals fixes ( #8224 )
...
Co-authored-by: Roo Code <roomote@roocode.com>
2025-09-22 12:54:56 -07:00
Chris Estreich
247da38b02
Add model info to eval runs table ( #7749 )
2025-09-06 22:10:08 -07:00
Chris Estreich
c7d7ad8197
Fix evals ( #7547 )
2025-08-29 20:10:58 -07:00
Chris Estreich
37330b0209
Bring back "Use @roo-code/cloud from npm" ( #6795 )
2025-08-06 20:51:41 -10:00
Matt Rubens
5c05762333
Revert "Migrate evals database when deploying roo-code-website" ( #6525 )
2025-07-31 19:27:55 -04:00
Chris Estreich
796ee5c0e3
Migrate evals database when deploying roo-code-website ( #6146 )
...
Evals db migrate
2025-07-30 20:12:21 -04:00
Chris Estreich
9956cc1f48
Use SIGKILL for command execution timeouts in the "execa" variant ( #6071 )
2025-07-23 01:47:10 -07:00
Chris Estreich
984d368f7a
Fix evals; broken by #5865 ( #6065 )
2025-07-22 09:18:38 -07:00
roomote[bot]
c96b399dd7
feat: add configurable timeout for evals (5-10 min) ( #5865 )
...
* feat: add configurable timeout for evals (5-10 min)
- Add timeout field to CreateRun schema with min 5, max 10, default 5
- Add timeout slider UI component to /runs/new page
- Update database schema to include timeout column in runs table
- Create migration to add timeout column with default value of 5
- Update runTask.ts to use configurable timeout from run settings
- Pass timeout parameter through the createRun action
* fix: remove unused EVALS_TIMEOUT import
* fix: add timeout field to createRun calls in copyRun test
- Added timeout: 5 to both createRun calls in copyRun.spec.ts
- This fixes the test failure caused by the new required timeout field in the runs schema
- The timeout field was added in the configurable timeout feature but the test was not updated
* fix: use configurable timeout for Redis key expiration in registerRunner
- Updated registerRunner function to accept timeoutSeconds parameter
- Modified call in runTask.ts to pass configurable timeout instead of hardcoded EVALS_TIMEOUT
- Removed unused EVALS_TIMEOUT import from redis.ts
- Ensures Redis keys remain valid for the entire duration of task execution (up to 10 minutes)
---------
Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: hannesrudolph <hrudolph@gmail.com>
2025-07-19 22:20:23 -04:00
roomote[bot]
5a459d0516
Fix Docker port conflicts for evals services ( #5909 )
...
Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
2025-07-19 00:10:10 -04:00
Chris Estreich
99448fc913
Minor fixes for local (non-Docker) evals ( #5604 )
...
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-07-11 12:15:21 -07:00
Noritaka Kobayashi
ab55854702
chore: fix typos in comment & document ( #5569 )
2025-07-10 11:33:26 -04:00
Steven T. Cramer
fca4bea7c3
Update evals Docker setup to work on Windows. ( #4656 )
2025-06-17 11:23:43 -07:00
Chris Estreich
62c3914034
Farewell jest ( #4607 )
...
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-06-16 21:39:45 -07:00
Chris Estreich
fe52c19fde
Evals fixes ( #4599 )
2025-06-12 13:01:59 -07:00
Chris Estreich
395f55b31f
Convert jest tests to vitest and disable default watch mode for vitest ( #4568 )
...
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-06-11 14:48:01 -07:00
Chris Estreich
ab01fb3bdb
Move to Postgres for evals on roocode.com ( #4520 )
2025-06-11 14:17:53 -07:00
Chris Estreich
083ac9333a
Revert "chore(deps): update postgres docker tag to v17" ( #4557 )
2025-06-11 10:32:11 -07:00
renovate[bot]
cf7c4a13a0
chore(deps): update postgres docker tag to v17 ( #4350 )
...
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2025-06-10 13:50:29 -07:00
Chris Estreich
8d5dab3518
GHA evals ( #4472 )
2025-06-10 11:01:54 -07:00
John Richmond
736c4b1810
Improve eval docker ports ( #4489 )
2025-06-09 21:41:18 -07:00
Chris Estreich
f5c53f1ed7
Retry tasks that disconnect before completion or timeout ( #4461 )
2025-06-09 14:26:51 -07:00
Chris Estreich
2bf4347cc3
Document how to add evals ( #4470 )
2025-06-09 14:26:19 -07:00
Chris Estreich
73ed9f2b26
Docker cleanup script ( #4469 )
2025-06-09 14:25:13 -07:00
Chris Estreich
52673b3721
Harden evals with retry logic + centralize logs on Docker host ( #4440 )
...
* Harden evals with retry logic + centralize logs on Docker host
* Don't delete controller container on exit
* Add randomness
* More tweaks
* More tweaks
2025-06-07 08:58:09 -07:00
Chris Estreich
6a232a99c6
Prevent hanging processes from preventing container cleanup ( #4428 )
2025-06-06 17:07:31 -07:00
Chris Estreich
39d7f559d2
Update evals README.md ( #4420 )
...
* Update README.md
* Update README.md
2025-06-06 09:04:11 -07:00
Chris Estreich
f61b4e46a8
Fix git clone in evals setup script ( #4418 )
2025-06-06 08:47:43 -07:00
Chris Estreich
24851b96db
Improve the evals "run status" section ( #4400 )
2025-06-05 22:00:15 -07:00
Chris Estreich
39d6535f7c
Fix rust evals (run cargo with bash) + logging tweaks ( #4392 )
2025-06-05 16:40:34 -07:00
Chris Estreich
cb5b9c3718
Improve Docker setup for evals ( #4327 )
2025-06-05 12:16:36 -07:00
Chris Estreich
d87f890556
Move evals into pnpm workspace, switch from SQLite to Postgres ( #4278 )
2025-06-03 01:38:08 -07:00
