Chris Estreich
a4d2de4534
Add pass / fail events for evals ( #2656 )
2025-04-15 11:29:38 -07:00
Chris Estreich
5b48a2d708
More small evals tweaks ( #2620 )
2025-04-14 21:13:08 -07:00
nobuo kawasaki
30c3a65b7b
Fix eslint error about --ext option by remove it ( #2543 )
2025-04-14 16:25:09 -07:00
Chris Estreich
d73789cfe4
Update default settings for evals ( #2601 )
2025-04-14 08:15:20 -07:00
Chris Estreich
4ecec98384
Support all providers in evals settings ( #2573 )
2025-04-13 10:30:06 -07:00
Chris Estreich
352a6b9c75
More unit test kill -9 fixes ( #2570 )
...
* More unit test kill -9 fixes
* Fix linter warning
2025-04-13 09:44:45 -07:00
Chris Estreich
ef9b339027
Evals improvements ( #2555 )
...
* Evals improvements
* Remove debugging
2025-04-13 00:57:33 -07:00
mecab
37cfd75ebf
Add Anthropic option to pass API Token as Authorization header instead of X-Api-Key for the custom base URL ( #2531 )
...
Add Anthropic option to use authToken over apiKey
2025-04-12 22:20:03 -04:00
Chris Estreich
2c8304ea31
Fix node version string when running asdf install nodejs ( #2524 )
2025-04-12 01:06:03 -07:00
Chris Estreich
91178628aa
Evals enhancements: delete runs, show all run instead of just completed runs ( #2520 )
2025-04-12 00:07:30 -07:00
Chris Estreich
df01e7948b
Add reasoningEffort to provider settings schema ( #2518 )
2025-04-11 22:45:14 -07:00
Chris Estreich
6ab9aa9f15
Control evals concurrency in web app ( #2265 )
2025-04-11 22:21:15 -07:00
Chris Estreich
2eba534dd6
Evals fixes ( #2505 )
...
* Allow Turso URLs, add support for API providers beyond OpenRouter
* Make the git branch name unique
2025-04-11 14:54:03 -07:00
ronyblum
a031b74bf5
Modification of AWS Bedrock to Amazon Bedrock ( #2473 )
...
* Modification of AWS Bedrock to Amazon Bedrock
* Duplicated comment removal
2025-04-10 15:47:34 -04:00
Chris Estreich
4e716263d0
Add a script to copy eval run results to Turso ( #2452 )
2025-04-09 21:18:07 -07:00
Chris Estreich
5fa555e60d
Fix gh fork command ( #2442 )
2025-04-09 13:12:55 -07:00
Chris Estreich
6d9ebe3fcf
More sane evals default concurrency + staggered startup ( #2441 )
2025-04-09 13:04:36 -07:00
Ross McFarland
58149a083e
Clean up global rateLimitSeconds & fully shift to provider version ( #2408 )
...
* Directly use provider rateLimitSeconds and remove uneeded default
* remove a bunch of unused rateLimitSeconds references
* rateLimitSettings field def in GlobalSettingsRecord isn't needed for migration
2025-04-08 11:21:54 -04:00
Chris Estreich
f747215988
Improve evals settings import ( #2298 )
2025-04-04 10:33:32 -07:00
Chris Estreich
8f356711f1
Ironing out some kinks when running evals with high parallelism ( #2280 )
2025-04-04 09:08:12 -07:00
Chris Estreich
49a61ca30f
Support bash 3.x in our evals setup script ( #2248 )
2025-04-02 20:53:36 -07:00
Matt Rubens
3d53386f1e
Remove the experimental unified diff ( #2246 )
2025-04-02 23:13:46 -04:00
Chris Estreich
e050043e58
Typos + promisify exec calls ( #2243 )
2025-04-02 17:12:03 -07:00
Chris Estreich
d15f813514
Evals ( #2238 )
...
* Evals
* Remove redundant line
* Fix tsc error
* Disable debug mode
* Add option to kill run
2025-04-02 16:49:41 -07:00
