diff --git a/benchmark/README.md b/benchmark/README.md index a40d7a3bdd..7f6c8810eb 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -1,5 +1,11 @@ # Benchmark Harness +## Get Started + +[Install](https://docs.docker.com/desktop/) Docker Desktop. + +## Local Debugging + Install nvm: ```sh @@ -27,11 +33,46 @@ Configure database: ```sh cp packages/server/.env.sample packages/server/.env # Update BENCHMARKS_DB_PATH as needed in `packages/server/.env`. -pnpm --filter @benchmark/server db:migrate +pnpm --filter @benchmark/db db:push ``` -Run the web ui: +Run the web app: ```sh pnpm web ``` + +Run an exercise via the cli: + +```sh +pnpm cli run [cpp|go|java|javascript|python|rust|all] [|all] +``` + +## Extension Runner + +The extension runner (`@benchmark/runner`) is responsible for spawning VSCode instances and running Roo Code on the benchmark exercises. It leverages the [`@vscode/test-electron`](https://github.com/Microsoft/vscode-test) package to do this. + +VS Code provides two CLI parameters for running extension tests, `--extensionDevelopmentPath` and `--extensionTestsPath`. + +For example: + +```sh +# - Launches VS Code Extension Host +# - Loads the extension at +# - Executes the test runner script at +code \ + --extensionDevelopmentPath= \ + --extensionTestsPath= +``` + +If you make extension code changes then you need to re-build it so that `extensionDevelopmentPath` has the latest version transpiled: + +```sh +pnpm build:extension +``` + +If you make changes to `@benchmark/runner` or any of its dependencies then you should rebuild it: + +```sh +pnpm build:runner +``` diff --git a/benchmark/packages/runner/README.md b/benchmark/packages/runner/README.md deleted file mode 100644 index 8c747e9572..0000000000 --- a/benchmark/packages/runner/README.md +++ /dev/null @@ -1,27 +0,0 @@ -# Extension Runner - -VS Code provides two CLI parameters for running extension tests, `--extensionDevelopmentPath` and `--extensionTestsPath`. - -For example: - -```sh -# - Launches VS Code Extension Host -# - Loads the extension at -# - Executes the test runner script at -code \ - --extensionDevelopmentPath= \ - --extensionTestsPath= -``` - -If you make extension code changes then you need to re-build it so that `extensionDevelopmentPath` has the latest version transpiled: - -```sh -# From the repository root: -pnpm build:extension -``` - -If you make changes to `@benchmark/runner` or any of its dependencies then you should rebuild it: - -```sh -pnpm build:runner -```