| .. | ||
| .docker | ||
| scripts | ||
| src | ||
| .env.development | ||
| .env.test | ||
| .gitignore | ||
| ARCHITECTURE.md | ||
| docker-compose.yml | ||
| Dockerfile.runner | ||
| Dockerfile.web | ||
| drizzle.config.ts | ||
| eslint.config.mjs | ||
| package.json | ||
| README.md | ||
| tsconfig.json | ||
| vitest-global-setup.ts | ||
| vitest.config.ts | ||
Run Roo Code Evals
Prerequisites
- Docker Desktop
- git
- That's it!
Setup
Clone the Roo Code repo:
git clone https://github.com/RooCodeInc/Roo-Code.git
cd Roo-Code
Add your OpenRouter API key:
echo "OPENROUTER_API_KEY=sk-or-v1-[...]" > packages/evals/.env.local
Run
Start the evals service:
docker compose -f packages/evals/docker-compose.yml --profile server --profile runner up --build --scale runner=0
The initial build process can take a minute or two. Upon success you should see ouput indicating that a web service is running on localhost:3000:
Additionally, you'll find in Docker Desktop that database and redis services are running:
Navigate to localhost:3000 in your browser and click the 🚀 button.
By default a evals run will run all programming exercises in Roo Code Evals repository with the Claude Sonnet 4 model and default settings. For basic configuration you can specify the LLM to use and any subset of the exercises you'd like. For advanced configuration you can import a Roo Code settings file which will allow you to run the evals with Roo Code configured any way you'd like (this includes custom modes, a footgun prompt, etc).
After clicking "Launch" you should find that a "controller" container has spawned as well as N "task" containers where N is the value you chose for concurrency:
The web app's UI should update in realtime with the results of the eval run:
Advanced Usage / Debugging
The evals system runs VS Code headlessly in Docker containers for consistent, reproducible environments. While this design ensures reliability, it can make debugging more challenging. For debugging purposes, you can run the system locally on macOS, though this approach is less reliable due to hardware and environment variability.
To configure your MacOS system to run evals locally, execute the setup script:
cd packages/evals && ./scripts/setup.sh
The setup script does the following:
- Installs development tools: Homebrew, asdf, GitHub CLI, pnpm
- Installs programming languages: Node.js 20.19.2, Python 3.13.2, Go 1.24.2, Rust 1.85.1, Java 17
- Sets up VS Code with required extensions
- Configures Docker services (PostgreSQL, Redis)
- Clones/updates the evals repository
- Creates and migrates a Postgres database
- Prompts for an OpenRouter API key to add to
.env.local - Optionally builds and installs the Roo Code extension from source