Commit graph

25 commits

Author SHA1 Message Date
Daniel
6cfa82f571
Revert to pre-AI-SDK state (January 29, 2026) (#11462)
Revert to pre-AI-SDK state (commit 67e568f6b)

This commit reverts the codebase to the state before AI SDK migration work began.

Target commit: 67e568f6b - refactor: replace fetch_instructions with skill tool and built-in skills (#10913)
Date: January 29, 2026

This removes approximately 152 commits of AI SDK migration work.
A follow-up PR will add back bug fixes and features that are unrelated to AI SDK.

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-13 16:45:18 -05:00
Danger Mouse
571be71005
Pr 11144 (#11315)
* Latest main branch snapshot from API

* feat: add dedicated Azure OpenAI provider using @ai-sdk/azure package

* feat: add Azure provider UI component and translations

* feat: add Azure provider translations for all locales

* chore: add missing Azure placeholder translations

* Delete .changeset/azure-ai-sdk-migration.md

* fix: add Azure provider validation for onboarding workflow

- Add azureApiKey to SECRET_STATE_KEYS for proper configuration detection
- Add Azure validation case in validateModelsAndKeysProvided
- Add validation translations for azureResourceName and azureDeploymentName across all 18 locales

This fixes the issue where the Finish button does nothing when setting up Azure provider in the onboarding workflow.

* feat(azure): add model metadata, model picker, rename to Azure AI Foundry

- Add static model metadata for 29 Azure models (from models.dev)
  with Roo-specific flags (reasoning, tools, verbosity) matching
  openAiNativeModels
- Add model picker dropdown to Azure provider settings for model
  capability detection (context window, max tokens, pricing)
- Rename provider label from 'Azure OpenAI' to 'Azure AI Foundry'
  across all 18 locales
- Make API key optional (supports Azure managed identity / Entra ID)
- Update default API version from 2024-08-01-preview to 2025-04-01-preview
- Fix maxOutputTokens validation (filter invalid values <= 0)
- Handler separates deployment name (API calls) from model ID
  (capability lookup) with azureDefaultModelInfo (gpt-4o) fallback
- Remove unhelpful 'Get Azure AI Foundry Access' button
- Prevent stale model IDs from other providers carrying over
- Suppress validation errors on fresh provider selection

* fix(azure): add missing isAiSdkProvider() override for reasoning block preservation

* Azure Fixes for Hannes

* Quick Fix for Respones API Only (for Hannes)

* fix: use explicit azureOpenAiDefaultApiVersion fallback when apiVersion is empty

Addresses review feedback: the UI placeholder shows '2025-04-01-preview' via
azureOpenAiDefaultApiVersion, so the handler should use the same constant as
fallback instead of silently deferring to the SDK's internal default.

* fix: remove stale Cerebras references (retired provider)

* fix: add missing retiredProviderMessage translations for all locales

* fix: do not map promptCacheMissTokens to cacheWriteTokens for Azure

Azure uses OpenAI-compatible caching which does not report cache write
tokens separately. promptCacheMissTokens represents tokens NOT found in
cache (processed from scratch), not tokens written to cache. This aligns
the Azure handler with the OpenAI native handler behavior.

---------

Co-authored-by: Hannes Rudolph <hrudolph@gmail.com>
Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: daniel-lxs <ricciodaniel98@gmail.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
2026-02-09 14:11:12 -07:00
Hannes Rudolph
8de9337e63
chore: remove XML tool calling support (#10841)
Co-authored-by: daniel-lxs <ricciodaniel98@gmail.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
2026-01-20 20:25:08 -05:00
Hannes Rudolph
41c5ff6076
feat(web-evals): remember last Roo model selection + add evals skill (#10470)
* feat(web-evals): remember last Roo model selection

* fix(web-evals): reset model selections on provider switch and fix lint warning

- Add useEffect to reset model selections when switching between providers
  This prevents OpenRouter model IDs from persisting when switching to Roo,
  which was causing Roo's stored selection to be overwritten with wrong IDs

- Remove unused 'executionMethod' from onSubmit dependency array to fix
  react-hooks/exhaustive-deps warning

* fix(web-evals): add missing executionMethod to test cases

* fix(web-evals): harden localStorage + keep provider selections
2026-01-07 08:00:34 -07:00
Chris Estreich
424bce6078
Add an option to use our cli for evals (#10456)
Co-authored-by: Roo Code <roomote@roocode.com>
2026-01-05 11:33:19 -08:00
Hannes Rudolph
84c5d2fd61
feat(evals): improve evals UI with tool groups and duration fix (#10133)
Co-authored-by: Roo Code <roomote@roocode.com>
2025-12-16 12:43:39 -08:00
Hannes Rudolph
325410955c
feat(web-evals): improve run logs and formatters (#10081) 2025-12-14 17:36:00 -08:00
Hannes Rudolph
c103a4a639
feat: streaming tool stats + token usage throttling (#9926)
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
2025-12-08 19:40:12 -08:00
Hannes Rudolph
c10d1d9ffc
feat(web-evals): add multi-model launch and UI improvements (#9845)
Co-authored-by: Roo Code <roomote@roocode.com>
2025-12-04 20:48:38 -05:00
Hannes Rudolph
8433eafb05
feat(evals-ui): Add filtering, bulk delete, tool consolidation, and run notes (#9837) 2025-12-04 14:28:37 -07:00
Chris Estreich
648e009b8d
Update the evals keygen command (#9754) 2025-12-02 19:44:11 -08:00
Hannes Rudolph
d2017c80cf
Add web-evals updates and kill run functionality (#9681)
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Matt Rubens <mrubens@users.noreply.github.com>
Co-authored-by: Roo Code <roomote@roocode.com>
2025-11-29 13:08:56 -07:00
Hannes Rudolph
3f0a6971ca
feat(web-evals): add task log viewing, export failed logs, and new run options (#9637)
Co-authored-by: roomote[bot] <219738659+roomote[bot]@users.noreply.github.com>
2025-11-27 21:15:31 -07:00
Hannes Rudolph
4442397507
feat(web-evals): enhance dashboard with dynamic tool columns and UX improvements (#9592)
Co-authored-by: Roo Code <roomote@roocode.com>
2025-11-26 00:15:29 -05:00
Chris Estreich
2ca9eac9e0
Enable the Roo Code Cloud provider in evals (#9492) 2025-11-22 00:06:03 -08:00
Chris Estreich
ef340806c7
Evals web app fixes / tweaks (#7299) 2025-08-21 12:36:33 -07:00
Chris Estreich
9956cc1f48
Use SIGKILL for command execution timeouts in the "execa" variant (#6071) 2025-07-23 01:47:10 -07:00
Chris Estreich
984d368f7a
Fix evals; broken by #5865 (#6065) 2025-07-22 09:18:38 -07:00
roomote[bot]
c96b399dd7
feat: add configurable timeout for evals (5-10 min) (#5865)
* feat: add configurable timeout for evals (5-10 min)

- Add timeout field to CreateRun schema with min 5, max 10, default 5
- Add timeout slider UI component to /runs/new page
- Update database schema to include timeout column in runs table
- Create migration to add timeout column with default value of 5
- Update runTask.ts to use configurable timeout from run settings
- Pass timeout parameter through the createRun action

* fix: remove unused EVALS_TIMEOUT import

* fix: add timeout field to createRun calls in copyRun test

- Added timeout: 5 to both createRun calls in copyRun.spec.ts
- This fixes the test failure caused by the new required timeout field in the runs schema
- The timeout field was added in the configurable timeout feature but the test was not updated

* fix: use configurable timeout for Redis key expiration in registerRunner

- Updated registerRunner function to accept timeoutSeconds parameter
- Modified call in runTask.ts to pass configurable timeout instead of hardcoded EVALS_TIMEOUT
- Removed unused EVALS_TIMEOUT import from redis.ts
- Ensures Redis keys remain valid for the entire duration of task execution (up to 10 minutes)

---------

Co-authored-by: Roo Code <roomote@roocode.com>
Co-authored-by: hannesrudolph <hrudolph@gmail.com>
2025-07-19 22:20:23 -04:00
Chris Estreich
ab01fb3bdb
Move to Postgres for evals on roocode.com (#4520) 2025-06-11 14:17:53 -07:00
Chris Estreich
8d5dab3518
GHA evals (#4472) 2025-06-10 11:01:54 -07:00
Chris Estreich
52673b3721
Harden evals with retry logic + centralize logs on Docker host (#4440)
* Harden evals with retry logic + centralize logs on Docker host

* Don't delete controller container on exit

* Add randomness

* More tweaks

* More tweaks
2025-06-07 08:58:09 -07:00
Chris Estreich
24851b96db
Improve the evals "run status" section (#4400) 2025-06-05 22:00:15 -07:00
Chris Estreich
cb5b9c3718
Improve Docker setup for evals (#4327) 2025-06-05 12:16:36 -07:00
Chris Estreich
d87f890556
Move evals into pnpm workspace, switch from SQLite to Postgres (#4278) 2025-06-03 01:38:08 -07:00