mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-07 02:59:05 +00:00
ci(codspeed): pin benchmark runner to ubuntu-24.04 (#31746)
* ci(codspeed): pin benchmark runner to ubuntu-24.04 ubuntu-latest resolves to different runner images between the BASE (main/staging) and HEAD (PR) runs, so CodSpeed reports 'Different runtime environments detected' and emits false-positive regressions (e.g. a -25.2% swing on test_completion_multi_turn in #31684, an MCP auth fix with no LLM code changes). Pinning the runner to a fixed image keeps BASE and HEAD on the same hardware so 1 ms swings on a ~3 ms benchmark stop blocking unrelated PRs. Fixes #31738 * ci(codspeed): stop running benchmarks on litellm_internal_staging The CodSpeed check flip-flops on internal staging and on PRs targeting it (e.g. "+11.75% improvement" on one run, "-25.36% regression" on the next) because the comparison flags "different runtime environments" and the benchmarks are only 3-4 ms, so sub-millisecond runner noise swings the result by 25-30%. Pinning the runner to ubuntu-24.04 in this PR helps the head side, but the internal_staging base is still recorded on the old unpinned runner, so comparisons keep flapping until the pin merges and the base is re-baselined. Until that settles, the red X's on internal staging make the OSS project look unhealthy and confuse contributors, so drop the litellm_internal_staging push and pull_request triggers and keep CodSpeed running on main only. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
This commit is contained in:
parent
a0b26d2c3c
commit
ae8084de74
1 changed files with 1 additions and 3 deletions
4
.github/workflows/codspeed.yml
vendored
4
.github/workflows/codspeed.yml
vendored
|
|
@ -4,11 +4,9 @@ on:
|
|||
push:
|
||||
branches:
|
||||
- main
|
||||
- litellm_internal_staging
|
||||
pull_request:
|
||||
branches:
|
||||
- main
|
||||
- litellm_internal_staging
|
||||
# Allow CodSpeed to trigger backtest performance analysis
|
||||
# in order to generate initial data
|
||||
workflow_dispatch:
|
||||
|
|
@ -23,7 +21,7 @@ concurrency:
|
|||
|
||||
jobs:
|
||||
benchmarks:
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue