mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
ae8084de74
* ci(codspeed): pin benchmark runner to ubuntu-24.04 ubuntu-latest resolves to different runner images between the BASE (main/staging) and HEAD (PR) runs, so CodSpeed reports 'Different runtime environments detected' and emits false-positive regressions (e.g. a -25.2% swing on test_completion_multi_turn in #31684, an MCP auth fix with no LLM code changes). Pinning the runner to a fixed image keeps BASE and HEAD on the same hardware so 1 ms swings on a ~3 ms benchmark stop blocking unrelated PRs. Fixes #31738 * ci(codspeed): stop running benchmarks on litellm_internal_staging The CodSpeed check flip-flops on internal staging and on PRs targeting it (e.g. "+11.75% improvement" on one run, "-25.36% regression" on the next) because the comparison flags "different runtime environments" and the benchmarks are only 3-4 ms, so sub-millisecond runner noise swings the result by 25-30%. Pinning the runner to ubuntu-24.04 in this PR helps the head side, but the internal_staging base is still recorded on the old unpinned runner, so comparisons keep flapping until the pin merges and the base is re-baselined. Until that settles, the red X's on internal staging make the OSS project look unhealthy and confuse contributors, so drop the litellm_internal_staging push and pull_request triggers and keep CodSpeed running on main only. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
56 lines
1.4 KiB
YAML
56 lines
1.4 KiB
YAML
name: CodSpeed Benchmarks
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- main
|
|
pull_request:
|
|
branches:
|
|
- main
|
|
# Allow CodSpeed to trigger backtest performance analysis
|
|
# in order to generate initial data
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
id-token: write
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
benchmarks:
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 15
|
|
|
|
steps:
|
|
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
|
|
with:
|
|
python-version: "3.12"
|
|
|
|
- name: Set up uv
|
|
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
|
|
with:
|
|
version: "0.10.9"
|
|
|
|
- name: Run benchmarks
|
|
uses: CodSpeedHQ/action@1c8ae4843586d3ba879736b7f6b7b0c990757fab # v4.12.1
|
|
with:
|
|
mode: simulation
|
|
run: >
|
|
env PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
|
|
uv run --frozen --no-default-groups
|
|
--with pytest==8.3.5
|
|
--with pytest-codspeed==4.3.0
|
|
--with "mcp>=1.26.0,<2.0"
|
|
--with "a2a-sdk>=1.1.0,<2.0"
|
|
pytest
|
|
-p pytest_codspeed.plugin
|
|
tests/benchmarks/
|
|
--codspeed
|