mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-22 00:31:44 +00:00
9928da27f3
Addresses three Bugbot concerns flagged on PR #28027 that are real behavioral / coverage gaps: 1. matrix_builder._aggregate_cell now joins every failing tier's error in the published cell instead of silently dropping all but the first. When Haiku 429s and Opus times out on the same cell, both diagnostics land in the matrix JSON so docs-page triage can name both outliers. 2. _aggregate_cell treats 'not_tested' rows as absent data: they're dropped before computing the cell status. Previously a mixed (pass, not_tested) cell silently fell through to 'not_tested', discarding the passing tiers and hiding real coverage from the published matrix. A cell still aggregates to 'not_tested' when *every* row is 'not_tested' (or there are no rows at all). 3. test_v0_layout.py now structurally validates every feature declared in manifest.yaml (directory exists, __init__.py exists, every per-provider test_<provider>.py exists), not just the original six v0 rows. The EXPECTED_FEATURE_IDS / EXPECTED_PROVIDERS anchor constants still pin v0 positions; the new manifest-driven tests extend the same structural guarantees to every post-v0 row so a broken directory in 'count_tokens', 'tool_search', 'web_search', etc. fails CI instead of silently becoming a 'not_tested' cell. Three new builder tests pin the new aggregation behavior: - mixed pass + not_tested surfaces as pass - all-not_tested stays not_tested - multiple fail errors are joined with '; ' Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| fixtures | ||
| __init__.py | ||
| test_matrix_builder.py | ||
| test_v0_layout.py | ||