This commit is contained in:
N 2026-10-11 12:24:45 +01:00 • committed by GitHub
commit 8564b2a512
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 16 additions and 7 deletions

View file

@ -99,7 +99,7 @@ Given target availability + window, computes:
- Allowed downtime in the window
- Multi-window burn-rate thresholds per Google SRE Workbook (Chapter 5):
- **Fast burn** — page if 2% of monthly budget consumed in 1 hour
- **Slow burn** — page if 10% consumed in 6 hours, ticket if 10% in 3 days
- **Slow burn** — page if 5% consumed in 6 hours, ticket if 10% in 3 days
- Recommended alerting rules (PromQL-shaped output)
```bash
@ -140,8 +140,13 @@ See `references/sli_design.md` for examples and anti-patterns.
For 99.9% SLO over 30 days:
- Allowed unavailability: `0.1% × 30 × 24 × 60 = 43.2 minutes`
- 1-hour fast-burn threshold (2% of monthly budget burned): `2% × 43.2 / 60 ≈ 1.44 ratio multiplier`
- 6-hour slow-burn threshold (10% in 6h): `10% × 43.2 / 360 ≈ 0.6 ratio multiplier`
- Burn rate is the observed bad-event fraction divided by the allowed bad-event fraction: `observed_bad_fraction / (1 - target_percent / 100)`.
- Threshold for consuming a chosen budget fraction over an alert window: `budget_fraction × SLO_window_hours / alert_window_hours`.
- 1-hour fast-burn threshold (2% of 30-day budget): `0.02 × 720 / 1 = 14.4`.
- 6-hour slow-burn threshold (5% in 6h): `0.05 × 720 / 6 = 6`.
- 3-day ticket threshold (10% in 72h): `0.10 × 720 / 72 = 1`.
These are dimensionless burn-rate multipliers, not error-rate percentages. For a 99.9% target, a burn rate of 14.4 corresponds to a bad-event fraction of `14.4 × 0.001 = 0.0144` (1.44%). Thresholds scale with the SLO reporting window: the corresponding 28-day multipliers are 13.44, 5.6, and approximately 0.933.
`error_budget_calculator.py` does this math for you and emits ready-to-paste alert rules.

View file

@ -27,7 +27,7 @@ abort_if: "p99 > 1000ms OR error_rate > 1%"
After (SLO-driven):
```
abort_if: "burn_rate.fast > 14.4 over 1h (per SLO checkout-success)"
abort_if: "burn_rate.fast > 14.4 over 1h (per 30-day SLO checkout-success)"
```
Wire-up:
@ -79,7 +79,7 @@ Goal: ship a new checkout flow.
2. **Compute burn-rate alerts** (slo-architect):
```bash
error_budget_calculator.py --target 99.9 --window-days 28
# → fast_burn threshold = 14.4
# → fast_burn threshold = 13.44 (2% of a 28-day budget in 1h)
```
3. **Define rollout** (feature-flags-architect):
@ -91,7 +91,7 @@ Goal: ship a new checkout flow.
4. **Wire the abort** (feature-flags-architect):
```yaml
abort_if: "burn_rate.fast > 14.4 (per SLO slo-checkout-svc-...)"
abort_if: "burn_rate.fast > 13.44 (per 28-day SLO slo-checkout-svc-...)"
```
5. **Validate via chaos** before going wide (chaos-engineering):

View file

@ -45,13 +45,17 @@ The alert fires only when BOTH windows show high burn. This filters spikes (only
### Recommended thresholds
For a 30-day SLO reporting window:
| Alert | Long window | Short window | Burn rate threshold | % budget at fire | Severity |
|---|---|---|---|---|---|
| Fast burn | 1h | 5m | 14.4 | 2% in 1h | page |
| Slow burn | 6h | 30m | 6 | 5% in 6h | page |
| Ticket | 3d | 6h | 1 | 10% in 3d | ticket |
The numbers come from: `burn_rate × bad_event_rate > slo_target_violation_rate`.
Burn rate is `observed_bad_fraction / allowed_bad_fraction`, where `allowed_bad_fraction = 1 - target_percent / 100`. To consume a chosen budget fraction over a long alert window, use `burn_rate_threshold = budget_fraction × SLO_window_hours / long_alert_window_hours`. An error-ratio alert compares `observed_bad_fraction > burn_rate_threshold × allowed_bad_fraction` in both its long and short windows.
For 30 days (720 hours), the multipliers above are `0.02 × 720 / 1 = 14.4`, `0.05 × 720 / 6 = 6`, and `0.10 × 720 / 72 = 1`. For 28 days (672 hours), the same budget fractions and alert windows give 13.44, 5.6, and approximately 0.933. The target determines the allowed bad-event fraction; it does not change these multipliers.
`error_budget_calculator.py` computes these for any target+window. Output is PromQL-shaped: