diff --git a/engineering/skills/slo-architect/SKILL.md b/engineering/skills/slo-architect/SKILL.md index e1758733..298f3d2f 100644 --- a/engineering/skills/slo-architect/SKILL.md +++ b/engineering/skills/slo-architect/SKILL.md @@ -99,7 +99,7 @@ Given target availability + window, computes: - Allowed downtime in the window - Multi-window burn-rate thresholds per Google SRE Workbook (Chapter 5): - **Fast burn** — page if 2% of monthly budget consumed in 1 hour - - **Slow burn** — page if 10% consumed in 6 hours, ticket if 10% in 3 days + - **Slow burn** — page if 5% consumed in 6 hours, ticket if 10% in 3 days - Recommended alerting rules (PromQL-shaped output) ```bash @@ -140,8 +140,13 @@ See `references/sli_design.md` for examples and anti-patterns. For 99.9% SLO over 30 days: - Allowed unavailability: `0.1% × 30 × 24 × 60 = 43.2 minutes` -- 1-hour fast-burn threshold (2% of monthly budget burned): `2% × 43.2 / 60 ≈ 1.44 ratio multiplier` -- 6-hour slow-burn threshold (10% in 6h): `10% × 43.2 / 360 ≈ 0.6 ratio multiplier` +- Burn rate is the observed bad-event fraction divided by the allowed bad-event fraction: `observed_bad_fraction / (1 - target_percent / 100)`. +- Threshold for consuming a chosen budget fraction over an alert window: `budget_fraction × SLO_window_hours / alert_window_hours`. +- 1-hour fast-burn threshold (2% of 30-day budget): `0.02 × 720 / 1 = 14.4`. +- 6-hour slow-burn threshold (5% in 6h): `0.05 × 720 / 6 = 6`. +- 3-day ticket threshold (10% in 72h): `0.10 × 720 / 72 = 1`. + +These are dimensionless burn-rate multipliers, not error-rate percentages. For a 99.9% target, a burn rate of 14.4 corresponds to a bad-event fraction of `14.4 × 0.001 = 0.0144` (1.44%). Thresholds scale with the SLO reporting window: the corresponding 28-day multipliers are 13.44, 5.6, and approximately 0.933. `error_budget_calculator.py` does this math for you and emits ready-to-paste alert rules. diff --git a/engineering/skills/slo-architect/references/composition.md b/engineering/skills/slo-architect/references/composition.md index 993b6096..94142302 100644 --- a/engineering/skills/slo-architect/references/composition.md +++ b/engineering/skills/slo-architect/references/composition.md @@ -27,7 +27,7 @@ abort_if: "p99 > 1000ms OR error_rate > 1%" After (SLO-driven): ``` -abort_if: "burn_rate.fast > 14.4 over 1h (per SLO checkout-success)" +abort_if: "burn_rate.fast > 14.4 over 1h (per 30-day SLO checkout-success)" ``` Wire-up: @@ -79,7 +79,7 @@ Goal: ship a new checkout flow. 2. **Compute burn-rate alerts** (slo-architect): ```bash error_budget_calculator.py --target 99.9 --window-days 28 - # → fast_burn threshold = 14.4 + # → fast_burn threshold = 13.44 (2% of a 28-day budget in 1h) ``` 3. **Define rollout** (feature-flags-architect): @@ -91,7 +91,7 @@ Goal: ship a new checkout flow. 4. **Wire the abort** (feature-flags-architect): ```yaml - abort_if: "burn_rate.fast > 14.4 (per SLO slo-checkout-svc-...)" + abort_if: "burn_rate.fast > 13.44 (per 28-day SLO slo-checkout-svc-...)" ``` 5. **Validate via chaos** before going wide (chaos-engineering): diff --git a/engineering/skills/slo-architect/references/error_budget.md b/engineering/skills/slo-architect/references/error_budget.md index 3b3d0a05..a5d9adcd 100644 --- a/engineering/skills/slo-architect/references/error_budget.md +++ b/engineering/skills/slo-architect/references/error_budget.md @@ -45,13 +45,17 @@ The alert fires only when BOTH windows show high burn. This filters spikes (only ### Recommended thresholds +For a 30-day SLO reporting window: + | Alert | Long window | Short window | Burn rate threshold | % budget at fire | Severity | |---|---|---|---|---|---| | Fast burn | 1h | 5m | 14.4 | 2% in 1h | page | | Slow burn | 6h | 30m | 6 | 5% in 6h | page | | Ticket | 3d | 6h | 1 | 10% in 3d | ticket | -The numbers come from: `burn_rate × bad_event_rate > slo_target_violation_rate`. +Burn rate is `observed_bad_fraction / allowed_bad_fraction`, where `allowed_bad_fraction = 1 - target_percent / 100`. To consume a chosen budget fraction over a long alert window, use `burn_rate_threshold = budget_fraction × SLO_window_hours / long_alert_window_hours`. An error-ratio alert compares `observed_bad_fraction > burn_rate_threshold × allowed_bad_fraction` in both its long and short windows. + +For 30 days (720 hours), the multipliers above are `0.02 × 720 / 1 = 14.4`, `0.05 × 720 / 6 = 6`, and `0.10 × 720 / 72 = 1`. For 28 days (672 hours), the same budget fractions and alert windows give 13.44, 5.6, and approximately 0.933. The target determines the allowed bad-event fraction; it does not change these multipliers. `error_budget_calculator.py` computes these for any target+window. Output is PromQL-shaped: