Commit graph

6 commits

Author SHA1 Message Date
Ryan Crabbe
4fa6742b01 Add Prometheus child_exit cleanup for gunicorn workers
When a gunicorn worker exits (e.g. from max_requests recycling), its
per-process prometheus .db files remain on disk. For gauges using
livesum/liveall mode, this means the dead worker's last-known values
persist as if the process were still alive. Wire gunicorn's child_exit
hook to call mark_process_dead() so live-tracking gauges accurately
reflect only running workers.
2026-02-27 16:11:15 -08:00
Ryan Crabbe
0ee8cb5f02 refactor: remove shutdown cleanup, rely solely on startup wipe
Phase 2 (per-worker mark_process_dead on shutdown) only ever fired
when all workers shut down together, making it redundant — Phase 1
wipes everything on next startup anyway. This aligns with the
prometheus_client docs: just wipe the directory between runs.
2026-02-26 12:17:31 -08:00
Ryan Crabbe
acfb5ade97 refactor: remove periodic dead PID cleanup, trim redundant tests
Startup wipe + graceful shutdown cleanup are sufficient. Remove
hourly mark_dead_pids scan, its helpers, and redundant test cases.
2026-02-26 10:03:51 -08:00
Ryan Crabbe
bfceefbb26 parameterize test 2026-02-25 17:30:59 -08:00
Ryan Crabbe
9157c1ebe2 test: redundant tests 2026-02-25 17:29:17 -08:00
Ryan Crabbe
9ea100a9f7 feat: add prometheus multiprocess directory cleanup
Adds cleanup utilities for PROMETHEUS_MULTIPROC_DIR to prevent
unbounded RAM/disk growth from stale .db files in multi-worker setups.

Three-part lifecycle aligned with upstream prometheus_client docs:
1. Startup: wipe entire directory before workers fork (clean slate)
2. Shutdown: mark_process_dead() for own PID (removes gauge_live* only)
3. Periodic (hourly): scan for dead PIDs and call mark_process_dead()

Counter/histogram files are never individually deleted at runtime to
avoid partial counter resets that cause false spikes in rate()/increase().

Also auto-creates PROMETHEUS_MULTIPROC_DIR when prometheus callback is
configured with multiple workers and the env var is not already set.
2026-02-25 17:20:46 -08:00