Commit graph

7 commits

Author SHA1 Message Date
yuneng-jiang
8dec9f46e1 fix: handle string callback values in prometheus multiproc setup
When callbacks are configured as a plain string (e.g., `callbacks: "my_callback"`)
instead of a list, the proxy crashes on startup with:
  TypeError: can only concatenate str (not "list") to str

Normalize each callback setting to a list before concatenating.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 10:07:36 -07:00
Ryan Crabbe
4fa6742b01 Add Prometheus child_exit cleanup for gunicorn workers
When a gunicorn worker exits (e.g. from max_requests recycling), its
per-process prometheus .db files remain on disk. For gauges using
livesum/liveall mode, this means the dead worker's last-known values
persist as if the process were still alive. Wire gunicorn's child_exit
hook to call mark_process_dead() so live-tracking gauges accurately
reflect only running workers.
2026-02-27 16:11:15 -08:00
Ryan Crabbe
0ee8cb5f02 refactor: remove shutdown cleanup, rely solely on startup wipe
Phase 2 (per-worker mark_process_dead on shutdown) only ever fired
when all workers shut down together, making it redundant — Phase 1
wipes everything on next startup anyway. This aligns with the
prometheus_client docs: just wipe the directory between runs.
2026-02-26 12:17:31 -08:00
Ryan Crabbe
acfb5ade97 refactor: remove periodic dead PID cleanup, trim redundant tests
Startup wipe + graceful shutdown cleanup are sufficient. Remove
hourly mark_dead_pids scan, its helpers, and redundant test cases.
2026-02-26 10:03:51 -08:00
Ryan Crabbe
bfceefbb26 parameterize test 2026-02-25 17:30:59 -08:00
Ryan Crabbe
9157c1ebe2 test: redundant tests 2026-02-25 17:29:17 -08:00
Ryan Crabbe
9ea100a9f7 feat: add prometheus multiprocess directory cleanup
Adds cleanup utilities for PROMETHEUS_MULTIPROC_DIR to prevent
unbounded RAM/disk growth from stale .db files in multi-worker setups.

Three-part lifecycle aligned with upstream prometheus_client docs:
1. Startup: wipe entire directory before workers fork (clean slate)
2. Shutdown: mark_process_dead() for own PID (removes gauge_live* only)
3. Periodic (hourly): scan for dead PIDs and call mark_process_dead()

Counter/histogram files are never individually deleted at runtime to
avoid partial counter resets that cause false spikes in rate()/increase().

Also auto-creates PROMETHEUS_MULTIPROC_DIR when prometheus callback is
configured with multiple workers and the env var is not already set.
2026-02-25 17:20:46 -08:00