Cancelling every in-flight job the moment shutdown reached the scheduler
dropped the rows a write job had already popped: flush_gateway_requests
drains its accumulator before committing and does not restore it on
CancelledError, and update_spend requeues its batch only after the
shutdown drain had already run.
Shutdown now waits up to JOB_FINISH_TIMEOUT_SECONDS for in-flight jobs
to finish on their own, cancels the ones still running, and does both
before the shutdown flushes so a requeued batch is still written. The
cleanup run never finishes inside the grace, so it is still cancelled
and still records outcome="aborted".
Resolves LIT-6990
Review follow-ups on #41213:
- Pause the scheduler as the first shutdown step so a job whose fire time
falls inside the shutdown window does not start only to be cancelled.
Jobs already running keep the whole window and are cancelled and
awaited before the database disconnects, as before.
- Keep the cleanup run's progress in a task-scoped ContextVar rather than
on the cleaner instance, so two runs overlapping on one cleaner
(APSCHEDULER_MAX_INSTANCES above 1 without a Redis lock) each report
their own rows and batches on cancellation.
- Drop the module docstrings the repository comment policy does not
allow; the rationale lives in the PR description.
cleanup_old_spend_logs only caught Exception, so a run cut short by
CancelledError recorded no outcome and logged nothing. Under uvicorn the
job was never cancelled at all: uvicorn re-raises the captured SIGTERM as
soon as the lifespan shutdown returns, before asyncio cancels outstanding
tasks, so an in-flight scheduler job simply died with the process.
The cleanup now handles CancelledError by logging elapsed time, rows
deleted and batch count at error level, recording outcome="aborted", and
re-raising. The lifespan shutdown stops the scheduler and awaits the jobs
it cancels while the database is still connected, so that handler runs
under uvicorn too, and the pod lock is released instead of orphaned.
Resolves LIT-6990
