Our nightly cleanup job used to assume it ran exactly once and finished. Then a run overlapped with the next one, they both deleted the same temp files, and one crashed mid-way leaving a half-cleaned state. On the retry it did the wrong thing because it assumed the previous run had completed.
I now prompt specifically for idempotency and single-instance locking: a flock so two copies never overlap, and every operation written so running it 5 times equals running it once. Turns out "idempotent" is a word models nod at but don't implement unless you enumerate what it means.
How do you all handle the case where the lock holder is killed and leaves a stale lock? flock's fd-based locking mostly solves it for me but I'd love other approaches.
Write a maintenance script (`{LANG}`: bash or Python, your call, say why) meant to run from cron/systemd on a Linux host. It should {MAINTENANCE_TASK}.
It MUST be safe to run repeatedly and concurrently:
- Single-instance lock: use `flock` on a fixed fd/lockfile (bash) or `fcntl.flock` on an opened file (Python). If another instance holds the lock, exit 0 with a log line, do not queue or block forever. Stale locks must self-resolve (fd locks release on process death).
- Idempotency: every action is a no-op if already done. Deleting: ignore 'already gone'. Creating: 'create if not exists'. Moving: check destination first. Running N times must equal running once.
- Dry-run: `--dry-run` prints intended actions and touches nothing.
- Bounded work: cap how much it processes per run ({MAX_ITEMS}) so a backlog can't make one run take hours.
- Logging: structured single-line logs with timestamp + level to stderr, suitable for journald. Exit 0 success, non-zero only on real failure so cron alerts mean something.
- Time safety: never delete based on mtime younger than {MIN_AGE}; guard against clock skew.
End with a TEST PLAN: the exact commands to prove (a) two concurrent invocations don't collide, (b) running twice leaves the same end state, (c) --dry-run changes nothing.