Every scheduler runs twice

Part 10 of an ongoing series on Moty, a fitness data platform I build and run solo in production: one multi-tenant API, two Next.js frontends. All numbers come from production measurements and git history.

The premise nobody re-checks

Production runs two instances of the API, not for scale but because of how it deploys. A rolling deployment keeps the old and new tasks alive together, so at least two copies of the application are always running.

The consequence is easy to state and easy to forget: every scheduled job in the codebase fires on every instance. A cleanup that runs "hourly" runs twice an hour, and a nightly batch runs twice a night. The scheduler annotation in your code says when; your deployment topology decides how many times.

The standard answer, and the one that got away

The standard answer is a distributed lock. Before a scheduled method runs, it takes a named lock in shared storage (ours live in Valkey); whichever instance wins runs the job, the other sees the lock and exits. The lock's timeout doubles as dead-instance handling: if the winner crashes mid-run, the lock expires and the next firing takes over instead of deadlocking forever.

We had this on every scheduler. Except one.

An hourly job that cleans up empty workout sessions had no lock, and had been double-firing on production for as long as it existed. When I found it during an audit, I asked myself whether this had been a considered choice ("it's idempotent, locks are unnecessary overhead"). The honest answer was no, it was an omission. The job happened to be idempotent, so double-running was wasteful rather than harmful, but happened to be is the operative phrase. Nobody had decided that. The safety was an accident that looked like a design.

Idempotency or locks? Wrong question

There's a real debate here, and the fashionable position is "make jobs idempotent and stop caring how many times they run." I half agree. Idempotency is the property you want anyway, because locks reduce duplicate scheduling, not duplicate effects from retries and crashes mid-job.

But relying on idempotency alone has a governance problem, because idempotency is invisible. Nothing in a job's signature says "safe to run concurrently with itself," and nothing breaks when a new job isn't. Every future scheduler has to re-earn the property, and no signal fires when one doesn't. A lock, by contrast, is visible, mechanical, and cheap. So the answer isn't either/or. Idempotency for correctness, and locks for cost, noise, and not betting on every future author's discipline.

Making the rule enforce itself

The fix for the one missing lock was one annotation. The interesting work was making sure this class of omission can't recur, because it belongs to a nasty category: rules that compile fine and fail silently. A missing lock doesn't throw or log anything. The job runs twice, forever, until someone audits.

Documentation never holds that line for long, so an architecture test holds it instead. We added one that scans the codebase and fails the build if any scheduled method lacks a lock annotation. And because a test that finds nothing might be a test that sees nothing, we verified it the only convincing way: inject a violation, watch it fail, remove the violation. A rule-checking test you've never seen fail is folklore with a green checkmark.

That one test joined three siblings in the same build: other conventions of the same silent-failure species that had lived only in documentation until then. That's the pattern I'd generalize: whenever a code review comment starts with "remember, we always…", ask whether a build step could remember instead.

What I'd pass on

• Count your instances before trusting your cron. "Hourly" in code really means "hourly per instance," whatever the comment above it intends.
• "It happens to be safe" is not a design. Idempotency nobody decided on is a coincidence, not a property. Write the lock, or write the decision down.
• Silent rules deserve loud tests. Anything that compiles when violated and fails without a log line is a candidate for an architecture test, and the test itself isn't real until you've watched it catch a planted violation.