Cron job heartbeat monitoring, catch jobs that silently stop

Cron was designed to start programs on a schedule, not to tell anyone how they went. When a job fails, cron may send its output to a local mailbox that nobody reads, or to nowhere at all. When a job does not start, there is no output and no error to send.

Jobs stop running for ordinary reasons. A server is rebuilt and the crontab is not copied. A container image changes and the scheduler is no longer part of it. A deploy changes a path, and the command now fails on the first line. A disk fills up, a lock file from a crashed run is never removed, a credential expires, a timezone setting changes. None of these produce a visible outage. The only symptom is missing work, discovered when someone needs a restore, a report or a payment run.

• Backups that stopped after a migration
• Imports and syncs that fail on a changed file format
• Billing, invoicing and payout runs that never start
• Queue workers that crash and are not restarted
• Cleanup jobs whose absence fills a disk weeks later

An HTTP check asks a server a question and waits for an answer. A scheduled job has no address to ask. It runs for a few seconds or minutes and then disappears until the next run. To see it, the direction of the check has to reverse: the job reports in, and the monitor notices when the report does not arrive.

Each job gets a unique URL. When the job finishes successfully, it sends a request to that URL. The monitor knows the schedule of the job, so it knows when the next request should arrive. If the request does not arrive within the expected time plus a grace period, the monitor marks the job as down and sends an alert.

The pattern is simple enough to add to any job in one line, and it works the same for a crontab entry, a scheduled task in a framework, a serverless function on a timer or a long running worker that reports every few minutes.

0 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 https://ping.example/your-unique-id

The double ampersand matters. The ping is sent only if the job exits with code 0, so a job that runs and fails is reported the same way as a job that never ran. The timeout and retry options stop a slow network from hanging the job or losing the ping.

A single success ping tells you that the job finished. Two more signals make the picture complete.

• A start ping at the beginning of the run lets the monitor measure how long the job takes and alert you when it starts but never finishes.
• A failure ping, sent when the job catches an error, turns a late alert into an immediate one, and can carry the exit code or the last lines of output.
• A success ping at the end closes the run and records the duration.

curl -fsS -m 10 https://ping.example/your-unique-id/start
/usr/local/bin/import.sh
code=$?
if [ "$code" -eq 0 ]; then
  curl -fsS -m 10 https://ping.example/your-unique-id
else
  curl -fsS -m 10 https://ping.example/your-unique-id/fail
fi

The monitor can only know a ping is late if it knows when the ping was due. There are two common ways to describe that. A simple period, such as every 24 hours, works for jobs that run at a fixed rhythm. A cron expression, such as 0 2 * * 1-5, matches the real schedule, including jobs that skip weekends or run several times a day at uneven hours. Use the same expression as the job itself, so the monitor and the scheduler agree.

The grace period is the time the monitor waits after the expected ping before raising an alert. Set it from what you observe, not from a guess. A backup that usually takes 12 minutes and sometimes 25 should have a grace period of about 40 minutes. A job that takes seconds can have a grace period of a few minutes. Too short a grace period produces alerts on every slow night, and the team learns to ignore them. Too long a grace period delays the alert until the damage is done.

When a start ping is used, set a separate limit for the run time, so a job that hangs is reported even if it started on time.

Schedulers usually run in the server timezone, and many servers run in UTC while the team thinks in local time. Make sure the monitor uses the same timezone as the scheduler. Jobs scheduled between 1 and 3 in the morning in a timezone with daylight saving time can be skipped once a year or run twice. For critical jobs, schedule them outside that window or in UTC.

When a job takes longer than the interval between runs, two copies can run at the same time and compete for the same files or rows. Use a lock, such as flock, so a new run exits early while the previous one is still working. An early exit should not send a success ping, otherwise a stuck run hides behind a stream of skipped runs that each report success.

Shell scripts often return the exit code of the last command, not of the step that failed. Start scripts with strict settings, such as set -euo pipefail, so a failed step stops the script and returns a non zero code. Without that, a backup that failed to upload but printed a final message still ends with code 0 and sends a success ping.

A queue worker or a daemon is not scheduled, but it can still use a heartbeat. Let it ping every few minutes from its main loop, and set the monitor period accordingly. If the process dies or hangs, the pings stop and the alert follows.

Start with the jobs whose failure costs money or data, then add the rest.

1. 01 Database and file backups, including the upload to off site storage.
2. 02 Billing, invoicing and payout runs.
3. 03 Data imports, exports and partner feeds.
4. 04 Queue workers that process orders, emails or webhooks.
5. 05 Certificate renewal jobs, which are covered best together with ssl certificate monitoring.
6. 06 Cleanup and rotation jobs that protect disk space.

A missed backup belongs to whoever runs infrastructure. A failed invoice run belongs to whoever owns billing. Group heartbeats by owner or by system, and send alerts to the channel that person watches. Use escalation for the jobs that cannot wait until morning, and maintenance windows for planned downtime, so a migration does not produce a wave of missed pings.

• One heartbeat per job, named after what the job does, not after the script file
• The ping sent only after success, with a timeout and retries
• The schedule copied from the job, in the same timezone
• A grace period based on the longest normal run, not the average
• A start ping and a run time limit for jobs that can hang
• A lock against overlapping runs that does not report success on an early exit
• Strict shell settings so failures return a non zero exit code
• Alerts routed to the owner of the job, with escalation for critical ones

UptimePilot gives each job a unique ping URL, understands simple periods and cron expressions, supports start and failure pings, and alerts by email, SMS, Slack, Microsoft Teams or webhook. Heartbeats are included on every plan, from 10 on Starter to 1,000 on Enterprise. See cron job monitoring for the details and uptime monitoring pricing for the limits.