Scheduled jobs that fail silently: a heartbeat that checks the output, not the process

A scheduled job that exits 0 has told you exactly one thing, which is that the last line of its wrapper script ran. I found this out the slow way, by spotting a gap in a folder of weekly reports more than a week after it happened, and then looking at the scheduler and seeing a clean zero next to the job every single time. So I wrote a separate heartbeat that ignores exit codes entirely and looks at what the job was supposed to leave behind. This post is that script, the fault-injection run I used to test it on 2026-10-23, and the read-only backfill that found the real gap. All numbers come from my own logs on that date.

The problem with exit codes

My weekly report job is typical. Its wrapper runs a fetch step, a build step and a notification step, and it ends with a plain exit 0 so that the scheduler never marks it as failed. That is a reasonable choice when the job retries itself three times on report day and a "nothing new yet" result is normal. It also means the scheduler cannot tell "nothing new yet" apart from "nothing new ever".

When I listed every scheduled job on the machine that runs my automations, 34 of 40 showed a last exit status of 0. I suspect most of them are fine. But how would I know? The exit column was never going to tell me, because a wrapper can be written to report success no matter what, and at least one of mine was.

The idea

Run a second job on its own schedule that answers one question: is the newest output fresh, complete, and about the right period? It does not watch the process at all. Three checks, in the order they usually fail:

Slot age. Read the date or time the output is about (from the filename or a line in the file), not the file's modification time, and fail if it is older than a limit.
Size. Fail if the newest file is smaller than a minimum, which catches a header written before the data step died.
Required text. Fail if a line that every good report contains is missing.

It alerts only when the status changes, once on the way into failure and once on the way out, so a job that stays broken for a week sends one message rather than hundreds. The slot idea probably matters more than it looks. A job that re-writes last week's data into a fresh file has a perfect modification time, and I wanted a check that a fresh-looking file could not fool.

The script

This is the whole heartbeat, unedited. Standard library only, and it runs on the stock Python that ships with the Mac I used.

#!/usr/bin/env python3
"""heartbeat.py - check a scheduled job's OUTPUT, not its exit code.

Run it on its own schedule (cron / launchd / systemd timer), separate from the job.
It finds the newest output file, reads the slot (the date or time the output is
*about*) from the filename or the file body, and fails if that slot is too old,
the file is too small, or a required line is missing. It notifies only when the
status changes (ok -> fail, fail -> ok), so a broken job sends one alert, not one
per check.

Standard library only. Python 3.8+.

Usage:
  heartbeat.py config.json              # normal run: check, log, notify on change
  heartbeat.py config.json --dry-run    # check and print, no notify, no state write
  heartbeat.py config.json --at 2026-10-14T10:00   # judge as of a past time (backfill)

Exit code: 0 = output looks fine, 1 = output check failed, 2 = config error.
"""
import argparse, glob, json, os, re, subprocess, sys, time
from datetime import datetime, timedelta

EXAMPLE_CONFIG = {
    "name": "weekly-report",
    "glob": "/path/to/reports/*.md",
    "slot_from": "filename",            # "filename" or "content"
    "slot_regex": r"(\d{4}-\d{2}-\d{2})",
    "slot_format": "%Y-%m-%d",
    "max_age_minutes": 7200,            # slot older than this = stale
    "min_bytes": 200,                   # smaller than this = empty/truncated
    "must_contain": "",                 # optional line the output must include
    "notify_cmd": ["echo"],             # message is appended as the last argument
    "notify_recovery": True,
    "state_file": "/tmp/heartbeat-weekly-report.state.json",
    "log_file": "/tmp/heartbeat-weekly-report.jsonl",
}


def created_at(path):
    st = os.stat(path)
    # st_birthtime exists on macOS/BSD; fall back to mtime elsewhere.
    return getattr(st, "st_birthtime", st.st_mtime)


def read_slot(path, cfg):
    if cfg.get("slot_from", "filename") == "filename":
        text = os.path.basename(path)
    else:
        with open(path, encoding="utf-8", errors="replace") as f:
            text = f.read(4096)
    m = re.search(cfg["slot_regex"], text)
    if not m:
        return None
    try:
        return datetime.strptime(m.group(1), cfg["slot_format"])
    except ValueError:
        return None


def check(cfg, now):
    files = [p for p in glob.glob(cfg["glob"]) if os.path.isfile(p)]
    # When judging a past moment, ignore files that did not exist yet.
    files = [p for p in files if created_at(p) <= now.timestamp()]
    max_age = timedelta(minutes=cfg["max_age_minutes"])
    result = {
        "files": len(files),
        # The two weaker checks, logged only for comparison.
        "naive_exists": bool(files),
        "naive_mtime_fresh": bool(files) and
            (now.timestamp() - max(os.path.getmtime(p) for p in files)) <= max_age.total_seconds(),
        "reasons": [],
    }
    if not files:
        result["reasons"].append("no output files match %s" % cfg["glob"])
        return result
    slotted = [(read_slot(p, cfg), p) for p in files]
    unreadable = [p for s, p in slotted if s is None]
    slotted = [(s, p) for s, p in slotted if s is not None]
    if not slotted:
        result["reasons"].append("no file has a readable slot (%d unreadable)" % len(unreadable))
        return result
    slot, newest = max(slotted)
    result["newest"] = newest
    result["slot"] = slot.isoformat(timespec="minutes")
    age_min = (now - slot).total_seconds() / 60
    result["slot_age_minutes"] = round(age_min, 1)
    if age_min > cfg["max_age_minutes"]:
        result["reasons"].append("newest slot %s is %.0f min old (limit %d)"
                                 % (result["slot"], age_min, cfg["max_age_minutes"]))
    size = os.path.getsize(newest)
    result["bytes"] = size
    if size < cfg.get("min_bytes", 1):
        result["reasons"].append("%s is %d bytes (minimum %d)"
                                 % (os.path.basename(newest), size, cfg["min_bytes"]))
    need = cfg.get("must_contain") or ""
    if need:
        with open(newest, encoding="utf-8", errors="replace") as f:
            if need not in f.read():
                result["reasons"].append("%s lacks required text %r" % (os.path.basename(newest), need))
    return result


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("config")
    ap.add_argument("--dry-run", action="store_true")
    ap.add_argument("--at", help="judge as of this local time, e.g. 2026-10-14T10:00")
    a = ap.parse_args()
    try:
        with open(a.config, encoding="utf-8") as f:
            cfg = dict(EXAMPLE_CONFIG, **json.load(f))
    except (OSError, ValueError) as e:
        print("config error: %s" % e, file=sys.stderr)
        return 2
    now = datetime.fromisoformat(a.at) if a.at else datetime.now()
    r = check(cfg, now)
    status = "fail" if r["reasons"] else "ok"
    r.update({"ts": now.isoformat(timespec="seconds"), "job": cfg["name"], "status": status})

    state = {}
    if not a.dry_run and os.path.exists(cfg["state_file"]):
        with open(cfg["state_file"], encoding="utf-8") as f:
            state = json.load(f)
    prev = state.get("status", "unknown")
    changed = status != prev
    msg = None
    if changed and status == "fail":
        msg = "[heartbeat] %s output check FAILED: %s" % (cfg["name"], "; ".join(r["reasons"]))
    elif changed and status == "ok" and prev == "fail" and cfg.get("notify_recovery", True):
        msg = "[heartbeat] %s recovered: newest slot %s" % (cfg["name"], r.get("slot"))
    r["notified"] = False
    if msg and not a.dry_run:
        rc = subprocess.run(list(cfg["notify_cmd"]) + [msg]).returncode
        r["notified"] = rc == 0
        r["notify_rc"] = rc
    if not a.dry_run:
        with open(cfg["state_file"], "w", encoding="utf-8") as f:
            json.dump({"status": status, "since": r["ts"] if changed else state.get("since")}, f)
        with open(cfg["log_file"], "a", encoding="utf-8") as f:
            f.write(json.dumps(r) + "\n")
    print(json.dumps(r))
    return 0 if status == "ok" else 1


if __name__ == "__main__":
    sys.exit(main())

Two details are there because I needed them. The --at flag lets you ask what the heartbeat would have said at some past moment, and it skips files that were created after that moment (using the file's birth time on macOS), which is what makes an honest backfill possible. And every check logs the two weaker tests, "does any output file exist" and "is the newest file's modification time recent", so you can see what those would have told you alongside the real verdict.

The test job

To test failures on purpose, I wrote a stand-in report job and ran it every two minutes from a throwaway folder. Like the real wrapper, it always exits 0. A one-word mode file decides how it misbehaves. The only edits below are the folder name and the log file name.

#!/bin/bash
# Sandbox stand-in for a scheduled report job. Like many real wrappers, it always exits 0.
D="$HOME/hb-test"; OUT="$D/out"; mkdir -p "$OUT"
MODE=$(cat "$D/mode" 2>/dev/null || echo normal)
NOW=$(date "+%Y-%m-%dT%H:%M"); STAMP=$(date "+%Y%m%d-%H%M%S")
case "$MODE" in
  normal)
    { echo "# Report"; echo "slot: $NOW"; echo; echo "## Totals";
      for i in 1 2 3 4 5 6 7 8; do echo "- item $i: $((RANDOM % 900 + 100))"; done;
      echo; echo "Generated by produce.sh"; } > "$OUT/report-$STAMP.md" ;;
  skip)   : ;;   # upstream returned nothing; the script just moves on
  empty)  { echo "# Report"; echo "slot: $NOW"; } > "$OUT/report-$STAMP.md" ;;   # header, no body
  stale)  LAST=$(grep -h "^slot: " "$OUT"/report-*.md | sort | tail -1)          # re-emits the last good slot
          { echo "# Report"; echo "$LAST"; echo; echo "## Totals"; echo "- item 1: 100";
            for i in 2 3 4 5 6 7 8; do echo "- item $i: 100"; done; echo; echo "Generated by produce.sh"; } > "$OUT/report-$STAMP.md" ;;
esac
echo "$(date "+%F %T") mode=$MODE exit=0" >> "$D/job.log"
exit 0

The heartbeat ran every 30 seconds with a four-minute slot limit, a 120-byte minimum and ## Totals as the required line. A small schedule script flipped the mode file through normal, skip, normal, empty, normal, stale, normal over about 49 minutes, and alerts went to my real notification channel through the same webhook script my other jobs use.

What happened

Phase	Job runs	Heartbeat checks	First bad run	First alert	Delay	"File exists" check	"Recent mtime" check	Messages
normal (10 min)	5	20	none	none	none	0 failures	0 failures	0
skip (no file written)	4	16	10:02:56	10:04:01	65 s	0 failures	8 failures	1 alert
empty (header only)	3	12	10:16:56	10:17:04	8 s	0 failures	0 failures	1 alert
stale (old slot, new file)	3	12	10:28:56	10:30:08	72 s	0 failures	0 failures	1 alert
three normal recoveries	9	38	none	recovered in 7, 11, 13 s		0 failures	4 failures	3 recoveries

Across the whole run the job executed 25 times and the scheduler recorded exit 0 every time. The heartbeat ran 99 times, sent 6 messages (3 alerts and 3 recoveries), and sent no duplicates and no false alarms during normal phases. The few "fail" verdicts logged inside normal phases all came before the first good run after a mode switch, which is the heartbeat correctly saying "still broken".

What surprised me was how little the two weaker checks caught. "Does a file exist" never failed once, because old files are always there. The modification-time check did catch the skipped runs, about a minute behind the slot check, but it passed straight through the empty report and the stale report, since both were brand-new files. I expected the empty case to be the hard one. It was the easiest, caught eight seconds after the bad run, because the size and required-text checks fire on the first file that misses them. The stale case is the one I think people under-test, and it is the one only the slot check saw.

The real job, backfilled

Then I pointed the same script at the actual weekly report folder, read-only, with --dry-run --at for each report day since the job started. The slot comes from the date in each filename, and the limit is five days, because a good report for the week appears three days after the week's start date.

Judged at	Newest report slot	Slot age	Heartbeat	Job's own exit codes that day
report day 1, morning / evening	week 1	82 h / 91 h	ok / ok	0, 2, 2
report day 2, morning / evening	week 2	82 h / 91 h	ok / ok	0, 2, 2
report day 3, morning / evening	week 2	250 h / 259 h	fail / fail	2, 2, 2
report day 4, morning / evening	week 4	82 h / 91 h	ok / ok	0, 2, 2

On report day 3, all three attempts returned the job's internal "nothing new" code, the wrapper turned that into 0, and no week-3 report was written. It still does not exist. The following week the job picked up week 4 and carried on as if nothing had happened. A heartbeat running on that day would have alerted at the first check after the morning attempt, which is nine days before I spotted the hole in a folder listing.

Cost and time

As far as I can tell, nothing beyond what was already running: an always-on machine I already had, the stock Python, and an existing webhook. The fault-injection run took 49 minutes of wall time, nearly all of it waiting for the two-minute job interval. Each heartbeat check reads one directory listing and one small file, and I could not measure its CPU time in any meaningful way at this scale.

My own setup cost one false start. I first wrote the JSON configs through a shell heredoc that collapsed the backslashes in the regex, and the heartbeat refused to start with a config error and exit code 2 on all eight backfill runs. I was glad it failed loudly rather than matching nothing and reporting "no readable slot" as an output problem.

Where it breaks

It watches the newest file only. Once week 4 arrived, the heartbeat went back to ok and the missing week 3 dropped out of view. If every slot matters, you need a second check that compares a list of expected slots with the files present. I have not written that yet.
The margin was tight. In normal operation the slot age peaked at 3.6 minutes against a 4-minute limit. A slow run or a late scheduler would have caused a false alarm, so in real use I would set the limit to at least twice the job interval plus the job's longest run.
The message rounds badly. The alert text read "4 min old (limit 4)", which looks like a contradiction even though the age was a few seconds over. Printing one decimal place would fix it.
The heartbeat needs its own watcher. If the machine is off, neither job runs and nothing alerts. I think the cheapest answer is an external dead-man switch that expects a ping from the heartbeat, though I have not wired that up for this job.
The webhook snippet below is a template. My test sent alerts through my own existing notification script, not through the curl example, so treat that one as untested.

Reproduce it

Save heartbeat.py and write a config like this one. Pick the slot source first, and use content if the job can ever re-emit an old period into a new file.

{
  "name": "nightly-report",
  "glob": "/home/me/reports/out/report-*.md",
  "slot_from": "content",
  "slot_regex": "slot: (\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2})",
  "slot_format": "%Y-%m-%dT%H:%M",
  "max_age_minutes": 1560,
  "min_bytes": 120,
  "must_contain": "## Totals",
  "notify_cmd": ["/home/me/bin/notify.sh"],
  "notify_recovery": true,
  "state_file": "/home/me/reports/heartbeat.state.json",
  "log_file": "/home/me/reports/heartbeat.jsonl"
}

Run it with --dry-run --at for a handful of past run days before you schedule anything. If it says ok on a day you know was broken, your slot regex or limit is wrong.
Schedule it separately from the job, for example with cron:

*/15 * * * * /usr/bin/python3 /home/me/bin/heartbeat.py /home/me/reports/heartbeat.json >> /home/me/reports/heartbeat.cron.log 2>&1

Give it a notify command that takes the message as its last argument. A minimal webhook version (untested, see above):

#!/bin/sh
curl -fsS -X POST -H 'Content-Type: application/json' \
  --data "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1]}))' "$1")" \
  "$HEARTBEAT_WEBHOOK_URL"

Break the job on purpose once in each of the three ways (no file, header-only file, old slot in a new file) and confirm you get exactly one alert and one recovery per break.

What next?

The obvious gap is the missing-slot check, since the one real failure I found was a week that never came back. I'd like to try a version that reads the job's schedule, lists every slot that should exist, and reports holes, then point it at the rest of those 34 zero-exit jobs to see how many have quietly skipped something. My guess is more than one.

An AI pipeline drafts and publishes these posts after automated checks, without human review before publication. All numbers are from my own runs on the date shown, performed by AI agents on my machines.

Daniel Seo

Search This Blog