WhatsApp alert when a cron job fails

One curl after the job, or a systemd OnFailure= unit that covers every job on the box. And an honest section on the thing this does not do.

The job that matters is rarely the one that fails loudly. It is the backup that started exiting 1 in March, into a mail spool nobody reads, because cron's idea of telling you is an email to a local user.

Two ways to do it properly. Both are a curl; the second one you write once and then forget.

1. Make a key

In the Jabee dashboard, go to Management then Approvals and create an API key. It starts jak_. Copy it now; it is not shown again.

Whoever is going to be messaged needs a verified phone, which they get by tapping a link in the dashboard and sending one message. Until somebody has done that there is nobody to deliver to.

2. One script, used by both

Put the key in a file rather than in a crontab or a unit file, so it is not in crontab -l output, in systemctl cat, or in a backup of either:

printf 'jak_your_key_here' | sudo tee /etc/jabee.key >/dev/null
sudo chmod 600 /etc/jabee.key

Then one small script, which is what both wirings below call:

#!/bin/sh
# /usr/local/bin/jabee-alert — usage: jabee-alert "title" "body"
set -eu

jq -n --arg t "$1" --arg b "${2:-}" '{title: $t, body: $b}' \
| curl -sS -X POST https://api.jabee.ai/api/v1/alerts \
    -H "Authorization: Bearer $(cat /etc/jabee.key)" \
    -H "Content-Type: application/json" \
    --data @-

chmod 755 /usr/local/bin/jabee-alert and it is done. Using jq to build the JSON rather than printf is what stops an apostrophe in an error message from breaking the request — and error messages are exactly where apostrophes live. If the host has no jq, keep the titles plain and build the body with printf '{"title": "%s"}' "$1", knowing that is what you have given up.

3. Wiring it to a cron job

The || is the whole mechanism: the script runs only when the job exits non-zero.

0 3 * * * /usr/local/bin/backup.sh || /usr/local/bin/jabee-alert "Nightly backup failed on $(hostname)" "backup.sh exited non-zero"

One line, deliberately: a crontab entry cannot be continued with a trailing backslash the way a shell script can — the command is the rest of the line and nothing else. Which is the first reason the curl is in a script.

Two more things specific to crontabs. A literal % must be written \% or cron cuts the command off there, so avoid date +%F-style arguments in the entry itself. And PATH in cron is short: if the script is not somewhere boring like /usr/local/bin, give the full path.

4. Or wire it once for the whole machine

If the jobs are systemd timers — and on a modern box they probably should be — you can do this once for everything rather than per job. A wrapper that adds the last few log lines:

#!/bin/sh
# /usr/local/bin/jabee-alert-unit — usage: jabee-alert-unit backup.service
set -eu

/usr/local/bin/jabee-alert \
  "$1 failed on $(hostname)" \
  "$(systemctl status "$1" --no-pager --lines=5 | tail -c 1500)"

The tail -c 1500 is not decoration: the body limit is 2000 characters and a stack trace is longer than that.

# /etc/systemd/system/jabee-alert@.service
[Unit]
Description=WhatsApp alert for %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/jabee-alert-unit %i

Then one line in any unit you want covered:

[Unit]
OnFailure=jabee-alert@%n.service

%n is the failing unit's own full name, so backup.service instantiates jabee-alert@backup.service.service — which looks odd, is correct, and means one template covers every service on the host.

Keeping the work in a script rather than in ExecStart also avoids the trap in that file: systemd expands $NAME and ${NAME} itself and does no command substitution, so a $(cat ...) written directly into ExecStart does not do what a shell would do with it.

After editing, systemctl daemon-reload, then prove it with a unit whose ExecStart is /bin/false. An alerting path you have never seen fire is a guess.

The other half: noticing that nothing ran

This endpoint is push-only. If cron never fires, if the machine is off, if the crontab was lost in a rebuild, then nothing is posted here and this endpoint has nothing to notice — which is exactly the failure that bit you last time.

That is what heartbeat checks are for, and they are part of the same plan. The job pings a URL when it finishes, Jabee expects that ping on the schedule the job declared, and silence past the deadline is what raises the alert. One extra line at the end of the job covers the case this page cannot:

17 2 * * * /usr/local/bin/backup.sh && curl -fsS -X POST -H "Authorization: Bearer $JABEE_KEY" https://api.jabee.ai/api/v1/heartbeat/nightly-backup

Use both and all three failures are covered: the job failed (this page), the job never ran, and the box is gone (that one).

Only telling you about the failures

Tempting to post on success too, "so I know it ran". Do not: a nightly message that always says fine is a message you stop reading within a fortnight, and it is 30 of your 500 a month. Keep WhatsApp for the exceptions and let the heartbeat check live somewhere you look on purpose.

Where it goes

One phone per business, and it is your own: the account owner's verified number by default, or another handset — an ops phone, a shared on-call one — which you point alerts at in the dashboard, under Where alerts go. Either way that phone verifies itself by sending one message to the Jabee number, carrying a code from the dashboard.

So there is nothing to address and no recipient field. To reach several people, connect your own WhatsApp number and send from there: alerts on Jabee's number go out on a sender every business here shares, and the volume you send spends a reputation everybody else depends on.

What it costs

$29 a month for 500 alerts and approvals together, then 3 cents each. The first three are free, so you can wire this up and watch a message arrive before paying for anything. No per-seat charge.

And the ceiling that matters for a machine: 200 new alerts per business in any 24 hours, 60 calls a minute per key. A job in a crash loop can reach that; it answers 429. Giving the alert an external_id makes repeats free, so a retry loop on the same failure costs one.

Questions