CRON Job Failure Cheatsheet: Diagnose, Alert, Recover

For: A backend engineer or engineering lead at a 20–100-person B2B SaaS company whose scheduled jobs — billing runs, report generation, data syncs — are silently failing in production with no alert, no log, and no one noticing until a customer complains

If your cron job fails silently, the fix is not better error logging — it is inverting your monitoring to assert the presence of a success signal within a deadline, not the presence of a failure signal. A crashed job and a job that never ran look identical to your log aggregator. This cheatsheet covers the failure modes, the detection patterns that actually work, and the recovery playbook when a scheduled job goes missing in production.

The core insight

Most teams monitor cron by writing a log line at the end of a successful run and alerting on error strings. This is backwards. A job that dies before writing anything — OOM kill, container eviction, unhandled promise rejection, DNS timeout during boot — produces zero log output. Your alerting sees silence and treats silence as health.

The only reliable pattern: the job must ping an external heartbeat service on successful completion. That service alerts when the ping does not arrive within an expected window.

Failure modes: what actually breaks cron in production

The detection hierarchy

Layer these. Each catches what the layer above misses.

1. Heartbeat on success — external service alerts on missing ping. Catches crashes, evictions, cron daemon death.
2. Exit-code alerting — non-zero exit fires an alert. Catches handled errors.
3. Business-metric assertion — job asserts rows_processed > 0 or invoices_sent == expected before pinging success. Catches exit-0 no-op bugs.
4. Duration bounds — alert if job runs <10% or >300% of median. Catches stalls and empty runs.
5. Overlap detection — distributed lock (Redis SET NX EX, Postgres advisory lock). Catches runaway concurrent executions.

Heartbeat tools comparison

Minimal heartbeat pattern (any language)

start_ping(job_id)      # optional: signals job started
try:
    result = do_work()
    assert result.rows > 0    # business assertion
    success_ping(job_id)
except Exception as e:
    fail_ping(job_id, error=str(e))
    raise

The start_ping matters: without it, you can't distinguish "job started but crashed" from "job never triggered."

Kubernetes CronJob gotchas

• concurrencyPolicy: Forbid — prevents overlapping runs. Default Allow is dangerous for non-idempotent jobs.
• startingDeadlineSeconds — if unset and the controller lags, missed runs are skipped silently. Set it explicitly.
• successfulJobsHistoryLimit / failedJobsHistoryLimit — keep at least 3-5 for debugging.
• Node eviction — set resource requests/limits; a job pod without requests is first to be evicted.
• Time zone — pre-1.25, all schedules were UTC. On 1.25+, use spec.timeZone explicitly.

The recovery playbook

A customer just told you the Tuesday report never ran. Work through in order:

1. Confirm the miss. Check heartbeat history, job runner logs, and the downstream artifact (S3 file, DB row, sent email). Don't trust any single source.
2. Determine idempotency. Can you safely re-run? If the job sends emails or charges cards, you need a dedupe key before re-running.
3. Check for partial completion. Job may have processed 80% of records before dying. Query for the boundary (last updated_at, max processed ID).
4. Re-run scoped. Pass a date range or ID range to the job explicitly rather than re-running the default "since last run" logic.
5. Backfill missed windows. If several runs were missed, replay in chronological order — some jobs are order-dependent.
6. Write the postmortem trigger. Add an assertion or heartbeat that would have caught this specific failure. Every silent failure teaches you what your monitoring was blind to.

Alert routing that doesn't get ignored

Tradeoffs to acknowledge

• Heartbeats add a dependency. Your monitoring service going down triggers false alarms. Whitelist the heartbeat endpoint from firewall changes and pick a provider with a public status history.
• Business-metric assertions require domain knowledge. They're the highest-signal alerts but the hardest to write, and they drift as the product changes.
• Distributed locks add complexity. If the lock holder dies without releasing, you need a TTL — but a TTL too short causes duplicate runs anyway.

How CodeNicely can help

When we rebuilt the accounting engine for GimBooks, a YC-backed fintech SaaS, we inherited exactly this problem: recurring invoice generation and GST-filing jobs that failed silently, with customers finding out before the team did. The fix wasn't a monitoring tool purchase — it was restructuring every scheduled task around heartbeat assertions, idempotent re-runs keyed on invoice periods, and business-metric checks (row counts, expected filing volumes) before any job reported success.

If your team is running scheduled jobs at production scale but doesn't have the bandwidth to rebuild the observability layer, our digital transformation and engineering teams do this kind of work as targeted engagements — no vendor lock-in, full IP handover. Reach out if silent job failures are burning your on-call rotation.

Frequently Asked Questions

Why does my cron job not run but shows no error?

Most common causes: the cron daemon itself is down, the container was evicted before the schedule fired, startingDeadlineSeconds was exceeded in Kubernetes, or a timezone mismatch means it ran at a different time than you expect. Check the cron daemon status and the Kubernetes CronJob controller events first — the job may never have been triggered at all.

How do I get alerted when a cron job fails to run entirely?

Use a heartbeat-based monitoring service (Healthchecks.io, Cronitor, Sentry Cron Monitoring) where your job pings a URL on successful completion, and the service alerts when the ping doesn't arrive within a deadline. This is the only pattern that catches jobs that crash before writing any log or never trigger at all.

What's the difference between exit-code alerting and heartbeat monitoring?

Exit-code alerting only fires if the job runs, exits with non-zero, and something is watching stdout. Heartbeat monitoring fires when an expected success signal doesn't arrive — catching total crashes, OOM kills, container evictions, and cron daemon failures that exit-code alerting is blind to. You want both layered.

How do I safely re-run a cron job after a missed execution?

First verify idempotency — if the job sends emails or writes non-idempotent side effects, add a dedupe key before re-running. Check for partial completion by querying the last processed record. Then invoke the job with an explicit date or ID range rather than relying on "since last run" logic, which may skip or duplicate work.

Can CodeNicely audit our scheduled job infrastructure?

Yes — we do targeted reliability audits covering scheduled jobs, background workers, and event pipelines, then implement the fixes. Scope and timeline depend on your job count, tech stack, and existing observability. Contact CodeNicely for a personalized assessment.