I've been running into this issue where my backup cron jobs "succeed" but create empty files.
For example:
Database backup cron runs, exit code 0, but creates a zero-byte dump file
Disk space runs out mid-backup, leaving an empty file but script still exits successfully
Network timeout during remote backup creates empty file, but no error is raised
File permission issues prevent writes, but the backup script doesn't check if anything was actually written
The logs say everything's fine, but when you actually need the backup, it's not there. Actually, the errors are probably in the logs somewhere, but who checks logs proactively? I'm not going through log files every day to see if something silently failed.
I've tried:
Adding file size checks in scripts - works, but you still need to check the logs or set up alerts
Monitoring tools that watch exit codes - but they don't catch "successful" failures
Manual verification - but that defeats the purpose of automation
I ended up building a simple monitoring tool that watches backup results instead of just execution - you send it the actual file size and it alerts if the backup is empty or suspiciously small. No need to dig through logs.
But I'm curious: how do you all handle this? Are you actually checking backup file sizes regularly, or do you have something that proactively alerts you when backups are empty or incomplete?
Ever had a critical cron job silently fail for days? I've seen backups stop running, cleanup scripts fail, and monitoring jobs die—all without any notification.
This guide covers practical ways to detect cron failures before they become disasters. What's your worst cron failure story? How do you monitor your scheduled tasks?
Those are all solid practices! I use chronic too, and proper exit codes are essential.
The gap I found is that even with all of that, you can still have "successful" jobs (exit code 0, no errors) that produce wrong results. Like a backup script that runs successfully but creates empty files because the source directory was empty, or a sync that only processes 10% of records because of a logic bug.
But there's another issue: chronic doesn't detect when cron jobs don't run at all. If your crontab gets corrupted, the server time changes, or cron daemon stops, chronic won't alert you because there's no output to email.
That's why I built result validation monitoring - it expects a ping after your job completes. If the ping doesn't arrive (job didn't run, crashed before completion, etc.), it alerts. Plus it validates the actual results (file sizes, record counts, content validation) and alerts if they don't match expectations.
Works alongside chronic/exit codes, but adds detection for jobs that never executed and validation of the actual outputs.
That's a clever approach! I used to do something similar, but found I'd miss emails or they'd get buried.
The tool I built does something similar but automated - it watches for a ping after your job completes and can validate the results (file sizes, timestamps, counts, etc.). You set validation rules once, and it only alerts when something's actually wrong. No need to read through daily emails.
For your backup example, you'd ping it with the file size, and it alerts if the size is unexpectedly small or zero. Same idea, just automated.
For example:
Database backup cron runs, exit code 0, but creates a zero-byte dump file Disk space runs out mid-backup, leaving an empty file but script still exits successfully Network timeout during remote backup creates empty file, but no error is raised File permission issues prevent writes, but the backup script doesn't check if anything was actually written
The logs say everything's fine, but when you actually need the backup, it's not there. Actually, the errors are probably in the logs somewhere, but who checks logs proactively? I'm not going through log files every day to see if something silently failed.
I've tried:
Adding file size checks in scripts - works, but you still need to check the logs or set up alerts Monitoring tools that watch exit codes - but they don't catch "successful" failures Manual verification - but that defeats the purpose of automation
I ended up building a simple monitoring tool that watches backup results instead of just execution - you send it the actual file size and it alerts if the backup is empty or suspiciously small. No need to dig through logs.
But I'm curious: how do you all handle this? Are you actually checking backup file sizes regularly, or do you have something that proactively alerts you when backups are empty or incomplete?