I've been running into this issue where my backup cron jobs "succeed" but create empty files.
For example:
Database backup cron runs, exit code 0, but creates a zero-byte dump file
Disk space runs out mid-backup, leaving an empty file but script still exits successfully
Network timeout during remote backup creates empty file, but no error is raised
File permission issues prevent writes, but the backup script doesn't check if anything was actually written
The logs say everything's fine, but when you actually need the backup, it's not there. Actually, the errors are probably in the logs somewhere, but who checks logs proactively? I'm not going through log files every day to see if something silently failed.
I've tried:
Adding file size checks in scripts - works, but you still need to check the logs or set up alerts
Monitoring tools that watch exit codes - but they don't catch "successful" failures
Manual verification - but that defeats the purpose of automation
I ended up building a simple monitoring tool that watches backup results instead of just execution - you send it the actual file size and it alerts if the backup is empty or suspiciously small. No need to dig through logs.
But I'm curious: how do you all handle this? Are you actually checking backup file sizes regularly, or do you have something that proactively alerts you when backups are empty or incomplete?
For example:
Database backup cron runs, exit code 0, but creates a zero-byte dump file Disk space runs out mid-backup, leaving an empty file but script still exits successfully Network timeout during remote backup creates empty file, but no error is raised File permission issues prevent writes, but the backup script doesn't check if anything was actually written
The logs say everything's fine, but when you actually need the backup, it's not there. Actually, the errors are probably in the logs somewhere, but who checks logs proactively? I'm not going through log files every day to see if something silently failed.
I've tried:
Adding file size checks in scripts - works, but you still need to check the logs or set up alerts Monitoring tools that watch exit codes - but they don't catch "successful" failures Manual verification - but that defeats the purpose of automation
I ended up building a simple monitoring tool that watches backup results instead of just execution - you send it the actual file size and it alerts if the backup is empty or suspiciously small. No need to dig through logs.
But I'm curious: how do you all handle this? Are you actually checking backup file sizes regularly, or do you have something that proactively alerts you when backups are empty or incomplete?