Prevent duplicate cron job running(pankajtanwar.in)
pankajtanwar.in
Prevent duplicate cron job running
https://www.pankajtanwar.in/blog/prevent-duplicate-cron-job-running
47 comments
You can also just use systemd timers :)
Did not know. Thanks for sharing this. This looks mighty useful.
Is there a TUI to manage systemd-timers?
cron is a lot nicer imo. All your schedules are in one place and everyone knows how to access it.
> cron is a lot nicer imo.
How, exactly ?
I've been a cron user forever, I had never heard of systemd timers until this thread.
A quick internet search shows:
But having seen how easy it is to achieve the same thing with systemd timers whilst not requiring a whole lot of boilerplate code in each shell file ... why wouldn't you switch to systemd timers ? I sure will be.
How, exactly ?
I've been a cron user forever, I had never heard of systemd timers until this thread.
A quick internet search shows:
- Nice clean config in a file (instead of having to edit crontab)
- Lots of great features that you would have to manually implement and maintain in cron (e.g. missed launches, duplicate runs, random delay, timeouts)
But having seen how easy it is to achieve the same thing with systemd timers whilst not requiring a whole lot of boilerplate code in each shell file ... why wouldn't you switch to systemd timers ? I sure will be.
The general consensus: "because it's systemd", "systemd is unstable", "systemd does too much", "systemd is non-unix".
I like systemd. It's a good clone of launchd
I like systemd. It's a good clone of launchd
[deleted]
dcron is very lightweight and has most of those features. Dunno about random delay/timeouts, but worst case those sound like a 10 line shell script away.
https://github.com/dubiousjim/dcron
https://github.com/dubiousjim/dcron
> dcron is very lightweight and has most of those features.
Hmm, let's see ...
Hmm, let's see ...
- Choice 1: Use what comes with my system (two choices, systemd or, if for the anti-systemd people... cron)
- Choice 2: Install some random third-party that has not had any substancial git commits since 2014 (the 2019 commits are just admin metadata).There is a systemd command for that: systemctl list-timers
You can also just use systemd-cron :)
https://github.com/systemd-cron/systemd-cron
https://github.com/systemd-cron/systemd-cron
We used to run crons using a custom wrapper which handled locking, logging, state reporting and enable/disable via puppet. However, systemd timer units basically do all of this for you, so I'd recommend that instead.
Can you please share any resources to understand it more in detail? With flock, I am facing issue with lock files, if cron crashes or system reboots, lock files are a big pain.
IME, flock works fine across SIGKILLs and forced reboots. Are you trying to use flock on a filesystem that runs across multiple kernels (i.e., networked fs), by any chance?
If you use consul, you can use `consul lock` [0]. Allows you to have the cron configured on multiple servers too.
[0]: https://www.consul.io/commands/lock
[0]: https://www.consul.io/commands/lock
I don't get why people downvotes you. Clearly consul is not that widespread but if you happen to use it, consul lock is pretty cool, it also permits locks between a cluster of machines
OpenBSD's crontab provides a `-s command` flag which does similar thing in principle:
https://man.openbsd.org/crontab.5
"-s command
Only a single instance of command will be run concurrently. Additional instances of command will not be scheduled until the earlier one completes."
But I don't think it saves the state somewhere permanently, which isn't a big problem until cron's process dies for some reason.
https://man.openbsd.org/crontab.5
"-s command
Only a single instance of command will be run concurrently. Additional instances of command will not be scheduled until the earlier one completes."
But I don't think it saves the state somewhere permanently, which isn't a big problem until cron's process dies for some reason.
If you need to save state across executions, you should write a daemon instead.
Then "rm" should be a daemon too
cron applies this very method of attempting to handle state by merely being a daemon, and it's not enough because cron can crash.
The approach you're suggesting is what led to the problem you're proposing it as a solution for.
The approach you're suggesting is what led to the problem you're proposing it as a solution for.
A fairly common pattern would be trying to create a specifically named file with the O_CREAT and O_EXCL flags. That's atomic, so you don't need flock. If it fails, then exit. This pattern is also often extended where you write your pid to the file and subsequent runs do a sanity check that your pid is actually running, and unlink the file if not.
That said systemd-timers, as mentioned in many other comments, is probably the right answer here.
That said systemd-timers, as mentioned in many other comments, is probably the right answer here.
flock locks are released when the file is no longer open. This means that no matter how your command exits the OS will close open files and release the lock for you.
The problem with creating files is that you then have the stale lock file problem to deal with which adds a lot more complication.
The problem with creating files is that you then have the stale lock file problem to deal with which adds a lot more complication.
Is there any other way without using lock files? Using socket or something?
I suppose you could try listening on a socket at a specific port. The next process would get EADDRINUSE if you were still running.
There's also sysv or posix semaphores.
There's also sysv or posix semaphores.
Huh, does python not have a way to flock a file itself?
A simple flock doesn't always work all that well - the devil is in the details. And there is the issue that you miss a run of the script and have to wait another half hour for the next run.
I've used the Highlander, "There can be only one!" code pattern for forever, and it works really well:
https://www.perlmonks.org/?node_id=14260
You could change things to check on the status of your lock every minute or so for 15 minutes or whatever.
Also in module form,
https://metacpan.org/pod/App::Highlander
And much more updated:
https://medium.com/booking-com-development/highlander-b67172...
No need to attempt to stuff all this in a cronjob IMHO.
A simple flock doesn't always work all that well - the devil is in the details. And there is the issue that you miss a run of the script and have to wait another half hour for the next run.
I've used the Highlander, "There can be only one!" code pattern for forever, and it works really well:
https://www.perlmonks.org/?node_id=14260
You could change things to check on the status of your lock every minute or so for 15 minutes or whatever.
Also in module form,
https://metacpan.org/pod/App::Highlander
And much more updated:
https://medium.com/booking-com-development/highlander-b67172...
No need to attempt to stuff all this in a cronjob IMHO.
> Huh, does python not have a way to flock a file itself?
It does. (https://docs.python.org/3/library/fcntl.html#fcntl.flock)
It does. (https://docs.python.org/3/library/fcntl.html#fcntl.flock)
Python has SQLite. This means you can lock a db file on all platforms and synchronize processes using SQLite.
That seems a little inefficient to pull in SQLite just so that you can have it lock a file, when you can just lock your own file, don't you think?
glances around nervously is flocking one of those "what's old is new again" concept that people are just now relearning? Surely a blog post on how not to have two concurrent cronjobs running is kind of a weird thing to read when some Linux grey beard figured this out in the 90's on their lunch break.
glances around nervously is flocking one of those "what's old is new again" concept that people are just now relearning? Surely a blog post on how not to have two concurrent cronjobs running is kind of a weird thing to read when some Linux grey beard figured this out in the 90's on their lunch break.
[deleted]
SQLite comes for free in Python distribution. The only reason why I mentioned it. Of course, just having a nice cross-platform file locking interface would be nice (not the case for Python).
I meant in a complexity/memory way. Exclusive locks themselves aren't actually cross platform - it would vary on your fs. That's why you don't see a nice cross-platform file locking interface. I'm not sure if you can guarantee an exclusive lock in SQLite on something like a nfs?
This is from SQLite docs:
> SQLite uses POSIX advisory locks to implement locking on Unix. On Windows it uses the LockFile(), LockFileEx(), and UnlockFile() system calls. SQLite assumes that these system calls all work as advertised. If that is not the case, then database corruption can result. One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under Windows. Your best defense is to not use SQLite for files on a network filesystem.
I guess cross-platform for naive me is POSIX + Windows.
> SQLite uses POSIX advisory locks to implement locking on Unix. On Windows it uses the LockFile(), LockFileEx(), and UnlockFile() system calls. SQLite assumes that these system calls all work as advertised. If that is not the case, then database corruption can result. One should note that POSIX advisory locking is known to be buggy or even unimplemented on many NFS implementations (including recent versions of Mac OS X) and that there are reports of locking problems for network filesystems under Windows. Your best defense is to not use SQLite for files on a network filesystem.
I guess cross-platform for naive me is POSIX + Windows.
Welcome to Debian's dpkg::start-stop-daemon circa 2001
https://git.dpkg.org/cgit/dpkg/dpkg.git/commit/utils/start-s...
https://git.dpkg.org/cgit/dpkg/dpkg.git/commit/utils/start-s...
Or install moreutils from your distro's package repo so you can use lckdo and get some other super useful shell helpers too.
https://joeyh.name/code/moreutils/
https://joeyh.name/code/moreutils/
One would prefer not to lug a tool cabinet to a job site. Is there a standalone package for a tool like this?
Well, the flock command mentioned in the article will also work, but you might have to deal with having a bit more boilerplate around everything. The moreutils things are pretty small, some are just implemented as small perl scripts even.
If you're writing the program, it's fairly easy to do yourself in it with Flock or Fcntl (you can usually just get an exclusive lock in the absolute path to the running executable/script.
That said, quite a few of those moreutils items are useful in their own right (e.g. ts to prefix timestamps to each line of output is something so simple but so useful for logging output, chronic to silence stdout unless exit error is also very useful for cron, etc).
If you're writing the program, it's fairly easy to do yourself in it with Flock or Fcntl (you can usually just get an exclusive lock in the absolute path to the running executable/script.
That said, quite a few of those moreutils items are useful in their own right (e.g. ts to prefix timestamps to each line of output is something so simple but so useful for logging output, chronic to silence stdout unless exit error is also very useful for cron, etc).
But doesn’t flock also require installing a utils package? The docker people, recently being forced to choose a life of mass migration, would greatly appreciate tools which do not introduce additional overhead to be carried across the whaling sea to the kubed podlands.
$ rpm -qi moreutils | grep Size
Size : 194354
$ rpm -qi util-linux | grep Size
Size : 11565734
$ rpm -ql moreutils | grep -v build-id | xargs du -hc
4.0K /usr/bin/chronic
4.0K /usr/bin/combine
20K /usr/bin/errno
24K /usr/bin/ifdata
16K /usr/bin/ifne
20K /usr/bin/isutf8
16K /usr/bin/lckdo
16K /usr/bin/mispipe
16K /usr/bin/pee
20K /usr/bin/sponge
8.0K /usr/bin/ts
8.0K /usr/bin/vidir
4.0K /usr/bin/vipe
4.0K /usr/bin/zrun
24K /usr/share/doc/moreutils
4.0K /usr/share/man/man1/chronic.1.gz
...
4.0K /usr/share/man/man1/zrun.1.gz
260K total
But honestly, a package made that excludes the perl portions would be ideal for containers. From what I can tell, there's no other libraries used except glibc, and then you get most of those implemented in ~20k or less each.
$ rpm -ql moreutils | grep bin/ | xargs -n1 ldd | cut -d'(' -f1 | sort | uniq
/lib64/ld-linux-x86-64.so.2
libc.so.6 => /lib64/libc.so.6
linux-vdso.so.1
not a dynamic executable
I have made a simple "cron with extras" project https://github.com/umputun/cronn handling duplication issue as well as a bunch of other things, like restarting failed jobs, jitter, automatic reload and so on.
How to use systemd timers as a replacement for cron jobs:
https://opensource.com/article/20/7/systemd-timers
Systemd is much more powerful than cron and manages your exact use case. You will need to deal with process hangs though if you have self dependencies like this.
Thanks for sharing. I will definitely look into this.
I think systemd timers are vastly superior to cron mostly because of this problem.
Thank you. I will surely look into this.
Add and verify a last_started and last_completed time to your local database.
I have 100+ crons & I don't want to write some specific logic in my code for handling this, now what? would you mind sharing your thoughts on the same?
The other solutions based on file locking tools all seemed fragile.
You have to change _something_!