GNU Make Jobserver Implementation(make.mad-scientist.net)

59 pointsby krakensdenvor 10 Jahren20 comments

http://make.mad-scientist.net/papers/jobserver-implementation/

20 comments

Toward the very end, the author gives a brief one paragraph explanation of why they didn't use multiple threads:

    Obviously one idea is to use threads. It’s common to have a separate thread
    for dealing with signals and combining threads with semaphores, etc. would
    allow GNU make to wait on two different types of asynchronous events
    simultaneously. This was rejected because so far GNU make was single-threaded,
    and requiring multi-threading was deemed too much of a price to pay.

I'm confused as to why adding multi-threading would be quite so onerous. As I understand it, one could restrict the use of threads to that specific feature and thus avoid large architectural changes.

Do I not understand how Make is designed, or is "the price to pay" in reference to performance or some other metric?

hyc_symasvor 10 Jahren

As I noted in my own mention http://highlandsun.com/hyc/#Make I originally wrote that patch back in 1991. There was no such thing as threads back then. Even in 1999 the POSIX threads spec was only up to Draft 6 or so, not yet finalized. Nobody sane would write production code for such a moving target (oh, except maybe Microsoft, using Draft4 to create the Windows NT threads API).

JdeBPvor 10 Jahren

That parenthetical history is wrong, at least according to Dave Cutler. He traced Windows NT's threading back to his work on VAXELN in 1983, years before POSIX was even thought of. He also was quite obviously deriving the NT API in part from OS/2, again not POSIX, given the whole "mutant"/"mutex" affair.

hyc_symasvor 10 Jahren

Windows NT RPC is DCE RPC - that is an indisputable fact. DCE used POSIX Draft 4, and Windows copied DCE pretty completely there. https://en.wikipedia.org/wiki/DCEThreads

JdeBPvor 10 Jahren

Well now we know where all of your errors are coming from. Regrettably, you've learned a lot of bunkum, that you now have to un-learn.

The source of that bunkum, or at least the source that you've pointed to, is a canonical example of how rubbish turns into gospel via Wikipedia. It started out, in August 2005, as an intentionally opinionated rant written by someone who didn't supply xyr name, who clearly didn't know the subject, and who pointed to entities that didn't exist at the time in question as xyr authorities.

* https://en.wikipedia.org/w/index.php?title=DCEThreads&oldid=...

Other people came along and tried to take the opinions out and leave the facts. But the underlying problem was that the facts weren't factual at all, and were as warped as the rest of the rant was; and no-one with any actual knowledge of the topic, or research ability, has touched the article significantly in eleven years.

* https://en.wikipedia.org/w/index.php?title=DCEThreads&type=r...

And in the meantime, people come along, believe Wikipedia uncritically, and regurgitate a load of old tripe.

Here are some of the things that you have to learn in place of the un-learned junk:

* There were such things as threads in 1991. There were such things as threads in 1988 when Dave Cutler started on "NT OS/2" at Microsoft. There were such things as threads in 1987 when OS/2 version 1.0 was announced. And indeed in 1983 in VAXELN according to Dave Cutler.

* Threads are not RPC, DCE RPC or otherwise.

* The design of Windows NT threads pre-dated the existence of 1003.4a draft 4 (dated 1990-08-10). By years.

For far better things than Wikipedia to read on this subject, see:

* Inside Windows NT. Particularly page 92 of the 1st edition.

* Cutler's own Windows NT Mutant Specification, written in 1989.

* Mark Russinovich's writings on the subject of Windows NT and its VMS heritage.

* Any good book on OS/2 version 1.0 programming that details the threads API.

* Michael Yam's 1996 Dr Dobbs article on porting his wrapper class library from DCE to Win32 threads.

* Any number of books on the subject of DCE, explaining the two quite different things that are DCE RPC and DCE threads.

0xcde4c3dbvor 10 Jahren

It might be a portability concern. GNU make runs on a lot of platforms.

Samathyvor 10 Jahren

Surely the solution there would be to make it a compile-time option. Enable threaded jobs, or process jobs.

Avshalomvor 10 Jahren

Implementing both sounds like a price to pay.

hyc_symasvor 10 Jahren

That would be stupid. Double the amount of work for no increment in functionality.

borisvor 10 Jahren

GNU make is a giant string-chopping hairball written in C over the past 30 years by numerous authors. Paul D. Smith (the current maintainer and the author of this article) has actually been doing a lot of codebase cleanup over the past years. But still any major change is a monumental task.

nullcvor 10 Jahren

It may be that the bulk of the codebase was not concurrency safe.

ameliusvor 10 Jahren

> a traditionally hairy problem in UNIX programming: how to wait for two different types of events (signals and file descriptors) at the same time.

I've always thought one of the weak points of POSIX was that the event sources were not unified. Imho, is a pity the relevant committee didn't work towards a clean solution in the mean time.

okiganvor 10 Jahren

Same here! Also annoying is inability to wake up when any (vs all) child threads are done.

Even more annoying is that Windows API allows to wait on any combination of file/socket descriptor and/or message[1].

[1] MsgWaitForMultipleObjects https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...

uepvor 10 Jahren

I know we're talking about POSIX here, but Linux does have the ability to wait on signals the same way it waits on file descriptors. It does this by also allowing signals to be delivered on file descriptors.

http://man7.org/linux/man-pages/man2/signalfd.2.html

hyc_symasvor 10 Jahren

Gratuitous fluff. Writing to a pipefd inside a signal handler works on all POSIX systems.

JdeBPvor 10 Jahren

Winding forward to the present: I recommend a look at kevent/kqueue.

some1elsevor 10 Jahren

We need to use Make for sensible (partial) rebuilds of static websites. I'm exploring ways to build a dependency graph out of Handlebars files. That might be enough to generate a Makefile and leave Gulp/Grunt out of the equation. Anyone doing anything similar?

saosebastiaovor 10 Jahren

Not for static websites, but I've definitely done very similar with Knitr for repeatable analyses in R (except for autogenerating the makefile). I create R scripts, markdown templates, database extract jobs, and even take care of sending the analysis results via email. I've left the job where I was responsible for those reports, but they're still in use with a distribution list that goes up to the SVP level at Amazon, so apparently they've been pretty successful.

jahitrvor 10 Jahren

GNU make is one of the most complete task manager I have seem.

But there are a few major problems with it:

Newies often complain a lot about the esoteric syntax.

It's written in C.

It doesn't gets updated as fast as it should.

These points make it not very suitable for our current time.

I belive we all just need to pick something like gulp and port the "good parts" into it.

There is no point in having 20 tools that do the same!

hyc_symasvor 10 Jahren

"written in C" is an asset, not a liability. It is still the one language that is portable across the widest spectrum of platforms, as well as the most efficient high level language for execution.