A stupid simple make wrapper that makes my life easier(github.com)
github.com
A stupid simple make wrapper that makes my life easier
https://github.com/wurosh/cake
43 comments
> It is NOT a repository for "dev commands" whatever the hell that means.
What it means is that it doesn't matter if I run `make test`, my coworker runs `make test` or the CI system runs `make test`, we all end up with the same result. Namely that the test suite will be run, after all stuff is done that is needed to setup the dev environment. Basically you codify all actions you as a developer would need, to setup, verify or use a project and all the dependencies between them. This makes your projects portable and when you have multiple projects (in different languages), getting it setup and running the test suite is the same for each one of them.
Why not use a bash script? That's how I started off but once a coworker introduced me to Make I never looked back. Bash scales really badly in terms of readability and maintainability.
What it means is that it doesn't matter if I run `make test`, my coworker runs `make test` or the CI system runs `make test`, we all end up with the same result. Namely that the test suite will be run, after all stuff is done that is needed to setup the dev environment. Basically you codify all actions you as a developer would need, to setup, verify or use a project and all the dependencies between them. This makes your projects portable and when you have multiple projects (in different languages), getting it setup and running the test suite is the same for each one of them.
Why not use a bash script? That's how I started off but once a coworker introduced me to Make I never looked back. Bash scales really badly in terms of readability and maintainability.
Right, I wasn't asking what "single source of truth is" I was asking what a "dev command" is. If you have commands and you want to store them so that multiple people can run them then the solution is NOT make, using make to execute commands is really quite like using machine gun to turn your TV on. Make is a very big and complex tool with lots of nuance and lots of different and not-cross-compatible versions. Moreover, it doesn't always use the same shell on every machine you run it on so using it to store bash scripts (for example) isn't really portable either.
I'm sorry to be the one to tell you this but neither you nor your coworker know what make is if all you think it is is a way to store subcommands.
If you want shell scripts write shell scripts. If you want them all to stem from one command (as subcommands), use a switch statement. These solutions will actually likely be far more reliable and portable (if you know what you're doing, I would imagine it's easier with portable shell than portable shell in make since even most of the programmers I know who use make for the purpose it was designed for don't seem to understand it very well).
I'm sorry to be the one to tell you this but neither you nor your coworker know what make is if all you think it is is a way to store subcommands.
If you want shell scripts write shell scripts. If you want them all to stem from one command (as subcommands), use a switch statement. These solutions will actually likely be far more reliable and portable (if you know what you're doing, I would imagine it's easier with portable shell than portable shell in make since even most of the programmers I know who use make for the purpose it was designed for don't seem to understand it very well).
Take a look at one of my Makefiles[0]. It uses most features Make has to offer (to much so). It doesn't matter what state you start from, fresh clones repo or outdated virtualenv. Running `make run` will do everything needed to run the application instance, `make test` will run the testsuite. Each command is it's own "functional" block of shell script with it's inputs (prerequisites) and outputs (target). All recipies will run in Bash on all system btw, see the first line of the Makefile.
Still, a Makefile with only targets, no prerequisites is less cruft than a Bash script with a switch statement. But eventually your (team) needs will grow and you start to introduce more logic and dependencies. This is where Make shines for me.
[0] https://gitlab.com/internet-cleanup-foundation/web-security-...
Still, a Makefile with only targets, no prerequisites is less cruft than a Bash script with a switch statement. But eventually your (team) needs will grow and you start to introduce more logic and dependencies. This is where Make shines for me.
[0] https://gitlab.com/internet-cleanup-foundation/web-security-...
"Dependencies" in your makefile are literally just functions. This can literally all be done with a bash script and 100% less backslashes to make things run in one shell. I really seriously think that this would be much cleared if written in bash.
That being said, while perusing the original makefile I found multiple issues. The one which stands out the most is that running make with a -j flag (which is entirely likely if your end user aliases that in their bashrc, I know multiple people that do). I am pretty sure fix and test can't run concurrently, at least I can't imagine how in-place fixing of files while running tests would work reliably. There's probably more cases where this would break things. Another issue is using `echo -e` and other bashisms, except that it's not guaranteed make will be running bash.
I really don't understand why your makefile keeps creating files, if it didn't create most of those files it would behave basically identically except for not needing to periodically run clean. In fact, by not providing a full DAG your makefile is just annoying in that there really isn't any option other than deleting random files or redoing all the steps. I don't see how having to delete .make.test and running make test is any better than having ./run-testsuite or ./run testsuite just run the testsuite every time.
Edit: I noticed your makefile sets SHELL. But this will in turn break this line:
https://gitlab.com/internet-cleanup-foundation/web-security-...
That being said, while perusing the original makefile I found multiple issues. The one which stands out the most is that running make with a -j flag (which is entirely likely if your end user aliases that in their bashrc, I know multiple people that do). I am pretty sure fix and test can't run concurrently, at least I can't imagine how in-place fixing of files while running tests would work reliably. There's probably more cases where this would break things. Another issue is using `echo -e` and other bashisms, except that it's not guaranteed make will be running bash.
I really don't understand why your makefile keeps creating files, if it didn't create most of those files it would behave basically identically except for not needing to periodically run clean. In fact, by not providing a full DAG your makefile is just annoying in that there really isn't any option other than deleting random files or redoing all the steps. I don't see how having to delete .make.test and running make test is any better than having ./run-testsuite or ./run testsuite just run the testsuite every time.
Edit: I noticed your makefile sets SHELL. But this will in turn break this line:
https://gitlab.com/internet-cleanup-foundation/web-security-...
I think you are missing a fundamental feature of what makes Make special, which is how it handles dependencies. It uses file timestamps to decide what to do and especially what things to skip because they don't need to be done. All those .make.* files serve a purpose, as Make will compare the timestamp of those files to the prerequisite files (source files) and only run the shell scripts if they are outdated (in the process recreating the target with an updated timestamp). So if I didn't change any of the source files, there is no need to lint or test again, since nothing changed, there can be no different result. Same goes for installing requirements (a lengthy task). If the requirements files don't change, nothing needs to be reinstalled. But if I pull in new changes from a colleague with an updated requirements files, running any Make target that has a dependency on those requirements will run the scripts to update them without me having to think about it.
I know how make works. I really only looked at .make.test because it has a pretty glaring bug in it and assumed the rest of the . targets you have have similar issues. I think actually it may be the only one with this issue. Here's the issue:
By depending on ${pysrc} which is set above with `pysrc = $(shell find ${pysrcdirs} -name *.py)` you ignore the possibility that a file may be removed. This would happen in a bisect for example or if someone were to manually delete a file. This will prevent tests from running.
I think the bigger issue with your makefile is that there's about 5 targets in total which actually make use of any of make's features, and a lot of targets which make incorrect assumptions about how make works.
Here's a list of targets which appear to not use any of make's features at all (and no, depending on ${app} being set up before they are ran does not constitute using make's features): audit, run_no_backend, run-frontend, run-nonlocal-frontend, app, run-worker, run-broker, testcase, test_integration, test_system, test_datasets, test_deterministic, test_mysql, test_postgres, clean, clean_virtualenv, mrproper, pip-sync, _mark_outdated, _commit_update, ${python} ${pip}.
Moreover, here's an example of a target with fundamental issues:
update_requirements: _mark_outdated requirements.txt requirements-dev.txt requirements-deploy.txt _commit_update
This is just plain wrong. You're treating make dependencies as some kind of sequential list of commands. That's not how make works. If I was running this with make -j or a gnu make implementation with by-default nondeterministic target build order choice this would just break.
I'm not saying your makefile saves zero purpose. There's a few targets there which you should keep. But it should really be 90% shorter with all the removed bits in a shell script so that you're not relying on nuances of your particular version of GNU make to make it work.
By depending on ${pysrc} which is set above with `pysrc = $(shell find ${pysrcdirs} -name *.py)` you ignore the possibility that a file may be removed. This would happen in a bisect for example or if someone were to manually delete a file. This will prevent tests from running.
I think the bigger issue with your makefile is that there's about 5 targets in total which actually make use of any of make's features, and a lot of targets which make incorrect assumptions about how make works.
Here's a list of targets which appear to not use any of make's features at all (and no, depending on ${app} being set up before they are ran does not constitute using make's features): audit, run_no_backend, run-frontend, run-nonlocal-frontend, app, run-worker, run-broker, testcase, test_integration, test_system, test_datasets, test_deterministic, test_mysql, test_postgres, clean, clean_virtualenv, mrproper, pip-sync, _mark_outdated, _commit_update, ${python} ${pip}.
Moreover, here's an example of a target with fundamental issues:
update_requirements: _mark_outdated requirements.txt requirements-dev.txt requirements-deploy.txt _commit_update
This is just plain wrong. You're treating make dependencies as some kind of sequential list of commands. That's not how make works. If I was running this with make -j or a gnu make implementation with by-default nondeterministic target build order choice this would just break.
I'm not saying your makefile saves zero purpose. There's a few targets there which you should keep. But it should really be 90% shorter with all the removed bits in a shell script so that you're not relying on nuances of your particular version of GNU make to make it work.
Your shell file with a switch isn't going to do incremental parallel builds.
If you understand make, your makefile will behave as you expect it to. If you try to read a makefile as a shell script you'll constantly be surprised.
If you understand make, your makefile will behave as you expect it to. If you try to read a makefile as a shell script you'll constantly be surprised.
The people this tool is aimed for don't use make to do incremental parallel builds. They use it to as a glorified shell wrapper which produces sub commands. That's what the use-case for this tool seems to be at least.
I know what make is and I know how to use it, thanks.
I know what make is and I know how to use it, thanks.
And you have concluded this how? Again, in what way is this arcane power of make that you describe subdued by this tool?
I've concluded this buy looking at some of the makefiles that people are referring to and the way the description of the tool is worded.
Calling makefiles a "single source of truth" for "dev commands" sounds to me like a glorified shell script aggregator. That's not what make is.
All in all I have two issues:
1. The naming of this tool. It's not make specific, you could literally make this tool run anything else and it would still work identically. But in this case it seems to be encouraging misuses of make.
2. The concept described by the tool's documentation that makefiles are "a single source of truth for dev commands".
Calling makefiles a "single source of truth" for "dev commands" sounds to me like a glorified shell script aggregator. That's not what make is.
All in all I have two issues:
1. The naming of this tool. It's not make specific, you could literally make this tool run anything else and it would still work identically. But in this case it seems to be encouraging misuses of make.
2. The concept described by the tool's documentation that makefiles are "a single source of truth for dev commands".
> why not just do a bash script with a switch in it?
Because with bash scripts you'd be re-inventing the wheel when it comes to dependencies, among other things.
Make certainly might not be the right tool for having a bunch of "dev commands" that aren't targets/recipes, but bash isn't it either. There is plenty of task runners that allow you to have a single source of truth for dev commands.
Because with bash scripts you'd be re-inventing the wheel when it comes to dependencies, among other things.
Make certainly might not be the right tool for having a bunch of "dev commands" that aren't targets/recipes, but bash isn't it either. There is plenty of task runners that allow you to have a single source of truth for dev commands.
Most "dependencies" in the makefiles I see which are like this are just other phony targets. In bash you can do this with functions, and if you're worried they'll get ran more than once you can write another 1 line function which only runs a function if it hasn't been ran yet during that execution of the script.
In fact, nothing is stopping people from using make for some things and leaving the bulk of everything else to a separate shell script.
In fact, nothing is stopping people from using make for some things and leaving the bulk of everything else to a separate shell script.
Fair enough, but this wrapper does not take any power away from make. How much of that power ends up being used is up to the developer. It can literally anything a regular instance of make can do - it just supplies the dependencies. In what way does that have nothing to do with make? If you think there is a better way of phrasing it, I'd be more than happy to adopt it.
Changed from "dev commands" to "the build process". Hopefully that makes the intent of the tool clearer.
> If you want a glorified shell script which runs commands
Give my Run tool a try:
Run: Easily manage and invoke small scripts and wrappers https://github.com/TekWizely/run
Give my Run tool a try:
Run: Easily manage and invoke small scripts and wrappers https://github.com/TekWizely/run
> why not just do a bash script with a switch in it?
Here is another alternative:
https://github.com/TekWizely/run
Here is another alternative:
https://github.com/TekWizely/run
Hey thanks for the bump !
I sometimes chime in on posts where OP makes a tool in the same domain as Run.
When I hit this article yesterday, the bash script convo hadn't started yet and I saw the OPs creation as not being in the same realm as Run, so didn't chime in.
But now that its been mentioned :)
Anyway I got a few extra subs from your call out, so thanks again for that!
(actually I just spent the last 30 mins reviewing 24 hours of HN posts to see where the call out might have come from, and finally found it :) -- Surprisingly google hasn't indexed this post yet so didn't show up in my saved search )
I sometimes chime in on posts where OP makes a tool in the same domain as Run.
When I hit this article yesterday, the bash script convo hadn't started yet and I saw the OPs creation as not being in the same realm as Run, so didn't chime in.
But now that its been mentioned :)
Anyway I got a few extra subs from your call out, so thanks again for that!
(actually I just spent the last 30 mins reviewing 24 hours of HN posts to see where the call out might have come from, and finally found it :) -- Surprisingly google hasn't indexed this post yet so didn't show up in my saved search )
A bit of a plug, but if you are using python in your project already, give a try to pydoit (pydoit.org/). It completely replaced make, task runners or build systems for me.
Pydoit is the sweet spot for most of my use cases:
- it scales up (graph of deps, file watcher, etc) but above all, it scales DOWN (simple stuff are dead simple)
- it promotes a declarative task definition
- it embraces the shell yet let you use python if you need to
- it gets out of your way and doesn't try to rule your project. It just runs what you want and disapear.
- the doc is nice
The biggest drawback is that you need to pip install it, which means non python devs will avoid it. I wish it was provided as an stand alone executable
Pydoit is the sweet spot for most of my use cases:
- it scales up (graph of deps, file watcher, etc) but above all, it scales DOWN (simple stuff are dead simple)
- it promotes a declarative task definition
- it embraces the shell yet let you use python if you need to
- it gets out of your way and doesn't try to rule your project. It just runs what you want and disapear.
- the doc is nice
The biggest drawback is that you need to pip install it, which means non python devs will avoid it. I wish it was provided as an stand alone executable
a meta-plug is in order for pipx, then: https://pipxproject.github.io/pipx/
which wholly abstracts "pip install this thing that has an entry_point without hating your life."
which wholly abstracts "pip install this thing that has an entry_point without hating your life."
Thanks! New URL: https://pypa.github.io/pipx/
Having to work with multi-language codebases cures one real fast of any interest in single-language build systems.
Still, make kinda sucks. I've yet to meet a build system that ticks these boxes: multi-language, simple, fast.
Still, make kinda sucks. I've yet to meet a build system that ticks these boxes: multi-language, simple, fast.
pydoit is not single language in the sense it limits your to tasks for your language. It's completely agnostic in that sense, and I use it for non python projects as well.
Even the fact it's using python to declare your tasks is not a problem: it's basically a function signature and a mapping, nothing you can't master in 5 minutes, and certainly simpler than make files DSL. Also easier to get right and debug.
The problem is the fact you need python to install it and run it, which non python dev will rightfully not care to do.
Even the fact it's using python to declare your tasks is not a problem: it's basically a function signature and a mapping, nothing you can't master in 5 minutes, and certainly simpler than make files DSL. Also easier to get right and debug.
The problem is the fact you need python to install it and run it, which non python dev will rightfully not care to do.
The README says it all. I kept writing Makefiles that ran build targets in Docker in a shoddy way. I just wrote a slightly less shoddy POSIX sh wrapper that does it for me. It's nothing fancy, but it makes my life easier and it's super simple to use.
Before releasing a script, it's always a good idea to fix the errors reported by Shellcheck. There are several here. Also the script has a /bin/sh shebang and README says it's POSIX but it has bashims (for example "echo -e" will fail in default /bin/sh in Debian).
there are more issues after fixing all of those: grep -w is not POSIX, the exit status of type is not clearly defined by POSIX, realpath is not POSIX, basename is POSIX but can be replaced with ${path##*/} for most well-formed paths (gives different results for /, but this script doesn't work properly for / anyways?), inodes are not unique across devices, --directory=dir has ugly handling but seems to work but --directory dir doesn't (i think it silently sets the directory to --directory and then passes dir as a separate argument?)
I fixed everything you mentioned. I kept basepath because it handles more edge cases. I improved the option parsing to handle --directory dir, although that is neither here nor there (it's GNU.. -ish). The handling is ugly for the sake of POSIX compatibility - if you can think of a more elegant way to write it let me know, I'd be glad to incorporate it. The exit status of type is clearly defined by POSIX (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t...). Either way, thank you for the exhaustive list of problems, it really helped - please don't hesitate to open issues with any further observations.
more specifically the problem with type is it's not clear whether "An error occurred" includes the case where the command is not found. one could reasonably argue that printing "command not found" is a successful run. additionally, it's only present in the XSI extension. for these reasons, command -v is usually preferred for better POSIX compatibility.
I usually have it on. Turns out I didn't install the binary on this PC which was silently ignored by my editor. Fixed! Thanks for pointing it out!
I see this post has finally devolved into a discussion around using make as a task runner (it hadn't yet when I first read the article yesterday)
With that in mind, I toss my tool into the ring:
Run: Easily manage and invoke small scripts and wrappers - https://github.com/TekWizely/run
With that in mind, I toss my tool into the ring:
Run: Easily manage and invoke small scripts and wrappers - https://github.com/TekWizely/run
Looks interesting! Not to put a dampener on the project but the name clashes with Cake (C# Make) project which could be confusing - https://cakebuild.net/ None the less this looks like a useful tool.
[deleted]
Mm. I've gone this way before, and eventually it breaks down. make just isn't that composeable. I have even made fancy Makefiles that allow me to pass command-line arguments to targets. At the end of the day, what you really want is a wrapper program that gives you specific composeable functionality, and Make isn't that.
I did want that functionality in the past, but it's not exactly what I'm trying to achieve (but if you find a tool that does, please post it here). I don't want to run arbitrary commands in a container. I want to build software in a container so I don't have to think about its dependencies (as much). I should probably clarify that in the README
I like the idea, but since I’m so used to using make for everything, I would probably just bake in a Makefile with the shell bits I found most useful for each container instead.
on macOs I had to skip the in_docker_group check.
And the escape sequences did not work
Interesting. I don't have access to a Mac, so a bit of extra information would be helpful. Could you open an issue on the repo? I know docker desktop uses a virtialized linux kernel on Mac, but I assumed access would still be controlled through user groups. Is there an alternative check I can do? I'd be more than happy to add MacOS specific logic. I'll look into how escape sequences work on Mac as well.
I ended up disabling the check - it's more trouble than it's worth.
This feels like when you set up a club for a hobby you enjoy, let's say it's beer pong, and you go on holiday for a month, you come back to find that your beer pong club is now full of people just chugging lots of beer and completely ignoring the game.
Please, make is a sophisticated tool with lots of good features. It is NOT a repository for "dev commands" whatever the hell that means. Make is a tool which takes targets, dependencies and recipes and turns them into a DAG which it then executes based on a set of criteria. If you want a glorified shell script which runs commands, why not just do a bash script with a switch in it? It's certainly going to behave a lot more like you expect. Why call it cake when it has almost nothing to do with make?