Managing dotfiles with Make(matheusmoreira.com)
matheusmoreira.com
Managing dotfiles with Make
https://www.matheusmoreira.com/articles/managing-dotfiles-with-make
48 comments
Is there a reason to not include your makefile in the git repo? Is it not useful to anyone else?
Speaking for myself, reasons why I don’t immediately share my own tooling:
Perhaps I want to hold it to my own standard for tooling, not the team’s.
Perhaps I want to write it in a language that the team doesn’t really embrace.
I might not think it’s worth anyone’s time to code review changes I make to my own tools.
I might want to do something dumb like bake an API key into the tool.
Maybe the project already has a similar tool, and I don’t want to explain why I am wrapping/ignoring it.
In short: sometimes the cost to collaborate on idiosyncratic tools is just higher than I’m willing to pay.
Perhaps I want to hold it to my own standard for tooling, not the team’s.
Perhaps I want to write it in a language that the team doesn’t really embrace.
I might not think it’s worth anyone’s time to code review changes I make to my own tools.
I might want to do something dumb like bake an API key into the tool.
Maybe the project already has a similar tool, and I don’t want to explain why I am wrapping/ignoring it.
In short: sometimes the cost to collaborate on idiosyncratic tools is just higher than I’m willing to pay.
Sadly, I do the same at times.
I just want to hack together something that works for my use case. If I can share that with the wider team, that is superb. But I really don't enjoy my hacky utils being subject to tedious code review scrutiny they don't deserve.
To be clear, I _do_ add utils that I think will be useful to the common repo, subject to review, and mostly they will be merged. But if it gets painful, I just retract the PR and keep it locally.
I just want to hack together something that works for my use case. If I can share that with the wider team, that is superb. But I really don't enjoy my hacky utils being subject to tedious code review scrutiny they don't deserve.
To be clear, I _do_ add utils that I think will be useful to the common repo, subject to review, and mostly they will be merged. But if it gets painful, I just retract the PR and keep it locally.
Have you ever tried to submit a PR of your Makefile to the maintainers of, say, an open source npm package? If you have, what compelling story were you able to tell about that Makefile that got the maintainers to merge your PR? And after they successfully merged it, did you continue putting up PR after PR for each successive round of tweaks to that Makefile?
What an extraordinary amount of unnecessary effort that would be. My workflow does not belong to the repo, it belongs to me. The only thing that belongs in the repo is all the shared workflows (or the elements they comprise.)
What an extraordinary amount of unnecessary effort that would be. My workflow does not belong to the repo, it belongs to me. The only thing that belongs in the repo is all the shared workflows (or the elements they comprise.)
Articles like this one are more likely to drive people away from make(1) than teach them to appreciate it.
This Makefile is anything but simple. It uses advanced features that are meant for complicated situations, just to create a few symlinks. The same symlinks, every time. And if you introduce a new dotfile to the repo, you have to update the Makefile too.
It also makes no use of the main feature of make(1) - to track files for changes, and update them.
For demonstration, here is the same functionality, in sh(1):
This Makefile is anything but simple. It uses advanced features that are meant for complicated situations, just to create a few symlinks. The same symlinks, every time. And if you introduce a new dotfile to the repo, you have to update the Makefile too.
It also makes no use of the main feature of make(1) - to track files for changes, and update them.
For demonstration, here is the same functionality, in sh(1):
files=$(find files -type f | sed s=^files/==)
echo "$files" | sed s=[^/]*\$== | sort -u | { cd; xargs mkdir -p; }
echo "$files" | xargs -I{} -n1 ln -sf $PWD/files/{} ~/{}
What's funny is people use make as just an imperative task runner. For some reason people prefer to write a makefile with 10 commands rather than write 10 short shell scripts.
The real point of make is it's a declarative build system. You just tell it to make and it will do what needs to be done by figuring out dependencies and rebuilds.
The real point of make is it's a declarative build system. You just tell it to make and it will do what needs to be done by figuring out dependencies and rebuilds.
Make is also a highly discoverable entry point to a project.
A Makefile with tasks that just run external scripts is much easier to find than the scripts directly. Add some help texts and it’s a great DevEx multiplier.
A Makefile with tasks that just run external scripts is much easier to find than the scripts directly. Add some help texts and it’s a great DevEx multiplier.
As opposed to bash scripts documented in a README file, that will be understood by more people and not have to deal with the quirks of make?
Yes.
With a Makefile I can generally just run `make` or `make <tab>` to have a feel for what's available. It's right there on my terminal, which is usually the first thing I interact with when I open a repository. If enough of the engineers use it, it stays up-to-date as it gets fixed and updated along with any other changes.
Documentation OTOH has a tendency to be forgotten and left for dead. IME this is especially true for internal documentation, and the closest to the code the docs are, the less attention they receive - since higher level documentation is more likely to be consumed by people outside of the team.
With a README, I need to:
* remember to read it * trust that it will have the information * trust that the information is up-to-date * finally, figure out paths to scripts, arguments and/or copy-paste commands
With a Makefile I can generally just run `make` or `make <tab>` to have a feel for what's available. It's right there on my terminal, which is usually the first thing I interact with when I open a repository. If enough of the engineers use it, it stays up-to-date as it gets fixed and updated along with any other changes.
Documentation OTOH has a tendency to be forgotten and left for dead. IME this is especially true for internal documentation, and the closest to the code the docs are, the less attention they receive - since higher level documentation is more likely to be consumed by people outside of the team.
With a README, I need to:
* remember to read it * trust that it will have the information * trust that the information is up-to-date * finally, figure out paths to scripts, arguments and/or copy-paste commands
INSTALL is also a standard location, for the commands that are related to building and installing the software.
I think people like using makefile as a simple task runner because it's pretty much ubiquitous and also a kind of auto-descriptive standard. Interactive shells usually do autocompletion on makefile targets so it's easy to see what you can run on a project (more so on old or foreign projects)
> This Makefile is anything but simple.
This makefile does look simple to me, although it is not easy; but TFA never claimed it was supposed to be "easy".
The shell you mentioned is not in any way "simpler" nor even "easier". The same thing can be done with bash substitutions and loops (all supported by zsh), and someone would argue that it'd be simpler by virtue of not suffering e.g gnu vs bsd vs posix sed nor needing xargs; also, pipefail and such.
Also it doesn't do "the same thing": you have to be at a specific pwd and not in any subdir + you can't do a subset (e.g "make git")
Note I'm not saying it's better or worse.
I really like the article as it showcased a few advanced make features in a concrete and limited use case (symlinks). The result is simple to use and maintain.
This makefile does look simple to me, although it is not easy; but TFA never claimed it was supposed to be "easy".
The shell you mentioned is not in any way "simpler" nor even "easier". The same thing can be done with bash substitutions and loops (all supported by zsh), and someone would argue that it'd be simpler by virtue of not suffering e.g gnu vs bsd vs posix sed nor needing xargs; also, pipefail and such.
Also it doesn't do "the same thing": you have to be at a specific pwd and not in any subdir + you can't do a subset (e.g "make git")
Note I'm not saying it's better or worse.
I really like the article as it showcased a few advanced make features in a concrete and limited use case (symlinks). The result is simple to use and maintain.
> but TFA never claimed it was supposed to be "easy".
TFA> Another reason to use it is this turned out to be a surprisingly easy task.
TFA> Another reason to use it is this turned out to be a surprisingly easy task.
I stand corrected.
I suppose I'm biased. I'm used metaprogramming it by now.
I'm very prone to being entranced by the eerily lisp-like nature of GNU Make. Got side tracked in one of my projects trying to recreate GNU autoconf inside it. That was the day I finally understood what autoconf was even doing.
I'm very prone to being entranced by the eerily lisp-like nature of GNU Make. Got side tracked in one of my projects trying to recreate GNU autoconf inside it. That was the day I finally understood what autoconf was even doing.
> And if you introduce a new dotfile to the repo, you have to update the Makefile too.
Not at all. The rules are generic. I can give make any path inside my home directory and it will try to match it against an equivalent file in the repository.
It looks complicated because I used functions to compute all these paths from the items in the repository. Here's what the final result looks like:
The bulk of the makefile is dealing with environment variables such as XDG_*_HOME and GNUPGHOME variables which affect where the links are supposed to end up.
Not at all. The rules are generic. I can give make any path inside my home directory and it will try to match it against an equivalent file in the repository.
$ make /home/matheus/.vimrc
ln -snf /home/matheus/.files/~/.vimrc /home/matheus/.vimrc
It looks complicated because I used functions to compute all these paths from the items in the repository. Here's what the final result looks like:
$ phony-targets GNUmakefile
git
/home/matheus/.config/git/config
mpv
/home/matheus/.config/mpv/mpv.conf
vim
/home/matheus/.vimrc
# ...
The bulk of the makefile is dealing with environment variables such as XDG_*_HOME and GNUPGHOME variables which affect where the links are supposed to end up.
Adding to the sh implementation, `find -L "$HOME" -maxdepth 1 -type l -exec rm {} +` will clean up dead symlinks in your home directory, e.g., if a dotfile is deleted because it's not needed anymore.
What worked best for me is to turn my user's home dir into a Git repository. My .gitignore contains this:
*
And if I need to add something, I just "git add -f ...". Works surprisingly well. Combines well with git-secret for things like SSH keys, certificates and API keys.
*
And if I need to add something, I just "git add -f ...". Works surprisingly well. Combines well with git-secret for things like SSH keys, certificates and API keys.
I wouldn't trust that... If you by mistake run `git clean -xfd` in your home, well, you know what happens
Same thing as running `rm -rf *` by mistake. Don't do that and keep backups.
Ha, I'll probably never run that command but perhaps some git commands can be disabled if they're run in ~/
my dotfiles follow atlassian's tutorial where they do this, but put the git dir somewhere else and alias away the git command. It has worked pretty well for me.
https://www.atlassian.com/git/tutorials/dotfiles
https://www.atlassian.com/git/tutorials/dotfiles
Only downside to this are tools that default to only acting on stuff under version control. Whenever I use rg inside my home directory I'm caught out by this.
I was caught by that too. But a recent read has given me a tip. Add a ripgrep specific ignore files that undo the home gitignore.
I found Andrew Burgess's method approachable and ended up cribbing from his dotfiles concept [1] and making my own repository [2]. His method uses bash instead of make to create the symlinks defined in the links.prop file in each subdir of the repo.
I typically checkout the dotfiles repo to a personal project folder on every new OS build and then run the installer to get all my configs in place.
- [1] https://shaky.sh/simple-dotfiles/ - [2] https://gitlab.com/jgonyea/dotfiles
I typically checkout the dotfiles repo to a personal project folder on every new OS build and then run the installer to get all my configs in place.
- [1] https://shaky.sh/simple-dotfiles/ - [2] https://gitlab.com/jgonyea/dotfiles
I use a combination of stow and make to manage my dotfiles. I added a makefile well after using stow for a decade. The makefile is more for new system setup than day to day management. I might try out replacing stow with make based on this blog, more for fun than anything. I'm a bit reluctant to replace what has been working so well for a decade, but I'm very intrigued by this. Make has always interested me. It seems like it could be incredibly powerful in the right hands.
Nothing beats nix + home manager. But it is fun to see how many different ways people go about solving this particular problem. All of them have their tradeoffs and annoyances.
I advocate (and use) nix + home manager too but when you need a one-off change or test and realize you need to do the whole commit and switch thing, or when you are debugging and spelunk through the read only nix store, or when you set up a new (non-nix) computer and do the nix install, home-manager install, run the switch and get a deluge of errors...
it's simultaneously awesome but "can I really recommend this to <colleague>?"
it's simultaneously awesome but "can I really recommend this to <colleague>?"
With nix + home manager, you can use `mkOutOfStoreSymlink` to make symlinks between the dotfile repo and the target destination in `.config`. I've found this to be the most ergonomic way to have nix-managed dotfiles. Because the out-of-store dotfile repo is symlinked, you can make little changes to your system without doing the whole commit and switch dance.
For example, here's a snippet pulled from my dotfiles that does this for multiple dotfiles at once:
For example, here's a snippet pulled from my dotfiles that does this for multiple dotfiles at once:
home.file =
builtins.mapAttrs
(key: value: {
# symlink ~/dotfiles/configs/{value} to ~/{key}
source = config.lib.file.mkOutOfStoreSymlink "${config.home.homeDirectory}/dotfiles/configs/${value}";
})
{
".zshrc" = "zsh/zshrc";
".p10k.zsh" = "zsh/p10k.zsh";
".config/sway/config" = "sway/config";
".config/nvim/init.lua" = "nvim/init.lua";
};Oh right, I do this too!
At the same time it often feels like a veneer of control, like you can control exactly where to place the door, but what's in the messy room (like emacs profiles if you do that) might be hidden behind the very nice and solid door.
It's like in python projects I lock python3 and uv, and beyond that it's wild west. Still beats everything else, still feels a bit incomplete, and still feels somewhat unresolvable.
At the same time it often feels like a veneer of control, like you can control exactly where to place the door, but what's in the messy room (like emacs profiles if you do that) might be hidden behind the very nice and solid door.
It's like in python projects I lock python3 and uv, and beyond that it's wild west. Still beats everything else, still feels a bit incomplete, and still feels somewhat unresolvable.
Caveat: This works for literal dotfiles in your repo but it doesn’t help you if you use nix declared config, e.g. programs.git
The configs I tweak are git, bash and emacs, and each has their own way to load extra config from a local file. You can use this for stateful config local to a machine and out of the nix store.
It depends on buy-in from the tool so it’s not a panacea but it works for my use cases at least. I also don’t like to switch config for every change and it turns out I rarely have to.
It depends on buy-in from the tool so it’s not a panacea but it works for my use cases at least. I also don’t like to switch config for every change and it turns out I rarely have to.
This whole thread is a case study in “nix is worse ROI for every of the individual things it can do, but it’s the same I for all of them”.
Nix is definitely The Way to solve this problem, but I wouldn’t recommend it if the only thing you’ll ever use it for is home-manager. Once you’ve paid the investment, though… I never want to rsync a dotfile ever again in my life.
Nix is definitely The Way to solve this problem, but I wouldn’t recommend it if the only thing you’ll ever use it for is home-manager. Once you’ve paid the investment, though… I never want to rsync a dotfile ever again in my life.
Advantages of using nix are so numerous, it would be a sin to even try to list all of them. Suffice it to say that using HM and Nix to manage dotfiles is the ultimate peak of what one can do. I just wanted to write +1 but thought that would be too short.
Until and unless you're on a system which doesn't or can't have nix (RAM size preventing evaluation, $job policy, ...)
I really appreciate and use nixpkgs+nix-darwin as well as NixOS and flakes and per-project nix-shell/nix develop through and through; even then, Home Manager is a bridge too far for me.
One of the features of my dotfiles setup is that it has zero dependencies (beyond a POSIX shell).
I really appreciate and use nixpkgs+nix-darwin as well as NixOS and flakes and per-project nix-shell/nix develop through and through; even then, Home Manager is a bridge too far for me.
One of the features of my dotfiles setup is that it has zero dependencies (beyond a POSIX shell).
I've been using Chezmoi for a little while now. It simplifies what I want to do better than anything else I have tried.
Chezmoi is probably the best thing I've found. But I do have aome complaints about it. I don't love the filenames, and using symlinks often feels like a second class citizen.
It also has the downside that you have to install and set up chezmoi first, and it isn't included in the debian or ubuntu repos.
It also has the downside that you have to install and set up chezmoi first, and it isn't included in the debian or ubuntu repos.
This is very clever, but I'd have to relearn Make's subtleties each time I tried to debug it or add a feature (count me in the "prefer to avoid it" crowd).
I ended up writing a Go CLI to symlink my dotfiles. It's probably 10x more lines of code, but it uses a "plan, then execute" pattern, similar to Terraform that I find easier to test and clearer to use too. And it's a single binary so it's easy to install on Windows too.
I ended up writing a Go CLI to symlink my dotfiles. It's probably 10x more lines of code, but it uses a "plan, then execute" pattern, similar to Terraform that I find easier to test and clearer to use too. And it's a single binary so it's easy to install on Windows too.
Nice article, I love Make and your blog looks sharp.
I would suggest to provide the beginner friendly version in which all the rules are hard coded, so that a beginner can profit and don't be freightened.
And then, introduce the problem of needing to add rules by hand for each new dotfile, and then provide your solution using Make's builtins.
I also suggest putting the complete final version of the Makefile at then end, so I can copy it.
Regards,
I also suggest putting the complete final version of the Makefile at then end, so I can copy it.
Regards,
Been using yadm for a long time and i love it. super simple and has the "bootstrap" concept as an escape hatch for when i want to get weeeird
I use Make with GNU Stow for dotfiles targeting different operating systems, with a Nix Flake for shell tools: https://github.com/ADGEfficiency/dotfiles/blob/main/Makefile
It works fine - most of my time is still spent in configuring Neovim ^^
It works fine - most of my time is still spent in configuring Neovim ^^
It’s always fun to see what other people do for dotfile mgmt.
I have a simple bash script that does something similar: https://erock-git-dotfiles.pgs.sh/tree/main/item/dotfiles.sh...
I have a simple bash script that does something similar: https://erock-git-dotfiles.pgs.sh/tree/main/item/dotfiles.sh...
I do like Make so quite an interesting article, but the author seems overly impressed with their idea of using a literal ~ for the name of the subdirectory. That adds an unnecessary WTF factor to the code in my opinion :)
I don't understand what "managing" this accomplishes?
Thanks for sharing!
It is good to see that other ppl uses make for dotfiles.
Here is mine: https://github.com/balazs4/dotfiles/blob/canary/makefile
I am pretty happy about it nowadays; it works for me.
Cheers
Here is mine: https://github.com/balazs4/dotfiles/blob/canary/makefile
I am pretty happy about it nowadays; it works for me.
Cheers
I also tend to put a Makefile into the root of any repo I work in (ignored via .git/info/exclude), so that shell commands relevant to my workflow can accumulate under that Makefile. I've been doing this for at least 10 years and love it. Most repos have some sort of a cli interface, but that doesn't mean that they're any good. Make is nice way to wrap all of them, and to standardize on at least the build/test/run commands across many disparate repos.
Here's an example of one of those from a long-abandoned project: https://gist.github.com/staticshock/0e88a3232038d14a2817e105...