The practical (Unix) problems with .cache and its friends(utcc.utoronto.ca)
utcc.utoronto.ca
The practical (Unix) problems with .cache and its friends
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/DotCacheDirectoriesProblem
43 comments
It's because of articles like this that computing is an inconsistent mess. When you make a technical proposal in public like this, it's your responsibility to ask yourself "What would it look like if everyone did this?". If the answer is "a personal kind of hell", don't make the proposal.
Also pretty easy to implement disk quotas on a modern system.
Heck, you could even display a warning or the top directories consuming space upon login.
Not a great take - this is a operations problem, not a design problem.
Heck, you could even display a warning or the top directories consuming space upon login.
Not a great take - this is a operations problem, not a design problem.
What software should be doing is using XDG_CACHE_HOME, if it is set (along with the other XDG environment variables). That way cache directories will be where the user expects.
The XDG specification (https://specifications.freedesktop.org/basedir-spec/latest/) is
Unfortunately, we have nothing but social pressure and incensed HN comments to get developers to follow it. That's why the mobile world adopted a "you shall" model: a developer won't spent five minutes to save a million years of user time unless there's some social or technical guardrail forcing his hand.
1. short and easy to read,
2. good for users, and
3. can be implemented in five minutes.
Unfortunately, we have nothing but social pressure and incensed HN comments to get developers to follow it. That's why the mobile world adopted a "you shall" model: a developer won't spent five minutes to save a million years of user time unless there's some social or technical guardrail forcing his hand.
Agreed. What op wants in this case, is for XDG_CACHE_HOME to default to ~/cache when unset.
The idea is quite reasonable: suddenly all tools that show disk usage will show this directory clearly and you can easily find what’s eating up you space with the same usual tools as always.
The idea is quite reasonable: suddenly all tools that show disk usage will show this directory clearly and you can easily find what’s eating up you space with the same usual tools as always.
Are dot folders actually a problem? I have show hidden files turned on globally everywhere always. I don't really understand how people operate otherwise.
So it sounds like having them hidden is a problem for you, given that you've disabled this behaviour globally.
Some folk just don't want to configure every single application to unhide them (or have no idea that they exist in the first place).
Some folk just don't want to configure every single application to unhide them (or have no idea that they exist in the first place).
I think they'll find they're in the minority on that opinion, so they should just set XDG_CACHE_HOME, and maybe symlink ~/.cache to ~/cache for those programs that have just hardcoded ~/.cache
> Dear everyone writing Unix programs that cache things in dot-directories (.cache, .local, etc): please don't. Create a non-dot directory for it.
If it doesn't live inside ~/.cache or ~/.program/cache then I won't know if I can delete it or not.
If you are not putting your cache in one of those folders, I'd expect the application itself to have a clear-cache option or for it to do it automatically from time to time.
If it doesn't live inside ~/.cache or ~/.program/cache then I won't know if I can delete it or not.
If you are not putting your cache in one of those folders, I'd expect the application itself to have a clear-cache option or for it to do it automatically from time to time.
To me, on OpenBSD that kind of breaks the purpose of unveil.
Firefox and chrome on OpenBSD is patched to use pledge and unveil. That restricts where these programs can read/write data. I would not want a program to have access to all cache data for every program.
To me, if you see dir cache, you can clear it. In fact, I have a cron job to clear cache data older than a week for Firefox. No issues doing this for many years.
Firefox and chrome on OpenBSD is patched to use pledge and unveil. That restricts where these programs can read/write data. I would not want a program to have access to all cache data for every program.
To me, if you see dir cache, you can clear it. In fact, I have a cron job to clear cache data older than a week for Firefox. No issues doing this for many years.
> I would not want a program to have access to all cache data for every program.
Then XDG_CACHE_DIR doesn't change anything?
The path is going to be $XDG_CACHE_DIR/$PROGRAM which is what would get allowed (and neighbour subdirs disallowed); no program would get to see the others' cache.
I just checked and I have a ~/.cache/firefox right there.
Having it at a single root point means it's trivial to make all cache ephemeral in any way one chooses.
Then XDG_CACHE_DIR doesn't change anything?
The path is going to be $XDG_CACHE_DIR/$PROGRAM which is what would get allowed (and neighbour subdirs disallowed); no program would get to see the others' cache.
I just checked and I have a ~/.cache/firefox right there.
Having it at a single root point means it's trivial to make all cache ephemeral in any way one chooses.
I am not sure if XDG_CACHE_DIR is set or not. But for some reason I think it is unveiled in the firefox config file in /etc. My OpenBSD system is powered down right now so I cannot check.
How do you know it's a cache dir and not configuration or something else? I don't say programs should have access to the cache directories of other programs just to their own inside ~/.cache.
If I see the name cache I hope it is cache :)
So far that has not failed me yet, maybe some day I will get burned.
So far that has not failed me yet, maybe some day I will get burned.
I thing there was a confussion. I was arguing against putting it somewhere that is not ~/.cache/program or ~/.program/cache, for example in ~/program, which would not have the string `cache` in the name.
Can ex. Firefox not unveil($HOME/.cache/firefox) ? Or actually I'd prefer unveil($HOME/.cache/firefox/$PROFILE)
Not sure what is being asked.
But there is a config file in /etc on OpenBSD that is used to determine what directories are unveiled. I do not remember the name.
So you can customize that as you see fit, veiling or unveiling items.
But there is a config file in /etc on OpenBSD that is used to determine what directories are unveiled. I do not remember the name.
So you can customize that as you see fit, veiling or unveiling items.
That isn't my impression of how it works? https://man.openbsd.org/unveil describes a system call that a program uses at runtime, so it should be easy for Firefox to start up and then use unveil() to limit itself to paths it knows it will need (but it can pick those paths dynamically when it starts)
> I'd expect the application itself to have a clear-cache option or for it to do it automatically from time to time.
Android has this. Though, the reason they included a "cache partition" back in the day was to make sure there was always enough space to download flash system.img over-the-air (as wiping clean the cache partition would provide enough room).
Android has this. Though, the reason they included a "cache partition" back in the day was to make sure there was always enough space to download flash system.img over-the-air (as wiping clean the cache partition would provide enough room).
I usually agree with the things that Chris writes but this isn't one of them.
For starters, ~/.cache was invented because everyone was doing their own thing and the previous situation was FAR worse.
I don't agree with eliminating the concept of dot directories in unix just because some people haven't learned about them yet.
And of course he acknowledges that their environment is unique.
What I would do: run a script to symlink '.cache' to 'cache' in entry users home dir. Then it's not hidden anymore.
For starters, ~/.cache was invented because everyone was doing their own thing and the previous situation was FAR worse.
I don't agree with eliminating the concept of dot directories in unix just because some people haven't learned about them yet.
And of course he acknowledges that their environment is unique.
What I would do: run a script to symlink '.cache' to 'cache' in entry users home dir. Then it's not hidden anymore.
I don't think that putting cache in .cache is bad, as long as everyone does it. But some programs (like VS Code and Chromium) decided that .config is a good place for their cache and local files, which I think is a bigger problem.
XDG_CACHE_HOME, as mentioned by numerous other posters, is a good remedy to deal with this kind of cached-artifacts-sprawl - but it does need a will to cooperate and care from the original software developer in the first place, which ain't a given.
So does another standard that I've come to appreciate in all software that supports it: CACHEDIR.TAG (https://github.com/LineageOS/charter/blob/main/device-suppor...) - I'm just here to spread the word! ;)
So does another standard that I've come to appreciate in all software that supports it: CACHEDIR.TAG (https://github.com/LineageOS/charter/blob/main/device-suppor...) - I'm just here to spread the word! ;)
Dammit, my clipboard manager messed up the link I was about to post, and now it's too late to edit :/
For posterity: https://bford.info/cachedir/ is what I was meant to link to above. Sorry for the inconvenience.
For posterity: https://bford.info/cachedir/ is what I was meant to link to above. Sorry for the inconvenience.
I went on GitHub to see if anyone was using this cachedir tag signature. Found many results so that’s cool :)
But ironically, if I’m reading correctly the command line ncurses based disk usage tool ncdu seems like it will intentionally ignore any directory that has this cachedir tag.
https://github.com/rofl0r/ncdu/blob/a216bc2d35b6edf4dadf5535...
https://github.com/rofl0r/ncdu/blob/a216bc2d35b6edf4dadf5535...
Which is the very sort of thing the original post was complaining about that happens with invisible .cache directories too :^)
But ironically, if I’m reading correctly the command line ncurses based disk usage tool ncdu seems like it will intentionally ignore any directory that has this cachedir tag.
https://github.com/rofl0r/ncdu/blob/a216bc2d35b6edf4dadf5535...
https://github.com/rofl0r/ncdu/blob/a216bc2d35b6edf4dadf5535...
Which is the very sort of thing the original post was complaining about that happens with invisible .cache directories too :^)
I get kinda pissed at any program that creates a visible directory.
Also dot files and directories in my home. Why do programs decide to treat my home like a junkyard?
At one point I markedy home directory as read only. Subdirs were read-write.
The number of programs that would silently or cryptically fail was too much. Had to cave in and allow them free reign.
The number of programs that would silently or cryptically fail was too much. Had to cave in and allow them free reign.
If your users are not fluent enough in Linux/Unix tools and conventions to be able to figure out how large a single standardized dotfile in their home directory is, I really don’t think the problem is “application developers are following a well-known standard.”
Wow I disagree so much.
Use XDG dirs: https://wiki.archlinux.org/title/XDG_Base_Directory#User_dir...
Use XDG dirs: https://wiki.archlinux.org/title/XDG_Base_Directory#User_dir...
"Dot" files are a terrible idea to begin with.
It began as a dirty hack in "ls", in order to skip displaying "." and "..", it looked only at the first character of a file name. Exact string comparison is costly, and early UNIX could not afford that price.
Then people noticed that other ".whatevers" will also be "hidden", unless "-a" is specified.
Then it snowballed into hell.
Both Plan 9 and OS X tried to fix it, with "~/lib" and "~/Library" respectively, but especially as the latter was and is a certified UNIX, there's no avoiding a hundred random dot files and directories. I try to delete stuff I know I won't need again but...
Advice for all programs: follow the XDG spec[1]. Advice for OS vendors: default them to something more reasonable; ~/Library sounds appealing since we already use ~/Desktop, ~/Downloads, and so on.
[1]: https://specifications.freedesktop.org/basedir-spec/latest/
It began as a dirty hack in "ls", in order to skip displaying "." and "..", it looked only at the first character of a file name. Exact string comparison is costly, and early UNIX could not afford that price.
Then people noticed that other ".whatevers" will also be "hidden", unless "-a" is specified.
Then it snowballed into hell.
Both Plan 9 and OS X tried to fix it, with "~/lib" and "~/Library" respectively, but especially as the latter was and is a certified UNIX, there's no avoiding a hundred random dot files and directories. I try to delete stuff I know I won't need again but...
$ uname
Darwin
$ ls -a | grep '^\.' | wc -l
61
Advice for all programs: follow the XDG spec[1]. Advice for OS vendors: default them to something more reasonable; ~/Library sounds appealing since we already use ~/Desktop, ~/Downloads, and so on.
[1]: https://specifications.freedesktop.org/basedir-spec/latest/
Hidden files with . are just not a good design.
the `du -s *` really shows that.
I wish they showed by default and you added a term to hide them.
Apple does not help here as they used to use ~/Library/Caches for all but now with sandboxes each Container has its own Cache and the sirectory they are in sometimes has a random name not simply the app name.
the `du -s *` really shows that.
I wish they showed by default and you added a term to hide them.
Apple does not help here as they used to use ~/Library/Caches for all but now with sandboxes each Container has its own Cache and the sirectory they are in sometimes has a random name not simply the app name.
Fun fact, hidden dot files were not designed, but were the result of an early UNIX bug that users and developers came to depend on…
http://xahlee.info/UnixResource_dir/writ/unix_origin_of_dot_...
http://xahlee.info/UnixResource_dir/writ/unix_origin_of_dot_...
> As noted by Haelwenn /элвэн/, a plain 'du' will find such dotfiles. The problem is that plain 'du' is more or less useless for most people; to really take advantage of it, you have to know the right trick (not just the -h argument but feeding it to sort to find things). How I think most people use 'du' to find space hogs is they start in their home directory with 'du -s ' (or maybe 'du -hs ') and then they look at whatever big things show up. This will completely miss things in dot-directories in normal usage.
I don't understand. `du` indeed finds the dotfiles (and dot-folders); and if their contents are large, then they'll be noticed by someone who "looks at whatever big things show up". It's not at all clear to me why I'd be using a combination like `du -s *`.
Besides, there are more ways to hide the data from yourself (e.g. symlinking your desktop from somewhere else).
That said, I have found `ncdu` invaluable for navigating through the results, swapping between different views etc.
> And on Linux desktops, I believe that common GUI file browsers will omit dot-directories by default and may not even have a particularly accessible option to change that (this is certainly the behavior of Cinnamon's 'Files' application and I can't imagine that GNOME is different, considering their attitude).
"Cinnamon's 'Files' application" is Nemo, wherein you can right-click on any window and check the "show hidden files" option in the context menu (admittedly a strange placement, since it's a global setting).
I don't understand. `du` indeed finds the dotfiles (and dot-folders); and if their contents are large, then they'll be noticed by someone who "looks at whatever big things show up". It's not at all clear to me why I'd be using a combination like `du -s *`.
Besides, there are more ways to hide the data from yourself (e.g. symlinking your desktop from somewhere else).
That said, I have found `ncdu` invaluable for navigating through the results, swapping between different views etc.
> And on Linux desktops, I believe that common GUI file browsers will omit dot-directories by default and may not even have a particularly accessible option to change that (this is certainly the behavior of Cinnamon's 'Files' application and I can't imagine that GNOME is different, considering their attitude).
"Cinnamon's 'Files' application" is Nemo, wherein you can right-click on any window and check the "show hidden files" option in the context menu (admittedly a strange placement, since it's a global setting).
I'd like to second the recommendation of ncdu.
If anything can be taken home from this article, it's that perhaps desktop environments need to do a better job helping users to understand disk utilisation.
The article's own recommendation would just make things worse. Application authors should stick to XDG standard directories.
If anything can be taken home from this article, it's that perhaps desktop environments need to do a better job helping users to understand disk utilisation.
The article's own recommendation would just make things worse. Application authors should stick to XDG standard directories.
I'm not sure I agree with this. The `~/.cache` directory is a standard, and it's usually the first place I check when I suspect my disk usage is too high. I do most of my personal work on a 15-inch MacBook Air with 16 GB of memory but only a tiny 256 GB disk.
I agree that putting cache files in `~/.cargo/cache` is worse than using the global `~/.cache`, but imagine the chaos if everyone decided to add their cache directory directly under the home folder without putting a dot in front of it.
Also, I believe the XDG spec should be followed on both macOS and Linux. Doing it differently on Mac is just a pain.
I agree that putting cache files in `~/.cargo/cache` is worse than using the global `~/.cache`, but imagine the chaos if everyone decided to add their cache directory directly under the home folder without putting a dot in front of it.
Also, I believe the XDG spec should be followed on both macOS and Linux. Doing it differently on Mac is just a pain.
Yeah, no. The author's suggestion is reminiscent of the way Windows apps tend to dump their icons and directories straight onto the Desktop. It creates a God-awful mess that effectively hides anything the user tries to put there.
Imagine what your home directory would look like if you did 'mv ${HOME}/.cache/* ${HOME}'. No thank you.
App developers should not clutter my home directory with visible application-specific subdirectories. That's a fast way to get your application kicked to the curb.
Imagine what your home directory would look like if you did 'mv ${HOME}/.cache/* ${HOME}'. No thank you.
App developers should not clutter my home directory with visible application-specific subdirectories. That's a fast way to get your application kicked to the curb.
I aliased ls to ls -A ages ago. While of course this doesn't solve any specific problems, it keeps one much more aware of what all's floating around in your home directory dotfiles.
> It's my view that this is a mistake and that everyone should put their big caches in a clearly visible directory or directory hierarchy, one that people can actually find in practice.
ls -a ~
ls -a ~
You can put `.cache` in RAM (reducing wear on your SSD), with a fixed size limit. If something tries to exceed that limit, you'll know:
$ df ~/.cache
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 819200 66360 752840 9% /home/user/.cacheWith the manner exposed by the most feature-full DEs, Iʼd expect setting default to ~/Cache (not ~/cache, the latter is for the userʼs discretion).
But, I tend to agree with the main articleʼs premise that without a special teaching for the issue most newcomers will lose the issue.
But, I tend to agree with the main articleʼs premise that without a special teaching for the issue most newcomers will lose the issue.
Well looks like everyone has already said what i came to say.
XDG for the win.
The problem isn't the standard, its all the applications that don't follow it.
XDG for the win.
The problem isn't the standard, its all the applications that don't follow it.
I really like ncdu, which helps interactively detect and delete big unused directories.
Use an appropriately named folder (hi Nvidia, I hate your ".nv") in XDG_CACHE_HOME like every well behaved program in the past decade and let the disk space utilities solve the visibility issue.