The trouble with symbolic links(lwn.net)
lwn.net
The trouble with symbolic links
https://lwn.net/Articles/899543/
123 comments
> You can't normally hard link directories.
That's only to avoid loops, as far as I understand. Symlinks do allow loops, but require application programmers to handle them. So maybe we just need better APIs/API contracts around loops, rather than two types of links?
> If a file has multiple links, finding them all normally requires scanning the entire file system
Couldn't this pretty easily be solved at the file system level? Just store a back pointer from a file to each of its names.
The fact that it's possible to break symlinks very easily by deleting the pointed-to file (name) is a problem as well: Wouldn't application developers usually, or at least sometimes, want to know about the fact that they are about to break a link (or conversely, not deleting the final copy of a file and not just a reference to it)?
> So there are more reasons for symlinks than just "hard links are restricted to linking within the same filesystem"
I think this might be the only real (technical/historical) limitation. The rest could probably be worked around, but maybe having two distinct types of links, with these other binary decisions (allowing loops, making deletion explicit vs. a matter of referenc counting) being more or less arbitrarily bucketed into those two types based on what was easier to implement.
That's only to avoid loops, as far as I understand. Symlinks do allow loops, but require application programmers to handle them. So maybe we just need better APIs/API contracts around loops, rather than two types of links?
> If a file has multiple links, finding them all normally requires scanning the entire file system
Couldn't this pretty easily be solved at the file system level? Just store a back pointer from a file to each of its names.
The fact that it's possible to break symlinks very easily by deleting the pointed-to file (name) is a problem as well: Wouldn't application developers usually, or at least sometimes, want to know about the fact that they are about to break a link (or conversely, not deleting the final copy of a file and not just a reference to it)?
> So there are more reasons for symlinks than just "hard links are restricted to linking within the same filesystem"
I think this might be the only real (technical/historical) limitation. The rest could probably be worked around, but maybe having two distinct types of links, with these other binary decisions (allowing loops, making deletion explicit vs. a matter of referenc counting) being more or less arbitrarily bucketed into those two types based on what was easier to implement.
I tried to construct my argument to make it clear that I'm aware there are ways to solve the issues with hard links, but they have their own sets of trade-offs.
For hard links, it's not only that they can cause loops. There are the other issues I outlined (linking across file systems, no single canonical representation of the file in the file system, finding all the links to the file, etc).
There's no "just store a back pointer." That will obviously introduce its own set of complexities and trade-offs. Where do you store the pointers? What's the API for viewing them? What's the CLI for viewing them? Is it a new switch to `ls`? A new CLI entirely? How do you keep the pointers up to date? What sort of locking is needed when updating the pointers? What about `fsck`? How do you get this implemented across the multitude of Unix and Unix-like OS's and file systems?
(As an aside, I've been really trying to stop using the word "just" lately as I've learned that things are rarely so simple to justify the word.)
Again, I'm not saying there isn't a better solution, but I don't think it's patching up hard links. I think it's something outside the box of both hard links and symbolic links.
For hard links, it's not only that they can cause loops. There are the other issues I outlined (linking across file systems, no single canonical representation of the file in the file system, finding all the links to the file, etc).
There's no "just store a back pointer." That will obviously introduce its own set of complexities and trade-offs. Where do you store the pointers? What's the API for viewing them? What's the CLI for viewing them? Is it a new switch to `ls`? A new CLI entirely? How do you keep the pointers up to date? What sort of locking is needed when updating the pointers? What about `fsck`? How do you get this implemented across the multitude of Unix and Unix-like OS's and file systems?
(As an aside, I've been really trying to stop using the word "just" lately as I've learned that things are rarely so simple to justify the word.)
Again, I'm not saying there isn't a better solution, but I don't think it's patching up hard links. I think it's something outside the box of both hard links and symbolic links.
Re: Symlink analysis: Well said.
> (As an aside, I've been really trying to stop using the word "just" lately as I've learned that things are rarely so simple to justify the word.)
Me too! I realized how it immediately frustrated me to hear it used about my domains. I’m constantly having to work to not seem as short/blunt/know-it-all as I feel. I think this word is a connotation trap, because when I use it feels inoffensive, but when I hear it seems blunt and dismissive and I’m quick to assume the person doesn’t understand or empathize with the complexities of the situation. That’s a long way of saying I really enjoyed your aside.
> (As an aside, I've been really trying to stop using the word "just" lately as I've learned that things are rarely so simple to justify the word.)
Me too! I realized how it immediately frustrated me to hear it used about my domains. I’m constantly having to work to not seem as short/blunt/know-it-all as I feel. I think this word is a connotation trap, because when I use it feels inoffensive, but when I hear it seems blunt and dismissive and I’m quick to assume the person doesn’t understand or empathize with the complexities of the situation. That’s a long way of saying I really enjoyed your aside.
Totally agree with “just”.
I’ve also tried to eliminate “but” since it usually comes across as “throw out whatever I just said and focus on this instead”.
The language we use is important and worth optimizing.
I’ve also tried to eliminate “but” since it usually comes across as “throw out whatever I just said and focus on this instead”.
The language we use is important and worth optimizing.
I noticed recently that I often preface statements with "just wanted to say" or "just chiming in here" and similar. I cringed hard when I realized and am working on eliminating that use of "just". Seems like the same general thing: it's never "just" X.
Right? And that’s a double whammy. You’re minimizing the statement you want to make, and you’re minimizing your place in that conversation.
> [...] I think it's something outside the box of both hard links and symbolic links.
Absolutely agreed – given your examples and all the other challenges around backwards compatibility with decades of application code, I'd also assume it would be something new entirely.
But my guess is that it would be able to meet the existing use cases of both.
Absolutely agreed – given your examples and all the other challenges around backwards compatibility with decades of application code, I'd also assume it would be something new entirely.
But my guess is that it would be able to meet the existing use cases of both.
[deleted]
Hard links don't have a canonical name though - they're all equally the same file, and this is really a problem: opening and editing a file in one location, edits it in all of them without you knowing what those locations might be.
Symlinks at least explicitly declare the dependency and how it should mutate.
A classic being /etc/resolve.conf symlinks - if I'm untarring and restore a symlink for it, I'm currently saying the file should have content from somewhere else on the system - not that the file is specific content.
Symlinks at least explicitly declare the dependency and how it should mutate.
A classic being /etc/resolve.conf symlinks - if I'm untarring and restore a symlink for it, I'm currently saying the file should have content from somewhere else on the system - not that the file is specific content.
> Hard links don't have a canonical name though - they're all equally the same file, and this is really a problem: opening and editing a file in one location, edits it in all of them without you knowing what those locations might be.
That is something the filesystem could store tho, in the same way it stores the number of links to a file it could be a bit more capable and store the links themselves (possibly in a xattr).
> Symlinks at least explicitly declare the dependency and how it should mutate.
They only declare one dependency one way, it's not like a symlink gives you all the other symlinks to the terminal location it will affect.
That is something the filesystem could store tho, in the same way it stores the number of links to a file it could be a bit more capable and store the links themselves (possibly in a xattr).
> Symlinks at least explicitly declare the dependency and how it should mutate.
They only declare one dependency one way, it's not like a symlink gives you all the other symlinks to the terminal location it will affect.
Symlinks do that too even inevitably: no matter how you change the file, it changes at all links and you can't prevent it; systemd uses this feature when it creates dependency references (the linked dependency must never differ from the source, what hard links don't ensure).
The difference is a symlink at least declares this explicitly. A hard link on the other hand looks and works like an independent file...but isn't one.
> Couldn't this pretty easily be solved at the file system level?
It will not solve a problem that does not even exist in the first place, but will rather badly break the semantics of the UNIX file system precisely at the file system level.
> Just store a back pointer from a file to each of its names.
UNIX file systems do not have files in the conventional sense. They have disk block allocations referenced to by an inode and one or more directory entries pointing back to a specific block allocation via the associated inode. This makes hard links easily possible and very cheap. It is a one to many relationship (one block allocation to many directory entries), and turning it into a many to many relationship, with each directory entry pointing to every single possible permutation of other directory entries across the entire file system a nightmare in every imaginable way.
It is even possible to zero directory entries pointing to an inode (if you poke around with the file system debugger, you can manually delete the last remaining directory entry without releasing allocated blocks into the disk block pool but the next fsck run will reclaim them anyway).
It will not solve a problem that does not even exist in the first place, but will rather badly break the semantics of the UNIX file system precisely at the file system level.
> Just store a back pointer from a file to each of its names.
UNIX file systems do not have files in the conventional sense. They have disk block allocations referenced to by an inode and one or more directory entries pointing back to a specific block allocation via the associated inode. This makes hard links easily possible and very cheap. It is a one to many relationship (one block allocation to many directory entries), and turning it into a many to many relationship, with each directory entry pointing to every single possible permutation of other directory entries across the entire file system a nightmare in every imaginable way.
It is even possible to zero directory entries pointing to an inode (if you poke around with the file system debugger, you can manually delete the last remaining directory entry without releasing allocated blocks into the disk block pool but the next fsck run will reclaim them anyway).
> It is even possible to zero directory entries pointing to an inode.
Historically, fsck would link such anonymous inodes into lost+found using their inode number as their name in the lost+found directory, but I admit having no idea whether this still applies to modern journaled file systems.
Historically, fsck would link such anonymous inodes into lost+found using their inode number as their name in the lost+found directory, but I admit having no idea whether this still applies to modern journaled file systems.
File system journals have reduced the likelihood of unlinked inodes ending up in /lost+found but have not eliminated it completely. There is still a non-zero chance a journal corruption as well during a unexpected shutdown or complete power loss during the journal update and something turning up after a full fsck run later.
>> You can't normally hard link directories.
> That's only to avoid loops, as far as I understand
Later HFS+ does support directory hard links, a feature introduced for Time Machine IIRC, but generally unavailable to the user.
> That's only to avoid loops, as far as I understand
Later HFS+ does support directory hard links, a feature introduced for Time Machine IIRC, but generally unavailable to the user.
> Couldn't this pretty easily be solved at the file system level? Just store a back pointer from a file to each of its names.
In theory yes, but no filesystem does this as far as I know.
In theory yes, but no filesystem does this as far as I know.
In Windows both NTFS and ReFS keep backpointers to all their names. They store the file ID of the directory, and the name of the file in the directory. In NTFS these are stored as a special attribute, and in ReFS they reside as rows in the file table.
It's required for a few reasons. Historically NTFS has had an API to query all of a files names and this needs to be done efficiently. And when a file is opened by ID, the file system needs to construct a canonical path for it in the name space.
Source: I am the Microsoft developer that added hardlink support to ReFS. All opinions are my own.
It's required for a few reasons. Historically NTFS has had an API to query all of a files names and this needs to be done efficiently. And when a file is opened by ID, the file system needs to construct a canonical path for it in the name space.
Source: I am the Microsoft developer that added hardlink support to ReFS. All opinions are my own.
[deleted]
ELOOP errno
symlinks are great, I don't see why we would remove such feature. The author pointed out a bunch of issues around atomic operations related to symlinks which in my view are valid. Similar TOCTOU race exists with PIDs, see https://lwn.net/Articles/773459/
Not sure whether the pid issue was ever resolved, havn't checked in on that in a while.
Not sure whether the pid issue was ever resolved, havn't checked in on that in a while.
Symlinks are great from a "just make it work!" point of view but they're absolutely terrible from a "make it robust, sane and secure" point of view.
All of the points in the article are valid but there's even simpler stuff like the fact that you can't canonicalise paths (resolve ..) without reading the filesystem.
This should be required reading: https://9p.io/sys/doc/lexnames.html
All of the points in the article are valid but there's even simpler stuff like the fact that you can't canonicalise paths (resolve ..) without reading the filesystem.
This should be required reading: https://9p.io/sys/doc/lexnames.html
You can resolve some .. components without reading the filesystem and there are situations where it is useful to do that, while refusing to treat .. that cannot be resolved without accessing the filesystem.
One example is inside the implementation of a function that calculates relative paths:
The function is very useful with this restriction, which can be externally worked around if ever necessary, e.g. by tacking the absolute path of the current directory onto both arguments:
One example is inside the implementation of a function that calculates relative paths:
1> (rel-path "a/b" "../c")
"../../../c"
2> (rel-path "../c" "a/b")
** rel-path: from path uses .. to escape common prefix: "../c" "a/b"
** during evaluation at expr-2:1 of form (rel-path "../c" "a/b")
The function is very useful with this restriction, which can be externally worked around if ever necessary, e.g. by tacking the absolute path of the current directory onto both arguments:
3> (rel-path `@(pwd)/../c` `@(pwd)/a/b`)
"../kaz/a/b"I take it an “alias” isn’t good in this case? An alias does follow a move of the original - usually.
Interesting; how does that work under the hood?
It's macOS magic that requires HFS/APFS and doesn't work at the POSIX layer. It would not work for my use case, no.
An alias is like a hybrid between a symbolic link and a hard link. Like a symbolic link, it's its own file type whose contents point to the original, but like a hard link it points to the original using its ID, not its path. So an alias works even if the original is moved, but it does not increase the original's link count and is its own distinct entity in the file system.
An alias is like a hybrid between a symbolic link and a hard link. Like a symbolic link, it's its own file type whose contents point to the original, but like a hard link it points to the original using its ID, not its path. So an alias works even if the original is moved, but it does not increase the original's link count and is its own distinct entity in the file system.
A posix filesystem by itself is not a defensible security perimeter. Symlinks introduce security problems but there are other sources as well. If you have a system where processes with different trust profiles share a common view of a file system you have to assume one can manipulate the filesystem state to subvert the other.
Android has dealt with this via locking down and isolating apps to their own filesystems. Cross app communication and data sharing utilizes IPC primitives that have rich caller/Callie information that can be used to build capabilities and authn/authz checks.
The posix filesystem just doesn't have the abstractions/expressiveness one would need to build a robust security perimeter between untrusted apps.
Android has dealt with this via locking down and isolating apps to their own filesystems. Cross app communication and data sharing utilizes IPC primitives that have rich caller/Callie information that can be used to build capabilities and authn/authz checks.
The posix filesystem just doesn't have the abstractions/expressiveness one would need to build a robust security perimeter between untrusted apps.
> The posix filesystem just doesn't have the abstractions/expressiveness one would need to build a robust security perimeter between untrusted apps.
This, absolutely, but I think it's even worse than that; in my mind the value prop of k8s is twofold: declarative configuration and isolation that forces apps to be able to interact over a very small boundary, the network overlay.
This, absolutely, but I think it's even worse than that; in my mind the value prop of k8s is twofold: declarative configuration and isolation that forces apps to be able to interact over a very small boundary, the network overlay.
IMO the article's conclusion is backwards. There's nothing wrong with symlinks when files are opened with openat(), since in principal the program (or controlling program) should always be in control of the filesystem layout. It's open() that causes problems, and complex interactions with symlinks in attacker-controlled directories are just one of them.
The POSIX file API was designed before the concept of capability passing (arguably, before the concept of computer security in general). A modern replacement would look more like Fuchsia, where child processes are provided file access scoped to their parent process's authority. This same scoping can also be used within a process, for example to implement a server that can "self-chroot".
A better example might have been listxattr() vs flistxattr(), because the former works on a file without read permissions, but the latter fails on a descriptor opened with O_PATH.
The POSIX file API was designed before the concept of capability passing (arguably, before the concept of computer security in general). A modern replacement would look more like Fuchsia, where child processes are provided file access scoped to their parent process's authority. This same scoping can also be used within a process, for example to implement a server that can "self-chroot".
> So, more functions following the pattern of openat() had to be created
> [...]
> Some are still missing, like getxattrat() and setxattrat().
A better example might have been listxattr() vs flistxattr(), because the former works on a file without read permissions, but the latter fails on a descriptor opened with O_PATH.
listxattr("xattr-chmod000.txt", NULL, 0) = 14
listxattr("xattr-chmod000.txt", "user.testattr\0", 14) = 14
getxattr("xattr-chmod000.txt", "user.testattr", NULL, 0) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "xattr-chmod000.txt", O_RDONLY|O_PATH) = 3
flistxattr(3, NULL, 0) = -1 EBADF (Bad file descriptor)"There's nothing wrong with symlinks when files are opened with openat(), since in principal the program (or controlling program) should always be in control of the filesystem layout. It's open() that causes problems, and complex interactions with symlinks in attacker-controlled directories are just one of them."
One of my minor annoyances with new languages is the continued persistence of open-based file APIs, with openat APIs shoved off to the side if they are even implemented at all. If you start from scratch with an openat-based API, it's not even that hard; you basically get a file object, just one with some different attributes and methods (or appropriate local ideas), most of which you don't care about, and it's not that hard to work with if you start with that from day one. It can be quite hard to backport something deeply based on string-based path manipulation into the *at-based APIs, though.
I haven't deeply studied it but you ought to be able to simulate an openat-based API on a conventional filesystem that doesn't support it. It may not immunize you to security issues, but at least the code ought to be as portable as any other code that starts getting detailed about its interactions with filesystems, which is already "kinda, not very, some elbow grease required"... it's not like the bar is sky high because all that stuff already works perfectly across all platforms and filesystems anyhow.
One of my minor annoyances with new languages is the continued persistence of open-based file APIs, with openat APIs shoved off to the side if they are even implemented at all. If you start from scratch with an openat-based API, it's not even that hard; you basically get a file object, just one with some different attributes and methods (or appropriate local ideas), most of which you don't care about, and it's not that hard to work with if you start with that from day one. It can be quite hard to backport something deeply based on string-based path manipulation into the *at-based APIs, though.
I haven't deeply studied it but you ought to be able to simulate an openat-based API on a conventional filesystem that doesn't support it. It may not immunize you to security issues, but at least the code ought to be as portable as any other code that starts getting detailed about its interactions with filesystems, which is already "kinda, not very, some elbow grease required"... it's not like the bar is sky high because all that stuff already works perfectly across all platforms and filesystems anyhow.
Well there is the solution: work with file descriptors and not with paths. POSIX should be extended to make sure all functions that take a path has also the version that takes the file descriptor (to avoid the /proc/self/fd/%d hack, that is not portable to non-Linux OS that don't have /proc, and on Linux requires /proc to be mounted that is not always the case for example in sandboxes and chroots).
You don't also only have problem with symlinks if you work with paths, but with any kind of paths. For example is wrong to check with the path if a file exists and then do something with it, because it can as well be deleted, modified, etc. You have to work with file descriptors, and use only one function (open) to resolve the path into a descriptor one time (that is also more efficient, since resolving a path is computationally expensive, especially on modern filesystems).
You don't also only have problem with symlinks if you work with paths, but with any kind of paths. For example is wrong to check with the path if a file exists and then do something with it, because it can as well be deleted, modified, etc. You have to work with file descriptors, and use only one function (open) to resolve the path into a descriptor one time (that is also more efficient, since resolving a path is computationally expensive, especially on modern filesystems).
"X is fundamentally broken" is a tired trope. To me, something is broken if it is no longer working as intended. It used to work, but now it does not - it is broken.
If something works as intended, but its utility is limited, and it can be improved, it is not broken.
If something works as intended, but its utility is limited, and it can be improved, it is not broken.
Symlinks work as intended, but they cause a lot of unintended security vulnerabilities, i.e. they break lots of otherwise-functioning code.
You can play with your words and redefine their meanings, but the vulnerabilities remain.
You can play with your words and redefine their meanings, but the vulnerabilities remain.
> You can play with your words and redefine their meanings, but the vulnerabilities remain.
I don't think OP was trying to play with words, I do think there's an absolute "this symlink thing is a vuln" vs an absolute "I use symlinks to make X work" argument. Symlinks have always been at the line between the absolutes. They do enable a great deal of functionality but they can be a security risk, and source of bugs when developers don't handle them correctly. That said, they are heavily used feature on unix like oses. My /usr/bin on Ubuntu has 48 of them (most were put there by apt installed packages).
I don't think OP was trying to play with words, I do think there's an absolute "this symlink thing is a vuln" vs an absolute "I use symlinks to make X work" argument. Symlinks have always been at the line between the absolutes. They do enable a great deal of functionality but they can be a security risk, and source of bugs when developers don't handle them correctly. That said, they are heavily used feature on unix like oses. My /usr/bin on Ubuntu has 48 of them (most were put there by apt installed packages).
> they break lots of otherwise-functioning code.
There is no otherwise! POSIX has symbolic links: if your software does not function with symbolic links, it does not function on POSIX.
There is no otherwise! POSIX has symbolic links: if your software does not function with symbolic links, it does not function on POSIX.
Of course there is otherwise. Windows (more or less) doesn't have symlinks. Plan9 doesn't have symlinks. You don't have to have symlinks.
Can you not imagine anything other than POSIX?
Can you not imagine anything other than POSIX?
They have been around for 40+ years, they don't break code unless we are talking about code predating their introduction. It is not me playing with words, I am just pointing out a tired trope.
[deleted]
> but they cause a lot of unintended security vulnerabilities
No, bad coders on UNIX platforms do this.
The code may be valid code, but if it's intending to support running on UNIX it should do it properly not assuming it's on a FAT32 filesystem in 2022.
Or, even better, run the code you don't trust on fat32 filesystems, see how far that gets.
No, bad coders on UNIX platforms do this.
The code may be valid code, but if it's intending to support running on UNIX it should do it properly not assuming it's on a FAT32 filesystem in 2022.
Or, even better, run the code you don't trust on fat32 filesystems, see how far that gets.
Part of the problem is that handles on directories on which one can then use the the *at family of syscalls are not first-class citizens in many programming languages. Which in turn might be due to portability concerns with windows, e.g. Java's SecureDirectoryStream isn't available there[0].
Apparently windows does have an openat-like API[1], but it's low-level.
Programmers aren't using them because the language standard libraries point them in the wrong direction.
[0] https://github.com/google/guava/wiki/Release21#user-content-... [1] https://github.com/rust-lang/rust/blob/1c63ec48b8cbf553d291a...
Programmers aren't using them because the language standard libraries point them in the wrong direction.
[0] https://github.com/google/guava/wiki/Release21#user-content-... [1] https://github.com/rust-lang/rust/blob/1c63ec48b8cbf553d291a...
Weird that you need to dip down to an Nt* function to open a child given a parent handle. It's not like it's unheard of to be able to do that in the Windows API. The registry is also a hierarchical system and opening any key requires passing in a parent key handle (the root handles are predefined).
It's a quirk of history, imho. If Win32 had been written without concern for what came before, it probably would have more closely followed NT conventions.
But it wasn't. It followed on from Win16 and DOS so, to an extent, it emulated DOS-style path and file handling. After all, that's what developers and users were familiar with. The Windows registry did not have all this baggage so it followed the style of the NT kernel.
Though this doesn't explain why Win32 never added CreateFileRelativeToDirectoryHandleW
But it wasn't. It followed on from Win16 and DOS so, to an extent, it emulated DOS-style path and file handling. After all, that's what developers and users were familiar with. The Windows registry did not have all this baggage so it followed the style of the NT kernel.
Though this doesn't explain why Win32 never added CreateFileRelativeToDirectoryHandleW
This a serious issue in support of the language for running on a UNIX system.
Frankly blaming the kernel/filesystem for this is like saying I want my 720p monitor to display 8k correctly, it must be the display-drivers fault...
Frankly blaming the kernel/filesystem for this is like saying I want my 720p monitor to display 8k correctly, it must be the display-drivers fault...
Don't victim blame.
Do you really think that using open(), stat(), lstat() (!), realpath(), mkdir(), rename() etc etc etc is a sign of a bad coder? The problem is that the APIs set you up for unexpected failure, and even some of the provided workarounds to 'safely' handle symlinks don't do it well enough.
In the case of symlinks, I think it's fair to blame the tools rather than the workman.
Do you really think that using open(), stat(), lstat() (!), realpath(), mkdir(), rename() etc etc etc is a sign of a bad coder? The problem is that the APIs set you up for unexpected failure, and even some of the provided workarounds to 'safely' handle symlinks don't do it well enough.
In the case of symlinks, I think it's fair to blame the tools rather than the workman.
API is always simplification and is not supposed to be used without understanding concepts and reality under the hood.
Example: wanna show 1M POI in browser on some small territory. Openmaps/googlemaps API allows that, no prob. Looks good, yeah? Sorry, doesn't work. Because 1M is too large to show and browser gets stuck.
The API do not prevent _all_ kinds of legshooting engineers invent.
Example: wanna show 1M POI in browser on some small territory. Openmaps/googlemaps API allows that, no prob. Looks good, yeah? Sorry, doesn't work. Because 1M is too large to show and browser gets stuck.
The API do not prevent _all_ kinds of legshooting engineers invent.
Don't worry, they don't listen to speaking out against bigG or others for being bad
This is not victim blame. RTFM READ IT!!!
Most sane languages and low level tools describe what you want and how to work correctly.
If you don't want this feature in the filesystem, move to one that doesn't support it, or better yet submit a patch to run the filesytem you want with this feature deactivted for "security concerns".
Demanding a whole OS change the way it works for bad/lazy/inept coders is akin to 2 people getting blind drunk and blaming the other person or the drink for the stupid things they did. Take some responsibility.
Most sane languages and low level tools describe what you want and how to work correctly.
If you don't want this feature in the filesystem, move to one that doesn't support it, or better yet submit a patch to run the filesytem you want with this feature deactivted for "security concerns".
Demanding a whole OS change the way it works for bad/lazy/inept coders is akin to 2 people getting blind drunk and blaming the other person or the drink for the stupid things they did. Take some responsibility.
> Demanding a whole OS change the way it works for bad/lazy/inept coders is akin to 2 people getting blind drunk and blaming the other person or the drink for the stupid things they did. Take some responsibility.
And if people were just more careful, none of rust's memory safety stuff is needed! Also, why do modern languages hand hold multi-threading so much, just give developers some mutex primitives and let them have at it, the good coders will be just fine!
Of course the rest of us will have to deal with machines getting pwned due to security bugs, but hey, at least the "well written" programs won't have those problems...
And if people were just more careful, none of rust's memory safety stuff is needed! Also, why do modern languages hand hold multi-threading so much, just give developers some mutex primitives and let them have at it, the good coders will be just fine!
Of course the rest of us will have to deal with machines getting pwned due to security bugs, but hey, at least the "well written" programs won't have those problems...
Stop defending poor coding and lack of skill.
Everything you're saying is an excusory situation for hiring poor coders at minimum wage who can't or won't read documentation. This is 80IQ points South of frankly most of the conversations on here.
Yes the rest of us cope with security incidents. There will always be security incidents. Stop defending practices that leads to them.
Everything you're saying is an excusory situation for hiring poor coders at minimum wage who can't or won't read documentation. This is 80IQ points South of frankly most of the conversations on here.
Yes the rest of us cope with security incidents. There will always be security incidents. Stop defending practices that leads to them.
> Stop defending poor coding and lack of skill.
This is a hopelessly elitist attitude. Also it is a useless one, over 1000 CVEs, yelling "be better at your job!" is just going to result in another 1000 CVEs. That is exactly what happened for decades with buggy C code, buffer overflows and use after frees, for a long time the refrain was "just do better!".
Well millions of dollars of damages later, it turns out berating people to "just do better" doesn't actually make things any better. A combination of static analysis and runtime tooling, and then the eventual creation of new programming languages that allow for correct modeling of memory ownership, is what the industry en masse has decided on.
For APIs that get misused? The solution is to provide higher level APIs that allow programmers to easily accomplish the correct thing in a secure manner.
As an aside, and in general, when designing software, I want to maximize the amount of brain power I am dedicating to solving the business problem at hand. Dealing with poorly designed insecure APIs detracts from me getting my actual job done.
> Stop defending practices that leads to them.
The practice in this case is the direct use of filesystem APIs that were designed in the 1970s for a very different security ecosystem than what exists today.
Lots of things designed in the 1970s are not secure by default. Heck most things designed in the 1970s, outside of maybe some IBM Mainframe stuff, was not designed to be secure by default.
What you are arguing is that instead of buying a fire extinguisher to put in the kitchen of an old house, people should just try and not set things on fire.
I mean, yeah, sure, good goal, but buy the fire extinguisher anyway.
This is a hopelessly elitist attitude. Also it is a useless one, over 1000 CVEs, yelling "be better at your job!" is just going to result in another 1000 CVEs. That is exactly what happened for decades with buggy C code, buffer overflows and use after frees, for a long time the refrain was "just do better!".
Well millions of dollars of damages later, it turns out berating people to "just do better" doesn't actually make things any better. A combination of static analysis and runtime tooling, and then the eventual creation of new programming languages that allow for correct modeling of memory ownership, is what the industry en masse has decided on.
For APIs that get misused? The solution is to provide higher level APIs that allow programmers to easily accomplish the correct thing in a secure manner.
As an aside, and in general, when designing software, I want to maximize the amount of brain power I am dedicating to solving the business problem at hand. Dealing with poorly designed insecure APIs detracts from me getting my actual job done.
> Stop defending practices that leads to them.
The practice in this case is the direct use of filesystem APIs that were designed in the 1970s for a very different security ecosystem than what exists today.
Lots of things designed in the 1970s are not secure by default. Heck most things designed in the 1970s, outside of maybe some IBM Mainframe stuff, was not designed to be secure by default.
What you are arguing is that instead of buying a fire extinguisher to put in the kitchen of an old house, people should just try and not set things on fire.
I mean, yeah, sure, good goal, but buy the fire extinguisher anyway.
How is it hopelessly elitist to call out insecure code as being INSECURE!!!
My whole point is the same as yours fix it at the source. You seem to think hacking off the hands of some coders is safer (I may agree). But why not try to EDUCATE THEM?!?
Education costs 1000s of dollars at most rather than your hypothetical billion dollar APPLICATION LEVEL hack.
Why are you so elitist to assume people can't cope with these concepts?
My whole point is that they need to be tought they're running on a Unix server rather than an 1998 SD card. The rest of your complaining is either you don't understand this or are trying to excuse bad or insecure practice as acceptable. If this is your case. RUN THE CODE IN AN ENVIRONMENT WHERE THIS CAN'T HAPPEN. Seriously there are filesystems and options for this.
Calling for these features to be removed from extX, ZFS or other shows you don't understand storage technologies well enough.
My whole point is the same as yours fix it at the source. You seem to think hacking off the hands of some coders is safer (I may agree). But why not try to EDUCATE THEM?!?
Education costs 1000s of dollars at most rather than your hypothetical billion dollar APPLICATION LEVEL hack.
Why are you so elitist to assume people can't cope with these concepts?
My whole point is that they need to be tought they're running on a Unix server rather than an 1998 SD card. The rest of your complaining is either you don't understand this or are trying to excuse bad or insecure practice as acceptable. If this is your case. RUN THE CODE IN AN ENVIRONMENT WHERE THIS CAN'T HAPPEN. Seriously there are filesystems and options for this.
Calling for these features to be removed from extX, ZFS or other shows you don't understand storage technologies well enough.
> How is it hopelessly elitist to call out insecure code as being INSECURE!!!
If a lot of code, written by a lot of different engineers, all ends up being insecure, it is worth asking, why is code dealing with this particular domain so often insecure?
> But why not try to EDUCATE THEM?!?
You can do that, and of course we should, but here is the thing about security:
The good guys have to write secure code every time, or else the attackers guys win.
Eternal vigilance is inhumanly hard to maintain. A better solution is to write higher level APIs or API wrappers that don't have these flaws.
> Why are you so elitist to assume people can't cope with these concepts?
Sure they can, but how many concepts can people cope with at once? Humans have a limit for how much they can juggle in their head. A huge part of software engineering is picking what abstraction layer to operate at. If I am writing code that deals with tons of string parsing and manipulation, I'd be a fool to write it in C or C++. Now I've done that when I needed the performance, but managing a massive number of strings in native code is easily 5x the work compared to using a GC language that also automatically tracks string length.
C is the wrong abstraction there. And indeed an obscene number of security holes have historically centered around string processing in C. That is because on top of managing all the business logic (which may be obscenely complicated by itself!) engineers now have to do so in a language that is really bad at dealing with strings and they have to do a lot of mental work to ensure the code is correct.
If I am manually flipping bits in hardware, well, JS can do it (I have seen it!) but honestly, that shouldn't be anyone's language of choice for directly interfacing with hardware.
(Doing that in C, really fun!)
> Calling for these features to be removed from extX, ZFS or other shows you don't understand storage technologies well enough.
I am not saying that. I am saying that the original POSIX APIs make writing secure code around symlinks hard, and I am saying that solely based on the fact that a bunch of security holes around POSIX APIs and symlinks exist!
This isn't some shocking statement. The original POSIX APIs make a lot of things hard.
If a lot of code, written by a lot of different engineers, all ends up being insecure, it is worth asking, why is code dealing with this particular domain so often insecure?
> But why not try to EDUCATE THEM?!?
You can do that, and of course we should, but here is the thing about security:
The good guys have to write secure code every time, or else the attackers guys win.
Eternal vigilance is inhumanly hard to maintain. A better solution is to write higher level APIs or API wrappers that don't have these flaws.
> Why are you so elitist to assume people can't cope with these concepts?
Sure they can, but how many concepts can people cope with at once? Humans have a limit for how much they can juggle in their head. A huge part of software engineering is picking what abstraction layer to operate at. If I am writing code that deals with tons of string parsing and manipulation, I'd be a fool to write it in C or C++. Now I've done that when I needed the performance, but managing a massive number of strings in native code is easily 5x the work compared to using a GC language that also automatically tracks string length.
C is the wrong abstraction there. And indeed an obscene number of security holes have historically centered around string processing in C. That is because on top of managing all the business logic (which may be obscenely complicated by itself!) engineers now have to do so in a language that is really bad at dealing with strings and they have to do a lot of mental work to ensure the code is correct.
If I am manually flipping bits in hardware, well, JS can do it (I have seen it!) but honestly, that shouldn't be anyone's language of choice for directly interfacing with hardware.
(Doing that in C, really fun!)
> Calling for these features to be removed from extX, ZFS or other shows you don't understand storage technologies well enough.
I am not saying that. I am saying that the original POSIX APIs make writing secure code around symlinks hard, and I am saying that solely based on the fact that a bunch of security holes around POSIX APIs and symlinks exist!
This isn't some shocking statement. The original POSIX APIs make a lot of things hard.
People being educated for a specialist job, I wonder how they will cope. Well we hire in enough others for less serious problems, sure why not hire less than competent people to slap a fixed badge on the side of it.
If you want this level of abstraction it should be built into the framework or language you're using. I'm not saying go away and rewrite chrome in C you'd be chasing segfaults for 5 years.
If you want to hire people who don't care again. Run this in an environment where this doesn't matter. If you're ultra paranoid insist in a layer to compensate for people's failings.
Again there is nothing wrong with the tried and tested API that can't be fixed with a modicum of effort. Calling it broken or insisting it change to fit problems caused by people so far removed they don't know what architecture they're working code for is not a reason to change the POSIX system. It's a reason to fix your abstraction.
If you want this level of abstraction it should be built into the framework or language you're using. I'm not saying go away and rewrite chrome in C you'd be chasing segfaults for 5 years.
If you want to hire people who don't care again. Run this in an environment where this doesn't matter. If you're ultra paranoid insist in a layer to compensate for people's failings.
Again there is nothing wrong with the tried and tested API that can't be fixed with a modicum of effort. Calling it broken or insisting it change to fit problems caused by people so far removed they don't know what architecture they're working code for is not a reason to change the POSIX system. It's a reason to fix your abstraction.
I think we're arguing the same thing.
Write an abstraction on top of the hard to use API for the majority to use, and people who need to use the low level API directly can learn to do so.
Write an abstraction on top of the hard to use API for the majority to use, and people who need to use the low level API directly can learn to do so.
That “X − designed in the 70s when we had no idea of anything regarding computers − is fundamentally broken” isn't so surprising after all.
In fact, computers are probably the only place in the entire technology landscape where we keep using almost unmodified stuff from the 70s and decided we cannot change it because there's too much things relying on it.
I don't like breaking everything all the time more than anyone, but maybe one time every 20 or 30 years is OK…
In fact, computers are probably the only place in the entire technology landscape where we keep using almost unmodified stuff from the 70s and decided we cannot change it because there's too much things relying on it.
I don't like breaking everything all the time more than anyone, but maybe one time every 20 or 30 years is OK…
> In fact, computers are probably the only place in the entire technology landscape where we keep using almost unmodified stuff from the 70s and decided we cannot change it because there's too much things relying on it.
Bridges and buildings from the 1970s (and much older) are still working fine today.
The thing is, if I do decide to replace my bridge or building because it's old and outdated then I can just replace that one thing without affecting much else. With computers, that is obviously not the case: you need to replace the entire city.
Plus, it's not really the case that we "keep using almost unmodified stuff from the 70s"; while many concepts remained the same and things remained compatible, things have been greatly extended and modified since; it's like those old buildings that were built during the middle ages (or sometimes even earlier) that have been changed and upgraded extensively throughout the centuries to the point you really need to know where to look to see it's actually a centuries-old building.
Bridges and buildings from the 1970s (and much older) are still working fine today.
The thing is, if I do decide to replace my bridge or building because it's old and outdated then I can just replace that one thing without affecting much else. With computers, that is obviously not the case: you need to replace the entire city.
Plus, it's not really the case that we "keep using almost unmodified stuff from the 70s"; while many concepts remained the same and things remained compatible, things have been greatly extended and modified since; it's like those old buildings that were built during the middle ages (or sometimes even earlier) that have been changed and upgraded extensively throughout the centuries to the point you really need to know where to look to see it's actually a centuries-old building.
> Bridges and buildings from the 1970s (and much older) are still working fine today.
With modern earthquake straps added, and I bet the locks got replaced a few times over, also the building likely had its insulation improved, better venting added, a sprinkler system, fire exits, and a wheelchair ramp put in at some point.
Are there a few quaint stone bridges from 1700 still in use? Sure, going over the neighborhood creek. But all the bridges around me have undergone serious upgrades or retrofitting over the decades.
With modern earthquake straps added, and I bet the locks got replaced a few times over, also the building likely had its insulation improved, better venting added, a sprinkler system, fire exits, and a wheelchair ramp put in at some point.
Are there a few quaint stone bridges from 1700 still in use? Sure, going over the neighborhood creek. But all the bridges around me have undergone serious upgrades or retrofitting over the decades.
> Bridges and buildings from the 1970s (and much older) are still working fine today.
I am not sure about that, floods here go past the 200 years average line at the time that many bridges or buildings was designed. And actually breaks a lot of buildings.
Climate change these days is just as unexpected as hackers these days to who we were.
I am not sure about that, floods here go past the 200 years average line at the time that many bridges or buildings was designed. And actually breaks a lot of buildings.
Climate change these days is just as unexpected as hackers these days to who we were.
It would be like upgrading the train network to increase the distance between rails, to increase the size of cargo that can be transported.
Not just one city needs the upgrade, all of them will need it, and all the related infrastructure like bridges and tunnels too.
Not just one city needs the upgrade, all of them will need it, and all the related infrastructure like bridges and tunnels too.
I don't think that's entirely true. There's plenty of major systems that have made fundamentally incompatible breaking changes in order to move things forward. Windows did that with Vista, Android did that with Linux (eg, app sandboxing per UID, heavily restricted filesystem access to shared directories), etc..
It's kinda mainly desktop/server Linux where there's this inability to move forward.
It's kinda mainly desktop/server Linux where there's this inability to move forward.
Android was for a type of device and userbase that had never run Linux, and so there were no pre-existing notions about what it should do, no pre-existing programs that needed to run, etc.
A better example would be Apple which makes breaking changes to both iOS and macOS absolutely all the time.
A better example would be Apple which makes breaking changes to both iOS and macOS absolutely all the time.
For the launch version sure but Android has made no shortage of breaking changes since then, like all the storage changes. Or selinux clamp down. Or permission changes. Or background restrictions. Or etc...
Internal plumbing is largely unchanged. Sure, there's more flexible pipe, and a lot more plastic pipe, and a lot more quarter turn valves, but thread pitch and pipe diameters are largely unchanged and unchangable.
Does writing count as technology? The Chinese characters are thousands of years old.
Writing is a craft or art learned and refined through skill.
I'd argue that it is.
I would probably draw a line between writing and vernacular speech, in the sense of language acquired simply through assimilation and not specifically trained or drilled.
There might be writing which fails that test (e.g., very basic literacy), and of speech which passes (advanced rhetoric, debate, accents and impressions, singing, etc.).
But as a socially-acquired means-to-an-end refined by practice and study, yes, technology.
I'd argue that it is.
I would probably draw a line between writing and vernacular speech, in the sense of language acquired simply through assimilation and not specifically trained or drilled.
There might be writing which fails that test (e.g., very basic literacy), and of speech which passes (advanced rhetoric, debate, accents and impressions, singing, etc.).
But as a socially-acquired means-to-an-end refined by practice and study, yes, technology.
I take it that here "technology landscape" means something like "the gui portion of the software stack", right?
If it used to work without letting others pwn your account or box, and not it fails to, it's broken.
The word "broken" came from before we had constant arms races in technology. Obviously we don't call clubs broken because we now have rapid artillery, but there was enough time between clubs and swords, and swords and guns to allow transitions away.
When I'm exposed to a core OS feature I expect by default that it should not come with expected, critical security vulnerabilities. It is rational to expect a user to use the basic features of an OS and expect them not to cause severe issues. You can say it's not broken, and that's true in as much as they're still functional, but if by broken one means the larger question of "is this reliable and safe?" then I think the answer is pretty clear that symlinks are broken.
When I'm exposed to a core OS feature I expect by default that it should not come with expected, critical security vulnerabilities. It is rational to expect a user to use the basic features of an OS and expect them not to cause severe issues. You can say it's not broken, and that's true in as much as they're still functional, but if by broken one means the larger question of "is this reliable and safe?" then I think the answer is pretty clear that symlinks are broken.
In the modern world, the demand seems to be that every tool be perfectly safe in every situation no matter what you do (and it seems practically nothing lives up to this demand, given the ever increasing river of silly CVEs for almost every component, like regex DoS on build tools).
It's important to understand the scope of the issue. If you create and operate on your own symlinks in your own folders, there is no problem. The problem is when a more privileged user operates on folders that can be written to by less privileged users, for example system daemons (like a /tmp cleanup, or a web server serving /home/*/www), or suid binaries. These things need to be written very carefully, it is now clear.
But if I'm working with my own files, media, source code, build tools, web pages, etc, in my own folders, then symlinks are still fine.
And there is an existing setting to mitigate a couple common forms of the issue that does exist when accessing folders other users can write to: https://www.kernel.org/doc/html/latest/admin-guide/sysctl/fs...
It's important to understand the scope of the issue. If you create and operate on your own symlinks in your own folders, there is no problem. The problem is when a more privileged user operates on folders that can be written to by less privileged users, for example system daemons (like a /tmp cleanup, or a web server serving /home/*/www), or suid binaries. These things need to be written very carefully, it is now clear.
But if I'm working with my own files, media, source code, build tools, web pages, etc, in my own folders, then symlinks are still fine.
And there is an existing setting to mitigate a couple common forms of the issue that does exist when accessing folders other users can write to: https://www.kernel.org/doc/html/latest/admin-guide/sysctl/fs...
> In the modern world, the demand seems to be that every tool be perfectly safe in every situation no matter what you do
The problem is that there is such a huge number of tools in widespread use that each one causing even a few security vulnerabilities means that the ecosystem overall is constantly vulnerable
The problem is that there is such a huge number of tools in widespread use that each one causing even a few security vulnerabilities means that the ecosystem overall is constantly vulnerable
I appreciate the reply, but after thinking about it I think it's more akin to someone having been sold a house only to be told eight years later that the seller of the house knew that if someone tied a shoelace to the front door and pulled on it, then the entire house would explode.
Could we consider it a broken doorknob Pierce?
Could we consider it a broken doorknob Pierce?
It's more akin to pulling the shoelace, and the door closes on your fingers. Oh no, doors are unsafe, how utterly broken.
Or, more like, if there's an attacker hiding in your house, while you're setting up the shoelace door thing for some odd reason, they could slam the door on your fingers. Oh no, how were we ever allowed to have doors, so criminally unsafe.
Or, more like, if there's an attacker hiding in your house, while you're setting up the shoelace door thing for some odd reason, they could slam the door on your fingers. Oh no, how were we ever allowed to have doors, so criminally unsafe.
Interestingly, Windows actually did exactly what's proposed at the end when MS added them in Vista. To minimize the security issues with symlinks, you had to elevate to admin to create them.
It was only during the life of Windows 10 that they even added the option to not have to elevate to create them. It was done specifically because symlinks are often shared across systems since they end up in places like git repos and npm packages: https://blogs.windows.com/windowsdeveloper/2016/12/02/symlin...
It was only during the life of Windows 10 that they even added the option to not have to elevate to create them. It was done specifically because symlinks are often shared across systems since they end up in places like git repos and npm packages: https://blogs.windows.com/windowsdeveloper/2016/12/02/symlin...
> It was only during the life of Windows 10 that they even added the option to not have to elevate to create them
Nah it was available from the start in local security policy settings.
https://docs.microsoft.com/en-us/previous-versions/windows/i...
Nah it was available from the start in local security policy settings.
https://docs.microsoft.com/en-us/previous-versions/windows/i...
Thanks for the correction. I actually think I did use this option way back, but just completely forgot about it.
There is an option (that is proposed when you install git) to allow normal users to create symlinks. The fact is that symlinks are very useful to a developer, and a lot of development tools make use of them.
Maybe I'm being naive, but I don't get how "pathnames as a concept are now utterly broken in POSIX". Isn't this "merely" a problem that the resolution of the path name is dynamic and can change between inspection and use? Wouldn't a practice of resolving pathnames once (recursively, atomically, whatever) into an immutable, opaque, direct handle, such as file descriptor, before use solve this issue? I realize what I just said may be tantamount to "all file io ops taking path strings are broken" - but that seems like a problem with the initial API design, not with the concept of having a level of indirection in path name resolution itself.
This is basically what I was going to say. The article spends a lot of time arguing that TOCTOU patterns introduce security vulnerabilities, which I think all programmers (should!) already know but then comes to the weird conclusion that we'd just be better off without symlinks instead of designing an API to work with them atomically.
Kinda reminds me of how a lot of UX changes happen: "This really popular feature is a bit kludgy and hard to maintain, let's just rewrite the whole app without it! (Instead of doing the work required to make it not suck.)"
Kinda reminds me of how a lot of UX changes happen: "This really popular feature is a bit kludgy and hard to maintain, let's just rewrite the whole app without it! (Instead of doing the work required to make it not suck.)"
Almost all the TOCTOU examples given in the article could be modified not to involve symlinks and still be valid.
Don't hard links suffer from the issue that because they're actually links to a specific file, not path pointers, that you can replace the target file thinking you have updated something in the system and instead have stale hard links lying around, referencing the older version when you intended to replace the older version for all users?
I think that in a hypothetical world where symlinks worked like hard links, we'd be swapping out the security complaints in this post for articles about how hard it is to upgrade a POSIX system properly, tools and tricks you can use to make sure you truly replaced all instances of a binary with a known vulnerability, and so on.
I think that in a hypothetical world where symlinks worked like hard links, we'd be swapping out the security complaints in this post for articles about how hard it is to upgrade a POSIX system properly, tools and tricks you can use to make sure you truly replaced all instances of a binary with a known vulnerability, and so on.
Hardlinks exist, so you already have this problem. Tar has to keep track of inode numbers to recreate hard links for example.
What I mean is that it's pretty SOP to have a package that installs a new command to work by installing to /opt/my-package and then symlinking /usr/bin/my-cmd to /opt/my-package/my-cmd.
In the absence of symlinks, that link would be hard and dpkg et. al would have to do package management by deleting the /usr/bin/my-cmd link and re-creating it instead of letting it ride, trusting that it will point to the correct thing when the update completes because the target bin will have changed.
In the absence of symlinks, that link would be hard and dpkg et. al would have to do package management by deleting the /usr/bin/my-cmd link and re-creating it instead of letting it ride, trusting that it will point to the correct thing when the update completes because the target bin will have changed.
Another problem with hard links: you can not hard link a directory. It would make ".." ambiguous.
Also with hard link directories, you would want to be able "rmdir" non-empty directories, just to delete the link. But then you have the problem of reference loops, so how do you reclaim space reliably? You would need a garbage collection algorithm to find data not reachable by root.
Also with hard link directories, you would want to be able "rmdir" non-empty directories, just to delete the link. But then you have the problem of reference loops, so how do you reclaim space reliably? You would need a garbage collection algorithm to find data not reachable by root.
Whether directories can be hardlinked depends on the filesystem and OS. When macOS switched from HFS+ to APFS, one of the changes was that they dropped support for directory hardlinks.
https://developer.apple.com/library/archive/documentation/Fi...
https://developer.apple.com/library/archive/documentation/Fi...
I'm not a macOS user, but these sure seem to add complexity:
https://stackoverflow.com/questions/80875/what-is-the-unix-c...
The filesystem has to check for and disallow loops.
https://stackoverflow.com/questions/80875/what-is-the-unix-c...
The filesystem has to check for and disallow loops.
The GC algorithms for hardlinked directories can be the same as for harlinked files: reference counting.
Only if there are no loops..
> An application running as root may try to check that /data/mydir is a regular directory (not a symlink) before opening the file /data/mydir/passwd. In between the time the program does the directory check and the file open, an attacker could replace the mydir directory with a symlink to /etc, and now the file opened is, unexpectedly, /etc/passwd. This is a kind of race condition known as a time-of-check-to-time-of-use (TOCTOU) race.
That application is doing the wrong check; it should be validating that every component of the path is a directory which is only writable to root.
First you stat("/"). OK, that is a directory and writable only to root: so no non-root process can put a symlink there. Next we check "/data". OK, that's a directory, and since we know / is owned by root and not world-writable, /data cannot be replaced by a symlink.
And so on ...
This can easily be made into a function like safe_path("/data/dir/path/to/mypasswd") which returns true only if no pathname component is something which a user other than the caller, or root, could tamper with to point to a different file.
The open system call should have a flag for this, O_SAFE. That would alter the behavior of the name resolution function (traditionally, "namei") to do these checks along the path.
The path could have symlinks, if they are not tamperable from the POV of the calling user.
Typically superuser applications in Unix rely on filesystem structure. They set environment variables like PATH carefully, and stick to accessing data in known directories that had better be safe. If /data/mydir/passwd is something that is manipulated by a root application, then the system is misconfigured if any of these is writable to a non-root user: /, /data, /data/mydir or /data/mydir/passwd.
If that is the case, you don't need symlinks to wreak havoc on the application. You can, for instance, write your own password into that password file and then falsely authenticate with that app.
That application is doing the wrong check; it should be validating that every component of the path is a directory which is only writable to root.
First you stat("/"). OK, that is a directory and writable only to root: so no non-root process can put a symlink there. Next we check "/data". OK, that's a directory, and since we know / is owned by root and not world-writable, /data cannot be replaced by a symlink.
And so on ...
This can easily be made into a function like safe_path("/data/dir/path/to/mypasswd") which returns true only if no pathname component is something which a user other than the caller, or root, could tamper with to point to a different file.
The open system call should have a flag for this, O_SAFE. That would alter the behavior of the name resolution function (traditionally, "namei") to do these checks along the path.
The path could have symlinks, if they are not tamperable from the POV of the calling user.
Typically superuser applications in Unix rely on filesystem structure. They set environment variables like PATH carefully, and stick to accessing data in known directories that had better be safe. If /data/mydir/passwd is something that is manipulated by a root application, then the system is misconfigured if any of these is writable to a non-root user: /, /data, /data/mydir or /data/mydir/passwd.
If that is the case, you don't need symlinks to wreak havoc on the application. You can, for instance, write your own password into that password file and then falsely authenticate with that app.
There doesn't seem to be a way to batch together operations that involve walking through directories and symlinks to do something to a file. This seems to be a major source of complexity.
I always thought Unix v7+ should have added some kind way to do atomic groups of syscalls, eg:
It could also solve the whole thing with ending up with zero-length files because you didn't use the right incantation to update a file atomically on ext4 (https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-a...).
begin_transaction ();
lstat ("/path", ...);
lstat ("/path/foo", ...);
commit ();
It could also solve the whole thing with ending up with zero-length files because you didn't use the right incantation to update a file atomically on ext4 (https://thunk.org/tytso/blog/2009/03/12/delayed-allocation-a...).
A general purpose transactional interface widene the error space to include cross process deadlocks / denial of service not to mention performance issues.
Could you elaborate? Seems like there’s a bunch of things that can’t be batched together on an ordinary file, without involving symlinks.
What they (and TFA) are saying is that there is no transactional view of the FS. If you could work in “repeatable read” (only and always see the state of the FS before you started the transaction) symlink races wouldn’t be possible.
Hard linking files isn’t useful in my experience because it requires every tool working with that file to never delete it and recreate it. However that’s what exactly many tools do. So the only way to reliably “share” a file is with a symlink. At least this is true for my workflows.
I really think that an "open the file as this user or group (also constrained to the current user's permissions)" option will solve the security problems better than the "open the file relative to this root" one. Or, failing that, an usable capability system (not SELinux).
I don't think just checking for symlinks really solves anything. It may make your bugs harder to exploit, what is always good, but people use symlinks, so you have to support them, so the bugs will stay there.
I don't think just checking for symlinks really solves anything. It may make your bugs harder to exploit, what is always good, but people use symlinks, so you have to support them, so the bugs will stay there.
No, there is absolutely nothing broken about symbolic links. What is the issue here is accessing user files as root. That is inherently unsafe in POSIX and also affects hardlinks as well (even more so, since you have to "follow" hardlinks http://michael.orlitzky.com/articles/posix_hardlink_heartach...).
>Clients that have write access to the exported part of the file system under a share via SMB1 unix extensions or NFS can create symlinks that
So, it's not symlinks broken, but SMB1 unix extensions broken when exposed to the world for symlink creation. AIU this features doesn't even serve windows interoperability. And if the author wanted to disable all symlinks altogether, what is the purpose of these extensions?
So, it's not symlinks broken, but SMB1 unix extensions broken when exposed to the world for symlink creation. AIU this features doesn't even serve windows interoperability. And if the author wanted to disable all symlinks altogether, what is the purpose of these extensions?
The trouble with symbolic links is users who don't understand/use linking coming from alternate platforms and developing on/for UNIX.
This is a feature in the same way that shellsock was used as a feature for many years by experts. Thankfully I'm expecting something like POSIX to save us this time.
This is a feature in the same way that shellsock was used as a feature for many years by experts. Thankfully I'm expecting something like POSIX to save us this time.
Does anyone else read something like this
Jeremy Allison gave a talk titled "The UNIX Filesystem API is profoundly broken: What to do about it?".
and immediately shut down on the person saying it because he all know it's extremely hyperbolic given what is happening in reality?
It's great to not like something and point out its flaws as far as you use them and "here's the great idea to fix those issues" but to try to offend you audience and the Unix community as the first words you see in a talk is a great way to have people shut off and think "oh great another neckbeard with an overinflated sense of self"
Jeremy Allison gave a talk titled "The UNIX Filesystem API is profoundly broken: What to do about it?".
and immediately shut down on the person saying it because he all know it's extremely hyperbolic given what is happening in reality?
It's great to not like something and point out its flaws as far as you use them and "here's the great idea to fix those issues" but to try to offend you audience and the Unix community as the first words you see in a talk is a great way to have people shut off and think "oh great another neckbeard with an overinflated sense of self"
I personally think that hard links should go away. I have trouble thinking of any use for hard links that isn’t better served by CoW links. Hard links have quite surprising properties with respect to chmod, they are awkward to handle in archival tools, and even reliably identifying them is awkward to impossible in general.
Features add complexity and robust features add robust complexity. Robust features that span a core component like filesystem handling span many utilities.
Maybe we should do the same search on samba vulnerabilities have have him take a look in the mirror...
Maybe we should do the same search on samba vulnerabilities have have him take a look in the mirror...
Response in code: https://news.ycombinator.com/item?id=32201150
I was copying a file today and thought: "Why is it duplicating the data? Why does it not just create a new second filename pointer in the filesystem?"
Take a look at the cp man page -
https://www.man7.org/linux/man-pages/man1/cp.1.html
Look for "reflink" and you get your wish :)
https://www.man7.org/linux/man-pages/man1/cp.1.html
Look for "reflink" and you get your wish :)
FICLONE is the underlying ioctl:
https://www.man7.org/linux/man-pages/man2/ioctl_ficlone.2.ht...
As of Linux 5.18, it's supported by btrfs, xfs, overlayfs and some networking filesystems.
https://www.man7.org/linux/man-pages/man2/ioctl_ficlone.2.ht...
As of Linux 5.18, it's supported by btrfs, xfs, overlayfs and some networking filesystems.
Very cool! :)
The problem here is Linux.
As an example of all these issues, I manage a bunch of Mac build hosts with multiple Xcode versions installed. We only retain the most recent patch release of each major.minor version, but drop compatibility symlinks in place for the other versions. On macOS, an application is just a directory. So for example we'll have:
From a simple "ls" it's obvious which versions are installed and which are just compatibility shims. Symlinks are just so damn convenient for use cases like this. Hard links don't cut the mustard here.
So there are more reasons for symlinks than just "hard links are restricted to linking within the same filesystem", but yes, that too.
Probably I'm just lacking imagination and there's a solution that offers the advantages of symlinks with none of the downsides, but in my experience, we see this sort of indirection all over the computing landscape, so it seems like there's a fundamental need for it.