I've worked on projects with people who've insisted on these approaches, and I've never understood how this secures things in any deep way. As I understand the reasoning, open("/a/b",...) is discouraged in favor of openat(fd, "b", ...) because an attacker might be able to hijack /a. But as I see things, if an attacker has ever been able to hijack what /a contains or what "/a" refers to, they can change whatever "b" ends up naming, and the game is over.
As for the article's glnx_chaseat(), ISTM the 8 orthogonal flag bits are a warning indicator. If there need to be 256 ways to configure pathname resolution semantics, then there are 255 ways the flags compiled into program can be inappropriate for any particular use. Even if we stipulate that there's a problem here that both needs solving, if those flags aren't a hint that this is a flaky solution, I don't know what is.
So it has seemed to me as if the real security problem is the existence of a file system shared among unrelated sets of processes, and that if there is a secure alternative to that on existing operating systems, it probably looks more like embedding program configuration or data inside a program. (But this is handwavy: I'm envisioning stuffing data into ELF or Mach-O segments in signed binaries; some novel mechanism would need to be invented for shebang scripts.) But probably compartmentalizing all systems into distinct VMs is more practical than redesigning all software. (I would imagine that since the article's author works on Flatpak, they are motivated to want something less than VMs to serve as viable compartmentalization solutions, however.)
A while ago I stumbled across a technique for improving stream buffering that I wish more I/O library implementors knew about. Korn and Vo's sfio library (circa 1991) had a feature called "pooling", whereby distinct streams could be linked together. Any read or write operation to any stream in a pool implicitly synchronized all the other streams in the pool first. This way, when stdio and stderr were pooled, which was the default when both went to ttys, a write on stderr implicitly flushed stdout. I've implemented this feature for myself a couple times; it's fairly easy to do and basically eliminates the need to explicitly flush streams in client code.
Citation: https://archive.org/details/1991-proceedings-tech-conference... but note that the explanation of stream pools there is a little less precise and more general than really necessary. I believe that later versions of sfio simplified things somewhat, though I could be wrong. (I find their code fairly hard to read.)
Anyhow, ISTM a missed opportunity when new languages that don't actually use libc's routines for something reinvent POSIX's clunkier aspects.
Re: obtaining a legal copy of Genera, as of 2023 Symbolics still existed as a corporate entity and they continued to sell x86-64 laptops with "Portable Genera 2.0". I bought one from them then, and occasionally see them listing some on Ebay. (This isn't intended as an advertisement or endorsement, just a statement. I think it's quite unfortunate that Symbolics's software hasn't been made freely available, since it's now really only of historical interest.)
I don't know about the Unix certification process itself, but the Single Unix Specification explicitly mentions case-insensitivity among non-conforming file system behaviors that are allowed as extensions (in 2.1.1 item 4, third-to-last bullet):
So a conforming OS has to make case-sensitive file systems available (which MacOS does: you can create case-sensitive HFS or APFS volumes). But I'm not sure if a conforming OS instance (i.e., running system) has to have any case-sensitive mount points, and either way, AFAIK there's no practical and race-free way for a conforming application to detect whether any particular mount point behaves case-sensitively.
So I believe that as far as the standard goes, a conforming application might run on a conformingly-extended OS where no portion of the the file namespace behaves case-sensitively. IOW, a conforming application cannot rely on case-sensitivity for file names.
As for the article's glnx_chaseat(), ISTM the 8 orthogonal flag bits are a warning indicator. If there need to be 256 ways to configure pathname resolution semantics, then there are 255 ways the flags compiled into program can be inappropriate for any particular use. Even if we stipulate that there's a problem here that both needs solving, if those flags aren't a hint that this is a flaky solution, I don't know what is.
So it has seemed to me as if the real security problem is the existence of a file system shared among unrelated sets of processes, and that if there is a secure alternative to that on existing operating systems, it probably looks more like embedding program configuration or data inside a program. (But this is handwavy: I'm envisioning stuffing data into ELF or Mach-O segments in signed binaries; some novel mechanism would need to be invented for shebang scripts.) But probably compartmentalizing all systems into distinct VMs is more practical than redesigning all software. (I would imagine that since the article's author works on Flatpak, they are motivated to want something less than VMs to serve as viable compartmentalization solutions, however.)