Rich Felker of musl libc comments on Google's LLVM libc proposal(lists.llvm.org)
lists.llvm.org
Rich Felker of musl libc comments on Google's LLVM libc proposal
http://lists.llvm.org/pipermail/llvm-dev/2019-June/133308.html
163 comments
This, and I also strongly argue that compiler implementations should be untied from standard libraries, and this also applies to libc++/libstdc++. A compiler shipping its runtime is a bad idea that totally screws up compatibility (just look at how painful it is to use MINGW + C++ on Windows because it can't link with VS's C++ Runtime - I understand that MS headers are crap to parse, but with a bit of effort clang now does it fine), and ties them together and prevents reuse. In an ideal world shipping a runtime should be the OS's role, not the compiler's, and the compiler should make use of whatever the platform ships, instead of having N compilers shipping N different, incompatible libraries. The interface between the library and the compiler would then become a true language standard, instead of "library for compiler version X.Y.Z"
Raymond Chen wrote about this in his 2014 blog post "Windows is not a Microsoft Visual C/C++ Run-Time delivery channel"
https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12...
Stable ABIs are good, but the ability to break it and iterate/improve is even more so. GNU platforms are currently stuck with a shitty std::string implementation (32 bytes but only 15 bytes of SSO capacity) because breaking ABI means recompiling your entire Linux distro.
https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12...
Stable ABIs are good, but the ability to break it and iterate/improve is even more so. GNU platforms are currently stuck with a shitty std::string implementation (32 bytes but only 15 bytes of SSO capacity) because breaking ABI means recompiling your entire Linux distro.
As he notes there, Windows does ship a C/C++ runtime, but you're not allowed to use that one - it is only for internal Windows stuff and only ever has to be compatible with whatever software comes with Windows. Any Windows update can make it binary-incompatible with a third-party program. User axjs notes in another top-level comment that a Google libc(++) will always just serve Google's products, including keeping bugs that a Google product relies on - somewhat similar to the Microsoft C/C++ runtime that comes with Windows, which you are strongly encouraged not to use.
That ship has sailed a long time ago, anything compiled with MinGW links against MSVCRT.DLL and did that from the beginning. Considering they had to backpedal changes in that DLL because it broke some program, as mentioned in the linked blog, i doubt this DLL will ever be removed as that will break thousands of applications that have come to rely on it.
>anything compiled with MinGW links against MSVCRT.DLL
It has changed recently, MinGW-w64 supports linking with UCRT instead of msvcrt.dll. Unfortunately, it's almost undocumented[1].
[1] - https://sourceforge.net/p/mingw-w64/mailman/message/36621319...
It has changed recently, MinGW-w64 supports linking with UCRT instead of msvcrt.dll. Unfortunately, it's almost undocumented[1].
[1] - https://sourceforge.net/p/mingw-w64/mailman/message/36621319...
Considering that it isn't the default and it is only available on Windows 10 (and that only since 2015) i think that it is very safe to say that for all intents and purposes "anything compiled with MinGW links against MSVCRT.DLL" is valid and will be for a long time.
Though even if MinGW switched to UCRT tomorrow, it would still leave a ton of executables since 1998 (or whenever it started using it) relying on MSVCRT.DLL (which AFAIK is included by default in Windows since Windows 98 so it has a slightly wider support range than UCRT :-P).
Though even if MinGW switched to UCRT tomorrow, it would still leave a ton of executables since 1998 (or whenever it started using it) relying on MSVCRT.DLL (which AFAIK is included by default in Windows since Windows 98 so it has a slightly wider support range than UCRT :-P).
If it will only ever serve Google then it shouldn't be part of upstream.
Well glibc kind of proved that individual maintainers can be just as bad for stewardship[0].
[0]: https://news.ycombinator.com/item?id=7914022
[0]: https://news.ycombinator.com/item?id=7914022
Why not start with musl?
This is a common complaint about companies starting new open source projects, and it really annoys me.
Anyone who works with software knows there are tons of very good reasons why you might want to start a new project from scratch instead of contributing to an existing one. You might find the existing architecture limiting. The trade offs taken to be non optimal (for your use case or in general). Consider the existing code to be low quality. The code contribution process to lax (introducing bugs) or strict (slowing work). Find the community toxic. Want to utilize existing internal code bases. Etc.
There are tons of valid reasons to make a new project, and for most of them you don't want to say it out loud. Saying "your code sucks" or "we don't like you (i.e. toxic community)" is rude. Saying "we don't like the processes" is asking for a long political fight, and likely to be taken as rude by the people who invented them. Saying that they want to utilize internal code bases could be problematic internally. Etc.
In this case I get the feeling that quite a bit of it is disagreeing with the tradeoff's musl took (see: Wanting to make libc more like a library), which has to be about the only reason you are allowed to publicly say. Maybe there are other reasons as well, maybe not.
Sorry to jump down your throat, it's just a common type of comment that I wanted to give a general reply to. (Edit: To be clear I'm not involved with this project, or a google employee for that matter)
Anyone who works with software knows there are tons of very good reasons why you might want to start a new project from scratch instead of contributing to an existing one. You might find the existing architecture limiting. The trade offs taken to be non optimal (for your use case or in general). Consider the existing code to be low quality. The code contribution process to lax (introducing bugs) or strict (slowing work). Find the community toxic. Want to utilize existing internal code bases. Etc.
There are tons of valid reasons to make a new project, and for most of them you don't want to say it out loud. Saying "your code sucks" or "we don't like you (i.e. toxic community)" is rude. Saying "we don't like the processes" is asking for a long political fight, and likely to be taken as rude by the people who invented them. Saying that they want to utilize internal code bases could be problematic internally. Etc.
In this case I get the feeling that quite a bit of it is disagreeing with the tradeoff's musl took (see: Wanting to make libc more like a library), which has to be about the only reason you are allowed to publicly say. Maybe there are other reasons as well, maybe not.
Sorry to jump down your throat, it's just a common type of comment that I wanted to give a general reply to. (Edit: To be clear I'm not involved with this project, or a google employee for that matter)
This is reasonable.
A simple answer like "we are not starting with musl because ..." would be quite welcome, though.
A simple answer like "we are not starting with musl because ..." would be quite welcome, though.
I assume, that Google's answer would sound like this:
"We are not starting with musl, because we want to own all our software the same way people own hamsters or dogs: care for them, feed them, train them, single handedly decide whether to castrate or kill them. We don't need the risk of owning something we don't fully control."
As far, as I am aware, musl isn't owned by public company or foundation. Google can't exert desirable degree of control over musl project, because there is no board of directors to buy their way into, — it is all maintained by one guy (and chattel slavery is outlawed in US...)
"We are not starting with musl, because we want to own all our software the same way people own hamsters or dogs: care for them, feed them, train them, single handedly decide whether to castrate or kill them. We don't need the risk of owning something we don't fully control."
As far, as I am aware, musl isn't owned by public company or foundation. Google can't exert desirable degree of control over musl project, because there is no board of directors to buy their way into, — it is all maintained by one guy (and chattel slavery is outlawed in US...)
Bribing or hiring the dev can still work.
When a new project is proposing to do something that's been done before, potentially multiple times, I think it is reasonable to want to understand how the new project will compare and contrast to work that's already been done.
Asking "How will this project compare to <existing implementation>?" seems quite reasonable to me. In fact, I would expect the person making the proposal to address this question as part of the proposal. It is not desirable to reinvent the wheel without good reason.
It's totally fine if the reason is something like: we want to use a different license, or a different development process, or a different coding style, or whatever. Whatever the actual motivations are, clarifying them up front will be good for everyone.
Asking "How will this project compare to <existing implementation>?" seems quite reasonable to me. In fact, I would expect the person making the proposal to address this question as part of the proposal. It is not desirable to reinvent the wheel without good reason.
It's totally fine if the reason is something like: we want to use a different license, or a different development process, or a different coding style, or whatever. Whatever the actual motivations are, clarifying them up front will be good for everyone.
I agree this with as some one working at a company that avoids most open source like the plague. But in this particular case, the code quality in musl is pretty good. I haven't had an issue with it so far, and after trying to upstream some things in glibc I have to say I can't see why they wouldn't use musl.
Well, there are still huge holes in musl. towlower e.g uses a linear search. my binary search made it 2x faster, but we still have to wait for that to appear. Implemented Unicode standards are years old.
You don't know which standard is implemented.
Still many problems with dynamic linking.
And: musl is POSIX. POSIX blocking IO should not be used in a modern OS or language. You can never guarantee concurrency safety with POSIX. This is the Fuchsia problem. They went for unsafety and POSIX compat. L4 ditto. Microsoft's Singularity not.
A modern libc would need to get rid of blocking IO and implement strings as utf8 only. There's no need for anything else. The C standard made too many mistakes.
And: musl is POSIX. POSIX blocking IO should not be used in a modern OS or language. You can never guarantee concurrency safety with POSIX. This is the Fuchsia problem. They went for unsafety and POSIX compat. L4 ditto. Microsoft's Singularity not.
A modern libc would need to get rid of blocking IO and implement strings as utf8 only. There's no need for anything else. The C standard made too many mistakes.
I don't see how you're planning on writing a libc that chooses to interpret the C standard as full of "mistakes" without writing an entirely different language.
I'm already maintaining the libc security extensions, which are in the standard but all libc's refuse to implement.
I also added the missing string API's for the wrong wide API, but thinking about adding the proper u8 api instead. Nobody really should use the wide (UTF-32/UTF-16) api.
But the C standard committee is somewhere completely else. There are not even aware of the problems, even if I'm getting interviewed from time to time, so that they will change their attitudes towards insecurity and functionality. searching for strings should be a pretty common usage pattern, but it's still not implemented. coreutils or grep or sed would like to use it, I'm sure. But not 10x slower as the current multibyte patches. Which I'm also maintaining btw. for coreutils.
Non-blocking IO into the standard? I only know about libuv which was fine before the drama. Despised it since then, but I'm still using it. The standard mostly needs to provide a namespace to block blocking IO if someone needs a concurrency-safe subset. Then the unhealthy need for POSIX compat can be circumvented somewhat. And gets() can be finally.
But the C standard committee is somewhere completely else. There are not even aware of the problems, even if I'm getting interviewed from time to time, so that they will change their attitudes towards insecurity and functionality. searching for strings should be a pretty common usage pattern, but it's still not implemented. coreutils or grep or sed would like to use it, I'm sure. But not 10x slower as the current multibyte patches. Which I'm also maintaining btw. for coreutils.
Non-blocking IO into the standard? I only know about libuv which was fine before the drama. Despised it since then, but I'm still using it. The standard mostly needs to provide a namespace to block blocking IO if someone needs a concurrency-safe subset. Then the unhealthy need for POSIX compat can be circumvented somewhat. And gets() can be finally.
What problems with dynamic linking? Considering docker and most container Linux systems use musl im sure they wouldnt mind knowing. Also musl has no problem having utf8 strings, in fact its one of the highlighted features.
pthread and dynamic linking: Look at their bug reports and mailing list discussions.
musl has no problem with utf8, because musl doesn't implement an utf8 string api. musl is strict C11. This is one of the problems. Most libc's do have this problem. There is casefolding, which is small but too slow (as explained). And 6 years behind. (Unicode 6.0 vs 12.1) There's no normalization at all, so you cannot search for strings, just for memory buffers. Which leads to insecure identifier handling, such as all public names, paths, ... There is a lot of identifier API in C11 and various syscalls/kernels, which are all insecure. Not musl's or any libc fault, but they should lead the pack, not follow wrong standards. Having checked unicode collate handling, guess there's none neither. It's planned for later this year I read. Unicode (the "wide" api) is UTF-32 which is too big and too slow.
musl has no problem with utf8, because musl doesn't implement an utf8 string api. musl is strict C11. This is one of the problems. Most libc's do have this problem. There is casefolding, which is small but too slow (as explained). And 6 years behind. (Unicode 6.0 vs 12.1) There's no normalization at all, so you cannot search for strings, just for memory buffers. Which leads to insecure identifier handling, such as all public names, paths, ... There is a lot of identifier API in C11 and various syscalls/kernels, which are all insecure. Not musl's or any libc fault, but they should lead the pack, not follow wrong standards. Having checked unicode collate handling, guess there's none neither. It's planned for later this year I read. Unicode (the "wide" api) is UTF-32 which is too big and too slow.
Yes, there are reasons to rewrite a library, but people expect you to tell them why you are rewriting it.
So... why are they rewriting it?
So... why are they rewriting it?
anth_anm(1)
Which do you think looks better on your promotion packet?
1. Write a new libc and get it used all over Google, generating lots of users automatically; or
2. Shepard Google's adoption of code some other folks wrote.
1. Write a new libc and get it used all over Google, generating lots of users automatically; or
2. Shepard Google's adoption of code some other folks wrote.
I don't think this level of cynicism is warranted
However, as the person who actually runs the relevant promo committees for the org, both would be fine, i'd care a lot more about why the choice was made.
However, as the person who actually runs the relevant promo committees for the org, both would be fine, i'd care a lot more about why the choice was made.
I love how your repose with 1st hand experience is downvoted as opposed to a pointless jab of the original poster :)
Googlers have the experience but not the distance to meaningfully comment on google.
Sorry, but this is just BS through and through.
Painting 100k people with a brush of "can't be objective" based on no data is probably the most biased think i've actually seen in a long time.
Painting 100k people with a brush of "can't be objective" based on no data is probably the most biased think i've actually seen in a long time.
I’ll paint 7 billion with the “proximity changes perception” brush.
> Googlers have the experience but not the distance to meaningfully comment on google.
Yes, having experience is still better than outright making things up because you have neither the experience, nor data or any kind of basis for a claim.
Yes, having experience is still better than outright making things up because you have neither the experience, nor data or any kind of basis for a claim.
Either one looks completely fantastic, I assure you.
I have helped people on my team write promotion packets in both forms. Both worked perfectly well.
I have helped people on my team write promotion packets in both forms. Both worked perfectly well.
Option 1 will take a long time. Writing a libc takes a long time, and transitioning the whole of Google codebase to a new libc takes even longer to iron out all of the incompatibilities.
If one goes with option 2, he/she may have many more achievements to write on his/her promotion packet during the time he/she would have spent implementing a new libc. So I hardly think promotion is the incentive here.
If one goes with option 2, he/she may have many more achievements to write on his/her promotion packet during the time he/she would have spent implementing a new libc. So I hardly think promotion is the incentive here.
Instead of resorting to theoretical rhetorics on meta issues on the given proposal,Rich Felker could ask this question as well. But he did not, I am actually quite disappointed with his response, it looks unprofessional.
These are the highlights IMO:
> Avoiding monoculture preserves the motivation for consensus-based standards processes rather than single-party control
> there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class".
> Avoiding monoculture preserves the motivation for consensus-based standards processes rather than single-party control
> there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class".
Rich is of course totally right. There's not only the botched musl in Fuchsia, also Android Bionic, which is a libc written in C++. It's utterly unusable.
They had a better, older, safer libc implementation based on some HP code, but that is gone.
Some of Rich' points are moot though. Google already have their own libc's, in Android and Fuchsia with enhanced security measures. Similar to Huawei or Intel or Cisco. Better than Apple's freebsd libc. Now reaching out to the community to outsource their development efforts under llvm is a good thing. There's obviously a need, musl, bsd and glibc cannot address. I'm sympathetic to that.
Some of Rich' points are moot though. Google already have their own libc's, in Android and Fuchsia with enhanced security measures. Similar to Huawei or Intel or Cisco. Better than Apple's freebsd libc. Now reaching out to the community to outsource their development efforts under llvm is a good thing. There's obviously a need, musl, bsd and glibc cannot address. I'm sympathetic to that.
And the need is? I read the thread and I still don't know.
This sounds like Android NDK development will get quite painful in a few years. It previously included a botched libc++ missing a bunch of features. This is libc, but I imagine subtle feature disparities and incompatibilities there will cause plenty of pain as well.
Android NDK development was been quite painful since ever, to the point sometimes I miss Symbian, and Symbian C++ patterns were not to fall in love with.
Not sure about you, but for me it was felt like a 20% project that someone on the Android team does something when they happen to be free of work.
If it wasn't for JetBrains coming up with CLion we still wouldn't have a replacement for Eclipse CDT, multiple reboots of NDK build tools, no real story on how to share libraries between NDK projects, native APIs are C wrappers to C++ code that we wrap again in C++ for proper type safety, permissions or file access require JNI, no AndroidX/JetPack like libraries and the list goes on.
The IO talk was more of state of nation than anything else.
Apple and Microsoft platforms are definitly more C++ developer friendly regarding the overall tooling, and mixed language development support as well.
Not sure about you, but for me it was felt like a 20% project that someone on the Android team does something when they happen to be free of work.
If it wasn't for JetBrains coming up with CLion we still wouldn't have a replacement for Eclipse CDT, multiple reboots of NDK build tools, no real story on how to share libraries between NDK projects, native APIs are C wrappers to C++ code that we wrap again in C++ for proper type safety, permissions or file access require JNI, no AndroidX/JetPack like libraries and the list goes on.
The IO talk was more of state of nation than anything else.
Apple and Microsoft platforms are definitly more C++ developer friendly regarding the overall tooling, and mixed language development support as well.
There are already some real pain points with bionic. I primarily write Linux code that sometimes has to run on Android (notably not any other UNIX). I've run into tons of problems porting to old versions of bionic, despite the kernel being the same.
I don’t think the Android team is the interested party here, given the original poster’s interest only in x86_64
Google doesn't do "only x86_64". Even when I was there (years ago) at a minimum all software had to run on PowerPC as well, and non-portable assembly/intrinsics required explicit approval from "Eng Elders" and a portable fallback. I wouldn't be surprised if ARM compat is also required now.
That might be true, but the email posted here explicitly lists x86_64 as the architecture they are targeting initially.
In my experience NDK development is quite painful now. Most of my NDK work is a little old now but I found the debugging process quite painful. This was for C++ code that was running on other platforms so almost all of the debugging was due to platform incompatibilities or bugs.
But I did notice in the linked article that they are only targeting x86_64, so I don't think this libc is targeted at Android.
But I did notice in the linked article that they are only targeting x86_64, so I don't think this libc is targeted at Android.
Make vs buy decisions work differently for companies with virtually unlimited buying power. The absolute control that doing it yourself provides cannot be bought. And Google has to care only the platforms it wants to support (as some guy at BSDcon said, you have Windows, Linux Intel/AMD/ARM and rounding errors).
Do you think the 100000 cookie cutter game developers for Android will stop developing for it because the platform libc has subtle issues? They rightfully don't give a shit.
Do you think the 100000 cookie cutter game developers for Android will stop developing for it because the platform libc has subtle issues? They rightfully don't give a shit.
Doesn't google use newlib already?
Wait, weren't they using Bionic on android?
Google is too big for the left hand to know what the right hand is doing. Various parts of Google use bionic, newlib, musl, and multiple forks of glibc.
So does that mean all that propaganda about them having a monorepo is bullshit?
Android (bionic), Chrome (forks of glibc and newlib in NaCl), ChromeOS (glibc with patches), and Fuchsia (musl) are all in separate Gerrit repositories. The monorepo for Google proper probably contains all of these, plus the libc used for Google.
What does “LLVM implementation of Libc” mean here? Does that just mean an implementation of libc maintained by the LLVM project?
Sounds like an LLVM "recommended" libc?
Loved the third point in Rich's reply:
"there is tremendous value in non-monoculture of libc implementations, or implementations of any important library interfaces or language runtimes. Likewise there's tremendous value in non-monoculture of tooling (compilers, linkers, etc.). Avoiding monoculture preserves the motivation for consensus-based standards processes rather than single-party control (see also: Chrome and what it's done to the web) and the motivation for people writing software to write to the standards rather than to a particular implementation. A big part of making that possible is clear delineation of roles between parts of the toolchain and runtime, with well-defined interface boundaries. Some folks have told me that I should press LLVM to make musl the "LLVM libc" instead of whatever Google wants to do, but that misses the point: there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class"."
Loved the third point in Rich's reply:
"there is tremendous value in non-monoculture of libc implementations, or implementations of any important library interfaces or language runtimes. Likewise there's tremendous value in non-monoculture of tooling (compilers, linkers, etc.). Avoiding monoculture preserves the motivation for consensus-based standards processes rather than single-party control (see also: Chrome and what it's done to the web) and the motivation for people writing software to write to the standards rather than to a particular implementation. A big part of making that possible is clear delineation of roles between parts of the toolchain and runtime, with well-defined interface boundaries. Some folks have told me that I should press LLVM to make musl the "LLVM libc" instead of whatever Google wants to do, but that misses the point: there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class"."
> there is tremendous value in non-monoculture of libc implementations
> any one library implementation that's "first class" for use with LLVM while others are only "second class"."
I agree with Zachary here. LLVM's really good at mixing and matching w/target libs IMO (and linkers too, while we're at it). FWIW lld and libc++/abi have been a part of the llvm project for a long time now and BFD ld and libstdc++ are still the defaults for the linux clang driver.
> any one library implementation that's "first class" for use with LLVM while others are only "second class"."
I agree with Zachary here. LLVM's really good at mixing and matching w/target libs IMO (and linkers too, while we're at it). FWIW lld and libc++/abi have been a part of the llvm project for a long time now and BFD ld and libstdc++ are still the defaults for the linux clang driver.
Correct. clang is always going to default to targeting whatever the platform ecosystem is. I don't see linux distros moving away from glibc any time soon.
The notion that a LLVM libc would in some way make other libc's "second class" is completely unfounded. libc++ vs. libstdc++ is proof of this.
The notion that a LLVM libc would in some way make other libc's "second class" is completely unfounded. libc++ vs. libstdc++ is proof of this.
The benefits of not having a monoculture resonate heavily with me. I'm convinced it is in line with antifragility.
Odd that so much of the goals folks make involve building monocultures.
Odd that so much of the goals folks make involve building monocultures.
Monocultures are easier to control. Control is easier to monetize. People thus seek to create Monocultures so they can make money.
If you think about it, one of the reasons to avoid Monocultures is because it reduces your control, and that's often seen through being price gouged in some way.
If you think about it, one of the reasons to avoid Monocultures is because it reduces your control, and that's often seen through being price gouged in some way.
This makes sense from a corporate perspective. But consumers seem to prefer it, as well. As do workers.
Consumers and workers prefer it when it matches what they want to a high enough degree. Too much variation means too many decisions that either do not matter or you have to be careful to get right. That's less of an issue when there's less choice.
The problem is that this requires the group controlling ecosystem to accurately follow the needs of the masses. If what they provide starts mismatching what people want, more and more people begin to be unhappy, and it's very easy for the goals of the public to misalign with the goals of the controlling group over time. Just because people seem to prefer it now doesn't mean they will a decade from now, but a decade of control can cause real problems when trying to make changes. It's very easy to get into a local maxima type situation, where it takes a lot of time and resources to just come to parity with an alternative, much less capitalize on where it can exceed the status quo.
Imagine a situation where Firefox didn't exist. Since all the other major browsers have moved to Webkit (or Webkit forked) engine, what would the cost be in time and effort of highly skilled people to make a browser engine from scratch? I'm not sure any project would succeed, or if it did, it might be a decade in the making.
People like ease of use, but people also really dislike being stuck without a choice. I think people realize this, which is why even though they often vote for convenience with their dollars, they also express a preference that alternatives exist, even if they don't use them, because they recognize both the benefit of having that option as well as the pressure it puts on their preferred choice.
I think if you ask, most iPhone users would say they prefer their current phone. I also think if you asked them if they think they would be better off without Android existing, they would say no, for exactly the reasons above; it makes Apple move forward with advancements, and it provides an alternative if they ever decide they really don't like something Apple is doing.
The problem is that this requires the group controlling ecosystem to accurately follow the needs of the masses. If what they provide starts mismatching what people want, more and more people begin to be unhappy, and it's very easy for the goals of the public to misalign with the goals of the controlling group over time. Just because people seem to prefer it now doesn't mean they will a decade from now, but a decade of control can cause real problems when trying to make changes. It's very easy to get into a local maxima type situation, where it takes a lot of time and resources to just come to parity with an alternative, much less capitalize on where it can exceed the status quo.
Imagine a situation where Firefox didn't exist. Since all the other major browsers have moved to Webkit (or Webkit forked) engine, what would the cost be in time and effort of highly skilled people to make a browser engine from scratch? I'm not sure any project would succeed, or if it did, it might be a decade in the making.
People like ease of use, but people also really dislike being stuck without a choice. I think people realize this, which is why even though they often vote for convenience with their dollars, they also express a preference that alternatives exist, even if they don't use them, because they recognize both the benefit of having that option as well as the pressure it puts on their preferred choice.
I think if you ask, most iPhone users would say they prefer their current phone. I also think if you asked them if they think they would be better off without Android existing, they would say no, for exactly the reasons above; it makes Apple move forward with advancements, and it provides an alternative if they ever decide they really don't like something Apple is doing.
Yeah, I am also confused. Isn't there Clang already?
(Please correct me if I am wrong)
The C library is generally considered a part of the OS and the overall toolchain. Clang is "merely" a C compiler.
The C library provides implementations for the platform-neutral language functions like "open/read/write/close/ioctl/dup/dup2" and more.
The C library provides implementations for the platform-neutral language functions like "open/read/write/close/ioctl/dup/dup2" and more.
That is an UNIX thing, due to their symbiotic relationship, in every other OS out there the C library is whatever comes with the C compiler.
So implementations of those subroutines are written in assembly lang?
No, it's very common for them to be written in primarily C. But I'm obligated to mention (as a member of the Rust Evangelism Strike Force) that you don't have to write them in C. ;)
Many implementations do have performance-critical parts be target-dependent and implemented in assembly. IIRC glibc has more of this than musl/newlib/bionic, but it's been a while since I last looked.
Many implementations do have performance-critical parts be target-dependent and implemented in assembly. IIRC glibc has more of this than musl/newlib/bionic, but it's been a while since I last looked.
> No, it's very common for them to be written in primarily C
Then can't you just get the LLVM IR out of it using Clang?
Then can't you just get the LLVM IR out of it using Clang?
The root of the issue is that, while you can write most of the code in C (but not all), it has to be different for different operating system kernels. It will use different syscall numbers, argument size and order and "extra options" will be different, some kernels will have a more general syscall with options that enable it to be used for a handful of similar standard libc wrapper functions ...
On macOS, you're really supposed to always dynamically link to the libSystem.dylib libc implementation, because the macOS kernel has subtle backwards-incompatible changes in the raw syscall interfaces on every macOS release (maybe even in point-release updates, I'm not sure). Go tried to have its own internal static libc replacement for macOS like it does for Linux, but just a year or two ago Go gave that up, and now dynamically links the macOS libc.
On macOS, you're really supposed to always dynamically link to the libSystem.dylib libc implementation, because the macOS kernel has subtle backwards-incompatible changes in the raw syscall interfaces on every macOS release (maybe even in point-release updates, I'm not sure). Go tried to have its own internal static libc replacement for macOS like it does for Linux, but just a year or two ago Go gave that up, and now dynamically links the macOS libc.
I think that's immaterial but you're right that you could dump the IR.
For the most part, C is a much easier language to program in than the IR. The IR is also very unstable: it changes from release to release.
For the most part, C is a much easier language to program in than the IR. The IR is also very unstable: it changes from release to release.
Somebody has to write the library to begin with. The typical Unix thing is for the OS to have one everybody links to. Which means, among other things, variation between libcs is a factor in portability of C programs from one OS to another.
[deleted]
> platform-neutral language functions like "open/read/write/close/ioctl/dup/dup2"
These are all syscall wrappers. They are present in libc but they are going to be very boring stubs that merely call into the kernel. (Also I don't think you can call them platform neutral or language functions as they are POSIX rather than being from the C standard...)
The more "implementation-heavy" parts of a libc... Things like stdio, string calls, malloc, ...
These are all syscall wrappers. They are present in libc but they are going to be very boring stubs that merely call into the kernel. (Also I don't think you can call them platform neutral or language functions as they are POSIX rather than being from the C standard...)
The more "implementation-heavy" parts of a libc... Things like stdio, string calls, malloc, ...
Those stubs have to correctly handle POSIX thread cancellation, which is "exciting" in the worst possible way.
Some of them also have to translate between the userspace version of particular argument types and the kernel version, which don't always line up.
Some of them also have to translate between the userspace version of particular argument types and the kernel version, which don't always line up.
pthread cancellation was a terrible idea when it was added, and it's even worse today. It's an API that provides no actual value (you can implement it yourself via signals), is a giant minefield if you care about not leaking or deadlocking, and infects the entirety of libc with pointless checks for functionality that no one in their right mind should use.
> pthread cancellation was a terrible idea when it was added, and it's even worse today
> you can implement it yourself via signals
It appears, that you are contradicting yourself. Either it is "bad idea" or it is easy to correctly implement by oneself, not both.
musl has well though-out implementation of pthread cancellation, so it is clearly doable, even if glibc developers have failed at it.
> you can implement it yourself via signals
It appears, that you are contradicting yourself. Either it is "bad idea" or it is easy to correctly implement by oneself, not both.
musl has well though-out implementation of pthread cancellation, so it is clearly doable, even if glibc developers have failed at it.
The problem with pthread cancellation is that it is fundamentally broken when used on a thread that can ever acquire resources (opening an fd, taking a mutex, mallocing a buffer, etc.). If it were just "call pthread_testcancel to figure out if you were cancelled", that would be fine, but no, you either get blown up immediately, or you get blown up at arbitrary function call boundaries (that don't really make any sense; e.g. why is strerror_r allowed to be a cancellation point?)
It's easy to correctly implement the parts that aren't just adding a call to pthread_testcancel to every single syscall wrapper, just reserve an RT signal and do your thread teardown when you receive it, using pthread_sigmask to implement enable/disable. It's just that it's just a terrible idea.
It's easy to correctly implement the parts that aren't just adding a call to pthread_testcancel to every single syscall wrapper, just reserve an RT signal and do your thread teardown when you receive it, using pthread_sigmask to implement enable/disable. It's just that it's just a terrible idea.
strerror[_r]() is allowed to be cancellation point, because it's implementation may involve calls to read() which is required to be cancellation point.
> It's an API that provides no actual value (you can implement it yourself via signals)
> is a giant minefield if you care about not leaking or deadlocking
Aren't these two in opposition?
> is a giant minefield if you care about not leaking or deadlocking
Aren't these two in opposition?
No. pthread_cancel implemented completely correctly is inherently a giant minefield. (Details in a reply to a sibling post)
Clang is a C language family frontend for LLVM. That is, it's a compiler for C and C++ and related languages.
Libc is the C standard library, and there are multiple implementations of it, e.g. glibc and musl. Clang does not include an implementation of libc, though it does have an implementation of the C++ standard library called libc++: https://libcxx.llvm.org/
It's not immediately clear to me why an LLVM implementation of the C++ standard library would be desirable while an implementation of the C standard library would be undesirable.
Libc is the C standard library, and there are multiple implementations of it, e.g. glibc and musl. Clang does not include an implementation of libc, though it does have an implementation of the C++ standard library called libc++: https://libcxx.llvm.org/
It's not immediately clear to me why an LLVM implementation of the C++ standard library would be desirable while an implementation of the C standard library would be undesirable.
> It's not immediately clear to me why an LLVM implementation of the C++ standard library would be desirable while an implementation of the C standard library would be undesirable.
Worth noting the c++ standard library has had a lot of changes in recent years. libc has been hugely more stable in the same timeframe. (Notably there was C11, but the changes since C99 are not nearly what has been going on in c++ land.) The odds that whatever libc you already have is good enough are very high.
Worth noting the c++ standard library has had a lot of changes in recent years. libc has been hugely more stable in the same timeframe. (Notably there was C11, but the changes since C99 are not nearly what has been going on in c++ land.) The odds that whatever libc you already have is good enough are very high.
One of the replies is this email from an LLVM developer working at Google: http://lists.llvm.org/pipermail/llvm-dev/2019-June/133315.ht...
I can't say it looks particularly good for someone at Google to be using their moderator powers on this unrelated-to-Google list to tell Felker not to include parts of his message which reflect badly on Google.
I can't say it looks particularly good for someone at Google to be using their moderator powers on this unrelated-to-Google list to tell Felker not to include parts of his message which reflect badly on Google.
I see nothing wrong with this. The last thing you want in a technical discussion are petty personal attacks. The moderator sounds a lot like something you would see dang say on HN.
I don't think they're petty personal attacks. They are important concerns which I share.
Regardless, since this particular member of the project has clear incentives to be biased in favor of Google, they should have recused themselves, and asked another member of the project/moderator of the list to judge whether the message was appropriate.
Regardless, since this particular member of the project has clear incentives to be biased in favor of Google, they should have recused themselves, and asked another member of the project/moderator of the list to judge whether the message was appropriate.
Chandler is as related to the person responsible for putting musl in Fuchsia as are any two randomly selected software developers from anywhere in the industry. Having to disqualify someone from moderating a mailing list based on the union of all the conflicts of interest of tens of thousands of unrelated people is very silly.
The conflict of interest is not related to putting musl in Fuschia. The conflict of interest is created by Chandler's employment by Google.
Someone paid by a company should not be making moderation decisions about comments related to that company.
Someone paid by a company should not be making moderation decisions about comments related to that company.
Him moderating is a volunteer position, and he's been doing it forever.
(He's also an llvm foundation board member, etc).
Truthfully, I think the drive by commentary by HN on a community they appear to have about 10 minutes experience dealing with is a much worse look than what chandler did here.
It's funny to watch an entire group of people assume they understand the situations, motives, and community, based on a sample size of "1-2 emails".
If anyone actually bothered to look at the history of his moderation, you'd see he's incredibly consistent in his moderation. In fact, he's publicly called out me before for something similar, despite the fact that I was his boss's boss.
So I don't think it's reasonable to claim what should be happening here when the community itself has not asked for anything different - in fact, it's been quite happy with it.
If you want to become part of that community and then suggest change, by all means, you are more than welcome to join and contribute.
(He's also an llvm foundation board member, etc).
Truthfully, I think the drive by commentary by HN on a community they appear to have about 10 minutes experience dealing with is a much worse look than what chandler did here.
It's funny to watch an entire group of people assume they understand the situations, motives, and community, based on a sample size of "1-2 emails".
If anyone actually bothered to look at the history of his moderation, you'd see he's incredibly consistent in his moderation. In fact, he's publicly called out me before for something similar, despite the fact that I was his boss's boss.
So I don't think it's reasonable to claim what should be happening here when the community itself has not asked for anything different - in fact, it's been quite happy with it.
If you want to become part of that community and then suggest change, by all means, you are more than welcome to join and contribute.
Not just employed by google, but apparently the /lead/ for the team proposing this in the first place!
Felker's comment was that a big corporation might release botched code that contained hacks, but their influence would mean developers would be stuck coding against it.
If that's insulting at all, it's several levels less insulting than the sort of thing I see dang moderating.
If that's insulting at all, it's several levels less insulting than the sort of thing I see dang moderating.
The argument would be that he is dragging in incidents and companies, which could derail the discussion unnecessary. That’s reasonable but this is not:
> I don't think this project poses any risk there, much like Zach points out in his reply. Google is specifically discussing [...]
Now he’s using his moderator rights to push his opinion which unsurprisingly defends his employer.
He should have sent two mails, one moderating and one participating in the discussion.
> I don't think this project poses any risk there, much like Zach points out in his reply. Google is specifically discussing [...]
Now he’s using his moderator rights to push his opinion which unsurprisingly defends his employer.
He should have sent two mails, one moderating and one participating in the discussion.
scarface74(2)
There's a clear conflict of interest: moderating comments critical of the organisation who employs you can easily be considered biased to a disinterested party. It doesn't matter if there is bias or not, the possibility that people will see it as biased is enough for people to recognise it and lost trust. This kind of moderation should be done by a disinterested party, and the Google employee should recuse themselves from addressing it (just refer it to another moderator).
Guess who just finished their Conflict of Interest yearly refresher!
Guess who just finished their Conflict of Interest yearly refresher!
But the criticism had nothing to do with "petty personal attacks", rather it is just tone policing mildly blunt language.
Copy-pasting a comment from "cup" on lobsters, because I think it's on-point:
>1000 times yes. If Google wants their own libc that’s their business. But LLVM should not be part of their “manifest destiny”. The corporatization of OSS is a scary prospect, and should be called out loud and clear like this every time it’s attempted.
https://lobste.rs/s/dgmoyi/response_libc_llvm#c_hg8dvg
>1000 times yes. If Google wants their own libc that’s their business. But LLVM should not be part of their “manifest destiny”. The corporatization of OSS is a scary prospect, and should be called out loud and clear like this every time it’s attempted.
https://lobste.rs/s/dgmoyi/response_libc_llvm#c_hg8dvg
There has been huge corporate involvement and investment in LLVM from its earliest days as anything beyond a research project.
When Apple open-sourced Clang, or their C++ standard library implementation and contributed them to the LLVM project, was that something to be afraid of as well? It seems to have gone pretty well.
When Apple open-sourced Clang, or their C++ standard library implementation and contributed them to the LLVM project, was that something to be afraid of as well? It seems to have gone pretty well.
Fair point, but Google is not Apple. Google has a history of railroading standards to suit their needs, to the point where the standard becomes "what Google is doing". See the Chrome project.
There are plenty of examples on Apple's side, too. To name a few: Floppy disks. CDROM. USB. Firewire. WebKit.
Uh... Firewire.
Unfortunately, the good deeds does not protect from bad consequences.
Google has shown little regard to standards and community, specially as of recent, based on their handling of the musl libc in fuchsia, libc++ in Android, Go the language and it's community, and the elephant in the room: Google Chrome. Google has a bad culture of forking things and breaking things for good business reasons which does not align with the community and people outside Google.
The concerns around Google fucking things up with their influence is not unreasonable.
Google has shown little regard to standards and community, specially as of recent, based on their handling of the musl libc in fuchsia, libc++ in Android, Go the language and it's community, and the elephant in the room: Google Chrome. Google has a bad culture of forking things and breaking things for good business reasons which does not align with the community and people outside Google.
The concerns around Google fucking things up with their influence is not unreasonable.
I'm only a casual observer of LLVM, and have really only used tools such as clang-format.
That said, I haven't observed Apple applying a heavy hand in regards to LLVM/Clang development.
As a C++ developer, clang has been a god send, amd not just for Clang users. For a long while, they had the best error messages around. Actually truly pointing you to precisely where the error is and giving meaningful errors. For a long fucking time, C++ compilers gave really verbose, shitty error messages that took a whole other level of understanding to decipher, which really hindered novices from progressing. Clang put pressure on GCC and Visual Studio to up their game and produce useful errors during compilation.
It was bad enough at one point, I used to present interview candidates with example code that failed to compile and the compiler output and ask the candidatea to fix the code (it was always a trivial mistake in the code involving templates, bit it'd generate a few screen fulls of errors). It wasnt my sole judge of whether I'd recommend them or not, but it was a good indicator of how well they understood the C++ environment and how to decipher compiler errors (sample program was less than 20 lines, but generated at least 3 screen full of errors due to template spewage).
That said, I haven't observed Apple applying a heavy hand in regards to LLVM/Clang development.
As a C++ developer, clang has been a god send, amd not just for Clang users. For a long while, they had the best error messages around. Actually truly pointing you to precisely where the error is and giving meaningful errors. For a long fucking time, C++ compilers gave really verbose, shitty error messages that took a whole other level of understanding to decipher, which really hindered novices from progressing. Clang put pressure on GCC and Visual Studio to up their game and produce useful errors during compilation.
It was bad enough at one point, I used to present interview candidates with example code that failed to compile and the compiler output and ask the candidatea to fix the code (it was always a trivial mistake in the code involving templates, bit it'd generate a few screen fulls of errors). It wasnt my sole judge of whether I'd recommend them or not, but it was a good indicator of how well they understood the C++ environment and how to decipher compiler errors (sample program was less than 20 lines, but generated at least 3 screen full of errors due to template spewage).
Just so we're completely clear, Google has been an incredibly long-term contributor to basically all parts of LLVM. So have sooooo many other companies. This is a really cool project IMO, but it doesn't represent a significant change in who or how much anyone is contributing to LLVM... It's a drop in the bucket.
And for the record, LLVM is doing fine with all these corporate contributors.
I'm not trying to say that this can't be a concern for OSS communities and projects, but LLVM seems to have a quite successful and sustainable model here.
And for the record, LLVM is doing fine with all these corporate contributors.
I'm not trying to say that this can't be a concern for OSS communities and projects, but LLVM seems to have a quite successful and sustainable model here.
The problem is that neither the original thread, nor the threads here (which seem to be quite heavy on Google commenters) answers the question 'why?'.
I think this wouldn't have caused nearly as much commotion as it does now, if the original post explained the technical reasons for starting a new libc. What problems do existing implementations have? How could a new libc address these issues?
Instead, this has an air of (regardless whether this is true): we are going to write a new libc for completely opaque company-driven reasons, and either you accept it or you we will make it happen anyway.
I think this wouldn't have caused nearly as much commotion as it does now, if the original post explained the technical reasons for starting a new libc. What problems do existing implementations have? How could a new libc address these issues?
Instead, this has an air of (regardless whether this is true): we are going to write a new libc for completely opaque company-driven reasons, and either you accept it or you we will make it happen anyway.
I think its primarily because Chris Lattner[0] is at Google now. I wonder even if they pull out this effort how much of that would still be maintained if Lattner changes jobs.
[0]: https://en.wikipedia.org/wiki/Chris_Lattner
[0]: https://en.wikipedia.org/wiki/Chris_Lattner
Chris Lattner said that he has nothing to do with this project
http://lists.llvm.org/pipermail/llvm-dev/2019-June/133274.ht...
http://lists.llvm.org/pipermail/llvm-dev/2019-June/133274.ht...
FWIW Roland McGrath, the original glibc author, also works at Google. But of course not every possible person will be involved here, no matter how relevant.
No. Chris is working in the AI/ML space, and specifically commented in this thread that he is not involved with this project.
Filed under "This is why we need the GPL"
How does the GPL help with this? Google could build their own libc replacement, and if they push it hard enough to gain widespread community support, it could become the (or at least, a) new standard. At that point Google is free to do whatever they want with it as long as they make it useful enough that the community won't abandon it. The GPL just means that it could be forked later, but if Google is still putting significant resources into developing the main branch then no-one will switch to the forks.
GPL is literally made to allow forks like these, what exactly did you mean with your post?!
This. I definitely sounds like they want a libc that they are free to fork and add proprietary incompatibilities to, which they can do with the GNU libc (under LGPL) as long as they don't want to distribute it, so we can assume they want to distribute it and that this is a further closing of their consumer tech like android. I would've though musl would be a better starting point for them.
> A corporate-led project does not have to answer to the community, and will leave whatever bugs they introduce in place for the sake of bug-compatibility with their own software rather than fixing them.
This speaks volumes very clearly. This highlights an immense hazard. Enterprise scale companies contributing to open-source is a fantastic thing, but enterprise scale companies thrusting their own proprietary libraries onto the open-source world is not. I'm already actively avoiding becoming beholden to Google in my work as it is already, let alone in the world where important software uses a libc written by Google.
If you're not concerned by this, refer to the immense power that Google already wields over the extremely ubiquitous web-standards through the market dominance that Chrome has.
This speaks volumes very clearly. This highlights an immense hazard. Enterprise scale companies contributing to open-source is a fantastic thing, but enterprise scale companies thrusting their own proprietary libraries onto the open-source world is not. I'm already actively avoiding becoming beholden to Google in my work as it is already, let alone in the world where important software uses a libc written by Google.
If you're not concerned by this, refer to the immense power that Google already wields over the extremely ubiquitous web-standards through the market dominance that Chrome has.
(Disclosure, Googler here, not working on projects related to any of the things being discussed, speaking in a purely personal capacity.)
I hate to disagree with Rich Felker. I think most people have a lot of respect for musl and the amount of effort it took. But, I disagree with the basic conclusions.
For one thing, I don't think it's fair to compare this to Chrome, an end user product. libc is a fundamental piece of architecture, like (edit: as in, fundamental in stacks that they are used in, to clarify) protobufs, gRPC, Kubernetes, etc. I heavily sympathize with folks uneasy about the Chrome situation - like I've mentioned maybe hundreds of times now I use and recommend Firefox personally, but I feel this particular discussion will become more productive when that stops being a focal point.
Their other points are not really related, but I disagree with them also. I don't think there's a good motivation to keep bugs for backwards compatibility: Google runs non-Google software, too; keeping subtle incompatibilities could be a disaster. Also, Google has a pretty good corpus of code to test their libc on, so I think they have a unique opportunity to fast-track a pretty battle-tested libc (I did not know of this project before this Hacker News post, so this is purely a hot take, but for now I stand by it.)
No idea what their plan is, but open governance would probably work wonders here. After all, LLVM has plenty of corporate contributors, and it seems to do fine.
I hate to disagree with Rich Felker. I think most people have a lot of respect for musl and the amount of effort it took. But, I disagree with the basic conclusions.
For one thing, I don't think it's fair to compare this to Chrome, an end user product. libc is a fundamental piece of architecture, like (edit: as in, fundamental in stacks that they are used in, to clarify) protobufs, gRPC, Kubernetes, etc. I heavily sympathize with folks uneasy about the Chrome situation - like I've mentioned maybe hundreds of times now I use and recommend Firefox personally, but I feel this particular discussion will become more productive when that stops being a focal point.
Their other points are not really related, but I disagree with them also. I don't think there's a good motivation to keep bugs for backwards compatibility: Google runs non-Google software, too; keeping subtle incompatibilities could be a disaster. Also, Google has a pretty good corpus of code to test their libc on, so I think they have a unique opportunity to fast-track a pretty battle-tested libc (I did not know of this project before this Hacker News post, so this is purely a hot take, but for now I stand by it.)
No idea what their plan is, but open governance would probably work wonders here. After all, LLVM has plenty of corporate contributors, and it seems to do fine.
The problem was stated in Felker's response: "...corporate development teams are uniquely qualified to utterly
botch a libc, yet still push it into widespread use..."
This can come innocently as well. For a simple non-code example: google has a code style guide. For C++ it says "don't use exceptions" and even (or used to?) explains that the writers consider exceptions to be a good thing but they have enough legacy / third party code base that doesn't use them that they can't switch. Lots of people use the google style guide because it exists, rather than write one themselves, or rather than spend time hashing one out. So they end up shackling themselves to a constraint entirely resulting from an environment they are not in.
I see this indirectly in your own comment, where you casually referred to gRPC or protobufs as "fundamental piece[s] of architecture". Well they aren't standards, they are technologies that google likes. Protobufs is a perfect example of a technology in, well not widespread use, but is used by a bunch of projects, and as discussed elsewhere contains a number of decisions that reflect the google environment, probably not resulting in a great fit for a number of places where it is used.
There's nothing malign in that; the whole point of writing code is to meet your needs. But when you're an 800 pound gorilla, don't presume your every step is what everyone else needs. They are what you need and without proactive effort may not help many others....but can crowd out what could.
This can come innocently as well. For a simple non-code example: google has a code style guide. For C++ it says "don't use exceptions" and even (or used to?) explains that the writers consider exceptions to be a good thing but they have enough legacy / third party code base that doesn't use them that they can't switch. Lots of people use the google style guide because it exists, rather than write one themselves, or rather than spend time hashing one out. So they end up shackling themselves to a constraint entirely resulting from an environment they are not in.
I see this indirectly in your own comment, where you casually referred to gRPC or protobufs as "fundamental piece[s] of architecture". Well they aren't standards, they are technologies that google likes. Protobufs is a perfect example of a technology in, well not widespread use, but is used by a bunch of projects, and as discussed elsewhere contains a number of decisions that reflect the google environment, probably not resulting in a great fit for a number of places where it is used.
There's nothing malign in that; the whole point of writing code is to meet your needs. But when you're an 800 pound gorilla, don't presume your every step is what everyone else needs. They are what you need and without proactive effort may not help many others....but can crowd out what could.
>I see this indirectly in your own comment, where you casually referred to gRPC or protobufs as "fundamental piece[s] of architecture". Well they aren't standards, they are technologies that google likes. Protobufs is a perfect example of a technology in, well not widespread use, but is used by a bunch of projects, and as discussed elsewhere contains a number of decisions that reflect the google environment, probably not resulting in a great fit for a number of places where it is used.
I am not suggesting they're fundamental to the world, rather just fundamental pieces of architecture where they're being used at. That is to say, fundamental in a similar (though not identical) way to a libc would be. This is more of a miscommunication than a disagreement I think.
>There's nothing malign in that; the whole point of writing code is to meet your needs. But when you're an 800 pound gorilla, don't presume your every step is what everyone else needs. They are what you need and without proactive effort may not help many others....but can crowd out what could.
So, I'm not in the know, but... To quote the original post:
> So, what do you think about incorporating this new libc under the LLVM project?
That would, to me, imply under LLVM governance, not Google, and I think that's the primary thing people should focus on if they are worried. Kubernetes is a good example of where I think this ended up going extremely well; Kubernetes is under control of CNCF and not Google.
Moving governance elsewhere is surely beneficial for the project's health, and since it's a low level piece of architecture rather than an end user product, I see no reason why there wouldn't be motivation to do that. It appears it is the intent to do that in the first place.
I am not suggesting they're fundamental to the world, rather just fundamental pieces of architecture where they're being used at. That is to say, fundamental in a similar (though not identical) way to a libc would be. This is more of a miscommunication than a disagreement I think.
>There's nothing malign in that; the whole point of writing code is to meet your needs. But when you're an 800 pound gorilla, don't presume your every step is what everyone else needs. They are what you need and without proactive effort may not help many others....but can crowd out what could.
So, I'm not in the know, but... To quote the original post:
> So, what do you think about incorporating this new libc under the LLVM project?
That would, to me, imply under LLVM governance, not Google, and I think that's the primary thing people should focus on if they are worried. Kubernetes is a good example of where I think this ended up going extremely well; Kubernetes is under control of CNCF and not Google.
Moving governance elsewhere is surely beneficial for the project's health, and since it's a low level piece of architecture rather than an end user product, I see no reason why there wouldn't be motivation to do that. It appears it is the intent to do that in the first place.
Hi, full disclosure, it's my team working on this at Google FWIW, and the technical lead for much of Google's contributions to LLVM.
When we asked the community if they were interested in us developing this in the open as part of LLVM, we meant as part of LLVM. We're very committed to the LLVM open source project, and think it would be great for this to be developed within that framework (technical, project, community, the whole works). I'm hopeful that the community is in fact interested.
But we also were asking -- we have some specific technical goals that we'd like to make sure we can accomplish. It really is the community's decision whether this makes sense. =]
When we asked the community if they were interested in us developing this in the open as part of LLVM, we meant as part of LLVM. We're very committed to the LLVM open source project, and think it would be great for this to be developed within that framework (technical, project, community, the whole works). I'm hopeful that the community is in fact interested.
But we also were asking -- we have some specific technical goals that we'd like to make sure we can accomplish. It really is the community's decision whether this makes sense. =]
Hi Chandler!
Is it just me, or the original Googler's posting on llvm-dev does not explain their motivation beyond the vague "doesn't quite address"? I guess some solid list of problems hard to resolve could help.
Also, as you say, it's a community decision now, what will be a plan if community NACKs the proposal? A BoringLibc from scratch? Commitment to contribute hard fixes to glibc and/or mail?
Is it just me, or the original Googler's posting on llvm-dev does not explain their motivation beyond the vague "doesn't quite address"? I guess some solid list of problems hard to resolve could help.
Also, as you say, it's a community decision now, what will be a plan if community NACKs the proposal? A BoringLibc from scratch? Commitment to contribute hard fixes to glibc and/or mail?
I think it's hard to predict what a community will be interested in deep-diving to understand. And I'm not sure that HN is super representative of its interests either. The email was aimed at LLVM folks, not HN.
I generally trust the LLVM community to ask for any details they need to reasonably evaluate a proposal like this, and I also trust the folks on my team to work to address those requests to the extent we can.
I don't think it makes sense to try and speculate about what option will make the most sense if LLVM says "nope". Generally, I plan to encourage the team to see if there is a good way for us to address concerns the LLVM community has while still getting the technical things we need. IMO, it would be somewhat surprising if there were no reasonable path where this could both be reasonable for the LLVM community and Google. Doesn't mean it is impossible, but having detailed and precise plans don't seem like a priority. IMO, the priority is finding a good way to work with the LLVM community here.
On a more meta level, I also think it would be good for lots of folks (HN, twitter, etc.) to be a bit less harsh in their criticism of initial posts proposing new efforts/projects. I've seen this several times recently (ranging from this to the V language stuff). I'd suggest folks maybe ask questions and give people a chance to flesh out their thoughts and provide missing context rather than hammering in feedback. In many cases, I think the feedback is actually good, but the method of delivery makes it much harder for people to learn from and respond to constructively.
Anyways, enough meta...
I generally trust the LLVM community to ask for any details they need to reasonably evaluate a proposal like this, and I also trust the folks on my team to work to address those requests to the extent we can.
I don't think it makes sense to try and speculate about what option will make the most sense if LLVM says "nope". Generally, I plan to encourage the team to see if there is a good way for us to address concerns the LLVM community has while still getting the technical things we need. IMO, it would be somewhat surprising if there were no reasonable path where this could both be reasonable for the LLVM community and Google. Doesn't mean it is impossible, but having detailed and precise plans don't seem like a priority. IMO, the priority is finding a good way to work with the LLVM community here.
On a more meta level, I also think it would be good for lots of folks (HN, twitter, etc.) to be a bit less harsh in their criticism of initial posts proposing new efforts/projects. I've seen this several times recently (ranging from this to the V language stuff). I'd suggest folks maybe ask questions and give people a chance to flesh out their thoughts and provide missing context rather than hammering in feedback. In many cases, I think the feedback is actually good, but the method of delivery makes it much harder for people to learn from and respond to constructively.
Anyways, enough meta...
> I've seen [harsh criticism] several times recently (ranging from this to the V language stuff).
It's probably not a good idea to put yourself on one level with a scammer.
Seriously though, any effort made by Google will of course be scrutinized much more vigorously than the same effort made by a smaller org or a single person. If I decide to build my own libc, no one will care because they probably don't need to care. But big companies like Google have the power and inertia to just shove decisions in people's faces. I think for most people here, a large part of the dayjob is spent dealin with and working around the stupid decisions of third-party hardware/software vendors.
It's probably not a good idea to put yourself on one level with a scammer.
Seriously though, any effort made by Google will of course be scrutinized much more vigorously than the same effort made by a smaller org or a single person. If I decide to build my own libc, no one will care because they probably don't need to care. But big companies like Google have the power and inertia to just shove decisions in people's faces. I think for most people here, a large part of the dayjob is spent dealin with and working around the stupid decisions of third-party hardware/software vendors.
There are also many example of Google doing just that to various degrees when it comes to open source projects, security research, their own products, dealing with vendors, customers or employment among other things. This of course isn't something unique to Google. What is unique however is their position.
They are probably the most powerful company when it comes computers, software and the Internet. They have the culture, means and goodwill to do a lot more damage than almost anyone else. I am sure that is at part of why people want to work for Google. Not to do damage of course, but because you can make a difference. Unfortunately making a difference for Google frequently isn't the best for rest of us.
They are probably the most powerful company when it comes computers, software and the Internet. They have the culture, means and goodwill to do a lot more damage than almost anyone else. I am sure that is at part of why people want to work for Google. Not to do damage of course, but because you can make a difference. Unfortunately making a difference for Google frequently isn't the best for rest of us.
Snark aside, Google's desire to own the de facto non-GPL libc is worthy of a discussion in the broader community. Your expressed desire to limit the discussion and unwillingness to directly address the issues raised by Rich Felker speaks volumes.
- Google doesn’t claim to want to own it, at least not anywhere that I’ve seen written. They just want to design it, for their needs.
- Nowhere do they say it should be the “defacto” libc. If anything, Clang and libc++ remain not defacto. Glibc would likely continue to be the preferred libc in my personal opinion. Nothing wrong with that.
- Nowhere do they say it should be the “defacto” libc. If anything, Clang and libc++ remain not defacto. Glibc would likely continue to be the preferred libc in my personal opinion. Nothing wrong with that.
My concern is that if Google's libc becomes part of the LLVM package, it will push out Clang and musl and any others. It should be a separate package like the others.
I assume you mean glibc instead of Clang here.
Well, I think personally making it part of LLVM works to the community’s advantage. If Google is the sole owner and maintainer, external forces will have less impact, much to the detriment of the library quality (imo.) As a part of LLVM, there are many huge players with interest in improving the library and many maintainers externally.
Google and other big companies already contribute to LLVM, so I don’t think the status quo is changing at all.
Well, I think personally making it part of LLVM works to the community’s advantage. If Google is the sole owner and maintainer, external forces will have less impact, much to the detriment of the library quality (imo.) As a part of LLVM, there are many huge players with interest in improving the library and many maintainers externally.
Google and other big companies already contribute to LLVM, so I don’t think the status quo is changing at all.
>And I'm not sure that HN is super representative of its interests either. The email was aimed at LLVM folks, not HN.
>On a more meta level, I also think it would be good for lots of folks (HN, twitter, etc.) to be a bit less harsh in their criticism of initial posts proposing new efforts/projects.
I'm sure it's super annoying when folks in the larger community with vested interests in the continuing success of projects like LLVM start noticing harsh criticism coming from credible experts like Rich Felker. =]
And then post it to HN or tweet it out to others so the concern spreads. =]
But all feedback is good feedback, right? And anyways, I'm sure the folks at LLVM will let Google have their way in the end because there's no other foreseeable outcome of this discussion, is there? At least, none worth discussing. =]
>On a more meta level, I also think it would be good for lots of folks (HN, twitter, etc.) to be a bit less harsh in their criticism of initial posts proposing new efforts/projects.
I'm sure it's super annoying when folks in the larger community with vested interests in the continuing success of projects like LLVM start noticing harsh criticism coming from credible experts like Rich Felker. =]
And then post it to HN or tweet it out to others so the concern spreads. =]
But all feedback is good feedback, right? And anyways, I'm sure the folks at LLVM will let Google have their way in the end because there's no other foreseeable outcome of this discussion, is there? At least, none worth discussing. =]
Rich Felker was being civil. You are not being civil. In the spirit of civility and honestly just adhering to Hacker News community guidelines, please reconsider your tone. I believe this kind of behavior is damaging to open source, and mocking the person you are replying to reflects pretty poorly on yourself.
I agree that I was out of line, which is why I posted a more civil reply (I hope) afterwards, since at that point I wasn't able to delete it.
"...corporate development teams are uniquely qualified to utterly botch a libc, yet still push it into widespread use..."
That doesn't seem to be uniquely corporate, there are lots of botched, poorly implemented, or designed, community projects not backed or funded by corporations that have come into widespread use.
There are also lots of attempts by corporations to force adoption of some piece of open source software only to fail, look no further than Polymer for example, or NaCL/PNaCL for that matter.
Sounds to me like what they're proposing is a simplified subset of libc, since the exact justifications for what's being supported and what's being left out haven't been discussed, it really seems premature to dismiss it out of hand. There could be plenty of valid reasons for doing it.
Look at OpenGL, a hideously complex spec that is onerous to implement for full compatibility and compliance. John Carmack forked it to create mini-GL drivers that were useful for games. Eventually this became MCD, and eventually the largest most cash flush vendors funded full ICD drivers. Carmack was correct to fork OpenGL for consumer hardware to support games as the fully OpenGL was really incompatible with market needs and backwards compatibility with the full spec was needed by a tiny minority, but held back drivers for millions of consumers.
And now, decades after that decision, we have Vulkan, because honestly, the original OpenGL design needed to be replaced.
Perhaps libc is reaching that point where it is suffering under technical debt?
That doesn't seem to be uniquely corporate, there are lots of botched, poorly implemented, or designed, community projects not backed or funded by corporations that have come into widespread use.
There are also lots of attempts by corporations to force adoption of some piece of open source software only to fail, look no further than Polymer for example, or NaCL/PNaCL for that matter.
Sounds to me like what they're proposing is a simplified subset of libc, since the exact justifications for what's being supported and what's being left out haven't been discussed, it really seems premature to dismiss it out of hand. There could be plenty of valid reasons for doing it.
Look at OpenGL, a hideously complex spec that is onerous to implement for full compatibility and compliance. John Carmack forked it to create mini-GL drivers that were useful for games. Eventually this became MCD, and eventually the largest most cash flush vendors funded full ICD drivers. Carmack was correct to fork OpenGL for consumer hardware to support games as the fully OpenGL was really incompatible with market needs and backwards compatibility with the full spec was needed by a tiny minority, but held back drivers for millions of consumers.
And now, decades after that decision, we have Vulkan, because honestly, the original OpenGL design needed to be replaced.
Perhaps libc is reaching that point where it is suffering under technical debt?
> ... there are lots of botched ...
Yes, but have they been "push(ed) into widespread use ..."? That was the point Rich was trying to make, I think.
Yes, but have they been "push(ed) into widespread use ..."? That was the point Rich was trying to make, I think.
Yes. Does OpenSSL ring any bells? How about PHP and its ecosystem? How about Sendmail? Or GNOME.
There have been OSS projects 'push'ed into widespread use by the popularity of a few strong personalities and herd mentality, but have questionable design. As someone with a long history of Perl, I'd lump the entire Perl language and ecosystem into that category, along with CGI and FastCGI which existed far longer than they should have before someone put their foot down and made a better open-source application interface.
Developer adoption is by no means a process by which 'he who has the most money wins', there's plenty of examples of big failures to thrust commercial designs into the public space, and there are plenty examples of shitty OSS designs getting uptake, I'm sure if you ask Linus, he could go on a long rant.
As I mentioned with Carmack's mini-GL/MCD model, new forks often get traction if they solve very real customer problems. In Carmack's case, he was developing on NeXT Hardware, for PCs, and PC consumer cards didn't run OpenGL, but often ran either DirectX or a proprietary driver API. If you are in the video game market, multiplatform portability is a given, so forking OpenGL to contain only the high performance subset for games that could run on consumer hardware was so obvious in retrospect, and solved a real program for game developers, who quickly latched onto it, and for 2nd and 3rd tier 3D accelerator vendors who couldn't get devs to use their proprietary API but could benefit from the easy to implement mini-GL.
I have no clue why they want to fork or subset libc, but before dismissing it out of hand, maybe there's a real compelling reason for it, and if there isn't, it'll have as much shelf life as proprietary Unix forks.
But using ad hominem critique in absence of a concrete proposal seems out of place to me.
There have been OSS projects 'push'ed into widespread use by the popularity of a few strong personalities and herd mentality, but have questionable design. As someone with a long history of Perl, I'd lump the entire Perl language and ecosystem into that category, along with CGI and FastCGI which existed far longer than they should have before someone put their foot down and made a better open-source application interface.
Developer adoption is by no means a process by which 'he who has the most money wins', there's plenty of examples of big failures to thrust commercial designs into the public space, and there are plenty examples of shitty OSS designs getting uptake, I'm sure if you ask Linus, he could go on a long rant.
As I mentioned with Carmack's mini-GL/MCD model, new forks often get traction if they solve very real customer problems. In Carmack's case, he was developing on NeXT Hardware, for PCs, and PC consumer cards didn't run OpenGL, but often ran either DirectX or a proprietary driver API. If you are in the video game market, multiplatform portability is a given, so forking OpenGL to contain only the high performance subset for games that could run on consumer hardware was so obvious in retrospect, and solved a real program for game developers, who quickly latched onto it, and for 2nd and 3rd tier 3D accelerator vendors who couldn't get devs to use their proprietary API but could benefit from the easy to implement mini-GL.
I have no clue why they want to fork or subset libc, but before dismissing it out of hand, maybe there's a real compelling reason for it, and if there isn't, it'll have as much shelf life as proprietary Unix forks.
But using ad hominem critique in absence of a concrete proposal seems out of place to me.
Its not really ad hominem, Android itself us a massive pile of hacks (zygote process, project treble, binder, etc) that while it worked for google in fixing their issues it is far from ideal. Also what API has replaced FCGI and CGI in general? Hopefully not in process modules.
What's replaced it is running services or application servers that contain an embedded HTTP handler, and putting a load balancer in front of it.
Also, rather than guilt by association (Android et al, which was done under different circumstances compared with the hundreds of other libraries Google has released), why not actually hear out the concrete proposal and then criticize it?
Saying no one in category X can ever build anything good is just illogical.
Also, rather than guilt by association (Android et al, which was done under different circumstances compared with the hundreds of other libraries Google has released), why not actually hear out the concrete proposal and then criticize it?
Saying no one in category X can ever build anything good is just illogical.
NDK still feels like it.
'getting uptake' is not the same as being 'pushed'. In fact, the former sounds a lot like 'pull'.
Have you used OpenSSL? It sucks. The API returns strange and inconsistent error codes. Error reporting was terrible. Strange non orthogonalities abound. It is a lesson on how not to design an API. The only reason it got traction was it was first and no one bothered to do better.
systemd?
pulseaudio?
Bad examples. Lennart benefited from Redhat backing his work.
Great examples. What happens if Google botches up LLVM? They're far more powerful than Redhat.
They were bad examples of non-enterprise backed software being pushed into widespread use.
They are great examples of poorly-designed replacements for existing open source software being pushed into widespread adoption solely because the largest Enterprise distribution of Linux willed it.
And also great examples of software developers lying about their intentions early on to silence criticism of their design and the subsequent ramifications of adopting their solutions.
Multiply that by 100 when comparing Google's influence with Redhat's.
If Google wants to develop their own libc, they can. I'm f they want to make it open source, they can. There's nothing about doing those things that necessitates doing so as part of the LLVM project.
And also great examples of software developers lying about their intentions early on to silence criticism of their design and the subsequent ramifications of adopting their solutions.
Multiply that by 100 when comparing Google's influence with Redhat's.
If Google wants to develop their own libc, they can. I'm f they want to make it open source, they can. There's nothing about doing those things that necessitates doing so as part of the LLVM project.
Seems to me that trying to upstream is common courtesy before forking. If there is some reason it needs to be integrated with changes to LLVM itself then you might end up with an LLVM fork similar to WebKit/Blink, is that a better outcome?
The point is that no existing libc is part of the LLVM project; i.e., under that umbrella. What's different about Google wanting their own libc and following what Clang and musl do currently?
> Also, Google has a pretty good corpus of code to test their libc on
I left Google more than 5 years ago but at the time the subset of the language (C++) one was allowed to use was quite limited. Testing your implementation of libc on that codebase could still lead to poor coverage.
I left Google more than 5 years ago but at the time the subset of the language (C++) one was allowed to use was quite limited. Testing your implementation of libc on that codebase could still lead to poor coverage.
> Google has a pretty good corpus of code to test their libc on...
That is a good point, but I'm in no way disputing that Google could do a great job of creating their own libc. I would never be foolish enough to challenge the merit of Google's engineers, the proof of this is clear in the tasting of the pudding that is Google's software. My concerns lie in the open-source community becoming further beholden to Google, or even worse with Google dictating the direction of development on what could become a cornerstone of the architecture of many critical pieces of software.
That is a good point, but I'm in no way disputing that Google could do a great job of creating their own libc. I would never be foolish enough to challenge the merit of Google's engineers, the proof of this is clear in the tasting of the pudding that is Google's software. My concerns lie in the open-source community becoming further beholden to Google, or even worse with Google dictating the direction of development on what could become a cornerstone of the architecture of many critical pieces of software.
(as stated elsewhere, this effort is in my team at Google)
FWIW, while I think libc is super important, and I'm really excited about this project and the opportunity it represents, I think the LLVM project and community is much larger than libc. =D I don't think you need to worry about this effort changing anything about how LLVM operates. And we're currently just asking if they're interested. =]
FWIW, while I think libc is super important, and I'm really excited about this project and the opportunity it represents, I think the LLVM project and community is much larger than libc. =D I don't think you need to worry about this effort changing anything about how LLVM operates. And we're currently just asking if they're interested. =]
What people are afraid of isn't Google having a negative effect by lack of motivation or technical competence, but the opposite. That Google will be very successful in doing what Google think is important while not taking anyone else's merit, interest or concerns seriously.
Being excited and not seeing a problem isn't so much reassuring as what people are afraid of. If you do end up being wrong and Google does gain too much influence from the perspective of someone outside of Google not only would you probably be getting a promotion but also be in a better position with more influence yourself since you are a part of Google.
So unless you are specifically convinced that what people are concerned about won't happen, which would be pretty hard to argue unless this activity differs substantially from others at Google, your comment doesn't really say much. As it doesn't speak to anyone else's interests.
Being excited and not seeing a problem isn't so much reassuring as what people are afraid of. If you do end up being wrong and Google does gain too much influence from the perspective of someone outside of Google not only would you probably be getting a promotion but also be in a better position with more influence yourself since you are a part of Google.
So unless you are specifically convinced that what people are concerned about won't happen, which would be pretty hard to argue unless this activity differs substantially from others at Google, your comment doesn't really say much. As it doesn't speak to anyone else's interests.
Google has been a contributor to LLVM for about 10 years, and a major contributor for most of that time.
At no point have they tried to force something into LLVM that goes against the wishes of the community. They have always worked with the LLVM community to improve LLVM, and LLVM has benefited greatly from Google's contributions. I expect this project to be no different.
At no point have they tried to force something into LLVM that goes against the wishes of the community. They have always worked with the LLVM community to improve LLVM, and LLVM has benefited greatly from Google's contributions. I expect this project to be no different.
In the beginning of Google's letter they let us understand they are going to create simplified version for their own needs. It does mean they don't care about compatibility and bugs, if it doesn't affect their software. That's not how this kind of libraries should be implemented.
> Chrome, an end user product. libc is a fundamental piece of architecture
The web browsers is not the end user products anymore. The web sites(and these days, smartphone apps which is just a wrapper of the browser) are the end user products. The web browser is, just like libc, fundamental piece of architecture.
Sure, the users directly interact with web browsers GUI, but average users can't recognize the difference between the Browser, Internet, Google and Android. They are all same things to most of them.
In that sense, browser and libc are the same, all users use it without noticing it.
The web browsers is not the end user products anymore. The web sites(and these days, smartphone apps which is just a wrapper of the browser) are the end user products. The web browser is, just like libc, fundamental piece of architecture.
Sure, the users directly interact with web browsers GUI, but average users can't recognize the difference between the Browser, Internet, Google and Android. They are all same things to most of them.
In that sense, browser and libc are the same, all users use it without noticing it.
The browser is a architectural in the electron case where it is shipped as part of a product, a whole browser is a product by itself.
Same way as LCD screens are achitectural pieces but a television is a product.
Same way as LCD screens are achitectural pieces but a television is a product.
I think trying to work with protobufs by analogy here fails because the scopes of power are so vastly different.
I'm not making a comment dressed as a question, I'm genuinely curious. Is this super different from the linux kernel policy that you don't break user space? Both in intent and in effect.
In this scenario "userspace" refers to Google's code, and "breaking" it would be necessary to fix it for everyone else.
Quite relevant then: "Why I'm donating $150/month (10% of my income) to the musl libc project"[1].
[1] https://andrewkelley.me/post/why-donating-to-musl-libc-proje...
[1] https://andrewkelley.me/post/why-donating-to-musl-libc-proje...
This would have been a much better email without the needless stone throwing from one libc maintainer's glass house.
I'm surprised the proposed initial scope would include x86-64 but not ARMv7 or AArch64. That suggests Google's motivation here is their server software, not mobile devices running Android or Fuchsia.
Also, portability and consistency is an important attribute of a libc. Supporting multiple architectures from the beginning would help shape a portable design. For example, the original target processor for Windows NT development was the Intel i860 specially because it wasn't the standard i386 architecture. Fun fact: the Intel i860 was codenamed "N10" aka "N-Ten", which may have been the origin of the Windows "NT" moniker.
Also, portability and consistency is an important attribute of a libc. Supporting multiple architectures from the beginning would help shape a portable design. For example, the original target processor for Windows NT development was the Intel i860 specially because it wasn't the standard i386 architecture. Fun fact: the Intel i860 was codenamed "N10" aka "N-Ten", which may have been the origin of the Windows "NT" moniker.
> mobile devices running Android
They already have Bionic!
> or Fuchsia
They already have a weird fork of musl that Rich described in the email!
typical Google "Not Invented in this Team" approach..
They already have Bionic!
> or Fuchsia
They already have a weird fork of musl that Rich described in the email!
typical Google "Not Invented in this Team" approach..
Bionic was a very hard child birth.
Check NDK roadmap to see how much years (yes years) has taken them between Android and NDK releases to provide a proper libc.
Check NDK roadmap to see how much years (yes years) has taken them between Android and NDK releases to provide a proper libc.
[deleted]
Google are going to make it with or without LLVM's cooperation.
If it's done with LLVM, it's highly probable that LLVM won't let them extend,extinguish libc. Google's code will be carefully evaluated by a relatively neutral party: LLVM.
If Google does it alone, they will write whatever code they want. They'll be free to extend, extinguish libc.
If it's done with LLVM, it's highly probable that LLVM won't let them extend,extinguish libc. Google's code will be carefully evaluated by a relatively neutral party: LLVM.
If Google does it alone, they will write whatever code they want. They'll be free to extend, extinguish libc.
Why do you think Google has the ability to EEE libc by themselves but somehow be voluntarily reigned in by LLVM?
> Some folks have told me that I should press LLVM to make musl the "LLVM libc" instead of whatever Google wants to do, but that misses the point: there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class".
If they want to create a great libc - or fuck up another libc - they're much welcome to do that, but they need to go do it outside of LLVM.
Dumping a libc on LLVM is just (1.) appropriating credibility from LLVM, (2.) sticking others with the maintenance bill, and (3.) horseshit.