Apple curl security incident 12604(daniel.haxx.se)
daniel.haxx.se
Apple curl security incident 12604
https://daniel.haxx.se/blog/2024/03/08/the-apple-curl-security-incident-12604/
70 comments
Never attribute to stupidity that which could be explained by malice (especially if the malicious actor uses Hanlon's razor in defense).
Minor correction: it falls back to Apples CA store if the initial check fails. So wouldn’t add more computation for nothing.
However I do agree that it’s not the expected result and thus bad behaviour.
Given how Apple generally favour backwards incompatible changes and that this feature is added into curl, this does feel like there is more to this story than Apple is letting on. Perhaps it’s used for developer diagnostic tools or AppStore validation?
However I do agree that it’s not the expected result and thus bad behaviour.
Given how Apple generally favour backwards incompatible changes and that this feature is added into curl, this does feel like there is more to this story than Apple is letting on. Perhaps it’s used for developer diagnostic tools or AppStore validation?
This is the kind of behaviour (corporate and software) that I would anticipate if some government has ordered them to secretly install it as a back door — a government-controlled item in the CA store (which isn't necessarily obvious, I think it just means at least one private key got copied), a requirement that all first-party software must use that store, and a requirement not to admit that this is a problem.
> Minor correction: it falls back to Apples CA store if the initial check fails. So wouldn’t add more computation for nothing.
Not GP but that is extra computation? It should just fail at that point, instead it's doing computation. According to the result, it's then either 'for nothing', or in order to give the incorrect result.
Not GP but that is extra computation? It should just fail at that point, instead it's doing computation. According to the result, it's then either 'for nothing', or in order to give the incorrect result.
I disagree, but please read my comment through because if you take the first part of the comment alone then it might read as defending Apple, when I'm really not.
That's a failure scenario though. If your application is failing a CA check against an authority you've pinned then your infra and application is already broken in ways that the extra computation is meaningless. It is your certificates or application that then need correcting.
In terms of computational overhead (and only computational overhead!), where it matters is the happy path because That's where the overhead of additional computation would affect the performance for your operational infra and thus your users (and we are talking about this specific scenario and not some more generalised discussion about TLS). If you follow the happy path then Apples change, as serious as it is, doesn't affect the behaviour or your application.
I'm not defending Apple here. Whether intended as one or not, this change is a literal backdoor. And it needs to be framed as such. Arguing it as a computation overhead is, at best, missing the significance of this change in behaviour.
So the real problem isn't computational overhead, it's the question of what signing authorities are given permissions to write certificates for your pinned endpoints that wouldn't / shouldn't have had under curl's default code and expected behaviour. ie who now has a backdoor?
That's a failure scenario though. If your application is failing a CA check against an authority you've pinned then your infra and application is already broken in ways that the extra computation is meaningless. It is your certificates or application that then need correcting.
In terms of computational overhead (and only computational overhead!), where it matters is the happy path because That's where the overhead of additional computation would affect the performance for your operational infra and thus your users (and we are talking about this specific scenario and not some more generalised discussion about TLS). If you follow the happy path then Apples change, as serious as it is, doesn't affect the behaviour or your application.
I'm not defending Apple here. Whether intended as one or not, this change is a literal backdoor. And it needs to be framed as such. Arguing it as a computation overhead is, at best, missing the significance of this change in behaviour.
So the real problem isn't computational overhead, it's the question of what signing authorities are given permissions to write certificates for your pinned endpoints that wouldn't / shouldn't have had under curl's default code and expected behaviour. ie who now has a backdoor?
I did read your whole comment, and I didn't really take it as 'defending Apple', at least not in the sort of blind fanboyism way that implies.
If I understand correctly, you're assuming that the user is in control of both running the curl command, and the host it's pointed at? And thus the behaviour in this case is, you argue, arbitrary because you shouldn't let it get there anyway?
I was just thinking of it as the former, say I want to curl -X POST a comment to secretnews.ycombinator.com (not real, as far as I know!) and verify that someone hasn't MITM'd it so that I accidentally disclose my comment on news.ycombinator.com. I don't control either domain, I'm just running a curl command.
If I understand correctly, you're assuming that the user is in control of both running the curl command, and the host it's pointed at? And thus the behaviour in this case is, you argue, arbitrary because you shouldn't let it get there anyway?
I was just thinking of it as the former, say I want to curl -X POST a comment to secretnews.ycombinator.com (not real, as far as I know!) and verify that someone hasn't MITM'd it so that I accidentally disclose my comment on news.ycombinator.com. I don't control either domain, I'm just running a curl command.
I’m saying calling this a problem because it might add a computational overhead is missing the severity of the problem.
This is a problem because it creates a backdoor.
I also disagree it creates a computation overhead because it’s only engaged outside of normal operation (ie when your application is already broken or if someone is intentionally exploiting that backdoor). but the computation argument is more of a distraction from the real problem.
So definitely not defending Apple. I’m saying it’s more serious than just a computation problem.
This is a problem because it creates a backdoor.
I also disagree it creates a computation overhead because it’s only engaged outside of normal operation (ie when your application is already broken or if someone is intentionally exploiting that backdoor). but the computation argument is more of a distraction from the real problem.
So definitely not defending Apple. I’m saying it’s more serious than just a computation problem.
Hm, ok. I don't think they did miss the severity though, at least by my reading of:
> That means that this "feature" by Apple either adds more computation for nothing, or breaks the verification model that is expected.
they're saying one of two things happens: you get the wrong result (more severe than amount of computation) or you (because the cert it does have is not in Apple's store either) get the correct result, but wasted cycles doing something that you didn't want it to do and risked getting the wrong result.
> That means that this "feature" by Apple either adds more computation for nothing, or breaks the verification model that is expected.
they're saying one of two things happens: you get the wrong result (more severe than amount of computation) or you (because the cert it does have is not in Apple's store either) get the correct result, but wasted cycles doing something that you didn't want it to do and risked getting the wrong result.
You, not they. The “they” in this context is the author of curl. And Daniel is fully aware of the problems this change makes.
Whereas you’re the one barking on about computational overhead.
Whereas you’re the one barking on about computational overhead.
Unfortunately, this kind of behavior, where the Apple policies always override whatever the "owner" of an Apple device wants to accomplish, is not surprising but it is what should always be expected from Apple.
This is also what keeps me from them. The devices seem really well-built and the OS polished, but with their lock-in to their own ways and own ecosystem (can't get access to your own hardware), it's just not compatible with being a hacker in the HN sense of the word, or at least not for that part of your digital life, and it's not like you'd want to buy it as a device on the side given those prices. I keep doubting about trying out Apple products, the latest one being their AR headset, but so far they've always seemed too hostile towards developers/tinkerers
Apple is the owner, you are the user ;)
Unfortunately, there’s always someone who is more than willing to read far too much into a low-level technical decision if it serves to, in their eyes, ‘prove’ their preconceived worldview.
A company as large as Apple making this particular decision in service of a wider vision to “own user devices, is attributing to Apple a degree of organisational and coordination prowess that is well beyond unheard of in any organisation even one tenth of Apple’s size.
But it’s Apple, right!?
A company as large as Apple making this particular decision in service of a wider vision to “own user devices, is attributing to Apple a degree of organisational and coordination prowess that is well beyond unheard of in any organisation even one tenth of Apple’s size.
But it’s Apple, right!?
It would be a lot easier to agree with you if Apple didn't go out of their way in software, hardware and firmware to block third-parties from getting a fair shake. This is the company that stops you from repairing your own device using software that you cannot control - it's not a lack of organizational and coordination prowess that makes these things happen. Nobody cakewalks their way into manufacturing things on the Chinese mainland.
Similarly, some intern at Apple did not authorize this change, write the patch, and then defend the decision when upstream questioned them about it. That requires coordination, which is frankly why this whole brouhaha is offensive. Apple could be working with upstream to close a critical vulnerability, but instead they're defending the merits of a faulty and dangerous fork. That is brazen and irresponsible behavior that requires just as much effort as working together with the community to solve the issue. Even companies a tenth Apple's size can get this right.
Similarly, some intern at Apple did not authorize this change, write the patch, and then defend the decision when upstream questioned them about it. That requires coordination, which is frankly why this whole brouhaha is offensive. Apple could be working with upstream to close a critical vulnerability, but instead they're defending the merits of a faulty and dangerous fork. That is brazen and irresponsible behavior that requires just as much effort as working together with the community to solve the issue. Even companies a tenth Apple's size can get this right.
Run open source curl. You now have the desired behavior.
Presumably apple are setting this[0]? Emphasis mine.
CURLSSLOPT_NATIVE_CA
Tell libcurl to use the operating system's native CA store for certificate verification. If you set this option and *also set a CA certificate file* or directory then during verification those certificates are searched in addition to the native CA store.
No, that's not the case. `curl` the binary knows how to call `libcurl` the library.
Right. I'm more suggesting that this combination of options should be handled differently inside of libcurl. E.g. either you want the native store for validation, or you don't. If you "also set a CA certificate file" as per the libcurl docs, presumably you want to use that cert only, and no others.
I'm not too familiar with how the library works internally, but I was imagining something like the above would be possible.
I'm not too familiar with how the library works internally, but I was imagining something like the above would be possible.
It isn't (lib)curl that's doing this, it's Apple's own version of LibreSSL that curl is built against on macOS.
There's nothing curl can do about it.
There's nothing curl can do about it.
This is a backdoor.
I'm not saying it's intentional, or malicious. But as a matter of fact it is indeed a backdoor.
If you start adding keys to your users authentication scheme, then you've added a backdoor.
I'm not saying it's intentional, or malicious. But as a matter of fact it is indeed a backdoor.
If you start adding keys to your users authentication scheme, then you've added a backdoor.
Seems like exactly the sort of thing that the NSA pressures large American tech companies to do.
Also the sort of issue which the large American tech companies conspicuously dismiss as unimportant.
As much as the default behaviour is questionable, I don't fully agree with the assessment: this is in fact a curl documentation problem.
As a multi-protocol lib, curl does not in fact implement all of them directly, instead relying on transitive dependencies which act as "backends" to outsource, in most cases, decoding the low-level bits. For some of the protocols, and for good reason, it incorporates support for multiple alternative libraries.
The downside of that approach is that it is hard, if not impossible, to ensure common behaviour across independent backends which may not cover all of the same features, provide incomplete APIs, or, such as in this case do not provide a way to patch some of its default behaviour. Libressl is not a bit-for-bit reimplementation of openssl, and are under no obligation to mimic its API fully (however convenient that would be for us all).
In such cases, and given the reluctance of upstream to fix it, curl is left with 2 possibilities: drop support for said library, or document the quirk. Given the downside of breaking user code of following the former, it should at least do the latter.
All this being said, I do agree with the main sentiment that this is a flawed approach from a security standpoint of libressl, and there's probably reason to open a CVE, but it should be on libressl.
As a multi-protocol lib, curl does not in fact implement all of them directly, instead relying on transitive dependencies which act as "backends" to outsource, in most cases, decoding the low-level bits. For some of the protocols, and for good reason, it incorporates support for multiple alternative libraries.
The downside of that approach is that it is hard, if not impossible, to ensure common behaviour across independent backends which may not cover all of the same features, provide incomplete APIs, or, such as in this case do not provide a way to patch some of its default behaviour. Libressl is not a bit-for-bit reimplementation of openssl, and are under no obligation to mimic its API fully (however convenient that would be for us all).
In such cases, and given the reluctance of upstream to fix it, curl is left with 2 possibilities: drop support for said library, or document the quirk. Given the downside of breaking user code of following the former, it should at least do the latter.
All this being said, I do agree with the main sentiment that this is a flawed approach from a security standpoint of libressl, and there's probably reason to open a CVE, but it should be on libressl.
The thing I'm not quite getting is how building curl from sources apparently gets around the issue, if the root cause is in the system ssl library?
The user who submitted the bug report was using Homebrew, which does not build curl with the system version of OpenSSL: https://github.com/Homebrew/homebrew-core/blob/547d6bb9b289c...
Reminds me of the SQLite F_BARRIERFSYNC debacle.
They just don't care.
https://bonsaidb.io/blog/acid-on-apple/
They just don't care.
https://bonsaidb.io/blog/acid-on-apple/
I remember a while back when they broke one of the ancient Unix tools they actually ship by hacking in some option badly. Their attitude to maintainership is like two levels below a Debian guy breaking OpenSSL.
If Daniel says your curl is broken, you fix it, Apple. It’s that simple.
Daniel can be wrong. He can even be wrong about curl.
He's probably correct here, as far as I can determine in 2 minutes, but "he's correct because it's Daniel" is the worst possible argument.
This reminds me of this old exchange about the history of C (specifically, Eric S. Raymond's "contribution" to it): https://scienceblogs.com/deltoid/2011/10/14/dennis-ritchie-h...
He's probably correct here, as far as I can determine in 2 minutes, but "he's correct because it's Daniel" is the worst possible argument.
This reminds me of this old exchange about the history of C (specifically, Eric S. Raymond's "contribution" to it): https://scienceblogs.com/deltoid/2011/10/14/dennis-ritchie-h...
I think the counter-argument is that "curl" is a project maintained by Daniel. If you are going to ship a product that doesn't behave in the same way you should call it something else. So Daniel is right about what "curl" should do, but Apple can be right about what "applecurl" which is very similar to curl should do.
I think it should warrant more dialogue than that first line support shut down though, even if the Apple team's gut reaction is to disagree.
Not 'Daniel' specifically, but in general when you're packaging a tool and its author/maintainer(s) contact you about it.
Not 'Daniel' specifically, but in general when you're packaging a tool and its author/maintainer(s) contact you about it.
He’s been maintaining an essencial part of the computing infrastructure for 25 years and doing a great job at it.
A fair coin can land on heads 20 times in a row, but the odds are not great.
A fair coin can land on heads 20 times in a row, but the odds are not great.
I think parent doesn't mean Daniel is never wrong even without proof but if one side claims a bug is a bug and the other says it's not than you can pretty sure it's a bug if Daniel considers it as one.
It's about the evaluation of a finding.
It's about the evaluation of a finding.
Thanks for the warning! FWIW, I'm using MacPorts to override many of the tools that come bundled with macOS (curl being one of them), because they tend to be out of date or otherwise broken. I've long since lost trust in Apple's bundled software.
I wonder if Apple relies on that behavior for something important...
Not sure if you're being snarky, but obviously this is very very important.
It'd be entirely reasonable for someone to create a script that is meant to only use the internal private CA. When you run this command then you KNOW that you are ONLY talking to an internal corporate resource, because this is an internal corporate CA.
And then apple adds a backdoor the size of domain validation, to this.
Hell, it'd be entirely valid to use a dummy name, and just rely on the fact that it's signed by the corp CA, to establish that you are connecting to a corporate server. Now Apple's added a security vulnerability to that, by breaking this extremely reasonable assumption.
Not cool.
It'd be entirely reasonable for someone to create a script that is meant to only use the internal private CA. When you run this command then you KNOW that you are ONLY talking to an internal corporate resource, because this is an internal corporate CA.
And then apple adds a backdoor the size of domain validation, to this.
Hell, it'd be entirely valid to use a dummy name, and just rely on the fact that it's signed by the corp CA, to establish that you are connecting to a corporate server. Now Apple's added a security vulnerability to that, by breaking this extremely reasonable assumption.
Not cool.
So much about Apple caring for the security of its users.
> Obviously I think differently.
:)
:)
A default behaviour is not the same thing as a fallback behaviour. Seems that Apple's security team has a reading comprehension problem.
I guess it's a general MIT license problem. Somebody, be it Google, Apple or whoever else, can deliberately break your program and distribute it under the original name. And although you kinda own the name you really don't.
Wouldn't the same problem happen with most other (if not all?) FOSS licenses?
It's not. You can do the same with GPL-licensed code. You can use trademark law in both cases to prevent people from distributing modified versions, like Firefox did for years with Debian
https://en.wikipedia.org/wiki/Debian–Mozilla_trademark_dispu...
https://en.wikipedia.org/wiki/Debian–Mozilla_trademark_dispu...
Would it be theoretically possible to counter this with a trademark? And then say (in legalese) "if you're just packaging it then I grant you permission to use the trademark, but if you fork it (or otherwise substantially modify it) then permission is revoked"
It's not that simple.
https://curl.se/docs/thename.html
https://curl.se/docs/thename.html
[deleted]
threeseed(3)
That means that this "feature" by Apple either adds more computation for nothing, or breaks the verification model that is expected.
Neither of these are an expected result.