Linux developers patch security holes faster than anyone else, says Google(zdnet.com)
zdnet.com
Linux developers patch security holes faster than anyone else, says Google
https://www.zdnet.com/article/google-project-zero-finds-linux-developers-patch-security-holes-faster-than-anyone-else/
60 comments
It's very simple. You assume your favorite OS is written by more skillful developers or steered by more skillful architects/tech-lead or overseen by more skillful team leads.
Any of these are wrong.
Propriety systems are closed and obfuscate their flaws.
Any of these are wrong.
Propriety systems are closed and obfuscate their flaws.
I don't think we'll ever get a total picture on the first two items from Windows or MacOS, so that's difficult to compare.
What's the issue with Linux's architecture? Hybrid kernels aren't shown to be any more secure than a monokernel to my knowledge. There is something to be said that we don't make kernels for device drivers, which are basically their own operating systems, but that's common to all kernels.
What's the issue with Linux's architecture? Hybrid kernels aren't shown to be any more secure than a monokernel to my knowledge. There is something to be said that we don't make kernels for device drivers, which are basically their own operating systems, but that's common to all kernels.
What architectural/impact differences does Linux have vs others?
See e.g. https://madaidans-insecurities.github.io/linux.html for a detailed discussion.
There's some very good points in there, but (4) is unfair. It's true that there's no boundary between a sudoer and root in Linux, but there's also no boundary between an Administrator and SYSTEM in Windows. UAC, even in the "secure" AlwaysNotify mode which uses the secure desktop, has countless unpatched bypasses[1].
Also, (3) should raise some eyebrows for readers paying attention. Cool, Microsoft removed font parsing from the kernel, how wise of them. Wait a second, why was font parsing in the kernel to begin with? With win32k.sys, it shouldn't be surprising that Microsoft has to do more legwork to bring the attack surface back down to the level of other OSes. They're also exploring the use of eBPF in the Windows kernel too[2].
[1]: https://github.com/hfiref0x/UACME [2]: https://github.com/Microsoft/ebpf-for-windows
Also, (3) should raise some eyebrows for readers paying attention. Cool, Microsoft removed font parsing from the kernel, how wise of them. Wait a second, why was font parsing in the kernel to begin with? With win32k.sys, it shouldn't be surprising that Microsoft has to do more legwork to bring the attack surface back down to the level of other OSes. They're also exploring the use of eBPF in the Windows kernel too[2].
[1]: https://github.com/hfiref0x/UACME [2]: https://github.com/Microsoft/ebpf-for-windows
In a world where bugs are certain, unpatched (because a bug is known but a patch hasn’t been released) e.g. iOS is nominally not more secure than patched Linux, unless possibly if you are a nationstate actor hoarding zerodays. The speed of patching bugs is important because most targets aren’t big enough to be on the receiving end of a zero day. Though I do agree that this doesn’t really say anything about whether one system is more or less secure than others from a design standpoint, it certainly matters from a practical standpoint.
If you're not concerned about zero days, then why does it matter how long project zero knows about the bug before it's patched?
[deleted]
In practice, once it is known that a bug exists, it’s a race against time until someone discovers it.
This is an example of why "average" is not a good statistic for long-tailed distributions. What does it mean that Apple took 69 days, or Microsoft 83? Does that mean that 90% of the bugs were fixed in 2 weeks, but 10% took months (or years)? Or were half fixed in a month, but the remainder took 3 months or more?
A much more useful number would be the median. When were half the bugs fixed? And it would be nice to know about the slowest 10%.
A much more useful number would be the median. When were half the bugs fixed? And it would be nice to know about the slowest 10%.
It’s not in the linked article but there’s significant more information from the source:
https://googleprojectzero.blogspot.com/2022/02/a-walk-throug...
https://googleprojectzero.blogspot.com/2022/02/a-walk-throug...
I think you want to look at cumulative time. If, hypothetically, linux fixes one bug in one day for 365 days, and windows fixes one bug in ten days one time, is linux 10x better?
The problem with any of these measures is that we do not know how severe the bugs are.
But you are certainly correct, average (or median) time does not tell us how many bugs were fixed.
But you are certainly correct, average (or median) time does not tell us how many bugs were fixed.
The Linux kernel developers don't release security fixes directly to end users, unlike e.g. Apple and Microsoft.
Perhaps the Linux numbers should also include the integration/QA lead time of the distros if it's meant to be a competition.
Perhaps the Linux numbers should also include the integration/QA lead time of the distros if it's meant to be a competition.
You make a good point here -- the time-to-deployment is significant as well as the time-to-resolve.
It's also true that once a fix is available in any Linux source tree, advanced users can begin to use it from that moment onwards.
Arguably some users could develop their own workaround or solution even before one is available upstream, should they want to (because the code and tools to compile it are all available to them).
(similarly it'd be valuable to analyze how long it takes for the majority of end-users to upgrade across common operating systems. something like this chart[1] of Firefox version traffic history, but for security fix rollout)
[1] - https://web.archive.org/web/20201220010043/https://paulcalva...
It's also true that once a fix is available in any Linux source tree, advanced users can begin to use it from that moment onwards.
Arguably some users could develop their own workaround or solution even before one is available upstream, should they want to (because the code and tools to compile it are all available to them).
(similarly it'd be valuable to analyze how long it takes for the majority of end-users to upgrade across common operating systems. something like this chart[1] of Firefox version traffic history, but for security fix rollout)
[1] - https://web.archive.org/web/20201220010043/https://paulcalva...
Sure - there is definitely a real benefit for advanced users of the Linux kernel.
Epecially if your organization has someone on the Linux kernel security team, I guess.
Epecially if your organization has someone on the Linux kernel security team, I guess.
This can also be a downside where the fix reveals the vulnerability and it begins to be exploited before distros get fixes out.
Iirc there recently was a big one where the fix was snuck into a large Linus patch and not revealed until all distros had a chance to release updated kernels.
Iirc there recently was a big one where the fix was snuck into a large Linus patch and not revealed until all distros had a chance to release updated kernels.
Tech companies are run by product managers these days and fixing a bug is not a product priority if the buggy software is still making money
One of many reasons I’m a fan of SAFe (Scaled Agile Framework). There are 2 backlogs, one run by product and one run by dev/ops/arch/sec. Over the course of a PI (about 1 quarter), capacity planning usually allocated 70/30 between the two backlogs.
This allows your tech staff to prioritize things in their 30 without having to make a business case for people who don’t intimately understand why it’s important. It’s beautiful.
They are called the Program and Solution backlogs.
https://www.scaledagileframework.com/program-and-solution-ba...
This allows your tech staff to prioritize things in their 30 without having to make a business case for people who don’t intimately understand why it’s important. It’s beautiful.
They are called the Program and Solution backlogs.
https://www.scaledagileframework.com/program-and-solution-ba...
My favorite part of the SAFe is the image you get right on the homepage:
https://www.scaledagileframework.com/
It has a section "agile release train" which is a literal train! Everybody loves trains. It also has an icon with the text "Built-in quality", so you can sleep soundly knowing that you will not have quality issues anymore.
/s of course, but that image is really something. It's so vague it can be anything you want it to; a middle managers' dream.
https://www.scaledagileframework.com/
It has a section "agile release train" which is a literal train! Everybody loves trains. It also has an icon with the text "Built-in quality", so you can sleep soundly knowing that you will not have quality issues anymore.
/s of course, but that image is really something. It's so vague it can be anything you want it to; a middle managers' dream.
Don’t get me wrong, that image is dumb. I don’t know why they insist on promoting it so much.
Whenever I teach a class or tell people about SAFe I always let them know 2 things:
1. All of the principles are sound and amounts to a smooth combination of best practices working together. Many companies are doing a lot of what’s involved because of that.
2. There are a lot of buzzwords. You can just assume you’re putting “lean-agile” in front of anything like they do with “quantum” in movies. It’s exhausting if you don’t let yourself chuckle. Because of it, reading the documentation can be confusing if you haven’t taken a class.
Whenever I teach a class or tell people about SAFe I always let them know 2 things:
1. All of the principles are sound and amounts to a smooth combination of best practices working together. Many companies are doing a lot of what’s involved because of that.
2. There are a lot of buzzwords. You can just assume you’re putting “lean-agile” in front of anything like they do with “quantum” in movies. It’s exhausting if you don’t let yourself chuckle. Because of it, reading the documentation can be confusing if you haven’t taken a class.
Well, I found this money quote on release train:
" The more alignment you have, the more autonomy you can grant. The one enables the other.
© Scaled Agile, Inc. Include this copyright notice with the copied content.
Read the FAQs on how to use SAFe content and trademarks here: https://www.scaledagile.com/about/about-us/permissions-faq/ Explore Training at: https://www.scaledagile.com/training/calendar/ "
Note that I copied only the quote. But you can see this quote is eminently copyrightable. Hence the copyright notice.
" The more alignment you have, the more autonomy you can grant. The one enables the other.
© Scaled Agile, Inc. Include this copyright notice with the copied content.
Read the FAQs on how to use SAFe content and trademarks here: https://www.scaledagile.com/about/about-us/permissions-faq/ Explore Training at: https://www.scaledagile.com/training/calendar/ "
Note that I copied only the quote. But you can see this quote is eminently copyrightable. Hence the copyright notice.
I think it’s include for anything copied.
That image (and the rest of the gallery) screams "expensive low quality" with all its strength. I have never seen anything that uses an image like that to be worth one's time.
And yes, that style is fashionable on some circles. I usually prefer to avoid those circles.
And yes, that style is fashionable on some circles. I usually prefer to avoid those circles.
> There are 2 backlogs, one run by product and one run by dev/ops/arch/sec. Over the course of a PI (about 1 quarter), capacity planning usually allocated 70/30 between the two backlogs.
Does "run by" mean that dev has to justify their projects to product and vice versa? Or is this a "this is our area, we'll take care of it thank you" kind of thing?
In my team at AWS we have ~5 backlogs each planning cycle, which includes areas like 'operational risk reduction,' 'operational burden reduction,' and 'adoption'. What makes it work in our case is that both dev and product jointly own all the backlogs, and have to come to a single recommendation on allocation they put forward to the service GM. Dev has to defend projects they want to do in the more engineering focused backlogs, while product has to do the same for the feature oriented backlog. Over time, allocations have shifted significantly depending on where the service is and what customers are asking for. What I like about this is that it forces everybody to get along and to keep their eye on what's actually important (the customer). This does require that both product and dev are experienced enough to see the other perspective. Product will have to understand that it's not in the customer's interest if dev has a super high burden, while dev will have to accept that a perfect solution is usually worse than a good enough one.
Does "run by" mean that dev has to justify their projects to product and vice versa? Or is this a "this is our area, we'll take care of it thank you" kind of thing?
In my team at AWS we have ~5 backlogs each planning cycle, which includes areas like 'operational risk reduction,' 'operational burden reduction,' and 'adoption'. What makes it work in our case is that both dev and product jointly own all the backlogs, and have to come to a single recommendation on allocation they put forward to the service GM. Dev has to defend projects they want to do in the more engineering focused backlogs, while product has to do the same for the feature oriented backlog. Over time, allocations have shifted significantly depending on where the service is and what customers are asking for. What I like about this is that it forces everybody to get along and to keep their eye on what's actually important (the customer). This does require that both product and dev are experienced enough to see the other perspective. Product will have to understand that it's not in the customer's interest if dev has a super high burden, while dev will have to accept that a perfect solution is usually worse than a good enough one.
The tech backlog just has to be justified among the technical contributors. Product is not involved at all.
The only place where there may be some crossover is “architectural runway” which works to ensure that your future goals are possible on the system that you have. This could mean that if you know you’re going to have a need to scale a system based on a near-future feature request from product that you put the work into the tech backlog to get things ready in advance.
The only place where there may be some crossover is “architectural runway” which works to ensure that your future goals are possible on the system that you have. This could mean that if you know you’re going to have a need to scale a system based on a near-future feature request from product that you put the work into the tech backlog to get things ready in advance.
Sounds horrible. Product justified git [version] tagging should be done with names and not semver/other. It was very simple: "Marketing pitches it better so more sells". Threw devs and DevOps into the pits of hell, adding unnecessary cost to many tasks.
Interesting insight. I probably never was part of a real by the book SAFe environment then.
In all SAFe projects I took part never was there a second backlog. Or if there was one (once) the reasoning was:
"Let'sput everything from tech in there. We will revisit it once we have done the important/business-relevant stuff."
My take away message is that no matter what, business either cares for security or it doesn't. No framework will help against the priorities of product/management if they are aligned against security.
In all SAFe projects I took part never was there a second backlog. Or if there was one (once) the reasoning was:
"Let'sput everything from tech in there. We will revisit it once we have done the important/business-relevant stuff."
My take away message is that no matter what, business either cares for security or it doesn't. No framework will help against the priorities of product/management if they are aligned against security.
Your description is really why it’s so important. The two backlogs combined with PI planning that lets developers lay out the best course of action based on the priorities they’ve been handed is core to the entire approach.
Developers are expected to have a lot more control in SAFe.
Like anything else though, it boils down to leadership understanding that giving up total control is more effective for the company. Usually technical leaders understand that.
Developers are expected to have a lot more control in SAFe.
Like anything else though, it boils down to leadership understanding that giving up total control is more effective for the company. Usually technical leaders understand that.
A company and managers not respecting refactor is a culture problem, having two backlogs that are weighted differently is just a workaround to the culture problem. That culture usually isn't from managers being ignorant of the value of refactor, it's that incentives for them (bonuses, etc) are aligned against the latter backlog. If you fix the incentive, all the other pieces will fall into place.
This sounds pretty smart. It'd sound even better if it was made 50/50.
You can agree to make it whatever balance you want. 70/30 is a solid recommendation to start with though.
I’ve found that in many cases, because the developers understand what needs to be done so clearly in that backlog that even big items go quickly.
The biggest thing is just to make sure that you’re not in a situation of 100% features where everything else is being ignored until it all catches up to you. 70/30 at least forces the issue.
I’ve found that in many cases, because the developers understand what needs to be done so clearly in that backlog that even big items go quickly.
The biggest thing is just to make sure that you’re not in a situation of 100% features where everything else is being ignored until it all catches up to you. 70/30 at least forces the issue.
Where I work we brag about doing SaFe but we don't actually do it :))
Then let’s make it a priority.
We can talk about it in the next sprint, after a story has been created and groomed. Then it’s just a matter of someone taking it off the backlog after they’ve gotten through their current workload and scheduled a meeting with product to make sure expectations are aligned.
We can talk about it in the next sprint, after a story has been created and groomed. Then it’s just a matter of someone taking it off the backlog after they’ve gotten through their current workload and scheduled a meeting with product to make sure expectations are aligned.
This. At most companies Agile = delivering new buggy software on top of old buggy software every few weeks.
I disagree with the "these days" part. Tech companies never fixed bugs. In fact, since I can remember, they have only improved on that (but they still aren't any good).
Yeah: Google employees routinely leave patches for bugs in the PR queue because someone hasn't either update the existing tests (which clearly were adding negative value if they fail on the fixed code) or write new regression tests, as if "the code is already known to be wrong shouldn't trump "maybe someone in the future will break it again".
[deleted]
this runs counter to the trend observed in the article.
[deleted]
Linux is just a kernel. Others provide a complete os i cluding ways to distribute security patches. Apples and oranges.
What does “Linux developers patch a security hole”? Time until the patch is merged into Linus’s tree? Or more reasonably time until a patched kernel is available in a given distro?
Either an MR or a merge into the kernel, yes. But, regardless, the comparison omits open auditing vs a closed one.
With closed systems being able to decide what to disclose and obfuscating their own system, the [power] user is less likely to have actual numbers of anything. Be it how many flaws or how many were discovered internally and placed in the backlog. Then, you throw in some PR requirements into the mix and you'll never have a clear picture of what you're using. Just a sales pitch in a different medium.
This is an important dimension.
With closed systems being able to decide what to disclose and obfuscating their own system, the [power] user is less likely to have actual numbers of anything. Be it how many flaws or how many were discovered internally and placed in the backlog. Then, you throw in some PR requirements into the mix and you'll never have a clear picture of what you're using. Just a sales pitch in a different medium.
This is an important dimension.
Your last question is impossible to answer because distros are all so different.
Which is the point - time until patched in the kernel source tree is a relatively useless metric unless it’s pretty close to when those fixes ship to Ubuntu, Debian, CentOS, Amazon Linux - whatever the majority of Linux is these days.
Otherwise a fix in Linus’ tree is about as useful as a committed fix to the Windows source repository.
Otherwise a fix in Linus’ tree is about as useful as a committed fix to the Windows source repository.
>Otherwise a fix in Linus’ tree is about as useful as a committed fix to the Windows source repository.
I don't see why. Users can manually update their kernels in this case. I don't believe that's an option you'll get with a proprietary OS.
I don't see why. Users can manually update their kernels in this case. I don't believe that's an option you'll get with a proprietary OS.
It's a bit more useful, but the breakage has to be seriously impacting my use of the system to roll my own kernel outside of the distro's packaging (the only one I've done is manually applying a fix to a Gentoo kernel and even then I just throw the patch on after the normal stuff).
The *BSDs are nonexistent in the raw data [1].
Of course less code leads to less security holes. But 0/376? Or is project zero excluding BSDs entirely?
[1] https://bugs.chromium.org/p/project-zero/issues/list?sort=id...
Of course less code leads to less security holes. But 0/376? Or is project zero excluding BSDs entirely?
[1] https://bugs.chromium.org/p/project-zero/issues/list?sort=id...
I don't believe Chrome provides support for BSD, which explains why it's missing.
How? I referenced the underlying data. They have Microsoft kernel bugs. Zoom bugs. Even systemd. What am I missing? I think they coincidentally happen to have their bug tracker hosted on chromium’s site.
[deleted]
The last time I saw such a comparison HP was in the lead with over 200 days to fix. This time Microsoft is in the lead with 90. HP doesn't seem to produce SW anymore. Or Google's project zero team doesn't care about them.
HP spun its software out in HP Software Division and sold the whole thing to everyones favorite legacy software company: Micro Focus.
Microsoft has improved dramatically compared to the days of Windows 95. Apple, on the other hand, is mired in complacency.
I'm always really disappointed every time I hear an account from security researchers about how difficult it is to deal with Apple. I realize that Apple cannot possibly be alone in this, but there's something especially frustrating about a company that aggressively denies choice to its users, cites security as a major benefit of that, and then ignores critical feedback about vulnerabilities.
If you don't admit to vulnerabilities they don't exist, right?
Just one example, reneging on bug bounties is mind-blowingly stupid for a $3T company. Security researchers will sell what they find to the likes of NSO Pegasus instead of Apple. Also, the bad blood thus created makes them less likely to follow "responsible disclosure" and instead just disclose them as soon as discovered for kudos.
Doesn't the data in that blog show Apple improving faster than MicroSoft? Going from taking longer than to fix bugs in 2019 to being faster than them now?
It's pretty impressive that everyone seems to have reduced their time to fix bugs. Linux is doing an amazing job there.
It's pretty impressive that everyone seems to have reduced their time to fix bugs. Linux is doing an amazing job there.
I don't see how the Linux developers' higher speed of fixing security bugs would in any way substantiate the author's claim that Linux is not "less secure than proprietary systems". He completely leaves aside
- the number of security bugs per unit time,
- the severity of the bugs in question, and
- the overall system architecture and the potential impact of security bugs in general. Needless to say, this is where the situation on Linux looks pretty grim compared to current "proprietary systems" (which has nothing to do with the fact that they're proprietary).