Multiple Bugs in OpenBSD Kernel(marc.info)
marc.info
Multiple Bugs in OpenBSD Kernel
http://marc.info/?l=oss-security&m=146853062403622&w=2
68 comments
Compared to other OSes and distros this is one in a million.
More readable archived copy:
http://permalink.gmane.org/gmane.comp.security.oss.general/1...
(I'm not a fan of gmane, but it did a better job with this particular mail than the alternatives.)
(I'm not a fan of gmane, but it did a better job with this particular mail than the alternatives.)
The gmane interface is so bad it beggars belief.
Classic view is better for nested threads:
http://thread.gmane.org/gmane.comp.security.oss.general/1994...
Yes, but at least it bothered to parse the email and display it? The original URL seems to have very nearly just echoed the raw MIME message into an HTML page.
FWIW, I find Gmane much easier to use than Google Groups. It doesn't take ages to asynchronously load via JS and then hijack common browser behavior or make it harder to use the browsers history mechanism to return to where the tab was closed. Gmane's servers are slow, but the interface is much easier to use because it's a simple design.
Fuzzing is not something I have looked at seriously (to be honest it seems like asking clients to take up running before walking) but the outcomes are ... Impressive.
How come? What better way to test your function then giving them random/unexpected input and see how that works out.
Would you not want to try out what happens if I pulled a door you built that is only meant for pushing? There will always be that one person who ignores the sign and tries pulling just like there is always that person who uses your function in an unexpected way.
Would you not want to try out what happens if I pulled a door you built that is only meant for pushing? There will always be that one person who ignores the sign and tries pulling just like there is always that person who uses your function in an unexpected way.
The issue I have with tests, is that I feel like I'm just implementing my algorithm a second time, in a different wording (language/domain), hoping that this will catch some unknown bug(s).
If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much.
Furthermore: if our algorithms require a second "test implementation", shouldn't we also write a test for the test, to confirm that the written test actually tests correctly? What good is a broken test? If you've built a plywood door, and a machine for testing that door, wouldn't it be useful to have a machine that tests that your testing machine actually tests the door properly?
All jokes aside, I really enjoy automated, high-level tests, which mimic actual user behaviour. I think this is different because we're no longer implementing an algorithm a second time around (as a test), but rather encoding user behaviour (of the algorithm) into an algorithm (new code).
If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much.
Furthermore: if our algorithms require a second "test implementation", shouldn't we also write a test for the test, to confirm that the written test actually tests correctly? What good is a broken test? If you've built a plywood door, and a machine for testing that door, wouldn't it be useful to have a machine that tests that your testing machine actually tests the door properly?
All jokes aside, I really enjoy automated, high-level tests, which mimic actual user behaviour. I think this is different because we're no longer implementing an algorithm a second time around (as a test), but rather encoding user behaviour (of the algorithm) into an algorithm (new code).
> If a test suite covers all aspects of an implementation, it's basically a second implementation, and then the question becomes whether the time spent writing this test code would actually have been better spent looking over the actual code that defines your algorithm. If we can't detect errors in code by reading it, I don't think tests will help us much.
The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code.
The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code.
And it if I understand OpenBSD goal correctly, they take any kind of edge they can get to find those missed parts.
Their formula for what is worth essentially boils down to "if this approach can find at least one bug no other approach would do it, we will use it".
Their formula for what is worth essentially boils down to "if this approach can find at least one bug no other approach would do it, we will use it".
> The OpenBSD code has been audited many times by very experienced and careful C experts. Fuzzing apparently found the very few things they hadn't already found by reading the code.
That's definitely comforting, but I guess my point is that the fact that bugs can even hide in plain sight in the first place is the root cause of the problem. If we can look at a specification of what something is supposed to do - the code - and not see that it does something entirely different, then the problem is with the programming/specification language.
The very purpose of a programming language is to make the specification of computer programs readable (rather than reading assembly). If a specification language needs a a computer program to go through the specification and see if it specifies things that we didn't intend to, then that specification language misses the very point of being a language in which to specify things in the first place.
That's definitely comforting, but I guess my point is that the fact that bugs can even hide in plain sight in the first place is the root cause of the problem. If we can look at a specification of what something is supposed to do - the code - and not see that it does something entirely different, then the problem is with the programming/specification language.
The very purpose of a programming language is to make the specification of computer programs readable (rather than reading assembly). If a specification language needs a a computer program to go through the specification and see if it specifies things that we didn't intend to, then that specification language misses the very point of being a language in which to specify things in the first place.
I don't think it's that simple. A lot of bugs are the result of disparate pieces of code -- each of which is fairly easy to verify by reading it -- interacting in an unforeseen way.
I would argue that this effect is not a product of code but of global state. If you have a pure function that transforms some input to an output, all you can do wrong is use the wrong tool for the wrong job, which the type system should, preferably, prevent you from doing. Using the wrong function for the wrong job means we haven't restricted well enough the input the function takes, through clearly defining the types of the arguments, not allowing for meaningless invariants.
If a specification language requires the writer/reader to be aware of the entire specification in order to verify whether an isolated piece of specification is valid, that specification language is not of much value, I would argue.
It occurs to me that the problem with traditional languages is that they do not allow the direct reference to information (values); only to variables which contain values (information). In Haskell, if you want a place to store information you create a TVar (variable), while you are forced to do this, always, in traditional languages, if you want to work with information/values properly. You're forced to keep information in a register if you want to manipulate it, and always refer to the register, when the information it contains is what you're really concerned with. Having to both think about the values/information and where you stored this information is double work. Why not reference the information directly?
If a specification language requires the writer/reader to be aware of the entire specification in order to verify whether an isolated piece of specification is valid, that specification language is not of much value, I would argue.
It occurs to me that the problem with traditional languages is that they do not allow the direct reference to information (values); only to variables which contain values (information). In Haskell, if you want a place to store information you create a TVar (variable), while you are forced to do this, always, in traditional languages, if you want to work with information/values properly. You're forced to keep information in a register if you want to manipulate it, and always refer to the register, when the information it contains is what you're really concerned with. Having to both think about the values/information and where you stored this information is double work. Why not reference the information directly?
I agree that many bugs occur due to state. I disagree that languages, particularly imperative languages (which you seem to be attacking here), are significantly responsible for bugs.
Syntax is not really an issue for any programmer with a reasonable amount of experience. I very rarely have trouble accurately transcribing ideas to code. More often, my ideas are faulty.
Physics engines in games are a perfect example. They're a perfect use case for functional programming, there's not a lot of complex state ideally, and yet even simple simulations are notoriously fraught with unexpected behavior. See: thousands of YouTube videos of physics glitches in games.
Syntax is not really an issue for any programmer with a reasonable amount of experience. I very rarely have trouble accurately transcribing ideas to code. More often, my ideas are faulty.
Physics engines in games are a perfect example. They're a perfect use case for functional programming, there's not a lot of complex state ideally, and yet even simple simulations are notoriously fraught with unexpected behavior. See: thousands of YouTube videos of physics glitches in games.
> Syntax is not really an issue for any programmer with a reasonable amount of experience.
> They're a perfect use case for functional programming, there's not a lot of complex state ideally,
> and yet even simple simulations are notoriously fraught with unexpected behavior. See: thousands
> of YouTube videos of physics glitches in games.
If an implementations of 3D scene rasterization contains glitches, the language used is most definitely the issue, because producing glitch-free 3D animations is a solved problem. No programmer writes a physics engine to produce glitches, so if they magically appear, even though they aren't visible in the spec, the spec language is faulty.
I guess I'm arguing that the problem is that most popular languages allow you to write programs/specs that are invalid (will crash when executed). So we're writing specifications in languages where we can't even say whether the specification is valid or not (actually implementable). If humans produce buggy programs, that is proof that we're using the wrong tool, I'm arguing, because no one intends to produce bugs.
Haskell is a bit scary, at least it was for me in the beginning, because it can seem so hard just to get your program to compile. But that's because the compiler actually verifies that your program is valid - that no unhandled exception will occur. If people are having writing valid, purely functional programs, perhaps this is, in part, because their idea is unimplementable, and Haskell is telling them this via not being able to compile your spec, while other languages inform you of this via an unhandled exception that crops up a year later, after the code is in production.
> No programmer writes a physics engine to produce glitches, so if they magically appear, even though they aren't visible in the spec, the spec language is faulty.
I'm saying that popular physics engines generally have bulletproof, tried-and-tested code, and are even implemented in a very functional way on a conceptual level. Despite this, they exhibit glitches. The glitches don't appear magically out of the spec language, they are an inevitable result of the discrete nature of realtime physics simulations.
This is meant to be a counter-example to your assertion that bugs come from miscommunication between computers and humans. My argument is that more often than not, we communicate our ideas perfectly, but our ideas are flawed. In the case of physics engines, they are flawed by design in order to compromise accuracy for performance.
I'm saying that popular physics engines generally have bulletproof, tried-and-tested code, and are even implemented in a very functional way on a conceptual level. Despite this, they exhibit glitches. The glitches don't appear magically out of the spec language, they are an inevitable result of the discrete nature of realtime physics simulations.
This is meant to be a counter-example to your assertion that bugs come from miscommunication between computers and humans. My argument is that more often than not, we communicate our ideas perfectly, but our ideas are flawed. In the case of physics engines, they are flawed by design in order to compromise accuracy for performance.
> In the case of physics engines, they are flawed by
> design in order to compromise accuracy for performance.
> The glitches don't appear magically out of the spec language,
> they are an inevitable result of the discrete nature of
> realtime physics simulations.
> the compiler actually verifies that your program is valid - that no unhandled exception will occur
That's not actually correct. The compiler verifies (modulo your transitive use of unsafe primitives) that certain things won't occur. Unhandled exceptions are not one of those things.
That's not actually correct. The compiler verifies (modulo your transitive use of unsafe primitives) that certain things won't occur. Unhandled exceptions are not one of those things.
That depends on how you define an unhandled exception :)
You're right of course: no compiler can guarantee lack of exceptions when it comes to IO, but I would argue that GHC, indeed, does verify that - unless a function explicitly throws an exception - no exception will occur. This is very different from C, where an exception can be the result of just about any operation (if you do something wrong), rather than only an effect that appears when you use "throwIO" (or similar). It's the difference between exceptions being used as a tool by the user (programmer), rather than a way for the compiler to tell the user (after your program is compiled and is running) that it has no idea what to do now, and will abort. In Haskell I write the "error"/"throwIO" statement causing an abort, in a C program I might have intended something completely different, but the compiler throws one at runtime.
We mustn't mix up IO with everything else, because reality is inherently unreliable, and it's just the nature of it that a certain file won't necessarily be readable tomorrow - we can never know that. But we can know whether a pure function, sans compiler bugs, will execute without throwing an exception. And I would argue that adding "error" or "fail" does not constitute an unhandled exception, as that statement was put there intentionally by the author of the program (perhaps to test out something).
You're right of course: no compiler can guarantee lack of exceptions when it comes to IO, but I would argue that GHC, indeed, does verify that - unless a function explicitly throws an exception - no exception will occur. This is very different from C, where an exception can be the result of just about any operation (if you do something wrong), rather than only an effect that appears when you use "throwIO" (or similar). It's the difference between exceptions being used as a tool by the user (programmer), rather than a way for the compiler to tell the user (after your program is compiled and is running) that it has no idea what to do now, and will abort. In Haskell I write the "error"/"throwIO" statement causing an abort, in a C program I might have intended something completely different, but the compiler throws one at runtime.
We mustn't mix up IO with everything else, because reality is inherently unreliable, and it's just the nature of it that a certain file won't necessarily be readable tomorrow - we can never know that. But we can know whether a pure function, sans compiler bugs, will execute without throwing an exception. And I would argue that adding "error" or "fail" does not constitute an unhandled exception, as that statement was put there intentionally by the author of the program (perhaps to test out something).
I think the difference is that the test shows what the idea you (as the architect) have is, while the code shows how you (as the developer) implemented it.
And then you run the tests and they will, hopefully, tell you if the how matches the what, and if it doesn't where the two differ, i.e. where the the code doesn't do what you intended...
And then you run the tests and they will, hopefully, tell you if the how matches the what, and if it doesn't where the two differ, i.e. where the the code doesn't do what you intended...
I took it more as there are fundamentals that could be focused on first. If you know your door is a piece of plywood leaning against the opening, yout time may be better spent getting a real door before you start testing all the ways plywood can fail.
If you aren't in the habit of using debris for structural items, sure, spend some time testing failure modes. :)
If you aren't in the habit of using debris for structural items, sure, spend some time testing failure modes. :)
Exactly - these days table stakes is having a CI test suite with unit tests with good coverage, good integration tests (with the attendant "so I need four servers plus a couple to fire the tests ... Oh") and wait what you want UI testing ? Oh.
Isn't fuzzing responsible for most CVEs the last couple of years?
Google have been blasting holes in Flash and the Windows kernel for quite a while now, by virtue of having a huge number of machines performing effective fuzzing.
See http://googleprojectzero.blogspot.co.uk/ for some neat writeups
See http://googleprojectzero.blogspot.co.uk/ for some neat writeups
A couple of years ago, I listened to a Podcast on the general topic of malware, vulnerabilities and so forth (mainly malware and bot nets, though), and the guest speaker mentioned that apparently Microsoft, too, had set up a number of machines (I think he said something like a couple of hundred) to do nothing but fuzzing on a large body of Microsoft software (including, I guess, stuff like Office, Exchange, SQL Server, Internet Explorer / Edge, ...).
I do not know if this is true (but it makes sense), and I have no clue how many bugs were discovered that way. But I would not be surprised if Microsoft quietly fixed those. The descriptions of their patches are often... a little vague (from the point of view of a sysadmin, I am by no means a security expert).
I do not know if this is true (but it makes sense), and I have no clue how many bugs were discovered that way. But I would not be surprised if Microsoft quietly fixed those. The descriptions of their patches are often... a little vague (from the point of view of a sysadmin, I am by no means a security expert).
http://queue.acm.org/detail.cfm?id=2094081 says
"Since 2008, SAGE has been running 24/7 on an average of 100-plus machines/cores automatically fuzzing hundreds of applications in Microsoft security testing labs."
"Since 2008, SAGE has been running 24/7 on an average of 100-plus machines/cores automatically fuzzing hundreds of applications in Microsoft security testing labs."
Thanks!
[deleted]
While a bit surprising to see so many at once in OBSD kudos to the team for the rapid response and to those who found the bugs for their responsible disclosure.
Multiple (!) security issues in OpenBSD (!).
OK.
We're doomed.
OK.
We're doomed.
At least these (seem to) result in a kernel panic instead of privileged escalation or reading unauthorized memory.
But in a world where even djbdns had a (minor) security issue, I don't think it's reasonable to expect any non-trivial software—particularly software written in C—to be fully secure.
But in a world where even djbdns had a (minor) security issue, I don't think it's reasonable to expect any non-trivial software—particularly software written in C—to be fully secure.
And for those not aware, Project Triforce, is NCC's effort to run the wonderful fuzzer American Fuzzy Lop, on everything:
https://www.nccgroup.trust/us/about-us/newsroom-and-events/b...
https://www.nccgroup.trust/us/about-us/newsroom-and-events/b...
yeukhon(3)
It sounds like one local privilege escalation (possibly?) and a series of crashers?
Honestly walking away with those being the highest severity bugs is a credit to the OpenBSD team and their focus on security. They're totally bugs and it sounds like they're getting fixed immediately, but... many kernels fix these types of things all the time and don't even consider them security bugs.
Honestly walking away with those being the highest severity bugs is a credit to the OpenBSD team and their focus on security. They're totally bugs and it sounds like they're getting fixed immediately, but... many kernels fix these types of things all the time and don't even consider them security bugs.
Those bugs come from syscall fuzzing. Local privilege escalation is the worst thing you can get from it, so the results are as bad as they can be (apart from finding multiple privilege escalations that is).
I see your point, but it doesn't happen to be true, a bug in recv, send or other networking syscalls certainly could be much worse than a local priv escalation.
And seriously, only one local priv escalation and some crashers is still a pretty light haul for a good fuzzing session from a competent team. The sky is not falling for OpenBSD today.
And seriously, only one local priv escalation and some crashers is still a pretty light haul for a good fuzzing session from a competent team. The sky is not falling for OpenBSD today.
I'm not sure I follow. Certainly, bugs in the TCP/IP subsystem can be worse than syscall bugs. But what's the bug in recv(2) that is going to be worse than that?
I don't think anyone thinks the sky is falling. To me what's interesting about this is the approach Jesse and Tim took, and how quickly it generated very straightforward panics.
I don't think anyone thinks the sky is falling. To me what's interesting about this is the approach Jesse and Tim took, and how quickly it generated very straightforward panics.
If there was a systematic bug in receive or send where a certain string of data corrupted a buffer allowing for code execution, exploiting it remotely would be simple if there's any code on the machine using that syscall.
I'm not saying this is a likely bug, or even something vaguely reasonable, but it could totally be found with local syscall fuzzing, so local priv escalation isn't the worst bug you can uncover.
I'm mostly being pedantic here.
I'm not saying this is a likely bug, or even something vaguely reasonable, but it could totally be found with local syscall fuzzing, so local priv escalation isn't the worst bug you can uncover.
I'm mostly being pedantic here.
send/recv is the syscall side of the kernel, so no that's still local attackers.
"I'm going to try to find bugs by calling 'recv' with strange parameters" will not find bugs that are triggered by any type of data coming off of the wire.
Yes, it's not impossible, but seems about as likely to me as fuzzing syscalls locally would be at finding security holes in www.facebook.com by connecting to it and fuzzing up an SSL connection.
"I'm going to try to find bugs by calling 'recv' with strange parameters" will not find bugs that are triggered by any type of data coming off of the wire.
Yes, it's not impossible, but seems about as likely to me as fuzzing syscalls locally would be at finding security holes in www.facebook.com by connecting to it and fuzzing up an SSL connection.
> a bug in recv, send or other networking syscalls certainly could be much worse than a local priv escalation
I think that's very unlikely. Since most of the socket API ignores the payload (it's just opaque data), I don't see how recv/send issue could be triggered from outside. Syscall vulnerabilities are usually based on bad flags, bad addresses, or type confusions. Normally you have to have local access to control those.
I think that's very unlikely. Since most of the socket API ignores the payload (it's just opaque data), I don't see how recv/send issue could be triggered from outside. Syscall vulnerabilities are usually based on bad flags, bad addresses, or type confusions. Normally you have to have local access to control those.
It looks like most of them cause panics; which are still pretty bad. An unprivileged user shouldn't be able to crash your system.
OpenBSD entire underlying theme and goals are based around security, and their record is actually really good in that regard.
The fact that it's taken this long and a powerful fuzzer to find all these shows their commitment to security while also showing what we can accomplish with fuzzers today.
I've got an old X1 I plan on installing/trying out OpenBSD.
OpenBSD entire underlying theme and goals are based around security, and their record is actually really good in that regard.
The fact that it's taken this long and a powerful fuzzer to find all these shows their commitment to security while also showing what we can accomplish with fuzzers today.
I've got an old X1 I plan on installing/trying out OpenBSD.
Right? As much as anything this seems like they've found places where OpenBSD perhaps drifted from their best practices. How many panics are in their kernel today and how many should be there? Knowing the OpenBSD guys this has already triggered at review at that level of their entire source code.
Ya I was quite impressed.
[deleted]
Firstly glad to see these reported and fixed.
Secondly how many of these were remotely exploitable? Yes OpenBSD is limited in it's exposure with the "base system", but it seems like few of these pose as "holes" for the system? Arguably pledge(2) could factor into this, maybe? I'll let someone better qualified comment.
Again glad to see these fixed. But is the baseline free user access to the whole system for NetSec/ OpSec these days? I don't know maybe it is.
I'm just reluctant to have to read through the HN, "OMG OpenBSD had CVEs" and "C is insecure". Arguably the later has some merit but C isn't going away anytime soon, for better or for worse.
Secondly how many of these were remotely exploitable? Yes OpenBSD is limited in it's exposure with the "base system", but it seems like few of these pose as "holes" for the system? Arguably pledge(2) could factor into this, maybe? I'll let someone better qualified comment.
Again glad to see these fixed. But is the baseline free user access to the whole system for NetSec/ OpSec these days? I don't know maybe it is.
I'm just reluctant to have to read through the HN, "OMG OpenBSD had CVEs" and "C is insecure". Arguably the later has some merit but C isn't going away anytime soon, for better or for worse.
Direct remote privilege escalation exploits in kernels are relatively rare.
The usual attack path is exploiting an application bug for local user access, followed by a local privilege escalation.
The usual attack path is exploiting an application bug for local user access, followed by a local privilege escalation.
> Secondly how many of these were remotely exploitable?
I don't think this is a valid question. The vulnerabilities are in syscalls. Unless some application exposes them in a very unexpected way, they're local-only by design.
I don't think this is a valid question. The vulnerabilities are in syscalls. Unless some application exposes them in a very unexpected way, they're local-only by design.
I wish someone would release something like Fuzzing At Home. I've got computer power to throw at it, but I don't really have the expertise to do the setup work...
The last time I tried to compile an executable using all the afl fuzzer magic-compile-time stuff, I gave up, so I have the impression that CPU time isn't the bottleneck here.
It took me a day to understand the tooling required to get AFL working. Now I can spin up a new test case for a library within a couple hours. Once you have the test case, then it is CPU bound. I had on one series of tests, of the baresip sip library libre, running across 4 machines(24 cores each) for a day before it was 100% sure it had tested every code path looking for a SIP protocol decoding error through fuzzing.
How did you split up the work across the machines?
AFL has documentation on how to setup that up. eg:
https://github.com/mcarpenter/afl/blob/master/docs/parallel_fuzzing.txtThis is part of my problem. The other part is that I just don't know where to look. I'd much rather have some board choose submitted areas of open source projects, have the submitter/community create a harness, and then run something like Folding@home in a Docker container.
Computing power isn't the bottleneck of fuzzing. It's being smart about choosing your targets, tools and seeds.
My rule of thumb: It's very rare that I see any results beyond running a fuzzing job on an average machine for a day.
My rule of thumb: It's very rare that I see any results beyond running a fuzzing job on an average machine for a day.
It's striking how many of these issues cause panics because of assertions that were already in the code. Without good assertion use, I would assume that many of these would have been much worse.
Ted U. announced that usermount will be removed in OpenBSD 6.0
https://marc.info/?l=openbsd-announce&m=146854517406640&w=2
https://marc.info/?l=openbsd-announce&m=146854517406640&w=2