Bugs found by jsfunfuzz(bugzilla.mozilla.org)
bugzilla.mozilla.org
Bugs found by jsfunfuzz
https://bugzilla.mozilla.org/show_bug.cgi?id=jsfunfuzz
9 comments
For anyone interested it's called fuzz testing: http://en.wikipedia.org/wiki/Fuzz_testing
This is really interesting, and a great approach to testing. I wonder if other JS engines like V8 are using this or something similar?
Indeed interesting. I wonder if there is a tool for generating input in a way that guarantees that all locations in the program are actually covered (in other words, that all reachable code has been reached).
Of course, this is no guarantee that the program actually works, but it would make me sleep better :)
Of course, this is no guarantee that the program actually works, but it would make me sleep better :)
An instrumented fuzzer like AFL purports to do this:
http://lcamtuf.coredump.cx/afl/
This blog post is a fascinating description of its potential:
http://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thi...
I haven't had as much success with it, but it's so interesting that I'll keep trying. I'm also interested in KLEE, which I found in a similar HN story, but it has very specific build requirements:
https://klee.github.io
[Edit: bhouston posted the exact same links minutes before I did. Anyway, cool stuff.]
http://lcamtuf.coredump.cx/afl/
This blog post is a fascinating description of its potential:
http://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thi...
I haven't had as much success with it, but it's so interesting that I'll keep trying. I'm also interested in KLEE, which I found in a similar HN story, but it has very specific build requirements:
https://klee.github.io
[Edit: bhouston posted the exact same links minutes before I did. Anyway, cool stuff.]
KLEE is pretty cool. To compensate for the atrocious build instructions there's a docker image which contains KLEE built and ready to use (https://registry.hub.docker.com/u/kleeweb/klee/).
There's also a web interface to just play around with KLEE without having to download and install anything that a few other people and I worked on available at http://klee.doc.ic.ac.uk:55080/, which we open-sourced https://github.com/klee-web/klee-web.
There's also a web interface to just play around with KLEE without having to download and install anything that a few other people and I worked on available at http://klee.doc.ic.ac.uk:55080/, which we open-sourced https://github.com/klee-web/klee-web.
One type of testing (sorry, not a tool yet) which can help with this is combinatorial testing. Basically, when testing against a set of multiple inputs, create a test which will test every possible combination of input for a pair of inputs.
i.e. for inputs A, B, C, try every combination of A-B, A-C, and B-C
It sounds like a lot, but its not too onerous with the speed of computers we have today. This type of testing will flush out most bugs which will result from a particular combination of inputs, and is deterministic, when compared to fuzz testing.
This is not to detract from fuzz testing, just a note that it is effectively designed to be non-deterministic, which can result in a delay in bug detection.
i.e. for inputs A, B, C, try every combination of A-B, A-C, and B-C
It sounds like a lot, but its not too onerous with the speed of computers we have today. This type of testing will flush out most bugs which will result from a particular combination of inputs, and is deterministic, when compared to fuzz testing.
This is not to detract from fuzz testing, just a note that it is effectively designed to be non-deterministic, which can result in a delay in bug detection.
Just to be clear, taking inputs in pairs does not guarantee complete coverage, even when trying all permutations of those pairs. I mention this because it is what the OP of this thread asked about.
There is a tool like that for compiled binaries. Details on its usage here:
http://lcamtuf.coredump.cx/afl/
http://lcamtuf.blogspot.ca/2014/11/pulling-jpegs-out-of-thin...
http://lcamtuf.coredump.cx/afl/
http://lcamtuf.blogspot.ca/2014/11/pulling-jpegs-out-of-thin...
Yes, they are running various internal testing suites in google, check here for example: https://code.google.com/p/chromium/issues/detail?id=360298