Show HN: OpenAPI fuzzer – fuzzing APIs based on OpenAPI specification(github.com)
github.com
Show HN: OpenAPI fuzzer – fuzzing APIs based on OpenAPI specification
https://github.com/matusf/openapi-fuzzer
24 comments
Fuzzing: “Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program”
[Wikipedia]: https://en.m.wikipedia.org/wiki/Fuzzing
(Just in case others are looking for a definition, though I guess it must be relatively well known)
[Wikipedia]: https://en.m.wikipedia.org/wiki/Fuzzing
(Just in case others are looking for a definition, though I guess it must be relatively well known)
Very impressive, particularly for an undergrad thesis! It's almost magical how quickly fuzzing can find bugs, even in mature software.
Fantastic, great to see. I was looking into this problem several years back and recall wanting more semantics than openapi provided to be able to drive fuzzing. Maybe you have a clever solution. Will check out the thesis. Cheers!
Love this. Sad it’s GPL though.
Fun fact but there are very little hazards for GPL with testing software. Since it doesn't ship with the actual software it can be restricted to the source being distributed to the "users" of the tests which are the developers. If source must be provided to the contractor (in the case of say the US gov or a corporation), the tests only need be run in the view of a restricted set of individuals which allows them to receive the source but none of the standard users of the main, non-testing software.
Why so?
This is fantastic!
I was a just looking for a fuzzer for OpenAPI stuff. I'm building a project with FastAPI that I'd love to try this on.
I was a just looking for a fuzzer for OpenAPI stuff. I'm building a project with FastAPI that I'd love to try this on.
You can already do this using Hypothesis with the Pydantic plugin, https://pydantic-docs.helpmanual.io/hypothesis_plugin/. Here is an example for Quart-Schema (similar setup to FastAPI) https://pgjones.dev/blog/automatic-api-testing-2021.
Excellent! It's always nice to find useful tools you didn't know about.
[deleted]
Great job! I wonder if that could be used with other fuzzers in OSS fuzz (giving tools more options to use for fuzzers and APIs)
Right now, it's probably not well suited for use in CI, since it runs until it's terminated. But I definitely plan to add a support for better usage in CI along with some examples. Thanks for the suggestion. I'll look into it!
Looks like a good start. Things can get complicated when going into request bodies, especially oneOf, anyOf relations.
I went through this journey while writing a similar tool: https://github.com/Endava/cats. Currently it has 72 Fuzzers registered which cover quite a lot of scenarios.
There's another one here by Microsoft - this is cool though! great to see more Rust tools.
https://github.com/microsoft/restler-fuzzer
https://github.com/microsoft/restler-fuzzer
Yes, we looked into it. There is a chapter in my thesis[0] about RESTler and comparison with OpenAPI fuzzer. The main difference between those two fuzzer is that RESTler is a statefull fuzzer and OpenAPI fuzzer is a stateless fuzzer.
Thanks to being statefull, RESTler is able to analyze a dependencies between a requests. For example, it will not call and endpoint to get user details before calling endpoint to create a user. This should make it more efficient because it does not waste requests to calling endpoint that will return 404.
On the other hand, fuzzing is all about trying to supply unexpected input to the software. Therefore, OpenAPI fuzzer makes those requests, since it may cause some undefined behavior when we try to get user before creating one.
So while RESTler tries to check some invariant (for example if deleted user cannot be accessed) OpenAPI fuzzer tries to cause the service to crash by invalid input. RESTler is usually not able to crash the service by providing invalid input, because it needs to keep the fuzzing dictionary small (bigger dictionary would make the dependency finding slow). So I think, they complements each other nicely.
Another difference it in reporting error. RESTler considers only status code 500 as an error. However, OpenAPI specification states all possible status codes that we can get from an endpoint. OpenAPI fuzzer utilizes this and reports every time it receives and unexpected status code.
0: https://github.com/matusf/bachelor-thesis/releases/download/...
Thanks to being statefull, RESTler is able to analyze a dependencies between a requests. For example, it will not call and endpoint to get user details before calling endpoint to create a user. This should make it more efficient because it does not waste requests to calling endpoint that will return 404.
On the other hand, fuzzing is all about trying to supply unexpected input to the software. Therefore, OpenAPI fuzzer makes those requests, since it may cause some undefined behavior when we try to get user before creating one.
So while RESTler tries to check some invariant (for example if deleted user cannot be accessed) OpenAPI fuzzer tries to cause the service to crash by invalid input. RESTler is usually not able to crash the service by providing invalid input, because it needs to keep the fuzzing dictionary small (bigger dictionary would make the dependency finding slow). So I think, they complements each other nicely.
Another difference it in reporting error. RESTler considers only status code 500 as an error. However, OpenAPI specification states all possible status codes that we can get from an endpoint. OpenAPI fuzzer utilizes this and reports every time it receives and unexpected status code.
0: https://github.com/matusf/bachelor-thesis/releases/download/...
Thanks for the reply! Nice to know those differences, I will definitely try your project on a spec in the future.
Does it support OAS 3.1?
No, it does not. As for the parsing of the OAS, it relies on openapiv3[0], which does not yet support it.
0: https://github.com/glademiller/openapiv3
0: https://github.com/glademiller/openapiv3
That's too bad. It's starting to feel like 3.1 is never going to gain adoption unless we get entirely new tools built for it.
I'm less worried about 3.1 as I was with 3.0. It took years (like more than three) for the move from 2 to 3, and still there's a couple of popular tools that still haven't fully supported 3.0 yet.
[deleted]
I've just managed to get this set up with an API but it appears to get stuck testing only one of the endpoints. For example, if I set it up to test an endpoint with a query param like so:
> /search?q=?
Then it seems to try every single value for q it can think of (there are quite a few different possible strings) without ever moving onto the next endpoint. Is there anyway to configure its behaviour or provide hints as to what kind of fuzzing to do?
> /search?q=?
Then it seems to try every single value for q it can think of (there are quite a few different possible strings) without ever moving onto the next endpoint. Is there anyway to configure its behaviour or provide hints as to what kind of fuzzing to do?
Thanks for the report. However, without more information I'm not able to help you. What is your setup? How did you run it? could you please share the specification file? Also, let's probably move this discussion to GitHub issues: https://github.com/matusf/openapi-fuzzer/issues
As for the choice of language, Rust proved to be a good decision, even though, one would think that dynamic languages are better suited for fuzzing (at least that was the choice for API fuzzers that I looked into). Thanks to Rust's type system, I was able to deserialize the OpenAPI specification to structs and traverse them when creating a fuzzing payload in a type-safe way. Other fuzzers load the specification to a dictionary/hashmap and then fail during the traversal because of some missing key they expected.
0: https://github.com/matusf/bachelor-thesis/releases/download/...
1: https://github.com/matusf/openapi-fuzzer#findings