codesoap

il y a 2 ansdiscuss

A while ago I wrote https://github.com/codesoap/osmar, a tool for searching PostgreSQL databases with open streetmap data from the command line. Now I have rewritten the tool to read PBF files directly instead of querying a database. This article gives some insight into the performance optimizations needed to make this approach viable.

codesoap

il y a 2 ansdiscuss

Good point. Maybe a small 'hp2url' or 'hp2curl' tool, which takes httpipe input and prints URLs or curl commands for each given request, would be a better solution.

codesoap

il y a 2 ansdiscuss

Only briefly. HAR is extremely verbose, which makes it impractical for storing large amounts of requests and responses. It's also not line based, not designed for easy streaming/filtering (it has header/config fields) and the more "granular structure" (e.g. HTTP headers are separate JSON objects) allows for less freedom in creating malformed requests, which can be desirable when trying to find bugs in web applications.

codesoap

il y a 2 ansdiscuss

It was a tradeoff. The solution with "base" is more compact and arguably easier to read. On the other hand, it's easier to filter or manipulate httpipe, if the fields are separated. For example, to filter out all requests to a certain host, independent of port or protocol, one could just use this:

    jq -c 'select(.host == "test.net")' my_stored_results.jsonl

The port of some stored requests can be swapped out like this:

    jq -c '.port = 8080' my_stored_requests.jsonl

In other words: I chose to sacrifice compactness for "ease of tinkering".

However, I was also thinking about specifying an optional/informational "url" field or something similar, which also includes the path; "https://test.net/src" for your example. This field would be ignored by tools, but one could more easily copy things over into curl or a browser for further investigations.

codesoap

il y a 2 ansdiscuss

I haven't been on the hunt much since I wrote the tools, but the first two bug bounties I ever received where issued for things I found with a web fuzzer. No doubt I would have made that money with pfuzz, if I had it back then :-)

codesoap

il y a 2 ansdiscuss

Wow, didn't expect you to see this, but glad to have you here :-) Don't hesitate to contact me if you have any feedback!

codesoap

il y a 2 ansdiscuss

To be frank, I hadn't even considered looking at the file format of wireshark when thinking of existing file formats, that I could reuse. I've now taken a brief look and it seems like wireshark supports quite a lot of different formats [1], but the preferred one seems to be PcapNG. At a first glance, there are several attributes that make them less suitable for my purposes:

1. PcapNG as well as the other file formats look like they are storing packets, which is a lower level than HTTP requests and unnecessarily verbose for my intended purposes.

2. They are binary formats, which makes them less suitable for printing to stdout. This also means, that they are not line based, which means UNIX tools, like grep, cannot be used effectively.

3. They are not designed for streaming. The httpipe format is line-based and contains no header/global fields. Thus it is trivial to, for example, build a filtering program: it would just read one line at a time and print it again, if it matches filter criteria; the output would automatically be valid httpipe again.

4. Lastly, parsing and composing JSON is something most developers have done before and basically every programming language has libraries for it. This makes it easy for the ecosystem to grow and enables users to build custom tools without too much initial effort.

[1] https://wiki.wireshark.org/FileFormatReference

[2] https://pcapng.com/

codesoap

il y a 2 ansdiscuss

I have thought about this a lot, but came to the conclusion, that people are most likely to write tools if the format is easy to parse and construct in many programming language. I think it's hard to find an encoding that suits this criteria better than JSON.

It also has the benefit of being single-lined, since newline characters are encoded, which is necessary for a line based format. This, in turn, allows the use of many UNIX tools, like grep.

However, it certainly is not the most compact format, when encoding large amounts of binary data. Gzipping for storage will alleviate the overhead somewhat. Overall, dealing with large amounts of binary data in web application testing seems like a less common use case, so I felt like I shouldn't put too much focus on it.

codesoap

il y a 2 ansdiscuss

It seems to me like "fuzzing" has a different meaning in web application penetration testing. Here, "fuzzer" is a term for tools that just generate different request using wordlists, without adding any mutations. For example, the two popular tools ffuf [1] and wfuzz [2] also call themselves fuzzers.

I see how reusing a term for a different concept is bothersome, but I feel like "fuzzer" is the term that people learning about bug bounty hunting are familiar with.

[1] https://github.com/ffuf/ffuf

[2] https://wfuzz.readthedocs.io/en/latest/

codesoap

il y a 3 ansdiscuss

> go is the worst because the unix-hater club at the root of their development team [...]

I was surprised to find this sentiment about Go. I always felt Go focused on simplicity, modularity, good tooling and even good integration with C. Those are qualities that seem UNIX-affirmative to me.

Where does De Raadt's opinion come from? Could it be referring to Rob Pike's blasphemic Plan 9 adventure or is it just a provocative rambling about a certain ioctl-incident?

codesoap

il y a 4 ansdiscuss

Good idea. I'll think about it. Thanks!

codesoap

il y a 4 ansdiscuss

I just discovered https://osmq.eu and https://osmq.org. I guess this name is taken :(

codesoap

il y a 4 ansdiscuss

Hmm, I'm afraid I'm a bit biased against anything "explore", because I always associate it with Microsoft (internet explorer and file explorer).

codesoap

il y a 4 ansdiscuss

Oh, I like that one. I'll think about it for a bit, but maybe I'll take it. Thanks!

codesoap

il y a 4 ansdiscuss

Author here. I've heard that before. Maybe I should really rename. Got any suggestions? "osms" (OSM search) might be misleading because it reminds me of SMS. "osmfind" is a little long for my taste, but well, maybe it's better than "osmf".

Submissions

[untitled]

[untitled]

Questioning Go's range-over-func Proposal

Show HN: Pfuzz, a web fuzzer following the Unix philosophy

Do the Go developers hate Unix?

Figment: A simple script I wrote to A/B test audio quality

comments