Tracing HTTP Requests with tcpflow(ananthakumaran.in)
ananthakumaran.in
Tracing HTTP Requests with tcpflow
https://ananthakumaran.in/2022/11/12/trace-http-requests.html
16 comments
For a CLI Wireshark, use tshark
There is also netpeek[1] which has better filtering capabilities and UX overall. It supports ngrep like bpf filters which we found useful.
1: https://github.com/darshanime/netpeek
1: https://github.com/darshanime/netpeek
This assumes you have shell and those tools installed via your Dockerfile. Best practice is to have a multi stage build and just have the working binary, keeping the image as light as possible.
So when do you make a fat image with debug tools and when do you keep it skinny?
So when do you make a fat image with debug tools and when do you keep it skinny?
Or you’re running on a VM and can install these tools for debugging and then blow it away when you’re done
A thought. I would usually start with my application logs. Reverting to a tcp tool for a http app problem seems cobtrived
Yeah the app looked like it was delegating to another service and needed a bearer token for that. The app logs should have showed this straight away. Furthermore, I’ve been burned enough to know that if I have a service that’s dependent on other services then I setup tasks in my service to ping the health of those services periodically and log warnings/errors if they’re unreachable. This allows me to divert blame as quickly as possible if alerted in the middle of the night.
This looks handy, better ergonomics than strace to just hook into http requests of a running process.
I wonder how it works with TLS if it’s working at the socket level..
I wonder how it works with TLS if it’s working at the socket level..
There seems to be an issue open for this https://github.com/simsong/tcpflow/issues/58
I’m guessing it doesn’t work with TLS as it’d need the keys to decrypt the traffic
I recall seeing a thread somewhere saying tcpflow would not add this capability and they point people to ssldump [1][2] and even that has some limitations.
[1] - https://github.com/adulau/ssldump
[2] - https://linux.die.net/man/1/ssldump
[1] - https://github.com/adulau/ssldump
[2] - https://linux.die.net/man/1/ssldump
With TLS 1.3 and Perfect Forward Safety, even knowing the secret key is no longer enough. Otherwise, ssldump is a tool that would dump sessions if you knew the server private key and were somehow able to force negotiating a non-PFS TLS 1.2 cipher suite.
in this space I have successfully used mitmproxy, charles and, with the best experience, fiddler
mitmproxy.org www.charlesproxy.com https://www.telerik.com/fiddler/fiddler-classic
mitmproxy.org www.charlesproxy.com https://www.telerik.com/fiddler/fiddler-classic
+1 for Fiddler, though it doesn't sound like the author was in a position to forward traffic to an HTTP proxy at the time in prod
There's also termshark, a TUI for tshark inspired by Wireshark.
You also have mitmproxy which can be useful if TLS is involved.
You also have mitmproxy which can be useful if TLS is involved.
Just because I know them better, I would have used `netstat -putln` and `tcpdump -i lo -n -A port 8000` to do the same thing. I'll take a look at tcpflow and ss though - it's always nice to know more tools.
It's good knowing this tool exists, but I think I'll stick to tcpdump and Wireshark. In Wireshark it's trivial to use the `http.request` filter to do this, and following the TCP conversation with decoded bodies, or specifying a TLS cert, is equally simple.