A visual guide to SSH tunnels(robotmoon.com)
robotmoon.com
A visual guide to SSH tunnels
https://robotmoon.com/ssh-tunnels/
81 comments
I’ve been using ssh for 25+ years, I wrote a (in hindsight, pretty awful) article about it for Linux Journal, and I never knew this.
Since this tip is popular I'll share another related tip.
When using nested ssh session, you can send escape commands to any of them by the number of tildes. For example, if you ssh into Server A and then from Server A to Server B:
When using nested ssh session, you can send escape commands to any of them by the number of tildes. For example, if you ssh into Server A and then from Server A to Server B:
~~. # drop session from Server A to Server B
~. # drop original session into Server AI sometimes have to hit a <return> before the ~. disconnect works. Not sure why.
Because iirc, ssh only interprets the tilde if it's immediately preceded by a newline.
Is ~ a literal ~ character? Or notation for some modifier key? I only knew about ^ to mean "control".
Yes, but you need to press enter before using it. Try typing `⏎~?' in an SSH session.
Literal ~. Its input as a sequencenot as a modifier. Iirc the sequence ~h or ~? prints a little help page
Warning, if you read comradesmith's comment even more literally and type "~." (tilde and period) in an SSH session, that will close the connection and exit the client. It is useful when a remote application hangs and does not let you exit gracefully: https://apple.stackexchange.com/questions/35524/what-can-i-d...
Well, there's something I didn't know. Thank you for the time and tab savings!
There's a really fancy new -R feature that I love (added in 2017 I think).
ssh <target> -R <just a port number>
This opens up a localhost port on the target that acts as a socks server which tunnels all your traffic through the source machine.
This is great for machines that you can SSH into, but are otherwise completely isolated from the network, or are monitored heavily. You can jump on, and pull down everything you need from the existing SSH connection, rather than using the machine to make requests out to the internet directly.
This is also good if the source machine is a web server or something on a secured network which can SSH out, but not much else, and the destination is your command and control server on the internet. Then it opens up a socks port on the C&C machine that gives full access to the internal network, impersonating the source machine.
Every other -R is a point to point TCP connection, but setting up a SOCKS proxy with -R is magic. More analogous to a reverse -D than a reverse -L. Super useful.
ssh <target> -R <just a port number>
This opens up a localhost port on the target that acts as a socks server which tunnels all your traffic through the source machine.
This is great for machines that you can SSH into, but are otherwise completely isolated from the network, or are monitored heavily. You can jump on, and pull down everything you need from the existing SSH connection, rather than using the machine to make requests out to the internet directly.
This is also good if the source machine is a web server or something on a secured network which can SSH out, but not much else, and the destination is your command and control server on the internet. Then it opens up a socks port on the C&C machine that gives full access to the internal network, impersonating the source machine.
Every other -R is a point to point TCP connection, but setting up a SOCKS proxy with -R is magic. More analogous to a reverse -D than a reverse -L. Super useful.
> (added in 2017 I think)
Indeed: committed in September 2017, released in v7.6 in October 2017.
Indeed: committed in September 2017, released in v7.6 in October 2017.
This[1] is probably the best, easiest and most (visually) effective way to learn about SSH tunnels.
[1] : https://unix.stackexchange.com/a/118650/289353
[1] : https://unix.stackexchange.com/a/118650/289353
Drawing some diagrams helped me understand remote port forwarding and OpenSSH's port:host:port syntax: http://dirk-loss.de/ssh-port-forwarding.htm
Thanks! I pinned your page on my evernote.. I always get confuse everytime I need to make a ssh port forward. After a while I get it but I spend time (and some bad words :P)
I think it's way clearer!
One common use is to secure vnc, which has no crypto built into it.
On the remote host, have the vnc server listen only on localhost:5901
This assumes everyone who can ssh to the server should be allowed access.
On your workstation, form an ssh tunnel to that remote host and its port, and link it your own localhost:5901
Open your vnc client GUI and set it to open 127.0.0.1:5901, voila, vnc session.
More fun things: say for example you have a big remote xen dom0 with a lot of unique qemu HVM VMs running on it. Configure each of their .cfg files for xen to spawn a vnc server on localhost only and a unique port number per domU (5902,5903,5904,etc). Then use the same method to connect.
On the remote host, have the vnc server listen only on localhost:5901
This assumes everyone who can ssh to the server should be allowed access.
On your workstation, form an ssh tunnel to that remote host and its port, and link it your own localhost:5901
Open your vnc client GUI and set it to open 127.0.0.1:5901, voila, vnc session.
More fun things: say for example you have a big remote xen dom0 with a lot of unique qemu HVM VMs running on it. Configure each of their .cfg files for xen to spawn a vnc server on localhost only and a unique port number per domU (5902,5903,5904,etc). Then use the same method to connect.
Note that it still exposes the connection to all local processes, on both hosts (except network-restricted services or with non-typical host-local firewall rules), though it should be fine, if you're using semi-reasonable passwords.
SSH supports unix domain socket forwarding and even cross unix/ip forwarding with a syntax like:
Unfortunately I haven't been able to convince tigervnc viewer to connect to a unix domain socket, so I don't have a fully scripted connection, but remmina and tigervnc server work fine.
Also, forwardings / connections fail if the socket already exists when the listening side starts up (often an unclosed leftover), so I rm them before starting the ssh connection / vnc server and it works reliably.
SSH supports unix domain socket forwarding and even cross unix/ip forwarding with a syntax like:
ssh -L /path/to/sock:/path/to/sock
ssh -L /path/to/sock:localhost:5901
ssh -L 5091:/path/to/sock
Unfortunately I haven't been able to convince tigervnc viewer to connect to a unix domain socket, so I don't have a fully scripted connection, but remmina and tigervnc server work fine.
Also, forwardings / connections fail if the socket already exists when the listening side starts up (often an unclosed leftover), so I rm them before starting the ssh connection / vnc server and it works reliably.
Wow that's amazing, I've been cobbling together socat processes because I didn't know this was possible.
I do this for a number of hosts simultaneously with macos and map them specifically in my .ssh/config:
to remote into machine mac-b, in a terminal window:
After that, you can right-click on screen-sharing in the dock - even when it's not running - and you will see a context menu item for "mac-b 5904.vncloc"
By the way, mac to mac screen sharing is very well done (will they axe it next?)
- it is very fast
- the keyboard stuff just works
- you can copy/paste even rich text between the local and remote macs
- you can drag and drop files from the remote desktop to the local one
Host mac-a
...
LocalForward 5903 localhost:5900
Host mac-b
...
LocalForward 5904 localhost:5900
to remote into machine mac-b, in a terminal window:
ssh mac-b
After that, you can right-click on screen-sharing in the dock - even when it's not running - and you will see a context menu item for "mac-b 5904.vncloc"
By the way, mac to mac screen sharing is very well done (will they axe it next?)
- it is very fast
- the keyboard stuff just works
- you can copy/paste even rich text between the local and remote macs
- you can drag and drop files from the remote desktop to the local one
I am always amazed at what is possible with the linux command line, and it‘s something one can really fall in love with.
These are simply amazing. There's a stack overflow answer similar to this which I found very intuitive to understand https://unix.stackexchange.com/a/115906/254044
SSH tunnels are my favorite tool for NAT-busting. I always have to look up the cryptic syntax, but with one strategically placed Raspberry Pi, you can basically get from anywhere to anywhere.
And it works just as well on a locked down corporate network as it does on a home network. Why people put up with garbage VPN software is beyond me.
And it works just as well on a locked down corporate network as it does on a home network. Why people put up with garbage VPN software is beyond me.
One can further tunnel SSH inside of TLS with Stunnel [0] to get through firewalls that block SSH but permit arbitrary TLS connections. Loading a website through this requires three layers of TCP but assuming the connection is good this should be okay.
A couple reasons for a regular VPN: they can be made automatic and persistent so the tunnel is always ready for use, they can tunnel other protocols besides TCP, they can tunnel entire networks, and using UDP as the tunnel protocol helps avoid the TCP-in-TCP congestion issues.
[0] https://en.wikipedia.org/wiki/Stunnel
A couple reasons for a regular VPN: they can be made automatic and persistent so the tunnel is always ready for use, they can tunnel other protocols besides TCP, they can tunnel entire networks, and using UDP as the tunnel protocol helps avoid the TCP-in-TCP congestion issues.
[0] https://en.wikipedia.org/wiki/Stunnel
Stunnel and ssh tunnels don't put TCP inside TCP so it shouldn't risk any TCP-in-TCP congestion issues?
Yes, and piecewise TCP is often even faster than point-to-point TCP, especially on flaky connections, or when the transient connections are on hosts with lower RTTs (outgoing gateway closer to destination). It will likely be faster than a UDP VPN.
It still does have overhead, so lower bandwidth. Also, TCP, SSH and/or TLS forwarding of UDP, IP or Ethernet can have issues with delays, re-transmits etc (TCP-in-TCP).
It still does have overhead, so lower bandwidth. Also, TCP, SSH and/or TLS forwarding of UDP, IP or Ethernet can have issues with delays, re-transmits etc (TCP-in-TCP).
SSH is a TCP connecting though, isn't it? How do you avoid TCP inside TCP when both are TCP? Oram I missing something?
It just relays the packet payload (i.e the actual data bytes inside the TCP stream are read out from the source, then sent along to the destination inside a different TCP stream. TCP/IP headers aren't included or replayed).
SSH isn’t a replacement for VPNs. I’d argue that they complement each other.
For example, SSH port forwarding is TCP only, so if you need UDP, you‘ll need a VPN.
For example, SSH port forwarding is TCP only, so if you need UDP, you‘ll need a VPN.
[note I am wrong here - it doesn't support UDP - see below]
The ssh SOCKS proxy supports UDP. I use Firefox with a SOCKS proxy via ssh (e.g. ssh -D 2222 hostname) to get around geoblocking and accessing papers via an institutional account. The tsocks program can also let you use a SOCKS proxy for an arbitrary command in linux. I find it much more convenient than a VPN, as it can be easily applied to single programs.
The ssh SOCKS proxy supports UDP. I use Firefox with a SOCKS proxy via ssh (e.g. ssh -D 2222 hostname) to get around geoblocking and accessing papers via an institutional account. The tsocks program can also let you use a SOCKS proxy for an arbitrary command in linux. I find it much more convenient than a VPN, as it can be easily applied to single programs.
For single applications that are able to "cooperate" with being proxied, a proxy server can make much more sense than a VPN, yes.
The less cooperative your applications are (even though cooperation can be somewhat enforced using tsocks, as you mention, which can be further made to apply system-wide using sshuttle [1]), the more a VPN will start to make sense.
[1] https://github.com/sshuttle/sshuttle
The less cooperative your applications are (even though cooperation can be somewhat enforced using tsocks, as you mention, which can be further made to apply system-wide using sshuttle [1]), the more a VPN will start to make sense.
[1] https://github.com/sshuttle/sshuttle
Are you about this? Last time I've checked SSH SOCKS5 only supports TCP traffic, UDP isn't supported and needs extra steps through tools like socat to wrap UDP in TCP.
http://lists.mindrot.org/pipermail/openssh-unix-dev/2017-Jan...
http://lists.mindrot.org/pipermail/openssh-unix-dev/2017-Jan...
It looks like I was wrong about UDP being supported by ssh. Sorry about misleading everyone. I've verified with nc that it doesn't actually forward UDP. I'm amazed that streaming media over this seems to work fine on most sites, though.
Many applications these days can fall back to TCP because UDP is often blocked (almost never for good reasons, but that's another discussion). For real-time applications, that fallback usually comes with a loss in quality/interactivity, though.
Alternatively, you might be unknowingly using un-proxied UDP: Many AV systems only enforce IP restrictions on the signalling protocol (which is often TCP); the actual media can then take whatever path it wants.
By the way, it seems like you can forward UDP over sshuttle [1] when used with tproxy (which seems to be a Linux-only feature)!
[1] https://sshuttle.readthedocs.io/en/stable/tproxy.html
Alternatively, you might be unknowingly using un-proxied UDP: Many AV systems only enforce IP restrictions on the signalling protocol (which is often TCP); the actual media can then take whatever path it wants.
By the way, it seems like you can forward UDP over sshuttle [1] when used with tproxy (which seems to be a Linux-only feature)!
[1] https://sshuttle.readthedocs.io/en/stable/tproxy.html
Note that the default port for SOCKS(5) is 1080, and the de facto standard for (forward) web proxies (e.g., Squid) is 3128, so one may want to make of habit of using one of those ports.
3128 is a common http/s proxy sure, but so is 8080, and I've seen 80 a few times too.
1080 is often default for socks, but I've also seen places where it's 1085 (for socks5 -- 1084 for socks4)
Ultimately it can be whatever you want.
1080 is often default for socks, but I've also seen places where it's 1085 (for socks5 -- 1084 for socks4)
Ultimately it can be whatever you want.
Actually, OpenSSH has built-in TUN/TAP support using -w<local-tun-number>:<remote-tun-number>. You can create a layer 3/point-to-point/ TUN tunnel or layer 2/ethernet/TAP tunnel.
I used to run a site-to-site VPN between two sites a couple of decades ago with a simple script doing just this. These days we'd use OpenVPN or strongSwan for such use-cases but despite the 'hackiness' of the former approach, it worked reliably for years.
I used to run a site-to-site VPN between two sites a couple of decades ago with a simple script doing just this. These days we'd use OpenVPN or strongSwan for such use-cases but despite the 'hackiness' of the former approach, it worked reliably for years.
OpenSSH VPNs suffer from TCP-over-TCP [1]. I'd only use them as a last resort.
Sometimes I actually wish SSH would offer UDP support for precisely this use case, but arguably, TCP VPN support is already a stretch in terms of scope for SSH.
[1] http://sites.inka.de/sites/bigred/devel/tcp-tcp.html
Sometimes I actually wish SSH would offer UDP support for precisely this use case, but arguably, TCP VPN support is already a stretch in terms of scope for SSH.
[1] http://sites.inka.de/sites/bigred/devel/tcp-tcp.html
Indeed. For VPN, use Wireguard. It’s open source, easy to configure, and works excellently.
https://www.wireguard.com/
https://www.wireguard.com/
Why don't people use unix domain sockets for local connections? They are pretty much the same thing as local IP connections, except access rights apply as on files, so you're not exposing connections to pretty much all processes.
Domain sockets can be forwarded via ssh with the same -L and -R arguments, including cross unix/ip forwardings.
Domain sockets can be forwarded via ssh with the same -L and -R arguments, including cross unix/ip forwardings.
It's a relatively recent feature in openssh (in the past decade for upstream; not sure when it first landed in an Ubuntu LTS, which for practical purposes is the earliest one might start using it regularly)
last time I tried ssh does not delete the socket file when the connection closes and complains the file already exists on a new connection.
For those interested in the topic of tunneling, I highly recommend The Cyber Plumber’s Handbook [1] to gain an even deeper understanding and examples beyond SSH.
[1] https://cph.opsdisk.com/
[1] https://cph.opsdisk.com/
Appreciate the shout out magicconch! Just added a HN family discount to pick up the book, lab guide, and lab access for $10.
https://gum.co/cph_book_and_lab/hackernews20210207
https://gum.co/cph_book_and_lab/hackernews20210207
Very cool, I was unaware of this product. Thanks for the deal! :D
Thanks for the HN discount! Purchased.
Is it just me or is this site a bit of a disaster on mobile?
Yes gave up it’s too awkward to read.
Could read just fine on iPhone 12p after zooming out; even in portrait
On desktop firefox all of the code snippets are black-on-black so I have to select the code to read it.
Yep, but it works well for me in landscape view.
Nice reference.
Reaching target hosts multiple jump-hosts away, and going the wrong way through multiple firewalls segregating those lans, is another use-case of tunnels I've found handy.
i.e.
your_remote_client_host->internet->fw1->lan1->fw2->lan2->fw3->target_host
Target_host can be reached by doing tunnel-in-tunnel-in-tunnel-in-tunnel. Each tunnel gets you past one firewall. The final tunnel you can just ssh to fw3 via a local tunnel and a local port on your_remote_client now takes you straight to target_host.
Reaching target hosts multiple jump-hosts away, and going the wrong way through multiple firewalls segregating those lans, is another use-case of tunnels I've found handy.
i.e.
your_remote_client_host->internet->fw1->lan1->fw2->lan2->fw3->target_host
Target_host can be reached by doing tunnel-in-tunnel-in-tunnel-in-tunnel. Each tunnel gets you past one firewall. The final tunnel you can just ssh to fw3 via a local tunnel and a local port on your_remote_client now takes you straight to target_host.
Nicely written!
I once also wrote a guide to access remote IoT devices using SSH tunnels: https://phillip.dornauer.cc/unix/iot/2019/04/12/set-up-ssh-t...
I once also wrote a guide to access remote IoT devices using SSH tunnels: https://phillip.dornauer.cc/unix/iot/2019/04/12/set-up-ssh-t...
Absolutely brilliant! I've had to explain local/reverse/dynamic port forwarding to people enough times that I've written a text file I can just send them, but I can finally replace that with a link to this page.
Really nice and clear. One addition: the final example uses ProxyCommand, I find ProxyJump much more useful: you can specify multiple hops clearly and even specify different private keys for each hop.
ProxyCommand is useful for proxying with a non-SSH protocol. For example when combined with proxytunnel or similar you can run your SSH connections over an HTTPS proxy, which appears as regular web traffic to the network operator.
ProxyJump indeed seems more useful, but has less support (it's a "newer" SSH feature). Do you know if it needs to be supported by the openssh libs of intermediate hosts, or just the initiating client?
As far as I remember, it‘s implemented on the client only. (It can be replicated with ProxyCommand and a few invocations of cat/nc on the proxy host, if I remember correctly.)
That comes in very handy. Bookmarked.
I also found the following very nice: (Same info but presented in a different way.)
https://unix.stackexchange.com/questions/115897/whats-ssh-po...
I also found the following very nice: (Same info but presented in a different way.)
https://unix.stackexchange.com/questions/115897/whats-ssh-po...
https://unix.stackexchange.com/a/115906/9745
I’ve always shared this with my team. I think OP’s post is good, but prefer the visuals of this for the `-L` and `-R` features.
Great resource. Most tutorials on SSH tunnels are strangely incomplete. This one seems to cover all the important points, and does so in an approachable way.
Every once in a while I wonder how the SOCKS5 stuff works at the protocol level, I would like to be able to SSH out without running a proxy process locally, just talk to the ssh deamon on the relay server directly. The OpenSSH code is a bit dense but it looks like it creates dynamic tunnels that are torn down when the connection closes. Anyone confirm that is what happens behind the scenes?
Great, I can refer people (and myself) to this page !
A lot of tunneling tutorial, medium articles and blog post mixes local and remote port forwarding and use the words interchangeably which cause a lot of confusion.
A lot of tunneling tutorial, medium articles and blog post mixes local and remote port forwarding and use the words interchangeably which cause a lot of confusion.
There’s also -w which sets up a point to point tunnel which you can route over - even more powerful than the SOCKS server
In tmux, when opening multiple sessions to the same remote server, how do you avoid typing the password repeatedly?
Look up ‘shh multiplexing’. This makes subsequent sessions share the same connection. I don't know a particularly good tutorial; maybe see https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing
Add your ssh pub key (id_rsa.pub) to authorized_ssh keys on the remote servers will automatically authenticate you.
https://www.ssh.com/ssh/copy-id
https://www.ssh.com/ssh/copy-id
Why not visualize them as actual tunnels?
Really nice guide!
This stuff really helps getting into the topic!
I was hoping for a series of tubes.
Thanks for this amazing guide!
In a computer networks (the Internet), if I can see you, you can see me.
You might want to be a bit more specific than computer networks. Monitor/ promiscuous mode is a thing, and so is dropping packets. It's not much use outside of wifi anymore, but there's still a few dumb hubs around.
One thing to add is that you can even open tunnels during an interactive session without disconnection.
To do this, type the escape command sequence ~C (will not show) and it will drop you to the control prompt. You can then add tunnels.