Closing a stale SSH connection(davidisaksson.dev)
davidisaksson.dev
Closing a stale SSH connection
https://davidisaksson.dev/posts/closing-stale-ssh-connections/
90 comments
> Problem can be that sometimes the connection is only broken one way
Many network engineers who have to deal with asynchronous routing have had to deal with this particular issue. You got into the habit of doing a ctrl+u ctrl+k just in case, and you get hyper-aware that pressing the enter key has consequences.
Many network engineers who have to deal with asynchronous routing have had to deal with this particular issue. You got into the habit of doing a ctrl+u ctrl+k just in case, and you get hyper-aware that pressing the enter key has consequences.
Another note for nordic layouts and Linux: tilde (~) is considered to be a "dead key" on most nordic layouts, meaning you may need to follow AltGr + ~ with a space before typing out the dot. Otherwise you might get weird looks from the terminal trying to figure what you meant with the tilde-dot.
Incidentally, while testing on Windows (in both WSL and cmd.exe) with a Finnish layout: you do not need a space after typing out the tilde.
Incidentally, while testing on Windows (in both WSL and cmd.exe) with a Finnish layout: you do not need a space after typing out the tilde.
For this reason you can change the tilde with `EscapeChar` configuration in your `~/.ssh/config` (or `-e` option, but I bet you want it to be permanent).
~ is also the (very common) shorthand for home directory. How do Nordic keyboard users deal with that?
As a former user of ISO layout keyboards, IMHO a great quality of life improvement is to switch to ANSI.
A US ANSI keyboard has all programming symbols in the right place. Yet it makes Latin/Nordic and other symbols easy to type via compose key (e.g. Right Alt), or similar.
For example, Compose a + ' = á, a + e = æ, o + / = ø, s + s = ß, s + o = §, etc.
An annoying issue is that most brands won't sell ANSI keyboards abroad. Apple is one of the few that get this right. A trick is to import from countries where ANSI is the default, such as NL.
A US ANSI keyboard has all programming symbols in the right place. Yet it makes Latin/Nordic and other symbols easy to type via compose key (e.g. Right Alt), or similar.
For example, Compose a + ' = á, a + e = æ, o + / = ø, s + s = ß, s + o = §, etc.
An annoying issue is that most brands won't sell ANSI keyboards abroad. Apple is one of the few that get this right. A trick is to import from countries where ANSI is the default, such as NL.
I use the french azerty layout (fr-pc) on macOS (equivalent fr-latin9 on linux). It is an ISO keyboard, I am able to do all of the examples you wrote, and more: option b is ß, ã is option ~ and a, "é" is its own key, so is "à" or "ç".
I am able to write 120 words per minute, not so special, but good enough for me. I find the ansi/qwerty too slow with its compose system (I tried on the canadian-intl layout made for Québec).
I am able to write 120 words per minute, not so special, but good enough for me. I find the ansi/qwerty too slow with its compose system (I tried on the canadian-intl layout made for Québec).
Is this a physical Azerty keyboard? How do you get around having to hold shift to type any number?
What are you in that your local ISO layout has these issues? The Irish layouts on X and Windows don't have this issue and makes typing just about any diacritic easy, though I go a bit further and use the UnicodeExpert variant. MacOS, OTOH, has issues unless I switch to an ANSI layout.
I was not claiming any local ISO has trouble typing other characters with compose key or similar.
My point was that ANSI is more ergonomic than ISO for programming, and all local characters are still easy to type.
My point was that ANSI is more ergonomic than ISO for programming, and all local characters are still easy to type.
Maybe I'm misunderstanding, but it's not an issue on a Mac. The Option + ^ will produce a ~ with no need for a space. I don't recall it ever being an issue for me on Linux either, so I wonder if there's a difference between a Nordic and a purely Danish keyboard.
Personally I always found that ^] for various terminal commands was much more of a hassle, to the point where it's easier to just close the terminal window.
Some Danes will just use a US keyboard layout, but that's not really as common as sites like HN will have you believe. In terms of programming it does make a little sense though. The 8 and 9 key is doing a lot of heavy lifting, Shift + 8 is (, Option + 8 is [ and Shift + Option + 8 is { and the matching close characters on 9.
There's not a lot of room for special characters on a Nordic keyboard, all most all of the require a modifier key. I don't know if that makes Nordic keyboard users more adverse to the use of these characters in commands and programming languages.
Personally I always found that ^] for various terminal commands was much more of a hassle, to the point where it's easier to just close the terminal window.
Some Danes will just use a US keyboard layout, but that's not really as common as sites like HN will have you believe. In terms of programming it does make a little sense though. The 8 and 9 key is doing a lot of heavy lifting, Shift + 8 is (, Option + 8 is [ and Shift + Option + 8 is { and the matching close characters on 9.
There's not a lot of room for special characters on a Nordic keyboard, all most all of the require a modifier key. I don't know if that makes Nordic keyboard users more adverse to the use of these characters in commands and programming languages.
“Personally I always found that ^] for various terminal commands was much more of a hassle (…)”
Yeah, tell me ’bout it. Back when I used Windows at work, a friend told me that instead of Ctrl ] one should press Ctrl ¨ – because on a regular Swedish keyboard, ¨ is on the same location that has ] on a US keyboard. (Not sure if this works with Linux/Mac, never tried there.)
Yeah, tell me ’bout it. Back when I used Windows at work, a friend told me that instead of Ctrl ] one should press Ctrl ¨ – because on a regular Swedish keyboard, ¨ is on the same location that has ] on a US keyboard. (Not sure if this works with Linux/Mac, never tried there.)
With agony. It sucks.
${HOME}
Yes, programming with a nordic keyboard is slightly inconvenient because ~, {, }, [, ], and | all require AltGr. (Isn't that the same for German keyboard?)
I had to switch over 25 years ago when moving. I just considered it a minor inconvenience. I don't think my programming output really suffers, there are so many other factors. Some (very few) people (native or foreigners) do use different keyboards for that reason. I have not noted that they would be better or faster programmers for that. I would claim the correlation between programming fast and introducing more bugs is much clearer.
As for the dead key, pressing tilde space is not the optimal solution. Especially if you are using ssh over ssh. That means you need to produce 2 tilde characters to control the inner ssh. I prefer pressing tilde tilde. With nested ssh that makes 4 tilde characters. Much easier to type than tilde space because you just hold the AltGr key during all for 4 key presses.
I had to switch over 25 years ago when moving. I just considered it a minor inconvenience. I don't think my programming output really suffers, there are so many other factors. Some (very few) people (native or foreigners) do use different keyboards for that reason. I have not noted that they would be better or faster programmers for that. I would claim the correlation between programming fast and introducing more bugs is much clearer.
As for the dead key, pressing tilde space is not the optimal solution. Especially if you are using ssh over ssh. That means you need to produce 2 tilde characters to control the inner ssh. I prefer pressing tilde tilde. With nested ssh that makes 4 tilde characters. Much easier to type than tilde space because you just hold the AltGr key during all for 4 key presses.
Anecdata, but after having some programs break when I used `~` to refer to my home directory (can't remember, was a long time ago), I got into the habit of either using `${HOME}` if I'm in a shell, or just writing the full path explicitly if it's a config file.
So nowadays I only use `~` when I'm navigating in an interactive shell session, and never in commands or config files.
So nowadays I only use `~` when I'm navigating in an interactive shell session, and never in commands or config files.
[deleted]
Pro tip: if you used ssh to get into host A and then another ssh to get from host A into host B, to break the A-B connection you need to issue `~~.`
("control up-arrow Q" song playing in the background.)
("control up-arrow Q" song playing in the background.)
Alternatively just use jump hosts: $(ssh -J bastion target)
Oh, I didn't know about the -J option. At $WORK, after ssh-ing onto the jump host, we've to do 'sudo -u special_user sudosh' (no password) before we ssh further. I don't suppose there's any way around that, is there?
ProxyCommand may help you; see [0]
[0] https://sshmenu.sourceforge.net/articles/transparent-mulitho...
[0] https://sshmenu.sourceforge.net/articles/transparent-mulitho...
And if you use host A like that enough you can also change the ssh escape key in your config.
Thanks.
Lots of words to just say "Enter then ~."
I feel old. I remember when we moved from telnet and rlogin to ssh and this was in the man page. I'm also happy to learn kids are still using terminals and ssh.
I suspect many linux users nowadays don't even know that manpages exist or don't want to use them. Otherwise blogposts like this would not exist.
They know the man pages exist and they want to use them, but most are written to merely be consistent in format and not necessarily in content. The worst thing to omit is examples.
What users do nowadays is rely on alternatives such as https://tldr.inbrowser.app/
What users do nowadays is rely on alternatives such as https://tldr.inbrowser.app/
Man pages are too long and technical for 70% of casual use when the answers are a Google away. Normal usage patterns don't involve using every option, and that's what man pages are focused on documenting.
I'm not sure why a video of a 60's TV show would be useful when you're trying to get an explanation of command line options for common posix commands. But you do you.
I’ve been around Unixes since the early 90s and was around for the transition away from telnet and I only learned about this about 5 years ago. Shoulda read the whole man page, I guess.
Well... terminal emulators. Bonus points to everyone who has an actual VT240 on their physical desktop.
> Bonus points to everyone who has an actual VT240 on their physical desktop.
Bonus points to anyone who can find one and get it to work. I want one, but it's a huge investment (both in price and effort)
Bonus points to anyone who can find one and get it to work. I want one, but it's a huge investment (both in price and effort)
Yikes. Yeah. Went on ebay thinking they couldn't possibly be THAT expensive. I was wrong. I wonder if there's a market for a $200 VT420 clone built with a FPGA that has a USB connection for a keyboard and a HDMI connector for a monitor?
The MAME emulator will actually run a VT260 rom, allowing the emulated serial port to be output to a socket.
Are guides available on how to setup and use?
Edit: Found this https://zork.net/~st/jottings/Real-VT102-emulation-with-MAME...
Edit: Found this https://zork.net/~st/jottings/Real-VT102-emulation-with-MAME...
Here are a bunch of URLs that I collected for the VT240.
http://www.bitsavers.org/pdf/dec/terminal/
https://github.com/larsbrinkhoff/terminal-simulator/issues/1
https://www.mail-archive.com/[email protected]/msg09086...
https://forums.bannister.org/ubbthreads.php?ubb=showflat&Num...
https://simh.trailing-edge.narkive.com/qgzCvrl8/dec-vt-emula...
The command that I ended up using was:
http://www.bitsavers.org/pdf/dec/terminal/
https://github.com/larsbrinkhoff/terminal-simulator/issues/1
https://www.mail-archive.com/[email protected]/msg09086...
https://forums.bannister.org/ubbthreads.php?ubb=showflat&Num...
https://simh.trailing-edge.narkive.com/qgzCvrl8/dec-vt-emula...
The command that I ended up using was:
mame -rp . vt240 -window -nothrottle -host null_modem -bitb socket.foo.com:23MAME is just crazy. I only learned recently it contains the best Sega Genesis emulator and now you tell me it does terminal emulation, too.
Primary source is the SSH manpage, Section "ESCAPE CHARACTERS". Read it from your terminal with the `man ssh` command.
One of the most useful SSH tricks I've ever learned. I wonder if there's a way to detect a stale session and force reconnect when I turn on the computer? Like mosh[0], but with SSH.
[0]: https://mosh.org/
[0]: https://mosh.org/
I wonder if there's a way to detect a stale session and force reconnect
SSH uses TCP and if the session is gone it will be an invalid session in iptables/nftables and likely have timed out on the remote end depending on state table timeouts and how long your laptop were offline. If there were no firewall in the path then one could play with long SO_KEEPALIVE sessions which I have done in the past when rebooting datacenter-wide diskless NFS clients and NAS's but I dont believe this will work with SSH due to session keys. As you alluded to, Mosh is the best current way to deal with broken or roaming sessions as Mosh uses a nonce and is designed to be stateless.
If the sshd and ssh client timeouts are high enough, a UDP VPN can at times work around intermittent timeouts.
SSH uses TCP and if the session is gone it will be an invalid session in iptables/nftables and likely have timed out on the remote end depending on state table timeouts and how long your laptop were offline. If there were no firewall in the path then one could play with long SO_KEEPALIVE sessions which I have done in the past when rebooting datacenter-wide diskless NFS clients and NAS's but I dont believe this will work with SSH due to session keys. As you alluded to, Mosh is the best current way to deal with broken or roaming sessions as Mosh uses a nonce and is designed to be stateless.
If the sshd and ssh client timeouts are high enough, a UDP VPN can at times work around intermittent timeouts.
> "As you _eluded_ to"
should be "alluded to"; eluded: evaded / dodged, vs alluded to: mentioned / referred to
NOT being pedantic about spelling, just trying to be helpful (esp. for non-native English readers).
should be "alluded to"; eluded: evaded / dodged, vs alluded to: mentioned / referred to
NOT being pedantic about spelling, just trying to be helpful (esp. for non-native English readers).
Thanks. I'm sure I will get it wrong again but I try. Another one that my subconscious types out wrong is queue vs cue out of habit. Fixed in above comment.
You can turn on TCP keeyalive on ssh using a handful of -o flags if that's the behavior you prefer.
https://news.ycombinator.com/item?id=5017108
https://news.ycombinator.com/item?id=5017108
Another good tip; when randomly generating passwords (especially for other users), filter out anything starting with a tilde to prevent strange behavior.
Your new password is: +++ATH0
This doesn't work.
The Hayes modem protocol specifies that a significant pause must be inserted after "+++" for it to have signaling effect.
https://en.wikipedia.org/wiki/Hayes_AT_command_set#Hayes'_so...
The Hayes modem protocol specifies that a significant pause must be inserted after "+++" for it to have signaling effect.
https://en.wikipedia.org/wiki/Hayes_AT_command_set#Hayes'_so...
... that link describes why it does sometimes work
Being able to change port forwarding without reconnecting is really useful too.
I wish there is an easy way to have the reverse: a “sticky” ssh session.
It’s so annoying that the connection is lost when going to sleep or network issues. And the solutions to fix this are not really worth the effort.
It’s so annoying that the connection is lost when going to sleep or network issues. And the solutions to fix this are not really worth the effort.
I run ssh in a loop and auto-attach to a persistent tmux session on the other end. I also kill ssh processes as part of suspend so that the new one gets launched in the loop as soon as we wake up.
WireGuard[1] might help here, at least with network issues. It's literally designed to be able to keep a connection even when moving to one network to another ("roaming").
And as a side-benefit, if your SSH daemon only listens on the WireGuard interface, that's another layer of defense you get for free (not to mention you'll stop getting noise in your logs).
Ironically though, here you actually need to know about `<Enter>~.` because if the remote host actually goes down, WG will keep trying to contact the remote peer for some time; this is the same behavior that allows you to keep a connection open even when roaming, but seen from the other side.
[1]: https://www.wireguard.com/
And as a side-benefit, if your SSH daemon only listens on the WireGuard interface, that's another layer of defense you get for free (not to mention you'll stop getting noise in your logs).
Ironically though, here you actually need to know about `<Enter>~.` because if the remote host actually goes down, WG will keep trying to contact the remote peer for some time; this is the same behavior that allows you to keep a connection open even when roaming, but seen from the other side.
[1]: https://www.wireguard.com/
et survives reboots and IP roaming, pretty much anything: https://eternalterminal.dev/
I’m not sure about resuming from sleep, but does mosh address your network stability issues?
https://mosh.org/
https://mosh.org/
The main thing about Mosh is you need both on the sever and the client. Installing on random servers you might be ssh -in only once feels gross.
Has Mosh crypto been reviewed? Last I checked they were using some custom crypto on top of UDP instead of using something like DTLS or QUIC. Given SSH is one of the most battle tested protocols out there I am wary of replacing it with something else.
https://mosh.org/#faq:~:text=Q%3A%20What%20is%20Mosh%27s%20s...
The cryptography is standard AES-128 in OCB3 mode. It's been around long enough, and has had enough security scrutiny to at least discover a few minor DoS vulnerabilities, that it isn't entirely unreviewed.
For the cipher itself, see https://en.wikipedia.org/wiki/OCB_mode#Attacks
The cryptography is standard AES-128 in OCB3 mode. It's been around long enough, and has had enough security scrutiny to at least discover a few minor DoS vulnerabilities, that it isn't entirely unreviewed.
For the cipher itself, see https://en.wikipedia.org/wiki/OCB_mode#Attacks
> QUIC
Mosh has been effectively unmaintained since long before QUIC even existed. It should be rewritten to use QUIC, except that QUIC's requirement for TLS certificates rather than generic asymmetric cryptography basically breaks any integrations like this.
Mosh has been effectively unmaintained since long before QUIC even existed. It should be rewritten to use QUIC, except that QUIC's requirement for TLS certificates rather than generic asymmetric cryptography basically breaks any integrations like this.
It was AES-OCB last time I looked at it. Not sure if this is good/bad.
Also very helpful: There is ~C which opens a "command" mode where you can add local or remote forwardings after you opened a SSH session (e.g. -L8080:localhost:8080)
Probably too late to post here but the timing of this article was interesting. I use the ~. on a daily basis because something in the firewall at work was causing my idle ssh connections to hang. But just yesterday morning right before reading this article I added ServerAliveInterval 30 in the client ssh config to see if it would help and guess what, the connections were not stuck today.
You might also want to set ServerAliveCountMax to something high, the default is 3 which means if your network is down for 3*30s the SSH client will disconnect.
Pretty sure the answer is no, cause I've been searching for a way to do this for a while, but is there any way to trigger these escapes from script?
Use case is bouncing through to an RDS that only allows access from specific EC2 instances. The RDS endpoint in question is highly specific to the EC2 - not a big deal to hit ~L and create the forward manually, but doing this automatically would be great.
Use case is bouncing through to an RDS that only allows access from specific EC2 instances. The RDS endpoint in question is highly specific to the EC2 - not a big deal to hit ~L and create the forward manually, but doing this automatically would be great.
Can you not use SSH proxyjump?
https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump
https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump
Yup. Though doesn't help much for my use case (I don't think) - I have an ever change set of ec2's, each with access to a single RDS endpoint.
Most debug/config is done via sqlcl which is straight forward as that's run from the ec2 and so has access to the DB, but sometimes I need to fall back to eg TOAD or sqldeveloper which require the forward to be setup - either adhoc via ~L or -L or via .ssh/config
Each ec2 knows it's RDS end point and I have an extracted list which I use to generate a list of .ssh/config entries periodically to automatically set up the forwards, but just being able to run a command from the ec2s that translates to eg:
~L > 1521:<extracted rds endpoint from app config:1521
would make my life marginally easier
Most debug/config is done via sqlcl which is straight forward as that's run from the ec2 and so has access to the DB, but sometimes I need to fall back to eg TOAD or sqldeveloper which require the forward to be setup - either adhoc via ~L or -L or via .ssh/config
Each ec2 knows it's RDS end point and I have an extracted list which I use to generate a list of .ssh/config entries periodically to automatically set up the forwards, but just being able to run a command from the ec2s that translates to eg:
~L > 1521:<extracted rds endpoint from app config:1521
would make my life marginally easier
ssh only looks for those escape sequences on the input side / on the controlling pty.
I think you would have to do something like have the script on the remote ec2 instance emit some pattern that you configure your local terminal emulator to watch for and then somehow tell your terminal to emit the ~L sequence of keystrokes.
iterm2 can do this kind of thing, see https://iterm2.com/triggers.html .
I think you would have to do something like have the script on the remote ec2 instance emit some pattern that you configure your local terminal emulator to watch for and then somehow tell your terminal to emit the ~L sequence of keystrokes.
iterm2 can do this kind of thing, see https://iterm2.com/triggers.html .
Maybe take a look at expect:
https://linux.die.net/man/1/expect
https://linux.die.net/man/1/expect
Adding for completeness sake, if using Alpine Linux the ssh escape sequence menu and output will not display correctly if using /bin/ash from BusyBox as your login shell. One work-around is to type 'cat [Return]' and then use the escape sequences, or to carefully change your shell to /bin/bash
With all that glory of Alpine, my personal choice is to avoid having it on server side, it's too much "if" with it to keep in my memory.
Alpine Linux user here, i could not reproduce that. How does the used shell even matter for that? The ssh client talks to the tty directly, not via shell.
This thread [1] explains the issue people run into. I ran into it and thus found that explanation. Verified I can still reproduce it on Alpine 3.17.3 up to date on patches
For clarification regarding what I mean with menu and output will not display the disconnect command will still work but there will be no feedback. So disconnecting will work but one won't get the menu or feedback on changing verbosity or dropping to a command line or displaying forwarded connections, etc... and a few of the sub-commands will not work.
[1] - https://superuser.com/questions/985437/ssh-escape-key-only-w...
For clarification regarding what I mean with menu and output will not display the disconnect command will still work but there will be no feedback. So disconnecting will work but one won't get the menu or feedback on changing verbosity or dropping to a command line or displaying forwarded connections, etc... and a few of the sub-commands will not work.
[1] - https://superuser.com/questions/985437/ssh-escape-key-only-w...
Due to my use of the ControlMaster feature, I usually just do `ssh -O exit foo` in another terminal to close an existing connection. Either that or just close the terminal and open a new one.
I would love the opposite, how to keep a SSH connection alive when sleeping the computer? XD
Using a tmux or screen session to maintain persistence instead might be a better idea, depending on your use case.
On a desktop machine, sure... but on a laptop, I'd want it to enter sleep and subsequently deep sleep when I close the lid or lock it.
tilde dot is my jam.
that typeface makes the tilde prefix (~) almost looks like a hyphen (-)
Why is this better than CTRL-C?
Generally ssh will just forward signals (SIGINT, SIGQUIT) to the remote host. If that side is not responsive, you can hit Ctrl+C all you want, but it won't do anything.
It's client-side, so works even if the remote system is totally hung and did not clearly disconnect.
For example, running `systemctl suspend` will not terminate active SSH connections before putting the destination machine into a sleep state, and thus Ctrl+C (which isn't processed by SSH) will do nothing until the remote host is woken up by some mechanism.
For example, running `systemctl suspend` will not terminate active SSH connections before putting the destination machine into a sleep state, and thus Ctrl+C (which isn't processed by SSH) will do nothing until the remote host is woken up by some mechanism.
Control-C won't stop a connection that's already been established. It only aborts if it hasn't completed the handshake yet.
Otherwise how could you send a control-C without any special guard sequence while ssh-ing?
Otherwise how could you send a control-C without any special guard sequence while ssh-ing?
[deleted]
If you enjoyed this article, you may also find other useful tips in a piece I submitted to HN a while ago on "advanced SSH usage": https://johannes.truschnigg.info/writing/2022-07_advanced_ss...
Loved your article, thank you for writing it !
If you enjoyed this article, you may like to learn about man pages... Try "man man".
> (Note that escapes are only recognized immediately after newline.)
This means that it is easy to pick up a habit to smash the enter-button a few times before doing this dance, and as noted on a nordic layout it can be a bit tricky and since you might seldom do it you might do it a few times.
Problem can be that sometimes the connection is only broken one way, what you are typing goes to the server but the responses don't. So you might end up wreaking some kind of havoc on the remote server when you just want to kill the session. Maybe you had a half-written command. Maybe you had just done an up-arrow to get to the previous command, maybe you redid that up-arrow one or more times before you realized that the connection was broken. If you press enter now you will re-run one of your previous commands. Could be quite scary.
To save you from some of that, you could do a ctrl+c which will clear your current line, before pressing enter. But whether that is a good idea depends on the context...
The most apparent issue this has been for me is if on the remote you have IRC or something and you type a bunch of garbage to whatever channel you are on. No biggie, but the old restart the terminal isn't too bad either.