Making an SSH client the hard way(tailscale.com)
tailscale.com
Making an SSH client the hard way
https://tailscale.com/blog/ssh-console/
107 comments
You‘d have to redirect all network usage (i.e. the sockets API or your platform‘s equivalent) through the custom stack, which is possible if you can rebuild the source or by using something like LD_PRELOAD for binaries, but can get very tricky in the general case.
There‘s an utility called SSHuttle that does something similar for SSH instead of Tailscale/Wireguard: It redirects all sockets usage to go through an SSH connection, to allow usage of SSH port forwarding without explicit SOCKS support on the app‘s side.
There‘s an utility called SSHuttle that does something similar for SSH instead of Tailscale/Wireguard: It redirects all sockets usage to go through an SSH connection, to allow usage of SSH port forwarding without explicit SOCKS support on the app‘s side.
SSHuttle was created by the CEO/co-founder of Tailscale :)
Interesting. Sshuttle is great. I've used that a couple of times to create a simple vpn without having to deal with openvpn. Simple and effective.
that's a bit worrying
there's a rather large misunderstanding on its github page:
> You can't use openssh's PermitTunnel feature because it's disabled by default on openssh servers; plus it does TCP-over-TCP, which has terrible performance.
it doesn't do TCP over TCP, it's a bytestream over TCP (exactly the same as shuttle)
something like OpenVPN running in TCP mode would be TCP over TCP
there's a rather large misunderstanding on its github page:
> You can't use openssh's PermitTunnel feature because it's disabled by default on openssh servers; plus it does TCP-over-TCP, which has terrible performance.
it doesn't do TCP over TCP, it's a bytestream over TCP (exactly the same as shuttle)
something like OpenVPN running in TCP mode would be TCP over TCP
Are you sure you are not misunderstading PermitTunnel yourself?
That SSH feature is indeed used for packet forwarding over SSH using TUN/TAP, i.e. packet-over-TCP, and by extension TCP-over-TCP.
That SSH feature is indeed used for packet forwarding over SSH using TUN/TAP, i.e. packet-over-TCP, and by extension TCP-over-TCP.
Indeed, both the name and the very clever combination of existing components sounded familiar when I first heard about Tailscale :)
. You can use tools like proxychains to leverage LD_PRELOAD to redirect all network traffic via a socks5 proxy.
Tailscaled comes with a socks5 proxy
Tailscaled comes with a socks5 proxy
Could the Tailscale client be packaged as an extension so I can visit sites on my Tailnet without having to install a client? Sometimes I want to visit a "internal" site without having to install the client, if I'm using a temporary box for something. I'm not sure how much more work would have to be done, might have to dig into the open source pieces of this.
Putting my vote in for this feature as well... Also, why not compile VNC into WASM and get full remote desktop experience for graphical apps. It seems that hard work has already been done!
> Also, why not compile VNC into WASM and get full remote desktop experience for graphical apps. It seems that hard work has already been done!
Yup. :)
In fact, that's mentioned in the original public bug: https://github.com/tailscale/tailscale/issues/3157
Yup. :)
In fact, that's mentioned in the original public bug: https://github.com/tailscale/tailscale/issues/3157
This is really cool and fun, but is this a safe way to run SSH clients?
If, say, the adblock Chrome extension you're using gets bought by a malware operator and backdoored[0], now it also has SSH and VPN access.
[0]: https://www.wired.co.uk/article/fake-chrome-extensions-malwa...
If, say, the adblock Chrome extension you're using gets bought by a malware operator and backdoored[0], now it also has SSH and VPN access.
[0]: https://www.wired.co.uk/article/fake-chrome-extensions-malwa...
The Tailscale VPN client, the same one which runs on other devices, is compiled to WASM. It handles all of the key exchanges to connect to the tailnet. The SSH session is running as a WASM Tailscale client.
The browser, opening connections from within the browser engine, doesn't have the keys for SSH or VPN access.
The browser, opening connections from within the browser engine, doesn't have the keys for SSH or VPN access.
Sounds like about "as good as this gets" if you happen to want to do this in browsers. Good job.
It doesn’t have the keys, but it can inject any javascript and do whatever the user can do.
To me it seems they've taken all precautions they can reasonably take -- "what if the user installs a keylogger" isn't fixable by anyone.
"Installing a keylogger" is a vast oversimplification, even if it is outside of their threat model.
Installing almost anything in your browser is usually a matter of a couple of clicks.
Installing almost anything in your browser is usually a matter of a couple of clicks.
Like intercepting your oauth token next time you login into SSO and then use that to access your tailnet.
This was true even before this new feature.
The new threat model is entirely psychological.
This was true even before this new feature.
The new threat model is entirely psychological.
The wasm code runs in the browser, hence the keys live in browser memory, so the browser has access to the keys.
And then the addon intercepts the loading of the Wasm code, injects it’s own payload into it and has access to the keys.
What keys? I think the implementation does not use regular SSH keys for SSH authentication, but rather something custom (I believe traffic to port 22 on each SSH enabled client is intercepted and the daemon handles authentication itself).
For the downvotes: Am I misunderstanding Tailscale's implementation?
I'm not commenting on whether it's a good or bad idea, but we should at least be talking about the same thing.
I'm not commenting on whether it's a good or bad idea, but we should at least be talking about the same thing.
citation needed
If you have an actively malwared extension, there's probably an easier way to exploit any given target-- the simplest being recording passwords.
There's not really a "safe" website when the browser is malicious.
There's not really a "safe" website when the browser is malicious.
This is turning a remote server into a previously "safe website" and expanding the threat model to include remote web browser attacks.
I believe that replacing vscode.exe that you just downloaded is entirely possible with extension. So if you’ve got compromised browser, it’s gotta be hard to stay safe elsewhere. Lots of attack vectors. Better don’t get infected.
>To make this possible, we ported the following to WebAssembly: the Tailscale client, WireGuard®, a complete userspace network stack (from gVisor), and an SSH client.
Wow, and they're proud of it.
Wow, and they're proud of it.
It's a pretty good approach and pretty hard to pull off. I'd be proud of it too.
Cool feature! I was just looking at boringtun last night and wondering if it could compile to WASM, to get a virtualized network interface in the browser.
Did you experiment with the new WebTransport API [0] at all? It's only supported in Chromium browsers, but seems promising for this kind of use case.
[0] https://chromestatus.com/feature/4854144902889472
Did you experiment with the new WebTransport API [0] at all? It's only supported in Chromium browsers, but seems promising for this kind of use case.
[0] https://chromestatus.com/feature/4854144902889472
I can't shake the feeling that Tailscale's SSH authentication mechanism is at the wrong layer of the stack. It appears to work by looking at the (source, dest) IP address pair and mapping that to a Tailscale identity. But this may mean that any user or anyone who can initiate TCP connections from an authenticated user's IP can authenticate to the destination over Tailscale SSH.
If Tailscale's client was a userspace construct bound to a specific user SSH program, maybe fine. But Tailscale's client is a regular VPN client. What happens if you connect to the Tailscale VPN, open a malicious but sandboxed app of some sort, and that app connects to the target on TCP port 22.
For all that it's a seriously unfinished product, Cloudflare's SSH offering seems better thought out. Perhaps Tailscale should find a way to issue a short-lived certificate and use that in addition?
(It looks like regular sshd could almost be convinced to handle this. If the SSH_CONNECTION environment variable were passed to the AuthorizedPrincipalsCommand helper or if the source and destination were available as '%' tokens, then AuthorizedPrincipalsCommand could do the Tailscale tuple lookup and use it as a second factor in addition to a short-lived certificate (or regular SSH key or whatever). I bet openssh would accept a patch for this.)
If Tailscale's client was a userspace construct bound to a specific user SSH program, maybe fine. But Tailscale's client is a regular VPN client. What happens if you connect to the Tailscale VPN, open a malicious but sandboxed app of some sort, and that app connects to the target on TCP port 22.
For all that it's a seriously unfinished product, Cloudflare's SSH offering seems better thought out. Perhaps Tailscale should find a way to issue a short-lived certificate and use that in addition?
(It looks like regular sshd could almost be convinced to handle this. If the SSH_CONNECTION environment variable were passed to the AuthorizedPrincipalsCommand helper or if the source and destination were available as '%' tokens, then AuthorizedPrincipalsCommand could do the Tailscale tuple lookup and use it as a second factor in addition to a short-lived certificate (or regular SSH key or whatever). I bet openssh would accept a patch for this.)
I agree.
I feel like there should be a lightweight way to use SSH certificates, so you could use it independently or on top of Tailscale. Like a server (CA), client and daemon on your machines that should be reachable via SSH that handles short lived certs and authentication of clients. But I'm not aware of anything like that.
I have used Teleport before but it seemed not that great for machines not publicly reachable, because then all the traffic goes through a proxy.
I feel like there should be a lightweight way to use SSH certificates, so you could use it independently or on top of Tailscale. Like a server (CA), client and daemon on your machines that should be reachable via SSH that handles short lived certs and authentication of clients. But I'm not aware of anything like that.
I have used Teleport before but it seemed not that great for machines not publicly reachable, because then all the traffic goes through a proxy.
This is how Okta's Advanced Server Access works: https://www.okta.com/products/advanced-server-access/
Smallstep and cloudflared do this. Sadly, both of them seem to use essentially identical client-side hacks. Smallstep is a small company that I wouldn’t trust with the keys to the kingdom, and Cloudflare seems to treat their SSH product as something thrown over the fence with nothing resembling support.
Gravitational’s Teleport seems pretty good, but it’s heavyweight and doesn’t have any pricing appropriate for small businesses.
Gravitational’s Teleport seems pretty good, but it’s heavyweight and doesn’t have any pricing appropriate for small businesses.
I'm mildly curious what client side hacks you're talking about?
https://developers.cloudflare.com/cloudflare-one/identity/us...
The configuration is:
Host vm.example.com ProxyCommand bash -c '/usr/local/bin/cloudflared access ssh-gen --hostname %h; ssh -tt %[email protected] >&2 <&1'
Host cfpipe-vm.example.com HostName vm.example.com ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h IdentityFile ~/.cloudflared/vm.example.com-cf_key CertificateFile ~/.cloudflared/vm.example.com-cf_key-cert.pub
Think about this a bit. Contemplate what happens if you use sftp, vscode remoting, or anything else nontrivial. Hint: “cloudflared access ssh-gen” is not actually any sort of proxy, and the ssh -tt command is a kludge that should, if openssh were more on the ball about inherited file descriptors, should not work at all.
The right way to to this is to use Match … exec. Or to ask openssh to add an option for a command to execute before reading IdentityFile. Or to ask for an IdentityFileCommand option. Or to use a custom ssh agent.
The configuration is:
Host vm.example.com ProxyCommand bash -c '/usr/local/bin/cloudflared access ssh-gen --hostname %h; ssh -tt %[email protected] >&2 <&1'
Host cfpipe-vm.example.com HostName vm.example.com ProxyCommand /usr/local/bin/cloudflared access ssh --hostname %h IdentityFile ~/.cloudflared/vm.example.com-cf_key CertificateFile ~/.cloudflared/vm.example.com-cf_key-cert.pub
Think about this a bit. Contemplate what happens if you use sftp, vscode remoting, or anything else nontrivial. Hint: “cloudflared access ssh-gen” is not actually any sort of proxy, and the ssh -tt command is a kludge that should, if openssh were more on the ball about inherited file descriptors, should not work at all.
The right way to to this is to use Match … exec. Or to ask openssh to add an option for a command to execute before reading IdentityFile. Or to ask for an IdentityFileCommand option. Or to use a custom ssh agent.
A step-up prompt or notification wouldn't be a bad idea to approve ssh connections before they are established -- This is the same issue with ssh agents. I think there's an ACL setting in tailscale where you check ssh connections with a re-auth. it's time-based with a minimum of 1 minute though.
[deleted]
Tailscale SSH's check mode (https://tailscale.com/kb/1193/tailscale-ssh/#configure-tails...) is meant to address the issue of "rogue process starts an SSH connection". For truly sensitive applications, you can set the check period to be "1s" to always require it.
Hmm. If the problem is that Tailscale SSH doesn’t strongly associate the person authenticating with the connection being authenticated, asking the person to reauthenticate seems like a pretty weak solution.
Unless I'm misunderstanding something, the check solution creates that strong association. Logging in gives you a link you have to go to and auth, authing let's your session connect. Disconnect, and you have to do this again.
No check mode reuses the auth of the tailscale client, check mode authenticates the ssh connection itself
No check mode reuses the auth of the tailscale client, check mode authenticates the ssh connection itself
And then that tailscale client is authorized to log in over ssh for however long the check lasts, and anyone else who can initiate a TCP session over the link can also get in.
This significantly increases the threat model for your remote servers to include all sorts of remote attacks through the web, including:
Realistically, this essentially (actually, literally) opens a remote root shell into your entire infrastructure through a web page, with apparently nothing more than matching an IP address pair (https://news.ycombinator.com/item?id=33361837) to authenticate.
What could go wrong?
This design with its loose coupling between authenticated user and IP addresses for high-value targets makes me view Tailscale's security model in a whole new light.
* garden-variety web attacks (i.e., XSS, CRSF, etc)
* attacks that might become viable against the browser (for example, Mobile Safari has a history of vulnerabilities)
* various attacks against the backend web server (API attacks)
* attacks against the WASM layer
* CDN injections
* Tailscale's backend (various types of injections, timing attacks, or deeper attacks on Tailscale's infrastructure like the nightmares of HeartBleed, Shellshock, Meltdown, etc)
Realistically, this essentially (actually, literally) opens a remote root shell into your entire infrastructure through a web page, with apparently nothing more than matching an IP address pair (https://news.ycombinator.com/item?id=33361837) to authenticate.
What could go wrong?
This design with its loose coupling between authenticated user and IP addresses for high-value targets makes me view Tailscale's security model in a whole new light.
This seems to really expand the scope of how keys are handled and authorized. It at least appears to be optional, however.
Or just use userify's public key distribution model (then you can stick with a simple and proven design that just automates the standard SSH design that's been around for decades)
Or just do it the old way, by hand: drop your teams' public keys on groups of remote servers, setup their sudo access, and you're done.
Or just use Ansible or Chef or Terraform, or any of those other server orchestration tools, too (just without the permission layer/bulk remote session kills/user deletes of Userify).
of course, if you really need an ssh terminal in a browser, then you have to go with one of these sorts of crazy things, and at least it's not using passwords!
Or just use userify's public key distribution model (then you can stick with a simple and proven design that just automates the standard SSH design that's been around for decades)
Or just do it the old way, by hand: drop your teams' public keys on groups of remote servers, setup their sudo access, and you're done.
Or just use Ansible or Chef or Terraform, or any of those other server orchestration tools, too (just without the permission layer/bulk remote session kills/user deletes of Userify).
of course, if you really need an ssh terminal in a browser, then you have to go with one of these sorts of crazy things, and at least it's not using passwords!
[deleted]
That's unnecessary sensationalism. Most of those vectors are behind an SSO login and are not exposed to Internet at all (from the article: "your browser becomes a Tailscale client, and joins your tailnet in the same way as any other device that you run Tailscale on").
Or, did you mean attacks on SSO? If that's the case, then SSH web wouldn't make any difference. Someone authenticating themselves could use regular SSH or whatever.
Similarly, Tailscale backend is already subject to the vectors you mentioned (API, side-channel attacks). This feature doesn't add any new attack vectors.
Again, attacks on browser means end of game already. Someone can use that vector to access to your local network in other ways. They don't need Tailscale's SSH web client for that.
Or, did you mean attacks on SSO? If that's the case, then SSH web wouldn't make any difference. Someone authenticating themselves could use regular SSH or whatever.
Similarly, Tailscale backend is already subject to the vectors you mentioned (API, side-channel attacks). This feature doesn't add any new attack vectors.
Again, attacks on browser means end of game already. Someone can use that vector to access to your local network in other ways. They don't need Tailscale's SSH web client for that.
> Again, attacks on browser means end of game already.
A bad Chrome extension does not allow the bad guys to open a terminal on my machine, load my ssh keys, launch an authenticated SSH connection, and launch an authenticated SSH connection into an enumerated list of remote servers.
A bad Chrome extension does not allow the bad guys to open a terminal on my machine, load my ssh keys, launch an authenticated SSH connection, and launch an authenticated SSH connection into an enumerated list of remote servers.
A malicious browser extension can access your email, SSO prompts, password manager, etc, and therefore gain access to your Tailscale network anyway. SSH web doesn't add a new threat vector here. It's already game over.
A browser extension cannot access your email or password manager. But it can use this new security hole created by tailscale.
If the extension in question has the read/modify all websites permission, why would it not be able to access your email or password manager?
Not OP, but...my email and password managers are not websites. I have a local app that does email, I have a local app that does password management. So the extension could certainly access passwords I put into my browser, but I don't see a vector to the "keys to the kingdom" so to speak.
The most popular password managers (1Password, Bitwarden, Lastpass) have a web UI.
Sure, and I use Bitwarden. I just don't use the web UI, specifically to avoid issues like malicious extensions, Firefox exploits, etc.
I'm aware that web-based email and credential managers exist, but GP asked "...why would it not be able to access your email or password manager?" I answered that, with my app choice, I don't see how they could.
I'm aware that web-based email and credential managers exist, but GP asked "...why would it not be able to access your email or password manager?" I answered that, with my app choice, I don't see how they could.
Who's using extensions in a work context anyway? I only use an ad-blocker because it's safer from a malware perspective.
> Who's using extensions in a work context anyway?
literally everyone?
literally everyone?
React Developer Tools, for example.
Ah, thanks. I'm more on sysadmin side for my day job, and didn't even know about this. I'll have to keep it in mind as I've been dabbling with learning a modern web framework and React was a candidate.
Well, this exists: https://engineering.fb.com/2022/03/10/security/code-verify/
[deleted]
(Tailscale employee)
It is not a web page with a shell open.
It is the Tailscale client, compiled to WASM, maintaining the keys for connections to nodes on the tailnet. Connections opened from the browser engine don't get the ability to reach the tailnet.
It is not a web page with a shell open.
It is the Tailscale client, compiled to WASM, maintaining the keys for connections to nodes on the tailnet. Connections opened from the browser engine don't get the ability to reach the tailnet.
Given the apparent (yet possibly unfounded) concern, it might be useful to discuss this issue in the blog post. Would you consider updating it?
Doesn't the post say exactly what the parent comment did? It explains the process that they came up with the WASM Tailscale, and how it authorizes and works. If anything, the parent comment just paraphrased the article.
It doesn't paragraph, this is the important sentence
> Connections opened from the browser engine don't get the ability to reach the tailnet.
That was not really explained in the article. Maybe it's obvious to some people, but I'm sure it's not for many.
> Connections opened from the browser engine don't get the ability to reach the tailnet.
That was not really explained in the article. Maybe it's obvious to some people, but I'm sure it's not for many.
I wouldn't be the target market for this, however more power to them. I understand that to an extent we should never roll our own, so to speak, however, I think that we should not put all of our eggs in one basket. In that regard, I think that once the beginning kinks are ironed out it will be a better thing and we should ultimately embrace these types of endeavors.
Let me say again, though I admire it, I'd never use this. I like to sleep soundly, as irrational as that may be.
Let me say again, though I admire it, I'd never use this. I like to sleep soundly, as irrational as that may be.
I just learned there's a POSIX kernel in Go that can actually run apps.
https://www.usenix.org/conference/osdi18/presentation/cutler
I think this should be compiled with WASM and deployed via web page. Then we can explore the idea of browser-hosted POSIX kernels.
I think this should be compiled with WASM and deployed via web page. Then we can explore the idea of browser-hosted POSIX kernels.
> To make this possible, we ported the following to WebAssembly: the Tailscale client, WireGuard®, a complete userspace network stack (from gVisor), and an SSH client.
I love that they were clearly inspired by fly.io. Warms my heart that a random blog post with a good idea can spread like this.
I love that they were clearly inspired by fly.io. Warms my heart that a random blog post with a good idea can spread like this.
Which blog post are you referring to?
But yes, we love Fly and use them (and they use us) and we share a slack channel between our two companies for casual banter.
But yes, we love Fly and use them (and they use us) and we share a slack channel between our two companies for casual banter.
I mean if you want to go back, this is the original post from the horse’s mouth:
https://lists.zx2c4.com/pipermail/wireguard/2021-January/006...
https://lists.zx2c4.com/pipermail/wireguard/2021-January/006...
Hi Mihai! Great work! I would love to see where this goes!
Forgive my ignorance but is there any sort of native client besides the browser running in the background to help with websocket to tcp? Or a tunnel to a cloud service to help there?
Forgive my ignorance but is there any sort of native client besides the browser running in the background to help with websocket to tcp? Or a tunnel to a cloud service to help there?
No native client is running. The browser makes a WebSocket connection to our relay server, and we run the WireGuard tunnel over that.
You mean WireGuard (in userspace w/ netstack) tunneled in WebSockets to DERPServers, correct? What sorcery is this. (:
In the old days, people said you shouldn't write crypto in JavaScript because it was somehow insecure. Have those concerns gone away with WebAssembly and https everywhere?
I think the consensus is still that you can't write side-channel/timing proof crypto in (most) Javascript (runtimes) - but that with webcrypto(?) most runtimes will provide the secure crypto primitives you need in order to do (secure) crypto with Javascript?
To add, certain concerns still stand since the Web crypto API is extremely limited in capabilities (Example it doesn't support secp256k1).
However on the bright side performance.now() has a much worse resolution on most browsers, making timing attacks harder.
However on the bright side performance.now() has a much worse resolution on most browsers, making timing attacks harder.
It will never go away because you can't guarantee constant time algorithms will be implemented as such when transformed by a JIT.
If machine code can issue the necessary hints to the hardware to skip all time-variant optimizations, why couldn‘t the same work for a WASM runtime?
It could, but you need a lot of work on a lot of layers, possibly routed through an entire standards committee, to get it done. If it ever happens it's going to be a long time.
I don't think the argument was ever that "JavaScript was insecure." It was that the websites hosting it may be compromised or may change the script at any time without any indication (or the FBI forcing a site to backdoor the JS for some investigation)
Yeah, but see also Code Verify (and sub-resource integrity): https://blog.cloudflare.com/cloudflare-verifies-code-whatsap...
Both javascript and browsers have been hardened a lot since then. Back in the day people were using javascript on plain http websites, without https. Cross site scripting prevention in browsers simply did not exist.
Nowadays, it's a lot harder to get past browser security and people run all sorts of applications in browsers such as banking, business critical SAAS, email, etc. So, perfectly fine to include some crypto in that and probably not optional to do so for a lot of applications.
What's still true is that you should not be rolling your own crypto libraries and instead use libraries from reputable sources that have been scrutinized by people that know what they are doing. That's true whether you use javascript, C or whatever. And of course with web assembly you can just compile those libraries and use them in a browser sandbox.
Nowadays, it's a lot harder to get past browser security and people run all sorts of applications in browsers such as banking, business critical SAAS, email, etc. So, perfectly fine to include some crypto in that and probably not optional to do so for a lot of applications.
What's still true is that you should not be rolling your own crypto libraries and instead use libraries from reputable sources that have been scrutinized by people that know what they are doing. That's true whether you use javascript, C or whatever. And of course with web assembly you can just compile those libraries and use them in a browser sandbox.
Interesting, I could see this as a nice replacement for Gravitational Teleport.
One of the things that Teleport lacks (IMO) is Wireguard
One of the things that Teleport lacks (IMO) is Wireguard
Will Tailscale SSH support FreeBSD any time soon?
Most of the products from tailscale just seem to be "look at the inner platforms we can build that replace the outer platforms".
Having an SSH client in your browser join your VPN violates all the principles of modern computing.
Having an SSH client in your browser join your VPN violates all the principles of modern computing.
> Having an SSH client in your browser join your VPN violates all the principles of modern computing.
I think that's a compliment? You're welcome? :)
I think that's a compliment? You're welcome? :)
absolutely. the lines weren't bad, but they were off in places and now they are set in stone for no good reason.
edit: _especially_ when it comes to security
edit: _especially_ when it comes to security
It's not a compliment. Or rather, within my understanding of how things should be architected, it's not. I certainly wouldn't claim that my own beliefs about network architecture should trump others, and I work in a different domain from most of the people with your use case. Whether the disruptive work you're doing is good for the world in the long run is still a very open question in my mind.
I used to think that everything in the world should move to the browser (in my case, that would be high performance molecular graphics and microscope control) and ChromeOS was the logical extension. Web Assembly to handle the existing C++ codebases, a collection of standard web tech to handle the user interface. Big fan of SSH extension because it meant that machines that only ran a browser could be useful command line programming terminals. But didn't like that SSH extension was written in a dead-end container technology (NaCl) and the CHrome folks sort of messed up multiple ways for the extension to integrate better.
But after working with web tech for enough time I came to conclude that putting things in the browser like this is an antipattern, in particular increasing the surface complexity of the software space while not actually replacing existing systems (openssh continues to exist, OS-level VPNs continue to exist, even after you port the VPN and client to web assembly and run it in a browser).
In my mental model, it makes more sense to put the browser in a VM and then wire the VM's networking to a VPN handled by the OS, rather than putting what is more or less a significant fraction of a virtual machine manager's capabilities into the browser. If you're going to do that, why not go whole-hog and add a VM to chrome so that it can run linux with a full networking stack and then host an SSH client inside that? So you can run linux in your chrome in your linux.
My compliment is: I am impressed at how well you parlayed several technical projects into a thought leadership position, but we have fundamentally different architectural principles and work in different domains. Your work disrupts mine, but mine doesn't disrupt yours. My enterprise actually disallows me from visiting your company's website on my work computer because users installing their own VPNs is considered a security risk (fwiw, I bought into BeyondCorp, which eschews VPNs, a long time ago, and would prefer my enterprise eliminate VPNs, as they don't really protect our users).
I used to think that everything in the world should move to the browser (in my case, that would be high performance molecular graphics and microscope control) and ChromeOS was the logical extension. Web Assembly to handle the existing C++ codebases, a collection of standard web tech to handle the user interface. Big fan of SSH extension because it meant that machines that only ran a browser could be useful command line programming terminals. But didn't like that SSH extension was written in a dead-end container technology (NaCl) and the CHrome folks sort of messed up multiple ways for the extension to integrate better.
But after working with web tech for enough time I came to conclude that putting things in the browser like this is an antipattern, in particular increasing the surface complexity of the software space while not actually replacing existing systems (openssh continues to exist, OS-level VPNs continue to exist, even after you port the VPN and client to web assembly and run it in a browser).
In my mental model, it makes more sense to put the browser in a VM and then wire the VM's networking to a VPN handled by the OS, rather than putting what is more or less a significant fraction of a virtual machine manager's capabilities into the browser. If you're going to do that, why not go whole-hog and add a VM to chrome so that it can run linux with a full networking stack and then host an SSH client inside that? So you can run linux in your chrome in your linux.
My compliment is: I am impressed at how well you parlayed several technical projects into a thought leadership position, but we have fundamentally different architectural principles and work in different domains. Your work disrupts mine, but mine doesn't disrupt yours. My enterprise actually disallows me from visiting your company's website on my work computer because users installing their own VPNs is considered a security risk (fwiw, I bought into BeyondCorp, which eschews VPNs, a long time ago, and would prefer my enterprise eliminate VPNs, as they don't really protect our users).
(I'm a tailscale cofounder) I think of Tailscale more like a set of tools that lets you do any architecture you want. Nobody has to use Tailscale ssh console, but if you believe in the future of wasm -> apps -> web console -> ssh, now you can have it.
On the other hand, if you believe in the future of OS private network connectivity -> console -> ssh, then you had that already with native Tailscale and Tailscale ssh.
If you believe in OS private network connectivity -> browser -> javascript console -> ssh, then you can do that too, by installing tailscale in the native OS and then the browser can use it.
I actually agree with you, I'm very suspicious about a world where we just move everything into the web browser. But on the other hand, sometimes it's really handy to have that option.
On the other hand, if you believe in the future of OS private network connectivity -> console -> ssh, then you had that already with native Tailscale and Tailscale ssh.
If you believe in OS private network connectivity -> browser -> javascript console -> ssh, then you can do that too, by installing tailscale in the native OS and then the browser can use it.
I actually agree with you, I'm very suspicious about a world where we just move everything into the web browser. But on the other hand, sometimes it's really handy to have that option.
Hi Avery. I think we may have chatted when I worked at Google (you can figure out my username pretty easily).
To be honest I can't evaluate your product at work- to determine whether it helps our users and whether the idea of moving more of the network stack into the application makes sense- because my corporation (a large multinational pharma) disallows us from visiting the entire tailscale website because you sell a VPN product(!) which isn't our standard one. I'd love to change that policy but I'd still want to move to a BeyondCorp world (https w/ auth), not put a VPN in my browser. Or make Tailscale our standard VPN.
I see the point of "it's really handy". That's how we got Javascript which is a cost we now all have to pay.
To be honest I can't evaluate your product at work- to determine whether it helps our users and whether the idea of moving more of the network stack into the application makes sense- because my corporation (a large multinational pharma) disallows us from visiting the entire tailscale website because you sell a VPN product(!) which isn't our standard one. I'd love to change that policy but I'd still want to move to a BeyondCorp world (https w/ auth), not put a VPN in my browser. Or make Tailscale our standard VPN.
I see the point of "it's really handy". That's how we got Javascript which is a cost we now all have to pay.
"https with auth" is fine and good, and obviously the world has been heading in this direction. But I secretly suspect this is because 90%+ of developers nowadays don't know how to hack on any layer below http.
Tailscale is not a typical VPN; it's just a system that attempts to provide beyondcorp-like behaviour at a lower level of the stack, so that you don't have to rewrite all your apps (ssh in this case!) to use https, and don't have to have open ports in your firewall, and don't have to run everything through the cloud if you don't want.
As in my post above, there's more than one way to do it. You can also build traditional-beyondcorp-over-https on top of a Tailscale network, so you get all the improved network connectivity and also all the benefits of a "pure" beyondcorp architecture.
Tailscale is not a typical VPN; it's just a system that attempts to provide beyondcorp-like behaviour at a lower level of the stack, so that you don't have to rewrite all your apps (ssh in this case!) to use https, and don't have to have open ports in your firewall, and don't have to run everything through the cloud if you don't want.
As in my post above, there's more than one way to do it. You can also build traditional-beyondcorp-over-https on top of a Tailscale network, so you get all the improved network connectivity and also all the benefits of a "pure" beyondcorp architecture.
SSH in the web browser is actually the best practice today. Here are some examples of why SSH in your browser actually compliments modern computing:
- An SSO-authenticated web interface, integrated with a host agent on your instances, means you don't have to manage SSH keys.
- If you just need a disposable CLI that inherits permissions from your SSO-authenticated user role, you can do that from a disposable box in a web interface after authenticating via the web interface. Google Cloud Shell is a good example.
- Cloud-native development is easier if developers can just start working on a unified environment, without having to set up & maintain a local environment. Utilizing the web browser avoids the need to consider separate tools and separate methods of network connection.
- SSH'ing to "private" instances is impossible without going through a bastion or VPN. The bastion then becomes a single point of attack, and is hard to maintain and secure. Similarly the VPN is an additional attack vector, maintenance headache, and requires client-side software, configuration, troubleshooting. Instead of deploying a bunch of bastions or setting up a VPN, if you can use the backend control plane through an SSO-authenticated API gateway, along with a backend proxy to internal networks, you can avoid bastions altogether. This is the best practice for Zero-Trust. Google Cloud IAP Proxy is a good example.
The implementation of it, with WebAssembly/WebSockets/WireGuard/DERP, may be lamentable for several reasons. But it probably (I assume?) solves problems that other SSH Web Interfaces didn't. I hate that the web browser has monopolized computing interfaces :) But in this case it seems to solve many problems.
- An SSO-authenticated web interface, integrated with a host agent on your instances, means you don't have to manage SSH keys.
- If you just need a disposable CLI that inherits permissions from your SSO-authenticated user role, you can do that from a disposable box in a web interface after authenticating via the web interface. Google Cloud Shell is a good example.
- Cloud-native development is easier if developers can just start working on a unified environment, without having to set up & maintain a local environment. Utilizing the web browser avoids the need to consider separate tools and separate methods of network connection.
- SSH'ing to "private" instances is impossible without going through a bastion or VPN. The bastion then becomes a single point of attack, and is hard to maintain and secure. Similarly the VPN is an additional attack vector, maintenance headache, and requires client-side software, configuration, troubleshooting. Instead of deploying a bunch of bastions or setting up a VPN, if you can use the backend control plane through an SSO-authenticated API gateway, along with a backend proxy to internal networks, you can avoid bastions altogether. This is the best practice for Zero-Trust. Google Cloud IAP Proxy is a good example.
The implementation of it, with WebAssembly/WebSockets/WireGuard/DERP, may be lamentable for several reasons. But it probably (I assume?) solves problems that other SSH Web Interfaces didn't. I hate that the web browser has monopolized computing interfaces :) But in this case it seems to solve many problems.
I'm fine with ssh in a browser. I used Chrome SSH Extension for many years to connect to a VM running tmux. And I use RDP if I truly need a remote desktop. However, it (browser SSH) not a replacement for, it's an augmentation of, the OS-level ssh client.
Turning this around. Let's take the idea of using WASM to put a full environment in the user's browser. This is a logical idea, after all- WASM exists to make it possible to write applications in Not-Javascript and deploy them in a browser. IE, don't stop with SSH: you should have a web server, a shell, multiprocessing, scripting languages, everything necessary to host VSCode server and a self-hosted compilation toolchain in a browser. Full linux user space in a browser, enough to compile ChromeOS and boot into a browser running linux
What have you achieved? A very expensive (in terms of porting cost, CPU usage, and deployment size) inner platform that does what an OS does already. But it's inside the browser, with a patched version of code (because WASM always trails native apps), with each sub-application maybe linking in its own TCP stack. So it will always trail innovations in desktops, since it's not a full replacement for the existing system. So it makes the world more complicated and exposes more surface areas for security management.
Turning this around. Let's take the idea of using WASM to put a full environment in the user's browser. This is a logical idea, after all- WASM exists to make it possible to write applications in Not-Javascript and deploy them in a browser. IE, don't stop with SSH: you should have a web server, a shell, multiprocessing, scripting languages, everything necessary to host VSCode server and a self-hosted compilation toolchain in a browser. Full linux user space in a browser, enough to compile ChromeOS and boot into a browser running linux
What have you achieved? A very expensive (in terms of porting cost, CPU usage, and deployment size) inner platform that does what an OS does already. But it's inside the browser, with a patched version of code (because WASM always trails native apps), with each sub-application maybe linking in its own TCP stack. So it will always trail innovations in desktops, since it's not a full replacement for the existing system. So it makes the world more complicated and exposes more surface areas for security management.
Talking about inner platforms (and in case there's one person left who hasn't seen it): https://www.destroyallsoftware.com/talks/the-birth-and-death...
https://en.wikipedia.org/wiki/Inner-platform_effect
"""The inner-platform effect is the tendency of software architects to create a system so customizable as to become a replica, and often a poor replica, of the software development platform they are using. This is generally inefficient and such systems are often considered to be examples of an anti-pattern."""
Like I said elsewhere, I'm not completely opposed to the idea of the browser as a complete and fully functional application container for an inner platform. And I want to encourage creative people to try new technologies, especially WASM to explore the idea of "how much can we move to the browser". However, I see the container as the mediator of the network, not the application.
"""The inner-platform effect is the tendency of software architects to create a system so customizable as to become a replica, and often a poor replica, of the software development platform they are using. This is generally inefficient and such systems are often considered to be examples of an anti-pattern."""
Like I said elsewhere, I'm not completely opposed to the idea of the browser as a complete and fully functional application container for an inner platform. And I want to encourage creative people to try new technologies, especially WASM to explore the idea of "how much can we move to the browser". However, I see the container as the mediator of the network, not the application.
How does a VM or sandboxing help anything? The whole point of wireguard is to get the (shitty) OS VPN out of the picture entirely.
Give me a stream of bytes and let the whole world see it for all I care--wireguard will build a secure private network entirely on that stream of bytes. It could be totally public coffee shop wifi with zero encryption (basically yelling your passwords and secrets out in the open) and yet wireguard will make it secure and private for me.
So in this case who cares if its a websocket to the browser vs a 'proper' (bloated, shitty) OS VPN. Give me a stream of bytes and I'll build my own secure and trusted network on it thank you very much.
Give me a stream of bytes and let the whole world see it for all I care--wireguard will build a secure private network entirely on that stream of bytes. It could be totally public coffee shop wifi with zero encryption (basically yelling your passwords and secrets out in the open) and yet wireguard will make it secure and private for me.
So in this case who cares if its a websocket to the browser vs a 'proper' (bloated, shitty) OS VPN. Give me a stream of bytes and I'll build my own secure and trusted network on it thank you very much.
The VM encapsulates the OS component for networking and networking virtualization. In my experience, the virtualized software stack in the VM host is very reliable and predictable, and all the existing OS tools I need as a sysadmin (like tshark, ip, and other commands) all work just fine for debugging.
I understand the desire for moving more TCP logic to applications but, given my experience with network technology, I would predict that ten years from now, people will hate the experience of having to update 30 apps to get 1 fix to TCP performance that would have just been a kernel upgrade. IE, like everything that happened with the web and inner platforms, it's more work, doesn't replace the existing system, and just makes the admin's life harder for the ostensible purpose of being more convenient for the developer on their own machine.
I understand the desire for moving more TCP logic to applications but, given my experience with network technology, I would predict that ten years from now, people will hate the experience of having to update 30 apps to get 1 fix to TCP performance that would have just been a kernel upgrade. IE, like everything that happened with the web and inner platforms, it's more work, doesn't replace the existing system, and just makes the admin's life harder for the ostensible purpose of being more convenient for the developer on their own machine.
Yeah, in a way we're already there.
It gives a new meaning to "my computer is a datacenter" phrase.
we easily run multiple VMs kn the same laptop with multiple kernels with multiple TCP/IP implementations; even withing the same OS user space programs come with their own DNS resolvers that ignore system wide settings (Go without cgo, I'm thinking of you), and soon we're going to have a proliferation of user space networking stacks.
It gives a new meaning to "my computer is a datacenter" phrase.
we easily run multiple VMs kn the same laptop with multiple kernels with multiple TCP/IP implementations; even withing the same OS user space programs come with their own DNS resolvers that ignore system wide settings (Go without cgo, I'm thinking of you), and soon we're going to have a proliferation of user space networking stacks.
From the article
> Web-based SSH clients aren’t new. Nearly every VPS and cloud provider already lets you connect to your VMs from the web — so how is this different?
This is clearly isn't for everyone, but if you need or already use something like that, then I think this has a chance to be more secure than some other options.
> Web-based SSH clients aren’t new. Nearly every VPS and cloud provider already lets you connect to your VMs from the web — so how is this different?
This is clearly isn't for everyone, but if you need or already use something like that, then I think this has a chance to be more secure than some other options.
I really liked Chrome SSH extension but now I've returned to using the ssh command line client on all three platforms.
The issue is wiring up the browser-hosted application with a custom network inside the browser. It's a truly interesting but highly disruptive concept and I'm curious how it will play out. Perhaps in the future every program will statically compile its own TCP stack and talk over RAW sockets, but... that's sort of throwing away everything BSD and Linux and Windows achieved over the last few decades in terms of OS abstractions.
The issue is wiring up the browser-hosted application with a custom network inside the browser. It's a truly interesting but highly disruptive concept and I'm curious how it will play out. Perhaps in the future every program will statically compile its own TCP stack and talk over RAW sockets, but... that's sort of throwing away everything BSD and Linux and Windows achieved over the last few decades in terms of OS abstractions.
Funnily enough, the Chrome SSH extension uses pNaCl which is wasm-like in a roundabout way (non-ISA dependent bytecode running in the browser). So it's really not so different from what tailscale is doing.
> Perhaps in the future every program will statically compile its own TCP stack and talk over RAW sockets
A surprising number of common applications already do this.
I ran into a recent bug[1] caused by a Windows update where TLS handshakes would randomly fail. But this only presented in a few apps. Browsers and .NET apps were all completely unaffected because they don't use the OS level functionality to handle TLS.
[1]: This is a link to the KB that fixed the issue, which was introduced in the 22H2 cumulative update (search for "SEC_E_ILLEGAL_MESSAGE"): https://support.microsoft.com/en-us/topic/october-25-2022-kb...
> Perhaps in the future every program will statically compile its own TCP stack and talk over RAW sockets
A surprising number of common applications already do this.
I ran into a recent bug[1] caused by a Windows update where TLS handshakes would randomly fail. But this only presented in a few apps. Browsers and .NET apps were all completely unaffected because they don't use the OS level functionality to handle TLS.
[1]: This is a link to the KB that fixed the issue, which was introduced in the 22H2 cumulative update (search for "SEC_E_ILLEGAL_MESSAGE"): https://support.microsoft.com/en-us/topic/october-25-2022-kb...
Yes, I know pnacl very well- I used it as a sandbox for an idle cycle harvester that ran in the background of Google servers. It was never a workable technology (even when we did heroics to make it work) although many of the ideas were good.
TLS is different from TCP. TLS support might be provided by an OS, but it's certainly something an application can link in since it's really just a byte translator with some additional complex logic. TCP is an OS-level protocol for all the reasons that history chose it (having your network device and network protocol in the same ring).
TLS is different from TCP. TLS support might be provided by an OS, but it's certainly something an application can link in since it's really just a byte translator with some additional complex logic. TCP is an OS-level protocol for all the reasons that history chose it (having your network device and network protocol in the same ring).
Fair enough. I guess I just assumed that if the OS provided your TCP socket you'd also rely on it to provide TLS, but I'm sure I'm missing a lot of the nuance of why you'd mix and match.
[deleted]
Would it be possible to bundle the same into a portable application allowing you to use Tailscale without installing it? My understanding is that currently if you can't install Tailscale on a client you need to use Subnet Router. https://tailscale.com/kb/1109/devices-without-tailscale/