How Tailscale Works(tailscale.com)
tailscale.com
How Tailscale Works
https://tailscale.com/blog/how-tailscale-works/
81 comments
Well, this just listed out my concerns pretty well and I spent the last hour installing this on my devices. Specifically, how do I explain to a fortune 500 IT department how this is secure in a few simple sentences?
Large companies that want to remove us from the trusted group should run the coordination server on-prem.
We have considered other certification options, but so far they boil down to running a part of the coordination server on-prem. Still exploring the space though.
(I work at Tailscale)
We have considered other certification options, but so far they boil down to running a part of the coordination server on-prem. Still exploring the space though.
(I work at Tailscale)
Sure, large companies can easily run a on-premise coordination server. But large companies usually also have a network department and several other SDN and VPN solutions already. They are probably using highly tuned and standardized ipsec tunnels with hw accelerated AES encryption and special devices from different vendors.
I think tailscale could become the #1 solution for personal users and small / mid-sized companies (some of them might not even have an IT dept). At least the easy installation, the single process design (no separate IKE service etc.) and the opinionated modern cryptography - that deliberately does not allow any configuration at all - would make it a really good candidate for those use cases.
This users can probably rent a cheap $3 VPS easily and use it as an on-prem coordination server. However, this increases the installation effort considerable and trusting the cheap $3 VPS to be an essential part of your cooperate network might be a deal breaker. Many of those users might not even have a dedicated team for server maintenance and might not apply security updates regularly.
So, please add an (optional) additional PSK (or key pair / certificate) to each node that is not shared with the coordination server and can be used to sign / verify the configuration on each client. Users that do not care don't have to specify anything (they just have to log-in and everything works). Users that do care would have to login and provide the PSK on each device.
(And to be clear, the problem is not that I do not trust you. I am sure you are doing a great job and I would really like to use your service. But you are an US based company and with the new data protection laws within the EU it might not be easy to convince all customers that potentially sending all their data to a third-party company within the US is necessary.)
I think tailscale could become the #1 solution for personal users and small / mid-sized companies (some of them might not even have an IT dept). At least the easy installation, the single process design (no separate IKE service etc.) and the opinionated modern cryptography - that deliberately does not allow any configuration at all - would make it a really good candidate for those use cases.
This users can probably rent a cheap $3 VPS easily and use it as an on-prem coordination server. However, this increases the installation effort considerable and trusting the cheap $3 VPS to be an essential part of your cooperate network might be a deal breaker. Many of those users might not even have a dedicated team for server maintenance and might not apply security updates regularly.
So, please add an (optional) additional PSK (or key pair / certificate) to each node that is not shared with the coordination server and can be used to sign / verify the configuration on each client. Users that do not care don't have to specify anything (they just have to log-in and everything works). Users that do care would have to login and provide the PSK on each device.
(And to be clear, the problem is not that I do not trust you. I am sure you are doing a great job and I would really like to use your service. But you are an US based company and with the new data protection laws within the EU it might not be easy to convince all customers that potentially sending all their data to a third-party company within the US is necessary.)
Are large companies your target market?
Nice, UX looks good. Other alternatives are Zerotier [1], Pritunl [2], and Slack's Nebula [3].
1. https://www.zerotier.com/
2. https://pritunl.com/
3. https://github.com/slackhq/nebula
1. https://www.zerotier.com/
2. https://pritunl.com/
3. https://github.com/slackhq/nebula
I was coming here to say that it looks a lot like ZeroTier. The main differentiator as far as I know is the support for SSO and MFA.
With ZeroTier, the client connects to the network using a private shared key and then needs to be approved on the control plane independently.
With ZeroTier, the client connects to the network using a private shared key and then needs to be approved on the control plane independently.
ZeroTier original (and still main) author here. Objectively speaking another difference is the ACL scheme which seems perhaps less powerful than our own rules engine but easier to use. The ACLs are closer to intents while ZT rules require some TCP/IP knowledge (like raw iptables or pf).
We have plans to address this ourselves more with a higher level UI to build rules from intents and common patterns. Also have more auth integration planned, but the list is long.
We have plans to address this ourselves more with a higher level UI to build rules from intents and common patterns. Also have more auth integration planned, but the list is long.
Hi, thanks a lot for ZeroTier! I use it every day and it changed how I interact with my network of personal computers.
Do you have any news to ZeroTier 2.0? I'm really looking forward to it and being able to deploy my own controllers easily!
Significant performance improvements in the core, reworked CLI, crypto improvements, professional security audit of both design and code, easier root server federation, easier DIY controllers, better multicast, and of course bug fixes.
The firm doing our security audits is an extremely well known one. Don't want to reveal the name quite yet.
It's possible that some of that won't land fully formed right away in 2.0 but will follow shortly thereafter, but the performance, auditing, crypto, and multicast will be there.
The firm doing our security audits is an extremely well known one. Don't want to reveal the name quite yet.
It's possible that some of that won't land fully formed right away in 2.0 but will follow shortly thereafter, but the performance, auditing, crypto, and multicast will be there.
[deleted]
It's not clear from the docs. Can a two node Tailscale setup connect to each other when both sides are behind a IPv4 only CGNAT(Verizon/AT&T LTE)?
There is mention of STUN, but CGNAT is a whole new level of ugly and there is often NAT behind the CGNAT.
If not, do nodes that can get incoming connections become relay "super nodes?"
There is mention of STUN, but CGNAT is a whole new level of ugly and there is often NAT behind the CGNAT.
If not, do nodes that can get incoming connections become relay "super nodes?"
(Article author here.) Yes, this is what DERP is for. Nowadays automatically designating nodes as supernodes doesn’t work as well as it used to, since many devices are battery constrained and cloud devices tend to have metered bandwidth. We might someday have a way to add your own DERP relays for your domain (since the DERP code is open source anyway). The tricky part is guaranteeing reliable delivery in that case.
Also VERY interested in this, and the impending Android client status. Some of us are fully on mobile LTE data.
It sounds like these guys have done their homework (from a distributed systems PoV, I don’t know much about the security side).
IIUC it’s an application of one of my favorite design patterns: “this part needs CP, the rest is more flexible”.
It’s remarkable how far this gadget can take you.
IIUC it’s an application of one of my favorite design patterns: “this part needs CP, the rest is more flexible”.
It’s remarkable how far this gadget can take you.
Do you mean least privilege/ minimal centralization? I wrote this rant-ish post on the concept a while back.
http://adamierymenko.com/decentralization.html
It describes a route whereby I got to the concept, but I doubt I'm the first one there. Everything gets reinvented over and over again in this field.
http://adamierymenko.com/decentralization.html
It describes a route whereby I got to the concept, but I doubt I'm the first one there. Everything gets reinvented over and over again in this field.
Tailscale looks really cool. BUT for a modern zero-trust solution I would like to also see a way to connect Load Balanced Applications in addition to devices.
Seems like I would need to setup a bastion device (virtual machine / container / other) with port forwarding for each Load Balanced Application since ACLs are set on the device level. The bastion would then be a single point of failure. Is there another way?
Seems like I would need to setup a bastion device (virtual machine / container / other) with port forwarding for each Load Balanced Application since ACLs are set on the device level. The bastion would then be a single point of failure. Is there another way?
We have been thinking about load balancing as part of a general concept of services we want to roll out. Totally agree we need something. If you want to discuss it further, want to file an issue on github?
(Tailscale co-founder)
(Tailscale co-founder)
I was wondering, does this work with any domains that advertise (for example) OpenID Connect, or does it require one of the well-known providers?
It can, though right now we only expose a few well-known providers. For business customers we can add providers, send us an email!
Hopefully we can make this self-serve at some point. It's on the TODO list.
(Tailscale co-founder)
Hopefully we can make this self-serve at some point. It's on the TODO list.
(Tailscale co-founder)
Seems a bit like TINC-VPN
https://www.tinc-vpn.org/
https://www.tinc-vpn.org/
Tinc is really quite excellent, but it lacks the central authority to dynamically handle authentication and key management like tailscale.
I think the ideal case is Tinc continuing to manage the control plane for mesh networking, integrated with Wireguard as the actual VPN under the hood (given its rapid emergence as the de-facto "Linux VPN"), combined with a (currently absent) central identity management system like what tailscale provides.
I think the ideal case is Tinc continuing to manage the control plane for mesh networking, integrated with Wireguard as the actual VPN under the hood (given its rapid emergence as the de-facto "Linux VPN"), combined with a (currently absent) central identity management system like what tailscale provides.
I have what I think is a probably pretty common enterprise use case.
Windows Server with file share behind a sonicwall firwall. Large (now very large) number of remote users need to access file share.
Currently users connect using a VPN that terminates at the firewall, with access rules allowing access to file share.
Is this use case supported with tailscale? Ie, run something on server, allow users to run something, they can then connect to file share?
Anything to do on firewall to make this all go faster / easier?
As others have mentioned - we cannot let third party add nodes to our network without approval. Adding a PSK or something to let us lock network down a bit more would be good I think.
Edit: Installing at home (google wifi for router) and work machine (behind sonicwall) - can't get RDP/Remote Desktop to connect but client install / login is great and would be much easier than current approach if it worked.
Windows Server with file share behind a sonicwall firwall. Large (now very large) number of remote users need to access file share.
Currently users connect using a VPN that terminates at the firewall, with access rules allowing access to file share.
Is this use case supported with tailscale? Ie, run something on server, allow users to run something, they can then connect to file share?
Anything to do on firewall to make this all go faster / easier?
As others have mentioned - we cannot let third party add nodes to our network without approval. Adding a PSK or something to let us lock network down a bit more would be good I think.
Edit: Installing at home (google wifi for router) and work machine (behind sonicwall) - can't get RDP/Remote Desktop to connect but client install / login is great and would be much easier than current approach if it worked.
Is there any page that describes how to actually get VPN work end to end with tailscale?
I am a newbie and would like to understand.
I am a newbie and would like to understand.
Install the client on two devices and log in with a Gmail account. That's it.
They'll be assigned IPs and can reach each other.
(I'm a co-founder of Tailscale.)
They'll be assigned IPs and can reach each other.
(I'm a co-founder of Tailscale.)
That's really cool. Thank you.
what's local networking got to do with gmail?
There needs to be a mechanism to distribute keys to nodes / authenticate nodes / associate nodes.
Tailscale offers a centralized server to allow nodes to provision themselves. It bootstraps that trust/account system of of logging in with some account, such as via a google oauth login.
Tailscale offers a centralized server to allow nodes to provision themselves. It bootstraps that trust/account system of of logging in with some account, such as via a google oauth login.
Right! That makes sense, thanks.
I have need to connect all of my developers to multiple AWS accounts and have them work from hotels, home and company sites. AWSVPN only works for single account and needs certificates to be distributed to everyone. I like the way your solution is described but am unclear whether it will work for us?
I tried to use it for creating a mesh network of my family's computers, but switched to zerotier.
I use ansible to keep all our computers up to date and maintain an ldap server for some basic information; my parents are old and not tech savvy, so I do my best to make it easy for them.
Initially I set up tailscale, but the lack of an android client and the difficulty of setting up the raspberry service made me to switch to zerotier. Thing is, I really wanted that to work, as the ping with wireguard between the mesh instances was so low that surprised me.
Oh well, zerotier achieves what I want for now and it has the added benefit of letting you choose the private IP range that you like. I'll definitely keep an eye out for tailscale in the future though.
I use ansible to keep all our computers up to date and maintain an ldap server for some basic information; my parents are old and not tech savvy, so I do my best to make it easy for them.
Initially I set up tailscale, but the lack of an android client and the difficulty of setting up the raspberry service made me to switch to zerotier. Thing is, I really wanted that to work, as the ping with wireguard between the mesh instances was so low that surprised me.
Oh well, zerotier achieves what I want for now and it has the added benefit of letting you choose the private IP range that you like. I'll definitely keep an eye out for tailscale in the future though.
Oh man, I have a PoC of something like this written up in Python. I was more targeting a OpenVPN replacement than a mesh network, but it'd literally just be changing the central server's backend with something like etcd.
Good job on open sourcing your work and best of luck.
Good job on open sourcing your work and best of luck.
Great post! Can Tailscale be used to create a network between 10,000 IoT devices and a few servers? Is it scalable enough to support 10,000 peers on each node? Is it possible to script the registration of each IoT device to the control plane?
Is Tailscale SDWAN for edge VPNs?
Great write-up, when I checked out TS a month or more ago, documentation was severely lacking, and the repo wasnt out there. Lot of docs confused things as they talked about some logging services (pivoted product??).
In any case the one concern I had was the coordination server being able to add new nodes as another commenter mentioned. Indeed when I first set it up, they added some dev node for chat to my peers by default! That worries me a bit. You basically have to trust their coordinator is super secure. Otherwise if someone can abuse it, they could add nodes to your very private network.
In any case the one concern I had was the coordination server being able to add new nodes as another commenter mentioned. Indeed when I first set it up, they added some dev node for chat to my peers by default! That worries me a bit. You basically have to trust their coordinator is super secure. Otherwise if someone can abuse it, they could add nodes to your very private network.
Add UDP relays and TCP hole punching and it'll finally be a better take on Hamachi :)
https://swapped.cc/hamachi
https://swapped.cc/hamachi
(Tailscale employee here) UDP relays - that's roughly our DERP relay network. It currently operates over HTTPS because that's the last-resort "most likely to work in hard networks" transport, but adding a UDP option for efficiency is in the books.
TCP hole punching - I don't understand this, can you elaborate? By running Tailscale you can just talk to the machines on your Tailscale network, no hole punching required. At the transport layer we might do NAT traversal shenanigans to get the mesh network up, but that's invisible to the "user layer". I'm guessing you mean something slightly other and I'm just not parsing correctly?
TCP hole punching - I don't understand this, can you elaborate? By running Tailscale you can just talk to the machines on your Tailscale network, no hole punching required. At the transport layer we might do NAT traversal shenanigans to get the mesh network up, but that's invisible to the "user layer". I'm guessing you mean something slightly other and I'm just not parsing correctly?
Relays are helpful not only when UDP is blocked, but also when both nodes are behind NAT that does random port overloading and port prediction doesn't work.
TCP hole punching uses symmetrical TCP open trick to establish a proper TCP session between nodes each behind its own NAT. Roughly the same idea as with UDP h/p, but requires a coordinating server to make it work. Useful for cases when UDP punching doesn't work or when UDP is blocked completely.
TCP hole punching uses symmetrical TCP open trick to establish a proper TCP session between nodes each behind its own NAT. Roughly the same idea as with UDP h/p, but requires a coordinating server to make it work. Useful for cases when UDP punching doesn't work or when UDP is blocked completely.
Oh, that kind of TCP hole punching. No, we don't do that, because we run over UDP only. So far, I don't think we've seen network conditions where UDP is blocked but the NAT is friendly enough to permit TCP traversal (same problems as UDP traversal, only with extra state to manage). I suppose we might get to it eventually, but there's a lot of simpler cases we can handle better first :)
Hamachi was amazing. Just amazing. What.. 15 years ago? I still miss it after LogMeIn bought them.
This is a good post,answered most of my questions. Is it free though? It seems I can't find out the price without signing up.
My first concern was how it will be abused by malware and blocked by different networks. For example ngrok.io is classified as "proxy avoidance" which means if I use it and go to a client site that blocks it,it will create an awkward situation. But I think requiring 3rd party idP might solve this issue with tailscale.
My first concern was how it will be abused by malware and blocked by different networks. For example ngrok.io is classified as "proxy avoidance" which means if I use it and go to a client site that blocks it,it will create an awkward situation. But I think requiring 3rd party idP might solve this issue with tailscale.
How much of a drop-in replacement is this for a Remote Desktop Gateway connecting through to users' workstations via RDP? Our users love RDP, but i'm more in love with the security that something like this could provide.
A more relevant question for me is What is Tailscale? Didn't get that from either the article or really from the start page. Is it a VPN with "networking features"?
I guess you can call it that. You can start it on all your devices (e.g. laptop, PC, phone, tablet, Docker containers, etc.) and it will assign a fixed IP to each of them. Using this fixed IP all those devices are able to communicate securely with each other, no matter where those devices are located at the moment. Tailscale discovers the fastest, direct route between each device and routes the packages accordingly.
Example: Start it on all your devices and you can always access the shared files from your PC at home on your phone or tablet, no matter where you are. If you happen to be at home, no packages will be sent over your internet connection (so you can easily use it to stream 4k movies for example), but the way to access those files is always the same (fixed IP) and always secure.
An additional advantage of Wireguard (which is used behind the scenes) is that it maintains a strict mapping of IP addresses and identities. Therefore, you can usually restrict access to services by IP (instead of using TLS certificates or complicated authentication mechanisms).
Example: Start it on all your devices and you can always access the shared files from your PC at home on your phone or tablet, no matter where you are. If you happen to be at home, no packages will be sent over your internet connection (so you can easily use it to stream 4k movies for example), but the way to access those files is always the same (fixed IP) and always secure.
An additional advantage of Wireguard (which is used behind the scenes) is that it maintains a strict mapping of IP addresses and identities. Therefore, you can usually restrict access to services by IP (instead of using TLS certificates or complicated authentication mechanisms).
Really fascinating work! I enjoyed reading this article.
Do you actually have tunnels between every device? Wondering, how does this scale?
Tunnels are created between any two devices that try to talk.
It scales great, not concentrating connections makes it easy. Tunnels are very lightweight and the only machines with lots of tunnels are servers that are already provisioned for talking to lots of machines.
(Tailscale co-founder)
It scales great, not concentrating connections makes it easy. Tunnels are very lightweight and the only machines with lots of tunnels are servers that are already provisioned for talking to lots of machines.
(Tailscale co-founder)
Interesting :) Follow up, trying to cement my understanding here.
If two devices may talk to each other (as defined in the ACL), only then they get to know each other's public keys, right?
If so, how would this ACL be negated later- do the devices forget eachother after this disallow?
Yes, you only get the public keys of machines you can talk to. Later the server asks the client to forget the key if they can no longer talk.
Additionally, we rotate keys. For security sensitive users we can rotate these keys daily.
Additionally, we rotate keys. For security sensitive users we can rotate these keys daily.
Thank you! Fascinating work. Easily readable, yet highly informative article.
Hope you are the ones that kill off VPNs!
from where can I learn the fundamentals on this topic. I tried searching for data plane but most of them are k8s specific and few of them were pointing to layer3 of OSI model but I couldn't understand maybe my fundamentals are not too strong.Any reference such as books or articles will be much appreciated
This could be too high level, but I stumbled onto the founder's (crawshaw) website and found one of his blog posts very helpful for understanding basic fundamentals of the product (i.e. Zero Trust networks and microsegmentation)
https://crawshaw.io/blog/zero-trust
https://crawshaw.io/blog/zero-trust
Can i use tailscale to access the internet as if i was in another country?
Is there an open source version of this?
tailscale is open source: https://github.com/tailscale/tailscale
update: apparently, only Linux client is open-source. My bad.
update: apparently, only Linux client is open-source. My bad.
That is only the linux client?
Tailscale co-founder here: that's the guts of all the clients. (It almost works on macOS too as is, just a little more work to do.) We are still open sourcing our code, there's more to come but we haven't found the time to clean it up and get it ready to release. Soon hopefully!
Is the client the important part of the software, or is there a server side?
The client contains our most sophisticated code. It's also the most important from a trust perspective, it is responsible for securing your data, our servers cannot see anything of yours because of the client. So it's vital the important parts of it are open source.
That said, I'd like to open source a server. We need a reference implementation of our control protocol so it can be properly analyzed by security experts. Not sure what form that should take yet, but I'd like it to be usable.
That said, I'd like to open source a server. We need a reference implementation of our control protocol so it can be properly analyzed by security experts. Not sure what form that should take yet, but I'd like it to be usable.
It is, admittedly, impossible for me to be unbiased in this discussion (coauthor of Nebula, hi), but I strongly disagree that your client code is your most important component, from a trust perspective.
Your coordination server tells every node about every other node and distributes the keys for the entire network. Everything on a tailscale network implicitly trusts your coordination service.
If an individual client is compromised, code or otherwise, the effect is more limited than your coordination service being compromised, in which case the entire system's trust is broken.
Your coordination server tells every node about every other node and distributes the keys for the entire network. Everything on a tailscale network implicitly trusts your coordination service.
If an individual client is compromised, code or otherwise, the effect is more limited than your coordination service being compromised, in which case the entire system's trust is broken.
hi crawshaw, I've started playing with Tailscale as potential alternative for how we're using LISP in Project EVE (Edge Computing Mesh Network usecase) and was wondering whether there's any way today to make Linux clients negotiate (even if through a manual process without) going through your control plane?
What license schemes etc?
Also how do you integrate with IdPs?
Are you a SAML service provider?
If so where is the documentation for it?
Any else getting a 500 on the blog?
Our FD limit was misconfigured on our new blog. Whoops.
Working now after 3/4 tries
Down for me as well
500 internal server error
Nice job
[deleted]
HN promoted Tailscale worth of million dollars of advertising so far for free. This is really disgusting and the biggest proof that relations >>> product.
Rename this website to Tailscale News.
Rename this website to Tailscale News.
I mean... the people that founded the company have been long time contributors to the site. You've probably read a hundred of Avery's articles here.
What are you doing that's interesting? Post it here and you can get the free advertising too.
What are you doing that's interesting? Post it here and you can get the free advertising too.
When every Wireguard post gets immediately hijacked to become a promotion for this company, when at least 2 threads get to the HN frontpage each week then something is totally very fishy. I understand they are popular and have strong connections with many powerful people here. But at least this systematic promotion should be mentioned as promotion and not some innocent thread because it's totally organized and intentional.
nif2ee(2)
My main concern currently is the coordination server which does not fit the zero trust claim.
I know that the traffic between peers is end-to-end encrypted and you did a good job designing your DERP protocol. However, the ability of the coordination server (login.tailscale.com) to add arbitrary nodes to my private network without my consent scares me.
Maybe you can use Wireguard's PSK to add an additional pre-shared-key to all nodes that is not managed by the coordinator (and never transferred to it)? This would make the setup slightly more difficult (you need to login to tailscale.com AND you need to provide your PSK), but it would at least ensure that the system can never talk to foreign nodes added by the coordinator itself (because the foreign nodes do not know the PSK).
If you already use the PSK for something else, another passphrase/key-file that is never transmitted and is XOR'ed over the PSK will do the trick. The firewall configuration that is pushed to all clients should be probably signed by a local key too, since it is rather critical.
Another possible attack of the coordinator would be if he pushes a configuration with correct VPN IPs and correct public keys, but changes the mapping between them. Since the IP within a wireguard network is usually used as an identity, this might be a huge problem.