Headscale: Open-source implementation of the Tailscale control server(github.com)
github.com
Headscale: Open-source implementation of the Tailscale control server
https://github.com/juanfont/headscale
69 comments
Oh yes, Tailscale/Zerotier actually has a centralized control plane such that access can be revoked centrally and users be managed centrally too. and Tailscale has very nice blog posts explaining their infrastructure [1]
[1] https://tailscale.com/blog/how-tailscale-works/
[1] https://tailscale.com/blog/how-tailscale-works/
Thanks! I've read (again) the article, but it doesn't really answer my question. Here's a concrete example:
An organization uses Tailscale. There's 'server102' that is connected to the Tailscale network that all users of the `devops` team have access to. A new employee, Anne, joins the company. Sysadmins set up her SSO account, as well as makes her part of `devops` on Tailscale.
Anne gets her company computer, sets it up, connects to Tailscale, fires up her shell, types in `ssh anne@server102`, presses Enter.
What happens?
An organization uses Tailscale. There's 'server102' that is connected to the Tailscale network that all users of the `devops` team have access to. A new employee, Anne, joins the company. Sysadmins set up her SSO account, as well as makes her part of `devops` on Tailscale.
Anne gets her company computer, sets it up, connects to Tailscale, fires up her shell, types in `ssh anne@server102`, presses Enter.
What happens?
Is there anything among these that incorporates a basic configurable firewall policy?
In the more distant past, I used sshuttle to create “one way” poor man’s VPN; it is slow, but it was enough to saturate the remote connections I had at the time; and —- unlike many other systems at the time —- I knew I could trust the cryptography and key distribution, which piggybacks ssh.
At the minimum,I want to have connections going only one way between sine hosts, or no way in the case of two edge devices - and possibly also list specific ports and protocols. Sshuttle only provided directionality - and not intentionally either…
Sshuttle was conceived and written by Avery Pennarun, who later went to co-create … tailscale.
In the more distant past, I used sshuttle to create “one way” poor man’s VPN; it is slow, but it was enough to saturate the remote connections I had at the time; and —- unlike many other systems at the time —- I knew I could trust the cryptography and key distribution, which piggybacks ssh.
At the minimum,I want to have connections going only one way between sine hosts, or no way in the case of two edge devices - and possibly also list specific ports and protocols. Sshuttle only provided directionality - and not intentionally either…
Sshuttle was conceived and written by Avery Pennarun, who later went to co-create … tailscale.
No one mentioned yet Netmaker[1]
It's a fully meshed network based on wireguard, it's open source including the web ui
[1]: https://github.com/gravitl/netmaker
It's a fully meshed network based on wireguard, it's open source including the web ui
[1]: https://github.com/gravitl/netmaker
Does it use a central control plane?
With nearly all websites running https now, is it safe enough to surf without vpn these days.
Adding dns-over-https so your ISP can not collect where you're going.
As far as google etc collecting your info, it will work the same as long as you're using their service, with/without VPN.
I do use VPN(tailscale) for work so I can access corporate internal network, but for general surfing purpose, do I really need vpn these days.
Adding dns-over-https so your ISP can not collect where you're going.
As far as google etc collecting your info, it will work the same as long as you're using their service, with/without VPN.
I do use VPN(tailscale) for work so I can access corporate internal network, but for general surfing purpose, do I really need vpn these days.
In general, no. This is a good rundown, imo: https://gist.github.com/joepie91/5a9909939e6ce7d09e29/
Another similar project to Tailscale, for those in the market:
https://github.com/slackhq/nebula
Crazy simple, fully open source, trivial to self-host. Maybe not as featureful as Tailscale, but imo that can be a feature unto itself.
https://github.com/slackhq/nebula
Crazy simple, fully open source, trivial to self-host. Maybe not as featureful as Tailscale, but imo that can be a feature unto itself.
Related, here's a surprisingly balanced comparison of Nebula and Tailscale, on tailscale.com: https://tailscale.com/kb/1148/tailscale-vs-nebula/.
> If you’re a system administrator or technical person looking for a completely open source, free peer-to-peer mesh VPN, and you’re willing to run a certificate authority and the control plane yourself, try out Nebula.
> If you’re looking for a polished, user-friendly peer-to-peer mesh VPN with a hosted control plane and integration with existing identity providers, give Tailscale a try.
> If you’re a system administrator or technical person looking for a completely open source, free peer-to-peer mesh VPN, and you’re willing to run a certificate authority and the control plane yourself, try out Nebula.
> If you’re looking for a polished, user-friendly peer-to-peer mesh VPN with a hosted control plane and integration with existing identity providers, give Tailscale a try.
Hm — there’s a middle ground here that’s missing. I’d like to see a managed mesh allow for disabling its key distribution for certain nodes. They don’t create wireguard peers for any but a predefined shortlist of public keys, but still accept route updates from those peers.
The threat model is someone adding peers to the control plane, including as a result of control plane takeover or the identity provider failing. These special nodes can’t then be made to talk to anybody they can’t authenticate, no matter what you do on the control plane. It assumes private keys are safe. Obviously this is a client side setting, which shouldn’t have any control plane API, just like the current Tailscale options to eg accept no incoming traffic. This comes from my experience with ZeroTier, which I wrote about here: https://news.ycombinator.com/item?id=28426664
Then you can run your own Wireguard key distribution if you like, but ideally you just distribute manually for a few nodes and leave it at that.
The threat model is someone adding peers to the control plane, including as a result of control plane takeover or the identity provider failing. These special nodes can’t then be made to talk to anybody they can’t authenticate, no matter what you do on the control plane. It assumes private keys are safe. Obviously this is a client side setting, which shouldn’t have any control plane API, just like the current Tailscale options to eg accept no incoming traffic. This comes from my experience with ZeroTier, which I wrote about here: https://news.ycombinator.com/item?id=28426664
Then you can run your own Wireguard key distribution if you like, but ideally you just distribute manually for a few nodes and leave it at that.
Tiny usability improvement for small networks: “freeze” mode where the current set of peer public keys is frozen and no new peers can be added. Tie this to a (G)UI on each node to accept new peers anyway with user interaction using Signal style key visualisation, and you’re cooking with gas. Probably not worth it though, virtually nobody with three devices total and the time to do this manually really needs it.
Nebula is great - super simple to set up and get started if you have a VM to use as a lighthouse. Lots of cloud providers free tiers are have enough resources to host a lighthouse as well.
Certificate management is its one weakness at the moment. There are a growing number of projects floating around attempting to solve that though:
- https://github.com/unreality/nebula-mesh-admin
- https://github.com/b177y/starship
- https://github.com/symkat/MeshMage
Plus im sure defined networks has their own solution in the works as well.
Certificate management is its one weakness at the moment. There are a growing number of projects floating around attempting to solve that though:
- https://github.com/unreality/nebula-mesh-admin
- https://github.com/b177y/starship
- https://github.com/symkat/MeshMage
Plus im sure defined networks has their own solution in the works as well.
Nebula creators also started their own organization called Defined Networking to focus more on this networking: https://www.defined.net/
I've really liked nebula and have been working on a web frontend. Basically define a network and nodes and it uses nebula to generate certificates and has scripts for installing.
https://github.com/symkat/MeshMage
https://github.com/symkat/MeshMage
I've been using Nebula for personal use and it's really great. I have a free Oracle Cloud vm as my "lighthouse."
The advantage of Nebula is that it's dead simple. Generate a keypair, copy it over, copy the config file, and go. It can do mesh routing for the vpn and traverse nat magically. You can delegate dns to the lighthouse and name resolution just works too.
That simplicity is awesome for personal use, and maybe it's good enough for a small operation, but I'm guessing it doesn't have all the bells and whistles you'd want for medium or larger companies.
The advantage of Nebula is that it's dead simple. Generate a keypair, copy it over, copy the config file, and go. It can do mesh routing for the vpn and traverse nat magically. You can delegate dns to the lighthouse and name resolution just works too.
That simplicity is awesome for personal use, and maybe it's good enough for a small operation, but I'm guessing it doesn't have all the bells and whistles you'd want for medium or larger companies.
Nebula transits every EC2-to-EC2 packet at Slack, across lots of AWS regions and tens of thousands of hosts. It’s probably doing petabits of traffic per second. And it’s a safer, more expressive firewall than EC2 security groups.
So, yes, it works for personal use-cases but it works for truly gigantic applications, too.
So, yes, it works for personal use-cases but it works for truly gigantic applications, too.
The big thing I see missing is the management piece, which is what makes Tailscale compelling. If you're just running Nebula as an individual user, or for a small org, there wouldn't be much overhead. Otherwise, for larger deployments, you need to "roll your own" solution to manage configs outside of Nebula itself.
You'd also want this to be self-service in some way - so road warriors can rotate their own certs, with auth backed by some kind of central SSO system. The last I looked, Nebula didn't offer this stuff.
You'd also want this to be self-service in some way - so road warriors can rotate their own certs, with auth backed by some kind of central SSO system. The last I looked, Nebula didn't offer this stuff.
How well does it handle public WiFi? Some hotspots may block any non-HTTP traffic, or traffic on nonstandard ports. IIRC ZeroTier will use relays when UDP traffic is blocked.
innernet[1] is another similar one
[1]: https://github.com/tonarino/innernet
[1]: https://github.com/tonarino/innernet
Also:
WireTrustee: https://news.ycombinator.com/item?id=27672715
Netmaker: https://github.com/gravitl/netmaker
Defined.net (from makers of Slack Nebula): https://www.defined.net/
WireTrustee: https://news.ycombinator.com/item?id=27672715
Netmaker: https://github.com/gravitl/netmaker
Defined.net (from makers of Slack Nebula): https://www.defined.net/
One thing I can't figure out with Nebula is - how do you join multiple different networks?
For example, I have a personal laptop - I want to join two different networks, that are for two different purposes, and be able to talk to hosts in each? (But hosts in each should not be able to talk to hosts in the other)
For example, I have a personal laptop - I want to join two different networks, that are for two different purposes, and be able to talk to hosts in each? (But hosts in each should not be able to talk to hosts in the other)
Tinc is also pretty good and predates both by about a decade :)
Ps if we're listing alternatives zerotier is also one
Ps if we're listing alternatives zerotier is also one
[deleted]
Yggdrasil is not the same as either of these, but it can do similar things.
https://yggdrasil-network.github.io/
https://yggdrasil-network.github.io/
Doesn’t Yggdrasil use WireGuard these days?
Interesting, what are you using for cert management?
I saw this a while ago but had not realized it was feature complete now. Fantastic work. I look forward to moving from pure-wireguard to Headscale/Tailscale soon.
As far as I could work out, you can’t use the iOS client for Tailscale without building your own binary for it. I’d love to be proven wrong.
It's in the App Store here:
https://apps.apple.com/us/app/tailscale/id1470499037?ls=1
https://apps.apple.com/us/app/tailscale/id1470499037?ls=1
Yes, but that version works only with the tailscale.com server, not a custom one.
Actually the version that's in the App Store does support alternate control servers. It just needs to be configured via MDM configuration (https://support.apple.com/guide/mdm/mdm-overview-mdmbf9e668/...).
That's how our larger on-prem customers use the iOS client when they run their own in-house control plane server.
That's how our larger on-prem customers use the iOS client when they run their own in-house control plane server.
Is there any intent to make that just a setting? Using MDM for something like that as an individual seems a little over the top. I couldn't immediately find any information about how to configure this if I wanted to using MDM, even if it was the only option.
You can create MDM profiles even without a MDM server etc by using Apple Configurator 2. You will need a Mac however. You can then install those profiles manually.
A simple setting in the app would be far easier yeah. I would suspect that the ease of people bypassing the paid service is probably not a priority for them ;)
Personally I never even tried tailscale as I try to avoid Google. I definitely don't want to use my Google account to log into it and give Google more information.
But tinc serves my usecase well. The peer injection is a bit of a worry there too though. Especially because tinc peers are able to add any peers on their own (it's a feature meant to provide easier configuration).
I tried nebula extensively too but it didn't add enough over tinc to make it worthwhile switching especially now that tinc has an Android app.
A simple setting in the app would be far easier yeah. I would suspect that the ease of people bypassing the paid service is probably not a priority for them ;)
Personally I never even tried tailscale as I try to avoid Google. I definitely don't want to use my Google account to log into it and give Google more information.
But tinc serves my usecase well. The peer injection is a bit of a worry there too though. Especially because tinc peers are able to add any peers on their own (it's a feature meant to provide easier configuration).
I tried nebula extensively too but it didn't add enough over tinc to make it worthwhile switching especially now that tinc has an Android app.
I realize, but it's still a pain to do, and how to actually configure TailScale like this is completely undocumented, I couldn't find any mention of this feature outside of a hacker news comment. On the OSX application there's a hidden menu (hold option) which allows you to choose a different master server, but not enter your own, which is so close to being usable but just out of reach.
Is there a similar setting for Android app? I've tried building .apk with forked version of tailscale, but stopped after I saw issue on github about inability to login via auth key (https://github.com/tailscale/tailscale/issues/675). I thought headscale didn't support auth through email account?
While this is great, of course, it's definitely not feature parity with Tailscale, which currently allows me to have nothing listening on the opening internet and still form a private network spanning hosts all over the world, share resources within that network with 3rd parties trivially and send files across my network, android/iOS apps, etc.
I still posit the alternative to Tailscale is simply just wireguard. I don't see huge value in hosting my own Tailscale over just using Tailscale.
I still posit the alternative to Tailscale is simply just wireguard. I don't see huge value in hosting my own Tailscale over just using Tailscale.
You don’t think Tailscale has things listening on the open internet? When you use their control plane you are just outsourcing doing this to someone else (who is probably better at it than you but it’s still someone else). Lack of “feature parity” is a strange term for something that completely replicates a third party service but requires self-hosting.
Of course they do -- that's why I use them. I don't have to.
> Lack of “feature parity” is a strange term for something that completely replicates a third party service but requires self-hosting.
It doesn't completely replicate it though, does it? No iOS app, no file sending -- heck the entire premise of Tailscale is oriented around SSO as they are explicitly not an IdP.
I don't mean to knock this effort -- it's great! -- but even if I were to switch to this today, there is a ton of missing functionality, and I'd need to do a ton of work beyond the scope of Headscale just to get this running in my corp.
> Lack of “feature parity” is a strange term for something that completely replicates a third party service but requires self-hosting.
It doesn't completely replicate it though, does it? No iOS app, no file sending -- heck the entire premise of Tailscale is oriented around SSO as they are explicitly not an IdP.
I don't mean to knock this effort -- it's great! -- but even if I were to switch to this today, there is a ton of missing functionality, and I'd need to do a ton of work beyond the scope of Headscale just to get this running in my corp.
The ZeroTier ecosystem has a number of open source self-hosted controller projects for those who want unlimited members/networks/admins.
- https://github.com/key-networks/ztncui (the most popular one, GUI)
- https://github.com/dec0dOS/zero-ui (GUI)
- https://github.com/thedunston/bash_cli_zt (CLI)
- https://github.com/key-networks/ztncui (the most popular one, GUI)
- https://github.com/dec0dOS/zero-ui (GUI)
- https://github.com/thedunston/bash_cli_zt (CLI)
IANAL How might this interact with the ZeroTier BSL license? Would self hosting an alternative controller combined with BSL licensed clients violate the license (for commercial purposes)?
No, not if you're not selling ZeroTier as a for-profit SaaS product or building it into a closed product.
I read through the README and I'm still not quite sure - given how simple it is to set up wireguard, what's the value add here?
Wireguard is fairly easy to add between two machines, which allows you to set up a client/server setup which all traffic routes through
You can add a new machine, so to set up the third machine takes 10 minutes, the fourth takes 10, the fifth 10, etc
If machine 3 wants to talk to machine 6, packets are routed via the single central "vpn concentrator" machine.
However to really benefit from wireguard, you don't want to tunnel all the traffic through a single machine - both from a security perspective and performance perspective.
To add 3 machines is fine, you need to set up 3 tunnels, from machine 1-2, 2-3 and 1-3.
A fourth machine needs 3 new tunnels - 1-4, 2-4 and 3-4. A fifth machine needs 4 new tunnels.
You then need to manage all those keys and cycle through them (you should change private keys regularly)
Things like tailscale automate all this. You want to add 19th a machine, you simply add one entry and it handles the rest.
You can add a new machine, so to set up the third machine takes 10 minutes, the fourth takes 10, the fifth 10, etc
If machine 3 wants to talk to machine 6, packets are routed via the single central "vpn concentrator" machine.
However to really benefit from wireguard, you don't want to tunnel all the traffic through a single machine - both from a security perspective and performance perspective.
To add 3 machines is fine, you need to set up 3 tunnels, from machine 1-2, 2-3 and 1-3.
A fourth machine needs 3 new tunnels - 1-4, 2-4 and 3-4. A fifth machine needs 4 new tunnels.
You then need to manage all those keys and cycle through them (you should change private keys regularly)
Things like tailscale automate all this. You want to add 19th a machine, you simply add one entry and it handles the rest.
Centralised management and simple NAT/firewall traversal, basically.
The peer to peer traffic also helps increase throughput and reduce latency. With traditional VPN, peers will never connect directly
so this whole zerocorp/zerotier/encrypted-mesh networking approach is pretty cool, but every time i see it i ask myself: how do you monitor for malicious nodes? in old setups, typically there would be some sort of passive monitoring system that would monitor the traffic between hosts and could be used for forensics/malicious traffic identification. but if you're encrypting traffic at each node for each other node, then only the participant nodes are privy to the traffic. if one or both are compromised, how would you ever know? sure you can run userland security agents on them that collect data, but if the machines are actually compromised, you can't really trust what they say, right? (that's the whole reason why you use a third system for monitoring!)
Some discussion in the Tailscale docs here: https://tailscale.com/kb/1011/log-mesh-traffic/
so that's a pretty cool and elegant solution. i suppose they don't have the log tampering detection stuff implemented yet, but it seems straightforward to implement and i'm sure it will happen eventually.
cool. netflow for encrypted mesh networking. still vulnerable if both nodes are compromised via a sidechannel and collude on their logs, but that's also getting pretty radical in terms of an attack vector.
what about actually logging the contents? i've seen big commercial systems that look pretty much like distributed wireshark, with capture points, storage systems and pretty guis for inspection... not sure how prevalent and useful they are, but having a step deeper than netflow style logs can be useful, both for debugging and security purposes. i suppose you could do this double entry for that as well, but that seems a pretty high cost if the tunnels are high bandwidth?
cool. netflow for encrypted mesh networking. still vulnerable if both nodes are compromised via a sidechannel and collude on their logs, but that's also getting pretty radical in terms of an attack vector.
what about actually logging the contents? i've seen big commercial systems that look pretty much like distributed wireshark, with capture points, storage systems and pretty guis for inspection... not sure how prevalent and useful they are, but having a step deeper than netflow style logs can be useful, both for debugging and security purposes. i suppose you could do this double entry for that as well, but that seems a pretty high cost if the tunnels are high bandwidth?
ZeroTier rules can be used to monitor traffic via the "tee" rule. You can send copies of any packet matching any criteria (whole packet or part of it) to a monitor. Both sender and receiver can match, so someone would have to compromise both sides to evade it.
Not sure what I think about this.
I don't use Tailscale because I don't trust their key distribution, and this open source project would solve that, but it might undermine Tailscale's sustainability.
This would be a shame because Tailscale is working well with the open source community: open source clients, working well with distros, working well with Linux DNS stack, supporting a more P2P secure Internet, and documenting their well through it.
I don't use Tailscale because I don't trust their key distribution, and this open source project would solve that, but it might undermine Tailscale's sustainability.
This would be a shame because Tailscale is working well with the open source community: open source clients, working well with distros, working well with Linux DNS stack, supporting a more P2P secure Internet, and documenting their well through it.
You buy a Tailscale contract for the same reason you buy a Red Hat contract. If something goes wrong, and you need to fix it fast, their experts will work on that, not yourself.
Also, Tailscale offers OAuth through large corporate providers; I'm not sure Headscale is going to support that. (Actually, this is why I don't use Tailscale for my private network: I don't want to depend on an external OAuth provider.)
Also, Tailscale offers OAuth through large corporate providers; I'm not sure Headscale is going to support that. (Actually, this is why I don't use Tailscale for my private network: I don't want to depend on an external OAuth provider.)
Headscale dev here.
Companies looking into this will pay Tailscale.com service. You really need commercial support if you plan a large enterprise deployment. Tailscale even now offers a self-hosted version of their service - for those with concerns about using the public SaaS.
Companies looking into this will pay Tailscale.com service. You really need commercial support if you plan a large enterprise deployment. Tailscale even now offers a self-hosted version of their service - for those with concerns about using the public SaaS.
You can possibly buy a subscription but tell the devs that you're using headscale. If enough people do this they might make a host-your-own version like bitwarden.
To clarify, I don't use Headscale either.
Both look like great products, I just don't need what they're offering enough to accept their downsides.
Both look like great products, I just don't need what they're offering enough to accept their downsides.
We're working on the key distribution trust problem. Stay tuned.
I agree - there's always a danger with companies that try and have a lot of their product as OS that someone will come along with an OS product that plugs the gap in the only place they're trying to make a profit!
I think because in this case they enforce that by hosting the control plane themselves, it doesn't really matter. I wouldn't use something I can't self-host anyway, not something as security-sensitive as a VPN.
So many users will not have even considered the paid version anyway, and their participation will give tailscale a higher marketshare and thus more viability.
So many users will not have even considered the paid version anyway, and their participation will give tailscale a higher marketshare and thus more viability.
The question is this. Say, I use one of the above to form a private mesh network for the nodes that an organization needs to have access to. So far so good. But on the machine side I would still want to have key (ideally certificate) based authentication, and some user management, such that access can be revoked. Is this an anti-pattern? Or do people use something like Go Teleport in combination with a zero trust mesh network?