Gravitational Wormhole: WireGuard for Kubernetes(gravitational.com)
gravitational.com
Gravitational Wormhole: WireGuard for Kubernetes
https://gravitational.com/blog/announcing_wormhole/
33 コメント
Author here. Feel free to reach out if you have questions or thoughts about the project.
Not gonna lie... I paused at reading the title and only clicked when I was expecting to read about physics.
(I had no idea of any of the Kubernetes products mentioned in the article prior to reading)
(I had no idea of any of the Kubernetes products mentioned in the article prior to reading)
Hi Kevin, I'm wondering where you see Wormhole fit into the ecosystem alongside heavier systems like Istio, which offer many more features than just mTLS. Do you think Wormhole and Istio could be deployed together, using Wormhole for mTLS and turning it off in Istio, and some advantage in doing so? Do you see Wormhole's focus as being tuned to situations where the relative heaviness of solutions like Istio are not necessary and only the lightest solution possible is desired? Does that mean that Wormhole should be viewed more as "include this with my application designed for small / lightweight environments" or "include this with my Kubernetes distro, and config applications running on that distro to best use Wormhole"?
> I'm wondering where you see Wormhole fit into the ecosystem alongside heavier systems like Istio?
I see wormhole as a simpler / lighter weight solution. Like flannel, but building an encrypted network instead of a vxlan (or xx plugin) network. The trade offs are optimized towards simplicity and being a lighter weight option with less capabilities.
>Do you think Wormhole and Istio could be deployed together, using Wormhole for mTLS and turning it off in Istio, and some advantage in doing so?
I don't think I have a deep enough understanding of Istio to know for sure how easy it is to use them together and have never tested this sort of configuration. I wouldn't consider wormhole a replacement for mTLS, as wormhole isn't capable of embedding identity of either end into a connection. You can think of it like establishing ipsec tunnels between networks.
Where I think wormhole would come into play in this theoretical setup, is if you want a subset of traffic to be encrypted, which either doesn't play well with istio for various reasons or you benefit from additional layers to the security model.
> Do you see Wormhole's focus as being tuned to situations where the relative heaviness of solutions like Istio are not necessary and only the lightest solution possible is desired?
Yes, exactly.
> Does that mean that Wormhole should be viewed more as "include this with my application designed for small / lightweight environments" or "include this with my Kubernetes distro, and config applications running on that distro to best use Wormhole"?
Yes. Although using wormhole doesn't require any application involvement. Wormhole operates as the networking configuration for the cluster, and simply uses an encrypted protocol for doing so. So any applications that run on say a flannel cluster, should not realize wormhole is there, we just provide a kubernetes network. Of course there is always an implementation detail we may do slightly different, but that's likely a defect in the implementation we would need to address.
I see wormhole as a simpler / lighter weight solution. Like flannel, but building an encrypted network instead of a vxlan (or xx plugin) network. The trade offs are optimized towards simplicity and being a lighter weight option with less capabilities.
>Do you think Wormhole and Istio could be deployed together, using Wormhole for mTLS and turning it off in Istio, and some advantage in doing so?
I don't think I have a deep enough understanding of Istio to know for sure how easy it is to use them together and have never tested this sort of configuration. I wouldn't consider wormhole a replacement for mTLS, as wormhole isn't capable of embedding identity of either end into a connection. You can think of it like establishing ipsec tunnels between networks.
Where I think wormhole would come into play in this theoretical setup, is if you want a subset of traffic to be encrypted, which either doesn't play well with istio for various reasons or you benefit from additional layers to the security model.
> Do you see Wormhole's focus as being tuned to situations where the relative heaviness of solutions like Istio are not necessary and only the lightest solution possible is desired?
Yes, exactly.
> Does that mean that Wormhole should be viewed more as "include this with my application designed for small / lightweight environments" or "include this with my Kubernetes distro, and config applications running on that distro to best use Wormhole"?
Yes. Although using wormhole doesn't require any application involvement. Wormhole operates as the networking configuration for the cluster, and simply uses an encrypted protocol for doing so. So any applications that run on say a flannel cluster, should not realize wormhole is there, we just provide a kubernetes network. Of course there is always an implementation detail we may do slightly different, but that's likely a defect in the implementation we would need to address.
Whats the performance impact?
Hi! It’s good to see Wireguard become more usable in more scenarios.
Are you in contact with the Wireguard author, and how do you plan to support the Wireguard project?
Are you in contact with the Wireguard author, and how do you plan to support the Wireguard project?
Hi.
No I'm not in touch with the WireGuard maintainers. I'd be happy to start a dialog.
As for contributing to the project, I'll likely take a crack at porting the netlink interface to the golang netlink project (https://github.com/vishvananda/netlink) so that the wireguard interface can be automated through golang. Right now, the project just automates this by execing the wg cli.
We're also integrating this with our commercial offering of gravity, which if we see traction I'll be making the case that we should contribute to the project financially as well.
No I'm not in touch with the WireGuard maintainers. I'd be happy to start a dialog.
As for contributing to the project, I'll likely take a crack at porting the netlink interface to the golang netlink project (https://github.com/vishvananda/netlink) so that the wireguard interface can be automated through golang. Right now, the project just automates this by execing the wg cli.
We're also integrating this with our commercial offering of gravity, which if we see traction I'll be making the case that we should contribute to the project financially as well.
https://github.com/mdlayher/wireguardctrl This is what you're looking for. We'll probably make these "official" at some point, as they seem to be pretty nicely done.
I'm looking forward to reading the source of your WireGuard integration. Congrats on launching.
Feel free to contact us if you need - team at wireguard dot com.
I'm looking forward to reading the source of your WireGuard integration. Congrats on launching.
Feel free to contact us if you need - team at wireguard dot com.
Creator of https://github.com/vishvananda/netlink here. Would be happy to have support added.
I had previously come across an in-tree WireGuard extension for Flannel, https://github.com/coreos/flannel/blob/master/dist/extension....
However, Flannel's WireGuard extension looked way too experimental, and unmaintained. I realise it is also early days for Gravitational Wormhole, but Gravitational Wormhole looks promising.
However, Flannel's WireGuard extension looked way too experimental, and unmaintained. I realise it is also early days for Gravitational Wormhole, but Gravitational Wormhole looks promising.
What advantage do you see using WG over TLS? A nice feature of TLS is that you can perform AuthZ on a per-service basis, and you get encryption for free.
TLS also has the nice property of working with all sorts of clients, even those outside the cluster that might not be using WG encryption at the network later.
TLS also has the nice property of working with all sorts of clients, even those outside the cluster that might not be using WG encryption at the network later.
I would say it really comes down to the use case. If you're already using TLS and have the infrastructure for key management, issuing certs, etc. It's not worth replacing.
To me, where WireGuard comes in is for services that aren't traditionally encrypted, or where an extra layer of security is desired.
The example case that we've seen a few times is cluster DNS. The DNS traffic internal to the cluster is just plain DNS, meaning it leaks internal information and can be tampered with on an untrusted network. Even though CoreDNS does support DNS over TLS, it's a large effort to update clients in some organizations, that is better spent elsewhere.
This is where we see the opportunity for WireGuard, is ensuring all internal traffic is encrypted. For taking applications where the security model traditionally requires a secure network, and allowing them to be forklifted into a kubernetes cluster that provides those capabilities. I'm not saying this is an ideal model over using mTLS for every connection, just not everyone has what's needed to deliver mTLS everywhere.
Unfortunately, we do lose the identity features that TLS offers in the x509 certs. Nothing prevents running TLS for services that need it though on top of the WireGuard network.
So like everything else, I think there are pros and cons to both approaches, and it depends on what applies to the specific application.
I hope that helps.
To me, where WireGuard comes in is for services that aren't traditionally encrypted, or where an extra layer of security is desired.
The example case that we've seen a few times is cluster DNS. The DNS traffic internal to the cluster is just plain DNS, meaning it leaks internal information and can be tampered with on an untrusted network. Even though CoreDNS does support DNS over TLS, it's a large effort to update clients in some organizations, that is better spent elsewhere.
This is where we see the opportunity for WireGuard, is ensuring all internal traffic is encrypted. For taking applications where the security model traditionally requires a secure network, and allowing them to be forklifted into a kubernetes cluster that provides those capabilities. I'm not saying this is an ideal model over using mTLS for every connection, just not everyone has what's needed to deliver mTLS everywhere.
Unfortunately, we do lose the identity features that TLS offers in the x509 certs. Nothing prevents running TLS for services that need it though on top of the WireGuard network.
So like everything else, I think there are pros and cons to both approaches, and it depends on what applies to the specific application.
I hope that helps.
I'm enthusiastic about WireGuard but I'm dubious about the "forklift" approach you describe. Actually, more seriously, I doubt in most cases that systems to be forklifted were truly secure originally and so while I don't think using WireGuard makes that worse, it may mean an opportunity is missed to fix more serious problems.
Presuming that by mTLS you mean mutually authenticated TLS, I don't see it as a good fit for internal systems - to me (as someone who has implemented mutual auth to considerable success) it's best where you have B2B applications, because you've got the three distinct parties that make a PKI worthwhile. Whenever I see a PKI in which all three of the parties are essentially identical I cringe, e.g. I keep seeing OpenVPN setups where one person runs the CA, runs the VPN server and mints all the private keys - the PKI is pure theatre in this approach, it makes no difference to anything.
Presuming that by mTLS you mean mutually authenticated TLS, I don't see it as a good fit for internal systems - to me (as someone who has implemented mutual auth to considerable success) it's best where you have B2B applications, because you've got the three distinct parties that make a PKI worthwhile. Whenever I see a PKI in which all three of the parties are essentially identical I cringe, e.g. I keep seeing OpenVPN setups where one person runs the CA, runs the VPN server and mints all the private keys - the PKI is pure theatre in this approach, it makes no difference to anything.
Speaking from deep inside The Enterprise...
I doubt in most cases that systems to be forklifted were truly secure originally...
Yeah, people generally understand that apps developed under a perimeter security mindset aren't considered secure by newer zero-trust standards.
...an opportunity is missed to fix more serious problems.
The security debt is so enormous that there will probably never be enough budget to fix it.
I doubt in most cases that systems to be forklifted were truly secure originally...
Yeah, people generally understand that apps developed under a perimeter security mindset aren't considered secure by newer zero-trust standards.
...an opportunity is missed to fix more serious problems.
The security debt is so enormous that there will probably never be enough budget to fix it.
Hi Kevin, how does this compare to other ways of doing encrypted overlay within Kubernetes? Things such ebpf-based tools or sidecars?
This is discussed a bit in the rationale and alternatives section of the original spec: https://github.com/gravitational/wormhole/blob/master/docs/r...
To try and summarize without speaking too much for projects that I'm not intimately familiar with, Wormhole takes the approach of automating the configuration of WireGuard to create a full mesh encrypted network.
Other approaches take a much more comprehensive approach, of trying to solve key signing, identity, revocation, offload to the kernel and intercept via sidecars (using ebpf), etc. These tools come with many features and an inherent complexity, where I believe there is room for a simpler tool that is easier to use and troubleshoot.
To try and summarize without speaking too much for projects that I'm not intimately familiar with, Wormhole takes the approach of automating the configuration of WireGuard to create a full mesh encrypted network.
Other approaches take a much more comprehensive approach, of trying to solve key signing, identity, revocation, offload to the kernel and intercept via sidecars (using ebpf), etc. These tools come with many features and an inherent complexity, where I believe there is room for a simpler tool that is easier to use and troubleshoot.
Do you feel bad about using your 'hiring process' as a means of free labour?
i.e. Reading kubeadm config, reusing system certs, and automating the configuration of a VPN overlay system: https://github.com/jameskeane/scrambler
--
FYI to readers, the above project was completed as part of a 'take-home assignment' that I was explicitly told was not a project being worked on.
The feedback I received indicated that this approach was not considered previously, and may indicate (also your commit dates) that this entire project may have been fabricated after my submission.
Here's the feedback from Gravitational around Oct 2018 (before this project was 'started'):
> * I do think re-using the kubernetes certs for IPSec is a compelling solution for the IPSec secrets
> * I like the re-use of the kubernetes keys/certs for ipsec as it avoids use of the PSK
At least it was released as OSS, but seriously you should be ashamed of yourselves.
i.e. Reading kubeadm config, reusing system certs, and automating the configuration of a VPN overlay system: https://github.com/jameskeane/scrambler
--
FYI to readers, the above project was completed as part of a 'take-home assignment' that I was explicitly told was not a project being worked on.
The feedback I received indicated that this approach was not considered previously, and may indicate (also your commit dates) that this entire project may have been fabricated after my submission.
Here's the feedback from Gravitational around Oct 2018 (before this project was 'started'):
> * I do think re-using the kubernetes certs for IPSec is a compelling solution for the IPSec secrets
> * I like the re-use of the kubernetes keys/certs for ipsec as it avoids use of the PSK
At least it was released as OSS, but seriously you should be ashamed of yourselves.
FYI, I was hired by Gravitational back in April 2018 and I was given that same take-home assignment as part of the interview process - to write a CNI plugin for k8s which created an encrypted mesh network between nodes. My work is here: https://github.com/webvictim/k8s-ipsec-cni
I found some other resources/projects online while I was writing this which indicated that I certainly wasn't the first interviewee to be asked to do this - I evidently wasn't the last either as your project shows.
My point is that it seems Gravitational has had this 'idea' for quite some time as they've been using it as an interview question for at least the year 2018. I don't feel like that constitutes them using anyone's interview work as free labour.
I found some other resources/projects online while I was writing this which indicated that I certainly wasn't the first interviewee to be asked to do this - I evidently wasn't the last either as your project shows.
My point is that it seems Gravitational has had this 'idea' for quite some time as they've been using it as an interview question for at least the year 2018. I don't feel like that constitutes them using anyone's interview work as free labour.
Big warning to readers:
If Gravitational asks you to complete an 'engineering challenge' they are using you for free labour.
See: https://news.ycombinator.com/item?id=19784787
If Gravitational asks you to complete an 'engineering challenge' they are using you for free labour.
See: https://news.ycombinator.com/item?id=19784787
James, we respectfully disagree.
I think it's helpful if you (or anyone reading this) compare two projects side by side.
This is the project you have submitted:
https://github.com/jameskeane/scrambler
It is a prototype of IPSec CNI plugin using Bash. We are happy you are promoting your project online and as we've discussed during the interview, you could use it as you wish as it's your work.
This is the project that we have created at Gravitational:
https://github.com/gravitational/wormhole
It is using Go, Wireguard and is designed for production use under Apache 2 license.
I understand that similarities in the problem statement may have led you to believe that we could re-use the project idea, I'm assuring that you that we have not.
There are many OSS CNI plugins in the market solving the similar problem in a similar fashion (e.g. flannel with IPSec backend).
As a matter of fact that's the main reason we have picked this as a problem statement - to make it easier for candidates to look around for prior work.
For example, this is a submission of the engineer we've hired based on their similar execution of the assignment from April 2018:
https://github.com/webvictim/k8s-ipsec-cni
James, I wish you the best and hope that my response clarifies some confusion around this.
Sasha, CTO and co-founder at Gravitational
I think it's helpful if you (or anyone reading this) compare two projects side by side.
This is the project you have submitted:
https://github.com/jameskeane/scrambler
It is a prototype of IPSec CNI plugin using Bash. We are happy you are promoting your project online and as we've discussed during the interview, you could use it as you wish as it's your work.
This is the project that we have created at Gravitational:
https://github.com/gravitational/wormhole
It is using Go, Wireguard and is designed for production use under Apache 2 license.
I understand that similarities in the problem statement may have led you to believe that we could re-use the project idea, I'm assuring that you that we have not.
There are many OSS CNI plugins in the market solving the similar problem in a similar fashion (e.g. flannel with IPSec backend).
As a matter of fact that's the main reason we have picked this as a problem statement - to make it easier for candidates to look around for prior work.
For example, this is a submission of the engineer we've hired based on their similar execution of the assignment from April 2018:
https://github.com/webvictim/k8s-ipsec-cni
James, I wish you the best and hope that my response clarifies some confusion around this.
Sasha, CTO and co-founder at Gravitational
So you're admitting that you asked candidates to work on a project you intended to monetize?
Were you lying when you explicitly told me the opposite or did you decide after my submission (going by commit dates) to make this a "real" project?
You didn't steal my code; you stole my _ideas_ and are now presenting them as your own original work. Shame on you.
Were you lying when you explicitly told me the opposite or did you decide after my submission (going by commit dates) to make this a "real" project?
You didn't steal my code; you stole my _ideas_ and are now presenting them as your own original work. Shame on you.
> If you’re running Kubernetes in a network you don’t fully trust or need to encrypt all pod network traffic between hosts for legacy applications or compliance reasons, Wormhole might be for you
Is there a good analysis somewhere about how typical kubernetes setups trust the network and what badness an advesary could do with kubernetes network access? How sound is this default deployment setup from security POV?
For example I think DNS is used internally for service discovery, and incoming TLS is often terminated and proxied onwards as HTTP - those could be both MITMed, right?
Is there a good analysis somewhere about how typical kubernetes setups trust the network and what badness an advesary could do with kubernetes network access? How sound is this default deployment setup from security POV?
For example I think DNS is used internally for service discovery, and incoming TLS is often terminated and proxied onwards as HTTP - those could be both MITMed, right?
I thought I understood this, and that it replaced (and no doubt did a better job of) what I'd already done - WG to get nodes on the same network, CNI on top.
But requirement 2 confused me: > A Kubernetes cluster with IPAM enabled (--pod-network-cidr= when using kubeadm based install)
So, do node machines need to already be on the same network or not?
But requirement 2 confused me: > A Kubernetes cluster with IPAM enabled (--pod-network-cidr= when using kubeadm based install)
So, do node machines need to already be on the same network or not?
Sorry for the confusion, this requirement isn't about the node networking. This is about assigning subnets to each node for the pod IP addresses.
Kubernetes includes an optionally enabled IPAM, that can allocate subnets from a large subnet, into the /24 subnet assigned to each node. This becomes the spec.podCIDR field in the node object when enabled.
Instead of implementing another IPAM for the node cidrs, wormhole uses the built in IPAM within kubernetes. The only thing is, the ipam needs to be turned on. On a kubeadm based install, this is done by setting the pod-network-cidr flags during init. This sets --cluster-cidr / --allocate-node-cidrs on kube-controller-manager.
Kubernetes includes an optionally enabled IPAM, that can allocate subnets from a large subnet, into the /24 subnet assigned to each node. This becomes the spec.podCIDR field in the node object when enabled.
Instead of implementing another IPAM for the node cidrs, wormhole uses the built in IPAM within kubernetes. The only thing is, the ipam needs to be turned on. On a kubeadm based install, this is done by setting the pod-network-cidr flags during init. This sets --cluster-cidr / --allocate-node-cidrs on kube-controller-manager.
But what's the value of that flag if I'm going to use Wormhole only afterwards?
I set up a Wireguard /17 (iirc) mesh, and then kubeadm'd with --pod-network-cidr as that node's wg endpoint /24.
I suppose what I'm not understanding is how I can set up the cluster sufficiently to be able to kubectl apply wormhole if I haven't already setup a wireguard (or...) network between the nodes?
I set up a Wireguard /17 (iirc) mesh, and then kubeadm'd with --pod-network-cidr as that node's wg endpoint /24.
I suppose what I'm not understanding is how I can set up the cluster sufficiently to be able to kubectl apply wormhole if I haven't already setup a wireguard (or...) network between the nodes?
Ok, I think I might understand my miss-understanding. If I understand correctly, you're running you're kubernetes API / connectivity over a WireGuard network established between hosts?
Because the kubernetes APIs are protected by mTLS between nodes, the API connectivity doesn't need to be over WireGuard, unless you don't have bi-direction connectivity for other reasons (like NAT). Although, I don't generally recommend splitting nodes between disparate networks, as it's not a good failure model for kubernetes.
How this is modelled, is when setting up a kubernetes cluster, you establish node trust using the installation method, and whatever network exists between the hosts.
This creates a kubernetes installation, but doesn't setup network. When using the flags to kubeadm, this creates a kubeadm config file, and turns on the kubernetes subnetting. So say you were to configure 10.100.0.0/16 as a pod-network-cidr. The would cause the kubernetes controller manager to assign a /24 to each node for pods. So node 1 gets 10.100.1.0/24, node 2 gets 10.100.2.0/24, etc. The addresses from the node cidr are then assigned the pods as they are scheduled by the cni plugin. The wormhole plugin, then takes care of setting up all the networking that covers 10.100.0.0/24, because kubernetes has effectively been granted management of this subnet.
So, yes, this assumes some sort of network exists between nodes, which is the network that has been used to create mTLS trust between kubelet/api on all members.
Because the kubernetes APIs are protected by mTLS between nodes, the API connectivity doesn't need to be over WireGuard, unless you don't have bi-direction connectivity for other reasons (like NAT). Although, I don't generally recommend splitting nodes between disparate networks, as it's not a good failure model for kubernetes.
How this is modelled, is when setting up a kubernetes cluster, you establish node trust using the installation method, and whatever network exists between the hosts.
This creates a kubernetes installation, but doesn't setup network. When using the flags to kubeadm, this creates a kubeadm config file, and turns on the kubernetes subnetting. So say you were to configure 10.100.0.0/16 as a pod-network-cidr. The would cause the kubernetes controller manager to assign a /24 to each node for pods. So node 1 gets 10.100.1.0/24, node 2 gets 10.100.2.0/24, etc. The addresses from the node cidr are then assigned the pods as they are scheduled by the cni plugin. The wormhole plugin, then takes care of setting up all the networking that covers 10.100.0.0/24, because kubernetes has effectively been granted management of this subnet.
So, yes, this assumes some sort of network exists between nodes, which is the network that has been used to create mTLS trust between kubelet/api on all members.
Very cool to see WireGuard being used in a mesh implementation. This reminds me of Weave[0] which has worked well for me. I'll definitely be experimenting with Wormhole.
[0] https://www.weave.works/oss/net/
[0] https://www.weave.works/oss/net/
We were recently discussing creating something like for this, the current set of CNI options is wide but shallow. As mentioned in the article CNI is something you want to be as simple as possible, we’ve had trouble with weave and all it’s complexity. Flannel plus encryption is perfect!
Check out Scrambler: https://github.com/jameskeane/scrambler
It's basically this, but with ~100 lines of bash. As simple as it gets.
It's basically this, but with ~100 lines of bash. As simple as it gets.
Just afraid that time will come, when searching for black hole, neutrinos, gravity will primarily yield these kinds of topics. Not actual ones.
Doesn't Istio provide inter-pod encryption, or am I totally off the reservation here?