Etcd 3.2 now with massive watch scaling and easy locks(coreos.com)
coreos.com
Etcd 3.2 now with massive watch scaling and easy locks
https://coreos.com/blog/etcd-3.2-announcement
3 comments
Is this in the critical path to enable Kubernetes to scale over 5000 nodes?
Every little bit helps, but no, etcd has not been an issue for Kube since 3.1. We already have a watch cache in the apiserver that supports large number of watches.
The issues right now are about optimizing how nodes communicate with the master for the tens of related resources used for pods (config maps, secrets, PVCs, etc). That's more classic optimization of how we batch that data.
I will say etcd 3.2 is fantastic because it removes the snapshot size limit for etcd2 - we've got Kubernetes clusters with 1-2Gb of etcd data and when the write rate gets high enough we are snapshotting so fast that if a member drops out you can't rsync the snapshot to the remote system fast enough to do a restore before the next snapshot hits. So this is a big win for us - thanks to the etcd team for getting that in at the last moment!
The issues right now are about optimizing how nodes communicate with the master for the tens of related resources used for pods (config maps, secrets, PVCs, etc). That's more classic optimization of how we batch that data.
I will say etcd 3.2 is fantastic because it removes the snapshot size limit for etcd2 - we've got Kubernetes clusters with 1-2Gb of etcd data and when the write rate gets high enough we are snapshotting so fast that if a member drops out you can't rsync the snapshot to the remote system fast enough to do a restore before the next snapshot hits. So this is a big win for us - thanks to the etcd team for getting that in at the last moment!
Watches in etcd were so bad, they needed a full http request respond cycle for each event. And when you weren't fast enough to reconnect again you could miss events..