We will open source highlander eventually, but it’s not quite there yet.
We use a VIP that the NUCs share... ie; one of the three will always have a VIP, and if it dies another NUC grabs it. This is a poor man’s load balancer in that sense, because we only have the NUC hardware onsite;
Iot device control, such as timers for food, and cameras tracking food (to name a few). These also output streams of data that is interesting to us, such that we want to exfiltrate them up to the cloud.
I’m not a fan of hype... k8s is one of the few hyped technologies that has delivered on its promises.
Credit card transactions, mobile orders, timer synchronization, order receiving (tablets etc), iot devices (cameras, cooking devices) and other things planned for the future.
Our tolerance requirements for synchronicity are broad enough that we can tolerate blips like this... at the end of the day we are automating away some simple human interactions (ie; fry the fries or track the food), we aren’t performing surgery with these systems.
So far we’ve been satisfied with Linux (Ubuntu 18.04 to be exact) and it’s overall capabilities.
Also, don’t forget that cost at scale is a big factor... doing things perfectly but expensively is not profitable at 2k locations.
Could you define what is not deterministic about containerized computing?
Examples of things we are controlling are timers for food, cameras that can recognize and track food items, drying machines that will automatically trigger and fry fries.
Well, it’s a 6 man team so most of our trade offs were for expedience, not the worlds greatest architecture. We cheated and sync’d data using a HA MongoDB setup amongst the cluster. RKE for K8s clustering was a life saver (nothing easier at this point on bare metal IMO), although RKE can be brittle at times.
The primary challenge was just reasoning with the template and using Helm at scale... ie; what exactly did we deploy on those hundreds of varying clusters?
Other issues included; tiller would sometimes become unstable... version mismatch issues between helm local and roller... lack of a clear, outage free canary deployment... we even found cases where helm would not cleanup after itself during a deployment and retain previous config settings within k8s.