True for long running services, but not necessarily ephemeral workloads. Those need a system to schedule them or else once they complete they just stop running.
Also, as long running services crash or reboot naturally they need to be rescheduled or else your cluster slowly dies, and as your cluster size increases the mtbf decreases and the need to reschedule workloads continually increases.
Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are.
Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul.
Nomad is an application/workload scheduler. You tell it what you need to run and what their memory/cpu requirements are and it finds a space in your physical infrastructure to run that. The apps it runs normally needs secrets from vault and communicate with services discovered via consul.
They all are well integrated and build off of each other, kind of like layers an onion where your app services are the outer layer of the onion. Consul failing this badly is like the core of the onion going rotten. There’s not much saving it at that point, you need to grow a new onion from the inside out, but that takes time.
Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 (https://www.hashicorp.com/blog/announcing-hashicorp-consul-1...) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers.
Furthermore, the services were down for long enough that the asset caches went fully cold, so spinning back up to 100% capacity would put far more load than usual on the servers meaning recovery needed to be a very slow incremental rollout to warm them.
Messy situation all around, but if you want the real root cause pay attention to the consul change log over the next few weeks I’d say.
Also, as long running services crash or reboot naturally they need to be rescheduled or else your cluster slowly dies, and as your cluster size increases the mtbf decreases and the need to reschedule workloads continually increases.