The replacement machine will start pulling its data from the remaining nodes when it joins the cluster. However, it's recommended to migrate the failed node's data first if it's greater than 50MB: https://github.com/coreos/etcd/blob/master/Documentation/op-...
This is exactly the conclusion we arrived at. We re-build our base images nightly (including all available updates) and then base the application images on those.
As Musk mentioned [1], landing on the barge is a necessity, due to the speed required to get into higher orbits. It's impossible to return the first stage to land.
We've moved to using Hashicorp's Consul-Template (https://github.com/hashicorp/consul-template). Ansible populates Consul with any required configuration changes during the deployment of a new version, and Consul-Template knows about these changes and automatically writes them to disk. Applications running on the host are then reloaded to pick up the changes.
We're running all the containers on Mesos hosts, so really all Ansible needs to do for us is talk to Marathon. We realized early on down this path that to accommodate scale we'd need to have some sort of scheduler. Mesos happened to be the most robust.
We originally tried the docker module in Ansible but found it had a few problems. There's been a lot of work on it since, and I expect it will be in a much better state when Ansible 2.0 is released.
We eventually settled on having Ansible build an AMI for us that can then be spun up by as part of a Cloudformation template (also initiated by Ansible).
We've actually been moving further and further away from having Ansible handle the configuration management side of things, and deal with Orchestration primarily.
I'd agree that jinja2 has a perfectly fine way of handling loops, but the problem is that functionality only works in Ansible template files.
The Ansible playbook can only make use jinja2 filters to act on variables:
https://docs.ansible.com/ansible/playbooks_filters.html
I think one thing to keep in mind about Ansible is that it's an orchestration tool that also does configuration management. We've integrated Ansible into our workflows in such a way that it kicks off everything we need to do, even if that involves just coordinating some info between APIs.
We don't mutate containers at all - merely get Ansible to make things happen around their deployment and communication.
We tend to handle that part inhouse. We're using Jenkins and have it set up to build a standard set of base images with all the latest updates daily. It can be run on-demand as well.
All containers running code are based on these images, so the updates are picked up on the next build/deploy.
I can second this recommendation. One thing to note: The small folding version is less resilient than the larger model. A fine tradeoff, given its size.
"The technology of deployment does not change 'minutes or hours' into 'days or months'"
I wouldn't say that's true. We're transitioning into multiple languages, and want to have an environment that will allow future languages to be added as required. Building a generic infrastructure to run containers lets us run everything on the same base platform. Otherwise, we'd need to tailor the images and configuration for the individual language type. When a new language is introduced, it can take 'days or months' to get everything working well.
That's not to say Docker doesn't require the same attention to security as other options. This seems to me akin to running a downloaded base VM image without first doing updates.
