Introduction to Containerization(medium.com)
medium.com
Introduction to Containerization
https://medium.com/devopslinks/the-missing-introduction-to-containerization-de1fbb73efc5
22 comments
A great history of containerization starting from chroot type jailed environments in the late 1970s to the dizzying plethora of options we have today.
So...this all seems insanely complicated. Is there any kind of rough guidance about how "big" an organization or codebase needs to be before it becomes a net win to go down this path? Either in terms of lines of code or number of services or requests served or servers needed or any other kind of metric? Obviously the breakeven point is somewhere between "CRUD app that runs for free in Heroku" and "millions of requests served per second" but I don't know where it is.
I don't know that it's about size so much as it's about mix.
The biggest, most complicated, most microservices-y organization I ever worked at didn't use containerization, and didn't want to use containerization. It also had the most smoothly-running system I've ever seen.
But they also had a very strictly controlled run-time environment. There was typically only one set of run-time dependencies that was consistent on all servers and in dev. Meaning that all services could be spun up and spun down on any server, without having to worry about fancy wrappers to make sure that they could maintain their own special snowflake needs that were incompatible with everyone else's special snowflake needs. But it also meant that developers couldn't create special snowflake needs, which in turn meant that taking on a new dependency wasn't a simple package manager operation; it involved coordination with operations and perhaps across many product teams.
As draconian as it sounds, for the most part everyone there loved it. It was just such an easy place to work, because the whole IT situation was uncannily straightforward and simple and consistent.
By contrast, I have also seen a company whose employees I could count on one hand, with a fairly small and tightly focused product, that absolutely needed to be Dockerized. Primarily because every single component was written in a different language, or a different version of the language, or used a different NoSQL datastore, etc. And at that point, with that much mix, you absolutely need some way to layer a consistent abstraction on top of all of it. Otherwise you'll be spending all your time fiddling with environment issues.
The biggest, most complicated, most microservices-y organization I ever worked at didn't use containerization, and didn't want to use containerization. It also had the most smoothly-running system I've ever seen.
But they also had a very strictly controlled run-time environment. There was typically only one set of run-time dependencies that was consistent on all servers and in dev. Meaning that all services could be spun up and spun down on any server, without having to worry about fancy wrappers to make sure that they could maintain their own special snowflake needs that were incompatible with everyone else's special snowflake needs. But it also meant that developers couldn't create special snowflake needs, which in turn meant that taking on a new dependency wasn't a simple package manager operation; it involved coordination with operations and perhaps across many product teams.
As draconian as it sounds, for the most part everyone there loved it. It was just such an easy place to work, because the whole IT situation was uncannily straightforward and simple and consistent.
By contrast, I have also seen a company whose employees I could count on one hand, with a fairly small and tightly focused product, that absolutely needed to be Dockerized. Primarily because every single component was written in a different language, or a different version of the language, or used a different NoSQL datastore, etc. And at that point, with that much mix, you absolutely need some way to layer a consistent abstraction on top of all of it. Otherwise you'll be spending all your time fiddling with environment issues.
Think about using containers if any of the below apply:
- You have heard or said the words "Hmm, it worked on my machine"
- You run or want to run multiple applications on the same host
- You care about software supply chain
- You think serverless patterns may apply to your workloads
- Scheduling applications on to your infrastructure is manual
- You have heard or said the words "Hmm, it worked on my machine"
- You run or want to run multiple applications on the same host
- You care about software supply chain
- You think serverless patterns may apply to your workloads
- Scheduling applications on to your infrastructure is manual
> - You have heard or said the words "Hmm, it worked on my machine"
I currently use VirtualBox + SaltStack to solve this problem, which has worked well and has significantly less overhead than going full Docker / K8s
I currently use VirtualBox + SaltStack to solve this problem, which has worked well and has significantly less overhead than going full Docker / K8s
Does anyone know if there's a similarly great "missing introduction to containerization," but for a less CS oriented audience.
I'm currently trying to convince my team members that using a dockerised version of our internal django project is great for loads of reasons (portability and use in continuous integration being the main two I'm trying to sell on as our team is rapidly growing from 4 developers last year to 12 this year)
However, these reasons aren't resonating with them, since they're not seeing the cracks starting to form with the growing development team, and to them the project is working just fine on the server as is and the very simple CI is working just fine as is.
Are there good case studies or historical perspective that would help show how containerization may help as we grow even further over the next while?
I'm currently trying to convince my team members that using a dockerised version of our internal django project is great for loads of reasons (portability and use in continuous integration being the main two I'm trying to sell on as our team is rapidly growing from 4 developers last year to 12 this year)
However, these reasons aren't resonating with them, since they're not seeing the cracks starting to form with the growing development team, and to them the project is working just fine on the server as is and the very simple CI is working just fine as is.
Are there good case studies or historical perspective that would help show how containerization may help as we grow even further over the next while?
TBH, if I were you, I would wait until they see the cracks starting to form for themselves.
Containerization isn't perfect, and has its own challenges. There will be growing pains as you move to it, especially if most the team is unfamiliar with the technology. If you try to push it before people perceive (for themselves) that they are experiencing some of the problems that it's designed to solve, then, on top of all that, you're going to have an audience that sees it as a solution in search of a problem.
Conditions like that are just about perfect for kicking off a chain of events that leads to yet another "Containerization is the worst idea ever, never containerize anything" blog post entering the world.
Containerization isn't perfect, and has its own challenges. There will be growing pains as you move to it, especially if most the team is unfamiliar with the technology. If you try to push it before people perceive (for themselves) that they are experiencing some of the problems that it's designed to solve, then, on top of all that, you're going to have an audience that sees it as a solution in search of a problem.
Conditions like that are just about perfect for kicking off a chain of events that leads to yet another "Containerization is the worst idea ever, never containerize anything" blog post entering the world.
Appreciate the measured warning, as you mentioned there's a lot of strong opinions either way so it's hard to find the more practical advice among it all which is what inspired the original question.
I'll take all the advice here and get better understanding of the overheads and pain myself (and hopefully a few other interested folk) so that when the cracks do start showing properly there's a working containerized solution to roll out more smoothly.
I'll take all the advice here and get better understanding of the overheads and pain myself (and hopefully a few other interested folk) so that when the cracks do start showing properly there's a working containerized solution to roll out more smoothly.
I would seriously reconsider this position. Containerization adds a significant amount of overhead to tasks that your team is probably already comfortable with. It's not a clean, simple tool or abstraction as this article demonstrates. I've seen teams get eaten by docker hype and it's not pretty.
Thanks for putting some brakes on the hype train!
We're a pretty diverse team of developers with a wide range of skill levels and development environments. I was hoping that docker may have helped some developers get their environment up and running quicker and more standardised, but I can totally see how the opposite could happen if we're adding yet more overhead to the process.
I'm going to keep tipping away at the idea in the background, since I've already learned a lot about building a better non-containerized version in the process, but I'm reconsidering the idea of rolling it out to the team.
We're a pretty diverse team of developers with a wide range of skill levels and development environments. I was hoping that docker may have helped some developers get their environment up and running quicker and more standardised, but I can totally see how the opposite could happen if we're adding yet more overhead to the process.
I'm going to keep tipping away at the idea in the background, since I've already learned a lot about building a better non-containerized version in the process, but I'm reconsidering the idea of rolling it out to the team.
Choosing to move to containers is somewhat similar to choosing to move to a service oriented architecture in the sense that it formalizes a division of labor. You will have to dedicate some of your people to making sure that anything works at all.
Some people prefer this because it's easier to understand what people are spending their time on. Whereas the cracks and seams of sharing work as a team grows are harder to identify. There's lots of an hour or two lost here because of a merge conflict or a half day there because the django project won't start up unless it can find redis.
The question for you is where are you in your product lifecycle? Are you in a place where you can take several people off of feature development and maintenance and dedicate them to, essentially, DevOps? Were you hiring with that in mind? Are people on your team interested in moving to that kind of work?
I really don't think there should be that many cracks in a 12-person dev team that drive a technical need to use containers (or SOA, for that matter).
Some people prefer this because it's easier to understand what people are spending their time on. Whereas the cracks and seams of sharing work as a team grows are harder to identify. There's lots of an hour or two lost here because of a merge conflict or a half day there because the django project won't start up unless it can find redis.
The question for you is where are you in your product lifecycle? Are you in a place where you can take several people off of feature development and maintenance and dedicate them to, essentially, DevOps? Were you hiring with that in mind? Are people on your team interested in moving to that kind of work?
I really don't think there should be that many cracks in a 12-person dev team that drive a technical need to use containers (or SOA, for that matter).
I got my team onboard by creating my own containers for our apps to use. As others saw my increased development productivity using those personal containers they got interested and I taught them.
The advantage is that I’d quietly figured out the pain and process so it was smooth sailing for them when they started using it.
After that, it was not difficult to start talking about using containers for CI and production.
The advantage is that I’d quietly figured out the pain and process so it was smooth sailing for them when they started using it.
After that, it was not difficult to start talking about using containers for CI and production.
> using a dockerised version of our internal django project is great for loads of reasons
It's an internal project? What is the uptime and DR requirements for it?
It might not be worth dockerizing if you don't need to scale out. Is it a traditional project that can just scale to larger machines as needed? Can you just run on multiple static environments for development?
If you are making something that needs to autoscale on kuberneties, do real CI (not just continuous build), etc, then sure.
It's an internal project? What is the uptime and DR requirements for it?
It might not be worth dockerizing if you don't need to scale out. Is it a traditional project that can just scale to larger machines as needed? Can you just run on multiple static environments for development?
If you are making something that needs to autoscale on kuberneties, do real CI (not just continuous build), etc, then sure.
Thanks for the input, given the generally negative outlook in the comments on this idea, maybe multiple static environments might solve enough of our issues.
For uptime requirements, it's the main tool our company uses to organise and interrogate our scientific output and processes, so it's important it gets more stable. My primary reason for heading towards containerization was the "Hmm, it worked on my machine" reason someone mentioned above and the fact we were using Gitlab for CI meant that dockerizing seemed like a natural step.
I'm definitely going to tread more carefully into this domain given all the feedback here, I appreciate all the comments.
For uptime requirements, it's the main tool our company uses to organise and interrogate our scientific output and processes, so it's important it gets more stable. My primary reason for heading towards containerization was the "Hmm, it worked on my machine" reason someone mentioned above and the fact we were using Gitlab for CI meant that dockerizing seemed like a natural step.
I'm definitely going to tread more carefully into this domain given all the feedback here, I appreciate all the comments.
You might want to start by implementing CD for dev and test using Docker and watchtower: https://github.com/containrrr/watchtower
Keep an eye out for cracks starting to form, or something just squeaking occasionally.
When it does, half-joke Y'know, that wouldn't happen with a container!
When it does, half-joke Y'know, that wouldn't happen with a container!
Great article.
But I feel there is a slight confusion about where LXC ends and LXD starts.
But I feel there is a slight confusion about where LXC ends and LXD starts.
Great start of an article. Question: why are we documenting this stuff in blog posts, rather than a Wiki?
The kind of article I started writing ! Only in much better than my draft :-)
But that picture... doesn't have any container ships in it!
Umm, no. It was Sun Solaris Containers (Oracle didn't purchase Sun until 2010), and it wasn't at all an implementation of Linux-VServer (which already ran on X86 and SPARC processors in any case). Actually the original inspiration for Solaris Containers (aka zones) was FreeBSD jails.