You Don't Need to Rebuild Your Development Docker Image on Every Code Change(vsupalov.com)
vsupalov.com
You Don't Need to Rebuild Your Development Docker Image on Every Code Change
https://vsupalov.com/rebuilding-docker-image-development/
65 comments
Did you try typescript? Now I can write for hours without trying out the app. And when I try... It just works
This is such a weird comment.. nothing about the OC suggests they were talking about front end dev; not to mention that writing for hours before actually running it is a pretty terrible way to code. Typescript is also not a workaround for volume binding, which can accommodate any stack.
> nothing about the OC suggests they were talking about front end dev
>> I'm never confident in the changes I make, but my style is to write a few lines at a time, save, re-run or refresh.
This is a pretty typical frontend development pattern, especially seeing "refresh."
>> I'm never confident in the changes I make, but my style is to write a few lines at a time, save, re-run or refresh.
This is a pretty typical frontend development pattern, especially seeing "refresh."
I read that as "re-run or refresh, as applicable", i.e. re-run a Go app or refresh a Node.js webapp. Besides, it absolutely can be a big issue with non-frontend development, so the point still stands.
Yes, that is why I specifically mentioned "re-run." About 75% of my work is actually API and backend data engineering work (Python Go, Node in that order). I do "save and re-run" less in that type of work, but definitely not enough to make frequent docker builds a hassle.
As another commenter mentioned, "refresh" tipped them off as a front end dev.
As a backend dev working in statically typed languages, I will sometimes code for hours without running, and I wouldn't say anything about it is terrible. I haven't worked with Typescript, but it wouldn't surprise me if it enabled a similar development process.
I wouldn't necessarily infrequent running as a technique worth emulating, but in certain situations it works pretty well.
As a backend dev working in statically typed languages, I will sometimes code for hours without running, and I wouldn't say anything about it is terrible. I haven't worked with Typescript, but it wouldn't surprise me if it enabled a similar development process.
I wouldn't necessarily infrequent running as a technique worth emulating, but in certain situations it works pretty well.
The majority of my work is actually in Python and Go on the backend. I do save/re-run less than when I'm doing front end, but docker builds are still a hassle. Flask will automatically reload on file changes, taking advantage of a docker volume. With Go, I'm doing go runs in development anyway up until deployment with go build.
Maybe I'm doing things wrong, but docker volumes are essential in how I like to dev.
Maybe I'm doing things wrong, but docker volumes are essential in how I like to dev.
Even there it is beneficial to run sooner. E.g, for servers as soon as the listen is up and dispatching, run it and see that the handler runs. Often times the only times my error handling code runs is during this initial code writing time. (For certain annoying to test for types of error). Fast iterations around a known good baseline.
A star fruit is also known as a carambola.
Honestly I'm quite blown away to know there are people who weren't doing it this way. I think it's due to a fundamental misunderstanding of what docker is for and that is caused by using a solution without knowing the problem.
How awesome would it be if we could exclude specific dirs or files from bind mounts. Think of cache directories, log files, and, of course, node_modules.
The common workaround to this, is to create an empty volume and map whatever you don't want mounted to it. You'll end up with an empty directory in your host system, but its contents are available from within the container.
It has many problems though, mostly permission and ownership related, but it somehow makes Docker for Mac even slower and crash even more often than usual.
Example minimal docker-compose.yaml file for a custom "devcontainer" we use in VSCode that showcases the workaround:
The common workaround to this, is to create an empty volume and map whatever you don't want mounted to it. You'll end up with an empty directory in your host system, but its contents are available from within the container.
It has many problems though, mostly permission and ownership related, but it somehow makes Docker for Mac even slower and crash even more often than usual.
Example minimal docker-compose.yaml file for a custom "devcontainer" we use in VSCode that showcases the workaround:
version: '3.8'
services:
project:
image: devcontainer:dev
container_name: devcontainer
build:
context: .
dockerfile: Dockerfile
args:
ALPINE_VERSION: '3.13'
NODE_VERSION: '14.16.1'
ports:
- 3000:3000
volumes:
- ..:/project
- node_modules:/project/node_modules:cached
volumes:
node_modules:
@bdcravens mentioned this as a solution in another comment:
http://docker-sync.io/
http://docker-sync.io/
Thanks! I came across that project in the past, bookmarked it, and then completely forgot it even existed. Trying it out tonight :)
Using host volumes can have big performance problems on OSX and Windows. Node or frontend development with huge node_modules folders in particular can run into problems like flakey file change notifications or just general slowness accessing files.
Linux users aren't immune from trouble either. Host volume permissions are a big source of pain--if your mounted files don't have the same UID/GUID as the user baked into the dockerfile then you'll hose access to the files on your local machine. Most simple dockerfiles run as root (especially dockerfiles created by windows or mac developers who never deal with this problem) and all the created files inside them show up as root owned on your localhost. You have to take special care to craft your dockerfile to not run as a root user, launch with explicit user UID/GUID that matches your host machine, and use a tool like fixuid as an entrypoint to fix up mismatches.
A good workaround though is to use docker's internal volumes instead of host mounts. This fixes all the Linux permission issues (the internal volumes live in their own root/docker managed location), and fixes the slowness on other platforms (no VM folder mount slowdown). But you give up the freedom of seeing your files locally and need to go all in with something like VS code remote containers, or running your IDE directly in the container.
Linux users aren't immune from trouble either. Host volume permissions are a big source of pain--if your mounted files don't have the same UID/GUID as the user baked into the dockerfile then you'll hose access to the files on your local machine. Most simple dockerfiles run as root (especially dockerfiles created by windows or mac developers who never deal with this problem) and all the created files inside them show up as root owned on your localhost. You have to take special care to craft your dockerfile to not run as a root user, launch with explicit user UID/GUID that matches your host machine, and use a tool like fixuid as an entrypoint to fix up mismatches.
A good workaround though is to use docker's internal volumes instead of host mounts. This fixes all the Linux permission issues (the internal volumes live in their own root/docker managed location), and fixes the slowness on other platforms (no VM folder mount slowdown). But you give up the freedom of seeing your files locally and need to go all in with something like VS code remote containers, or running your IDE directly in the container.
Docker truthfully told is unusable for Dev envs that need to change constantly. It was never built to handle this use case.
I haven't seen a good docker compose file that can be reliably used as a development environment in multiple oses with good performance. There's all sorts of edge cases where volumes don't work.
In my opinion docker is very useful for thing like dbs, queues and other processes where the underlying code doesn't change.
But for everyday frontends and backends it's not worth it.
Nowadays I write a shell.nix file which contains all the dependencies the project needs, it works but is definitely not as easy to learn as a docker file
Dev should look just like production if you are doing it right, and you are correct dev was basically an afterthought in Docker.
If you run this in a screen session it will auto rebuild as you make changes, which is the closest I've been able to come to containerizing my development style: https://gist.github.com/mikedamm/53dc6a78b976eeac88893427424...
If you run this in a screen session it will auto rebuild as you make changes, which is the closest I've been able to come to containerizing my development style: https://gist.github.com/mikedamm/53dc6a78b976eeac88893427424...
When people say docker helps in making dev look like prod, I assume they are running everything on docker in production, but that is very rarely the case with dbs, queues and cloud specific services.
Interesting I'll check this script, but I have doubts doing a rebuild is going to be quick enough even with good cache layering in the Dockerfile. Use case is that I have frontend and backend code in one repo, and make changes across them that need to be reflected in the ui.
Interesting I'll check this script, but I have doubts doing a rebuild is going to be quick enough even with good cache layering in the Dockerfile. Use case is that I have frontend and backend code in one repo, and make changes across them that need to be reflected in the ui.
Hi HN! First time I see one of my articles on here. What a nice surprise.
If the above link caught your attention, you might also enjoy the following ones:
* For the quickest ROI: https://vsupalov.com/improve-your-docker-images/
* Stuff I WISH I knew: https://vsupalov.com/12-docker-facts/
* If your image builds are slow: https://vsupalov.com/5-tips-to-speed-up-docker-build/
Looking forward to join the discussion later!
If the above link caught your attention, you might also enjoy the following ones:
* For the quickest ROI: https://vsupalov.com/improve-your-docker-images/
* Stuff I WISH I knew: https://vsupalov.com/12-docker-facts/
* If your image builds are slow: https://vsupalov.com/5-tips-to-speed-up-docker-build/
Looking forward to join the discussion later!
I have done this but if possible I just run my code locally and reserve docker for infra (e.g. DB). Java, nodejs, go are all very easily run locally.
Maintaining a complex Dockerfile is just more crap that makes programming tedious.
Maintaining a complex Dockerfile is just more crap that makes programming tedious.
Honestly my experience is basically the opposite: Maintaining a coherent development environment is what makes programming tedious.
I'd much prefer to just specify in a dockerfile that I need this compiler, these tools and these env-variables, and then I never have to worry about that stuff again.
I'd much prefer to just specify in a dockerfile that I need this compiler, these tools and these env-variables, and then I never have to worry about that stuff again.
It's hotcode reload and debugging where I find docker starts to annoy me. (Also resource consumption and performance to an extent)
[deleted]
I have the same experience. If cross platform loading of environment variables and shell scripting is needed one can use Powershell. Together with Windows Terminal and tmux for Linux and Mac, you can start every microservice in different panes and tabs. You won't get the full Docker experience where you know for a certainty that your code will run the same way in production, but leave that to ci/cd, perhaps a local kubernetes cluster you can use for production-like testing.
Sure, nodejs is easy to run locally. nodejs+npm of the exact version that a specific project needs, which is different and incompatible with the version some other project needs starts to get more annoying.
Basically all of our projects have some level of external dependencies in the form of libraries or binary tools. Some projects overlap dependencies, but on incompatible versions. This gets really hairy.
Trying to do this without containers (from experience, this is what the devs were doing before) turns into a massive documentation project and a nightmare of trying to get all these incompatible things to exist on a single system where we don't even have a consistent target because everyone's local dev environment is different enough to make it a nightmare. And then continually hounding the devs to keep the documentation up-to-date. But even that doesn't solve the problem because "apt-get install nodejs" today is not necessarily the same as "apt-get install nodejs" in six months or a year and nobody's thinking about this so it's just "worked for me last time!". Onboarding people at one point was a _days_ long process and a group effort.
Instead we just re-use the same 30-odd line Dockerfile we use for deploying out to the infra for local dev. Instead of pages upon pages of out-of-date documentation. a bunch of tribal knowledge, and continually running into new and interesting bugs, we now have a short and simple source of truth which _has_ to be up-to-date because otherwise the environment the project is deployed to is broken. Onboarding people takes 5 minutes.
If you're a single person working on a single project, yeah, containerizing stuff is maybe some annoying overhead. Even then I still use it because once you've gotten past the learning curve, it's actually a really effective way to document and define your project's environment and ensure it stays consistent.
Basically all of our projects have some level of external dependencies in the form of libraries or binary tools. Some projects overlap dependencies, but on incompatible versions. This gets really hairy.
Trying to do this without containers (from experience, this is what the devs were doing before) turns into a massive documentation project and a nightmare of trying to get all these incompatible things to exist on a single system where we don't even have a consistent target because everyone's local dev environment is different enough to make it a nightmare. And then continually hounding the devs to keep the documentation up-to-date. But even that doesn't solve the problem because "apt-get install nodejs" today is not necessarily the same as "apt-get install nodejs" in six months or a year and nobody's thinking about this so it's just "worked for me last time!". Onboarding people at one point was a _days_ long process and a group effort.
Instead we just re-use the same 30-odd line Dockerfile we use for deploying out to the infra for local dev. Instead of pages upon pages of out-of-date documentation. a bunch of tribal knowledge, and continually running into new and interesting bugs, we now have a short and simple source of truth which _has_ to be up-to-date because otherwise the environment the project is deployed to is broken. Onboarding people takes 5 minutes.
If you're a single person working on a single project, yeah, containerizing stuff is maybe some annoying overhead. Even then I still use it because once you've gotten past the learning curve, it's actually a really effective way to document and define your project's environment and ensure it stays consistent.
If only life was such a beauty.
Now you have bunch of new and interesting problems with docker ecosystem.
Newer tools are much much better in this cross-dependency problem as they evolved too, not only docker. I don't remember dependency hell nowdays while it was more or less common 15 years ago.
I had the same project with and without docker and I totally prefer without docker. There are few quirks here and there but problems are mostly solvable within minutes unlike basically anything you can get with docker.
Now you have bunch of new and interesting problems with docker ecosystem.
Newer tools are much much better in this cross-dependency problem as they evolved too, not only docker. I don't remember dependency hell nowdays while it was more or less common 15 years ago.
I had the same project with and without docker and I totally prefer without docker. There are few quirks here and there but problems are mostly solvable within minutes unlike basically anything you can get with docker.
My own fix for the node version issue is that we use a single version of nodejs (currently latest v12), and when we update, we update everything (which happens every two years, and is generally not too painful in my experience), so we just have to have this version locally.
Yes, it's do-able. As others have pointed out, nvm also works.
1. There isn't nvm for things that aren't node. Some things have an equivalent (e.g., pyenv). Many external dependencies do not.
2. I've inherited a spiderweb of like two dozen different major projects that are all interconnected. Doing an upgrade across everything at once means a stop-work for months while all the developers update tooling, then fix all their issues so everything works, then we try and coordinate testing and deploys of everything without breaking anything.
Or, alternatively:
1. We don't need extra tooling that doesn't exist.
2. We can update projects one at a time instead of mass-migrating several million lines of code across half a dozen different frameworks.
1. There isn't nvm for things that aren't node. Some things have an equivalent (e.g., pyenv). Many external dependencies do not.
2. I've inherited a spiderweb of like two dozen different major projects that are all interconnected. Doing an upgrade across everything at once means a stop-work for months while all the developers update tooling, then fix all their issues so everything works, then we try and coordinate testing and deploys of everything without breaking anything.
Or, alternatively:
1. We don't need extra tooling that doesn't exist.
2. We can update projects one at a time instead of mass-migrating several million lines of code across half a dozen different frameworks.
nvm (node version manager) is your friend. Allows you to use different node versions for different projects.
I used to use nvm, nave, n and similar things, but I think having a single version of node for all projects is easier and better. I just have unattended upgrade setup everywhere, so I have always the latest v12 without having to think about it.
It's my impression that there are lots of platforms that ask very little of their environment, and therefore don't benefit much from Docker, and therefore can be developed with minimal friction in a local environment without using it. Node, Java, often Python, even Rust because of its static dependencies, etc often fall into this category. And of course front-ends, especially.
In practice you'll still use Docker to deploy because it's a nearly-unavoidable deployment protocol at this point. But I see very little reason to have it house your in-progress code in these cases.
In practice you'll still use Docker to deploy because it's a nearly-unavoidable deployment protocol at this point. But I see very little reason to have it house your in-progress code in these cases.
Works remotely on kubernetes, for us the game changer was running inotify_wait on cygwin for our windows users, changes are up in about 600ms.
1. Configure a service 2. Use start command sleep infinity 3. Install inotify_wait on windows 4. Do a loop like this
Look for changes, Rsync changes to pod, Pkill java, Run start.sh
We do a grep on changed files and only kill java if jar/classes and other deps has changed, that makes it possible to edit html in pod and get fast updates.
1. Configure a service 2. Use start command sleep infinity 3. Install inotify_wait on windows 4. Do a loop like this
Look for changes, Rsync changes to pod, Pkill java, Run start.sh
We do a grep on changed files and only kill java if jar/classes and other deps has changed, that makes it possible to edit html in pod and get fast updates.
I have never understood kubernetes development in local development workflow unless kubernetes functionality being developed or some yaml magic. It’s super overhead IMHO. If needed to test api’s as unit test you can always port-forward traffic of desired enpoint or use something like telepresence. Less kubernetes is always better
It’s cpu/memory overhead yes, but between matching prod and dev, extremely simple config (redis is always located at “redis:6379”), zero setup instructions (skaffold dev), remote/local hybrid development, and fostering prod-ready skills for developers (still working on that devops koolaid), I find it -extremely- worthwhile. I’ve yet to meet someone who doesn’t hate it to start (because kubernetes) and winds up loving it and swearing by it.
It’s essentially a vastly superior “vagrant up” on anabolic steroids
It’s essentially a vastly superior “vagrant up” on anabolic steroids
I agree, for e2e/integration testing, excellent
Have you seen https://skaffold.dev/ ?
When you have a team that already know the out and ins of Kubernetes it's easier to just use something that has all the integrations you need locally, certs, end points, firewall. Especially when you are not allowed to install things on your local machine.
Garden[1] is a tool we built (yes, I'm affiliated :)) that has a lot of this functionality built-in. Might fit your use case.
We recently re-wrote the hot reload functionality to use Mutagen[2] under the hood and it's insanely fast (<200ms anecdotally). It also does two way sync which can be useful. The old implementation used rsync but a lot of our Windows users struggled with that. So I figured I'd share in case that sounds familiar.
What happens after a sync event depends on the stack, but we've had pretty good success with Entr[3]. We often have it watch a single file so that multiple watchers in a shared dev cluster don't eat up all the node's resources.
1: https://github.com/garden-io/garden 2: https://mutagen.io/ 3: http://eradman.com/entrproject/
We recently re-wrote the hot reload functionality to use Mutagen[2] under the hood and it's insanely fast (<200ms anecdotally). It also does two way sync which can be useful. The old implementation used rsync but a lot of our Windows users struggled with that. So I figured I'd share in case that sounds familiar.
What happens after a sync event depends on the stack, but we've had pretty good success with Entr[3]. We often have it watch a single file so that multiple watchers in a shared dev cluster don't eat up all the node's resources.
1: https://github.com/garden-io/garden 2: https://mutagen.io/ 3: http://eradman.com/entrproject/
This is great, until you need to watch a tree of files in macOS. Then docker takes a core to do the fsnotify/watch.
There are caching settings to make it better, but there are race conditions between the watch and when file contents change, so sometimes the JS stack will compile an inconsistent file.
OTOH, on linux, it's grand.
There are caching settings to make it better, but there are race conditions between the watch and when file contents change, so sometimes the JS stack will compile an inconsistent file.
OTOH, on linux, it's grand.
docker-sync fixes this by using an intermediate container that handles sync (I believe there are other options which work essentially the same way). It supports ignoring files you don't need on your main machine, like tmp folders and the like, which can improve performance further.
http://docker-sync.io/
http://docker-sync.io/
While this is great for people with a fundamental understanding of containers and your prod environment this will usually lead to some issues with developers that don't need to, or want to, have context in these areas.
In the past, to make a very similar workflow possible, I've built tools that automatically watch your source files and rebuild & restart only what is needed [0]. This was built for bazel + docker-compose but there isn't a reason one couldn't watch the "build:" contexts for what files are important.
At a previous company one of our engineers was a huge fan of this volume mount approach and every single time something broke (which was very frequent due to some prod/dev env magic we had) I had to assist quite a few more junior devs figure out what was wrong with their machine. For those with scripting languages, was it their system's newline endings? For compiled languages, was their system SDK different then what was in the container? For prod bugs, did they forget to rebuild & test the container before opening their PR (we had no automated integration testing)?
In my opinion, if you can make your build system in charge of building/packaging things you'll have a much happier time.
[0] - https://github.com/CaperAi/bazel_compose
In the past, to make a very similar workflow possible, I've built tools that automatically watch your source files and rebuild & restart only what is needed [0]. This was built for bazel + docker-compose but there isn't a reason one couldn't watch the "build:" contexts for what files are important.
At a previous company one of our engineers was a huge fan of this volume mount approach and every single time something broke (which was very frequent due to some prod/dev env magic we had) I had to assist quite a few more junior devs figure out what was wrong with their machine. For those with scripting languages, was it their system's newline endings? For compiled languages, was their system SDK different then what was in the container? For prod bugs, did they forget to rebuild & test the container before opening their PR (we had no automated integration testing)?
In my opinion, if you can make your build system in charge of building/packaging things you'll have a much happier time.
[0] - https://github.com/CaperAi/bazel_compose
What does it mean to "make your build system in charge of building/packaging things"? Like push your code out to Jenkins and have it build it? Or have Gradle make something that runs for you?
For devs on macOS, keep in mind that your filesystem is case insensitive!
If you're using a linux image that expects a directory like `UpperCase` and you named it `uppercase` locally it would work.. until you bake the source for your production release and you get errors around that directory not existing.
"But it works locally!"
If you're using a linux image that expects a directory like `UpperCase` and you named it `uppercase` locally it would work.. until you bake the source for your production release and you get errors around that directory not existing.
"But it works locally!"
there can also be subtle issues with git, I recommend then to convert HFS / APFS to case-sensitive. I understand why normal people do not care, but Dev types in unix alike environments do not have a choice with decades of case-sensitive filesystems before them
A little known feature is that the Docker integration in Visual Studio gives you this automatically.
We use this where I work and love it. It has the added benefit of letting you package preferred vscode extensions with the devcontainer as well.
I think PyCharm Professional's docker-compose remote interpreter[0] does too. It was a lot of setup but it's the only way I know to have your code run inside containers and actually be able to stop at breakpoints inside them. (I don't know much about Docker.)
0: https://www.jetbrains.com/help/pycharm/using-docker-compose-...
0: https://www.jetbrains.com/help/pycharm/using-docker-compose-...
Container fast run or something like that right ? If you observe the commands it does, it simply mounts everything it needs on the container and uses the first stage of the docker file (which is usually a base image) to run your stuff.
Pretty sure they are talking about https://code.visualstudio.com/docs/remote/containers
Nix has saner repeatable dev envs, and makes updating or adding deps a snap.
Can be used to build scratch containers. Can be used to cross compile.
The only major downside is it requires learning and understanding nix which I get is a hurdle, but one that's well worth it.
Docker is only one container tool of many now, and it's worth exploring what else is out there.
Can be used to build scratch containers. Can be used to cross compile.
The only major downside is it requires learning and understanding nix which I get is a hurdle, but one that's well worth it.
Docker is only one container tool of many now, and it's worth exploring what else is out there.
In the past few weeks, I have spent some time and released dew [0]. It helps encapsulating this kind of setups in configuration and minimising typing. dew is still evolving, but it has served me well.
[0]: https://github.com/efrecon/dew
[0]: https://github.com/efrecon/dew
For any rails dev out these
volumes:
volumes:
- .build/.bundle-cache-dir/app:/usr/local/bundle/[deleted]
Skaffold is a great tool for this and employs hot reloading techniques without having to rebuild the image. https://skaffold.dev/
I find it very interesting the idea of separating development image from production one and skaffold promotes it in a way ("skaffold dev" vs "skaffold run")
Same as in frontend for example you don't "npm build" your way through development, instead you want hot-reload and other similar features.
Same as in frontend for example you don't "npm build" your way through development, instead you want hot-reload and other similar features.
We wrote a tool to make using development Docker images easy - https://metaswitch.github.io/floki/
Looks awesome!
Another thing I've recently found helpful was docker layer caching inside github actions. Pretty easy to integrate, saves a lot of build minutes.
I've been doing this for about 6 months now, it saves a lot of time. Especially if you need to upload changed docker images to AWS.
I have meant to try if VMWare's vctl is any better regarding detecting file changes
[deleted]
Maybe it's because I'm never confident in the changes I make, but my style is to write a few lines at a time, save, re-run or refresh. I can't imagine having to do a docker build every code change. If you do the later, it's just nominally better than developing on a remote server, which brings its own challenges.