The Slippery Slope of Docker Dev Environments(blog.testdouble.com)
blog.testdouble.com
The Slippery Slope of Docker Dev Environments
https://blog.testdouble.com/posts/2020-02-11-the-slippery-slope-of-docker-dev-environments/
77 comments
The problem we have with that at my workplace is that it's hard to do things like run IDEs/set breakpoints, run gdb or strace, get a REPL (with useful REPL tooling), etc. inside a production-style container. I understand the arguments that you don't want to do this in actual production, you want to kill the offending container and figure out the problem from logs - but once you've got an idea of where the problem lies, for local dev you do actually want to debug problems this way. Even editing a random config file or editing source code to add a printf is more difficult inside a production-style container than in a local checkout. Have you found good approaches for this?
For me, the other big complication is that we do backend stuff in Java... and God help you if you want a self-contained image that can intelligently and quickly reload Java code for anything that's not a very specific arrangement of frameworks.
It's night and day compared to the Javascript world, where it's nearly a given that the bare minimum should be watchers that recompile code and restart a server before you finish switching window focus.
It's night and day compared to the Javascript world, where it's nearly a given that the bare minimum should be watchers that recompile code and restart a server before you finish switching window focus.
Yeah, I wonder to what extent "Just put it all in Docker" works for specific language communities, and if you're not in those communities the advice falls flat. (My employer is pretty polyglot, but primarily Java; my team is mostly Python; we also build our own C code as well as third-party C code and need to debug it every so often.)
I mean, what Docker does is, it turns a network of executables and scripts into a single executable with no outside dependencies.
It works extremely well for language communities that need this, in particular it caught on because you would build something and test it locally and it would work just fine, but then you are running the wrong version of Python or pip installed the wrong version of a library, or you have the wrong version of PHP or Node.js relative to Prod.
For Java and Go, I believe they have already settled these problems, no? You might shell out to other commands and have a weak dependency on your runtime environment that way, but if you stay within your lane, then you already have an executable with all of its dependencies bundled in. They just have a lot less to gain from Docker. I can't see why they would have more to lose.
It works extremely well for language communities that need this, in particular it caught on because you would build something and test it locally and it would work just fine, but then you are running the wrong version of Python or pip installed the wrong version of a library, or you have the wrong version of PHP or Node.js relative to Prod.
For Java and Go, I believe they have already settled these problems, no? You might shell out to other commands and have a weak dependency on your runtime environment that way, but if you stay within your lane, then you already have an executable with all of its dependencies bundled in. They just have a lot less to gain from Docker. I can't see why they would have more to lose.
As someone who mostly works on the JVM, I think this is exactly right. Docker-based JVM development environments add annoying layers of indirection and complexity for close to zero benefit.
All the times I can think of when we've run into "works in dev, not in prod" issues that weren't pilot error, it has been down to stuff like firewall configuration or the fact that production is distributed across more nodes than people can realistically simulate on a single dev machine. Never once that I can remember have we had a single production problem caused by the JVM working differently in dev environments.
It is pretty telling that if you're looking online for tutorials about how to effectively use Docker in a development workflow, you will find essentially none that involve running a compiler.
All the times I can think of when we've run into "works in dev, not in prod" issues that weren't pilot error, it has been down to stuff like firewall configuration or the fact that production is distributed across more nodes than people can realistically simulate on a single dev machine. Never once that I can remember have we had a single production problem caused by the JVM working differently in dev environments.
It is pretty telling that if you're looking online for tutorials about how to effectively use Docker in a development workflow, you will find essentially none that involve running a compiler.
OK, but, even for Python that solves the problem once you're in the container, but it doesn't really address the problem of whether my local Python installation, the one my IDE uses, is using the same version and the same libraries - unless you either decide I don't get to use an IDE or that my IDE goes into the container. Right?
Yes, you are exactly right.
I mentioned it further down, but my team and I are having preliminary success by integrating with VSCode's "Remote: Docker" extension. They've got it rigged so that all language support can run as remote servers inside the container. They give you a special .json file to control how you'd like to start the image, and you can supply a project-specific dockerfile that takes any additional IDE-specific steps that you want to add in as layers when opening up the project's dev container.
It still has some rough edges - for some reason, you can't supply Build-time arguments to the devcontainer Dockerfile I mentioned above, which makes it hard to get permissions to be non-annoying if you're worried your users will have odd UIDs. And there are a couple other random ticky tack things. But it's by far the closest to actually "developing out of a container" as you and I are imagining it, with all of the attendant benefits. When you want a new dev to start working on the project, they basically do this:
1. git clone newproject 2. Open that project in vscode 3. Respond to a pop-up that yes, I do want to open this in a container 4. <wait for the dev-specific layers to be built> 5. Do a normal project build 6. Enjoy full intellisense, syntax highlighting, import resolution, etc. in exactly the environment that's going to run.
In addition, by including the Dockerfile that builds the base image (the one that the dev container FROM's) into the source, developers can now change library versions and dependencies as a git commit, and then coerce other developers into automatically updating their environments by pushing a new image with a new tag to your gitlab / w/e, and then updating the tag mentioned in the FROM in your .devcontainer/Dockerfile. It's like how you would expect development to feel in 2020.
I mentioned it further down, but my team and I are having preliminary success by integrating with VSCode's "Remote: Docker" extension. They've got it rigged so that all language support can run as remote servers inside the container. They give you a special .json file to control how you'd like to start the image, and you can supply a project-specific dockerfile that takes any additional IDE-specific steps that you want to add in as layers when opening up the project's dev container.
It still has some rough edges - for some reason, you can't supply Build-time arguments to the devcontainer Dockerfile I mentioned above, which makes it hard to get permissions to be non-annoying if you're worried your users will have odd UIDs. And there are a couple other random ticky tack things. But it's by far the closest to actually "developing out of a container" as you and I are imagining it, with all of the attendant benefits. When you want a new dev to start working on the project, they basically do this:
1. git clone newproject 2. Open that project in vscode 3. Respond to a pop-up that yes, I do want to open this in a container 4. <wait for the dev-specific layers to be built> 5. Do a normal project build 6. Enjoy full intellisense, syntax highlighting, import resolution, etc. in exactly the environment that's going to run.
In addition, by including the Dockerfile that builds the base image (the one that the dev container FROM's) into the source, developers can now change library versions and dependencies as a git commit, and then coerce other developers into automatically updating their environments by pushing a new image with a new tag to your gitlab / w/e, and then updating the tag mentioned in the FROM in your .devcontainer/Dockerfile. It's like how you would expect development to feel in 2020.
I just figured out a similar workflow using PyCharm Pro. I set up a Deployment mapping from my Windows laptop to my Linux workstation which deploys code whenever I save. Then the Linux folder is volume-mounted into the Docker container. I set up a Remote Debugging configuration, and then after the Docker container is brought up, it will attach via SSH to the Python instance running in the remote Docker container. (Your code needs only have the pydevd_pycharm library installed.) From there you can breakpoint, watch variables, interact at the console, etc. It was a bit of bear to set up, but now I find it very tractable to run and immensely useful.
If you're looking for an IDE with remote interpreter support I've been using JetBrains products for years with Vagrant and Docker and they work great with full support for launching tests in the remote, setting breakpoints etc.
[deleted]
I solve this problem by having everything testable. I'm not saying this is a replacement for running your code as a assembled app before committing... But like, you should know that the code is going to work as intended before you incur the cost of running the application.
For exploratory programming I used to just enable remote debugging and set break points. This should be going away in favor of the REPL introduced in Java 9 tho... Haven't met lots of developers that have integrated it into their gradle builds yet tho...
For exploratory programming I used to just enable remote debugging and set break points. This should be going away in favor of the REPL introduced in Java 9 tho... Haven't met lots of developers that have integrated it into their gradle builds yet tho...
Are you testing your code inside or outside of Docker?
How do you do exploratory programming in languages/environments that don't give you a remote REPL? (For instance, how do you do it in C?)
How do you do exploratory programming in languages/environments that don't give you a remote REPL? (For instance, how do you do it in C?)
The common setups I've worked with are Docker instances for the automated CI tests and then running tests on your dev machine w/o a container.
If I had to do exploratory programming without a REPL I'd probably leverage my test environment to write small bits of code, step through them with a debugger, and then use the results to structure a larger program.
If I had to do exploratory programming without a REPL I'd probably leverage my test environment to write small bits of code, step through them with a debugger, and then use the results to structure a larger program.
Fat jars is the way - works fine with containers; java ee is another story - but should be considered legacy these days.
JVM has remote debugging as well. Works great with Intellij and Eclipse.
I've done some initial investigation a while ago. You should be able to attach gdb to a process in a different cgroup.
To more quickly edit configs just make it so your containers build and start quickly. This is very easy to do if you are aware of the docker layer cache.
Hot reload is basically `while (true) build() && restart()`
To more quickly edit configs just make it so your containers build and start quickly. This is very easy to do if you are aware of the docker layer cache.
Hot reload is basically `while (true) build() && restart()`
Oh, you can definitely attach gdb from outside the container to processes in the container. But I need to load debug symbols, libraries, etc. I'd like gdb to be able to see the source code I'm editing. I've done it and it doesn't quite work (I have stories about gdb's strange attempts at container-awareness... it spawns a thread and tries to stick that thread in the container so it can find files).
Also, now your gdb setup isn't containerized, and that is genuinely part of your "dev environment" (it's a thing you want to share with other developers), so you lose out on many of the advantages of Dockerizing your dev environments. Note that you can't gdb/strace across containers because you can't see pids in one container from another container. (Similarly, a Kubernetes sidecar with gdb/strace isn't really going to work.)
Not all of my software restarts quickly, and it's not because I don't know how to make containers do things - it's because the underlying build system is slow even on the local machine, or because they do a lot of processing at startup, or whatever. (Very few of the things I work on are webapps!)
Also, now your gdb setup isn't containerized, and that is genuinely part of your "dev environment" (it's a thing you want to share with other developers), so you lose out on many of the advantages of Dockerizing your dev environments. Note that you can't gdb/strace across containers because you can't see pids in one container from another container. (Similarly, a Kubernetes sidecar with gdb/strace isn't really going to work.)
Not all of my software restarts quickly, and it's not because I don't know how to make containers do things - it's because the underlying build system is slow even on the local machine, or because they do a lot of processing at startup, or whatever. (Very few of the things I work on are webapps!)
> Note that you can't gdb/strace across containers because you can't see pids in one container from another container. (Similarly, a Kubernetes sidecar with gdb/strace isn't really going to work.)
In Kubernetes, you can configure a Pod to share a single process namespace among its containers.
https://kubernetes.io/docs/tasks/configure-pod-container/sha...
In Kubernetes, you can configure a Pod to share a single process namespace among its containers.
https://kubernetes.io/docs/tasks/configure-pod-container/sha...
Wow, I totally missed this - good to know!
there are solutions to your issues. One is sharing the namespaces. Another is putting gdb in your container (or a debug version) and bind mounting your build/dev directories
> I think it's odd to get hung up on prefixing your commands with "docker-compose exec" or ...
We're already doing "bundle exec rails s" or "bundle exec capistrano production deploy" commands. So a "docker-compose up" shouldn't matter at all.
To be fair, I've had frontenders complain that it was hard for them to remember the capistrano deploy commands. And that they'd love a GUI butten for deployments. Rather than just look at our wiki or write it down.
We're already doing "bundle exec rails s" or "bundle exec capistrano production deploy" commands. So a "docker-compose up" shouldn't matter at all.
To be fair, I've had frontenders complain that it was hard for them to remember the capistrano deploy commands. And that they'd love a GUI butten for deployments. Rather than just look at our wiki or write it down.
It's not like providing a GUI is impossible, a single file python server outputting HTML with buttons would do it. :)
It's not impossible. But we're a small dev team with several active clients. So most of us are swamped with client work.
For the few frontend devs that need to deploy the applications it's way easier for them to just save the command to their notes.
For the few frontend devs that need to deploy the applications it's way easier for them to just save the command to their notes.
Hm. Actual deploys? How about a GitLab CI pipeline with "when: manual" tasks for PROD/LIVE deploy?
But, yes, putting the how to deploy into the README.md can be done in 5 seconds whereas setting up a CI/CD pipeline is a lot more effort. So I guess frontend devs just have to be tamed :)
But, yes, putting the how to deploy into the README.md can be done in 5 seconds whereas setting up a CI/CD pipeline is a lot more effort. So I guess frontend devs just have to be tamed :)
If your goal is detecting earlier configuration issues or limitations induced by the prod environment, it can be enough to have well maintained integration/test environments. And I guess it’s basically the situation the author ended up with.
You can have these running on the same stack as your prod (for instance in a similar kubernetes cluster, or at least on a mirrored config by the same provider if you’re on plain VMs), getting that much closer to test/prod parity.
Most companies can afford one virtual env per dev, so the remote test environment becomes an extension of their local one. Demoing features offline should still be OK as well, you need everything setup locally for dev anyway.
If the goal is to get people used to what’s running in production and train them to access the containers and debug them remotely, it’s another beast altogether, and reminds me of all the discussion about learning vim. It brings tremendous value to be proficient in coding in CLI, and not just for simple edits, but to actually be able to use it as the main coding tool.
But then is it an efficient time investment for most people ? It’s pretty close to living your life in dim light because one day you’ll have to deal with a blackout and you’ll be super efficient then. It’s good to have a few people doing that, I’m not convinced it’s good for a whole team.
You can have these running on the same stack as your prod (for instance in a similar kubernetes cluster, or at least on a mirrored config by the same provider if you’re on plain VMs), getting that much closer to test/prod parity.
Most companies can afford one virtual env per dev, so the remote test environment becomes an extension of their local one. Demoing features offline should still be OK as well, you need everything setup locally for dev anyway.
If the goal is to get people used to what’s running in production and train them to access the containers and debug them remotely, it’s another beast altogether, and reminds me of all the discussion about learning vim. It brings tremendous value to be proficient in coding in CLI, and not just for simple edits, but to actually be able to use it as the main coding tool.
But then is it an efficient time investment for most people ? It’s pretty close to living your life in dim light because one day you’ll have to deal with a blackout and you’ll be super efficient then. It’s good to have a few people doing that, I’m not convinced it’s good for a whole team.
> Does [Docker] make some things more difficult as this author says? At first, yes. As time moves forward you'll end up with many nock on effects that you can't get any other way.
I agree that docker provides many efficiency improvements, especially with respect to testing, reproducing bugs, and deployment.
But what about pure development of new features? In a bare-metal development environment, you just compile and run or launch a debugger. With docker, you need to compile, build a docker image, run the docker image, then launch your app/debugger. Would love to hear solutions on how to eliminate/ameliorate these additional steps.
I agree that docker provides many efficiency improvements, especially with respect to testing, reproducing bugs, and deployment.
But what about pure development of new features? In a bare-metal development environment, you just compile and run or launch a debugger. With docker, you need to compile, build a docker image, run the docker image, then launch your app/debugger. Would love to hear solutions on how to eliminate/ameliorate these additional steps.
You can mount your working source folder in the running container.
If you are working with a scripting language that can be enough to have your changes reflected on the running instance, otherwise you can set a hook on these changes to recompile and/or redeploy the new code automatically.
If you are working with a scripting language that can be enough to have your changes reflected on the running instance, otherwise you can set a hook on these changes to recompile and/or redeploy the new code automatically.
Amen, like you mention there are a couple of hurdles to get around but the reproducibility far outweighs that cost. Before docker trying to get virtually anything running on a variety of machines was a total nightmare
This.
Every company I’ve ever worked for, to setup a dev environment and get the product passing test suites has been days to weeks. And without the automation and standard operating environment, every release was filled with missing dependencies because developers only tested on their local precious system. A side benefit to moving to containers is not contaminating your own machine with libraries or even breaking it because of versioning or config issues.
Docker gets developers and production machines on the same page.
... I would never go back to bare metal development
Every company I’ve ever worked for, to setup a dev environment and get the product passing test suites has been days to weeks. And without the automation and standard operating environment, every release was filled with missing dependencies because developers only tested on their local precious system. A side benefit to moving to containers is not contaminating your own machine with libraries or even breaking it because of versioning or config issues.
Docker gets developers and production machines on the same page.
... I would never go back to bare metal development
> Docker gets developers and production machines on the same page. ... I would never go back to bare metal development
Instead of reading this article as "docker containers are bad" (wrong), try reading it as "there are problems with docker containers that can be improved" (right).
Containers fix a large number of issues, and many of these fixes directly benefit developers. But the article is right to point out inefficiencies that docker introduces in developer workflows. Docker may be taking developers and ops three steps forward, but for some, it's also one step backwards.
Instead of reading this article as "docker containers are bad" (wrong), try reading it as "there are problems with docker containers that can be improved" (right).
Containers fix a large number of issues, and many of these fixes directly benefit developers. But the article is right to point out inefficiencies that docker introduces in developer workflows. Docker may be taking developers and ops three steps forward, but for some, it's also one step backwards.
Sure there may be some issues, but as you say, we're moving two steps forward :)
Another downside of the containers-for-dev approach is that your company's stack might be too big to fit in a 16GB laptop when run as containers in a VM with dedicated memory.
Sure, but do you need production data for development? Most people would just synthesizer what’s needed to run
I mean the code -- having to spin up 30 JVMs for a medium sized org's working stack takes a while and brings a MacBook Pro to its knees. This could probably be improved, if it were deemed sufficiently organizationally important.
Well if your environment is 100 services and each one takes 512mb of ram (that's being generous w/ what I've seen for the typical Spring Java app) at idle...
There's nothing particularly unique about docker that allows you to achieve this - many of us have been doing this for over a decade with a plain ol' VM. Which docker is using anyways if you're on windows or mac.
I've found that given the many docker specific issues I've ran into in development, the "matches your production" argument doesn't always bear completely true unless you're also using containers in production.
I've found that given the many docker specific issues I've ran into in development, the "matches your production" argument doesn't always bear completely true unless you're also using containers in production.
For development, I’ve found containers to be useful for “arms-length” daemons (that is: databases and similar) as the containers need not change often. It makes it easier to ensure that every project uses the correct version of these daemons, as sometimes the OS package manager may not offer the correct version, or different projects necessitate running different versions simultaneously.
As for everyday running of the actual software you’re writing: no way. The tooling for integrating an editor or IDE with the container process is pretty poor (so one often ends up just setting up everything on the host anyway), image rebuilds are frequent and slow, and most language ecosystems already have tools for installing and switching between different versions. They are useful if you want to more closely replicate your CI or production environment for specific debugging, though. Nevertheless, the belief that running a container locally will “never create disparity between your development environment and your production environment” is a farce. There are so many other variables that are often still different. It helps, but it’s far from a silver bullet.
As for everyday running of the actual software you’re writing: no way. The tooling for integrating an editor or IDE with the container process is pretty poor (so one often ends up just setting up everything on the host anyway), image rebuilds are frequent and slow, and most language ecosystems already have tools for installing and switching between different versions. They are useful if you want to more closely replicate your CI or production environment for specific debugging, though. Nevertheless, the belief that running a container locally will “never create disparity between your development environment and your production environment” is a farce. There are so many other variables that are often still different. It helps, but it’s far from a silver bullet.
Disclaimer: I would consider myself one of the developers who tries docker out every couple of years, but just can't seem to get it.
> the developer environment just doesn’t square well with whatever system is running your code. In the olden days of yore, this disparity was so great that we often had to tunnel in to an environment that was exactly like our production server
For web apps, I've gotten stuff working very easily across linux, freebsd, and mac os. Thanks a lot due to package managers, up to date library packages, and any python/ruby/node packages with c/c++ parts generally being able to find the header/.so files. Lockfiles (package-lock.json, poetry.lock) have gotten better as well, so I feel secure knowing that there won't be hiccups there.
Thank you for your comment on docker CLI.
I find docker commands more confusing and obtuse to work with than the commands of postgres, python, node, webpack, combined. All of them. Because it's about having to think about copying files, proxying against the file system, layers of containers. By the time stuff is running I feel so far away from the roots, I'm asking myself how much of a favor is this really doing?
The other thing is deploying these images is a lot of work compared to just running SSH commands via fabric, pulling code, and restarting services. I also think developers benefit from jumping in and wiring in the daemons themselves on whatever system. I think it's part of what fullstack is. It'll come in useful should stuff go wrong in production.
I also found my experimentation with docker inconsistent, even when I cloned projects from git repos with docker compose, I have a high failure rate. To the point I'd be happier if they just gave instructions for me to wire in the daemons and db myself. And when I google, I go down a rabbit hole of configurations and github issues that don't seem to fit (perhaps docker or docker-compose is out of date somehow? Or is it the project itself? Or me?)
> the developer environment just doesn’t square well with whatever system is running your code. In the olden days of yore, this disparity was so great that we often had to tunnel in to an environment that was exactly like our production server
For web apps, I've gotten stuff working very easily across linux, freebsd, and mac os. Thanks a lot due to package managers, up to date library packages, and any python/ruby/node packages with c/c++ parts generally being able to find the header/.so files. Lockfiles (package-lock.json, poetry.lock) have gotten better as well, so I feel secure knowing that there won't be hiccups there.
Thank you for your comment on docker CLI.
I find docker commands more confusing and obtuse to work with than the commands of postgres, python, node, webpack, combined. All of them. Because it's about having to think about copying files, proxying against the file system, layers of containers. By the time stuff is running I feel so far away from the roots, I'm asking myself how much of a favor is this really doing?
The other thing is deploying these images is a lot of work compared to just running SSH commands via fabric, pulling code, and restarting services. I also think developers benefit from jumping in and wiring in the daemons themselves on whatever system. I think it's part of what fullstack is. It'll come in useful should stuff go wrong in production.
I also found my experimentation with docker inconsistent, even when I cloned projects from git repos with docker compose, I have a high failure rate. To the point I'd be happier if they just gave instructions for me to wire in the daemons and db myself. And when I google, I go down a rabbit hole of configurations and github issues that don't seem to fit (perhaps docker or docker-compose is out of date somehow? Or is it the project itself? Or me?)
I didn't really buy wholly into containers for a while either, but the tools have gotten better, the process makes it easier to tick the security boxes, and the confidence brought about by immutable deployments and having good visibility into the service state has been really valuable in a growing environment. No more "oh yeah, we poked at ulimit on that server late Friday night, that's why the other ones don't work as well" and so on. No fiddling around with multiple python or java environments. Keep after the details and you can end up not even having anything to ssh into at all--really get out of the game of "server touching" if you will.
At my company we just finished setting up a brand new docker dev container that we work out of, and it's awesome. However, it was absolute pain to get it to the point where it's awesome, and the only way it worked at all was because of the fantastic new "Remote: Docker" docker extension that has been released with vscode. Our dev environment is intimately tied to vscode and its ability to interpret a special .devcontainer folder to automagically add in special additional dev layers with project-specific vscode extensions, run all relevant language tools as remote servers inside the container, and provide full intellisense over your project.
My questions is, how the hell is anybody getting anything done _without_ all the features I just mentioned? I've found initial, paltry support for particular languages in JetBrains, etc. IDEs, and that's about it. None of the other IDEs have any real solution go getting multi-language intellisense from inside the container. Are people just like... exec'ing into the container and then dicking around in Vim? Or are people mapping all of the source in as a volume, running the IDE outside the container, and just forcing the user to have all the relevant dependencies installed on their host system if they want to see auto-completions, go-to-definition, etc. (thus completely defeating the purpose of containerizing your environment)?
In my eyes, the only reason Docker as a dev environment makes any sense at all is because of the uniquely good support vscode offers, but I see it mentioned all over the docker website that it can help you make your development environment repeatable and exactly like your production environment. But... that doesn't actually work, like, at all...
Can somebody help me understand what I'm missing?
My questions is, how the hell is anybody getting anything done _without_ all the features I just mentioned? I've found initial, paltry support for particular languages in JetBrains, etc. IDEs, and that's about it. None of the other IDEs have any real solution go getting multi-language intellisense from inside the container. Are people just like... exec'ing into the container and then dicking around in Vim? Or are people mapping all of the source in as a volume, running the IDE outside the container, and just forcing the user to have all the relevant dependencies installed on their host system if they want to see auto-completions, go-to-definition, etc. (thus completely defeating the purpose of containerizing your environment)?
In my eyes, the only reason Docker as a dev environment makes any sense at all is because of the uniquely good support vscode offers, but I see it mentioned all over the docker website that it can help you make your development environment repeatable and exactly like your production environment. But... that doesn't actually work, like, at all...
Can somebody help me understand what I'm missing?
Having the environment containerized, you can run the software on your hardware in conditions equivalent to production. That’s it; whether or not your IDE also uses the environment inside the container doesn’t detract from that.
Personally, I don’t care for auto-completion or go-to-definition, but I want to be as sure as possible that if my software doesn’t break in development then it won’t break in staging and production.
Personally, I don’t care for auto-completion or go-to-definition, but I want to be as sure as possible that if my software doesn’t break in development then it won’t break in staging and production.
Interesting.
>Personally, I don’t care for auto-completion or go-to-definition
What about just plain old syntax highlighting to make sure your imports are resolving? Do you just run vim from inside the container or something, and then find out whether it builds / works by running the container?
Like, say you had a Python project that you were going to containerize. Do you perhaps do the actual editing of the source on your dev machine, with all of the relevant libraries installed into a virtualenv or something? If that's the case, then it's basically a guess-and-check workflow, right? There's nothing strongly enforcing your virtualenv to match what's in the container. And there's nothing about having the container that saves you from having to setup your local environment in a way that's suitable for that particular project. That just doesn't seem like "developing out of a container" to me. More like... using containers as a local test environment, or something like that.
Maybe you're just explaining why the Docker folk feel it's reasonable to say that Docker improves the development experience, putting aside whether they imagine you literally sync'ing up your IDE up with a container as I was describing.
>Personally, I don’t care for auto-completion or go-to-definition
What about just plain old syntax highlighting to make sure your imports are resolving? Do you just run vim from inside the container or something, and then find out whether it builds / works by running the container?
Like, say you had a Python project that you were going to containerize. Do you perhaps do the actual editing of the source on your dev machine, with all of the relevant libraries installed into a virtualenv or something? If that's the case, then it's basically a guess-and-check workflow, right? There's nothing strongly enforcing your virtualenv to match what's in the container. And there's nothing about having the container that saves you from having to setup your local environment in a way that's suitable for that particular project. That just doesn't seem like "developing out of a container" to me. More like... using containers as a local test environment, or something like that.
Maybe you're just explaining why the Docker folk feel it's reasonable to say that Docker improves the development experience, putting aside whether they imagine you literally sync'ing up your IDE up with a container as I was describing.
the second thing. we do this heavily.
you map the source directory as a volume and then run the editor outside the container.
you map the source directory as a volume and then run the editor outside the container.
Do you mind if I ask whether you're using some kind of stack where auto-completion, intellisense, etc. is not dependent on any system libraries or dependencies? Or do you just require the developer to install all of those libraries and dependencies on bare metal as well? If it's the latter, isn't that defeating half the purpose of containerizing it?
My prediction is that you might be using a stack where it's not a big deal to isolate project-specific dependencies, npm or virtual-env style. And so in that context, it's not a big deal to ask developers to do a pip install -r requirements.txt or whatever, and then the docker container's job is just to emulate the runtime environment.
It has just always made sense to me that the win of dockerizing your dev environment would be that you could reproduce it on another machine with a docker pull, so those sort of "half-way" workflows were unsatisfying to me. I'm always curious to hear how other people are doing it though - thanks for responding.
My prediction is that you might be using a stack where it's not a big deal to isolate project-specific dependencies, npm or virtual-env style. And so in that context, it's not a big deal to ask developers to do a pip install -r requirements.txt or whatever, and then the docker container's job is just to emulate the runtime environment.
It has just always made sense to me that the win of dockerizing your dev environment would be that you could reproduce it on another machine with a docker pull, so those sort of "half-way" workflows were unsatisfying to me. I'm always curious to hear how other people are doing it though - thanks for responding.
I think the idea others are getting at is using docker as a means of setting up a testing environment in your dev env, not using it as a dev env. I tend to do this (though the way you describe would be great too!). I develop locally with intellij but run my integration tests in a docker container (or a few) by packaging up my app in a container. I just make that container look like whats running in prod. Thats the magic. dev looks like prod.
I'm not using intellisense or anything. the editor is left to the developer whos workstation the environment is running on. The docker approach for this is just to manage the build in a way that is like the CI pipeline.
we have a shell script that adds some syntactic sugar around running docker in a consistent way, but other than that it's left to the dev how they want to manage the editor process.
we have a shell script that adds some syntactic sugar around running docker in a consistent way, but other than that it's left to the dev how they want to manage the editor process.
Containers are great, but everything else that Docker does is very meh. Slow, inconvenient, hard to work with.
What I'd really want is a system that takes nix files (see Nix & NixOS), and gets that running as a container. Or something like that: Guix or whatever else. Free package-level caching and deduplication, and so on.
What I'd really want is a system that takes nix files (see Nix & NixOS), and gets that running as a container. Or something like that: Guix or whatever else. Free package-level caching and deduplication, and so on.
https://nixery.dev/ does some of this. `docker pull nixery.dev/shell/git/htop` and you get an image built on the fly with bash, git and htop in it. Each loaded in their individual layers.
For a push approach, I am maintaining this project: https://github.com/nix-community/docker-nixpkgs
Both work well for CLI-like applications. What is missing is to get something similar for services where there is a mix of configuration coming from environment variables and files.
For a push approach, I am maintaining this project: https://github.com/nix-community/docker-nixpkgs
Both work well for CLI-like applications. What is missing is to get something similar for services where there is a mix of configuration coming from environment variables and files.
Thanks. I will look at this and other comments, but I don't want stuff that involves docker in any way. Only NixOs + containers.
I asked about this a few months ago on here and was pointed to this:
https://grahamc.com/blog/nix-and-layered-docker-images
It's far from a native solution, and is a bit hacky due to the 256 layer limitation in docker, but it's most of the way to what you want, I think.
https://grahamc.com/blog/nix-and-layered-docker-images
It's far from a native solution, and is a bit hacky due to the 256 layer limitation in docker, but it's most of the way to what you want, I think.
Guix has that built in, effectively - you can build a Guix operating-system as a container to run on the same system (this shares the host machine's store for deduplication), or a Docker image if you want to ship it to a machine that's not running Guix.
See docker-image and container here: https://guix.gnu.org/manual/en/html_node/Invoking-guix-syste...
See docker-image and container here: https://guix.gnu.org/manual/en/html_node/Invoking-guix-syste...
Doesn't the NixOS CLI do exactly that with virtual machines?
Yes, the nixos-container command. That's the imperative approach, you can also do it declaratively.
https://nixos.org/nixos/manual/index.html#ch-containers
https://nixos.org/nixos/manual/index.html#ch-containers
Have you taken a look at NixOps[1] and Disnix[2]? They extend the concepts of Nix and NixOS into networks of machines and distributed services (TL;DR version).
If you are using only one system, the NixOS containers support might be enough, though.
[1] https://nixos.org/nixops/ [2] https://nixos.org/disnix/
If you are using only one system, the NixOS containers support might be enough, though.
[1] https://nixos.org/nixops/ [2] https://nixos.org/disnix/
I did used nixops. I quite enjoyed it, though the local state database was annoying. I wish it was more like k8s, keeping its state internally in a distributed db. I'll look into disnix. Thanks!
True, are you aware of the following Github issue related to this?
https://github.com/NixOS/nixops/issues/565
It has links to solutions people are using, and other related issues.
https://github.com/NixOS/nixops/issues/565
It has links to solutions people are using, and other related issues.
The past few days I tried putting a personal project with Docker containers together consisting of nginx, certbot, static frontend files (built from source), backend web server. My goal was to be able to call docker-compose up. I’ve done a similar project before without Docker so I know what the end result should look like. Integrating the Docker containers with each other and getting the configurations right is a nightmare. Today I realized I can’t justify the time spent as “learning experience” anymore. What I did was premature optimization that failed terribly and cost me days.
Docker is a savior when you have several different projects with several different versions of a language or slightly different stacks.
Very much this. If you only have one project and one set of dependencies, I can see how people don’t see the value in Docker.
As soon as you have multiple projects with a mix of dependencies (especially when you’re dealing with incompatible versions or obsolete packages) it really becomes a huge win. Doubly so when you have out of date dependencies that (for whatever bad reason) you can’t update, but also don’t want leaking out into your dev environment.
As soon as you have multiple projects with a mix of dependencies (especially when you’re dealing with incompatible versions or obsolete packages) it really becomes a huge win. Doubly so when you have out of date dependencies that (for whatever bad reason) you can’t update, but also don’t want leaking out into your dev environment.
Now you create an interesting set of issues for the quality and security tests.
How do you test code quality and code security as part of your pipeline for all of those different versions on the fly ?
eg some run python 2.7 and some run python 3.0
How do you test code quality and code security as part of your pipeline for all of those different versions on the fly ?
eg some run python 2.7 and some run python 3.0
Let me break your points down (using python 2.7/3.0 as the example):
a. Code quality - tackled at code review time, and probably not dependent on python 2.7/3.0 differences; if they are version dependent then again it's a code review process. b. Code security - again, code review
Additional:
c. Third party dependency vulnerabilities: detection should be automated; github now supports automatic checks that open pull requests for vulnerable artifacts.
d. user acceptance tests - seems unlikely that two versions of python are used for the same part of the system.
Docker (especially docker-compose) makes it more visible as to what will be deployed. In fact, it may actually be better for security when those security checks of a system can be made before it goes to production, because the whole stack can be built, audited and executed locally.
a. Code quality - tackled at code review time, and probably not dependent on python 2.7/3.0 differences; if they are version dependent then again it's a code review process. b. Code security - again, code review
Additional:
c. Third party dependency vulnerabilities: detection should be automated; github now supports automatic checks that open pull requests for vulnerable artifacts.
d. user acceptance tests - seems unlikely that two versions of python are used for the same part of the system.
Docker (especially docker-compose) makes it more visible as to what will be deployed. In fact, it may actually be better for security when those security checks of a system can be made before it goes to production, because the whole stack can be built, audited and executed locally.
So you dont use automation like sonarcube / qualys and other tools ?
Our corp requires this at a minimum to even be deployed to development. Let alone staging / prod
Our corp requires this at a minimum to even be deployed to development. Let alone staging / prod
Your concerns are real, but they are a separate problem.
Regardless of the downstream testing/integration concerns, I work with projects with multiple environments, ranging from current Rails apps to a Rails 2 application (that lives in a VERY isolated environment as an internal-app for a cheap "customer"), with various things in between. I also contribute to some open source projects that want weird environments.
I've got those projects on my plate regardless of your downstream concerns (which are very valid). However, Docker lets me work on each of them in isolation from one another, and also in a way where none of the goofiness is leaking out into my dev workstation.
Prior to spinning it up in Docker, the Rails 2 app I mentioned could only be worked on in production because setting up a dev environment for it that didn't require removing all the modern stuff was effectively impossible. So while I'm embarrassed to still support that project, at least I have a way to do so and a transition path forward.
Regardless of the downstream testing/integration concerns, I work with projects with multiple environments, ranging from current Rails apps to a Rails 2 application (that lives in a VERY isolated environment as an internal-app for a cheap "customer"), with various things in between. I also contribute to some open source projects that want weird environments.
I've got those projects on my plate regardless of your downstream concerns (which are very valid). However, Docker lets me work on each of them in isolation from one another, and also in a way where none of the goofiness is leaking out into my dev workstation.
Prior to spinning it up in Docker, the Rails 2 app I mentioned could only be worked on in production because setting up a dev environment for it that didn't require removing all the modern stuff was effectively impossible. So while I'm embarrassed to still support that project, at least I have a way to do so and a transition path forward.
I always though of containers as production-level artifacts. In development, I prefer to run most of my environment directly (if possible). Some components can be run as containers if necessary.
Containers provide consistency and isolation. I dont think container, as an abstraction, is needed at the development stage.
Once development is done, then you pass around a resultant container.
For example, I run my Rails projects directly on my Mac (Rails + Native MySQL), my coworker on Linux runs it there natively as well. That means we have to deal with runtime platform issues, true. However when we're ready to "release", we build a container that is then tested, and released to prod.
Containers provide consistency and isolation. I dont think container, as an abstraction, is needed at the development stage.
Once development is done, then you pass around a resultant container.
For example, I run my Rails projects directly on my Mac (Rails + Native MySQL), my coworker on Linux runs it there natively as well. That means we have to deal with runtime platform issues, true. However when we're ready to "release", we build a container that is then tested, and released to prod.
Docker is great for local dev. I can have a complex docker-compose file with all my dependencies and It Just Works.
Nice article, I feel like I learned a lot of these lessons over the last 1.5 years. Here's a Django/Vue.js project that I have been working on that uses 11 containers for local development (postgres, redis, redis-commander, nginx, node, django, channels, celery, beat, flower and mailhog). I think the weirdest things I have to do for the development environment are all related to hot-reloading, which involves tweaked versions of container start scripts. I run an environment similar to this one daily for work and for some other side projects.
https://gitlab.com/verbose-equals-true/django-postgres-vue-g...
The readme for this project addresses some of the issues that you might have with integration testing. I use GitLab CI and I'm able to take advantage of GitLab runner to simulate my CI environment in containers that run on my local machine.
I explored a few different ways of running integration tests in my development environment, one using docker-compose with docker in docker, and another method using services in GitLab CI. I learned a lot from https://gitlab.com/gitlab-org/gitter/webapp.
https://gitlab.com/verbose-equals-true/django-postgres-vue-g...
The readme for this project addresses some of the issues that you might have with integration testing. I use GitLab CI and I'm able to take advantage of GitLab runner to simulate my CI environment in containers that run on my local machine.
I explored a few different ways of running integration tests in my development environment, one using docker-compose with docker in docker, and another method using services in GitLab CI. I learned a lot from https://gitlab.com/gitlab-org/gitter/webapp.
I have only used this development environment Ubuntu, I don't know how well it would work on Mac, but I would be curious to try it on Mac and Windows
We use docker in production and hand a dev setup. Despite months of work, I don't use the dev setup. One still needs to know a lot about the apps and setting them up to get things going. Rebuilding containers takes a long time and is generally a waste of time. Running workers that loop and check ses, for example, eats up massive amounts of cpu while essentially just looping and doing nothing else. It's hard to know the current configuration and keep track of logs. It's a shit experience compared to some apache vhosts. They are useful for some dependencies and our apps that user end of life tech that would conflict with everything else, but otherwise it's not worth the bother. Even on bare metal Linux it's not worth the hassle. In my osx vm, the docker daemon needs restarting every day. The disk needs to be cleaned regularly. So much overhead for no gain. No thanks. I gave up on this years ago. Great for production though.
From my experience, docker dev environments are great if your host OS is Linux (i.e. if you don't need to run a Linux VM inside of another OS). Otherwise, performance can be very bad.
We very successfully run Kubernetes locally, in CI (via kind) and in npe/prod. There are some differences but they have been reduced over time, very strongly recommend this approach to teams for medium+ complexity apps.
We basically had Kube in prod, so worked backwards to make everything match that as closely as possible, solving the required problems along the way. docker compose for example is less likely to be used in prod and thus a poor choice.
We basically had Kube in prod, so worked backwards to make everything match that as closely as possible, solving the required problems along the way. docker compose for example is less likely to be used in prod and thus a poor choice.
I use docker-compose because it makes things easier and portable.
Instead of opening four terminals and running ./manage.py runserver, npm run dev (twice) and hugo serve, and having project instances of Postgres, Redis and Celery running all the time, I just run docker compose up.
I want to move to a new machine? docker-compose up.
docker and docker-compose address blooming system complexity very well. I think that's more a reflection of the complexity than docker.
Instead of opening four terminals and running ./manage.py runserver, npm run dev (twice) and hugo serve, and having project instances of Postgres, Redis and Celery running all the time, I just run docker compose up.
I want to move to a new machine? docker-compose up.
docker and docker-compose address blooming system complexity very well. I think that's more a reflection of the complexity than docker.
People conserved with the configuration concerns notes in the article should check out https://github.com/NixOS/nixpkgs
Developing with Docker is pretty great for my colleagues who use Linux laptops.
Sadly performance on Macs is woeful (due to architectural differences between the two systems) but even so it's still a very useful tool.
Sadly performance on Macs is woeful (due to architectural differences between the two systems) but even so it's still a very useful tool.
[deleted]
Obligatory LXD reference: https://linuxcontainers.org/
1. Developers can now help you debug that one issue in prod that no one can figure out.
2. Deployment is as easy as adding a line to a config (load dev yaml + patch)
3. Good devs will start automating the boring stuff. Good operators will look at, learn to use, and love these tools.
4. Developers will make sure containers and roll out systems work lightening fast.
5. "We need to run this migration in production. Tested it locally" actually means something now.
6. Demos can be done locally with absolutely no network connection. Demoing to the security minded? Worried about showing off your, often down, production env? No problem! Bring a dev laptop.
7. Developers can test QPS quotas, build dashboards for metric collection, benchmark performance, etc
I think it's odd to get hung up on prefixing your commands with "docker-compose exec" or putting "docker-compose up -d --build" under watchman/inotify or investing in setting up minikube + (tilt | skaffold).
Sure it's more stuff for your developers to know but it's very powerful to see your change as it effects the entire living system.