Ask HN: How long does it take to setup your dev environment?
I thought this would get better with docker, but it seems to always be a multi-day process to get up and running. How does your team solve this problem and how long does it take a new dev to be writing code?
38 comments
Pretty proud to say that at Buffer it's a few minutes. We've written a CLI that sets up team members including getting them a new SSH key to use with github. If you are running the install fresh, you simply use ./dev setup. If you are running some services for the day, ./dev up.
This CLI wraps around docker-compose and does some gnarly stuff using Bash.
We now have multiple services and products and at any given time, a team member could be working on any one of them. So to avoid burning people's computers out we even have the ability to spin up only a subset of services specified.
So ./dev up account session-service <product-name> will get you all the required stuff to work on the product you want to work on for example. This involves a surprising amount of work such as re-configuring config files for our reverseproxy and deciding which repos to actually pull updates for.
We found that with dev environments, it's not just setting up. It's also the continued running of things. Architectures change. Dependencies change. Each of these can break people's dev environments and we try to use the tool to contain as much as possible. We are currently adding some functionality in to diagnose and auto resolve errors such as services not starting up due to node dependencies not being installed for example.
This area excites me a lot, so if you want to chat about it, my email is in my user profile :) . Happy to share the work we've done.
This CLI wraps around docker-compose and does some gnarly stuff using Bash.
We now have multiple services and products and at any given time, a team member could be working on any one of them. So to avoid burning people's computers out we even have the ability to spin up only a subset of services specified.
So ./dev up account session-service <product-name> will get you all the required stuff to work on the product you want to work on for example. This involves a surprising amount of work such as re-configuring config files for our reverseproxy and deciding which repos to actually pull updates for.
We found that with dev environments, it's not just setting up. It's also the continued running of things. Architectures change. Dependencies change. Each of these can break people's dev environments and we try to use the tool to contain as much as possible. We are currently adding some functionality in to diagnose and auto resolve errors such as services not starting up due to node dependencies not being installed for example.
This area excites me a lot, so if you want to chat about it, my email is in my user profile :) . Happy to share the work we've done.
About a day.
We have about a dozen external dependencies (databases, caches, front end build systems, etc), custom scripts to set up/populate some of the data stores, and even some third party tools that need to be custom-compiled on every system. We don't use Docker. So it's not as if setup is trivially simple.
But with a clear and specific guide, anyone who isn't terrified of the command line can follow along. Since we added docs, the only person who wasn't working on their first ticket by the end of day 2 was an intern.
When I came on we didn't have docs, and setup time was closer to a week.
Depending on just how low-hanging the fruit is that we can point the new developer at, most have a bug fix in QA by day 4.
Since we grow slowly and turn over slowly, it's really not a huge deal to have the new person spend a day setting up their environment a couple times a year.
It gives them a little insight into how the infrastructure fits together, and an excuse to interact with the team when they need to ask for help (being referenced once or twice a year, the docs are invariably just enough out of date to occasion a few error messages) and discover that we are in fact friendly and helpful.
The best also ask why things are the way they are and gain a little insight into the history of the codebase and the areas that we know are clean or messy or drowning in tech debt.
Are you growing so fast that you have new hires setting up a dozen new development environments from scratch every month? Then yeah, it's probably worth a week of developer time to decrease setup time from 12 hours to 2. If you're growing at a more reasonable pace - even a new hire every month - the ROI is probably not there.
We have about a dozen external dependencies (databases, caches, front end build systems, etc), custom scripts to set up/populate some of the data stores, and even some third party tools that need to be custom-compiled on every system. We don't use Docker. So it's not as if setup is trivially simple.
But with a clear and specific guide, anyone who isn't terrified of the command line can follow along. Since we added docs, the only person who wasn't working on their first ticket by the end of day 2 was an intern.
When I came on we didn't have docs, and setup time was closer to a week.
Depending on just how low-hanging the fruit is that we can point the new developer at, most have a bug fix in QA by day 4.
Since we grow slowly and turn over slowly, it's really not a huge deal to have the new person spend a day setting up their environment a couple times a year.
It gives them a little insight into how the infrastructure fits together, and an excuse to interact with the team when they need to ask for help (being referenced once or twice a year, the docs are invariably just enough out of date to occasion a few error messages) and discover that we are in fact friendly and helpful.
The best also ask why things are the way they are and gain a little insight into the history of the codebase and the areas that we know are clean or messy or drowning in tech debt.
Are you growing so fast that you have new hires setting up a dozen new development environments from scratch every month? Then yeah, it's probably worth a week of developer time to decrease setup time from 12 hours to 2. If you're growing at a more reasonable pace - even a new hire every month - the ROI is probably not there.
Depends. Is it a fresh OS install on a brand new machine, or just getting your specific Dockerized app running on my machine?
If it's just your app, then it really should be nothing more than a docker-compose up --build, and perhaps a few 1 off tasks (such as setting up and / or seeding the DB, which should be callable commands you can run or exec).
The above should work for most web apps (there are exceptions of course), but if you have a fairly standard app I would strive to accomplish the above.
It should take only as long as it takes for the up --build to complete for the most part. All of my Rails, Flask, Phoenix and Node apps tend to be up and running in less than 10-15 minutes with 99% of that time spent on waiting for the images to build.
I also happen to teach Docker[0] and Flask[1] courses and out of the tens of thousands of people who have taken it, most people get the apps and examples up and running very quickly (spinning up a complex Flask app). If there's any struggle it's usually getting Docker set up initially, but that's quickly fixed in nearly every case.
[0]: https://diveintodocker.com/ [1]: https://buildasaasappwithflask.com/
If it's just your app, then it really should be nothing more than a docker-compose up --build, and perhaps a few 1 off tasks (such as setting up and / or seeding the DB, which should be callable commands you can run or exec).
The above should work for most web apps (there are exceptions of course), but if you have a fairly standard app I would strive to accomplish the above.
It should take only as long as it takes for the up --build to complete for the most part. All of my Rails, Flask, Phoenix and Node apps tend to be up and running in less than 10-15 minutes with 99% of that time spent on waiting for the images to build.
I also happen to teach Docker[0] and Flask[1] courses and out of the tens of thousands of people who have taken it, most people get the apps and examples up and running very quickly (spinning up a complex Flask app). If there's any struggle it's usually getting Docker set up initially, but that's quickly fixed in nearly every case.
[0]: https://diveintodocker.com/ [1]: https://buildasaasappwithflask.com/
We've built tools to get new team members spun up quickly on projects. Maybe an hour, if that?
First, a lot of our developers use our wp-local-docker [1] setup, which includes a lot of the tools we use daily for WordPress development, including Elasticsearch.
Then, we use WP Snapshots [2] to pull a "snapshot" of the project from a private S3 bucket. It's up to the lead developer on the project to keep it up to date by pushing a fresh snapshot up to S3.
[1] https://github.com/10up/wp-local-docker
[2] https://github.com/10up/wpsnapshots
First, a lot of our developers use our wp-local-docker [1] setup, which includes a lot of the tools we use daily for WordPress development, including Elasticsearch.
Then, we use WP Snapshots [2] to pull a "snapshot" of the project from a private S3 bucket. It's up to the lead developer on the project to keep it up to date by pushing a fresh snapshot up to S3.
[1] https://github.com/10up/wp-local-docker
[2] https://github.com/10up/wpsnapshots
On my personal machine I keep everything that's not part of a default Ubuntu installation and not a package inside a folder called $HOME/lab. Then I have a shell script that I run to install packages and create sym links for various config files that live inside this "lab" folder. It makes it so I can switch to a new machine fairly painlessly by copying the folder and running the script. I try to keep changes to the "install script" consistent with changes I make to the system. It's simple and works pretty well.
This is a very subjective topic, as each developer has their preferences in tools, editors, etc. Also there needs to be consideration for the local dev environment for quick/dirty prototyping and the "official" dev environment for the product or company you are working with.
My personal/local dev environment consists mainly of brew, docker, sublime text, python, golang, ipython, jq, Vagrant/virtualbox, and my git repo of dot files. On a new mac i can be setup in about 2 hours with the majority of the time being spent in downloading or building software in brew.
My work dev environment consists of a series of tagged docker images in a private repo. These are golden masters of what we use in beta/test/prod. Has a continous build and test pipeline that is powered by a CI/CD tool (aka jenkens, circleci, drone.io, etc). Assuming they have credentials and are granted access, new developers can be up to speed w/in 15 mins.
Lastly, I want to share my internal struggle with reducing and optimizing the multi-day dev environment bootstrapping without discounting the inherent value it brings. Shouldn't a new dev run through the bootstrap process manually to gain intimate knowledge about the tools and systems that they will be working with for the next 1-2 years? Just seems like a worthy investment. Honestly, the ramp up the product architecture and code layout will take longer and this remedial task creates an opportunity to get their hands dirty, update outdated wikis, and potentially improve processes based on their experience. As i can see the merit on both sides, the tie breaker for me is team culture.
My personal/local dev environment consists mainly of brew, docker, sublime text, python, golang, ipython, jq, Vagrant/virtualbox, and my git repo of dot files. On a new mac i can be setup in about 2 hours with the majority of the time being spent in downloading or building software in brew.
My work dev environment consists of a series of tagged docker images in a private repo. These are golden masters of what we use in beta/test/prod. Has a continous build and test pipeline that is powered by a CI/CD tool (aka jenkens, circleci, drone.io, etc). Assuming they have credentials and are granted access, new developers can be up to speed w/in 15 mins.
Lastly, I want to share my internal struggle with reducing and optimizing the multi-day dev environment bootstrapping without discounting the inherent value it brings. Shouldn't a new dev run through the bootstrap process manually to gain intimate knowledge about the tools and systems that they will be working with for the next 1-2 years? Just seems like a worthy investment. Honestly, the ramp up the product architecture and code layout will take longer and this remedial task creates an opportunity to get their hands dirty, update outdated wikis, and potentially improve processes based on their experience. As i can see the merit on both sides, the tie breaker for me is team culture.
Funny you say this, I'm in the middle of configuring a new desktop for my own use. I've written puppet manifests to configure the desktop, install the tools, etc.
I solemnly swear I'll keep it up to date, but then again I said the same thing with the system I'm replacing - which I'd configured with ansible. At the very least all my dotfiles and projects are stored in git, so mostly getting a workable system is trivial - it's things like setting up arduino-studio, firefox (installed from binary releases to /opt, because I want to be current), and similar niggles that are hard.
I can configure hundreds of servers with puppet, but desktops tend to be used for so much adhoc work that is hard to script, that keeping them current and automated is a lot of work.
(e.g. My wife suddenly says "I want XXXX", so I login and set it up, then forget about it for a few years.)
I solemnly swear I'll keep it up to date, but then again I said the same thing with the system I'm replacing - which I'd configured with ansible. At the very least all my dotfiles and projects are stored in git, so mostly getting a workable system is trivial - it's things like setting up arduino-studio, firefox (installed from binary releases to /opt, because I want to be current), and similar niggles that are hard.
I can configure hundreds of servers with puppet, but desktops tend to be used for so much adhoc work that is hard to script, that keeping them current and automated is a lot of work.
(e.g. My wife suddenly says "I want XXXX", so I login and set it up, then forget about it for a few years.)
This will change your life. What you're looking for is called MDM or mobile device management. This is essentially configuration management for laptops or phones, but designed so that these devices can be used in an ad-hoc manner. Apple was the first to really introduce this, but it's available for ChromeOS and Windows 10 (plus Android/iOS). For your personal desktops I would take a look at fleetsmith.com
My personal development hosts, desktops, and laptops all run Debian, but it was interesting to hear about such things regardless!
Usually a couple of minutes:
apt/brew install docker docker-compose
git clone <repo>
docker-compose up
./gradlew build
docker-compose brings up all required dependencies (databases, message queues,...) and all tasks related to building, formatting, codestyle, testing are specified in the gradle build file. Optionally the dev can import the gradle project into the IDE.
apt/brew install docker docker-compose
git clone <repo>
docker-compose up
./gradlew build
docker-compose brings up all required dependencies (databases, message queues,...) and all tasks related to building, formatting, codestyle, testing are specified in the gradle build file. Optionally the dev can import the gradle project into the IDE.
> I thought this would get better with docker, but it seems to always be a multi-day process to get up and running.
I never thought of it till you said this but it has almost always been this way (roughly) -
Every gain in performance has been met with a gain in bloat.
From web page size, to databases, to our dev environments we just keep piling more crap in. Some gives us marginal gains in productivity and performance to accommodate another set of likely marginal features. Someone once described a coworker as a "technological magpie" who would latch on to the newest shiny object, and I think that lots of us have this affliction and are unaware of it or don't know how to fight it.
I never thought of it till you said this but it has almost always been this way (roughly) -
Every gain in performance has been met with a gain in bloat.
From web page size, to databases, to our dev environments we just keep piling more crap in. Some gives us marginal gains in productivity and performance to accommodate another set of likely marginal features. Someone once described a coworker as a "technological magpie" who would latch on to the newest shiny object, and I think that lots of us have this affliction and are unaware of it or don't know how to fight it.
Five seconds.
I type ‘ngp CLIENT PROJECT’
I then have the latest versions of my project template checked out (front end and backend), site served by a local web server, a new database setup, a config file written to connect to the database, and a git repo setup.
I type ‘ngp CLIENT PROJECT’
I then have the latest versions of my project template checked out (front end and backend), site served by a local web server, a new database setup, a config file written to connect to the database, and a git repo setup.
I tried five or six different google queries but have not been able to figure out what `ngp` is. Can you elaborate?
My guess would be a custom made script around "ng new <project>" from angular-cli. Do I win anything?
It takes 1-2 hours to setup a new laptop, email, other services with most of the laptop work automated using a MDM (Jamf in our case, although I would suggest looking at alternative vendors). Day of, it should take the new developer 1-2 hours to get fully set up with their dev environment since we use Docker Compose and keep the onboarding instructions pretty smooth. We used to have devs start working on a ticket after lunch, although now we're doing various training throughout the day.
I used to optimise for this, but different customers have different requirements anyway.
Also dev en to me is my macbook pro + terminal + text editors, but on top of that there's set up for each project.
I used to optimise for this, but it took a week or so 3 years ago and I now I don't care how painful it was.
I'm sure it matters a lot more for open source projects, where you want to encourage lots of ad hoc contributors who can set up easily.
About a day or so from a fresh install of macOS.
The vast majority of projects (both internal and for clients) that I work on use a vagrant dev environment.
Besides that it’s mostly just local tools. I try to use Mac App Store versions of apps where possible because it makes that scenario simpler (open purchases tab, click a bunch of “install” buttons)
For other stuff like hg, git, vagrant, etc which are .pkg based I’m still looking for a good automation solution.
The vast majority of projects (both internal and for clients) that I work on use a vagrant dev environment.
Besides that it’s mostly just local tools. I try to use Mac App Store versions of apps where possible because it makes that scenario simpler (open purchases tab, click a bunch of “install” buttons)
For other stuff like hg, git, vagrant, etc which are .pkg based I’m still looking for a good automation solution.
You should check out brew and brew cask if you're on a mac.
You can install Chrome/Firefox from the command line, atom, etc. Very simple and convenient.
You can install Chrome/Firefox from the command line, atom, etc. Very simple and convenient.
I know about Brew.
Brew's philosophy is "I know you have a powerful multi-user UNIX workstation, with literally decades of best practices about user permissions, installing software etc, but just for lulz we're going to install everything as user-owned, so you dont have to type your password.
Fuck. That.
Brew's philosophy is "I know you have a powerful multi-user UNIX workstation, with literally decades of best practices about user permissions, installing software etc, but just for lulz we're going to install everything as user-owned, so you dont have to type your password.
Fuck. That.
About 3 hours. We have a CLI tool which is essentially a wrapper around docker and containers. It also manages DNS so that containers (backend, frontend, microservices, etc.) can talk to each other.
We also have "medics" who are people who have been around a while and know how to debug and troubleshoot issues that arise. As well as some documentation to assist. The documentation could be better, though.
We also have "medics" who are people who have been around a while and know how to debug and troubleshoot issues that arise. As well as some documentation to assist. The documentation could be better, though.
I haven't had to completely reset my machine in probably five years. Occasionally, I think I should come up with some kind of ansible script that would take me from a bare bones of Ubuntu installed to having all of my software installed and reconfigured.
I do have a script that I wrote that will take me from a default terminal/VPS to ready to code in no time.
I do have a script that I wrote that will take me from a default terminal/VPS to ready to code in no time.
From a fresh macOS, 1-2 hours. Go makes things very easy. Install go, install an editor, install that editor's Go plugin, and let that plugin install whatever other go utils it needs to do code completion, refactoring, etc.
Have you looked into docker-compose?
1. Install Docker
2. `docker-compose up`
3. Run tasks to seed the database
1. Install Docker
2. `docker-compose up`
3. Run tasks to seed the database
We've taken this a step further: `docker-compose.yml` uses a database image that has seed data baked right in. It's great.
If I own the hardware, it takes ~ 20 minutes of human time spread over 2 hours.
With restrictions from an employer, usually 4-8 hours.
With restrictions from an employer, usually 4-8 hours.
I do it instantly with Nix, which keeps my system configuration and my dotfiles in a declarative way.
macOS setup here.
I use https://github.com/mathiasbynens/dotfiles to get my setup done. Comes pretty much with a lot OTB!
I use https://github.com/mathiasbynens/dotfiles to get my setup done. Comes pretty much with a lot OTB!
how do I find a list of everything I have installed so I can then use that to make a script to install it all on a fresh machine?
On a Debian based system
dpkg -l
dpkg -l | awk '/^rc/ {print $2}'I am on OSx and use homebrew /cask heavily.
But I also have a few things installed with Pip and and a few things manually installed and it just seems a bit tricky to traverse all of my install sources and points.
But I also have a few things installed with Pip and and a few things manually installed and it just seems a bit tricky to traverse all of my install sources and points.
[deleted]
from scratch, less than 4 hours, including all the tools needed
usually half a day.
- docker - nodejs - mysql - redis
- docker - nodejs - mysql - redis