Ask HN: How to deploy into production large software project?
I work as software developer for a very large corporation...
And yet, in my department we have absolutely outdated process of deploying large software projects. We literary have 2 people, they take a massive spreadsheet and drag and drop (from repository) software onto a large number of servers, after that they manually change .config files (info comes from another massive spreadsheet) and manually start/stop services in correct sequence. After that someone else comes in and verifies that software is up & running correctly, by checking logs. All software are Windows services, most don't have .MSI installation packages.
Problem is, last Friday deployment took 21h and we have several per year! I was at work from 7 in the morning till 2 in the morning. I was completely drained after being on my feet for 21 and following the morning we had issues, because of misconfigured applications (connection strings...)
HN help me sleep better. How do you deploy large software projects ?
Thank you !
6 comments
Set a conservative schedule for improvement (months, not weeks).
Automate the low-hanging fruit first (copying files, maybe, or centralized health-checking so you don't have to open logs).
For a lot of improvements you can make a business case other than your quality of life. Health-checking improves reliability, for example. But accept that these will be hard conversations that take a long time. Don't let others convince you this isn't a problem.
Research the tools that solve your problems -- it's easy to argue for adopting an industry-standard tool that solves an industry-standard problem.
Accept that certain pieces of the process will stay manual for a while.
Don't be afraid to go after small improvements -- if you can halve the difficulty, the remaining steps will fall away easily.
Automate the low-hanging fruit first (copying files, maybe, or centralized health-checking so you don't have to open logs).
For a lot of improvements you can make a business case other than your quality of life. Health-checking improves reliability, for example. But accept that these will be hard conversations that take a long time. Don't let others convince you this isn't a problem.
Research the tools that solve your problems -- it's easy to argue for adopting an industry-standard tool that solves an industry-standard problem.
Accept that certain pieces of the process will stay manual for a while.
Don't be afraid to go after small improvements -- if you can halve the difficulty, the remaining steps will fall away easily.
https://octopus.com/ is a decent option. But, I wrote a windows service that knows how to deploy different types of applications windows service, iis app, scheduled tasks etc.
The service listens for manifest change message on a topic, when a team wants to release a new version of an app, they update the url of the jenkins artifacts in a manifest file in git and publish manifest changed message.
Deployment service on all the servers downloads the latest version of the manifest file and updates or installs apps as needed.
I also wrote a custom prometheus exporter to export health and version of services installed on a box. That allows us to track the deployment status.
This setup has worked super reliability for us, very little maintenance and training required. Since deployment occur in parallel, deployment on hundreds of servers takes only a few seconds.
The service listens for manifest change message on a topic, when a team wants to release a new version of an app, they update the url of the jenkins artifacts in a manifest file in git and publish manifest changed message.
Deployment service on all the servers downloads the latest version of the manifest file and updates or installs apps as needed.
I also wrote a custom prometheus exporter to export health and version of services installed on a box. That allows us to track the deployment status.
This setup has worked super reliability for us, very little maintenance and training required. Since deployment occur in parallel, deployment on hundreds of servers takes only a few seconds.
Automate how you set up your development environment, then automate the process of running this automation on your production servers.
You can get incredibly fancy and automate everything and deploy tens of times a day etc. but you don't have to. You just have to pick practices that would improve your current situation, and in most cases more automation helps.
In my current job, despite us not having everything down perfectly, we've got to a point where I can tag my branch and run a script that deploys the tagged code as a separate container locally if I'm doing development.
Then once we want to deploy to production, another set of configuration files (also versioned, with the smallest diff possible from the dev config) is fed to the same script, and the service is deployed on the production server.
Then you automate running this process across your servers (are your servers your own? Are they on the cloud? Use the tools relevant to your situation).
Then, there is no need to perform manual monitoring of the logs. Set up a monitoring service to notify you of any worrying logs (warn level and up? Specific patterns? Up to you and depending on what you're logging) and feed that service your logs. Hell, automate pinging these services so you can check whether they're in the state you expect, and make a colour coded minimal dashboard that anyone can check in a sec and find out if the everything is in the state it should be.
If the business is paranoid, then do smoke testing at the end, but you shouldn't need to.
You can get incredibly fancy and automate everything and deploy tens of times a day etc. but you don't have to. You just have to pick practices that would improve your current situation, and in most cases more automation helps.
In my current job, despite us not having everything down perfectly, we've got to a point where I can tag my branch and run a script that deploys the tagged code as a separate container locally if I'm doing development.
Then once we want to deploy to production, another set of configuration files (also versioned, with the smallest diff possible from the dev config) is fed to the same script, and the service is deployed on the production server.
Then you automate running this process across your servers (are your servers your own? Are they on the cloud? Use the tools relevant to your situation).
Then, there is no need to perform manual monitoring of the logs. Set up a monitoring service to notify you of any worrying logs (warn level and up? Specific patterns? Up to you and depending on what you're logging) and feed that service your logs. Hell, automate pinging these services so you can check whether they're in the state you expect, and make a colour coded minimal dashboard that anyone can check in a sec and find out if the everything is in the state it should be.
If the business is paranoid, then do smoke testing at the end, but you shouldn't need to.
Hey there, I can help you with our product. We provide deployments of software as service.
We also provide customised solution for large enterprises.
Take a look at http://grandappstudio.com/roll-out
Let us know if you have any questions at [email protected]
Write a daemon yourself to control the deployment on the machines. A kind of trojan but make it nice and safe. Do it on your spare time at the office and wait for a good opportunity to show it.
You probably have many levels of hierarchy working against you. If you have the opportunity to lead or manage anything, take it.
You probably have many levels of hierarchy working against you. If you have the opportunity to lead or manage anything, take it.
I'm not familiar with your setup, so I'm not sure how much of this is applicable, but I've seen a few different approaches, the all more or less go like this:
* choose a configuration state manager like Salt, Ansible, Chef, Puppet, etc. Of all these I'd probably say Salt is the simplest for somebody approaching this methodology.
* choose a source control management system -- I'd just go with git, and host something on github or gitlab or bitbucket or whatever.
* choose a secure storage medium for the variables that are going to be loaded into your configuration state manager.
* choose some form of automation management tool, Jenkins, Thoughtworks Go, TravisCI, CircleCI, w/e.
You're going to use these in conjunction to gain the following: when a developer pushes code to a specific branch, this push triggers a build on the automation server, which is able to package the code, and distribute it to the nodes managed by your configuration state manager. Aka, a push to your repo makes it all happen (with testing, etc as gatekeepers to deployment of course.)
I'll describe a basic setup I've encountered in the wild using Salt in a master-agent setup (since it's the most basic imo, aside from headless):
-> developer writes some code
I use Jenkins Groovy pipelines (which I'm not that crazy about) to ensure that the Jenkins jobs themselves are captured as code and are repeated the same way each time across my make build steps. I work in AWS, so the machines I use, the bootstraps they boot with, the desired state config files, and everything are defined as code in some regard.
You should attempt to push for 'desired state' and move towards 'immutable state'.
I would also strongly consider looking at the other options aside from salt, it's only so-so.
* choose a configuration state manager like Salt, Ansible, Chef, Puppet, etc. Of all these I'd probably say Salt is the simplest for somebody approaching this methodology.
* choose a source control management system -- I'd just go with git, and host something on github or gitlab or bitbucket or whatever.
* choose a secure storage medium for the variables that are going to be loaded into your configuration state manager.
* choose some form of automation management tool, Jenkins, Thoughtworks Go, TravisCI, CircleCI, w/e.
You're going to use these in conjunction to gain the following: when a developer pushes code to a specific branch, this push triggers a build on the automation server, which is able to package the code, and distribute it to the nodes managed by your configuration state manager. Aka, a push to your repo makes it all happen (with testing, etc as gatekeepers to deployment of course.)
I'll describe a basic setup I've encountered in the wild using Salt in a master-agent setup (since it's the most basic imo, aside from headless):
-> developer writes some code
-> developer pushes that code to github
-> a POST webhook containing branch info is sent to my Jenkins box
-> Jenkins initiates a build job that pulls down that branch of code
-> Jenkins uses make (we use make) to build the code branch.
-> Jenkins deposits the built code artifacts on a fileshare somewhere.
-> Jenkins calls to the salt-master to update all nodes.
-> the salt master (which could just be hosted on the Jenkins i guess) tells all minions to update
-> all minions pull the new code and HUP themselves.
I use Jenkins Groovy pipelines (which I'm not that crazy about) to ensure that the Jenkins jobs themselves are captured as code and are repeated the same way each time across my make build steps. I work in AWS, so the machines I use, the bootstraps they boot with, the desired state config files, and everything are defined as code in some regard.
You should attempt to push for 'desired state' and move towards 'immutable state'.
I would also strongly consider looking at the other options aside from salt, it's only so-so.