Ask HN: Describe deploy process at your work/project
7 comments
For context, we have a different AWS account for each environment (production, staging, etc.) and have as much parity as possible between them (although resource sizes might vary - for example different RDS instance sizes for staging vs. production). All of our services run on AWS ECS and CI is done through CircleCI.
Our deployments start with a Github PR needing to have 2 approvals from peers and have all CI checks pass. Once the PR has been approved it can be merged into master which will trigger CI workflows again with the additional step of deploying the changes to staging and building a new release for production. The author of the changes can then verify they work as expected and promote the release in production or rollback their changes if things aren't working.
All of our AWS resources are created and tracked through Terraform but services are created through Convox[1]. We use Convox to create "releases" (ECS task definitions), manage their lifecycle (creation, promotion, releases, rollbacks, etc.) and it's CLI is pretty convenient (promoting a new release is as simple as running `convox releases promote <ID>`). That being said we've had some shortcomings with it and have outgrown it's flexibility. We'll be starting our migration into k8s early next year but it's worked great for us for the past 2 years.
[1] https://convox.com
Our deployments start with a Github PR needing to have 2 approvals from peers and have all CI checks pass. Once the PR has been approved it can be merged into master which will trigger CI workflows again with the additional step of deploying the changes to staging and building a new release for production. The author of the changes can then verify they work as expected and promote the release in production or rollback their changes if things aren't working.
All of our AWS resources are created and tracked through Terraform but services are created through Convox[1]. We use Convox to create "releases" (ECS task definitions), manage their lifecycle (creation, promotion, releases, rollbacks, etc.) and it's CLI is pretty convenient (promoting a new release is as simple as running `convox releases promote <ID>`). That being said we've had some shortcomings with it and have outgrown it's flexibility. We'll be starting our migration into k8s early next year but it's worked great for us for the past 2 years.
[1] https://convox.com
I work on two machines - a 'dirty' one and a secure one.
The 'dirty' machine is used for day-to-day tasks. I write code, download libraries, research things on the internet, but the machine doesn't have access to any of my production systems. In other words, if the dirty machine were to be compromised, the attacker couldn't take over my website with it.
All code is pushed to a private fork of my website, which is the 'staging' branch. A separate GitLab account and SSH identity is used so that there is a clear separation of privileges.
Then, when I'm ready to deploy to production, I turn around and log on to my secure machine. I create a merge request to merge the dirty 'staging' branch with the clean production branch. Before accepting the merge request, I thoroughly review the diff of my own code in order to make sure that an attacker didn't sneak something in without me noticing.
If I need to make minor corrections before merging (e.g. if I notice a spelling mistake at the last minute), I can commit this straight to master from the secure machine and then pull the change back down to staging afterwards.
Doing it this way does take slightly longer, but if you have both of the machines physically next to each other (but with separate peripherals and isolated network connections of course), then the security benefit massively outweighs the extra time requirement.
The 'dirty' machine is used for day-to-day tasks. I write code, download libraries, research things on the internet, but the machine doesn't have access to any of my production systems. In other words, if the dirty machine were to be compromised, the attacker couldn't take over my website with it.
All code is pushed to a private fork of my website, which is the 'staging' branch. A separate GitLab account and SSH identity is used so that there is a clear separation of privileges.
Then, when I'm ready to deploy to production, I turn around and log on to my secure machine. I create a merge request to merge the dirty 'staging' branch with the clean production branch. Before accepting the merge request, I thoroughly review the diff of my own code in order to make sure that an attacker didn't sneak something in without me noticing.
If I need to make minor corrections before merging (e.g. if I notice a spelling mistake at the last minute), I can commit this straight to master from the secure machine and then pull the change back down to staging afterwards.
Doing it this way does take slightly longer, but if you have both of the machines physically next to each other (but with separate peripherals and isolated network connections of course), then the security benefit massively outweighs the extra time requirement.
For work, we use Concourse-CI self-hosted on EC2 using this script[0]. We have blue-green deploys to Pivotal Web Services (PWS) which is hosted CloudFoundary in AWS. We've written our own blue-green deployment scripts for Spring Boot, Node.js/React/Vue, and Python. We also have a pipeline for deploying directly to EC2 Autoscaling groups.
We have a Go service that is deployed directly to EC2 with a Makefile from its source repo. We deploy it so rarely that we haven't made a pipeline.
The deployment script is used to deploy first to a 'staging' environment and if all is well, a manual trigger to continue deploying the service to production. These deployments run from the 'master' branch from Github repos. We also have 'dev' branches that deploy to a separate dev environment for ad-hoc use for dev/integration.
Databases are PostgreSQL in AWS RDS with Liquibase used for migrations.
When performing a blue-green deployment to each environment, the test suite is run against the 'new' blue/green deployment. After passing this suite it becomes the current version in the environment and an additional 'Journey Test' sequence is run which verifies that the top 10 actions work as designed. We also run a 'Health Check' of the current production environment using the journey tests every 10 minutes.
[0] https://github.com/karmakaze/concourse-ec2
For personal projects, I've used Netlify for static sites, or webhooks/polling of Github/Gitlab repos to update prod deployments as well as a Makefile 'deploy-prod' target from the project repo.
We have a Go service that is deployed directly to EC2 with a Makefile from its source repo. We deploy it so rarely that we haven't made a pipeline.
The deployment script is used to deploy first to a 'staging' environment and if all is well, a manual trigger to continue deploying the service to production. These deployments run from the 'master' branch from Github repos. We also have 'dev' branches that deploy to a separate dev environment for ad-hoc use for dev/integration.
Databases are PostgreSQL in AWS RDS with Liquibase used for migrations.
When performing a blue-green deployment to each environment, the test suite is run against the 'new' blue/green deployment. After passing this suite it becomes the current version in the environment and an additional 'Journey Test' sequence is run which verifies that the top 10 actions work as designed. We also run a 'Health Check' of the current production environment using the journey tests every 10 minutes.
[0] https://github.com/karmakaze/concourse-ec2
For personal projects, I've used Netlify for static sites, or webhooks/polling of Github/Gitlab repos to update prod deployments as well as a Makefile 'deploy-prod' target from the project repo.
ftp some php scripts, written in notepad++
Thanks :)