Show HN: A Bot to Deploy to AWS, Digital Ocean Etc.(deploybot.com)
deploybot.com
Show HN: A Bot to Deploy to AWS, Digital Ocean Etc.
https://deploybot.com/
40 comments
The biggest issue I have with this model -- and git-flow as an opinion about how deploy works in general -- is that it doesn't take any account of rollbacks, and it doesn't scale well to arbitrary numbers of deployments.
The history of my production deploys is not monotonically forward: if something breaks, it rolls back. Nor does production roll forward as one piece: different components of the stack roll in separate motions (the db schema vs the frontend servers for example).
Git, tied to the project dev history, does not well represent these things. Reverts in the deploy branch are not semantically identical to rollbacks in production: and it's not necessarily safe or wise to merge them back into dev history.
A separate git repo, referencing release numbers, or dev repo commit hashes, would work pretty well on the other hand...
The history of my production deploys is not monotonically forward: if something breaks, it rolls back. Nor does production roll forward as one piece: different components of the stack roll in separate motions (the db schema vs the frontend servers for example).
Git, tied to the project dev history, does not well represent these things. Reverts in the deploy branch are not semantically identical to rollbacks in production: and it's not necessarily safe or wise to merge them back into dev history.
A separate git repo, referencing release numbers, or dev repo commit hashes, would work pretty well on the other hand...
I'd say rollback is possible under a 'git driven' workflow.
That said, sincerely I find rollback one of those inherently complex ideas:
- Rolling back assumes going back to the previous commit will fix everything, an unproven (unprovable?) hypothesis in the face of database migrations, job queues, etc.
- Making database migrations reversible can be nearly impossible (particularly at scale), aside from a significant engineering effort (for something that should absolutely never happen)
So I just don't contemplate the possibility of rolling back a deploy.
Instead I try things (particularly migrations) on staging rigurously:
- staging environments always ephemeral - created from scratch for a given relase
- always load fresh production DB into staging
- check that all my model objects are still `.valid?` (http://api.rubyonrails.org/classes/ActiveRecord/Validations....) after the migration
- leave staging running a few days.
- if you really can (not easy), forward production traffic to staging as well.
If things go wrong (which under my proposed discipline would be a massive screw-up), then the fix would require analysis, a regular fix (no time travelling), and a regular deploy.
Reacting instantly (i.e. without analysis) is kind of delusional thinking. I'd rather stay broken a little longer for avoiding further complications!
That said, sincerely I find rollback one of those inherently complex ideas:
- Rolling back assumes going back to the previous commit will fix everything, an unproven (unprovable?) hypothesis in the face of database migrations, job queues, etc.
- Making database migrations reversible can be nearly impossible (particularly at scale), aside from a significant engineering effort (for something that should absolutely never happen)
So I just don't contemplate the possibility of rolling back a deploy.
Instead I try things (particularly migrations) on staging rigurously:
- staging environments always ephemeral - created from scratch for a given relase
- always load fresh production DB into staging
- check that all my model objects are still `.valid?` (http://api.rubyonrails.org/classes/ActiveRecord/Validations....) after the migration
- leave staging running a few days.
- if you really can (not easy), forward production traffic to staging as well.
If things go wrong (which under my proposed discipline would be a massive screw-up), then the fix would require analysis, a regular fix (no time travelling), and a regular deploy.
Reacting instantly (i.e. without analysis) is kind of delusional thinking. I'd rather stay broken a little longer for avoiding further complications!
I heartily agree with your sentiment "rollback one of those inherently complex ideas" :) It's true that sometimes it's not even well-defined, such as for database migrations.
Some of this also depends on context. If I'm shipping a single primary deployment of a massive fairly monolithic SaaS product, I can do this time-marches-on stuff. If I'm actually shipping shrinkwrapware -- and as a sibling comment says, doing rolling blue-green deploys also looks like this, if briefly -- switching something back to a previous code version is very worth minding.
Some of this also depends on context. If I'm shipping a single primary deployment of a massive fairly monolithic SaaS product, I can do this time-marches-on stuff. If I'm actually shipping shrinkwrapware -- and as a sibling comment says, doing rolling blue-green deploys also looks like this, if briefly -- switching something back to a previous code version is very worth minding.
Code rollbacks are about immediate mitigation, not about pie in the sky snapshot rollback. If you are sane about deployment and don't go to 100% of traffic instantly, then halting a broken deploy and rolling back is certainly better than shifting into analysis mode.
> I'd rather stay broken a little longer for avoiding further complications!
Unfortunately analysis is slow and an unbounded process, and high leverage businesses where every second of downtime has actual measurable loss simply cannot accept this trade-off.
> I'd rather stay broken a little longer for avoiding further complications!
Unfortunately analysis is slow and an unbounded process, and high leverage businesses where every second of downtime has actual measurable loss simply cannot accept this trade-off.
Good point. Few solutions are apt for every scale and every business!
OTOH, if you really essay a given deployment again and again, you can become really confident that the operation will succeed in production.
Real example: the most important feature I've developed this year has been put 5+ times in staging across a couple months. Every time I've asserted all kind of stuff, gathered feedback from the business owner, etc.
The deployment going bad in production is just not a possibility.
At a larger scale than mine, I would probably introduce 'dark launching' as well. That would further reduce the possibility of needing rollback.
OTOH, if you really essay a given deployment again and again, you can become really confident that the operation will succeed in production.
Real example: the most important feature I've developed this year has been put 5+ times in staging across a couple months. Every time I've asserted all kind of stuff, gathered feedback from the business owner, etc.
The deployment going bad in production is just not a possibility.
At a larger scale than mine, I would probably introduce 'dark launching' as well. That would further reduce the possibility of needing rollback.
Could that not just be shuffled into the CI side of things
issues occur -> redeploy last-1 deployment
issues occur -> redeploy last-1 deployment
I use a very similar system to this one, but instead of building from environment-specific branches, we branch from a release branch.
We use a similar git flow system, but we build when a deployable bit of work has been merged into the release branch for this sprint of work (e.g. release-0.1). Once this merge has occurred, a build is triggered, and is then sent to our deployment tool (Octopus Deploy). Our branch naming structure partners with the build number for this specific bit of work to create a build number that matches semantic versioning.
From Octopus, we push these releases from our automated dev deployments up to UAT, staging, and eventually to live. Once we have pushed a release to live, the release branch is merged into master, making that release our new master build. When we move on to the next sprint/release cycle, we reset the build number and merge into a new release branch
We were keen to use this approach as it allows QA and non-technical people like project managers to deploy releases rather than developers. It was a bit weird to move to a workflow where you never actually deploy from master, but it's worked really well for our use case. It's also reassuring to know that the build that works on your dev box is the exact same build being deployed to your other boxes.
With that aside, I definitely agree with the idea behind your post. These are problems that can be solved with existing tools. You can do it using plain-old pull requests, or using a dedicated deployment tool. These tools already have plugins that you can throw into your favoured chat platform, or through email. It seems like there are an ever-increasing number of build/deploy solutions out there that seem to do basically the same thing, but slightly differently.
We use a similar git flow system, but we build when a deployable bit of work has been merged into the release branch for this sprint of work (e.g. release-0.1). Once this merge has occurred, a build is triggered, and is then sent to our deployment tool (Octopus Deploy). Our branch naming structure partners with the build number for this specific bit of work to create a build number that matches semantic versioning.
From Octopus, we push these releases from our automated dev deployments up to UAT, staging, and eventually to live. Once we have pushed a release to live, the release branch is merged into master, making that release our new master build. When we move on to the next sprint/release cycle, we reset the build number and merge into a new release branch
We were keen to use this approach as it allows QA and non-technical people like project managers to deploy releases rather than developers. It was a bit weird to move to a workflow where you never actually deploy from master, but it's worked really well for our use case. It's also reassuring to know that the build that works on your dev box is the exact same build being deployed to your other boxes.
With that aside, I definitely agree with the idea behind your post. These are problems that can be solved with existing tools. You can do it using plain-old pull requests, or using a dedicated deployment tool. These tools already have plugins that you can throw into your favoured chat platform, or through email. It seems like there are an ever-increasing number of build/deploy solutions out there that seem to do basically the same thing, but slightly differently.
Please be aware of the gotcha of protected branch: https://help.github.com/articles/about-branch-restrictions/
Many teams - why, I don't know - dislike automated deployments, but do like being able to trigger a deployment without needing to setup local deploy environments. In this case, having a bot is useful.
In a way, my setup doesn't automate deployments - i.e. you never inadvertently deploy by committing some code or merging a PR (when your intent was merely approving some code).
That's why I made the master vs. master-xxx distinction: deployments are an explicit action, which you are free to defer as needed.
That's why I made the master vs. master-xxx distinction: deployments are an explicit action, which you are free to defer as needed.
So why is a bot better than hitting the build/deploy button on your CI/CD server?
An issue I have with these git driven deployment process (not your specifically which looks well thought-out) is that your are starting from source each time.
I like my deployment across environments to use the exact same binaries that have been validated. Is it just me that is afraid of introducing issues by not doing that ?
I like my deployment across environments to use the exact same binaries that have been validated. Is it just me that is afraid of introducing issues by not doing that ?
Long shot, but you might be interested in http://repeatr.io/ -- it's a tool for our brave-new-container-world intended to make it easy (or at least easier) to suss out reproducible builds.
Or to answer your direct question: "no", it's definitely not just you who's afraid of that :)
Or to answer your direct question: "no", it's definitely not just you who's afraid of that :)
Thanks for sharing!
Seems like a pretty neat service. To save others some time, they don't have a free tier, you can't host it yourself, and they use Docker for builds before deployments:
http://support.deploybot.com/article/1028-plans-and-pricing
http://support.deploybot.com/article/1028-plans-and-pricing
Suggestion to all companies: please have a /pricing/ page.
Fair enough. They do list their prices but on their signup page https://signup.deploybot.com/signup/new
DeployBot certainly does have a free use plan, afaik
If it does, it's not obvious.
Granted, I might have been grandfathered on the Free Plan:
http://imgur.com/a/GZU14
Free Plan includes 1 repository, unlimited servers, unlimited deployments, and unlimited users.
http://imgur.com/a/GZU14
Free Plan includes 1 repository, unlimited servers, unlimited deployments, and unlimited users.
So Beanstalk does have a free tier still. And it sounds like Beanstalk includes some or all DeployBot functionality?
Does one need to have Docker configured to use this? I haven't gotten around to learning Docker yet.
Docker's fairly easy to get your head around in a few hours if you do a bit of reading. It's being used as a base tool for more and more projects now, so I'd definitely recommend having an understanding of how it works at a basic level, at least.
We[1] use DeployBot every day and we can't endorse them enough!
The combination of DeployBot, Github and AWS Elastic Beanstalk is awesome and is the closest thing to having Heroku in Australia.
We used to just use Elastic Beanstalk, but when AWS moved their deploy method away from git to using zips of S3 bundles, it meant that you needed to reupload the entire app whenever you made a change (not just the delta). This can take a long time on ADSL. DeployBot saved the day here, and allowed us to pull the code from Github.
[1] http://littlebirdelectronics.com
The combination of DeployBot, Github and AWS Elastic Beanstalk is awesome and is the closest thing to having Heroku in Australia.
We used to just use Elastic Beanstalk, but when AWS moved their deploy method away from git to using zips of S3 bundles, it meant that you needed to reupload the entire app whenever you made a change (not just the delta). This can take a long time on ADSL. DeployBot saved the day here, and allowed us to pull the code from Github.
[1] http://littlebirdelectronics.com
The problem with generic deployment services like this is that they don't account for scalability.
Different stacks have different requirements. I built my own deployment service for my open source project/stack specifically so that it would handle scalability too. See https://baasil.io/ I know Laravel followed this approach too with https://forge.laravel.com
I think using more sepcialized deployment stacks (as opposed to generic ones) is the best approach for non-trivial apps.
Though I guess if you use a microservices approach, you could have a different deployer for different kinds of services.
Different stacks have different requirements. I built my own deployment service for my open source project/stack specifically so that it would handle scalability too. See https://baasil.io/ I know Laravel followed this approach too with https://forge.laravel.com
I think using more sepcialized deployment stacks (as opposed to generic ones) is the best approach for non-trivial apps.
Though I guess if you use a microservices approach, you could have a different deployer for different kinds of services.
Migrated a complex Jenkins setup to Deploybot in 2015, saves our company a ton of time managing deploys. I'd highly recommend deploybot to anyone.
If I could critique even just one thing it would probably be its pricing structure for personal use, I can't justify $15/m just for deployments. I'd love if they had some kind of personal "developer" tier with support for more repos. On the business side, $15/m is ridiculously cheap for what service we're getting.
If I could critique even just one thing it would probably be its pricing structure for personal use, I can't justify $15/m just for deployments. I'd love if they had some kind of personal "developer" tier with support for more repos. On the business side, $15/m is ridiculously cheap for what service we're getting.
How does this compare to something like Laravel Forge[0]? Is it just that Forge focuses on PHP?
[0] https://forge.laravel.com
[0] https://forge.laravel.com
Kudos to WildBit. They're undeniably great in all the ways.
Their team photos look so happy that they seem about to explode in a sparkling cloud of rainbows and unicorns.
http://wildbit.com/
http://wildbit.com/
Sent them cupcakes once. Nat's pretty awesome at customer stuff.
Any suggestions on reading material/designs for deploying a complete CI process for a new development team using these newer processes/applications?
Definitely take a look onto Ansible[1]. We are using it on daily basis and to deploy and provision servers.
Deployment from CI was just 4 lines into CI config (fetch repo, set-up keys, deploy in the very same way as we do)
edit: we are using quite widely. Wanna create new servers in AWS - just run a playbook. Wanna setup new database cluster - just run a playbook. So easy
[1]: https://www.ansible.com/
Deployment from CI was just 4 lines into CI config (fetch repo, set-up keys, deploy in the very same way as we do)
edit: we are using quite widely. Wanna create new servers in AWS - just run a playbook. Wanna setup new database cluster - just run a playbook. So easy
[1]: https://www.ansible.com/
Thanks, will do! We are starting a new dev team and I would like to get these practices going immediately instead of "oh, I lost the code cause it was on my laptop" (which happens right now)
If you haven't read it already, read Continuous Delivery.
https://www.amazon.com/dp/0321601912
https://www.amazon.com/dp/0321601912
Heh, my company uses this. I didn't realise it was so new that it deserved a HN post to it's front page.
I have used deploybot for over a year, great engineering. My favourite deploy tool
Happy user of DeployBot here. Does exactly what it's advertised it does.
Any suggestions on similar open-source tools?
What many don't seem aware of is that plain pull requests, in combination with CI, entirely kill the need for a deploy app/bot.
This is how I do it at my current company:
This setup has the same benefits (and then some more) than competitors:
Hope it helps someone!