Ask HN: Looking for Good Articles on CICD/Release-Engineering
9 comments
My number one tip for learning more about CI is to read other people's CI code.
Thanks to GitHub Actions there are thousands of great examples of CI configuration in YAML files on GitHub.
I use GitHub code search all the time for these: I'll search for a tool I want to use in CI and filter for YAML files in the .github/workflows directory to see what other people are doing.
I also have a couple of test repos of my own which exist just so I can try out new CI patterns in GitHub Actions before I try using them on a real project. Here's one of those: https://github.com/simonw/playing-with-actions
Thanks to GitHub Actions there are thousands of great examples of CI configuration in YAML files on GitHub.
I use GitHub code search all the time for these: I'll search for a tool I want to use in CI and filter for YAML files in the .github/workflows directory to see what other people are doing.
I also have a couple of test repos of my own which exist just so I can try out new CI patterns in GitHub Actions before I try using them on a real project. Here's one of those: https://github.com/simonw/playing-with-actions
On a high level I agree with Mr Conway. “Orgs which design systems are constrained to produce designs which are copies of the communication structures of these organizations.”
You will rarely see two release processes being exactly the same in two companies or amongst teams in the same company.
The release process is completely different if you deploy microservices vs frontend, mobile or SPAs. Languages differ. Artifacts too.
Some tribes gave dedicated QA teams, others enforce developers to pass a testing threshold level during CI. Some do nothing, just test in production.
Some promote the same artifact, others rebuild in every stage running slightly different end to end tests.
We have a pretty customized rollback mechanism these days to enable quick rollbacks because we use gitOps, which doesn’t support rollbacks (the rollback happens when you revert a commit). But there there is a process, a lock tag and couple of many small things that same to be from experience.
Monorepos vs multi-repos. So many decisions depend on that…
Without knowing the specifics is hard to help. What I can tell you is that “metrics are king”. Make sure you measure time and success rate for each part of the build, test and deployment process. Developers can be like spoiled kids at times but they are your “clients”, make sure really, really listen.
Good luck!
You will rarely see two release processes being exactly the same in two companies or amongst teams in the same company.
The release process is completely different if you deploy microservices vs frontend, mobile or SPAs. Languages differ. Artifacts too.
Some tribes gave dedicated QA teams, others enforce developers to pass a testing threshold level during CI. Some do nothing, just test in production.
Some promote the same artifact, others rebuild in every stage running slightly different end to end tests.
We have a pretty customized rollback mechanism these days to enable quick rollbacks because we use gitOps, which doesn’t support rollbacks (the rollback happens when you revert a commit). But there there is a process, a lock tag and couple of many small things that same to be from experience.
Monorepos vs multi-repos. So many decisions depend on that…
Without knowing the specifics is hard to help. What I can tell you is that “metrics are king”. Make sure you measure time and success rate for each part of the build, test and deployment process. Developers can be like spoiled kids at times but they are your “clients”, make sure really, really listen.
Good luck!
It's an interesting area. I worked in that part of a company for a few years, and work with that area as a consumer.
Part of it is using Jenkins/GitLab/GitHub effectively, etc... but... there are also some real shortcomings to setting up elaborate, complicated systems with those tools.
As a first step, devs should have a sane, efficient incremental build setup for their project using CMake/maven/gradle/whatever that works well on their laptop (i.e., forget about all the centralized web-based apps). If they can't do that, no amount of Jenkins/GitLab/GitHub/ADO can help. If their build is slow on its own... then fix it! Running a bad/slow build on a centralized platform just makes things slow for everyone, and then there's no incentive for anyone to fix their own thing because everyone will stil be the victim of everyone else.
Only once that is sorted out, should things be hosted on Jenkins/GitLab/GitHub/ADO... and even then, using a bare minimum, simple, boilerplate setup, not some elaborate configuration that means that the software can not be built without Jenkins/GitLab/GitHub/ADO any more, and nothing can ever be migrated.
Part of it is using Jenkins/GitLab/GitHub effectively, etc... but... there are also some real shortcomings to setting up elaborate, complicated systems with those tools.
As a first step, devs should have a sane, efficient incremental build setup for their project using CMake/maven/gradle/whatever that works well on their laptop (i.e., forget about all the centralized web-based apps). If they can't do that, no amount of Jenkins/GitLab/GitHub/ADO can help. If their build is slow on its own... then fix it! Running a bad/slow build on a centralized platform just makes things slow for everyone, and then there's no incentive for anyone to fix their own thing because everyone will stil be the victim of everyone else.
Only once that is sorted out, should things be hosted on Jenkins/GitLab/GitHub/ADO... and even then, using a bare minimum, simple, boilerplate setup, not some elaborate configuration that means that the software can not be built without Jenkins/GitLab/GitHub/ADO any more, and nothing can ever be migrated.
Apart from the resources in the other comments, I also benefited greatly by understanding the theory and principles behind CI and CD.
The Continuous Delivery (2010) book is a good place to start, along with Accelerate (2018).
The Phoenix Project (2013) is a fun read, inspired in many ways by The Goal (1984) which although fantastic, is quite a dry read.
https://trunkbaseddevelopment.com/ is a great resource and entirely free to read online.
Might not be relevant to you, but I work with mobile apps so “Continuous Deployment of Mobile Software at Facebook” (2016) was quite insightful: https://research.facebook.com/publications/continuous-deploy...
The Continuous Delivery (2010) book is a good place to start, along with Accelerate (2018).
The Phoenix Project (2013) is a fun read, inspired in many ways by The Goal (1984) which although fantastic, is quite a dry read.
https://trunkbaseddevelopment.com/ is a great resource and entirely free to read online.
Might not be relevant to you, but I work with mobile apps so “Continuous Deployment of Mobile Software at Facebook” (2016) was quite insightful: https://research.facebook.com/publications/continuous-deploy...
If videos are also appreciated I would really recommend binging the official CD Foundation channe on Youtube.
The CD Foundation is, as far as I understand, a fork of the CNCF (Cloud Native Computing Foundation) and is primarily focused on the (open) CI/CD ecosystem containing services such as Jenkins, Spinnaker, Screwdriver, Tekton and Argo.
Like the CNCF they also host conferences, workshops and talks which are recorded and can be viewed through YT for free.
https://www.youtube.com/channel/UC7HcWhSetq6nTlpMXPHKz_A
The CD Foundation is, as far as I understand, a fork of the CNCF (Cloud Native Computing Foundation) and is primarily focused on the (open) CI/CD ecosystem containing services such as Jenkins, Spinnaker, Screwdriver, Tekton and Argo.
Like the CNCF they also host conferences, workshops and talks which are recorded and can be viewed through YT for free.
https://www.youtube.com/channel/UC7HcWhSetq6nTlpMXPHKz_A
I would recommended a book for understanding the bigger picture, and why are we doing CI/CD:
"Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation" By David Farley and Jez Humble
Dave’s channel on YouTube (just called Continuous Delivery I think) is also really good.
Take a look at this article: https://gitprotect.io/blog/exploring-best-practices-and-mode...
Also, you can find more articles on CI/CD on this blog
Also, you can find more articles on CI/CD on this blog
Am an SRE who has been tasked to make my Org's (2k+ devs) CICD Infra more robust, updated, further scaled and automated. And most importantly, though it may not be in the purview of SRE, help also Devs improve their productivity - and this is the part where I need your help.
I have basic idea of GitOps/CICD, and am looking to learn lot more. In my org we use Jenkins/GitLab/GitHub (don't ask why all 3, and I am trying to phase out Jenkins). I am watching videos on Youtube, but its difficult to get great ideas or deep insight from most of them. Most are repetitive.
I came across the following articles: 1. https://lepape.me/stop-using-gitops-to-sell-your-products/ 2. https://abseil.io/resources/swe-book/html/ch18.html
These gave me some good insights into release-engineering and methodologies practiced in different ways. I am looking for similar articles, or may be anything good that can help me learn more and improve my basics as well as my horizon in this matter. Since I want to help Devs, who work on code related to backend/frontend/mobile, I think I will have to learn a lot.
Can you please throw any good article that you may have bookmarked in this matter and might help me? Release engineering, good branching strategies, possible automation, things to learn (gradle/maven). Anything which you think might be related to these.
Thanks in Advance.