Ask HN: How does your company release software?
I’m working as a software engineer for an enterprise grade company whose core domain is not software itself. It’s my first company and I lack comparison how other companies, from smaller start-ups to larger software companies, release software and ensure quality. I’m mainly interested in how heavy- or lightweight the process is, who is held accountable and if you have a heavy-weight process what do you think is the main reason for that? Thanks for sharing your thoughts!
12 comments
We built some tools in house. I would say we have a fairly pedestrian system. We do B2B software, so there is some build time configuration and enforcement we like to apply from a project management perspective.
90% of our struggle is addressed by the fact that our product is a single binary distribution that only needs to be installed on 1 host. The self-contained deployments option provided by .NET is very helpful too.
The way our process works - someone merges a PR down and a project manager is looking for a build containing this for testing. So, they will log into a system that is integrated with github to trigger a build at the desired commit. Once it completes, the release artifacts are zipped up and sent to a public S3 bucket using a csprng token as the file name. This link is surfaced in our interface for easy handoff to the customer in email. For internal purposes, the project manager can optionally perform automatic deployment of the completed build to one or more internal environments. Automatic, scheduled delivery of the release email to customers is also an option.
We also employ some 3rd party tools, such as github actions to run checkbuilds and gate merging of PRs.
Ultimately, the theme of our toolset is to empower non developers to move mountains. You can get some of this with 3rd party, but you'll never have it fit as well as our solution does for us.
90% of our struggle is addressed by the fact that our product is a single binary distribution that only needs to be installed on 1 host. The self-contained deployments option provided by .NET is very helpful too.
The way our process works - someone merges a PR down and a project manager is looking for a build containing this for testing. So, they will log into a system that is integrated with github to trigger a build at the desired commit. Once it completes, the release artifacts are zipped up and sent to a public S3 bucket using a csprng token as the file name. This link is surfaced in our interface for easy handoff to the customer in email. For internal purposes, the project manager can optionally perform automatic deployment of the completed build to one or more internal environments. Automatic, scheduled delivery of the release email to customers is also an option.
We also employ some 3rd party tools, such as github actions to run checkbuilds and gate merging of PRs.
Ultimately, the theme of our toolset is to empower non developers to move mountains. You can get some of this with 3rd party, but you'll never have it fit as well as our solution does for us.
So we work at the same company or there's just a lot of shared practices in older monolithic .Net shops.
I think we are practicing the same dark arts from different towers.
The .net ecosystem does seem to guide developers to looking inward rather than out to 3rd parties and cloud sprawl. A big part of this is the amount of functionality you get out of the box. We have a ~200meg trimmed distribution and we only need 2 non-microsoft nugets to build it.
The .net ecosystem does seem to guide developers to looking inward rather than out to 3rd parties and cloud sprawl. A big part of this is the amount of functionality you get out of the box. We have a ~200meg trimmed distribution and we only need 2 non-microsoft nugets to build it.
When I worked in enterprise we didn't release software until we ensured quality.
The dev was responsible to test their software before they push their commit - QA's job was not to find bugs from lazy developers.
Our time estimations for tickets included additional time for writing tests.
All PR had to be reviewed for bugs, security, performance etc.. the author ws responsible for getting someone to review it if they wanted it released.
The dev was required to communicate to with QA and explain if any other systems or functionalities were be affected by a change.
Everything was be tested on staging before going to prod - absolutely everything with no exceptions.
Regression tests were never skipped.
If something couldn't be tested then we had to logically deduce why it could not cause an error.
After a release we ran smoke tests and would immediately roll back if any issues.
The dev was responsible to test their software before they push their commit - QA's job was not to find bugs from lazy developers.
Our time estimations for tickets included additional time for writing tests.
All PR had to be reviewed for bugs, security, performance etc.. the author ws responsible for getting someone to review it if they wanted it released.
The dev was required to communicate to with QA and explain if any other systems or functionalities were be affected by a change.
Everything was be tested on staging before going to prod - absolutely everything with no exceptions.
Regression tests were never skipped.
If something couldn't be tested then we had to logically deduce why it could not cause an error.
After a release we ran smoke tests and would immediately roll back if any issues.
Our system is all managed by Gitlab Premium.
Release when ready. Via CI/CD process. Code review from Dev, signoff (approval button on merge request) from Product via Gitlab Review Apps.
This can result in multiple releases a day.
No release branches - everything off main (a la trunkbaseddevelopment.com)
Release when ready. Via CI/CD process. Code review from Dev, signoff (approval button on merge request) from Product via Gitlab Review Apps.
This can result in multiple releases a day.
No release branches - everything off main (a la trunkbaseddevelopment.com)
What about documentation, quality/test reports, change logs etc.?
Based on "This can result in multiple releases a day.", I'm guessing this is a web application, so there's rarely documentation (outside of an externally managed help center, that might get outdated). No change logs. Quality - a lot of web apps rely on automated testing and user reports. If there's manual QA, it's done on the pull/merge request level or periodically on master/main/trunk. When someone is done and says "it works", it can be deployed on production, potentially passing through staging.
The above is just a hypothetical based on how I've seen a lot of web apps being done.
The above is just a hypothetical based on how I've seen a lot of web apps being done.
auto-generated e2e test reports using cypress.io + gerkhin...
for release reporting, it's (currently) quite informal and ad-hoc. Technical releases are often behind feature-flags so we can separate this from customer-facing launch comms.
for release reporting, it's (currently) quite informal and ad-hoc. Technical releases are often behind feature-flags so we can separate this from customer-facing launch comms.
CI builds a docker image. That image goes to a registry on GitLab.
I update the docker-compose on the server I want to deploy the update with the latest version number. Then run docker-compose up -d
I update the docker-compose on the server I want to deploy the update with the latest version number. Then run docker-compose up -d
For WeKan Open Source kanban https://wekan.github.io , very lightweight process. I release new versions very often https://github.com/wekan/wekan/blob/master/CHANGELOG.md .
WeKan has MIT license, used in most countries of the world https://snapcraft.io/wekan . Everything is Open Source, all updates are free, no registration required. WeKan is translated to about 70 languages https://transifex.com/wekan/wekan so it's about 3x more translations than Trello or Jira.
Someone sends PR, I check it, merge if it works. I'll create new release for all platforms. For Snap, it updates all servers automatically worldwide. For other platforms, users update when they like.
If there are any problems, like something does not work, someone adds new issue to https://github.com/wekan/wekan/issues . It is fixed by me or some other contributor, and I make new release, usually same or following day problem has been reported.
WeKan users usually keep daily backups, and restore if something gets messed up.
About who is held accountable:
- If I broke something, I try to fix it.
- If some other contributor sent PR that broke something, I usually revert that change, release new version of WeKan, and that other contributor will send fixed PR later, that is then added to WeKan.
- If some customer has problems, customer contacts Commercial Support https://wekan.team/commercial-support/ , I fix it and document steps to wiki like for example https://github.com/wekan/wekan/wiki/Repair-MongoDB , and for anyone else asking about same thing I send link to that wiki page.
- If someone has installed WeKan to their own server, has not made backups, and has not paid anything for support, that someone is responsible for not making backups.
WeKan has MIT license, used in most countries of the world https://snapcraft.io/wekan . Everything is Open Source, all updates are free, no registration required. WeKan is translated to about 70 languages https://transifex.com/wekan/wekan so it's about 3x more translations than Trello or Jira.
Someone sends PR, I check it, merge if it works. I'll create new release for all platforms. For Snap, it updates all servers automatically worldwide. For other platforms, users update when they like.
If there are any problems, like something does not work, someone adds new issue to https://github.com/wekan/wekan/issues . It is fixed by me or some other contributor, and I make new release, usually same or following day problem has been reported.
WeKan users usually keep daily backups, and restore if something gets messed up.
About who is held accountable:
- If I broke something, I try to fix it.
- If some other contributor sent PR that broke something, I usually revert that change, release new version of WeKan, and that other contributor will send fixed PR later, that is then added to WeKan.
- If some customer has problems, customer contacts Commercial Support https://wekan.team/commercial-support/ , I fix it and document steps to wiki like for example https://github.com/wekan/wekan/wiki/Repair-MongoDB , and for anyone else asking about same thing I send link to that wiki page.
- If someone has installed WeKan to their own server, has not made backups, and has not paid anything for support, that someone is responsible for not making backups.
I describe ours as a hippo’s ass.
Enterprise software also for reference. Huge cargo cult.
Enterprise software also for reference. Huge cargo cult.
PR for a ticket to master. Reviewed by team. Merged to master. Builds, is automatically deployed to a kubernetes cluster.