Ask HN: Companies of one, how do you ensure business continuity?
25 comments
Keep everything simple. Dumb, boring. Complexity will drive you nuts. You want to be a boring pilot of a floating boat, rather than a super hero in a sinking ship.
Keep a solid stack of cash. 6-9 months worth of potato diet (consider fixed costs). Never let bills run out on you. Never trust dynamic pricing.
Make systems work for you 24/7, keep measuring, have alerts set, incl. recurring jobs via calendar.
Keep a solid log of changes and most recordable things.
Offload everything related to accounting and focus on one thing, the tech.
Keep your customer's expectations in check. They should know you're not a multinational VC funded grunt cave. Keep it human, keep it real. So that when stuff does go bad, you are on the right foot to answer for it, and the narrative is based around compassion and not SLA's.
Keep a solid stack of cash. 6-9 months worth of potato diet (consider fixed costs). Never let bills run out on you. Never trust dynamic pricing.
Make systems work for you 24/7, keep measuring, have alerts set, incl. recurring jobs via calendar.
Keep a solid log of changes and most recordable things.
Offload everything related to accounting and focus on one thing, the tech.
Keep your customer's expectations in check. They should know you're not a multinational VC funded grunt cave. Keep it human, keep it real. So that when stuff does go bad, you are on the right foot to answer for it, and the narrative is based around compassion and not SLA's.
I can't emphasise enough the keep things dumb.
Sometimes you lose motivation for 6 months... and that business should be able to tick along just fine.
In many ways a business of one needs simplicity and over-engineering. Monitoring, alerts, automation... but all really simple. Nothing clever, nothing difficult, composed of entirely standard parts, tech, systems. I say over-engineering as you don't want to need a human (you) to be doing anything for this thing to function. Your job is literally to put yourself out of your job... and for the business to cruise along. Automate everything... and if you cannot do so, make it so simple that toil is virtually nothing.
Sometimes you lose motivation for 6 months... and that business should be able to tick along just fine.
In many ways a business of one needs simplicity and over-engineering. Monitoring, alerts, automation... but all really simple. Nothing clever, nothing difficult, composed of entirely standard parts, tech, systems. I say over-engineering as you don't want to need a human (you) to be doing anything for this thing to function. Your job is literally to put yourself out of your job... and for the business to cruise along. Automate everything... and if you cannot do so, make it so simple that toil is virtually nothing.
As an example for the simplicity.
I have a small search engine for chemical properties[0] running with a boring go stack, with just a stupid SQLite database (but a custom in memory index for some particular "chemical" search part). It has been humming around for the past 10 years without downtime.
The stack is incredibly simple and monolithic. A complete redeploy is made by a simple push on master, simple CI/CD with Woodpecker[1] building Docker images and triggering a refresh of the system with docker compose.
Every single step can be down manually if a given step or component is failing, but in the "everyday" scenario, everything is automated.
So, keep it as simple as possible with minimal "interconnections" with other services. If possible do not depend on any external services.
[0]: https://www.chemeo.com
[1]: https://woodpecker-ci.org/
I have a small search engine for chemical properties[0] running with a boring go stack, with just a stupid SQLite database (but a custom in memory index for some particular "chemical" search part). It has been humming around for the past 10 years without downtime.
The stack is incredibly simple and monolithic. A complete redeploy is made by a simple push on master, simple CI/CD with Woodpecker[1] building Docker images and triggering a refresh of the system with docker compose.
Every single step can be down manually if a given step or component is failing, but in the "everyday" scenario, everything is automated.
So, keep it as simple as possible with minimal "interconnections" with other services. If possible do not depend on any external services.
[0]: https://www.chemeo.com
[1]: https://woodpecker-ci.org/
As a company of one for 20 years, I have made lots of websites. My advice is to intentionally design the product to be low maintenance from the start. When considering a new feature, always think about future maintenance costs.
For example, integrating a 3rd party API? How reliable is that API, what happens if it goes down, is there an alternative API to have on backup, etc? I reinvent the wheel sometimes, if its not too hard, just to decrease external dependencies.
Thinking of allowing user generated content such as uploading images, or writing comments/posts? Prepare to spend a good deal of your time moderating that content and managing spam. I go out of my way now to avoid having UGC on my sites, even when it would enhance the product.
Going to use the latest bleeding edge technology? What happens in 3 years when it's obsolete or the next upgrade has breaking changes? I try to use dumb boring old LAMP with vanilla JS whenever possible. I have several 10+ year old websites using this boring tech stack that are still chugging along that I literally haven't looked at in years. Whereas I have an old angular 1.0 website for which I have forgotten the build step because I failed to take notes and I doubt "npm install" would work anymore for its ancient dependencies, so its stuck in dinosaur land unless I want to do a total rewrite. Keep it dumb.
For example, integrating a 3rd party API? How reliable is that API, what happens if it goes down, is there an alternative API to have on backup, etc? I reinvent the wheel sometimes, if its not too hard, just to decrease external dependencies.
Thinking of allowing user generated content such as uploading images, or writing comments/posts? Prepare to spend a good deal of your time moderating that content and managing spam. I go out of my way now to avoid having UGC on my sites, even when it would enhance the product.
Going to use the latest bleeding edge technology? What happens in 3 years when it's obsolete or the next upgrade has breaking changes? I try to use dumb boring old LAMP with vanilla JS whenever possible. I have several 10+ year old websites using this boring tech stack that are still chugging along that I literally haven't looked at in years. Whereas I have an old angular 1.0 website for which I have forgotten the build step because I failed to take notes and I doubt "npm install" would work anymore for its ancient dependencies, so its stuck in dinosaur land unless I want to do a total rewrite. Keep it dumb.
Structure.
For me it's all about keeping things simple, automating everything that can be automated and keeping structure in my day. Simple things like always handle all operational stuff as soon as it pops up so you'll never get behind on anything. Once you start getting behind on administration or e-mails it becomes progressively harder to catch up.
And yes, this means that if you're going on holiday your laptop goes with you.
Also, I prioritize eating, exercise and sleeping. If it means I get to spend less billable hours doing an integration project for a customer then so be it. It's almost impossible to overstate how important it is to keep physically and thereby mentally fit when running a business on your own.
For me it's all about keeping things simple, automating everything that can be automated and keeping structure in my day. Simple things like always handle all operational stuff as soon as it pops up so you'll never get behind on anything. Once you start getting behind on administration or e-mails it becomes progressively harder to catch up.
And yes, this means that if you're going on holiday your laptop goes with you.
Also, I prioritize eating, exercise and sleeping. If it means I get to spend less billable hours doing an integration project for a customer then so be it. It's almost impossible to overstate how important it is to keep physically and thereby mentally fit when running a business on your own.
For the last few years with PlotAPI [1], it's been mostly about communicating early and managing expectations. The emails hit my phone, so I always reply within a reasonable time to confirm I've seen their email, thank them for their business, and some timeframe of when I can get around to addressing the issue.
Besides that, I've kept things simple and built my own tools around deployment - so making quick changes/fixes are easy for me as a team of one.
I'm refreshing the docs site [2] right now, which I hope will reduce the number of emails I get and make it easier for customers to get the answers they need (even to questions they didn't know they had) themselves.
[1] https://PlotAPI.com
[2] https://PlotAPI.com/docs
Besides that, I've kept things simple and built my own tools around deployment - so making quick changes/fixes are easy for me as a team of one.
I'm refreshing the docs site [2] right now, which I hope will reduce the number of emails I get and make it easier for customers to get the answers they need (even to questions they didn't know they had) themselves.
[1] https://PlotAPI.com
[2] https://PlotAPI.com/docs
You're essentially on call 24/7, so better get used to it. If you travel, securing a solid Internet connection is your priority, and your family needs to understand you can be called away at a moment's notice. And if you're on a long flight, it can make sense to pay up for Internet connectivity as well.
As soon as it makes sense, you should start hosting somewhere in the cloud so you can offload at least some of the potential problems to SREs. I also found that moving your database to the cloud can be a good trade-off when running solo.
As soon as it makes sense, you should start hosting somewhere in the cloud so you can offload at least some of the potential problems to SREs. I also found that moving your database to the cloud can be a good trade-off when running solo.
As others said, you're on call 24/7, but you can make it easier on yourself. My philosophy:
- Keep everything as simple as it can sensibly be. Obviously this will depend on your application but if you can, you're better off having a simple docker compose stack than a complex k8s cluster (as an example), if you don't really need it. This will make it quicker to fix and get back to your vacation when you need to.
- If you can afford it, pay for cloud hosting and other services (like backup services, Route53 domains with certificates etc). Their SRE efforts are going to be infinitely more capable and available than yours are.
- Use password managers and secure them well. Make it so that you could lose your laptop and be up and running again within 30 minutes of buying a new one.
- Keep your development process consistent and documented. I develop everything inside a Docker container and use the same application/service template for everything. Again, this makes it much faster to troubleshoot and bug fix on another machine if required. I do all my development on an EC2 instance using VSCode Remote SSH + Containers so I can connect to the same environment from any machine. This also means if I need to fix something from somewhere with spotty internet, I only need SSH access and I have access to 10Gb internet etc, so uploading new images and patches etc is not affected by my location.
All these things probably cost me a few thousand dollars a year (tax deductible) but to me they are worth it for the peace of mind and the few times I have had to fix something in a pinch, it has more than paid for itself.
- Keep everything as simple as it can sensibly be. Obviously this will depend on your application but if you can, you're better off having a simple docker compose stack than a complex k8s cluster (as an example), if you don't really need it. This will make it quicker to fix and get back to your vacation when you need to.
- If you can afford it, pay for cloud hosting and other services (like backup services, Route53 domains with certificates etc). Their SRE efforts are going to be infinitely more capable and available than yours are.
- Use password managers and secure them well. Make it so that you could lose your laptop and be up and running again within 30 minutes of buying a new one.
- Keep your development process consistent and documented. I develop everything inside a Docker container and use the same application/service template for everything. Again, this makes it much faster to troubleshoot and bug fix on another machine if required. I do all my development on an EC2 instance using VSCode Remote SSH + Containers so I can connect to the same environment from any machine. This also means if I need to fix something from somewhere with spotty internet, I only need SSH access and I have access to 10Gb internet etc, so uploading new images and patches etc is not affected by my location.
All these things probably cost me a few thousand dollars a year (tax deductible) but to me they are worth it for the peace of mind and the few times I have had to fix something in a pinch, it has more than paid for itself.
I remember some guy ranting that Apple would take his laptop for a week of repairs without providing spare for him.
Having a spare laptop and phone is cheap - it does not have to be latest and greatest.
Having a spare laptop and phone is cheap - it does not have to be latest and greatest.
Off topic but Apple don’t really mind if you buy a new one and then return it after a week once your old one is fixed. Apple support even told my father to do this.
I was a company of one, providing specialised web software powering research funding for multiple clients. I deliberately chose a dead-simple tech stack, and had a backup agreement with a suitably experienced former colleague who could step in if required. I've recently sold the business and now provide the same backup for the new owner.
I'm the sole developer of HashBackup. An important part of planning a solo venture is choosing something that is doable by a single person, and if it fails for a short period of time, say a day or two, the failure will not impact customers too much.
For example, the HashBackup website is for just for documentation, and if it fails, having documentation offline is not a huge deal. If it had user signups, accounts, and all that, it still would not be a big deal to be down as long as the installed software continued to work, at least for a few days. And backup software is not "mission critical" to the same extent that hosting a company's tech support website would be, for example. All of these have been very conscious choices.
For example, the HashBackup website is for just for documentation, and if it fails, having documentation offline is not a huge deal. If it had user signups, accounts, and all that, it still would not be a big deal to be down as long as the installed software continued to work, at least for a few days. And backup software is not "mission critical" to the same extent that hosting a company's tech support website would be, for example. All of these have been very conscious choices.
Don’t sweat too much. Automate things as much as you can (adopt a DevOps philosophy), make sure you can switch laptop anytime (everything is backed up to the cloud, password manager, git repos, etc) and run your app exclusively from a cloud provider (no on-premises). For customer support, they can usually wait a bit for a response.
If there is too much to handle, it means there is some cash flowing in and might be time to get a first employee (or outsource some tasks).
If there is too much to handle, it means there is some cash flowing in and might be time to get a first employee (or outsource some tasks).
As another take — I'd say automate judiciously. If you're a team of one, the benefits of things like CI/CD and containers are reduced.
They're definitely still worthwhile, but that needs to be weighed against how comfortable you are setting up/maintaining these to begin with.
If they don't divert too much energy because you're already well versed in them, they're probably a win.
But needing to shell into a couple of EC2 boxes & manually deploying/upgrading stuff might be a safer bet for some people.
You don't want to answer a P1 ticket at 3am & be getting some esoteric Kubernetes error and have no-one to ask. I think it's tough to learn & adopt new ops tools as a team of one if they're things that are on the critical path.
They're definitely still worthwhile, but that needs to be weighed against how comfortable you are setting up/maintaining these to begin with.
If they don't divert too much energy because you're already well versed in them, they're probably a win.
But needing to shell into a couple of EC2 boxes & manually deploying/upgrading stuff might be a safer bet for some people.
You don't want to answer a P1 ticket at 3am & be getting some esoteric Kubernetes error and have no-one to ask. I think it's tough to learn & adopt new ops tools as a team of one if they're things that are on the critical path.
Yes I should have balanced my comment with « use the things you’re comfortable with ». From my perspective, setting up a QA suite + CI/CD + CaaS type hosting is done in an hour at the beginning of the project. But yeah you’re right YMMV.
However, I disagree with the small benefits of DevOps for a solo dev: if done properly, you start to have complete 8+ hours night sleep!
However, I disagree with the small benefits of DevOps for a solo dev: if done properly, you start to have complete 8+ hours night sleep!
I don't have a company, but I do host a few play projects. My advise is mostly around using cloud providers..
1) build for limited/easy observational/maintenance. Adding observers and maintaining systems is time you could spend elsewhere. 2) favor self-healing. I generally do this by using severless services -- code won't suddenly execute differently in a new lambda/function or container, so if the data center has an issue, my issue is resolved when there's is. 3) keep your running stack simple. The less services you leverage, the less you are impacted by issues. It may make more sense to use the database for locking instead of redis because that's one less thing to learn. 4) reproducibility of your environment. DNS is almost the only thing I need to set up manually (because the effort to automate making SOA/NS links). But after that, I can rebuild my entire code stacks and make a new dev environment in minutes. I then can load up the recent DB backup. Having this also makes it easier to deploy in multiple regions, if you get to that level of redundancy needed.
I guess my advise is mostly keep it simple so your less likely to have an issue.
1) build for limited/easy observational/maintenance. Adding observers and maintaining systems is time you could spend elsewhere. 2) favor self-healing. I generally do this by using severless services -- code won't suddenly execute differently in a new lambda/function or container, so if the data center has an issue, my issue is resolved when there's is. 3) keep your running stack simple. The less services you leverage, the less you are impacted by issues. It may make more sense to use the database for locking instead of redis because that's one less thing to learn. 4) reproducibility of your environment. DNS is almost the only thing I need to set up manually (because the effort to automate making SOA/NS links). But after that, I can rebuild my entire code stacks and make a new dev environment in minutes. I then can load up the recent DB backup. Having this also makes it easier to deploy in multiple regions, if you get to that level of redundancy needed.
I guess my advise is mostly keep it simple so your less likely to have an issue.
Do you mean if you get old and retire or just for vacation? Vacation is easy, just grow to the point where you can hire someone and build the product to stay online without any intervention. Retiring is the hard part. My plan is for my company to become obsolete before I want to retire due to the fast-changing tech landscape.
Automate
Simplify business + product design and make arch boring
Prioritize getting more revenue so you can hire
Simplify business + product design and make arch boring
Prioritize getting more revenue so you can hire
Never promise 100% availability unless you have 24/7 real humans monitoring. It will eventually come back to bite you.
How many deals would you need to close to afford an employee before end of year? Part-time? I would make that your immediate goal.
Keep in mind that that does not actually solve the issue of continuity:
- What if the employee quits
- What if you're unavailable as before, and the employee gets stuck
- What about the plethora of tasks you cannot expect the employee to do
All the while adding complexity of hiring and managing someone
- What if the employee quits
- What if you're unavailable as before, and the employee gets stuck
- What about the plethora of tasks you cannot expect the employee to do
All the while adding complexity of hiring and managing someone
The likely goal is decreasing the bite out of 24/7 on-call wrt customer responsiveness and service uptime
Technical user support & service uptime: can simplify by design, docs, self-healing. For a focused service, remaining timely human support can go to a new hire. If they can even handle 50% of remaining load here, and defer responses to next day/few days, awesome. If they leave, can get another. Bonus points if they can help further reduce the sting here over time by writing more docs or writing more code.
Sales: Unfortunately, someone good at technical support is often not good for sales support, and a minimum, not at the level of the founder who has seen so much more. So self-serve sales automation helps, and eventually another hire here too. Luckily, sales is generally a lower cadence, and tech support can depressure via an acknowledgement of reciept etc till actual reach out.
Technical user support & service uptime: can simplify by design, docs, self-healing. For a focused service, remaining timely human support can go to a new hire. If they can even handle 50% of remaining load here, and defer responses to next day/few days, awesome. If they leave, can get another. Bonus points if they can help further reduce the sting here over time by writing more docs or writing more code.
Sales: Unfortunately, someone good at technical support is often not good for sales support, and a minimum, not at the level of the founder who has seen so much more. So self-serve sales automation helps, and eventually another hire here too. Luckily, sales is generally a lower cadence, and tech support can depressure via an acknowledgement of reciept etc till actual reach out.
Work on reliability. Then there are sysadm companies out there you can pay for part-time services.
Very interested to know more about your experiences!
[1]https://news.ycombinator.com/item?id=32960033