Ask HN: Founders, how much automated tests do you write?
41 comments
The answer is so dependant on what you're doing, and your industry.
The answers here are all correct for the use case of the person answering it. We don't know your use case so we can't give you a good answer.
For example a seed funded health tech company needs 100% test coverage to be compliant with regulations.
A bootstrapped online collaboration tool with no users doesn't need any tests.
A seed funded online collaboration tool needs some tests so the developers can work together without breaking each others code.
The answers here are all correct for the use case of the person answering it. We don't know your use case so we can't give you a good answer.
For example a seed funded health tech company needs 100% test coverage to be compliant with regulations.
A bootstrapped online collaboration tool with no users doesn't need any tests.
A seed funded online collaboration tool needs some tests so the developers can work together without breaking each others code.
Another aspect is the cost and risk to the company if it ships defects to production (assuming it actually has customers, has largely de-risked product-market fit, etc). Suppose a company is in the hardware business and locks its cash into building and shipping a run of their v1 consumer product, then discovers after shipping there's a defect in the entire manufacturing run that means the product needs to be recalled and repaired. That's likely sufficient to bankrupt a young company that doesn't have revenue flowing in from other products or lines of business. You'd want to be doing a lot more QA than merely unit testing.
Not a founder, but a serial side-project'er. Initial prototype typically has no tests, at least for the bits that are wired. Specific bits of complexity will have unit tests. At some stage might add integration tests using something like docker compose.
Use TDD for bits that have very limited scope and function. For instance, an order book class or similar. Otherwise code first, test bugs if needed. For smaller projects (that aren't critical espc.) overtesting can kill your momentum and motivation, which is an often overlooked resource.
Use TDD for bits that have very limited scope and function. For instance, an order book class or similar. Otherwise code first, test bugs if needed. For smaller projects (that aren't critical espc.) overtesting can kill your momentum and motivation, which is an often overlooked resource.
While working on my startup as a solo-founder I used to follow a TDD approach supported mostly by Integration tests. I mostly focused on the user happy paths with limited coverage outside the Integration tests.
The goal is to have the major functionality of the platform tested automatically so we can be confident that new features don’t introduce new bugs.
The goal is to have the major functionality of the platform tested automatically so we can be confident that new features don’t introduce new bugs.
Precisely what I do with personal side projects, and for greenfield development at the company at which I’m a cofounder (and which has been around for over a decade now).
Coverage is the best example of Goodhart's Law, never focus on it.
Integration tests are typically best bang for buck: https://kentcdodds.com/blog/write-tests
Once manual testing starts causing noticeable pain, you can start automating some of it. This may very well be beyond $10-100M in valuation.
A lot of people say unit tests give them confidence, but that's a delusion. Losing confidence in your code means you have outgrown your architecture. However, you can temporarily address it with more tests, because addressing architecture issues takes some time. The secret is to write as little tests as possible to bring confidence back to acceptable level.
Don't spend more than 10% time on tests.
Don't let junior engineers write tests.
Integration tests are typically best bang for buck: https://kentcdodds.com/blog/write-tests
Once manual testing starts causing noticeable pain, you can start automating some of it. This may very well be beyond $10-100M in valuation.
A lot of people say unit tests give them confidence, but that's a delusion. Losing confidence in your code means you have outgrown your architecture. However, you can temporarily address it with more tests, because addressing architecture issues takes some time. The secret is to write as little tests as possible to bring confidence back to acceptable level.
Don't spend more than 10% time on tests.
Don't let junior engineers write tests.
I don’t think this is correct. I write scientific computing code in Rust at the moment for example — I write unit tests to make sure it actually works.
Being confident my stuff works without unit tests would require me to write unit tests and then delete them.
Being confident my stuff works without unit tests would require me to write unit tests and then delete them.
I very much agree with James Coplien approach to unit tests:
- Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value.
Though I'd say that for low-level code (without dependencies) there's basically no difference between unit/integration/e2e tests. They are just tests.
The only difference I found is that with unit tests people will also test implementation details. So it's still better to have integration test mindset even when your only option is unit tests.
- Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value.
Though I'd say that for low-level code (without dependencies) there's basically no difference between unit/integration/e2e tests. They are just tests.
The only difference I found is that with unit tests people will also test implementation details. So it's still better to have integration test mindset even when your only option is unit tests.
E2E for happy paths and unit tests for algorithms that are nontrivial, that’s about it.
[deleted]
Almost Never test-first. It impedes my concentration and flow and inspiration. A new idea to me is like a fresh slab of marble you have to sculpt. Over time, you play with the code and api ergonomics datamodel and abstractions and UI until it “feels” right and takes on a more defined form. Working on tests at that point would be like continuously stopping to check the foundation of the marble.
But for self contained code that is guaranteed to have edge cases (working with dates, date math, unstructured user input like SMS responses, almost always yes, but after some of the code is written and is getting cumbersome to test different cases manually.
But for self contained code that is guaranteed to have edge cases (working with dates, date math, unstructured user input like SMS responses, almost always yes, but after some of the code is written and is getting cumbersome to test different cases manually.
Not a founder, but I work at a very large fortune 500 firm.
The vast majority of our custom apps have little to no tests.
However our INFRASTRUCTURE is loaded with tests, heartbeat monitors, etc.
You name it, we have it.
On front end, almost none. Apps are very test-hostile, so much so that just setting up the framework can increase complexity by ^3.
Well, unit tests are fine and often we do it for things like testing that email parsers and unit conversion works. As long as it's easier than manual testing.
It's also easier to do tests than debug sometimes.
Back end is a different story - the ROI on tests increases drastically because tests are easier to write and failure is more expensive.
Well, unit tests are fine and often we do it for things like testing that email parsers and unit conversion works. As long as it's easier than manual testing.
It's also easier to do tests than debug sometimes.
Back end is a different story - the ROI on tests increases drastically because tests are easier to write and failure is more expensive.
[deleted]
Code is a liability. Founder or not I write as few tests as I feel necessary for things I don't trust in my code.
I generally write as little code as possible overall, and try to sell it as fast as possible.
I hate source code, it never ever is the result for me. So if i do not have to write code than I get to not write tests.
My goal is always to write as little code as absolutely possible. That's it
I generally write as little code as possible overall, and try to sell it as fast as possible.
I hate source code, it never ever is the result for me. So if i do not have to write code than I get to not write tests.
My goal is always to write as little code as absolutely possible. That's it
Are you coding in bash script - just kidding
I mean how does it even works ? What if you want to build an app for a company which is wishing for digitization of their business - you need to have code - Don't you ?
Secondly how come founder are not liability - I have seen a stage in start ups when it comes to the situation where startup is either sold or purchased by other competitive business and after some months when the startup is absorbed the founders are gone or have a silent positions... Doesn't it make them liability afterwards?
I mean how does it even works ? What if you want to build an app for a company which is wishing for digitization of their business - you need to have code - Don't you ?
Secondly how come founder are not liability - I have seen a stage in start ups when it comes to the situation where startup is either sold or purchased by other competitive business and after some months when the startup is absorbed the founders are gone or have a silent positions... Doesn't it make them liability afterwards?
"I hate code, and I want as little of it as possible in our product." (Jack Diederich)
The best practice is to write the test by knowing all the scenarios.
Usually this is the practice I have seen successful in all the big projects
- unit test for each function
- Functional Integration test to test the building blocks and functions together (for example rest API with business layer and database
- BDD Behavior driven development where you write automated test to test your business scenarios by initializing the seed data in the test systems
- Performance testing
If you have freelancer who charge per hour or if you have consultants which are expensive - they will write too many test and there code will be so complex in the end that when you want to hire permanent employees or other freelancers they will say "I would like to rewrite it"
The developers now a days like to do clean coding and for simple things they discuss too much even for a program of 2 + 2. there will be endless discussions and in some companies (diverse or multi culture) developers are always under mining each other with what they like and don't like
Usually this is the practice I have seen successful in all the big projects
- unit test for each function
- Functional Integration test to test the building blocks and functions together (for example rest API with business layer and database
- BDD Behavior driven development where you write automated test to test your business scenarios by initializing the seed data in the test systems
- Performance testing
If you have freelancer who charge per hour or if you have consultants which are expensive - they will write too many test and there code will be so complex in the end that when you want to hire permanent employees or other freelancers they will say "I would like to rewrite it"
The developers now a days like to do clean coding and for simple things they discuss too much even for a program of 2 + 2. there will be endless discussions and in some companies (diverse or multi culture) developers are always under mining each other with what they like and don't like
It depends.
I've seen so many absolutist opinions on testing that I'm distrustful of the wisdom of. When I see people say that there's only 1 way to test or not test things, and you always have to do it in a certain way, and fully test everything, I'm not sure that they're right. I think your need for code testing, and how much testing you do, matters more based on the context and risk you see through experience and good judgement.
> Making software for nuclear power plants? Sure, go slow and test the shit out of it.
> Making software for cars or space ships? Sure, go slow and test the shit out of it.
> Making large scale software for a big financial company where millions of dollars are at stake in the case of a fuckup? Sure, test the shit out of it. You can afford to throw 8 full teams of developers on every continent into nothing but testing, and then testing the tests, and then testing the tests for the tests.
> Building an API that multiple teams and customer are going to depend on? Yeah, testing that sounds like a good idea to ensure everything is working as expected.
> Running a profitable SaaS? Sure, at a certain point where you're not in wartime mode struggling for survival, fully testing and formalizing a lot of things makes some sense.
> But trying to get a tiny 1-human startup off of the ground that no lives depend on? I don't know what works for you best, but would you rather write code to make you money, or spend lots of time writing tests on stuff that's probably going to change a lot later anyway or that you don't even know if it really is a realistic business?
> Building a shitty app for some dumb marketing firm or whatever? I've literally never seen anybody in many types of businesses demand formal code testing. The only thing that matters to them is user-acceptance: the client and/or their QA team will manually test all of the functionality and tell you whether or not it's acceptable to them.
I've seen so many absolutist opinions on testing that I'm distrustful of the wisdom of. When I see people say that there's only 1 way to test or not test things, and you always have to do it in a certain way, and fully test everything, I'm not sure that they're right. I think your need for code testing, and how much testing you do, matters more based on the context and risk you see through experience and good judgement.
> Making software for nuclear power plants? Sure, go slow and test the shit out of it.
> Making software for cars or space ships? Sure, go slow and test the shit out of it.
> Making large scale software for a big financial company where millions of dollars are at stake in the case of a fuckup? Sure, test the shit out of it. You can afford to throw 8 full teams of developers on every continent into nothing but testing, and then testing the tests, and then testing the tests for the tests.
> Building an API that multiple teams and customer are going to depend on? Yeah, testing that sounds like a good idea to ensure everything is working as expected.
> Running a profitable SaaS? Sure, at a certain point where you're not in wartime mode struggling for survival, fully testing and formalizing a lot of things makes some sense.
> But trying to get a tiny 1-human startup off of the ground that no lives depend on? I don't know what works for you best, but would you rather write code to make you money, or spend lots of time writing tests on stuff that's probably going to change a lot later anyway or that you don't even know if it really is a realistic business?
> Building a shitty app for some dumb marketing firm or whatever? I've literally never seen anybody in many types of businesses demand formal code testing. The only thing that matters to them is user-acceptance: the client and/or their QA team will manually test all of the functionality and tell you whether or not it's acceptable to them.
When working with Regex and PyParsing, I find it imperative to maintain unit tests on everything small because I am dealing with over 2,400 abstract keyword combination built out of some 423 EBNF syntax of ISC Bind9 named.conf configuration file and all in Python.
Tiny mistakes that I make are caught early on, and development gets a solid foundation in which to build the proper abstract syntax tree.
Oh, test code is about 1/4th the size of the original source code, line-wise, but it has 100% coverage on syntax linkage (and not necessarily 100% data value validation due to dependency on other field settings).
source: https://github.com/egberts/bind9_parser
Tiny mistakes that I make are caught early on, and development gets a solid foundation in which to build the proper abstract syntax tree.
Oh, test code is about 1/4th the size of the original source code, line-wise, but it has 100% coverage on syntax linkage (and not necessarily 100% data value validation due to dependency on other field settings).
source: https://github.com/egberts/bind9_parser
The things that keep breaking get automated tests. The things that can't possibly break or it will cost me significant revenue or time get tests. Everything else gets to wait until it falls into one of the above two buckets.
This is something I have been considering as I work on firestarter [0] which is a going to be an open source starting point for building a SaaS.
I decided tests are a good thing to have, and especially when already set up and you can easily add more by following the pattern.
I prefer unit tests that are narrow in scope and avoid IO interactions (for example databases)
Of course anyone using my framework can ignore the tests if they desire. But my gut feeling is tests save you time in the short run as well as long run.
[0] https://github.com/mcapodici/firestarter
I decided tests are a good thing to have, and especially when already set up and you can easily add more by following the pattern.
I prefer unit tests that are narrow in scope and avoid IO interactions (for example databases)
Of course anyone using my framework can ignore the tests if they desire. But my gut feeling is tests save you time in the short run as well as long run.
[0] https://github.com/mcapodici/firestarter
In our backend team we almost exclusively write integration tests.
For the frontend we do automated ui tests with a service called Autify.
This is not by any means extensive, BUT with very little time investment we get catch most of our bugs.
For the frontend we do automated ui tests with a service called Autify.
This is not by any means extensive, BUT with very little time investment we get catch most of our bugs.
I pretty much have the same strategy. We mostly want to ensure that crucial functionality doesn’t break for users. Since we are moving fast, we try not to break things.
Ghost Inspector used to be my go to, but I’ve been looking for something in the $20-30 per month range. Makes it a much easier decision when we aren’t yet collecting much revenue.
Ghost Inspector used to be my go to, but I’ve been looking for something in the $20-30 per month range. Makes it a much easier decision when we aren’t yet collecting much revenue.
I write tests as I go along. It helps me challenge my assumptions about what I wrote, and it helps me write more robust code. I usually end up with 80-90% coverage.
Last company, no tests for the first 6 months. Current company after 2 months, have some unit tests for complicated parts in place, will start writing tests once bugs are resolved.
I guess it really depends on what your company is doing, experience of the team and the language/tech you are using.
I guess it really depends on what your company is doing, experience of the team and the language/tech you are using.
Tip: resolve the bugs by writing tests
technical perspective, ignoring business perspective: resolving the bugs by writing tests is excellent advice, especially once a component is sufficiently complex. get regression tests in place first, so you can sense how may new unknown defects a "fix" for a known defect is introducing. Michael Feathers' book "Working Effectively with Legacy Code" describes one approach.
somewhat political perspective: if the org doesn't value testing or software quality, wait until there's obviously a business need to lift quality. E.g. the CTO is saying things such as "constant defects in component X are destroying most of the value of our product", users complain that the Y feature keeps getting broken in successive releases, customers refuse to renew subscriptions or upgrade to a new version until Y feature works reliably. Then propose the plan to improve QA by setting up an automated regression test suite, to allow refactor-to-test of the offending code to happen in a controlled, low risk way.
somewhat political perspective: if the org doesn't value testing or software quality, wait until there's obviously a business need to lift quality. E.g. the CTO is saying things such as "constant defects in component X are destroying most of the value of our product", users complain that the Y feature keeps getting broken in successive releases, customers refuse to renew subscriptions or upgrade to a new version until Y feature works reliably. Then propose the plan to improve QA by setting up an automated regression test suite, to allow refactor-to-test of the offending code to happen in a controlled, low risk way.
Agree with this totally, additionally if you are unsure if you will proceed with an idea with no users, getting something done and receiving feedback as fast as possible is better than doing it perfectly the first time.
That said at my last company after 12 months we hired a few engineers so we had to transition to requiring tests for new functionality. This was just to stop regressions and make sure we could continue moving quickly.
That said at my last company after 12 months we hired a few engineers so we had to transition to requiring tests for new functionality. This was just to stop regressions and make sure we could continue moving quickly.
I partly agree with you, but I also think that using debugger/stepping through code/test manually will take longer times in most cases. And it's time wasted that does not produce anything "permanent" - unlike tests.
I write tests after fixing bugs to make sure they don't happen again. Mostly integration-tests.
I only write tests in two situations:
1) mission critical parts of my system that will cause damage if they break
2) small functions that I can't wrap my head around (eg some complex regex extraction)
Everything else can be fixed fast with minimal damage as it comes up. My test coverage is less than 1%.
1) mission critical parts of my system that will cause damage if they break
2) small functions that I can't wrap my head around (eg some complex regex extraction)
Everything else can be fixed fast with minimal damage as it comes up. My test coverage is less than 1%.
Depends. For web-app and business logic, we have about 95% coverage. All our models and back-end processes are well-tested with error cases. Our front-end & E2E is much lighter and we focus on happy cases.
Firmware is mostly tested manually.
Firmware is mostly tested manually.
E2E for the core & commonly used features. But I plan to add more E2E tests for less commonly used features that I’d still consider core (these are manually tested for now due to time constraints).
Around 15-20% test coverage sadly enough. As a solo founder, small team, not enough funds in the bank, the most important thing is shipping the next change and iterating from there
The second you "think" a piece of software/a feature works, you should be verifying that this is the case automatically. It's often just as easy to write an automated test as test manually, just do it and that test will be there forever saving you from your own idiocy. You don't need to write the tests before you know how the feature is going to shake down to its implementation, but you'd sure do better to test before pushing to prod. You save yourself so much hassle in the long run with automated CI.
Most code is tested, even for hobby projects. I find I save more time down the road with tests.
I only write tests for important stuff that I must continually work on. Although in the past 12 months, my tests have been written by either chatgtp or playground codex models before that. Which makes me lose 0-time testing.
> Full coverage?
No, risk-based.
No, risk-based.
Prototyping and figuring out product? Zero.
Zero
How do you go about it? Full coverage? End to end only? Integration only? How much dev time does it end up taking you?