Save the code or save the tests? (2020)(qntm.org)

26 pointsby feltsensevor 5 Jahren30 comments

Save the code or save the tests? (2020)

https://qntm.org/tests

32 comments

> Doesn't matter how good the tests are, by the time you've "reconstructed the code" from tests you're out of business. Running code with no tests makes money. Tests with no running code don't.

^^ comment from someone named hobbs

Exactly. Users can tolerate an app that's kinda broken in some ways, but tests don't make money. In fact tests can often cost money anyway if they're not testing what's really important. No matter what, there are going to be parts of your code that simply cannot be described by the tests that were written.

cogman10vor 5 Jahren

I look at it from a pure time perspective. I generally spend something like a 10:1 ratio of writing code vs tests. That is, it takes me 1/10th the time to write testing stuff vs the actual code.

With that perspective, it's beyond silly to prefer the tests. You are saying "Should I throw away 90% of the work or 10% of the work?"

Said another way, you are asking "Should I save the chapter headings of my book or the content?". I can make new chapter headings, but writing the actual chapters is where I spent all my time.

aliasElivor 5 Jahren

Saving only the tests is completely nonsensical. Even with 100% code coverage, tests will only cover a tiny fraction of the programs behavior. The original program of course fully describes its behavior. If you are lucky you might even have a description in the comments why some part of this behavior was chosen.

pettersvor 5 Jahren

I have worked in the airline industry with scheduling algorithms. We had a very comprehensive test suite, but reconstructing the algorithms from that would take a decade. That surely only works for toy code bases. No one saves the tests.

refactor_mastervor 5 Jahren

Exactly. Verifying an algorithm is easy. Writing it is hard.

postalratvor 5 Jahren

add(a, b) => a + b

How would you verify that algorithm?

zeroimplvor 5 Jahren

Something like:

    assertEquals(0, add(0,0))
    assertEquals(1, add(0,1))
    assertEquals(2, add(1,1))
    assertEquals(8, add(3,5))
    assertEquals(-3, add(2,-5))

PS: This kind of proves why a test is a lot less useful than the algorithm. Try figuring out how to implement "add" from the above test.

anonytraryvor 5 Jahren

Personally I see this a lot and I'm not a fan:

  it('should work', () => {
    inputOutputPairs.forEach(([input, output]) => {
      expect(testedFunction(input)).equals(output);
    })
  })

Tests are way more useful when they are written with the intent of acting as documentation. If I had to give this test a grade (from F- to A+), it would get a D+. It catches regressions, yet does an absolutely horrid job of explaining the intent behind testedFunction.

ramses0vor 5 Jahren

    cases = [
      { in: 1, arg: 2, expected: 3, message: "base case" },
      { in: 2, arg: -2, expected: 0, message: "negative numbers" },
      ...
    ]
    for ( c in cases ) {
      actual = obj.do( c.in, c.arg )
      assert( actual == c.expected, c.message )
    }

see: http://fit.c2.com/ ... table-based testing, "cases" and "drivers"

cronin101vor 5 Jahren

    Forall A:
    add(A, 0) == A

    Forall A B C:
    add(add(A,B), C) == add(A, add(B, C))

I think that’s sufficient. Rule 101 of useful testing is understanding the actual constraints you rely on.

Maybe throw in add(A, B) == add(B, A) if you’re angling for a promotion…

postalratvor 5 Jahren

    add(a, b) => a == 1 && b == 1 ? 1 : a + b

Kinda disappointing your test would take so long to run and still catch a simple error.

milkey_mousevor 5 Jahren

> your test would take so long to run

I think GP's test would probably be implemented with property testing (or even better, symbolic execution). It wouldn't necessarily take super long to run.

postalratvor 5 Jahren

How difficult would it be to write a proper test with one of those systems for something more complex when even this trivial example doesn't have solution yet?

[deleted]vor 5 Jahren

knightofmarsvor 5 Jahren

This is absolutely ridiculous. The code is the product, without it, the company is out revenue the entire time that the application is being re-written to match the tests.

FartyMcFartervor 5 Jahren

Tests are a development tool: they make it easier for developers to change code confidently.

So to rephrase the question: save the code, or save something that makes it easier to change code you no longer have? It seems a bit easier to answer if asked this way...

scaryglidersvor 5 Jahren

If I were to lose all the code for my Elite: Dangerous 3rd party application, I'd not be losing just the code - I'd be losing all the UI stuff.

Even IF I ever wrote unit tests for it - which I don't do and have never done for anything - those unit tests wouldn't have anything in them from which I could reconstruct the UI.

No. You save the code and the surrounding UI and the UI glue code.

slvervor 5 Jahren

> More recently, out of curiosity, I pitched the same question as a poll on Twitter. On Twitter, roughly 80% of respondents said that they would save the application code and 20% said they would save the tests.

Surprise, common sense.

Saving the tests is like needing to know the answer to the Ultimate Question of Life, the Universe, and Everything (which is 42) and discarding the question.

cogman10vor 5 Jahren

I'm shocked that 20% of devs value tests over working code. This should have been a 99% to 1% thing.

aszenvor 5 Jahren

I'll try to justify, imagine you have working code with no tests, it's been running for years but no one can understand it or change any of it. It's providing a very fixed value. Now imagine you have all the tests laying out exactly how the features should work. Sure you don't have anything to show for yet but it's not going to take a lot of time to implement these features. The hardest part in most companies is not building features but what features to build and how should they work. This usually takes domain knowledge and a lot of trial and error. So let me put it this way, if there was a competitor in the business, I would definitely like to see their tests rather than their working code. Since tests actually define how their app should work and not their code which merely tells me how their app is working.

cogman10vor 5 Jahren

I have worked on exactly that sort of code base. I've also worked on codebases which did super test coverage.

And honestly, the 0 test coverage base was actually easier to deal with.

Tests are good, but like all code they can be implemented poorly.

aszenvor 5 Jahren

Some test cases are really important and cover the whole domain, take as example the tc39 tests, they are really all you need to fulfill to create a compliant javascript implementation.

lakisvor 5 Jahren

> it's not going to take a lot of time to implement these features That's so wrong. Have you ever worked in a complex program?

aszenvor 5 Jahren

Yes ofcourse. Having tests helps a lot, take as example someone building a markdown parser. If u have the knowledge of how to build parsers, then an existing suite of tests is going to speed your work quite a bit by ensuring that you build the right thing and covering the exact features you need without any ambiguity.

Ofcourse I do realise that in real world examples there's not many test cases which cover a feature fully. But this was all thinking hypothetically.

jokethrowawayvor 5 Jahren

Peer pressure and the TDD religion

drewcoovor 5 Jahren

> When would this hypothetical, or anything like it, ever happen?

Having test code and dev code in separate repos is a pattern in some places. Where that exists, save the dev code. Any test code that's not allowed to mingle with dev code is highly suspect and almost certainly not a working spec.

jspashvor 5 Jahren

Well I use elixir so of course I would save the code. The tests are built into the comments!

[deleted]vor 5 Jahren

lmilcinvor 5 Jahren

I see how I am going to be downvoted to hell, but let's give it a shot:

I just refuse to write tests. I think it is complete waste of time.

Here, I said it.

My preferred style of working is writing something and then refactoring it to my satisfaction. I toy with interfaces as I discuss meaning of various things, learning in the process. I am mercilessly cutting stuff that is not necessary, finding ways to make everything shorter, simpler, easier to understand. In the process I am constantly moving stuff around, changing concepts for other concepts, removing code that has already been written that I find no longer necessary, etc.

Writing tests might make sense if you know how the code will be structured, what is going to be exact contract for that method, and so on. It doesn't make sense if you are going to be changing it in a moment, multiple times, with 20% probability you are going to remove it anyway.

I tried to give it a shot, but writing tests breaks my flow as I try to put some kind of structure as quickly as possible and then when I am polishing it to perfection.

Writing tests after the code has been written seems like a waste of time for another reason. I have already written the code and thought through the contract. Writing complete unit test suite would take about as much time as time necessary to write the code in the first place. But if I put so much time writing that code correctly there is relatively little chance somebody is going to be reworking it. That's basically the whole point, to make it less likely that it is going to be hotspot of work in the future. I put effort in that module so that I don't have to go back to it.

Spending a lot of time now to have some probability it is going to save some amount of time in the future, most likely less or comparable with the amount of time invested in the tests -- total waste of time.

Later, assuming I actually try to refactor code that is tested, the tests are getting in my way. Thanks to tools like IDE I can edit a lot of code efficiently and with high degree of reliability. I can move stuff around, simplify, then move again, then simplify again and so on. But there is still no automation to correct unit tests accordingly -- every time I changed the structure of the code significantly I have to go to correct those tests.

So it kinda seems it doesn't help me with the refactoring task either. There might be a chance I make a mistake that could have been caught by a unit test -- but my experience shows that it is rather rare occasion that a unit tests prevents me from doing something stupid.

Instead I am spending effort now to edit all those unit tests for a speculative benefit of saving some time later, and likely less than I am spending now.

Add to that that most codebases I have seen in the past consist mostly of bullshit unit tests -- tests that are there only to prop up test coverage to a level required by CI pipeline and usually test only simplest parts of the contract or even plainly stuff that is completely self explanatory.

Why the bad state of unit tests? I think reasons are quite simple:

1. If the test is written after the code it means the developer has already written the code and is now not that much interested in doing something else that doesn't seem to be immediately improving the product. About the same reason why most documentation is shitty and out of date.

2. Production code is constantly being tested and there is feedback if it is not correct. There is no feedback if tests are not correct. The only real requirement for unit tests is that they pass and that there is enough of them.

3. If the deadline is looming, what do you think is getting cut first? Expect absolute least effort into writing tests or documentation.

4. It is hard enough to get developers to review production code. I have yet to see a reviewer that would do anything else than skip or only gloss over test code.

Having to make a choice -- do I want more well written code or less code but with test coverage that is most likely going to bring little value... my choice is rather simple.

yoyonamitevor 5 Jahren

I do think it's important to think critically and not just do something for the sake of doing it. Testing well is not easy to get right. If your tests are constantly changing, then the tests are probably too coupled with the implementation. However, I think it's safe to say that the user wants some level of consistency in the product. If your tests captured what needs to stay consistent, you should theoretically see less churn in the test code.

There are other ways to get a stable system. It seems like the code that you're working on doesn't see much code churn and you have a good understanding of the system. This is definitely a situation where I see testing not being as valuable.

I think one of the big reasons testing is considered a good practice is we're relying more on dependencies that need to be updated somewhat frequently, and semver is not taken as seriously, especially in certain web ecosystems. There's a decent chance you don't work in such an ecosystem where it's not an issue, but that's the biggest reason for me to advocate for testing.

The other big reason is that it's easier to make changes in a codebase that you aren't familiar with when tests are available. This is especially true with software that may have non-obvious corner cases.

aszenvor 5 Jahren

If you are trying to put structure to your code, you would like to get instant feedback on how that structure works from the outside. That is how the api works, the best way to do is to make a test which actually uses your code, and asserts some invariants on it.

With tdd the idea is you write a test first to explore how your code should be called, then you make that code, satisfy the test and add more requirements and repeat the whole thing. Once you have a bunch of tests which are defining the use cases in your code, you can refactor/structure your code internally without affecting the tests or if you think the whole api could be simpler, change the whole thing along with tests. The refactoring at this stage is generally of higher quality since you have a concrete idea of all the ways your code needs to work while as if you refactor early there is little guarantee that it's any good except for intuition.

If you are structuring or abstracting without examples of how your code is to be used, then the design often turns out to be inefficient, overly complex, coupled or over abstracted.

Admittedly this TDD approach is a bottom up design method, it takes time and effort and it's not well suited for problems where you already have a good idea of how code should look like/feel/behave.

You don't need to run these tests every time in ci, you can delete them or keep them as documentation or run them whenever you have to work on the code again.

Diggseyvor 5 Jahren

I agree with you somewhat: especially when starting a project, it just makes no sense to write tests up-front (again, unless the product is the tests).

That said, when making incremental changes to an existing project, tests (and in some cases TDD) have a lot of value. There comes a point where the project changes from a creative endeavor to a product. Then you need tests.

The problem is how you create a test suite at that point in time without being negatively influenced by having written the code. Most unit tests I've seen in the wild are BS for this reason: the person writing them just repeats the code they wrote to implement the feature, or at least makes the same bad assumptions. In most cases they mock out so much that the test bears no relation to the real world, and breaks with the slightest change to the code.

For this reason I prefer to write tests from the top-down instead of bottom-up. Start with a few end-to-end tests, making sure to cover the happy paths and a few of the more probable error cases. These few initial tests are more valuable than any that will come later, because they best correspond to how the product will be used, and they are least likely to break during later refactorings since they don't depend on any internals, making them very cheap to maintain. For smaller projects, these tests may be all you need.

Every level you go below that incurs an increasingly large maintenance cost, so you have to weigh it against how often the design of that component will change.

When I do write unit tests it will be for essentially pure functions where the domain of inputs is sufficiently small that you can do fairly exhaustive testing. For example, a function that determines whether a floating point value can be losslessly converted to a 32-bit integer is perfect for unit testing. Nothing needs to be mocked - whatever the code does under test is what it will do in production.

All of this is my preference, but the only thing that really matters is that you actually use your brain when deciding what tests you write: weigh up the costs of the test against the set of potential bugs that it can prevent. The real sin of TDD is that it teaches you to write thoughtless tests.