Theory of Cost Driven Testing

Purpose

In this article we shall explore guidelines that can help us reap the benefits of automated software testing with the least effort and time possible. There's always a trade off between writing bug free software and delivering results quickly. Thanks to the law of diminishing returns, we know that every additional effort produces less returns than before. This is especially true in startups and small dev teams with tight deadlines. Why are some software "bug free" despite having no automated test? Why do some software have many bugs despite high code coverage? Can we write tests that we are very likely to benefit from while not writing any test that is unlikely to help us?

Critical Features

Critical features are those that must be working correctly or there will be dire consequences. A bug in the feature could have catastrophic consequences such as losing your job, losing money, someone dying etc. This should always be the first to have automated testing. Thorough testing is highly recommended for expected and unexpected values since any bug could have potentially disastrous effect. Everything else can come later.

We have to be strict when deciding whether a feature is critical, or we might end up saying that 90% of the code is critical. Would a bug cause the app to crash? That's not critical. Would it make customer angry? That's not critical since it can be used as an excuse for every bug.

Bug Prone Code

Once all critical features are fully tested, we can move on to bug prone areas starting from the most likely to the least likely. Some of the factors that increase the chance of bugs are: amount of custom code, number inputs and outputs, logical complexity, high usage. That usually means testing high level functions such as controllers, instead of low level utility functions. This will let you get more code coverage with less tests, and also cover more flows, inputs and outputs. Functions which are made up of mostly third party party code will have lower priority since most of it are already be well tested. Features that are very frequently used should have higher priority simply because bug discovery is much more likely and more people will be affected by the bug. Compare this to a buggy feature that is barely used by anyone. There's a lower chance of discovering the bug. And even if the bug goes unnoticed, very few people would be affected by it. It is quite unnecessary to test functions that are simple and infrequently used since they are 1) less likely to be buggy, 2) less likely to be affected by code changes, 3) much easier to achieve 0 bug.

Since frequent usage increase the chance and impact of bug, it stands to reason that we should prioritize testing with data that users are more likely to enter. That means we should have lower priority for testing edge cases and bad input. Or simply not test them at all. For example, a user who enters his email in a date field might cause a horrifying error page, but he can easily rectify this by trying again and hopefully realizing this mistake. In the worst case, 1 user will have an unpleasant experience. Compare this to a user who enters the correct value into the correct field. There is simply no way to help himself. And everyone would be affected by it.

That's All

Since the goal is to write as little test as possible, any additional test means we waste more time and increase our technical debt. Hopefully with these guidelines we can write only 20% of the tests yet achieve 80% of the benefits of testing.

Keywords: unit testing, software testing, automated testing, programming