Finding the right Test-Mix

The topic of Test Driven Development (TDD) and unit-testing usually creates heated discussions among developers. We at Qafoo are not an exception and the topic how to apply TDD in our daily work regularly leads to heated discussions. One reason for this is that both TDD and unit-testing combined are by some people seen dogmatically as the only way to do software development.

This blog post tries to summarize our current state of discussion on TDD, software testing and finding the right test-mix. It is written from my own perspective and all errors in description are mine, not my colleagues'. I also try to review the topic in context of the recent discussions of others.

Let's start with Uncle Bob, who is the primary voice of TDD as the only approach to software development. He recently wrote about this in his article "The startup trap" spurring several discussions (HackerNews , Storify, Greg Youngs Blog ). However, he also acknowledges in a blog post the day later, that he does not apply TDD dogmatically to every aspect of software development. Notably, he doesn't write tests for:

• state of objects, for example getters/setters and attributes themselves.

• GUIs, because they require lots of tinkering and small adjustments to configuration variables.

• Third-party libraries

• while experimenting with code (what I would call periods of "Spikes").

Most importantly, check how Uncle Bob never mentions the word unit-test in both blog posts. This is because TDD is about software design, not about testing your individual units (objects, functions, ...) or 100% code coverage. We think, TDD can be applied on any level of the testing pyramid: the acceptance-, integration- and unit-test level. Kent Beck describes this in his book "Extreme Programming explained".

Why do I mention this? We have made the experience first hand that starting a new project with TDD and mostly unit-tests can actually slow down development considerably. This happens because during the development of new projects and features the requirements change frequently. This conflicts with tests locking the code to the current functionality through high test-coverage. Johann Peter Hartmann discussed this topic with Toby in an interview on our blog some weeks ago.

Unit-tests are necessary to stabilize your code, but when you know the requirements are not stable yet, then having too many unit-tests can be a burden as well. It is very difficult to write unit-tests that are immune to change, especially if you don't know what changes will happen. Combined with a dynamic language like PHP that has poor automatic refactoring support and you will suddenly see yourself wasting lots of time adjusting unit-tests to changing requirements.

There are several learnings from this that we try to teach our customers in our trainings:

1. Don't focus on unit-tests exclusively. A good test-mix is required and uses acceptance-, integration- and unit-tests. A "good" ratio for software is something like 70% unit, 20% integration and 10% acceptance tests.

2. During development-spikes, with frequent requirement changes and uncertainty, it can be better to drive the design using acceptance- and integration-tests instead of unit-tests. Unit-test only those parts of the application that are mission critical, highly reused parts of the infrastructure (high cohesion) or stable already.

3. Once the software stabilizes, you should refactor tests from the acceptance- and integration-levels towards the unit-test-level. This makes the tests run much faster, less prone to failure due to side-effects, and allows you to write much more tests for different input combinations and edge-cases. Failures in unit-tests are also much easier to analyze than failures in acceptance tests.

4. Apply risk management to different components of your software: Having a lot of unit-tests is only important for those parts of your software that have a high business value. Features that are less important don't need the same test-coverage as those features that generate the majority of the business value. A few acceptance-tests might already be enough for those less important components.

5. Learning how to write tests that don't break on every occasion. This is beyond the scope of this blog-post.