Test-Driven Development Violates the Dichotomies of Testing

Kent Beck, Three Rivers Institute

June 2007

 

Tests are often described by dichotomies: unit vs. functional, black-box vs. white-box, testing vs. design, tester vs. coder. Test-driven development (TDD) doesn’t fit comfortably in any of these dichotomies. This paper points out how TDD “breaks the rules”.

Explanation

This or that. Western logic is based on dichotomies. Dichotomies help us understand the world. Red state/blue state. Offence/defense. Once you know what category something falls in, you know what to do with it.

Dichotomy is a tricksy tool, turning easily in the hand. The quick, comfortable division into this or that inevitably misses the nuances of the real situation. By insisting that dichotomies are real we close off our minds from options that don’t fit, like finding ways for defenders to score. Especially when encountering a new idea, one learning strategy is to discard comfortable dichotomies and try to experience the idea on its own terms.

My topic for today is TDD and its uncomfortable encounter with the dichotomies used to describe more familiar styles of tests. The dichotomies don’t fit well, which has hampered understanding of TDD, what it can be used to do and what it can’t be used to do.

TDD inverts the familiar design/code/test workflow of programming. To test-drive development, start with an outline of tests that need to be satisfied by the system. Then, write one test at a time (which may involve referring to program elements that don’t exist yet), create stubs so the test compiles even if it doesn’t run, fill out the implementation of the stubs so the test runs, then refactor the design and implementation to comfortably support the code. Repeat this cycle until all the tests in the outline, plus any other tests discovered along the way, are implemented and satisfied.

By rearranging the steps of programming and intimately interleaving testing and development, TDD confounds the categories that have been used to understand testing. The first is the dichotomy between unit and functional testing. A TDD test can operate at a high level, especially when beginning implementation of a new part of a system. While such a test covers a lot of ground conceptually, the initial implementation is likely to be simple, refined to satisfy later tests. In TDD you write the test that suggests writing the next bit of functionality, regardless of scale.

The unit/functional distinction seems to be valuable primarily when running tests. A unit test runs fast and has a small number of reasons it could fail. When a unit test fails, it points an accusing finger at a small part of the program. A failing functional test requires more exploration to find the cause of failure and often runs slower. In TDD, large-scale tests check that systems are wired together correctly, while smaller-scale tests check the detailed functioning of the parts. Since programmers think thoughts at both levels, both kinds of tests are useful.

Another misleading dichotomy is between black-box and white-box tests. Since TDD tests are written before the code they are to test, they are black-box tests. However, I commonly get the inspiration for the next test from looking at the code, a hallmark of white-box testing. The advantage of black-box testing is you aren’t influenced by the code so, even if you write some redundant tests, you have a good chance of writing tests that weren’t imagined by the programmer. The advantage of white-box testing is you can use the structure of the code to identify classes of equivalent inputs and avoid wasted tests. TDD doesn’t take advantage of independent verification (although you could apply that technique on top of TDD), but it gets some of the advantages of black-box testing along with the advantages of white-box testing.

Another dichotomy confounded in TDD is the distinction between tester and coder. The person applying TDD is certainly a tester, because they are writing and running automated tests. At the same time, they are certainly a coder, because they are writing and running code as part of the same workflow. The old Taylorist divide between those concerned with producing and those concerned with quality begins to fade when applying TDD.

The distinction between testing and design is also blurred in TDD. Tests written first are a design notation, at least the design of the programming interface. I was always taught to separate logical and physical design, but no one in college explained how. TDD helps programmers separate the two, since they are encountered at different parts of the cycle—logical design while writing the test and physical design after the test is written and is already failing.

None of these dichotomies—unit vs. functional tests, white- vs. black-box test, tester vs. coder, and testing vs. design—apply comfortably to TDD. TDD uses tests to motivate and inform design, refine and communicate intent, and smooth the troubled emotional waters of programming. More about that in a future paper.