Large numbers of failed tests are rare

What is the lifetime cost of developer tests? Do they smooth development into the future or are they like mud, clinging ever thicker to your boots until you can’t move at all?

I used a data set from DevCreek to explore this question. DevCreek is a site that helps programmers gather statistics on their development activities. The data in this set comes from the programmers at Intelliware, a Toronto software consultancy and the developer of DevCreek. Their data in DevCreek covers almost 100 person-years of development.

DevCreek gathers data on testing by recording the results of every test run during coding. There are nearly 300,000 test runs in the data set. Mining this data provides clues about the cost of tests.

Data

Tests that run incur only the cost of waiting for the test to finish (this can add up, but that’s the topic of another post). It’s the tests that fail that require programmer time and attention. So, how many tests fail on average every time a programmer presses “Run”? Actually, that’s a trick question. A large majority (85%) of test runs have either zero or one failure. Plotted on a log-log scale, the number of failures per run forms a fairly straight line, characteristic of a power law distribution:

The data show that occasionally large numbers of tests do fail (5049 was the most tests that failed in a single run). This doesn’t necessarily mean that such jackpot test runs are expensive to fix. There could be a single cause for all the failures. More analysis of the data is necessary to see what happens downstream of large numbers of test failures.

Perhaps the number of test failures is limited because people aren’t running many tests. In that case the percentage of test failures should be relatively constant. Here is the graph of test failure percentages:

The big spike at 100% failures is mostly (55000 out of 63000) caused by test runs with a single test that fails (common in test-driven development). Even the little blip at 50% is mostly caused by test runs with two tests, one of which fails. Excluding test runs that pass, the percentage of test failures is evenly distributed.

Conclusion

Snow avalanche magnitude follows a power law distribution. When planning for avalanches, the important question is, “What is the worst avalanche we’re likely to see in the next 100 years?” The average avalanche magnitude is not helpful in answering this question. Instead, you need to know the slope of the line in the log-log graph. Following the slope out to the end gives you a clue about the maximum magnitude. As reported in the referenced paper, the slope varies by location from 2 to 5. Data sets from each location could have the same average magnitude , but the likely maximum magnitude would be dramatically different.

Similarly, in speaking of test failure rates the average number of failed tests is not a helpful measure. Averages are misleading in power law distributed data because the outliers have a disproportionate impact on the result. To characterize the data and draw conclusions about the future you need to begin with the slope. This helps predict how frequent large numbers of failures will be and what is meant by “large”. Tracking changes to the slope over time could give you clues to the health of the system.

The original question remains unanswered so far. The number of failures is a crude approximation of the overall cost of keeping tests passing. The MTBF of tests and the time required to get a test passing once it has failed would be useful measures to get closer to a fact-based estimate of cost.

A potential limitation in this analysis is the small data set. The programmers at Intelliware have been practicing developer testing for many years. To be able to make broad conclusions, data sets from a variety of development contexts, programming languages, and problem domains need to be analyzed.

Appendix: Test Count versus Test Failures

Another way to look at the relationship of total test count in a run to the number of failures is to plot them independently. One pillar of the “tests are costly” argument seems to be that the cost (approximated here by the failure rate) of tests are constant–more tests, more cost. Here is a 3D visualization of tests per run versus failures per run:

Most of the time test runs succeed, or a few tests fail, regardless of the size of the test suite. The only exception is for runs with a single test, where 2/3rds of the time the sole test fails. This is characteristic of test-driven development (in my own practice I usually run more than one test at a time). In any case, more tests does not equal more failures.