CD Survey: How often do you run functional tests?

Following up on the continuous deployment survey question about frequency of deployment, the question for today is, “How often do you run functional tests?” Here is the data:

What I take from this is that there are a considerable number (~30%) of people using continuous integration in the poll audience. The ~10% of people who have brought functional testing into the inner loop of coding is also a bit surprising, although we do it to great effect while developing JUnit. Another surprise to me is seeing that the majority of these projects get feedback from functional tests every day.

All the panelists (Jez Humble, Timothy Fitz and I) agreed on how essential automated functional testing is to applying continuous deployment. I think of it as a stack of filters, each removing defects–personal habits, social processes (like pair programming or reviews), unit tests, and functional tests. If the functionality flowing from the whole stack is high enough quality, then continuous deployment is possible.

Manual testing has a place in projects applying continuous deployment, just not on the critical path to deployment. It might seem like a big step to remove that last human “sanity check” before deployment. If it seems like a big step, then you’re likely not ready. Keep working on the filters to improve the deployability (is that a word?) of each one’s output. When you heard someone say, “Why are we checking this by hand? We never find anything,” then you’re ready.

Next question: how often do folks run their unit tests?

Commercial plugs: Check out my new series of screencasts on intermediate test-driven development. If you run unit tests for Java in Eclipse, check out JUnit Max, the continuous testing plugin.