Some thoughts:

• Business-based operations metrics. One of the key insights of continuous deployment is using business-oriented metrics to monitor operations instead of the more natural (for programmers, anyway) technical metrics. If you expect 50 sign-ups per minute and the rate suddenly dips to 20/minute after a deployment, it’s time to roll back. The practice is not in common use.
• Kanban versus iterations. Iterations still dominate, even though the additional flexibility of kanban is a better match for continuous deployment.
• Pair programming. For all the complaints I hear about pair programming, I would have expected this number to be lower than 25%.
• Test-driven development. 50% is higher than I would have expected. Adoption of TDD is excellent preparation for teams wishing to deploy more frequently (see my commercial screencasts for more details).
• Continuous integration. I expected this number to be higher. CI was the first practice from Extreme Programming to spread widely, but, at least among this audience, it is not pervasive.
• More than 75% of teams test manually before deployment. This is a sensible practice until the defect rate is brought down and the operations infrastructure made robust in the face of errors, but I expect the number to drop as teams mature in their application of continuous deployment.

Change generally happens on a time scale of decades. Mass production and then lean production each took upwards of fifty years to become widespread. I don’t mean to be overconfident, but the picture above (skewed as it is by selection bias) paints a picture of software development that is substantially different than common practice twenty or even ten years ago. There’s still a long way to go until software development pours out the stream of value it is capable of, but we’re making progress.

Commercial plugs: Check out my series of screencasts on intermediate-level test-driven development, $25 for four episodes. If you run unit tests for Java in Eclipse, check out JUnit Max, the continuous testing plugin, $50/year until August 1, 2010.

One of the essential practices of continuous deployment is that everyone stays very close to the mainline, typically by checking progress into HEAD frequently. This requires developers to be careful to only make changes that are safe and to only reveal functionality when it is ready for real users. The change of task sequence this requires is an acquired skill, but once mastered results in a low-risk development style.

The data shows that many projects are checking in very frequently:

The question needs followup, as it doesn’t ask whether there is a single mainline that people continually integrate with, and which direction the integration goes.

Tomorrow is the final question of the survey, which asks which practices folks are applying.

Commercial plugs: Check out my new series of screencasts on intermediate test-driven development. If you run unit tests for Java in Eclipse, check out JUnit Max, the continuous testing plugin.

Props to Chartpart.com for the lovely front end for Google Charts.

I was a bit surprised at how many people are running tests on every change, but that’s good news for JUnit Max, which shaves seconds off the delay waiting for feedback from tests. If those delays happen a hundred times a day, the cumulative effect of a tool like Max can be substantial. I’m not sure what to make of the teams that run unit tests daily or weekly. I’d like to talk to such a team to understand more about their process.

Unit testing wasn’t explicitly addressed by any of the presenters at the Continuous Deployment webinar, I think because unit testing is assumed when you’re deploying on a short cycle. I don’t recall who said it first, but the model I use is that the defects leaving any stage are proportional to the defects entering that stage. No matter how good your functional tests are, defects leaving the programmer’s desk need to be rare if the software is always going to be ready to deploy.

What I take from this is that there are a considerable number (~30%) of people using continuous integration in the poll audience. The ~10% of people who have brought functional testing into the inner loop of coding is also a bit surprising, although we do it to great effect while developing JUnit. Another surprise to me is seeing that the majority of these projects get feedback from functional tests every day.

All the panelists (Jez Humble, Timothy Fitz and I) agreed on how essential automated functional testing is to applying continuous deployment. I think of it as a stack of filters, each removing defects–personal habits, social processes (like pair programming or reviews), unit tests, and functional tests. If the functionality flowing from the whole stack is high enough quality, then continuous deployment is possible.

Manual testing has a place in projects applying continuous deployment, just not on the critical path to deployment. It might seem like a big step to remove that last human “sanity check” before deployment. If it seems like a big step, then you’re likely not ready. Keep working on the filters to improve the deployability (is that a word?) of each one’s output. When you heard someone say, “Why are we checking this by hand? We never find anything,” then you’re ready.

Next question: how often do folks run their unit tests?

Commercial plugs: Check out my new series of screencasts on intermediate test-driven development. If you run unit tests for Java in Eclipse, check out JUnit Max, the continuous testing plugin.

My co-presenters were Timothy Fitz of IMVU and Jez Humble of ThoughtWorks. Timothy talked about continuous deployment in a large web/client application with ~50 developers while Jez covered CD for IT applications. Both made a nice contrast to my experiences with CD for individual or pair projects, and immediately addressed the “sure, but it won’t scale” objection. My thanks to Timothy, Jez, and Alan Zeichick of SD Times for their work setting up and delivering the webinar. (I’ll post here when a recording is available.)

For the poll, the questions were posted two days before the webinar. We had ~200 responses out of the 1100 registrants and 430 attendees. Beware of extrapolating too far from the data because the poll suffers from both selection bias and an amateur question author (me). However, the responses do provide some idea of how at least a slice of our industry works.

The first question I will cover from the poll is, “How often do you deploy to production?” That’s the fundamental measure of continuous deployment, at least as narrowly viewed (more about that later). Here’s the raw data:

Deployment cycle starts with the orange slice for annual deployment at the bottom right. Going clockwise from there, the three biggest slices (quarterly, monthly, weekly) take up most of the chart. Completing the circle, there are almost as many people deploying on every commit as deploying annually, so continuous deployment already has significant traction.

One of the surprises of the webinar was Jez Humble’s flexibility with regards to how far towards real deployment Continuous Delivery (his phrase and the title of his forthcoming book) needs to go. His strategy is to take the rapid iteration that is made possible in development by Extreme Programming and gradually extend it down the deployment pipeline, even if that stops short of real actual live deployment with real actual live users. My general preference is to work the other way around. I like to get the whole team–business, development, and operations–working with the same rhythm, then gradually increase that rhythm. There’s no right or wrong approach, it’s just interesting to see the contrast.

Next question: how often do you run functional tests?

Commercial plugs: Check out my new series of screencasts on intermediate test-driven development. If you run unit tests for Java in Eclipse, check out JUnit Max, the continuous testing plugin.

XIX Olympiad

I was seven in 1968 when the Mexico City Olympics rolled around. They were the first sporting event I devoted myself to completely. I had been happy earlier that year when the first Super Bowl was played, proving once and for all the superiority of what is now the NFC, but with the Olympics I fell in love.

I bought the whole Avery Brundage/Chariots of Fire line: amateurs competing only for the glory of sport and the honor of their nation. I was happy to US athletes do well, but I was happier to see an athlete surpass his own limitations. Bob Beamon’s jump was remarkable not because he was American, but because it was such a remarkable performance. It was possible for people to become gods, if only for a moment.

Tommie Smith & John Carlos brought a discordant note to my proceedings with their medal stand protest. My father was outraged but I understood that they had a legitimate complaint and no better way to speak of it. Still, sport was supposed to be about sport, right?

By the closing ceremony/party my discomfort was washed away in a music-powered dancing unification of all humanity. Sport was the highest and best in us.

1968 was a confusing time to be seven: body counts on the nightly news, “duck and cover” drills at school (and we lived on the Moffett Field final approach, so this was an immediate, if futile, concern), hippies, and anti-war protests. At seven I had the intellectual capacity to understand a surprising amount of this, but no emotional tools with which to deal with it personally. Don’t worry, in sport I had a perfect refuge.

XX Olympiad

Four years later I couldn’t wait for Munich to start. At first everything was as I hoped it would be. There was even a rare note of international harmony in the house because my rabidly anti-communist father, then 140 kg, idolized Alexiev. There was Dave Wottle’s incredible kick–zero to hero on the final straight. Bruce Jenner, local boy making good on the biggest stage in the world.

Then came the reports of the massacre. All day we sat glued to the TV, stunned. I was eleven at the time, so my emotions were a violent tempest in an extremely small tea pot. Bitterly furious at the Palestinians, terrified for the hostages, utterly baffled at why someone would do this to sport, to me, blindly hopeful when the choppers took off, inconsolable when it became clear that everyone was dead.

The lasting emotion was rage, rage that someone would violently pervert sport to suit their own needs. In sport I had a refuge where the world made sense and that had been ripped from my hands and smashed on the floor. I was ready to drink blood.

Psychologists tell us that when a trauma happens, people often aren’t prepared to deal with the intensity of emotions in the moment, so they bottle it up. When this happens, emotional development pauses. The theory is that by working through those stored emotions later in a safe environment, emotional development can continue. I fell like I’ve experienced a bit of this effect thanks to the book Vengeance.

Vengeance

Vengeance tells the story of an Israeli hit squad put together to assassinate eleven high-ranking Palestinians involved in terrorism, one for each dead Israeli athlete. By the end of the book this attempt to balance the books has wrought a terrible vengence of its own on the surviving members of the team. In the end, the author/protaganist is estranged even from his own country.

Debate raged when the book came out: fact or fiction? good or evil? I won’t address any of these questions here. I also don’t intend to debate the ethics of the actions described or the preceding events. What I’d like to highlight here is what the book did for me.

While I was reading Vengeance, that furious little eleven-year-old that I’d put in a box was able to come out and read along. I got my wish–kill those responsible with my own two (borrowed) hands. What happens then? The human cost to the protaganists is horrifying, along with the effects on those close to the murdered. Yes, vengeance is physically possible, but at the end what has been achieved? I felt those long-ago emotions again and was able to let them go.

In the months since reading Vengeance, I’ve noticed that I am able to read about and think about Middle Eastern politics without the same surges of rage that I used to experience. I can see that both sides have unmet needs. I can see that many of their actions will not lead to those needs being met. At the same time I can have compassion for those making those decisions. If I was filled with rage for 38 years for something that happened five thousand miles away, how much more difficult must it be to move past personal experiences of terror, grief, and rage?

I’m sadder and wiser (I hope) in my view of sports now. Sport can’t solve problems, but, at its best, it can remind us that we can solve our problems, we can overcome. Shalom. Salam.

This piece one entitled “One Team”, about the evolution of my thinking about software development teams. It was initially published in July of 2001. It was interesting to me to re-read it in light of Steve Blank’s Customer Development and the Problem Team in Lean Startups.

Stupid Mistake

The biggest error in Extreme Programming Explained is the implicit assumption that you have a technical team serving a single customer.

The lonely customer orbits a team of programmers

I had this picture in my mind because of two experiences. At a mutual fund company I had a customer who had been a customer service representative for seven years, a supervisor and trainer for three years, and then threatened to quit if the crummy old system wasn’t replaced. At a large manufacturing company I had a customer who had run a finance department for years, knew all the ins, outs and exceptions, and was willing to work very hard to keep up with the programmers.

The picture of a single person feeding expectations to a whole team of programmers is seductive. You don’t have to worry so much about disagreement in detail, and political disagreements are even rarer. You don’t have to worry about finding the appropriate expert to answer a question.

Don’t get me wrong. If you can find that one person who knows the whole domain, is willing to make quick decisions and fix them later, can speak to you both concretely and abstractly, and can put up with a room full of nerds you are likely to be successful. First, though, such folks are understandably rare. Second, this picture seems to exclude many people who now dedicate their talents to software development, like analysts, testers, and modelers.

It Gets Worse

The problem is compounded in XP Explained (and subsequent writings and conversations) by referring to the programmers when using the phrase “the team,” as in “the customer speaks to the team with one voice.” This usage always bothered me, but I didn’t know what to do about it.

Cracks Appear

The first inkling I had that “the team” shouldn’t mean just programmers came out of my experiences at Evant Solutions (evant.net). We (I’m a sometime coach and stock holder) have about ten programmers on (what we’ll call for nearly the last time) the team. But the product, a demand chain management tool for retailers, requires specialized knowledge of many different aspects of retailing. No one customer has sufficient knowledge or perspective to guide the programmers in implementing the whole system.

“The customer” consists of six product managers. One (I’ll call him “Jim” because that’s his name) is the uber-product manager, overseeing the competitive vision of the entire product. Working with him are five product managers specializing in aspects of retailing: assortment planning, purchase orders, inventory, and some other stuff I don’t understand even that well (contact Evant for useful details).

Evant chose a three week iteration, curiously beginning on Tuesday (Jim says, “Planning sucks and Monday sucks, why put them together?”) On Tuesday, the product managers present their stories to the programmers for tasking and estimation. Sometimes one product manager dominates the iteration, sometimes the stories are spread around, and sometimes they are shared. Curious. Even curiouser is the frequent appearance of fresh bruises and bandages on the product managers at the iteration planning meeting.

A Team Is Formed

What gives? Jim says,

The short of it is that we hold a pre-planning meeting before each cycle to discuss which use cases we want to attack. Usually, we have a theme/focus to each cycle that is provided by our clients (i.e., incorporating feedback from XYZ Inc. regarding planning) that makes the discussion around which use cases to select somewhat “non-toxic.” Although our process is very collaborative and democratic, my “uber” nature does make it easy to “break the tie” when we have competing requests.¹

Regardless of Monday’s disagreements, Tuesday morning the customer team presents their story choices to the development team with one voice. It is as if they agree, at least as far as the programmers can tell.

Stranger in a Strange Land

The second crack in my model of the lone customer orbiting “the team” came during a visit to Japan in April 2001. The Extreme Hour is an exercise invented by Peter Merel to simulate the dialog used to choose scope in an XP project. A group of eight people is split into four developers who draw pictures of a coffee maker, two customers who write stories about a coffee maker, and two testers who verify that the coffeemaker as drawn satisfies the stories.

For ten minutes the customers write stories and the developers estimate how long they will take to draw. Then comes the interesting part. In five minutes the customers have to choose which stories will go into the first release of the coffee maker based on the estimates on the stories and the developers’ estimate of their overall speed.

I say interesting, because the choosing phase always takes too long, at least a minute or two. The customers disagree on the relative priorities of the stories, they argue with the developers over estimates and speed, and they are reluctant to make a final decision until absolutely compelled to do so.

I gave a private XP lecture to a group of 20 programmers from Technologic Arts, the sponsor of my trip. The first phase of the exercise went as expected—they wrote a bunch of stories. The second phase was interesting.

Instead of six or seven minutes to pick the stories for the initial release, each of the teams was done in two minutes. Strange, but after all these were bright folks who had been studying XP for a while.

The next day I gave an all-day lecture to 200 programmers and project managers. My hosts (hi Marika and Yoshi!) insisted we could run the Extreme Hour with the whole group. I was unconvinced, but it was their show, so what the heck.

The attendees were sitting four to a table, four tables across. The easiest way to form the teams was to have every other row turn around. The backward-facing attendees were the customers and the forward-facing attendees were the developers (we skipped the testers for simplicity).

The first ten minutes went better than I expected. We had them work on a tea maker (although my translators told me later they would have preferred coffee since there are two words for tea and they didn’t know which one I meant). Every team wrote plenty of stories (with a little prompting). We were ready for the first release plan.

In a giant room, with 200 people who had never tried the exercise before, giving instructions through simultaneous translators (who were excellent, but still), the entire room, all 25 teams, had their release plan finished in, you guessed it, two minutes.

Westerners hearing this story often comment, “Sure. Japan is a hierarchical society. The senior customer decided and the others followed.” Au contraire, my ethnocentric friend. In the conversations I watched, every customer talked about the same amount. Body language was similar in the whole customer team. Without knowing Japanese I can’t tell for sure, but I’m guessing the four customers simply formed a team faster than I could recognize and got on with working together. Yoshihide Nagase comments,

Japan has many hierarchical styles. Software development in Japan is hierarchical, too. But manufacturing development (such as in Toyota, Sony, Epson and so on) uses small teams. Maybe attendees of your XP seminar here felt like these small teams, not the usual Japanese Software Engineering way.

The Last Straw

My identification of “the team” with “the programmers” finally died when I had the chance to visit a testing conference and a modeling conference in quick succession. In both places the primary question was, “I’d like to try this. Where do I fit into the XP picture?” As long as the XP picture contains a lone customer orbiting a team of programmers, the answer is “nowhere.” I know that many of the skills and attitudes of modelers and testers would be a huge asset on my teams. If they are right, then my XP picture must be wrong.

One Team

What if we redrew the picture to reflect what happens at Evant, and what I observed in Japan?

The new picture reminds me of American football teams. The offensive and defensive teams have different skills, values, and perspectives but they share a single goal, winning the game.

Where do our good testers and modelers fit in? On the Business team. Helping scout, write, slice, dice, and verify are everyone whose lives will be affected by the scope and quality of the released software:

• Analysts
• Representative users
• Testers
• Marketers
• Customer support
• Sales
• Modelers
• Interface designers that watch what real users do and help “pave the paths”
• Business strategists

Their jobs are different than they have been in the past. No longer can any one of these groups grab a phase to themselves, either before development starts or after it finishes.

Looked at from Here

Once I shifted to viewing one team, some problems I had been wrestling with for months came into focus. Martin Fowler reported a team at Thoughtworks where the programmers brought their workload and stress level under control with the Planning Game, but only at the expense of overloading the analysts. Analysts need liberation, too. ThoughtWorks has since been using teams of analysts and QA in its larger XP projects for a year or so now with reasonable success http://martinfowler.com/articles/planningXpIteration.html.

Another problem that came into focus for me was the growth of the customer team on the C3 project, XP’s spawning ground. At first there was only one customer. After a few months another payroll expert joined the team to help prepare test data. A few months later a third payroll expert joined. The One Team picture explains this. As the business sponsors discover how responsive the development team could be they chose to increase their business investment.

Far from being the norm, perhaps the original “one customer/many programmer” picture is weird. A picture with the teams more nearly balanced in size may be more healthy.

But Wait, There’s More

Our happy little family would be complete, if only we didn’t want to get paid. And if we always got along. And probably eleven other impossible things before breakfast.

I spoke at the University of Oregon CS commencement this year. One of my old professors, Andrzej Proskurowksi, heard the XP story and likened the picture above to a two-legged stool. And he’s absolutely right. There are questions that are outside of the scope of the people on the team as we’ve drawn it so far.

• How does the project get started?
• How is investment increased, reduced, or terminated?
• How are disagreements resolved that aren’t handled by business and development?
• How are relative priorities set between this team and all the other projects that need doing?

I would love to have a better name for the team that answers these questions. The only one I can think of threatens to unstopper the bottle into which we have just so laboriously stuffed the Demons Control, Hierarchy, and Certainty. But, my failings as a thinker being what they are, I’ll just have to call this, for the moment, the Management Team.

But I really mean “team”, not “manager” and certainly not “roomful of hens squabbling over too few grains of corn.” To scale XP practices to the next size of organization (up to about 150), the members of the management team must work together to solve problems at the scale of the whole organization, typically by breaking them into bite-sized chunks that can be chewed and swallowed by individual teams.

Being on the management team is no easy ride. XP Management is not there to Make Decisions, and certainly not there to Fix Things. Managers are there primarily to notice when the words and the deeds don’t match, to market and sell the achievements of the teams, and to make sure there aren’t obviously better ways to spend money.

In redrawing the picture, I’m forced to switch metaphors. If I draw three circles, whichever one ends up on top will think it’s in charge. If two are on top, whichever one ends up on the bottom will think it’s being oppressed. Circles just won’t cut it. However, Andrzej’s suggestion has merit:

I like this picture. First, the three legs are equal partners. Without the support of any one team, the stool collapses. Second, it’s an Aalto design, and I’m a big Aalto fan. Third, imagining the stool in use, the three sub-teams have to work together to withstand the pressure from the …, well, from above.

Making the Shift

I’ve been using “the team” to mean the whole team for a week or so now. Old habits die hard, but it’s starting to feel natural. If you want to try it, talk about “the team” to mean customers, programmers, and managers. Talk about the customer team, the development or programming team, and those clueless suits (just kidding, talk about the management team) when you need to discuss narrower concerns. Give it a week and see how it goes.

Some Predictions

Where will it end? I’m reminded of engineering driven startups, where engineers often form 80-90% of the employees. As these firms grow and mature, the proportion of engineers drops to something like 10-15%.

Will we see this as XP teams mature? Will we ever see a customer team of 10 marketers, domain experts, representative users, sales people, analysts, testers, and usability engineers feed stories and acceptance tests to 4 programmers with 3 managers? What would happen if there was such a team?

Roles

The division of the One Team into three sub-teams smells of taylorist division of labor. I can imagine someone reading this and saying, “Oh, I have to decide once and for all whether I’m a manager or a customer.” In practice, people move between the teams, sometimes even iteration-to-iteration. And once the team is clicking, the artificially hard divisions blur, and rightly so.

However, the social contract of work is different between the members of One XP Team. Before you can begin messing with the roles you have to get used to how they are different. Then you can begin experimenting while working together towards the ultimate goal: getting as much of the most valuable stuff done as possible by a given date.

Sidebar: Customer Team at Thoughtworks

Martin Fowler writes:

At ThoughtWorks the notion of “one team” makes a lot of sense. We often staff projects with our own analysts, who are often business experts. Indeed in some business sectors our analysts are more familiar with the specialized domain than our clients. QA is also obviously an agent of the customer, as indeed it was in the early days at C3.

Our largest XP-ish project has a full team of analysts and a full QA team—all as part of the overall project team. The analysts need to collaborate to present the stories with one voice to the programmers. They’re also a mixed bunch combining both business experts with pure analysts.

They’ve recently set out some of their conclusions about making the customer team work.

• They’ve missed having some kind of “big picture” to act as a road map
• They’ve found it hard to divide up functionality into suitable story-sized chunks
• They’ve found it frustrating to not give an indication to developers of which direction future function is likely to go
• They’ve found that specialized analysts are better at looking at the thorny details of exceptional cases – in other projects testers have reported a similar phenomenon.

The danger with an analyst group is always that they can end up being a communication barrier rather than a communication enabler. The notion of the single business expert customer helps prevent that, but I don’t think there’s any way to structurally avoid a barrier. The key lies in the attitude of the people involved. Particularly on the customer side of the team you have to build the roles around the people – not the other way around.

Certainly the programmer-centric attitude of XP has caused frustration. Part of XP’s guiding mission is to heal the rift between business and technology. For this to work we need to better understand the various aspects of the customer team. In many ways it’s a more complex team than the developer team. It’s also one where the techniques are less developed.

Here’s what happened this time. I read Michael S. Malone’s piece berating the tech press for covering Apple softly. Until I read this, I wasn’t too upset about iPhoneGate, but Mike’s article framed the situation differently for me: as a free speech issue.

Since my point seemed to be widely misunderstood on Twitter, I will say right up front that I am not commenting on the behavior of Gizmodo or the gentleman who found the phone and subsequently sold it. I would like to think that in the same situation I would return the phone to Apple, although I would certainly be tempted to post some pictures just because I think Apple’s paranoia deserves tweaking whenever possible. Not returning the phone to its (obviously) rightful owner was wrong, but it’s an everyday kind of wrong and it doesn’t affect my life so I’m not going to get too upset about it.

What bothers me is Apple’s reaction. Without knowing the details of the activities that went on, from up here in southern Oregon it looks like Apple enlisted the police to make an example of Gizmodo. If I had lost a hardware prototype and I called the San Mateo police or the REACT group or the Superior Court judge, there’s no way they would have reacted in the way they did. Apple got special treatment because they are large. That’s wrong.

I’ve learned to look for the intent behind behavior that I don’t understand. What could Apple’s intent have been in having Mr. Chen’s house searched in the manner it was? The explanation that makes sense to me is that they wanted to intimidate anyone who was thinking about writing about Apple “secrets”. The course of justice doesn’t demand what they caused to happen. They were looking for effect, is how I read it.

Here’s where it starts to affect my life–what if I figure out something about Apple next “big thing”, purely legitimately? Am I going to write about it? Well, if it’s me I probably will because 1) I’m clueless about what’s going to upset people (obviously) and 2) I hate people telling me what to do. I’d certainly think about it, though. What if Apple took umbrage at what I wrote? Josephine County sheriffs would be battering down my door in no time, and there’s nothing I could do about it. I don’t like that feeling.

My conclusion is that Apple has read too many of its own press releases and thinks the world should run for its benefit. I expect more boorish behavior and increasing alienation from the needs of customers and developers. That’s too bad, because I like my Apple products.

I’ve spent maybe a quarter of my career doing user interface design and implementation. One skill I was proud of was my ability to take in a variety of constraints on the UI and produce a design to satisfy them. It takes a while, though, and if I’m going to be honest with myself the success rate isn’t that great. That skill, the single blinding flash of design insight, I discovered yesterday, is no longer essential. Iteration is the new inspiration.

One of the fundamental changes driving lean startups is the incredible reduction in the capital required to deliver the first iterations of a product. Capital is still required to fuel growth, but finding out if you have a market and pivoting when you discover you don’t, is one or two orders of magnitude cheaper than it used to be.

I had previously experienced this efficiency in programming (open source tools) and deployment (AppEngine). Yesterday I got a taste of it in user interface design. Back in the day, developers (those with design and/or programming skills) would argue endlessly about designs. Along came usability labs, with mirrored walls, video taping, and real users (gasp) invited in. Watching someone fumble with your perfectly logical UI design was always an education, but it was expensive, both in terms of capital and time.

Enter Speed

Yesterday I started using two services, UserTesting.com and fivesecondtest.com, that bring the same capital- and time-efficiency to UI design that open source tools brought to programming. Rather than work extremely hard getting the design right, then failing if you guess wrong, it is now quick and cheap to iterate. Have a crazy idea? Test it. Three alternatives? Test them.

At the Startup Lessons Learned conference Siqi Chen of Zynga recommended UserTesting.com. UserTesting.com is the online version of the usability labs of old. For $39 you get a 15 minute screen cast of a real user using your system while thinking aloud. Problems in flow are glaringly apparent. ($39 is approximately the cost of a ten minute argument between two designers, and in my experience arguments can last a lot longer than ten minutes.)

Granted, you don’t get everything out of UserTesting.com that you get out of a usability lab, but it’s quick and it’s cheap. I can see iterating a design several times a day with UserTesting.com. By the time you’d rounded up a subject for a usability lab session you could already have the results from UserTesting.com.

Fivesecondtest.com becomes useful earlier in the design cycle. Given a screenshot, users are asked what they remember about it or they are asked to click on notable sections of the screen. You can pay to have customized tests, asking particular questions, for example. Since it works on screen shots, you can iterate much faster with fivesecondtest than you can with UserTesting. It’s a tradeoff between fidelity and speed. For a quick, “is this too cluttered?” kind of question, fivesecondtest is ideal.

You Still Have to Design

Neither of these services substitute for building real, person-to-person, in-depth relationships with real customers. And you still have to have the ideas to test in the first place. Rapid iteration with lousy ideas will result in a lousy interface. However, user interface inspiration just got rollerskates.

You could use these services to produce a similar quality of design compared to what you are producing now, but in less time and at less cost. I’m more interested in spending a little more time and money than that, and trying out crazy UI ideas. Crazy ideas that just happen to work provide pizzaz, differentiation, and extend the lead you have on competitors.

Note to Programmers

If you are working with a designer who begins using these services, you are going to have to up your game. You need to find ways to provide testable variations of your application much faster than you do today. That will require better designs to effectively separate the parts of the system that change from the parts that don’t, better coordination and communication to keep track of the variations, and better tools to support the whole process. You will also need to be disciplined about removing the traces of failed experiments, lest your code begin to resemble a Hollywood backlot.

Get the whole team in a conference room Monday morning. Choose a feature you are going to add. Have one person come up to the one machine in the room. Write one line of code. Deploy.

If you can’t deploy, erase that line of code and write a different line of code first. Deploy.

Once the deployment is done, write another line of code. Deploy.

About this time some of the rest of the team is going to be bored watching all the manual steps in the deployment. Have them go off and automate the most tedious, boring, or error-prone step. Bring the automation back to the conference room and start using it.

By about Wednesday you’ll be feeling pretty good about your ability to deploy (even if you aren’t feeling so good about progress working one line at a time). You’ll likely make a mistake. Roll back manually. Split off a team to be able to roll back automatically.

At some point you’re likely to need to change some existing code. If you don’t have automated tests for the current behavior, split off a team to write a test before changing the code.

By Friday afternoon you may not have made much progress on your feature (but you may be pleasantly surprised), but you will understand deployment much better than you do now. You will also understand how to slice development more finely and rearrange the pieces. You will also understand the value of automation.

But don’t do this. It’s a crazy idea. Never work. Forget it. Sorry to waste your time.