The Learn/Measure/Build Cycle

Learn Measure Build Cycle

In preparing for Startup Lessons Learned next Friday I took a page full of notes during a conversation Eric Ries and I had about my keynote. I will wait for the conference to unleash the Big Message, but here is a smaller lesson that I found interesting.

Ordinarily we talk about the Build/Measure/Learn cycle (as in these slides from Ash Maurya, another SLL speaker). When Eric and I were talking, I realized that the loop is backwards.

From a programmer’s perspective, the loop is certainly correct. You write a program. You watch how someone uses the program. You learn something that you use to improve the program. I have had this experience many times. I have an idea for a program, I write it, I learn something.

Contrast that with the Lean principle of Pull: avoid the waste of overproduction by only creating based on demonstrated need. The customer orders a product, then you build it and deliver it. If that takes too long, make it take less long, don’t pile up a bunch of inventory in hopes that someone will buy it some day. Being able to instantly deliver a product no one wants is false economy.

I am working on two products right now that contrast nicely with respect to which way the loop runs. One product is a set of narrated screencasts illustrating test-driven development. For this product I’m running the loop forwards. I made a screencast, I saw how many people watched the unedited excerpt, I learned something about how many people are interested and how effective Twitter is as a marketing tool. Yes I learned something but now I’m at a bit of a loss. I’m not even sure what else I need to learn. I’ll make a couple more screencasts and then make the whole series available for sale. I hope they do well, but because I’m running the loop forwards I don’t expect to learn more about my audience until the casts are out there.

Contrast that with my other project, a poker-related game. Yes, I “primed the pump” with a very simple version of the game, but I’m now in a learning directed loop, running the cycle backwards. I start with an assumption, for example that I can make the game sticky. If the game is sticky, then people playing it will play several game per session and come back for multiple sessions. Now, what do I need to build to make it stickier?

The feeling is completely different between the two projects. With the screencasts I feel a bit lost. Yes, I know people are interested and I’ll sell quite a few units. However, I have no idea how to improve their value short of getting them out there and incorporating what people say into the next series of casts. It feels a bit like writing a book or the bad old days of writing a program for a year before trying to sell it.

Running the loop the other way, pulling the “build” from the needs of the learning, is much more energizing. I know why I am building when I build. I don’t create anything just grins. Programming’s still joyful, but it also has purpose. I know when to stop–when I’m ready to learn.

Here’s an exercise, gentle reader. Explain how to run the learning loop backwards for screencasts. One of the constraints is that people will only make one transaction per series, at least the way they are sold now.

16 Comments

Mike FinneyApril 15th, 2010 at 5:08 pm

Nice! Yes, I agree. Programming with a purpose is an energizing event that feeds itself. I had a similar experience when creating a simple screen with information that might interest our customers. The screen caught on aka “got sticky”. We got feedback in terms of useful information of various levels. The experience followed the loop backwards as you described it.

Ash mauryaApril 15th, 2010 at 5:19 pm

I’ve taken a much looser interpretation to the word “Build”.

From a programming project perspective, When I have an idea for a product, I don’t build the product first but instead test the underlying problem assumptions first with my supposed target audience by building a “problem presentation” (from 4STE). Then I’ll build a “product presentation” which will typically have a demo-able but not yet usable version of the product. It’s only after those 2 Build/measure/learn loops that I’ll build the product MVP.

From a non-programming project perspective like writing my book, I actually started building the blog. A book was never in my plans until a few readers suggested it. I then promptly built a landing page and tested interest in the idea as well as pricing, preferred formats, etc. I set a target date for this summer and while some of the content for the book will come from my current blog posts, a lot of the content is still being determined based on reactions and comments I receive on future posts.

In both instances, I think the key is building an audience of users/customers that provide ongoing feedback on how to make the yet-to-be materialized finished product better. In the case of the screencasts, is the content for the series pre-determined or can you engage your audience to help define that?

adminApril 15th, 2010 at 6:08 pm

Ash,

I’d love to hear what you learned about book pricing and format. I have quite a bit of Responsive Design material but a traditional book seems like a bad bet at the moment.

Regarding the screencast series, I have released the first unedited ten minutes of the first two episodes on Vimeo. I got some good general comments on it, but nothing that I would change direction about. I’m trying to tread the line between getting good feedback while at the same time not cannibalizing my eventual sales by giving away too much. That’s still not a line I’m comfortable with.

I agree with the liberal interpretation of “build”. If you can get a question answered with a Sharpie and an index card, do it.

Nigel ThorneApril 15th, 2010 at 8:17 pm

I’m still a little confused Kent.

At the point that you have decided what experiment you want to perform, don’t you then, Build it, measure your results (based on the metrics you defined), then learn from comparing the results to your expectations. Sounds more like a pendulum swinging back and forth.

OR

Make Assumption => Define Metric => Define Experiment => Build => Measure => Learn =>> Repeat

Chris NeumannApril 15th, 2010 at 8:21 pm

Hi Kent,

I’m a business guy who can barely figure out how code works when I look at it, so in some ways, I’m forced to run the loop in reverse since it’s relatively expensive to hire an engineer to build it for me, even though I’ve gotten good at hiring less expensive but very smart offshore talent.

As Ash suggests, what I do is come up with an idea, then write down a bunch of assumptions (it’s key to write them down, I’ve found), and then see if I can find my target customers, which helps me learn what the marketing channels look like. Based on the conversations (learning), I pivot into some new assumptions. Once I’m comfortable, I’ll build a fake-o prototype which includes a buy now page, and see what it will cost me to get people through the funnel. That gives me a good sense of viability of the idea. If needed, I’ll speak to people who tried to buy the product for additional feedback, and adjust further, then invest in actually building some sort of functional MVP.

For your screencast idea, can you register a domain you’re willing to throw away and then consider “selling” these screencasts, with the post-credit card page being some form of 404 error? If you’re not comfortable with that (I would imagine a lot of the marketing of this product rests on your reputation) then what about pre-selling them? Maybe you could offer a discount if they pre-pay, and even be explicit that you’re not going to offer the product and will refund their money unless you get X buyers. You might even end up with some word-of-mouth juice out of it!

Another product idea for you: how about some sort of packaged CI/CD engine? The rest of the stack seems to be plug and play, but doing TDD requires a bunch of deployment scripts which end up being pretty hard to do because there don’t seem to be tools out there to make it easy, or at least the devs that I’ve been working with push back and say it’s a lot of work.

adminApril 15th, 2010 at 8:29 pm

Your last line expresses the way I have experienced the process to work best. Building is pulled through the process based on the needs of the metrics which are based on the needs of the learning.

adminApril 15th, 2010 at 8:36 pm

Thanks for the ideas, Chris. I like the idea of pre-selling the screencasts. I’m still in the position where I can’t think of what else I can learn before releasing the whole series, though. I’m okay with the remaining risk, but I don’t want to turn down any learning that’s available.

I think you’re right that there is a big hole in the toolset for continuous integration as soon as you extend it to continuous deployment. I’m not enough of an expert on CD to build the product, but someone out there will, I’m sure.

Kyle MathewsApril 16th, 2010 at 7:42 am

My thought was break all the potential content of the screencasts into small atomic chunks. Then for each of those chunks, write a post for each and post as normal. Build a landing page that links to all of them and promote that page the most. Then measure which of the chunks get the most views, shared links, comments, etc. and that would be fairly good gauge I think of interest in potential screencasts. Not perfect as probably there’s a difference in how people read vs. how people watch screencasts but it’d be a pretty good indicator.

Nick BarendtApril 16th, 2010 at 8:34 am

I have not tried it, but Wistia (http://wistia.com/) offers a video sharing service with “video heatmaps” that helps isolate which portions of videos viewers concentrate on (by monitoring pause/rewind).

Whether “hot” portions of a screencast indicate confusion (“wait, what did he just say?”) or value (“wow, I need so hear that bit again!”), though, would seem highly variable.

One possible way to eke out some more user data from videos.

adminApril 16th, 2010 at 10:16 am

Nick,

Thanks for the suggestion. I hadn’t heard of Wistia before. That sounds like it’s worth an experiment.

adminApril 16th, 2010 at 10:18 am

I can see how that would work between series, but a topic like “Intro to TDD” seems atomic to me. Perhaps there’s another way of looking at the topics, though.

Nigel ThorneApril 19th, 2010 at 5:06 am

Comparing Ryan Bates’ free http://railscasts.com/ with http://peepcode.com/ or http://pragprog.com/screencasts may help.

Ryan does a great job of picking a new feature in rails, or a new gem that solves a problem and makes a quick (around 3 mins) hands on guided tutorial. The consistant stream of great quality screencasts leads to a dedicated readership that lends itself to advertising.

The peepcode guys take a much more in-depth look at a topic or tool (performance tuning, sinatra, jquery, rspec) and bring the expert advice. These take more like an hour. He charges $9 for each.

The pragmatic programmers have a set of topics and create episodes that are individually sold.

You could consider picking a few topics for your screencasts, then creating a first episode of each and see which sells the most, and use the feedback to decide which topics are worth extending with another episode. I know Ryan gets lots of ‘how to’ questions that he uses to decide on topics for his screencasts.

I hope this helps.

adminApril 19th, 2010 at 5:11 am

Nigel,

Thank you for the suggestion. A lot of my tweets could use a little screencast explanation. Give away the 2-3 minute ones, advertise the 30 minute ones relentlessly on them. I should prototype this.

As far as doing one paid screencast to see whether the series would sell, people will generally make one buying decision per series. If there’s one cast, they’ll buy one and they won’t come back to buy the rest. That’s the observed customer behavior. One strategy is to keep the series short (like 4 episodes) so you can do relatively a lot of series, even if it’s one fourth as many. Another would be to sell subscriptions so people receive the first episode and they get charged as new episodes are delivered. You’d need to verify that people will actually do this, though.

Nigel ThorneApril 19th, 2010 at 5:14 am

Also.. this seems to relate nicely to your article. http://thisisindexed.com/2010/04/how-to-get-unstuck/

[...] [...]

[...] The Learn/Measure/Build Cycle [...]