The Economic Case for JUnit Max

In preparing to relaunch JUnit Max, I’ve been try to articulate exactly why it is worth the price. I’m still conflicted about charging for Max, although realistically if I had no chance to be paid for it I couldn’t afford to work on it. If I’m going to charge money, though, I’d like to know that Max is worth it.

For me this isn’t an issue. I run tests all the time and I appreciate the time savings and additional focus Max gives me. Of course, I had to give up the things that the time I spent implementing Max could have brought, but that’s a sunk cost for me. I simply like programming more when I have Max, which is a big part of my motivation in the re-launch.

If you’re out there with $100 in your pocket, though, I can imagine that you need convincing. If you’re going to ask someone else for the $100, then they need convincing. So, does Max make economic sense?

Here’s an envelope calculation based entirely on time savings. I’ll take the JUnit unit tests as a baseline. They take about 10 seconds to run. While we’re programming we run them ~50 times per hour. Scaling this up to a full-time job, we would be spending 10 seconds/run * 50 runs/hour * 1000 programming hours/year = 139 hours/year waiting for tests to finish. At $200/hour, your employer is paying $27,000 for you to wait for test results.

With Max, wait times are reduced because of test prioritization. With the JUnit test suite we get results in 2 seconds instead of 10. Even if the savings were only 50%, though, Max would still be worth $13,000/year (and the savings on longer-running tests suites will be larger than the 80% we get for the JUnit tests). In other words, Max pays for itself roughly every two working days.

And that doesn’t count all the times I stay focused on programming because it’s only two seconds instead of getting distracted during a 10-second or 30-second pause and not getting anything done for several minutes. I can only conclude that at $100/year Max is seriously underpriced. My conscience is assuaged.

29 Comments

Martin DSeptember 13th, 2010 at 2:57 pm

Fifty times an hour?! I must really be doing this testing thing wrong…
It certainly looks like an interesting product though.

Mario FalcoSeptember 13th, 2010 at 3:00 pm

I know, Mr. Beck, that very few software engineers in the world may even dream of being as good (and as valuable) as you are but…
200$/hour?
That’s what I get paid A DAY!
And I’m not a junior either…
OMG, is Italy so deep in shit that companies here can only afford to pay IT professional one eigth of the US wage? :-|

Ramon LeonSeptember 13th, 2010 at 4:01 pm

> While we’re programming we run them ~50 times per hour.

That’s a huge flaw in your reasoning, most programmers aren’t you. I think you’d be lucky to find that most programmers who use unit test tools still only run tests once every few minutes at most. Most just aren’t *that* test infected. I’d say more people use them to protect against regressions than they do as a test first design tool.

adminSeptember 13th, 2010 at 4:12 pm

What would you say the numbers are, then?

adminSeptember 13th, 2010 at 4:13 pm

Pay scales are different in different parts of the world, certainly. As long as you can live comfortably…

Even if the payoff time is 20 days, though, Max still looks like a bargain to me.

adminSeptember 13th, 2010 at 4:15 pm

You can see my TDD videos (the free version is at http://vimeo.com/10789674 ) for examples of how I use tests.

Mate GulyasSeptember 13th, 2010 at 9:37 pm

I second the fact that few developer run tests that often. I run tests like 10 times a day. Also, they are definietly not last 10 seconds. Two maybe. I guess Max makes economic sense in large applications.

The vimeo link you provided gives a 404. Can you repost, please?

adminSeptember 13th, 2010 at 10:20 pm

I’d like to have more data on the distributions of test runs/day and test duration/run. I wouldn’t be surprised to be an outlier, at least at the moment. Part of what I’m trying to accomplish with Max is demonstrate that tests that are run more often are more valuable, leading to more tests, more value, etc.

I’ve fixed the Vimeo link.

SteveSeptember 14th, 2010 at 8:32 am

I think the basic flaw in this assessment is forgetting that “You are not your user”.

Start by understanding how regular developers work (even running tests 10 times a day seem like a lot based on my experience) and then build a product for them instead of for you.

adminSeptember 14th, 2010 at 2:25 pm

Steve,

Thanks for the feedback. I don’t mind a product that leads practice by a bit. Too far and I won’t have a business, but the built-in tool does a just fine job of supporting 10 test runs a day, so I won’t have a business supporting those programmers either. It’s an interesting tradeoff.

Kent

stevenSeptember 14th, 2010 at 2:40 pm

@Mario,
I don’t know if that’s for real. I’m in Australia and our average contract amount is $60p/h. And thats just contract (no benefits at all).
Then again, I’ve heard Australia’s I.T. pay is quite low.

ArneSeptember 14th, 2010 at 11:12 pm

I think that the main problem in marketing JUnit Max is, that developers expect development tools to be free. They can use eclipse for no money, and get an enourmous featureset for free. Even if JUnit Max make economic sense, it looks rather expensive. I fear most people are not rational when dealing with money and that makes selling of development tools … tricky.

A second, minor, problem is, that there is no test version and nearly no documentation on the website. How should I as a potential customer decide if it is worth the price?

Personally I have testet a competing product in the past. I was happy with it (was free for that time) until I get to one nasty bug it has (an endless loop in my code let eclipse stale). For this reason I can’t use it, because the time it saves I loose every time eclipse hangs. Without a test version I can not verify if there are such blockers in the product …

Sebastian KübeckSeptember 14th, 2010 at 11:26 pm

Congratulations Kent!
Looking forward to try it out!

In case you don’t know it yet, you might like Eric Ries’ experiences startups.
Reflects many of my own experiences…

http://ecorner.stanford.edu/authorMaterialInfo.html?mid=2329

Mario FalcoSeptember 15th, 2010 at 6:07 am

I run tests very often when I’m developing new stuff; I run them more rarely when I’m refactoring; I admit I (almost) gave up on them for bugfixing and legacy code, expecially with C++ which is a mighty PITA. Anyway I think that JUnitMax might be a wonderful assistant.

Of course my company should buy JUnitMax, but no one here is nearly as infected as me…

@Steve
Pay here is considerably lower, as I said, but I have (some sort of) benefits: I have a long term contract and work really near home in a zone where the cost of living is (relatively) low, so I can’t complain really… unless that leads Mr. Beck in giving me some kind of discount, of course ;-)

Ramon LeonSeptember 15th, 2010 at 6:14 pm

Making money selling development tools is hard, programmers aren’t a large market. Making money selling tools to a very tiny portion of the programmer market is even harder, there simply aren’t that many TDD or BDD programmers compared to the size of the pool of programmers. And with free test frameworks available, it’s probably a case of worse is better. I think most of us spend most of our time maintaining existing software that’s nearly impossible to test because of how badly it was originally written; testing much of it seems nearly impossible without a ton of rewriting that we simply don’t have time to do.

The only tool I recall considering a must have is JetBrains Resharper, I’ve made every employer purchase it for me because I simply refuse to use Visual Studio without it. Developer tools is just a tough market. The only communities I’ve seen where testing is part of the general culture seems to be Ruby and Smalltalk. Testing is just not mainstream in the Java and .Net markets as far as I’ve seen.

adminSeptember 15th, 2010 at 10:24 pm

Thanks for the encouragement, Ramon :-) I’m not looking to build a billion dollar business, just support my kids through college. I looked at my assets and the huge adoption of JUnit seemed like the thing I could most easily leverage. I see the challenges much as you do, but I think I have a chance. We’ll see.

[...] This post was mentioned on Twitter by iteman, Alltop Agile. Alltop Agile said: The Economic Case for JUnit Max http://bit.ly/aLCKPy [...]

Marko SchulzSeptember 16th, 2010 at 8:36 am

Quite surprised by the low number of test runs people report here.

For me it varies vastly by the project, language and tools.

Most of the (Java based) test suites I use take longer than 10 seconds, so I often run only a subset of the tests and the whole suite occasionally. Most of the time I run some tests probably 5-20 times per hour.

When using jspec as continuous testing tool with Javascript I maxed out at estimated 200 runs per hour. This is why I am very eager to give JUnit Max a spin (I acquired a license but didn’t find the time for it yet).

Ramon LeonSeptember 16th, 2010 at 9:33 am

> I’m not looking to build a billion dollar business, just support my kids through college.

Yea, I sympathize, and wish you the best of luck. Some people in the Ruby community seem to be making a pretty good bit of money selling screencasts about programming; considering your reputation among programmers, I’d bet you could as well. Just an idea.

VladimirSeptember 22nd, 2010 at 7:40 am

so… test driven development costs $27000 per developer ?
:)

the idea to speed up tests is a very good. But once you are down to earth, there are many other annoying things that slow down developers.
Don’t want to be negative but I don’t see a market for JUnitMax.
Maybe it’s just me working in terribly disfuctional organizations.

Regards,

Vladimir

p.s. Kent, are you participating in any real software development projects? eg you are writing books and giving speeches, but any development?

MeinteSeptember 23rd, 2010 at 12:34 pm

How about a per-use payment model with a curve that doesn’t discourage to do more tests as the Max tool proves its continuous worth? I don’t think any individual developer is going to shell out $100 (since companies generally are very reluctant to buy anything which hasn’t got a IBM-like signature to it) without a trial.

adminSeptember 23rd, 2010 at 7:19 pm

Certainly some style of free trial is in our future.

adminSeptember 23rd, 2010 at 7:21 pm

Vladimir,

I’m quite satisified with sales and adoption at this point, so my experience is different than yours. And yes, I do participate in “real” software development project: Max, JUnit itself, & a large (~200 person year) insurance system on a regular basis. Why do you ask?

adminSeptember 23rd, 2010 at 7:23 pm

I do have a screencast of TDD with the Pragmatic Programmers: http://www.pragprog.com/screencasts/v-kbtdd/test-driven-development . I don’t know about “quite a bit”, but it will turn out to well per hour I spent on it.

VladimirSeptember 24th, 2010 at 4:49 am

Kent,

I’m glad it’s going well. You obviously know your market better and have enough followers.

I asked a question about your involvement with development because I’m getting a feeling that many people who promote various development methodologies are a bit detached from reality. Max and JUnit I guess both your project and you set rules. Insurance system is more like real world project… but the thing is, they hired you because of who you are, so I assume you can set rules as well which makes it more of an exception than a norm.

anyway, it’s good to know that there are people and teams out there caring about 10s vs 2s difference in unit test run times. I hope I didn’t upset you too much with my comments. Have a good weekend.

btw, I have a question. Do you think unit test should test only one class or it’s ok to have unit tests which test several classes working together?

Vladimir

Thomas Ferris NicolaisenSeptember 27th, 2010 at 4:32 am

Hi Kent,

As much as I respect your work and believe that JUnit Max will benefit me beyond the cost, Arne has a point. Developers are used to free tools, and many IT companies do not have budget windows for software of this nature and size. Some refuse everything that costs money, and others only allow “enterprise packages”. And in general, it’s hard to justify a 100$ purchase with managers that don’t even grok TDD.

I know you aren’t asking for pricing suggestions, but here’s an idea nonetheless: If you can get into the developer’s personal budget, where he’ll pay out of his own pocket to make his working day easier, you could perhaps reach a much wider field. I’ve seen this work for BareTailPro (tail for Windows): many people buy a personal copy because they are forced to work on Windows machines by IT, and this is the same kind of IT that does not cash out for “personal” software.

Just to use myself as an example: I map F13 to “Rerun test” in Eclipse, and hit this repeatedly when developing. I also run tests by packages when practical. This get’s annoying when working on larger features across packages, of course, but I’m not dishing 100$ out of my own pocket to reduce this pain. Call me greedy, but it simply doesn’t hurt that much most of the time. Instead I focus on working with one test or package at a time.

If JUnit Max was the same price as a paperback book on Amazon, I would definitely check it out. And I’m sure a lot of other test-driven developers in my situation would do the same. I don’t know how big this market is though, but I’m sure you’ve got an idea of whether this would work.

This is just a humble idea, so please disregard it if your current setup works fine ;)

adminOctober 28th, 2010 at 3:19 pm

Thomas,

Thank you for the feedback. Pricing is definitely not an exact science, and the optimal price will exclude some potential buyers, as it sounds $100/year has for you. I find it rather strange than an employer would pay $100K/year for a developer but wouldn’t pay another $100 to increase productivity much more than .1%, but that’s their decision.

The real money to be made in tools is in bulk licenses in any case. I won’t ever sell 10,000 individual licenses, which is my benchmark for a viable business. I’m sorry if this sounds cold, but I find that having two kids in college sharpens my interest in numbers over ideology.

Juan BernabóJanuary 13th, 2011 at 12:35 pm

Before Eclipse how many times a medium java programmer used to build their projects by hour?

After Eclipse as building became as simple and fast as saving the file, a lot of people changed instantaneously their behavior and the rate of buildings per hour became some orders of magnitude higher, and reduce the cost of delay to know if there is a compile error.

If there is a tool that allows the same instant change on behavior, so that anyone starts to run the tests every time they save the files transparently, you can reduce the cost of delay to obtain feedback on the tests and improve flow and throughput by 10% to 30%…. this number would be really nice to have.

Some how with just making this process cost so little in terms of time, make it possible to put it on a normal behavior programmers already have at a rate much more higher than the other.

One suggestion can’t be offered for free for individual programmers but for a fee for organizations?

Daniel AshtonJune 5th, 2011 at 4:45 am

Hi Kent,

The JUnitMax website is showing an error this morning (June 5 2011), as follows:

This webpage is not found
No webpage was found for the web address: http://www.junitmax.com/
Error 6 (net::ERR_FILE_NOT_FOUND): The file or directory could not be found.