Field Annotations Unexpectedly Rock

I’d like to introduce you to an unexpected effect Saff and I ran into the other night while working on JUnit: annotations on fields are much more powerful than annotations on classes or methods.

Suites

JUnit 4 introduced the use of annotations to tag methods that were intended to be executed as tests. We extended this idea to annotating classes to tell JUnit to run them in a non-standard manner. For example, the old JUnit test suite functionality now looks like this:

@RunWith(Suite.class)
@SuiteClasses({
  AssumptionTest.class,
  ...
})
public class AllTests {}

I always found this syntax ugly and awkward, but I didn’t have a better alternative that was declarative, easy to write, and easy to read.

Annotated Fields: Rules

A recent innovation in JUnit is the introduction of rules. We noticed that may people who implemented custom test runners (like the Suite runner above) we over-served by having to subclass one of JUnit’s existing runners. The extraneous detail made learning how to write runners harder than it needed to be and most requirements from custom runner could be met with just a few entry points. All customer runner writers need is a way to insert their code into the flow of control that results in a test running.

We created rules, annotated fields that are called before and after a test is run. Here is a rule that writes to the console when a test is about to run:

@Rule public MethodRule logger= new TestWatchman() {
  public void starting(FrameworkMethod method) {
    System.out.println(method.getName());
  };
};

We were surprised by how flexible rules turned out to be. People began writing new rules for all sorts of unanticipated purposes. We had stumbled on a really useful API.

Categories: Take 1

For many years people have been asking JUnit to support categories, being able to choose subsets of tests according to a user-generated classification scheme. I thought (and still do) that this is properly the job of the IDE, but since we had implemented everything else higher priority we gave categories a try.

@Category(Fast.class)
public class B {
  @Test
  public void c() {}
}
@RunWith(Categories.class)
@IncludeCategory(Fast.class)
@SuiteClasses({ A.class, B.class, C.class })
public class FastSuite {}

By tagging the class B as belonging to the Fast category, the FastSuite, tagged to include Fast tests, will run the tests in B but not A or C.

The API is ugly but functional. As we got into the twisty little corners of the implementation, though, I had a hard time following what was going on (the true measure of a design is whether the main flow remains clear after all the corner cases have been accounted for). I was ready to quit programming for the life of an itinerant goat cheese maker when I thought, “Maybe the problem isn’t the implementation, it’s the API!”

Categories: Take 2

Putting our experience with rules to use, Saff suggested we try annotating fields instead of classes. Categories are an example of the more general concept of a filter:

@RunWith(Suite.class)
public class FastSuite {
   @SuiteClasses
   public Classes classes= new Listed(A.class, B.class, C.class);

   @FilterWith
   Filter filter= CategoryFilter.include(Fast.class);
}

We transformed both class annotations into field annotations, one to tell the suite which classes to run and one to tell the suite how to filter out unwanted tests.

Here’s the magic part. As soon as we had written this we could imagine several other ways of filtering tests–by recency of failure, by name, and so on. Implementing these changes requires no modification of the Suite runner. In fact, these changes don’t require any modification of the core of JUnit at all.

Likewise, using an annotated field opened up possibilities for retrieving the classes to run as part of a suite. Johannes Link’s handy ClasspathSuite can be recast as an alternative implementation of Classes above, along with a host of other ways to find the classes to run.

Taking a Step Back

I’m a big fan of declarative expression in programming languages. Annotations are superior to convention (like naming all test methods “test…”) for declaring intent because they can be checked by the compiler. The addition of parameters to annotations makes them even more expressive. However, annotations attached to elements fully determined at compile time (like classes and methods) are fundamentally limited.

The problem is the coupling between intention and implementation. An annotation attached to a class specifies both intention (i.e. “run this class as a test in a certain way”) with an implementation (“this exact class right here”). Conflating intention and implementation limits the flexibility and power of annotations.

Annotations on fields separates intention and implementation. The intention is attached to the field while the implementation is specified by the value of the field, which can vary at runtime. It is this separation, and the flexibility it buys, that is the secret behind rules and this proposed new test suite semantics and syntax.

I’m still getting used to annotations and what they are capable of. I know I’ll be looking for ways to annotate fields in the future instead of classes or methods if I have the option.