The Open/Closed/Open Principle

The Open/Closed Principle always bothered me. I agree with it philosophically–good designs make it possible to add functionality without disturbing existing features–but in my experience there are no permanently closed abstractions. Superclasses or APIs might be stable for a (relatively) long time, but eventually even the most fundamental classes and interfaces need updating to meet emerging needs. I got to thinking about Kuhn‘s The Structure of Scientific Revolutions and its relationship to software design and found, much to the surprise of my jetlagged mind, a resolution to this dilemma.

Kuhn divides science into ordinary and revolutionary. Ordinary science is the kind that goes on all the time–gathering data, extending ideas a little at a time, publishing papers. Once in a while, though, some of the facts stubbornly refuse to fit the theory. Eventually the misfit is large enough to trigger revolutionary science. Crackpots propose wild ideas. Eventually one of them catches on, triggering another round of ordinary science.

Software Design

I got to wondering if software design goes through ordinary and revolutionary phases, and if it does, so what?

Ordinary design is the kind we do every day–extract a method, extract an object, move a bit of logic or state closer to where it belongs. The open/closed principle pretty much works. Superclasses sit. APIs sit. New features fit the design without much change.

Then comes a feature that really doesn’t fit the design. The fundamental elements and relationships have to be twisted to implement it. The fact (feature) just doesn’t fit the theory (design).

Kuhn describes revolutionary science as a time of chaos, with various odd ideas competing for attention. Each theory fits the new fact, but at the cost of not explaining the existing facts. In software, though, there is a preliminary phase. When the need for design change becomes apparent, software designers can isolate the part of the system that is to change from the part that will remain stable. Assumptions about a wire protocol, for example, may be scattered about the code, but before changing the protocol the designer can gather these assumptions in a single element. Then the parts of the system that don’t care about the wire protocol are unlikely to be accidentally affected. Isolation reduces the cost of change by reducing risk.

Revolutionizing designs without first isolating change puts a bigger burden on the designer. The challenges of revolutionizing a design while working in safe steps is generally enough for me without adding the challenge of keeping track of changes all over the code base. Isolating change is low-risk and fairly mechanical, while giving me an overview of areas of the system that are about to be overhauled.

An alternative (or sometimes companion) strategy is to start from scratch. If you have no idea how to support a new wire protocol, you can build a system that just has the wire protocol and nothing else. You will be missing many of the feature you expect, but you will be able to thoroughly explore designs for the protocol without distractions. What comes out of a de novo design is generally an understanding of the needs of the new design to be folded back into the existing system. Occasionally, though, you discover that you really don’t need all that other stuff and you have a tidy, and much smaller, system to work with.

For example, we started from scratch when we implemented JUnit 4. First we made sure we could run tests marked with annotations. When we understood that thoroughly we made sure the new tests worked alongside older tests. For the recent introduction of interceptors, though, we carefully isolated the code that ran tests and made it modular before finding a new way of modifying test running.

Revolutionary design violates the open/closed principle, almost by definition. The feature you want to add needs new elements and relationships that don’t fit with the existing design. The basic abstractions need to be reopened to modification. Once the feature is added, they can close again. Further development can use the new elements and relationships as vocabulary for further extension. This extension takes place against a background of ordinary responsive design.

Example

I’ve been interested for some time in better support for remote pair programming. The assymmetrical response time of screen sharing makes equal contribution impossible. What is needed is multi-local editing, where every user sees immediate feedback from their keystrokes and later processing handles remote updates and conflict resolution.

The Eclipse editor, not surprisingly, is not designed to handle such a feature. There is no clean separation between user events and changes to the internal model of the source file. This is not intended as a criticism of the Eclipse design. Design in advance of need is waste. The fact of multi-local editing simply doesn’t fit in the theory of the current design.

Following the outline above, the first step would be to modify the editor to clearly separate event processing from model updating. With that in place it should be possible to revolutionize updating to include the possibility of several sources for updates and for trying various architectures, peer-to-peer and master-slave, for detecting and resolving conflicts.

Conclusion

As I said in the opening paragraph, so what? Being aware of when I’m doing ordinary design and when I’m doing revolutionary design helps me cut down on the design space. When I’m doing ordinary design (which is my default), I try to support a feature with small incremental changes at the fringe of the design. Only if limited changes don’t work do I change hats and look for more fundamental and far-reaching changes. Ordinary design changes are easy to communicate and can be created at full speed.

Revolutionary changes requires much more care, both technically and socially. The change may require succession so the audience can absorb it. Revolutionary changes may also require many technical steps to achieve, both in the isolation phase, the experimentation phase, and the execution phase. Some experiments will fail–yes this design supports the new feature but it will never support the old feature. Sigh… back to the sketch pad.

UPDATE–Revolutionary design also requires different values from ordinary design. In ordinary design, I ignore design changes that make the code worse. Deliberately introducing duplication, for example, is unthinkable. In revolutionary design, though, I will happily duplicate code as long as I suspect I can eliminate it later. Sometimes an effective revolutionary strategy is just to inline absolutely everything in a class and helpers into a single method and begin re-extracting from there. I visualize this with the Design is an Island metaphor. Ordinary design is uphill, revolutionary design is under water.

Ordinary and revolutionary design are both necessary for responsive design. While most features can be supported with no design changes or only ordinary changes, new classes of features require revolutionary design. Staying in ordinary design as long as possible but shifting to revolutionary design as necessary and only as long as necessary helps keep the design lean, pliable, and a good platform for the needs of stories as they emerge.

18 Comments

Peter WilliamsJune 21st, 2009 at 11:20 pm

That’s funny. I always thought the most important lesson from science for software development was its rigorous empiricism. Ronald Fisher expressed this eloquently:

In relation to any experiment we may speak of this hypothesis as the “null hypothesis,” and it should be noted that the null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation. Every experiment may be said to exist only in order to give the facts a chance of disproving the null hypothesis.

The null hypothesis in software development is that your software works as intended.

UnclebobmartinJune 22nd, 2009 at 2:58 am

A funamental truth, nicely described.

KentBeckJune 22nd, 2009 at 3:19 am

That sounds like TDD (likewise proofs of “correctness”, or rather correspondance). Each one is an experiment that can disprove “the software works as intended”. If it passes, as you say, that doesn’t mean the software works as expected. If all the tests pass, though, and they are good tests, experience shows that the next “test” introduced by the user is unlikely to disprove the null hypothesis.

Robert ElliotJune 22nd, 2009 at 4:08 am

Perhaps my understanding of the open/closed principle is faulty, but it seems to me to demand a level of clairvoyance from the developer about where a system will need extension in the future that will lead to breaking the YAGNI principle an awful lot. My understanding of TDD and XP would suggest that you don’t add abstraction layers for extensibility until you actually need them, and then when you do you happily alter the code secure in the knowledge your tests will prevent any regressions.

I guess it’s more important when writing a library – but even then, it is surely the API contracts that should be closed for change, not the actual implementing code?

But as I say, perhaps I’ve misunderstood the open/closed principle. Or perhaps it’s more of a thing to strive for than an absolute where you have “failed” if you find yourself altering a superclass?

Chris AniszczykJune 22nd, 2009 at 5:04 am

“The Eclipse editor, not surprisingly, is not designed to handle such a feature. There is no clean separation between user events and changes to the internal model of the source file. This is not intended as a criticism of the Eclipse design. Design in advance of need is waste. The fact of multi-local editing simply doesn’t fit in the theory of the current design.”

So, the editor may not have the optimal design for this, but it hasn’t prevented us from trying. Inside the Eclipse Communications Framework (ECF), we came up with COLA and some code to handle this case. Here’s a video:

http://live.eclipse.org/node/543

The ECF project is now looking at implementing Google Wave too:

http://eclipsesource.com/blogs/2009/06/15/google-wave-and-ecf/

Mike BriaJune 22nd, 2009 at 6:02 am

Calling out an explicit distinction between “ordinary” and “revolutionary” design, interesting. I wonder if keeping track of how much of either you’re doing might hint at any useful insight on how fit your design actually is? Dunno, just thinking out loud.

Great post, thanks for it.
MB

KentBeckJune 22nd, 2009 at 6:04 am

Chris,

It looks like you made a lot of progress although the documentation I could find was a bit discouraging (no cut/paste support, for example). I’d love to talk to you about your experiences and the current state of the project. This is definitely the future of collaboration.

KentBeckJune 22nd, 2009 at 6:07 am

Michael Feathers made the observation that OCP was an emergent property, not a goal. I agree, in which case my observation is that it is an emergent property that comes and goes in a healthy system.

John KavadiasJune 22nd, 2009 at 7:20 am

Nice post! Nice comment from Mr Williams too. Very succint.

Re: Kuhn, I think he also observed that maverick or unfashionable ideas (“wild ideas” proposed by “crackpots”) may also be around for quite some time and struggle along in parallel with the mainstream, which is pursued as the most successful theory/framework of the day.

When the mainstream stalls, old as well as new “wild ideas” are (re-)examined in the search for a way to progress.

So as well as new crackpot ideas, when engaging in revolutionary design, we also look for previously existing niche or “less popular” or simply different ideas and approaches.

Perhaps their merits and breadth of application are poorly appreciated/known by the mainstream.
[ Example: new (computer) science becoming more accessible with documentation and implementation (like ratpack parsers making more natural DSLs easier to build). ]

Perhaps the ideas are understood, but were considered too high an exploration-cost or investment before now (archipelagos of designs in your island metaphor).

And for a bit of cheek and humour:

Ordinary design is about accreting to the design, and simple Darwinian evolution (slow evolution through small variations of valued characteristics).

Revolutionary design is about r?evolving the design. To r?evolve is to dramatically evolve (punctuated evolution?) and potentially turn (around, or to a new perspective, or on its head) the design.

Rick DeNataleJune 22nd, 2009 at 7:36 am

I can’t help but think that this is the normal refactoring process, maybe on a longer time-scale.

The paradigm which represents “normal science” or “the current design/framework” represents technical debt, which becomes more and more apparent as new facts, or requirements, which don’t fit the paradigm are uncovered.

So the paradigm shift is a refactoring of the old paradigm to incorporate the new facts/requirements whether they be a new wire protocol or data from physics experiments where relativity comes into play.

Rafael Peixoto de AzevedoJune 22nd, 2009 at 9:16 am

High quality system designs are based on modular and expressive abstractions.

Modularity is a design quality a syntactical level.
A design’s abstraction is a module if its internal implementation and external context are coupled only through the specification of its responsibilities.

Modular designs enable local implementation change and global reuse in new contexts.
Modularity enables locality and reusability and facilitates human minds to engineer highly complex and adaptable systems.

Expressiveness is a design quality at semantic level.
A design’s abstraction is expressive if its specification represents a domain concept.

Expressive designs can easily articulate the system’s abstractions to clearly express the relevant concepts, problems and solutions. Expressiveness facilitates understanding and communication that enable human minds to engineer relevant and valid solutions.

Identifying, representing and separating concerns
is the key guiding principle for our design process.
It bridges syntax and semantics to define abstractions and assign responsibilities
that compose and decompose modular and expressive designs.

When our world view or design objectives are radically changed,
new concerns emerge as relevant.
Due to their central role in our design process,
the new related concepts, problems and solutions
can only be expressed after the new concerns are also represented
and separated in the design.

That is when the Open-Closed Principle usually breaks at some scope.
The “revolution” requires a major refactoring
before continuing “evolutive” design based on iterations and increments.

Remy Chi Jian SuenJune 22nd, 2009 at 10:37 am

Kent, if you have inquiries about the Cola shared editor, please post them to the eclipse.technology.ecf newsgroup or the ecf-dev mailing list.

Technically speaking, a user should be able to cut/copy/paste text but the integrity of the document may be lost as I’m not sure if the conflict resolution algorithm currently takes batch operations into account. However, if only one user is editing the document, he or she is free to perform batch deletion or insertion operations.

Currently, you can only edit one document at a time though we hope to address that with bug 238966.

[...] and Small Scale Refactoring Reading Kent Beck’s post about Ordinary and Extraordinary Design reminded me of a long-running discussion on a long-running project about what refactoring is for [...]

MortenJuly 11th, 2009 at 1:41 pm

As Rick mentioned above, it sounds just like refactoring. Some features needs major refactoring to implement in exisiting code, others dont. The purpose of the open/closed principles is to minimize the amount of refactoring needed. But your code is just a model, not a concrete mapping of the users actual needs. (In fact the users words are just mappings of the needs)

The model will change, and break, just as the users needs will change. All we can hope to do is to manage change. Open-closed principle is just a principle, but sometimes you have to break principles to go forward. Sometimes you have to leave your comfort zone.

James AbleyJuly 24th, 2009 at 6:25 am

Kent, this seems like the best place to leave a comment about your search collaborative editing tools in Eclipse.

http://etherpad.com/

Web-based, but has some nice features. I guess having access to Google Wave will similarly give you a rich stream of ideas about how you might want this to work.

[...] [...]

Shane CourtrilleSeptember 24th, 2009 at 7:57 am

This to me kind of fits with Ayende’s view of Concepts & Features @ http://ayende.com/Blog/archive/2009/03/06/application-structure-concepts-amp-features.aspx.

Revolutionary changes are concepts where as ordinary changes are features.

[...] Special thanks to Kent Beck for the enlightening inspiration! [...]