Multiple Deployed Versions: The Once and Future Practice
I gave a talk recently entitled “Software G Forces“, where I outlined how “sensible” software engineering changes when you deploy software annually, quarterly, monthly, weekly, daily, and hourly. Preparing the talk reminded me forcefully that the answer to every interesting question in software development begins with, “It depends…” What looks like solid, responsible development with quarterly deployments seems ludicrous when new versions are deployed every day, and vice versa. The principles are the same but the practices are wildly different.
One of the intellectual games I played in preparing the talk was to look for a practice that only made sense for a range of deployment cycles, that was effective for, say, quarterly to weekly releases but not for longer or shorter cycles. Anders Haugeto, who was visiting Three Rivers Institute world headquarters with his team, challenged me to find practices that are the opposite, that make sense for very long deployment cycles and very short deployment cycles, but not in between. I finally found one–deploying multiple versions.
At very long deployment cycles having multiple versions in the field is necessary because you need to have patch releases to address deployed defects. Customers can’t wait a year to get their defects fixed. Also, because the upgrade path for infrequently deployed software is often painful and/or expensive, customers will stay with downrev software, sometimes for years.
Multiple versions is expensive because you have to port defect fixes to (potentially, at least) all the active branches. I’ve seen teams grind to a halt on new development because porting fixes became so expensive. One of the advantages and enablers of more frequent deployments around the quarterly/monthly transition is moving to having a single deployed version.
I was talking with Timothy Fitz about what it would take to be able to deploy their client software hourly instead of daily, and, surprise surprise, he said that they would have to add the ability to have multiple versions of the software deployed simultaneously. Because there is a non-zero chance of a deployment needing to be rolled back, to avoid disruption to the service you need to roll out new versions gradually. This implies that you need to be able to run the two versions at the same time.
It turns out that the online computer gaming community is grappling with this same issue. The tough situation is when you need to roll out incompatible changes. They use techniques like having all players with the older version only able to interact with other players with the same version. As players log off and log back on, they get the new software.
Anyway, there you have it–a practice that makes perfect sense with annual or hourly deployments, but not in between.