Continuous Deployment Immersion
I had this idea while I was sitting in the Startup Lessons Learned conference. You definitely don’t want to try this. It’s a crazy idea. And you don’t want to hire me to come facilitate. That would be piling madness on madness. Here’s what you don’t want to do:
Get the whole team in a conference room Monday morning. Choose a feature you are going to add. Have one person come up to the one machine in the room. Write one line of code. Deploy.
If you can’t deploy, erase that line of code and write a different line of code first. Deploy.
Once the deployment is done, write another line of code. Deploy.
About this time some of the rest of the team is going to be bored watching all the manual steps in the deployment. Have them go off and automate the most tedious, boring, or error-prone step. Bring the automation back to the conference room and start using it.
By about Wednesday you’ll be feeling pretty good about your ability to deploy (even if you aren’t feeling so good about progress working one line at a time). You’ll likely make a mistake. Roll back manually. Split off a team to be able to roll back automatically.
At some point you’re likely to need to change some existing code. If you don’t have automated tests for the current behavior, split off a team to write a test before changing the code.
By Friday afternoon you may not have made much progress on your feature (but you may be pleasantly surprised), but you will understand deployment much better than you do now. You will also understand how to slice development more finely and rearrange the pieces. You will also understand the value of automation.
But don’t do this. It’s a crazy idea. Never work. Forget it. Sorry to waste your time.
Awesome idea.
If you’re playing 2 tables (so to speak) as a single Developer/Entrepreneur, then you really need to be doing this exercise constantly.
The lazy triggers in start-ups for hiring more people are usually things like “I’m tired of doing QA” or “I’m tired of build/deploy”…. hire someone else to do it.
Keep the whole thing automated from the beginning.
What a good, terrible idea, Kent
I think that you could motivate some improvements simply by telling everyone that a week as you described it is scheduled. The “dead give-away” bottlenecks that nobody wants to be watch repeatedly on the first day may simply disappear before the week actually starts …
There are only two things I can think of that are missing.
1. I would consider adding some sort of poisonous gas which is slowly seeping into the room that they guy writing the line of production code and manually deploying it is in. That other team has to automate the deployment to prevent his untimely demise.
2. The server room is fille with lions. (This serves no practical purpose other than entertainment.)
Just add a film crew and a manical Perl programming and you will have a reality TV show.
Awesomely (is this even in the dictionary) radical idea, the big question though – how to convince your boss, who wants fast results and quick turnarounds. Selling TDD is like climbing Mt. Everest and now add continuous deployment…still really good if we can do this on a real life projects.
I joke, but this is really an awesome idea. The pain of non automating deployment tasks is spread out thinly over time, so it is not felt. By bringing the pain point close to home, it forces the realization of the cost of non-automation.
“If you don’t have automated tests for the current behavior, split off a team to write a test before changing the code.”
If a team hasn’t reach the point of having an automated (in some fashion, even via cron) test suite, I have trouble believing that you’d be able to sell this to management or even the team itself. At this point they haven’t even accepted the importance of continuous improvement, I don’t see continuous deployment being a priority.
Empiricism!!!! Sometimes you win … sometimes you loose … but you always learn.
Why? In the real world (not the consulting world) we don’t need to deploy every day… nor do we really want to. Yes, feedback is awesome and necessary but customers get annoyed with too frequent deploys of installed software (ours have told us so). Web software might be a different story but even here, the usability factor of constantly changing features means that continuous deployment might not be desirable.
So.. if I am only deploying once a month, why would I spend an entire week having 5 developers shave 30 minutes off my deploy time? Burning 200 developers hours to save 30 minutes a month doesn’t seem like a good idea for a startup or any other business.
If continuous deployment is part of the core strategy of my business process then focusing on that process makes sense. OpenOffice release about 3 to 4 times a year, Microsoft Office about every 3 to 4 years… yet from a business standpoint, Microsoft is still winning. I am not convinced that continuous deployment is a goal that every software team should have.
I understand the philosophical extension of TDD to continuous deployment but creating value needs to be the focus rather than the process. If you never pay attention to the process, you might not be good at creating value. However if all you pay attention to is the process, then you will create a great process that may or may not create good value.
In other words, maybe it matters more what you deploy than how often you deploy. Just a contrarian thought.
Matt,
I appreciate your point that what you deploy matters more than how you deploy it. In learning situations, how you deploy affects what you deploy. The more loops through the learning the cycle and the more you get out of each loop, the better the “what”.
Kent
P.S. I’m 90% product developer and 10% consultant, so the initial ad hominem doesn’t contribute to understanding.
Kent,
Sorry about the ad hominem… it wasn’t meant to be but I see how it turned out that way. I should have said “in my world” rather than the “real world” because the consulting world is as real as the development world.
My point was that a consultant faces a different problem set than most developers face. In the consulting world your exercise is valuable because it points to flaws in the process which presumably is the problem you are trying to solve.
I agree that learning cycles are important. When we deployed a new product a few years ago, we were excited to have our release schedule down to a matter of days. This was critical to the early success of our product as we were able to quickly turn around new ideas based on user feedback.
Creating knowledge is a legitimate goal and trimming the deploy process is a means to that end. I think you and I probably agree that the means shouldn’t become the end. Unfortunately I have seen that happen too often in Agile shops (and well… everywhere else for that matter!)
@Matt,
There is a point you are missing here. You do deploy more often than you think.
Do you deploy to a development environment?
How about an integration or staging environment?
Do developers have to do some sort of deploy to their local machine?
If your answer to any of those questions is “That is a different kind of deployment.” Then that itself is a problem which stems from not having a good automated deployment solution.
Matt,
How do you deploy demo versions for your customers? How do you deploy to an internal server for staging and integration testing? How do you confirm that your current build deploys correctly? How do you deploy a web app to a laptop so a developer can take it out for a network-independent demonstration? Continuous deployment solves all these problems, ensuring you can always do all of the above with one click. Until you have tried it, don’t knock it.
John,
Good points. We have a variety of deploys including native X86, obfuscated .NET, ASP.NET and Silverlight apps. We have discovered that one-size does not fit all when it comes to deploys!
I think the discordant note this article struck in me is that our shop embraces continuous improvement. We aim for incremental, holistic process improvement. Sitting down in a room and focusing on one component in isolation can provide some unique insights… but it can also lead to myopic decision making.
I am being contrarian but only to a point. I am not arguing that the deploy isn’t important, I am arguing that it is only important in context. In and of itself, a great deploy process is meaningless so I question the value of viewing it in isolation.
I am probably wrong but I am certainly not trying to be mean-spirited or narrow-minded, just learning by questioning.
Matt,
Don’t worry no hard feeling here. We are all learning together.
To address your point specifically about holistic process improvement:
In my mind it comes down to this. You find a pain point. You address that pain point. You see where you are and adapt.
You repeat the process over and over. If your not able to consistently build the software, you probably don’t need to worry about continuous deployment. Address the bigger pain point first.
If you are not able to get any kind of unit tests written. Focusing on automated system tests, probably is not a high priority.
Just have to make sure you are attacking one thing at a time. Deployment, gets ignored, because it is a pain point developers don’t feel as much. (Or they are not aware they feel.) In most of the places where I have implemented what I call “push button deployment”. The time it took to actually implement it was tiny compared to the 1000s of hours it saved over the life of the project.
I real love the idea of continuous deployment, even if it’s not possible to implement all the way into production every step taken towards that goal provides a benefit.
My team have been slowly moving towards a continuous deployment model, but I can’t see any way around changes that require a database schema update.
As far as I can see there isn’t an easy way around this problem, so for now, continuous deployment for me means continuous deployment to a staging environment.
Anyone got an ideas of how to better to handle this?
I think this can be done during some training ( workshop, conference or etc )
Nice idea as a motivational technique. Going to have to try that sometime
[...] Beck recently proposed a different experiment in extremely short iterations, for a different reason, continuous deployment [...]