Why agile is so hard for corporates
This post is an attempt to explain why corporates struggle with software projects. To do so, I’m going to use the analogy of driving to Cornwall. Because why not…
If you want to know what it’s like running a software project in a transport corporate, imagine being a car driver heading off on your family holiday.
You know that you live in Southgate in North London and that you’re heading for Perranporth on the Cornish Coast. You know that you need to arrive by 5PM as you are collecting the keys to a holiday cottage.
From this, you calculate that this means you should leave at 6AM to give you plenty of contingency. You also decide that you’re going to go via the M25, M3, A303 and A30. But you’ll have your SatNav switched on and your radio tuned for traffic updates, and will keep an eye on how things are going.
Now imagine that your spouse doesn’t drive themselves, doesn’t understand maps, doesn’t trust your ability and has decided that you need a more robust schedule so they can be confident they know it’s all going to plan. So they (from now on, we’ll assume that you are female and your spouse is male) have created a milestone plan.
At 0630 you must be at South Mimms. At 0700 you must be passing the M40. At 0730 you must be on the slipway joining the M3. They calculate the mileage (to the nearest mile), estimate your petrol costs, decide on the precise venues for breakfast, lunch and tea and write it all down in a Gannt chart.
You protest that this is madness. You know where Perranporth is and you can judge the best way to get there. But the best way may change according to circumstances and these constraints aren’t going to make it easier; they’ll actually make it harder.
But he’s unrelenting. He doesn’t want to get to 5PM and discover you’ve ended up in Dover. Given that he doesn’t understand driving and maps and stuff like that, how else will he know that you’re on track to achieve your goal than through clear time-bound milestones that will enable him to hold you to account for your progress in delivery. Reluctantly, you agree and get on with packing. Grumpily.
The day of the journey starts badly. The kids are in a strop and you leave at 0645 not 0630. At 0700 you haven’t even reached the M25, never mind South Mimms. By 0730, you’re passing Potters Bar when you should have reached the M40. Your project is now in a state of crisis. Your spouse is livid: given how many milestones you’ve missed, the chances of successful delivery look remote. You try to persuade him that, actually, it’s completely fine and that nothing that matters has gone wrong and that you’ve still got plenty of time to reach Perranporth. Indeed, at this stage, what has happened is entirely irrelevant. But he points out that you’re now 50% off target and your pleadings get lost in a hail of metrics.
You continue round the M25 and your SatNav and radio spark into life. They tell you that there’s been an accident on the M3 at Frimley. The entire motorway is backed up. Anyone going that way is likely to be there for hours. You quickly tell your spouse that you should go via the M4 and M5. Luckily you’ve discovered the situation before you hit the M4 interchange so no harm done.
He hits the roof! Going via the M4 is going to add £40 in petrol costs. Not only are you late, you’re also over budget! Moreover, he’s disbelieving. Partly disbelieving that there’s a problem that requires a £40 solution (he points at the clear road out of the windscreen as evidence) and partly disbelieving that you didn’t see this coming. He points out that you appear to have been taken by surprise by traffic. Traffic! Surely anyone could have predicted that traffic would be an issue when driving from London to Perranporth. Why didn’t you plan for it so that you could have hit the milestone plan and budget even with the “traffic”.
You point out that had you gone via the A303, you wouldn’t have incurred the extra cost and that the original plan was an estimate. Had you been asked to, you’d have come up with a range of costs based on all the different routes that might come into play but even that wouldn’t have been certain (do you include the costs for the insurance excess just in case you have a minor scrape? You’re not planning to but it might happen?). But all he hears is that you’re spending more money in order to change a plan which is already going wrong.
You can’t cope with this. You pull off the M25 at Rickmansworth, pull into a lay-by next and refuse to move until you’ve got a better way of handling things. You explain that you understand his need to be sure that you’re going the right way. That you understand that he doesn’t do the driving, doesn’t understand where Perranporth is, doesn’t understand about traffic or routes or motorways or any of the other things that affect this journey and that, therefore, he needs confidence it’s all OK. But you explain that he is acting against his own interests by making it impossible for you, the expert, to actually get you to your holiday.
You make a proposal. From now on, you’ll be crystal clear what the next milestone looks like. Now you know there’s traffic on M3, it would be madness to go via the M3. So you’ll go via the M4. You’ll give him an estimate of where you’ll be in thirty minutes (Slough) and how much petrol you’ll have used to get there. Thirty minutes later, you’ll do the same again. And again. And again. That way, he can build confidence you know what you’re talking about. And if you get it wrong, you’ll talk about what you got wrong and work out whether it was a one-off or is going to need to be built into future estimates.
You’ll also keep him appraised with your honest assessment of whether the things that are happening are going to affect your likelihood of getting to Perranporth by 5PM. Without having your head filled to bursting with hitting every single milestone, it will be much easier to keep focused on the actual end goal.
After a bit (ok, a lot!) of persuasion he agrees that, maybe, he can live with this. He’s not happy but he’ll agree to try it out.
What you’ve just done is invented agile.
A few key points: what you’ve not done is eliminate controls. Your spouse is entitled to know that you’re being held to targets. It’s just that the targets are focused on the most important things: the end goal and what you’re doing right now. Moreover, by focusing on what matters most, you eliminate distractions onto things that matter less.
You also haven’t eliminated the need to plan. You should plan. Thoroughly and in detail. The key point is to try to work out whether leaving Southgate at 6AM with a goal of reaching Perranporth by 5PM is a sensible one. Your spouse’s milestone plan was a very sensible way of testing that. Having a plan for where to have lunch wasn’t a stupid idea either. Agile doesn’t mean undeliverable and it doesn’t mean unplanned. Your spouse needed to prove it could be done, and so should you. But, having proved it, the milestone plan should have been thrown away.
And it doesn’t mean a lack of financial controls. Your spouse was quite right to insist on a budget. Your mistake was allowing the estimate of the cost of the best-case scenario to become the budget. Your budget should have been for a reasonable case scenario in which there’s some traffic and some need for diversions. And then you keep monitoring what happens. If you get stuck in a queue behind an accident for seven hours and need to stop in a hotel to avoid driving while exhausted then this is a new cost that wasn’t reasonable to budget. But it doesn’t change the fact that the cost is now necessary to achieve the goal (or as close as possible to the goal that can be achieved in the new circumstances). If your spouse doesn’t like this new reality, it doesn’t change it. The only alternative to incurring the cost is to abandon the goal. And, sometimes, as in this case, even abandoning the goal doesn’t prevent the cost emerging. The key things are to avoid such a tight plan that honesty about costings becomes an admission of failure .
In the scenario we’re describing here, you switched to the M4, got caught in a bit more traffic, had to shell out for the extra petrol and arrived in Perranporth at 4.24PM. But, of course, early on there was a moment of truth where the justifiable desire of your spouse for controls risked tying you to a plan that was no longer optimised and, had you not had your little talk, would have made you both grumpy - and late.
As it is, you’re right on time for a lovely holiday: and a much better relationship with your spouse…
Having spent a career in corporate transport before founding Snap, I’d heard a lot about agile development.
I also knew various corporate teams that professed to do it.
But it was only when entering Startupland that I properly understood agile development - and why it is so incredibly hard for corporates to actually do.
Most software development follows the traditional ‘waterfall’ method of project management (called waterfall because each element of the project cascades into the next). A waterfall plan is detailed and prescriptive. Everything is pre-planned with detailed milestones and sequences. This leads to a precise work schedule and a precise budget. If you are building something like Crossrail, you really need to use the Waterfall method!
Unfortunately, corporate software projects - which invariably use the waterfall method - frequently go ‘wrong’ - in the sense that neither the milestones nor the budget are achieved. Everyone gets very cross and insists on doing better next time. Unfortunately ‘better’ frequently means more milestones and tighter controls - which is the exact opposite of what is needed.
In both Arriva and the RDG, I’d been involved in software projects that made every mistake described above. When I founded a startup, I had to go through a big process of re-educating myself. I didn’t always find it easy. As a non-software engineer, it can feel really hard to let go of the milestone plan. I didn’t always succeed. But we always should.
