A blog post about the importance of Testing and learning

One of the things I teach organisations is the benefits of incrementalism.

Modern Business Case culture is designed to prevent waste by ensuring that a project has a sound Business Case before any money is spent. Unfortunately, it often achieves precisely the opposite.

As Exhibit Number One, m’lud, I would like to present to you the launch of Dot2Dot by my old employer National Express.

I use the example not because it is unusual or egregious. I use this example solely because I was there and saw it happen.

I therefore want to emphasise that I’m not saying that those involved were individually culpable. What happened was what often happens in corporates: money wasted not because of human incompetence but because of the way we make decisions.

I would also like to clarify that I was at NX at the time and did absolutely nothing to intervene to try to course-correct. My wisdom is entirely in hindsight (as a result of all the experiences and research that has caused me to create a business focused on precisely these questions). As a naive 26 year-old, I certainly wasn’t on the phone to the top team to suggest they were getting it wrong.

So, with that out of the way, what exactly happened?

A decent idea

The idea wasn’t a stupid one.

National Express already had a big presence in the airport transfer market. National Express coaches was the largest operator of public transport from Heathrow to places other than London, while Gatwick and Stansted Express trains (both then run by NX) were the dominant airport transfers from those airports. All of these were mass-market public transport, but NX spotted that many other global airports are served by minibuses connecting the airport directly with hotels. London airports, by contrast, were served by rail links that only served a central London terminus, with passengers then expected to change to a tube or taxi (this was all long before the Elizabeth line).

*National Express already had a strong airport expertise*

They felt there was an opportunity to provide minibuses. The idea was for an online-only business that dynamically routed the vehicles in response to passengers’ bookings. Today we’d call it DRT (Demand Responsive Transport).

So far, so good

This reasoning was sensible.

What should have happened next is that the coach business, as the business unit already closest to this market, should have been given the task of incrementally testing and learning whether this was a good idea.

They could have run a whole bunch of Google ads for this new proposition and compared the price and popularity with the existing coach service, which they already understood. The fact the proposition didn’t exist would not have mattered. Those who clicked on it could have been shown a “sign up for an email when it goes live” form. This digital data would have given them a steer on the potential market.

If it passed this test, they could have hired a fleet of minibuses for a couple of weekends and offered the service manually. No need to build a website, no need to build a routing algorithm. Just put a whole bunch of graduate trainees into the airport to sell paper tickets and get one of the more intelligent controllers to manually route the hired minibuses. It would have cost a few thousand quid. If they’d done it for three consecutive weekends at three different price points, they’d have got a pretty good sense of whether passengers wanted it, how much they were willing to pay and what it was worth.

This approach is the approach of the Lean Startup, a fantastic book by Eric Reis. I don’t really blame National Express for not having adopted it in 2007: he only published it in 2011. But I used it for my startup Sn-ap in 2016 and it works like a treat.

The first fundraise we publicised was £2m. But, in fact, we’d quietly raised £50k, which we spent running a series of digital and real-life experiments so that by the time we launched in public, we already understood that the market existed and what it was worth.

Not being able to mind-read Eric Reis, National Express didn’t do any of this.

*The approach National Express could have taken - had they been able to see into the future*

Instead, the project was costed up by finance and found to be expensive. As a result, instead of being delegated down it was delegated up. The Group central finance team took control of it and commissioned a major management consultancy to produce the business case. The business case cost a large six figure sum to produce. Of course, once so much time and money had been invested, it was psychologically harder to abandon it.

Luckily, it didn’t seem like there was a need to abandon it. As is so often the case, the professionally-produced business case looked very professional. It had both excel and powerpoint and lots of numbers. It included a full P&L for each year and showed when it would break even. It was a really good piece of work. It was also, of course, completely wrong.

The Launch

The business case was hugely encouraging.

National Express expanded its ambitions for the product. As well as serving the airports, it was also decided to serve Canary Wharf (where National Express had recently acquired a successful commuter coach business) and Kings Cross (where National Express had taken over the East Coast rail franchise).

A fleet of seventy vehicles would be needed for launch, with drivers recruited and trained.

Creating the website, routing algorithm and depots was a huge task. Recruiting and training the drivers was an even bigger one. The team did a superb job. I wasn’t close to it, but it was clearly impressive.

By late 2007, it was ready to launch.

The flop

And it was a total failure.

It was very quickly clear that the market just wasn’t there. It was a premium product and people just weren’t willing to pay the prices being asked for. While it was direct, it was also slow. It seemed that, if asked to spend a lot of money, people preferred to get the Heathrow or Gatwick Express and then a taxi. The service lost £5m in the first (partial) year and £8m in the first full year. Just 15 months after launching, National Express sold it for £1. Its new owners closed it seven months later.

The fundamental absence of a market could have been learned in advance but the business case (while expensive and thorough) was theoretical.

Not doing those small-scale experiments also meant that details that could have been fixed were missed. It wasn’t obvious until it launched, but the name was a disaster. The proposition was web based. It was before apps. It needed a website. The website was dot2dot.com. Now imagine you’re a hotel concierge explaining this to a prospective customer: “Dot, that’s D-O-T, 2, that’s the digit 2 not the word two, dot again D-O-T, then dot as in an actual dot not the word dot, com”. So obvious once live but not remotely obvious in planning. Susan Boyle made a similar mistake when she invited people to the Susan Album Party. No-one spotted the url of susanalbumparty could also be read differently if you assume the first three letters are Su’s. It happens.

The lesson

Now I’m much older and a bit wiser, it is obvious what went wrong. A good idea was escalated up not down. Instead of people close to the market running experiments to discover if the market actually existed, a huge and expensive business case exercise was kicked off. The irony, of course, is that because it was such an expensive idea, it was felt necessary to spend lots of money on the business case. Not all ideas are good ones, and a business case is not reality. So get into the field and learn.

Take action!

Freewheeling teams show a bias to action. Take action now:

If you’re a leader:

Read The Lean Startup. While the title suggests it’s just for startups, it really isn’t
Remember that excel isn’t life. If you want to know if something will work, it can often be quicker and cheaper to try it than try to model it
Set your team the challenge of coming up with the cheapest, lowest cost experiment possible
Remember that making decisions based on incomplete information is no worse than making decisions based on inaccurate complete information

If you’re a team member:

When you make a proposal, accompany it with the specific low-cost experiments you can run to learn if your proposal is a good one
Treat your proposal like a scientific hypothesis. We can get wedded to our ideas. Instead, be clear that you have a hypothesis to test. Your proposal is for the experiments, not the idea

What do you think? Have you seen similar examples? Can we do things differently? Join the discussion on LinkedIn.

👋 I'm 𝗧𝗵𝗼𝗺𝗮𝘀. I help organisations like yours drive 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻, deliver 𝗰𝗵𝗮𝗻𝗴𝗲, and achieve 𝗳𝗮𝘀𝘁𝗲𝗿 results, drawing on 20 years of leadership across public and private sectors.

🚀 I offer 𝘀𝗽𝗲𝗮𝗸𝗶𝗻𝗴, 𝗺𝗲𝗻𝘁𝗼𝗿𝗶𝗻𝗴, and 𝗰𝗼𝗻𝘀𝘂𝗹𝘁𝗶𝗻𝗴 that energise teams, shape strategies and remove barriers to change. Whether you aim to accelerate innovation, drive change, or inspire your people, I’m here to help. Let’s talk!

Join the Dot2Dots: A Case Study In too big, too fast