TLDR: When there is no evidence for what works, decisions fall to whoever is most senior or most sure. That is the default in most businesses, and it is expensive. Even the best instincts are wrong most of the time, good ideas from junior people never get heard, and a failed decision by a senior person leaves no fingerprints. A test changes it. It is the one thing in a business that outranks the person, and it gives the intern’s idea and the chief executive’s idea the same fair hearing.
Click to discuss this article with your favourite AI
The last article ended on a question. If nobody can tell what is actually working, who has been deciding what the business does? The decision does not disappear just because the evidence is missing. It still gets made. It just gets made by opinion, and usually by the most senior or the most certain opinion in the room.
The default setting is the boss
In the absence of evidence, hierarchy fills the gap. The redesign gets the green light because a senior stakeholder liked the mockups. The feature ships because the founder is sure customers want it. The campaign runs because the person who signs off preferred the blue one. It all feels like leadership, decisive people making calls. Strip out the evidence, though, and what you are watching is the highest paid person’s opinion standing in for data. The industry even has a name for it, the HiPPO, the highest paid person’s opinion, and in most businesses it is the single biggest input into what gets built.
The gut call that would have cost ten times the saving
Here is one example, with real money on the table. A payment provider the business relied on changed its fee structure in a way that was going to cost us a lot, and the CFO wanted it removed from the site out of frustration. His position was firm. Customers would simply pay another way. The data said otherwise. That provider was one of the strongest payment options we had. So before anyone pulled it, I asked to test it. Hiding it at the checkout increased drop-off at the payment step by 22 percent, and the revenue walking out the door was more than ten times what the new fees were costing the business. The CFO changed his mind. Not because someone more senior overruled him, there was no one more senior, but because the test showed the reaction would cost ten times what it would have saved.
It is tempting to file the CFO story under a #hippoFail tag. But it is not always the most senior person overruling the room on a whim, and it is usually quieter and more collective than one loud call. You do it too. Every time you rank a backlog on gut, wave through an obvious improvement or kill an idea because it does not sound promising, you are doing what the CFO nearly did, just without the title. Which is why the sharpest version of the problem does not come from a boardroom at all. It comes from a team of some of the best engineers on the planet.
The best idea Bing ever had almost did not happen
In 2012 an engineer at Bing suggested a small change to the way ad headlines displayed. It was rated low value and left in the backlog for more than six months, because nobody who looked at it thought it was worth the effort. Eventually a developer built it on a whim, because it was quick to code, and put it into a test. Revenue rose so sharply that the team assumed something was broken. Nothing was broken. That one change was worth around one hundred million dollars a year, the best revenue idea in Bing’s history.
Sit with that for a second. The idea did not improve between the backlog and the test. The only thing that changed was that somebody finally measured it instead of rating it on instinct. For six months, the collective judgement of one of the best engineering organisations on earth had a hundred-million-dollar idea in front of it and called it low priority.
Seniority buys confidence, not accuracy
If a miss like that can happen at Bing, be honest about what your Monday prioritisation meeting is doing. Even the best product teams are wrong about most of the ideas they are sure about, as we saw in the opening piece, roughly two in three go nowhere or backwards. Experience does not fix that. It does not make you right more often. It makes you wrong more confidently, and confidence is the currency that wins an argument when there is no data on the table.
What deferring to opinion actually costs
The obvious cost is the wrong bets. There are two quieter costs that are worse.
The first is that opinion is unaccountable. When a senior person’s idea fails and it was never tested, there is no counterfactual to hold it against, so it is never clearly the idea’s fault. The result gets blamed on the market, the timing, bad luck or worse for you, how it was implemented. The same decision, run as a test, would have been caught in a fortnight. Left untested, it just leaks money quietly and nobody has to own it.
The second is that it silences the room. When the way to win an argument is rank and volume, the person with the right idea and the wrong title learns to stop offering it. You lose your best thinking twice. Once because it never gets heard, and again because the people who have it give up on being heard.
A test is the only thing that outranks the boss
Here is the part most people miss. Experimentation is not only a way to measure things. It is a way to govern them. A test is the one thing in a business that outranks the person. It does not care who proposed the idea or what their title is. The intern’s hunch and the chief executive’s conviction go into the same test and come back with the same kind of answer. That is uncomfortable if you are used to winning by seniority, and it is a relief for everyone else, because it means you can disagree with the most powerful person in the building and simply be right, on the record. That is what settled the payment fight earlier. Not rank, not the better argument, a number that did not care who was in the room.
The bonus the room never sees
There is one more thing a test does that an opinion never can. Win or lose, it leaves the business knowing something it did not know an hour ago. An opinion that turns out wrong teaches you nothing, it just quietly costs you. A test that turns out wrong is money well spent, because you bought a fact you get to keep. That compounding is the next article.
Want to prevent a costly mistake?
If the biggest calls in your business are settled by whoever is most senior or most sure, I help teams put a cheaper and fairer arbiter in the room.