TLDR: Once the data is trustworthy, analysis is a discipline, and the same discipline whether a person or an AI runs it: start from a hypothesis, decompose before you theorise, find where a change lives, and be willing to prove yourself wrong. AI has made the mechanical part, writing the query and pulling the data, almost free. It has not touched the judgement, and a fast wrong answer is more dangerous than a slow one. And analysis can only tell you what happened and why. To learn whether a change you make actually works, you have to experiment. That is where this goes next.
Click to discuss this article with your favourite AI
You can now point an AI at your data and ask it almost anything in plain English, and it will write the query, pull the numbers and draw you a chart in about the time it takes to read this sentence. It feels like analysis has been solved.
It has not, and it is worth being precise about why. Writing the query can be genuinely hard, structuring the logic, getting the joins right, handling the edge cases. But that is mechanical hard, the kind a machine is now genuinely good at. The part that never came in a box is the judgement: knowing which question to ask, whether the answer you get back is real and what you should actually do about it. AI got dramatically better at the mechanical half and left the judgement half exactly where it was.
Analysis is a discipline, not a query
Good analysis follows a shape, and it is worth knowing that shape whether you run it yourself or check an AI that ran it for you.
It starts with a hypothesis, not a question. “I wonder about mobile” gets you a data dump. “I think mobile revenue fell because the cart is hard to use on a small screen, and if so the drop should be worst at the cart step” gets you something you can test. A hypothesis tells you what to look for and, just as importantly, what would prove you wrong.
When a number moves, the discipline is not to explain it straight away. First I check it is even real. Before I theorise about a 15% drop in sales, I check whether a tag broke when the site last changed, whether it was a public holiday, whether the same week last year looks identical. A surprising number of dramatic drops vanish the moment you do that.
Once you know it is real, you narrow it down before you reach for a reason. Is revenue down because fewer people came, or because the people who came spent less. Is it happening everywhere, or only on mobile, only from paid ads, only with new customers. Every one of those questions makes the problem smaller, and by the time you have worked through them you are not staring at the whole business wondering what happened, you are looking at one specific corner of it. A cause is much easier to find in a small room than a large one.
And it stays honest, which mostly means being willing to kill your own theory. On one site, the shortest and highest-converting path to purchase looked obviously like loyal repeat customers who knew exactly what they wanted. The tidy story. When we actually checked, two thirds of them were brand new, arriving from organic search. The opposite of the assumption. Ship a loyalty campaign off the tidy story and you would have aimed it at precisely the wrong people.
The same number can hide five different causes
Here is the part an eager AI is worst at, and where a person earns their keep.
On that same site, customers who left checkout, went back to the cart and returned converted about ten points worse than those who did not leave the checkout. Easy to spot. The trouble is that one fact had at least five plausible explanations. Shipping cost sticker shock. The full order total landing for the first time. The friction of a checkout that would not let you edit your cart in place. Re-entering details on the way back. Or simply the psychological break of having quit the flow once. Five causes, and they point at five completely different fixes.
An AI asked why the number dropped will happily hand you one confident answer, usually the tidiest one, shipping cost. It will read beautifully. It has no way of knowing which of the five it is, because the data alone cannot separate them, and neither can a person until they design a way to tell them apart, which here means letting some customers edit their cart inside checkout and watching whether that ten point gap closes. That is the whole point. The value was never the answer. It was seeing that one clean number hid five questions.
Where the machine genuinely helps
None of this is anti-AI. Used well it is a real advance, just not the one people advertise.
What it collapses is the mechanical time. Writing the SQL, shaping the data, running the same breakdown across twenty segments, the work that used to eat a day now takes a minute. That is real, and it matters, because it means the scarce human hours go to the judgement instead of the plumbing. The catch is that it raises the stakes on the judgement rather than lowering them. A wrong answer produced in a minute, in fluent and confident prose, is more dangerous than a wrong answer that took a day, because nobody thinks to check it. The faster the machine gets, the more the human has to stay involved, not less.
And all of it, every hypothesis and every clever breakdown, rests on the one thing this whole series has been about. Run the sharpest analysis in the world on data you cannot trust and you get sharp nonsense, faster. The reason we spent five articles on questions, mapping, auditing, defining and fixing is that analysis is the easy part once the foundation is real, and close to worthless when it is not.
What analysis still cannot do
So say you have got there. Trustworthy data, a reporting layer that flags what matters, the discipline to interrogate it properly. There is still a hard limit on what any of it can tell you, and it is the most important thing to understand before you spend real money acting on an insight.
Analysis works with the data your customers have already generated. It is very good at what happened, and at its best it gets you to a strong theory about why. What it cannot do is tell you whether a change you make will work. Take a real one. On that site, people who used the search bar bought far more than people who only browsed. Tempting to conclude that putting a big search box in front of everyone would lift sales. But you cannot tell, from watching alone, whether searching makes people buy, or whether people who already knew what they wanted were always going to search. Observation cannot separate the two. Redesign the site around that hunch, be wrong, and you have spent a quarter making the number worse.
The only way to actually know is to change one thing on purpose, show the bigger search box to half your visitors and the current site to the other half, and measure whether the half that got it actually bought more. That is an experiment, and it is a different discipline from analysis. Trustworthy data and honest analysis get you to a hypothesis worth betting on. Experimentation is how you find out whether the bet pays before you wager the whole site on it.
Which is exactly where this series ends and the next one begins. We have spent it building data you can trust and learning to read it honestly. The next series is about the thing reading can only ever point you toward: designing and running experiments that tell you, for certain, what actually moves the number.
Ready to get your data in order?
I help ecommerce businesses build data they can trust, then use it to test what actually works rather than guess. If that is the order you want to do things in, let’s talk.