Don’t trust the data

Dixon says: “You need to think about what you are trying to measure.

Buyer behaviour is changing dramatically with the advent of AI. Users are going to be sending out their little AI bots to ask questions on your website instead of going there themselves.

If you're trying to buy a vacuum cleaner, you might ask, ‘What's the best vacuum cleaner for pet hair?’ Of course, Shark, Dyson, and Hoover all want to be in that list of recommendations, but it's the AI that's going to do the investigating.

Potentially, an AI is going to go and find these brands, have a look at those brands, compare the information about pet hair, and then come back and provide an answer to the user that says, ‘X brand is slightly better than Y brand for this particular type of pet hair.’ At that point, the user doesn't need to click on any of the websites, but they will still buy a Shark, a Dyson, an Electrolux, or whatever it may be. They'll have made their decision.

All of the metrics that we've been using for the last 20 years have measured the visitors that come to the website. That's been a key performance metric: has your visitor come from search, from pay-per-click, from direct, affiliate, etc.? That's been the mentality, but that doesn't really work in an AI-driven world.

Firstly, that’s because AI is doing the search for you. Secondly, the AI doesn't typically trigger a visit on most web analytics systems. Most web analytics systems are JavaScript-based: the web page loads, it triggers a JavaScript call, and that will record the visitor. However, LLMs are really lazy when it comes to crawling the site. They just want the text. If the text doesn't appear, they can't be bothered to call the JavaScript.

Often, it won't even come up as a click in your systems, so you're going to have to change the way you measure success.”

Is it the LLMs that are making the decision to visit your website?

“Absolutely. We don't need to think too deeply to realise that is what’s happening. We've probably all done it ourselves, even if we haven't realised it. We’re using our phones, interacting on Facebook, there’s Grok on Twitter – and you’re asking a question of an AI machine.

All of the tools and methodologies that I've heard from most people seem to assume that you're still trying to get traffic to the website. The truth of the matter is that the human is trying to get an answer to a question, and the answer to the question doesn't necessarily mean going to the website.

It might mean sending their little bot out to the website, even if that's not what they think they’re doing. In that case, their decision is made prior to any visit to a website.”

Is it a significant percentage of traffic that is being affected by AI search?

“It depends on the demographic and the kind of person that you're talking to. Younger people and students have been using AI models for their homework for quite a while. They're very comfortable using ChatGPT, Gemini, etc., and they probably use it considerably more than older demographics do. Then, there are demographics like the SEO industry that are ahead of the curve as well.

In the middle, there's still a lot of learning for individuals. What is clear to me, though, is that this stuff is coming through to people even though they don't realise it. They're using their phones, and it’s built into the operating system. They don't know that they are using an AI. They're just choosing to ask their question on whatever device is in front of them, or using whatever methodology is in front of them. They don't necessarily know whether it's using ChatGPT, Gemini, or a traditional search engine.

They're changing their behaviour, and they're changing their behaviour because they're getting their answer quicker, and more in depth. They don't have to go to all of those 10 blue links and read all that information to make their decision. They just have to ask one question and everything's done for them. That's a significant difference.”

Are there any tools that are doing a better job of measuring visits from AI crawlers?

“Each business needs to really think about what is happening in practical terms, because it's going to be different for different types of businesses.

I run a Software as a Service, and for anybody to buy my service, they're going to have to come to the website and interact with the website. That's not the same for a product-based business like Dyson and Electrolux. Those users could make their decision and then go and buy on Amazon. They could make their decision and go and buy in a store. They might still require the information on the website to make that decision, but they don’t necessarily need to visit the website themselves to find it.

For some businesses, it's still about the visits to the website. For product-based businesses, it's not about that. It's understanding where that user journey decision is being made for the user. Clicks may be completely irrelevant in the future, but that's not to say that the website is irrelevant. It's just that the information is being collected in a way that isn't tracked on a per-visit basis.

You've got to think about your own user journey and make your own decisions as to what's important and what key performance indicators you're going to use. Ultimately, it is hard to track, and I don't think many are doing it very well.

One thing that you can do is go back to one of the old systems we used to use, which was log tracking. Back in the ‘90s and early ‘00s, the JavaScript approach to tracking visitors didn't really happen. We were using log tracking software, and the LLMs will get picked up in that.

Looking at more recent hosting packages, people seem to have forgotten about logs. They're not always there, they’re not always easy to find, and you need a methodology for analysing lines and lines of data. Log file tracking came with a whole bunch of other problems as well, because it was making requests that made it very difficult to understand what was a user and what wasn't. There are problems with all the methodologies out there, but it's a real opportunity to stop and think.

The world is changing. User behaviour is changing. People may be on their phone in a mobile app, or they might be on Facebook and using Meta AI. There is no search per se, and no Google lookup. Google might still power the answer, but it's coming through a completely different set of resources. You might even be asking your television.”

If clicks become irrelevant, how do you measure organic search success?

“I don't know the answer, but I do know that assuming the old systems are the way to go is a mistake.

With Waikay, we're trying to work out the prominence of your brand in LLM responses, in the context of a particular topic or idea, compared to the other brands that are out there. We're trying to record a share of voice within LLM responses, and that's one measure that we're working on.

The other is how well the LLM response reflects what your website says. We use vector analysis to look at the topics that the LLM says are important to your business, or what it knows about your business in the context of a particular topic. Then, we compare that to the topics that you're talking about on the website. Is it accurate? Are there hallucinations? Then, we turn that into a score between 0 and 100.

It's an interesting methodology, but it's a predictive methodology. It's not easy to tie that back to your marketing budget. We've got all the problems that you had in the '60s and '70s, where you know that 80% of your advertising budget is wasted; you just don't know which 80%.

The mistake people are making is that they have something in place, but it’s dependent on trying to track the click back to the original data source or where the money was spent. That's already flawed, and it could be incredibly flawed for some businesses.”

Should each business try to develop its own multi-attribution model, incorporating visibility within generative search as part of that?

“I don't think so. You should be looking at the qualitative response that you get from generative search. In other words, it's the words on the page that come back.

When you say, ‘What's the best vacuum cleaner for dog hair?’, the best thing to measure is the quality of that response and how well your brand is represented within it. Then, you want to consider how many people are generally using that channel to give you some idea of where people are convinced.

They will be convinced by that answer. If the AI says, ‘X brand is markedly better than the others,’ they're sold, and you can't necessarily track that individual person to a sale. It's the same as putting up a big banner on the M4 as you drive into London, seeing an uplift, but none of them actually scanned the QR code on the banner.

That purchase decision is being made in that AI result, and they're either going to buy from you or your competitor. All you can do is track your share of voice in that AI result.”

Are stakeholders receptive to the idea of using brand visibility within AI search as a new metric for success?

“I don't think they are very receptive at the moment.

Particularly at the CMO level, marketing directors want a dashboard. They've got £100 to spend, and they want to be able to spend £4 over here, £17 over there, £24 over there, and then see the return on investment of each of those pots of money. That's how they want it to work, but it doesn't make it so.

You can wish that's the way the world is, but it never was anyway. Word of mouth has never been properly tracked, and word of mouth is the most powerful marketing tool for many brands.”

Dixon, what’s the key takeaway from the tip you shared today?

“Don't trust the stats.

Don't trust the data that you're seeing on the screen, especially if that data is from systems that were built before AI came along”

Dixon Jones is CEO at InLinks and Waikay. Find out more over at Waikay.io.