1, 0, 2 out of 15?

There is a new kind of counter in the store now. A man with a problem no longer walks the aisles of search results; he asks a chatbot, which answers like a clerk, naming two or three products the chatbot trusts and saying nothing about the rest.

Being one of the products the chatbot names matters more every year. SparkToro found that 68% of US Google searches this year ended without a click to any website, and one indie founder traced about 48% of his users to ChatGPT. When the chatbot leaves your product out, the reader never hears of your product.

In September I asked three chatbots the same five questions on three separate days. Out of fifteen chances each day, the chatbots named my app once, then not at all, then twice, which is not much of a reputation, but the way the score moved taught me a lot.

The app is ZeroPhase. ZeroPhase lays the Navy's 26-week BUD/S prep guide, a good guide buried in a long PDF, out one day at a time. What follows is how I checked, why I first blamed Google for leaving my pages out when Google had indexed the pages all along, and how to run the same check without fooling yourself the way I did.

The ten-minute check

The check is plain work. You write down five questions the way a stranger would word the trouble, without your product's name, something like "Is there an app for the 26-week Navy SEAL prep guide?" You put each question, word for word, to ChatGPT, Claude, and Google AI Mode, and write down every product each chatbot names, in order. You mark each of the fifteen answers yes or no, date the page, and ask the same questions again two weeks later, because the value of the check is in the repetition.

A notebook and a row of browser tabs are enough for the job. You can also hand the check to an AI agent, which asks the fifteen questions faster than you can and files the answers in your repo as a marketing to-do. I used an agent, mostly for the speed. The agent keeps a log in the ZeroPhase repo with every run dated.

The diagnosis I got wrong

On September 3, ChatGPT named ZeroPhase first when asked whether an app existed for the guide, and no other answer mentioned ZeroPhase. I took the mention as a beginning and wrote five new pages, one for each question. On September 17 the count was zero.

I thought I knew why: a site: search for zerophase.app showed only my homepage, and Google's documentation says a page must be "indexed and eligible to be shown in Google Search with a snippet" to appear in AI Overviews or AI Mode. I decided Google had not indexed my new pages.

Before I acted on the guess, I asked Google directly through the URL Inspection tool in Search Console. Google had indexed forty of my 69 pages, including every page written for the five questions; the site: search had undercounted, and I believed the undercount because the undercount agreed with me. Getting into Google's index had never been the trouble; getting chosen by the chatbots was. As Kevin Indig put the difference, a crawled page only sits "in a pool of sources to be considered for citation."

On September 29 the count rose to two: ChatGPT named ZeroPhase first again, and Claude named ZeroPhase too, though I had written nothing new. The lost mention and the returned mention came from the same page. Machines are so moody nowadays.

Why the count moves when nothing changes

Glen Allsopp at Detailed asked 1,300-odd buying questions every day for 28 days, more than 70,000 answers. On ChatGPT, the brands named on most days changed about 13% from day to day, while the long tail of small brands changed 78%.

An app that brings in $60 a year lives far out in the long tail, and my 1, 0, and 2 was the long tail behaving as Allsopp describes. One run tells you as much as one day's weather tells you about a season; the count across runs is the number to watch. I nearly wrote that my five new pages had cost me a mention, a confident sentence about nothing.

What to write when the count stays low

An answer engine lifts a piece of a page, not the whole page, so a page with one clear fact at a time is easier to carry off. Naming the source of each fact right in the sentence helps too: the GEO paper (KDD 2024) found that quotes, statistics, and cited sources each raised a page's visibility by 30 to 40%, while stuffing in keywords did nothing at all. Two more habits help: answering the question at the top of the page, and showing up on the sites the answer engines read. Profound's study of 680 million citations found Reddit among the top three sources on every major answer engine.

This week I added a page for a sixth question, "My PST is less than 26 weeks away. Can I still follow the guide?", and every claim on the new page quotes the guide. The new page scored zero out of three on launch day, before Google had seen the page, and the zero goes in the log.

In short

An answer-engine agency on TrustMRR shows $160,821 in revenue over the last 30 days (as of September 29), so people plainly pay for the answer. You can learn most of the answer yourself in ten minutes, if you ask the same questions again and judge your product on the count, not the latest score.

After three runs, my count stands at 3 mentions out of 45 (not exactly billboard-worthy). I will run the check again in mid-October and report the new count, whichever way the count goes.go