The full AEO playbook: mining AI-bot 404s, rewriting 12,000 TLDRs, guarding SEO, and a control-group re-measurement. Raw growth: 37x. The part our work can claim: about 2x.
Loved this breakdown of AEO in practice. The “your server is the sensor” mindset and using 404s as a roadmap are especially sharp. The honest accounting of the 37x headline into a ~2x true lift, with a proper control, is rare and really valuable.
Also really liked the distinction between “getting seen” and “getting chosen” and how you tied that to TLDR design, grounding, and deeper Q&A. The demand driven page creation stats vs bulk generation were eye opening. This is one of the most concrete and intellectually honest case studies on AI oriented growth I have read in a while.
The question we would genuinely like to hear answers to: what does your AEO dashboard actually measure? Ours fixated on bot request counts for months, and the number that mattered most (bot volume flat, human sessions 37x) was the one we almost did not chart.
If you run Cloudflare, Vercel, or Fastly, here is the one tactic from this piece you can try this week: read your AI-bot 404 log. It is a content roadmap almost nobody opens.
And the honest caveat one more time: of the 37x, the tide was 3.5x and our work was worth about 2x. If you measure your own AEO against an on-domain control, we would love to compare notes.
Using 404 data to identify unmet demand and content gaps is a clever approach for deciding what to build next.
I’m curious whether you can identify which ChatGPT model or product surface generated each request, and whether results vary across them. Also, how do you account for major model updates that may change retrieval or citation behavior independently of your own optimizations.
On surfaces: the logs identify the bot, not the model. Cloudflare separates ChatGPT-User (live fetches during a user's prompt), OAI-SearchBot (search indexing), and GPTBot (training), and the same split exists for other operators. So we can tell what function a request served and who sent it, but not which underlying model made the call; user agents do not carry that. Across operators the differences are large: ChatGPT dominates our AI referrals (everything else was under 1% of AI-referred sessions in the study window), and during a firewall incident this summer, Copilot citations held steady while ChatGPT sessions fell, which is exactly how we isolated the cause.
On model updates: that is what the on-domain control is for. An update that changes retrieval or citation behavior hits our treated and untreated pages alike, so the treated-to-control ratio nets it out, the same way it nets out platform growth. The honest limitation is an update that interacts with the treatment itself, say a new model that happens to favor the exact format we rewrote toward. That we cannot separate from our own effect, and the paper lists it among the threats to validity.
Loved this breakdown of AEO in practice. The “your server is the sensor” mindset and using 404s as a roadmap are especially sharp. The honest accounting of the 37x headline into a ~2x true lift, with a proper control, is rare and really valuable.
Also really liked the distinction between “getting seen” and “getting chosen” and how you tied that to TLDR design, grounding, and deeper Q&A. The demand driven page creation stats vs bulk generation were eye opening. This is one of the most concrete and intellectually honest case studies on AI oriented growth I have read in a while.
Thank you! Glad you loved it!
The question we would genuinely like to hear answers to: what does your AEO dashboard actually measure? Ours fixated on bot request counts for months, and the number that mattered most (bot volume flat, human sessions 37x) was the one we almost did not chart.
If you run Cloudflare, Vercel, or Fastly, here is the one tactic from this piece you can try this week: read your AI-bot 404 log. It is a content roadmap almost nobody opens.
And the honest caveat one more time: of the 37x, the tide was 3.5x and our work was worth about 2x. If you measure your own AEO against an on-domain control, we would love to compare notes.
No Substack account? We are also on Reddit: https://www.reddit.com/r/glasp/
Thank you for sharing your learnings!
Using 404 data to identify unmet demand and content gaps is a clever approach for deciding what to build next.
I’m curious whether you can identify which ChatGPT model or product surface generated each request, and whether results vary across them. Also, how do you account for major model updates that may change retrieval or citation behavior independently of your own optimizations.
Thanks Jose, great questions!
On surfaces: the logs identify the bot, not the model. Cloudflare separates ChatGPT-User (live fetches during a user's prompt), OAI-SearchBot (search indexing), and GPTBot (training), and the same split exists for other operators. So we can tell what function a request served and who sent it, but not which underlying model made the call; user agents do not carry that. Across operators the differences are large: ChatGPT dominates our AI referrals (everything else was under 1% of AI-referred sessions in the study window), and during a firewall incident this summer, Copilot citations held steady while ChatGPT sessions fell, which is exactly how we isolated the cause.
On model updates: that is what the on-domain control is for. An update that changes retrieval or citation behavior hits our treated and untreated pages alike, so the treated-to-control ratio nets it out, the same way it nets out platform growth. The honest limitation is an update that interacts with the treatment itself, say a new model that happens to favor the exact format we rewrote toward. That we cannot separate from our own effect, and the paper lists it among the threats to validity.