The 2026 Robots.txt Playbook: How to Allow AI Search Bots While Blocking AI Training Crawlers
Your robots.txt file might be costing you AI visibility and you would never know. A Hostinger analysis of 66.7 billion web crawler requests revealed a striking pattern: websites are increasingly welcoming AI search bots while blocking AI training crawlers, and the gap is widening dramatically. OAI-SearchBot (OpenAI’s search crawler) now has 55.67% website coverage. Meanwhile, GPTBot (OpenAI’s training crawler) dropped from 84% coverage to just 12%. This is the new robots.txt reality. AI companies have split their crawlers into separate user agents for search and training, and your robots.txt needs to reflect that distinction. Block the wrong bot and you disappear from ChatGPT’s search results. Allow the wrong bot and your content feeds someone’s training data for free. ...