In the first half of 2026, ChatGPT-User accessed pages that had been disallowed via robots.txt in 54% of recorded cases, according to the platform TollBit. Among European sites in the sample, this was the highest rate: Bytespider stood at 48%, and PerplexityBot at 42%. Overall, across European and North American sites in TollBit's sample, about 15% of AI bot requests hit pages with an explicit robots.txt disallow.
It is important to note that this is not about all websites on the internet, but about traffic that TollBit observed within its own network. The company defines a bypass as a successful request to a URL that the publisher explicitly disallowed for the respective bot. Therefore, the statistics show that requests to disallowed pages occurred, but they do not by themselves establish the cause of each request or rule out user-agent spoofing.
The peculiarity here is related to the very purpose of ChatGPT-User. According to OpenAI's documentation, this agent may access a page when a user asks ChatGPT or a custom GPT a question that requires fetching the page's content. OpenAI specifically notes that the agent is not used for automatic crawling and that, because the request is initiated by the user, robots.txt rules may not apply to it. This differs from the OAI-SearchBot tool, which determines whether content can be used in ChatGPT search responses, and GPTBot, which is designed to crawl pages that may be used to improve base models.
Thus, robots.txt is not a technical protection against a page's content being retrieved. It conveys the publisher's instructions to automated agents, but it does not prevent the server from delivering the file upon request. If a resource truly must not be provided to external agents, technical access control mechanisms are needed—authentication, authorization, network rules, or server-level blocking. For teams managing web resources, this means they need to separately control site visibility for search and AI services and actual data access.
