OpenAI and Anthropic are ignoring an established rule that prevents bots scraping online content

OpenAI and Anthropic have stated publicly that they respect robots.txt and blocks to their specific web crawlers, GPTBot and ClaudeBot.

However, according to TollBit’s findings, such blocks are not being respected, as claimed. AI companies, including OpenAI and Anthropic, are simply choosing to “bypass” robots.txt in order to retrieve or scrape all of the content from a given website or page.

A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions “into account each time we train a new model.” A spokesperson for Anthropic did not respond to emails seeking comment.

Robots.txt is a single bit of code that’s been used since the late 1990s as a way for websites to tell bot crawlers they don’t want their data scraped and collected. It was widely accepted as one of the unofficial rules supporting the web.

With the rise of generative AI, startups and tech companies are racing to build the most powerful AI models. A key ingredient is high-quality data. The thirst for such training data has undermined robots.txt and the unofficial agreements supporting the use of this code.

OpenAI is behind the popular chatbot ChatGPT. The company’s largest investor is Microsoft. Anthropic is behind another relatively popular chatbot, Claude. It’s largest investor is Amazon.

Both chatbots serve up answers to user questions in the tone of a human. Such answers are only possible because the AI models they are built on include massive amounts of written text and data scraped from the web, much of it under copyright or otherwise owned by creators.

Several tech companies last year argued to the US Copyright Office that nothing on the web should be considered under copyright when it comes to AI training data.

OpenAI has struck a few deals with publishers for access to content, including Axel Springer, which owns BI. The US Copyright Office is set to update its guidance on AI and copyright later this year.

Are you a tech employee or someone else with a tip or insight to share? Contact Kali Hays at khays@businessinsider.com or on secure messaging appSignal at +1-949-280-0267. Reach out using a non-work device.