Reddit is now blocking major search engines and AI bots — except the ones that pay

Reddit is ramping up its crackdown on web crawlers. Over the past few weeks, Reddit has started blocking search engines from surfacing recent posts and comments unless the search engine pays up, according to a report from 404 Media.

Right now, Google is the only mainstream search engine that shows recent results when you search for posts on Reddit using the “site:reddit.com” trick, 404 Media reports. This leaves out Bing, DuckDuckGo, and other alternatives — likely because Google has struck a $60 million deal that lets the company train its AI models on content from Reddit.

“This is not at all related to our recent partnership with Google,” Reddit spokesperson Tim Rathschmidt says in a statement to The Verge. “We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”

Last month, to enforce its policy against scraping, Reddit updated the site’s robots.txt file, which tells web crawlers whether they can access a site. “It’s a signal to those who don’t have an agreement with us that they shouldn’t be accessing Reddit data,” Ben Lee, Reddit’s chief legal officer, told my colleague Alex Heath in Command Line.

In a statement to The Verge, Microsoft spokesperson Caitlin Roulston said, “Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models,” adding that Bing stopped crawling Reddit when the platform updated its robots.txt file on July 1st.

It’s a bold move for a massive website like Reddit to block some of the most popular search engines, but it’s not all that surprising. Over the past year, Reddit has become more protective of its data as it looks to open up another source of revenue and appease new investors. After making its API more expensive for some third-party developers, Reddit reportedly threatened to cut off Google if it didn’t stop using the platform’s data to train AI for free.

With AI chatbots filling the internet with questionable content, finding things written by a fellow human has never been more important. I, like many others, have started appending “Reddit” to many of my searches just to get human answers, and it’s pretty frustrating to know that I’ll now only be able to do that on Google (or search engines that rely on it) — especially when I do many of my searches on Bing.

Update, July 24th: Added a statement from Reddit.

Update, July 25th: Added a statement from Microsoft.