Reddit CEO Steve Huffman is defending Reddit’s decision to block companies from scraping the site without an AI agreement. This decision has resulted in search engines other than Google no longer listing recent Reddit posts, as Reddit updated its Robots Exclusion Protocol to block bots.
This move aligns with Reddit’s stance on preventing the misuse of public content. Despite this, OpenAI announced SearchGPT, which can still show recent Reddit results, indicating ongoing negotiations and adaptations within the AI industry.
This blocking of free scraping follows Reddit’s year-long effort to stop AI companies from profiting off its content without compensation. As part of this initiative, Reddit began charging for API access, which led to the shutdown of several third-party Reddit apps due to high costs.
Huffman confirmed in an interview with The Verge that only Google has a current agreement with Reddit, reportedly worth $60 million a year. The financial details of Reddit’s deal with OpenAI remain undisclosed, but these agreements ensure Reddit has control over how its data is used.
Huffman criticized companies like Microsoft, Anthropic, and Perplexity for not negotiating in good faith regarding data usage. He claimed Microsoft previously used Reddit data without proper agreements and sold it through the Bing API to other search engines.
Huffman expressed frustration over the difficulty of blocking these companies, indicating ongoing tensions between Reddit and major tech firms over data privacy and usage rights.
Microsoft responded by asserting that it respects the robots.txt standard and does not use content against a site’s wishes for its generative AI models. However, Microsoft VP Jordi Ribas highlighted that Reddit’s changes favor Google, impacting competition.
Huffman also referenced comments from Mustafa Suleyman of Microsoft AI, who suggested that content on the open web has traditionally been considered fair use, a position Huffman argues against, emphasizing that not all internet content is free for AI companies to use.
Reddit has not disclosed the financial requirements for scraping agreements with Microsoft, Perplexity, or Anthropic. Reddit spokesperson Tim Rathschmidt mentioned ongoing discussions with multiple search engines and an openness to partnerships.
As Reddit seeks new revenue streams to achieve profitability, it faces challenges, including user protests against API rule changes and the broader debate over AI’s use of online content. Reddit’s reliance on user-generated content adds complexity to these issues, highlighting the need for clear agreements and policies in the evolving AI landscape.
