Google now places AI-driven responses at the top of its search results, potentially deterring users from visiting the websites whose data fuels these answers. However, many website owners feel they can’t afford to block Google’s AI from summarizing their content.
The challenge is that the same tool Google uses to generate AI responses also crawls the web for search results. Blocking this tool could diminish a site’s visibility on Google, similar to how some competitors’ AI tools have been excluded from websites.
Google’s dominance and publisher challenges
Google’s dominance in search, recently deemed an illegal monopoly by a US federal judge, gives it a significant edge in the growing AI landscape. This power imbalance leaves publishers with a tough choice: either allow their content to feed AI models—potentially making their sites obsolete—or risk losing visibility on Google, a key traffic source.
Joe Ragazzo, publisher of Talking Points Memo, describes it as an existential crisis for companies: “You drop out, and you die immediately, or you partner with them, and you probably just die slowly because eventually, they’re not going to need you either.”
AI overviews and their impact on website traffic
Google’s AI Overviews, summaries that appear at the top of search results, are part of its push to deliver higher-quality information and opportunities to users. Since its inception, Google has crawled millions of websites to build a vast index, making it difficult for even well-resourced companies like Microsoft to compete.
Emerging AI competitors
Generative AI has birthed a new wave of firms offering search products that deliver concise, AI-generated answers. But before these challengers can take on Google, they must first crawl the web—a daunting task. Website owners lose money, processing power, and storage when their sites are crawled, so they often provide guidelines to bots. Google and Microsoft’s Bing typically enjoy the most access because they drive significant traffic to sites.
Despite the AI boom sparking deals between media companies and startups, Google has been notably absent from such agreements. Insiders say Google has quietly communicated to publishers that it’s not interested in negotiating, except possibly for a $60 million deal with Reddit. This deal gives Google access to valuable data for its AI while boosting Reddit’s visibility in search results.
The risk of blocking Googlebot
In contrast to other AI companies, iFixit’s relationship with Google is “much more tenuous,” says Kyle Wiens, CEO of the website that offers free online repair tips. “I can block ClaudeBot from indexing us without harming our business,” he explains, referring to the bot from AI startup Anthropic. “But if I block Googlebot, we lose traffic and customers.”
Ragazzo emphasizes that publishers must take control of their futures and avoid over-reliance on any single tech platform, including Google. “We believe the only way to create a publication that can endure multiple eras is to develop genuine relationships with readers,” he says.
(Tashia Bernardus)