google.com/goto: Google's anti-scraping update
summary
Google is replacing direct organic result links with opaque redirect links to prevent automated scrapers from bulk-extracting URLs. By requiring a separate request to resolve each link, Google increases the operational cost and visibility of scraping activities. This change forces developers to perform more requests, making it easier for Google to detect and block bot behavior. The shift is part of a larger effort to combat AI crawlers and SEO tools that harvest search data at scale.
key points
- The new /goto links use custom encoding that cannot be decoded offline, requiring a request to Google's servers to retrieve the final destination from the Location header.
- This mechanism increases the noise and latency for scrapers, as each single result must now be resolved individually rather than being parsed from HTML.
- Google is combining this redirect system with other restrictions, such as theremoving of the num=100 parameter, to make naive scraping increasingly difficult.
who should read it
Software developers and SEO professionals who rely on automated search engine result page data for their tools and indexes.
discussion skeptical
consensus
Commenters generally agree that Google Search has deteriorated into an ad-driven, 'enshittified' experience that prioritizes corporate interests over user utility. While many view Kagi as a viable, privacy-focused alternative, a significant divide exists regarding its pricing, with some seeing $10/month as a small price for privacy and others finding it prohibitively expensive. A minority of users report that even Kagi's search quality and speed have recently declined, leading them back to LLMs or DuckDuckGo.
top insight
One user provides a specific prompt to trick Google's AI-based search into behaving like the 2004 version of the engine by commanding it to return ten relevant links with short snippets and avoid non-matching pages. This reveals that the underlying data and capability for a high-quality, simple search experience still exist within Google's infrastructure, but are intentionally suppressed in favor of the current AI and ad-centric UX.
Generated by an LLM from the article and thread. Check the sources before relying on it.