Web intelligence lead Stephen Burns outlines how search optimization for Google and visibility for artificial intelligence models have diverged into distinct tasks. The findings originate from an interview on the Odys Podcast regarding data collection practices at the Common Crawl Foundation.
Crawling Metrics and Technical Barriers
Common Crawl processes approximately 2.3 billion web pages each month. These datasets form the training foundation for large language models including GPT, Claude, and Llama. Unlike Google, which relies on PageRank, the foundation evaluates site importance through Harmonic Centrality. This metric measures how close a domain sits to the core of the global network. Higher centrality scores increase crawl frequency and the likelihood of content entering AI training sets.Default configuration settings on content delivery networks block AI crawlers on roughly 35 percent of websites. Site owners often remain unaware of these restrictions. Children’s Hospital Los Angeles serves as a documented case.
The facility does not appear in AI responses regarding pediatric leukemia treatment because its CDN settings prevent crawler access. Burns notes that such configuration errors extend beyond technical limitations and may affect public information access.
Site administrators can verify crawler access by reviewing robots.txt files, CDN configurations, server logs, and CCBot activity records. Fast HTML delivery improves crawl efficiency, as many AI bots process JavaScript with limited capability. Serving pre-rendered markup ensures content reaches training datasets.
Publishers operating in non-English languages should maintain English versions of their materials. English iterations increase inclusion rates in English-language models. Newly launched domains start with lower Harmonic Centrality scores.
Backlinks from established websites accelerate network proximity growth. Regular monitoring through Common Crawl tools allows administrators to track inclusion status.