Getting Googlebot to index more pages on large sites

Looking for ideas around how to get Google to index more pages across large sites? 

Hello. Thanks for listening to SEO tips today.

This is tip comes from Oliver Mason – who wrote a post about an alternative approach to XML sitemap and other non-conventional ideas to get Google to index more of your pages if you work on large sites. 

I have not tried these yet myself, but I was intrigued by the quote he had from Barry Adams about how he’s found that limiting sitemaps to only 10,000 URLs (yes, I know Google says they can crawl up to 50,000 per XML sitemaps) leads to more processing and indexing. And I can’t help but wonder if this would hold true to Bing as well as they have more limited crawling resources.

The other ideas that he recommends require developer engagement and set up but there were so intriguing that I wanted to share them with you.

The first is to set up a dynamic XML sitemap creation and monitoring system to encourage Googlebot to find and index the URLs it has not yet indexed.

And his steps go like this: 

• You create a list of URLs you want Googlebot to crawl. (Those URLs that you’ve seen in Google Search Console that have not yet been indexed – though test first as sometimes updated data in that report is delayed).

• You generate an XML sitemap based on a small set of those URLs (10K maybe? And name it uncrawled.xml or notcrawled.xml so that you remember) 

• You monitor your log files for Googlebot requests. 

• When you notice that Googlebot makes a request to one of the URLs you are monitoring, you remove that URL from that XML sitemap and from monitoring. 

• Then you can add that URL that has now been crawled to one of your existing XML sitemaps by page type.

His other idea I think is also brilliant and would also require developer involvement but the basic idea is this: 

You create and populate an internal links widget which is on every page of your site that dynamically lists a few of the URLs that Googlebot has not crawled. Once you notice that Googlebot has crawled the URL that you’re monitoring, it’s dropped from the widget. 

So if you work on a large site and have the ear of your developers, I think you should give these a try. I’m going to see if I can work them into client projects. And if you do succeed in implementing one of these, drop me a note and let me know how it goes.

Thanks for listening. Come back tomorrow for another SEO tip.