Watier Ong Strategies

Main Nav

  • Home
  • About
    • Our Results and Case Studies
    • Clients
    • Approach
    • Values
    • Testimonials
    • Katherine Watier Ong Speaker Info
    • Published Thought Leadership
  • Services
    • AI SEO Services
    • Technical SEO Services
    • Traffic Loss Investigation
    • SEO for Associations and Academic Journals
    • Video Optimization
    • Voice Search and Wearable Tech Optimization Services
    • White Label SEO Audit Services
    • Google Analytics Audits
  • Training
    • Member Login
    • Specialized Technical SEO Training
    • SEO Coaching
    • SEO Bootcamp Course Registration
    • SEO Bootcamp Course
      • SEO Bootcamp Course questions
  • Podcasts
    • Daily SEO Tips
    • Digital Marketing Victories
  • Digital Marketing Resources
    • Newsletter Signup
    • WO Strategies Recommendations
    • Blog
    • Digital Marketing How-To Videos
    • Join Our Free Content Library Membership
  • Contact
You are here: Home / Podcast Episodes / Getting Googlebot to index more pages on large sites

Getting Googlebot to index more pages on large sites

August 5, 2020 by Katherine Watier-Ong Leave a Comment

Getting Googlebot to index more pages on large sites

https://wostrategies.com/wp-content/uploads/2020/08/GettingGooglebottoindexmorepagesonlargesites.mp3

Looking for ideas around how to get Google to index more pages across large sites? 

Hello. Thanks for listening to SEO tips today.

This is tip comes from Oliver Mason – who wrote a post about an alternative approach to XML sitemap and other non-conventional ideas to get Google to index more of your pages if you work on large sites. 

I have not tried these yet myself, but I was intrigued by the quote he had from Barry Adams about how he’s found that limiting sitemaps to only 10,000 URLs (yes, I know Google says they can crawl up to 50,000 per XML sitemaps) leads to more processing and indexing. And I can’t help but wonder if this would hold true to Bing as well as they have more limited crawling resources.

The other ideas that he recommends require developer engagement and set up but there were so intriguing that I wanted to share them with you.

The first is to set up a dynamic XML sitemap creation and monitoring system to encourage Googlebot to find and index the URLs it has not yet indexed.

And his steps go like this: 

  • You create a list of URLs you want Googlebot to crawl. (Those URLs that you’ve seen in Google Search Console that have not yet been indexed – though test first as sometimes updated data in that report is delayed).
  • You generate an XML sitemap based on a small set of those URLs (10K maybe? And name it uncrawled.xml or notcrawled.xml so that you remember) 
  • You monitor your log files for Googlebot requests. 
  • When you notice that Googlebot makes a request to one of the URLs you are monitoring, you remove that URL from that XML sitemap and from monitoring. 
  • Then you can add that URL that has now been crawled to one of your existing XML sitemaps by page type.

His other idea I think is also brilliant and would also require developer involvement but the basic idea is this: 

You create and populate an internal links widget which is on every page of your site that dynamically lists a few of the URLs that Googlebot has not crawled. Once you notice that Googlebot has crawled the URL that you’re monitoring, it’s dropped from the widget. 

So if you work on a large site and have the ear of your developers, I think you should give these a try. I’m going to see if I can work them into client projects. And if you do succeed in implementing one of these, drop me a note and let me know how it goes.

Thanks for listening. Come back tomorrow for another SEO tip.

Filed Under: Daily SEO Tips

About Katherine Watier-Ong

Katherine is an online marketing instructor, public speaker, and a consultant who has over 20 years of experience in communications strategy and online delivery of communications messages, including thirteen years of SEO, social media, SEM, and web analytics management.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search This Website

Topics

Get a Free SEO Consultation

For a 30-minute strategy call book a call

Find Katherine online

  • My Resume
  • Linkedin
  • Bluesky
  • Pinterest
  • Facebook
  • Moz.com
  • Quora
  • YouTube
  • Podcast Feed

Subscribe to our YouTube Channel

Recent Posts

  • What AI Content Policies Should Your Organization Have?
  • Google’s AI Overviews & Perspectives
  • Why .govs need an SEO strategy
  • Google Analytics 4 – The 411
  • A look back at 2021, and some 2022 optimism.

Check out our podcast!

Daily SEO Tips

155 episodes

Digital Marketing Victories - All Episodes

32 episodes

Let’s Connect

  • Facebook
  • LinkedIn
  • Pinterest
  • RSS
  • Twitter
  • YouTube

Resources

  • Free list - 300+ AI Tools to Boost Your Marketing Output
  • AI Workflows for SEO

Terms of Service and Cookie Policy

Terms and conditions | Cookie policy

Monthly Newsletter

Sign up for our monthly newsletter to get updates on all things SEO.

Industry Presence

Find out more about Katherine’s contributions to the digital marketing industry.

Alignable Winner 2023 – Local Business Person of The Year 2023 

Digital Marketing Victories Podcast

  • Listen to Digital Marketing Victories Podcast

Get a free SEO consultation

For a free consultation call 202-930-1744

Federal Capabilities Statement

Download our federal capabilities statement

Capabilities Deck

Download our capabilities deck

© 2025 Watier Ong (WO) Strategies LLC - A WASHINGTON, DC DIGITAL MARKETING FIRM

Privacy Policy