How your website's crawl budget can impact Google crawling

Hello. Thanks for listening to SEO tips today.

Google released another great SEO mythbusing video about crawl budget that is a “must watch”. I re-watched it multiple times.

Here are my crib notes:

They talked about how it’s more accurate to speak of the two terms:
- Crawl rate – How much stress Google puts on your server before it crashes.
- Crawl demand – Google needs to be conservative with its resources, but if it notices that the demand for your content increases, it will increase the amount it’s crawling your site. Google is focused on when the content was last changed and needs to keep track of how often content/information is changing (the change frequency).
Only large sites need to be worried about crawl limits (millions of URLs) unless you have a flaky server set up.
If you see a wave of Google increase then decrease in your log files, it’s Googlebot testing your server capacity.
Google explains that excluded URLs in the Google Search Console report are URLs that Google has not found valuable.
Network requests made during the rendering phase will also impact your crawl budget. To solve this, you should allow Google to cache your resources. FYI. If your API calls
GET requests, Google can’t cache them, and it will impact your crawl budget. They are trying to cache as aggressively as possible, and they might ignore caching headers.

And related to keeping that flaky server, up, Gary Illyes from Google stated that you really need to fix pages that take a lot of CPU

Here’s his tweet:

So that’s your tip for today, if you haven’t watched Google’s SEO Myth Busting video related to crawl budget, you should check it out.

Thanks for listening. Come back tomorrow for another SEO tip.

? Listen to the previous tip: Can you trust SEO studies

? Subscribe and play on your favorite player

How your website’s crawl budget can impact Google crawling