Optimizing for Google Scholar vs Googlebot

Are you working on a Google Scholar site that could be more optimized for Google search? Then this post is for you.

Over the past year, I’ve been working with professional journals that are in Google Scholar but need to improve their visibility in Google Search. My focus has been to ensure that I don’t recommend changes that impact their Google Scholar visibility, but definitely enhance their discoverability in Google search.

It turns out that documentation on the difference in those crawlers and algorithms is hard to find. So after a ton of research, I pulled together a “cheat sheet” – which is below. I hope it helps other SEOs working on journal sites.

Purpose of Optimizing for Google

Google scholar is a sub-search engine of Google that is solely focused on scholarly literature. From the custom search engine, you can search across many disciplines and sources. Types of literature include articles, theses, books, abstracts, and court opinions. Sources include academic publishers, professional societies, online repositories, universities, and other web sites.

Anyone that has published academic articles can be included as an individual in Google scholar, in fact, here’s my profile. To be included as an institution, Google recommends being hosted by one of their Google Scholar partners.

Difference between Google Search and Google Scholar

The difference between Google and Google Scholar is that Google Scholar focuses on the scholarly literature available on the Internet. Google, on the other hand, has a broader scope and retrieves resources regardless of where online they come from.

The other big difference is that the estimated searches per month for Google Scholar is the number of people using each engine per month. As you can see from the Ahrefs data below, there are just under 3 million searches on Google Scholar each month:

Google.com has 201 million searches per month:

Optimizing for the larger Google traffic is a great opportunity to get you more traffic to your site.

Google Scholar crawl frequency

But, unlike Googlebot, Google Scholar only crawls new sites/content every 6-9 months!

With a slow crawl schedule, it’s more critical than ever that journal sites intensely focus on ensuring that Googlebot can find their information as that bot crawls more frequently. In many ways, this should mirror your Bing SEO strategy, as they are also limited in their crawl frequency.

Duplication on Google Scholar vs Googlebot

It seems as though Google Scholar can group files based on DOI (Digital Object Identifier). This means there’s no additional work to ensure that Scholar knows about the variants of the files (pdf, image pages, etc.) are a part of the same article if you have provided proper references in the metadata in the root file. All variants combine to increase the citation count.

Despite so many similarities mentioned earlier, Googlebot works differently. It sees all of those files as individual objects. Distinct URLs that are not grouped, from Googlebot’s perspective, don’t acquire a “group ranking” metric. To ensure that Googlebot sees those variants as a part of the whole, the following is recommended (based on Googlebot/Google Webmaster Help files best practices):

• Resolving any server issues
• Rel=canonical tag from .pdf version (and other versions) to the main article once it’s a year old.
• Ensuring that all the variants of the article URLs are captured via XML sitemaps. Ideally, one for article variants that are older than one year, and one for the current year and split by variant type to allow us to troubleshoot any indexing issues.

Here’s how Google Scholar is the same as Google Search/Googlebot

Google Scholar crawling – just like Googlebot

Google scholar needs to crawl your site like Googlebot to discover the content, and that means a variety of things including that you need to ensure that you are using 301 redirects when moving content. Here’s the Google Scholar help reference:

Since Google Scholar crawls like Googlebot, your navigation needs to conform to what Googlebot can crawl. That means JavaScript navigation and navigation that relies on form submissions/internal search should be converted to something that works with Googlebot and Google Scholar.

Impact of Crawl Errors and Page Speed

Slow crawl speed or crawl errors hurt every aspect of the crawl, and in this case, it will delay Google Scholar finding updates to existing papers:

The Scholar help files even mention using the Google webmaster help references for Googlebot to troubleshoot crawling issues:

Issues like page speed and server errors can impact Google scholar crawling just like Googlebot. Here’s the reference from the Google Scholar help files:

Looking for Additional Guidance? Check out Darcy Dapra

Darcy is the previous Partner Manager for Google Scholar presentations, so she’s a fantastic resource on such matters. In the presentation below, she specifically mentions one-to-one 301 redirects for each URL when migration/changing URLs. She also mentions multiple times the importance of avoiding interstitials for URLs when redirecting.

Darcy recommends getting in touch with the Google Scholar team one month before a move, which is a solid tip. Here’s a summary of what is in her presentation as technical requirements for a site move or URL change:

She also mentions following the references Google Webmaster Forums and Google Webmaster Tools help forums related to site moves around the 32-minute marker.

Here’s another presentation that is a follow-on to Darcy’s presentation. At 14:13 of this presentation, she astutely describes how to optimize digital repositories for both Google and Google Scholar.

Watch the video here.

My Recommendation: Optimize for Google Scholar AND Google Search

Hopefully, you’re now convinced that as a journal site you should optimize your content for larger Google search and you have some sense of the differences between Google Scholar indexing and visibility versus Google search indexing and visibility opportunities.

If you think you could get more traffic by optimizing your site for Google search, drop me a note and I can provide you with more information about how much additional traffic you could receive as a site optimized for Google search.

PS: Thanks for taking the time out to read my post and geek out about SEO with me! I get my inspiration for post topics from other SEOs and in-house marketers struggling with SEO strategy and implementation questions, so if you like this post, please…

1. Leave a comment
2. Share it with your network
3. Follow me for future posts