If you’ve ever wanted to illustrate to your developers why Googlebot needs to be considered a target website audience segment, this post is for you.

Why create a Googlebot persona?

What I’ve found amazing over my 15+ years of working as an SEO is that the unique needs and requirements of Google and specifically Googlebot (Google’s crawler) are not included when web developers build websites. This results in sites that are built in such a way that Googlebot can not crawl them, can not find important pages or sections of the website are never added to the Google index. This post (and the included persona) are my response to what I think is an unmet need in the web strategy process.

Let’s start with a definition of personas. 

Most website projects start with some sort of persona creation activity. In today’s ever personalized world, it’s impossible to create effective marketing unless you know “who” you are talking to.

Personas are a tool used by digital marketing strategists to turn insights around your audience into tactical “snapshots” of who those audience targets are, what their emotional state is when they interact with your website, and what specifically is motivating them to visit your site. (If you are new to personas, Hubspot has a great overview which includes a simple persona creation tool). These personas are usually shared with the developers working on the website project along with use cases and acceptance criteria to help them build a website that will work for your audience. 

Why create a Googlebot “persona”?

Most of my work focuses on enterprise-level sites where technical issues can sink or heavily suppress organic traffic success.  I think some of the confusion stems from the fact that many strategies (and many developers) don’t realize that there are  4 stages to “ranking” in Google which include:

1. Crawling
2. Rendering
3. Indexing
4. Ranking

Most importantly, the sophistication of the technology Google uses during those 4 stages changes quite a bit depending on the stage.

For instance, crawling is (in some ways) still stuck with 1994 technology. Old school HTML links and formatting still really matter. So do internal linking. This is even more so the case if you pivot to Bingbot whose crawling resources are more limited. 

Once you get past crawling,  to indexing, your competition will matter more.  Your content needs to “deserve” to be indexed because it offers a unique perspective.

When you look at the ranking process, Google’s AI algorithms and entity understanding come into play.

Do you see?

Google takes an ever more sophisticated approach as it moves from indexing to ranking.

Increasing your developer’s understanding of crawling with a Persona

My theory is that Googlebot is not included in the persona work because no one has created a persona that encapsulates what Googlebot needs to crawl your site and how it behaves. You have tools for developers like Moz’s Web Developer Cheat Sheet and Google’s videos for developers, but it’s important that you (and your developers) need to also envision what motivates Googlebot and what will limit its crawling.

So, I created a persona for Googlebot and it’s below (you can click on the file to enlarge it):

While a persona should help when your establishing technical requirements the real test to see if you built a site that Googlebot can crawl is to your site like Googlebot. You can set up that type of crawl with a tool like Deepcrawl, ScreamingFrog, or Sitebulb. All of them provide reports of any issues on your site encountered by Googlebot. You can also personalize the persona based on the data around how Googlebot interacts with your specific site and what kind of crawl frequency you’re seeing from Googlebot in your log files.

At WO Strategies, we use a combination of ScreamingFrog and Deepcrawl for our client’s technical SEO audits in combination with ScreamingFrog’s Log Analyzer tool where you can see Googlebot’s actual behavior on your site. 

Feel free to download and share the Googlebot persona and use it for your own SEO projects. I’m working on a Bingbot persona next. Let me know if this persona is helpful in the comments!

PS: Thanks for taking the time out to read my post and geek out about SEO with me!  I get my inspiration for post topics from other SEOs and in-house marketers struggling with SEO strategy and implementation questions, so if you like this post, please…

1. Leave a comment
2. Share it with your network
3. Follow me for future posts