Are you looking to get your feet wet with SEO testing?

I’ve tried to pull together all of the resources that you’ll need to think through the benefits, risks, and challenges you need to think through as you set up and measure your first SEO test.

Let’s start with:

The benefits of testing for SEO

There are a variety of reasons why you might be interested in launching an SEO testing program and are looking for a tutorial of sorts of how to get started.

Some in-house SEOs see it as a way to get internal buy-in for changes that they see as necessary for improvement, but for which there is internal pushback on implementing. Others are in-house marketers with a high traffic site and they don’t want to make a change that will accidentally tank organic traffic.

Both are valid.

There are two really big benefits to SEO testing.

1. Results from the test can provide justification for further investment in various recommended SEO changes. (This bullet seems obvious)

2. Test results can also help you avoid making changes that would have negative consequences.

I want to provide an example for that second bullet to illustrate the point.

Here’s an example of why you only run SEO experiments with an established testing plan

This example is from the Pinterest engineering team who wrote a post in 2015 about their work with SEO experiments. In that post, they outline that for UX (not SEO) reasons the Pinterest team really wanted to be able to render content client side in JavaScript.

Luckily they just made that change to a limited number of pages and tracked the impact. (Sidenote: Google notoriously has issues indexing JavaScript  – in fact, I worked with a client that lost almost all of their organic traffic because of a development driven move to Angular JS 1).

Because as they tested the JavaScript design, they could see the significant and sustained drop in organic traffic and were able to turn off the experiment and cancel plans to roll out the change to the rest of the site.

Fact: One cannot run a “true” A/B SEO Test

Sorry to disappoint you if that’s what you were looking for.

But let me explain why you can’t.

First of all, you can’t really do a true SEO A/B split test as it would require that you split your users into two groups and you have two identical samples, then you change just one condition on “A” and leave “B” alone and then measure the results.

Here’s an illustration of an A/B test:

That would mean you would need two pages on the same domain with the same Domain Authority and backlink profile, internal link profile, content, etc.

This creates two problems:

1. Creating this type of duplicity might run you into a cloaking penalty with Google (if they index it).
2. Additionally, there’s a really good chance that you couldn’t even get Google to index “A” because it would look like duplicate content.  Which means you can’t even get started running the test as you need both pages indexed and ranked to see if one change over the other is going to positively impact your organic traffic.

Here’s the process for running a statistically significant SEO test on just one page:

1. Measure your site’s traffic for at least 10 days without that keyword in the title tag, and no other changes to the site, or marketing to that page.
2. Change the title tag.
3. Wait until it shows up in Google’s cache (that way you know it’s indexed), then measure the site traffic for at least another 10 days (again no changes or marketing to that page).
4. Change the title tag back, wait until it shows up in the cache again, and measure for at least 10 days.
5. Put the keyword back in, wait for it to be in Google’s cache, and measure again for 10 days.
6. Run a sample t-test to compare the traffic on the days with the cached keyword and without.  Here’s a free tool to help you crunch the t-test data: http://www.usablestats.com/calcs/2samplet

Why so many days of no other changes?

You need a total sample size of at least 40 before you can run a test without worry about skewed data.  You’re switching it back and forth to minimize the impact of seasonality or other outside trends impacting your data. You also need to pay attention to Google’s cached page because that’s the copy that they are evaluating for ranking. You need to start measuring your “days” above from when you see the cached copy.

If the above seems impossible to execute, that’s the point.

There are a few other options.

What sort of SEO testing CAN you do?

Since strict A/B testing is out from a statistical perspective, what kind of testing is left? You can test changes to a randomly selected group of pages and measure their performance vs. their projected performance.

The process would look like this:

1. Identify a set of pages you want to improve
2. Choose the test to run across those pages
3. Randomly group the pages into a control and a variant group
4. Note the date of the Google cached page (the date that Google added the changed pages to their index)
5. Measure the impact and declare the test a success if the variant group outperforms the forecasted performance (and if the control does not)

Because you’re now conducting a test on a group of pages your pages are not statistically identical, you can’t just compare performance across those pages but instead need to forecast the performance of both sets of pages.

If you’re interested in diving into the statistics and math involved in this process, I would encourage you to check out Distilled’s resources which include:

• Predicting the present with Bayesian structural time series [PDF]
• Inferring causal impact using Bayesian structural time series [PDF]
• CausalImpact R package
• Finding the ROI of title tag changes
• Ben Estes’ post about R and analytics forecasting

If all of this seems too challenging, there is always Distilled’s OCN software (which I’ll discuss later in this post).

But let’s assume that you still want to set up an SEO test manually…

Here are some things to keep in mind when setting up your first SEO test

You’ve decided that you’re going to launch an SEO test focused on which tactics are going to drive the organic visitors that stay on your site.

Keep the following in mind when setting up and launching that test:

1. Focus on measuring what’s important

Make sure you’re focused on changes that impact the deep KPIs for your site and business (like lifetime visitor value, and organic purchases) vs. rankings.

With Google’s use of Machine Learning, Artificial Intelligence, and their complex personalized algorithms, you can’t create a test with scientific accuracy about a factor that will impact your rankings. Besides, have you seen the study that shows how much rankings change on an hourly basis? If you’re only tracking ranking improvements, it’s likely you’re going to miss the changes that improve organic clickthrough rate but not rankings. Ultimately, those changes will increase the organic traffic to your site, which is the desired end result!

You also want to make sure that any ranking improvements improve (or keep steady) the time that users are spending on your ranking pages and site; otherwise, your temporary ranking improvements might become longer term ranking declines if Google sees that your high ranking pages are resulting in visitors that bounce back to search.

2. Make sure you’re using good analytics data

As you’re pulling your analytics data to see if there’s a trend, make sure to:

1. Pull only non-paid search traffic.
2. Pull it from a filtered account that removes any spam traffic. Unfortunately, organic keyword driven spam IS a thing.

3. Randomness is important

You need to make sure that you’re not biasing things based your favorite products, an existing promotion, Google algorithm update, or seasonality. Since SEO Testing is Page Oriented – vs CRO (conversion rate optimization)  testing which is user oriented – you need to make the same tactical SEO change across a random group of pages. You can see Distilled’s example below:

4. Change only one thing across those pages

You need to test only one SEO tactic at a time, otherwise, you won’t know which factor resulted in your traffic lift.

5. Testing on stable SERPs is ideal

If you are going to go the route of your own SEO test, then you really want as much outside stability in your control and test group as possible. Testing on keywords that have a lot of SERPs (Search Engine Result Page) volatility is going to skew your data and make it tough to see a test result that is insightful.

As you’re building pages for your test group, see what their rankings look like for their target phrases. If they bounce around a lot in the rankings, along with other people’s pages bouncing in rankings, then those pages would not be ideal for testing.

6. Pick pages that have the potential to move

Here are the facts: Ranking in positions 1 through 6 is hard. There are TONS of factors that go into those rankings (and that’s not even taking into account Knowledge Graph, shopping results, instant answers, etc) which is why it might make more sense to take pages that are ranking 8 through 30 of the SERP for their associated terms. This is an area where a minor change (like a meta title change) might make more of an impact.

7. Set a time frame for your test

Distilled has a few suggestions:

“In deciding how long to run tests for, you first need to decide on an approach. If you simply want to verify that tests have a positive impact, then due to the rational and consistent nature of Google, you can take a fairly pragmatic approach to assessing whether there’s an uplift — by looking for any increase in rankings for the variant pages over the control group at any point after deployment — and roll that change out quickly.”

“If, however, you are more cautious or want to measure the scale of impact so you can prioritize future types of tests, then you need to worry more about statistical significance. How quickly you will see the effect of a change is a factor of the number of pages in the test, the amount of traffic to those pages, and the scale of impact of the change you have made. All tests are going to be different.”

8. Yes, setting up an SEO test might be difficult

Some CMS systems will make it difficult to make changes to an arbitrary group of pages. It might even be hard to gather and analyze the data in a way in which you are sure of the trend and will come to the right conclusions. This approach will also make it difficult to test things like changing the internal linking structure or adding different schema markup.

Here’s the easiest SEO testing option: Distilled ODN

Distilled’s ODN (Optimisation Delivery Network) works like a CDN (Content Delivery Network) and is displayed in front of your website to users and Googlebot.

Here’s how Distilled’s SEO testing process works:

1. Make a change within their CMS interface to a group of similar pages (like product pages).
2. The platform will then automatically make that change to half of the pages on your website. (setting up your control and variant groups for you).
3. The platform then analyzes the organic search traffic to your test pages and will show you how they perform against the original pages.
4. Once you see the results (and if it’s positive), you can then make that change to all of your pages across your site via the ODN CMS interface.

This platform also allows you to run a small test, and if the results are not positive you can rollback the changes with minimal negative organic traffic impact. However, Distilled’s OCN is an enterprise tool with an enterprise price tag.

The software might be best for you if:

• You’re short on in-house statistical staff
• Pulling and analyzing the results data from your analytics program might be challenging
• Your analytics data is not reliable due to spam issues, lack of a filtered account, or other issues
• You have a CMS that isn’t flexible enough to let you set up random groups
• You have a low-risk tolerance (high traffic that you’re concerned about losing)
• You want to get started testing quickly

As a side note Distilled OCN lets you make all sorts of changes to your site that might be stuck in your development queue. I’ve talked to them on behalf of other clients related to setting up Google Tag Manager or implementing schema. As an in-house marketer, you can take that out of your developer’s queue and make the changes yourself.

As a final note

Though it might seem like it, I’m not on Distilled’s sales team, though I’m a fan of any solution that will allow in-house marketers to quickly set up tests and tracking that gives them strong insights to enhance their digital campaigns. In many situations that are off the shelf software vs. creating a system from scratch if you have the budget.

Either way, you should think about how to set up an effective and efficient SEO test for your site. Throughout my career, I’ve found that data often wins over colleagues who are hesitant to change, and that data can make your job implementing an SEO strategy just THAT much easier.

Thanks for reading!

Katherine

PS: Thanks for taking the time out to read my post and geek out about SEO with me!  I get my inspiration for post topics from other SEOs and in-house marketers struggling with SEO strategy and implementation questions, so if you like this post, please…
1) Leave a comment
2.) Share it with your network
4) Follow me for future posts