A/B testing: Your questions, answered

Article topics

1. What is A/B testing?
2. How does A/B testing work?
3. What are the benefits of A/B testing?
4. What can you use A/B testing for?
5. How do you work out what to test?
6. Can you test more than one variant at the same time?
7. How long should you run an A/B test for?
8. How many visits do I need to ensure accurate results?
9. Does A/B testing affect my search engine optimisation efforts?
10. What’s the best software to use for A/B testing?
11. How can you make sure your A/B testing is a success?
12. What are the benefits of using an agency to carry out A/B testing?

When making decisions about your eCommerce store, you might go with your gut or act on a hunch.

However, you’ll always make the right choices with relevant and accurate data.

One of the best ways to get reliable data for your business is through A/B testing.

If you’re new to A/B testing, we’ve put together this comprehensive guide to using it on your website and other marketing channels.

What is A/B testing?

A/B testing is when you compare two virtually identical versions of a webpage to see which one performs best. It’s often referred to as split testing or bucket testing.

A/B testing is part of a larger marketing discipline called conversion rate optimisation, or CRO. This is when you make changes to your website to increase the amount of conversions you get.

How does A/B testing work?

Let’s imagine that you have a landing page on your website. Your existing page header is performing well, but you want to see if it can do even better.

Your original landing page is called the ‘control.’ You then create another version of your landing page, known as the ‘test.’ This is identical to the control in every way – just with a different header.

You then send half of your website traffic to the control page and the rest to the test page.

After a certain amount of time (we’ll talk about this in more detail later), you analyse the results to see which page got the most conversions.

What are the benefits of A/B testing?

The key benefit of A/B testing is that you get valuable, actionable data that’s unique to your business.

There are many studies out there advising the best action to take to increase conversions on your website. However, just because something works for one business doesn’t mean it will work for yours.

A/B testing means you can test what’s most important to you and find out what your target audience prefers.

What can you use A/B testing for?

You can use A/B testing to test virtually any element on your web pages, including:

• Images
• Videos
• Text
• Call-to-action buttons
• Forms
• Page layouts

You can use A/B testing on your other marketing channels too. For example, you can see which email subject lines get the best open rates or which Google Display ads get the most clicks.

How do you work out what to test?

An excellent place to start is to identify your goals. For example, do you want more organic traffic, increased sales, or extra signups for your newsletter? This will ensure you carry out the A/B tests that benefit your business the most.

Existing data can help. For example, if you get customer feedback advising that your product pages are hard to navigate or your Google Analytics data shows high bounce rates on your landing pages.

You can also use CRO to identify what to test. For example, let’s say you use heat maps on your website and determine that web visitors aren’t engaging with your data capture forms. You can then use A/B testing to see which elements of your forms you can change to make them more appealing.

Ultimately you want to create a hypothesis for A/B testing, which may sound like this:

Our landing page has a call-to-action button that says ‘register now’ to sign up for a webinar.

If we change the button text from ‘register now’ to ‘reserve your spot,’ we will create a sense of urgency, which we predict will lead to a 20% increase in signups.

During this A/B test, we will measure button click-through rate, number of webinar signups, and landing page bounce rate.

A/B testing is a lot like a science experiment! By looking at existing data and predicting what changes testing will bring, you’ll be on the right track.

Can you test more than one variant at the same time?

Technically you can, but we wouldn’t recommend it. The more elements you add into the mix, the more complicated your testing becomes and the more data you need to assure validity. It’s best to keep your tests as simple as possible.

Let’s say you want to see which colour call-to-action button leads to the most conversions – red, yellow, blue and green.

Rather than test all the colours at the same time, you’ll get a more accurate result if you test, say, red and yellow together, and blue and green together. You can then pit the winners against each other to find out which colour your customers ultimately prefer!

How long should you run an A/B test for?

Unfortunately there isn’t a definitive answer!

It’s essential to wait until your test has reached ‘statistical significance.’ This means you’re confident that the difference in your A/B testing results is a genuine improvement and is not down to random chance.

Let’s say you’re testing two versions of a product page, and one is doing better than the other. An influencer you’re working with may put out a social media post about your product, meaning a sudden surge in traffic and conversions. It’s crucial to ensure that this event doesn’t have a significant impact on your results.

Most CRO specialists recommend that you aim for at least 95% statistical significance and run your A/B test as long as it takes to achieve this.

In our experience, we recommend running an A/B test for no less than two weeks and no more than eight weeks.

An overly short test means you don’t get the data you need, and an excessively long test increases the chances of your test becoming skewed by external factors.

How many visits do I need to ensure accurate results?

Like the amount of time your A/B test runs for, there isn’t a definitive answer to this question either! You need enough website visitors to achieve statistical significance.

As a rule of thumb, the smaller the increase in conversion rate you expect, the more visitors you need to make your A/B test viable. This is because larger increases are easier to validate than smaller ones.

There are sample size calculators that make it easy to identify how many visits per page you need.

Can you run A/B tests on smaller sites that get less traffic? You can, but you need to optimise your tests accordingly. We recommend focusing on simpler A/B tests and running them for an extended period of time (although, of course, not so long they’re affected by external biases.)

You can also use alternative CRO techniques, like heatmaps, session recordings, and user testing.

Does A/B testing affect my search engine optimisation efforts?

Some people think that A/B testing can have a negative effect on search engine rankings. This is because A/B testing leads to duplicate content on a website, which can confuse search engines like Google and Bing.

The good news is that Google supports A/B testing as long as you don’t do anything to trick the search algorithm intentionally.

What’s the best software to use for A/B testing?

44% of businesses that run A/B tests use specialist software.

Specialist A/B testing software, like VWO, Unbounce, and Omniconvert, make it easy to change pages, drive and segment traffic, and track results.

Do you need specialist software? It can be handy if you regularly run A/B tests. However, if you only carry out A/B testing occasionally, you can use your own eCommerce platform to manage your testing.

Some eCommerce platforms have their own apps and plugins for conducting A/B tests. For example, Shoplift and Visually in Shopify, or Nelio in WordPress and WooCommerce.

If you run A/B tests on certain marketing channels, these channels often offer their own A/B testing functionality. Meta, Google Ads, and email marketing platforms like MailChimp all provide testing tools.

How can you make sure your A/B testing is a success?

A/B tests don’t produce a winning outcome all the time. According to VWO, only one in seven tests have a viable outcome.

We’ve looked at some things you can do to improve the accuracy of your A/B testing. For example, running your test for the right length of time and only testing two elements. Here are some additional things to consider.

• Make sure your tests aren’t too vague. It’s better to carry out many small tests than one large one. This means you can pinpoint the exact elements that generate the right results
• Avoid busy times of the year, like Christmas, Black Friday, or the summer sales. It’s essential to run tests at times when user behaviour is more representative of your typical audience
• If you don’t trust the results, rerun your test. If you run the same test multiple times and get the same outcome each time, it’s a good sign
• And finally, don’t panic if your test doesn’t work out. Look at what went wrong and what you can do to improve the process next time

What are the benefits of using an agency to carry out A/B testing?

A/B testing can give you the data you need to transform your site and skyrocket your sales. However, it’s important to get it right – you don’t want to waste your time on hundreds of A/B tests that aren’t accurate.

If you’re not confident in your A/B testing abilities, a digital marketing agency specialising in CRO can help. Here are some of the reasons you should partner with an agency:

• An agency will use its years of experience to suggest the right tests to carry out, how long to test for, and what you can do to increase the odds of a successful experiment
• Good A/B testing software can be expensive. An agency has access to enterprise A/B testing tools, meaning you don’t have to factor subscription costs into your marketing budget
• An agency will carry out testing without bias. Imagine there’s a page on your website you love – you may unconsciously skew the results of your A/B tests in that page’s favour. An agency will carry out all testing fairly and objectively
• An agency can decipher the results of your A/B testing, make changes on your behalf and recommend what to look at next

We specialise in CRO at Xigen. Our skilled team will work with you to understand your goals, gather a range of data, and identify the right tests to complete.

Let us help take the guesswork out of your website – get in touch today to learn more.

Need help with your next big eCommerce project? Find out how we can support.