A/B Testing

A/B testing (also called split testing) is a method through which variables on a landing page or email are tested against one another to identify which one produces a better response rate.

Running A/B tests are a great way of recognizing initiatives, both big and small, that drive the most traffic to your webpage, and that ultimately convert into the most sales. It is though that A/B testing of landing pages can generate up to 30-40% more leads for B2B sites.

A/B testing works by comparing a ‘control’ – the original element – and the ‘treatment’ – the tweaked version of that element.

Smashing Magazine has produced a great image to help visualise the A/B Testing process: https://align.me/files/A%3AB%20Testing%20Image%20.gif

Guidelines for A/B Testing

With the help of HubSpot, 10 guidelines for effective A/B Testing have been outlined:

1. Only conduct one test at a time

If you were to test both a landing page and an email that direct to a landing page, it is easy to see how the results could become unclear.

2. Test only one variable at a time

Similar to the first point, conclusions may be unclear if there are a number of elements that have been altered.

3. Test minor changes as well

Small details can be just as important as large ones. Simple colour or positioning changes can actually result in significant improvements.

4. You can A/B Test the entire element

Testing two completely different landing pages or emails against one another can be a good starting point before looking into the nitty-gritty aspects.

5. Measure as far down the funnel as possible

Don’t just measure your opening or conversion rate, but also how these have translated into demo requests or sales, for example.

6. Set up control and treatment

The control is the original form of the element you are testing, while the treatment is a variation of this form that will be tested against the control.

7. Decide what you want to test

Elements to be tested do not necessarily have to be as instantly recognizable as colour, wording, layout or design. You may choose to test different target audiences, timing or alignment between an email and a landing page.

8. Split your sample group randomly

Email recipient groups must be as similar as possible in order to draw the most valuable conclusions from your results. Equality between sample groups is absolutely crucial.

9. Test at the same time

A rapidly changing business environment means that it is very important that tests A and B are conducted at the same time to ensure recipients are subject to the same test conditions.

10. Decide on necessary significant before testing

Before beginning your test, decide at what level your results will have to reach to warrant a change in your email or website execution. Make sure to set the statistical significant goal for your winning variation before you start testing.

Frequency of A/B Testing

Unfortunately there is no one-size-fits-all answer here. Every business is characterised by different customers, market conditions, and financial circumstances, all of which influence how often you should test.

The most important thing to remember with frequency is to not restrict yourself in terms of the data you collect. There is always more you can learn from your webpage or email layout that will allow your to deepen your understanding of the likes and dislikes of your customers. You are able to acquire this information by running as many tests as possible. Many businesses overlook the importance of continuous testing and are content with being content.

Taking advantage of that feeling of content is where businesses can become better than the average. This may mean testing a specific segments of your customer base or another element altogether. Consumer preferences and tastes are continuously evolving so it is vital for businesses to evolve right along side them in order to remain effective. Having said this, to attain the most valuable information, only one test should be conducted any point in time.

What Elements of a Webpage should be Tested?

Webics have outlined a number of elements that businesses should consider testing:

Call-to-Action (CTA) Buttons

Slight changes to the colour of CTA buttons, or the verbs used within them, can impact your conversion rate.

Page Layout

While this can take some time, testing alternative layout arrangements can help you understand combinations that allow visitors to flow through your content, and down your conversion funnel with greater ease.

Navigation

It is well documented that simple, self-explanatory navigation leads to the highest conversion rate. Alterations to elements that can potentially improve webpage navigation should be tested to see what impact that have on conversion rates.

Page Copy

Companies tend to overlook testing page copies to their own detriment. When split testing your copy you should; reword your offer, reword the entire body, test variant headings and alternative formatting options.

Promotional Offers

The wording of offers can heavily impact on the success of the particular promotion.

Checkout Process

Just because a customer has clicked “Add to Cart” does not mean the they will actually go through with the purchase. In order to minimise this, test as many variants in your checkout process as you can.

Images Product Selection Process

Is your product selection process easy to understand or does it leave your visitors wondering if they have successfully added the item to the cart or need to try again. The key here is absolute simplicity and obviousness. When your users choose to read more about a product after clicking through from a category page or from suggested items, you should test the difference in performance from when you take them to a completely separate product page or when an info pop-up within the same page is shown to them. Consider testing as many elements of your product selection process as you can.

Security Fields

It is imperative to make sure your customers feel their details are safe and that your system is secure. Test elements of your webpage that instill feelings of trust in your customer and that encourage them to continue with their purchase. This is also important with other forms of data collection such as a contact form.

Example of A/B Tests for Email Campaigns

Subject Lines and Article Headings

  • Specific vs Generic
  • Poses a problem vs Provides a solution
  • Targets reader i.e. 'you' vs General heading

Sender name

  • From the CEO vs From the company

Layout

  • Change the ordering of the items you offer
  • Include pictures to accompany each item vs No pictures

Sending

  • Send the email at different times on the same day
  • Send the email at the same time on different days

Marketo's List of Elements to Test

1. Subject line

2. From name

3. Day of the week

4. Time of day

5. Frequency

6. Mostly-images vs. mostly-text

7. Short copy vs. long copy

8. Links vs. buttons

9. Number of links

10. Unsubscribe at the top

11. First name personalization — in the subject line

12. First name personalization — in the email body

13. Animated gifts

14. Font colors

15. Font styles

16. Opt-down

17. Social sharing icons

18. Social connecting icons

19. Delivery by time zone

20. Call to action — number

21. Call to action — placement

22. Post-click landing page

23. Social proof

24. Tone — human vs. corporate

25. Copy length

Sample Size

An easy trap to fall into with testing is to try to test too many variables at once. The more variables you test, the larger your sample size need to be. Here's a little tool for determining how big your sample size needs to be.

A/B Testing in the Real World

  • At align.me, we have conducted our own A/B Testing for various reasons. Click here to see some of the lessons we have learned.
  • Vision 6 is a messaging platform provider. Their 2011 Email Metrics Report outlines benchmarks and other information regarding A/B Testing that may be useful to take into consideration for your own tests.
  • A/Btests.com is a site that helps you test web A/B versions of sites/applications and share that knowledge with colleagues or clients. This site also allows you to view and learn from other people's test results.
  • Unbounce is a self-serve hosted service that provides marketers doing paid search, banner ads, email or social media marketing, with the easiest way to create, publish & test promotion specific landing pages without the need for IT or developers.
  • Explore the following collection of Case Studies which demonstrate the effectiveness of A/B Testing
  • Hyundai experienced a 62% increase in requests for test drives through their website after testing the following to find the most effective combination:
    • New (SEO friendly) text vs control text
    • Extra Call to action buttons vs no extra buttons
    • Large photo of the car vs thumbnails
  • By simply changing the senders name from the generic “HubSpot” to a personal name of someone from the marketing team, Hubspot was able to increase their leads generated from email marketing by 131.
  • According to a blog post in Marketo, you should perform tests that can't be ignored.
  • This test duration calculator can help you determine the maximum number of days a test should be run before giving up.