Lessons learned from align.me's own A/B tests

align.me recently changed the way it communicates with its contact database. We replaced the process of having our contacts divided into various logical campaigns, which had become difficult to manage and prevented quality measurability, and decided to enter them all into a single email stream.

This single stream would receive a three weekly email series called Funnel Vision, containing a hand-picked selection of B2B tips, techniques and advice on the fine art of Funnel Management.

There were many motivations for the change, but one of the biggest was that we wanted to perfrom A/B Testing to find out what our audience engaged with and how we could influence this. So, we built A and B versions of our emails and went to work testing one elememt at a time.


What have we learned so far?

Our tests have helped us confirm that our audience:

  • Likes the generic Funnel Vision subject line (changing increased our opens, but only out of curiosity – our audience has been trained to know what to expect in Funnel Vision)
  • Likes the classic articles written by one of our frequent contributors
  • Prefers test only emails without images accompanying articles
  • Clicks headings and not icons


Whilst the results below are thruths only for align.me's Funnel Vision eDM series, we hope the lessons learned can help you build your own A/B tests.

Contents

Results of our A/B Tests

"Do emails containing the word B2B generate more opens?"

  • Email A: How to set a strategy for advertising
  • Email B: How to set a strategy for B2B advertising
  • Measure of success: Article clicks
  • Result: Emails sent on containing B2B in the subject line generated a near 25% increase in article clicks, this was present across all articles.

"Do emails sent on Saturdays or Sundays generated more opens?"

  • Email A: Sent on Saturday
  • Email B: Sent on Sunday
  • Measure of success: Open Rate
  • Result: Emails sent on Saturdays generated 6% more open rate and 24% more CTR compared to emails sent on Sundays.


"Do subject lines that include statement or question earned more open rates?"

  • Email A: Statement
  • Email B: Question
  • Measure of success: Open Rate
  • Result: Emails that included question in the subject line earned 7% more open rates.


"Do emails that include Digital Assets or exclude Digital Assets generate more clicks?"

  • Email A: Include Digital Assets
  • Email B: Exclude Digital Assets
  • Measure of success: CTR
  • Result: Emails that excluded digital assets generated 24% more CTR.


"Do emails that have regular design or responsive design generates more clicks?"

  • Email A: Regular Design
  • Email B: Responsive Design
  • Measure of success: CTR
  • Result: Inconclusive. Open rates and CTR for both were similar.


"Do emails that have generic subject line or subject line tailored to FV content generate more opens?"

  • Email A: Generic subject line
  • Email B: Subject line tailored to FV content
  • Measure of success: Open rate
  • Result: Subject line tailored to FV content generated 10% more open rate.


"Do emails sent on Thursday, Friday or Saturday generate more opens?"

  • Email A: Saturday
  • Email B: Thursday and Friday
  • Measure of success: Open Rate
  • Result: Emails sent on Saturdays consistently generated more open rate and CTR compared to Thursdays and Fridays.


"Does an emphasis on video blog or no emphasis attracts more opens?"

  • Email A: No Indication of Video
  • Email B: Emphasis on Video
  • Measure of success: Open Rate
  • Result: Text blog with no clear indication of video generated 8% more open rate and 58% more CTR. No indication of video generated 2% more open rate and 40% more CTR. Our audience prefers no clear indication of video compared to emphasis on video.


"Does CTA location in title or below teaser generate more clicks?"

  • Email A: In title
  • Email B: Below teaser
  • Measure of success: CTR
  • Result: CTA in title generated 76% more CTR than CTA below teaser.


"Do our audience prefer 4 articles or 2 articles?"

  • Email A: 4 articles
  • Email B: 2 articles
  • Measure of success: CTR
  • Result: 2 articles generated 9 % more CTR than 4 articles. Thus our audience prefers 2 articles.


"Does blank template or basic design templates generate more clicks?"

  • Email A: Blank template
  • Email B: Basic design template
  • Measure of success: CTR
  • Result: Inconclusive. On average, the CTR for both was identical.


"Do emails sent on Friday or Saturday generate more opens?"

  • Email A: Sent on Friday
  • Email B: Sent on Saturday
  • Measure of success: Open Rate
  • Result: Emails sent on Saturday consistently generated more open rates.


"Do emails sent on Wednesday or Saturday generate more opens?"

  • Email A: Sent on Wednesday
  • Email B: Sent on Saturday
  • Measure of success: Open rate
  • Result: On average emails sent on Saturdays have 2% more open rates than emails sent on Wednesdays.


"Do emails sent on Wednesday or Friday generate more opens?"

  • Email A: Sent on Wednesday
  • Email B: Sent on Friday
  • Measure of success: Open Rate
  • Result: Inconclusive. Both have similar open rates, no clear winner could be identified.


"Do our audience prefer our emails to list each article on its own (block layout) or have each article woven into a more narrative styled email?"

  • Email A: Block
  • Email B: Narrative
  • Measure of success: CTR
  • Result: Email with block layout earned 47% more CTR compared to narrative styled email.


"Do our audience prefer our emails to list each article on its own (block layout) or have each article woven into a more narrative styled email?"

  • Email A: Block
  • Email B: Narrative
  • Measure of success: CTR
  • Result: Despite earning a .2% higher CTR, Email B's narrative version is not a clear enough winner to become the new control. We will re-test this item.


"Do our content is positioned clearly in the email layout?"

  • Our email series contains several articles. Some of which are more substantial offerings, in the form of a white-paper or video. The audience who views these particular digital assets submit their details in a survey. To this end, we wish to increase the volume of individuals who click this. We are testing its position in our email layout.
  • Email A: Digital Asset last
  • Email B: Digital Asset first
  • Measure of success: CTR
  • Result:
  • Placing the DA first in Email B helped to generate 13 more clicks. However, in comparison to previous editions this is not a significant number. Previous editions have generated more clicks despite the DA being in the last position.
  • Overall Email A received a CTR that was 2% higher than Email B. Having the DA first clearly had a negative effect on our engagement rate in Email B.
  • Interestingly, an article written by one of our frequent contributors generated the highest amount of clicks in Email A and came a close second in Email B. Our audience obviously loves his content, so we will keep him at number one and leave the DA last.

"Do our emails have clear design?"

  • Our previous tests helped us improve our content so now we want to test email design.Our emails series presents each article with a relevant image/icon which is also hyperlinked to the full article and we want to find out if our readers are clicking on the icons or the article headings. Our results will tell us whether or not to include them.
  • Email A Icons present:
  • Email B No icons:
  • Measure of success: CTR
  • Result: The no-icon version received a higher CTR so it will become our control.


"Do article heading that targets readers ie. 'you' attract more open than general headings?"

  • Email A article headings:
  1. How to make money from failure
  2. The outsourcing dilemma: 6 steps for B2B marketers
  3. Funnel Wiki highlights
  4. Free white paper - Forget About Branding: How to Get Marketing Closer to Revenue
  • Email B article headings:
  1. You can make money from failure
  2. The outsourcing dilemma: 6 steps to help you make the right choice
  3. What are your peers doing on Funnel Wiki?
  4. Free white paper - Forget About Branding: How to Get Marketing Closer to Your Revenue
  • Measure of success: CTR
  • Result: The general article headings were the clear winner, generating 26% more clicks that the targeted headings. This is sufficient data to retain general headings for the email series.


"Does a generic subject line or a specific subject line, attract more opens?"

  • Email A subject line: Funnel Vision - B2B marketing and sales performance tips
  • Email B subject line: How to allocate budget, better understand your buyers and improve sales conversion rates.
  • Measure of success: Open rate.
  • Result: Inconclusive.
  • Action taken: The specific subject line generated a slightly higher open rate, however the margin was not large enough to warrant naming it the new control. Therefore the generic heading will remain the control and Test 4 will assess click-through rate based on different article headings.

"Does the word order in the subject line have an impact on open rates?"

  • Email A subject line: Funnel Vision - B2B marketing and sales performance tips
  • Email B subject line: B2B marketing and sales performance tips - Funnel Vision
  • Measure of success: Open rate
  • Result: Open rates were identical.
  • Action taken: The control element (Email A subject line) will be retained and retested.

"Does the word 'Examples' in the subject line have an impact on open rates?"

  • Email A subject line: B2B Marketing Strategy
  • Email B subject line: B2B Marketing Strategy Examples
  • Result: Open rates were identical.
  • Action taken: A/B Testing Road map created here.

"Does our audience click on article headings, or the picture icons that accompany them?"

  • Measure of success: Click Through Rate
  • Result: CTR on article headings accounted for over 90% of all clicks.
  • Action taken: As the article icons add aesthetic appeal and structure to the email, we have decided to retain them for future emails. However, later on we may redesign the icons so that they are more attractive at which point we could re-run this test.

"Do shorter, sharper sentences, following journalistic principles generate higher blog, and CTA clicks?"

  • Email A (Control): Long, fluffier, more descriptive sentences.
  • Email B (Test): Short, sharp, less than 25 word sentences.
  • Result: While the difference was not huge, all three sends of the test cycle resulted in A (the control), generating higher click-throughs.
  • Action taken: Considering A resulted in higher blog and CTA clicks with each send, we conclude that for this particular audience, we should continue using longer, more descriptive sentences. View the full report here [1].