Have you ever run an AB test or an experiment? We do them all the time, but in B2B marketing there’s a few little traps that can catch you because the data can tell you lies and you’ll form the wrong conclusions. I’ll show you how to avoid those wrong conclusions and to use tests to optimize all of your marketing.
The first thing I want to address is why we do these tests and why we take them through to the conclusion of trying to be statistically significant. First, we can get really big improvements or really big drop-offs by making the smallest changes. That’s why you test. Because you can make quite big improvements with converge optimization by making small changes. Why do we need statistical significance? Surely, a directional result is evidently better than the other one. Today, I’m going to show you why that’s not the case – that’s how the data lies.
Our first conclusion is make your tests simple. Complex tests will confuse you and you’ll draw incorrect conclusions. Second one is make your tests big – don’t test little changes. Test big changes and make the little changes later. Third one is isolate the source. That is, don’t let your different tests or your A and B landing pages (if that’s what you’re testing), receive their data from different sources. If you think source is an important difference, then actually test for source, not the landing page. Do the same with time. Either isolate time of day, time of week, time of month, time of year, or test for it. Finally, be patient. Resist early guesses. You will be wrong. I’ll share with you the results of our tests and show you the pre-launch campaign that all this fuss is about at the very end.
I’m not going to bore you with too much of the background to the tests that we’re actually trying to find a result for, but I will tell you at the end, because it’s actually quite exciting. However, it’s not meaningful for the conclusions I’m trying to reach with you today. What I will do though, is show you the tests as they played out for us, because the journey itself is quite telling. When we first set this up, we didn’t have time to set a test up, so we didn’t have time to hypothesize. To set it up, we just had to sort out the landing page. We were getting just short of a 14% conversion. That’s not bad, but we always know we can do better, so we invested the time. The Join Fast Funnel Club is the call to action that pops up at the bottom of the video screen. It’s a logical call direction because the whole page is selling membership to the Fast Funnel Club. I’ll explain again what Fast Funnel Club is at the end. That’s what we were selling and it does make sense – but it’s a tax, it’s a cost, it’s a thing the visitor has to do rather than the benefit they will get as a result of doing that thing. So, we changed it, to Get Early Access and Discounts, because that’s the benefit.
What were the results? At the time of today’s recording, between them they were averaging, just short of 14%, at 13.66%, almost the same. Here’s something really weird – 15% for the control and 12% for the tests. Now, the weird thing is not that the test failed or the test got a lesser result. The old company was actually winning. The weird thing is that the control was now getting 15%, not 13.6% – why? It’s because we were running those at different times of the year. The original landing page that didn’t have a test, started over the Christmas break and had different data sources. Therefore, our first conclusion is that source actually matters. We should also add that time does have an impact – weekends, holidays, et cetera. Ignore for the moment the fact our control went from 13.66% to 15%.
Let me show you the results as they revealed themselves in a genuine parallel test. I’m going to do that by talking through a couple of LinkedIn posts that I did during the break. I paused the original page, which I still conceptually think of as a 15% converter, and created an identical copy of that with different texts and the call to action. I explained what those differences were already. The two versions of the landing page were now getting the traffic not from the original emails, but from LinkedIn posts, Twitter feeds, referrals, and other sources. At the time we ran an interim test or when we looked at the results, D was beating E by 12%. That’s off a pretty low traffic rate. D’s button copy was, ‘Get Early Access’ – that’s the change. So the change was beating the control by 12%. This is quite a good result and I was excited by that. Now, the landing page platform gave us a 17% confidence in that conclusion, so we clearly couldn’t trust it yet.
Over the next four days, the winner was E, which was the new one, then back to D, which was the control, then back to E, the new test, and back to E again. We’d need to see a bigger margin between D and E before we draw any conclusions. But how confident do we need to be? Do we need to get to 95% confidence? Remember, at this stage in the test, it was starting to look clearly like D – our change, was going to be the winner. If I had to make a call at that time, that was the one I would have made. Let me cycle forward now to another glance at the same data a couple of days later. I headlined this one in LinkedIn by saying, ‘We’re just two phone calls away from making a big mistake’.
If you have a look at the data in the video, if only two of the nine new members who had been presented with version E, which is the control, had received an inopportune phone call, then the conversion rate would have dropped to 6.9% rather than 8.8%. If only two of the 102 who actually got dragged away by phone calls when they were just about to join, had not received those calls, then version D would’ve been reporting 7.7% and version D would’ve been the winner. In short, we were two phone calls away from making a disastrous decision. You have to let the data play out.
Where are we now? Our champion seems to be getting around 8% to 9% when it’s receiving its feeds from sources other than e-mail, and around 25%, when it receives its traffic from e-mails. Now, we’re definitely getting a better result out of the champion, that is the original one that sold the action, from most sources. However, I’d be willing to bet e-mail users actually like the revised copy, and that’s what we’re testing next – to split out and create two versions of the landing page, each in turn have two versions. One version is just for e-mail, and there’s an AB test on that one, and one version for every other source and there’s an AB test on that.
Let me tie all this together with some conclusions and actions I would recommend when trying to use tests to inform your optimization. In B2B, even in large companies, we often deal in very low volumes. For that reason, we have to be smart in how we shape our tests, and we have to be patient. These are my five key conclusions about testing optimization for B2B. Make your test simple – don’t test multiple variables at one time, but make it something from which you can draw an absolute definitive conclusion. Learn from that conclusion and assume it into your next est. Try and draw conclusions that are concrete, allowing you to build a control based on your past knowledge in the same market.
Secondly, make your tests big – think of the largest changes or improvements you can make to your control, and test for those first. Remember, it will take a long time before you get a result so you want to make the wait worth it. Test the smaller changes later. Thirdly, isolate source, that is the source of the traffic – or test for it. We basically need to make sure that at any point in time, A and B are receiving their traffic from the exact same source. Of course, isolate for time of day, quarter, month, year, et cetera. Make sure, again, that you’re testing them in parallel, or if you think time is material, test for it. Although, don’t test time as well as different button copy for example. Finally, be patient and resist early guesses. You saw in one of my LinkedIn posts I was starting to lean towards the new test being the winner – that was D. But in fact, during the course of four days, we saw wild swings, and finally it played out that E, the test that was showing the action, was actually the winner. What action were we actually trying to achieve? Forget the test for a moment, focus on desired outcome.
Let me show you what all the fuss was about and why I’d like to invite you to do the same thing as we were inviting those who went to the landing page. You probably already know that Funnel Plan plays a critical role in our work with larger companies. It also plays an important role with our smaller clients, but in both cases, it’s very hands-on. One of our smartest consultants has to lead the planning. We’re launching a self-serve version of Funnel Plan that is not just for small companies who can’t afford the consulting, but actually the small divisions of very large companies. Think of a very large company like an IBM. They can afford consulting rates with companies like us but they’ve got tens of thousands of go-to-market plans they work on every year. That actually becomes wickedly expensive. Our self-serve Funnel Plan is not just for small companies, but also divisions within larger companies.
The great thing is, you take the plan that Funnel Plan generates, plus we hook it into the CRM and extract the actual data. Then we look at the plan, the actual data, and aggregated data from all the other plans that we’ve captured in Funnel Plan, allowing us to inform some really amazing automated coaching. That coaching manifests as a weekly e-mail that says, ‘you were expecting closure rates of only 17%, but you’ve actually been generating 25%, let’s look at why that’s the case, what things can you keep and what do you need to change’. Conversely, you thought the whole journey was going to take twelve weeks, it’s taken twenty-four, but when you win, it takes only eight. What does that mean? What can you do about it? We can automate all of that based on the data. We can give some really, really deep insights based on the plan, plus the actual, plus all the other plans that we have in the database.
The Fast Funnel Club is an early access array or a pre-release list. Members who’ve joined Fast Funnel Club get early access to Funnel Plan – at least two months earlier than the general market. Plus they get big discounts in return for joining Fast Funnel Club, and they get unique insights. You can join Fast Funnel Club too. It’s totally free and completely without any obligation. Go to funnelplan.com, join up to Fast Funnel Club, and enjoy that early access and super huge benefits.
