FulcrumTech's Email Split Testing 101

Split testing is a powerful tool for increasing the effectiveness of your email marketing campaigns. It’s a great way to determine what works best for your business and your customers, rather than basing marketing decisions solely on industry data.

What is a split test?

When we talk about testing, we’re typically trying to answer the following type of question:
Which is the best — A or B?
An A/B split test is when you send each of the variations that you are testing to two portions of your test group.  If you have set up your test correctly, you can determine which version is best for your purposes and then roll that out to a broader amount of your sample.

What to test

In marketing, there is so much that is worth testing.  In effect, marketing is all about testing.  While we have best practices, the best way to determine what’s right for you is to conduct a test.  Here are a few items we at FulcrumTech like to test:

• Headlines
• Email subject lines (testing open rates)
• From name
• Day of week to send
• Time of day to send
• Layout/design (looking at click-through rates)
• Personalization (testing whether Dear First Name or First Name in the subject line delivers better results)

Setting up the test

If you have 2 or more variables you’re trying to test (e.g. headlines, subject lines, graphics), you need to present them to a randomly selected group of people and see how they react or respond.  This is known as an A/B split test. The act of testing requires that you "split" them across a homogeneous sample group.  If you are testing more than 2 variables, you’d use what’s called multivariable testing.  To ensure that the test is as accurate as possible, you want to ensure that you have the following:

• An adequate sample size – the sample must be large enough that your results are statistically significant
• A homogeneous sample across all splits – the sample should be randomly selected identically for each split
• Identical timeframe – test all segments at the same time (unless, of course, you are testing time frames) because introducing variations in the test time period can lead to yet another variable that may inhibit your ability to compare your results accurately

 
It’s easy to get started with some basic tests, and I urge all clients to get started with some basics.  As you get deeper into testing, you’ll want to learn much more about sample size determination, as well as the risk factors potentially affecting your results.  Ultimately, you’ll discover that it’s much better to create a valid test than a poorly constructed test that leads you to make incorrect decisions based upon incomplete data.

Written by Mitch Lapides, President and CEO of FulcrumTech, who has over 15 years in online product development and e-marketing.