What's a Valid Email-Marketing Test?

Many of our clients and prospects hear us constantly note that marketing is testing. What subject line do you think will perform better? Which layout will drive more clicks? Which text on a button will deliver more click-throughs? Well, I promise that I always have an opinion. And so should you. That’s called a hypothesis. But the right answer comes out of testing. And that test must be valid. Only when it is valid will you confidently be able to make the best decision.

Now, I can’t possibly give you a course on testing in a couple hundred words, but I do want to briefly explain the concept of validity…statistical validity, that is. Without validity, you could end up making a decision with confidence, but using data that was not valid. That’s blind confidence, and that’s worse than doing no test at all.

I’ll use an example to explain statistical validity. Imagine that you have a quarter and a penny. I want you to flip each coin 100 times, and for each heads you get, pretend I’m going to pay you $20. Which coin will earn you more money?

I’m a nice guy, though, and I let you do a test before deciding which to use. So, you flip the quarter and get 2 heads in a row. You flip the penny and get 1 head and 1 tail. So, you quickly decide to use the quarter for your 100 flips.

Now, I know you’re smart enough to know, logically, that flipping twice isn’t enough to make such a conclusion. Suppose, though, that you flip each 50 times or 500 times. As you do more flips of each and count the results, you’ll get a closer approximation of the “real” answer, which we know to be 50/50.

With valid tests, we use statistical formulas to determine how big your sample size should be (the number of times you flip the coins in this case) and when the difference in results between the two options can be deemed truly “different,” statistically. We typically apply a confidence level to the result saying, for example, that we’re 95% confident that the difference in the two options is not simply a difference due to normal statistical variation but is truly different.

It can get much more complicated, but this is one of the core principles behind some of the types of testing we do. To confirm, though, there are many other factors that go into creating a valid test. Knowing how to do these tests correctly can give you the confidence that when you make a decision that it is, in fact, the best decision given the time and money you have to conduct them. And that’s one of the reasons I recently got certified in online testing…to be sure we’re doing the best job we can when we help you perform your tests.