Email service provider ExpertSender recently added a new feature that allows their users to know whether the results of their split tests are statistically significant. Marketers now get a graphical representation of their results and can make informed decisions on their split testing outcomes directly from the email provider’s interface.
Statistical Significance in testing
ExpertSender decided to combat one of the inherent weaknesses of split testing: split test results can lead you into making erroneous decisions. Having the appropriate confidence levels (both statistically and in your own assumptions) allows you to make better-informed business decisions.
Are Your Split Tests Statistically Significant?
Have you ever wondered whether the split tests that you are performing on your emails are statistically significant (in simple terms: if the test result really did mean something or if you happened to run it again you might just have had a completely different result)?
For example, a split test result on a subject line generating a 22% open rate might look like a clear winner compared to 20% generated by another, but can you tell whether this difference wasn’t just the result of chance?
Statistical suggestions
After running split tests (ExpertSender allows for up to 5 different versions) users are presented with a few different suggestions based on the results of the test group’s size, versions and variables that were tested. These suggestions allow the user to make actionable changes for future split tests.
Some of the possible suggestions a user may receive;
- Winning version is certain.
- Winning version is not certain. Increase the test groups in the next split test.
- Winning version is certain, but either the size of the test group was too big or the number of versions too small. You can try using a smaller test group or adding another version in the next split test.
- Winning version is not certain. Try using a larger test group or remove one test version in the next split test.
- Winning version is certain, but the result of the remaining versions is not certain. Try testing another variable next time.
The ExpertSender platform uses a 95% confidence interval for these calculations. This means that in 95 out of 100 samples your estimated mean will fall within your confidence interval.
In the following example, there are three different test groups with different subject lines and each of them has a different Unique Open Rates, 6.67%, 6.20% and 8.36% respectively. The final test group seems to be the obvious winner, but are you able to conclude whether it was statistically significant?
The Statistical significance result confirms that the third version was a certain winner, but there is no certainty in our remaining versions.
Availability
The statistical significance analysis is available to all ExpertSender users as of the latest release.
