ExpertSender - May 11, 2016

ExpertSender makes split tests actionable with statistical significance feature

Email service provider ExpertSender recently added a new feature that allows their users to know whether the results of their split tests are statistically significant. Marketers now get a graphical representation of their results and can make informed decisions on their split testing outcomes directly from the email provider’s interface.


Statistical Significance in testing

ExpertSender decided to combat one of the inherent weaknesses of split testing: split test results can lead you into making erroneous decisions. Having the appropriate confidence levels (both statistically and in your own assumptions) allows you to make better-informed business decisions.

Are Your Split Tests Statistically Significant?

Have you ever wondered whether the split tests that you are performing on your emails are statistically significant (in simple terms: if the test result really did mean something or if you happened to run it again you might just have had a completely different result)?

For example, a split test result on a subject line generating a 22% open rate might look like a clear winner compared to 20% generated by another, but can you tell whether this difference wasn’t just the result of chance?

Statistical suggestions

After running split tests (ExpertSender allows for up to 5 different versions) users are presented with a few different suggestions based on the results of the test group’s size, versions and variables that were tested. These suggestions allow the user to make actionable changes for future split tests.

Some of the possible suggestions a user may receive;

  • Winning version is certain.
  • Winning version is not certain. Increase the test groups in the next split test.
  • Winning version is certain, but either the size of the test group was too big or the number of versions too small. You can try using a smaller test group or adding another version in the next split test.
  • Winning version is not certain. Try using a larger test group or remove one test version in the next split test.
  • Winning version is certain, but the result of the remaining versions is not certain. Try testing another variable next time.

The ExpertSender platform uses a 95% confidence interval for these calculations. This means that in 95 out of 100 samples your estimated mean will fall within your confidence interval.

In the following example, there are three different test groups with different subject lines and each of them has a different Unique Open Rates, 6.67%, 6.20% and 8.36% respectively. The final test group seems to be the obvious winner, but are you able to conclude whether it was statistically significant?


The Statistical significance result confirms that the third version was a certain winner, but there is no certainty in our remaining versions.


The statistical significance analysis is available to all ExpertSender users as of the latest release.

About Jordie van Rijn

Jordie van Rijn is an independent email marketing consultant and analyst. He is the founder of Email Vendor Selection and specializes in smart email marketing, online marketing strategy, software selection, campaign management, optimisation and RFP / vendor selection. He tested, reviewed, and wrote about 100+ business software including email marketing services, CRMs, ecommerce platforms, and online course creators. Published in-depth email marketing guides for financial service, ecommerce, travel, restaurant, and fashion industries.
Named one of "50 Online Marketing Influencers to Watch" by Entrepreneur magazine. Companies like Scania, KLM, Unilever, AEGON, CZ, FNV, NRC Media have asked me for advice.


Email & Marketing Automation Software
Expertsender is rated best email marketing software for segmentation in 3 consecutive years. Marketers trust ExpertSender for their solid email deliverability and smart automated campaigns.
Read more.. Demo Price CSA
Enable registration in settings - general
Compare items
  • Total (0)