How to Take Testing Into Account When Evaluating ESPs

As a part of the work we do with organizations to help them choose the best email service provider (ESP). Email marketers regularly ask, “What do I need to look for in an ESP if I want to really delve into testing and optimization?”

This is one of those questions with a two-part answer…kind of like A/B testing, I guess. It’s a question I anticipate hearing with greater frequency. Marketers want to know what is or isn’t working, as email evolves into a sophisticated tool and key component of omni-channel marketing.

If you want to make sure you have a good handle on how to use email testing to improve email results and ROI, first see this blog post. This article is about what to look for in an ESP if you are keen on testing. So let’s break it down and provide the answer…

Consider ease of use

The first part of the answer has to do with the functionality of the ESP. If you want to be testing more, then you have to be able to do more tests. This means ease of use becomes critical when evaluating your email service provider options.

Tests based on data and segmentation

If you wanted to call on other information such as the data in your CRM system or your web analytics, how easily can you integrate that data with the data in the ESP for segmentation purposes? How easy is it to map outside data into the ESP?

Ideally you want the CRM and email data integrated so it is at your fingertips. And you want the segmentation to be as easy as drag-and-drop. The more segmenting you can quickly do (emphasis here on the quickly), the more optimizing you’ll be doing on those segments — and therefore even more segmenting as you break groups into sub groups and then subs of those sub groups.

Automation of the test process

Consider the ease of automating different part of the tests—or conversely the difficulty of doing so. Some ESP platforms will automatically send a winner for you. You set up the A/B test to a small portion of your list with criteria that determines the winner. The platform will automatically send the winner to the whole list…without you having to lift a finger. Some ESPs offer a split testing wizard that can automate much of the setup for you.

Number of variants

Many email marketers take the term A/B testing very literally and only use 2 versions in their A/B tests. We recommend testing more variations, if your list size allows for it while keeping tests statistically significant. But does your ESP let you test four or eight or even 16 different versions at the same time? With ease?

What about block-level testing that lets you test only a small part of an email rather than create different versions? Does the platform allow for multivariate testing?

Evaluating ease of use

I can’t stress enough the importance of ease of use because if it’s not easy to do, you won’t do it. Here’s one quick way to evaluate it: Ask the ESP how long it takes to set up a subject line test (or how many steps). Or better yet, ask for a test drive and discover for yourself how seamless or clunky the process is in real life.

Then there’s the actual testing. When something is easy to do and understand, you’ll do it more. That’s true for testing and optimizing as well: The easier it is for you to test and understand results, the more testing you will do and the more you will be able to optimize your email marketing effectiveness. Now, about understanding the results…

Understanding the results: email analytics

The other part of this answer is the optimizing part: being able to understand the results of all of your testing, and not just understanding, but being able to dig even deeper into the results.

For example, this screenshot from eMVision shows open rates at glance, by time of day, day of week, and day of month. With one click, a marketer can delve into the details of any of this data.


This sounds easy, but is it? No, because most analytics don’t work this way. When it comes to email analytics, the industry seems to be lagging behind a bit in what it offers marketers, namely that the analytics tools are not yet providing the capability needed. According to a Salesforce study on the state of email analytics, survey respondents cite the following challenges:

Relying on manual processes to get all the data in one view (53%) is one of the greatest challenges enterprises face today. Additional factors driving enterprises to integrate more data sources into their analytics applications include finding that too much data is left unanalyzed (53%), spending too much time updating spreadsheets (52%), and analysis is performance by business analysts, not end users of the data (50%).


Email analytics (a part of Business Intelligence) is important because it’s one thing to be able to test A vs. B but something else altogether to be able to identify segments, patterns and trends. Whether the trends are good or bad, you want to be able to act on them, taking advantage of the good trends to do more of the same and taking action to stop the bleeding when the trend is one that’s quickly spiraling downward.

However, you also need a big picture view because it could be an upswing in one of your metrics coincides with a downward trend in another, meaning you didn’t improve after all. If you get more opens with subject line A, but more click throughs with subject line B, you need to know this. (And this is also why you need to know what you are testing for before you start.)

In addition—as with the testing—ease of use is critical because you won’t do the analytics if it’s not easy to make sense of the data before you.

Solving multiple marketing challenges

According to research presented during the MarketingSherpa Summit 2016 summary webinar, the #7 challenge for marketers is “Design and management of experiments and A/B tests.” But #3 is “acting on data to improve marketing performance.” You might rephrase this as, you have to know what you’re going to do with the data and information.


Be realistic about how important testing is to your organization

Before making testing and analytics capabilities a high priority on your requirements list as you do your email service providers comparison, think through your real life use of these tools. If you haven’t been doing any serious testing before and are now just thinking about doing it, it might not be the biggest thing to consider.

It’s a good idea to start listing some of the tests you would like to run in the coming period, say for the coming year. That will help you determine the time it will take and the return testing will give you.

Caveat: Are we asking the right questions?!

Now, having said all of this, I am going to give you a big, huge caveat: If you’re serious about testing, it can also be a third-party vendor question rather than criteria for choosing an ESP. Chances are, some ESPs, even a top-tier ones, will fall short on the testing and email analytics features if you want something heavy duty. In addition, there might be times when the kind of analysis you want to do will be easier with a third-party solution.

Then the ESP question switches from, “Which ESP can meet my testing and analytics needs?” to, “How easy is it to bring testing from another program back into the ESP and use it?”

To research email analytics from this angle, you have two choices: Search for the third-party vendor first then make sure that solution integrates with the ESPs on your short list. Or go to the ESPs and ask which email analytics solutions can integrate with their platforms. (Or call on a consultant to help you sort this all out.)

You do not want to choose the analytics solution and then try to force it to work with an ESP, nor do you want to go the other route, choosing an ESP then trying to force email analytics software to integrate with it.

If you do evaluate the email analytics vendors first, all of the advice above on ease-of-use still applies. Also consider the growing number of analytics tools that are presenting results in a visual way that is much easier to understand and interpret.

About Marco Marini

Marco Marini is the CEO of ClickMail Marketing, a vendor-agnostic reseller of email marketing solutions. Marco is an acknowledged expert in e-marketing with over 15 years of experience in the field. Before taking over as CEO at ClickMail, he was the VP of Marketing & Operations. Marco has also held key marketing positions with CyberSource, eHealthInsurance, DoveBid and IBM Canada.

Enable registration in settings - general
Compare items
  • Total (0)