17 Questions to Ask to Find the Best Recommendation Engine

If you’re reading this article, you’re either intrigued or already seriously looking to find the best Recommendation Engine for your email marketing program.

To properly evaluate recommendation technology for your business, below you’ll find a list of the criteria you should keep in mind.

First of all, it’s important to understand your goals in addition to the tasks a Recommendation Engine must fulfill:

  • Do you want to build your brand?
  • Increase cross-channel engagement?
  • Drive email opens, clicks, and conversions?
  • What about customer satisfaction?
  • Is your focus on it growth and/or profits?

Once these strategic considerations are defined, you can move on to evaluating what is the best recommendation engine technology for you. The questions to ask can be grouped into related areas, including those related to data, model design, speed, onboarding, simplicity and industry relevance.


Let’s start with the questions about data:

What data is used to generate the content recommendations?


  1. Engines that use both explicit and implicit data of website visitor behavior. For example, information that is provided intentionally via a purchase or download as well as dwell time and pages browsed. Avoid Engines with data inputs that are too limited, which may result in over-weighted bias towards popular or trending items.
  2. Also look for Engines that get running quite fast, or have the ability to supplement the model with starting data to avoid the “cold start problem.” Because these models must capture and analyze behavior before they can start producing good recommendations, it might take time to become smart enough. Depending on the sophistication of the algorithms and the volume of your website traffic, the Engine could take several weeks to become productive. The ability to upload history gives the system a jumpstart.
  3. Another important data topic is previous browsing behavior that occurred before a transaction took place, because consumers will often browse several times before taking that first explicit step. That earlier behavior is important for the model to understand broader categories of interest and ensure those next best recommendations are well-informed. Irrelevant recommendations will cause consumers to lose trust in your brand.
  4. Finally, supplementing your first-party data with third-party data may also be useful in some industries, so be sure to discover whether that applies to your business case.

Does the data model use predictive analytics and machine learning to produce the recommendations ?


  1. The Best Recommendation Engines use multiple algorithmic methods such as collaborative filtering and content-based filtering.
  2. Reports, case studies or data should support an improvement in performance after technology implementation, as well as demonstrate a continuing trend of self-optimization over time.
  3. The system should pay attention to how subscribers interact with the email recommendations and feed that data back into the model. In most cases the faster this loop is closed the better.

Does the Recommendation Engine work in real-time?


  1. Technologies that are running continuously, communicating instantly, and work in real time from end-to-end.
  2. New behaviors of a customer on a website should instantly inform the predictive data model, which should immediately update the content recommended. It’s worth noting that even if an email campaign has already been sent, the latest systems can push the updated content into each email, right inside each inbox.
  3. Watch out for technologies that use batch processing to update recommendations on a less frequent basis such as daily or weekly, as well as systems that have recommendations frozen at send time – these may not be ideal for your business needs and expectations.

Is the technology easy to integrate?


  1. Testimonials, satisfaction surveys and references should be available that demonstrate ease of integration and deployment.
  2. You’ll want a transparent overview of the costs involved beyond price. One of the biggest problems of Recommendation Engines today is that they are embroiled in complex systems that are expensive to buy and require teams or agencies to run. You may even have to replace your entire marketing automation platform to obtain this capability, and then pay the vendor extra for technical service support.
  3. If the workload and cost seem too heavy don’t be discouraged – there are also new advances in technology that avoid complex integrations and can work with your existing email platform.

Is the interface application easy to use?


  1. Applications that you can demo as well as testimonials from customers about what it takes to use the technology should help you assess the time to create one personalized campaign and confirm that it works.
  2. AVOID complexity. Studies show that despite the oodles of bells and whistles in some Recommendation Engines, the majority of email marketers aren’t using them. Furthermore, when given the opportunity to adjust recommendations presented to them, subscribers don’t engage. These manual fine-tuning features might seem important, but in reality they are for edge cases if anything at all, and their value has not been proven. The best approach is to go with something simpler that’s optimized for the 90% and then rely on machine learning to fine-tune the engine.

Is the Recommendation Engine proven in your vertical?


  1. Case studies should be available for your vertical demonstrating a positive impact on the traditional email metrics, as well as return-on-investment (ROI) versus cost. Aggregate data by vertical is also useful to evaluate. If there are no case studies or testimonials relating to your type of business, you should probably walk away, unless you have agreed to be in a beta or pilot of some kind.
  2. Be sure to confirm how much data the model was “trained on.” Low numbers of profiles mean the model has not had to deal with enough long-tail cases. Several million subscribers or consumer profiles is a solid basis for a more intelligent data model.

Click here to download these questions in PDF format.

About Sandy Hathaway

Sandy Hathaway (@StartupSandy) has been driving business growth via strategic marketing for nearly two decades. She's was the co-founder and CMO of AVARI, a predictive intelligent content technology that can automatically personalize any type of email campaign, using any email platform.

    Enable registration in settings - general
    Compare items
    • Total (0)