November 14, 2014

Responsive Design A/B Testing Leads to a 130% Increase in Clicks

[ 0 By Lauren Smith

With a return on investment (ROI) of 4,300%, it’s no wonder why email is used throughout the entire lifecycle of a customer. From initial interactions to purchases to post-purchase messages, email is ever-present. And, through A/B testing your emails, you can make this ROI even greater—all while gaining insights into your audience. Through testing, you can learn what resonates best with your audience so you can create better performing emails.

When it comes to A/B testing your emails, ignore the benchmarks. No one has the same audience and email strategy as you—your audience is unique. However, using others’ A/B tests, as well as industry trends, as inspiration is a great idea to get your testing ideas flowing. In this post, we’ll take a look at how Act-On Software reacted towards a continuous increase in mobile opens to create an A/B test revolving around responsive design. And, who knows—maybe you’ll find inspiration from their test!

Testing the effectiveness of responsive design

With mobile opens often as high as 50%, there has been a big push in the email industry to use responsive or mobile-friendly design. With this trend in mind, Act-On Software wanted to test the impact responsive design had on their emails.

Let’s take a look at their hypothesis, the test itself, and the results.

The hypothesis

Would Act-On subscribers be more engaged with responsive emails? Would responsive email design lead to an increase in their sales ready leads (SRLs)? Act-On Software’s hypothesis was that the responsive email would lead to an increase in both engagement and SRLs, but, since they had never done this type of test before, they were unsure of what to expect.

The control: Non-responsive design

The control version is a simple HTML design that involves mostly live text and a large image. There is a large bulletproof button with the call-to-action (CTA) to “Download the toolkit.” The image also links to the landing page. In addition, it’s not very mobile friendly. The text is extremely small and difficult to read.

The test: Responsive design

While using similar content and the same subject line and “from” name as the control, the test version is different in a number of ways. For starters, with the exception of the header image, logos, and social sharing links, the entire email is live text. They have removed the large image from the center of the email. In addition, they have simplified the email by removing the “About Act-On Software” paragraph at the bottom of the email. While the CTA button is the same, they have hyperlinked “free toolkit” in the text as a secondary CTA (rather than the huge image in the control version).

Since this email is responsive, it looks great and is easy to interact with on a mobile device. The text is enlarged and legible, and the CTA button is easily tappable.

The process: How did they go about the test?

After coming up with their hypothesis and the data points they would be measuring (unique open rates and SRLs), they designed both variants of the email. Before sending, they used Litmus’ Email Testing tools to ensure that both versions rendered properly across 30+ clients. Then, they set up the A/B test within the Act-On Software platform.

The audience for this test was those with marketing titles, across industries, US-based, and those that were part of Act-On’s “top of funnel” segment. They all had a lead score of zero, which means that they had not participated in any lead scoring behaviors within the last 90 days. The list size was 40,000 and both versions of the email were sent on the same day, only 15 minutes apart.

• Version A (control): Sent to 20,000 at 11:30am PT on Wednesday
• Version B (test): Sent to 20,000 at 11:45am PT on Wednesday

Responsive design for the win!

48 hours after sending the test, Act-On looked at the results—and they were even more drastic than they imagined.

The test version resulted in a 130% increase in clicks and a 93.3% increase in SRLs. Talk about a successful test! With such significant results, Act-On converted all of their emails to responsive

Why were the results so significant?

Perhaps the significant increase in clicks and SRLs is a result of the simplified email and removal of the large image, but it’s likely a result of the responsive elements. With larger text and a touch-friendly CTA, version B is much easier to interact with on a mobile device. If viewed on a mobile device, version A would require the subscriber to zoom in and navigate the email in order to properly read it—this would likely frustrate the subscriber and cause them to abandon it. In fact, 80% of people delete an email if it doesn’t look good on their mobile device.

MORE RESPONSIVE DESIGN TESTING INSPIRATION

Act-On Software isn’t the only one that has A/B tested responsive design and seen significant results. Let’s take a look at others who have seen success from going responsive:

• Deckers saw a 10% increase in clicks when A/B testing responsive design vs. non responsive.
• Kitbag saw a 9% open rate increase when testing responsive design vs. non responsive.
• While these were complete design overhauls rather than A/B tests, Edinburgh Bicycle Cooperative saw a 13% increase in mobile clicks and 30% increase in mobile site visits and Auto Trader saw an 89% increase in open rate and 391% increase in click rate after moving their designs over to responsive.

Like we said earlier, don’t use these results as baselines for your testing; however, use these testing experiments (and successful results!) as inspiration to kickstart A/B tests of your own. While we definitely recommend using mobile-friendly designs due to a continuous increase in mobile opens, there are so many other things to test, too—subject lines, from names, etc. It’s all about learning more about your audience and optimizing your emails for them!

OPTIMIZE EMAILS FOR YOUR AUDIENCE

Optimizing your emails for your audience is a win-win—they’ll receive emails that they want and you’ll get better results. From the results above, it’s obvious that A/B testing is a great way to learn what resonates best with your subscribers.

Before planning your tests, discover everything you can about your audience to base your tests off of—where are they opening your emails? Are they engaged with your emails and actually reading them, or simply deleting them? If you’re seeing a large percentage of your audience opening on mobile devices, perhaps you should try a responsive design test like Act-On. You can use the engagement data as baselines to compare your testing results to—are they more engaged with the responsive email? All of this information can be found through Litmus’ Email Analytics tool.

Once you determine your A/B tests, be sure to preview your designs. Two versions of an email means even more to test and more that could go wrong. With Litmus’ Email Testing tool you can preview both versions of our email in over 30+ clients in just minutes!

Get started with responsive email templates from Litmus