Design + Data = A Match Made in Heaven?

[ 0 By

About a month ago I had the opportunity to moderate a panel discussion at the Email Insider Summit in Captiva Island, Florida. The idea was to bring together designers and marketers to talk about creative testing and the pros and cons of data-driven email design. No matter which side of the coin you’re on (I have the advantage — or disadvantage — of being both at Litmus), you’ll probably agree that balancing data-driven marketing decisions with a desire to be creative and inspirational is no easy task. Add the fact that A/B testing with most email tools is relatively cheap and easy, and there should be no excuse for testing only your subject line for the millionth time. The possibilities for testing email creative are endless, and can range from link colors to complete differences in look and feel.

I had some really smart and talented folks on the panel: Brian Brown (a partner in brand experience agency ideapark), Jay Jhun (who heads up email services at Engauge) and Lisa Dick (online marketing manager at Helzberg Diamonds) offered up a ton of great insight along with some case studies.

Test the small (or even tiny) things. Lisa had amazing stories about testing small changes in creative, sometimes for tremendous gain. In one example, moving product pricing and descriptions from one side of the email to directly under each product nearly doubled revenue generated from the send. In another, her team tested two button designs: one with an arrow, and another without. The buttons with arrows had a higher click rate!


It seemed for every win, though, was another test that showed inconclusive results. All the panelists agreed that even no result is a result, and can help you learn what types of tests or tweaks to try in the future. The bottom line? You never know until you try.

Set expectations for branding vs. performance. Brian told us about a test where marketing management was so laser-focused on performance that they had started sacrificing brand elements in their email design. The email was performing extraordinarily well with each test iteration, but was becoming increasingly less recognizable as part of the brand. In another example, marketing was dead-set on using non-web-safe brand fonts on all digital communications, but in a secret test, a web-safe alternative was shown to outperform the control since text was visible with images turned off. At what point does brand trump performance, or vice versa?

Is email design art or commerce? Everyone agreed that it’s a little bit of both. Collaboration and communication are key in organizations where data-focused marketers and designers striving for creativity and inspiration are working together. A beautiful design may perform very well, or could return negative results. Encourage designers to brainstorm new tests and invite them to participate in setting goals.

For the marketer: Don’t suck the soul from designers. It’s important not to start from a soulless position. Adopt a “you work with me” vs. “you work for me” attitude.

For the designer: Get excited about testing and data. What do you want to test? Participate in the testing process by thinking of new things to test. For every design principle there is probably a test you could run! Also, see Josh Porter’s presentation (in Resources, below) for 5 reasons why metrics are a designer’s best friend.

The takeaway? Design and data can be a match made in heaven as long as everyone respects each other’s position. Remember that everyone has a job they are trying to do. If you’re not testing creative or design elements already, you could be leaving money – and creativity – on the table.


Joshua Porter, a designer and co-founder of Performable, gave an excellent talk at SXSW on metrics-driven design. You can download his presentation, and he also has eight Principles of Metrics-Driven Design:

  • Optimize in small steps; innovate with daring leaps.
  • No design survives contact with the user.
  • Small improvements, taken together, yield amazing results.
  • Testing is empowering, reversion is cleansing.
  • Metrics are not creative: human beings are.
  • All team members are responsible for the user experience.
  • If metrics aren’t actionable, they aren’t useful.
  • Design is never done.

MarketingExperiments and ExactTarget published this case study and webinar on design testing. They redesigned emails for three different clients (using three different agencies) and did an A/B/C/D test against the control to determine winners.

A Beginner’s Guide to A/B Testing: Email Campaigns That Convert: This blog post by KISSmetrics is a great guide to getting started with email testing, and has some great suggestions for things to test.

Which Test Won? is a website dedicated to displaying the results of digital A/B tests.

Data, Not Design, Is King in the Age of Google: This article from The New York Times highlights a designer that left Google for focusing too much on data when it came to design.

You can also watch a recording of the panel discussion from the Email Insider Summit or view the slides with examples of creative testing.