A/B Split Tests

A/B split tests allow you to automate the tedious steps of testing different message types and provide access to valuable testing data with a minimal investment of your time.

Do you ever have a problem deciding between a few different subject lines? Maybe you were sure that one was better, but a co-worker disagreed? Well, maybe you're both wrong. A/B split tests allow you to quickly compare multiple versions of a message in a way that lets you discover the best performing option. Whether it's the best subject line, or a choice between two images, A/B split tests take judgment calls out of the equation.

An A/B split test starts with between two and twenty groups of contacts chosen randomly from the contacts targeted in a delivery. A message is sent to those groups with a variable, such as different subject lines, From Name & Addresses, different message content, or date and time of send variations. After the messages are delivered, you can review the results and identify which version performed better. This information can drive future marketing choices. The following tips will help you set up a successful A/B split test:

  • The name “A/B split” indicates that you are choosing between varying versions of an email message. Always remember to minimize the difference between the different versions you create. If you change many things in each test, then you may have a hard time understanding why one group outperformed another. Rather than having multiple variables, you should consider changing only one thing at a time and running several splits over a few deliveries.
  • Have you ever had a subject line that was pretty good, but didn't make the cut? Try a Champion/Challenger test with it, and you may be surprised by the results. Try that every couple of deliveries, and you might find yourself doing a lot more A/B-Winner splits.
  • From Name & Address tests are good for a new list or mailing, but aren't a great idea on an established list. Remember that changing your From name or address can cause confusion amongst contacts and even cause you to lose inbox whitelisting with some ISPs.
  • With Full Message Content tests, remember not to get pulled into the trap of using two drastically different versions of a message. Rather, use this type of test to examine how simple design tweaks, such as order and positioning of links and images, affects the amount of clicks your message receives. You may be surprised to see how simple changes, like moving your call to action to the area of the message above the fold, will help drive clicks in your message.