Send An A/B Split Test

About this task

An A/B split test is scheduled similar to a normal delivery. First you configure the settings for your split and then you set up the delivery like a normal send.

To setup an A/B splits test:

Procedure

  1. Go to Messages > All Messages.
  2. Click on Email.
  3. Click on the name of an email message you want to perform an A/B split test on.
  4. Click the A/B Split radio button in the Schedule Delivery section.
  5. Click Schedule.
  6. Select what you'd like to perform the test from the Split Content list:
    • Subject Line: lets you test different subject lines for your message. Use this if you want to see how varying subject lines affect open rates.
      Tip: For tips on how to write effective subject lines, see Effective Email Subject Lines.
    • From Name & Address: lets you test different from names and from addresses used in your message. Use this if you want to see how varying the from name and from address affects the open rate for your message.
    • Full Message Content: allows you to monitor how minor tweaks to message content can affect the metrics associated with user actions that occur after the message is opened, such as clicks and conversions.
    • Date and Time of Send: allows you to see how altering the send date and time can affect the open rate for your message.
  7. Select the type of split test to run in the Type Of Split section:
    • A/B-Winner: tests messages with two or more groups to see which is most effective. After a set period of time, a winner is chosen based on preselected criteria, such as highest open rate or highest number of clicks. After the winner is selected, the winning message is sent to the remainder of your targeted contacts.
    • Champion/Challenger: similar to A/B-Winner tests, except there isn't a winner group. Instead you have an established way of doing something, called the Champion, and you want to test if it is still the best by introducing a Challenger. For example, use this to test a new template design or header image compared to your current standard. When creating a Champion/Challenger split, it is common to send most of your contacts the champion, while setting aside a smaller portion for the challenger. Once the test is run, there is no winner group who receives a message, but you do have a good idea about whether or not your champion is as good as you thought it was.
  8. If you chose A/B-Winner: Select the winning criteria from the Winning Criteria list.

    This allows you to specify which metric determines the best performing group.

    Note: If you choose Champion/Challenger, skip this and go to the next step.

    The winning criteria you can choose from are:

    Criteria Description Of Criteria Description Of Metric
    Highest Open Rate The winner will be determined by which version of the message has the highest open rate. For more information on the open rate metric, see Open Rate Metric.

    The open rate represents the percentage of emails that were opened, as compared to the number of emails that were successfully delivered. DMA/EEC equivalent: Render Rate The calculation does not take into account the send history for individual messages. Therefore, on Over Time and Date Range reports, it is possible for the rates to equal a value greater than 100% if enough messages sent on a date prior to the start of the report are included in the calculation. For example, a message was sent 100 times a previous day and 100 times on a date within the report range. The message is opened 125 times within the report range. In this case, the Open Rate on the report is 125% because the message was opened more times than it was sent.

    Highest Click Rate The winner will be determined by which version of the message has the highest click rate. For more information on the click rate metric, see Click Rate Metric.

    The click rate represents the number of unique clicks that were recorded, as compared to the number of unique opens recorded. DMA/EEC equivalent: Click to Open Rate (CTO) The calculation does not take into account the history for individual messages. Therefore, on Over Time and Date Range reports, it is possible for the rates to equal a value greater than 100% if enough messages sent on a day prior to the start of the report are included in the calculation. For example, a contact may have opened a message on a previous day, but clicked on the message on a day within the report range.

    Highest Conversion Rate The winner will be determined by which version of the message has the highest conversion rate. For more information on the conversion rate metric, see Conversion/Delivered Rate Metric.

    The conversion/delivered rate represents the percentage of conversions made, as compared to the number of emails delivered.

    Highest Revenue Per Contact The winner will be determined by which version of the message has generated the most revenue per contact. The revenue per contact is determined by dividing the conversions by the sent number. For more information on the conversions and sent metrics, see Conversions Metric and Sent (Emails) Metric.
    Highest Revenue Per Click The winner will be determined by which version of the message has generated the most revenue per click. The revenue per click is determined by dividing the conversions by the clicks For more information on the conversions and clicks metrics, see Conversions Metric and Clicks Metric.
    Highest Click Through Rate The winner will be determined by which version of the message has the highest click through rate. For more information on the click through rate metric, see Click Through Rate Metric.

    The click through rate represents the number of clicks that were recorded, as compared to the number of emails delivered.

    Highest Conversion/Delivered Rate The winner will be determined by which version of the message has the highest conversion/delivered rate. For more information on the conversion/delivered rate metric, see Conversion/Delivered Rate Metric.

    The conversion/delivered rate represents the percentage of conversions made, as compared to the number of emails delivered.

  9. Specify your split groups in the Split Groups section.
    1. Enter a name for each group in the Name box.
    2. Set the percentage for each group in the Size box.

      The contacts in split groups are randomly chosen from your targeted contacts. Here you are only setting the size of those groups, which is a percentage of the whole group.

    3. To add more groups, click Add Another Group.

      You can create a maximum of 20 split groups.

    For A/B-Winner split tests, try to keep the total for your test groups to under 50% of your list. This ensures your winner group will be the largest group. If you plan to use two groups, then 20% and 20% is good choice. Reduce the size of the groups as you add more groups. As you create groups and specify sizes, the Winner group is automatically updated to contain the remainder of the percentage.

    >

    For Champion/Challenger, there is no Winner group so you need to allocate a full 100% amongst the groups you create.

  10. Click Next.
  11. Assign the content or time for each split group.

    The options here will vary based on what you selected in step 6. If you choose to test on Full Message Content, you will need to select an alternate message version to use for the test.

  12. Click Next.
  13. In the Sender Options section, specify the following:
    • From Name: Also called an alias, this is the name that appears to the recipient of the email message.
    • From Address: The email address that your email message will appear from.
    • Reply-To Address: The address that the application will forward email replies to.
  14. In the Select Recipients section, select the lists and segments that you would like to send to.
  15. In the Delivery Scheduling section, specify when you want the test to go out.

    For an A/B-Winner test, you also need to pick a time for the winner to be sent. The winner will be selected and Bronto will begin delivery preparation at this time. This means there could be a delay of 30-60 minutes from the time you select and when the winning message is sent.

    To ensure the best winner is selected, make sure you give enough time for your contacts to engage with your test messages. If you throttle the message, it may take more time for the message to go out.

    When you specify the lists and segments who will receive your message, remember that the percentage you select for your Split Groups applies to the total number of contacts selected as recipients. For example, you choose 2 lists and 3 segments as recipients. Your test uses 2 Split Groups, each sized at 50%. In this case 50% of the total number of contacts selected as recipients receive Split A, and the other 50% receive Split B. So, if the total number of contacts on the lists and segments you selected as recipients was 100, then that means that 50 contacts received Split A, and the other 50 would receive Split B.

  16. Optional: Click Show Advanced Options to view and configure the advanced sending options.
    For more information about
  17. Click Next.

    The verification screen is displayed.

  18. Double check all of your selections and then click Send Delivery to begin the A/B Split test.