Recently, one of our customers tried to run an A/B test as part of a blast email he was sending out. He ran into multiple issues while trying to keep his old email blast program structure intact, and contacted us for help. The A/B tests he wanted to set up were:

- The email’s content (‘Whole Email’)

- The email’s subject line (‘Subject Line’)

- The email’s date and time (‘Date/Time’)

As part of the A/B test, our customer wanted to use the ‘declare winner - automatic’ option. It was most important to him to keep all processes intact since the batch campaigns program templates worked very well for him until now. He just wanted to add the A/B test functionality to the flow. As you may know, Marketo’s built-in A/B testing is only meant to be used in an Email Program. Using an email program easily allows you to create a test, to determine the date and time of the test and the date and time of the main blast. The A/B testing dashboard is most helpful. It has a step-by-step structure, ending at the last step of approving the program.

 
 Send Email Program - Case Study
 

Our customer's program template had nine smart campaigns. The first of these was designed for sending the email, and the other eight were used for ‘program status change.’ This means the actions that the lead performs after receiving the email. Some of these actions were connected directly to the email statistics – such as ‘open’ and ‘click’, while others were related to activities in Salesforce.

 

All of the triggers and filters used in these eight smart campaigns (see below for more details on the send email smart campaign) used the original email name.

 
Email Campaigns Marketo
 Email Campaigns Marketo
 

As some of you may know, when using an email within an email program for A/B testing, Marketo will NOT recognize it anywhere else. This can be a major hindrance for some users, and for our customers, it definitely was one. The old structure of the program became irrelevant, and it was certainly possible that the tracking system of the program may have become irrelevant as well.

 This may sound like a disaster, but the solution is quite simple:
 The first step is to approve the email program (the lower right rectangle). Make sure not to skip this step or it will be impossible for Marketo to recognize the email.

The next step is to fill in all of the fields in all triggers and filters that require the email name with the following syntax:

[parent program name].[email program name].[A/B test name] test

 

For example:

Parent program name: Black Friday

 Email program name: Send Email
 A/B test name: subject line
 Final syntax: Black Friday. Send Email. subject line test.
 

Send email smart campaign:

 A control group is used to check how an email affects the population. A "random sample" of 10% of the population is taken out, and the email is sent to the other 90% of the group.
 

Using an email program makes this process impossible since it was built by Marketo as a smart campaign with one flow step – send an email.

 

Our solution was to create a "preparation" smart campaign. Its smart list included the population for the program, and the flow step separated it into a control group and a target group (via change program status).


 
Send Email Program - Case StudyChange Program Status

 

The last step in the process is to set up the smart list of the email program to include only the recipients marked as "Target Group.”

 Just don't forget to run the preparation campaign before the A/B testing time.