How to Run Automation Experiments

Automation experiments are an advanced feature available within Sendlane that you can use to run split tests (aka A/B tests) on your automation email messages!

With automation experiments, you're able to create a message and test multiple versions to a small portion of your audience to see which one performs best. Once the winner is determined, that version will be sent to the rest of your contacts. The best part of all? It's 100% automated!


Before Getting Started

In order to set up different automation experiments, you should first understand how to build an automation

You'll also want to be familiar with how to create a campaign


Running Automation Experiments

  1. Inside of the automation you'd like to experiment with, click through to choose a Send Message action, then select Email
  2. Depending on your current setup, either click Create New Email, or use the dropdown to duplicate an existing message
  3. Fill in any necessary details about your message and check the Experiments box

  4. In the Experiment Settings section, you'll set basic parameters for your test
    • Components - The specific features of the email you'd like to test (subject line, preheader text and/or content)
    • Duration - The specific parameters for your experiment:
      • Specific amount of time - send experiment versions for a set period of time (hours, days, weeks)
      • Number of emails - send experiment versions until a specific number of emails have been sent out
      • Ongoing - send experiment versions indefinitely
        Please Note

        If you choose specific amount of time or total emails to be sent, you'll also be prompted to decide which metric to focus on for the Winning Version:

        • Factor - Open rateclick rate, or revenue will be the metric that determines the winner
        • Time - Processing time once the duration has been achieved (this is similar to a wait time between actions)
  5. Click Next: Content to begin building your first email (Test A) by filling in any content sections you'd like to have across all versions (those items will be up top) and the test versions (towards the bottom)
  6. Once you've finished test A, either click Blank Experiment to start from scratch or Copy A to duplicate your original version
    Please Note

    You can create up to 4 different test versions within each experiment and select any combination of components to test.

  7. Click Save, then repeat steps 5 and 6 until you've created the desired amount of test email versions (A-D) for your Experiment
  8. Once you're ready to begin your experiment, click Activate to launch your test versions!

Next Steps

For any email sent using the experiments feature, you'll see the experiments icon in that node:

If you ever want to choose a winner manually, simply click into the Send Email node, and you'll see the Make this a winner button available for each of your test versions:

From there, the experiment will end, and the winning version will be sent to all new contacts flowing through your automation!


Additional Resources


TL;DR

Inside of automation   → Send Message → Email → Create New Email (or duplicate existing) → Fill in details → Check experiments → Set parameters (component, duration and winning version if applicable) → Next: Content → Build Test A (fill in content) → Click Blank Experiment or Copy A → Repeat steps until desired tests are created → Save Activate

Still need help? Get in touch! Get in touch!