After you create an experiment, you're ready to add variants to it based on the experiment type you selected.
For some examples of the changes you might consider for the type of variant you're creating, see Planning an Experiment.
TIPS
To generate the most conclusive results and to limit the effort required to assess experiment results, limit the number of variants you create—one or two should be enough to test your hypothesis. We also recommend that you test one hypothesis at a time.
If you intend to make changes to the control (original) campaign you're experimenting with, do so before you create experiment variants. Any changes you make to a control campaign after creating variants for an experiment are not reflected in the variants.
Be sure to review all variants before launching the experiment—you want to avoid changing an experiment after you launch it, as doing so may cause inconsistent results.
If you're using Send Time Optimization with a campaign and set up an experiment with the “test with subset” strategy, we recommend ensuring that the experiment duration is at least as long as the Send Time Optimization send window. This will ensure that all users in the experiment group will have received the campaign and have time to convert before a winner is declared.
This article focuses on email campaigns, but variants are created the same way for other campaign types, with different fields to experiment with.
# In this article
# Creating variants for from name, subject line, or preheader
The steps involved in creating a variant that tests opens and open rates is the same whether you're testing from name and sender, subject line, or preheader, you just specify values for different fields.
-
In the Variant setup section, click Add variant.
For each variant you want to create, enter the values you want to experiment with. For example, maybe you want to create a variant of a subject line that includes an emoji.
You can enter values manually, or click Iterate for Copy Assist to generate choices for you.
-
To add another variant, click Save and add variant, otherwise, click Save.
NOTE
If your campaign uses locales, be sure to update each variant for each locale. Simply select the locale from the button at the top right of the screen and make the changes you made to the first locale.
Configure the experiment. See Configuring Experiments.
# Creating send time variants
You've already specified a send time for the control campaign. Now, specify send times for each variant you create.
In the Variant setup section, edit Variation name, if needed.
-
Specify a date and time for this variant.
IMPORTANT
If your campaign's send lists have highly dynamic membership, launch the send time experiment as close to your campaign's first send as possible. This way, the list's membership at the experiment's start will be similar to what it was at the start of the campaign.
To add another variant, click Save and add variant, otherwise, click Save.
Configure the experiment. See Configuring Experiments.
# Creating variants of a message body
Experiment with the message body to measure the effectiveness of a campaign (and its variants) on clicks, purchases, or custom events.
For example, maybe you want to experiment with a winback campaign that currently offers churned users a 25% discount on their next purchase. Perhaps you want to see whether they convert more readily when you highlight what they've missed while they've been gone.
TIP
To create a variant of a campaign that tests large-scale changes, try breaking them into discrete experiments. For example, create one experiment to test messaging, and another to test a new layout. This way, you can evaluate which change has the greatest impact.
In the Variant setup section, click Edit variant in template editor.
-
In the template editor view, make the changes you want to test, and click Save design.
For more information, see See Template Editors Overview
NOTE
If your campaign uses locales, be sure to edit each variant for each locale available from the drop down at the top left of the screen.
From the Variant setup page, click Save & Review.
To add another variant, click Save and add variant, otherwise, click Save.
Configure the experiment. See Configuring Experiments.
# Creating a Send Time Optimization test
To test whether a blast campaign has a better open rate with Send Time Optimization enabled (compared to the campaign's scheduled send time), create an STO experiment.
NOTE
You can't disable Send Time Optimization after a campaign starts sending.
To associate an STO experiment with a new campaign, after you attach a template and update its content, in Conversions and Experiments, click Edit and then Add Experiment.
-
On the New experiment page, choose Send Time Optimization and click Create experiment.
If Send Time Optimization isn't visible, your Iterable project doesn't yet have enough historical user engagement data to calculate optimized send times.
From the Variant setup page, you can see that the experiment and STO variation are configured for you. No additional work is needed. The experiment will launch when the campaign is sent.
-
(optional) On the Review & Launch page for the campaign, under Delivery | Enable Send Time Optimization, you can change the maximum number of hours (between 6 and 24) after the campaign's configured send time (whether immediate or scheduled) that Send Time Optimization can send messages. STO optimizes send times at a one-hour granularity and sends messages at the top of the hour.
WARNING
If you add a rate limit to your campaign, some messages may send outside of your STO window. Rate limits are a beta feature; talk to your Iterable customer success manager to add them to your account.
-
Schedule the campaign to send now or at a specific time.
NOTE
Specifying a recurrence pattern for a Send Time Optimization campaign causes each recurrence to use Send Time Optimization. However, STO experiments do not recur.
# Holdout group tests
It's not possible to create a standalone holdout group experiment. Instead, create one of the types of experiments mentioned in this article, and when you're setting it up, add a holdout group.
You don't have to add a variant to a holdout experiment if you don't need one. If you're only interested in measuring campaign conversions and holdout group conversions, simply skip the variant setup step. When your experiment is launched, it will be sent to the users who are not part of the holdout group.
NOTE
For any given experiment, you can create a single holdout group.
WARNING
If you're testing out Iterable's holdout groups functionality to get a
sense of how it works, and you make an API call to track a custom conversion
event for one of the holdout group's members, do not include a campaignId
in
the API request body. Iterable uses attribution periods to track holdout group
conversions. For a real campaign, a holdout group member wouldn't have received
the message, and therefore wouldn't have a campaign ID to associate with their
conversion event. If you do include a campaignId
for such a user, their
conversion won't be associated with the holdout group.
# Next steps
Configure the experiment. See Configuring Experiments.