Communicating with your users is a tricky business. There are so many variables involved in creating the right message, at the right time, for the right audience. Fortunately, Iterable experiments can help you determine, with confidence, which of your ideas users respond to the most and which versions of a campaign truly contribute the most to conversions.
This article explains how to experiment with Iterable campaigns so you can create experiences that users respond to favorably, while promoting the outcomes you seek.
In this article
A/B testing
A/B testing is the type of experimentation used in Iterable. With this type of experimentation, you apply a specific change to a variant of an original (control) campaign, then run both campaigns to test which is most effective.
Let's consider a case where you've created a welcome campaign that new users really respond to. You don't have a lot of time to craft new messaging, but wonder whether simply adding emojis to the subject line might lead to even more opens.
To test whether users respond favorably to this change before implementing it for all users who receive your welcome campaign, set up an experiment to send the original campaign to one group of users and a variant of the campaign with an emoji in the subject line to another group. Once the experiment ends, evaluate the impact of the change by reviewing the performance metrics in Experiment Analytics. If a variant performs better than the original campaign (perhaps it has more Opens than the original), consider using it as your new welcome message.
Use Iterable experiments to:
Test the effect of certain parts of a message (such as from name, subject line, or preheader) on opens and open rates.
Find a send time that maximizes opens and conversions.
Evaluate the potential to improve conversions with updates to a message's body.
Assess the impact of Send Time Optimization on opens and conversions.
Compare conversion rates for a campaign to those of users who don't receive it (a holdout group).
Create a variant of a campaign that you can use for future sends if it outperforms your original control campaign.
Required permissions
When working with experiments, you must have the following permissions:
- To view the settings for your Iterable project, you need the Manage Settings permission.
- To create campaigns and experiments, you need the Draft Journeys, Campaigns, and Experiments permission.
- To schedule or activate campaigns, you need the Manage and Launch Campaigns permission.
Terms you should know
As you work with experiments in Iterable, understanding these terms will be helpful:
| Term | What it means |
|---|---|
| Confidence | The likelihood that a variant's message will increase (if lift is positive) or decrease (if lift is negative) the conversion rate compared to the control, based on the selected success metric. |
| Control campaign or template | The original campaign or template on which you base a variant, identified as variant A. |
| Holdout group | A group of users you exclude from a campaign, used to test the results of sending a campaign compared with not sending it. |
| Lift value | The difference in performance between a test variant and the control with respect to the success metric that was selected during campaign setup. |
| Confidence interval | Provided with the Lift value, this reflects the conversion rate you can expect (based on the selected metric) if a variant is sent to all users. |
| Variant (or variation) | A version of the campaign you want to use for A/B testing. |