This document describes how to create Send Time Optimization experiments with email blast campaigns. It's not currently possible to use Send Time Optimization experiments with push notifications.
For email blast campaigns, Send Time Optimization (STO) attempts to maximize the Email Open Rate metric (defined as Unique Email Opens or Clicks / Total Emails Delivered). To do this, it uses machine learning to analyze each recipient's historical engagement behavior (in the same Iterable project), and then chooses a send time when they're likely to open the message.
Send Time Optimization experiments use optimized times for half of a blast campaign's recipients and the campaign's configured send time for everyone else. They allow you to measure the improvement Send Time Optimization makes on a campaign's Email Open Rate.
For information about using Send Time Optimization with campaigns and workflows, read:
Table of contents
Currently, Send Time Optimization:
- Can be used with:
- Blast email campaigns (with or without a Send Time Optimization experiment)
- Workflow Send Email nodes (without a Send Time Optimization experiment)
- Blast push notification campaigns (without a Send Time Optimization experiment)
- Workflow Send Push nodes (without a Send Time Optimization experiment)
- Can only be used with marketing messages (not transactional).
- Can only optimize send times based on project-specific data (it does not analyze cross-project data for the same user).
- Can only select optimized send times aligned to the top of the hour.
- Can only use a 50/50 control/variation split for Send Time Optimization experiments (this is not adjustable).
- Does not guarantee higher engagement.
Additionally, note that:
- For users with insufficient historical project data from which to derive optimized send times, Iterable always sends campaigns at their configured send times.
- Iterable updates its Send Time Optimization machine learning models once per week, using the newest available data for each project (weighting recent data more heavily). Given the large number of projects in Iterable, all models are updated over the course of the week and not on any set schedule.
To create an email blast campaign with an associated STO experiment:
1. Create an email blast campaign
To create an email blast campaign, navigate to Messaging > Campaigns and click Create New Campaign. On the New Campaign Screen:
Specify a Campaign Name, choose whether to use a Custom Conversion Metric, and add Labels as necessary.
For Campaign Type, choose Blast.
For Message Medium, choose Email.
Select the list to which you'd like to send the campaign.
Click Continue to Templates.
Select an existing template and click Continue to Design, or click Create a New Template. Configure your template as necessary.
You'll now be on the Edit Template screen.
2. Create an STO experiment
To use STO for half of your campaign's recipients and a configured send time for the rest, create an STO experiment. To do this:
On the Edit Template screen (where you left off in the previous step), click Create A/B Experiment:
On step 2 (Setup) of the experiment creation sequence, enter an Experiment Name and select Send Time Optimization in the Experiment With section:
If the Send Time Optimization option isn't visible, it's because your Iterable project does not have enough historical user engagement data to calculate optimized send times.
On step 3 (Design) of the experiment creation sequence, click Review (in the upper-right corner) to continue on.
On step 4 (Review) of the experiment creation sequence, click Save & Continue (in the upper-right corner) to continue on.
You'll now be on the campaign's Review & Launch screen.
3. Send or schedule the campaign
On the Review & Launch screen:
Notice that the Send Time Optimization toggle has been enabled.
If the toggle isn't visible, it's because your Iterable project doesn't yet have enough historical engagement data to support the Send Time Optimization machine learning models.
Specify the maximum number of hours (between 6 and 24) after the campaign's configured send time (whether immediate or scheduled) that Send Time Optimization can send messages. Send Time Optimization optimizes send times at a one-hour granularity and sends messages at the top of the hour.
To send the campaign immediately, click Send Campaign Right Now.
To schedule the campaign for later or have it recur, click Schedule Campaign For Later. Specifying a recurrence pattern for an Send Time Optimization campaign causes each recurrence to use Send Time Optimization. However, STO experiments do not recur.
After a campaign begins to send, you cannot disable Send Time Optimization.
3. Monitor experiment results
To monitor the results of an STO experiment:
Wait until the experiment is active (after the campaign's specified send time).
Navigate to Messaging > Experiments.
Select a tab: Completed, Running, Draft/Scheduled, or Canceled/Errored (depending on the experiment's state).
Click an experiment's name to view its variations and their associated metrics. The variations will look similar to this:
For STO experiments, do not click the Use Variation button.
Iterable sends the experiment's Control variation to half of the campaign's recipients at its configured send time, and the STO variation to the other recipients at per-user optimized send times.
Use the Showing X Metrics drop-down menu in the upper-right corner of the table to choose which columns the table should display.
To choose metrics to analyze with charts and graphs, use the Metrics to Analyze drop-down menu below the table:
Since STO optimizes Email Open Rate, this is a good metric to analyze.
To learn more about experiment metrics, read A/B Experiment Metrics.