Experiment Analytics

A variant that's selected by Iterable as the winner of an experiment has been found to have performed better than the other variants in the experiment as measured by opens, clicks, or custom conversions (defined when the campaign was set up).

This article explains how to evaluate and export the data that reflect the performance of variants that are part of an experiment. For information about the metrics that Iterable uses to select a winning variant, see Experiment Winner Selection.

To view the analytics associated with an experiment, navigate to Messaging > Experiments and click the experiment you're interested in. From there, view high-level details about an experiment on the Overview tab, and more detailed performance-related information on the Performance tab.

# In this article

# Experiments Overview tab

The Overview tab provides a link to the campaign on which an experiment was based and summarizes certain details that might be helpful to share with senior stakeholders who want to understand how experiments are being used to improve messaging.

• View details about the experiment and the associated campaign, including a contextualized banner that summarizes the state or outcome of the experiment.

• View pre-selected analytics and details about each variant.

• View details about the experiment's configuration.

• End an experiment early without declaring a winner, perhaps because the experiment results were inconclusive, by clicking End Experiment. The control will be used for all future sends.

• End an experiment early, perhaps because a variant was clearly performing better than others, by clicking Declare winner. The selected variant will be used for all future sends.

# Evaluating experiment analytics (Overview tab)

In addition to the summary at the top of the page, this view shows the following metrics for each campaign in the experiment:

• Sends represents the total number of messages sent for each variant. If all sends for a campaign total an amount that's less than the list size, Iterable may have skipped sending some messages for one of the reasons described Send Skips.

• Unique conversion rate column reflects the success metric selected for this experiment. See Metrics Definitions for information about the metric you see.

• Lift represents the difference in performance between a test variant and the control with respect to the success metric that was selected during campaign setup (see Select winner based on).

  For example, if an experiment was set up to use Clicks as a success metric (measured as a unique conversion rate), a Lift value of 5.0pp indicates that the test variant's click rate (Unique Clicks / Total Sends) was 5% higher than the control variant.

• Confidence interval reflects the conversion rate you can expect (based on the selected metric) if a variant is sent to all users. For example, if an experiment has an open rate of 50% for a subset of users, ± 2 means that you can expect a 48% to 52% lift in opens if the selected variant is sent to all users.

• Confidence represents the likelihood that a variant's message will increase (if lift is positive) or decrease (if lift is negative) the conversion rate compared to the control, based on the selected success metric.

# Experiments Performance tab

The Performance tab allows you to dig deeper into your experiment results, providing details that may be helpful to share with a BI team or a more experienced data analytics team.

• Filter variant performance by time (Experiment Period, Experiment Start to Now, or Custom Timeframe). The experiment period is selected by default, but you can expand the view to a period of your choosing. For example, if you want to see how a campaign performed after the winning variant was selected (when it sent to the users who didn't receive it during the experiment period), extend the Viewing metrics recorded value past the end of the experiment.

• Review each variant's performance using a default set of funnel metrics or metrics you select.

• Export experiment metrics to share with others or to analyze further. You can export all available metrics or just those you select.

• Review variant performance over time for any available metric.

# Customizing experiment analytics (Performance tab)

In addition to the default metrics that are displayed on the Performance tab, other metrics may provide additional helpful insights. For example, if a variant that tested a new subject line was selected as the winner of an experiment, the metrics displayed by default tell you by how much it outperformed the control in terms of conversions. But, what if you also want to verify that the variant didn't lead users to unsubscribe? In that case, adding Total Unsubscribes to the metrics you review on the Performance tab would be helpful.

# Exporting experiment metrics

The Performance tab also provides ways to export data so you can share it with other stakeholders or tools. You can export all metrics, or just the metrics that are selected in your current view.

• To export the metrics for the selected experiment in a CSV file, from the Performance tab in Experiment Analytics, click the Export current view or Export all metrics.

• To pull historic experiment data across all experiments, or export multiple experiments at once, use the GET /api/experiments/metrics API endpoint.