Articles in this section

See more

Evaluating Holdout Groups

When you set up an experiment, you can set aside a portion of a send list to whom Iterable won't send a campaign, but for whom conversions are still tracked. This allows you to compare the conversion performance of users who don't receive a campaign to the conversion performance of those who do.

Read this article to learn how to view holdout group analytics and to identify users who were included in a holdout group. For information about setting up a holdout group, see Configuring Experiments.

In this article

Analyzing holdout group conversions (Campaign Analytics)

To use Campaign Analytics to analyze holdout group conversions:

  1. Navigate to Messaging > Campaigns, and open your campaign.

  2. On the Campaign Review page, click View analytics.

  3. On the Overview tab in Campaign Analytics, select the holdout-related metrics you want to view, depending on the message channel for the campaign. For example, for email:

    • Total Email Holdout reflects the number of email sends that were skipped because of an experiment holdout group.

    • Conversions / Email Holdouts reflects the total percentage of
      conversions for users who were part of a holdout group in an experiment that's associated with an email campaign.

Analyzing holdout group conversions (Experiment Analytics)

To use Experiment Analytics to analyze holdout group conversions:

  1. Navigate to Messaging > Experiments and open the experiment associated with your campaign.

  2. On the Performance tab, select a purchase or conversion metric for Variant breakdown or for Variant performance over time. Total Email Holdout and Conversions / Email Holdouts are holdout-specific metrics that might be helpful.

Viewing holdout group metrics in a user's profile

When an experiment includes a holdout group, Iterable adds a Send Skip event to the user profile for each user who doesn't receive the campaign. The result looks something like this:

{
    "reason": "ExperimentHoldOut",
    "createdAt": "2021-03-10 23:50:30 +00:00",
    "campaignId": 2087681,
    "itblInternal": {
        "documentCreatedAt": "2021-03-10 23:50:30 +00:00",
        "documentUpdatedAt": "2021-03-10 23:50:30 +00:00"
    },
    "messageTypeId": 21356,
    "contentId": 11349462,
    "messageId": "9c438a20d26e4e9f8ad4edea23f811cd",
    "eventType": "emailSendSkip",
    "templateId": -1,
    "email": "user@example.com",
    "channelId": 19477
}

Determining which users were in a holdout group

To find out which users received an Email Send Skip event:

  1. Navigate to Audience > Segmentation.

  2. Set up a query to find users who received a send skip event for the campaign with reason set to ExperimentHoldOut.

  3. Click Refresh Results to list users who were in the campaign's holdout group.

    Finding holdout group members

Want to learn more?

Related articles