Evaluating Holdout Groups

Read this article to learn how to view holdout group analytics and find the users who were included in a holdout group. For information about setting up a holdout group, see Setting up an Experiment.

# In this article

# Analyzing holdout group conversions (Campaign Analytics)

To use the Campaign Analytics page to analyze holdout group conversions:

1. Navigate to Messaging > Campaigns, and open your campaign.

2. Click the drop-down menu in the upper-right corner of the page and select the metrics you're interested in.

   

3. Click the drop-down menu on the left side of the page and select the holdout group.

   

4. Scroll down and review the purchase or conversion metrics that you selected.

# Analyzing holdout group conversions (Experiment Analytics)

To use the Experiment Analytics page to analyze holdout group conversions:

1. Navigate to Messaging > Experiments and open the experiment associated with your campaign.

2. In the upper-right corner of the variations table, select the purchase or conversion metrics you're interested in analyzing, and use the table to compare the results:

   

3. Below the table, select a purchase or conversion Metric to analyze, and use the charts and graphs to compare the holdout group to your campaign.

   

# Viewing holdout group metrics in a user's profile

When an experiment includes a holdout group, Iterable adds a Send Skip event to the user profile for each user who doesn't receive the campaign. The result looks something like this:

{
    "reason": "ExperimentHoldOut",
    "createdAt": "2021-03-10 23:50:30 +00:00",
    "campaignId": 2087681,
    "itblInternal": {
        "documentCreatedAt": "2021-03-10 23:50:30 +00:00",
        "documentUpdatedAt": "2021-03-10 23:50:30 +00:00"
    },
    "messageTypeId": 21356,
    "contentId": 11349462,
    "messageId": "f81b735493064cc39b650ed467f1c98d",
    "eventType": "emailSendSkip",
    "templateId": -1,
    "email": "user@example.com",
    "channelId": 19477
}

# Determining which users were in a holdout group

To find out which users received an Email Send Skip event:

1. Navigate to Audience > Segmentation.

2. Set up a query to find users who received a send skip event for the campaign with reason set to ExperimentHoldOut.

3. Click Refresh Results to list users who were in the campaign's holdout group.

   

# Want to learn more?

• Creating Blast Campaigns
• Tracking Purchases and Revenue
• Tracking Custom Conversions
• Metric Definitions
• Managing Custom Events