Glossary of Statistical Terms:
Confidence: The likelihood that the difference in conversion rates between a given variation and the control or baseline is not due to chance. Your statistical significance level reflects your confidence level.
Difference Interval: Difference intervals tell you the range of values where the difference between the original and the variation actually lies. The difference interval is a confidence interval of the conversion rates that you can expect to see when implementing a given variation. Think of it as your "margin of error".
For example if my open rate is 50% and my difference interval is + 2%, then my difference interval is from 48% to 52%.
Confidence Level: What percentage of confidence you can say the given outcome will occur within the difference interval--this is set at 90% for experiments in Iterable.
The A/B Test Report
The A/B test report uses email events or conversions (purchases or custom events) to calculate the confidence intervals, open, click, purchase or conversion rates, and statistical significance.
An example of an A/B test report is shown here:
- Open Rate: this column shows the percentage of unique email opens as well as the difference interval
- Click Rate: this column shows the percentage of unique clicks opens as well as the difference interval
- Revenue: this is the total revenue generated from the variation
- Purchases: this is the number of purchases made from that variation
- Revenue / M: this is the revenue generated from the variation per thousand emails sent
- Purchase (or Conversion) Rate: this column reports the number of conversion events received and the number of view events received as well as the difference interval
- Improvement: this column reports the percentage Improvement of the Test variation compared to the baseline
- Confidence: the likelihood that the difference in conversion rates between a given variation and the control or baseline is not because of chance
The values used in the report are calculated as noted below.
Conversion Rate and Conversion Rate Change for Variations
For each variation the following is calculated:
- Conversion rate:
- The percentage change of the conversion rate between the Test variation and the Control variation: