Glossary of statistical terms
Confidence: The likelihood that the difference in conversion rates between a given variation and the control or baseline is not due to chance. Your statistical significance level reflects your confidence level.
Difference interval: Difference intervals tell you the range of values where the difference between the original and the variation actually lies. The difference interval is a confidence interval of the conversion rates that you can expect to see when implementing a given variation. Think of it as your "margin of error". For example if an open rate is 50% and the difference interval is +2%, then the difference interval is from 48% to 52%.
Confidence level: What percentage of confidence you can say the given outcome will occur within the difference interval. This is set at 90% for experiments in Iterable.
#The A/B test report
The A/B test report uses email events or conversions (purchases or custom events) to calculate the confidence intervals, open, click, purchase or conversion rates, and statistical significance.
An example of an A/B test report is shown here:
- Open Rate
- Percentage of unique email opens and the difference interval
- Click Rate
- Percentage of unique clicks opens as well as the difference interval
- Total revenue generated from the variation
- Number of purchases made from that variation
- Revenue / M
- Revenue generated from the variation per thousand emails sent
- Purchase Rate
- Number of conversion events received and the number of view events received as well as the difference interval
- Difference between the click rate for the control and for the variation
- Difference in conversion rates between a given variation and the control or baseline is not because of chance
The values used in the report are calculated as noted below.
#Conversion rate and conversion rate change for variations
For each variation the following is calculated:
The percentage change of the conversion rate between the test variation and the control variation: