Glossary of statistical terms

Confidence: The likelihood that the difference in conversion rates between a given variation and the control or baseline is not due to chance. Your statistical significance level reflects your confidence level.

Difference interval: Difference intervals tell you the range of values where the difference between the original and the variation actually lies. The difference interval is a confidence interval of the conversion rates that you can expect to see when implementing a given variation. Think of it as your "margin of error". For example if an open rate is 50% and the difference interval is +2%, then the difference interval is from 48% to 52%.

Confidence level: What percentage of confidence you can say the given outcome will occur within the difference interval. This is set at 90% for experiments in Iterable.
#The A/B test report
The A/B test report uses email events or conversions (purchases or custom events) to calculate the confidence intervals, open, click, purchase or conversion rates, and statistical significance.
An example of an A/B test report is shown here:
 Open Rate
 Percentage of unique email opens and the difference interval
 Click Rate
 Percentage of unique clicks opens as well as the difference interval
 Revenue
 Total revenue generated from the variation
 Purchases
 Number of purchases made from that variation
 Revenue / M
 Revenue generated from the variation per thousand emails sent
 Purchase Rate
 Number of conversion events received and the number of view events received as well as the difference interval
 Improvement
 Difference between the click rate for the control and for the variation
 Confidence
 Difference in conversion rates between a given variation and the control or baseline is not because of chance
The values used in the report are calculated as noted below.
#Conversion rate and conversion rate change for variations
For each variation the following is calculated:

Conversion rate:

The percentage change of the conversion rate between the test variation and the control variation:
Comments
0 comments
Please sign in to leave a comment.