Is your test result significant? Does it have enough power?

Play with the controls and get a better feel for how a lower confidence level will boost the power or how an increase in test size can make a small CR-difference significant!

Pre-test calculation or post-test evaluation?

Pre-test analysis

Test evaluation

Test data

Unique visitors expected per variation

Number of expected conversions Control

Expected uplift (%)

Visitors A

Conversions A

Visitors B

Conversions B

Settings

Hypothesis ^(?)

One-sided

Two-sided

Confidence ^(?)

90%

95%

99%

Test result

link copied to clipboard!

Variation B's observed conversion rate () was higher than variation A's conversion rate (). You can be 95% confident that this result is a consequence of the changes you made and not a result of random chance.

The observed difference in conversion rate () isn't big enough to declare a significant winner. There is no real difference in performance between A and B or you need to collect more data.

Possible SRM alert

Assuming you intented to have a 50% / 50% split, a Sample Ratio Mismatch (SRM) check indicates there might be a problem with your distribution.

The expected distributions of variation A and B.

Conversion Rate Control

Conversions A / Visitors A

Conversion Rate B

Conversions B / Visitors B

Relative uplift in Conversion Rate

CR_B - CR_A / CR_A

Observed Power

p value

Z-score

( CR_B - CR_A ) / SE_difference

Standard error A

( CR_A * (1-CR_A ) / Visitors_A)^1/2

Standard error B

( CR_B * (1-CR_B ) / Visitors_B)^1/2

Std. Error of difference

SE_difference = ( SE_A² + SE_B² )^1/2

More tools

A/B Test Size Calculator - How many visitors do you need?
Bayesian A/B-test Calculator - What is the probability that your test variation beats the original?
False Discovery Rate Calculator