Evaluate conversion rate experiment outcomes with two-tailed Z-score calculations, p-values, and 95% confidence intervals, or compute pre-test visitor sample size targets to prevent underpowered experiments. Executed 100% in local browser memory.
Calculations reflect two-tailed Gaussian distribution probability.
| Decision / Reality | Variation is Truly Equal | Variation is Truly Superior |
|---|---|---|
| Declare Winner | Type I Error (False Positive, α) Deploying an ineffective change based on random noise. |
True Positive (Statistical Power, 1 - β) Accurately identifying and scaling genuine growth. |
| Conclude Inconclusive | Correct Retention of Control Avoiding wasteful engineering rollouts. |
Type II Error (False Negative, β) Discarding a winning variant due to an underpowered sample. |
Never terminate an experiment early simply because p < 0.05 on day 2. Reaching pre-computed sample size targets is mandatory to counteract "peeking" alpha inflation.
Run tests for a minimum of 14 continuous days (two full business cycles) to capture weekend browsing trends and pay-period purchasing variations.
If traffic was configured for a 50/50 split, verify that actual visitor counts do not deviate significantly via Chi-Square testing, which indicates redirect or caching bugs.
Learn how modern growth teams separate genuine conversion uplifts from random sampling variance using inferential mathematics, statistical power, and rigorous experiment stopping rules.
Evaluates pooled conversion variance under normal Gaussian distribution approximations, ensuring two-sided risk coverage for both positive and negative treatments.
Computes the exact visitor volume required per branch before launch, preventing underpowered tests and eliminating false discovery rates.
Enforces strict sample horizon stopping rules to counter alpha error inflation caused by repeatedly evaluating p-values prior to experiment completion.
In digital marketing, enterprise e-commerce, and high-ticket lead generation, conversion rate optimization (CRO) is frequently misunderstood as a series of creative design guesses. Marketing teams debate button colors, hero imagery, and headline phrasings based on subjective aesthetics. However, professional digital architecture treats every page layout as an empirical business hypothesis that must be rigorously tested using inferential statistics.
Without statistical rigor, teams routinely succumb to the "law of small numbers"—mistaking random variance, seasonality surges, and natural traffic fluctuations for true conversion lift. This client-side A/B Testing Statistical Significance and Sample Size Calculator provides performance marketing directors, frontend engineers, and product managers with the exact mathematical infrastructure needed to make definitive, data-backed deployment decisions.
At the heart of frequentist hypothesis testing lies the concept of the Null Hypothesis ($H_0$). In an A/B test, the null hypothesis posits that there is no true underlying difference in conversion propensity between the Control ($A$) and the Variation ($B$). Any observed difference in conversion rates is assumed to be the result of random sampling noise.
The alternative hypothesis ($H_1$) posits that the two versions possess fundamentally different conversion probabilities, and that the observed divergence reflects a genuine behavioral change triggered by the treatment.
Because conversion rate data consists of independent Bernoulli trials (where each visitor either converts or does not), the distribution of sample proportions follows a binomial distribution. By the Central Limit Theorem, when sample sizes exceed several hundred visitors, this binomial distribution approximates a standard normal Gaussian distribution. We evaluate the difference using the two-tailed pooled Z-test:
Where:
The p-value represents the probability of observing an uplift as large as (or larger than) the measured result, assuming that the null hypothesis is strictly true. A p-value of 0.03 does not mean that the variation has a 3% chance of failing; it means that if there were zero real difference between the designs, you would observe a result this extreme only 3 times out of 100 purely due to random chance.
The Statistical Confidence Level is defined as $1 - \text{p-value}$. When your confidence level surpasses the pre-set threshold (commonly 95%, corresponding to $\alpha = 0.05$ and $Z \ge 1.96$), you reject the null hypothesis and declare the variation a statistically significant winner.
The single most common error committed in conversion rate experimentation is early stopping (the peeking problem). In standard CRO platforms, software dashboards display live p-values updated in real time. A marketer launches an experiment on Tuesday morning; by Wednesday afternoon, after only 800 visitors, the platform flashes "96% Statistical Significance! Winner Found!"
Eager to capture gains, the marketer terminates the test and pushes the variation live. Four weeks later, company revenue remains entirely flat, or worse, declines. What happened?
Standard hypothesis testing mathematics assumes a fixed horizon: you determine your sample size in advance, collect the entire dataset, and evaluate the p-value exactly once at the conclusion of the test. When you continuously monitor a test day after day, you give random chance multiple opportunities to cross the 95% threshold.
Even when comparing an A/A test (where the Control is tested against an identical duplicate of itself), continuous monitoring over 30 days yields an actual false-positive rate exceeding 30% to 40%, despite setting the confidence threshold at 95%! To maintain statistical integrity, you must determine your sample size beforehand and refuse to declare a winner until the full quota of visitors has been reached.
Before launching any test, marketing leaders must determine whether their website possesses sufficient traffic velocity to achieve statistical power within a reasonable business window. Sizing an A/B test balances four critical parameters:
To calculate the minimum sample size required per variant for a balanced two-tailed test, our engine executes Evan Miller's pooled variance formula:
Notice the inverse-square relationship in the denominator: because the required sample size is inversely proportional to the square of the difference $(p_1 - p_2)^2$, halving your MDE requires quadrupling your sample size. If a test requires 10,000 visitors to detect a 20% uplift, detecting a 10% uplift will require approximately 40,000 visitors, and detecting a 5% uplift will require nearly 160,000 visitors.
A Sample Ratio Mismatch (SRM) occurs when the observed ratio of visitors assigned to each variation deviates significantly from the intended allocation. For example, if you configure your testing platform to deliver a 50/50 split across 10,000 total visitors, you expect approximately 5,000 visitors in Control and 5,000 in Variation.
If the actual result logs 5,420 visitors in Control and 4,580 in Variation, this divergence cannot be dismissed as random luck. A Chi-Square goodness-of-fit test will reveal a p-value far below 0.001, indicating severe structural bias. Common root causes of SRM include:
Whenever an SRM is detected, all conversion metrics and p-values are invalid. The experiment must be aborted immediately, the engineering routing pipeline debugged, and the test re-launched from scratch.
Modern experimentation tools generally fall into two philosophical camps:
The classical framework implemented in this tool. It assumes the true conversion rate is a fixed, unknown constant and treats data as random observations. It provides clear, actionable binary thresholds based on p-values and confidence intervals, making it the benchmark standard for academic peer review and regulatory compliance.
Bayesian testing incorporates prior knowledge (prior distributions) and calculates the probability that Variation B is superior to Control A given the observed data ($P(B > A | \text{Data})$). While intuitive for non-technical stakeholders (e.g., "There is a 94% probability that B is better"), Bayesian engines depend heavily on the subjective selection of prior models, which can inadvertently bias results in low-sample environments.
Consumer purchasing behavior varies drastically across days of the week. E-commerce sites frequently experience high browsing volume with low conversion intent on Monday and Tuesday, followed by high transaction velocity on Thursday evening and Friday. B2B websites experience heavy conversion volume during working hours (10 AM to 4 PM, Tuesday through Thursday) and near-zero conversions on weekends.
Therefore, an A/B test must always run in multiples of full 7-day weeks (minimum 14 days; ideally 21 to 28 days). Even if your website receives 500,000 visitors in 48 hours and reaches p < 0.01 by Wednesday, terminating the test mid-week ignores weekend consumer behavioral shifts and introduces systemic temporal bias.