Skip to main content
How Can We Help?
Identifying Winning Variants
Once your experiment has collected enough data, you can begin evaluating whether one of your Variants is outperforming the original Control.
A winning Variant is the version that produces the strongest results for the Conversion Goal you selected when creating your experiment. However, the Variant with the highest conversion rate isn’t automatically a winner. You should also consider how much data has been collected and how confident you can be in the results.
Compare the Control and Variants
Start by reviewing the performance of your Control and each Variant in the experiment report.
Pay attention to:
- Number of visitors
- Number of conversions
- Conversion rate
- Difference in performance between variations
- Statistical confidence
Your Control provides the baseline for the experiment. Each Variant should be evaluated based on how its performance compares with that original experience.
Look Beyond the Highest Conversion Rate
It’s tempting to assume that whichever Variant has the highest conversion rate is the winner.
For example:
Variant A is currently performing better, but that doesn’t necessarily mean the experiment is ready to be concluded.
Results can fluctuate as additional visitors participate, particularly when an experiment has a small sample size.
Before choosing a winner, consider whether you’ve collected enough data to make the result meaningful.
Consider Statistical Confidence
Statistical confidence helps you understand how likely it is that the difference you’re seeing represents a real performance improvement rather than normal variation in visitor behavior.
Higher confidence gives you greater reason to believe that a Variant will continue to outperform the Control.
Avoid declaring a winner based solely on an early increase in conversions.
Consider Your Sample Size
The number of visitors participating in your experiment also matters.
An experiment with only a small number of visitors can produce large swings in conversion rates from just a few additional conversions.
As more visitors participate, the results generally provide a stronger foundation for making a decision.
Compare the Results to Your Hypothesis
Return to the hypothesis you documented when creating your experiment.
Ask:
- Did the Variant produce the improvement you expected?
- Did visitor behavior change in the way you predicted?
- Was the difference meaningful enough to justify making the change?
- Did the experiment reveal something unexpected?
The goal isn’t simply to find a winner. It’s to understand what the experiment taught you about your visitors.
What If the Control Wins?
Sometimes the original Control will outperform every Variant.
That’s still a useful experiment.
A winning Control indicates that the proposed changes didn’t improve the Conversion Goal you were measuring. Instead of implementing the Variant, you can keep the existing experience and use what you learned to develop your next hypothesis.
What If There Is No Clear Winner?
Not every experiment produces a clear winner.
Similar results between the Control and Variants may mean:
- The experiment needs more visitors.
- The difference between the variations is too small.
- The change didn’t significantly influence visitor behavior.
- The Conversion Goal wasn’t strongly affected by the change.
In these situations, you may choose to continue collecting data or use what you’ve learned to create a new experiment.
Before Declaring a Winner
Before ending your experiment, make sure you’ve considered:
- Whether enough visitors participated.
- Whether enough conversions occurred.
- Whether the performance difference has remained consistent.
- Whether statistical confidence supports the result.
- Whether the result supports or challenges your hypothesis.
Once you’re comfortable with the results, you can move forward with selecting the winning experience.
Related Articles
Continue with:
