Skip to main content
How Can We Help?
Statistical Confidence
Statistical confidence helps you understand how much trust you can place in the results of an experiment.
During an A/B test, differences between your Control and Variants can occur naturally as visitors interact with your website. Statistical confidence helps determine whether the performance difference you’re seeing is likely the result of your changes rather than normal variation.
Bluvia’s Default Confidence Level
Bluvia uses a 95% confidence level by default.
A 95% confidence level provides a strong threshold for evaluating whether the difference between your Control and a Variant is likely to represent a meaningful result rather than normal variation in visitor behavior.
For most experiments, we recommend keeping the default 95% confidence level.
However, Bluvia allows you to adjust the confidence level based on the needs of your experiment.
Adjusting the Confidence Level
You can increase or decrease the confidence level when configuring your experiment.
A higher confidence level requires stronger evidence before a result reaches your selected threshold. This can provide greater certainty but may require more traffic, conversions, and time.
A lower confidence level may allow an experiment to reach the selected threshold sooner, but it also increases the risk of making a decision based on normal variation rather than a true difference between your variations.
For most users, 95% provides a good balance between confidence and the amount of data required.
Why Statistical Confidence Matters
Imagine a Variant receives five visitors and two of them convert.
That Variant may have a much higher conversion rate than the Control, but five visitors aren’t enough to confidently conclude that the Variant is actually better.
As more visitors participate and conversions are recorded, you gain more information about how each variation performs.
Statistical confidence helps you avoid making optimization decisions too early based on results that may still change.
Understanding Confidence Levels
Confidence is expressed as a percentage.
The higher the confidence level, the stronger the evidence required before a result meets that threshold.
For example:
- 90% confidence requires less evidence than 95%, but carries greater uncertainty.
- 95% confidence is Bluvia’s default and is recommended for most experiments.
- 99% confidence requires stronger evidence and may require considerably more data.
Changing your confidence level doesn’t make an experiment inherently better or worse. It changes how much evidence you require before considering the results convincing.
Results Can Change During an Experiment
It’s normal for experiment results to fluctuate.
Early in a test, a few conversions can dramatically change the performance of a Control or Variant. As more visitors participate, those fluctuations generally become less significant.
You may see:
- A Variant perform strongly at first and later fall behind.
- The Control initially lead before a Variant begins outperforming it.
- Two variations remain very close throughout the experiment.
- Confidence increase or decrease as additional data is collected.
For this reason, avoid declaring a winner simply because one Variant takes an early lead or reaches a confidence threshold unusually quickly.
Statistical Confidence and Sample Size
Statistical confidence and sample size are closely related, but they aren’t the same thing.
Sample size tells you how many visitors have participated in your experiment.
Statistical confidence helps you evaluate the strength of the evidence produced by those visitors.
An experiment may show a promising difference between variations but still need additional visitors before you can confidently act on the results.
See Understanding Sample Size for more information about how experiment traffic affects your results.
Statistical Confidence Isn’t the Only Consideration
Reaching your selected confidence level is important, but it shouldn’t be the only factor you consider before making a decision.
Also consider:
- How many visitors participated.
- How many conversions occurred.
- The difference in conversion rate between variations.
- How long the experiment has been running.
- Whether unusual traffic or events affected the results.
- Whether the improvement is meaningful for your website.
- Whether the results support your original hypothesis.
A result can meet your confidence threshold while producing an improvement that’s too small to matter in practice.
Best Practices
When working with statistical confidence:
- Use Bluvia’s default 95% confidence level for most experiments.
- Only adjust the confidence level when you understand the tradeoff.
- Avoid making decisions based on early results.
- Consider confidence alongside sample size and conversion volume.
- Look for consistent performance rather than temporary spikes.
- Consider whether the measured improvement is meaningful in practice.
- Don’t assume every experiment needs to produce a winner.
Statistical confidence is one part of making a well-informed optimization decision.
Related Articles
Continue with:
