Skip to main content
How Can We Help?

Understanding Sample Size

Posted
Updated
ByEmily Gomez
Sample size refers to the number of visitors who participate in your experiment.
The more visitors your experiment receives, the more information you have to compare your Control and Variants. Having an appropriate sample size helps reduce the influence of short-term fluctuations and gives you a stronger foundation for evaluating your results.

Why Sample Size Matters

A small number of visitors can produce results that look more meaningful than they actually are.
For example, imagine a Control and Variant have each received 10 visitors:
  • Control ā€” 1 conversion
  • Variant ā€” 3 conversions
The Variant currently has a much higher conversion rate, but only a few additional visitors could dramatically change the results.
As more visitors participate, individual conversions have less influence on the overall conversion rate, giving you a clearer picture of how each variation is performing.

There Isn’t One Ideal Sample Size

There isn’t a single number of visitors that guarantees a reliable A/B test.
The amount of data you need depends on several factors, including:
  • Your existing conversion rate.
  • The size of the improvement you’re trying to detect.
  • The number of Variants in your experiment.
  • How much traffic the tested content receives.
  • Your selected statistical confidence level.
An experiment testing a high-traffic page may collect useful data quickly, while a test on a lower-traffic page may need to run considerably longer.

Sample Size and Statistical Confidence

Sample size and statistical confidence work together when evaluating your experiment.
Sample size tells you how many visitors have participated.
Statistical confidence helps you understand the strength of the evidence produced by those visitors.
Bluvia uses a 95% confidence level by default, although you can adjust this threshold based on the needs of your experiment.
Increasing your confidence level generally requires more evidence, which may mean allowing additional visitors and conversions to accumulate before reaching your desired threshold.
See Statistical Confidence for more information about choosing and interpreting confidence levels.

Traffic Allocation Affects Sample Size

Your Traffic Allocation settings determine how participating visitors are distributed between the Control and Variants.
For example, with an even 50/50 allocation:
  • 50% of eligible visitors see the Control.
  • 50% see Variant A.
If you add additional Variants, traffic is divided among more experiences.
This means experiments with several Variants may need more total traffic before each version has received enough visitors to make a useful comparison.

Conversions Matter Too

Visitor count alone doesn’t tell the entire story.
An experiment may receive thousands of visitors but very few conversions if the selected Conversion Goal occurs infrequently.
When evaluating whether your experiment has collected enough data, consider both:
  • The number of participating visitors.
  • The number of completed conversions.
This is particularly important when testing goals that occur less frequently, such as purchases or form submissions.

Avoid Stopping at a Round Number

Don’t assume an experiment is ready to end simply because it has reached a particular number of visitors.
For example, reaching 100, 1,000, or 10,000 visitors doesn’t automatically make the results reliable.
Instead, evaluate sample size alongside:
  • Conversion volume.
  • Conversion rates.
  • Statistical confidence.
  • The consistency of the results.
  • How long the experiment has been running.
These factors provide more context than visitor count alone.

Give Your Experiment Enough Time

Sample size isn’t only about reaching a certain number of visitors.
Visitor behavior can change throughout the week or during different traffic periods. Ending an experiment too quickly may mean your results only represent a small portion of your normal audience.
Whenever possible, allow your experiment to run long enough to capture normal variations in website traffic and visitor behavior.

What If My Website Has Low Traffic?

Lower-traffic websites can still benefit from experimentation, but collecting enough data may take longer.
If an experiment is progressing slowly, consider:
  • Testing higher-traffic pages.
  • Choosing Conversion Goals that occur more frequently.
  • Testing larger, more meaningful changes.
  • Limiting the number of Variants competing for traffic.
  • Allowing the experiment to run longer.
Avoid lowering your standards simply to produce a result faster. An inconclusive experiment is more useful than a confident decision based on insufficient data.

Best Practices

When considering sample size:
  • Don’t declare a winner based on a small number of visitors.
  • Consider conversions as well as total visitors.
  • Account for how traffic is divided between Variants.
  • Give lower-traffic experiments additional time.
  • Consider normal changes in visitor behavior over time.
  • Evaluate sample size alongside statistical confidence.
  • Avoid using an arbitrary visitor count as your only stopping point.
The goal isn’t to collect as much data as possible. It’s to collect enough meaningful data to make a well-informed optimization decision.

Related Articles

Continue with: