Skip to main content
How Can We Help?
Reading Test Reports
Once your experiment begins collecting data, Bluvia provides a test report to help you understand how your Control and Variants are performing.
Test reports bring together the key performance metrics for your experiment so you can compare results, understand visitor behavior, and determine whether the changes you’re testing are having the desired effect.
This guide explains how to read your test report and what to consider when evaluating your results.
This guide explains how to read your test report and what to consider when evaluating your results.
Before Reviewing Your Results
Experiment results become more useful as additional visitors participate in your test.
It’s normal for results to change, particularly during the early stages of an experiment. A variant that appears to perform well initially may change as more visitors participate.
It’s normal for results to change, particularly during the early stages of an experiment. A variant that appears to perform well initially may change as more visitors participate.
Before making a decision based on your report, consider:
- How long the experiment has been running.
- How many visitors have participated.
- How many conversions each variation has received.Whether there have been unusual changes in website traffic.
- Whether the experiment has had enough opportunity to produce meaningful results.
Avoid declaring a winner based only on early performance.
Understanding Your Test Report
Your report compares the performance of the Control against each Variant in your experiment.
The Control represents the original experience, while each Variant represents one of the changes you’re testing.
Your report allows you to see how visitors responded to each version based on the conversion goal you selected when creating the experiment.
The Control represents the original experience, while each Variant represents one of the changes you’re testing.
Your report allows you to see how visitors responded to each version based on the conversion goal you selected when creating the experiment.
Visitors
Visitors represent the people who participated in your experiment and were shown a particular variation.
Reviewing visitor counts helps you understand how much traffic each variation has received.
If you’re using an even Traffic Allocation, your Control and Variants should receive approximately equal portions of eligible traffic over time.
Reviewing visitor counts helps you understand how much traffic each variation has received.
If you’re using an even Traffic Allocation, your Control and Variants should receive approximately equal portions of eligible traffic over time.
Small differences in visitor counts are normal.
Conversions
Conversions show how many visitors completed the goal you selected for the experiment.
What counts as a conversion depends on your selected Conversion Goal.
What counts as a conversion depends on your selected Conversion Goal.
For example, a conversion could represent:
- Clicking a CTA.
- Reaching a target page.
- Submitting a form.
- Reaching a specified scroll percentage.
- Completing another supported conversion action.
Conversions should always be interpreted in the context of the goal you’re trying to improve.
Conversion Rate
Conversion Rate shows the percentage of participating visitors who completed your selected goal.
For example, if a Variant receives 100 visitors and 15 complete the conversion goal, that Variant has a 15% conversion rate.
Conversion Rate makes it easier to compare variations even when they haven’t received exactly the same number of visitors.
A higher conversion rate may indicate that a Variant is outperforming the Control, but you should avoid making decisions based on conversion rate alone.
For example, if a Variant receives 100 visitors and 15 complete the conversion goal, that Variant has a 15% conversion rate.
Conversion Rate makes it easier to compare variations even when they haven’t received exactly the same number of visitors.
A higher conversion rate may indicate that a Variant is outperforming the Control, but you should avoid making decisions based on conversion rate alone.
Comparing Your Control and Variants
The primary purpose of your report is to understand how each Variant performs compared with the original Control.
When reviewing the results, ask:
When reviewing the results, ask:
- Which variation has the highest conversion rate?
- How large is the difference between the Control and Variant?
- Has the difference remained consistent as more visitors participate?
- Has enough data been collected to make a confident decision?
- Does the result support your original hypothesis?
The goal isn’t simply to find the variation with the largest number. It’s to determine whether the results provide enough evidence to make an informed optimization decision.
Don’t Rush Your Results
One of the most common A/B testing mistakes is ending an experiment as soon as one Variant begins outperforming another.
Early results can fluctuate significantly.
Whenever possible, allow your experiment to collect enough traffic and conversions before deciding which variation performed best.
A test that appears to have a clear winner after its first day may look very different after additional visitors participate.
Early results can fluctuate significantly.
Whenever possible, allow your experiment to collect enough traffic and conversions before deciding which variation performed best.
A test that appears to have a clear winner after its first day may look very different after additional visitors participate.
What If the Control Performs Better?
Not every experiment will produce a winning Variantāand that’s okay.
If your Control performs better, the experiment has still provided valuable information. You’ve learned that the proposed change didn’t improve the behavior you were trying to influence.
That insight can help inform your next hypothesis and prevent you from making a change that could negatively affect performance.
If your Control performs better, the experiment has still provided valuable information. You’ve learned that the proposed change didn’t improve the behavior you were trying to influence.
That insight can help inform your next hypothesis and prevent you from making a change that could negatively affect performance.
What If There Isn’t a Clear Winner?
Sometimes an experiment won’t produce a meaningful difference between the Control and Variants.
This may indicate that:
This may indicate that:
- The change didn’t have a significant impact on visitor behavior.
- The experiment needs more traffic.
- The selected conversion goal wasn’t strongly influenced by the change.
- The difference between the variations was too small.
- A different hypothesis may be worth testing.
An experiment doesn’t need to produce a winner to be useful. Every test can provide information that helps guide your next optimization decision.
From Results to Optimization
Your test report isn’t the end of the experimentation process.
Use what you’ve learned to determine your next action. You might implement a successful Variant, keep the Control, refine your hypothesis, or create another experiment based on what you discovered.
Over time, this process creates a continuous optimization cycle:
Use what you’ve learned to determine your next action. You might implement a successful Variant, keep the Control, refine your hypothesis, or create another experiment based on what you discovered.
Over time, this process creates a continuous optimization cycle:
- Identify an opportunity.
- Create a hypothesis.
- Run an experiment.
- Review the results.
- Apply what you learned.
- Identify the next opportunity.
The value of experimentation comes from continually applying what you learn.
Best Practices
When reviewing test reports:
- Give experiments enough time to collect meaningful data.
- Compare conversion rates rather than conversion totals alone.
- Consider your original hypothesis when interpreting results.
- Avoid making decisions based on short-term fluctuations.
- Treat unsuccessful experiments as learning opportunities.
- Use what you learn to inform your next experiment.
Related Articles
Continue with:
- Identifying Winning Variants
- Conversion Goals
- A/B Testing Best Practices
- Test Library
