This is a common question from The HubSpot Inbound Marketing Certification Exam.
Question:
How long should you let your conversion rate optimization experiments run (on average)?
- A) 1 week
- B) 4 weeks
- C) 12 weeks
- D) 48 weeks
Answer:
- B) 4 weeks
Factors that Affect Conversion Tests
In the real-world, there is no average duration for A/B conversion rate tests to get accurate data. In fact, it can be misleading to say it will take 2 weeks to run conversion rate experiments, when it may take a few days for a high-traffic page or even months for a site with low traffic.
Some of the factors that can affect the average time for tests include:
- The average number of daily visitors
- The percentage of visitors to include in the test
- The number of variations in the test
How A/B Testing Works
In A/B Testing, you have two versions of a webpage: Version A and Version B. The goal is for these webpages to have an increase in conversions from your visitors.
If you’re familiar with the basics of A/B testing, you might wonder how long the actual conversion rate experiment should last.
The answer is as long as your visitor data will last:
If you don't collect enough data to accurately compare one version to another, the results will be useless. You need a healthy sample size to get any kind of significance level. For that reason, it's important to run a conversion optimization experiment for a minimum of 7-10 days (about 1 and a half weeks). Beyond that, the results are too flaky.
One of the easiest ways is to use an A/B Test Duration Calculator which uses a formula to calculate the results for you.
The “Total number of days to run the test” is only an estimate.
Remember you need to get statistically significant data. For instance, you will have a more accurate estimate if say you get 250 conversions per week for 2 weeks for each variation. But this will still only be a probability. Even with 100% statistical significance you can be pretty confident, but still can’t be 100% sure to get any conversion boost.
You also need to consider that even a 5% conversion rate increase needs much more traffic and time to get to statistical significance than a 90% increase would. Slight differences may well be just a chance thing, but larger ones are the real deal.