A/B Testing on Black Friday, Posted by Chris Wallis, Conversion rate optimisation, Web Analytics in
Chris Wallis, Optimisation Executive at Mezzo Labs, explores the benefits and risks of testing during peak promotional periods.
Black Friday is fast approaching, bringing with it great opportunities for vendors great and small to test and optimise online. But it’s not all plain sailing. There are various precautions to bear in mind: does the risk of a drop in sales outweigh the potential for success?
What does this mean for CRO?
Let’s get down to it: What opportunities does this spike in traffic offer the conversion rate optimiser?
There are two schools of thought on A/B testing during peak periods… Some see it as an opportunity to squeeze out more sales. Others would avoid it – it’s too important a trading period to risk mistakes.
So what are the arguments for and against?
The Pros of testing on Black Friday
Higher visitors + increased conversion = even higher sales
Provided your baseline conversion rate doesn’t dip too much over the Black Friday period, more visitors means more sales.
Why? Let’s say you run a test and it gives you a healthy 5% boost for your conversion rate. A 5% increase of a larger number of visitors is going to give you more than a 5% increase of a smaller number, right? Any improvement will be a greater improvement.
Traffic spikes like this don’t last forever, so make the most of it, but only with tests you have great confidence in. Get it wrong and you’ll be missing sales. Hedge against it by only showing the variant to, say, 20% of visitors.
Accurate, confident results
It can often be difficult to reach a reliable conclusion for a test if the numbers behind it are small. It could take weeks or even months for conversion rate uplifts to stabilise to something you can trust.
Fortunately, the accuracy of results improves with more data in your sample. Maximise test data by running tests when there’s a lot of activity on-site.
Multivariate tests need even higher volumes. So Black Friday might be a chance for that MVT you’ve been wanting to run.
The Cons of testing on Black Friday
Risk of getting it wrong
If you test a variant that performs badly, the negative effects will hurt most over Black Friday. So carefully consider the possibility that the test variables won’t work. How radical is the change? And how could it go wrong?
Check how quickly your testing tool will report on performance, especially if your site receives a lot of traffic. Some tools have a lag of several hours before data appears, by which time it might be too late.
Read up on the statistical model that your tool uses. Some will find winners later than others to reduce the risk of declaring false positives.
Frankly I prefer this approach, as such models are going to provide more truthful final figures. The issue is that it could mean a longer test duration, and could potentially leave you with fewer reported successes. Will your promotional period last long enough to get you a significant result? Ask your tool vendor for their test duration calculator.
Server load slows your website
During events like Black Friday, testing platforms can be strained by the volume of server calls to their service. SAS tools like Optimizely and Adobe Target support hundreds, if not thousands, of tests on more than just your site. They will have a dizzyingly large number of test pages to render. This causes a huge strain on their infrastructures, potentially resulting in increased latency.
Top vendors have built-in timeouts to safe-guard you against extended response times. If a response is not received within a set timeframe (a matter of milliseconds), the site will just load the default experience. However, there is a risk that this doesn’t happen. It might be worth asking what measures your vendor has in place to ensure visitors aren’t impacted.
Recycle the results
I’ve seen a lot of tests where businesses frantically try to promote a seasonal sale in the best way possible. Image A? Or image B?
That’s all well and good if you reach a conclusion quickly, and can apply the winner for as long as possible during the promotion’s lifetime. However, if you don’t get a strong winner, the test might be a waste of time and resources.
Usually you can learn something from a test even if you don’t get a winner. But when it’s specific to a unique event, there’s nothing to carry forward.
Always make sure the learnings are transferrable in some way.
Watch your audience types
Events like Black Friday see businesses inundated with deal-hunters, many that have never visited your site before. They know what they want, but why should they buy it from you?
Make sure to get your offer and USPs are clear. This is a good opportunity to test how you should be promoting your brand values.
Test discount levels
This is a great opportunity to test price elasticity. How much should you cut your prices to get a sale? Cheaper stock may drive more sales, but will it erode net profits? The trade-off between transaction volumes and AOV is different for every brand and well worth testing to find the optimal balance.
Insights gained are perfect for carrying over to next year, when it all starts again.
In balance, I’d encourage running tests during the Black Friday peak. Provided you manage the risks, there are insights to be gained, sales to be made and new customers to be won.