Building a Culture of Optimization, Part 3: Know the Math
In part 3 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of bringing your organization up to speed on the math behind the tests. You can look back and see part 1 on the basics and part 2 on good test design.
Part 3: Know the Math! Give your peers a short stats lesson (but keep it light)!
What does someone running an A/B or MVT test need to know about math? How detailed should they be? Here is what I tell all of my coworkers:
Statistical Confidence = confidence in a repeated result
The confidence level, or statistical significance indicate how likely it is that a test experience’s success was not due to chance. A higher confidence indicates that:
– the experience is performing significantly different from the control
– the experience performance is not just due to noise
– If you ran this test again, it is likely you would see similar results
Confidence interval = a range within the true value that can be found at a given confidence level
Example: An experience’s conversion rate lift is 10%, it’s confidence level is 95% and it’s confidence interval is 5% to 15%. If you ran this test multiple times, 95% of the time, the conversion rate would fall between 5% to 15%
What impacts confidence interval?
– Sample size – as a sample grows the interval will shrink or narrow
– Standard deviation or consistency – similar performance over time reduces standard deviation
That’s it. That’s all people need to know.
In reality though, it’s not enough because it doesn’t go into details of how to calculate statistical confidence, but there are plenty of online tools to help with that. In my organization I give people a spreadsheet with a confidence calculator built in, all they have to do is plug in their numbers.
The real takeaway here, however, is that statistical confidence is important. I don’t actually care if my marketers know how to calculate it or even care to. All I really need them to know is that there is something mathematical that makes a difference for tests and that they should ask about it. In fact, after I gave a recent presentation on this topic, one of my coworkers said after that by tomorrow he’d forget what this stuff means, but he’d remember he should ask me about it. Mission accomplished!
Part 4 of this 5-part blog series on ‘Building a Culture of Optimization’ will focus on evangelizing your program, so stay tuned!