Building a Culture of Optimization, Part 1 – Educating the Basics

I’ve presented at several conferences, webinars, and meetings recently on the topic of ‘Building a Culture of Optimization.’ It’s a great topic and something our industry as a whole is still in the process of nailing down – most organizations I see haven’t yet reached maturity here. Given the importance of building such a culture in order to drive testing and optimization across our businesses I’m writing a mini-series of posts on some of the most important things I see, do, and recommend.

This will be a 5-part blog series on ‘Building a Culture of Optimization’ with parts 1-3 focusing on education, part 4 on process, and part 5 on spreading the word and shifting the culture.

Part 1: The Basics – Educate educate educate!

Educating your teammates is the first step in building a culture of optimization within your organization. Below are 5 basic principles & tips I generally cover with my teams early on when we talk about testing. Laying down these basics is important before you get into test design and execution.

1. What is Optimization?
Optimization is the ongoing, data driven process of continually discovering and quantifying the most effective experience, both online and offline, for your consumers and customers. Testing and testing platforms coupled with processes, enable organizations to adopt a culture of optimization.

2. What is a test?
There is a big difference between a true test and a time series test. Many people mistakingly think they are testing when they launch a new website and report on the impact by doing a before & after analysis. To be a true test, you must have a control and a variant within the same time period, while holding all other variables constant (note – this is one of the biggest mistakes I see, everything thinks they are running a test, when all they are really doing is a time-series analysis).

3. Testing vs Analysis
Which is appropriate depends on the question you are asking or the hypothesis you have.
Which layout works better? … test
What content do people look at more? … analysis

While they are different things, testing & analysis do go hand in hand. You should analyze your data set to find the opportunities to test, then test, then analyze the test results for insights and further test ideas. It’s a nice little circle.

4. Know your limits… Traffic
We’d all love to run 100s of tests at a time… but we only have so much traffic. The number of concurrent tests in market depends on traffic to each site.

Example: Most tests run in my organization for US English run on ~5% of website traffic per variation, whereas most regional tests (Brazil, Russia, India, etc) run on ~25% or more of traffic per variation because in these countries we get a lot less website traffic.

5. Given your traffic – is testing worth the effort?
Normally I’m the person in the organization who says we should test everything. Launching a new website? Test it. Want to change the main CTA? Test it. New headline on your homepage? Test it. BUT…

No doubt you will get requests to run a pretty vanilla test on a low traffic website. For example, changing the subtext or H2 of a campaign landing page that gets 500 visits per week. Is it worth it to test? Maybe. But before you go to all the effort of designing a test, developing the assets, executing & analyzing, be sure that this is something you absolutely need to test, because very likely, this test will need to run for a long time to reach statistically significant volumes, and even then may not reach confidence if the changes are too similar to the control.

It’s hard to say no to a test idea because the page doesn’t get enough traffic. It’s especially hard to say no when you’ve worked so hard to convince people to think about testing first. But knowing when it’s appropriate to test, and when you should just launch is an important step in the maturity cycle for your testing program.

So that’s the basics! Have you tried these out with your organization?

Part 2 of this 5-part blog series on ‘Building a Culture of Optimization’ will focus on building a good test design, so stay tuned!

And here’s a cute little meme I usually attach to quarterly test input emails:

Happy testing!

Next ArticleBuilding a Culture of Optimization, Part 2: Good Test Designs