Testing with a Rapid Optimization Plan

If you’ve ever set out to A/B test a whole site redesign, you must have come across the question of ‘What do we do if the new site, that we’ve spent so much time and money on, doesn’t win?’ That’s a fair question. A very fair one. In fact, if you are not asking yourself that question before starting down the road of testing a site redesign, you should reevaluate your testing plan, because it’s a very real possibility that the new site will not, in fact, perform better than the old one. That could happen for many reasons: users are used to your old site, and seeing a new one may be a jarring or disorienting experience to them you may have optimized the heck out of the old
Continue reading...

Data Driven Design

Designing a new website is a big task. You must take many things into consideration: ease of use & functionality, mobile responsiveness, content, flow, graphics, etc. On top of that, you need to ensure that all of the analytics tracking is properly setup and collecting the necessary data for you to report on success. With so many considerations, it’s important to look at what your users are already telling you about it’s ease of use and helpfulness before you begin to make decisions about how to redesign and change it. Key metrics to consider when thinking about a website redesign: – number of unique users & sessions in a given time period – top content by pageviews/events/goal conversions/etc – funnel success (newsletter signups, contact form submits, checkouts, etc) – device
Continue reading...

Step-by-Step: Setting up a Google Content Experiment on Your Website

Setting up a Google Analytics Content Experiment is easy! Follow this four-step process and you’ll be on your way to running your first test. To start, first go to the ‘Experiments’ section of Google Analytics and click on ‘Create Experiment’. Step 1: Setup the test Advanced: if you are working with a high volume page and want to analyze more than one goal at a time, you can set up a ‘fake goal’ so that the test will not optimize towards a single winner. Use a ‘fake goal’ to run the test longer than 2 weeks: Multi-armed bandit: Content Experiments uses a traffic splitting method called Multi-armed bandit (MAB) which essentially weights the traffic towards the variation(s) that appear to be winning, away from losing variations. In theory, this could
Continue reading...

Building a Culture of Optimization, Part 2: Good Test Designs

In part 2 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of teaching good test designs. You can see part 1 about educating the basics here. Part 2: Good Test Designs You can’t have a good test without a good test design. One of the first things I do when a new test idea surfaces is sit down with the key stakeholders & test proposers to understand the details of what they’d like to test. We’ll talk through the variables that are going to be tested, how best to setup & design the test, and ensure we are on the same page in terms of potential test outcomes and how to ensure we are testing in a clean and consistent manner.
Continue reading...

Building a Culture of Optimization, Part 1 – Educating the Basics

I’ve presented at several conferences, webinars, and meetings recently on the topic of ‘Building a Culture of Optimization.’ It’s a great topic and something our industry as a whole is still in the process of nailing down – most organizations I see haven’t yet reached maturity here. Given the importance of building such a culture in order to drive testing and optimization across our businesses I’m writing a mini-series of posts on some of the most important things I see, do, and recommend. This will be a 5-part blog series on ‘Building a Culture of Optimization’ with parts 1-3 focusing on education, part 4 on process, and part 5 on spreading the word and shifting the culture. Part 1: The Basics – Educate educate educate! Educating your teammates is the
Continue reading...