Testing with a Rapid Optimization Plan

If you’ve ever set out to A/B test a whole site redesign, you must have come across the question of ‘What do we do if the new site, that we’ve spent so much time and money on, doesn’t win?’ That’s a fair question. A very fair one. In fact, if you are not asking yourself that question before starting down the road of testing a site redesign, you should reevaluate your testing plan, because it’s a very real possibility that the new site will not, in fact, perform better than the old one. That could happen for many reasons: users are used to your old site, and seeing a new one may be a jarring or disorienting experience to them you may have optimized the heck out of the old
Continue reading...

Data Driven Design

Designing a new website is a big task. You must take many things into consideration: ease of use & functionality, mobile responsiveness, content, flow, graphics, etc. On top of that, you need to ensure that all of the analytics tracking is properly setup and collecting the necessary data for you to report on success. With so many considerations, it’s important to look at what your users are already telling you about it’s ease of use and helpfulness before you begin to make decisions about how to redesign and change it. Key metrics to consider when thinking about a website redesign: – number of unique users & sessions in a given time period – top content by pageviews/events/goal conversions/etc – funnel success (newsletter signups, contact form submits, checkouts, etc) – device
Continue reading...

Test Design: The Official Doc

If you are running a testing program, then you’ve more than likely had to think about what should go into a test you are running. This could include the problem statement, your hypothesis, how many variants, how different these variants will be, what your measures of success will be, screen captures, and more. It’s important to create a doc or some sort of accessible page/application for your teams to be able to reference this information. This helps to foster and open and collaborative culture of optimization. It will also help you as you look back to understand what your test objectives were and how the test did compared to those objectives. I track all of this via a Google Doc for each test I run. I used the same template for
Continue reading...

ga logo 2

I’m Joining the Google Analytics Team to Advocate for Digital Analytics Best Practices

After working as a practitioner of analytics and optimization for the past six years (at Adobe, the Apollo Group, and most recently Google), I’m excited to announce that I’m ‘officially’ making the move to the vendor side of the house! I’m joining the Google Analytics team as a ‘Best Practice Advocate’ for analytics and optimization. Five years ago, I would have said this role was my dream job. Two years ago, when I joined Google, I still would have said that this role was my dream job. And today, now 5 days in, I’m happy to say I’ve actually landed my dream job! 🙂 I say ‘officially’ with quotes for two main reasons: 1. I’ve already been at Google for two years, many might think that I’ve been on the
Continue reading...

Your Optimization Program’s First Hire

Just starting out in web optimization? I recently spoke on a panel at Optimizely’s Opticon and one of the questions that came up was ‘who would be your first hire’ for a new optimization program. There were a few different opinions on the panel, ranging from someone who gets stuff done, to an analytics rockstar, to that rare unicorn who can do it all. While all of these are good places to start, I tend to take the viewpoint of optimization through a solid analytics background as the best place to start. (Of course there are many optimization all-stars who didn’t come from an analytics background.) Why? Here are a few of the reasons why I’d look at hiring an analytics rockstar as your optimization lead: 1. They know data
Continue reading...

Building a Culture of Optimization, Part 5: So You’ve Found a Big Win… Now What?

In part 5 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of sharing your wins and bringing your organization along with you. You can see part 1, part 2, part 3 & part 4 here. Part 5: So you’ve found a big win. Now what? Ensure you’ve double triple checked your results! Are they statistically confident? Did you control for external variables? Why is this important? A personal example… I ran a test where we found significant uplift over our control from a couple of test variations, but one version stood out as the clear winner. After closing the test, reviewing and analyzing the data, I communicated the results and recommendation to launch the winner to the rest of my organization. Most people
Continue reading...

Building a Culture of Optimization, Part 4: Evangelize the Process

In part 4 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about evangelizing your process within your organization. You can see part 1, part 2, & part 3 here. Part 4: Evangelize the process Process is important. Process leads to consistency, repeatability, and authority in a testing program. Sharing that process and getting others in your organization bought in and supportive is even more important. One source of truth One of the best ways to make your optimization program better known within your organization is to evangelize it via a widely accessible & visible roadmap. Here’s an example roadmap that I use within my organization: I host this roadmap in a Google doc that is accessible to everyone in my organization, from analysts
Continue reading...

Building a Culture of Optimization, Part 3: Know the Math

In part 3 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of bringing your organization up to speed on the math behind the tests. You can look back and see part 1 on the basics and part 2 on good test design. Part 3: Know the Math! Give your peers a short stats lesson (but keep it light)! What does someone running an A/B or MVT test need to know about math? How detailed should they be? Here is what I tell all of my coworkers: Statistical Confidence = confidence in a repeated result The confidence level, or statistical significance indicate how likely it is that a test experience’s success was not due to chance. A higher confidence indicates that: – the
Continue reading...

Building a Culture of Optimization, Part 2: Good Test Designs

In part 2 of this 5-part blog series about ‘Building a Culture of Optimization’ I’m going to talk about the importance of teaching good test designs. You can see part 1 about educating the basics here. Part 2: Good Test Designs You can’t have a good test without a good test design. One of the first things I do when a new test idea surfaces is sit down with the key stakeholders & test proposers to understand the details of what they’d like to test. We’ll talk through the variables that are going to be tested, how best to setup & design the test, and ensure we are on the same page in terms of potential test outcomes and how to ensure we are testing in a clean and consistent manner.
Continue reading...