A/B Testing Dried Mangoes for Fun and Learning
Recently, a coworker brought back a bag of dried mangoes from the Philippines as a gift for me, telling me that they are the best dried mangoes in Southeast Asia (hereafter, SEA). This coworker happened to be from the Philippines and was for sure a bit biased in her opinion here…
Another coworker sitting next to me overheard the convo and piped in, saying that everyone wants to claim that mangoes from their country are the best. Apparently, this was an ongoing and playfully heated convo in the office.
I’ve been known to host data-driven social tasting events in Mountain View for the Google Analytics team, such as a Scotch tasting, Craft Beer and Guac-off, and Whiskey and baked goods. These events are always fun, get people talking and interacting, and because we’re the GA team, have a data element to them. Since I’m currently located in Singapore, I thought it would be fun to bring a small social event to the office here, so when I got wind of this great mango debate, I knew it was the perfect opportunity!
And thus, the Dried Mango A/B Test was scheduled! I picked up some dried mangoes from Indonesia while there the other week, and along with the mangoes from the Philippines and a generous supply of Thai mangoes available in our Micro-kitchen, we had enough to host a small A/B test tasting.
The setup:
– small bowls of each dried mango type, labeled for each country
– small bottles of water as palette cleansers
– a voting sheet and marker
We got a lot of feedback that this should have been a blind test, for example, labeling the mangoes 1, 2, 3, instead of labeling them by country. True. We should have set it up that way. Some on me for not thinking to control for that variable, as many potential testers may be biased by just knowing the country each mango is from.
Other variables we couldn’t control for:
– All mangoes are not created equal: one bag of dried mangoes is not necessarily representative of a whole countries supply
– Brands: we only tested one brand from each country
– Freshness: we had no idea how fresh or not each batch of dried mangoes was
For more info on controlling for variables and good test design, check out my post on Teaching Good Test Design.
The Results:
In total, 27 people participated in our A/B test. The numbers are not large, so we cannot draw true statistical significance, but we can infer a winner for fun.
1st place: Philippines (14)
2nd place: Thailand (10)
3rd place: Indonesia (3)
Qualitative feedback and observations: We asked people to give us feedback as to what they did and did not like about each mango, what they valued in a dried mango, and if each met their expectations.
– Color: each countries mangoes were quite different in color. The Indonesia mangoes were quite yellow, whereas the Philippines mangoes were pretty orange, and the Thai mangoes were somewhere in between.
– Thickness: Similar to color, they all differed. The Indo mangoes were cut very thick, the Philippines mangoes quite thin, and the Thai mangoes were again in between.
– Texture: was it dry or chewy? The overwhelming feedback was that the Indonesia mangoes were quite chewy, whereas the Philippines mangoes were drier.
– Sweetness: Both the Thailand mangoes and the Indonesia mangoes were detailed as sweet, with the Indo mangoes being the sweetest, more candy-like than fruit-like. The Philippines mangoes were detailed as more fruit-like, less dessert-like, and easier to eat more of because they lacked the overwhelming sweetness. I think this may be related to the thickness of the mangoes as well.
– A few people commented on the strong aroma of the various mangoes
So, was this a good test? Scientifically, only kind of. But fun-wise, definitely! The email chain circulating about the tasting provided some good entertainment, and we even offered up a prize to the most enthusiastic taster. Yoshi, from the SMB Adwords support team won due to his love of mangoes, his detailed analysis and description, and the fact that he brought his own palette cleanser to the tasting because he was quite serious about doing it right.
It also sparked several conversations with colleagues who were curious as to why we were doing this about A/B testing, creating enthusiasm for testing, and good test design (as I pointed out the potential flaws in our fun little test). It was a great way to spread interest in testing around the office to teams who don’t always have it top of mind 🙂
Catherine and me presenting Yoshi with his prize. When asked how he felt, he said “Mangotastic!”
Don
The other thing you should have controlled for is social bias. By revealing the results as you went along, it likely affected the votes of others, especially when two varieties were close calls or when they weren’t invested in the process.
One other point – you also wouldn’t want to label them 1, 2, and 3, because that also creates conscious or unconscious bias (because 1st place is better than 2nd, etc.). We oftentimes select 3 consonants from the middle of the alphabet M, K, and R.
Love your blog. I just stumbled onto it, and am trolling through it looking for gems. I’m a bit jealous of your vacations too!