The Danger of Data-Driven Decision-Making

[thehobbster]

The Danger of Data-Driven Decision-Making​

With the advance of computing power, there is a new game in town: Crowd-Sourcing. Why spend time, man-hours, paid labor, and server time beta testing your software when there are millions of people on the internet whose behaviors you can measure and extrapolate the data you need to tweak your software?

The caveat is this happens after launch.

Many companies have figured this out. For instance, Blizzard just released Diablo 3 to a ton of hungry gamers who have waited six years. They did a measley "beta" test, but the true beta test was the launch. They will put out a patch to take us into a legit release, and they did this to save money and time despite it being disrespectful to their buyers.

They used gamers as testers. Even though everyone was upset with the authentication servers being down, etc., what were they going to do? Go play Biablo 3? There is only one, and Blizzard leveraged this to obtain free beta testing.

Crowd Sourcing for Data: Blessing or Curse?
The kings of data-driven decision-making are Amazon and Google. I'd suspect there is no official version of Google. Traffic is being funneled to several versions that are live at once. They are constantly split-testing and collecting data. Every decision is not made by a boss somewhere. The decision is made by the numbers, which represent user behavior. Whatever version of the site converts better is the one that will be used, while more variations are rolled on it as well.

Can you see the potential problem here? There are these old adages such as "Think outside of the box." These didn't come about for fun. They are nuggets of wisdom. If you have much of a mathematical background, especially experience with algebra and statistics, you will remember little terms such as local minima and maxima, and the global minimums and maximums.

What this means is there are waves along, perhaps, a graph representing profit. If you present a certain amount of options for your consumers to choose from, you are cornering yourself into a box. You may collect the data and optimize accordingly, but you could be optimizing into a local maximum! Take a look at the image above.

Start at the left and move right. You may think you are bank-rolling because you've found a nice peak in your revenue. When you begin to slide to the right towards the question mark, you freak out and push back left with all of your data-driven might. But just beyond the horizon there is the true profit potential at the global maximum. Do you want to make cents or dollars? Will you ever know which you are even making if you constantly trust the data?

The Danger of Data & the Gamble of Intuition
There are some people out there who have played the game long enough that they have developed some sense of intuition about things. They know exactly how to optimize on the first try. Or perhaps they don't optimize at all. Take the classic Henry Ford example for instance. Henry Ford, the founder of Ford Motor Company, said, and I paraphrase, "If I'd asked my customers what it was they wanted, they'd have said 'faster horses!' The customer doesn't know what they want. You tell them what to want."

If Henry Ford had listened to the consumers, he'd have been in the business of optimizing riding crops, hanging out in the local maximum. Instead, he found the global maximum of producing cars, and he didn't bother with optimizing. You had one choice of car, because that's the one he manufactured. Case closed. And he balled out of control.

Another amazing case of marketing prowess was Steve Jobs. When asked how much testing he did on the iPad, he answered, "None." Zilch. Nada. Nathan. You'll take the white iPad and love it. These guys were amazing, but there are few marketing geniuses out there like this with a sixth sense about things. The king of all of them was Sigmund Freud, but that's a whole different story for another time.

The Fakers of Intuition - The HiPPO's
I recently read an article in Wired magazine about this very topic that introduced me to a term I hadn't heard before, which is the HiPPO. The HiPPo usually makes the decisions in companies that don't know how to read the data and make these data-driven decisions. They are the end-all-be-all authorities to making decisions, and that is the "Highest Paid Person's Opinion." And guess how that usually goes? Spiraling downward faster than the second season of The Walking Dead (although the last episode gave me hope!).

So I suppose the title of this article could be misleading, because it's just as dangerous to pretend to be a HiPPO and let your ego take the lead as it is to blindly follow the data without asking the philosophical questions of "why" the consumer is behaving in a certain way.

But let's get real. The smartest thing any of us could do is leverage our sensibilities of what works and doesn't, because we are all bombarded by advertising every day and know what's what, and to also do some data-driven decision-making. So how is this done?

Multivariate Testing A.K.A. The A/B Test
Mutlivariate is a big word that scares me. So to make it less intimidating, I break it down into its constituents which consist of "multiple" and "variations". Oh snap! Essentially you need to create multiple versions of your site or page and send a percentage of traffic to each one to see which performs better for your task. What makes someone click your advertisement? What makes them punch in their credit card information?

Just about anything you could think to tweak will affect your conversion rate. The number of columns on your page, the font size, the content of the images, the number of fields on your opt-in form, the call to action language, the color of your "buy now" button... all of it matters. Do people "buy now because it's a limited time offer" or because its a "limited time offer so buy now"? Collect the data and find out! Optimize, optimize, optimize!

But use your brain. If you aren't trying wildly different designs, welcome to the local maximum where you aren't reaching your true profit potential. Use your intuition, but not your ego, and then let the data guide you in for the smooth landing. Once you achieve your best, make a wildly different version of your lander and see if it has a chance to be optimized into being your new "local maxima."

Then do it again.

Originally Posted on March 17th, 2012 on