I vividly remember my stats professor, back when I took my undergrad degree. He had a nearly unintelligible Croatian accent, barely spoke above a whisper, and liked to write on the overhead with squiggly scribbles as he spoke/whispered. In short, I could hardly understand him, hear him, or read what he was writing, but he instilled in me that statistics are among the most misused metrics out there.
I’ll explain why. It’s easy to say that “a majority of people believe in aliens.” Or that “the average weight of hamsters in Slovenia is 200 grams.” Or “90% of survey respondents agreed that a Provincial Task Force on Walking While Texting” is a good idea. What do these numbers really mean? What constitutes a “majority?” How many hamsters were weighed in Slovenia? Did the researchers take into accounts things like breed, age and sex of the hamsters? Of this “90%” did the researchers consider things like: how many people did you ask? What was the survey return rate? Are the responses liked to other influencing factors? It’s just to easy to use words like “average” or “majority” and not give the context of sample size. It makes a pretty big difference if you asked 10 people in a survey versus 1000. How would you know if those 10 people were representative of the general population?
We at Saturday Morning Productions collected some data over the past two months, regarding people’s perceptions of a Goal Achieving App. We wanted to gauge people’s interest in using a goal app, as well as ascertain what types of goals they might want to focus on.
When we looked at the raw data, we found that 85% of respondents were interested in a goal-achieving app. But when you look at the numbers behind that, it is actually 129 people out of 152 respondents. Are those 152 respondents a representative sample of the entire population? Do they equally represent age, sex, socio-economic factors, and other demographic categories that might affect the outcome? We have no way of really knowing, plus that level of detail would be something akin to a graduate thesis.
We found that 83% of respondents had set goals in the past – that’s 125/100 if you’re wondering. When people did set goals, saving money (59%), improving health (58%) and being physically active (61%) were singled out as being important. That was reassuring for us, because in our early interations of Goal Buddy, we had singled those out too. It was good to see that what we saw as important goals aligned with what most people saw as important!
The numbers dropped when we asked if people were successful in achieving those goals. Only 36% (44/124) were successful in achieving their chosen goal. When asked why they did not, motivation emerged as the most frequently implicated factor (39% of the time). We asked survey respondents what they needed to help them achieve a goal, and only 1% said an app would be the solution. Instead, people pointed out support (37%) and motivation (51%) would be more influential.
So if you look at that last set of numbers, it seems like survey respondents don’t think an app is a good idea. It might seem that way on the surface, but the whole point of an app is to provide motivation (such as the drive to keep beating your personal best when you start keeping track of what you are doing) and support (through an online community, for example). It’s not really about the app – it’s the philosophy behind it. That’s where numbers and stats don’t always give the whole picture.
As much as I enjoy analysing numbers, it’s important to keep them in a broader context. Even my stats prof hammered that point home, and he’d grown up behind the Iron Curtain where you don’t analyse numbers…numbers analyse YOU!