Some Ethical Limits of Statistics

My piece titled "Some Benefits and Limits of Statistics" has been in the top five of my all-time most viewed posts for a while now. I have no clue why this is so, but in any case, I am glad people like to learn! (Or there are some very zealous cyber bots searching for posts about statistics in the United States and Russia.)

Either way, an observation that my latest statistics professor, Nicholas Jewell, made on the first day of class this semester (yes, I am taking more statistics) really stuck with my heart, and I thought it a good time to share. In so many ways, I am preaching to myself in this post, as I have had many a moment of poor character in my thoughts and words over the years, many times as a result of my numbers bent, and it is something I am constantly praying for God to redeem in me. So here goes.

In statistics, we typically do not have information about every single person or thing we want to understand. Instead, we take samples from a study population and try to generalize what we find in the sample to the whole population. Think of this like asking one hundred random strangers in your town whether they like your hair do. If 80 of the 100 people you stop all over your town tell you it's a bad hair do, you might reasonably guess that about 80% of people in the whole town would think the same, regardless of whether you have actually asked every single one of them. If you care much about what others think, you go get a new hair cut, or if not, you pride yourself on being a hair rebel.

Intuitively, we generally prefer larger sample sizes, because, all other things being equal, they make our results more credible. For example, if you stopped 5 random strangers some place in your town, and 4 of them did not like your hair do, you might chalk it up to bad luck in your stranger selection. You might figure, hey, if I had asked five other people across the street or across town, they might have said something different. But if you ask 1,000 random strangers, and 800 say you look like Medusa, you would be less likely to assume it was an "unlucky 800." As it turns out, this intuitive understanding is also mathematically true in statistics, although I won't get into too many details here.

"The top portion of this graphic depicts probability densities that show the relative likelihood that the "true" percentage is in a particular area given a reported percentage of 50%. The bottom portion shows the 95% confidence intervals (horizontal line segments), the corresponding margins of error (on the left), and sample sizes (on the right). In other words, for each sample size, one is 95% confident that the 'true' percentage is in the region indicated by the corresponding segment. The larger the sample is, the smaller the margin of error is."
(Source: http://en.wikipedia.org/wiki/Margin_of_error)

As Professor Jewell reminded us during our first class, in public health in particular, we are often using epidemiological methods to compare people who experience a certain health outcome with people who do not. For example, we might study the characteristics of people who get pancreatic cancer in their lifetime versus people who do not get pancreatic cancer in their lifetime, in the hopes of figuring out what those who get cancer share in common and how they are different from those who do not get cancer. If we conduct a study of 1,000 people, where we compare 999 individuals who did not get the cancer with one person who did, we are not likely to feel confident that the one person who got cancer in our sample is statistically representative of the multitude of people who get pancreatic cancer around the world. (This is just like the hair-do example above, where we believe the results more when we ask more people.) Let's assume that the single cancer patient in our study is also the only one of the 1,000 people we study who has ever visited the North Pole - so our one cancer patient has visited the North Pole, but our 999 non-cancer patients have never visited the North Pole. Do we then have evidence that the North Pole is associated with pancreatic cancer? Most of us would say no, because we intuitively sense that we cannot draw a conclusion based solely on a single person's experience. On the other hand, if we compare 500 people who did not get cancer with 500 people who did and find that a history of North Pole vacations is much more common among the cancer patients, we are more likely to believe and perhaps further investigate an association between cancer and the North Pole.

In the above example, more cases of disease in our study population gives us more confidence in our conclusions. We are more likely to learn something and believe it could be true more generally by studying 500 cancer patients than by studying only one cancer patient. As statisticians, then, we may find ourselves in the position of wishing for more cases of disease in our sample size in order to have greater confidence in our results. And this is where I really appreciated the insight of Professor Jewell. In moments like those, when we are wrapped up in data and numbers, we should take a moment to step back and realize exactly what we have just thought. Should anyone really wish for more cases of a disease, just for the sake of analytic rigor? The obvious answer should be no. We may wish that we had crafted a better study design to interview more individuals who already have the disease, but it is very easy to forget that each of those data points is a real person.

Every single cancer patient enrolled in a study represents a person and family and community who had to go through an agonizing experience. Participating in a study in and of itself is often no walk in the park, either. Furthermore, the results of a study into what causes a given outcome may benefit future patients, but are often too late to benefit the ones under study at the time. This pain and trial are not to be so callously wished on anyone.

And it is in this sense that as students, teachers, researchers, etc., those who use statistics have an ethical responsibility to consider the human costs of what we quantify. Professor Jewell said he reminds himself of this by watching one particularly compelling and emotionally complex movie about human subjects research every year, which helps re-center him on the human aspects of his work. You may find your own unique ways to heighten your awareness. I believe this need applies not only to public health, but also to many fields that utilize mathematical inquiry beyond public health - education, public policy, social work, and business, just to name a few. It is not that our methods will necessarily be changed in any concrete way by exercising compassion in our thoughts (although they might, in some cases); rather, we can seek an attitude of empathy and a posture of humility as acts of internal integrity. It is not that we should neglect the study of disease or other negative life outcomes, as our numerical inquiry is typically not the cause of such outcomes, and is often ultimately aimed at improving health. But even as we work for these good causes, we should take time to reflect, empathize, and get outside our little boxes to just be human - to appreciate the lived experiences of those studied or affected, and to stop ourselves from so recklessly throwing around words and cold thoughts about those who it is all too easy to see as data point number 732.

Funny Statistics

One of the things on the internet that often sends me into a minor rage (because, you know, there are just so few offensive things out there that I had to find something, anything to be annoyed at) is misuse of statistics. It is so easy for a manipulative writer to "deceive the masses" with numbers (myself often included), and then there are those who unintentionally misuse statistics in their articles and go about their merry days guilt free. That is among the reasons I wrote a post a while back about the benefits and limits of statistics. The point was really to encourage readers to think critically. Interestingly enough, that is my most popular post ever, so maybe I am onto something!

I seem to encounter statistically infuriating writing in the news or other websites on a near weekly basis, often written by "experts." Yikes! But this week, I found something statistically hilarious on buzzfeed.com: "The 10 Most Bizarre Correlations." Ok, I know I have a dorky sense of humor. But come on people, these are funny!

This is one of my favorites, because it realistically portrays what I actually see in articles on a regular basis:

Source: http://boingboing.net/2013/01/01/correlation-between-autism-dia.html

For those still scratching their heads about why this is absurd, notice that the two graphs are presented in total counts (total sales and total diagnoses), not as proportions of the total population represented by the graph. My best guess is that the population growth curve would match up very closely to these two metrics or that the population growth curve plus other factors would do so, but without that information specifically provided, we cannot know. It really doesn't mean that much statistically to say that there were four times as many disease diagnoses counted in 2008 compared to 1998 if there were also four times as many people alive in general (or if the methods for diagnoses became better, such that the disease was simply identified with greater ease). If that were the case, the rate of disease may have stayed constant. The same is true with the purchase or organic produce - it could be an increase based solely on having a larger population (or another possibility is that the number of organic produce purchases remained constant, with the price of the organic produce quadrupling, thus increasing total dollar sales). With this graph, we really have no way of knowing what this data really means, as it has no reference to how many people were in the population each year, nor do we have information on many other associated metrics. Without asking critical thinking questions, some might look at this and automatically conclude that there is a link between organic food and autism. Gotta love the misuse of science! I'll stick with my pesticide fruit, thank you very much!

This also reminds me of a funny xkcd comic:

Source: http://xkcd.com/1138/

As the comic author writes in the sub-text (when you hold your mouse cursor for a couple of seconds over the comic on their website), "There are also a lot of global versions of this map showing traffic to English-language websites which are indistinguishable from maps of the location of internet users who are native English speakers." Yes!

Another one of my favorites from the buzzfeed article is a 1960's study demonstrating that countries with slower growing economies have men with longer "male organs."

Source: http://www.nerve.com/news/love-sex/new-study-finds-correlation-between-penis-size-and-national-gdp-0

Or, as a slightly different, but more recent study showed:

Source: http://www.nerve.com/news/love-sex/new-study-finds-correlation-between-penis-size-and-national-gdp-0

I knew there was a silver lining to the recession! Ok, but seriously now - can you think of other factors that might create this effect, beyond the economic growth of a country? This is a test of your analytic prowess!

To see the rest of the odd examples of correlation, check out this awesome article: http://www.buzzfeed.com/kjh2110/the-10-most-bizarre-correlations.

Some Benefits and Limits of Statistics

One of my readers commented on my post about sex work and childhood abuse that I was essentially dehumanizing the issue by using statistics. While I think the reader may have made some incorrect assumptions about my motives, I think he or she brought up a valid point on some level, which is that statistics have their limits. So I thought I would write a brief post about the benefits and limits of biostatistical information. This is by no means intended to be an exhaustive list of all the issues that arise around statistics, but perhaps it can get your mental juices flowing to think more deeply about the articles you read.

Firstly, I would like to give a little pitch for why statistics are important. Have you ever heard someone say something like this (or have you said it yourself)?

• My friend got in a high speed car accident, and the only injury she sustained was a bruise from the seat belt; if she had not been wearing it, she would have no seat belt bruise, and that proves that you're safer without wearing seat belts!
• I have gone to McDonalds every week for fifty years and am in great health, so those fast food nay-sayers don't know what they're talking about.
• My cousin did not go to college, and she is making twice as much as I do, despite my degree. College does not help you get a higher paying job. I would not recommend that others go.
• None of my sexually active friends got infections back when we were teenagers, so it's not something I really worry about for my own teenage daughter.
• My aunt smoked for 75 years and lived to be 90 years old, so that just goes to show that smoking isn't as bad as they say.

These statements represent a very common logical fallacy in our culture, which is the assumption that a personal experience can be used to make statements about the entire population. How do you know that your personal experience is the norm and not the exception? Even if you believe it is the norm among your group of friends or acquaintances, how do you know your group of friends is the norm and not the exception compared to other groups? The answer is, without information about the experiences of others, you don't actually know.

This is one of the reasons statistics are useful. They show what the norm of experience is. That doesn't mean they show what happens with every single person in every single case, but they can show what is most likely to happen in a given population. So to counter one of the sample statements above, even though you think college was useless because you have a lower paying job than your less educated cousin, statistics can show that you and your cousin are the anomalies. Based on information I shared in a previous blog post, the median income for someone with a bachelors degree is 65% more than the median income for someone with only a high school diploma. That median is not the amount every person makes; it just implies that you probably have a better shot of making more with the degree. In all likelihood, you are one of the 50% of people with a degree who earns below the median income for your education level, and your cousin is above the median for his or her education level. That doesn't negate the usefulness of the median information for the population on average. Similarly, if a report shows that 75% of prostitutes were abused as children, as I wrote about in another earlier blog post, and you personally are a sex worker who was not abused, then that means you fall into the minority 25%, as do one in four of your colleagues; that is a large enough minority that we would expect there to be significant numbers who were not abused, and the fact that you are someone or know someone outside of the majority does not prove that the 75% figure is wrong.

Because statistics represent what happens over an entire population, they are especially useful for making health policy decisions. It is natural for us to want to take actions that feel right to us, but we must also consider if they are based on the reality of a population. For example, one politician might feel that smoking is a good thing, because it curbs his appetite and relaxes him after a long day. None of his friends have experienced major smoking-related health problems, so he believes very strongly that it is good for his health. Based on his feelings, he may want to ensure that everyone has greater access to cigarettes at as young an age as possible. His intuition tells him that this will benefit the health of the population. Yet because we have statistical information showing that smoking is harmful and addictive, on average, we pay attention to the statistics rather than one person's experience and feelings, and we have laws against selling cigarettes to minors. This seems like a common sense approach, but we often forget to ask if policy decisions have a basis in reality or in statistically proven effectiveness. Of course, we cannot always rely on statistics. Sometimes, we want to try something new, in which case we cannot know how it will pan out. And often times, the particulars of one population may be so different than those of another population that we do not trust the results that happened elsewhere to apply to the new population. But when it comes to many health decisions and initiatives that have been tested in a variety of groups, we can use statistics above our feelings to figure out what will be most effective.

So we have established that statistics help ground us in the realities of a population as a whole, beyond our own experiences and feelings. But as I stated in the introduction, statistics do have their limits. And it is often left up to us as consumers of the media to filter through the plethora of information and to figure out how reliable/useful a given statistic is. So how do you comb through the constant bombardment of data?

One thing to consider is that not all statistics are created equal. The structure of a study, such as the number of people studied, the way the questions were worded, the way lab tests were taken, or the types of people sampled, can all lead the data to represent underlying realities differently. When researchers craft questions, for example, they should try to maintain some neutrality that will not influence the answer. Imagine a study to determine patient opinions of their doctor. One question might read, "A lot of people in this town hate Doctor John Smith. On a scale of one to ten, how much do you hate Dr. Smith?" This question automatically introduces bias by telling the interviewee that others have a negative opinion and may affect the thoughts of the participant or their comfort with expressing a certain opinion. A less biased question might say, "On a scale of one to ten, how do you feel as a whole about Doctor John Smith, with ten being highly satisfied and one being highly dissatisfied?" Some people might even read that question and see issues, since highly satisfied is stated before highly dissatisfied, and the question is very vague about what aspects of the doctor the interviewee likes or does not like. Is this a question about accurate diagnosis, turnaround time, personality, appearance? As you can see, collecting statistics in an interview or survey format takes a lot of forethought. Even biostatistics collected from lab results, which do not involve direct questioning, can have elements of bias if not all labs were analyzed using the same standard procedure, if the standard procedure is more likely to erroneously produce a certain result, or if the participants were chosen in a biased fashion, etc.  (For an example of how sample selection and response choice can skew a statistic, read about the 1936 presidential campaign polling gaff.) Sometimes I even read articles that cause me to question whether the outcome being tested answers the question it is intending to quantify.

Let's look at another way statistics can misrepresent a situation. Imagine that you read a study stating that 75% of 13-year-old students at St. Catherine's school menstruate, but only 35% of 13-year-old students at Sedgwick menstruate. At first glance, you would say, wow, this is a major discrepancy! Something strange must be going on to cause this big gap between the two schools! But what if I told you St. Catherine's was an all-girls school and Sedgwick was a mixed gender school. You would have to ask, does the 35% statistic at Sedgwick include the boys (who of course would not menstruate at any age)? If so, that changes everything! This is a somewhat silly and made-up example, as this sort of glaring reporting difference is not common in my experience. But it illustrates the point well - we need to consider what is really being represented by a statistic, not just what the face-value number is.

Another common issue you may face in understanding biostatistics is confusing correlation with causation. Just because two things are statistically associated does not mean that one causes the other. For example, in the above scenario, you could say that a student is more likely to menstruate if he or she attends St. Catherine's than if he or she attends Sedgwick. That is a correlation which is proven by the statistics 75% and 35%. A causation statement would say that attending St. Catherine's causes menstruation to begin by age 13. Even though we know menstruation and school are correlated statistically, without more information on how many girls are at Sedgwick (compared to how many boys), it would be silly to conclude that the school attended is what actually causes menstruation. The statistic alerts us to something else underlying the issue, indicating that we need to dig deeper to find the cause - and it turns out the cause for the correlation may simply be the fact that there are only girls at St. Catherine's and both sexes at Sedgwick. If we adjusted the data to account for this difference, we might find that the rates among girls alone were identical at both schools. There is really no way to know just by looking at the values presented.

Let's take another example of causation versus correlation, this time from real life statistics. Imagine you are deciding between moving to California or to Oklahoma and are concerned about the health impact of where you live. You read on the CDC website that California has a lung cancer rate under 59.5 per 100,000 people, but Oklahoma is over 73.8. Based on this information, you see a correlation between living in Oklahoma and getting lung cancer. At face value, this indicates that you may want to move to California to decrease your risk. However, you must ask yourself, is it the act of living in Oklahoma that causes lung cancer, or is there something else going on under the surface of these statistics?

As it turns out, when you dig deeper, you find that the Huffington Post reported on a Gallup-Healthways pole showing 15.7% of individuals smoke in California and 26.6% of individuals smoke in Oklahoma. So the cause of the lung cancer may actually be cigarettes, at least in part, as opposed to the physical placement of one's home in Oklahoma. Other factors could also contribute, such as a higher percentage of people working in industries exposed to chemicals, socioeconomic concerns, access to health care, a population that eats certain foods, or a population that exercises less, etc. These things seem more likely to cause the different cancer rates, even though the cancer rate is correlated to the placement of your home. (Note: almost any factor may  be correlation and not causation - specific studies are needed to show that one thing actually causes another.)  It is true that even taking into account these lifestyle differences, it is still possible that there are environmental factors contributing to the health in each state, such as air pollution or water contaminants, or certain laws that affect population health. These things could affect the cancer rates and may legitimately lead you to prefer living in one state over the other for health reasons. But it is also possible that if you personally do not smoke or work with chemicals, and if you personally take care of your overall health, it may be that your chances of getting lung cancer are more or less the same whether you live in California or Oklahoma. The high-level information we have above is not enough to say for sure. You would need to dig deeper than the surface level cancer rates to determine your actual risk in each location.

Although there is plenty of reason to question the validity of the statistics you read, there is also a lot of misinformation floating around the internet about what makes a good study. For example, I have often seen people commenting on articles, "Well, this is ridiculous. They studied 3,000 people, but there are 300 million people in the U.S., so this is junk science and is not at all representative of everyone." However, if you take an introductory statistics class, you will learn that a sample size of 3,000, if properly selected to represent the population well, is actually a pretty large sample. Statistics is largely based around the ability to apply results from a sample to explain an entire population, and there is a lot of fancy math that goes into determining how accurately a sample represents a population. Now, if all 3,000 of the people sampled lived in one town or were all of the same ethnicity or all shared the same diet, then you would have to take other factors into consideration before simply assuming the results apply to everyone in the country. (Then again, if everyone was different with regards to those attributes, you would have to consider whether those differences affected the outcome being measured.)  But many studies consider a wide variety of factors and adjustments when crafting a sample or analyzing the results. The type of study being conducted and the factors that may influence the outcomes can change the number and type of subjects needed to produce a reliable result. The lesson here is not to throw out all statistics that come from a sample. Rather, what is important is to think critically about what a statistic is saying, to question how it was constructed, and to put these factors into consideration when you read about a study.

Yet another pitfall with statistics is the one my reader brought up in his or her comments, which is the tendency to lose site of the individuals represented. For example, we may talk about a program that will help 95% of people in a particular quandary and feel that the job is done once the program is implemented; yet the statistic may mask the dreadful plight of the 5% who are not helped. Without a connection to the actual people affected, it is hard to understand the human impact. Or when we talk about the 75% of prostitutes being abused as children, it is easy to link any conclusions about that sub-group with the entire group, when certain conclusions may or may not apply to every member of that 75% or not at all to the other 25%.

Some people believe that you should not use statistics to speak about an issue at all if you have not personally interacted with the population or been affected by the issue. They might call that dehumanizing. I tend to disagree and would simply say that you ought to be careful. As I wrote in my response to the comments on my earlier blog post, I do not believe it is inherently wrong to comment on statistical findings when I am not the individual who conducted the research myself or when I am not friendly with individuals affected. If an issue is important, I believe that many people who care should undertake the task of bringing others' research into the public consciousness, not just those with firsthand knowledge. I write about a wide variety of mental and physical health-related subjects, and I rely on the entirety of academia to support this undertaking. Of course, I think it is preferable to have both statistical findings and personal experience in a blog, as long as personal experiences are not presented as if they are statistically significant. Depending on the subject, I will have more or less personal experience to contribute. But just because an issue affects someone else's friend or family member and not mine does not mean I should not care or should not try to discuss a topic. Many important movements for change or justice rely on the dissemination of information among the general population and the engagement of a broad group of individuals, not just those with their own skin in the game.

Personally, I try to write about issues based on population information, but I can see how this could easily offend someone who does not fall within the bounds of the high-level statistic I am using. I do ask that readers give me some grace when it comes to this statistical issue and try to think critically as you yourself read. If I write that 60% of a population falls under a certain umbrella, I hope you will remember that means that 40% fall outside that umbrella. Additionally, consider whether the statistic I present is stating a causation or a correlation. Consult multiple sources yourself, if that will help you form a more complete opinion. I also try to use multiple sources when time allows and to site these sources; I encourage you to check them out for yourself and to judge their validity. If a statistic is obviously skewed to me, I try to track it down and make some sort of comment about my observation of the potential issue (such as noting a small sample size), but this is also the reader's responsibility when trying to understand almost any piece. We should all use our analytic thinking skills to evaluate arguments.

Ultimately, my blog represents my own opinion. I am biased. I freely admit it. I have my own motivations, experiences, and tendencies. As a writer, I need to consider the implications of using statistics and be careful/honest in how I present arguments based on the numbers. But that also means you need to engage your mind to judge the validity of the statistical statements you read to form your own opinion. It's kind of like reading the newspaper with an eye for the truth that even some articles presented as fact are actually opinion. Shocking, I know! And if you find a journal piece or a news article that counters something that I have written personally, by all means, share it! I love a lively and respectful discussion!


Cartoons copied from: http://lovestats.wordpress.com/dman/
For originals, see Dilbert or XKCD.com