One of my readers commented on my post about
sex work and childhood abuse that I was essentially dehumanizing the issue by using statistics. While I think the reader may have made some incorrect assumptions about my motives, I think he or she brought up a valid point on some level, which is that statistics have their limits. So I thought I would write a brief post about the benefits and limits of biostatistical information. This is by no means intended to be an exhaustive list of all the issues that arise around statistics, but perhaps it can get your mental juices flowing to think more deeply about the articles you read.
Firstly, I would like to give a little pitch for why statistics are important. Have you ever heard someone say something like this (or have you said it yourself)?
- My friend got in a high speed car accident, and the only injury she sustained was a bruise from the seat belt; if she had not been wearing it, she would have no seat belt bruise, and that proves that you're safer without wearing seat belts!
- I have gone to McDonalds every week for fifty years and am in great health, so those fast food nay-sayers don't know what they're talking about.
- My cousin did not go to college, and she is making twice as much as I do, despite my degree. College does not help you get a higher paying job. I would not recommend that others go.
- None of my sexually active friends got infections back when we were teenagers, so it's not something I really worry about for my own teenage daughter.
- My aunt smoked for 75 years and lived to be 90 years old, so that just goes to show that smoking isn't as bad as they say.
These statements represent a very common logical fallacy in our culture, which is the assumption that a personal experience can be used to make statements about the entire population. How do you know that your personal experience is the norm and not the exception? Even if you believe it is the norm among your group of friends or acquaintances, how do you know your group of friends is the norm and not the exception compared to other groups? The answer is, without information about
the experiences of others, you
don't actually know.
This is one of the reasons statistics are useful. They show what the norm of experience is. That doesn't mean they show what happens with every single person in every single case, but they can show what is most likely to happen in a given population. So to counter one of the sample statements above, even though you think college was useless because you have a lower paying job than your less educated cousin, statistics can show that you and your cousin are the anomalies. Based on information I shared in a
previous blog post, the median income for someone with a bachelors degree is 65% more than the median income for someone with only a high school diploma. That median is not the amount every person makes; it just implies that you probably have a better shot of making more with the degree. In all likelihood, you are one of the 50% of people with a degree who earns below the median income for your education level, and your cousin is above the median for his or her education level. That doesn't negate the usefulness of the median information for the population on average. Similarly, if a report shows that 75% of prostitutes were abused as children, as I wrote about in another
earlier blog post, and you personally are a sex worker who was not abused, then that means you fall into the minority 25%, as do one in four of your colleagues; that is a large enough minority that we would expect there to be significant numbers who were not abused, and the fact that you are someone or know someone outside of the majority does not prove that the 75% figure is wrong.
Because statistics represent what happens over an entire population, they are especially useful for making health policy decisions. It is natural for us to want to take actions that
feel right to us, but we must also consider if they are based on the reality of a population. For example, one politician might feel that smoking is a good thing, because it curbs his appetite and relaxes him after a long day. None of his friends have experienced major smoking-related health problems, so he believes very strongly that it is good for his health. Based on his feelings, he may want to ensure that everyone has greater access to cigarettes at as young an age as possible. His intuition tells him that this will benefit the health of the population. Yet because we have statistical information showing that smoking is harmful and addictive,
on average, we pay attention to the statistics rather than one person's experience and feelings, and we have laws against selling cigarettes to minors. This seems like a common sense approach, but we often forget to ask if policy decisions have a basis in reality or in statistically proven effectiveness. Of course, we cannot always rely on statistics. Sometimes, we want to try something new, in which case we cannot know how it will pan out. And often times, the particulars of one population may be so different than those of another population that we do not trust the results that happened elsewhere to apply to the new population. But when it comes to many health decisions and initiatives that have been tested in a variety of groups, we can use statistics above our feelings to figure out what will be most effective.
So we have established that statistics help ground us in the realities of a population as a whole, beyond our own experiences and feelings. But as I stated in the introduction, statistics do have their limits. And it is often left up to us as consumers of the media to filter through the plethora of information and to figure out how reliable/useful a given statistic is. So how do you comb through the constant bombardment of data?
One thing to consider is that not all statistics are created equal. The structure of a study, such as the number of people studied, the way the questions were worded, the way lab tests were taken, or the types of people sampled, can all lead the data to represent underlying realities differently. When researchers craft questions, for example, they should try to maintain some neutrality that will not influence the answer. Imagine a study to determine patient opinions of their doctor. One question might read, "A lot of people in this town hate Doctor John Smith. On a scale of one to ten, how much do you hate Dr. Smith?" This question automatically introduces bias by telling the interviewee that others have a negative opinion and may affect the thoughts of the participant or their comfort with expressing a certain opinion. A less biased question might say, "On a scale of one to ten, how do you feel as a whole about Doctor John Smith, with ten being highly satisfied and one being highly dissatisfied?" Some people might even read that question and see issues, since highly satisfied is stated before highly dissatisfied, and the question is very vague about what aspects of the doctor the interviewee likes or does not like. Is this a question about accurate diagnosis, turnaround time, personality, appearance? As you can see, collecting statistics in an interview or survey format takes a lot of forethought. Even biostatistics collected from lab results, which do not involve direct questioning, can have elements of bias if not all labs were analyzed using the same standard procedure, if the standard procedure is more likely to erroneously produce a certain result, or if the participants were chosen in a biased fashion, etc. (For an example of how sample selection and response choice can skew a statistic, read about the
1936 presidential campaign polling gaff.) Sometimes I even read articles that cause me to question whether the outcome being tested answers the question it is intending to quantify.
Let's look at another way statistics can misrepresent a situation. Imagine that you read a study stating that 75% of 13-year-old students at St. Catherine's school menstruate, but only 35% of 13-year-old students at Sedgwick menstruate. At first glance, you would say, wow, this is a major discrepancy! Something strange must be going on to cause this big gap between the two schools! But what if I told you St. Catherine's was an all-girls school and Sedgwick was a mixed gender school. You would have to ask, does the 35% statistic at Sedgwick include the boys (who of course would not menstruate at any age)? If so, that changes everything! This is a somewhat silly and made-up example, as this sort of glaring reporting difference is not common in my experience. But it illustrates the point well - we need to consider what is really being represented by a statistic, not just what the face-value number is.
Another common issue you may face in understanding biostatistics is confusing correlation with causation. Just because two things are statistically associated does not mean that one causes the other. For example, in the above scenario, you could say that a student is more likely to menstruate if he or she attends St. Catherine's than if he or she attends Sedgwick. That is a
correlation which is proven by the statistics 75% and 35%. A
causation statement would say that attending St. Catherine's
causes menstruation to begin by age 13. Even though we know menstruation and school are correlated statistically, without more information on how many girls are at Sedgwick (compared to how many boys), it would be silly to conclude that the school attended is what actually causes menstruation. The statistic alerts us to something
else underlying the issue, indicating that we need to dig deeper to find the cause - and it turns out the cause for the correlation may simply be the fact that there are only girls at St. Catherine's and both sexes at Sedgwick. If we adjusted the data to account for this difference, we might find that the rates among girls
alone were identical at both schools. There is really no way to know just by looking at the values presented.
Let's take another example of causation versus correlation, this time from real life statistics. Imagine you are deciding between moving to California or to Oklahoma and are concerned about the health impact of where you live. You read on the
CDC website that California has a lung cancer rate under 59.5 per 100,000 people, but Oklahoma is over 73.8. Based on this information, you see a correlation between living in Oklahoma and getting lung cancer. At face value, this indicates that you may want to move to California to decrease your risk. However, you must ask yourself, is it the act of living in Oklahoma that
causes lung cancer, or is there something else going on under the surface of these statistics?
As it turns out, when you dig deeper, you find that the
Huffington Post reported on a Gallup-Healthways pole showing 15.7% of individuals smoke in California and 26.6% of individuals smoke in Oklahoma. So the cause of the lung cancer may actually be cigarettes, at least in part, as opposed to the physical placement of one's home in Oklahoma. Other factors could also contribute, such as a higher percentage of people working in industries exposed to chemicals, socioeconomic concerns, access to health care, a population that eats certain foods, or a population that exercises less, etc. These things seem more likely to cause the different cancer rates, even though the cancer rate is
correlated to the placement of your home. (Note: almost any factor may be correlation and not causation - specific studies are needed to show that one thing actually causes another.) It is true that even taking into account these lifestyle differences, it is still possible that there are environmental factors contributing to the health in each state, such as air pollution or water contaminants, or certain laws that affect population health. These things could affect the cancer rates and may legitimately lead you to prefer living in one state over the other for health reasons. But it is also possible that if you personally do not smoke or work with chemicals, and if you personally take care of your overall health, it may be that your chances of getting lung cancer are more or less the same whether you live in California or Oklahoma. The high-level information we have above is not enough to say for sure. You would need to dig deeper than the surface level cancer rates to determine your actual risk in each location.
Although there is plenty of reason to question the validity of the statistics you read, there is also a lot of misinformation floating around the internet about what makes a good study. For example, I have often seen people commenting on articles, "Well, this is ridiculous. They studied 3,000 people, but there are 300 million people in the U.S., so this is junk science and is not at all representative of everyone." However, if you take an introductory statistics class, you will learn that a sample size of 3,000, if properly selected to represent the population well, is actually a pretty large sample. Statistics is largely based around the ability to apply results from a sample to explain an entire population, and there is a lot of fancy math that goes into determining how accurately a sample represents a population. Now, if all 3,000 of the people sampled lived in one town or were all of the same ethnicity or all shared the same diet, then you would have to take other factors into consideration before simply assuming the results apply to everyone in the country. (Then again, if everyone was different with regards to those attributes, you would have to consider whether those differences affected the outcome being measured.) But many studies consider a wide variety of factors and adjustments when crafting a sample or analyzing the results. The type of study being conducted and the factors that may influence the outcomes can change the number and type of subjects needed to produce a reliable result. The lesson here is not to throw out all statistics that come from a sample.
Rather, what is important is to think critically about what a statistic is saying, to question how it was constructed, and to put these factors into consideration when you read about a study.
Yet another pitfall with statistics is the one my reader brought up in his or her comments, which is the tendency to lose site of the individuals represented. For example, we may talk about a program that will help 95% of people in a particular quandary and feel that the job is done once the program is implemented; yet the statistic may mask the dreadful plight of the 5% who are not helped. Without a connection to the actual people affected, it is hard to understand the human impact. Or when we talk about the 75% of prostitutes being abused as children, it is easy to link any conclusions about that sub-group with the entire group, when certain conclusions may or may not apply to every member of that 75% or not at all to the other 25%.
Some people believe that you should not use statistics to speak about an issue at all if you have not personally interacted with the population or been affected by the issue. They might call that dehumanizing. I tend to disagree and would simply say that you ought to be careful. As I wrote in my response to the comments on my earlier blog post, I do not believe it is inherently wrong to comment on statistical findings when I am not the individual who conducted the research myself or when I am not friendly with individuals affected. If an issue is important, I believe that many people who care should undertake the task of bringing others' research into the public consciousness, not just those with firsthand knowledge. I write about a wide variety of mental and physical health-related subjects, and I rely on the entirety of academia to support this undertaking. Of course, I think it is preferable to have both statistical findings and personal experience in a blog, as long as personal experiences are not presented as if they are statistically significant. Depending on the subject, I will have more or less personal experience to contribute. But just because an issue affects someone else's friend or family member and not mine does not mean I should not care or should not try to discuss a topic. Many important movements for change or justice rely on the dissemination of information among the general population and the engagement of a broad group of individuals, not just those with their own skin in the game.
Personally, I try to write about issues based on population information, but I can see how this could easily offend someone who does not fall within the bounds of the high-level statistic I am using. I do ask that readers give me some grace when it comes to this statistical issue and try to think critically as you yourself read. If I write that 60% of a population falls under a certain umbrella, I hope you will remember that means that 40% fall outside that umbrella. Additionally, consider whether the statistic I present is stating a causation or a correlation. Consult multiple sources yourself, if that will help you form a more complete opinion. I also try to use multiple sources when time allows and to site these sources; I encourage you to check them out for yourself and to judge their validity. If a statistic is obviously skewed to me, I try to track it down and make some sort of comment about my observation of the potential issue (such as noting a small sample size), but this is also the reader's responsibility when trying to understand almost any piece. We should all use our analytic thinking skills to evaluate arguments.
Ultimately, my blog represents my own opinion. I am biased. I freely admit it. I have my own motivations, experiences, and tendencies. As a writer, I need to consider the implications of using statistics and be careful/honest in how I present arguments based on the numbers. But that also means you need to engage your mind to judge the validity of the statistical statements you read to form your own opinion. It's kind of like reading the newspaper with an eye for the truth that even some articles presented as fact are actually opinion. Shocking, I know! And if you find a journal piece or a news article that counters something that I have written personally, by all means, share it! I love a lively and respectful discussion!
Cartoons copied from:
http://lovestats.wordpress.com/dman/
For originals, see Dilbert or XKCD.com
Starting in 1967, a consortium of companies led by Texaco spent twenty-three years drilling for oil in the Amazon. Today, what remains is a vast array of environmental damage that has altered the face of the treasured rain forest and harmed the health of the populations who once enjoyed life there. 
