This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.
News
6 Questions to Ask About the Science Behind a Study
Have you noticed that we spend a lot of time talking about scientific breakthroughs and information you need to know now? If you watch local or national news in the U.S., there seems to be something new daily we must do, must avoid, or at least must heed. Maybe it’s about exercising more or a key nutrient that we just have to add into our diet.
John Oliver did a terrific send-up of how science often gets “translated” to make it more sexy for the nightly news.
Of course, it’s not just health & fitness and it’s not just about the sensationalizing of otherwise sound science. The demand for studies to report on has also led to far more reporting on invalid, often ideologically driven studies that lack scientific integrity.
At the American Institute of Architects 2016 conference, for example, several panelists focused on what they claim are bad chemicals in some materials and products. There was even comparison of product labels to food labels. Sure, chlorine is dangerous if you inhale a room full of it or ingest a gallon of it. But as an ingredient in making vinyl resin—or added to purify your public pool—not so much.
Six questions to help separate the facts from the pseudoscience.
To help separate the science from the squishy stuff, we’ve put together six questions to ask to understand the validity of the next study that lands in the headlines or on your desk.
- Is it an observational study? If it is, the results are inconclusive. Any study based solely on observing the behavior of a group of participants lacks proper scientific credibility because any number of unknowns can be contributing factors.
- Is the study peer-reviewed? If it is not, then it represents only the views of the authors (without any independent validation).
- Is causation proven? Studies often confuse correlation with causation. Just because two data points may share a relationship doesn’t mean one causes the other. For example, let’s say people who like bananas tend to drink more coffee. Does that mean eating bananas causes someone to drink coffee? Probably not.
- What’s the sample size? If the number of people in a given study is statistically insignificant, how can any reliable scientific conclusions be drawn from the results?
- Is the study an outlier? A lone study that challenges a body of scientific literature may prove correct—or not. Either way, a lot of fact checking should take place before concluding these researchers know something no one else does.
- Where was the study published? Like outlier studies, research that appears in obscure outlets that fail to garner the respect of the broader scientific or medical community should be approached with caution. It may be that the research supports an ideological perspective held by the publisher; or maybe the research wasn’t strong enough to appear in a more mainstream publication.
As John Oliver pointed out, most studies are based on sound science—and often it’s the publicists and news shows looking for an angle that bias the way they’re reported. That said, it’s always important to review the science and the rigor of the research just to be safe.