{"id":1778,"date":"2014-10-21T20:27:15","date_gmt":"2014-10-21T20:27:15","guid":{"rendered":"http:\/\/www.lifeandnews.com\/articles\/?p=1778"},"modified":"2016-08-19T19:15:03","modified_gmt":"2016-08-19T19:15:03","slug":"the-10-stuff-ups-we-all-make-when-interpreting-research","status":"publish","type":"post","link":"https:\/\/www.lifeandnews.com\/articles\/the-10-stuff-ups-we-all-make-when-interpreting-research\/","title":{"rendered":"The 10 stuff-ups we all make when interpreting research"},"content":{"rendered":"

By Will J Grant<\/a>, Australian National University<\/em> and Rod Lamberts<\/a>, Australian National University<\/em><\/p>\n

UNDERSTANDING RESEARCH: What do we actually mean by research and how does it help inform our understanding of things? Understanding what\u2019s being said in any new research can be challenging and there are some common mistakes that people make.<\/em><\/p>\n

Have you ever tried to interpret some new research to work out what the study means in the grand scheme of things?<\/p>\n

Well maybe you\u2019re smart and didn\u2019t make any mistakes \u2013 but more likely you\u2019re like most humans and accidentally made one of these 10 stuff ups.<\/p>\n

1. Wait! That\u2019s just one study!<\/h2>\n

You wouldn\u2019t judge all old men based on just Rolf Harris or Nelson Mandela. And so neither should you judge any topic based on just one study.<\/p>\n

If you do it deliberately, it\u2019s cherry-picking. If you do it by accident, it\u2019s an example of the exception fallacy<\/a>.<\/p>\n

The well-worn and thoroughly discredited case of the measles, mumps and rubella (MMR<\/a>) vaccine causing autism<\/a> serves as a great example of both of these.<\/p>\n

People who blindly accepted Andrew Wakefield\u2019s (now retracted<\/a>) study – when all the other evidence was to the contrary – fell afoul of the exception fallacy. People who selectively used it to oppose vaccination were cherry-picking.<\/p>\n

2. Significant doesn\u2019t mean important<\/h2>\n

Some effects might well be statistically significant, but so tiny as to be useless in practice.<\/p>\n

\"\"<\/a><\/figure>\n

You know what they say about statistics?<\/span>
\nFlickr\/Frits Ahlefeldt-Laurvig<\/a>, CC BY-ND<\/a><\/span><\/p>\n

Associations (like correlations<\/a>) are great for falling foul of this, especially when studies have huge number of participants. Basically, if you have large numbers of participants in a study, significant associations tend to be plentiful, but not necessarily meaningful.<\/p>\n

One example can be seen in a study<\/a> of 22,000 people that found a significant (p<0.00001) association between people taking aspirin and a reduction in heart attacks, but the size of the result was miniscule.<\/p>\n

The difference in the likelihood of heart attacks between those taking aspirin every day and those who weren\u2019t was less than 1%. At this effect size \u2013 and considering the possible costs associated with taking aspirin \u2013 it is dubious whether it is worth taking at all.<\/p>\n

3. And effect size doesn\u2019t mean useful<\/h2>\n

We might have a treatment that lowers our risk of a condition by 50%. But if the risk of having that condition was already vanishingly low (say a lifetime risk of 0.002%), then reducing that might be a little pointless.<\/p>\n

We can flip this around and use what is called Number Needed to Treat (NNT<\/a>).<\/p>\n

In normal conditions if two random people out of 100,000 would get that condition during their lifetime, you\u2019d need all 100,000 to take the treatment to reduce that number to one.<\/p>\n

4. Are you judging the extremes by the majority?<\/h2>\n

Biology and medical research are great for reminding us that not all trends are linear.<\/p>\n

We all know that people with very high salt intakes<\/a> have a greater risk of cardio-vascular disease than people with a moderate salt intake.<\/p>\n

\"\"<\/a><\/figure>\n

Too much or too little salt – which as worse?<\/span>
\nFlickr\/JD Hancock<\/a>, CC BY<\/a><\/span><\/p>\n

But hey \u2013 people with a very low salt intake may also have a high risk of cardio-vascular disease<\/a> too.<\/p>\n

The graph is U shaped, not just a line going straight up. The people at each end of the graph are probably doing different things.<\/p>\n

5. Did you maybe even want to find that effect?<\/h2>\n

Even without trying, we notice and give more credence to information that agrees with views we already hold. We are attuned to seeing and accepting things that confirm what we already know, think and believe.<\/p>\n

There are numerous example of this confirmation bias but studies such as this<\/a> reveal how disturbing the effect can be.<\/p>\n

In this case, the more educated people believed a person to be, the lighter they (incorrectly) remembered that person\u2019s skin was.<\/p>\n

6. Were you tricked by sciencey snake oil?<\/h2>\n
A classic \u2013 The Turbo Encabulator.<\/figcaption><\/figure>\n

You won\u2019t be surprised to hear that sciencey-sounding stuff is seductive. Hey, even the advertisers like to use our words!<\/a><\/p>\n

But this is a real effect that clouds our ability to interpret research.<\/p>\n

In one study<\/a>, non-experts found even bad psychological explanations of behaviour more convincing when they were associated with irrelevant neuroscience information. And if you add in a nice-and-shiny fMRI scan, look out!<\/p>\n

7. Qualities aren\u2019t quantities and quantities aren\u2019t qualitites<\/h2>\n

For some reason, numbers feel more objective than adjectivally-laden descriptions of things. Numbers seem rational, words seem irrational. But sometimes numbers can confuse an issue.<\/p>\n

For example, we know people don\u2019t enjoy waiting in long queues at the bank. If we want to find out how to improve this, we could be tempted to measure waiting periods and then strive to try and reduce that time.<\/p>\n

But in reality you can only reduce the wait time so far. And a purely quantitative approach may miss other possibilities.<\/p>\n

If you asked people to describe how waiting made them feel, you might discover it\u2019s less about how long it takes, and more about how uncomfortable they are.<\/p>\n

8. Models by definition are not perfect representations of reality<\/h2>\n

A common battle-line between climate change deniers and people who actually understand evidence is the effectiveness and representativeness of climate models.<\/p>\n

But we can use much simpler models to look at this. Just take the classic model of an atom. It\u2019s frequently represented as a nice stable nucleus in the middle of a number of neatly orbiting electrons.<\/p>\n

While this doesn\u2019t reflect how an atom actually looks, it serves to explain fundamental aspects of the way atoms and their sub-elements work.<\/p>\n

This doesn\u2019t mean people haven\u2019t had misconceptions about atoms based on this simplified model. But these can be modified with further teaching, study and experience.<\/p>\n

9. Context matters<\/h2>\n

The US president Harry Truman once whinged about all his economists giving advice, but then immediately contradicting that with an \u201con the other hand<\/a>\u201d qualification.<\/p>\n

Individual scientists – and scientific disciplines – might be great at providing advice from just one frame. But for any complex social, political or personal issue there are often multiple disciplines and multiple points of view to take into account.<\/p>\n

To ponder this we can look at bike<\/a> helmet laws<\/a>. It\u2019s hard to deny that if someone has a bike accident and hits their head, they\u2019ll be better off if they\u2019re wearing a helmet.<\/p>\n

\"\"<\/a><\/figure>\n

Do bike helmet laws stop some people from taking up cycling?<\/span>
\nFlickr\/Petar<\/a>, CC BY-NC<\/a><\/span><\/p>\n

But if we are interested in whole-of-society health benefits, there is research<\/a> suggesting that a subset of the population will choose not to cycle at all if they are legally required to wear a helmet.<\/p>\n

Balance this against the number of accidents where a helmet actually makes a difference to the health outcome, and now helmet use may in fact be negatively impacting overall public health.<\/p>\n

Valid, reliable research can find that helmet laws are both good and bad for health.<\/p>\n

10. And just because it\u2019s peer reviewed that doesn\u2019t make it right<\/h2>\n

Peer review<\/a> is held up as a gold standard in science (and other) research at the highest levels.<\/p>\n

But even if we assume that the reviewers made no mistakes or that there were no biases in the publication policies (or that there wasn\u2019t any straight out deceit), an article appearing in a peer reviewed publication just means that the research is ready to be put out to the community of relevant experts for challenging, testing, and refining.<\/p>\n

It does not mean it\u2019s perfect, complete or correct. Peer review is the beginning of a study\u2019s active public life, not the culmination.<\/p>\n

And finally \u2026<\/h2>\n

Research is a human endeavour and as such is subject to all the wonders and horrors of any human endeavour.<\/p>\n

Just like in any other aspect of our lives, in the end, we have to make our own decisions. And sorry, appropriate use even of the world\u2019s best study does not relieve us of this wonderful and terrible responsibility.<\/p>\n

There will always be ambiguities that we have to wade through, so like any other human domain, do the best you can on your own, but if you get stuck, get some guidance directly from, or at least originally via, useful experts.<\/p>\n

\n

This article is part of a series on Understanding Research<\/a>.<\/strong><\/p>\n

Further reading:<\/strong>
\nWhy research beats anecdote in our search for knowledge<\/a><\/strong>
\nClearing up confusion between correlation and causation<\/a><\/strong>
\nWhere\u2019s the proof in science? There is none<\/a><\/strong>
\nPositives in negative results: when finding \u2018nothing\u2019 means something<\/a><\/strong>
\nThe risks of blowing your own trumpet too soon on research<\/a><\/strong>
\nHow to find the knowns and unknowns in any research<\/a><\/strong>
\nHow myths and tabloids feed on anomalies in science<\/a><\/strong><\/p>\n

\"The<\/p>\n

Will J Grant owns shares in a science communication consultancy. He has previously received funding from the Department of Industry. <\/em><\/p>\n

Rod Lamberts has received funding from the ARC in the past. He also holds shares in a science facilitation consultancy.<\/em><\/p>\n

This article was originally published on The Conversation<\/a>.
\nRead the original article<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

By Will J Grant, Australian National University and Rod Lamberts, Australian National University UNDERSTANDING RESEARCH: What do we actually mean by research and how does it help inform our understanding of things? Understanding what\u2019s being said in any new research can be challenging and there are some common mistakes that people make. Have you ever […]<\/p>\n","protected":false},"author":39,"featured_media":6718,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[8],"tags":[],"_links":{"self":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts\/1778"}],"collection":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/users\/39"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/comments?post=1778"}],"version-history":[{"count":2,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts\/1778\/revisions"}],"predecessor-version":[{"id":6719,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/posts\/1778\/revisions\/6719"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/media\/6718"}],"wp:attachment":[{"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/media?parent=1778"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/categories?post=1778"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lifeandnews.com\/articles\/wp-json\/wp\/v2\/tags?post=1778"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}