Thread Status:
Not open for further replies.
  1. Steerpike
    Offline

    Steerpike Felis amatus Supporter Contributor

    Joined:
    Jul 5, 2010
    Messages:
    11,075
    Likes Received:
    5,272
    Location:
    California, US

    It is Important to Approach Science with a Critical Mind

    Discussion in 'Debate Room' started by Steerpike, Nov 3, 2013.

    Came across a Scientific American blog post today that reminded me of the fact that so many people, and particularly media outlets, will view any statement in the primary scientific literature as fact. There is only one way to evaluate research in the primary literature, and that is to read the actual research study critically, not just cite the conclusions of the researchers or, worse yet, a media report on the results of the research.

    When I was in a Ph.D. program in biochemistry, we had one class where all we did was read critically what was in the primary literature. The professor would bring in the current issue of Cell, or whatever journal, and we'd go through the reports one by one. Many times, the researchers conclusions were over-stated or not supporting by the evidence they were reporting. Not a majority of the time - most of the articles were pretty solid - but often enough that it taught me to never accept something in the primary literature at face value.

    The SciAm blog post is here: http://blogs.scientificamerican.com/cross-check/2013/11/02/a-dig-through-old-files-reminds-me-why-im-so-critical-of-science/

    The guy is writing about his personal experience as a science journalist. He links to a fairly well-known essay in PLOS Medicine by Ioannidis, where the author makes some rather bold claims, the sentence that received the most attention being "It can be proven that most claimed research findings are false."

    Ioannidis was interviewed by Scientific American and elaborated by saying "False positives and exaggerated results in peer-reviewed scientific studies have reached epidemic proportions in recent years. The problem is rampant in economics, the social sciences and even the natural sciences, but it is particularly egregious in biomedicine."

    The Ioannidis report is here: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124

    I'm not prepared to accept the extent of the author's conclusions, but anyone who has read through the primary literature themselves (or at least done so critically) knows that there are plenty of cases where the reported results are questionable.

    People who have a dislike or distrust of science will take these links to mean we should discard science (or at least discard science they don't like, personally), but I don't think that's the point in either case. The point is we should think critically about these things, and not just accept something because it is reported in the literature. In fact, even when there is a "scientific consensus," we shouldn't just accept it blindly. The history of science is full of examples where the scientific consensus turned out to be wrong (and in some cases the consensus was very strong indeed, to the point it had become dogma but was still ultimately proven wrong).

    The reason something false becomes dogma or consensus is often that scientists stop looking at or questioning the basic assumptions of their work. They're already down the road a bit, and their results are being interpreted in light of the accepted dogma that everyone knows is true. Sometimes, too, it is because the prominent voices in the scientific establishment have built careers on a certain viewpoint, staked reputations on it, and aren't going to budge even in the face of new evidence (see Clovis first, for example).

    The strong point of science is that it looks to get these things largely right - ultimately. Sometimes it can take a while, but ultimately, despite the false direction, steps backwards, biases, institutionalized dogma, and so on, science progresses overall in the right direction.

    But don't let someone toss a news report at you, or link you to what the NY Times says about a research study, or even be taken in by the conclusions section of a reference in the primary scientific literature, without thinking about what the scientists actually did, what their results were, and deciding for yourself whether the conclusions are warranted.
     
    GingerCoffee likes this.
  2. thirdwind
    Offline

    thirdwind Contributing Member Contest Administrator Reviewer Contributor

    Joined:
    Jul 17, 2008
    Messages:
    7,351
    Likes Received:
    2,891
    Location:
    Boston
    I agree with what you're saying, but what about all those poor souls who don't know much about science? While I'm not completely clueless (i.e., I understand the simple things), there are some topics, like particle physics for example, that I no absolutely nothing about. So in those cases I can't really comment on if the experiment was conducted properly, if the results make sense, etc.
     
  3. Steerpike
    Offline

    Steerpike Felis amatus Supporter Contributor

    Joined:
    Jul 5, 2010
    Messages:
    11,075
    Likes Received:
    5,272
    Location:
    California, US
    Sure, that's certainly true. When it comes to a lot of the math, I just have to trust the researchers did it correctly. In many cases, I couldn't check it if I wanted to, and in no case do I really want to. But even setting that aside, I think there is a lot more an average non-scientific person can do when reading some studies. It certainly takes time, however. I'm not saying everyone can do it all, or should want to do it all, but the key is to be aware and don't just accept a research report at face value. If you can't do the critical analysis yourself, chances are someone else can and is, and by simply having a critical mindset and not just accepting what is reported, you're more likely to do some further looking and come across an analysis by someone else (which also has to be read critically, of course. Yeah, you can see why people don't do this).

    Ideally, we'd have a lot more reliable media sources that do a great job of digging into reports and analyzing them critically, then presenting their own views for others. That way at least you could see the results of such an analysis and having more information to work with in trying to figure out what's true, what's hyperbole, and where the researchers could be off track.
     
  4. mammamaia
    Offline

    mammamaia nit-picker-in-chief Contributor

    Joined:
    Nov 21, 2006
    Messages:
    19,316
    Likes Received:
    1,014
    Location:
    Coquille, Oregon
    a great big 'AMEN' to that, sp!

    see my post in the thread mentioning fluoride and the one re afterlife, for confirmation...

    http://www.writingforums.org/threads/tim-minchin-science-inspires-so-dont-let-your-art-rule-your-head.128815/#post-1162736

    http://www.writingforums.org/threads/the-afterlife-are-skeptics-just-cynics.128835/#post-1162756
     
  5. Cogito
    Offline

    Cogito Former Mod, Retired Supporter Contributor

    Joined:
    May 19, 2007
    Messages:
    35,935
    Likes Received:
    2,043
    Location:
    Massachusetts, USA
    If you don't understand a topic, whether it is science or anything else, take ANY absolute assertions about it with a large grain of salt. If the matter under discussion is important to you, make the effort to learn enough about the subject to follow whatever controversy may exist. Otherwise, you are at least partially a slave to ignorance.

    No real scientist will be offended by skepticism. Skepticism is the lifeblood of science.
     
  6. GingerCoffee
    Offline

    GingerCoffee Web Surfer Girl Contributor

    Joined:
    Mar 3, 2013
    Messages:
    17,604
    Likes Received:
    5,877
    Location:
    Ralph's side of the island.
    One need not become an expert in every scientific field in order to evaluate scientific research. But everyone should make an effort to become media literate.

    I was struck by this from the first OP citation:
    My first thought was, who does he believe is responsible for the hype?

    Horgan neglect's to mention the mainstream news' business models that produce a commodity rather than information. News as a commodity means sensation, breakthroughs, scandals are what sells. Information in an informative, accurate and neutral presentation does not sell the news.

    And it's not just the news, it's also all the so called science programs on TV. Most of the better quality programs have disappeared or become watered down. They were few and far between in the first place. What do we have instead? Numerous pseudoscience programs that spew false facts without rebuttal, claiming the conclusions of eye witnesses (not the observations, but the conclusions) are facts, plaster prints of Bigfoot and films of animals (or costumed humans) are analyzed using unquestioned questionable scientific assertions, and misstated statistics like how many ships and planes have 'disappeared' in the Bermuda Triangle.

    Horgan continues:
    Again, the scientific community is not the main source of the hyping, they are instead doing the complaining, the checking, and the peer reviewing, as they should.

    Here is Horgan, educated as a journalist with an interest in science reporting, revealing what the problem is, but it would appear he has missed the forest for the trees. The underlying problem of the bad reporting is brushed over while instead he appears to be blaming the scientists for the problem.

    Yes, bad science gets through, peer review is not perfect, scientists include as many deceitful greedy personalities as one would find in other professions.

    While Horgan asks:
    He gives a valid answer, but overlooks the bulk of the problem:
    If you only look at the few dishonest or over confident researchers like Petrofsky or the even more infamous, Andrew Wakefield, I imagine it does look like the scientists are mostly duping the reporters.

    But if you look at the average science news reporting, every day you can hear about a pilot study described as, "we now know" or a "breakthrough" or a "cure" when the truth is no such thing. Science reporters often don't know the most basic things like the difference between a bacteria and a virus, nor do they make an effort to learn the most basic things about the scientific field they are reporting on.

    Wakefield's scam went on for years and the anti-vaxxer movement he started based on falsified research will do lingering damage for years to come. Public health and vaccine researchers were screaming the entire time, 'Wakefield' is wrong. The damage done by the media has yet to be corrected to this day.

    As for what can the scientific communities do about the problem? I'm happy to report that at least in the medical field, what Horgan suggests we do has been going on for years, that is, taking a much more critical look at research conclusions. Horgan may just now be reporting on this, but a movement has been prevalent in the scientific circles (including the skeptical community ;)) for a number of years. It's been especially prevalent in the movement toward scientific evidence based medicine.

    "Snake Oil Science" doesn't just look at the ineffectiveness of alternative medicine, it explores the problems with the research that is cited supporting it.
    Ben Goldacre has been promoting a critical look at research for years:
    Goldacre reported on Ioannidis' work in a Bad Science column in Aug of 2011.
    The problem with only positive results being published is one of the focuses of Goldacre's work.


    Bottom line, it's great to see these revelations getting to a wider audience. I think Horgan could have noted the additional problems with typical mainstream news science reporting, but oh well. Revelations about the flaws in research conclusions, especially in the medical field, began to emerge in earnest as a focus of scientific evidence based medicine practitioners in ~2008. It's the scientists that have detected and set out on a corrected course. The reporters are simply now starting to notice.
     
  7. GingerCoffee
    Offline

    GingerCoffee Web Surfer Girl Contributor

    Joined:
    Mar 3, 2013
    Messages:
    17,604
    Likes Received:
    5,877
    Location:
    Ralph's side of the island.
    It depends on if the results matter to a person how far one goes to verify conclusions.

    There are additional basics people can learn without having to understand the technicalities in a paper. For example, look for repeatable results from different researchers. And now, given what we know about negative results disproportionately under-reported compared to positive results, it also helps to look at the funding sources of the studies, looking for diversity.

    Media literacy provides methods of sorting through distorted reporting. But that's a whole topic in itself and a bit off topic here.


    People can develop basic skills of critical thinking without needing the most technical depth of any one scientific field.
     
  8. Steerpike
    Offline

    Steerpike Felis amatus Supporter Contributor

    Joined:
    Jul 5, 2010
    Messages:
    11,075
    Likes Received:
    5,272
    Location:
    California, US
    Anyone who looks on science as infallible, or even on scientific consensus as necessarily being the truth simply by virtue of there being a consensus, isn't a very good 'scientist' in their own right. Those sorts of things should be anathema to science.

    When it comes to mis-stating the results of scientific studies, the media tends to be truly awful. They want to create sensationalism. But the scientific community does miss things as a result of being too blinded by accepted dogma. To its credit, science is better at correcting itself than, say, religion. But that doesn't mean that at any given time scientists may widely accept something that is completely false, and fail to examine it further until one person comes along and demonstrates the problem.

    I've even seen scientific reports in very prestigious journals (PNAS) have statements removed by the scientists themselves prior to submission for non-scientific (aka political) reasons.
     
    123456789 likes this.
  9. GingerCoffee
    Offline

    GingerCoffee Web Surfer Girl Contributor

    Joined:
    Mar 3, 2013
    Messages:
    17,604
    Likes Received:
    5,877
    Location:
    Ralph's side of the island.
    And they are.

    Individuals are fallible, the process is fallible, but the principles of the scientific process includes acknowledgement of these tenets.
     
  10. Steerpike
    Offline

    Steerpike Felis amatus Supporter Contributor

    Joined:
    Jul 5, 2010
    Messages:
    11,075
    Likes Received:
    5,272
    Location:
    California, US
    Yes, they are anathema to science. But also part of human nature, and scientists have the same human failings as anyone else.
     
  11. GingerCoffee
    Offline

    GingerCoffee Web Surfer Girl Contributor

    Joined:
    Mar 3, 2013
    Messages:
    17,604
    Likes Received:
    5,877
    Location:
    Ralph's side of the island.
    Out of curiosity, who/what is your OP addressing? It's a bit of choir preaching, isn't it?
     
  12. Steerpike
    Offline

    Steerpike Felis amatus Supporter Contributor

    Joined:
    Jul 5, 2010
    Messages:
    11,075
    Likes Received:
    5,272
    Location:
    California, US
    Maybe. Being SciAm, seems like the audience is scientists and the scientifically literate, who aren't immune from these problems.
     
  13. GingerCoffee
    Offline

    GingerCoffee Web Surfer Girl Contributor

    Joined:
    Mar 3, 2013
    Messages:
    17,604
    Likes Received:
    5,877
    Location:
    Ralph's side of the island.
    There's plenty to address in the SciAm article, which I did. And it's true lay people (and certainly more than a few scientists) need to be reminded of the fraud and misleading that goes on in published research. But then you slipped into choir preaching, "Anyone who looks on science as infallible, or even on scientific consensus as necessarily being the truth", as if no scientists or critical thinkers understood these issues, as if the whole process was consistently overrated, and, as if you had a better alternative.
     
  14. Steerpike
    Offline

    Steerpike Felis amatus Supporter Contributor

    Joined:
    Jul 5, 2010
    Messages:
    11,075
    Likes Received:
    5,272
    Location:
    California, US
    No, this is just reading into it things I didn't say. If I'd meant to say those things, I would have typed them into my post.
     
  15. GingerCoffee
    Offline

    GingerCoffee Web Surfer Girl Contributor

    Joined:
    Mar 3, 2013
    Messages:
    17,604
    Likes Received:
    5,877
    Location:
    Ralph's side of the island.
    Oh the irony. :rolleyes:
     
    LizaKaye likes this.
  16. LizaKaye
    Offline

    LizaKaye New Member

    Joined:
    Sep 25, 2013
    Messages:
    2
    Likes Received:
    0
    Location:
    Chicago, IL
    Approaching anything with a preset for your vision closely parallels ignorance. Whether you're good at science or not, an open mind should be used, just as it should for anything else. Approaching critically now forces you to break down that wall before getting to your own true, raw, conclusion. Those that opt to believe everything they are told from science are probably guilty for believing that everything on the internet is true. I don't agree with this at all.
     
    GingerCoffee likes this.
  17. GingerCoffee
    Offline

    GingerCoffee Web Surfer Girl Contributor

    Joined:
    Mar 3, 2013
    Messages:
    17,604
    Likes Received:
    5,877
    Location:
    Ralph's side of the island.
    It's unscientific to think all scientifically arrived at conclusions are correct.


    Welcome to the forum, BTW. :)
     
    LizaKaye likes this.
  18. GingerCoffee
    Offline

    GingerCoffee Web Surfer Girl Contributor

    Joined:
    Mar 3, 2013
    Messages:
    17,604
    Likes Received:
    5,877
    Location:
    Ralph's side of the island.
    duplicate post
     
  19. AJC
    Offline

    AJC Active Member

    Joined:
    Nov 1, 2013
    Messages:
    149
    Likes Received:
    60
    I was part of an outreach program back in grad school that taught people with non-science backgrounds to read and interpret science articles critically. The one thing I noticed is how dedicated scientists are in this regard. A lot of the people I knew back then worked with local schools to improve scientific literacy. You do need a lot of patience however. Some of the people I worked with were either unable or unwilling to learn.
     
    GingerCoffee likes this.
Thread Status:
Not open for further replies.

Share This Page