Opinionista Ivo Vegter 15 September 2015

Science says science is shoddy

It’s a common refrain: if you’re sceptical of the latest headline-grabbing scientific papers, you’re a 'denier', who 'rejects science'. But 'science' is not truth. Science is a procedure for discovering truth. There’s a difference. And while science is powerful, it is far from perfect. Happily, there is science that says so, and aims to improve matters.

Until last month, the best available scientific estimate of how many trees we had on Earth was 400-billion. As of this month, there are 3-trillion trees on earth.

Despite this dramatic under-estimate of tree numbers, this new research will not restore the Amazon forest to its mythical position as the so-called “lungs of the earth”. Trees play only a subordinate role in converting carbon dioxide to oxygen. The Amazon never was the lungs of the earth. The majority of the world’s oxygen is produced by phytoplankton in the ocean.

Such are the numbers, the bare facts of science, upon which great public policy decisions hang.

The Banting Diet, as made famous by sports scientist Tim Noakes, preaches the virtues of bacon, and who can argue with that? But if claims that a diet high in saturated fat and low in carbohydrates will prevent diabetes, cancer and heart disease sound too good to be true, they probably are.

One reason why Banting has become so popular is the idea that some people react negatively to gluten, which are proteins commonly found alongside starch in grains. It is well known that patients with Celiac disease, an uncommon and serious auto-immune condition that causes inflammation of the intestinal tract, cannot tolerate gluten. However, patients who do not suffer from Celiac disease have also been reporting adverse effects from eating gluten. They, scientists would have us believe, might suffer from something known as non-Celiac gluten intolerance.

Two years later, those same scientists discovered that what appeared to be gluten-sensitivity is more likely sensitivity to a group of sugars known as fermentable oligosaccharides, disaccharides, monosaccharides and polyols (clumsily abbreviated as FODMAP). A follow-up study of self-diagnosed sufferers of gluten intolerance confirmed the culprit was FODMAP, not gluten.

Such are the flimsy facts of science upon which billion-dollar industries are built. In the US, about 1.8-million people suffer from Celiac disease, and 18-million (6%) claim to have non-Celiac gluten intolerance. Yet three times as many people buy gluten-free food, and a staggering 30% of the population would like to do so. The market for gluten-free products is worth north of $10-billion, and growing rapidly. This hive of food-marketing quackery is based on nothing more than a few speculative papers, since disproven by their own authors.

C Glenn Begley, former head of global cancer research at Amgen, a biopharmaceutical firm, began a project to replicate 53 academic studies considered to be landmarks in oncology. The result? Only six of the replication attempts succeeded, despite the fact that replication (verifying the conclusions given the same data) is a far lower hurdle than reproducability (reaching the same conclusions in an entirely new study).

A similar paper by Bayer, another pharmaceutical company, found that only 20% to 25% of published academic papers could be validated by replication. Many studies expand upon previous work without trying to validate the original research, and sometimes “an entire field” is built on such shaky foundations.

The reasons for these failures are many, but among them is pressure to publish significant findings. Sometimes, a result that was obtained only once was included in a paper “because it made the best story”. According to Ferric Fang, quoted by Reuters speaking to a National Academy of Sciences panel: “The surest ticket to getting a grant or job is getting published in a high-profile journal. This is an unhealthy belief that can lead a scientist to engage in sensationalism and sometimes even dishonest behaviour.”

Everything in science is based on publishing a peer-reviewed paper in a high-ranking journal. Absolutely everything,” Ivan Oransky, founder of the blog Retraction Watch, told the Washington Post. “You want to get a grant, you want to get promoted, you want to get tenure. That’s how you do it. That’s the currency of the realm.”

Such is the science on which billions of dollars in research funding, and millions of lives, hang.

According to a map published in 2005 by the United Nations Environment Programme (Unep), the world would have 50-million “climate refugees” by 2010. The claim was an exaggerated restatement of a guess by the legendary environmentalist Norman ‘heroic extrapolations’ Myers, who in 1979 made up the “40,000 per year” species extinction number that is still quoted by luminaries like Al Gore. Myers based his estimate on the total populations of regions that might be affected. Among his assumptions was that everybody would move. In reality, populations increased in the supposedly catastrophe-struck regions. Gavin Atkins of Asian Correspondent broke this story, and published a follow-up with more detail. Another retelling, with an archive of the offending Unep page and map, can be found here.

The UN’s response to the embarrassing non-appearance of 50-million refugees by 2010 was to remove the map from its website and simply substitute the discredited prediction with a new one: “The world will have 50-million environmental refugees by 2020”.

Such is the science on which major global policy initiatives are based.

It is commonly believed that psychotic people are often violent. But as it turns out, this is a long-lived myth. A recent study of violent offenders found that the association between psychosis and violence is tenuous, at best. Violence is only rarely preceded by psychosis, and there is no evidence for a specific sub-group of psychotic criminals. Although bipolar disorder has a higher association with criminal behaviour, it still accounts for only 10% of cases. Our caricature of the violent, deranged criminal may not be as accurate as we’d like.

Surprisingly, the field of psychology does better on the reproducability front than the cancer research papers we looked at earlier. Still, studies in psychology are non-reproducible more often than not, according to work conducted by Brian Nosek, executive director of the Center for Open Science.

Such is the science upon which our mental health is judged.

In politics, academic studies are used as weapons,” Peter Coy wrote for Bloomberg, in his 2013 analysis of the so-called Reinhart/Rogoff controversy. These economists had calculated the correlation between high public debt levels and lower economic growth, to confirm the belief that austerity was necessary to curb debt and spur growth. However, they made a few elementary spreadsheet mistakes along the way. Those who opposed austerity measures, and advocated stronger public spending in the face of a weak economy, had a field day.

Such is the science upon which interventions in major economies are based.

John PA Ioannidis took all this a step further. His thesis, published in 2005, is that most published research findings are false. It is more likely that a study is false if effect size is small, sample size is small, there is financial interest in the result, or when many scientists are all chasing statistically significant results. “For many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias,” he writes.

In an effort to make academic research available to a wider audience, open access journals were established. Instead of readers paying for journal access, the authors of papers would cover the costs and commercial interest of the publisher. Not surprisingly, this led to the rise of dodgy pay-to-play journals, some of which have publish laughably bad ‘science’.

The media is, of course, complicit. Its own need for sensational headlines has declared any number of cancer cures, climate disasters, or miracle foods to be the latest in scientific truth. A single study can serve as a peg upon which to hang a dramatic story. Of course, scientists know this, and use it to manipulate the media. And they also know that few mere journalists will be brave enough to express scepticism if the study appears in a major, peer-reviewed journal.

But the major journals are not immune. Nobel Prize winner Randy Schekman made waves in 2013 when he accused top journals Nature, Cell and Science of damaging science. “The big journals’ reputations are only partly warranted,” he wrote, declaring a boycott. “While they publish many outstanding papers, they do not publish only outstanding papers. Neither are they the only publishers of outstanding research.”

The problem, he felt, was with the incentive structure the journals offer academics, career-wise. They encourage bold, citable and controversial claims, while placing less emphasis on the workaday business of replicating or reproducing studies.

Projects are under way in an effort to improve the reliability and transparency of research results. Some notable examples are the PLOS/Science Exchange Reproducability Initiative, the AllTrials register of clinical trial results, and the Open Science Framework’s guidelines for Transparency and Openness Promotion. In the economics field, replicating empirical results is also high on the agenda, and a prediction market has been established to bet on which results will hold up to public scrutiny.

As an argument, “but it’s science” is essentially meaningless. When someone questions a particular thesis, prediction or result, that is not “denying science£. On the contrary. The reason these projects exist is because a great deal of ‘science’ – and in some fields a majority of newly-published work – is wrong.

Science is never ‘settled’. It is not conducted by ‘consensus’. Nothing is elevated to the level of truth simply by being published in a peer-reviewed scientific journal. Peer review can be corrupted, selection criteria can be imperfect, and incentives can be perverse.

The term ‘science’ does not describe one side of a public debate. In fact, whenever you hear it used in such a manner, it is probably designed to shut down debate and advance a particular political goal.

‘Science’ describes a process by which wrong results are corrected, and correct results are confirmed. When the data, methods, trials and results published in scientific papers are open to undergo replication and reproduction studies – whether they pass or fail – that is when ‘science’ becomes a profoundly powerful force for progress. Anything else is professional posturing, or worse, politics. DM


Comments - share your knowledge and experience

Please note you must be a Maverick Insider to comment. Sign up here or if you are already an Insider.


Financial Times Africa Summit: Ramaphosa and Rupert’s unlikely pas de deux

By John Battersby

Speaking Kurdish in Turkey was illegal until the 1990s.