Many of my columns involve scientific studies of one sort or another. Often, they raise issues that most people – including your humble columnist – are not expert at. This week, I propose to highlight some of the common problems with research studies. Lest I be accused of confirmation bias, I’ll use as examples studies with findings that suit my own views, and some that do not.
A few months ago, a lovely little study by Jonathan Klick and Joshua Wright appeared that suggested using reusable shopping bags leads to a higher numbers of lethal e.coli infections (food poisoning) turning up at local hospitals.
It seemed simple enough: San Francisco, where plastic bags had been banned, showed a big jump (46%) in this potentially fatal infection, while the data from surrounding areas, showed no change. According to the study, there’s even a death toll: “Our results suggest that the San Francisco ban led to, conservatively, 5.4 annual additional deaths.”
Sounds like a sweet gotcha moment for opponents of nanny-state intervention, does it not? Laws intended to enforce politically correct behaviour actually kill people!
It makes sense, too. Disposable plastic bags are sanitary. Reusable bags hardly ever get washed, and too often carry meat, dairy and vegetables in the same bag, causing cross-contamination.
As if to underscore the validity of the results, the reaction from supporters of the plastic bag ban has been indirect, attacking the researchers, the lack of peer review, or otherwise avoiding the paper’s substantive claims. If opponents have to attack the source or their motives, it might well indicate that they cannot challenge the facts. However, while the credentials of researchers or publication in a reputable journal is important, they do not trump the substance of a study.
So, do plastic bag bans kill people? Not so fast.
As it turns out, there are a few reasons to question this study. It was a very isolated study, which coincided with a rise of a particular strain of e.coli in various other locations, where the incidence was not correlated with reusable bag use. As Klick rightly told the San Francisco Chronicle at the time, he “rule out the possibility that there was something peculiar that happened in San Francisco.”
Moreover, the study did show a correlation between reusable bag use and e.coli cases, but it did not establish a causal link. It did not test whether the e.coli sufferers were also users of reusable bags they failed to wash.
So, not so deadly after all? Perhaps not. Besides, arguments for supporting a ban on plastic bags have not just disappeared. They are dangerous to animals, both on land and, if they make it through the storm-water or sewerage system, at sea. They are not very biodegradable, and can clog drains and waste treatment facilities.
These seem like compelling arguments, but the other side of the debate has arguments that are at least as compelling. Besides for aversion to coercive nanny-statism, the SF Chronicle lists a few: “Single-use bags are not the problem. In 2011, they represented 0.13% of California’s total waste stream. According to the British Environment Agency, consumers have to reuse a cloth bag 131 times to present lesser global warming impact than conventional bags. When consumers wash reusable cloth bags [it] undermines the environmental benefits.”
Whatever the arguments, the study about e.coli infections is not a good one. It merely raises a question, and points towards a need for further research. Relying on it would suggest a clear case of confirmation bias – that is, cherry-picking research that confirms your preconceived opinions.
There are many similar examples in the field that has occupied a great deal of my attention these last two years, namely shale gas drilling.
A study by Theo Colborn et al, for a lobby group known as The Endocrine Disruption Exchange, finds some airborne pollution near shale gas drilling sites. That this exists is, of course, not surprising. There is always at least some methane leakage, as well as venting of associated gases. It seems likely that the process of drilling, hydraulic fracturing, and the handling of waste water could add to the airborne chemical load. Trucks and pumps also aren’t entirely non-polluting.
The question is, does one need to fear the air quality near shale gas wells?
The paper notes that contaminant levels increased at the time of drilling, but did not increase during hydraulic fracturing. Therefore, “fracking” as such cannot be to blame. It notes that most contaminants were detected only infrequently, and at low levels. It also notes that “the chemicals reported in this exploratory study cannot … be causally connected to natural gas operations,” though it seems reasonable to presume an association, given the otherwise rural setting of the study.
It adds: “A literature search of the health effects of the [chemicals] revealed that many had multiple health effects, including 30 that affect the endocrine system, which is susceptible to chemical impacts at very low concentrations, far less than government safety standards.”
This raises the non-trivial question of why “government safety standards” do not recognise the supposed endocrine disruption dangers of these chemicals and set limits accordingly.
One hint might be the observation that some of the chemicals were found, “at concentrations greater than those at which prenatally exposed children in urban studies had lower developmental and IQ scores.”
This sort of comparison raises alarm bells, my doctor friends tell me. Sure, endocrine disruptors exist, but one ought to be aware that studies have different levels of evidence, ranging from merely anecdotal evidence, from which one can only conclude that something is possible, to correlative but not causative evidence, that is a little stronger, to controlled clinical research studies that offer the most significant findings.
If you’re comparing rural air quality with an urban prenatal study, for example, there are dozens of factors that can make the comparison invalid. The first issue to look at would be possible confounding factors. In an urban environment, it is extremely hard to isolate potential other causes of particular symptoms, especially when those “symptoms” are as hard to measure objectively as IQ. Others include the size and randomness of the sample.
The same problems can be found in carcinogen studies. Carcinogens are all around us. Whether they ultimately cause cancer is highly dependent on concentrations, individual physical characteristics, and even pure chance. Smoking is one of the few fields in which enough subjects have been studied for long enough to reach conclusive results: smoking can cause cancer, and alarmingly often does.
However, few field studies, especially in the field of endocrine disruption, have the sample sizes and rigorous controls necessary to reach conclusive results. Most are weak, or even entirely speculative. They may be scientifically interesting, but useless for policy-making. That, in turn, may explain why regulators have not set new, more rigorous exposure limits. The science simply isn’t there.
The Colborn study may (or may not) offer some valuable data, but it does not, on its own, say very much that is conclusive. It certainly is far from enough to overturn the much more significant observation that electricity generated from natural gas reduces carbon dioxide emissions by 50%, and almost completely eliminates most other pollutants – like sulphur, nitrous oxides, mercury and particulates.
Tables that mark “health effects” of a given chemical with an X, as this study includes, tell us very little. Is the effect rare or common? Does concentration matter? What are the chances of human exposure, and in what doses? Does atmospheric dispersal and natural breakdown via chemical reactions reduce the risks?
The Colborn study offers an interesting set of data, that might prove of some value in a broader context to direct future research into risks that are not commonly, or adequately, assessed.
To be fair, the study does admit to several serious shortcomings and limitations, but such caveats seldom turn up in the abstract, or the subsequent scary media headlines.
A study by Oswald and Bamberger, on the veterinary risks posed by chemical spills during drilling operations was dismissed by Ian Rae, a professor at the University of Melbourne and advisor to the United Nations Environment Programme on chemicals in the environment, with these scathing words: “It certainly does not qualify as a scientific paper but is, rather, an advocacy piece.”
The infamous Cornell study by Robert Howarth and Anthony Ingraffea on how methane leaks make gas worse than coal as a fuel for generating electricity was downplayed as based on low-quality data even by its own authors. It was denounced by many scientists as seriously flawed in its assumptions and conclusions. Even the environmental organisation that part-funded it, the National Resources Defence Council, cautioned against using it for policy analysis. Another, the Clean Air Task Force, which opposes all fossil fuels, including natural gas, declared: “This paper is selective in its use of some very questionable data and too readily ignores or dismisses available data that would change its conclusions.”
A pair of papers from Duke University researchers Osborn and Vengosh about methane contamination in water wells near gas operations, show correlation, but no causation. The result of their first study was much worse than their second, which raises questions about the validity of the findings, though the second does show a stronger statistical link. Another study by the same authors points to natural migration pathways, instead, and neither of the water well studies found any of the tell-tale chemicals that would prove a causal link between shale gas drilling and water well contamination. Both propose further research.
Locally, a report by the late Tony Twine, of Econometrix, commissioned by Shell, which examined the potential economic benefits of a local shale gas industry, suffers many of the same problems. It is based on assumptions that are hard to justify without actual exploration. It uses a “Keynesian multiplier” method that, as advocates of stimulus funding have discovered recently, is highly questionable. It is true that its results remain significant at only a fraction of the findings, but citing it as evidence of an inevitable shale gas boom would be to fall into the same cherry-picking sensationalism trap that those opposed to shale gas development routinely fall.
Some people might say that science should be left to scientists (and economics to economists). They will fight it out among themselves in the pages of refereed academic journals. However, in the real world, laypeople like journalists frequently report on important findings, and government officials routinely face pressure backed up by selected academic papers. Policy makers and the media wade through a minefield when it comes to scientific evidence. Healthy skepticism and a sharp eye for issues that affect the quality of the research and the validity of policy conclusions drawn from it, are as rare as they are valuable.
Relying on studies just because they happen to support your point of view is dangerous. If they are easily discredited, or don’t say as much, as strongly, as you say they do, you only open yourself to charges of gullibility and partisan cherry-picking. Best to avoid those if you want to make a convincing argument, especially when writing to policy-makers.
That said, banning things is rarely a good idea, so I’ll certainly keep an eye out for more conclusive evidence that opposing plastic bags (or, indeed, shale gas drilling) causes more unintended harm than intended good. DM
Support DAILY MAVERICK & get FREE UBER vouchers every month
An increasingly rare commodity, quality independent journalism costs money, though not nearly as much as its absence can cost global community. No country can live and prosper without truth - that's why it matters.
Every Daily Maverick article and every Scorpio exposé is proof of our dedication to this unshakeable mission. Investing in our news media is by far the most effective investment into South Africa's future.
You can support Independent and Investigative journalism by joining Maverick Insider. If you contribute R150 or more per month you will receive R100 back in UBER vouchers. EVERY MONTH until October 2019.
So, if you'd like to help and do something meaningful for yourself and your country, then sign up to become a Maverick Insider. Together we can Defend Truth.
A Twitter analysis of Justin Bieber's account found that around half of his followers are fake accounts.