A new study claims to have found a link between flame-retardant chemicals in furniture and textiles and a range of behavioural problems in children. Instead of jumping on the hysteria train with a ban hammer, join me for a case study in bad science.
“Firepreventing chemicals in furniture may be turning children violent and hyperactive,” declares the UK’s Mirror tabloid. Other news outlets around the world, including science publications, ran similar alarming headlines.
These headlines are based on a new study by Shannon Lipscomb et al published in the journal Environmental Health, which claimed to study social behaviours in preschool children as a result of exposure to two classes of flame retardant chemicals. The study was accompanied by a press release from Oregon State University, headlined, “Flame retardant chemicals may affect social behaviour in young children”.
Flame retardants are in common use worldwide, in many fabrics, plastics and furniture. Many governments have required their use in consumer products for many decades. Their purpose is fairly obvious: to prevent or slow down fires. How do we sleep while our beds are burning?
As with other industrial and commercial chemicals, the toxicity of flame retardant chemicals has long been a popular concern. If you want the alarmist version of the chemical industry’s conspiracy to poison you instead of burning you alive, visit Quack Mercola, where everything is toxic and the facts don’t matter.
There are legitimate reasons to think flame retardant chemicals might be harmful. But the most-cited academic studies of the health effects of these chemicals are limited to mice. There have been a few studies in humans, but they are limited in scope, based on small sample sizes, involve weak correlations, and reach uncertain or even contradictory conclusions.
The study by Lipscomb et al is an excellent example of why a single study that claims to find an association should not be a reason to panic about your children’s health, or demand that government ban chemical substances.
In the press release, one of the co-authors, environmental epidemiologist and associate professor Molly Kile, claims that the study showed a “significant relationship between social behaviours among children and their exposure to widely used flame retardants”.
She elaborated: “When we analysed behaviour assessments and exposure levels, we observed that the children who had more exposure to certain types of the flame retardant were more likely to exhibit externalising behaviours such as aggression, defiance, hyperactivity, inattention and bullying.”
But even if you use the statistical definition of “significant”, which is “better than chance”, it isn’t clear that the results are significant at all. The study sampled only 69 children. It tested for 41 different chemicals in two broad classes. It asked preschool teachers to rate behaviour on 11 different subscales (communication, co-operation, assertion, responsibility, empathy, engagement, self-control, externalising, bullying, hyperactivity/inattention, and internalising). It then used multiple different models – including non-linear exposure-response models – to fish for relationships.
The more hypotheses you test against a data set, the higher the likelihood of finding a false positive correlation purely by chance. This is called the multiple testing problem. There are ways to control for this, but the authors give no indication that they did so.
On the contrary. When you have two chemical compound classes, and test them against 11 behaviour indicators, you get 22 data points per test subject. If your idea of statistical significance is a p-value of less than 0.05, you’d expect just over one positive correlation, just by chance.
So let’s have a look at the positive correlations the authors did find: one of the chemical classes was associated with less assertive behaviour, while the other was associated with less responsible behaviour and more aggression (externalising). For the last one, the statistical significance of the finding was marginal, so there really were only two significant positive findings. For one of those, the exposure-response curve turned out to be non-linear and counterintuitive. So, really, the study found just over one positive correlation, which is exactly what you’d expect from chance.
The study does not bother to explain how exposure to flame retardant chemicals causes the suspected behavioural problems. That makes this paper appear to be a classical case of p-hacking: just test enough hypotheses until you have one or two that your data appears to confirm, even though that can happen purely by chance. (On a larger scale, this practice is known as data fishing. Wikipedia has a great example of such a correlation, showing a clear correlation between the number of letters in the winning word of the National Spelling Bee, and the number of people in the US killed by venomous spiders. Absent a causal link, such as a surprising surfeit of venomous spiders at locations where spelling bees are held, this is merely coincidence.)
There are other problems with the Lipscomb study. It does not report what other environmental exposure might have confounded the results, because the researchers did not test for other chemicals. It is entirely possible that children with higher flame retardant exposure were also exposed to other toxic substances, such as heavy metals or common household cleaners, which in turn could have caused the supposed effects.
It is also hard to rule out subjective errors in their findings. They neatly express the various social behaviour indicators in numbers, but you can’t actually measure behaviour in an exact fashion. Different teachers, for example, would probably rate children differently, depending on their own subjective knowledge and experience. There weren’t enough test subjects in the sample to smooth over such biases, so this would have added a degree of randomness to the data that further reduces the significance of the results.
The Lipscomb findings partially contradict a 2009 study of 62 children conducted in the Netherlands, which found weak correlations between flame retardant chemical exposure and “worse fine manipulative abilities, worse attention, better co-ordination, better visual perception, and better behaviour”.
That study also did not explain causation, and in particular did not explain why some correlations were positive and some were negative. In fact, if you read it, you’ll find it suffers from many of the same problems as the Lipscomb study.
Studies that go fishing for statistically significant results in multiple tests will inevitably find a few. They might be indicative of a need for further research, using bigger sample sizes, better control groups, and more focused methodology. But that’s it. They prove nothing.
But that’s not what the authors conclude: “We observed a cross-sectional association between children’s exposure to flame retardant compounds and teacher-rated social behaviours among preschool-aged children. Children with higher flame retardant exposures exhibited poorer social skills in three domains that play an important role in a child’s ability to succeed academically and socially.”
That’s an even stronger statement than the “may” phrasing in the press release headline, and it is not a true summary of the study. It generalises from the one or two weak associations found among 22 hypotheses to suggest a broad association between social behaviours and skills in general and all flame retardant compounds.
As science, this stinks. As propaganda aimed at people who are paranoid about industrial chemicals, it will do nicely.
Besides the annoying fact that such paranoia is exploited by naturopathic quacks, bad science such as this causes very real harm. Weak studies that make big headlines are meat and drink to activists who propose bans or regulations upon the flimsiest of evidence. If the risk of toxic exposure is significant, it is reasonable to seek to protect consumers. However, when there is little evidence of danger, the consequence of over-regulation is to stifle free enterprise by increasing the regulatory burden, raising consumer prices, and ultimately making everyone less prosperous.
While wealthy elites may be able to afford to indulge their petty neuroses, an over-abundance of regulatory caution is harmful to the beleaguered middle class and poor people around the world.
If you’re really looking for explanations for why kids act up, look no further than a throwaway line deep in the Lipscomb study itself. It says children’s early adverse experiences, such as abuse, neglect, parent mental illness, or parental substance use, were a stronger predictor for the observed behaviours than fire retardant chemicals. This actually does have a causal explanation and is consistent with well-established psychological theories.
So if you’re concerned about your children’s well-being, by all means buy a non-flammable couch. Your kids will be just fine, unless you sit on it all day smoking weed and beating them. DM
Riding a Black Unicorn Down the Side of an Erupting Volcano While Drinking from a Chalice Filled with the Laughter of Small Children is the title of a dark cabaret album by 'Voltaire'