“I’ve never seen a trial where there were four interim analyses; that may be the Olympic record,” said Eric Topol, editor-in-chief of Medscape, a website offering clinical information for health-care professionals, and director of the Scripps Research Translational Institute in La Jolla, California. “It’s obvious why it is being done: so you can just keep looking at the data to try to win a race.”
A wide range of symptoms and severity makes the evaluation of Covid-19 vaccines tricky. The U.S. Food and Drug Administration has said that to be approved, vaccines should cut the number of symptomatic cases by half. Yet documents released by the drugmakers show each has its own approach to defining which symptoms count, and when to count them.
Big drug studies usually allow a panel of monitors to get an early peek at the data once or twice before the planned end. The panel can stop the trial early if a treatment is judged overwhelmingly effective — or alternatively, a total dud. Four early looks may give Pfizer an “easy route” to making sure it has results soon, said Marie-Paule Kieny, a former World Health Organization official who’s now a research director at the French health-science institute Inserm.
“It seems that there are different levels of stringency,” she said in an interview. “I wouldn’t say that Pfizer-BioNTech comes out as a star of stringency.”
Moderna, working with the U.S. National Institute of Allergy and Infectious Diseases, won’t dive in until 53 cases have occurred; its ultimate goal is to make a judgment at 151 diagnoses. Cancer powerhouse AstraZeneca, collaborating with the University of Oxford, will take its first look at 75 cases, and not again until the trial is complete with 150.
“All trials have set the bar quite low for what they test against,” said Rasmus Bech Hansen, Airfinity’s chief executive officer.
Pfizer’s trial was designed to evaluate its vaccine candidate “as fast as possible,” said Amy Rose, a spokeswoman, in an email. The company has worked with government scientists to develop best practices for testing and based its schedule for interim analyses on the vaccine’s “strong profile” in early human trials and animal tests, she said.
Moderna’s plan was agreed upon with U.S. regulators, and the company has been open about the numbers since before the trial began in late July, spokesman Ray Jordan said in an email. Case totals for interim analysis were based on probabilities of success and “were not selected based on timeframes,” he said.
AstraZeneca said its trials are conducted under regulatory oversight and its plans have evolved over time to make sure they produce robust information in a timely way. All the companies said their trials will continue beyond points such as a preliminary readout or potential authorization.
Pfizer says its study will likely yield conclusive results in October. None of the drugmakers will likely know whether their vaccines lower hospitalizations until February, according to Airfinity. But it’s the earlier assessments that have observers most concerned.
Treatments such as Gilead Sciences Inc.’s antiviral remdesivir and convalescent plasma — a soup of immune factors taken from the blood of recovered Covid-19 patients — have already been authorized on the basis of somewhat limited data.
But those are mainly given to extremely sick patients whose lives are in danger. Authorizing a vaccine, on the other hand, might lead to use in hundreds of millions of uninfected people. If that happens, the data ought to be pretty convincing.
“A small number of additional events occurring or not occurring can tip the balance of a trial as to whether the findings are valid or not,” said Jonathan Kimmelman, director of the Biomedical Ethics Unit at McGill University in Montreal. “If you want robust findings that are going to be generalizable, you’re going to want to accumulate a reasonable number of events.”
Some vaccines are approved after just a few cases appear in trials. Merck & Co.’s Ebola vaccine was approved last year based on a 3,500-patient study that identified 10 cases, according to its label. But that shot had perfect effectiveness and was designed to prevent a disease with a sky-high death rate.
Billions of healthy people around the world at low risk of life-threatening disease could get a coronavirus shot, raising the bar for safety. An authorization that’s based on early results could mean that doctors know very little about a vaccine, other than the bare-bones results. The issue is gaining attention as questions persist about the halt of AstraZeneca’s human tests in the U.S. following the appearance of side effects that Oxford said were unlikely to be linked to the vaccine.
If Pfizer’s vaccine were to receive emergency authorization based on results from October, for example, most patients would have had less than two months of follow-up. At that point, there will be little known about its duration of protection, and its impact on severe disease may not be clear.
Along with a positive test for the coronavirus, Moderna requires most patients to have two or more symptoms to count as a case for judging vaccine efficacy, unless they have a telltale marker like shortness of breath.
But Pfizer’s trial reaches deeper into patients with mildly symptomatic cases. It officially counts even a coronavirus-positive patient with fever alone, one of the disease’s most common manifestations, as a symptomatic case.
This may allow Pfizer to tally cases faster, but also could mean an early result based mostly on mild cases, according Airfinity’s Hansen.
“We should really have hospitalizations or deaths as endpoints because ultimately we want to prevent people from getting seriously ill,” he said.
The rush for results may make it harder to get clear answers about how well the vaccines work, said Topol, the Scripps scientist.
“We want to know this vaccine has strong efficacy,” he said. “And that means two things: that it works in the majority of people and that it works to prevent serious infections, not sore throats or muscle aches.”