Dodgy data: COVID vaccine clinical trials show counting window issues, exaggerating efficacy

A hugely important set of articles has appeared in the prestigious Journal of Evaluation in Clinical Practice recently. The Fung, Jones, and Doshi paper outlines statistical biases, such as the case counting window bias, that likely lead to the COVID vaccines’ effectiveness being exaggerated in observational studies. The subsequent paper by Lataster then explained the situation is worse, as the case counting window bias is often accompanied by a definitional bias, and noted that this could exaggerate vaccine safety as well. Now Doshi and Fung return with a further paper. Source.

They state: “While both our commentary and Lataster’s critique focus on observational study designs, concerns about case counting windows also extend to the original phase 3 randomised trials of COVID-19 vaccines.” They found that case counting “only began once participants were 7 days (Pfizer) or 14 days (Moderna) post Dose 2, or approximately 4–6 weeks after Dose 1”. The obvious implication: “Decisions on when to initiate the case counting window affected calculations of vaccine efficacy. Because cases occurring in the 4–6 weeks between Dose 1 and the case counting window were excluded, reported vaccine efficacy against COVID-19 (the primary endpoint) at the time of Emergency Use Authorization was higher than what would have been calculated had all COVID-19 cases after Dose 1 been included, as in a conventional Intent-to-Treat analysis.” They also found that “different case counting windows” were used at different times, coincidentally (!?) yielding better results.

Okay then.