Effectiveness of BNT162b2 and mRNA-1273 covid-19 vaccines against symptomatic SARS-CoV-2 infection and severe covid-19 outcomes in Ontario, Canada: test negative design study

1Understanding how efficacy estimates from clinical trials of covid-19 vaccines translate into effectiveness estimates in the real world is crucial, given differences in populations, dosing intervals, and emerging variants. Because of constraints on the supply of covid-19 vaccines, Canada’s National Advisory Committee on Immunization recommended extending the interval between doses to a maximum of 16 weeks. With constraints on vaccine supply globally, determining the effectiveness of these vaccines after a single dose versus two doses is important for informing policy for many countries.

2We applied the test negative design to linked, population based health databases in Ontario, Canada (population 15 million) to evaluate vaccine effectiveness against symptomatic SARS-CoV-2 infection and severe outcomes (hospital admission or death) associated with SARS-CoV-2 infection for two mRNA vaccines: BNT162b2 (Pfizer-BioNTech) and mRNA-1273 (Moderna).

3We conducted a test negative design study among community dwelling Ontarians who had symptoms consistent with covid-19. The test negative design is comparable to a nested case-control design, with individuals who have symptoms and are tested for the presence of a pathogen of interest serving as the nesting cohort. All Ontarians aged ≥16 years, eligible for provincial health insurance, not living in long term care, and who were tested for SARS-CoV-2 between 14 December 2020 and 19 April 2021 were eligible for inclusion. We excluded those who tested positive for SARS-CoV-2 before 14 December 2020 and recipients of the ChAdOx1 (Oxford-AstraZeneca) vaccine. We restricted the analysis to individuals who had at least one relevant covid-19 symptom (based on self-report or observation, such as measured temperature), at the time of testing, which was collected on the SARS-CoV-2 test requisition form (see supplementary methods).

4Using unique encoded identifiers, we linked data from provincial SARS-CoV-2 laboratory testing, covid-19 vaccination, and health administrative datasets and analysed them at ICES (formerly the Institute for Clinical Evaluative Sciences).

5Our first primary outcome was symptomatic SARS-CoV-2 infection, ascertained by real time reverse transcription polymerase chain reaction (RT-PCR) tests on respiratory specimens, including samples from the nasopharynx (most common), nose, throat, saliva, and turbinates. Using data from the Ontario Laboratories Information System, which captured 91.8% (n=258 207) of all provincially reported cases of laboratory confirmed covid-19 (n=281 261) during the study period, people who tested positive were considered as cases and those who tested negative were considered as controls. Since the dates for symptom onset were inconsistently reported in the Ontario Laboratories Information System, we used the date of specimen collection as the index date. For cases with multiple positive test results, we used the date of the first positive test result. For controls with multiple negative test results, we used the date of a randomly selected negative test result as the index date.

6We obtained information on variants and mutations from the Public Health Case and Contact Management system, which contains information on the clinical course of cases and the results of screening tests for N501Y and E484K mutations and whole genome sequencing results that identify specific variant of concern lineages (alpha (B.1.1.7), beta (B.1.351), gamma (P.1)). All RT-PCR positive specimens with cycle threshold values ≤35 were tested for the N501Y mutation (starting 3 February 2021) and the E484K mutation (starting 22 March 2021). We considered samples with positive N501Y and negative E484K mutations as alpha variants, and samples with positive N501Y and E484K mutations as beta or gamma variants. We combined the latter two lineages for our analysis because there were small numbers of cases identified using whole genome sequencing.

7Our second primary outcome was severe disease associated with symptomatic SARS-CoV-2 infection, defined as either hospital admission or death with a recent positive test result, using the earliest of the specimen collection date or the hospital admission or death date as the index date. We identified these outcomes using the Case and Contact Management system (for both hospital admissions and deaths), the Canadian Institute for Health Information’s Discharge Abstract Database (for hospital admissions), and the Ontario Registered Persons Database (for deaths). For hospital admissions identified using the Discharge Abstract Database, a positive test result must have occurred within 14 days before or three days after admission. For deaths identified using the Registered Persons Database, a positive test result must have occurred within 30 days before death or within seven days post mortem. We used the same control group as for the first primary outcome analysis (ie, individuals with symptoms who tested negative for SARS-CoV-2).

8BNT162b2 became available in Ontario on 14 December 2020 and mRNA-1273 on 28 December 2020. The initial vaccination phase prioritised high risk populations such as older adults living in communal settings, healthcare workers (including non-patient facing staff working in healthcare institutions), adults living in Indigenous communities, and adults aged ≥80 years. Ontario had initially followed the manufacturers’ recommended dosing schedules (ie, a 21 day interval for BNT162b2 and a 28 day interval for mRNA-1273), but because of disruptions to vaccine supply in late January 2021, the interval was extended to 35-42 days for everyone except older adults living in communal settings, and Indigenous people. In early March, Ontario adopted the National Advisory Committee on Immunization’s recommendation to delay administering the second dose by up to 16 weeks for most individuals. Eligibility expanded over time, taking into account both age (ie, graduated expansion by decreasing age) and other high risk populations, such as people with certain health conditions and their care givers, certain essential frontline workers, and those aged ≥18 years living or working in communities with a high incidence of covid-19 (ie, those disproportionately affected by covid-19 and where transmission was still high). Adherence to these eligibility criteria varied across regions. As of 19 April 2021, 28% of adults in Ontario had received at least one dose of a covid-19 vaccine. Comprehensive documentation of all covid-19 vaccination events in Ontario, including product, date of being administered, and dose number, is recorded in real time into COVaxON, a centralised covid-19 vaccine information system. We used the COVaxON file containing events up to 25 April 2021 for these analyses, which likely had records of all vaccinations delivered by 19 April 2021.

9We obtained age, sex, and postal code of residence as of 14 December 2020 from the Registered Persons Database. We obtained the number of RT-PCR tests for each participant during the three months before 14 December 2020 from the Ontario Laboratories Information System to use as a proxy for highly tested individuals at increased risk of exposure to SARS-CoV-2 infection (eg, healthcare workers and care givers of long term care residents, who must also undergo serial SARS-CoV-2 testing). To capture temporal changes in viral activity and regional vaccine roll-out, we grouped dates of testing into two week periods. We determined the presence of comorbidities that increase the risk of severe covid-19, identified from various databases using validated algorithms and commonly accepted diagnostic codes, which have been described elsewhere. Receipt of influenza vaccination (a proxy for health behaviours) was ascertained during the 2019/20 or 2020/21 influenza season, or both, using physician and pharmacist billing claims in the Ontario Health Insurance Plan and Ontario Drug Benefit databases, respectively. We determined the public health unit of residence using the postal code and Statistics Canada Postal Code Conversion File Plus (version 7B) and grouped the units into larger regions. From 2016 census data we obtained information at the ecological level of dissemination area on four important social determinants of health (median household income, proportion of the working population employed as non-health essential workers (those unable to work from home), average number of people in each dwelling, and proportion of the population who self-identified as a visible minority). Dissemination areas generally contain 400-700 people. Supplementary eTable 1 provides details of these covariates.

10To compare characteristics between test positive cases and test negative controls and between vaccinated and unvaccinated individuals, we conducted descriptive analyses and calculated standardised differences.

11We used multivariable logistic regression models to estimate the odds ratio, comparing the odds of vaccination between test positive cases and test negative controls (with unvaccinated people as reference group). We estimated unadjusted and adjusted odds ratios accounting for all listed covariates. These covariates were selected a priori based on their known associations with SARS-CoV-2 infection or severity and receipt of a covid-19 vaccine and were assessed as potential confounders (supplementary eTable 2). Vaccine effectiveness was calculated using the formula: vaccine effectiveness=(1−odds ratio)×100%. We assumed those without information on exposures, outcomes, or covariates in ICES’ data holdings to not have the exposure, outcome, or covariate, and they were categorised as such within the analyses.

12For the primary analysis, we estimated overall vaccine effectiveness (for both mRNA vaccines combined) for those who received only one dose by their index date and those who received two doses by their index date. We considered index dates within varying intervals after vaccination.

13Vaccine effectiveness was also estimated ≥14 days after the first dose (among those who only received one dose) and ≥7 days after the second dose, stratified by vaccine product (BNT162b2 or mRNA-1273), age group (16-39, 40-69, and ≥70 years), sex, presence of any comorbidity, epidemic wave (index dates 14 December 2020 to 7 February 2021, representing wave 2 in Ontario; 8 February 2021 to 21 March 2021, representing the period between wave 2 and wave 3; and 22 March 2021 to 19 April 2021, representing wave 3), and variant (earlier variant versus alpha versus beta or gamma). We also estimated vaccine effectiveness by varying intervals after vaccination, stratified by age group.

14We repeated these analyses for severe outcomes, with adjustments to the intervals after vaccination due to reduced sample sizes. For example, we evaluated vaccine effectiveness for the entire period (≥0 days) after receipt of the second dose.

15Finally, to assess whether systematic differences between vaccinated and unvaccinated individuals were adequately controlled for in the main analyses, in a sensitivity analysis we assessed vaccine effectiveness against symptomatic infection and severe outcomes by varying intervals among only those who were vaccinated (treating people vaccinated 0-13 days before the test as reference group).

16All analyses were conducted using SAS version 9.4 (SAS Institute, Cary, NC). Tests were two sided, with P<0.05 considered as significant.

17Although study participants contributed in important ways to this research, it was not feasible to involve them in the design, conduct, reporting, or dissemination plans of our research. We did not involve members of the public in this research owing to resource and time constraints.

18Overall, 2 171 449 unique individuals were tested for SARS-CoV-2 from 14 December 2020 to 19 April 2021. After excluding those who had SARS-CoV-2 infection before the study period and those who had received the ChAdOx1 vaccine, 60.5% of those remaining did not have symptoms consistent with covid-19 or had no information on symptoms recorded in the Ontario Laboratories Information System, 24.4% were recorded as asymptomatic, and 15.1% had symptoms consistent with covid-19 recorded at the time of testing (fig 1). Grouped together, those with covid-19-like symptoms and those deemed asymptomatic had similar characteristics to the remaining individuals, except for covid-19 vaccine uptake, public health unit region, and number of previous SARS-CoV-2 tests (supplementary eTable 3).

19Of the 324 033 people with symptoms who were tested, 53 270 (16.4%) tested positive for SARS-CoV-2, 42 567 (79.9%) had information available on tests for variants, 21 272 (6.6%) had received at least one dose of mRNA vaccine, and 4894 (1.5%) had received two doses (table 1). Among test positive cases, 2479 (4.7%) had a severe outcome, of whom 2035 were admitted to hospital and 444 died. Most cases admitted to hospital tested positive before or on the admission date (1728, 84.9%) and nearly all tested positive before death. Test positive cases were more likely to be male, to reside in Peel region or Toronto, and to have had no SARS-CoV-2 tests during the three months before the vaccination programme, less likely to have received an influenza vaccine, and more likely to reside in neighbourhoods with lower income, more people in each dwelling, and greater proportions of essential workers and people from visible minorities (table 1). Vaccinated people were older, less likely to be male, and more likely to have had multiple SARS-CoV-2 tests during the three months before the vaccination programme, a comorbidity, and received an influenza vaccine. Compared with recipients of the mRNA-1273 vaccine, recipients of the BNT162b2 vaccine were younger, more likely to be female, and less likely to have a comorbidity (supplementary eTable 4). The distribution of these vaccine products also differed by public health unit regions. Most participants (77% for BNT162b2, 76% for mRNA-1273) had received only one dose by the index date. Supplementary eFigure 1 presents the distribution of vaccine product over the study period.

20Against symptomatic infection, adjusted vaccine effectiveness observed ≥14 days after only one dose was 60% (95% confidence interval 57% to 64%). This increased from 48% (41% to 54%) at 14-20 days to a plateau of 71% (63% to 78%) at 35-41 days (fig 2, supplementary eTable 5). A 16% increase in risk of symptomatic infection was observed 7-13 days after one dose (vaccine effectiveness −16%, −26% to −6%), but no increase was observed 0-6 days after one dose. Vaccine effectiveness observed ≥7 days after two doses was 91% (89% to 93%). Against severe outcomes of hospital admission or death, vaccine effectiveness observed ≥14 days after one dose was 70% (60% to 77%), increasing from 62% (44% to 75%) at 14-20 days to 91% (73% to 97%) at ≥35 days, whereas vaccine effectiveness observed ≥7 days after two doses was 98% (88% to 100%) (fig 3, supplementary eTable 5).

21In subgroup analyses of vaccine effectiveness against symptomatic infection, higher effectiveness was observed ≥14 days after only one dose of mRNA-1273 versus one dose of BNT162b2 (which was consistent across all age groups), for younger people versus adults aged ≥70 years, for those with no comorbidities versus those with comorbidities, and against the earlier variant and alpha variant versus beta or gamma variant (although 95% confidence intervals for vaccine effectiveness estimates for variants overlapped) (fig 4, supplementary eTable 6). Vaccine effectiveness estimates observed ≥7 days after two doses were, however, high (all ≥88%) and comparable across all subgroups, including against variants with the E484K mutation. Against severe outcomes, higher vaccine effectiveness was observed ≥14 days after one dose for those aged 16-39 years, but effectiveness estimates after two doses were mostly similar across subgroups (fig 4, supplementary eTable 7).

22Among adults aged ≥70 years, vaccine effectiveness against symptomatic infection after one dose was observed to be 64% (46% to 76%) at 28-34 days and 85% (38% to 97%) at 42-48 days, whereas comparable effectiveness estimates were achieved sooner after one dose for younger people (fig 5, supplementary eTable 8). Furthermore, vaccine effectiveness against severe outcomes was observed to be similar at ≥35 days after one dose (93%, 71% to 98%) as after two doses (97%, 86% to 99%).

23In the sensitivity analysis restricted to vaccinated people with those vaccinated 0-13 days before the index date serving as the reference group, vaccine effectiveness estimates against symptomatic infection were observed to be similar to those of the main analyses (supplementary eTable 5). For hospital admissions and deaths, however, the estimates differed for earlier vaccination intervals.

24Using the test negative study design, which mitigates selection bias from differences in health seeking behaviour between vaccinated and unvaccinated people, we estimated high (>90%) vaccine effectiveness of mRNA vaccines BNT162b2 and mRNA-1273 against symptomatic SARS-CoV-2 infection with full vaccination (≥7 days after a second dose), and moderate (about 50-70%) vaccine effectiveness with partial vaccination (≥14 days after only one dose). Estimates for both full and partial vaccination were about 10 percentage points higher against hospital admission or death than against symptomatic infection. Vaccine effectiveness generally was observed to increase over time after one dose; however, we also observed a slightly increased risk of symptomatic infection on days 7-13 after one dose, compared with no vaccination. In subgroup analyses, we observed lower vaccine effectiveness against symptomatic infection in adults aged ≥70 years and those with comorbidities, but a higher effectiveness after one dose of mRNA-1273 than after one dose of BNT162b2. Vaccine effectiveness was, however, observed to be consistently high across subgroups for fully vaccinated people, and also for older adults after longer intervals from a first dose.

25Our findings for fully vaccinated people are comparable with efficacy estimates from clinical trials and other real world effectiveness estimates reported in a range of settings. Existing evidence for effectiveness estimates of one dose of mRNA vaccines from observational studies is heterogeneous, with estimates for symptomatic infection ranging from 57% (95% confidence interval 50% to 63%) to 72% (58% to 86%) and post hoc calculations from efficacy trials about 90%. Similar heterogeneity has been found among one dose effectiveness estimates in older adults, with estimates generally lower for older adults after one dose, and increasing over time. In our analysis we observed an effectiveness against symptomatic infection of 63% (95% confidence interval 40% to 72%) ≥49 days after only one dose, in keeping with the findings of several other studies. In addition, we found that one dose of mRNA-1273 was associated with significantly higher effectiveness against symptomatic infection than one dose of BNT162b2. Differences in characteristics between recipients of the two vaccines might explain this finding, but similar results were also found in another Canadian province. Findings were, however, inconsistent in other studies that compared vaccine effectiveness between products after one dose; some found a trend towards higher effectiveness against infection using mRNA-1273, whereas others found no difference. However, the populations in each of these studies were more homogeneous than ours (eg, adults aged ≤40 years, healthcare workers, veterans). Our analysis also reflects extant evidence that vaccine effectiveness against symptomatic infection and covid-19 associated hospital admissions increases to high levels after a second dose, even in older adults. Lastly, our finding that two doses of mRNA vaccines was not associated with appreciable vaccine escape by lineage alpha or variants with the E484K mutation (beta and gamma) is notable.

26In our study, we observed an increased risk of infection 7-13 days after vaccination. Other studies also found an increased risk of SARS-CoV-2 infection up to 14 days after one dose. This could be due to an increase in exposures to SARS-CoV-2 after vaccination. Individuals might assume that they are protected against infection immediately after vaccination and engage in higher risk behaviours before a sufficient immune response has developed. Indeed, about 20% of the US public believe that protection is conferred either immediately or 1-2 weeks after the first vaccine dose. Future studies should examine the potential role of behavioural changes after the first dose of covid-19 vaccines. This finding could also be due a higher baseline risk of infection among those who were initially prioritised to receive the vaccine, which might not have been adequately controlled for in our models. A vaccine effectiveness estimate of close to 0% at 0-6 days after one dose, however, provides a level of validation that we had accounted for the differences between vaccinated and unvaccinated people.

27This study has some limitations. Firstly, our study sample was limited to those with symptoms of covid-19 recorded in the Ontario Laboratories Information System, which decreased our potential sample size considerably, from 2 171 449 people who were tested for SARS-CoV-2 to 324 033 who had relevant covid-19 symptoms recorded in the information repository. Not all laboratories in Ontario currently have the information technology infrastructure to submit information on symptoms (or documentation of asymptomatic testing) recorded on the SARS-CoV-2 laboratory requisition form into the Ontario Laboratories Information System. Thus, the generalisability of our findings to the broader population is uncertain and we could not estimate vaccine effectiveness against asymptomatic infection. However, the per cent positivity of our study sample (53 270/324 033=16.4%) did not differ much from that observed in Ontario during the study period (281 261/2 171 449=13.0%), and we would expect positivity to be higher for people with symptoms than those without symptoms. We also acknowledge that those with no information on symptoms recorded in the Ontario Laboratories Information System might have had symptoms at the time of testing, and those recorded as asymptomatic might have subsequently developed symptoms. In addition, covid-19 vaccination status might also be collected on the laboratory requisition form. This could bias the true vaccine effectiveness estimate, depending on whether symptoms were more likely to be documented on requisition forms for vaccinated people who ultimately test positive for SARS-CoV-2 (this would bias vaccine effectiveness towards the null) or less likely to be recorded (this would bias vaccine effectiveness away from the null). To minimise this selection bias, traditional test negative design studies collect vaccination status among all people with symptoms consistent with the pathogen under study. However, the congruence of our findings for fully vaccinated people with extant studies provides some reassurance that any underestimation or overestimation of vaccine effectiveness is likely to be small. Secondly, because the date of symptom onset is largely unavailable in the Ontario Laboratories Information System, and the Case and Contact Management system only has information on people who test positive, we used specimen collection date as the index date. This might have led to some vaccinated people being classified into an incorrect dose-to-index interval because symptom onset would have occurred several days before being tested. The impact of earlier intervals on vaccine effectiveness estimates if using the specimen collection date depends on the test results for vaccinated people (eg, vaccine effectiveness for earlier intervals would be overestimated if there were more vaccinated test positive cases and underestimated if there were more vaccinated test negative controls) or whether the lag between symptom onset and specimen collection dates resulted in misclassification of vaccinated people (false negatives), which would overestimate vaccine effectiveness for earlier intervals. Furthermore, we could not limit the study population to those tested within 10 days of symptom onset, a commonly used inclusion criterion for test negative studies. Prolonging the interval between symptom onset and testing increases the likelihood of false negative cases, which lowers vaccine effectiveness estimates. However, 89% of cases with dates for both symptom onset and specimen collection documented in the Case and Contact Management system (not the source of symptom data for this study) were tested within 10 days of symptom onset. Thirdly, our results might have been affected by outcome misclassification of severe outcomes due to unlinked case records and incomplete capture of severe outcomes in the Case and Contact Management system, and delays in identifying hospital admissions in the Discharge Abstract Database (which depends on people being discharged) and deaths in the Registered Persons Database. The direction of bias to vaccine effectiveness estimates depends on whether data completeness and lags are differential between vaccinated and unvaccinated test positive cases. For example, if vaccination status ascertained during the case management process influenced the degree of data collection (eg, if more complete among vaccinated cases), vaccine effectiveness would be biased towards the null, or if unvaccinated cases have a prolonged hospital admission because of more severe course of illness, their hospital record would not be available for analysis and vaccine effectiveness would be biased towards the null. Fourthly, some of our covariates might be subject to measurement error. We used frequency of previous SARS-CoV-2 tests as a proxy to identify those at higher risk of exposure (and increased likelihood to be targeted for early vaccination). However, we did not include point of care tests because they are incompletely captured in the Ontario Laboratories Information System. Furthermore, since access to testing is variable, we might not have adequately controlled for this situation. Finally, we might not have adequately accounted for confounding bias with the covariates that were available in the study databases, especially against hospital admissions and deaths.

28Our findings suggest that older people and those with comorbidities might benefit from risk based recommendations to minimise delays before a second dose of mRNA covid-19 vaccine. However, increasing protection against severe outcomes—arguably the more important outcomes—with increasing time after a first dose provides support for delaying the second dose in settings with constraints on vaccine supply. Mathematical modelling could be done to show how, particularly for jurisdictions with limited vaccine supply, vaccines should be distributed to maximise the protection of populations (eg, the relative benefits of providing second doses earlier to older populations versus providing more first doses to younger populations who respond better to one dose and thereby leading to more rapid achievement of herd immunity by maximising coverage with one dose). Since vaccine effectiveness against symptomatic infection after one dose is only moderate, and among older adults appears to be modest even at 14-20 days, people need to be informed that besides the absence of benefit during the first two weeks (and likely longer for older adults) after one dose of a mRNA covid-19 vaccine, they should continue to adhere to recommended public health measures, such as wearing a mask, physical distancing, and avoidance of social gatherings.

29Clinical trials and real world effectiveness studies have shown that mRNA covid-19 vaccines are protective against symptomatic SARS-CoV-2 infections and associated severe outcomes such as hospital admission and death

30Greater protection is conferred after two doses administered at least three weeks apart

31Protection against symptomatic infection and severe outcomes increases gradually after one dose of an mRNA covid-19 vaccine

32Although lower vaccine effectiveness was observed in older adults shortly after one dose, levels of protection similar to those of younger people were observed after longer intervals

33These findings support delaying second doses when vaccine supply is limited